2024-03-30T14:34:18.153 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T14:34:18.153 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} --archive /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207 --owner scheduled_teuthology@teuthology --name teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/orig.config.yaml 2024-03-30T14:34:18.184 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T14:34:18.276 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7630207' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) mon_bind_msgr2: false sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 5af0400050a123dc31f105425174f106e057276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8777 sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb sleep_before_teardown: 0 subset: 43/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 5af0400050a123dc31f105425174f106e057276a targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVqhWiaFRuPQmVUMJ9krG0HCU4jtyqrP3sL5xmlpkxyQzDhtnAtOPugxh6qmPPrimk3UnLRB2/rJn6khSFzfgA= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/zvpFpaQZcES9AAcTpvqAyiJ1CSm//iJurNQLeN7X0z0W/XK7ZGZaUVVslamF7yXjfMiSSjvzuOYsQFs5Bqjw= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T14:34:18.276 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa; will attempt to use it 2024-03-30T14:34:18.277 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_5af0400050a123dc31f105425174f106e057276a/qa/tasks 2024-03-30T14:34:18.277 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T14:34:18.279 INFO:teuthology.task.internal:Checking packages... 2024-03-30T14:34:18.302 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb' 2024-03-30T14:34:18.302 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T14:34:18.302 INFO:teuthology.packaging:ref: None 2024-03-30T14:34:18.302 INFO:teuthology.packaging:tag: None 2024-03-30T14:34:18.302 INFO:teuthology.packaging:branch: squid 2024-03-30T14:34:18.303 INFO:teuthology.packaging:sha1: 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:34:18.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-30T14:34:18.554 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1973.g5af04000 2024-03-30T14:34:18.555 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T14:34:18.562 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T14:34:18.562 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T14:34:18.600 INFO:teuthology.task.internal:Saving configuration 2024-03-30T14:34:18.612 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T14:34:18.625 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T14:34:18.650 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:29:15.400215', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVqhWiaFRuPQmVUMJ9krG0HCU4jtyqrP3sL5xmlpkxyQzDhtnAtOPugxh6qmPPrimk3UnLRB2/rJn6khSFzfgA='} 2024-03-30T14:34:18.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:29:15.401955', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/zvpFpaQZcES9AAcTpvqAyiJ1CSm//iJurNQLeN7X0z0W/XK7ZGZaUVVslamF7yXjfMiSSjvzuOYsQFs5Bqjw='} 2024-03-30T14:34:18.668 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T14:34:18.674 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-30T14:34:18.674 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-30T14:34:18.674 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T14:34:18.761 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe7952080d0>, signals=[15]) 2024-03-30T14:34:18.761 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T14:34:18.767 INFO:teuthology.task.internal:Opening connections... 2024-03-30T14:34:18.767 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T14:34:18.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:34:18.883 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-03-30T14:34:18.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:34:18.973 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T14:34:18.980 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-03-30T14:34:18.998 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-03-30T14:34:18.998 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-03-30T14:34:19.055 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T14:34:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T14:34:19.057 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-03-30T14:34:19.085 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-03-30T14:34:19.102 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-03-30T14:34:19.102 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-03-30T14:34:19.159 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T14:34:19.160 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T14:34:19.160 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-03-30T14:34:19.185 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T14:34:19.194 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T14:34:19.216 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T14:34:19.216 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-03-30T14:34:19.219 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-03-30T14:34:19.235 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T14:34:19.240 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T14:34:19.240 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-03-30T14:34:19.276 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-03-30T14:34:19.379 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T14:34:19.439 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T14:34:19.453 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T14:34:19.453 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T14:34:19.453 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T14:34:19.454 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T14:34:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:34:19.471 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-30T14:34:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:34:19.473 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-30T14:34:19.528 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-03-30T14:34:19.528 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-03-30T14:34:19.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-03-30T14:34:19.531 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-03-30T14:34:19.531 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-361.el9.x86_64 2024-03-30T14:34:19.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-03-30T14:34:21.683 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-30T14:34:21.766 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-30T14:34:23.158 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-30T14:34:23.159 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-30T14:34:24.033 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-30T14:34:24.048 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-30T14:34:25.874 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-30T14:34:38.816 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 664 kB/s | 6.1 MB 00:09 2024-03-30T14:34:40.595 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-30T14:34:42.337 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-30T14:34:47.342 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-30T14:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T14:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T14:34:49.678 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:34:49.687 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-03-30T14:34:49.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:34:49.689 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 76 M 2024-03-30T14:34:49.689 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 123 M 2024-03-30T14:34:49.689 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:34:49.844 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-30T14:34:50.436 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-30T14:34:50.894 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-30T14:34:51.211 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-30T14:34:51.215 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:34:51.215 INFO:teuthology.orchestra.run.smithi134.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-30T14:34:51.215 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:34:51.563 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:34:51.563 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:34:52.422 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:34:52.422 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:34:54.730 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:34:55.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:34:55.402 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:34:58.240 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:34:58.381 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:35:03.100 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:35:03.204 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:07.960 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:37.192 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:37.192 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T14:35:37.192 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:35:37.235 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:35:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:35:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:35:38.073 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:35:38.401 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:35:38.505 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.506 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:35:38.506 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T14:35:38.506 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T14:35:38.506 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T14:35:38.506 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T14:35:38.507 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T14:35:38.507 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:35:38.585 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:35:38.586 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:35:38.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-03-30T14:35:38.630 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T14:35:38.631 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T14:35:38.632 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-03-30T14:35:38.632 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:35:38.632 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T14:35:38.632 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T14:35:38.633 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-434.el9.x86_64 2024-03-30T14:35:38.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T14:35:39.244 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-03-30T14:35:40.663 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T14:35:40.690 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-03-30T14:35:40.693 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-30T14:35:40.708 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.ekWt6widMu 2024-03-30T14:35:40.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ekWt6widMu 2024-03-30T14:35:40.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.ekWt6widMu 2024-03-30T14:35:40.927 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.ekWt6widMu is 6KB 2024-03-30T14:35:40.946 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.ekWt6widMu 2024-03-30T14:35:40.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T14:35:41.024 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T14:35:41.024 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T14:35:41.024 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T14:35:41.024 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T14:35:41.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T14:35:41.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-03-30T14:36:11.180 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T14:36:11.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:36:11.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:36:29.599 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-03-30T14:36:38.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:36:38.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:36:41.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-03-30T14:36:53.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:36:53.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:36:56.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-03-30T14:37:11.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:37:11.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:37:14.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-03-30T14:37:32.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:37:32.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:37:33.321 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-30T14:37:33.696 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:37:33.697 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-30T14:37:33.697 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-30T14:37:33.712 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-434.el9.x86_64 2024-03-30T14:37:33.713 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T14:37:33.713 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T14:37:33.713 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T14:37:34.713 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T14:37:34.714 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T14:37:34.786 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-03-30T14:37:34.830 DEBUG:teuthology.parallel:result is None 2024-03-30T14:38:20.277 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 82 kB/s | 19 MB 03:55 2024-03-30T14:38:32.159 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 767 kB/s | 6.1 MB 00:08 2024-03-30T14:38:33.735 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-30T14:38:36.370 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 21 MB 00:02 2024-03-30T14:38:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-30T14:38:43.793 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T14:38:43.794 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T14:38:43.916 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:38:43.928 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-03-30T14:38:43.929 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:38:43.929 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 76 M 2024-03-30T14:38:43.929 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 123 M 2024-03-30T14:38:43.929 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:38:44.155 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 851 kB/s | 192 kB 00:00 2024-03-30T14:38:45.197 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-30T14:38:46.038 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-30T14:38:46.388 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-03-30T14:38:46.392 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:38:46.392 INFO:teuthology.orchestra.run.smithi028.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-30T14:38:46.392 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:38:46.855 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:38:46.855 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:38:47.724 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:38:50.153 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:38:50.761 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:38:50.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:38:53.742 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:38:53.883 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:38:58.953 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:38:58.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:04.085 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:34.922 INFO:teuthology.orchestra.run.smithi028.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T14:39:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:39:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T14:39:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T14:39:35.367 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:39:35.831 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:39:36.167 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:39:36.317 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:39:36.317 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:39:36.318 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T14:39:36.318 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T14:39:36.318 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T14:39:36.318 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T14:39:36.318 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T14:39:36.318 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T14:39:36.402 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T14:39:36.402 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T14:39:36.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-03-30T14:39:36.449 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T14:39:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:39:36.451 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T14:39:36.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T14:39:36.452 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-434.el9.x86_64 2024-03-30T14:39:36.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T14:39:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-03-30T14:39:38.414 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T14:39:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-03-30T14:39:38.454 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-03-30T14:39:38.469 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.a2R7hmPBJ7 2024-03-30T14:39:38.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.a2R7hmPBJ7 2024-03-30T14:39:38.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.a2R7hmPBJ7 2024-03-30T14:39:38.692 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.a2R7hmPBJ7 is 6KB 2024-03-30T14:39:38.741 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.a2R7hmPBJ7 2024-03-30T14:39:38.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T14:39:38.833 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T14:39:38.833 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T14:39:38.833 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T14:39:38.833 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T14:39:38.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T14:39:38.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-03-30T14:40:08.987 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T14:40:08.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-30T14:40:08.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:41:08.991 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T14:41:17.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-30T14:41:17.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:41:18.273 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-30T14:41:18.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-30T14:41:18.636 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T14:41:18.636 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-30T14:41:18.654 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-434.el9.x86_64 2024-03-30T14:41:18.654 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T14:41:18.654 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T14:41:18.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T14:41:19.656 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T14:41:19.656 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T14:41:19.728 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-03-30T14:41:19.767 DEBUG:teuthology.parallel:result is None 2024-03-30T14:41:19.768 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T14:41:19.774 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T14:41:19.774 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T14:41:19.776 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T14:41:19.794 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T14:41:19.868 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T14:41:19.968 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T14:41:19.968 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T14:41:19.971 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T14:41:20.014 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T14:41:20.046 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T14:41:20.046 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T14:41:20.048 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T14:41:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:41:20.093 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:41:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:41:20.108 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T14:41:20.109 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T14:41:20.116 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T14:41:20.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T14:41:20.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T14:41:20.179 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T14:41:20.225 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T14:41:20.225 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T14:41:20.228 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T14:41:20.244 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:41:20.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:41:20.378 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:41:20.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:41:20.504 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:41:20.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T14:41:20.574 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:41:20.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T14:41:20.675 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:41:20.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T14:41:20.805 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:41:20.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T14:41:20.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-03-30T14:41:20.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-03-30T14:41:20.926 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:41:20.957 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T14:41:21.471 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T14:41:21.481 INFO:teuthology.task.internal:Starting timer... 2024-03-30T14:41:21.481 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T14:41:21.624 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T14:41:21.741 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T14:41:21.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-03-30T14:41:21.808 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-03-30T14:41:21.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-03-30T14:41:21.874 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-03-30T14:41:21.876 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T14:41:21.876 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-03-30T14:41:21.904 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-03-30T14:41:21.905 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-03-30T14:41:21.929 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-03-30T14:41:21.929 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-03-30T14:41:21.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:41:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:41:21.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T14:41:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:41:22.003 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T14:41:22.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-03-30T14:41:22.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-03-30T14:41:22.081 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T14:41:22.091 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T14:41:22.107 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T14:41:22.108 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T14:45:44.751 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-03-30T14:45:44.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-30T14:45:44.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:45:44.845 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-30T14:45:44.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-30T14:45:44.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:45:44.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T14:45:45.057 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-30T14:45:45.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-30T14:45:45.141 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T14:45:45.151 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T14:45:45.151 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T14:45:45.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:45:45.155 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T14:45:45.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T14:45:45.184 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T14:45:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T14:45:45.209 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T14:45:45.225 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T14:45:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-03-30T14:45:45.244 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-03-30T14:45:45.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-03-30T14:45:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T14:45:45.268 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-03-30T14:45:45.278 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T14:45:45.284 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T14:45:45.301 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T14:45:45.330 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.333 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.356 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.359 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T14:45:45.360 INFO:teuthology.run_tasks:Running task pexec... 2024-03-30T14:45:45.369 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-30T14:45:45.370 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T14:45:45.370 DEBUG:teuthology.orchestra.run.smithi028:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T14:45:45.370 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2024-03-30T14:45:45.371 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T14:45:45.968 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:54 ago on Sat 30 Mar 2024 02:44:51 PM UTC. 2024-03-30T14:45:45.989 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 02:44:46 PM UTC. 2024-03-30T14:45:46.121 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:45:46.132 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:45:46.132 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.1 M 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 9.8 M 2024-03-30T14:45:46.133 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:45:46.138 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-30T14:45:46.139 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 9.8 M 2024-03-30T14:45:46.140 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:45:46.430 INFO:teuthology.orchestra.run.smithi134.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-30T14:45:46.430 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:45:46.430 INFO:teuthology.orchestra.run.smithi134.stdout:Total 11 MB/s | 3.1 MB 00:00 2024-03-30T14:45:46.467 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:45:46.472 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:45:46.472 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:45:46.488 INFO:teuthology.orchestra.run.smithi028.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-03-30T14:45:46.488 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:45:46.488 INFO:teuthology.orchestra.run.smithi028.stdout:Total 8.8 MB/s | 3.1 MB 00:00 2024-03-30T14:45:46.525 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:45:46.530 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:45:46.530 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:45:46.553 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:45:46.553 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:45:46.613 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:45:46.613 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:45:46.777 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:45:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:45:46.882 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:46.918 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:47.408 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:47.445 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:45:47.703 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:45:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-30T14:45:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:45:47.722 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:45:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-30T14:45:47.722 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:45:47.723 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:45:47.963 DEBUG:teuthology.parallel:result is None 2024-03-30T14:45:47.996 DEBUG:teuthology.parallel:result is None 2024-03-30T14:45:47.996 INFO:teuthology.run_tasks:Running task install... 2024-03-30T14:45:48.004 DEBUG:teuthology.task.install:project ceph 2024-03-30T14:45:48.005 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}} 2024-03-30T14:45:48.005 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'} 2024-03-30T14:45:48.005 INFO:teuthology.task.install:Using flavor: default 2024-03-30T14:45:48.013 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T14:45:48.013 INFO:teuthology.task.install:extra packages: [] 2024-03-30T14:45:48.013 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T14:45:48.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:45:48.016 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'tag': None, 'wait_for_package': False} 2024-03-30T14:45:48.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:45:48.165 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T14:45:48.165 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T14:45:48.232 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/ 2024-03-30T14:45:48.232 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1967.g4e50bbcd 2024-03-30T14:45:48.341 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T14:45:48.341 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:45:48.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T14:45:48.344 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T14:45:48.344 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:45:48.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T14:45:48.372 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T14:45:48.373 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T14:45:48.373 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T14:45:48.373 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T14:45:48.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T14:45:48.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T14:45:48.522 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T14:45:48.540 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T14:45:48.566 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-03-30T14:45:48.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-03-30T14:45:48.588 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-03-30T14:45:48.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-03-30T14:45:48.814 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-03-30T14:45:48.840 INFO:teuthology.orchestra.run.smithi028.stdout:74 files removed 2024-03-30T14:45:48.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-03-30T14:45:48.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2024-03-30T14:45:49.536 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-03-30T14:45:49.577 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 204 kB/s | 83 kB 00:00 2024-03-30T14:45:49.911 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 41 kB/s | 11 kB 00:00 2024-03-30T14:45:49.962 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-03-30T14:45:50.270 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-30T14:45:50.312 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-30T14:45:50.706 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-30T14:45:50.731 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-30T14:45:52.131 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-30T14:45:52.132 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-30T14:45:53.189 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-30T14:45:53.281 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-30T14:45:55.034 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-30T14:45:56.090 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-30T14:45:59.152 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-30T14:46:00.660 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-30T14:46:01.749 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.1 MB 00:02 2024-03-30T14:46:02.370 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-30T14:46:03.433 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-30T14:46:05.510 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-30T14:46:07.437 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-30T14:46:10.103 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options 2024-03-30T14:46:10.106 INFO:teuthology.orchestra.run.smithi028.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T14:46:10.107 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T14:46:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:Install 30 Packages 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:10.109 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 59 M 2024-03-30T14:46:10.110 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:46:10.510 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-30T14:46:10.763 INFO:teuthology.orchestra.run.smithi028.stdout:(1/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x86 9.4 MB/s | 5.3 MB 00:00 2024-03-30T14:46:10.805 INFO:teuthology.orchestra.run.smithi028.stdout:(2/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 609 kB/s | 25 kB 00:00 2024-03-30T14:46:10.888 INFO:teuthology.orchestra.run.smithi028.stdout:(3/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 8.7 MB/s | 745 kB 00:00 2024-03-30T14:46:10.955 INFO:teuthology.orchestra.run.smithi028.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 7.4 MB/s | 501 kB 00:00 2024-03-30T14:46:11.314 INFO:teuthology.orchestra.run.smithi028.stdout:(5/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-30T14:46:11.364 INFO:teuthology.orchestra.run.smithi028.stdout:(6/32): python3-ceph-argparse-19.0.0-1967.g4e50 905 kB/s | 45 kB 00:00 2024-03-30T14:46:11.414 INFO:teuthology.orchestra.run.smithi028.stdout:(7/32): python3-ceph-common-19.0.0-1967.g4e50bb 2.6 MB/s | 131 kB 00:00 2024-03-30T14:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout:(8/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 2.4 MB/s | 161 kB 00:00 2024-03-30T14:46:11.548 INFO:teuthology.orchestra.run.smithi028.stdout:(9/32): python3-rados-19.0.0-1967.g4e50bbcd.el9 4.7 MB/s | 321 kB 00:00 2024-03-30T14:46:11.615 INFO:teuthology.orchestra.run.smithi028.stdout:(10/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 4.4 MB/s | 298 kB 00:00 2024-03-30T14:46:11.682 INFO:teuthology.orchestra.run.smithi028.stdout:(11/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 1.5 MB/s | 99 kB 00:00 2024-03-30T14:46:11.791 INFO:teuthology.orchestra.run.smithi028.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 976 kB/s | 106 kB 00:00 2024-03-30T14:46:11.841 INFO:teuthology.orchestra.run.smithi028.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-30T14:46:11.874 INFO:teuthology.orchestra.run.smithi028.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-30T14:46:11.933 INFO:teuthology.orchestra.run.smithi028.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-30T14:46:11.975 INFO:teuthology.orchestra.run.smithi028.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-30T14:46:12.009 INFO:teuthology.orchestra.run.smithi028.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-30T14:46:12.042 INFO:teuthology.orchestra.run.smithi028.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-30T14:46:12.076 INFO:teuthology.orchestra.run.smithi028.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-30T14:46:12.110 INFO:teuthology.orchestra.run.smithi028.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-30T14:46:12.327 INFO:teuthology.orchestra.run.smithi028.stdout:(21/32): ceph-common-19.0.0-1967.g4e50bbcd.el9. 9.6 MB/s | 21 MB 00:02 2024-03-30T14:46:12.977 INFO:teuthology.orchestra.run.smithi028.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 356 kB/s | 309 kB 00:00 2024-03-30T14:46:13.085 INFO:teuthology.orchestra.run.smithi028.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-30T14:46:13.101 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:46:13.103 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:13.103 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:46:13.103 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:13.103 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 9.8 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.4 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.3 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 21 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 25 k 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T14:46:13.104 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 745 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 501 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 5.1 M 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 45 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 131 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 161 k 2024-03-30T14:46:13.105 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T14:46:13.106 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 321 k 2024-03-30T14:46:13.106 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 298 k 2024-03-30T14:46:13.106 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 99 k 2024-03-30T14:46:13.106 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T14:46:13.106 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout:Install 30 Packages 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-03-30T14:46:13.107 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:13.108 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 59 M 2024-03-30T14:46:13.108 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:46:13.202 INFO:teuthology.orchestra.run.smithi028.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-30T14:46:13.302 INFO:teuthology.orchestra.run.smithi028.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-30T14:46:13.461 INFO:teuthology.orchestra.run.smithi028.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-03-30T14:46:13.494 INFO:teuthology.orchestra.run.smithi028.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-30T14:46:13.562 INFO:teuthology.orchestra.run.smithi028.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-30T14:46:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-30T14:46:13.920 INFO:teuthology.orchestra.run.smithi028.stdout:(30/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 9.4 MB/s | 3.4 MB 00:00 2024-03-30T14:46:14.154 INFO:teuthology.orchestra.run.smithi028.stdout:(31/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 6.8 MB/s | 3.1 MB 00:00 2024-03-30T14:46:14.812 INFO:teuthology.orchestra.run.smithi028.stdout:(32/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9 2.1 MB/s | 9.8 MB 00:04 2024-03-30T14:46:14.818 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:14.818 INFO:teuthology.orchestra.run.smithi028.stdout:Total 12 MB/s | 59 MB 00:04 2024-03-30T14:46:14.819 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:46:14.861 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:46:14.861 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:46:15.269 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-common-19.0.0-1967.g4e50bbcd.el9.x 9.9 MB/s | 21 MB 00:02 2024-03-30T14:46:15.348 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:46:15.348 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:46:15.352 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-base-19.0.0-1967.g4e50bbcd.el9.x86 2.5 MB/s | 5.3 MB 00:02 2024-03-30T14:46:15.378 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-selinux-19.0.0-1967.g4e50bbcd.el9. 232 kB/s | 25 kB 00:00 2024-03-30T14:46:15.436 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): libradosstriper1-19.0.0-1967.g4e50bbcd. 8.4 MB/s | 501 kB 00:00 2024-03-30T14:46:15.487 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): libcephfs2-19.0.0-1967.g4e50bbcd.el9.x8 5.4 MB/s | 745 kB 00:00 2024-03-30T14:46:15.587 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-ceph-argparse-19.0.0-1967.g4e50 453 kB/s | 45 kB 00:00 2024-03-30T14:46:15.693 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-ceph-common-19.0.0-1967.g4e50bb 1.4 MB/s | 131 kB 00:00 2024-03-30T14:46:15.746 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): librgw2-19.0.0-1967.g4e50bbcd.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-30T14:46:15.771 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-cephfs-19.0.0-1967.g4e50bbcd.el 1.7 MB/s | 161 kB 00:00 2024-03-30T14:46:15.872 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): ceph-radosgw-19.0.0-1967.g4e50bbcd.el9 3.6 MB/s | 9.8 MB 00:02 2024-03-30T14:46:15.896 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-rados-19.0.0-1967.g4e50bbcd.el 2.1 MB/s | 321 kB 00:00 2024-03-30T14:46:15.922 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-rbd-19.0.0-1967.g4e50bbcd.el9. 1.9 MB/s | 298 kB 00:00 2024-03-30T14:46:15.972 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): python3-rgw-19.0.0-1967.g4e50bbcd.el9. 1.3 MB/s | 99 kB 00:00 2024-03-30T14:46:16.039 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-03-30T14:46:16.058 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-30T14:46:16.090 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-30T14:46:16.165 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 661 kB 00:00 2024-03-30T14:46:16.190 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-30T14:46:16.207 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 351 kB/s | 41 kB 00:00 2024-03-30T14:46:16.233 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-30T14:46:16.249 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-30T14:46:16.275 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 521 kB/s | 35 kB 00:00 2024-03-30T14:46:16.408 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-30T14:46:16.492 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-30T14:46:16.517 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-30T14:46:16.559 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-30T14:46:16.626 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-30T14:46:16.693 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-30T14:46:16.768 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-30T14:46:16.802 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-30T14:46:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:46:16.888 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T14:46:16.917 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T14:46:16.942 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T14:46:16.973 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T14:46:16.989 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T14:46:17.085 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): librbd1-19.0.0-1967.g4e50bbcd.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-03-30T14:46:17.133 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T14:46:17.188 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T14:46:17.385 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): librados2-19.0.0-1967.g4e50bbcd.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-03-30T14:46:17.389 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:17.390 INFO:teuthology.orchestra.run.smithi134.stdout:Total 14 MB/s | 59 MB 00:04 2024-03-30T14:46:17.390 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:46:17.424 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:46:17.424 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:46:17.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T14:46:17.585 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:17.661 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:17.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T14:46:17.778 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T14:46:17.841 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T14:46:17.862 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T14:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T14:46:17.906 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:46:17.907 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T14:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T14:46:18.053 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T14:46:18.088 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T14:46:18.200 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T14:46:18.239 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T14:46:18.288 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T14:46:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T14:46:18.349 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T14:46:18.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T14:46:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T14:46:18.736 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T14:46:18.827 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T14:46:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T14:46:18.927 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T14:46:19.361 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:46:19.405 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T14:46:19.430 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T14:46:19.443 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T14:46:19.482 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T14:46:19.485 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T14:46:19.515 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T14:46:19.538 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T14:46:19.554 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T14:46:19.555 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T14:46:19.591 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T14:46:19.732 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T14:46:19.786 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T14:46:20.128 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T14:46:20.193 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:20.268 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:20.358 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 8/34 2024-03-30T14:46:20.386 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T14:46:20.449 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 9/34 2024-03-30T14:46:20.477 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T14:46:20.521 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 11/34 2024-03-30T14:46:20.591 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T14:46:20.613 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T14:46:20.677 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 13/34 2024-03-30T14:46:20.712 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T14:46:20.823 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T14:46:20.855 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T14:46:20.895 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T14:46:20.936 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T14:46:20.965 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T14:46:20.998 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T14:46:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T14:46:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T14:46:21.426 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 22/34 2024-03-30T14:46:21.458 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 23/34 2024-03-30T14:46:21.502 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T14:46:21.956 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T14:46:21.963 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:21.981 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T14:46:22.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 26/34 2024-03-30T14:46:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 27/34 2024-03-30T14:46:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 28/34 2024-03-30T14:46:22.582 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:22.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T14:46:22.713 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T14:46:22.713 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T14:46:22.713 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T14:46:22.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:22.740 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:24.411 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:24.447 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:25.001 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 29/34 2024-03-30T14:46:25.042 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T14:46:25.113 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 30/34 2024-03-30T14:46:25.113 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T14:46:25.113 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T14:46:25.113 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:25.145 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-03-30T14:46:32.631 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-03-30T14:46:32.632 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-03-30T14:46:32.632 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-03-30T14:46:32.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:33.571 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T14:46:33.595 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T14:46:33.595 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:33.595 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T14:46:33.595 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T14:46:33.595 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T14:46:33.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:33.597 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T14:46:33.682 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T14:46:33.682 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:34.767 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:34.767 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T14:46:34.767 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T14:46:34.767 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T14:46:34.768 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T14:46:34.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:34.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T14:46:34.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T14:46:34.840 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-03-30T14:46:34.841 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:35.757 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T14:46:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/34 2024-03-30T14:46:35.782 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:35.782 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T14:46:35.782 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T14:46:35.782 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T14:46:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T14:46:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T14:46:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T14:46:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:35.970 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:46:36.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2024-03-30T14:46:36.878 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:29 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:46:37.016 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:37.016 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/34 2024-03-30T14:46:37.016 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_6 6/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 7/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9. 8/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x8 9/34 2024-03-30T14:46:37.017 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 10/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 11/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 12/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 13/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T14:46:37.018 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T14:46:37.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 31/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T14:46:37.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/34 2024-03-30T14:46:37.038 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:46:37.040 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:46:37.040 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:37.040 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:46:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:46:37.041 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:46:37.042 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-03-30T14:46:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:37.042 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 48 M 2024-03-30T14:46:37.042 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 205 M 2024-03-30T14:46:37.042 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:46:37.502 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 550 kB/s | 217 kB 00:00 2024-03-30T14:46:37.643 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-30T14:46:37.710 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 271 kB/s | 163 kB 00:00 2024-03-30T14:46:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 682 kB/s | 63 kB 00:00 2024-03-30T14:46:38.103 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 478 kB/s | 187 kB 00:00 2024-03-30T14:46:38.178 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T14:46:38.178 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:38.178 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.179 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.180 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:38.181 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:46:38.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-03-30T14:46:39.069 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:29 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:46:39.226 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:46:39.227 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:39.227 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:46:39.227 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:39.227 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 47 M 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 163 k 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-30T14:46:39.228 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:39.229 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-03-30T14:46:39.229 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 205 M 2024-03-30T14:46:39.229 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:46:39.419 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 21 MB/s | 47 MB 00:02 2024-03-30T14:46:39.422 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:39.422 INFO:teuthology.orchestra.run.smithi028.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-30T14:46:39.437 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:46:39.460 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:46:39.460 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:46:39.547 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-30T14:46:39.672 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-30T14:46:39.697 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): libcephsqlite-19.0.0-1967.g4e50bbcd.el9. 490 kB/s | 163 kB 00:00 2024-03-30T14:46:39.723 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-30T14:46:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:46:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:46:40.446 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:46:40.525 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T14:46:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T14:46:40.610 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T14:46:40.623 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 202 kB/s | 187 kB 00:00 2024-03-30T14:46:40.657 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T14:46:40.680 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T14:46:41.707 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-1967.g4e50bbcd.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-30T14:46:41.710 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:41.710 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-30T14:46:41.726 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:46:41.747 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:46:41.747 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:46:42.131 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:46:42.131 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:46:42.700 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:46:42.774 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T14:46:42.816 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T14:46:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T14:46:42.907 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T14:46:42.934 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T14:46:45.386 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T14:46:45.466 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T14:46:47.419 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/6 2024-03-30T14:46:47.501 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T14:46:47.783 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T14:46:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T14:46:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T14:46:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T14:46:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T14:46:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: jq-1.6-16.el9.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T14:46:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:46:48.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2024-03-30T14:46:49.003 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:42 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:46:49.165 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:46:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-03-30T14:46:49.168 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:46:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-03-30T14:46:49.168 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:46:49.168 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T14:46:49.169 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T14:46:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T14:46:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout:Install 36 Packages 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:49.172 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 30 M 2024-03-30T14:46:49.173 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 107 M 2024-03-30T14:46:49.173 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:46:49.827 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 6/6 2024-03-30T14:46:49.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/6 2024-03-30T14:46:49.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/6 2024-03-30T14:46:49.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T14:46:49.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T14:46:49.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T14:46:50.015 INFO:teuthology.orchestra.run.smithi028.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-30T14:46:50.173 INFO:teuthology.orchestra.run.smithi028.stdout:(2/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 3.2 MB/s | 1.5 MB 00:00 2024-03-30T14:46:50.241 INFO:teuthology.orchestra.run.smithi028.stdout:(3/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 4.4 MB/s | 2.4 MB 00:00 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T14:46:50.302 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:50.303 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:46:50.324 INFO:teuthology.orchestra.run.smithi028.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 2.9 MB/s | 248 kB 00:00 2024-03-30T14:46:50.441 INFO:teuthology.orchestra.run.smithi028.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T14:46:50.483 INFO:teuthology.orchestra.run.smithi028.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-30T14:46:50.508 INFO:teuthology.orchestra.run.smithi028.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-30T14:46:50.533 INFO:teuthology.orchestra.run.smithi028.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-30T14:46:50.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-03-30T14:46:50.558 INFO:teuthology.orchestra.run.smithi028.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-30T14:46:50.600 INFO:teuthology.orchestra.run.smithi028.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-30T14:46:50.667 INFO:teuthology.orchestra.run.smithi028.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-30T14:46:50.701 INFO:teuthology.orchestra.run.smithi028.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-30T14:46:50.735 INFO:teuthology.orchestra.run.smithi028.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-30T14:46:50.768 INFO:teuthology.orchestra.run.smithi028.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-30T14:46:50.977 INFO:teuthology.orchestra.run.smithi028.stdout:(15/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86 21 MB/s | 17 MB 00:00 2024-03-30T14:46:51.129 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:41 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:46:51.152 INFO:teuthology.orchestra.run.smithi028.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-30T14:46:51.289 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:46:51.291 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-30T14:46:51.291 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 6.6 k 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 2.4 M 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 1.5 M 2024-03-30T14:46:51.292 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 248 k 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 4.7 M 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 17 M 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T14:46:51.293 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T14:46:51.294 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T14:46:51.295 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T14:46:51.296 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T14:46:51.297 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-30T14:46:51.298 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-03-30T14:46:51.299 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:51.300 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-03-30T14:46:51.300 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 107 M 2024-03-30T14:46:51.300 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:46:51.428 INFO:teuthology.orchestra.run.smithi028.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 611 kB/s | 151 kB 00:00 2024-03-30T14:46:51.470 INFO:teuthology.orchestra.run.smithi028.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-30T14:46:51.570 INFO:teuthology.orchestra.run.smithi028.stdout:(19/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-30T14:46:51.618 INFO:teuthology.orchestra.run.smithi028.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-30T14:46:51.646 INFO:teuthology.orchestra.run.smithi028.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-30T14:46:51.671 INFO:teuthology.orchestra.run.smithi028.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 208 kB/s | 11 kB 00:00 2024-03-30T14:46:51.696 INFO:teuthology.orchestra.run.smithi028.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-30T14:46:51.721 INFO:teuthology.orchestra.run.smithi028.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-03-30T14:46:51.746 INFO:teuthology.orchestra.run.smithi028.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-30T14:46:51.772 INFO:teuthology.orchestra.run.smithi028.stdout:(26/36): python3-requests-2.25.1-8.el9.noarch.r 124 kB/s | 125 kB 00:01 2024-03-30T14:46:51.797 INFO:teuthology.orchestra.run.smithi028.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-30T14:46:51.822 INFO:teuthology.orchestra.run.smithi028.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 624 kB/s | 46 kB 00:00 2024-03-30T14:46:51.864 INFO:teuthology.orchestra.run.smithi028.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-30T14:46:51.889 INFO:teuthology.orchestra.run.smithi028.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-30T14:46:51.899 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-1967.g4e50bbcd.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-30T14:46:51.914 INFO:teuthology.orchestra.run.smithi028.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-30T14:46:51.956 INFO:teuthology.orchestra.run.smithi028.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 984 kB/s | 90 kB 00:00 2024-03-30T14:46:51.981 INFO:teuthology.orchestra.run.smithi028.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 538 kB/s | 36 kB 00:00 2024-03-30T14:46:52.015 INFO:teuthology.orchestra.run.smithi028.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-30T14:46:52.034 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-1967.g4e50bbcd.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-03-30T14:46:52.041 INFO:teuthology.orchestra.run.smithi028.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-30T14:46:52.065 INFO:teuthology.orchestra.run.smithi028.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 242 kB/s | 20 kB 00:00 2024-03-30T14:46:52.071 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:52.071 INFO:teuthology.orchestra.run.smithi028.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-30T14:46:52.096 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:46:52.142 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:46:52.143 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:46:52.558 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mds-19.0.0-1967.g4e50bbcd.el9.x86_ 2.7 MB/s | 2.4 MB 00:00 2024-03-30T14:46:52.573 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:46:52.573 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:46:52.659 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1967.g4e50 2.4 MB/s | 248 kB 00:00 2024-03-30T14:46:52.776 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-30T14:46:52.818 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-30T14:46:52.993 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): ceph-osd-19.0.0-1967.g4e50bbcd.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-30T14:46:53.018 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 158 kB/s | 32 kB 00:00 2024-03-30T14:46:53.043 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-30T14:46:53.068 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-30T14:46:53.094 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T14:46:53.161 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): ceph-mon-19.0.0-1967.g4e50bbcd.el9.x86 3.7 MB/s | 4.7 MB 00:01 2024-03-30T14:46:53.195 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-03-30T14:46:53.220 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 822 kB/s | 103 kB 00:00 2024-03-30T14:46:53.245 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-30T14:46:53.270 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-30T14:46:53.454 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 535 kB/s | 125 kB 00:00 2024-03-30T14:46:53.612 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 442 kB/s | 151 kB 00:00 2024-03-30T14:46:53.638 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 237 kB/s | 43 kB 00:00 2024-03-30T14:46:53.663 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 53 kB/s | 22 kB 00:00 2024-03-30T14:46:53.705 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-30T14:46:53.772 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-30T14:46:53.797 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-30T14:46:53.822 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-30T14:46:53.847 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 309 kB/s | 23 kB 00:00 2024-03-30T14:46:53.889 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-30T14:46:53.909 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:46:53.914 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-30T14:46:53.940 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-30T14:46:53.965 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-30T14:46:53.976 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T14:46:53.999 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-30T14:46:54.018 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T14:46:54.024 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-30T14:46:54.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T14:46:54.050 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-30T14:46:54.074 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-30T14:46:54.108 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-30T14:46:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T14:46:54.141 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-30T14:46:54.162 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T14:46:54.166 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-30T14:46:54.172 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:46:54.172 INFO:teuthology.orchestra.run.smithi134.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-30T14:46:54.194 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T14:46:54.195 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:46:54.238 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:46:54.238 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:46:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T14:46:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T14:46:54.431 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T14:46:54.506 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T14:46:54.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T14:46:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T14:46:54.652 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:46:54.653 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:46:54.662 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T14:46:54.726 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T14:46:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T14:46:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T14:46:55.020 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T14:46:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T14:46:55.084 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T14:46:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T14:46:55.203 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T14:46:55.245 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T14:46:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T14:46:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T14:46:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T14:46:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T14:46:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T14:46:55.562 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T14:46:55.737 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T14:46:55.877 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T14:46:55.990 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:46:56.044 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T14:46:56.047 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T14:46:56.068 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T14:46:56.078 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T14:46:56.097 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T14:46:56.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:56.167 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T14:46:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T14:46:56.253 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T14:46:56.346 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T14:46:56.415 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T14:46:56.456 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T14:46:56.528 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T14:46:56.582 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T14:46:56.619 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T14:46:56.677 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T14:46:56.733 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T14:46:56.919 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T14:46:56.965 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T14:46:57.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T14:46:57.047 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T14:46:57.077 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T14:46:57.153 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T14:46:57.187 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T14:46:57.220 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T14:46:57.261 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T14:46:57.295 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T14:46:57.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T14:46:57.428 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T14:46:57.461 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T14:46:57.533 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T14:46:57.628 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T14:46:57.655 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:57.672 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T14:46:57.802 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T14:46:57.961 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 31/36 2024-03-30T14:46:57.999 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T14:46:58.024 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 32/36 2024-03-30T14:46:58.024 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:58.024 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T14:46:58.024 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T14:46:58.025 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T14:46:58.025 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T14:46:58.153 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T14:46:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:46:58.480 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T14:46:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 33/36 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T14:46:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:46:59.965 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 34/36 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T14:46:59.988 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:00.215 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 35/36 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T14:47:00.238 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T14:47:00.305 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T14:47:00.306 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T14:47:00.308 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T14:47:00.313 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:01.455 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T14:47:01.456 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T14:47:01.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T14:47:01.458 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:01.458 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:01.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2024-03-30T14:47:02.239 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 36/36 2024-03-30T14:47:02.239 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 3/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 4/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 5/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9. 6/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T14:47:02.240 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T14:47:02.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T14:47:02.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T14:47:02.371 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:55 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:02.470 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:02.525 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:02.526 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:02.526 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:02.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2024-03-30T14:47:02.999 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T14:47:02.999 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:02.999 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:02.999 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T14:47:03.000 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T14:47:03.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:03.002 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:03.152 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:03.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-03-30T14:47:03.309 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:03.310 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 748 k 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 755 k 2024-03-30T14:47:03.311 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:03.762 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-03-30T14:47:03.762 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:03.762 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.6 MB/s | 748 kB 00:00 2024-03-30T14:47:03.762 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:03.768 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:03.768 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:03.774 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:03.774 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:03.857 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:03.858 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:53 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:03.957 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:04.011 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:04.012 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:04.012 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:04.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-03-30T14:47:04.328 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:04.427 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:04.651 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:54 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:04.800 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:04.800 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:04.800 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 748 k 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 748 k 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 755 k 2024-03-30T14:47:04.801 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:04.805 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:05.117 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:05.117 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:05.222 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-1967.g4e50bbcd.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-30T14:47:05.223 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:05.223 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-30T14:47:05.223 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:05.229 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:05.229 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:05.235 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:05.235 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:05.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T14:47:05.309 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:05.712 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:05.802 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:05.879 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:58 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:06.037 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 145 k 2024-03-30T14:47:06.038 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 435 k 2024-03-30T14:47:06.039 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:06.179 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:06.391 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 411 kB/s | 145 kB 00:00 2024-03-30T14:47:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:Total 409 kB/s | 145 kB 00:00 2024-03-30T14:47:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:06.401 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:06.401 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:06.455 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:06.456 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/1 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:06.462 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:06.674 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:06.674 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T14:47:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:06.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T14:47:07.263 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:07.416 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:07.417 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:07.417 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-30T14:47:07.417 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:07.417 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 145 k 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 435 k 2024-03-30T14:47:07.418 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:07.651 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:07.761 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-1967.g4e50bb 422 kB/s | 145 kB 00:00 2024-03-30T14:47:07.761 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:07.761 INFO:teuthology.orchestra.run.smithi134.stdout:Total 421 kB/s | 145 kB 00:00 2024-03-30T14:47:07.762 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:07.769 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:07.769 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:07.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2024-03-30T14:47:07.957 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:08.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:08.036 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:08.036 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:08.036 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T14:47:08.036 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:08.493 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:01 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:08.649 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:08.650 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:08.650 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:08.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T14:47:08.979 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd. 1/1 2024-03-30T14:47:08.979 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:08.979 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:08.979 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:08.980 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:08.980 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:09.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-03-30T14:47:09.279 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:09.436 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:09.437 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T14:47:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-03-30T14:47:09.439 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-03-30T14:47:09.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:09.440 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 8.6 M 2024-03-30T14:47:09.440 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 103 M 2024-03-30T14:47:09.440 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:09.807 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:09.927 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1967.g4e 161 kB/s | 31 kB 00:00 2024-03-30T14:47:09.953 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 70 kB/s | 15 kB 00:00 2024-03-30T14:47:09.957 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:09.958 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:09.958 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:10.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T14:47:10.094 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-30T14:47:10.145 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-30T14:47:10.329 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 5.9 MB/s | 3.5 MB 00:00 2024-03-30T14:47:10.412 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-30T14:47:10.599 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:00 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:10.729 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2024-03-30T14:47:10.751 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 3.5 M 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 31 k 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 15 k 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T14:47:10.753 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-30T14:47:10.754 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:10.755 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 8.6 M 2024-03-30T14:47:10.755 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 103 M 2024-03-30T14:47:10.755 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:10.904 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 722 kB/s | 548 kB 00:00 2024-03-30T14:47:11.005 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-30T14:47:11.121 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 369 kB/s | 144 kB 00:00 2024-03-30T14:47:11.147 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-30T14:47:11.406 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1967.g4e5 82 kB/s | 15 kB 00:00 2024-03-30T14:47:11.430 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1967.g4e 149 kB/s | 31 kB 00:00 2024-03-30T14:47:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-30T14:47:11.614 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-30T14:47:11.673 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1967.g4e50bbc 7.7 MB/s | 3.5 MB 00:00 2024-03-30T14:47:11.790 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-30T14:47:11.840 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-30T14:47:11.940 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-30T14:47:12.034 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 614 kB/s | 31 kB 00:00 2024-03-30T14:47:12.034 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 753 kB/s | 144 kB 00:00 2024-03-30T14:47:12.058 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-30T14:47:12.125 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.1 MB/s | 2.0 MB 00:00 2024-03-30T14:47:12.127 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:12.127 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.3 MB/s | 8.6 MB 00:01 2024-03-30T14:47:12.145 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:12.162 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:12.162 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:12.355 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:12.355 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:12.876 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:12.959 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T14:47:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T14:47:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T14:47:13.079 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T14:47:13.183 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T14:47:13.313 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T14:47:13.406 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T14:47:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T14:47:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T14:47:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T14:47:14.055 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 577 kB/s | 2.0 MB 00:03 2024-03-30T14:47:14.061 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:14.061 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.9 MB/s | 8.6 MB 00:04 2024-03-30T14:47:14.082 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:14.101 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:14.102 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:14.301 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:14.301 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:14.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T14:47:14.594 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T14:47:14.830 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T14:47:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T14:47:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T14:47:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T14:47:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T14:47:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T14:47:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T14:47:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T14:47:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T14:47:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 10/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T14:47:15.916 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T14:47:15.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T14:47:15.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T14:47:15.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T14:47:15.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T14:47:15.917 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T14:47:16.260 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:16.261 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:16.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T14:47:16.594 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 11/12 2024-03-30T14:47:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T14:47:17.083 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:07 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:17.230 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:17.231 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-30T14:47:17.231 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T14:47:17.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-30T14:47:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:17.234 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-03-30T14:47:17.234 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-03-30T14:47:17.234 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:17.632 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 215 kB/s | 32 kB 00:00 2024-03-30T14:47:17.687 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 322 kB/s | 17 kB 00:00 2024-03-30T14:47:17.749 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-30T14:47:17.774 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 400 kB/s | 34 kB 00:00 2024-03-30T14:47:17.917 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-30T14:47:17.983 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-30T14:47:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 12/12 2024-03-30T14:47:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el 1/12 2024-03-30T14:47:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noa 2/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9 3/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T14:47:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T14:47:18.009 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-30T14:47:18.342 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T14:47:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:18.354 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:18.484 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 7.4 MB/s | 7.4 MB 00:00 2024-03-30T14:47:18.543 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-30T14:47:18.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T14:47:18.610 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 732 kB/s | 195 kB 00:00 2024-03-30T14:47:18.735 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 981 kB/s | 245 kB 00:00 2024-03-30T14:47:18.738 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:18.738 INFO:teuthology.orchestra.run.smithi134.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-30T14:47:18.777 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:18.801 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:18.801 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:19.164 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:19.164 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:19.171 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:12 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:19.334 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:19.334 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-30T14:47:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:19.334 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-30T14:47:19.334 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 7.4 M 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T14:47:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-03-30T14:47:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:19.337 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 37 M 2024-03-30T14:47:19.337 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 209 M 2024-03-30T14:47:19.337 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:19.879 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-03-30T14:47:19.886 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:19.913 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-03-30T14:47:19.949 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T14:47:19.980 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T14:47:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-30T14:47:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 320 kB/s | 34 kB 00:00 2024-03-30T14:47:20.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T14:47:20.109 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T14:47:20.206 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-30T14:47:20.264 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-30T14:47:20.285 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T14:47:20.289 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-30T14:47:20.333 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T14:47:20.398 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 7.4 MB/s | 803 kB 00:00 2024-03-30T14:47:20.409 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T14:47:20.598 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-30T14:47:20.623 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 876 kB/s | 195 kB 00:00 2024-03-30T14:47:21.084 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T14:47:21.204 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T14:47:21.282 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.8 MB/s | 7.4 MB 00:01 2024-03-30T14:47:21.307 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 346 kB/s | 245 kB 00:00 2024-03-30T14:47:21.310 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:21.310 INFO:teuthology.orchestra.run.smithi028.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-30T14:47:21.354 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:21.379 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:21.379 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:21.753 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:21.753 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:22.457 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T14:47:22.594 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:22.672 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T14:47:22.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T14:47:22.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T14:47:22.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T14:47:23.033 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T14:47:23.063 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T14:47:23.142 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T14:47:23.484 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T14:47:23.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T14:47:23.830 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T14:47:23.946 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T14:47:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T14:47:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T14:47:26.322 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T14:47:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T14:47:26.453 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:26.454 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:26.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-03-30T14:47:27.480 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:17 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:27.634 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T14:47:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:27.636 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:27.637 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-03-30T14:47:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:27.637 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-03-30T14:47:27.637 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-03-30T14:47:27.637 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:27.815 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-03-30T14:47:27.840 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-30T14:47:27.882 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-30T14:47:27.907 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-30T14:47:27.932 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-30T14:47:27.958 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-30T14:47:27.983 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 167 kB/s | 49 kB 00:00 2024-03-30T14:47:28.150 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-03-30T14:47:28.175 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-30T14:47:28.276 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 592 kB/s | 59 kB 00:00 2024-03-30T14:47:28.301 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 638 kB/s | 203 kB 00:00 2024-03-30T14:47:28.368 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 975 kB/s | 90 kB 00:00 2024-03-30T14:47:28.401 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-03-30T14:47:28.405 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:28.406 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-30T14:47:28.406 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:28.419 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:28.419 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 12/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50b 1/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T14:47:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T14:47:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T14:47:29.026 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:29.214 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T14:47:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T14:47:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T14:47:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T14:47:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T14:47:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T14:47:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:29.265 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:29.269 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T14:47:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T14:47:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T14:47:29.412 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T14:47:29.504 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T14:47:29.605 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T14:47:29.662 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T14:47:29.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2024-03-30T14:47:30.271 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:23 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T14:47:30.431 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 49 k 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:30.432 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:30.433 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:30.434 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2024-03-30T14:47:30.434 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:30.434 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.0 M 2024-03-30T14:47:30.434 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-03-30T14:47:30.434 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T14:47:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T14:47:30.518 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T14:47:30.548 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T14:47:30.610 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-30T14:47:30.635 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-30T14:47:30.686 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-30T14:47:30.728 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-30T14:47:30.753 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-30T14:47:30.770 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-30T14:47:30.787 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): ceph-mgr-rook-19.0.0-1967.g4e50bbcd.el9 165 kB/s | 49 kB 00:00 2024-03-30T14:47:30.821 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 298 kB/s | 15 kB 00:00 2024-03-30T14:47:30.837 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 383 kB/s | 32 kB 00:00 2024-03-30T14:47:30.888 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 2.0 MB/s | 203 kB 00:00 2024-03-30T14:47:30.905 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-30T14:47:30.930 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-30T14:47:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-03-30T14:47:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:Total 3.7 MB/s | 2.0 MB 00:00 2024-03-30T14:47:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:31.005 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:31.005 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:31.102 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:31.103 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T14:47:31.334 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T14:47:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T14:47:31.335 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T14:47:31.637 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T14:47:31.687 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:31.688 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:31.818 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T14:47:31.883 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T14:47:31.938 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T14:47:31.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T14:47:31.974 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T14:47:32.016 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T14:47:32.113 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T14:47:32.231 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T14:47:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T14:47:32.526 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:32.679 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T14:47:32.680 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-30T14:47:32.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:32.682 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-03-30T14:47:32.682 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-03-30T14:47:32.682 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:33.042 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T14:47:33.097 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T14:47:33.139 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T14:47:33.188 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T14:47:33.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T14:47:33.271 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-30T14:47:33.329 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 776 kB/s | 248 kB 00:00 2024-03-30T14:47:33.388 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 368 kB/s | 139 kB 00:00 2024-03-30T14:47:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 300 kB/s | 58 kB 00:00 2024-03-30T14:47:33.580 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-30T14:47:33.614 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-30T14:47:33.616 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:33.616 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-03-30T14:47:33.629 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:33.639 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:33.640 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:33.727 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:33.728 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 13/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T14:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T14:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T14:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T14:47:34.029 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:34.187 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T14:47:34.232 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T14:47:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:34.335 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:34.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T14:47:34.748 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T14:47:34.826 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T14:47:34.895 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T14:47:34.918 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T14:47:35.128 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:35.286 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:35.286 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:35.286 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 139 k 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-03-30T14:47:35.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 6.8 M 2024-03-30T14:47:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 33 M 2024-03-30T14:47:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T14:47:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T14:47:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T14:47:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T14:47:35.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T14:47:35.674 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T14:47:35.674 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-03-30T14:47:35.832 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-30T14:47:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1967.g4e50bbcd.e 372 kB/s | 139 kB 00:00 2024-03-30T14:47:35.983 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T14:47:35.984 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:35.985 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:36.207 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-03-30T14:47:36.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-03-30T14:47:36.265 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2024-03-30T14:47:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 774 kB/s | 548 kB 00:00 2024-03-30T14:47:36.384 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:36.384 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-30T14:47:36.399 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:36.410 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:36.410 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:36.504 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:36.504 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:36.825 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:36.977 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:36.978 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:36.978 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:36.978 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 968 k 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-03-30T14:47:36.979 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T14:47:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T14:47:37.114 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 583 kB/s | 79 kB 00:00 2024-03-30T14:47:37.314 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 2.6 MB/s | 889 kB 00:00 2024-03-30T14:47:37.315 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:37.316 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.8 MB/s | 968 kB 00:00 2024-03-30T14:47:37.316 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:37.336 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:37.336 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T14:47:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T14:47:37.741 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T14:47:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T14:47:37.806 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T14:47:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T14:47:38.494 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T14:47:38.495 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T14:47:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 6/6 2024-03-30T14:47:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/6 2024-03-30T14:47:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T14:47:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T14:47:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T14:47:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T14:47:38.799 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T14:47:38.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:38.800 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:38.800 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T14:47:38.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:38.800 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:38.946 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T14:47:38.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:38.946 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:38.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:38.946 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:38.947 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:39.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-03-30T14:47:39.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2024-03-30T14:47:39.681 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:39.734 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:39.838 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:39.838 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-30T14:47:39.839 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:39.840 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 332 k 2024-03-30T14:47:39.840 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-03-30T14:47:39.840 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:39.892 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 889 k 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-03-30T14:47:39.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 968 k 2024-03-30T14:47:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 2.8 M 2024-03-30T14:47:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:39.960 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 573 kB/s | 69 kB 00:00 2024-03-30T14:47:40.017 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 640 kB/s | 79 kB 00:00 2024-03-30T14:47:40.202 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 729 kB/s | 264 kB 00:00 2024-03-30T14:47:40.203 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:40.203 INFO:teuthology.orchestra.run.smithi134.stdout:Total 913 kB/s | 332 kB 00:00 2024-03-30T14:47:40.204 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:40.212 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:40.213 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:40.249 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:40.249 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:40.342 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-fuse-19.0.0-1967.g4e50bbcd.el9.x86_ 1.9 MB/s | 889 kB 00:00 2024-03-30T14:47:40.343 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:40.343 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-03-30T14:47:40.344 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:40.396 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:40.443 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:40.443 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:40.577 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T14:47:40.621 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T14:47:40.638 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T14:47:40.638 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:40.638 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T14:47:40.638 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:40.641 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:40.789 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T14:47:40.864 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T14:47:41.355 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T14:47:41.538 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2/2 2024-03-30T14:47:41.538 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/2 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:41.672 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:41.843 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:41.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-03-30T14:47:42.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-volume 2024-03-30T14:47:42.574 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:42.722 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:42.727 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:42.728 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:42.728 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-30T14:47:42.728 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-03-30T14:47:42.729 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-1967.g4e50bbcd.el9 ceph-noarch 264 k 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-03-30T14:47:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:42.883 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 332 k 2024-03-30T14:47:42.883 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 1.5 M 2024-03-30T14:47:42.883 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:42.940 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 603 kB/s | 127 kB 00:00 2024-03-30T14:47:42.940 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:42.941 INFO:teuthology.orchestra.run.smithi134.stdout:Total 598 kB/s | 127 kB 00:00 2024-03-30T14:47:42.941 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:42.946 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:42.946 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:43.002 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:43.003 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:43.020 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 499 kB/s | 69 kB 00:00 2024-03-30T14:47:43.179 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-volume-19.0.0-1967.g4e50bbcd.el9.no 891 kB/s | 264 kB 00:00 2024-03-30T14:47:43.180 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:43.180 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-30T14:47:43.180 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:43.190 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:43.191 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:43.245 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:43.339 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T14:47:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T14:47:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2/2 2024-03-30T14:47:43.630 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:43.630 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T14:47:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:43.800 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:44.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:44.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:44.081 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:44.081 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:44.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:44.081 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:44.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-03-30T14:47:44.352 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 1/2 2024-03-30T14:47:44.687 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-1967.g4e50bbcd.el9.noarch 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:44.688 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:44.897 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:44.997 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:44.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2024-03-30T14:47:45.051 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:45.052 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:45.053 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:45.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-03-30T14:47:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:45.688 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:45.733 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:45.733 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 127 k 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 127 k 2024-03-30T14:47:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 456 k 2024-03-30T14:47:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:45.837 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:45.837 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 32 k 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 142 k 2024-03-30T14:47:45.838 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:45.942 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-19.0.0-1967.g4e50bbcd.el9.x86_64 613 kB/s | 127 kB 00:00 2024-03-30T14:47:45.943 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:45.943 INFO:teuthology.orchestra.run.smithi028.stdout:Total 608 kB/s | 127 kB 00:00 2024-03-30T14:47:45.943 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:45.949 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:45.949 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:46.006 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:46.007 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:46.084 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 129 kB/s | 32 kB 00:00 2024-03-30T14:47:46.085 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:46.085 INFO:teuthology.orchestra.run.smithi134.stdout:Total 128 kB/s | 32 kB 00:00 2024-03-30T14:47:46.085 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:46.088 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:46.089 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:46.112 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:46.113 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:46.198 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:46.212 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:46.302 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:46.332 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:46.744 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:47.004 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:47.022 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:47.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-03-30T14:47:47.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2024-03-30T14:47:47.765 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:47.815 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:47.865 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:47.917 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:47.918 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:47.918 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:47.919 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:47.974 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:47.975 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:47.975 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:47.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-03-30T14:47:48.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2024-03-30T14:47:48.557 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:48.614 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:48.657 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:48.710 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:48.711 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:48.711 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 32 k 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 32 k 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 142 k 2024-03-30T14:47:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:48.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-03-30T14:47:49.014 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-19.0.0-1967.g4e50bbcd.el9.x86_6 133 kB/s | 32 kB 00:00 2024-03-30T14:47:49.015 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:49.015 INFO:teuthology.orchestra.run.smithi028.stdout:Total 132 kB/s | 32 kB 00:00 2024-03-30T14:47:49.015 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:49.021 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:49.022 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:49.049 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:49.049 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:49.144 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:49.274 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:49.359 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:49.459 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:49.511 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:49.511 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:49.512 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:49.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-03-30T14:47:49.636 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:49.911 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:49.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:49.912 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:49.912 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:49.912 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:49.912 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:50.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2024-03-30T14:47:50.158 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:50.257 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:50.308 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:50.309 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:50.309 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:50.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-03-30T14:47:50.671 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:50.773 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:50.827 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:50.828 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:50.828 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:50.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2024-03-30T14:47:50.948 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:51.046 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:51.097 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:51.098 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:51.099 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:51.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-03-30T14:47:51.457 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:51.558 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:51.614 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:51.615 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:51.615 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:51.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2024-03-30T14:47:51.746 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:51.846 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:51.897 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:51.898 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-30T14:47:51.898 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:51.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-03-30T14:47:52.248 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:52.351 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:52.406 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:52.408 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:52.408 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:52.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2024-03-30T14:47:52.537 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:52.688 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:52.689 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:52.689 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-30T14:47:52.689 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:52.689 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:52.689 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-03-30T14:47:52.690 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:52.914 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 383 kB/s | 85 kB 00:00 2024-03-30T14:47:52.914 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:52.914 INFO:teuthology.orchestra.run.smithi134.stdout:Total 381 kB/s | 85 kB 00:00 2024-03-30T14:47:52.914 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:52.920 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:52.920 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:52.976 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:52.976 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:53.117 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:53.152 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:53.207 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:53.208 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:53.208 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:53.229 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:53.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2024-03-30T14:47:53.586 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:53.840 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:53.875 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:53.996 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:53.998 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:53.998 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:54.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2024-03-30T14:47:54.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-03-30T14:47:54.629 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:54.721 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:54.731 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 is already installed. 2024-03-30T14:47:54.787 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:54.788 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-03-30T14:47:54.788 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:54.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:54.878 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-03-30T14:47:54.879 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:55.417 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:55.572 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 85 k 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:55.573 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 85 k 2024-03-30T14:47:55.574 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 231 k 2024-03-30T14:47:55.574 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:55.582 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 4.4 MB/s | 3.1 MB 00:00 2024-03-30T14:47:55.582 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:55.583 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.4 MB/s | 3.1 MB 00:00 2024-03-30T14:47:55.583 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:55.592 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:55.592 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:55.666 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:55.666 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:55.892 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 268 kB/s | 85 kB 00:00 2024-03-30T14:47:55.893 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:55.893 INFO:teuthology.orchestra.run.smithi028.stdout:Total 267 kB/s | 85 kB 00:00 2024-03-30T14:47:55.893 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:55.900 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:55.901 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:55.959 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:55.959 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:56.089 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:56.100 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:56.158 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:56.183 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:56.184 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:56.184 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T14:47:56.184 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T14:47:56.184 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T14:47:56.184 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:56.210 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:56.589 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:56.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:56.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:56.903 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:47:56.903 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:56.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:56.904 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:47:57.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2024-03-30T14:47:57.281 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:57.282 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:57.282 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:57.282 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:57.282 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:57.282 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:57.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-03-30T14:47:57.745 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:47:57.904 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:47:57.904 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 3.1 M 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.1 M 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 13 M 2024-03-30T14:47:57.905 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:47:58.090 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 02:46:10 PM UTC. 2024-03-30T14:47:58.240 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-30T14:47:58.241 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-30T14:47:58.242 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-30T14:47:58.242 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:58.242 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 170 k 2024-03-30T14:47:58.242 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-03-30T14:47:58.242 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-30T14:47:58.352 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 6.9 MB/s | 3.1 MB 00:00 2024-03-30T14:47:58.353 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:58.353 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.9 MB/s | 3.1 MB 00:00 2024-03-30T14:47:58.353 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:47:58.363 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:47:58.363 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:47:58.441 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:47:58.441 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:47:58.593 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 485 kB/s | 170 kB 00:00 2024-03-30T14:47:58.594 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:47:58.594 INFO:teuthology.orchestra.run.smithi134.stdout:Total 483 kB/s | 170 kB 00:00 2024-03-30T14:47:58.594 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-30T14:47:58.602 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-30T14:47:58.602 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-30T14:47:58.659 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-30T14:47:58.659 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-30T14:47:58.807 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-30T14:47:58.880 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:47:58.924 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:58.945 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:58.969 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:58.969 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T14:47:58.969 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T14:47:58.969 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T14:47:58.969 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T14:47:58.970 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:47:59.340 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:47:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:59.630 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-30T14:47:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:47:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:47:59.630 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-30T14:47:59.852 DEBUG:teuthology.parallel:result is None 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:00.028 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:48:00.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2024-03-30T14:48:00.841 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Mar 2024 02:46:07 PM UTC. 2024-03-30T14:48:00.997 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-30T14:48:00.998 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:48:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-03-30T14:48:00.998 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-1967.g4e50bbcd.el9 ceph 170 k 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 170 k 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 494 k 2024-03-30T14:48:00.999 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-30T14:48:01.473 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-19.0.0-1967.g4e50bbcd.el9.x86_64.rpm 359 kB/s | 170 kB 00:00 2024-03-30T14:48:01.474 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:48:01.474 INFO:teuthology.orchestra.run.smithi028.stdout:Total 358 kB/s | 170 kB 00:00 2024-03-30T14:48:01.474 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-30T14:48:01.483 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-30T14:48:01.483 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-30T14:48:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-30T14:48:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-30T14:48:01.693 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-30T14:48:01.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:48:02.249 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 1/1 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-1967.g4e50bbcd.el9.x86_64 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:02.538 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-30T14:48:02.778 DEBUG:teuthology.parallel:result is None 2024-03-30T14:48:02.779 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:03.000 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T14:48:03.021 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T14:48:03.022 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T14:48:03.022 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T14:48:03.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:03.165 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T14:48:03.189 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-1967.g4e50bbcd.el9 2024-03-30T14:48:03.190 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1967.g4e50bbcd.el9 2024-03-30T14:48:03.190 INFO:teuthology.task.install:The correct ceph version 19.0.0-1967.g4e50bbcd is installed. 2024-03-30T14:48:03.191 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T14:48:03.192 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:48:03.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T14:48:03.221 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:48:03.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T14:48:03.262 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T14:48:03.262 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:48:03.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T14:48:03.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T14:48:03.359 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:48:03.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T14:48:03.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T14:48:03.458 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T14:48:03.458 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:48:03.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T14:48:03.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T14:48:03.555 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:48:03.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T14:48:03.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T14:48:03.653 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T14:48:03.653 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:48:03.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T14:48:03.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T14:48:03.751 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:48:03.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T14:48:03.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T14:48:03.850 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T14:48:03.950 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'cephadm_mode': 'root'} 2024-03-30T14:48:03.950 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:03.951 INFO:tasks.cephadm:Cluster fsid is 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:03.951 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T14:48:03.951 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.28:6789]', 'mon.c': '[v1:172.21.15.28:6790]', 'mon.b': '[v1:172.21.15.134:6789]'} 2024-03-30T14:48:03.951 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-03-30T14:48:03.952 INFO:tasks.cephadm:First mgr is y 2024-03-30T14:48:03.952 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T14:48:03.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-03-30T14:48:03.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-03-30T14:48:04.011 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-30T14:48:04.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:04.160 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54746/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1967-g4e50bbcd', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1967.g4e50bbcd'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-29 16:06:09.870176', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-30T14:48:04.301 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 4e50bbcdd1630c7760b708f3a5b4d38a121e66cb from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:04.303 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm 2024-03-30T14:48:04.303 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:04.919 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 767434 Mar 30 14:48 /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:04.920 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/4e50bbcdd1630c7760b708f3a5b4d38a121e66cb/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:05.346 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 767434 Mar 30 14:48 /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:05.347 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:05.365 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T14:48:05.393 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb on all hosts... 2024-03-30T14:48:05.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T14:48:05.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb pull 2024-03-30T14:48:06.210 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T14:48:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:48:33.688 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:48:39.808 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)", 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e", 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d" 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:48:39.809 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:48:39.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-03-30T14:48:39.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-03-30T14:48:39.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-03-30T14:48:39.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-03-30T14:48:39.993 INFO:tasks.cephadm:Writing seed config... 2024-03-30T14:48:39.994 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T14:48:39.994 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T14:48:39.994 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-30T14:48:39.995 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T14:48:39.996 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-30T14:48:39.997 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:48:39.997 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T14:48:40.011 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8345fd88-eea4-11ee-b647-cb9ed24678a4 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-30T14:48:40.011 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-30T14:48:40.053 DEBUG:teuthology.orchestra.run.smithi028:mgr.y> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-30T14:48:40.096 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T14:48:40.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -v bootstrap --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.28:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:48:40.291 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-30T14:48:40.291 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb', '-v', 'bootstrap', '--fsid', '8345fd88-eea4-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.28:6789]', '--skip-admin-label'] 2024-03-30T14:48:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T14:48:40.292 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-03-30T14:48:40.316 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T14:48:40.317 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-03-30T14:48:40.317 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-03-30T14:48:40.324 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T14:48:40.324 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T14:48:40.330 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T14:48:40.330 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:48:40.336 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-03-30T14:48:40.342 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-03-30T14:48:40.342 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-03-30T14:48:40.342 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-03-30T14:48:40.364 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-30T14:48:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-30T14:48:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-03-30T14:48:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-03-30T14:48:40.371 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T14:48:40.371 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T14:48:40.376 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T14:48:40.376 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:48:40.382 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139661158161472 on /run/cephadm/8345fd88-eea4-11ee-b647-cb9ed24678a4.lock 2024-03-30T14:48:40.389 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139661158161472 acquired on /run/cephadm/8345fd88-eea4-11ee-b647-cb9ed24678a4.lock 2024-03-30T14:48:40.390 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-03-30T14:48:40.390 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:6789], mon addrv is [v1:172.21.15.28:6789] 2024-03-30T14:48:40.393 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-03-30T14:48:40.393 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-03-30T14:48:40.396 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T14:48:40.396 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-30T14:48:40.396 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-30T14:48:40.398 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T14:48:40.398 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T14:48:40.398 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T14:48:40.398 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-30T14:48:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-03-30T14:48:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T14:48:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-03-30T14:48:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-30T14:48:40.399 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T14:48:40.400 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T14:48:40.868 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb... 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:70aeeb6837442a4655a911c520b59d95aba287a7bb7c28348ac976aba07eb7c4 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e 2024-03-30T14:48:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T14:48:41.815 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T14:48:41.815 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev) 2024-03-30T14:48:41.815 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T14:48:42.928 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-03-30T14:48:42.928 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-03-30T14:48:43.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBLJghmLd7WExAAKLpGFFOG/BeDreKMhcN8Xg== 2024-03-30T14:48:44.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBMJghm366iERAAkFSto2+K1jSVxB6evCjqmQ== 2024-03-30T14:48:45.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBNJghmaM/iDBAASJMVP8/pPF9V+OaMYZVLNA== 2024-03-30T14:48:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-03-30T14:48:46.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T14:48:46.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-30T14:48:46.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:46.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T14:48:46.824 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T14:48:46.825 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-03-30T14:48:46.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:46.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T14:48:46.825 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:48:46.825 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-03-30T14:48:47.870 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-03-30T14:48:48.077 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-30T14:48:48.221 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T14:48:48.392 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target. 2024-03-30T14:48:48.392 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target. 2024-03-30T14:48:48.621 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T14:48:48.621 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-30T14:48:48.776 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target.wants/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@.service. 2024-03-30T14:48:49.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:49 smithi028 podman[29351]: 2024-03-30 14:48:48.915243851 +0000 UTC m=+0.017484265 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:48:49.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:49 smithi028 podman[29351]: 2024-03-30 14:48:49.757542591 +0000 UTC m=+0.859783010 container init 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-03-30T14:48:49.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:49 smithi028 podman[29351]: 2024-03-30 14:48:49.76349882 +0000 UTC m=+0.865739226 container start 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-30T14:48:49.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:49 smithi028 ceph-mon[29389]: mkfs 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:49.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:49 smithi028 ceph-mon[29389]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T14:48:50.687 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T14:48:50.949 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-03-30T14:48:50.950 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 bash[29351]: 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 systemd[1]: Started Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: monmap epoch 1 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: last_changed 2024-03-30T14:48:46.261152+0000 2024-03-30T14:48:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: min_mon_release 19 (squid) 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: election_strategy: 1 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: fsmap 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T14:48:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:50 smithi028 ceph-mon[29389]: mgrmap e1: no daemons active 2024-03-30T14:48:53.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:48:53 smithi028 ceph-mon[29389]: from='client.? v1:172.21.15.28:0/1270749990' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T14:48:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 4s) 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T14:48:59.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.727+0000 7f339f925700 1 Processor -- start 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.728+0000 7f339f925700 1 -- start start 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.728+0000 7f339f925700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f339810abf0 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.728+0000 7f339d6c1700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3398104a30 0x7f3398104e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39442/0 (socket says 172.21.15.28:39442) 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.728+0000 7f339d6c1700 1 -- 172.21.15.28:0/4064028862 learned_addr learned my addr 172.21.15.28:0/4064028862 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224505195 0 0) 0x7f339810abf0 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33840035b0 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2992965720 0 0) 0x7f33840035b0 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339810bdd0 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3394001580 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3394001920 con 0x7f3398104a30 2024-03-30T14:48:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f338ffff700 1 -- 172.21.15.28:0/4064028862 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3394001ca0 con 0x7f3398104a30 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.729+0000 7f339f925700 1 -- 172.21.15.28:0/4064028862 >> v1:172.21.15.28:6789/0 conn(0x7f3398104a30 legacy=0x7f3398104e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 -- 172.21.15.28:0/4064028862 shutdown_connections 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 -- 172.21.15.28:0/4064028862 >> 172.21.15.28:0/4064028862 conn(0x7f3398100360 msgr2=0x7f33981027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 -- 172.21.15.28:0/4064028862 shutdown_connections 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 -- 172.21.15.28:0/4064028862 wait complete. 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 librados: starting msgr at 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 librados: starting objecter 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 Processor -- start 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 -- start start 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 librados: setting wanted keys 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.730+0000 7f339f925700 1 librados: calling monclient init 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339f925700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f339819a1d0 con 0x7f3398104a30 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339d6c1700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3398104a30 0x7f3398199ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39446/0 (socket says 172.21.15.28:39446) 2024-03-30T14:48:59.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339d6c1700 1 -- 172.21.15.28:0/1270749990 learned_addr learned my addr 172.21.15.28:0/1270749990 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 10 client.?.objecter ms_handle_connect 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684088486 0 0) 0x7f339819a1d0 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3384004560 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1748281749 0 0) 0x7f3384004560 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f339819a1d0 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3394001480 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 247295944 0 0) 0x7f339819a1d0 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339819a3a0 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f339819a690 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339f925700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f339819e1e0 con 0x7f3398104a30 2024-03-30T14:48:59.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.731+0000 7f339f925700 1 librados: init done 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3394001920 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3394004cc0 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3394005270 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f33940069a0 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 10 client.4098.objecter ms_dispatch 0x7f3398001550 osd_map(1..1 src has 1..1) 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f338e7fc700 20 client.4098.objecter dump_active .. 0 homeless 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.732+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c005150 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.735+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3394005840 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "status"} v 0) -- 0x7f337c005c50 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f338e7fc700 1 -- 172.21.15.28:0/1270749990 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+310 (unknown 1155462804 0 1452408986) 0x7f3394005ac0 con 0x7f3398104a30 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 10 librados: watch_flush enter 2024-03-30T14:48:59.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 10 librados: watch_flush exit 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 20 client.4098.objecter shutdown clearing up homeless session... 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 10 client.4098.objecter successfully canceled tick 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.809+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 >> v1:172.21.15.28:6789/0 conn(0x7f3398104a30 legacy=0x7f3398199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.810+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 shutdown_connections 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.810+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 >> 172.21.15.28:0/1270749990 conn(0x7f3398100360 msgr2=0x7f3398190cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.810+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 shutdown_connections 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.810+0000 7f339f925700 1 -- 172.21.15.28:0/1270749990 wait complete. 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:48:53.810+0000 7f339f925700 1 librados: shutdown 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-03-30T14:48:59.716 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T14:49:02.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:02 smithi028 ceph-mon[29389]: from='client.? v1:172.21.15.28:0/3338015316' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T14:49:02.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:02 smithi028 ceph-mon[29389]: from='client.? v1:172.21.15.28:0/3338015316' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T14:49:03.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:03.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.28:6789] 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T14:49:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.732+0000 7f68974b8700 1 Processor -- start 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.732+0000 7f68974b8700 1 -- start start 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.732+0000 7f68974b8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6890106b90 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.732+0000 7f6895254700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6890105050 0x7f6890105470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36780/0 (socket says 172.21.15.28:36780) 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.732+0000 7f6895254700 1 -- 172.21.15.28:0/3096410699 learned_addr learned my addr 172.21.15.28:0/3096410699 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311933813 0 0) 0x7f6890106b90 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f687c0035b0 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3138315407 0 0) 0x7f687c0035b0 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6890106d60 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f688c001580 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f688c001920 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f6887fff700 1 -- 172.21.15.28:0/3096410699 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f688c001ca0 con 0x7f6890105050 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.733+0000 7f68974b8700 1 -- 172.21.15.28:0/3096410699 >> v1:172.21.15.28:6789/0 conn(0x7f6890105050 legacy=0x7f6890105470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:03.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 -- 172.21.15.28:0/3096410699 shutdown_connections 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 -- 172.21.15.28:0/3096410699 >> 172.21.15.28:0/3096410699 conn(0x7f6890100c90 msgr2=0x7f68901030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 -- 172.21.15.28:0/3096410699 shutdown_connections 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 -- 172.21.15.28:0/3096410699 wait complete. 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 librados: starting msgr at 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 librados: starting objecter 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 Processor -- start 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 -- start start 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 librados: setting wanted keys 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.734+0000 7f68974b8700 1 librados: calling monclient init 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68974b8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f689019c500 con 0x7f6890105050 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f6895254700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6890105050 0x7f689019bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f6895254700 1 -- 172.21.15.28:0/3338015316 learned_addr learned my addr 172.21.15.28:0/3338015316 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 10 client.?.objecter ms_handle_connect 0x7f6890105050 2024-03-30T14:49:03.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968757570 0 0) 0x7f689019c500 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f687c004560 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2647859366 0 0) 0x7f687c004560 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f689019c500 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f688c001540 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3727689624 0 0) 0x7f689019c500 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689019c6d0 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f689019c9c0 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68974b8700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f68901a0510 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.735+0000 7f68974b8700 1 librados: init done 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f688c001920 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f688c004cc0 con 0x7f6890105050 2024-03-30T14:49:03.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f688c0051f0 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f688c006880 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 10 client.4100.objecter ms_dispatch 0x7f6890001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68867fc700 20 client.4100.objecter dump_active .. 0 homeless 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.736+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874005150 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.739+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f688c006ad0 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.810+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6874002430 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.813+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+317 (unknown 1213389831 0 2640779857) 0x7f688c0057d0 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.813+0000 7f68867fc700 1 -- 172.21.15.28:0/3338015316 <== mon.0 v1:172.21.15.28:6789/0 11 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f688c005ef0 con 0x7f6890105050 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.813+0000 7f68974b8700 10 librados: watch_flush enter 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.813+0000 7f68974b8700 10 librados: watch_flush exit 2024-03-30T14:49:03.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 20 client.4100.objecter shutdown clearing up homeless session... 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 10 client.4100.objecter successfully canceled tick 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 >> v1:172.21.15.28:6789/0 conn(0x7f6890105050 legacy=0x7f689019bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 shutdown_connections 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 >> 172.21.15.28:0/3338015316 conn(0x7f6890100c90 msgr2=0x7f6890192910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 shutdown_connections 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 -- 172.21.15.28:0/3338015316 wait complete. 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:02.814+0000 7f68974b8700 1 librados: shutdown 2024-03-30T14:49:03.458 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-03-30T14:49:04.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:04 smithi028 ceph-mon[29389]: from='client.? v1:172.21.15.28:0/1322628114' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:49:04.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7facf2e21700 1 Processor -- start 2024-03-30T14:49:04.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7facf2e21700 1 -- start start 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7facf2e21700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7facec10ad50 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7facf0bbd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7facec107240 0x7facec109630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36798/0 (socket says 172.21.15.28:36798) 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7facf0bbd700 1 -- 172.21.15.28:0/3097610869 learned_addr learned my addr 172.21.15.28:0/3097610869 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993545759 0 0) 0x7facec10ad50 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facdc0035b0 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2173916524 0 0) 0x7facdc0035b0 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facec10bf30 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.881+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7face0001580 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7face0001d30 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7faceb7fe700 1 -- 172.21.15.28:0/3097610869 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7face0005960 con 0x7facec107240 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7facf2e21700 1 -- 172.21.15.28:0/3097610869 >> v1:172.21.15.28:6789/0 conn(0x7facec107240 legacy=0x7facec109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7facf2e21700 1 -- 172.21.15.28:0/3097610869 shutdown_connections 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7facf2e21700 1 -- 172.21.15.28:0/3097610869 >> 172.21.15.28:0/3097610869 conn(0x7facec100c90 msgr2=0x7facec1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:04.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7facf2e21700 1 -- 172.21.15.28:0/3097610869 shutdown_connections 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.882+0000 7facf2e21700 1 -- 172.21.15.28:0/3097610869 wait complete. 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 librados: starting msgr at 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 librados: starting objecter 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 Processor -- start 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 -- start start 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 librados: setting wanted keys 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 librados: calling monclient init 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf2e21700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7facec19abf0 con 0x7facec107240 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf0bbd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7facec107240 0x7facec19a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36802/0 (socket says 172.21.15.28:36802) 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7facf0bbd700 1 -- 172.21.15.28:0/1322628114 learned_addr learned my addr 172.21.15.28:0/1322628114 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7face9ffb700 10 client.?.objecter ms_handle_connect 0x7facec107240 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7face9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424359273 0 0) 0x7facec19abf0 con 0x7facec107240 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facdc0056e0 con 0x7facec107240 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.883+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3678946814 0 0) 0x7facec10bf30 con 0x7facec107240 2024-03-30T14:49:04.525 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7facec19abf0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7face0001080 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3493324583 0 0) 0x7facec19abf0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facec19adc0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7facec19b0b0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7face0004ab0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7face0006380 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7face00069b0 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7facf2e21700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7facec19ec00 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7facf2e21700 1 librados: init done 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7face0007260 con 0x7facec107240 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 10 client.4102.objecter ms_dispatch 0x7facec001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:04.526 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.884+0000 7face9ffb700 20 client.4102.objecter dump_active .. 0 homeless 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.885+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facec063870 con 0x7facec107240 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.887+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7facdc0056e0 con 0x7facec107240 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.958+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7facec02dc50 con 0x7facec107240 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.958+0000 7face9ffb700 1 -- 172.21.15.28:0/1322628114 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+147 (unknown 1452402520 0 4115409571) 0x7face0005ce0 con 0x7facec107240 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.958+0000 7facf2e21700 10 librados: watch_flush enter 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 10 librados: watch_flush exit 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 10 client.4102.objecter successfully canceled tick 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 >> v1:172.21.15.28:6789/0 conn(0x7facec107240 legacy=0x7facec19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 shutdown_connections 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 >> 172.21.15.28:0/1322628114 conn(0x7facec100c90 msgr2=0x7facec1911a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 shutdown_connections 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 -- 172.21.15.28:0/1322628114 wait complete. 2024-03-30T14:49:04.527 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:03.959+0000 7facf2e21700 1 librados: shutdown 2024-03-30T14:49:04.528 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-03-30T14:49:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:04 smithi028 systemd[1]: Stopping Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:49:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:04 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a[29366]: 2024-03-30T14:49:04.787+0000 7f9903954700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T14:49:04.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:04 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a[29366]: 2024-03-30T14:49:04.787+0000 7f9903954700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T14:49:05.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:04 smithi028 podman[29720]: 2024-03-30 14:49:04.941117898 +0000 UTC m=+0.248562187 container died 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T14:49:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 podman[29720]: 2024-03-30 14:49:05.317193433 +0000 UTC m=+0.624637711 container cleanup 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-30T14:49:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 bash[29720]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T14:49:05.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 podman[29740]: 2024-03-30 14:49:05.593336331 +0000 UTC m=+0.656785364 container remove 7c1119966b5cb4ae0dd686bf34fb86c06c877f737776c684a3eb03ff2c90ed2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-30T14:49:05.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-03-30T14:49:05.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 systemd[1]: Stopped Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:49:05.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 systemd[1]: Starting Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:49:06.212 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 podman[29837]: 2024-03-30 14:49:05.935756991 +0000 UTC m=+0.107690115 container create b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T14:49:06.212 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:05 smithi028 podman[29837]: 2024-03-30 14:49:05.845733897 +0000 UTC m=+0.017667003 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 podman[29837]: 2024-03-30 14:49:06.129322798 +0000 UTC m=+0.301255878 container init b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 podman[29837]: 2024-03-30 14:49:06.134499801 +0000 UTC m=+0.306432881 container start b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 7 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: pidfile_write: ignore empty --pid-file 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: load: jerasure load: lrc 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: RocksDB version: 7.9.2 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Git sha 0 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: DB SUMMARY 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: DB Session ID: H7QV7J3MW3GVLS6SKIX3 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: CURRENT file: CURRENT 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-30T14:49:06.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T14:49:06.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94898 ; 2024-03-30T14:49:06.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.error_if_exists: 0 2024-03-30T14:49:06.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.create_if_missing: 0 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.paranoid_checks: 1 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.env: 0x555d2cf89c60 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.info_log: 0x555d2de41b20 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.statistics: (nil) 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.use_fsync: 0 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_log_file_size: 0 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T14:49:06.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_fallocate: 1 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.use_direct_reads: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.db_log_dir: 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.wal_dir: 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T14:49:06.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.write_buffer_manager: 0x555d2de141e0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.unordered_write: 0 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T14:49:06.217 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.row_cache: None 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.wal_filter: None 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.two_write_queues: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.wal_compression: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.atomic_flush: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.log_readahead_size: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T14:49:06.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_background_jobs: 2 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_background_compactions: -1 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_subcompactions: 1 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_open_files: -1 2024-03-30T14:49:06.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_background_flushes: -1 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Compression algorithms supported: 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kZSTD supported: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kXpressCompression supported: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kZlibCompression supported: 1 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kSnappyCompression supported: 1 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kLZ4Compression supported: 1 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: kBZip2Compression supported: 0 2024-03-30T14:49:06.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.merge_operator: 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_filter: None 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555d2de41c40) 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-30T14:49:06.221 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x555d2de45090 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-03-30T14:49:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-03-30T14:49:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression: NoCompression 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.num_levels: 7 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T14:49:06.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:49:06.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T14:49:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:49:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:49:06.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.ttl: 2592000 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:49:06.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6c37f2ff-048c-46e0-9f8c-1d41987e0f61 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810146163624, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810146164637, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90380, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 300, "table_properties": {"data_size": 88460, "index_size": 266, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12347, "raw_average_key_size": 48, "raw_value_size": 81765, "raw_average_value_size": 320, "num_data_blocks": 11, "num_entries": 255, "num_filter_entries": 255, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711810146, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6c37f2ff-048c-46e0-9f8c-1d41987e0f61", "db_session_id": "H7QV7J3MW3GVLS6SKIX3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-30T14:49:06.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810146164750, "job": 1, "event": "recovery_finished"} 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555d2df36000 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: DB pointer 0x555d2df20000 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:49:06.231 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 90.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 90.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T14:49:06.232 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 28.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x555d2de45090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:49:06.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: starting mon.a rank 0 at public addrs v1:172.21.15.28:6789/0 at bind addrs v1:172.21.15.28:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???) e1 preinit fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).mds e1 new map 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).mds e1 print_map 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T14:49:06.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 bash[29837]: b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 2024-03-30T14:49:06.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 systemd[1]: Started Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:49:06.235 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: monmap epoch 1 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: last_changed 2024-03-30T14:48:46.261152+0000 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: min_mon_release 19 (squid) 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: election_strategy: 1 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: fsmap 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T14:49:06.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:06 smithi028 ceph-mon[29876]: mgrmap e1: no daemons active 2024-03-30T14:49:07.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.712+0000 7f4dfb865700 1 Processor -- start 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.712+0000 7f4dfb865700 1 -- start start 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4dfb865700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4df4109a40 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df9601700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4df4105610 0x7f4df4105a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36812/0 (socket says 172.21.15.28:36812) 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df9601700 1 -- 172.21.15.28:0/890686039 learned_addr learned my addr 172.21.15.28:0/890686039 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2184069225 0 0) 0x7f4df4109a40 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ddc0035b0 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3260417826 0 0) 0x7f4ddc0035b0 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df410ac20 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.713+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4dec001900 con 0x7f4df4105610 2024-03-30T14:49:07.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f4dec0020b0 con 0x7f4df4105610 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4df3fff700 1 -- 172.21.15.28:0/890686039 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4dec005ce0 con 0x7f4df4105610 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4dfb865700 1 -- 172.21.15.28:0/890686039 >> v1:172.21.15.28:6789/0 conn(0x7f4df4105610 legacy=0x7f4df4105a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4dfb865700 1 -- 172.21.15.28:0/890686039 shutdown_connections 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4dfb865700 1 -- 172.21.15.28:0/890686039 >> 172.21.15.28:0/890686039 conn(0x7f4df4100bf0 msgr2=0x7f4df4103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4dfb865700 1 -- 172.21.15.28:0/890686039 shutdown_connections 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.714+0000 7f4dfb865700 1 -- 172.21.15.28:0/890686039 wait complete. 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 librados: starting msgr at 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 librados: starting objecter 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 Processor -- start 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 -- start start 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 librados: setting wanted keys 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 librados: calling monclient init 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4dfb865700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4df419cce0 con 0x7f4df4105610 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4df9601700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4df4105610 0x7f4df419c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36824/0 (socket says 172.21.15.28:36824) 2024-03-30T14:49:07.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4df9601700 1 -- 172.21.15.28:0/4165218489 learned_addr learned my addr 172.21.15.28:0/4165218489 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4df27fc700 10 client.?.objecter ms_handle_connect 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.715+0000 7f4df27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1868525603 0 0) 0x7f4df419cce0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ddc0056e0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2084539155 0 0) 0x7f4df410ac20 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4df419cce0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4dec004bf0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2715276799 0 0) 0x7f4df419cce0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df419ceb0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4df419d1a0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4dfb865700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4df41a0cf0 con 0x7f4df4105610 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4dfb865700 1 librados: init done 2024-03-30T14:49:07.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f4dec0020b0 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4dec0063b0 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4dec0069e0 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f4dec008080 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 10 client.14098.objecter ms_dispatch 0x7f4df4001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.716+0000 7f4df27fc700 20 client.14098.objecter dump_active .. 0 homeless 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.717+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de0005150 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.719+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4ddc0056e0 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.789+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f4de0005440 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4df27fc700 1 -- 172.21.15.28:0/4165218489 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (unknown 534127980 0 0) 0x7f4dec005a20 con 0x7f4df4105610 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 10 librados: watch_flush enter 2024-03-30T14:49:07.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 10 librados: watch_flush exit 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 20 client.14098.objecter shutdown clearing up homeless session... 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 10 client.14098.objecter successfully canceled tick 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 >> v1:172.21.15.28:6789/0 conn(0x7f4df4105610 legacy=0x7f4df419c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 shutdown_connections 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.791+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 >> 172.21.15.28:0/4165218489 conn(0x7f4df4100bf0 msgr2=0x7f4df4102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.792+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 shutdown_connections 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.792+0000 7f4dfb865700 1 -- 172.21.15.28:0/4165218489 wait complete. 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:06.792+0000 7f4dfb865700 1 librados: shutdown 2024-03-30T14:49:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T14:49:07.376 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:49:07.376 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-03-30T14:49:07.376 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-30T14:49:07.376 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-30T14:49:07.625 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T14:49:07.626 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y.service: Unit ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y.service not loaded. 2024-03-30T14:49:07.858 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4.target.wants/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y.service → /etc/systemd/system/ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@.service. 2024-03-30T14:49:07.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4165218489' entity='client.admin' 2024-03-30T14:49:07.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:07 smithi028 systemd[1]: Starting Ceph mgr.y for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:49:08.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:08 smithi028 podman[30117]: 2024-03-30 14:49:07.99402489 +0000 UTC m=+0.017360980 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb 2024-03-30T14:49:09.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:08 smithi028 podman[30117]: 2024-03-30 14:49:08.859748802 +0000 UTC m=+0.883084872 container create e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326) 2024-03-30T14:49:09.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 podman[30117]: 2024-03-30 14:49:09.078536116 +0000 UTC m=+1.101872200 container init e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc) 2024-03-30T14:49:09.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 podman[30117]: 2024-03-30 14:49:09.084155989 +0000 UTC m=+1.107492060 container start e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T14:49:09.195 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 bash[30117]: e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c 2024-03-30T14:49:09.210 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T14:49:09.210 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-30T14:49:09.218 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T14:49:09.218 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:49:09.218 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-30T14:49:09.218 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T14:49:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T14:49:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-30T14:49:09.234 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T14:49:09.234 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:49:09.234 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-30T14:49:09.234 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-30T14:49:09.234 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-03-30T14:49:09.235 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-03-30T14:49:09.467 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 systemd[1]: Started Ceph mgr.y for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:49:09.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:09.676+0000 7f5ae25e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:49:10.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3738300073' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:49:10.177 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:09 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:09.778+0000 7f5ae25e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:49:10.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:10.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T14:49:10.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T14:49:10.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T14:49:10.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:10.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T14:48:49.804783+0000", 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.770+0000 7f9863bea700 1 Processor -- start 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9863bea700 1 -- start start 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9863bea700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f985c079e70 con 0x7f985c1071d0 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9861986700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f985c1071d0 0x7f985c1095c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43804/0 (socket says 172.21.15.28:43804) 2024-03-30T14:49:10.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9861986700 1 -- 172.21.15.28:0/2774288786 learned_addr learned my addr 172.21.15.28:0/2774288786 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402959364 0 0) 0x7f985c079e70 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.771+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98480035b0 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.772+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3612438224 0 0) 0x7f98480035b0 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.772+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c10bcf0 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.772+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9858001580 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.772+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f9858001c60 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.772+0000 7f9860984700 1 -- 172.21.15.28:0/2774288786 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9858005960 con 0x7f985c1071d0 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.773+0000 7f9863bea700 1 -- 172.21.15.28:0/2774288786 >> v1:172.21.15.28:6789/0 conn(0x7f985c1071d0 legacy=0x7f985c1095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.773+0000 7f9863bea700 1 -- 172.21.15.28:0/2774288786 shutdown_connections 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.773+0000 7f9863bea700 1 -- 172.21.15.28:0/2774288786 >> 172.21.15.28:0/2774288786 conn(0x7f985c100ba0 msgr2=0x7f985c103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.773+0000 7f9863bea700 1 -- 172.21.15.28:0/2774288786 shutdown_connections 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.773+0000 7f9863bea700 1 -- 172.21.15.28:0/2774288786 wait complete. 2024-03-30T14:49:10.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 librados: starting msgr at 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 librados: starting objecter 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 Processor -- start 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 -- start start 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 librados: setting wanted keys 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 librados: calling monclient init 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9863bea700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f985c1a04b0 con 0x7f985c1071d0 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9861986700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f985c1071d0 0x7f985c106a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43808/0 (socket says 172.21.15.28:43808) 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9861986700 1 -- 172.21.15.28:0/3738300073 learned_addr learned my addr 172.21.15.28:0/3738300073 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9852ffd700 10 client.?.objecter ms_handle_connect 0x7f985c1071d0 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.774+0000 7f9852ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866696307 0 0) 0x7f985c1a04b0 con 0x7f985c1071d0 2024-03-30T14:49:10.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98480035b0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 80749079 0 0) 0x7f98480035b0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f985c1a04b0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9858001080 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3030283751 0 0) 0x7f985c1a04b0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c1a1690 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f985c1a06e0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9863bea700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f985c1a0c20 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9863bea700 1 librados: init done 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f9858001b30 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.775+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9858005fc0 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9858006570 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f9858007c50 con 0x7f985c1071d0 2024-03-30T14:49:10.517 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 10 client.14102.objecter ms_dispatch 0x7f985c001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9852ffd700 20 client.14102.objecter dump_active .. 0 homeless 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.776+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9840005120 con 0x7f985c1071d0 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.778+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f98580056a0 con 0x7f985c1071d0 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.870+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9840005410 con 0x7f985c1071d0 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.871+0000 7f9852ffd700 1 -- 172.21.15.28:0/3738300073 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 2324091294) 0x7f9858005880 con 0x7f985c1071d0 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 10 librados: watch_flush enter 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 10 librados: watch_flush exit 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 20 client.14102.objecter shutdown clearing up homeless session... 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 10 client.14102.objecter successfully canceled tick 2024-03-30T14:49:10.518 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 >> v1:172.21.15.28:6789/0 conn(0x7f985c1071d0 legacy=0x7f985c106a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 shutdown_connections 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 >> 172.21.15.28:0/3738300073 conn(0x7f985c100ba0 msgr2=0x7f985c191050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 shutdown_connections 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 -- 172.21.15.28:0/3738300073 wait complete. 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:09.872+0000 7f9863bea700 1 librados: shutdown 2024-03-30T14:49:10.519 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-03-30T14:49:10.712 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:10.472+0000 7f5ae25e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:49:10.712 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:10.568+0000 7f5ae25e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:49:10.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:10.711+0000 7f5ae25e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:49:10.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:10.809+0000 7f5ae25e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:49:10.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:10.894+0000 7f5ae25e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:49:11.460 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:11.161+0000 7f5ae25e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:49:11.461 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:11.273+0000 7f5ae25e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:11.461 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:11.362+0000 7f5ae25e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:49:11.461 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:11.460+0000 7f5ae25e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:49:11.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:11.638+0000 7f5ae25e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:49:12.896 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:12.642+0000 7f5ae25e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:49:12.896 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:12.781+0000 7f5ae25e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:49:13.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2802908976' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:49:13.221 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:13.128+0000 7f5ae25e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:49:13.221 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:13.220+0000 7f5ae25e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:49:13.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:13.425+0000 7f5ae25e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T14:49:13.648 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:13.649 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T14:49:13.650 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T14:49:13.658 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T14:49:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T14:48:49.804783+0000", 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.914+0000 7f3351e7a700 1 Processor -- start 2024-03-30T14:49:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.914+0000 7f3351e7a700 1 -- start start 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.914+0000 7f3351e7a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f334c079df0 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334b7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f334c107280 0x7f334c109670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43820/0 (socket says 172.21.15.28:43820) 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334b7fe700 1 -- 172.21.15.28:0/610123909 learned_addr learned my addr 172.21.15.28:0/610123909 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971455806 0 0) 0x7f334c079df0 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33340035b0 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 179427750 0 0) 0x7f33340035b0 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334c10bda0 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3340001580 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f3340001d30 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.915+0000 7f334a7fc700 1 -- 172.21.15.28:0/610123909 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3340005960 con 0x7f334c107280 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 -- 172.21.15.28:0/610123909 >> v1:172.21.15.28:6789/0 conn(0x7f334c107280 legacy=0x7f334c109670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 -- 172.21.15.28:0/610123909 shutdown_connections 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 -- 172.21.15.28:0/610123909 >> 172.21.15.28:0/610123909 conn(0x7f334c100bf0 msgr2=0x7f334c103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 -- 172.21.15.28:0/610123909 shutdown_connections 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 -- 172.21.15.28:0/610123909 wait complete. 2024-03-30T14:49:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.916+0000 7f3351e7a700 1 librados: starting msgr at 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 librados: starting objecter 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 Processor -- start 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 -- start start 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 librados: setting wanted keys 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 librados: calling monclient init 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3351e7a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f334c19ab00 con 0x7f334c107280 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f334b7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f334c107280 0x7f334c19a3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43834/0 (socket says 172.21.15.28:43834) 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f334b7fe700 1 -- 172.21.15.28:0/2802908976 learned_addr learned my addr 172.21.15.28:0/2802908976 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3348ff9700 10 client.?.objecter ms_handle_connect 0x7f334c107280 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3348ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497963480 0 0) 0x7f334c19ab00 con 0x7f334c107280 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.917+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33340056e0 con 0x7f334c107280 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1832938038 0 0) 0x7f334c10bda0 con 0x7f334c107280 2024-03-30T14:49:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f334c19ab00 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3340001080 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3672731283 0 0) 0x7f334c19ab00 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334c19acd0 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f334c19afc0 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3351e7a700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f334c19eb10 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3351e7a700 1 librados: init done 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f3340001c60 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3340006030 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3340006620 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f3340007d10 con 0x7f334c107280 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 10 client.14104.objecter ms_dispatch 0x7f334c001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.918+0000 7f3348ff9700 20 client.14104.objecter dump_active .. 0 homeless 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.919+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3338005120 con 0x7f334c107280 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.921+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f33340056e0 con 0x7f334c107280 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.995+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3338005410 con 0x7f334c107280 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.996+0000 7f3348ff9700 1 -- 172.21.15.28:0/2802908976 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 2027405913) 0x7f3340005710 con 0x7f334c107280 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 10 librados: watch_flush enter 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 10 librados: watch_flush exit 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 20 client.14104.objecter shutdown clearing up homeless session... 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 10 client.14104.objecter successfully canceled tick 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 >> v1:172.21.15.28:6789/0 conn(0x7f334c107280 legacy=0x7f334c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 shutdown_connections 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 >> 172.21.15.28:0/2802908976 conn(0x7f334c100bf0 msgr2=0x7f334c191060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:13.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 shutdown_connections 2024-03-30T14:49:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 -- 172.21.15.28:0/2802908976 wait complete. 2024-03-30T14:49:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:12.997+0000 7f3351e7a700 1 librados: shutdown 2024-03-30T14:49:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-03-30T14:49:13.905 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:13.613+0000 7f5ae25e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:49:14.167 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:13.905+0000 7f5ae25e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:14.167 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:14.166+0000 7f5ae25e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:14.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:14.407+0000 7f5ae25e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:49:15.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:15.229+0000 7f5ae25e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:49:16.094 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:15.809+0000 7f5ae25e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:49:16.094 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:15.911+0000 7f5ae25e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:49:16.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:16.137+0000 7f5ae25e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:49:16.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: Activating manager daemon y 2024-03-30T14:49:16.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: mgrmap e2: y(active, starting, since 0.00343601s) 2024-03-30T14:49:16.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: Manager daemon y is now available 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' 2024-03-30T14:49:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' 2024-03-30T14:49:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:16 smithi028 ceph-mon[29876]: from='mgr.14100 v1:172.21.15.28:0/2388721426' entity='mgr.y' 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T14:49:16.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T14:49:16.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:16.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T14:49:16.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T14:49:16.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T14:49:16.822 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T14:49:16.822 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T14:49:16.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T14:48:49.804783+0000", 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.112+0000 7f0a6e2bb700 1 Processor -- start 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.112+0000 7f0a6e2bb700 1 -- start start 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.112+0000 7f0a6e2bb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0a68079e50 con 0x7f0a681071f0 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.112+0000 7f0a67fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a681071f0 0x7f0a681095e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43840/0 (socket says 172.21.15.28:43840) 2024-03-30T14:49:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.112+0000 7f0a67fff700 1 -- 172.21.15.28:0/1358472229 learned_addr learned my addr 172.21.15.28:0/1358472229 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779379613 0 0) 0x7f0a68079e50 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a540035b0 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1238244764 0 0) 0x7f0a540035b0 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6810bd10 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0a50001580 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f0a50001d30 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.113+0000 7f0a66ffd700 1 -- 172.21.15.28:0/1358472229 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0a50005960 con 0x7f0a681071f0 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/1358472229 >> v1:172.21.15.28:6789/0 conn(0x7f0a681071f0 legacy=0x7f0a681095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/1358472229 shutdown_connections 2024-03-30T14:49:16.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/1358472229 >> 172.21.15.28:0/1358472229 conn(0x7f0a68100ba0 msgr2=0x7f0a68103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/1358472229 shutdown_connections 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/1358472229 wait complete. 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 librados: starting msgr at 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 librados: starting objecter 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 Processor -- start 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 -- start start 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 librados: setting wanted keys 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.114+0000 7f0a6e2bb700 1 librados: calling monclient init 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a6e2bb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0a6819ab10 con 0x7f0a681071f0 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a67fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a681071f0 0x7f0a6819a400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43850/0 (socket says 172.21.15.28:43850) 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a67fff700 1 -- 172.21.15.28:0/3764362803 learned_addr learned my addr 172.21.15.28:0/3764362803 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:16.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 10 client.?.objecter ms_handle_connect 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973503492 0 0) 0x7f0a6819ab10 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a540056e0 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 519946237 0 0) 0x7f0a6810bd10 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a6819ab10 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0a50001080 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1635415131 0 0) 0x7f0a6819ab10 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6819ace0 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.115+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6819afd0 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a6e2bb700 10 client.14106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a6819eb20 con 0x7f0a681071f0 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a6e2bb700 1 librados: init done 2024-03-30T14:49:16.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f0a50004ab0 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0a50006380 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a50006970 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f0a50007d50 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 10 client.14106.objecter ms_dispatch 0x7f0a68001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 3 client.14106.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 3 client.14106.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a657fa700 20 client.14106.objecter dump_active .. 0 homeless 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.116+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a48005120 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.119+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a540056e0 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.140+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mgrmap(e 2) ==== 45872+0+0 (unknown 4274141082 0 0) 0x7f0a50005ce0 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.201+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0a48005410 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.201+0000 7f0a657fa700 1 -- 172.21.15.28:0/3764362803 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (unknown 4201413639 0 2590913569) 0x7f0a500175d0 con 0x7f0a681071f0 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.202+0000 7f0a6e2bb700 10 librados: watch_flush enter 2024-03-30T14:49:16.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.202+0000 7f0a6e2bb700 10 librados: watch_flush exit 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.202+0000 7f0a6e2bb700 20 client.14106.objecter shutdown clearing up homeless session... 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.202+0000 7f0a6e2bb700 10 client.14106.objecter successfully canceled tick 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 >> v1:172.21.15.28:6789/0 conn(0x7f0a681071f0 legacy=0x7f0a6819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 shutdown_connections 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 >> 172.21.15.28:0/3764362803 conn(0x7f0a68100ba0 msgr2=0x7f0a68191030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 shutdown_connections 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 -- 172.21.15.28:0/3764362803 wait complete. 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:16.203+0000 7f0a6e2bb700 1 librados: shutdown 2024-03-30T14:49:16.830 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-03-30T14:49:17.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3764362803' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:49:17.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:17 smithi028 ceph-mon[29876]: mgrmap e3: y(active, since 1.0076s) 2024-03-30T14:49:19.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:19 smithi028 ceph-mon[29876]: mgrmap e4: y(active, since 2s) 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:20.160 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T14:49:20.161 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T14:49:20.161 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T14:49:20.171 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T14:49:20.172 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T14:48:49.804783+0000", 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-30T14:49:20.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f0222b47700 1 Processor -- start 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f0222b47700 1 -- start start 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f0222b47700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f021c109aa0 con 0x7f021c105670 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f02208e3700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f021c105670 0x7f021c105a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56240/0 (socket says 172.21.15.28:56240) 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f02208e3700 1 -- 172.21.15.28:0/497982659 learned_addr learned my addr 172.21.15.28:0/497982659 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3340242211 0 0) 0x7f021c109aa0 con 0x7f021c105670 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.332+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f020c0035b0 con 0x7f021c105670 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1315666277 0 0) 0x7f020c0035b0 con 0x7f021c105670 2024-03-30T14:49:20.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021c10ac80 con 0x7f021c105670 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0210001580 con 0x7f021c105670 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f0210001d30 con 0x7f021c105670 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f021b7fe700 1 -- 172.21.15.28:0/497982659 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0210005960 con 0x7f021c105670 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f0222b47700 1 -- 172.21.15.28:0/497982659 >> v1:172.21.15.28:6789/0 conn(0x7f021c105670 legacy=0x7f021c105a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f0222b47700 1 -- 172.21.15.28:0/497982659 shutdown_connections 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f0222b47700 1 -- 172.21.15.28:0/497982659 >> 172.21.15.28:0/497982659 conn(0x7f021c100c90 msgr2=0x7f021c1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f0222b47700 1 -- 172.21.15.28:0/497982659 shutdown_connections 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.333+0000 7f0222b47700 1 -- 172.21.15.28:0/497982659 wait complete. 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 librados: starting msgr at 2024-03-30T14:49:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 librados: starting objecter 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 Processor -- start 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 -- start start 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 librados: setting wanted keys 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 librados: calling monclient init 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0222b47700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f021c079a70 con 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f02208e3700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f021c105670 0x7f021c19d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56244/0 (socket says 172.21.15.28:56244) 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f02208e3700 1 -- 172.21.15.28:0/1601226182 learned_addr learned my addr 172.21.15.28:0/1601226182 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0219ffb700 10 client.?.objecter ms_handle_connect 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.334+0000 7f0219ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078091020 0 0) 0x7f021c079a70 con 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f020c0056e0 con 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2909050455 0 0) 0x7f021c10ac80 con 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f021c079a70 con 0x7f021c105670 2024-03-30T14:49:20.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0210004870 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1092733537 0 0) 0x7f021c079a70 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021c079c40 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f021c19e840 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f0210004a40 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f0210006350 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0222b47700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f021c19ebe0 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.335+0000 7f0222b47700 1 librados: init done 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 4) ==== 46070+0+0 (unknown 3787787261 0 0) 0x7f0210006510 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f02100483e0 con 0x7f021c105670 2024-03-30T14:49:20.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 10 client.14114.objecter ms_dispatch 0x7f021c001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 20 client.14114.objecter dump_active .. 0 homeless 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021c063870 con 0x7f021c105670 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.336+0000 7f0219ffb700 10 client.14114.objecter ms_handle_connect 0x7f020c03b9c0 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.341+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f020c0056e0 con 0x7f021c105670 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.627+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f021c192790 con 0x7f021c105670 2024-03-30T14:49:20.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.627+0000 7f0219ffb700 1 -- 172.21.15.28:0/1601226182 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 2958062076) 0x7f02100174e0 con 0x7f021c105670 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 10 librados: watch_flush enter 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 10 librados: watch_flush exit 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 20 client.14114.objecter shutdown clearing up homeless session... 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 10 client.14114.objecter successfully canceled tick 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 >> v1:172.21.15.28:6800/997592180 conn(0x7f020c03b9c0 legacy=0x7f020c03de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 >> v1:172.21.15.28:6789/0 conn(0x7f021c105670 legacy=0x7f021c19d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 shutdown_connections 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 >> 172.21.15.28:0/1601226182 conn(0x7f021c100c90 msgr2=0x7f021c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 shutdown_connections 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 -- 172.21.15.28:0/1601226182 wait complete. 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:19.629+0000 7f0222b47700 1 librados: shutdown 2024-03-30T14:49:20.179 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-03-30T14:49:20.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1601226182' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T14:49:21.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/878881185' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T14:49:21.530 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T14:49:21.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.650+0000 7f3595874700 1 Processor -- start 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.650+0000 7f3595874700 1 -- start start 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.650+0000 7f3595874700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3590109b30 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358effd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3590106d50 0x7f3590107170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56256/0 (socket says 172.21.15.28:56256) 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358effd700 1 -- 172.21.15.28:0/4171643910 learned_addr learned my addr 172.21.15.28:0/4171643910 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27161892 0 0) 0x7f3590109b30 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35780035b0 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240751 0 0) 0x7f35780035b0 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359010ad10 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3584001580 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f3584001c60 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.651+0000 7f358dffb700 1 -- 172.21.15.28:0/4171643910 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3584005960 con 0x7f3590106d50 2024-03-30T14:49:21.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 -- 172.21.15.28:0/4171643910 >> v1:172.21.15.28:6789/0 conn(0x7f3590106d50 legacy=0x7f3590107170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 -- 172.21.15.28:0/4171643910 shutdown_connections 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 -- 172.21.15.28:0/4171643910 >> 172.21.15.28:0/4171643910 conn(0x7f3590100c90 msgr2=0x7f35901030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 -- 172.21.15.28:0/4171643910 shutdown_connections 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 -- 172.21.15.28:0/4171643910 wait complete. 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 librados: starting msgr at 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 librados: starting objecter 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.652+0000 7f3595874700 1 Processor -- start 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3595874700 1 -- start start 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3595874700 1 librados: setting wanted keys 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3595874700 1 librados: calling monclient init 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3595874700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f359019acb0 con 0x7f3590106d50 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f358effd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3590106d50 0x7f359019a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56272/0 (socket says 172.21.15.28:56272) 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f358effd700 1 -- 172.21.15.28:0/878881185 learned_addr learned my addr 172.21.15.28:0/878881185 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:21.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 10 client.?.objecter ms_handle_connect 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528477165 0 0) 0x7f359019acb0 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35780035b0 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2507670024 0 0) 0x7f35780035b0 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f359019acb0 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.653+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3584001080 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2254330286 0 0) 0x7f359019acb0 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359019ae80 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f359019b170 con 0x7f3590106d50 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3595874700 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:21.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f359019ecc0 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3595874700 1 librados: init done 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f3584001c20 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3584005ff0 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.654+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 4) ==== 46070+0+0 (unknown 3787787261 0 0) 0x7f3584011610 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7f35840474b0 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 10 client.14116.objecter ms_dispatch 0x7f3590001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f357c005150 con 0x7f3590106d50 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 20 client.14116.objecter dump_active .. 0 homeless 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.655+0000 7f3594872700 10 client.14116.objecter ms_handle_connect 0x7f357803bb40 2024-03-30T14:49:21.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.660+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f35840174f0 con 0x7f3590106d50 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.860+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f357c002430 con 0x7f3590106d50 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.861+0000 7f3594872700 1 -- 172.21.15.28:0/878881185 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (unknown 1187553405 0 3640020501) 0x7f3584016df0 con 0x7f3590106d50 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 10 librados: watch_flush enter 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 10 librados: watch_flush exit 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 20 client.14116.objecter shutdown clearing up homeless session... 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 10 client.14116.objecter successfully canceled tick 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 >> v1:172.21.15.28:6800/997592180 conn(0x7f357803bb40 legacy=0x7f357803e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 >> v1:172.21.15.28:6789/0 conn(0x7f3590106d50 legacy=0x7f359019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 shutdown_connections 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 >> 172.21.15.28:0/878881185 conn(0x7f3590100c90 msgr2=0x7f3590078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 shutdown_connections 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 -- 172.21.15.28:0/878881185 wait complete. 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:20.863+0000 7f3595874700 1 librados: shutdown 2024-03-30T14:49:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-03-30T14:49:22.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/657846862' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T14:49:22.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setuser ceph since I am not root 2024-03-30T14:49:22.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setgroup ceph since I am not root 2024-03-30T14:49:22.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.904+0000 7fdd262c8700 1 Processor -- start 2024-03-30T14:49:22.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.904+0000 7fdd262c8700 1 -- start start 2024-03-30T14:49:22.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd262c8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd2010aba0 con 0x7fdd20106770 2024-03-30T14:49:22.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd20106770 0x7fdd20106b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56278/0 (socket says 172.21.15.28:56278) 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1ffff700 1 -- 172.21.15.28:0/923172559 learned_addr learned my addr 172.21.15.28:0/923172559 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722897300 0 0) 0x7fdd2010aba0 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd080035b0 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2875002301 0 0) 0x7fdd080035b0 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2010bd80 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdd10001580 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fdd10001d30 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.905+0000 7fdd1effd700 1 -- 172.21.15.28:0/923172559 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdd10005960 con 0x7fdd20106770 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 -- 172.21.15.28:0/923172559 >> v1:172.21.15.28:6789/0 conn(0x7fdd20106770 legacy=0x7fdd20106b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 -- 172.21.15.28:0/923172559 shutdown_connections 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 -- 172.21.15.28:0/923172559 >> 172.21.15.28:0/923172559 conn(0x7fdd20101d70 msgr2=0x7fdd20104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:22.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 -- 172.21.15.28:0/923172559 shutdown_connections 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 -- 172.21.15.28:0/923172559 wait complete. 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 librados: starting msgr at 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.906+0000 7fdd262c8700 1 librados: starting objecter 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd262c8700 1 Processor -- start 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd262c8700 1 -- start start 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd262c8700 1 librados: setting wanted keys 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd262c8700 1 librados: calling monclient init 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd262c8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd2019aa90 con 0x7fdd20106770 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd20106770 0x7fdd2019a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56292/0 (socket says 172.21.15.28:56292) 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1ffff700 1 -- 172.21.15.28:0/657846862 learned_addr learned my addr 172.21.15.28:0/657846862 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1d7fa700 10 client.?.objecter ms_handle_connect 0x7fdd20106770 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843806624 0 0) 0x7fdd2019aa90 con 0x7fdd20106770 2024-03-30T14:49:22.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.907+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd080056e0 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1283943035 0 0) 0x7fdd2010bd80 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd2019aa90 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdd10004870 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3177871981 0 0) 0x7fdd2019aa90 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2019ac60 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2019af50 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd262c8700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd2019eaa0 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd262c8700 1 librados: init done 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fdd10001d30 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.908+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdd10006030 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 4) ==== 46070+0+0 (unknown 3787787261 0 0) 0x7fdd100116b0 con 0x7fdd20106770 2024-03-30T14:49:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd0c005150 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7fdd10047530 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 10 client.14118.objecter ms_dispatch 0x7fdd20001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 20 client.14118.objecter dump_active .. 0 homeless 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.909+0000 7fdd1d7fa700 10 client.14118.objecter ms_handle_connect 0x7fdd0803b980 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:21.914+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdd080056e0 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.178+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fdd0c005440 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.195+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (unknown 2263024820 0 0) 0x7fdd100175a0 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.197+0000 7fdd1d7fa700 1 -- 172.21.15.28:0/657846862 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mgrmap(e 5) ==== 46081+0+0 (unknown 306405295 0 0) 0x7fdd10045f20 con 0x7fdd20106770 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 10 librados: watch_flush enter 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 10 librados: watch_flush exit 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 20 client.14118.objecter shutdown clearing up homeless session... 2024-03-30T14:49:22.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 10 client.14118.objecter successfully canceled tick 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 >> v1:172.21.15.28:6800/997592180 conn(0x7fdd0803b980 legacy=0x7fdd0803de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 >> v1:172.21.15.28:6789/0 conn(0x7fdd20106770 legacy=0x7fdd2019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 shutdown_connections 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 >> 172.21.15.28:0/657846862 conn(0x7fdd20101d70 msgr2=0x7fdd20104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.198+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 shutdown_connections 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.199+0000 7fdd262c8700 1 -- 172.21.15.28:0/657846862 wait complete. 2024-03-30T14:49:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:22.199+0000 7fdd262c8700 1 librados: shutdown 2024-03-30T14:49:23.052 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:22.770+0000 7fda391a0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:49:23.052 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:22.870+0000 7fda391a0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:49:23.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/657846862' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T14:49:23.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-mon[29876]: mgrmap e5: y(active, since 6s) 2024-03-30T14:49:23.571 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:23.540+0000 7fda391a0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:49:23.883 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:23.639+0000 7fda391a0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:49:23.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:23.784+0000 7fda391a0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:49:23.884 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:23.883+0000 7fda391a0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:49:24.164 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:23.969+0000 7fda391a0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.296+0000 7ff4a91df700 1 Processor -- start 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.296+0000 7ff4a91df700 1 -- start start 2024-03-30T14:49:24.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.296+0000 7ff4a91df700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4a410ab50 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.296+0000 7ff4a2d9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4a4106720 0x7ff4a4106b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56326/0 (socket says 172.21.15.28:56326) 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.296+0000 7ff4a2d9d700 1 -- 172.21.15.28:0/2168697611 learned_addr learned my addr 172.21.15.28:0/2168697611 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641989384 0 0) 0x7ff4a410ab50 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff48c0035b0 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2940259265 0 0) 0x7ff48c0035b0 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a410bd30 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff498001580 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7ff498001d30 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.297+0000 7ff4a1d9b700 1 -- 172.21.15.28:0/2168697611 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff498005960 con 0x7ff4a4106720 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 -- 172.21.15.28:0/2168697611 >> v1:172.21.15.28:6789/0 conn(0x7ff4a4106720 legacy=0x7ff4a4106b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 -- 172.21.15.28:0/2168697611 shutdown_connections 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 -- 172.21.15.28:0/2168697611 >> 172.21.15.28:0/2168697611 conn(0x7ff4a4101ca0 msgr2=0x7ff4a4104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 -- 172.21.15.28:0/2168697611 shutdown_connections 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 -- 172.21.15.28:0/2168697611 wait complete. 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 librados: starting msgr at 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.298+0000 7ff4a91df700 1 librados: starting objecter 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a91df700 1 Processor -- start 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a91df700 1 -- start start 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a91df700 1 librados: setting wanted keys 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a91df700 1 librados: calling monclient init 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a91df700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4a419cce0 con 0x7ff4a4106720 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a2d9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4a4106720 0x7ff4a419c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56328/0 (socket says 172.21.15.28:56328) 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff4a2d9d700 1 -- 172.21.15.28:0/3580217821 learned_addr learned my addr 172.21.15.28:0/3580217821 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:24.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff48bfff700 10 client.?.objecter ms_handle_connect 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.299+0000 7ff48bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274534155 0 0) 0x7ff4a419cce0 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff48c0056e0 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3887450706 0 0) 0x7ff4a410bd30 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4a419cce0 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff498004870 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 104798479 0 0) 0x7ff4a419cce0 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a419ceb0 con 0x7ff4a4106720 2024-03-30T14:49:24.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a419d1a0 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff4a91df700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4a41a0cf0 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff4a91df700 1 librados: init done 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7ff498001be0 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.300+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff498005fc0 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 5) ==== 46081+0+0 (unknown 306405295 0 0) 0x7ff498011570 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff490005120 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7ff4980473f0 con 0x7ff4a4106720 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 10 client.14122.objecter ms_dispatch 0x7ff4a4001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff4a259c700 1 -- 172.21.15.28:0/3580217821 >> v1:172.21.15.28:6800/997592180 conn(0x7ff48c03bac0 legacy=0x7ff48c03df80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:24.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.301+0000 7ff48bfff700 20 client.14122.objecter dump_active .. 0 homeless 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.306+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff48c0056e0 con 0x7ff4a4106720 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.502+0000 7ff4a259c700 1 -- 172.21.15.28:0/3580217821 >> v1:172.21.15.28:6800/997592180 conn(0x7ff48c03bac0 legacy=0x7ff48c03df80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.570+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ff490005c20 con 0x7ff4a4106720 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.571+0000 7ff48bfff700 1 -- 172.21.15.28:0/3580217821 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7ff498017400 con 0x7ff4a4106720 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 10 librados: watch_flush enter 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 10 librados: watch_flush exit 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 20 client.14122.objecter shutdown clearing up homeless session... 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 10 client.14122.objecter successfully canceled tick 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 >> v1:172.21.15.28:6800/997592180 conn(0x7ff48c03bac0 legacy=0x7ff48c03df80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 >> v1:172.21.15.28:6789/0 conn(0x7ff4a4106720 legacy=0x7ff4a419c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 shutdown_connections 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 >> 172.21.15.28:0/3580217821 conn(0x7ff4a4101ca0 msgr2=0x7ff4a41020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 shutdown_connections 2024-03-30T14:49:24.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 -- 172.21.15.28:0/3580217821 wait complete. 2024-03-30T14:49:24.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:23.573+0000 7ff4a91df700 1 librados: shutdown 2024-03-30T14:49:24.180 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-03-30T14:49:24.180 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-03-30T14:49:24.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3580217821' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T14:49:24.424 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:24.248+0000 7fda391a0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:49:24.424 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:24.362+0000 7fda391a0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:24.681 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:24.455+0000 7fda391a0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:49:24.681 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:24.552+0000 7fda391a0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:49:24.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:24 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:24.734+0000 7fda391a0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:49:25.973 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:25 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:25.720+0000 7fda391a0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:49:25.974 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:25 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:25.854+0000 7fda391a0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:49:26.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:26.193+0000 7fda391a0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:49:26.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:26.284+0000 7fda391a0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:49:26.966 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:26.483+0000 7fda391a0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:26.966 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:26.673+0000 7fda391a0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:49:27.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:26.964+0000 7fda391a0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:27.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:27.227+0000 7fda391a0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:27.690 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:27.469+0000 7fda391a0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:49:28.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:28.297+0000 7fda391a0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:49:29.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:28.909+0000 7fda391a0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:49:29.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:29.012+0000 7fda391a0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:49:29.685 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:29.235+0000 7fda391a0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: Active manager daemon y restarted 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: Activating manager daemon y 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: mgrmap e6: y(active, starting, since 0.00318611s) 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: Manager daemon y is now available 2024-03-30T14:49:29.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:29.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:29.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:29.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:29.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:49:29.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:29 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.680+0000 7ff6b4d27700 1 Processor -- start 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.680+0000 7ff6b4d27700 1 -- start start 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.680+0000 7ff6b4d27700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6b0079e10 con 0x7ff6b0105000 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ae59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff6b0105000 0x7ff6b0105420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56336/0 (socket says 172.21.15.28:56336) 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ae59c700 1 -- 172.21.15.28:0/3665295798 learned_addr learned my addr 172.21.15.28:0/3665295798 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060109701 0 0) 0x7ff6b0079e10 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff69c0035b0 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2867618301 0 0) 0x7ff69c0035b0 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b0106b40 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff698001580 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7ff698001d30 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.681+0000 7ff6ad59a700 1 -- 172.21.15.28:0/3665295798 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff698005960 con 0x7ff6b0105000 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.682+0000 7ff6b4d27700 1 -- 172.21.15.28:0/3665295798 >> v1:172.21.15.28:6789/0 conn(0x7ff6b0105000 legacy=0x7ff6b0105420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.682+0000 7ff6b4d27700 1 -- 172.21.15.28:0/3665295798 shutdown_connections 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.682+0000 7ff6b4d27700 1 -- 172.21.15.28:0/3665295798 >> 172.21.15.28:0/3665295798 conn(0x7ff6b0100bc0 msgr2=0x7ff6b0102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.682+0000 7ff6b4d27700 1 -- 172.21.15.28:0/3665295798 shutdown_connections 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.682+0000 7ff6b4d27700 1 -- 172.21.15.28:0/3665295798 wait complete. 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 librados: starting msgr at 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 librados: starting objecter 2024-03-30T14:49:30.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 Processor -- start 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 -- start start 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 librados: setting wanted keys 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 librados: calling monclient init 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6b4d27700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6b019cd20 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6ae59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff6b0105000 0x7ff6b019c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56340/0 (socket says 172.21.15.28:56340) 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6ae59c700 1 -- 172.21.15.28:0/2311192516 learned_addr learned my addr 172.21.15.28:0/2311192516 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6a77fe700 10 client.?.objecter ms_handle_connect 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.683+0000 7ff6a77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096393551 0 0) 0x7ff6b019cd20 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff69c0056e0 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2648997196 0 0) 0x7ff6b0106b40 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6b019cd20 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff698001080 con 0x7ff6b0105000 2024-03-30T14:49:30.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2426583530 0 0) 0x7ff6b019cd20 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b019cef0 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b019d1e0 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6b4d27700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6b01a0ce0 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6b4d27700 1 librados: init done 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7ff698001c60 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.684+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ff698006070 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 5) ==== 46081+0+0 (unknown 306405295 0 0) 0x7ff698006290 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6800/997592180 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff698001080 con 0x7ff69c03bbe0 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6add9b700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3878218402 0 0) 0x7ff6980484c0 con 0x7ff6b0105000 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 10 client.14124.objecter ms_dispatch 0x7ff6b0001550 osd_map(1..1 src has 1..1) 2024-03-30T14:49:30.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 3 client.14124.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 3 client.14124.objecter handle_osd_map decoding full epoch 1 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.685+0000 7ff6a77fe700 20 client.14124.objecter dump_active .. 0 homeless 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:24.886+0000 7ff6add9b700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:25.287+0000 7ff6add9b700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:26.088+0000 7ff6add9b700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:27.688+0000 7ff6add9b700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/997592180 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:29.239+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mgrmap(e 6) ==== 45883+0+0 (unknown 27629458 0 0) 0x7ff698013560 con 0x7ff6b0105000 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:29.239+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/997592180 conn(0x7ff69c03bbe0 legacy=0x7ff69c03e0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:29.684+0000 7ff6affff700 10 client.14124.objecter tick 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.240+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mgrmap(e 7) ==== 45975+0+0 (unknown 1388300821 0 0) 0x7ff6980490c0 con 0x7ff6b0105000 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.240+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6800/2767265670 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff698001080 con 0x7ff69c040450 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.241+0000 7ff6a77fe700 10 client.14124.objecter ms_handle_connect 0x7ff69c040450 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.242+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7ff698001080 con 0x7ff69c040450 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.248+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 --> v1:172.21.15.28:6800/2767265670 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff690002630 con 0x7ff69c040450 2024-03-30T14:49:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.248+0000 7ff6a77fe700 1 -- 172.21.15.28:0/2311192516 <== mgr.14120 v1:172.21.15.28:6800/2767265670 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (unknown 0 0 96372106) 0x7ff690002630 con 0x7ff69c040450 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 10 librados: watch_flush enter 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 10 librados: watch_flush exit 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 20 client.14124.objecter shutdown clearing up homeless session... 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 10 client.14124.objecter successfully canceled tick 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6800/2767265670 conn(0x7ff69c040450 legacy=0x7ff69c042840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 >> v1:172.21.15.28:6789/0 conn(0x7ff6b0105000 legacy=0x7ff6b019c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 shutdown_connections 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 >> 172.21.15.28:0/2311192516 conn(0x7ff6b0100bc0 msgr2=0x7ff6b0193210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 shutdown_connections 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 -- 172.21.15.28:0/2311192516 wait complete. 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:30.249+0000 7ff6b4d27700 1 librados: shutdown 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-03-30T14:49:30.852 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-03-30T14:49:31.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:31 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:31.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:31 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:31.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:31 smithi028 ceph-mon[29876]: mgrmap e7: y(active, since 1.00541s) 2024-03-30T14:49:31.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:31 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:32.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6dac735700 1 Processor -- start 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6dac735700 1 -- start start 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6dac735700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6da410ab80 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6daa4d1700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6da4106750 0x7f6da4106b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50132/0 (socket says 172.21.15.28:50132) 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6daa4d1700 1 -- 172.21.15.28:0/247744459 learned_addr learned my addr 172.21.15.28:0/247744459 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416588951 0 0) 0x7f6da410ab80 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d8c0035b0 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3906121616 0 0) 0x7f6d8c0035b0 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da410bd60 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6d9c001580 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f6d9c001d30 con 0x7f6da4106750 2024-03-30T14:49:32.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.386+0000 7f6da94cf700 1 -- 172.21.15.28:0/247744459 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6d9c005960 con 0x7f6da4106750 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- 172.21.15.28:0/247744459 >> v1:172.21.15.28:6789/0 conn(0x7f6da4106750 legacy=0x7f6da4106b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- 172.21.15.28:0/247744459 shutdown_connections 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- 172.21.15.28:0/247744459 >> 172.21.15.28:0/247744459 conn(0x7f6da4101d50 msgr2=0x7f6da4104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- 172.21.15.28:0/247744459 shutdown_connections 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- 172.21.15.28:0/247744459 wait complete. 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 librados: starting msgr at 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 librados: starting objecter 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 Processor -- start 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 -- start start 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 librados: setting wanted keys 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.387+0000 7f6dac735700 1 librados: calling monclient init 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6dac735700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6da419aac0 con 0x7f6da4106750 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6daa4d1700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6da4106750 0x7f6da419a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50142/0 (socket says 172.21.15.28:50142) 2024-03-30T14:49:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6daa4d1700 1 -- 172.21.15.28:0/2070778332 learned_addr learned my addr 172.21.15.28:0/2070778332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 10 client.?.objecter ms_handle_connect 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035663513 0 0) 0x7f6da419aac0 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d8c0056e0 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3023560510 0 0) 0x7f6da410bd60 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6da419aac0 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6d9c004870 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3327497898 0 0) 0x7f6da419aac0 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da419ac90 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6da419af80 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6dac735700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6da419ead0 con 0x7f6da4106750 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.388+0000 7f6dac735700 1 librados: init done 2024-03-30T14:49:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f6d9c001be0 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6d9c005fc0 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 7) ==== 45975+0+0 (unknown 1388300821 0 0) 0x7f6d9c006160 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d90005150 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f6d9c047180 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 10 client.14132.objecter ms_dispatch 0x7f6da4001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.389+0000 7f6da37fe700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.390+0000 7f6da37fe700 20 client.14132.objecter dump_active .. 0 homeless 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.390+0000 7f6da37fe700 10 client.14132.objecter ms_handle_connect 0x7f6d8c03b9b0 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.395+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6d8c0056e0 con 0x7f6da4106750 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.612+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6d90000bc0 con 0x7f6d8c03b9b0 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.618+0000 7f6da37fe700 1 -- 172.21.15.28:0/2070778332 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f6d90000bc0 con 0x7f6d8c03b9b0 2024-03-30T14:49:32.214 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 10 librados: watch_flush enter 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 10 librados: watch_flush exit 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 20 client.14132.objecter shutdown clearing up homeless session... 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 10 client.14132.objecter successfully canceled tick 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 >> v1:172.21.15.28:6800/2767265670 conn(0x7f6d8c03b9b0 legacy=0x7f6d8c03de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 >> v1:172.21.15.28:6789/0 conn(0x7f6da4106750 legacy=0x7f6da419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 shutdown_connections 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 >> 172.21.15.28:0/2070778332 conn(0x7f6da4101d50 msgr2=0x7f6da4104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.620+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 shutdown_connections 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.621+0000 7f6dac735700 1 -- 172.21.15.28:0/2070778332 wait complete. 2024-03-30T14:49:32.215 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:31.621+0000 7f6dac735700 1 librados: shutdown 2024-03-30T14:49:32.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:32 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:32.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:32 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:32.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:32 smithi028 ceph-mon[29876]: mgrmap e8: y(active, since 2s) 2024-03-30T14:49:33.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac950ef700 1 Processor -- start 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac950ef700 1 -- start start 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac950ef700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fac9010ac20 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac8ed9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fac901067f0 0x7fac90106c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50154/0 (socket says 172.21.15.28:50154) 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac8ed9d700 1 -- 172.21.15.28:0/2152334845 learned_addr learned my addr 172.21.15.28:0/2152334845 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.687+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850191896 0 0) 0x7fac9010ac20 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac780035b0 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2121178106 0 0) 0x7fac780035b0 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9010be00 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fac80001580 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fac80001d30 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac8dd9b700 1 -- 172.21.15.28:0/2152334845 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fac80005960 con 0x7fac901067f0 2024-03-30T14:49:33.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac950ef700 1 -- 172.21.15.28:0/2152334845 >> v1:172.21.15.28:6789/0 conn(0x7fac901067f0 legacy=0x7fac90106c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac950ef700 1 -- 172.21.15.28:0/2152334845 shutdown_connections 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac950ef700 1 -- 172.21.15.28:0/2152334845 >> 172.21.15.28:0/2152334845 conn(0x7fac90101db0 msgr2=0x7fac901041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.688+0000 7fac950ef700 1 -- 172.21.15.28:0/2152334845 shutdown_connections 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 -- 172.21.15.28:0/2152334845 wait complete. 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 librados: starting msgr at 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 librados: starting objecter 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 Processor -- start 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 -- start start 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 librados: setting wanted keys 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.689+0000 7fac950ef700 1 librados: calling monclient init 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac950ef700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fac9019aaf0 con 0x7fac901067f0 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac8ed9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fac901067f0 0x7fac9019a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50158/0 (socket says 172.21.15.28:50158) 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac8ed9d700 1 -- 172.21.15.28:0/1628431981 learned_addr learned my addr 172.21.15.28:0/1628431981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:33.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 10 client.?.objecter ms_handle_connect 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506502717 0 0) 0x7fac9019aaf0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac780056e0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 727249563 0 0) 0x7fac9010be00 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac9019aaf0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fac80004870 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 943205252 0 0) 0x7fac9019aaf0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9019acc0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac9019afb0 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac950ef700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fac9019eb00 con 0x7fac901067f0 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.690+0000 7fac950ef700 1 librados: init done 2024-03-30T14:49:33.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.691+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fac80004a40 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.691+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fac80006230 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.691+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7fac800117e0 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.691+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac7c005150 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.691+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7fac80048390 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.692+0000 7fac77fff700 10 client.14134.objecter ms_dispatch 0x7fac90001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.692+0000 7fac77fff700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.692+0000 7fac77fff700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.692+0000 7fac77fff700 20 client.14134.objecter dump_active .. 0 homeless 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.692+0000 7fac77fff700 10 client.14134.objecter ms_handle_connect 0x7fac7803bac0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.696+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fac780056e0 con 0x7fac901067f0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.902+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fac7c000bc0 con 0x7fac7803bac0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.903+0000 7fac77fff700 1 -- 172.21.15.28:0/1628431981 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (unknown 0 0 2070689548) 0x7fac7c000bc0 con 0x7fac7803bac0 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.904+0000 7fac950ef700 10 librados: watch_flush enter 2024-03-30T14:49:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.904+0000 7fac950ef700 10 librados: watch_flush exit 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.904+0000 7fac950ef700 20 client.14134.objecter shutdown clearing up homeless session... 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.904+0000 7fac950ef700 10 client.14134.objecter successfully canceled tick 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.904+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 >> v1:172.21.15.28:6800/2767265670 conn(0x7fac7803bac0 legacy=0x7fac7803df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 >> v1:172.21.15.28:6789/0 conn(0x7fac901067f0 legacy=0x7fac9019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 shutdown_connections 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 >> 172.21.15.28:0/1628431981 conn(0x7fac90101db0 msgr2=0x7fac901041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 shutdown_connections 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 -- 172.21.15.28:0/1628431981 wait complete. 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:32.905+0000 7fac950ef700 1 librados: shutdown 2024-03-30T14:49:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-03-30T14:49:34.721 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: Generating public/private rsa key pair. 2024-03-30T14:49:34.721 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: Your identification has been saved in /tmp/tmprdn7jl3g/key. 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: Your public key has been saved in /tmp/tmprdn7jl3g/key.pub. 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: The key fingerprint is: 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: SHA256:uHVYkoLY5sZQQM1HJH2p3+Qxy7mrVVaFMQAyyFTmOUQ ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: The key's randomart image is: 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: +---[RSA 3072]----+ 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | .o+o*++E.....oo.| 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | +oo=+o= .o | 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | o +..o* . . | 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | = .o =+ . | 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | + ..S=.=o | 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | . o..*o | 2024-03-30T14:49:34.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | . .. | 2024-03-30T14:49:34.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | .. | 2024-03-30T14:49:34.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: | .... | 2024-03-30T14:49:34.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:34 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: +----[SHA256]-----+ 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b9a475700 1 Processor -- start 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b9a475700 1 -- start start 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b9a475700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b94109b60 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b93fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b94106d30 0x7f8b94107150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50172/0 (socket says 172.21.15.28:50172) 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b93fff700 1 -- 172.21.15.28:0/1038286348 learned_addr learned my addr 172.21.15.28:0/1038286348 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838121065 0 0) 0x7f8b94109b60 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.985+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b7c0035b0 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 559595072 0 0) 0x7f8b7c0035b0 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9410ad40 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f8b84001580 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f8b84001c60 con 0x7f8b94106d30 2024-03-30T14:49:35.048 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b92ffd700 1 -- 172.21.15.28:0/1038286348 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f8b84005960 con 0x7f8b94106d30 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b9a475700 1 -- 172.21.15.28:0/1038286348 >> v1:172.21.15.28:6789/0 conn(0x7f8b94106d30 legacy=0x7f8b94107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b9a475700 1 -- 172.21.15.28:0/1038286348 shutdown_connections 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.986+0000 7f8b9a475700 1 -- 172.21.15.28:0/1038286348 >> 172.21.15.28:0/1038286348 conn(0x7f8b941003c0 msgr2=0x7f8b941027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 -- 172.21.15.28:0/1038286348 shutdown_connections 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 -- 172.21.15.28:0/1038286348 wait complete. 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 librados: starting msgr at 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 librados: starting objecter 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 Processor -- start 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 -- start start 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 librados: setting wanted keys 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 librados: calling monclient init 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b9a475700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b9419aca0 con 0x7f8b94106d30 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.987+0000 7f8b93fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b94106d30 0x7f8b9419a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50188/0 (socket says 172.21.15.28:50188) 2024-03-30T14:49:35.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b93fff700 1 -- 172.21.15.28:0/4246890425 learned_addr learned my addr 172.21.15.28:0/4246890425 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 10 client.?.objecter ms_handle_connect 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575059864 0 0) 0x7f8b9419aca0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b7c0035b0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3414944097 0 0) 0x7f8b7c0035b0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b9419aca0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f8b84001080 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 942691063 0 0) 0x7f8b9419aca0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9419ae70 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9419b160 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b9a475700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b9419ecb0 con 0x7f8b94106d30 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b9a475700 1 librados: init done 2024-03-30T14:49:35.050 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f8b84004a40 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.988+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f8b84006360 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7f8b84011930 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b80005120 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f8b84047660 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 10 client.14136.objecter ms_dispatch 0x7f8b94001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.989+0000 7f8b917fa700 20 client.14136.objecter dump_active .. 0 homeless 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.990+0000 7f8b917fa700 10 client.14136.objecter ms_handle_connect 0x7f8b7c03bbf0 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:33.994+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8b84017590 con 0x7f8b94106d30 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.200+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f8b80000b90 con 0x7f8b7c03bbf0 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.409+0000 7f8b917fa700 1 -- 172.21.15.28:0/4246890425 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f8b80000b90 con 0x7f8b7c03bbf0 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 10 librados: watch_flush enter 2024-03-30T14:49:35.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 10 librados: watch_flush exit 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 20 client.14136.objecter shutdown clearing up homeless session... 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 10 client.14136.objecter successfully canceled tick 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 >> v1:172.21.15.28:6800/2767265670 conn(0x7f8b7c03bbf0 legacy=0x7f8b7c03e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 >> v1:172.21.15.28:6789/0 conn(0x7f8b94106d30 legacy=0x7f8b9419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 shutdown_connections 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 >> 172.21.15.28:0/4246890425 conn(0x7f8b941003c0 msgr2=0x7f8b941047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 shutdown_connections 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 -- 172.21.15.28:0/4246890425 wait complete. 2024-03-30T14:49:35.052 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:34.411+0000 7f8b9a475700 1 librados: shutdown 2024-03-30T14:49:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:35 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:35 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:36.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxMskQz6JNzE/jOBSeOsI0Mr9Vj8FV1A05w4Hcks2ntX5JqbQwclfaVjaVF3ZJ3VuRbtLX2xs8ipZ23NBHQoObo+SSyHq1J7qeFvnRuK/5auYeveu27nnaie8py2tjRqLdkHd4kwerVUDdcUtu9fpsQAO7r4pKyF/7kMtdyytL8ef0dxOCSHqNsH1EQAdw8oos4u6R/8r9YNxMlhEWxPwC+Y+fQWy6EKAxYJ3fFfQhXb7QgRtXxPXmJ25FxDOkaVm6mv2gyayZJjM2eNf3AHCRaUKwAxHHRdg9lDm509xWcT5T0fSpB7/iO9XekYwcPONiFNVLFamf5zNPaVXa9Huy7n7SBiBy+YE0o2TsGPhjd2Mjk9JzINGmoN1zIlbLwWP5yJdThdA33Ke+BgrTOeyDxNHZFSVJ2C2mixoB0vhebX+F0UNcS55PWnCHQ2xm9GXD9tEbcLFt6Prq3GbsQhyXYUdjXk4RgCzF6owWicJnaTDGOvBCqgnKILUIuk08X3U= ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:49:36.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9d17fa700 1 Processor -- start 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9d17fa700 1 -- start start 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9d17fa700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9cc10abf0 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9caffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9cc104a30 0x7fd9cc104e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50198/0 (socket says 172.21.15.28:50198) 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9caffd700 1 -- 172.21.15.28:0/1624783429 learned_addr learned my addr 172.21.15.28:0/1624783429 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700618875 0 0) 0x7fd9cc10abf0 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b40035b0 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.525+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3634057602 0 0) 0x7fd9b40035b0 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc10bdd0 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd9bc001580 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fd9bc001d30 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9c9ffb700 1 -- 172.21.15.28:0/1624783429 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd9bc005960 con 0x7fd9cc104a30 2024-03-30T14:49:36.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1624783429 >> v1:172.21.15.28:6789/0 conn(0x7fd9cc104a30 legacy=0x7fd9cc104e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1624783429 shutdown_connections 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1624783429 >> 172.21.15.28:0/1624783429 conn(0x7fd9cc100360 msgr2=0x7fd9cc1027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1624783429 shutdown_connections 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.526+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1624783429 wait complete. 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 librados: starting msgr at 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 librados: starting objecter 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 Processor -- start 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 -- start start 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 librados: setting wanted keys 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 librados: calling monclient init 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9d17fa700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9cc19ab30 con 0x7fd9cc104a30 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9caffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9cc104a30 0x7fd9cc19a420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50214/0 (socket says 172.21.15.28:50214) 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9caffd700 1 -- 172.21.15.28:0/1837140590 learned_addr learned my addr 172.21.15.28:0/1837140590 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9b3fff700 10 client.?.objecter ms_handle_connect 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.527+0000 7fd9b3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915841680 0 0) 0x7fd9cc19ab30 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b40056e0 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941854807 0 0) 0x7fd9cc10bdd0 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9cc19ab30 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd9bc001080 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2966808915 0 0) 0x7fd9cc19ab30 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc19ad00 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc19aff0 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9d17fa700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9cc19eb40 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7fd9bc001b80 con 0x7fd9cc104a30 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9d17fa700 1 librados: init done 2024-03-30T14:49:36.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.528+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd9bc006200 con 0x7fd9cc104a30 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7fd9bc011890 con 0x7fd9cc104a30 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9cc063870 con 0x7fd9cc104a30 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7fd9bc047550 con 0x7fd9cc104a30 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 10 client.14138.objecter ms_dispatch 0x7fd9cc001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 20 client.14138.objecter dump_active .. 0 homeless 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.529+0000 7fd9b3fff700 10 client.14138.objecter ms_handle_connect 0x7fd9b403bc10 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.534+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd9b40056e0 con 0x7fd9cc104a30 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.740+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fd9cc02dce0 con 0x7fd9b403bc10 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.741+0000 7fd9b3fff700 1 -- 172.21.15.28:0/1837140590 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (unknown 0 0 1159999738) 0x7fd9cc02dce0 con 0x7fd9b403bc10 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 10 librados: watch_flush enter 2024-03-30T14:49:36.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 10 librados: watch_flush exit 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 20 client.14138.objecter shutdown clearing up homeless session... 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 10 client.14138.objecter successfully canceled tick 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 >> v1:172.21.15.28:6800/2767265670 conn(0x7fd9b403bc10 legacy=0x7fd9b403e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 >> v1:172.21.15.28:6789/0 conn(0x7fd9cc104a30 legacy=0x7fd9cc19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 shutdown_connections 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 >> 172.21.15.28:0/1837140590 conn(0x7fd9cc100360 msgr2=0x7fd9cc191060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 shutdown_connections 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 -- 172.21.15.28:0/1837140590 wait complete. 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:35.743+0000 7fd9d17fa700 1 librados: shutdown 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T14:49:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: Found migration_current of "None". Setting to last migration. 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14124 v1:172.21.15.28:0/2311192516' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14124 v1:172.21.15.28:0/2311192516' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:30] ENGINE Bus STARTING 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:30] ENGINE Serving on http://172.21.15.28:8765 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:30] ENGINE Serving on https://172.21.15.28:7150 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:30] ENGINE Bus STARTED 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14132 v1:172.21.15.28:0/2070778332' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14134 v1:172.21.15.28:0/1628431981' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14136 v1:172.21.15.28:0/4246890425' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: Generating ssh key... 2024-03-30T14:49:36.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:36 smithi028 ceph-mon[29876]: from='client.14138 v1:172.21.15.28:0/1837140590' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:38.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:38 smithi028 ceph-mon[29876]: from='client.14140 v1:172.21.15.28:0/1243642229' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:38.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:38 smithi028 ceph-mon[29876]: Deploying cephadm binary to smithi028 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f1524c5c700 1 Processor -- start 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f1524c5c700 1 -- start start 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f1524c5c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1520109ac0 con 0x7f1520105690 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f151e59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1520105690 0x7f1520105ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50228/0 (socket says 172.21.15.28:50228) 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f151e59c700 1 -- 172.21.15.28:0/4106940881 learned_addr learned my addr 172.21.15.28:0/4106940881 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976082689 0 0) 0x7f1520109ac0 con 0x7f1520105690 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.825+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f150c0035b0 con 0x7f1520105690 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435894888 0 0) 0x7f150c0035b0 con 0x7f1520105690 2024-03-30T14:49:42.002 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152010aca0 con 0x7f1520105690 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f1510001900 con 0x7f1520105690 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f15100020b0 con 0x7f1520105690 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f151d59a700 1 -- 172.21.15.28:0/4106940881 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f1510005ce0 con 0x7f1520105690 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f1524c5c700 1 -- 172.21.15.28:0/4106940881 >> v1:172.21.15.28:6789/0 conn(0x7f1520105690 legacy=0x7f1520105ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f1524c5c700 1 -- 172.21.15.28:0/4106940881 shutdown_connections 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f1524c5c700 1 -- 172.21.15.28:0/4106940881 >> 172.21.15.28:0/4106940881 conn(0x7f1520100c90 msgr2=0x7f15201030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.826+0000 7f1524c5c700 1 -- 172.21.15.28:0/4106940881 shutdown_connections 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 -- 172.21.15.28:0/4106940881 wait complete. 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 librados: starting msgr at 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 librados: starting objecter 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 Processor -- start 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 -- start start 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 librados: setting wanted keys 2024-03-30T14:49:42.003 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 librados: calling monclient init 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.827+0000 7f1524c5c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15201a0420 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f151e59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1520105690 0x7f1520102b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50236/0 (socket says 172.21.15.28:50236) 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f151e59c700 1 -- 172.21.15.28:0/1243642229 learned_addr learned my addr 172.21.15.28:0/1243642229 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 10 client.?.objecter ms_handle_connect 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3586372671 0 0) 0x7f15201a0420 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f150c0056e0 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 671816951 0 0) 0x7f152010aca0 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15201a0420 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f1510004bf0 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2878242484 0 0) 0x7f15201a0420 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15201a1600 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15201a0650 con 0x7f1520105690 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f1524c5c700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:42.004 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15201a0b90 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f1524c5c700 1 librados: init done 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f15100020b0 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.828+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f15100063b0 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7f1510011980 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1500005150 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f1510047c10 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 10 client.14140.objecter ms_dispatch 0x7f1520001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.829+0000 7f15177fe700 20 client.14140.objecter dump_active .. 0 homeless 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.830+0000 7f15177fe700 10 client.14140.objecter ms_handle_connect 0x7f150c03ba20 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:36.834+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f150c0056e0 con 0x7f1520105690 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:37.046+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f1500000bc0 con 0x7f150c03ba20 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.432+0000 7f15177fe700 1 -- 172.21.15.28:0/1243642229 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3724114767) 0x7f1500000bc0 con 0x7f150c03ba20 2024-03-30T14:49:42.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 10 librados: watch_flush enter 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 10 librados: watch_flush exit 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 20 client.14140.objecter shutdown clearing up homeless session... 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 10 client.14140.objecter successfully canceled tick 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 >> v1:172.21.15.28:6800/2767265670 conn(0x7f150c03ba20 legacy=0x7f150c03dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 >> v1:172.21.15.28:6789/0 conn(0x7f1520105690 legacy=0x7f1520102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 shutdown_connections 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 >> 172.21.15.28:0/1243642229 conn(0x7f1520100c90 msgr2=0x7f1520078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 shutdown_connections 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.435+0000 7f1524c5c700 1 -- 172.21.15.28:0/1243642229 wait complete. 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:41.436+0000 7f1524c5c700 1 librados: shutdown 2024-03-30T14:49:42.006 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-03-30T14:49:42.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:42 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:42.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:42 smithi028 ceph-mon[29876]: Added host smithi028 2024-03-30T14:49:42.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:42 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T14:49:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.461+0000 7f683efb3700 1 Processor -- start 2024-03-30T14:49:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.461+0000 7f683efb3700 1 -- start start 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f683efb3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6838109a40 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f683cd4f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6838105610 0x7f6838105a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53510/0 (socket says 172.21.15.28:53510) 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f683cd4f700 1 -- 172.21.15.28:0/875688032 learned_addr learned my addr 172.21.15.28:0/875688032 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752448337 0 0) 0x7f6838109a40 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68200035b0 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3680063647 0 0) 0x7f68200035b0 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683810ac20 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6830001900 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f68300020b0 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.462+0000 7f68377fe700 1 -- 172.21.15.28:0/875688032 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6830005ce0 con 0x7f6838105610 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 -- 172.21.15.28:0/875688032 >> v1:172.21.15.28:6789/0 conn(0x7f6838105610 legacy=0x7f6838105a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 -- 172.21.15.28:0/875688032 shutdown_connections 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 -- 172.21.15.28:0/875688032 >> 172.21.15.28:0/875688032 conn(0x7f6838100bf0 msgr2=0x7f6838103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 -- 172.21.15.28:0/875688032 shutdown_connections 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 -- 172.21.15.28:0/875688032 wait complete. 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 librados: starting msgr at 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.463+0000 7f683efb3700 1 librados: starting objecter 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683efb3700 1 Processor -- start 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683efb3700 1 -- start start 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683efb3700 1 librados: setting wanted keys 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683efb3700 1 librados: calling monclient init 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683efb3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6838079a70 con 0x7f6838105610 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683cd4f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6838105610 0x7f683819d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53516/0 (socket says 172.21.15.28:53516) 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f683cd4f700 1 -- 172.21.15.28:0/536948272 learned_addr learned my addr 172.21.15.28:0/536948272 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 10 client.?.objecter ms_handle_connect 0x7f6838105610 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150751782 0 0) 0x7f6838079a70 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68200056e0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1864235543 0 0) 0x7f683810ac20 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6838079a70 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6830004bf0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3579777474 0 0) 0x7f6838079a70 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.464+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6838079c40 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f683819e810 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7f6830004dc0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f68300065b0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f683efb3700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f683819ebb0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f683efb3700 1 librados: init done 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.465+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7f6830011cd0 con 0x7f6838105610 2024-03-30T14:49:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f68300478e0 con 0x7f6838105610 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 10 client.14142.objecter ms_dispatch 0x7f6838001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 20 client.14142.objecter dump_active .. 0 homeless 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6838063870 con 0x7f6838105610 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.466+0000 7f6835ffb700 10 client.14142.objecter ms_handle_connect 0x7f682003ba70 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.471+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f68200056e0 con 0x7f6838105610 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.685+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f683802dca0 con 0x7f682003ba70 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.690+0000 7f6835ffb700 1 -- 172.21.15.28:0/536948272 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f683802dca0 con 0x7f682003ba70 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 10 librados: watch_flush enter 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 10 librados: watch_flush exit 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 20 client.14142.objecter shutdown clearing up homeless session... 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 10 client.14142.objecter successfully canceled tick 2024-03-30T14:49:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 >> v1:172.21.15.28:6800/2767265670 conn(0x7f682003ba70 legacy=0x7f682003df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 >> v1:172.21.15.28:6789/0 conn(0x7f6838105610 legacy=0x7f683819d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 shutdown_connections 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 >> 172.21.15.28:0/536948272 conn(0x7f6838100bf0 msgr2=0x7f683818ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 shutdown_connections 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.692+0000 7f683efb3700 1 -- 172.21.15.28:0/536948272 wait complete. 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:42.693+0000 7f683efb3700 1 librados: shutdown 2024-03-30T14:49:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-03-30T14:49:43.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:43 smithi028 ceph-mon[29876]: from='client.14142 v1:172.21.15.28:0/536948272' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:43.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:43 smithi028 ceph-mon[29876]: Saving service mon spec with placement count:5 2024-03-30T14:49:43.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:43 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:45 smithi028 ceph-mon[29876]: from='client.14144 v1:172.21.15.28:0/1830900535' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:49:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:45 smithi028 ceph-mon[29876]: Saving service mgr spec with placement count:2 2024-03-30T14:49:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:45 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:45.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:45 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.004+0000 7efd946fc700 1 Processor -- start 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.004+0000 7efd946fc700 1 -- start start 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.004+0000 7efd946fc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd8c079e50 con 0x7efd8c107200 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd92498700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd8c107200 0x7efd8c1095f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53518/0 (socket says 172.21.15.28:53518) 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd92498700 1 -- 172.21.15.28:0/1241577548 learned_addr learned my addr 172.21.15.28:0/1241577548 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4250575309 0 0) 0x7efd8c079e50 con 0x7efd8c107200 2024-03-30T14:49:45.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd780035b0 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1692025875 0 0) 0x7efd780035b0 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c10bd20 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7efd7c001580 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7efd7c001d30 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.005+0000 7efd91496700 1 -- 172.21.15.28:0/1241577548 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7efd7c005960 con 0x7efd8c107200 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 -- 172.21.15.28:0/1241577548 >> v1:172.21.15.28:6789/0 conn(0x7efd8c107200 legacy=0x7efd8c1095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 -- 172.21.15.28:0/1241577548 shutdown_connections 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 -- 172.21.15.28:0/1241577548 >> 172.21.15.28:0/1241577548 conn(0x7efd8c100b90 msgr2=0x7efd8c103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 -- 172.21.15.28:0/1241577548 shutdown_connections 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 -- 172.21.15.28:0/1241577548 wait complete. 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 librados: starting msgr at 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.006+0000 7efd946fc700 1 librados: starting objecter 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 Processor -- start 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 -- start start 2024-03-30T14:49:45.419 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 librados: setting wanted keys 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 librados: calling monclient init 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd8c19ab20 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd92498700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd8c107200 0x7efd8c19a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53534/0 (socket says 172.21.15.28:53534) 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd92498700 1 -- 172.21.15.28:0/1830900535 learned_addr learned my addr 172.21.15.28:0/1830900535 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 10 client.?.objecter ms_handle_connect 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472971196 0 0) 0x7efd8c19ab20 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd780056e0 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1083037562 0 0) 0x7efd8c10bd20 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd8c19ab20 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7efd7c001080 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1715834978 0 0) 0x7efd8c19ab20 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c19acf0 con 0x7efd8c107200 2024-03-30T14:49:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.007+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c19afe0 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd946fc700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efd8c19eb30 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd946fc700 1 librados: init done 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1050+0+0 (unknown 965190600 0 0) 0x7efd7c001d30 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7efd7c005f80 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7efd7c011530 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.008+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70005120 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7efd7c0475e0 con 0x7efd8c107200 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 10 client.14144.objecter ms_dispatch 0x7efd8c001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 20 client.14144.objecter dump_active .. 0 homeless 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.009+0000 7efd837fe700 10 client.14144.objecter ms_handle_connect 0x7efd7803bc60 2024-03-30T14:49:45.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.013+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efd780056e0 con 0x7efd8c107200 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.233+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 --> v1:172.21.15.28:6800/2767265670 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7efd70000b90 con 0x7efd7803bc60 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.237+0000 7efd837fe700 1 -- 172.21.15.28:0/1830900535 <== mgr.14120 v1:172.21.15.28:6800/2767265670 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7efd70000b90 con 0x7efd7803bc60 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 10 librados: watch_flush enter 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 10 librados: watch_flush exit 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 20 client.14144.objecter shutdown clearing up homeless session... 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 10 client.14144.objecter successfully canceled tick 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 >> v1:172.21.15.28:6800/2767265670 conn(0x7efd7803bc60 legacy=0x7efd7803e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.239+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 >> v1:172.21.15.28:6789/0 conn(0x7efd8c107200 legacy=0x7efd8c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.240+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 shutdown_connections 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.240+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 >> 172.21.15.28:0/1830900535 conn(0x7efd8c100b90 msgr2=0x7efd8c191040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.240+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 shutdown_connections 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.240+0000 7efd946fc700 1 -- 172.21.15.28:0/1830900535 wait complete. 2024-03-30T14:49:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:44.240+0000 7efd946fc700 1 librados: shutdown 2024-03-30T14:49:46.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.890+0000 7f78b43bb700 1 Processor -- start 2024-03-30T14:49:46.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.890+0000 7f78b43bb700 1 -- start start 2024-03-30T14:49:46.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.890+0000 7f78b43bb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78ac10ad50 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.890+0000 7f78b2157700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f78ac107240 0x7f78ac109630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53546/0 (socket says 172.21.15.28:53546) 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.890+0000 7f78b2157700 1 -- 172.21.15.28:0/2676064416 learned_addr learned my addr 172.21.15.28:0/2676064416 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813912311 0 0) 0x7f78ac10ad50 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78980035b0 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 336999563 0 0) 0x7f78980035b0 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac10bf30 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f78a8001580 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f78a8004930 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b1155700 1 -- 172.21.15.28:0/2676064416 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f78a8005a70 con 0x7f78ac107240 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.891+0000 7f78b43bb700 1 -- 172.21.15.28:0/2676064416 >> v1:172.21.15.28:6789/0 conn(0x7f78ac107240 legacy=0x7f78ac109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- 172.21.15.28:0/2676064416 shutdown_connections 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- 172.21.15.28:0/2676064416 >> 172.21.15.28:0/2676064416 conn(0x7f78ac100c90 msgr2=0x7f78ac1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- 172.21.15.28:0/2676064416 shutdown_connections 2024-03-30T14:49:46.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- 172.21.15.28:0/2676064416 wait complete. 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 librados: starting msgr at 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 librados: starting objecter 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 Processor -- start 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- start start 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 librados: setting wanted keys 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 librados: calling monclient init 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b43bb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78ac19ac90 con 0x7f78ac107240 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.892+0000 7f78b2157700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f78ac107240 0x7f78ac19a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53562/0 (socket says 172.21.15.28:53562) 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78b2157700 1 -- 172.21.15.28:0/2050170185 learned_addr learned my addr 172.21.15.28:0/2050170185 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 10 client.?.objecter ms_handle_connect 0x7f78ac107240 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1425948194 0 0) 0x7f78ac19ac90 con 0x7f78ac107240 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78980056e0 con 0x7f78ac107240 2024-03-30T14:49:46.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 116553285 0 0) 0x7f78ac10bf30 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78ac19ac90 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f78a8001080 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4261876821 0 0) 0x7f78ac19ac90 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac19ae60 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac19b150 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78b43bb700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78ac19eca0 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78b43bb700 1 librados: init done 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f78a8004930 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.893+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f78a8006030 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7f78a80115e0 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7890005150 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f78a80475f0 con 0x7f78ac107240 2024-03-30T14:49:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78a37fe700 10 client.14146.objecter ms_dispatch 0x7f78ac001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78a37fe700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.894+0000 7f78a37fe700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.895+0000 7f78a37fe700 20 client.14146.objecter dump_active .. 0 homeless 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.895+0000 7f78a37fe700 10 client.14146.objecter ms_handle_connect 0x7f789803bcd0 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:45.899+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f78980056e0 con 0x7f78ac107240 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.104+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7890005440 con 0x7f78ac107240 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.106+0000 7f78a37fe700 1 -- 172.21.15.28:0/2050170185 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (unknown 2684808024 0 0) 0x7f78a8017510 con 0x7f78ac107240 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 10 librados: watch_flush enter 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 10 librados: watch_flush exit 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 20 client.14146.objecter shutdown clearing up homeless session... 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 10 client.14146.objecter successfully canceled tick 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 >> v1:172.21.15.28:6800/2767265670 conn(0x7f789803bcd0 legacy=0x7f789803e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.109+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 >> v1:172.21.15.28:6789/0 conn(0x7f78ac107240 legacy=0x7f78ac19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.110+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 shutdown_connections 2024-03-30T14:49:46.692 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.110+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 >> 172.21.15.28:0/2050170185 conn(0x7f78ac100c90 msgr2=0x7f78ac1911a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:46.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.110+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 shutdown_connections 2024-03-30T14:49:46.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.110+0000 7f78b43bb700 1 -- 172.21.15.28:0/2050170185 wait complete. 2024-03-30T14:49:46.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:46.110+0000 7f78b43bb700 1 librados: shutdown 2024-03-30T14:49:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:46 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:47.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2050170185' entity='client.admin' 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.179+0000 7f90e9618700 1 Processor -- start 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.179+0000 7f90e9618700 1 -- start start 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e9618700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90e410ad20 con 0x7f90e4104a70 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e2ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90e4104a70 0x7f90e4104e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53572/0 (socket says 172.21.15.28:53572) 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e2ffd700 1 -- 172.21.15.28:0/2069933540 learned_addr learned my addr 172.21.15.28:0/2069933540 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2620749232 0 0) 0x7f90e410ad20 con 0x7f90e4104a70 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d00035b0 con 0x7f90e4104a70 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3361375934 0 0) 0x7f90d00035b0 con 0x7f90e4104a70 2024-03-30T14:49:48.188 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e410bf00 con 0x7f90e4104a70 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f90cc001580 con 0x7f90e4104a70 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f90cc004930 con 0x7f90e4104a70 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.180+0000 7f90e1ffb700 1 -- 172.21.15.28:0/2069933540 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f90cc005a70 con 0x7f90e4104a70 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.181+0000 7f90e9618700 1 -- 172.21.15.28:0/2069933540 >> v1:172.21.15.28:6789/0 conn(0x7f90e4104a70 legacy=0x7f90e4104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.181+0000 7f90e9618700 1 -- 172.21.15.28:0/2069933540 shutdown_connections 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.181+0000 7f90e9618700 1 -- 172.21.15.28:0/2069933540 >> 172.21.15.28:0/2069933540 conn(0x7f90e4100410 msgr2=0x7f90e4102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.181+0000 7f90e9618700 1 -- 172.21.15.28:0/2069933540 shutdown_connections 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.181+0000 7f90e9618700 1 -- 172.21.15.28:0/2069933540 wait complete. 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 librados: starting msgr at 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 librados: starting objecter 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 Processor -- start 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 -- start start 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 librados: setting wanted keys 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 librados: calling monclient init 2024-03-30T14:49:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e9618700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90e419ab90 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e2ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90e4104a70 0x7f90e419a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53588/0 (socket says 172.21.15.28:53588) 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.182+0000 7f90e2ffd700 1 -- 172.21.15.28:0/219788854 learned_addr learned my addr 172.21.15.28:0/219788854 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 10 client.?.objecter ms_handle_connect 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131569032 0 0) 0x7f90e419ab90 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d00056e0 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2104017718 0 0) 0x7f90e410bf00 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90e419ab90 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f90cc001080 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 989591880 0 0) 0x7f90e419ab90 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e419ad60 con 0x7f90e4104a70 2024-03-30T14:49:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f90e419b050 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90e9618700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90e419eba0 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.183+0000 7f90e9618700 1 librados: init done 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.184+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f90cc004930 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.184+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f90cc0060a0 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.184+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7f90cc011650 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c4005150 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f90cc048480 con 0x7f90e4104a70 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 10 client.14148.objecter ms_dispatch 0x7f90e4001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 20 client.14148.objecter dump_active .. 0 homeless 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.185+0000 7f90dbfff700 10 client.14148.objecter ms_handle_connect 0x7f90d003bc60 2024-03-30T14:49:48.191 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.190+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f90d00056e0 con 0x7f90e4104a70 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.432+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f90c4005440 con 0x7f90e4104a70 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.434+0000 7f90dbfff700 1 -- 172.21.15.28:0/219788854 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (unknown 1496729260 0 0) 0x7f90cc017530 con 0x7f90e4104a70 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 10 librados: watch_flush enter 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 10 librados: watch_flush exit 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 20 client.14148.objecter shutdown clearing up homeless session... 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 10 client.14148.objecter successfully canceled tick 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 >> v1:172.21.15.28:6800/2767265670 conn(0x7f90d003bc60 legacy=0x7f90d003e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.436+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 >> v1:172.21.15.28:6789/0 conn(0x7f90e4104a70 legacy=0x7f90e419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.437+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 shutdown_connections 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.437+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 >> 172.21.15.28:0/219788854 conn(0x7f90e4100410 msgr2=0x7f90e41910e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.437+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 shutdown_connections 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.437+0000 7f90e9618700 1 -- 172.21.15.28:0/219788854 wait complete. 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:47.437+0000 7f90e9618700 1 librados: shutdown 2024-03-30T14:49:48.192 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-03-30T14:49:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/219788854' entity='client.admin' 2024-03-30T14:49:50.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:49 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:50.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3885532102' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T14:49:50.570 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setuser ceph since I am not root 2024-03-30T14:49:50.570 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setgroup ceph since I am not root 2024-03-30T14:49:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd13e35e700 1 Processor -- start 2024-03-30T14:49:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd13e35e700 1 -- start start 2024-03-30T14:49:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd13e35e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd138079e50 con 0x7fd138107260 2024-03-30T14:49:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd137fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd138107260 0x7fd138109650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59050/0 (socket says 172.21.15.28:59050) 2024-03-30T14:49:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd137fff700 1 -- 172.21.15.28:0/1009734171 learned_addr learned my addr 172.21.15.28:0/1009734171 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2976498156 0 0) 0x7fd138079e50 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.140+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1200035b0 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3439060189 0 0) 0x7fd1200035b0 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13810bd80 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd128001580 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd128004930 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd136ffd700 1 -- 172.21.15.28:0/1009734171 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd128005a70 con 0x7fd138107260 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd13e35e700 1 -- 172.21.15.28:0/1009734171 >> v1:172.21.15.28:6789/0 conn(0x7fd138107260 legacy=0x7fd138109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd13e35e700 1 -- 172.21.15.28:0/1009734171 shutdown_connections 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd13e35e700 1 -- 172.21.15.28:0/1009734171 >> 172.21.15.28:0/1009734171 conn(0x7fd138100c90 msgr2=0x7fd1381030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd13e35e700 1 -- 172.21.15.28:0/1009734171 shutdown_connections 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.141+0000 7fd13e35e700 1 -- 172.21.15.28:0/1009734171 wait complete. 2024-03-30T14:49:50.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 librados: starting msgr at 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 librados: starting objecter 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 Processor -- start 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 -- start start 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 librados: setting wanted keys 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 librados: calling monclient init 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd13e35e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd13819ab80 con 0x7fd138107260 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd137fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd138107260 0x7fd13819a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59054/0 (socket says 172.21.15.28:59054) 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.142+0000 7fd137fff700 1 -- 172.21.15.28:0/3885532102 learned_addr learned my addr 172.21.15.28:0/3885532102 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 10 client.?.objecter ms_handle_connect 0x7fd138107260 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638244104 0 0) 0x7fd13819ab80 con 0x7fd138107260 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1200056e0 con 0x7fd138107260 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 331176547 0 0) 0x7fd13810bd80 con 0x7fd138107260 2024-03-30T14:49:50.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd13819ab80 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd128001080 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3255515996 0 0) 0x7fd13819ab80 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13819ad50 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd13819b040 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd13e35e700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd13819eb90 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd13e35e700 1 librados: init done 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd128004c80 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.143+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd128006300 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 8) ==== 46081+0+0 (unknown 1524161843 0 0) 0x7fd1280118d0 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd124005120 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7fd1280476c0 con 0x7fd138107260 2024-03-30T14:49:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 10 client.14150.objecter ms_dispatch 0x7fd138001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.144+0000 7fd1357fa700 20 client.14150.objecter dump_active .. 0 homeless 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.145+0000 7fd1357fa700 10 client.14150.objecter ms_handle_connect 0x7fd12003bcd0 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.153+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd1200056e0 con 0x7fd138107260 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:49.414+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fd124005410 con 0x7fd138107260 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.246+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (unknown 1498667528 0 0) 0x7fd128017530 con 0x7fd138107260 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.248+0000 7fd1357fa700 1 -- 172.21.15.28:0/3885532102 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mgrmap(e 9) ==== 46094+0+0 (unknown 149190842 0 0) 0x7fd128046940 con 0x7fd138107260 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 10 librados: watch_flush enter 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 10 librados: watch_flush exit 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 20 client.14150.objecter shutdown clearing up homeless session... 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 10 client.14150.objecter successfully canceled tick 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 >> v1:172.21.15.28:6800/2767265670 conn(0x7fd12003bcd0 legacy=0x7fd12003e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.249+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 >> v1:172.21.15.28:6789/0 conn(0x7fd138107260 legacy=0x7fd13819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.250+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 shutdown_connections 2024-03-30T14:49:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.250+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 >> 172.21.15.28:0/3885532102 conn(0x7fd138100c90 msgr2=0x7fd1381910c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.250+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 shutdown_connections 2024-03-30T14:49:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.250+0000 7fd13e35e700 1 -- 172.21.15.28:0/3885532102 wait complete. 2024-03-30T14:49:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:50.250+0000 7fd13e35e700 1 librados: shutdown 2024-03-30T14:49:51.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-mon[29876]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:49:51.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-mon[29876]: from='mgr.14120 v1:172.21.15.28:0/2871544533' entity='mgr.y' 2024-03-30T14:49:51.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3885532102' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T14:49:51.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-mon[29876]: mgrmap e9: y(active, since 21s) 2024-03-30T14:49:51.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:50.848+0000 7fef84a25200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:49:51.112 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:50.952+0000 7fef84a25200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:49:51.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:51.633+0000 7fef84a25200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:49:52.060 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:51.738+0000 7fef84a25200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:49:52.060 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:51.882+0000 7fef84a25200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:49:52.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1731041461' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T14:49:52.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:51.981+0000 7fef84a25200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:49:52.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.067+0000 7fef84a25200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.378+0000 7fa5079c4700 1 Processor -- start 2024-03-30T14:49:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.378+0000 7fa5079c4700 1 -- start start 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.378+0000 7fa5079c4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa50010ab80 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa505760700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa500106750 0x7fa500106b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59078/0 (socket says 172.21.15.28:59078) 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa505760700 1 -- 172.21.15.28:0/227918545 learned_addr learned my addr 172.21.15.28:0/227918545 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641310463 0 0) 0x7fa50010ab80 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ec0035b0 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1499999955 0 0) 0x7fa4ec0035b0 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50010bd60 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fa4fc001580 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa4fc004930 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.379+0000 7fa4f7fff700 1 -- 172.21.15.28:0/227918545 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fa4fc005960 con 0x7fa500106750 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.380+0000 7fa5079c4700 1 -- 172.21.15.28:0/227918545 >> v1:172.21.15.28:6789/0 conn(0x7fa500106750 legacy=0x7fa500106b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.380+0000 7fa5079c4700 1 -- 172.21.15.28:0/227918545 shutdown_connections 2024-03-30T14:49:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.380+0000 7fa5079c4700 1 -- 172.21.15.28:0/227918545 >> 172.21.15.28:0/227918545 conn(0x7fa500101cf0 msgr2=0x7fa500104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.380+0000 7fa5079c4700 1 -- 172.21.15.28:0/227918545 shutdown_connections 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.380+0000 7fa5079c4700 1 -- 172.21.15.28:0/227918545 wait complete. 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 librados: starting msgr at 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 librados: starting objecter 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 Processor -- start 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 -- start start 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 librados: setting wanted keys 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 librados: calling monclient init 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa5079c4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa50019ab60 con 0x7fa500106750 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa505760700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa500106750 0x7fa50019a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59084/0 (socket says 172.21.15.28:59084) 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa505760700 1 -- 172.21.15.28:0/1731041461 learned_addr learned my addr 172.21.15.28:0/1731041461 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa4f67fc700 10 client.?.objecter ms_handle_connect 0x7fa500106750 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa4f67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266135962 0 0) 0x7fa50019ab60 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.381+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ec0056e0 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1671146185 0 0) 0x7fa50010bd60 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa50019ab60 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fa4fc001080 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2018333588 0 0) 0x7fa50019ab60 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50019ad30 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa50019b020 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa5079c4700 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa50019eb70 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa5079c4700 1 librados: init done 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa4fc004930 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.382+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fa4fc006000 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 9) ==== 46094+0+0 (unknown 149190842 0 0) 0x7fa4fc0116c0 con 0x7fa500106750 2024-03-30T14:49:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e4005150 con 0x7fa500106750 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7fa4fc0485e0 con 0x7fa500106750 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 10 client.14154.objecter ms_dispatch 0x7fa500001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa504f5f700 1 -- 172.21.15.28:0/1731041461 >> v1:172.21.15.28:6800/2767265670 conn(0x7fa4ec03bc10 legacy=0x7fa4ec03e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.383+0000 7fa4f67fc700 20 client.14154.objecter dump_active .. 0 homeless 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.391+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa4ec0056e0 con 0x7fa500106750 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.584+0000 7fa504f5f700 1 -- 172.21.15.28:0/1731041461 >> v1:172.21.15.28:6800/2767265670 conn(0x7fa4ec03bc10 legacy=0x7fa4ec03e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.667+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa4e4005c50 con 0x7fa500106750 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.667+0000 7fa4f67fc700 1 -- 172.21.15.28:0/1731041461 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7fa4fc017630 con 0x7fa500106750 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 10 librados: watch_flush enter 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 10 librados: watch_flush exit 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 20 client.14154.objecter shutdown clearing up homeless session... 2024-03-30T14:49:52.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 10 client.14154.objecter successfully canceled tick 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 >> v1:172.21.15.28:6800/2767265670 conn(0x7fa4ec03bc10 legacy=0x7fa4ec03e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.669+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 >> v1:172.21.15.28:6789/0 conn(0x7fa500106750 legacy=0x7fa50019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.670+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 shutdown_connections 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.670+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 >> 172.21.15.28:0/1731041461 conn(0x7fa500101cf0 msgr2=0x7fa500104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.670+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 shutdown_connections 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.670+0000 7fa5079c4700 1 -- 172.21.15.28:0/1731041461 wait complete. 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:51.670+0000 7fa5079c4700 1 librados: shutdown 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-03-30T14:49:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-03-30T14:49:52.596 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.336+0000 7fef84a25200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:49:52.596 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.452+0000 7fef84a25200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:52.597 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.541+0000 7fef84a25200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:49:52.849 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.644+0000 7fef84a25200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:49:52.849 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:52.824+0000 7fef84a25200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:49:53.956 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:53 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:53.822+0000 7fef84a25200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:49:53.957 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:53 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:53.956+0000 7fef84a25200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:49:54.586 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:54 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:54.293+0000 7fef84a25200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:49:54.586 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:54 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:54.385+0000 7fef84a25200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:49:54.586 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:54 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:54.585+0000 7fef84a25200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:54.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:54 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:54.775+0000 7fef84a25200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:49:55.326 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:55.065+0000 7fef84a25200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:49:55.326 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:55.325+0000 7fef84a25200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:49:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:55.567+0000 7fef84a25200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:49:56.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:56 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:56.392+0000 7fef84a25200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:49:57.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:56 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:56.969+0000 7fef84a25200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:49:57.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:57.070+0000 7fef84a25200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:49:57.291+0000 7fef84a25200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: Active manager daemon y restarted 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: Activating manager daemon y 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: mgrmap e10: y(active, starting, since 0.00320284s) 2024-03-30T14:49:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: Manager daemon y is now available 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:49:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:49:58.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.863+0000 7f4c50349700 1 Processor -- start 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.863+0000 7f4c50349700 1 -- start start 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.863+0000 7f4c50349700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4c4810abd0 con 0x7f4c48104a10 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.863+0000 7f4c4e0e5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4c48104a10 0x7f4c48104e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59090/0 (socket says 172.21.15.28:59090) 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.863+0000 7f4c4e0e5700 1 -- 172.21.15.28:0/3502715043 learned_addr learned my addr 172.21.15.28:0/3502715043 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966993605 0 0) 0x7f4c4810abd0 con 0x7f4c48104a10 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c340035b0 con 0x7f4c48104a10 2024-03-30T14:49:58.902 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3144119218 0 0) 0x7f4c340035b0 con 0x7f4c48104a10 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4810bdb0 con 0x7f4c48104a10 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4c44001580 con 0x7f4c48104a10 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4c44004930 con 0x7f4c48104a10 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.864+0000 7f4c4d0e3700 1 -- 172.21.15.28:0/3502715043 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4c44005a70 con 0x7f4c48104a10 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 -- 172.21.15.28:0/3502715043 >> v1:172.21.15.28:6789/0 conn(0x7f4c48104a10 legacy=0x7f4c48104e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 -- 172.21.15.28:0/3502715043 shutdown_connections 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 -- 172.21.15.28:0/3502715043 >> 172.21.15.28:0/3502715043 conn(0x7f4c48100340 msgr2=0x7f4c48102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 -- 172.21.15.28:0/3502715043 shutdown_connections 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 -- 172.21.15.28:0/3502715043 wait complete. 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 librados: starting msgr at 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 librados: starting objecter 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.865+0000 7f4c50349700 1 Processor -- start 2024-03-30T14:49:58.903 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c50349700 1 -- start start 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c50349700 1 librados: setting wanted keys 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c50349700 1 librados: calling monclient init 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c50349700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4c4819ab60 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c4e0e5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4c48104a10 0x7f4c4819a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59100/0 (socket says 172.21.15.28:59100) 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c4e0e5700 1 -- 172.21.15.28:0/2388418473 learned_addr learned my addr 172.21.15.28:0/2388418473 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 10 client.?.objecter ms_handle_connect 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830602239 0 0) 0x7f4c4819ab60 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c340056e0 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1044565249 0 0) 0x7f4c4810bdb0 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4819ab60 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4c44001080 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4253140145 0 0) 0x7f4c4819ab60 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4819ad30 con 0x7f4c48104a10 2024-03-30T14:49:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.866+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4819b020 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c50349700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c4819eb70 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c50349700 1 librados: init done 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4c44004930 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f4c440060a0 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.867+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 9) ==== 46094+0+0 (unknown 149190842 0 0) 0x7f4c44011740 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6800/2767265670 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c000d40 con 0x7f4c3403bc10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 2115785112 0 0) 0x7f4c44048620 con 0x7f4c48104a10 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c4d8e4700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c3f7fe700 10 client.14156.objecter ms_dispatch 0x7f4c48001550 osd_map(2..2 src has 1..2) 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c3f7fe700 3 client.14156.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c3f7fe700 3 client.14156.objecter handle_osd_map decoding full epoch 2 2024-03-30T14:49:58.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:52.868+0000 7f4c3f7fe700 20 client.14156.objecter dump_active .. 0 homeless 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:53.069+0000 7f4c4d8e4700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:53.470+0000 7f4c4d8e4700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:54.271+0000 7f4c4d8e4700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:55.872+0000 7f4c4d8e4700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/2767265670 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:57.294+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mgrmap(e 10) ==== 45896+0+0 (unknown 3631736865 0 0) 0x7f4c44013610 con 0x7f4c48104a10 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:57.295+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/2767265670 conn(0x7f4c3403bc10 legacy=0x7f4c3403e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:57.868+0000 7f4c4fb48700 10 client.14156.objecter tick 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.296+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mgrmap(e 11) ==== 45988+0+0 (unknown 1615899500 0 0) 0x7f4c44005010 con 0x7f4c48104a10 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.296+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6800/3720192247 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c000d40 con 0x7f4c34040570 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.297+0000 7f4c3f7fe700 10 client.14156.objecter ms_handle_connect 0x7f4c34040570 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.297+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f4c2c000d40 con 0x7f4c34040570 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 --> v1:172.21.15.28:6800/3720192247 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4c2c002840 con 0x7f4c34040570 2024-03-30T14:49:58.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c3f7fe700 1 -- 172.21.15.28:0/2388418473 <== mgr.14152 v1:172.21.15.28:6800/3720192247 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (unknown 0 0 3086460295) 0x7f4c2c002840 con 0x7f4c34040570 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 10 librados: watch_flush enter 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 10 librados: watch_flush exit 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 20 client.14156.objecter shutdown clearing up homeless session... 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 10 client.14156.objecter successfully canceled tick 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6800/3720192247 conn(0x7f4c34040570 legacy=0x7f4c34042960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 >> v1:172.21.15.28:6789/0 conn(0x7f4c48104a10 legacy=0x7f4c4819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 shutdown_connections 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 >> 172.21.15.28:0/2388418473 conn(0x7f4c48100340 msgr2=0x7f4c48191090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 shutdown_connections 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.311+0000 7f4c50349700 1 -- 172.21.15.28:0/2388418473 wait complete. 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:58.312+0000 7f4c50349700 1 librados: shutdown 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-03-30T14:49:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-03-30T14:49:59.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:57] ENGINE Bus STARTING 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:57] ENGINE Serving on http://172.21.15.28:8765 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:57] ENGINE Serving on https://172.21.15.28:7150 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: [30/Mar/2024:14:49:57] ENGINE Bus STARTED 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: from='client.14156 v1:172.21.15.28:0/2388418473' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: mgrmap e11: y(active, since 1.0057s) 2024-03-30T14:49:59.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:49:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:00.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:00 smithi028 ceph-mon[29876]: from='client.14156 v1:172.21.15.28:0/2388418473' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T14:50:00.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:00.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:00.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:00 smithi028 ceph-mon[29876]: mgrmap e12: y(active, since 2s) 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99ba962700 1 Processor -- start 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99ba962700 1 -- start start 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99ba962700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99b410acc0 con 0x7f99b4107e90 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99b3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99b4107e90 0x7f99b41082b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55510/0 (socket says 172.21.15.28:55510) 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99b3fff700 1 -- 172.21.15.28:0/1613794596 learned_addr learned my addr 172.21.15.28:0/1613794596 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682283453 0 0) 0x7f99b410acc0 con 0x7f99b4107e90 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.863+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f999c0035b0 con 0x7f99b4107e90 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2770142758 0 0) 0x7f999c0035b0 con 0x7f99b4107e90 2024-03-30T14:50:00.765 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b410bea0 con 0x7f99b4107e90 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f99a4001580 con 0x7f99b4107e90 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f99a4004930 con 0x7f99b4107e90 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99b2ffd700 1 -- 172.21.15.28:0/1613794596 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f99a4005a70 con 0x7f99b4107e90 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99ba962700 1 -- 172.21.15.28:0/1613794596 >> v1:172.21.15.28:6789/0 conn(0x7f99b4107e90 legacy=0x7f99b41082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99ba962700 1 -- 172.21.15.28:0/1613794596 shutdown_connections 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99ba962700 1 -- 172.21.15.28:0/1613794596 >> 172.21.15.28:0/1613794596 conn(0x7f99b41003c0 msgr2=0x7f99b41027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99ba962700 1 -- 172.21.15.28:0/1613794596 shutdown_connections 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.864+0000 7f99ba962700 1 -- 172.21.15.28:0/1613794596 wait complete. 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 librados: starting msgr at 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 librados: starting objecter 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 Processor -- start 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 -- start start 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 librados: setting wanted keys 2024-03-30T14:50:00.766 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 librados: calling monclient init 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99ba962700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99b41a05c0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99b3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99b4107e90 0x7f99b4107280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55524/0 (socket says 172.21.15.28:55524) 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99b3fff700 1 -- 172.21.15.28:0/2959116990 learned_addr learned my addr 172.21.15.28:0/2959116990 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99b17fa700 10 client.?.objecter ms_handle_connect 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.865+0000 7f99b17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370648323 0 0) 0x7f99b41a05c0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f999c0056e0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 216350907 0 0) 0x7f99b410bea0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99b41a05c0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f99a4001080 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2581230691 0 0) 0x7f99b41a05c0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b41a17a0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99b41a07f0 con 0x7f99b4107e90 2024-03-30T14:50:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99ba962700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99b41a0d30 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99ba962700 1 librados: init done 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f99a4004930 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.866+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f99a40060a0 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 11) ==== 45988+0+0 (unknown 1615899500 0 0) 0x7f99a40116d0 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a0005150 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f99a4047530 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 10 client.14164.objecter ms_dispatch 0x7f99b4001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 20 client.14164.objecter dump_active .. 0 homeless 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.867+0000 7f99b17fa700 10 client.14164.objecter ms_handle_connect 0x7f999c03bb00 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:49:59.872+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f999c0056e0 con 0x7f99b4107e90 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.082+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f99a0000bc0 con 0x7f999c03bb00 2024-03-30T14:50:00.768 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.175+0000 7f99b17fa700 1 -- 172.21.15.28:0/2959116990 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (unknown 0 0 3317752739) 0x7f99a0000bc0 con 0x7f999c03bb00 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.177+0000 7f99ba962700 10 librados: watch_flush enter 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.177+0000 7f99ba962700 10 librados: watch_flush exit 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.177+0000 7f99ba962700 20 client.14164.objecter shutdown clearing up homeless session... 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.177+0000 7f99ba962700 10 client.14164.objecter successfully canceled tick 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 >> v1:172.21.15.28:6800/3720192247 conn(0x7f999c03bb00 legacy=0x7f999c03dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 >> v1:172.21.15.28:6789/0 conn(0x7f99b4107e90 legacy=0x7f99b4107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 shutdown_connections 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 >> 172.21.15.28:0/2959116990 conn(0x7f99b41003c0 msgr2=0x7f99b41027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 shutdown_connections 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 -- 172.21.15.28:0/2959116990 wait complete. 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:00.178+0000 7f99ba962700 1 librados: shutdown 2024-03-30T14:50:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-03-30T14:50:01.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:01 smithi028 ceph-mon[29876]: from='client.14164 v1:172.21.15.28:0/2959116990' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:01.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:02.785 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SPV8gbDfnI6BWHgPcKWTreqEDSk38qYsc/5xdCtoQNe.gSGh480zi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711810201, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T14:50:02.785 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.285+0000 7fd616b19700 1 Processor -- start 2024-03-30T14:50:02.785 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.285+0000 7fd616b19700 1 -- start start 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd616b19700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd610079e70 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd6148b5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6101071d0 0x7fd6101095c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55530/0 (socket says 172.21.15.28:55530) 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd6148b5700 1 -- 172.21.15.28:0/1862276716 learned_addr learned my addr 172.21.15.28:0/1862276716 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789972368 0 0) 0x7fd610079e70 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6000035b0 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3931435915 0 0) 0x7fd6000035b0 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61010bcf0 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.286+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd604001580 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd604001c80 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd60f7fe700 1 -- 172.21.15.28:0/1862276716 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd604005960 con 0x7fd6101071d0 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd616b19700 1 -- 172.21.15.28:0/1862276716 >> v1:172.21.15.28:6789/0 conn(0x7fd6101071d0 legacy=0x7fd6101095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd616b19700 1 -- 172.21.15.28:0/1862276716 shutdown_connections 2024-03-30T14:50:02.786 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd616b19700 1 -- 172.21.15.28:0/1862276716 >> 172.21.15.28:0/1862276716 conn(0x7fd610100ba0 msgr2=0x7fd610103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd616b19700 1 -- 172.21.15.28:0/1862276716 shutdown_connections 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.287+0000 7fd616b19700 1 -- 172.21.15.28:0/1862276716 wait complete. 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 librados: starting msgr at 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 librados: starting objecter 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 Processor -- start 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 -- start start 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 librados: setting wanted keys 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 librados: calling monclient init 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd616b19700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6101a04e0 con 0x7fd6101071d0 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd6148b5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6101071d0 0x7fd610106a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55540/0 (socket says 172.21.15.28:55540) 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd6148b5700 1 -- 172.21.15.28:0/3330653033 learned_addr learned my addr 172.21.15.28:0/3330653033 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd60dffb700 10 client.?.objecter ms_handle_connect 0x7fd6101071d0 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.288+0000 7fd60dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144788695 0 0) 0x7fd6101a04e0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6000035b0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2556724030 0 0) 0x7fd6000035b0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6101a04e0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd604001080 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 436248817 0 0) 0x7fd6101a04e0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6101a16c0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6101a0710 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd6040017f0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fd6040063d0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd616b19700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6101a0c50 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.289+0000 7fd616b19700 1 librados: init done 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.290+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 12) ==== 46094+0+0 (unknown 471204694 0 0) 0x7fd6040119b0 con 0x7fd6101071d0 2024-03-30T14:50:02.788 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.290+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd610063870 con 0x7fd6101071d0 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7fd604047780 con 0x7fd6101071d0 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 10 client.14166.objecter ms_dispatch 0x7fd610001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 20 client.14166.objecter dump_active .. 0 homeless 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.291+0000 7fd60dffb700 10 client.14166.objecter ms_handle_connect 0x7fd60003bc60 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.296+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd604017620 con 0x7fd6101071d0 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.532+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd6101a0f40 con 0x7fd60003bc60 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.778+0000 7fd60dffb700 1 -- 172.21.15.28:0/3330653033 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (unknown 0 0 3731015500) 0x7fd6101a0f40 con 0x7fd60003bc60 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 10 librados: watch_flush enter 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 10 librados: watch_flush exit 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 20 client.14166.objecter shutdown clearing up homeless session... 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 10 client.14166.objecter successfully canceled tick 2024-03-30T14:50:02.789 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 >> v1:172.21.15.28:6800/3720192247 conn(0x7fd60003bc60 legacy=0x7fd60003e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 >> v1:172.21.15.28:6789/0 conn(0x7fd6101071d0 legacy=0x7fd610106a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 shutdown_connections 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 >> 172.21.15.28:0/3330653033 conn(0x7fd610100ba0 msgr2=0x7fd610191050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 shutdown_connections 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 -- 172.21.15.28:0/3330653033 wait complete. 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:01.780+0000 7fd616b19700 1 librados: shutdown 2024-03-30T14:50:02.790 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-03-30T14:50:03.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:02 smithi028 ceph-mon[29876]: from='client.14166 v1:172.21.15.28:0/3330653033' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:03.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:02 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:03.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2482609826' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T14:50:04.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce0e814700 1 Processor -- start 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce0e814700 1 -- start start 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce0e814700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce08109ac0 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce07fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fce08105690 0x7fce08105ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55556/0 (socket says 172.21.15.28:55556) 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce07fff700 1 -- 172.21.15.28:0/2511211038 learned_addr learned my addr 172.21.15.28:0/2511211038 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692920336 0 0) 0x7fce08109ac0 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdf00035b0 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.232+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3100553832 0 0) 0x7fcdf00035b0 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0810aca0 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fcdf8001580 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fcdf8004930 con 0x7fce08105690 2024-03-30T14:50:04.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce06ffd700 1 -- 172.21.15.28:0/2511211038 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fcdf8005960 con 0x7fce08105690 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce0e814700 1 -- 172.21.15.28:0/2511211038 >> v1:172.21.15.28:6789/0 conn(0x7fce08105690 legacy=0x7fce08105ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce0e814700 1 -- 172.21.15.28:0/2511211038 shutdown_connections 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce0e814700 1 -- 172.21.15.28:0/2511211038 >> 172.21.15.28:0/2511211038 conn(0x7fce08100c90 msgr2=0x7fce081030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce0e814700 1 -- 172.21.15.28:0/2511211038 shutdown_connections 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.233+0000 7fce0e814700 1 -- 172.21.15.28:0/2511211038 wait complete. 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 librados: starting msgr at 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 librados: starting objecter 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 Processor -- start 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 -- start start 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 librados: setting wanted keys 2024-03-30T14:50:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 librados: calling monclient init 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce0e814700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce08079a70 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce07fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fce08105690 0x7fce0819d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55560/0 (socket says 172.21.15.28:55560) 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce07fff700 1 -- 172.21.15.28:0/2482609826 learned_addr learned my addr 172.21.15.28:0/2482609826 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce057fa700 10 client.?.objecter ms_handle_connect 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce057fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018154036 0 0) 0x7fce08079a70 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.234+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdf00056e0 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2355513093 0 0) 0x7fce0810aca0 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce08079a70 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fcdf8001080 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2939523944 0 0) 0x7fce08079a70 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce08079c40 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce0819e860 con 0x7fce08105690 2024-03-30T14:50:04.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce0e814700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fce0819ec00 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce0e814700 1 librados: init done 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fcdf8004930 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.235+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fcdf8005f50 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 12) ==== 46094+0+0 (unknown 471204694 0 0) 0x7fcdf80115f0 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7fcdf8048570 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 10 client.14168.objecter ms_dispatch 0x7fce08001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 20 client.14168.objecter dump_active .. 0 homeless 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf4005150 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.236+0000 7fce057fa700 10 client.14168.objecter ms_handle_connect 0x7fcdf003bcb0 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.241+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fcdf00056e0 con 0x7fce08105690 2024-03-30T14:50:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.446+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcdf4005440 con 0x7fce08105690 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.446+0000 7fce057fa700 1 -- 172.21.15.28:0/2482609826 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fcdf8017580 con 0x7fce08105690 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 10 librados: watch_flush enter 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 10 librados: watch_flush exit 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 20 client.14168.objecter shutdown clearing up homeless session... 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 10 client.14168.objecter successfully canceled tick 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 >> v1:172.21.15.28:6800/3720192247 conn(0x7fcdf003bcb0 legacy=0x7fcdf003e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 >> v1:172.21.15.28:6789/0 conn(0x7fce08105690 legacy=0x7fce0819d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 shutdown_connections 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 >> 172.21.15.28:0/2482609826 conn(0x7fce08100c90 msgr2=0x7fce08103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 shutdown_connections 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 -- 172.21.15.28:0/2482609826 wait complete. 2024-03-30T14:50:04.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:03.448+0000 7fce0e814700 1 librados: shutdown 2024-03-30T14:50:04.334 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T14:50:04.334 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-30T14:50:04.341 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T14:50:04.341 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-30T14:50:04.341 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-30T14:50:04.341 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T14:50:04.342 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-03-30T14:50:04.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:04.342 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-03-30T14:50:04.342 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-03-30T14:50:04.343 INFO:teuthology.orchestra.run.smithi028.stdout: Password: 78y65l5qjn 2024-03-30T14:50:04.343 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:04.343 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config directory 2024-03-30T14:50:04.343 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-03-30T14:50:04.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:04 smithi028 ceph-mon[29876]: mgrmap e13: y(active, since 6s) 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f3728abe700 1 Processor -- start 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f3728abe700 1 -- start start 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f3728abe700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3724079e50 con 0x7f3724107260 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f372259c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3724107260 0x7f3724109650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55564/0 (socket says 172.21.15.28:55564) 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f372259c700 1 -- 172.21.15.28:0/678137507 learned_addr learned my addr 172.21.15.28:0/678137507 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923173393 0 0) 0x7f3724079e50 con 0x7f3724107260 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.801+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f370c0035b0 con 0x7f3724107260 2024-03-30T14:50:05.580 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1036176617 0 0) 0x7f370c0035b0 con 0x7f3724107260 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372410bd80 con 0x7f3724107260 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3714001580 con 0x7f3724107260 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3714004930 con 0x7f3724107260 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f372159a700 1 -- 172.21.15.28:0/678137507 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3714005a70 con 0x7f3724107260 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f3728abe700 1 -- 172.21.15.28:0/678137507 >> v1:172.21.15.28:6789/0 conn(0x7f3724107260 legacy=0x7f3724109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f3728abe700 1 -- 172.21.15.28:0/678137507 shutdown_connections 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f3728abe700 1 -- 172.21.15.28:0/678137507 >> 172.21.15.28:0/678137507 conn(0x7f3724100c90 msgr2=0x7f37241030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f3728abe700 1 -- 172.21.15.28:0/678137507 shutdown_connections 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.802+0000 7f3728abe700 1 -- 172.21.15.28:0/678137507 wait complete. 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 librados: starting msgr at 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 librados: starting objecter 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 Processor -- start 2024-03-30T14:50:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 -- start start 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 librados: setting wanted keys 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 librados: calling monclient init 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f3728abe700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37241a04a0 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f372259c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3724107260 0x7f3724106ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55576/0 (socket says 172.21.15.28:55576) 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f372259c700 1 -- 172.21.15.28:0/4204903887 learned_addr learned my addr 172.21.15.28:0/4204903887 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f37137fe700 10 client.?.objecter ms_handle_connect 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.803+0000 7f37137fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1238513747 0 0) 0x7f37241a04a0 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f370c0056e0 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3383574979 0 0) 0x7f372410bd80 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37241a04a0 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3714001080 con 0x7f3724107260 2024-03-30T14:50:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3308121027 0 0) 0x7f37241a04a0 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37241a1680 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37241a06d0 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f3728abe700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f37241a0c10 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f3728abe700 1 librados: init done 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3714004930 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.804+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f3714006030 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f3714011620 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3704005120 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f3714047770 con 0x7f3724107260 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 10 client.14170.objecter ms_dispatch 0x7f3724001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.805+0000 7f37137fe700 20 client.14170.objecter dump_active .. 0 homeless 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.806+0000 7f37137fe700 10 client.14170.objecter ms_handle_connect 0x7f370c03bdb0 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:04.810+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f370c0056e0 con 0x7f3724107260 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.012+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f3704005410 con 0x7f3724107260 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.013+0000 7f37137fe700 1 -- 172.21.15.28:0/4204903887 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (unknown 689008156 0 0) 0x7f3714017640 con 0x7f3724107260 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 10 librados: watch_flush enter 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 10 librados: watch_flush exit 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 20 client.14170.objecter shutdown clearing up homeless session... 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 10 client.14170.objecter successfully canceled tick 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 >> v1:172.21.15.28:6800/3720192247 conn(0x7f370c03bdb0 legacy=0x7f370c03e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 >> v1:172.21.15.28:6789/0 conn(0x7f3724107260 legacy=0x7f3724106ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 shutdown_connections 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 >> 172.21.15.28:0/4204903887 conn(0x7f3724100c90 msgr2=0x7f37241910c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 shutdown_connections 2024-03-30T14:50:05.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 -- 172.21.15.28:0/4204903887 wait complete. 2024-03-30T14:50:05.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:05.015+0000 7f3728abe700 1 librados: shutdown 2024-03-30T14:50:07.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.044+0000 7f9f3dc54700 1 Processor -- start 2024-03-30T14:50:07.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.044+0000 7f9f3dc54700 1 -- start start 2024-03-30T14:50:07.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f3dc54700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9f38109a40 con 0x7f9f38105610 2024-03-30T14:50:07.005 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f377fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f38105610 0x7f9f38105a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55580/0 (socket says 172.21.15.28:55580) 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f377fe700 1 -- 172.21.15.28:0/2404552126 learned_addr learned my addr 172.21.15.28:0/2404552126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427598908 0 0) 0x7f9f38109a40 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f240035b0 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1021895300 0 0) 0x7f9f240035b0 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3810ac20 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9f28001900 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 <== mon.0 v1:172.21.15.28:6789/0 4 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9f28004cb0 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.045+0000 7f9f367fc700 1 -- 172.21.15.28:0/2404552126 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9f28005ce0 con 0x7f9f38105610 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 -- 172.21.15.28:0/2404552126 >> v1:172.21.15.28:6789/0 conn(0x7f9f38105610 legacy=0x7f9f38105a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 -- 172.21.15.28:0/2404552126 shutdown_connections 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 -- 172.21.15.28:0/2404552126 >> 172.21.15.28:0/2404552126 conn(0x7f9f38100bf0 msgr2=0x7f9f38103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:07.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 -- 172.21.15.28:0/2404552126 shutdown_connections 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 -- 172.21.15.28:0/2404552126 wait complete. 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 librados: starting msgr at 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.046+0000 7f9f3dc54700 1 librados: starting objecter 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f3dc54700 1 Processor -- start 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f3dc54700 1 -- start start 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f3dc54700 1 librados: setting wanted keys 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f3dc54700 1 librados: calling monclient init 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f3dc54700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9f3819ab10 con 0x7f9f38105610 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f377fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f38105610 0x7f9f3819a400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55596/0 (socket says 172.21.15.28:55596) 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f377fe700 1 -- 172.21.15.28:0/1217413426 learned_addr learned my addr 172.21.15.28:0/1217413426 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f34ff9700 10 client.?.objecter ms_handle_connect 0x7f9f38105610 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f34ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:07.007 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241719611 0 0) 0x7f9f3819ab10 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.047+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f240056e0 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 923159567 0 0) 0x7f9f3810ac20 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f3819ab10 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9f28001400 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1510005506 0 0) 0x7f9f3819ab10 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3819ace0 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3819afd0 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9f28001890 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9f28006610 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f3dc54700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f3819eb20 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.048+0000 7f9f3dc54700 1 librados: init done 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f9f28011d70 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f38063870 con 0x7f9f38105610 2024-03-30T14:50:07.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f9f28047d60 con 0x7f9f38105610 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 10 client.14172.objecter ms_dispatch 0x7f9f38001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.049+0000 7f9f34ff9700 20 client.14172.objecter dump_active .. 0 homeless 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.050+0000 7f9f34ff9700 10 client.14172.objecter ms_handle_connect 0x7f9f2403bd50 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.054+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9f240056e0 con 0x7f9f38105610 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.369+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f9f3819ee10 con 0x7f9f38105610 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.372+0000 7f9f34ff9700 1 -- 172.21.15.28:0/1217413426 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (unknown 1169358022 0 0) 0x7f9f280179f0 con 0x7f9f38105610 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.373+0000 7f9f3dc54700 10 librados: watch_flush enter 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.373+0000 7f9f3dc54700 10 librados: watch_flush exit 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.373+0000 7f9f3dc54700 20 client.14172.objecter shutdown clearing up homeless session... 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.373+0000 7f9f3dc54700 10 client.14172.objecter successfully canceled tick 2024-03-30T14:50:07.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 >> v1:172.21.15.28:6800/3720192247 conn(0x7f9f2403bd50 legacy=0x7f9f2403e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 >> v1:172.21.15.28:6789/0 conn(0x7f9f38105610 legacy=0x7f9f3819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 shutdown_connections 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 >> 172.21.15.28:0/1217413426 conn(0x7f9f38100bf0 msgr2=0x7f9f38078fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 shutdown_connections 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 -- 172.21.15.28:0/1217413426 wait complete. 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-30T14:50:06.374+0000 7f9f3dc54700 1 librados: shutdown 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-30T14:50:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:07.012 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-03-30T14:50:07.046 INFO:tasks.cephadm:Fetching config... 2024-03-30T14:50:07.046 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:07.046 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T14:50:07.061 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T14:50:07.062 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:07.062 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T14:50:07.117 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T14:50:07.117 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:07.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-30T14:50:07.184 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T14:50:07.184 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:07.184 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T14:50:07.240 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T14:50:07.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxMskQz6JNzE/jOBSeOsI0Mr9Vj8FV1A05w4Hcks2ntX5JqbQwclfaVjaVF3ZJ3VuRbtLX2xs8ipZ23NBHQoObo+SSyHq1J7qeFvnRuK/5auYeveu27nnaie8py2tjRqLdkHd4kwerVUDdcUtu9fpsQAO7r4pKyF/7kMtdyytL8ef0dxOCSHqNsH1EQAdw8oos4u6R/8r9YNxMlhEWxPwC+Y+fQWy6EKAxYJ3fFfQhXb7QgRtXxPXmJ25FxDOkaVm6mv2gyayZJjM2eNf3AHCRaUKwAxHHRdg9lDm509xWcT5T0fSpB7/iO9XekYwcPONiFNVLFamf5zNPaVXa9Huy7n7SBiBy+YE0o2TsGPhjd2Mjk9JzINGmoN1zIlbLwWP5yJdThdA33Ke+BgrTOeyDxNHZFSVJ2C2mixoB0vhebX+F0UNcS55PWnCHQ2xm9GXD9tEbcLFt6Prq3GbsQhyXYUdjXk4RgCzF6owWicJnaTDGOvBCqgnKILUIuk08X3U= ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T14:50:07.319 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxMskQz6JNzE/jOBSeOsI0Mr9Vj8FV1A05w4Hcks2ntX5JqbQwclfaVjaVF3ZJ3VuRbtLX2xs8ipZ23NBHQoObo+SSyHq1J7qeFvnRuK/5auYeveu27nnaie8py2tjRqLdkHd4kwerVUDdcUtu9fpsQAO7r4pKyF/7kMtdyytL8ef0dxOCSHqNsH1EQAdw8oos4u6R/8r9YNxMlhEWxPwC+Y+fQWy6EKAxYJ3fFfQhXb7QgRtXxPXmJ25FxDOkaVm6mv2gyayZJjM2eNf3AHCRaUKwAxHHRdg9lDm509xWcT5T0fSpB7/iO9XekYwcPONiFNVLFamf5zNPaVXa9Huy7n7SBiBy+YE0o2TsGPhjd2Mjk9JzINGmoN1zIlbLwWP5yJdThdA33Ke+BgrTOeyDxNHZFSVJ2C2mixoB0vhebX+F0UNcS55PWnCHQ2xm9GXD9tEbcLFt6Prq3GbsQhyXYUdjXk4RgCzF6owWicJnaTDGOvBCqgnKILUIuk08X3U= ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:07.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxMskQz6JNzE/jOBSeOsI0Mr9Vj8FV1A05w4Hcks2ntX5JqbQwclfaVjaVF3ZJ3VuRbtLX2xs8ipZ23NBHQoObo+SSyHq1J7qeFvnRuK/5auYeveu27nnaie8py2tjRqLdkHd4kwerVUDdcUtu9fpsQAO7r4pKyF/7kMtdyytL8ef0dxOCSHqNsH1EQAdw8oos4u6R/8r9YNxMlhEWxPwC+Y+fQWy6EKAxYJ3fFfQhXb7QgRtXxPXmJ25FxDOkaVm6mv2gyayZJjM2eNf3AHCRaUKwAxHHRdg9lDm509xWcT5T0fSpB7/iO9XekYwcPONiFNVLFamf5zNPaVXa9Huy7n7SBiBy+YE0o2TsGPhjd2Mjk9JzINGmoN1zIlbLwWP5yJdThdA33Ke+BgrTOeyDxNHZFSVJ2C2mixoB0vhebX+F0UNcS55PWnCHQ2xm9GXD9tEbcLFt6Prq3GbsQhyXYUdjXk4RgCzF6owWicJnaTDGOvBCqgnKILUIuk08X3U= ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T14:50:07.379 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxMskQz6JNzE/jOBSeOsI0Mr9Vj8FV1A05w4Hcks2ntX5JqbQwclfaVjaVF3ZJ3VuRbtLX2xs8ipZ23NBHQoObo+SSyHq1J7qeFvnRuK/5auYeveu27nnaie8py2tjRqLdkHd4kwerVUDdcUtu9fpsQAO7r4pKyF/7kMtdyytL8ef0dxOCSHqNsH1EQAdw8oos4u6R/8r9YNxMlhEWxPwC+Y+fQWy6EKAxYJ3fFfQhXb7QgRtXxPXmJ25FxDOkaVm6mv2gyayZJjM2eNf3AHCRaUKwAxHHRdg9lDm509xWcT5T0fSpB7/iO9XekYwcPONiFNVLFamf5zNPaVXa9Huy7n7SBiBy+YE0o2TsGPhjd2Mjk9JzINGmoN1zIlbLwWP5yJdThdA33Ke+BgrTOeyDxNHZFSVJ2C2mixoB0vhebX+F0UNcS55PWnCHQ2xm9GXD9tEbcLFt6Prq3GbsQhyXYUdjXk4RgCzF6owWicJnaTDGOvBCqgnKILUIuk08X3U= ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:07.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T14:50:07.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1217413426' entity='client.admin' 2024-03-30T14:50:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.318+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1219305284 >> v1:172.21.15.28:6789/0 conn(0x7fdf381026a0 legacy=0x7fdf38104a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.318+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1219305284 shutdown_connections 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.318+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1219305284 >> 172.21.15.28:0/1219305284 conn(0x7fdf380fc030 msgr2=0x7fdf380fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.318+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1219305284 shutdown_connections 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1219305284 wait complete. 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 librados: starting msgr at 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 librados: starting objecter 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 Processor -- start 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 -- start start 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 librados: setting wanted keys 2024-03-30T14:50:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.319+0000 7fdf3cc45700 1 librados: calling monclient init 2024-03-30T14:50:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf3cc45700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf3819c740 con 0x7fdf381026a0 2024-03-30T14:50:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf3659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf381026a0 0x7fdf38101f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58220/0 (socket says 172.21.15.28:58220) 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf3659c700 1 -- 172.21.15.28:0/1829242279 learned_addr learned my addr 172.21.15.28:0/1829242279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 10 client.?.objecter ms_handle_connect 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724952600 0 0) 0x7fdf3819c740 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf200056e0 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1100009150 0 0) 0x7fdf381071c0 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf3819c740 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdf28001370 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.320+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2916276898 0 0) 0x7fdf3819c740 con 0x7fdf381026a0 2024-03-30T14:50:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3819d920 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3819c970 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf3cc45700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf3819ceb0 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf3cc45700 1 librados: init done 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdf28003740 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fdf28004ee0 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.321+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7fdf280050a0 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7fdf280479c0 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 10 client.14174.objecter ms_dispatch 0x7fdf38001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 20 client.14174.objecter dump_active .. 0 homeless 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf18005120 con 0x7fdf381026a0 2024-03-30T14:50:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.322+0000 7fdf277fe700 10 client.14174.objecter ms_handle_connect 0x7fdf2003bb90 2024-03-30T14:50:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.327+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdf200056e0 con 0x7fdf381026a0 2024-03-30T14:50:10.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.541+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 --> v1:172.21.15.28:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fdf18005410 con 0x7fdf381026a0 2024-03-30T14:50:10.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.543+0000 7fdf277fe700 1 -- 172.21.15.28:0/1829242279 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (unknown 3028693289 0 0) 0x7fdf280175b0 con 0x7fdf381026a0 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 10 librados: watch_flush enter 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 10 librados: watch_flush exit 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 20 client.14174.objecter shutdown clearing up homeless session... 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 10 client.14174.objecter successfully canceled tick 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 >> v1:172.21.15.28:6800/3720192247 conn(0x7fdf2003bb90 legacy=0x7fdf2003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 >> v1:172.21.15.28:6789/0 conn(0x7fdf381026a0 legacy=0x7fdf38101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 shutdown_connections 2024-03-30T14:50:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 >> 172.21.15.28:0/1829242279 conn(0x7fdf380fc030 msgr2=0x7fdf38104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 shutdown_connections 2024-03-30T14:50:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 -- 172.21.15.28:0/1829242279 wait complete. 2024-03-30T14:50:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:10.545+0000 7fdf3cc45700 1 librados: shutdown 2024-03-30T14:50:11.402 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T14:50:11.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T14:50:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1829242279' entity='client.admin' 2024-03-30T14:50:11.974 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:13.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:13.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:13.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:50:13.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:13.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:13.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.573+0000 7f42c46d9700 1 -- 172.21.15.28:0/984018592 >> v1:172.21.15.28:6789/0 conn(0x7f42bc100ab0 legacy=0x7f42bc100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- 172.21.15.28:0/984018592 shutdown_connections 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- 172.21.15.28:0/984018592 >> 172.21.15.28:0/984018592 conn(0x7f42bc0fc050 msgr2=0x7f42bc0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- 172.21.15.28:0/984018592 shutdown_connections 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- 172.21.15.28:0/984018592 wait complete. 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 librados: starting msgr at 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 librados: starting objecter 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 Processor -- start 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- start start 2024-03-30T14:50:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 librados: setting wanted keys 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 librados: calling monclient init 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c46d9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42bc195d30 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c2475700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f42bc100ab0 0x7f42bc195620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58232/0 (socket says 172.21.15.28:58232) 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.574+0000 7f42c2475700 1 -- 172.21.15.28:0/1061673208 learned_addr learned my addr 172.21.15.28:0/1061673208 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 10 client.?.objecter ms_handle_connect 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146963616 0 0) 0x7f42bc195d30 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42a80056e0 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2913821153 0 0) 0x7f42bc1060c0 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42bc195d30 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f42b8001480 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2743666976 0 0) 0x7f42bc195d30 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc195f00 con 0x7f42bc100ab0 2024-03-30T14:50:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc1961f0 con 0x7f42bc100ab0 2024-03-30T14:50:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42c46d9700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42bc199cf0 con 0x7f42bc100ab0 2024-03-30T14:50:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42c46d9700 1 librados: init done 2024-03-30T14:50:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f42b8003700 con 0x7f42bc100ab0 2024-03-30T14:50:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.575+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f42b8004e00 con 0x7f42bc100ab0 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f42b8004fc0 con 0x7f42bc100ab0 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a0005150 con 0x7f42bc100ab0 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f42b8047690 con 0x7f42bc100ab0 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 10 client.14176.objecter ms_dispatch 0x7f42bc001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.576+0000 7f42b37fe700 20 client.14176.objecter dump_active .. 0 homeless 2024-03-30T14:50:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.577+0000 7f42b37fe700 10 client.14176.objecter ms_handle_connect 0x7f42a803bb90 2024-03-30T14:50:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.581+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f42a80056e0 con 0x7f42bc100ab0 2024-03-30T14:50:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.792+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f42a0000bc0 con 0x7f42a803bb90 2024-03-30T14:50:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.795+0000 7f42b37fe700 1 -- 172.21.15.28:0/1061673208 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f42a0000bc0 con 0x7f42a803bb90 2024-03-30T14:50:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.797+0000 7f42c46d9700 10 librados: watch_flush enter 2024-03-30T14:50:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 10 librados: watch_flush exit 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 20 client.14176.objecter shutdown clearing up homeless session... 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 10 client.14176.objecter successfully canceled tick 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 >> v1:172.21.15.28:6800/3720192247 conn(0x7f42a803bb90 legacy=0x7f42a803e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 >> v1:172.21.15.28:6789/0 conn(0x7f42bc100ab0 legacy=0x7f42bc195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 shutdown_connections 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 >> 172.21.15.28:0/1061673208 conn(0x7f42bc0fc050 msgr2=0x7f42bc0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 shutdown_connections 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 -- 172.21.15.28:0/1061673208 wait complete. 2024-03-30T14:50:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:13.798+0000 7f42c46d9700 1 librados: shutdown 2024-03-30T14:50:14.631 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2024-03-30T14:50:14.631 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:50:14.632 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2024-03-30T14:50:14.650 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:50:14.651 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:14.708 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2024-03-30T14:50:14.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi134 2024-03-30T14:50:14.944 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='client.14176 v1:172.21.15.28:0/1061673208' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:16.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:15 smithi028 ceph-mon[29876]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 -- 172.21.15.28:0/769729292 >> v1:172.21.15.28:6789/0 conn(0x7f9be4106a30 legacy=0x7f9be4106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 -- 172.21.15.28:0/769729292 shutdown_connections 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 -- 172.21.15.28:0/769729292 >> 172.21.15.28:0/769729292 conn(0x7f9be40fc520 msgr2=0x7f9be40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 -- 172.21.15.28:0/769729292 shutdown_connections 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 -- 172.21.15.28:0/769729292 wait complete. 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 librados: starting msgr at 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.518+0000 7f9bea14d700 1 librados: starting objecter 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9bea14d700 1 Processor -- start 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9bea14d700 1 -- start start 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9bea14d700 1 librados: setting wanted keys 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9bea14d700 1 librados: calling monclient init 2024-03-30T14:50:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9bea14d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9be40fe870 con 0x7f9be4106a30 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be37fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9be4106a30 0x7f9be40fc9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58248/0 (socket says 172.21.15.28:58248) 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be37fe700 1 -- 172.21.15.28:0/1995032694 learned_addr learned my addr 172.21.15.28:0/1995032694 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be0ff9700 10 client.?.objecter ms_handle_connect 0x7f9be4106a30 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311604880 0 0) 0x7f9be40fe870 con 0x7f9be4106a30 2024-03-30T14:50:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.519+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bcc0056e0 con 0x7f9be4106a30 2024-03-30T14:50:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1774835803 0 0) 0x7f9bcc0056e0 con 0x7f9be4106a30 2024-03-30T14:50:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9be40fe870 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9bd40047f0 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3723614150 0 0) 0x7f9be40fe870 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be40fd0e0 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9be40fd3d0 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9bea14d700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9be41a1e60 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9bea14d700 1 librados: init done 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9bd4001a80 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.520+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f9bd4005f40 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f9bd4011530 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd0005120 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f9bd4047610 con 0x7f9be4106a30 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 10 client.14178.objecter ms_dispatch 0x7f9be4001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 20 client.14178.objecter dump_active .. 0 homeless 2024-03-30T14:50:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.521+0000 7f9be0ff9700 10 client.14178.objecter ms_handle_connect 0x7f9bcc03baf0 2024-03-30T14:50:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.526+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9be41075b0 con 0x7f9be4106a30 2024-03-30T14:50:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:16.735+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) -- 0x7f9bd0000b90 con 0x7f9bcc03baf0 2024-03-30T14:50:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:17 smithi028 ceph-mon[29876]: from='client.14178 v1:172.21.15.28:0/1995032694' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:17 smithi028 ceph-mon[29876]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:18.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:18 smithi028 ceph-mon[29876]: Deploying cephadm binary to smithi134 2024-03-30T14:50:19.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:19 smithi028 ceph-mon[29876]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.203+0000 7f9be0ff9700 1 -- 172.21.15.28:0/1995032694 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (unknown 0 0 2519024100) 0x7f9bd0000b90 con 0x7f9bcc03baf0 2024-03-30T14:50:21.206 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi134' with addr '172.21.15.134' 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 10 librados: watch_flush enter 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 10 librados: watch_flush exit 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 20 client.14178.objecter shutdown clearing up homeless session... 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 10 client.14178.objecter successfully canceled tick 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 >> v1:172.21.15.28:6800/3720192247 conn(0x7f9bcc03baf0 legacy=0x7f9bcc03dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.205+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 >> v1:172.21.15.28:6789/0 conn(0x7f9be4106a30 legacy=0x7f9be40fc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.206+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 shutdown_connections 2024-03-30T14:50:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.206+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 >> 172.21.15.28:0/1995032694 conn(0x7f9be40fc520 msgr2=0x7f9be4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.206+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 shutdown_connections 2024-03-30T14:50:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.206+0000 7f9bea14d700 1 -- 172.21.15.28:0/1995032694 wait complete. 2024-03-30T14:50:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:21.206+0000 7f9bea14d700 1 librados: shutdown 2024-03-30T14:50:21.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T14:50:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:22.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:22.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:22 smithi028 ceph-mon[29876]: Added host smithi134 2024-03-30T14:50:22.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:22.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:22 smithi028 ceph-mon[29876]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:22.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:23.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3954462760 >> v1:172.21.15.28:6789/0 conn(0x7ffb58100870 legacy=0x7ffb58100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3954462760 shutdown_connections 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3954462760 >> 172.21.15.28:0/3954462760 conn(0x7ffb580fc530 msgr2=0x7ffb580fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3954462760 shutdown_connections 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3954462760 wait complete. 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 librados: starting msgr at 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.665+0000 7ffb5e3fb700 1 librados: starting objecter 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb5e3fb700 1 Processor -- start 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb5e3fb700 1 -- start start 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb5e3fb700 1 librados: setting wanted keys 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb5e3fb700 1 librados: calling monclient init 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb5e3fb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffb5819f030 con 0x7ffb58100870 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb57fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffb58100870 0x7ffb5819e920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59712/0 (socket says 172.21.15.28:59712) 2024-03-30T14:50:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb57fff700 1 -- 172.21.15.28:0/3460120976 learned_addr learned my addr 172.21.15.28:0/3460120976 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb557fa700 10 client.?.objecter ms_handle_connect 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb557fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580345117 0 0) 0x7ffb5819f030 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.666+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb400056e0 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 377098286 0 0) 0x7ffb400056e0 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb5819f030 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ffb48001480 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3505718802 0 0) 0x7ffb5819f030 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5819f200 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffb5819f4f0 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb5e3fb700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffb581a1a60 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb5e3fb700 1 librados: init done 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ffb48001b60 con 0x7ffb58100870 2024-03-30T14:50:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.667+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7ffb48004f00 con 0x7ffb58100870 2024-03-30T14:50:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7ffb480050c0 con 0x7ffb58100870 2024-03-30T14:50:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7ffb48048340 con 0x7ffb58100870 2024-03-30T14:50:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 10 client.14180.objecter ms_dispatch 0x7ffb58001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 20 client.14180.objecter dump_active .. 0 homeless 2024-03-30T14:50:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb44005120 con 0x7ffb58100870 2024-03-30T14:50:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.668+0000 7ffb557fa700 10 client.14180.objecter ms_handle_connect 0x7ffb4003bbf0 2024-03-30T14:50:23.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.673+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ffb48017350 con 0x7ffb58100870 2024-03-30T14:50:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.883+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffb44000b90 con 0x7ffb4003bbf0 2024-03-30T14:50:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.884+0000 7ffb557fa700 1 -- 172.21.15.28:0/3460120976 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (unknown 0 0 3534459176) 0x7ffb44000b90 con 0x7ffb4003bbf0 2024-03-30T14:50:23.884 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:50:23.884 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}] 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 10 librados: watch_flush enter 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 10 librados: watch_flush exit 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 20 client.14180.objecter shutdown clearing up homeless session... 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 10 client.14180.objecter successfully canceled tick 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 >> v1:172.21.15.28:6800/3720192247 conn(0x7ffb4003bbf0 legacy=0x7ffb4003e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 >> v1:172.21.15.28:6789/0 conn(0x7ffb58100870 legacy=0x7ffb5819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 shutdown_connections 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 >> 172.21.15.28:0/3460120976 conn(0x7ffb580fc530 msgr2=0x7ffb580fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 shutdown_connections 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 -- 172.21.15.28:0/3460120976 wait complete. 2024-03-30T14:50:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:23.886+0000 7ffb5e3fb700 1 librados: shutdown 2024-03-30T14:50:24.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:24 smithi028 ceph-mon[29876]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:24.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:24 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:24.530 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-03-30T14:50:24.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables jewel 2024-03-30T14:50:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:25.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:25 smithi028 ceph-mon[29876]: from='client.14180 v1:172.21.15.28:0/3460120976' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 -- 172.21.15.28:0/1082097681 >> v1:172.21.15.28:6789/0 conn(0x7f6e20100420 legacy=0x7f6e20100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 -- 172.21.15.28:0/1082097681 shutdown_connections 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 -- 172.21.15.28:0/1082097681 >> 172.21.15.28:0/1082097681 conn(0x7f6e200fbcd0 msgr2=0x7f6e200fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 -- 172.21.15.28:0/1082097681 shutdown_connections 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 -- 172.21.15.28:0/1082097681 wait complete. 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 librados: starting msgr at 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.257+0000 7f6e28019700 1 librados: starting objecter 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 Processor -- start 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 -- start start 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 librados: setting wanted keys 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 librados: calling monclient init 2024-03-30T14:50:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6e2019cdc0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e25db5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6e20100420 0x7f6e2019c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59730/0 (socket says 172.21.15.28:59730) 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e25db5700 1 -- 172.21.15.28:0/2834740217 learned_addr learned my addr 172.21.15.28:0/2834740217 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 10 client.?.objecter ms_handle_connect 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870052890 0 0) 0x7f6e2019cdc0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e0c0056e0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2377716692 0 0) 0x7f6e0c0056e0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e2019cdc0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6e1c004750 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2483911798 0 0) 0x7f6e2019cdc0 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2019cf90 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.258+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2019d280 con 0x7f6e20100420 2024-03-30T14:50:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e28019700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e2019f7f0 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e28019700 1 librados: init done 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6e1c001c00 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f6e1c004d20 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f6e1c005030 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.259+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e04005120 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 869275787 0 0) 0x7f6e1c047790 con 0x7f6e20100420 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 10 client.14182.objecter ms_dispatch 0x7f6e20001550 osd_map(3..3 src has 1..3) 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 3 client.14182.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 3 client.14182.objecter handle_osd_map decoding full epoch 3 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 20 client.14182.objecter dump_active .. 0 homeless 2024-03-30T14:50:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.260+0000 7f6e16ffd700 10 client.14182.objecter ms_handle_connect 0x7f6e0c03bb40 2024-03-30T14:50:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.265+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6e1c0174e0 con 0x7f6e20100420 2024-03-30T14:50:26.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:26 smithi028 ceph-mon[29876]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:26.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:26.472+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) -- 0x7f6e04005410 con 0x7f6e20100420 2024-03-30T14:50:27.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.214+0000 7f6e16ffd700 1 -- 172.21.15.28:0/2834740217 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) ==== 120+0+0 (unknown 2099115640 0 0) 0x7f6e1c016de0 con 0x7f6e20100420 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 10 librados: watch_flush enter 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 10 librados: watch_flush exit 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 20 client.14182.objecter shutdown clearing up homeless session... 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 10 client.14182.objecter successfully canceled tick 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 >> v1:172.21.15.28:6800/3720192247 conn(0x7f6e0c03bb40 legacy=0x7f6e0c03e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 >> v1:172.21.15.28:6789/0 conn(0x7f6e20100420 legacy=0x7f6e2019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:27.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 shutdown_connections 2024-03-30T14:50:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 >> 172.21.15.28:0/2834740217 conn(0x7f6e200fbcd0 msgr2=0x7f6e20108940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 shutdown_connections 2024-03-30T14:50:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 -- 172.21.15.28:0/2834740217 wait complete. 2024-03-30T14:50:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:27.217+0000 7f6e28019700 1 librados: shutdown 2024-03-30T14:50:27.219 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to jewel 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:27.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:27 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:27.910 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-03-30T14:50:27.911 INFO:tasks.cephadm:Adding mon.c on smithi028 2024-03-30T14:50:27.911 INFO:tasks.cephadm:Adding mon.b on smithi134 2024-03-30T14:50:27.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b' 2024-03-30T14:50:28.145 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:28.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:28.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-30T14:50:28.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:28.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:28.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:28.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:28.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:29.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:29 smithi028 ceph-mon[29876]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:29.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:29 smithi028 ceph-mon[29876]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:29.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:29 smithi028 ceph-mon[29876]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:50:29.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 -- 172.21.15.134:0/1187028882 >> v1:172.21.15.28:6789/0 conn(0x7fbefc106a70 legacy=0x7fbefc106e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:29.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 -- 172.21.15.134:0/1187028882 shutdown_connections 2024-03-30T14:50:29.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 -- 172.21.15.134:0/1187028882 >> 172.21.15.134:0/1187028882 conn(0x7fbefc0fc580 msgr2=0x7fbefc0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 -- 172.21.15.134:0/1187028882 shutdown_connections 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 -- 172.21.15.134:0/1187028882 wait complete. 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 librados: starting msgr at 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.654+0000 7fbf03669700 1 librados: starting objecter 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf03669700 1 Processor -- start 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf03669700 1 -- start start 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf03669700 1 librados: setting wanted keys 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf03669700 1 librados: calling monclient init 2024-03-30T14:50:29.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf03669700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbefc198190 con 0x7fbefc106a70 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf01405700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbefc106a70 0x7fbefc197a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.134:47158/0 (socket says 172.21.15.134:47158) 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbf01405700 1 -- 172.21.15.134:0/2705712507 learned_addr learned my addr 172.21.15.134:0/2705712507 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbef27fc700 10 client.?.objecter ms_handle_connect 0x7fbefc106a70 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbef27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709340026 0 0) 0x7fbefc198190 con 0x7fbefc106a70 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.655+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbee80056e0 con 0x7fbefc106a70 2024-03-30T14:50:29.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3420326027 0 0) 0x7fbee80056e0 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbefc198190 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fbef80047f0 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1247956867 0 0) 0x7fbefc198190 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc198360 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc198650 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbf03669700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbefc19fcc0 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbf03669700 1 librados: init done 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbef8001a80 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.656+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7fbef8005e30 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee0005120 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7fbef8011420 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7fbef8047630 con 0x7fbefc106a70 2024-03-30T14:50:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 10 client.14184.objecter ms_dispatch 0x7fbefc001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.657+0000 7fbef27fc700 20 client.14184.objecter dump_active .. 0 homeless 2024-03-30T14:50:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.658+0000 7fbef27fc700 10 client.14184.objecter ms_handle_connect 0x7fbee803bb90 2024-03-30T14:50:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.662+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbefc1075f0 con 0x7fbefc106a70 2024-03-30T14:50:29.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.867+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b", "target": ["mon-mgr", ""]}) -- 0x7fbee0000c60 con 0x7fbee803bb90 2024-03-30T14:50:29.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.873+0000 7fbef27fc700 1 -- 172.21.15.134:0/2705712507 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7fbee0000c60 con 0x7fbee803bb90 2024-03-30T14:50:29.874 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2024-03-30T14:50:29.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 10 librados: watch_flush enter 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 10 librados: watch_flush exit 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 20 client.14184.objecter shutdown clearing up homeless session... 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 10 client.14184.objecter successfully canceled tick 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 >> v1:172.21.15.28:6800/3720192247 conn(0x7fbee803bb90 legacy=0x7fbee803e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 >> v1:172.21.15.28:6789/0 conn(0x7fbefc106a70 legacy=0x7fbefc197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 shutdown_connections 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.875+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 >> 172.21.15.134:0/2705712507 conn(0x7fbefc0fc580 msgr2=0x7fbefc108ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.876+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 shutdown_connections 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.876+0000 7fbf03669700 1 -- 172.21.15.134:0/2705712507 wait complete. 2024-03-30T14:50:29.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:29.876+0000 7fbf03669700 1 librados: shutdown 2024-03-30T14:50:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:30.483 DEBUG:teuthology.orchestra.run.smithi028:mon.c> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-30T14:50:30.485 DEBUG:teuthology.orchestra.run.smithi134:mon.b> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-30T14:50:30.488 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T14:50:30.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:50:31.179 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:31 smithi028 ceph-mon[29876]: from='client.14184 v1:172.21.15.134:0/2705712507' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:31 smithi028 ceph-mon[29876]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b;count:3 2024-03-30T14:50:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:31 smithi028 ceph-mon[29876]: Deploying daemon mon.b on smithi134 2024-03-30T14:50:32.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:32 smithi028 ceph-mon[29876]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:33.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:33 smithi028 ceph-mon[29876]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:34.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.360+0000 7f5d0de71700 1 -- 172.21.15.134:0/1840874313 >> v1:172.21.15.28:6789/0 conn(0x7f5d08106a60 legacy=0x7f5d08106e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.360+0000 7f5d0de71700 1 -- 172.21.15.134:0/1840874313 shutdown_connections 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.360+0000 7f5d0de71700 1 -- 172.21.15.134:0/1840874313 >> 172.21.15.134:0/1840874313 conn(0x7f5d08076570 msgr2=0x7f5d08076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.360+0000 7f5d0de71700 1 -- 172.21.15.134:0/1840874313 shutdown_connections 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.360+0000 7f5d0de71700 1 -- 172.21.15.134:0/1840874313 wait complete. 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 librados: starting msgr at 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 librados: starting objecter 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 Processor -- start 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 -- start start 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 librados: setting wanted keys 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 librados: calling monclient init 2024-03-30T14:50:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d0de71700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d0819bd80 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d077fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d08106a60 0x7f5d0819b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.134:57076/0 (socket says 172.21.15.134:57076) 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d077fe700 1 -- 172.21.15.134:0/3900798682 learned_addr learned my addr 172.21.15.134:0/3900798682 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d04ff9700 10 client.?.objecter ms_handle_connect 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.361+0000 7f5d04ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300622011 0 0) 0x7f5d0819bd80 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf00056e0 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2370243963 0 0) 0x7f5cf00056e0 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d0819bd80 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f5cf8001840 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1002065082 0 0) 0x7f5d0819bd80 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0819bf50 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0819e400 con 0x7f5d08106a60 2024-03-30T14:50:34.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d0de71700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d0819e7a0 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d0de71700 1 librados: init done 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5cf8003ae0 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.362+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2253323276 0 0) 0x7f5cf8004fd0 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf4005120 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f5cf8003dc0 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7f5cf8047960 con 0x7f5d08106a60 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 10 client.14186.objecter ms_dispatch 0x7f5d08001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.363+0000 7f5d04ff9700 20 client.14186.objecter dump_active .. 0 homeless 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.364+0000 7f5d04ff9700 10 client.14186.objecter ms_handle_connect 0x7f5cf003bb90 2024-03-30T14:50:34.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.368+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5cf801eb70 con 0x7f5d08106a60 2024-03-30T14:50:34.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.652+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5cf4005410 con 0x7f5d08106a60 2024-03-30T14:50:34.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.653+0000 7f5d04ff9700 1 -- 172.21.15.134:0/3900798682 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 1591659821) 0x7f5cf80040b0 con 0x7f5d08106a60 2024-03-30T14:50:34.654 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:50:34.654 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:48:46.261152Z","created":"2024-03-30T14:48:46.261152Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T14:50:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 10 librados: watch_flush enter 2024-03-30T14:50:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 10 librados: watch_flush exit 2024-03-30T14:50:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 20 client.14186.objecter shutdown clearing up homeless session... 2024-03-30T14:50:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 10 client.14186.objecter successfully canceled tick 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 >> v1:172.21.15.28:6800/3720192247 conn(0x7f5cf003bb90 legacy=0x7f5cf003e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 >> v1:172.21.15.28:6789/0 conn(0x7f5d08106a60 legacy=0x7f5d0819b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 shutdown_connections 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 >> 172.21.15.134:0/3900798682 conn(0x7f5d08076570 msgr2=0x7f5d081025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.655+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 shutdown_connections 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.656+0000 7f5d0de71700 1 -- 172.21.15.134:0/3900798682 wait complete. 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:34.656+0000 7f5d0de71700 1 librados: shutdown 2024-03-30T14:50:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:50:34.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:50:34.705 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:50:34.706 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:50:34.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/3900798682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:35.067 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:50:35.067 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:50:35.067 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:50:35.067 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:50:35.067 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.ttl: 2592000 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T14:50:35.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f007ac57-06de-4635-b8b3-adbcc577efeb 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810234712992, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810234713465, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711810234, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f007ac57-06de-4635-b8b3-adbcc577efeb", "db_session_id": "INLPY2QNICI0APD51QOF", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810234713560, "job": 1, "event": "recovery_finished"} 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5642dfe00000 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: DB pointer 0x5642dfdea000 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T14:50:35.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: ** DB Stats ** 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: ** Compaction Stats [default] ** 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:35.070 INFO:journalctl@ceph.mon.b.smithi134.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: ** Compaction Stats [default] ** 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:50:35.071 INFO:journalctl@ceph.mon.b.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: Cumulative compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: Interval compaction: 0.00 GB write, 0.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: Block cache BinnedLRUCache@0x5642dfd0f090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: using public_addrv v1:172.21.15.134:6789/0 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: starting mon.b rank -1 at public addrs v1:172.21.15.134:6789/0 at bind addrs v1:172.21.15.134:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(???) e0 preinit fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:35.072 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).mds e1 new map 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: e1 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: legacy client fscid: -1 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout: No filesystems configured 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T14:50:35.073 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:50:35.074 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.075 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='client.14184 v1:172.21.15.134:0/2705712507' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b;count:3 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: Deploying daemon mon.b on smithi134 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/3900798682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:34 smithi134 ceph-mon[33714]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T14:50:35.076 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:35 smithi134 bash[33640]: 1e3e5360cc5947739d616968b883eebafaea0c38938d9200319311f68ad87312 2024-03-30T14:50:35.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:35 smithi134 systemd[1]: Started Ceph mon.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:50:36.480 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T14:50:36.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:50:36.733 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:50:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.749+0000 7f9980749700 1 -- 172.21.15.134:0/1685567348 >> v1:172.21.15.28:6789/0 conn(0x7f9978100400 legacy=0x7f99781007e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.749+0000 7f9980749700 1 -- 172.21.15.134:0/1685567348 shutdown_connections 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.749+0000 7f9980749700 1 -- 172.21.15.134:0/1685567348 >> 172.21.15.134:0/1685567348 conn(0x7f9978077330 msgr2=0x7f9978077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.750+0000 7f9980749700 1 -- 172.21.15.134:0/1685567348 shutdown_connections 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.750+0000 7f9980749700 1 -- 172.21.15.134:0/1685567348 wait complete. 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.750+0000 7f9980749700 1 librados: starting msgr at 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.750+0000 7f9980749700 1 librados: starting objecter 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.750+0000 7f9980749700 1 Processor -- start 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f9980749700 1 -- start start 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f9980749700 1 librados: setting wanted keys 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f9980749700 1 librados: calling monclient init 2024-03-30T14:50:39.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f9980749700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99781a2e30 con 0x7f997819cf60 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f9980749700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99781a4010 con 0x7f9978100400 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f997e4e5700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9978100400 0x7f997819c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:44048/0 (socket says 172.21.15.134:44048) 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f997e4e5700 1 -- 172.21.15.134:0/1145329830 learned_addr learned my addr 172.21.15.134:0/1145329830 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 10 client.?.objecter ms_handle_connect 0x7f9978100400 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 10 client.?.objecter ms_handle_connect 0x7f997819cf60 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678391194 0 0) 0x7f99781a2e30 con 0x7f997819cf60 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.751+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99640035b0 con 0x7f997819cf60 2024-03-30T14:50:39.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3838908216 0 0) 0x7f99640035b0 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99781a2e30 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 2356309363 0 0) 0x7f99680017e0 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1915746381 0 0) 0x7f99781a2e30 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 >> v1:172.21.15.134:6789/0 conn(0x7f9978100400 legacy=0x7f997819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99781a51f0 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99781a3060 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f9980749700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99781a3560 con 0x7f997819cf60 2024-03-30T14:50:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.752+0000 7f9980749700 1 librados: init done 2024-03-30T14:50:39.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.753+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9978063870 con 0x7f997819cf60 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.754+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9968003b20 con 0x7f997819cf60 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.754+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 2356309363 0 0) 0x7f9968004980 con 0x7f997819cf60 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f996800ff70 con 0x7f997819cf60 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7f9968047050 con 0x7f997819cf60 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 10 client.14190.objecter ms_dispatch 0x7f9978001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.755+0000 7f996f7fe700 20 client.14190.objecter dump_active .. 0 homeless 2024-03-30T14:50:39.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.756+0000 7f996f7fe700 10 client.14190.objecter ms_handle_connect 0x7f996403bc10 2024-03-30T14:50:39.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:39.760+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9968015f40 con 0x7f997819cf60 2024-03-30T14:50:40.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.038+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f997810caa0 con 0x7f997819cf60 2024-03-30T14:50:40.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.038+0000 7f996f7fe700 1 -- 172.21.15.134:0/1145329830 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+912 (unknown 3557084514 0 1497509342) 0x7f9968015840 con 0x7f997819cf60 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: mon.a calling monitor election 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.039 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: mon.b calling monitor election 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: monmap epoch 2 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: last_changed 2024-03-30T14:50:34.734322+0000 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: min_mon_release 19 (squid) 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: election_strategy: 1 2024-03-30T14:50:40.040 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: fsmap 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: mgrmap e13: y(active, since 42s) 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: overall HEALTH_OK 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:40.041 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:40.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:50:40.042 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":2,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:50:34.734322Z","created":"2024-03-30T14:48:46.261152Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T14:50:40.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 10 librados: watch_flush enter 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 10 librados: watch_flush exit 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 20 client.14190.objecter shutdown clearing up homeless session... 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 10 client.14190.objecter successfully canceled tick 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 >> v1:172.21.15.28:6800/3720192247 conn(0x7f996403bc10 legacy=0x7f996403e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 >> v1:172.21.15.28:6789/0 conn(0x7f997819cf60 legacy=0x7f99781a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 shutdown_connections 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 >> 172.21.15.134:0/1145329830 conn(0x7f9978077330 msgr2=0x7f99781098b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 shutdown_connections 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 -- 172.21.15.134:0/1145329830 wait complete. 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:40.040+0000 7f9980749700 1 librados: shutdown 2024-03-30T14:50:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 2 2024-03-30T14:50:40.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: mon.a calling monitor election 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: mon.b calling monitor election 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: monmap epoch 2 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:40.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: last_changed 2024-03-30T14:50:34.734322+0000 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: min_mon_release 19 (squid) 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: election_strategy: 1 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: fsmap 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: mgrmap e13: y(active, since 42s) 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: overall HEALTH_OK 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:40.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:40 smithi134 ceph-mon[33714]: Deploying daemon mon.c on smithi028 2024-03-30T14:50:40.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/1145329830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:40.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:40 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:41.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:40 smithi028 ceph-mon[29876]: Deploying daemon mon.c on smithi028 2024-03-30T14:50:41.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/1145329830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:41.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:40 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:41.642 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T14:50:41.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T14:50:41.888 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:50:41.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:41 smithi134 ceph-mon[33714]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:42.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:41 smithi028 ceph-mon[29876]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.131 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:42 smithi028 systemd[1]: Starting Ceph mon.c for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:50:43.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- 172.21.15.134:0/2536592592 >> v1:172.21.15.28:6789/0 conn(0x7fb8c8100620 legacy=0x7fb8c8100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- 172.21.15.134:0/2536592592 shutdown_connections 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- 172.21.15.134:0/2536592592 >> 172.21.15.134:0/2536592592 conn(0x7fb8c80772c0 msgr2=0x7fb8c80776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- 172.21.15.134:0/2536592592 shutdown_connections 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- 172.21.15.134:0/2536592592 wait complete. 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 librados: starting msgr at 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 librados: starting objecter 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 Processor -- start 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 -- start start 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 librados: setting wanted keys 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.410+0000 7fb8ce488700 1 librados: calling monclient init 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8ce488700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb8c8102140 con 0x7fb8c8101cb0 2024-03-30T14:50:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8ce488700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb8c8102310 con 0x7fb8c8100620 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c7fff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fb8c8100620 0x7fb8c81015a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:44010/0 (socket says 172.21.15.134:44010) 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c7fff700 1 -- 172.21.15.134:0/2576209954 learned_addr learned my addr 172.21.15.134:0/2576209954 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 10 client.?.objecter ms_handle_connect 0x7fb8c8100620 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 10 client.?.objecter ms_handle_connect 0x7fb8c8101cb0 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371569915 0 0) 0x7fb8c8102310 con 0x7fb8c8100620 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b00035b0 con 0x7fb8c8100620 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611544825 0 0) 0x7fb8c8102140 con 0x7fb8c8101cb0 2024-03-30T14:50:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c8102310 con 0x7fb8c8101cb0 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2255990076 0 0) 0x7fb8b00035b0 con 0x7fb8c8100620 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8c8102140 con 0x7fb8c8100620 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2562307430 0 0) 0x7fb8c8102310 con 0x7fb8c8101cb0 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8b00035b0 con 0x7fb8c8101cb0 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 2356309363 0 0) 0x7fb8b80044f0 con 0x7fb8c8100620 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.411+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 2356309363 0 0) 0x7fb8bc0017e0 con 0x7fb8c8101cb0 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3957125931 0 0) 0x7fb8c8102140 con 0x7fb8c8100620 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 >> v1:172.21.15.28:6789/0 conn(0x7fb8c8101cb0 legacy=0x7fb8c81a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:43.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c81a7480 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c81a5460 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8ce488700 10 client.24099.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8c81a5960 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8ce488700 1 librados: init done 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb8b8001ba0 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 2356309363 0 0) 0x7fb8b8004c50 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7fb8b8004e90 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.412+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b4005120 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7fb8b80476d0 con 0x7fb8c8100620 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 10 client.24099.objecter ms_dispatch 0x7fb8c8001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 3 client.24099.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 3 client.24099.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 20 client.24099.objecter dump_active .. 0 homeless 2024-03-30T14:50:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.413+0000 7fb8c57fa700 10 client.24099.objecter ms_handle_connect 0x7fb8b003bd80 2024-03-30T14:50:43.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.418+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb8b8017470 con 0x7fb8c8100620 2024-03-30T14:50:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[29876]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 podman[41576]: 2024-03-30 14:50:43.131002826 +0000 UTC m=+0.111206485 container create 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 podman[41576]: 2024-03-30 14:50:43.037612245 +0000 UTC m=+0.017815923 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 podman[41576]: 2024-03-30 14:50:43.324768832 +0000 UTC m=+0.304972503 container init 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 podman[41576]: 2024-03-30 14:50:43.32992237 +0000 UTC m=+0.310126026 container start 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev), process ceph-mon, pid 7 2024-03-30T14:50:43.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pidfile_write: ignore empty --pid-file 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: load: jerasure load: lrc 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: RocksDB version: 7.9.2 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Git sha 0 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Compile date 2024-03-29 15:06:16 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: DB SUMMARY 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: DB Session ID: 9UBLV4X2TQ1W7W7F0JIP 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: CURRENT file: CURRENT 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-30T14:50:43.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.error_if_exists: 0 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.create_if_missing: 0 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.paranoid_checks: 1 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.env: 0x561261eb6c60 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.info_log: 0x561262aa1b00 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.statistics: (nil) 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.use_fsync: 0 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_log_file_size: 0 2024-03-30T14:50:43.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_fallocate: 1 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.use_direct_reads: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.db_log_dir: 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.wal_dir: 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T14:50:43.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.write_buffer_manager: 0x561262a741e0 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T14:50:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T14:50:43.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.465+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb8b8014f10 con 0x7fb8c8100620 2024-03-30T14:50:43.466 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:43 smithi134 ceph-mon[33714]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:43.693+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb8b4005410 con 0x7fb8c8100620 2024-03-30T14:50:43.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T14:50:43.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T14:50:43.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T14:50:43.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.unordered_write: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.row_cache: None 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.wal_filter: None 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.two_write_queues: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.wal_compression: 0 2024-03-30T14:50:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.atomic_flush: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.log_readahead_size: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T14:50:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_background_jobs: 2 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_background_compactions: -1 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_subcompactions: 1 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T14:50:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_open_files: -1 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_background_flushes: -1 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Compression algorithms supported: 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kZSTD supported: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kXpressCompression supported: 0 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kZlibCompression supported: 1 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kSnappyCompression supported: 1 2024-03-30T14:50:43.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kLZ4Compression supported: 1 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: kBZip2Compression supported: 0 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.merge_operator: 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_filter: None 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561262aa1c20) 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-30T14:50:43.728 INFO:journalctl@ceph.mon.c.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_type: 0 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_index_type: 0 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_shortening: 1 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: checksum: 4 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: no_block_cache: 0 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache: 0x561262aa5090 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_options: 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: capacity : 536870912 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: num_shard_bits : 4 2024-03-30T14:50:43.729 INFO:journalctl@ceph.mon.c.smithi028.stdout: strict_capacity_limit : 0 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_cache_compressed: (nil) 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: persistent_cache: (nil) 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size: 4096 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_size_deviation: 10 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_restart_interval: 16 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: index_block_restart_interval: 1 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: metadata_block_size: 4096 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: partition_filters: 0 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: use_delta_encoding: 1 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: filter_policy: bloomfilter 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: whole_key_filtering: 1 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: verify_compression: 0 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-30T14:50:43.730 INFO:journalctl@ceph.mon.c.smithi028.stdout: format_version: 5 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_index_compression: 1 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: block_align: 0 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: max_auto_readahead_size: 262144 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: prepopulate_block_cache: 0 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: initial_auto_readahead_size: 8192 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression: NoCompression 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.num_levels: 7 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T14:50:43.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T14:50:43.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T14:50:43.733 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T14:50:43.734 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.table_properties_collectors: 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.inplace_update_support: 0 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T14:50:43.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.bloom_locality: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.max_successive_merges: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.ttl: 2592000 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T14:50:43.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enable_blob_files: false 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.min_blob_size: 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 39cb13ee-4782-4125-aadb-36a6bde79a5f 2024-03-30T14:50:43.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810243441045, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810243441445, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711810243, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "39cb13ee-4782-4125-aadb-36a6bde79a5f", "db_session_id": "9UBLV4X2TQ1W7W7F0JIP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711810243441531, "job": 1, "event": "recovery_finished"} 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561262b96000 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: DB pointer 0x561262b80000 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** DB Stats ** 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:50:43.738 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-30T14:50:43.739 INFO:journalctl@ceph.mon.c.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T14:50:43.740 INFO:journalctl@ceph.mon.c.smithi028.stdout: Block cache BinnedLRUCache@0x561262aa5090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: using public_addrv v1:172.21.15.28:6790/0 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: starting mon.c rank -1 at public addrs v1:172.21.15.28:6790/0 at bind addrs v1:172.21.15.28:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(???) e0 preinit fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).mds e1 new map 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: e1 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: legacy client fscid: -1 2024-03-30T14:50:43.741 INFO:journalctl@ceph.mon.c.smithi028.stdout: 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout: No filesystems configured 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-03-30T14:50:43.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2834740217' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='client.14184 v1:172.21.15.134:0/2705712507' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:43.744 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi134:[v1:172.21.15.134:6789]=b;count:3 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Deploying daemon mon.b on smithi134 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/3900798682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.a calling monitor election 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.b calling monitor election 2024-03-30T14:50:43.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: monmap epoch 2 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: last_changed 2024-03-30T14:50:34.734322+0000 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: min_mon_release 19 (squid) 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: election_strategy: 1 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: fsmap 2024-03-30T14:50:43.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mgrmap e13: y(active, since 42s) 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: overall HEALTH_OK 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: Deploying daemon mon.c on smithi028 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/1145329830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 ceph-mon[41614]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-30T14:50:43.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 bash[41576]: 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a 2024-03-30T14:50:43.748 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:43 smithi028 systemd[1]: Started Ceph mon.c for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:50:48.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.412+0000 7fb8cdc87700 10 client.24099.objecter tick 2024-03-30T14:50:48.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.469+0000 7fb8c57fa700 1 -- 172.21.15.134:0/2576209954 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1127 (unknown 409899127 0 633015442) 0x7fb8b8016d70 con 0x7fb8c8100620 2024-03-30T14:50:48.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-30T14:50:48.471 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":3,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","modified":"2024-03-30T14:50:43.461080Z","created":"2024-03-30T14:48:46.261152Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-30T14:50:48.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.472+0000 7fb8ce488700 10 librados: watch_flush enter 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.472+0000 7fb8ce488700 10 librados: watch_flush exit 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.472+0000 7fb8ce488700 20 client.24099.objecter shutdown clearing up homeless session... 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.472+0000 7fb8ce488700 10 client.24099.objecter successfully canceled tick 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 >> v1:172.21.15.28:6800/3720192247 conn(0x7fb8b003bd80 legacy=0x7fb8b003e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 >> v1:172.21.15.134:6789/0 conn(0x7fb8c8100620 legacy=0x7fb8c81015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 shutdown_connections 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 >> 172.21.15.134:0/2576209954 conn(0x7fb8c80772c0 msgr2=0x7fb8c8109710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 shutdown_connections 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 -- 172.21.15.134:0/2576209954 wait complete. 2024-03-30T14:50:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:48.473+0000 7fb8ce488700 1 librados: shutdown 2024-03-30T14:50:48.475 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 3 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: mon.a calling monitor election 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: mon.b calling monitor election 2024-03-30T14:50:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/2576209954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: mon.c calling monitor election 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T14:50:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: monmap epoch 3 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: last_changed 2024-03-30T14:50:43.461080+0000 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: min_mon_release 19 (squid) 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: election_strategy: 1 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: 2: v1:172.21.15.28:6790/0 mon.c 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: fsmap 2024-03-30T14:50:48.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: mgrmap e13: y(active, since 51s) 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: overall HEALTH_OK 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:48.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: mon.a calling monitor election 2024-03-30T14:50:48.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: mon.b calling monitor election 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/2576209954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: mon.c calling monitor election 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T14:50:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: monmap epoch 3 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: last_changed 2024-03-30T14:50:43.461080+0000 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: min_mon_release 19 (squid) 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: election_strategy: 1 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: 2: v1:172.21.15.28:6790/0 mon.c 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: fsmap 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: mgrmap e13: y(active, since 51s) 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: overall HEALTH_OK 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: mon.a calling monitor election 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: mon.b calling monitor election 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/2576209954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: mon.c calling monitor election 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: monmap epoch 3 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: last_changed 2024-03-30T14:50:43.461080+0000 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: created 2024-03-30T14:48:46.261152+0000 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: min_mon_release 19 (squid) 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: election_strategy: 1 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: 0: v1:172.21.15.28:6789/0 mon.a 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: 1: v1:172.21.15.134:6789/0 mon.b 2024-03-30T14:50:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: 2: v1:172.21.15.28:6790/0 mon.c 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: fsmap 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: mgrmap e13: y(active, since 51s) 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: overall HEALTH_OK 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:49.066 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T14:50:49.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T14:50:49.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:49.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[29876]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:49.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:49.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[41614]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:49 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:49.856 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:49.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:49 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:49 smithi134 ceph-mon[33714]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:49 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:50:51.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:51.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 -- 172.21.15.28:0/3025702675 >> v1:172.21.15.28:6789/0 conn(0x7f9fd8106a30 legacy=0x7f9fd8106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 -- 172.21.15.28:0/3025702675 shutdown_connections 2024-03-30T14:50:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 -- 172.21.15.28:0/3025702675 >> 172.21.15.28:0/3025702675 conn(0x7f9fd80fc560 msgr2=0x7f9fd80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 -- 172.21.15.28:0/3025702675 shutdown_connections 2024-03-30T14:50:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 -- 172.21.15.28:0/3025702675 wait complete. 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 librados: starting msgr at 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.563+0000 7f9fdee4d700 1 librados: starting objecter 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 Processor -- start 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 -- start start 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 librados: setting wanted keys 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 librados: calling monclient init 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9fd81a4ab0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9fd81a5cf0 con 0x7f9fd8106a30 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdee4d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9fd81a6f30 con 0x7f9fd819aed0 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdd3ea700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9fd81a0da0 0x7f9fd81a31b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33800/0 (socket says 172.21.15.28:33800) 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fdd3ea700 1 -- 172.21.15.28:0/640144189 learned_addr learned my addr 172.21.15.28:0/640144189 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd7fff700 1 --1- 172.21.15.28:0/640144189 >> v1:172.21.15.28:6790/0 conn(0x7f9fd819aed0 0x7f9fd819f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58062/0 (socket says 172.21.15.28:58062) 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter ms_handle_connect 0x7f9fd819aed0 2024-03-30T14:50:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter ms_handle_connect 0x7f9fd81a0da0 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822602572 0 0) 0x7f9fd81a4ab0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fc8003bf0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter ms_handle_connect 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559511561 0 0) 0x7f9fd81a5cf0 con 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.564+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fd81a4ab0 con 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1956469762 0 0) 0x7f9fd81a4ab0 con 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fd81a5cf0 con 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9fcc004920 con 0x7f9fd8106a30 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120244163 0 0) 0x7f9fc8003bf0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fd81a4ab0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9fd0001b70 con 0x7f9fd81a0da0 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 538263462 0 0) 0x7f9fd81a4ab0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 >> v1:172.21.15.28:6790/0 conn(0x7f9fd819aed0 legacy=0x7f9fd819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 >> v1:172.21.15.134:6789/0 conn(0x7f9fd8106a30 legacy=0x7f9fd819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd81a8170 con 0x7f9fd81a0da0 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd81a4ce0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fdee4d700 10 client.14199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9fd81a51e0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.565+0000 7f9fdee4d700 1 librados: init done 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.566+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd8063870 con 0x7f9fd81a0da0 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.568+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9fd00051d0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.568+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9fd0005fc0 con 0x7f9fd81a0da0 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f9fd0011690 con 0x7f9fd81a0da0 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7f9fd0048730 con 0x7f9fd81a0da0 2024-03-30T14:50:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 10 client.14199.objecter ms_dispatch 0x7f9fd8001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 3 client.14199.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 3 client.14199.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 20 client.14199.objecter dump_active .. 0 homeless 2024-03-30T14:50:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.569+0000 7f9fd5ffb700 10 client.14199.objecter ms_handle_connect 0x7f9fc803fd70 2024-03-30T14:50:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.573+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9fd0017680 con 0x7f9fd81a0da0 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.788+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9fd806aa60 con 0x7f9fd81a0da0 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.788+0000 7f9fd5ffb700 1 -- 172.21.15.28:0/640144189 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+191 (unknown 2189217731 0 1202660526) 0x7f9fd0016f80 con 0x7f9fd81a0da0 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:50:51.789 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = 172.21.15.28:6789/0 172.21.15.134:6789/0 v1:172.21.15.28:6790/0 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 10 librados: watch_flush enter 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 10 librados: watch_flush exit 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 20 client.14199.objecter shutdown clearing up homeless session... 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 10 client.14199.objecter successfully canceled tick 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 >> v1:172.21.15.28:6800/3720192247 conn(0x7f9fc803fd70 legacy=0x7f9fc8042230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.790+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 >> v1:172.21.15.28:6789/0 conn(0x7f9fd81a0da0 legacy=0x7f9fd81a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.791+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 shutdown_connections 2024-03-30T14:50:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.791+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 >> 172.21.15.28:0/640144189 conn(0x7f9fd80fc560 msgr2=0x7f9fd80fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.791+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 shutdown_connections 2024-03-30T14:50:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.791+0000 7f9fdee4d700 1 -- 172.21.15.28:0/640144189 wait complete. 2024-03-30T14:50:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:50:51.791+0000 7f9fdee4d700 1 librados: shutdown 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: Reconfiguring daemon mon.a on smithi028 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/640144189' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: Reconfiguring daemon mon.a on smithi028 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/640144189' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:52.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:50:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T14:50:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: Reconfiguring daemon mon.a on smithi028 2024-03-30T14:50:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/640144189' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:52.859 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T14:50:52.860 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:52.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T14:50:52.890 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:52.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:52.957 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:50:52.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T14:50:52.992 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:50:52.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:50:53.063 INFO:tasks.cephadm:Adding mgr.y on smithi028 2024-03-30T14:50:53.063 INFO:tasks.cephadm:Adding mgr.x on smithi134 2024-03-30T14:50:53.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '2;smithi028=y;smithi134=x' 2024-03-30T14:50:53.338 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:50:53.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: Reconfiguring mon.c (monmap changed)... 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:53.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:53.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:53 smithi134 ceph-mon[33714]: Reconfiguring daemon mon.c on smithi028 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: Reconfiguring mon.c (monmap changed)... 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[29876]: Reconfiguring daemon mon.c on smithi028 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: Reconfiguring mon.c (monmap changed)... 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:53 smithi028 ceph-mon[41614]: Reconfiguring daemon mon.c on smithi028 2024-03-30T14:50:54.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.938+0000 7f531da22700 1 -- 172.21.15.134:0/2800940847 >> v1:172.21.15.28:6789/0 conn(0x7f5318100be0 legacy=0x7f5318101000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.938+0000 7f531da22700 1 -- 172.21.15.134:0/2800940847 shutdown_connections 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.938+0000 7f531da22700 1 -- 172.21.15.134:0/2800940847 >> 172.21.15.134:0/2800940847 conn(0x7f53180fc180 msgr2=0x7f53180fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.938+0000 7f531da22700 1 -- 172.21.15.134:0/2800940847 shutdown_connections 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.938+0000 7f531da22700 1 -- 172.21.15.134:0/2800940847 wait complete. 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 librados: starting msgr at 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 librados: starting objecter 2024-03-30T14:50:54.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 Processor -- start 2024-03-30T14:50:54.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 -- start start 2024-03-30T14:50:54.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 librados: setting wanted keys 2024-03-30T14:50:54.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.939+0000 7f531da22700 1 librados: calling monclient init 2024-03-30T14:50:54.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531da22700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53181a2110 con 0x7f531819e4a0 2024-03-30T14:50:54.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531da22700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53181a3330 con 0x7f5318100be0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531da22700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53181a4570 con 0x7f5318197000 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f5316ffd700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5318100be0 0x7f53181968f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:35896/0 (socket says 172.21.15.134:35896) 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f5316ffd700 1 -- 172.21.15.134:0/3367661863 learned_addr learned my addr 172.21.15.134:0/3367661863 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter ms_handle_connect 0x7f5318100be0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter ms_handle_connect 0x7f531819e4a0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter ms_handle_connect 0x7f5318197000 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 10 client.?.objecter resend_mon_ops 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324916812 0 0) 0x7f53181a3330 con 0x7f5318100be0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5300003c40 con 0x7f5318100be0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262928062 0 0) 0x7f53181a2110 con 0x7f531819e4a0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53181a3330 con 0x7f531819e4a0 2024-03-30T14:50:54.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087505942 0 0) 0x7f53181a4570 con 0x7f5318197000 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53181a2110 con 0x7f5318197000 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2270091776 0 0) 0x7f53181a3330 con 0x7f531819e4a0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53181a4570 con 0x7f531819e4a0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.940+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5310001b50 con 0x7f531819e4a0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2246212771 0 0) 0x7f5300003c40 con 0x7f5318100be0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53181a3330 con 0x7f5318100be0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2144613156 0 0) 0x7f53181a2110 con 0x7f5318197000 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5300003c40 con 0x7f5318197000 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53080044d0 con 0x7f5318100be0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f530c001600 con 0x7f5318197000 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3245144111 0 0) 0x7f53181a4570 con 0x7f531819e4a0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 >> v1:172.21.15.28:6790/0 conn(0x7f5318197000 legacy=0x7f531819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 >> v1:172.21.15.134:6789/0 conn(0x7f5318100be0 legacy=0x7f53181968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53181a57b0 con 0x7f531819e4a0 2024-03-30T14:50:54.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53181a2340 con 0x7f531819e4a0 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531da22700 10 client.14205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53181a2880 con 0x7f531819e4a0 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531da22700 1 librados: init done 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5310003f60 con 0x7f531819e4a0 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.941+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5310004ee0 con 0x7f531819e4a0 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f5310010310 con 0x7f531819e4a0 2024-03-30T14:50:54.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5304005150 con 0x7f531819e4a0 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7f5310046530 con 0x7f531819e4a0 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 10 client.14205.objecter ms_dispatch 0x7f5318001550 osd_map(4..4 src has 1..4) 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 3 client.14205.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 3 client.14205.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 20 client.14205.objecter dump_active .. 0 homeless 2024-03-30T14:50:54.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.942+0000 7f531ca20700 10 client.14205.objecter ms_handle_connect 0x7f530003bd00 2024-03-30T14:50:54.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:54.947+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5310016300 con 0x7f531819e4a0 2024-03-30T14:50:55.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.155+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi134=x", "target": ["mon-mgr", ""]}) -- 0x7f5304000bc0 con 0x7f530003bd00 2024-03-30T14:50:55.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.161+0000 7f531ca20700 1 -- 172.21.15.134:0/3367661863 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f5304000bc0 con 0x7f530003bd00 2024-03-30T14:50:55.162 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mgr update... 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 10 librados: watch_flush enter 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 10 librados: watch_flush exit 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 20 client.14205.objecter shutdown clearing up homeless session... 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 10 client.14205.objecter successfully canceled tick 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 >> v1:172.21.15.28:6800/3720192247 conn(0x7f530003bd00 legacy=0x7f530003e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 >> v1:172.21.15.28:6789/0 conn(0x7f531819e4a0 legacy=0x7f53181a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 shutdown_connections 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 >> 172.21.15.134:0/3367661863 conn(0x7f53180fc180 msgr2=0x7f53181035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 shutdown_connections 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 -- 172.21.15.134:0/3367661863 wait complete. 2024-03-30T14:50:55.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:50:55.163+0000 7f531da22700 1 librados: shutdown 2024-03-30T14:50:55.752 DEBUG:teuthology.orchestra.run.smithi134:mgr.x> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-30T14:50:55.754 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T14:50:55.755 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:50:55.755 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T14:50:55.773 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T14:50:55.774 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-30 14:50:42.865841725 +0000 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-30 14:45:21.993283203 +0000 2024-03-30T14:50:55.831 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-30 14:45:21.993283203 +0000 2024-03-30T14:50:55.832 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-30 14:45:21.993283203 +0000 2024-03-30T14:50:55.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T14:50:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-30T14:50:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-30T14:50:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000124581 s, 4.1 MB/s 2024-03-30T14:50:55.903 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T14:50:55.960 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-30 14:50:42.865841725 +0000 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-30 14:45:22.335273550 +0000 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-30 14:45:22.335273550 +0000 2024-03-30T14:50:56.021 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-30 14:45:22.335273550 +0000 2024-03-30T14:50:56.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: Reconfiguring mon.b (monmap changed)... 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: Reconfiguring daemon mon.b on smithi134 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='client.14205 v1:172.21.15.134:0/3367661863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi134=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: Saving service mgr spec with placement smithi028=y;smithi134=x;count:2 2024-03-30T14:50:56.070 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.071 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:55 smithi134 ceph-mon[33714]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: Reconfiguring mon.b (monmap changed)... 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: Reconfiguring daemon mon.b on smithi134 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='client.14205 v1:172.21.15.134:0/3367661863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi134=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: Saving service mgr spec with placement smithi028=y;smithi134=x;count:2 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[29876]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: Reconfiguring mon.b (monmap changed)... 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: Reconfiguring daemon mon.b on smithi134 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='client.14205 v1:172.21.15.134:0/3367661863' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi134=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: Saving service mgr spec with placement smithi028=y;smithi134=x;count:2 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:56.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:55 smithi028 ceph-mon[41614]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:56.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-30T14:50:56.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-30T14:50:56.089 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000116904 s, 4.4 MB/s 2024-03-30T14:50:56.090 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T14:50:56.148 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-30 14:50:42.866841697 +0000 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-30 14:45:22.676263925 +0000 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-30 14:45:22.676263925 +0000 2024-03-30T14:50:56.205 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-30 14:45:22.676263925 +0000 2024-03-30T14:50:56.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T14:50:56.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-30T14:50:56.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-30T14:50:56.273 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.0001053 s, 4.9 MB/s 2024-03-30T14:50:56.274 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T14:50:56.331 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-03-30T14:50:56.386 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T14:50:56.386 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.386 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-30 14:50:42.866841697 +0000 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-30 14:45:23.025254073 +0000 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-30 14:45:23.025254073 +0000 2024-03-30T14:50:56.387 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-30 14:45:23.025254073 +0000 2024-03-30T14:50:56.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T14:50:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-30T14:50:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-30T14:50:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.4117e-05 s, 5.4 MB/s 2024-03-30T14:50:56.453 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T14:50:56.509 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:50:56.509 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T14:50:56.526 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T14:50:56.526 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.581 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-30 14:50:33.568249766 +0000 2024-03-30T14:50:56.582 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-30 14:45:10.173355473 +0000 2024-03-30T14:50:56.582 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-30 14:45:10.173355473 +0000 2024-03-30T14:50:56.582 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-30 14:45:10.173355473 +0000 2024-03-30T14:50:56.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T14:50:56.649 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-30T14:50:56.649 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-30T14:50:56.649 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000207445 s, 2.5 MB/s 2024-03-30T14:50:56.650 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T14:50:56.707 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-03-30T14:50:56.762 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T14:50:56.762 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-30 14:50:33.568249766 +0000 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-30 14:45:10.529346516 +0000 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-30 14:45:10.529346516 +0000 2024-03-30T14:50:56.763 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-30 14:45:10.529346516 +0000 2024-03-30T14:50:56.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T14:50:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-30T14:50:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-30T14:50:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000198621 s, 2.6 MB/s 2024-03-30T14:50:56.831 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T14:50:56.886 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-03-30T14:50:56.940 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T14:50:56.940 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:56.940 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-30T14:50:56.940 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:56.940 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:56.941 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-30 14:50:33.569249741 +0000 2024-03-30T14:50:56.941 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-30 14:45:10.876337785 +0000 2024-03-30T14:50:56.941 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-30 14:45:10.876337785 +0000 2024-03-30T14:50:56.941 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-30 14:45:10.876337785 +0000 2024-03-30T14:50:56.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T14:50:57.006 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-30T14:50:57.007 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-30T14:50:57.007 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000238061 s, 2.2 MB/s 2024-03-30T14:50:57.008 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T14:50:57.064 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-03-30T14:50:57.119 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T14:50:57.119 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T14:50:57.119 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-30T14:50:57.119 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T14:50:57.119 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T14:50:57.120 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-30 14:50:33.569249741 +0000 2024-03-30T14:50:57.120 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-30 14:45:11.228328928 +0000 2024-03-30T14:50:57.120 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-30 14:45:11.228328928 +0000 2024-03-30T14:50:57.120 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-30 14:45:11.228328928 +0000 2024-03-30T14:50:57.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T14:50:57.186 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-30T14:50:57.186 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-30T14:50:57.186 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000265554 s, 1.9 MB/s 2024-03-30T14:50:57.187 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T14:50:57.245 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-03-30T14:50:57.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T14:50:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:50:57.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: Deploying daemon mgr.x on smithi134 2024-03-30T14:50:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:57 smithi134 ceph-mon[33714]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: Deploying daemon mgr.x on smithi134 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[29876]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:50:57.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: Deploying daemon mgr.x on smithi134 2024-03-30T14:50:57.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:57 smithi028 ceph-mon[41614]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:59.038 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:58 smithi134 podman[35795]: 2024-03-30 14:50:58.820312507 +0000 UTC m=+0.017475945 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-mon[33714]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 podman[35795]: 2024-03-30 14:50:59.142881707 +0000 UTC m=+0.340045126 container init 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 podman[35795]: 2024-03-30 14:50:59.148351396 +0000 UTC m=+0.345514816 container start 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-03-30T14:50:59.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 bash[35795]: 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 2024-03-30T14:50:59.431 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 systemd[1]: Started Ceph mgr.x for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0101533 s, 1.0 GB/s 2024-03-30T14:50:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-30T14:50:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[29876]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[41614]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:50:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:50:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:00.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-03-30T14:51:00.180 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:50:59.720+0000 7f80896af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:51:00.180 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:50:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:50:59.818+0000 7f80896af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:51:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:51:00.599 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:00.600 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:00.506+0000 7f80896af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:51:00.600 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:00.598+0000 7f80896af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:51:00.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:00.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:00 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:00.853 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:00.754+0000 7f80896af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:51:00.854 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:00.852+0000 7f80896af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:51:01.179 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:00.936+0000 7f80896af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:51:01.492 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:01.202+0000 7f80896af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:51:01.493 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:01.311+0000 7f80896af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:51:01.493 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:01.396+0000 7f80896af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:51:01.493 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:01.492+0000 7f80896af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T14:51:01.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: Reconfiguring daemon mgr.y on smithi028 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[29876]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:01.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: Reconfiguring daemon mgr.y on smithi028 2024-03-30T14:51:01.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:01 smithi028 ceph-mon[41614]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:01.929 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:01.668+0000 7f80896af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: Reconfiguring mgr.y (unknown last config time)... 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T14:51:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: Reconfiguring daemon mgr.y on smithi028 2024-03-30T14:51:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:01 smithi134 ceph-mon[33714]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.438+0000 7f0cab8c8700 1 -- 172.21.15.28:0/2793589438 >> v1:172.21.15.28:6789/0 conn(0x7f0ca4102050 legacy=0x7f0ca4102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.438+0000 7f0cab8c8700 1 -- 172.21.15.28:0/2793589438 shutdown_connections 2024-03-30T14:51:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.438+0000 7f0cab8c8700 1 -- 172.21.15.28:0/2793589438 >> 172.21.15.28:0/2793589438 conn(0x7f0ca40fd5f0 msgr2=0x7f0ca40ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:51:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.438+0000 7f0cab8c8700 1 -- 172.21.15.28:0/2793589438 shutdown_connections 2024-03-30T14:51:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.438+0000 7f0cab8c8700 1 -- 172.21.15.28:0/2793589438 wait complete. 2024-03-30T14:51:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 librados: starting msgr at 2024-03-30T14:51:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 librados: starting objecter 2024-03-30T14:51:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 Processor -- start 2024-03-30T14:51:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 -- start start 2024-03-30T14:51:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 librados: setting wanted keys 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 librados: calling monclient init 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ca41a47b0 con 0x7f0ca4102050 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ca41a59f0 con 0x7f0ca410a1c0 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0cab8c8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ca41a6c30 con 0x7f0ca4106480 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0ca9664700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ca4102050 0x7f0ca4075f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56472/0 (socket says 172.21.15.28:56472) 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0ca9664700 1 -- 172.21.15.28:0/99969592 learned_addr learned my addr 172.21.15.28:0/99969592 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter ms_handle_connect 0x7f0ca4102050 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter ms_handle_connect 0x7f0ca4106480 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter ms_handle_connect 0x7f0ca410a1c0 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.439+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650157406 0 0) 0x7f0ca41a47b0 con 0x7f0ca4102050 2024-03-30T14:51:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c900031f0 con 0x7f0ca4102050 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276771666 0 0) 0x7f0ca41a6c30 con 0x7f0ca4106480 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ca41a47b0 con 0x7f0ca4106480 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251874289 0 0) 0x7f0ca41a59f0 con 0x7f0ca410a1c0 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ca41a6c30 con 0x7f0ca410a1c0 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1464694901 0 0) 0x7f0c900031f0 con 0x7f0ca4102050 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ca41a59f0 con 0x7f0ca4102050 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0ca0001440 con 0x7f0ca4102050 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1028130450 0 0) 0x7f0ca41a47b0 con 0x7f0ca4106480 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c900031f0 con 0x7f0ca4106480 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c940018b0 con 0x7f0ca4106480 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 510416687 0 0) 0x7f0ca41a6c30 con 0x7f0ca410a1c0 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ca41a47b0 con 0x7f0ca410a1c0 2024-03-30T14:51:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c9c001bf0 con 0x7f0ca410a1c0 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 738916161 0 0) 0x7f0ca41a59f0 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 >> v1:172.21.15.28:6790/0 conn(0x7f0ca4106480 legacy=0x7f0ca419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 >> v1:172.21.15.134:6789/0 conn(0x7f0ca410a1c0 legacy=0x7f0ca41a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca41a7e70 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca41a6e60 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0cab8c8700 10 client.14217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ca41a73a0 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0cab8c8700 1 librados: init done 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0ca00010f0 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.440+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0ca0004e70 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c88005120 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 13) ==== 46137+0+0 (unknown 2211298249 0 0) 0x7f0ca00050f0 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 577553999 0 0) 0x7f0ca0047620 con 0x7f0ca4102050 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0c9a7fc700 10 client.14217.objecter ms_dispatch 0x7f0ca4001550 osd_map(4..4 src has 1..4) 2024-03-30T14:51:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0c9a7fc700 3 client.14217.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T14:51:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.441+0000 7f0c9a7fc700 3 client.14217.objecter handle_osd_map decoding full epoch 4 2024-03-30T14:51:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.442+0000 7f0c9a7fc700 20 client.14217.objecter dump_active .. 0 homeless 2024-03-30T14:51:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.442+0000 7f0c9a7fc700 10 client.14217.objecter ms_handle_connect 0x7f0c9003bb30 2024-03-30T14:51:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.446+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0ca00174c0 con 0x7f0ca4102050 2024-03-30T14:51:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:02.667+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0c88000b90 con 0x7f0c9003bb30 2024-03-30T14:51:02.929 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:02.644+0000 7f80896af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:51:02.930 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:02.774+0000 7f80896af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:51:03.398 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:03.111+0000 7f80896af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:51:03.398 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:03.201+0000 7f80896af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:03.397+0000 7f80896af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='client.14217 v1:172.21.15.28:0/99969592' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:03.399 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-mon[33714]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='client.14217 v1:172.21.15.28:0/99969592' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:03.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[29876]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='client.14217 v1:172.21.15.28:0/99969592' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:03.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:03 smithi028 ceph-mon[41614]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:03.679 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:03.584+0000 7f80896af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:51:04.135 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:03.870+0000 7f80896af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:51:04.135 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:04.134+0000 7f80896af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:51:04.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:04.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:04.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:04.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:04.371+0000 7f80896af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:51:04.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:04.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:04.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:04.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:04.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:04.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:05 smithi134 ceph-mon[33714]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:05.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:05.160+0000 7f80896af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:51:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:05 smithi028 ceph-mon[29876]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:05.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:05 smithi028 ceph-mon[41614]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:06.031 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:05.724+0000 7f80896af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:51:06.032 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:05.820+0000 7f80896af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:51:06.032 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:51:06.030+0000 7f80896af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:51:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:06.469+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mgrmap(e 14) ==== 91683+0+0 (unknown 998600420 0 0) 0x7f0ca0045f20 con 0x7f0ca4102050 2024-03-30T14:51:06.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]: dispatch 2024-03-30T14:51:06.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]': finished 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: Standby manager daemon x started 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:51:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1277396870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]: dispatch 2024-03-30T14:51:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]': finished 2024-03-30T14:51:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T14:51:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: Standby manager daemon x started 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1277396870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2521770051' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb350905-2b09-408a-9adc-4f2af2766479"}]': finished 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: Standby manager daemon x started 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:51:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:51:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:51:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/328680212' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:51:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1277396870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:07.441+0000 7f0cab0c7700 10 client.14217.objecter tick 2024-03-30T14:51:07.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[29876]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-30T14:51:07.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:51:07.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[29876]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[41614]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-30T14:51:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:51:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:07 smithi028 ceph-mon[41614]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:07.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:07 smithi134 ceph-mon[33714]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-30T14:51:07.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:07 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:51:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:07 smithi134 ceph-mon[33714]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:09.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:09 smithi028 ceph-mon[29876]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:09.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:09 smithi028 ceph-mon[41614]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:09.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:09 smithi134 ceph-mon[33714]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:12.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:12.442+0000 7f0cab0c7700 10 client.14217.objecter tick 2024-03-30T14:51:12.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[29876]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:12.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:51:12.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:12.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[41614]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:12.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:51:12.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:12 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:12.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:12 smithi134 ceph-mon[33714]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:12.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:12 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T14:51:12.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:12 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:13.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:13 smithi028 ceph-mon[29876]: Deploying daemon osd.0 on smithi028 2024-03-30T14:51:13.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:13 smithi028 ceph-mon[41614]: Deploying daemon osd.0 on smithi028 2024-03-30T14:51:13.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:13 smithi134 ceph-mon[33714]: Deploying daemon osd.0 on smithi028 2024-03-30T14:51:14.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:14 smithi028 ceph-mon[29876]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:14.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:14 smithi028 ceph-mon[41614]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:14.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:14 smithi134 ceph-mon[33714]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:15.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:15 smithi028 ceph-mon[29876]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:15.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:15 smithi028 ceph-mon[41614]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:15.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:15 smithi134 ceph-mon[33714]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:17.442+0000 7f0cab0c7700 10 client.14217.objecter tick 2024-03-30T14:51:17.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:17 smithi134 ceph-mon[33714]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:17 smithi028 ceph-mon[29876]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:17.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:17 smithi028 ceph-mon[41614]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:18.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:18.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:19.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:19 smithi134 ceph-mon[33714]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:19.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:19 smithi028 ceph-mon[41614]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:19.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:19 smithi028 ceph-mon[29876]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:21.123 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-03-30T14:51:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.122+0000 7f0c9a7fc700 1 -- 172.21.15.28:0/99969592 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2611477045) 0x7f0c88000b90 con 0x7f0c9003bb30 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 10 librados: watch_flush enter 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 10 librados: watch_flush exit 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 20 client.14217.objecter shutdown clearing up homeless session... 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 10 client.14217.objecter successfully canceled tick 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 >> v1:172.21.15.28:6800/3720192247 conn(0x7f0c9003bb30 legacy=0x7f0c9003dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 >> v1:172.21.15.28:6789/0 conn(0x7f0ca4102050 legacy=0x7f0ca4075f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 shutdown_connections 2024-03-30T14:51:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 >> 172.21.15.28:0/99969592 conn(0x7f0ca40fd5f0 msgr2=0x7f0ca4109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:51:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 shutdown_connections 2024-03-30T14:51:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 -- 172.21.15.28:0/99969592 wait complete. 2024-03-30T14:51:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:21.124+0000 7f0cab8c8700 1 librados: shutdown 2024-03-30T14:51:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:51:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:21.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:21.775 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T14:51:21.777 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_3... 2024-03-30T14:51:21.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T14:51:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:51:22.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:22 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:51:22.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:22 smithi134 ceph-mon[33714]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T14:51:22.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:22 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:22.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:22.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:22 smithi134 ceph-mon[33714]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:22.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:51:22.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[29876]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[29876]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[41614]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:22.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-mon[41614]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:22.707 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T14:51:22.277+0000 7f2e6cf70700 -1 osd.0 0 waiting for initial osdmap 2024-03-30T14:51:22.707 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:51:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T14:51:22.282+0000 7f2e68d63700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[29876]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[41614]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:23.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:23.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:23 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6801/3394323575' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:23.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:23 smithi134 ceph-mon[33714]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T14:51:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: osd.0 v1:172.21.15.28:6801/3394323575 boot 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[29876]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: osd.0 v1:172.21.15.28:6801/3394323575 boot 2024-03-30T14:51:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T14:51:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:24 smithi028 ceph-mon[41614]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:24.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: osd.0 v1:172.21.15.28:6801/3394323575 boot 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:51:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:24 smithi134 ceph-mon[33714]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T14:51:25.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:25 smithi028 ceph-mon[29876]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T14:51:25.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:25 smithi028 ceph-mon[41614]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0096453 s, 1.1 GB/s 2024-03-30T14:51:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-30T14:51:25.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:25 smithi134 ceph-mon[33714]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:26.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:26.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-03-30T14:51:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:51:26.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:27.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:27 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi028 2024-03-30T14:51:27.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:27 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-30T14:51:27.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:27 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi028 2024-03-30T14:51:27.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:27 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-30T14:51:27.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:27 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi028 2024-03-30T14:51:27.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:27 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 16069M 2024-03-30T14:51:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 -- 172.21.15.28:0/3310667585 >> v1:172.21.15.28:6790/0 conn(0x7fd968100ef0 legacy=0x7fd968101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 -- 172.21.15.28:0/3310667585 shutdown_connections 2024-03-30T14:51:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 -- 172.21.15.28:0/3310667585 >> 172.21.15.28:0/3310667585 conn(0x7fd9680fc520 msgr2=0x7fd9680fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:51:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 -- 172.21.15.28:0/3310667585 shutdown_connections 2024-03-30T14:51:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 -- 172.21.15.28:0/3310667585 wait complete. 2024-03-30T14:51:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 librados: starting msgr at 2024-03-30T14:51:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.230+0000 7fd96d05f700 1 librados: starting objecter 2024-03-30T14:51:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 Processor -- start 2024-03-30T14:51:28.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 -- start start 2024-03-30T14:51:28.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 librados: setting wanted keys 2024-03-30T14:51:28.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 librados: calling monclient init 2024-03-30T14:51:28.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd96806b470 con 0x7fd968109020 2024-03-30T14:51:28.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9681a3670 con 0x7fd968105390 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd96d05f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd9681a4890 con 0x7fd968100ef0 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd966d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd968100ef0 0x7fd96819a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42738/0 (socket says 172.21.15.28:42738) 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd966d9d700 1 -- 172.21.15.28:0/2099727032 learned_addr learned my addr 172.21.15.28:0/2099727032 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd94ffff700 10 client.?.objecter ms_handle_connect 0x7fd968109020 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd94ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd94ffff700 10 client.?.objecter ms_handle_connect 0x7fd968100ef0 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.231+0000 7fd94ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 10 client.?.objecter ms_handle_connect 0x7fd968105390 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471527096 0 0) 0x7fd9681a4890 con 0x7fd968100ef0 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9500035b0 con 0x7fd968100ef0 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156968504 0 0) 0x7fd96806b470 con 0x7fd968109020 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9681a4890 con 0x7fd968109020 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386512828 0 0) 0x7fd9681a3670 con 0x7fd968105390 2024-03-30T14:51:28.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96806b470 con 0x7fd968105390 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3597612831 0 0) 0x7fd9500035b0 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9681a3670 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd958004550 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1823278865 0 0) 0x7fd9681a4890 con 0x7fd968109020 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9500035b0 con 0x7fd968109020 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3058634927 0 0) 0x7fd96806b470 con 0x7fd968105390 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9681a4890 con 0x7fd968105390 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd960001b30 con 0x7fd968109020 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd95c001810 con 0x7fd968105390 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2780453942 0 0) 0x7fd9681a3670 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 >> v1:172.21.15.134:6789/0 conn(0x7fd968105390 legacy=0x7fd96819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 >> v1:172.21.15.28:6789/0 conn(0x7fd968109020 legacy=0x7fd9681a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9681a5ad0 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9681a26c0 con 0x7fd968100ef0 2024-03-30T14:51:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd96d05f700 10 client.34139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9681a2b80 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd96d05f700 1 librados: init done 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd958001ab0 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.232+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd958004e40 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.233+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd954005150 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 14) ==== 91683+0+0 (unknown 998600420 0 0) 0x7fd958003710 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(9..9 src has 1..9) ==== 1541+0+0 (unknown 711355364 0 0) 0x7fd958087ad0 con 0x7fd968100ef0 2024-03-30T14:51:28.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 10 client.34139.objecter ms_dispatch 0x7fd968001550 osd_map(9..9 src has 1..9) 2024-03-30T14:51:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 3 client.34139.objecter handle_osd_map got epochs [9,9] > 0 2024-03-30T14:51:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 3 client.34139.objecter handle_osd_map decoding full epoch 9 2024-03-30T14:51:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.234+0000 7fd94ffff700 20 client.34139.objecter dump_active .. 0 homeless 2024-03-30T14:51:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.235+0000 7fd94ffff700 10 client.34139.objecter ms_handle_connect 0x7fd950070460 2024-03-30T14:51:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.239+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd9580567c0 con 0x7fd968100ef0 2024-03-30T14:51:28.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:28.455+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd954000bc0 con 0x7fd950070460 2024-03-30T14:51:28.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:28 smithi028 ceph-mon[29876]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:28.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:28 smithi028 ceph-mon[41614]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:28.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:28 smithi134 ceph-mon[33714]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:29.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:29.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:29.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:29.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:29.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:29.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:29.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:30.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:30 smithi028 ceph-mon[29876]: from='client.34139 v1:172.21.15.28:0/2099727032' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:30.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:30 smithi028 ceph-mon[29876]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:30.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:30 smithi028 ceph-mon[41614]: from='client.34139 v1:172.21.15.28:0/2099727032' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:30.345 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:30 smithi028 ceph-mon[41614]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:30.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:30 smithi134 ceph-mon[33714]: from='client.34139 v1:172.21.15.28:0/2099727032' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:30.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:30 smithi134 ceph-mon[33714]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:32.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2181770864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]': finished 2024-03-30T14:51:32.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3755801278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2181770864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]': finished 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3755801278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2181770864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]: dispatch 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520"}]': finished 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3755801278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:33.233+0000 7fd967fff700 10 client.34139.objecter tick 2024-03-30T14:51:33.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:33 smithi134 ceph-mon[33714]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:33.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:33 smithi028 ceph-mon[29876]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:33.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:33 smithi028 ceph-mon[41614]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:35.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:35 smithi028 ceph-mon[29876]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:35.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:35 smithi028 ceph-mon[41614]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:35.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:35 smithi134 ceph-mon[33714]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:37.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:37 smithi028 ceph-mon[41614]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:37.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:37 smithi028 ceph-mon[29876]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:37.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:37 smithi134 ceph-mon[33714]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:38.234+0000 7fd967fff700 10 client.34139.objecter tick 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[29876]: Deploying daemon osd.1 on smithi028 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:38.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:38 smithi028 ceph-mon[41614]: Deploying daemon osd.1 on smithi028 2024-03-30T14:51:38.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T14:51:38.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:38.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:38 smithi134 ceph-mon[33714]: Deploying daemon osd.1 on smithi028 2024-03-30T14:51:39.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:39 smithi134 ceph-mon[33714]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:39.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:39 smithi028 ceph-mon[29876]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:39.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:39 smithi028 ceph-mon[41614]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:41.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:41 smithi134 ceph-mon[33714]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:41.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:41 smithi028 ceph-mon[29876]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:41.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:41 smithi028 ceph-mon[41614]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:43.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:43.235+0000 7fd967fff700 10 client.34139.objecter tick 2024-03-30T14:51:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[29876]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:43.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[41614]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:43.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:43 smithi134 ceph-mon[33714]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:43 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:43 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:44.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:44.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:44.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:45.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:45 smithi028 ceph-mon[29876]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:45.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:45 smithi028 ceph-mon[41614]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:45.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:45 smithi134 ceph-mon[33714]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.805+0000 7fd94ffff700 1 -- 172.21.15.28:0/2099727032 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3350336641) 0x7fd954000bc0 con 0x7fd950070460 2024-03-30T14:51:46.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:46 smithi028 ceph-mon[41614]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:51:46.807 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-03-30T14:51:46.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:46 smithi028 ceph-mon[29876]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 10 librados: watch_flush enter 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 10 librados: watch_flush exit 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 20 client.34139.objecter shutdown clearing up homeless session... 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 10 client.34139.objecter successfully canceled tick 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 >> v1:172.21.15.28:6800/3720192247 conn(0x7fd950070460 legacy=0x7fd950072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 >> v1:172.21.15.28:6790/0 conn(0x7fd968100ef0 legacy=0x7fd96819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 shutdown_connections 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 >> 172.21.15.28:0/2099727032 conn(0x7fd9680fc520 msgr2=0x7fd96810b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 shutdown_connections 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 -- 172.21.15.28:0/2099727032 wait complete. 2024-03-30T14:51:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:46.808+0000 7fd96d05f700 1 librados: shutdown 2024-03-30T14:51:46.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:46 smithi134 ceph-mon[33714]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T14:51:47.456 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T14:51:47.458 INFO:tasks.cephadm:Deploying osd.2 on smithi028 with /dev/vg_nvme/lv_2... 2024-03-30T14:51:47.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T14:51:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:51:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T14:51:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T14:51:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T14:51:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:47.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T14:51:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.014 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T14:51:47.861+0000 7f46ca8bb700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T14:51:48.014 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:51:47 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T14:51:47.864+0000 7f46c26a6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:51:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:51:48.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[41614]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' 2024-03-30T14:51:48.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[29876]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' 2024-03-30T14:51:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:48 smithi134 ceph-mon[33714]: from='osd.1 v1:172.21.15.28:6805/1850313547' entity='osd.1' 2024-03-30T14:51:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: osd.1 v1:172.21.15.28:6805/1850313547 boot 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:50.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[29876]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: osd.1 v1:172.21.15.28:6805/1850313547 boot 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:50.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:49 smithi028 ceph-mon[41614]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:50.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:51:50.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:51:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: osd.1 v1:172.21.15.28:6805/1850313547 boot 2024-03-30T14:51:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T14:51:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:51:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:49 smithi134 ceph-mon[33714]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T14:51:51.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:50 smithi028 ceph-mon[29876]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T14:51:51.032 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:50 smithi028 ceph-mon[41614]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T14:51:51.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:50 smithi134 ceph-mon[33714]: osdmap e14: 2 total, 2 up, 2 in 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00938043 s, 1.1 GB/s 2024-03-30T14:51:51.503 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-30T14:51:52.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-03-30T14:51:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:51:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:51:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:51:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:51:53.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:53 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi028 2024-03-30T14:51:53.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:53 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-30T14:51:53.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:53 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi028 2024-03-30T14:51:53.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:53 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-30T14:51:53.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:53 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi028 2024-03-30T14:51:53.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:53 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 8034M 2024-03-30T14:51:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- 172.21.15.28:0/3907047326 >> v1:172.21.15.28:6789/0 conn(0x7fe34c100f30 legacy=0x7fe34c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- 172.21.15.28:0/3907047326 shutdown_connections 2024-03-30T14:51:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- 172.21.15.28:0/3907047326 >> 172.21.15.28:0/3907047326 conn(0x7fe34c0fc4d0 msgr2=0x7fe34c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:51:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- 172.21.15.28:0/3907047326 shutdown_connections 2024-03-30T14:51:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- 172.21.15.28:0/3907047326 wait complete. 2024-03-30T14:51:54.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 librados: starting msgr at 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 librados: starting objecter 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 Processor -- start 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- start start 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 librados: setting wanted keys 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 librados: calling monclient init 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe34c197160 con 0x7fe34c105350 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe34c1a2950 con 0x7fe34c108fe0 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe352664700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe34c1a3b30 con 0x7fe34c100f30 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe34b7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe34c105350 0x7fe34c19caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49712/0 (socket says 172.21.15.28:49712) 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe34b7fe700 1 -- 172.21.15.28:0/605191981 learned_addr learned my addr 172.21.15.28:0/605191981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter ms_handle_connect 0x7fe34c105350 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter ms_handle_connect 0x7fe34c100f30 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter ms_handle_connect 0x7fe34c108fe0 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955485667 0 0) 0x7fe34c197160 con 0x7fe34c105350 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3340035b0 con 0x7fe34c105350 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003006257 0 0) 0x7fe34c1a3b30 con 0x7fe34c100f30 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.046+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe34c197160 con 0x7fe34c100f30 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124348350 0 0) 0x7fe34c1a2950 con 0x7fe34c108fe0 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe34c1a3b30 con 0x7fe34c108fe0 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4078478643 0 0) 0x7fe3340035b0 con 0x7fe34c105350 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe34c1a2950 con 0x7fe34c105350 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe340001870 con 0x7fe34c105350 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 584504178 0 0) 0x7fe34c197160 con 0x7fe34c100f30 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3340035b0 con 0x7fe34c100f30 2024-03-30T14:51:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe33c001430 con 0x7fe34c100f30 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2503343999 0 0) 0x7fe34c1a3b30 con 0x7fe34c108fe0 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe34c197160 con 0x7fe34c108fe0 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe344001c30 con 0x7fe34c108fe0 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1311861435 0 0) 0x7fe34c1a2950 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 >> v1:172.21.15.28:6790/0 conn(0x7fe34c100f30 legacy=0x7fe34c196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 >> v1:172.21.15.134:6789/0 conn(0x7fe34c108fe0 legacy=0x7fe34c1a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c1a4d50 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c1a3d60 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe352664700 10 client.14271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe34c1a42a0 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe352664700 1 librados: init done 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe340003b40 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.047+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe340005ca0 con 0x7fe34c105350 2024-03-30T14:51:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.048+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe338005120 con 0x7fe34c105350 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 14) ==== 91683+0+0 (unknown 998600420 0 0) 0x7fe3400036a0 con 0x7fe34c105350 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 1833+0+0 (unknown 1761192968 0 0) 0x7fe340087b60 con 0x7fe34c105350 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 10 client.14271.objecter ms_dispatch 0x7fe34c001550 osd_map(14..14 src has 1..14) 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 3 client.14271.objecter handle_osd_map got epochs [14,14] > 0 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 3 client.14271.objecter handle_osd_map decoding full epoch 14 2024-03-30T14:51:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 20 client.14271.objecter dump_active .. 0 homeless 2024-03-30T14:51:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.049+0000 7fe3497fa700 10 client.14271.objecter ms_handle_connect 0x7fe3340725f0 2024-03-30T14:51:54.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.054+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe3400566f0 con 0x7fe34c105350 2024-03-30T14:51:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:54.269+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe338000b90 con 0x7fe3340725f0 2024-03-30T14:51:54.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:54 smithi028 ceph-mon[29876]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:54.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:54 smithi028 ceph-mon[41614]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:54.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:54 smithi134 ceph-mon[33714]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:55.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[29876]: from='client.14271 v1:172.21.15.28:0/605191981' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:55.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:55.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:55.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[41614]: from='client.14271 v1:172.21.15.28:0/605191981' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:55.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:55 smithi134 ceph-mon[33714]: from='client.14271 v1:172.21.15.28:0/605191981' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:51:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:51:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:51:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:51:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:56 smithi028 ceph-mon[29876]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:56.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:56 smithi028 ceph-mon[41614]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:56.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:56 smithi134 ceph-mon[33714]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:57.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2536082102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[29876]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[29876]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]': finished 2024-03-30T14:51:57.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[29876]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2536082102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[41614]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[41614]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]': finished 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[41614]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T14:51:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:51:57.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2536082102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:57 smithi134 ceph-mon[33714]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]: dispatch 2024-03-30T14:51:57.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:57 smithi134 ceph-mon[33714]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c"}]': finished 2024-03-30T14:51:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:57 smithi134 ceph-mon[33714]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T14:51:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:51:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:58 smithi028 ceph-mon[29876]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4016532041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:58.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:58 smithi028 ceph-mon[41614]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:58.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4016532041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:58.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:58 smithi134 ceph-mon[33714]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:58.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4016532041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:51:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:51:59.048+0000 7fe351e63700 10 client.14271.objecter tick 2024-03-30T14:51:59.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:51:59 smithi134 ceph-mon[33714]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:51:59 smithi028 ceph-mon[29876]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:51:59.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:51:59 smithi028 ceph-mon[41614]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:01.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:01 smithi028 ceph-mon[29876]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:01 smithi028 ceph-mon[41614]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:01.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:01 smithi134 ceph-mon[33714]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:03.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:03 smithi028 ceph-mon[29876]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:03.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:03 smithi028 ceph-mon[41614]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:03.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:03 smithi134 ceph-mon[33714]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:04.049+0000 7fe351e63700 10 client.14271.objecter tick 2024-03-30T14:52:04.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:52:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:04 smithi134 ceph-mon[33714]: Deploying daemon osd.2 on smithi028 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[41614]: Deploying daemon osd.2 on smithi028 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:04 smithi028 ceph-mon[29876]: Deploying daemon osd.2 on smithi028 2024-03-30T14:52:05.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:05 smithi134 ceph-mon[33714]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:05.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:05 smithi028 ceph-mon[29876]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:05.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:05 smithi028 ceph-mon[41614]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:07.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:07 smithi134 ceph-mon[33714]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:07.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:07 smithi028 ceph-mon[29876]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:07.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:07 smithi028 ceph-mon[41614]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:09.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:09.050+0000 7fe351e63700 10 client.14271.objecter tick 2024-03-30T14:52:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[41614]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:10 smithi028 ceph-mon[29876]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:10 smithi134 ceph-mon[33714]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:12.409 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:12 smithi028 ceph-mon[41614]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:12.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:12 smithi028 ceph-mon[29876]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:12.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:12 smithi134 ceph-mon[33714]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:13.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.172+0000 7fe3497fa700 1 -- 172.21.15.28:0/605191981 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 596387677) 0x7fe338000b90 con 0x7fe3340725f0 2024-03-30T14:52:13.174 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 2 on host 'smithi028' 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.174+0000 7fe352664700 10 librados: watch_flush enter 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.174+0000 7fe352664700 10 librados: watch_flush exit 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.174+0000 7fe352664700 20 client.14271.objecter shutdown clearing up homeless session... 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.174+0000 7fe352664700 10 client.14271.objecter successfully canceled tick 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 >> v1:172.21.15.28:6800/3720192247 conn(0x7fe3340725f0 legacy=0x7fe334074ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 >> v1:172.21.15.28:6789/0 conn(0x7fe34c105350 legacy=0x7fe34c19caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 shutdown_connections 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 >> 172.21.15.28:0/605191981 conn(0x7fe34c0fc4d0 msgr2=0x7fe34c10b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 shutdown_connections 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 -- 172.21.15.28:0/605191981 wait complete. 2024-03-30T14:52:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:13.175+0000 7fe352664700 1 librados: shutdown 2024-03-30T14:52:13.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[29876]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:52:13.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:13 smithi028 ceph-mon[41614]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:13.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T14:52:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:13 smithi134 ceph-mon[33714]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:13.879 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T14:52:13.882 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_1... 2024-03-30T14:52:13.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T14:52:14.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:52:14.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[29876]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T14:52:14.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:14.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:14.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:52:14.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[41614]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T14:52:14.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:14.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:52:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:14 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T14:52:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:14 smithi134 ceph-mon[33714]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T14:52:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:14 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:14 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:14.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[58365]: 2024-03-30T14:52:14.654+0000 7f51bbff5700 -1 osd.2 0 waiting for initial osdmap 2024-03-30T14:52:14.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[58365]: 2024-03-30T14:52:14.658+0000 7f51b3de0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:15 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T14:52:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6809/3451325043' entity='osd.2' 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T14:52:15.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:15 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[29876]: osd.2 v1:172.21.15.28:6809/3451325043 boot 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[29876]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:16.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:52:16.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:52:16.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[41614]: osd.2 v1:172.21.15.28:6809/3451325043 boot 2024-03-30T14:52:16.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[41614]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T14:52:16.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:16 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:16.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:16 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:52:16.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:16 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:52:16.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:16 smithi134 ceph-mon[33714]: osd.2 v1:172.21.15.28:6809/3451325043 boot 2024-03-30T14:52:16.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:16 smithi134 ceph-mon[33714]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T14:52:16.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:16 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108532 s, 966 MB/s 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 2024-03-30T14:52:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-30T14:52:17.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:17 smithi028 ceph-mon[29876]: pgmap v78: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:17.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:17 smithi028 ceph-mon[41614]: pgmap v78: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:17.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:17 smithi134 ceph-mon[33714]: pgmap v78: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:18.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-03-30T14:52:18.529 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:52:18.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi028 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi028 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi028 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 5356M 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:19.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[29876]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[29876]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[41614]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:19.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:19 smithi028 ceph-mon[41614]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:19.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:19 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:19 smithi134 ceph-mon[33714]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T14:52:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:19 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:52:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:19 smithi134 ceph-mon[33714]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- 172.21.15.28:0/991671703 >> v1:172.21.15.28:6789/0 conn(0x7f88d4103630 legacy=0x7f88d4103a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- 172.21.15.28:0/991671703 shutdown_connections 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- 172.21.15.28:0/991671703 >> 172.21.15.28:0/991671703 conn(0x7f88d40fbcb0 msgr2=0x7f88d40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- 172.21.15.28:0/991671703 shutdown_connections 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- 172.21.15.28:0/991671703 wait complete. 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 librados: starting msgr at 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 librados: starting objecter 2024-03-30T14:52:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 Processor -- start 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 -- start start 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 librados: setting wanted keys 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.132+0000 7f88dae7b700 1 librados: calling monclient init 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88dae7b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d41a4930 con 0x7f88d4106600 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88dae7b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d41a5b70 con 0x7f88d410a1a0 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88dae7b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d41a6db0 con 0x7f88d4103630 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d8c17700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f88d4103630 0x7f88d4102c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58682/0 (socket says 172.21.15.28:58682) 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f88d4106600 0x7f88d419f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33536/0 (socket says 172.21.15.28:33536) 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d8c17700 1 -- 172.21.15.28:0/2364291628 learned_addr learned my addr 172.21.15.28:0/2364291628 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter ms_handle_connect 0x7f88d4106600 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter ms_handle_connect 0x7f88d4103630 2024-03-30T14:52:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter ms_handle_connect 0x7f88d410a1a0 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672552470 0 0) 0x7f88d41a4930 con 0x7f88d4106600 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88c40035b0 con 0x7f88d4106600 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453821445 0 0) 0x7f88d41a6db0 con 0x7f88d4103630 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d41a4930 con 0x7f88d4103630 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326189072 0 0) 0x7f88d41a5b70 con 0x7f88d410a1a0 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d41a6db0 con 0x7f88d410a1a0 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3937513971 0 0) 0x7f88c40035b0 con 0x7f88d4106600 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88d41a5b70 con 0x7f88d4106600 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88c0001850 con 0x7f88d4106600 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1593646711 0 0) 0x7f88d41a4930 con 0x7f88d4103630 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88c40035b0 con 0x7f88d4103630 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1082469314 0 0) 0x7f88d41a6db0 con 0x7f88d410a1a0 2024-03-30T14:52:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88d41a4930 con 0x7f88d410a1a0 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88c8004380 con 0x7f88d4103630 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.133+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88cc001af0 con 0x7f88d410a1a0 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1204640970 0 0) 0x7f88d41a5b70 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 >> v1:172.21.15.28:6790/0 conn(0x7f88d4103630 legacy=0x7f88d4102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 >> v1:172.21.15.134:6789/0 conn(0x7f88d410a1a0 legacy=0x7f88d41a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d41a7ff0 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88d41a5da0 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f88c0003b20 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88c0005c70 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88dae7b700 10 client.14298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f88d41a62e0 con 0x7f88d4106600 2024-03-30T14:52:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.134+0000 7f88dae7b700 1 librados: init done 2024-03-30T14:52:20.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.135+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d4068b30 con 0x7f88d4106600 2024-03-30T14:52:20.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.135+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 14) ==== 91683+0+0 (unknown 998600420 0 0) 0x7f88c0003680 con 0x7f88d4106600 2024-03-30T14:52:20.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 2511+0+0 (unknown 4048352426 0 0) 0x7f88c0087ce0 con 0x7f88d4106600 2024-03-30T14:52:20.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 10 client.14298.objecter ms_dispatch 0x7f88d4001550 osd_map(21..21 src has 1..21) 2024-03-30T14:52:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 3 client.14298.objecter handle_osd_map got epochs [21,21] > 0 2024-03-30T14:52:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 3 client.14298.objecter handle_osd_map decoding full epoch 21 2024-03-30T14:52:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 20 client.14298.objecter dump_active .. 0 homeless 2024-03-30T14:52:20.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.136+0000 7f88d1ffb700 10 client.14298.objecter ms_handle_connect 0x7f88c4070450 2024-03-30T14:52:20.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.140+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f88c0056630 con 0x7f88d4106600 2024-03-30T14:52:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:20.380+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f88d41a6640 con 0x7f88c4070450 2024-03-30T14:52:20.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[29876]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:52:20.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:20.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[29876]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T14:52:20.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[41614]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:52:20.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:20.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:20 smithi028 ceph-mon[41614]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T14:52:20.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:20 smithi134 ceph-mon[33714]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:52:20.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:20 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:52:20.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:20 smithi134 ceph-mon[33714]: osdmap e21: 3 total, 3 up, 3 in 2024-03-30T14:52:20.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62804]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:20.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:20.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62804]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:20.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62804]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:21.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:21.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:21.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:21.198 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 14:52:20 smithi028 sudo[62864]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:21.472 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62887]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:21.472 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62887]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:21.472 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62887]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:21.472 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62887]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: from='client.14298 v1:172.21.15.28:0/2364291628' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[29876]: pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: from='client.14298 v1:172.21.15.28:0/2364291628' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:21 smithi028 ceph-mon[41614]: pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:21.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: from='client.14298 v1:172.21.15.28:0/2364291628' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: osdmap e22: 3 total, 3 up, 3 in 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:21 smithi134 ceph-mon[33714]: pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:21.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62908]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:21.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62908]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:21.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62908]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:21.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62908]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:21.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:21.945+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f88c0051bf0 con 0x7f88d4106600 2024-03-30T14:52:21.991 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62919]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:21.991 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62919]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:21.991 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62919]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:21.991 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62919]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:22.258 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62922]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:22.258 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62922]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:22.258 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62922]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:22.258 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 14:52:21 smithi028 sudo[62922]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:22.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62947]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:22.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62947]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:22.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62947]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:22.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62947]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: Cluster is now healthy 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 sudo[36559]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 sudo[36559]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 sudo[36559]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:22.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:22 smithi134 sudo[36559]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:22.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:52:22.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: Cluster is now healthy 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62976]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62976]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62976]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 sudo[62976]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: Cluster is now healthy 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-30T14:52:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 sudo[36562]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 sudo[36562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 sudo[36562]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 sudo[36562]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:23.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T14:52:23.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:23.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63023]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:23.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63023]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:23 smithi134 ceph-mon[33714]: pgmap v85: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[29876]: pgmap v85: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T14:52:23.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 sudo[63026]: pam_unix(sudo:session): session closed for user root 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:23.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:52:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:52:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:52:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:23 smithi028 ceph-mon[41614]: pgmap v85: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1303484839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]': finished 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2170124953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1303484839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]': finished 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2170124953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1303484839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]: dispatch 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4305caf1-0023-438f-8520-87814130b457"}]': finished 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T14:52:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:24.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2170124953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:25.134+0000 7f88da67a700 10 client.14298.objecter tick 2024-03-30T14:52:25.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:25 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:52:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:25 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:52:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:25 smithi134 ceph-mon[33714]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:52:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:52:25.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[29876]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:25.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:52:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:52:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:25 smithi028 ceph-mon[41614]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:26.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:26 smithi134 ceph-mon[33714]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T14:52:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:26 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:26.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:26 smithi028 ceph-mon[29876]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T14:52:26.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:26 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:26.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:26 smithi028 ceph-mon[41614]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T14:52:26.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:26 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:27.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:27 smithi028 ceph-mon[29876]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:27.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:27 smithi028 ceph-mon[41614]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:27.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:27 smithi134 ceph-mon[33714]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:29.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:29 smithi134 ceph-mon[33714]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:29.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:29 smithi028 ceph-mon[29876]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:29 smithi028 ceph-mon[41614]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:30.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:30.135+0000 7f88da67a700 10 client.14298.objecter tick 2024-03-30T14:52:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:52:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[29876]: Deploying daemon osd.3 on smithi028 2024-03-30T14:52:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:52:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:30 smithi028 ceph-mon[41614]: Deploying daemon osd.3 on smithi028 2024-03-30T14:52:30.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:30 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T14:52:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:30 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:30 smithi134 ceph-mon[33714]: Deploying daemon osd.3 on smithi028 2024-03-30T14:52:32.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:32 smithi028 ceph-mon[29876]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:32.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:32 smithi028 ceph-mon[41614]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:32.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:32 smithi134 ceph-mon[33714]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:33.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:33 smithi134 ceph-mon[33714]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:33.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:33 smithi028 ceph-mon[29876]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:33.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:33 smithi028 ceph-mon[41614]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:35.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:35.136+0000 7f88da67a700 10 client.14298.objecter tick 2024-03-30T14:52:35.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:35 smithi028 ceph-mon[29876]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:35.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:35 smithi028 ceph-mon[41614]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:35.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:35 smithi134 ceph-mon[33714]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:36.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:36.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:36.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:37.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:37.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:37 smithi028 ceph-mon[29876]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:37.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:37 smithi028 ceph-mon[41614]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:38.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:37 smithi134 ceph-mon[33714]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:39.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:38 smithi028 ceph-mon[29876]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.081 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:38 smithi028 ceph-mon[29876]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:38 smithi028 ceph-mon[41614]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.082 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:38 smithi028 ceph-mon[41614]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:38 smithi134 ceph-mon[33714]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:38 smithi134 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T14:52:39.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.839+0000 7f88d1ffb700 1 -- 172.21.15.28:0/2364291628 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2140785129) 0x7f88d41a6640 con 0x7f88c4070450 2024-03-30T14:52:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-03-30T14:52:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 10 librados: watch_flush enter 2024-03-30T14:52:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 10 librados: watch_flush exit 2024-03-30T14:52:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 20 client.14298.objecter shutdown clearing up homeless session... 2024-03-30T14:52:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 10 client.14298.objecter successfully canceled tick 2024-03-30T14:52:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 >> v1:172.21.15.28:6800/3720192247 conn(0x7f88c4070450 legacy=0x7f88c4072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 >> v1:172.21.15.28:6789/0 conn(0x7f88d4106600 legacy=0x7f88d419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 shutdown_connections 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 >> 172.21.15.28:0/2364291628 conn(0x7f88d40fbcb0 msgr2=0x7f88d40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 shutdown_connections 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.841+0000 7f88dae7b700 1 -- 172.21.15.28:0/2364291628 wait complete. 2024-03-30T14:52:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:52:39.842+0000 7f88dae7b700 1 librados: shutdown 2024-03-30T14:52:40.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: osdmap e26: 4 total, 3 up, 4 in 2024-03-30T14:52:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:40.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: osdmap e26: 4 total, 3 up, 4 in 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='osd.3 v1:172.21.15.28:6813/1506157250' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: osdmap e25: 4 total, 3 up, 4 in 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T14:52:40.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: osdmap e26: 4 total, 3 up, 4 in 2024-03-30T14:52:40.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:40.539 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T14:52:40.541 INFO:tasks.cephadm:Deploying osd.4 on smithi134 with /dev/vg_nvme/lv_4... 2024-03-30T14:52:40.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T14:52:40.787 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:52:41.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:40 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:40 smithi028 ceph-mon[29876]: from='osd.3 ' entity='osd.3' 2024-03-30T14:52:41.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:40 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:40 smithi028 ceph-mon[41614]: from='osd.3 ' entity='osd.3' 2024-03-30T14:52:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:40 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:40 smithi134 ceph-mon[33714]: from='osd.3 ' entity='osd.3' 2024-03-30T14:52:41.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:52:41.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: osd.3 v1:172.21.15.28:6813/1506157250 boot 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[29876]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: osd.3 v1:172.21.15.28:6813/1506157250 boot 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:41.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:41 smithi028 ceph-mon[41614]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: osd.3 v1:172.21.15.28:6813/1506157250 boot 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:52:42.165 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:41 smithi134 ceph-mon[33714]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-30T14:52:42.613 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T14:52:42.614 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T14:52:42.614 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-03-30T14:52:42.614 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-03-30T14:52:42.614 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113185 s, 926 MB/s 2024-03-30T14:52:42.614 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-03-30T14:52:43.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-03-30T14:52:43.418 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:52:43.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:43 smithi134 ceph-mon[33714]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T14:52:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:43 smithi028 ceph-mon[29876]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T14:52:43.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:43 smithi028 ceph-mon[41614]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T14:52:44.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.508 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:52:44.509 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:44.509 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:44.509 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:52:44.509 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:52:45.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.016+0000 7fc9bea0b700 1 -- 172.21.15.134:0/700745617 >> v1:172.21.15.134:6789/0 conn(0x7fc9b8109040 legacy=0x7fc9b810b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:45.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.016+0000 7fc9bea0b700 1 -- 172.21.15.134:0/700745617 shutdown_connections 2024-03-30T14:52:45.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.016+0000 7fc9bea0b700 1 -- 172.21.15.134:0/700745617 >> 172.21.15.134:0/700745617 conn(0x7fc9b80fc500 msgr2=0x7fc9b80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:52:45.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.016+0000 7fc9bea0b700 1 -- 172.21.15.134:0/700745617 shutdown_connections 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.016+0000 7fc9bea0b700 1 -- 172.21.15.134:0/700745617 wait complete. 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 librados: starting msgr at 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 librados: starting objecter 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 Processor -- start 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 -- start start 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 librados: setting wanted keys 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 librados: calling monclient init 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9b806b4b0 con 0x7fc9b8100f10 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9b81a3a10 con 0x7fc9b8109040 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bea0b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9b81a4c30 con 0x7fc9b81053b0 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bcfa8700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc9b8109040 0x7fc9b81a12e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:40554/0 (socket says 172.21.15.134:40554) 2024-03-30T14:52:45.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9bcfa8700 1 -- 172.21.15.134:0/2574471497 learned_addr learned my addr 172.21.15.134:0/2574471497 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9b57fa700 10 client.?.objecter ms_handle_connect 0x7fc9b8109040 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.017+0000 7fc9b57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 10 client.?.objecter ms_handle_connect 0x7fc9b81053b0 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 10 client.?.objecter ms_handle_connect 0x7fc9b8100f10 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172001522 0 0) 0x7fc9b81a3a10 con 0x7fc9b8109040 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9a00035b0 con 0x7fc9b8109040 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618150450 0 0) 0x7fc9b81a4c30 con 0x7fc9b81053b0 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9b81a3a10 con 0x7fc9b81053b0 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608553008 0 0) 0x7fc9b806b4b0 con 0x7fc9b8100f10 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9b81a4c30 con 0x7fc9b8100f10 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3870749771 0 0) 0x7fc9a00035b0 con 0x7fc9b8109040 2024-03-30T14:52:45.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9b806b4b0 con 0x7fc9b8109040 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9b00017b0 con 0x7fc9b8109040 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1649262885 0 0) 0x7fc9b81a3a10 con 0x7fc9b81053b0 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9a00035b0 con 0x7fc9b81053b0 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9ac0018b0 con 0x7fc9b81053b0 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2893090056 0 0) 0x7fc9b81a4c30 con 0x7fc9b8100f10 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9b81a3a10 con 0x7fc9b8100f10 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9a8001870 con 0x7fc9b8100f10 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2363790111 0 0) 0x7fc9b806b4b0 con 0x7fc9b8109040 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 >> v1:172.21.15.28:6790/0 conn(0x7fc9b81053b0 legacy=0x7fc9b819dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 >> v1:172.21.15.28:6789/0 conn(0x7fc9b8100f10 legacy=0x7fc9b819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b81a5e70 con 0x7fc9b8109040 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b81a4e60 con 0x7fc9b8109040 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9bea0b700 10 client.24229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:52:45.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9b81a53a0 con 0x7fc9b8109040 2024-03-30T14:52:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9bea0b700 1 librados: init done 2024-03-30T14:52:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc9b0001a10 con 0x7fc9b8109040 2024-03-30T14:52:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.018+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9b00051f0 con 0x7fc9b8109040 2024-03-30T14:52:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.019+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a4005120 con 0x7fc9b8109040 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7fc9b0003e30 con 0x7fc9b8109040 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(28..28 src has 1..28) ==== 2803+0+0 (unknown 440757034 0 0) 0x7fc9b0088430 con 0x7fc9b8109040 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 10 client.24229.objecter ms_dispatch 0x7fc9b8001550 osd_map(28..28 src has 1..28) 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 3 client.24229.objecter handle_osd_map got epochs [28,28] > 0 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 3 client.24229.objecter handle_osd_map decoding full epoch 28 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.020+0000 7fc9b57fa700 20 client.24229.objecter dump_active .. 0 homeless 2024-03-30T14:52:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.021+0000 7fc9b57fa700 10 client.24229.objecter ms_handle_connect 0x7fc9a00704b0 2024-03-30T14:52:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.025+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc9b0056b10 con 0x7fc9b8109040 2024-03-30T14:52:45.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:45.234+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc9a4000b90 con 0x7fc9a00704b0 2024-03-30T14:52:45.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:45 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi028 2024-03-30T14:52:45.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:45 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-30T14:52:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:45 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi028 2024-03-30T14:52:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:45 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-30T14:52:45.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:45 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi028 2024-03-30T14:52:45.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:45 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 4017M 2024-03-30T14:52:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:46 smithi134 ceph-mon[33714]: from='client.24229 v1:172.21.15.134:0/2574471497' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:46 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:46 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:46 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:46 smithi134 ceph-mon[33714]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[41614]: from='client.24229 v1:172.21.15.134:0/2574471497' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[41614]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[29876]: from='client.24229 v1:172.21.15.134:0/2574471497' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:52:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:52:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:52:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:46 smithi028 ceph-mon[29876]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:47 smithi134 ceph-mon[33714]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:47.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:47 smithi028 ceph-mon[29876]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:47 smithi028 ceph-mon[41614]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/2625243173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:48 smithi134 ceph-mon[33714]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:48 smithi134 ceph-mon[33714]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]': finished 2024-03-30T14:52:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:48 smithi134 ceph-mon[33714]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T14:52:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:48 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:52:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/2625243173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[29876]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[29876]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]': finished 2024-03-30T14:52:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[29876]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T14:52:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:52:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/2625243173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[41614]: from='client.24235 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]: dispatch 2024-03-30T14:52:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[41614]: from='client.24235 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c"}]': finished 2024-03-30T14:52:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[41614]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T14:52:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:48 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:52:49.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/2769630420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:49 smithi134 ceph-mon[33714]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:49.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/2769630420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:49.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:49 smithi028 ceph-mon[29876]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:49.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/2769630420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:52:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:49 smithi028 ceph-mon[41614]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:50.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:50.018+0000 7fc9be20a700 10 client.24229.objecter tick 2024-03-30T14:52:52.320 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:52 smithi134 ceph-mon[33714]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:52 smithi028 ceph-mon[29876]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:52 smithi028 ceph-mon[41614]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:53.484 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:53 smithi134 ceph-mon[33714]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:53 smithi028 ceph-mon[29876]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:53.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:53 smithi028 ceph-mon[41614]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:54.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:54 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:52:54.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:54 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:54 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:52:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:54 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:54.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:54 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T14:52:54.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:54 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:52:55.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:52:55.019+0000 7fc9be20a700 10 client.24229.objecter tick 2024-03-30T14:52:55.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:55 smithi134 ceph-mon[33714]: Deploying daemon osd.4 on smithi134 2024-03-30T14:52:55.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:55 smithi134 ceph-mon[33714]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:55.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:55 smithi028 ceph-mon[29876]: Deploying daemon osd.4 on smithi134 2024-03-30T14:52:55.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:55 smithi028 ceph-mon[29876]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:55.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:55 smithi028 ceph-mon[41614]: Deploying daemon osd.4 on smithi134 2024-03-30T14:52:55.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:55 smithi028 ceph-mon[41614]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:57.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:57 smithi134 ceph-mon[33714]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:57 smithi028 ceph-mon[29876]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:57.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:57 smithi028 ceph-mon[41614]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:59.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:52:59 smithi134 ceph-mon[33714]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:52:59 smithi028 ceph-mon[29876]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:52:59.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:52:59 smithi028 ceph-mon[41614]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:00.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:00.020+0000 7fc9be20a700 10 client.24229.objecter tick 2024-03-30T14:53:00.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:00 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:00 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:00 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:00.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:00.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:00 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:01.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:01 smithi134 ceph-mon[33714]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:01.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:01 smithi028 ceph-mon[29876]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:01 smithi028 ceph-mon[41614]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:03.297 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 4 on host 'smithi134' 2024-03-30T14:53:03.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.296+0000 7fc9b57fa700 1 -- 172.21.15.134:0/2574471497 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2915527135) 0x7fc9a4000b90 con 0x7fc9a00704b0 2024-03-30T14:53:03.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.298+0000 7fc9bea0b700 10 librados: watch_flush enter 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.298+0000 7fc9bea0b700 10 librados: watch_flush exit 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.298+0000 7fc9bea0b700 20 client.24229.objecter shutdown clearing up homeless session... 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.298+0000 7fc9bea0b700 10 client.24229.objecter successfully canceled tick 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 >> v1:172.21.15.28:6800/3720192247 conn(0x7fc9a00704b0 legacy=0x7fc9a0072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 >> v1:172.21.15.134:6789/0 conn(0x7fc9b8109040 legacy=0x7fc9b81a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 shutdown_connections 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 >> 172.21.15.134:0/2574471497 conn(0x7fc9b80fc500 msgr2=0x7fc9b810b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 shutdown_connections 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 -- 172.21.15.134:0/2574471497 wait complete. 2024-03-30T14:53:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:03.299+0000 7fc9bea0b700 1 librados: shutdown 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:03.599 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:03 smithi134 ceph-mon[33714]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:03.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[41614]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:03.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:03 smithi028 ceph-mon[29876]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:03.909 DEBUG:teuthology.orchestra.run.smithi134:osd.4> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T14:53:03.912 INFO:tasks.cephadm:Deploying osd.5 on smithi134 with /dev/vg_nvme/lv_3... 2024-03-30T14:53:03.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T14:53:04.658 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:53:04.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:53:04.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-mon[33714]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:04.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-mon[33714]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T14:53:04.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:53:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[29876]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[29876]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T14:53:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T14:53:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[41614]: from='osd.4 v1:172.21.15.134:6800/3063775514' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[41614]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T14:53:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:04 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:05.048 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T14:53:04.838+0000 7f99c799b700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T14:53:05.048 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 14:53:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T14:53:04.841+0000 7f99bef85700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: osd.4 v1:172.21.15.134:6800/3063775514 boot 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T14:53:05.600 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:05 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T14:53:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: osd.4 v1:172.21.15.134:6800/3063775514 boot 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' 2024-03-30T14:53:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-30T14:53:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: osd.4 v1:172.21.15.134:6800/3063775514 boot 2024-03-30T14:53:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T14:53:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:05 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:53:06.541 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:06 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:06.541 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:06 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:06.541 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:06 smithi134 ceph-mon[33714]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T14:53:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[29876]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T14:53:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:06 smithi028 ceph-mon[41614]: osdmap e33: 5 total, 5 up, 5 in 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125616 s, 835 MB/s 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 2024-03-30T14:53:07.958 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-03-30T14:53:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T14:53:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:08 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: osdmap e34: 5 total, 5 up, 5 in 2024-03-30T14:53:08.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:08.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:08.636 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.636 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:08.636 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:08.636 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:08 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:08.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-03-30T14:53:08.913 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:53:09.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:09 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi134 2024-03-30T14:53:09.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:09 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 17088M 2024-03-30T14:53:09.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:09 smithi134 ceph-mon[33714]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T14:53:09.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi134 2024-03-30T14:53:09.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 17088M 2024-03-30T14:53:09.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[29876]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T14:53:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi134 2024-03-30T14:53:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 17088M 2024-03-30T14:53:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:09 smithi028 ceph-mon[41614]: osdmap e35: 5 total, 5 up, 5 in 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.481+0000 7fc7c10d6700 1 -- 172.21.15.134:0/602265944 >> v1:172.21.15.134:6789/0 conn(0x7fc7bc102030 legacy=0x7fc7bc102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.481+0000 7fc7c10d6700 1 -- 172.21.15.134:0/602265944 shutdown_connections 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.481+0000 7fc7c10d6700 1 -- 172.21.15.134:0/602265944 >> 172.21.15.134:0/602265944 conn(0x7fc7bc0fd5f0 msgr2=0x7fc7bc0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.481+0000 7fc7c10d6700 1 -- 172.21.15.134:0/602265944 shutdown_connections 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.481+0000 7fc7c10d6700 1 -- 172.21.15.134:0/602265944 wait complete. 2024-03-30T14:53:10.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 librados: starting msgr at 2024-03-30T14:53:10.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 librados: starting objecter 2024-03-30T14:53:10.484 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:10 smithi134 ceph-mon[33714]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:10.484 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:10 smithi134 ceph-mon[33714]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 Processor -- start 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 -- start start 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 librados: setting wanted keys 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 librados: calling monclient init 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.482+0000 7fc7c10d6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7bc197e60 con 0x7fc7bc10a1a0 2024-03-30T14:53:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7c10d6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7bc1a3650 con 0x7fc7bc102030 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7c10d6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7bc1a4830 con 0x7fc7bc106460 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7bad9d700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc7bc102030 0x7fc7bc197310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:33218/0 (socket says 172.21.15.134:33218) 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7bad9d700 1 -- 172.21.15.134:0/706814204 learned_addr learned my addr 172.21.15.134:0/706814204 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter ms_handle_connect 0x7fc7bc102030 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter ms_handle_connect 0x7fc7bc106460 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880722105 0 0) 0x7fc7bc1a3650 con 0x7fc7bc102030 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7a40035b0 con 0x7fc7bc102030 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter ms_handle_connect 0x7fc7bc10a1a0 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47765121 0 0) 0x7fc7bc1a4830 con 0x7fc7bc106460 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7bc1a3650 con 0x7fc7bc106460 2024-03-30T14:53:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2806642806 0 0) 0x7fc7a40035b0 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7bc1a4830 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7ac004780 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798177599 0 0) 0x7fc7bc197e60 con 0x7fc7bc10a1a0 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7a40035b0 con 0x7fc7bc10a1a0 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 326211816 0 0) 0x7fc7bc1a3650 con 0x7fc7bc106460 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7bc197e60 con 0x7fc7bc106460 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7b00014d0 con 0x7fc7bc106460 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1562064525 0 0) 0x7fc7bc1a4830 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 >> v1:172.21.15.28:6790/0 conn(0x7fc7bc106460 legacy=0x7fc7bc19d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 >> v1:172.21.15.28:6789/0 conn(0x7fc7bc10a1a0 legacy=0x7fc7bc1a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc1a5a50 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.483+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc1a4a60 con 0x7fc7bc102030 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.484+0000 7fc7c10d6700 10 client.24256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:53:10.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.484+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7bc1a4fa0 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.484+0000 7fc7c10d6700 1 librados: init done 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.484+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc7ac001bf0 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.484+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7ac005040 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.485+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a8005150 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7fc7ac0051e0 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(36..36 src has 1..36) ==== 3205+0+0 (unknown 2493561072 0 0) 0x7fc7ac088590 con 0x7fc7bc102030 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 10 client.24256.objecter ms_dispatch 0x7fc7bc001550 osd_map(36..36 src has 1..36) 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 3 client.24256.objecter handle_osd_map got epochs [36,36] > 0 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 3 client.24256.objecter handle_osd_map decoding full epoch 36 2024-03-30T14:53:10.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 20 client.24256.objecter dump_active .. 0 homeless 2024-03-30T14:53:10.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.486+0000 7fc7a3fff700 10 client.24256.objecter ms_handle_connect 0x7fc7a40702f0 2024-03-30T14:53:10.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.491+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc7ac05a590 con 0x7fc7bc102030 2024-03-30T14:53:10.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:10 smithi028 ceph-mon[29876]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:10.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:10 smithi028 ceph-mon[29876]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T14:53:10.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:10 smithi028 ceph-mon[41614]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 547 MiB used, 446 GiB / 447 GiB avail 2024-03-30T14:53:10.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:10 smithi028 ceph-mon[41614]: osdmap e36: 5 total, 5 up, 5 in 2024-03-30T14:53:10.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:10.690+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc7a8000bc0 con 0x7fc7a40702f0 2024-03-30T14:53:11.522 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:11 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:11.522 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:11 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:11.522 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:11 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:11.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:11 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:12.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:12 smithi134 ceph-mon[33714]: from='client.24256 v1:172.21.15.134:0/706814204' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:12.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:12 smithi134 ceph-mon[33714]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 148 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:53:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:12 smithi028 ceph-mon[29876]: from='client.24256 v1:172.21.15.134:0/706814204' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:12 smithi028 ceph-mon[29876]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 148 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:53:12.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:12 smithi028 ceph-mon[41614]: from='client.24256 v1:172.21.15.134:0/706814204' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:12.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:12 smithi028 ceph-mon[41614]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 148 MiB used, 447 GiB / 447 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: osdmap e37: 5 total, 5 up, 5 in 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/3653931535' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]': finished 2024-03-30T14:53:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T14:53:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/3245527626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:14.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:14.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: osdmap e37: 5 total, 5 up, 5 in 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/3653931535' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]': finished 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/3245527626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: osdmap e37: 5 total, 5 up, 5 in 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/3653931535' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: from='client.24262 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]: dispatch 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: from='client.24262 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45"}]': finished 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: osdmap e38: 6 total, 5 up, 6 in 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/3245527626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:15.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:15.485+0000 7fc7bbfff700 10 client.24256.objecter tick 2024-03-30T14:53:15.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:15 smithi134 ceph-mon[33714]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:15.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:15 smithi028 ceph-mon[29876]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:15.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:15 smithi028 ceph-mon[41614]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:53:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:17 smithi134 ceph-mon[33714]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:53:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:17 smithi028 ceph-mon[29876]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:53:17.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:17 smithi028 ceph-mon[41614]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:53:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:19 smithi134 ceph-mon[33714]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:19.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:19 smithi028 ceph-mon[29876]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:19.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:19 smithi028 ceph-mon[41614]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:20.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:20.486+0000 7fc7bbfff700 10 client.24256.objecter tick 2024-03-30T14:53:20.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:20 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:53:20.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:20 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:20.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:20 smithi134 ceph-mon[33714]: Deploying daemon osd.5 on smithi134 2024-03-30T14:53:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:53:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[29876]: Deploying daemon osd.5 on smithi134 2024-03-30T14:53:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T14:53:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:20 smithi028 ceph-mon[41614]: Deploying daemon osd.5 on smithi134 2024-03-30T14:53:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:21 smithi134 ceph-mon[33714]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:21.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:21 smithi028 ceph-mon[29876]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:21.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:21 smithi028 ceph-mon[41614]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:23 smithi134 ceph-mon[33714]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:23.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:23 smithi028 ceph-mon[29876]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:23.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:23 smithi028 ceph-mon[41614]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:25.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:25.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:25.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:25.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:25.487+0000 7fc7bbfff700 10 client.24256.objecter tick 2024-03-30T14:53:25.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:25.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:25.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:26.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:26 smithi134 ceph-mon[33714]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:26 smithi028 ceph-mon[29876]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:26 smithi028 ceph-mon[41614]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:27.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:27 smithi134 ceph-mon[33714]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:27.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:27 smithi028 ceph-mon[29876]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:27.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:27 smithi028 ceph-mon[41614]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:53:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T14:53:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:28 smithi134 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:29.054 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 5 on host 'smithi134' 2024-03-30T14:53:29.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.053+0000 7fc7a3fff700 1 -- 172.21.15.134:0/706814204 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 4057106283) 0x7fc7a8000bc0 con 0x7fc7a40702f0 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 10 librados: watch_flush enter 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 10 librados: watch_flush exit 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 20 client.24256.objecter shutdown clearing up homeless session... 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 10 client.24256.objecter successfully canceled tick 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 >> v1:172.21.15.28:6800/3720192247 conn(0x7fc7a40702f0 legacy=0x7fc7a40727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 >> v1:172.21.15.134:6789/0 conn(0x7fc7bc102030 legacy=0x7fc7bc197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 shutdown_connections 2024-03-30T14:53:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 >> 172.21.15.134:0/706814204 conn(0x7fc7bc0fd5f0 msgr2=0x7fc7bc10d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:53:29.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 shutdown_connections 2024-03-30T14:53:29.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 -- 172.21.15.134:0/706814204 wait complete. 2024-03-30T14:53:29.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:29.055+0000 7fc7c10d6700 1 librados: shutdown 2024-03-30T14:53:29.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[29876]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: osdmap e39: 6 total, 5 up, 6 in 2024-03-30T14:53:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='osd.5 v1:172.21.15.134:6804/3668151179' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:28 smithi028 ceph-mon[41614]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:29.661 DEBUG:teuthology.orchestra.run.smithi134:osd.5> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T14:53:29.663 INFO:tasks.cephadm:Deploying osd.6 on smithi134 with /dev/vg_nvme/lv_2... 2024-03-30T14:53:29.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T14:53:30.264 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.264 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: osdmap e40: 6 total, 5 up, 6 in 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.265 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:29 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: osdmap e40: 6 total, 5 up, 6 in 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: osdmap e40: 6 total, 5 up, 6 in 2024-03-30T14:53:30.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:29 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:30.504 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:53:31.480 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: osd.5 v1:172.21.15.134:6804/3668151179 boot 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T14:53:31.481 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:31 smithi134 ceph-mon[33714]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:31.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:31.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: osd.5 v1:172.21.15.134:6804/3668151179 boot 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[29876]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: osd.5 v1:172.21.15.134:6804/3668151179 boot 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: osdmap e41: 6 total, 6 up, 6 in 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: osdmap e42: 6 total, 6 up, 6 in 2024-03-30T14:53:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:31 smithi028 ceph-mon[41614]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:33.237 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T14:53:33.237 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T14:53:33.238 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-03-30T14:53:33.238 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-03-30T14:53:33.238 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134198 s, 781 MB/s 2024-03-30T14:53:33.238 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-03-30T14:53:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:33 smithi028 ceph-mon[29876]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T14:53:33.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:33 smithi028 ceph-mon[41614]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T14:53:33.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:33 smithi134 ceph-mon[33714]: osdmap e43: 6 total, 6 up, 6 in 2024-03-30T14:53:34.287 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T14:53:34.287 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:34.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:34 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-03-30T14:53:34.525 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:53:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T14:53:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: osdmap e44: 6 total, 6 up, 6 in 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:34 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:35.592 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:35 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi134 2024-03-30T14:53:35.593 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:35 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 8544M 2024-03-30T14:53:35.593 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:35 smithi134 ceph-mon[33714]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T14:53:35.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi134 2024-03-30T14:53:35.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 8544M 2024-03-30T14:53:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[41614]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T14:53:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi134 2024-03-30T14:53:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 8544M 2024-03-30T14:53:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:35 smithi028 ceph-mon[29876]: osdmap e45: 6 total, 6 up, 6 in 2024-03-30T14:53:36.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.083+0000 7fbea6cfa700 1 -- 172.21.15.134:0/1322689127 >> v1:172.21.15.134:6789/0 conn(0x7fbea0102030 legacy=0x7fbea0102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:36.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.083+0000 7fbea6cfa700 1 -- 172.21.15.134:0/1322689127 shutdown_connections 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.083+0000 7fbea6cfa700 1 -- 172.21.15.134:0/1322689127 >> 172.21.15.134:0/1322689127 conn(0x7fbea00fd5f0 msgr2=0x7fbea00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.083+0000 7fbea6cfa700 1 -- 172.21.15.134:0/1322689127 shutdown_connections 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.083+0000 7fbea6cfa700 1 -- 172.21.15.134:0/1322689127 wait complete. 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 librados: starting msgr at 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 librados: starting objecter 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 Processor -- start 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 -- start start 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 librados: setting wanted keys 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 librados: calling monclient init 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea0197e60 con 0x7fbea0106460 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea01a3650 con 0x7fbea010a1a0 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea6cfa700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbea01a4830 con 0x7fbea0102030 2024-03-30T14:53:36.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea5297700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbea010a1a0 0x7fbea01a0f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:53548/0 (socket says 172.21.15.134:53548) 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbea5297700 1 -- 172.21.15.134:0/2561251379 learned_addr learned my addr 172.21.15.134:0/2561251379 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter ms_handle_connect 0x7fbea010a1a0 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter ms_handle_connect 0x7fbea0102030 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter ms_handle_connect 0x7fbea0106460 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.084+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968221196 0 0) 0x7fbea01a3650 con 0x7fbea010a1a0 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe900035b0 con 0x7fbea010a1a0 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771463185 0 0) 0x7fbea01a4830 con 0x7fbea0102030 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea01a3650 con 0x7fbea0102030 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3637291558 0 0) 0x7fbea0197e60 con 0x7fbea0106460 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea01a4830 con 0x7fbea0106460 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3648062161 0 0) 0x7fbe900035b0 con 0x7fbea010a1a0 2024-03-30T14:53:36.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea0197e60 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe98004390 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 629916146 0 0) 0x7fbea0197e60 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 >> v1:172.21.15.28:6790/0 conn(0x7fbea0102030 legacy=0x7fbea0197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 >> v1:172.21.15.28:6789/0 conn(0x7fbea0106460 legacy=0x7fbea019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea01a5a50 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbea01a4a60 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbe980016f0 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe980060e0 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbea6cfa700 10 client.24283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbea01a4fa0 con 0x7fbea010a1a0 2024-03-30T14:53:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.085+0000 7fbea6cfa700 1 librados: init done 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.086+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea0068b30 con 0x7fbea010a1a0 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.086+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7fbe98001250 con 0x7fbea010a1a0 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(45..45 src has 1..45) ==== 3497+0+0 (unknown 2618941028 0 0) 0x7fbe980885e0 con 0x7fbea010a1a0 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 10 client.24283.objecter ms_dispatch 0x7fbea0001550 osd_map(45..45 src has 1..45) 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 3 client.24283.objecter handle_osd_map got epochs [45,45] > 0 2024-03-30T14:53:36.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 3 client.24283.objecter handle_osd_map decoding full epoch 45 2024-03-30T14:53:36.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 20 client.24283.objecter dump_active .. 0 homeless 2024-03-30T14:53:36.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.087+0000 7fbe9dffb700 10 client.24283.objecter ms_handle_connect 0x7fbe9006fd00 2024-03-30T14:53:36.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.091+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbe98056b50 con 0x7fbea010a1a0 2024-03-30T14:53:36.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:36.296+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbea01a5410 con 0x7fbe9006fd00 2024-03-30T14:53:36.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:36 smithi134 ceph-mon[33714]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:36.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:36 smithi028 ceph-mon[29876]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:36.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:36 smithi028 ceph-mon[41614]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:37.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:37 smithi134 ceph-mon[33714]: from='client.24283 v1:172.21.15.134:0/2561251379' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:37.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:37 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:37.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:37 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:37.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:37 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[29876]: from='client.24283 v1:172.21.15.134:0/2561251379' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[41614]: from='client.24283 v1:172.21.15.134:0/2561251379' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:53:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:53:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:53:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:37 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:38.260 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:38 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T14:53:38.260 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:38 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T14:53:38.260 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:38 smithi134 ceph-mon[33714]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:38.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T14:53:38.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T14:53:38.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[41614]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T14:53:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T14:53:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:38 smithi028 ceph-mon[29876]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:39.445 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/1049547227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.446 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:39 smithi134 ceph-mon[33714]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.446 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:39 smithi134 ceph-mon[33714]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]': finished 2024-03-30T14:53:39.446 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:39 smithi134 ceph-mon[33714]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T14:53:39.446 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:39 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:39.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/1049547227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[41614]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[41614]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]': finished 2024-03-30T14:53:39.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[41614]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/1049547227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[29876]: from='client.24289 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]: dispatch 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[29876]: from='client.24289 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d"}]': finished 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[29876]: osdmap e46: 7 total, 6 up, 7 in 2024-03-30T14:53:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:39 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:40.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:40 smithi028 ceph-mon[29876]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:40.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/3360730669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:40.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:40 smithi028 ceph-mon[41614]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:40.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/3360730669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:40 smithi134 ceph-mon[33714]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/3360730669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:53:41.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:41.086+0000 7fbea64f9700 10 client.24283.objecter tick 2024-03-30T14:53:41.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:41 smithi028 ceph-mon[29876]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:41.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:41 smithi028 ceph-mon[41614]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:41.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:41 smithi134 ceph-mon[33714]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:44.336 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:44 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:44.336 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:44 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:44.336 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:44 smithi134 ceph-mon[33714]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T14:53:44.336 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:44 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:44.337 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:44 smithi134 ceph-mon[33714]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[29876]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[29876]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[41614]: osdmap e47: 7 total, 6 up, 7 in 2024-03-30T14:53:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:44 smithi028 ceph-mon[41614]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:45.614 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:45 smithi134 ceph-mon[33714]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:45.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:45 smithi028 ceph-mon[29876]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:45.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:45 smithi028 ceph-mon[41614]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:46.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:46.087+0000 7fbea64f9700 10 client.24283.objecter tick 2024-03-30T14:53:46.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:46 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:53:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:46 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:46 smithi134 ceph-mon[33714]: Deploying daemon osd.6 on smithi134 2024-03-30T14:53:46.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:53:46.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:46.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[29876]: Deploying daemon osd.6 on smithi134 2024-03-30T14:53:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T14:53:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:46 smithi028 ceph-mon[41614]: Deploying daemon osd.6 on smithi134 2024-03-30T14:53:47.592 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:47 smithi134 ceph-mon[33714]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:47.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:47 smithi028 ceph-mon[29876]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:47 smithi028 ceph-mon[41614]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:50.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:50.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:50.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[29876]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T14:53:50.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:50.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[29876]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[41614]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T14:53:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:50 smithi028 ceph-mon[41614]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:50.559 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:50 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:50.559 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:50 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:50.559 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:50 smithi134 ceph-mon[33714]: osdmap e48: 7 total, 6 up, 7 in 2024-03-30T14:53:50.559 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:50.559 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:50 smithi134 ceph-mon[33714]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:51.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:51.088+0000 7fbea64f9700 10 client.24283.objecter tick 2024-03-30T14:53:52.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:52 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:52 smithi134 ceph-mon[33714]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[41614]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:52 smithi028 ceph-mon[29876]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:53.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:53 smithi134 ceph-mon[33714]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:53.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:53 smithi028 ceph-mon[41614]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:53 smithi028 ceph-mon[29876]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:55.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.009+0000 7fbe9dffb700 1 -- 172.21.15.134:0/2561251379 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 367834295) 0x7fbea01a5410 con 0x7fbe9006fd00 2024-03-30T14:53:55.011 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 6 on host 'smithi134' 2024-03-30T14:53:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 10 librados: watch_flush enter 2024-03-30T14:53:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 10 librados: watch_flush exit 2024-03-30T14:53:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 20 client.24283.objecter shutdown clearing up homeless session... 2024-03-30T14:53:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 10 client.24283.objecter successfully canceled tick 2024-03-30T14:53:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 >> v1:172.21.15.28:6800/3720192247 conn(0x7fbe9006fd00 legacy=0x7fbe900721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 >> v1:172.21.15.134:6789/0 conn(0x7fbea010a1a0 legacy=0x7fbea01a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 shutdown_connections 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 >> 172.21.15.134:0/2561251379 conn(0x7fbea00fd5f0 msgr2=0x7fbea010d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 shutdown_connections 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 -- 172.21.15.134:0/2561251379 wait complete. 2024-03-30T14:53:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:53:55.012+0000 7fbea6cfa700 1 librados: shutdown 2024-03-30T14:53:55.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:55.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:53:55.649 DEBUG:teuthology.orchestra.run.smithi134:osd.6> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T14:53:55.651 INFO:tasks.cephadm:Deploying osd.7 on smithi134 with /dev/vg_nvme/lv_1... 2024-03-30T14:53:55.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: osdmap e49: 7 total, 6 up, 7 in 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-mon[33714]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:56.361 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:53:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:53:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: osdmap e49: 7 total, 6 up, 7 in 2024-03-30T14:53:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[29876]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: osdmap e49: 7 total, 6 up, 7 in 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: from='osd.6 v1:172.21.15.134:6808/2135224194' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:53:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:56 smithi028 ceph-mon[41614]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:56.746 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T14:53:56.594+0000 7f92aed7d700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T14:53:56.746 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 14:53:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T14:53:56.601+0000 7f92a6b68700 -1 osd.6 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:53:57.301 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: osdmap e50: 7 total, 6 up, 7 in 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: osdmap e51: 7 total, 6 up, 7 in 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.302 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:57 smithi134 ceph-mon[33714]: from='osd.6 ' entity='osd.6' 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: osdmap e50: 7 total, 6 up, 7 in 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: osdmap e51: 7 total, 6 up, 7 in 2024-03-30T14:53:57.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[41614]: from='osd.6 ' entity='osd.6' 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: osdmap e50: 7 total, 6 up, 7 in 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: osdmap e51: 7 total, 6 up, 7 in 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:57 smithi028 ceph-mon[29876]: from='osd.6 ' entity='osd.6' 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: osd.6 v1:172.21.15.134:6808/2135224194 boot 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:58 smithi134 ceph-mon[33714]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:53:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:53:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: osd.6 v1:172.21.15.134:6808/2135224194 boot 2024-03-30T14:53:58.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[29876]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: osd.6 v1:172.21.15.134:6808/2135224194 boot 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: osdmap e52: 7 total, 7 up, 7 in 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:53:58.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:58 smithi028 ceph-mon[41614]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-30T14:53:59.337 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T14:53:59.337 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.337 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:59.338 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:53:59 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112081 s, 936 MB/s 2024-03-30T14:53:59.686 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-03-30T14:53:59.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: osdmap e53: 7 total, 7 up, 7 in 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:53:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:53:59 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:00.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:00 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi134 2024-03-30T14:54:00.285 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:00 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 5696M 2024-03-30T14:54:00.285 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:00 smithi134 ceph-mon[33714]: osdmap e54: 7 total, 7 up, 7 in 2024-03-30T14:54:00.285 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:00 smithi134 ceph-mon[33714]: pgmap v165: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:00.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi134:vg_nvme/lv_1 2024-03-30T14:54:00.524 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:00.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi134 2024-03-30T14:54:00.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 5696M 2024-03-30T14:54:00.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[29876]: osdmap e54: 7 total, 7 up, 7 in 2024-03-30T14:54:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[29876]: pgmap v165: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi134 2024-03-30T14:54:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 5696M 2024-03-30T14:54:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[41614]: osdmap e54: 7 total, 7 up, 7 in 2024-03-30T14:54:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:00 smithi028 ceph-mon[41614]: pgmap v165: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:01.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:01 smithi134 ceph-mon[33714]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T14:54:01.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:01 smithi134 ceph-mon[33714]: osdmap e55: 7 total, 7 up, 7 in 2024-03-30T14:54:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:01 smithi028 ceph-mon[29876]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T14:54:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:01 smithi028 ceph-mon[29876]: osdmap e55: 7 total, 7 up, 7 in 2024-03-30T14:54:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:01 smithi028 ceph-mon[41614]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T14:54:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:01 smithi028 ceph-mon[41614]: osdmap e55: 7 total, 7 up, 7 in 2024-03-30T14:54:02.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.147+0000 7f975f23f700 1 -- 172.21.15.134:0/2332130208 >> v1:172.21.15.134:6789/0 conn(0x7f9758100330 legacy=0x7f9758100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.147+0000 7f975f23f700 1 -- 172.21.15.134:0/2332130208 shutdown_connections 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.147+0000 7f975f23f700 1 -- 172.21.15.134:0/2332130208 >> 172.21.15.134:0/2332130208 conn(0x7f97580fbcb0 msgr2=0x7f97580fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.147+0000 7f975f23f700 1 -- 172.21.15.134:0/2332130208 shutdown_connections 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.147+0000 7f975f23f700 1 -- 172.21.15.134:0/2332130208 wait complete. 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 librados: starting msgr at 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 librados: starting objecter 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 Processor -- start 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 -- start start 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 librados: setting wanted keys 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 librados: calling monclient init 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97581982a0 con 0x7f9758100330 2024-03-30T14:54:02.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97581a3a90 con 0x7f9758106540 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f975f23f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f97581a4c70 con 0x7f975810a1a0 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9757fff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9758106540 0x7f975819dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34168/0 (socket says 172.21.15.134:34168) 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9757fff700 1 -- 172.21.15.134:0/1514833287 learned_addr learned my addr 172.21.15.134:0/1514833287 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter ms_handle_connect 0x7f9758106540 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592507583 0 0) 0x7f97581a3a90 con 0x7f9758106540 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97480035b0 con 0x7f9758106540 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter ms_handle_connect 0x7f975810a1a0 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter ms_handle_connect 0x7f9758100330 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957696371 0 0) 0x7f97581a4c70 con 0x7f975810a1a0 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.148+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97581a3a90 con 0x7f975810a1a0 2024-03-30T14:54:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405039313 0 0) 0x7f97581982a0 con 0x7f9758100330 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97581a4c70 con 0x7f9758100330 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1295076438 0 0) 0x7f97480035b0 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97581982a0 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9744001970 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2326906735 0 0) 0x7f97581a3a90 con 0x7f975810a1a0 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97480035b0 con 0x7f975810a1a0 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9750001aa0 con 0x7f975810a1a0 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1704634214 0 0) 0x7f97581982a0 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 >> v1:172.21.15.28:6790/0 conn(0x7f975810a1a0 legacy=0x7f97581a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 >> v1:172.21.15.28:6789/0 conn(0x7f9758100330 legacy=0x7f9758197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97581a5e90 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97581a4ea0 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9744003df0 con 0x7f9758106540 2024-03-30T14:54:02.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9744005e00 con 0x7f9758106540 2024-03-30T14:54:02.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f975f23f700 10 client.24310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:02.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97581a53e0 con 0x7f9758106540 2024-03-30T14:54:02.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.149+0000 7f975f23f700 1 librados: init done 2024-03-30T14:54:02.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.150+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9758068b30 con 0x7f9758106540 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.150+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f9744003950 con 0x7f9758106540 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(56..56 src has 1..56) ==== 3789+0+0 (unknown 3305213283 0 0) 0x7f9744088590 con 0x7f9758106540 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 10 client.24310.objecter ms_dispatch 0x7f9758001550 osd_map(56..56 src has 1..56) 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 3 client.24310.objecter handle_osd_map got epochs [56,56] > 0 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 3 client.24310.objecter handle_osd_map decoding full epoch 56 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 20 client.24310.objecter dump_active .. 0 homeless 2024-03-30T14:54:02.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.151+0000 7f9755ffb700 10 client.24310.objecter ms_handle_connect 0x7f9748070000 2024-03-30T14:54:02.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.155+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f97440568a0 con 0x7f9758106540 2024-03-30T14:54:02.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:02.361+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f97581a57e0 con 0x7f9748070000 2024-03-30T14:54:02.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:02 smithi134 ceph-mon[33714]: osdmap e56: 7 total, 7 up, 7 in 2024-03-30T14:54:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:02 smithi134 ceph-mon[33714]: pgmap v168: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:02.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:02 smithi028 ceph-mon[29876]: osdmap e56: 7 total, 7 up, 7 in 2024-03-30T14:54:02.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:02 smithi028 ceph-mon[29876]: pgmap v168: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:02.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:02 smithi028 ceph-mon[41614]: osdmap e56: 7 total, 7 up, 7 in 2024-03-30T14:54:02.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:02 smithi028 ceph-mon[41614]: pgmap v168: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T14:54:03.638 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:54:03.639 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:54:03.639 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:03 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:54:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:54:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T14:54:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T14:54:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:03 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:04 smithi134 ceph-mon[33714]: from='client.24310 v1:172.21.15.134:0/1514833287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:04 smithi134 ceph-mon[33714]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:54:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:04 smithi028 ceph-mon[29876]: from='client.24310 v1:172.21.15.134:0/1514833287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:04 smithi028 ceph-mon[29876]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:54:04.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:04 smithi028 ceph-mon[41614]: from='client.24310 v1:172.21.15.134:0/1514833287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:04.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:04 smithi028 ceph-mon[41614]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: Cluster is now healthy 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/261652349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]': finished 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T14:54:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:05 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T14:54:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: Cluster is now healthy 2024-03-30T14:54:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/261652349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]': finished 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: Cluster is now healthy 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/261652349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: from='client.24316 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]: dispatch 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: from='client.24316 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c"}]': finished 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: osdmap e57: 8 total, 7 up, 8 in 2024-03-30T14:54:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:05 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:06 smithi134 ceph-mon[33714]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/2710510790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:54:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:06 smithi028 ceph-mon[29876]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:06.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/2710510790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:54:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:06 smithi028 ceph-mon[41614]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/2710510790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T14:54:07.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:07.149+0000 7f975ea3e700 10 client.24310.objecter tick 2024-03-30T14:54:07.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:07 smithi134 ceph-mon[33714]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T14:54:07.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:07 smithi028 ceph-mon[29876]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T14:54:07.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:07 smithi028 ceph-mon[41614]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-30T14:54:09.425 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:09 smithi134 ceph-mon[33714]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T14:54:09.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:09 smithi028 ceph-mon[29876]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T14:54:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:09 smithi028 ceph-mon[41614]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-30T14:54:11.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:11 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:54:11.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:11 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:54:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:11 smithi134 ceph-mon[33714]: osdmap e58: 8 total, 7 up, 8 in 2024-03-30T14:54:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:11 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:11 smithi134 ceph-mon[33714]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[29876]: osdmap e58: 8 total, 7 up, 8 in 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[29876]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[41614]: osdmap e58: 8 total, 7 up, 8 in 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:11 smithi028 ceph-mon[41614]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:12.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:12.150+0000 7f975ea3e700 10 client.24310.objecter tick 2024-03-30T14:54:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:12 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:54:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:12 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:12 smithi134 ceph-mon[33714]: Deploying daemon osd.7 on smithi134 2024-03-30T14:54:12.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:54:12.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:12.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[41614]: Deploying daemon osd.7 on smithi134 2024-03-30T14:54:12.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T14:54:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:12 smithi028 ceph-mon[29876]: Deploying daemon osd.7 on smithi134 2024-03-30T14:54:13.662 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:13 smithi134 ceph-mon[33714]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:13.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:13 smithi028 ceph-mon[29876]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:13 smithi028 ceph-mon[41614]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:15 smithi134 ceph-mon[33714]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:15.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:15 smithi028 ceph-mon[29876]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:15.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:15 smithi028 ceph-mon[41614]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:17.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:17.151+0000 7f975ea3e700 10 client.24310.objecter tick 2024-03-30T14:54:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:18 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:18 smithi134 ceph-mon[33714]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[41614]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:18.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:18.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:18 smithi028 ceph-mon[29876]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:19 smithi134 ceph-mon[33714]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:19.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:19 smithi028 ceph-mon[29876]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:19.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:19 smithi028 ceph-mon[41614]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:20.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.476+0000 7f9755ffb700 1 -- 172.21.15.134:0/1514833287 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1241078275) 0x7f97581a57e0 con 0x7f9748070000 2024-03-30T14:54:20.478 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 7 on host 'smithi134' 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.478+0000 7f975f23f700 10 librados: watch_flush enter 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 10 librados: watch_flush exit 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 20 client.24310.objecter shutdown clearing up homeless session... 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 10 client.24310.objecter successfully canceled tick 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 >> v1:172.21.15.28:6800/3720192247 conn(0x7f9748070000 legacy=0x7f97480724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 >> v1:172.21.15.134:6789/0 conn(0x7f9758106540 legacy=0x7f975819dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 shutdown_connections 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 >> 172.21.15.134:0/1514833287 conn(0x7f97580fbcb0 msgr2=0x7f97580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 shutdown_connections 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 -- 172.21.15.134:0/1514833287 wait complete. 2024-03-30T14:54:20.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:20.479+0000 7f975f23f700 1 librados: shutdown 2024-03-30T14:54:20.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:20 smithi134 ceph-mon[33714]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:20.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:20 smithi134 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:20 smithi028 ceph-mon[29876]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:20 smithi028 ceph-mon[29876]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:20.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:20 smithi028 ceph-mon[41614]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:20.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:20 smithi028 ceph-mon[41614]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T14:54:21.057 DEBUG:teuthology.orchestra.run.smithi134:osd.7> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T14:54:21.059 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-30T14:54:21.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T14:54:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: osdmap e59: 8 total, 7 up, 8 in 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-mon[33714]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: osdmap e59: 8 total, 7 up, 8 in 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[41614]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: osdmap e59: 8 total, 7 up, 8 in 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='osd.7 v1:172.21.15.134:6812/3758190298' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:21.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:21 smithi028 ceph-mon[29876]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-30T14:54:22.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T14:54:21.829+0000 7fecba235700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T14:54:22.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 14:54:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T14:54:21.836+0000 7fecb181f700 -1 osd.7 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[29876]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[29876]: osdmap e60: 8 total, 7 up, 8 in 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[29876]: from='osd.7 ' entity='osd.7' 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[41614]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[41614]: osdmap e60: 8 total, 7 up, 8 in 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:22 smithi028 ceph-mon[41614]: from='osd.7 ' entity='osd.7' 2024-03-30T14:54:22.675 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:22 smithi134 ceph-mon[33714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-03-30T14:54:22.675 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:22 smithi134 ceph-mon[33714]: osdmap e60: 8 total, 7 up, 8 in 2024-03-30T14:54:22.675 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.675 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:22 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:22.675 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:22 smithi134 ceph-mon[33714]: from='osd.7 ' entity='osd.7' 2024-03-30T14:54:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.335+0000 7f5796add700 1 -- 172.21.15.28:0/3578370327 >> v1:172.21.15.28:6790/0 conn(0x7f579010a1c0 legacy=0x7f579010c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- 172.21.15.28:0/3578370327 shutdown_connections 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- 172.21.15.28:0/3578370327 >> 172.21.15.28:0/3578370327 conn(0x7f57900fd5f0 msgr2=0x7f57900ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- 172.21.15.28:0/3578370327 shutdown_connections 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- 172.21.15.28:0/3578370327 wait complete. 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 librados: starting msgr at 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 librados: starting objecter 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 Processor -- start 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- start start 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 librados: setting wanted keys 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 librados: calling monclient init 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57901a4770 con 0x7f579010a1c0 2024-03-30T14:54:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57901a59b0 con 0x7f5790102050 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f5796add700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57901a6bf0 con 0x7f5790106480 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f578ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5790106480 0x7f579019f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36572/0 (socket says 172.21.15.28:36572) 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.336+0000 7f578ffff700 1 -- 172.21.15.28:0/1667083517 learned_addr learned my addr 172.21.15.28:0/1667083517 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter ms_handle_connect 0x7f5790106480 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter ms_handle_connect 0x7f579010a1c0 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602770557 0 0) 0x7f57901a6bf0 con 0x7f5790106480 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5780005240 con 0x7f5790106480 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127972 0 0) 0x7f57901a4770 con 0x7f579010a1c0 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57901a6bf0 con 0x7f579010a1c0 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter ms_handle_connect 0x7f5790102050 2024-03-30T14:54:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480526240 0 0) 0x7f57901a59b0 con 0x7f5790102050 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57901a4770 con 0x7f5790102050 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 205029210 0 0) 0x7f5780005240 con 0x7f5790106480 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57901a59b0 con 0x7f5790106480 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f577c0019b0 con 0x7f5790106480 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3238683577 0 0) 0x7f57901a6bf0 con 0x7f579010a1c0 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5780005240 con 0x7f579010a1c0 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5788001bc0 con 0x7f579010a1c0 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2908201366 0 0) 0x7f57901a4770 con 0x7f5790102050 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57901a6bf0 con 0x7f5790102050 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2423255277 0 0) 0x7f57901a59b0 con 0x7f5790106480 2024-03-30T14:54:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 >> v1:172.21.15.134:6789/0 conn(0x7f5790102050 legacy=0x7f5790075f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 >> v1:172.21.15.28:6789/0 conn(0x7f579010a1c0 legacy=0x7f57901a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57901a7e30 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.337+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57901a49a0 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f577c001cb0 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f577c005dc0 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f5796add700 10 client.34313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f57901a4ee0 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f5796add700 1 librados: init done 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.338+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5790068b30 con 0x7f5790106480 2024-03-30T14:54:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f577c0039d0 con 0x7f5790106480 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4114+0+0 (unknown 2534484894 0 0) 0x7f577c0888e0 con 0x7f5790106480 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 10 client.34313.objecter ms_dispatch 0x7f5790001550 osd_map(61..61 src has 1..61) 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 3 client.34313.objecter handle_osd_map got epochs [61,61] > 0 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 3 client.34313.objecter handle_osd_map decoding full epoch 61 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.339+0000 7f578dffb700 20 client.34313.objecter dump_active .. 0 homeless 2024-03-30T14:54:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.340+0000 7f578dffb700 10 client.34313.objecter ms_handle_connect 0x7f5780070350 2024-03-30T14:54:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.344+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f577c056b00 con 0x7f5790106480 2024-03-30T14:54:23.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.551+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f579010e1d0 con 0x7f5790106480 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: osd.7 v1:172.21.15.134:6812/3758190298 boot 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:23.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[29876]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: osd.7 v1:172.21.15.134:6812/3758190298 boot 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:23.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:23 smithi028 ceph-mon[41614]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.551+0000 7f578dffb700 1 -- 172.21.15.28:0/1667083517 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v62) ==== 74+0+130 (unknown 442799425 0 3635336967) 0x7f577c056400 con 0x7f5790106480 2024-03-30T14:54:23.554 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 10 librados: watch_flush enter 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 10 librados: watch_flush exit 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 20 client.34313.objecter shutdown clearing up homeless session... 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 10 client.34313.objecter successfully canceled tick 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 >> v1:172.21.15.28:6800/3720192247 conn(0x7f5780070350 legacy=0x7f5780072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 >> v1:172.21.15.28:6790/0 conn(0x7f5790106480 legacy=0x7f579019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 shutdown_connections 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.553+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 >> 172.21.15.28:0/1667083517 conn(0x7f57900fd5f0 msgr2=0x7f5790109050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.554+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 shutdown_connections 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.554+0000 7f5796add700 1 -- 172.21.15.28:0/1667083517 wait complete. 2024-03-30T14:54:23.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:23.554+0000 7f5796add700 1 librados: shutdown 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: osd.7 v1:172.21.15.134:6812/3758190298 boot 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: osdmap e61: 8 total, 8 up, 8 in 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:54:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:23 smithi134 ceph-mon[33714]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:24.245 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":62,"num_osds":8,"num_up_osds":8,"osd_up_since":1711810462,"num_in_osds":8,"osd_in_since":1711810445,"num_remapped_pgs":1} 2024-03-30T14:54:24.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:54:24.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:24 smithi028 ceph-mon[41614]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T14:54:24.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1667083517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:54:24.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:24 smithi028 ceph-mon[29876]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T14:54:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1667083517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:54:24.496 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:54:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:24 smithi134 ceph-mon[33714]: osdmap e62: 8 total, 8 up, 8 in 2024-03-30T14:54:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1667083517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: Detected new or changed devices on smithi134 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 4272M 2024-03-30T14:54:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:25 smithi134 ceph-mon[33714]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:25.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: Detected new or changed devices on smithi134 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 4272M 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[29876]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: osdmap e63: 8 total, 8 up, 8 in 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: Detected new or changed devices on smithi134 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 4272M 2024-03-30T14:54:25.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:25.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:25 smithi028 ceph-mon[41614]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.293+0000 7f91d5aa0700 1 -- 172.21.15.28:0/542269298 >> v1:172.21.15.28:6789/0 conn(0x7f91d010eb30 legacy=0x7f91d0110fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 -- 172.21.15.28:0/542269298 shutdown_connections 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 -- 172.21.15.28:0/542269298 >> 172.21.15.28:0/542269298 conn(0x7f91d00761d0 msgr2=0x7f91d00765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 -- 172.21.15.28:0/542269298 shutdown_connections 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 -- 172.21.15.28:0/542269298 wait complete. 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 librados: starting msgr at 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 librados: starting objecter 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 Processor -- start 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 -- start start 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 librados: setting wanted keys 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.294+0000 7f91d5aa0700 1 librados: calling monclient init 2024-03-30T14:54:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91d5aa0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f91d01a7160 con 0x7f91d010eb30 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91d5aa0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f91d01a8380 con 0x7f91d010ae90 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91d5aa0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f91d01a95c0 con 0x7f91d0107060 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91cf7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f91d0107060 0x7f91d01010a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36592/0 (socket says 172.21.15.28:36592) 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91cf7fe700 1 -- 172.21.15.28:0/1833388143 learned_addr learned my addr 172.21.15.28:0/1833388143 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91cffff700 1 --1- 172.21.15.28:0/1833388143 >> v1:172.21.15.28:6789/0 conn(0x7f91d010eb30 0x7f91d01a5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41482/0 (socket says 172.21.15.28:41482) 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter ms_handle_connect 0x7f91d0107060 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter ms_handle_connect 0x7f91d010eb30 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter ms_handle_connect 0x7f91d010ae90 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2419325570 0 0) 0x7f91d01a95c0 con 0x7f91d0107060 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91b80031f0 con 0x7f91d0107060 2024-03-30T14:54:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183908673 0 0) 0x7f91d01a7160 con 0x7f91d010eb30 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d01a95c0 con 0x7f91d010eb30 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2842130331 0 0) 0x7f91d01a8380 con 0x7f91d010ae90 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d01a7160 con 0x7f91d010ae90 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1864072576 0 0) 0x7f91b80031f0 con 0x7f91d0107060 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91d01a8380 con 0x7f91d0107060 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f91c4001a30 con 0x7f91d0107060 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1999482194 0 0) 0x7f91d01a95c0 con 0x7f91d010eb30 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91b80031f0 con 0x7f91d010eb30 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 676479017 0 0) 0x7f91d01a7160 con 0x7f91d010ae90 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91d01a95c0 con 0x7f91d010ae90 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f91c8004720 con 0x7f91d010eb30 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.295+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f91c0001830 con 0x7f91d010ae90 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 168689725 0 0) 0x7f91d01a8380 con 0x7f91d0107060 2024-03-30T14:54:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 >> v1:172.21.15.134:6789/0 conn(0x7f91d010ae90 legacy=0x7f91d01a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 >> v1:172.21.15.28:6789/0 conn(0x7f91d010eb30 legacy=0x7f91d01a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d01aa800 con 0x7f91d0107060 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f91d01a7390 con 0x7f91d0107060 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91d5aa0700 10 client.34319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f91d01a7850 con 0x7f91d0107060 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91d5aa0700 1 librados: init done 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f91c4001c90 con 0x7f91d0107060 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.296+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f91c4005d10 con 0x7f91d0107060 2024-03-30T14:54:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.297+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91bc005150 con 0x7f91d0107060 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.297+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f91c4003930 con 0x7f91d0107060 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(64..64 src has 14..64) ==== 4081+0+0 (unknown 703345397 0 0) 0x7f91c40886f0 con 0x7f91d0107060 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 10 client.34319.objecter ms_dispatch 0x7f91d0001550 osd_map(64..64 src has 14..64) 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 3 client.34319.objecter handle_osd_map got epochs [64,64] > 0 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 3 client.34319.objecter handle_osd_map decoding full epoch 64 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 20 client.34319.objecter dump_active .. 0 homeless 2024-03-30T14:54:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.298+0000 7f91ccff9700 10 client.34319.objecter ms_handle_connect 0x7f91b8070490 2024-03-30T14:54:26.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.302+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f91c4056930 con 0x7f91d0107060 2024-03-30T14:54:26.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.509+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91bc005440 con 0x7f91d0107060 2024-03-30T14:54:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.510+0000 7f91ccff9700 1 -- 172.21.15.28:0/1833388143 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+11512 (unknown 1960508987 0 1244108026) 0x7f91c4056230 con 0x7f91d0107060 2024-03-30T14:54:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:54:26.511 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:48:49.804304+0000","modified":"2024-03-30T14:54:26.426425+0000","last_up_change":"2024-03-30T14:54:22.403545+0000","last_in_change":"2024-03-30T14:54:05.032145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:52:17.629184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb350905-2b09-408a-9adc-4f2af2766479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3394323575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3394323575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3394323575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3394323575}]},"public_addr":"172.21.15.28:6801/3394323575","cluster_addr":"172.21.15.28:6802/3394323575","heartbeat_back_addr":"172.21.15.28:6804/3394323575","heartbeat_front_addr":"172.21.15.28:6803/3394323575","state":["exists","up"]},{"osd":1,"uuid":"76bfea8d-68c7-448e-bb99-d3d003567520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1850313547}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1850313547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1850313547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1850313547}]},"public_addr":"172.21.15.28:6805/1850313547","cluster_addr":"172.21.15.28:6806/1850313547","heartbeat_back_addr":"172.21.15.28:6808/1850313547","heartbeat_front_addr":"172.21.15.28:6807/1850313547","state":["exists","up"]},{"osd":2,"uuid":"ad24917c-f5cb-4ca8-8a52-91d2256ff21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":3451325043}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":3451325043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":3451325043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":3451325043}]},"public_addr":"172.21.15.28:6809/3451325043","cluster_addr":"172.21.15.28:6810/3451325043","heartbeat_back_addr":"172.21.15.28:6812/3451325043","heartbeat_front_addr":"172.21.15.28:6811/3451325043","state":["exists","up"]},{"osd":3,"uuid":"4305caf1-0023-438f-8520-87814130b457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1506157250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1506157250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1506157250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1506157250}]},"public_addr":"172.21.15.28:6813/1506157250","cluster_addr":"172.21.15.28:6814/1506157250","heartbeat_back_addr":"172.21.15.28:6816/1506157250","heartbeat_front_addr":"172.21.15.28:6815/1506157250","state":["exists","up"]},{"osd":4,"uuid":"8180ec45-5260-4cdc-855c-be5302b1465c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6800","nonce":3063775514}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6801","nonce":3063775514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6803","nonce":3063775514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6802","nonce":3063775514}]},"public_addr":"172.21.15.134:6800/3063775514","cluster_addr":"172.21.15.134:6801/3063775514","heartbeat_back_addr":"172.21.15.134:6803/3063775514","heartbeat_front_addr":"172.21.15.134:6802/3063775514","state":["exists","up"]},{"osd":5,"uuid":"265992a1-46bc-4621-b9db-2b5ba1ee5e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6804","nonce":3668151179}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6805","nonce":3668151179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6807","nonce":3668151179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6806","nonce":3668151179}]},"public_addr":"172.21.15.134:6804/3668151179","cluster_addr":"172.21.15.134:6805/3668151179","heartbeat_back_addr":"172.21.15.134:6807/3668151179","heartbeat_front_addr":"172.21.15.134:6806/3668151179","state":["exists","up"]},{"osd":6,"uuid":"5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6808","nonce":2135224194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6809","nonce":2135224194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6811","nonce":2135224194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6810","nonce":2135224194}]},"public_addr":"172.21.15.134:6808/2135224194","cluster_addr":"172.21.15.134:6809/2135224194","heartbeat_back_addr":"172.21.15.134:6811/2135224194","heartbeat_front_addr":"172.21.15.134:6810/2135224194","state":["exists","up"]},{"osd":7,"uuid":"76f25ab9-af2f-4f45-afe3-13ece781d29c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6812","nonce":3758190298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6813","nonce":3758190298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6815","nonce":3758190298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6814","nonce":3758190298}]},"public_addr":"172.21.15.134:6812/3758190298","cluster_addr":"172.21.15.134:6813/3758190298","heartbeat_back_addr":"172.21.15.134:6815/3758190298","heartbeat_front_addr":"172.21.15.134:6814/3758190298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:49.151240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:13.079294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:27.871639+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:42.998495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:09.901960+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:29.301481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:53:55.240231+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:21.111874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2354623613":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/2838206829":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/1249955842":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1446120238":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/429160846":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/997592180":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/2767265670":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1590506004":"2024-03-31T14:49:57.293695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:54:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 10 librados: watch_flush enter 2024-03-30T14:54:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 10 librados: watch_flush exit 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 20 client.34319.objecter shutdown clearing up homeless session... 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 10 client.34319.objecter successfully canceled tick 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 >> v1:172.21.15.28:6800/3720192247 conn(0x7f91b8070490 legacy=0x7f91b8072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 >> v1:172.21.15.28:6790/0 conn(0x7f91d0107060 legacy=0x7f91d01010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 shutdown_connections 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 >> 172.21.15.28:0/1833388143 conn(0x7f91d00761d0 msgr2=0x7f91d0111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 shutdown_connections 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 -- 172.21.15.28:0/1833388143 wait complete. 2024-03-30T14:54:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:26.512+0000 7f91d5aa0700 1 librados: shutdown 2024-03-30T14:54:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:26 smithi028 ceph-mon[29876]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T14:54:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:26 smithi028 ceph-mon[41614]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T14:54:26.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:26 smithi134 ceph-mon[33714]: osdmap e64: 8 total, 8 up, 8 in 2024-03-30T14:54:27.140 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T14:54:27.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T14:54:27.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:54:27.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[29876]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T14:54:27.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1833388143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:54:27.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[29876]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:27.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[41614]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T14:54:27.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1833388143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:54:27.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:27 smithi028 ceph-mon[41614]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:27.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:27 smithi134 ceph-mon[33714]: osdmap e65: 8 total, 8 up, 8 in 2024-03-30T14:54:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1833388143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:54:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:27 smithi134 ceph-mon[33714]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 -- 172.21.15.28:0/602265880 >> v1:172.21.15.28:6789/0 conn(0x7f83fc103190 legacy=0x7f83fc103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 -- 172.21.15.28:0/602265880 shutdown_connections 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 -- 172.21.15.28:0/602265880 >> 172.21.15.28:0/602265880 conn(0x7f83fc0fc9e0 msgr2=0x7f83fc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 -- 172.21.15.28:0/602265880 shutdown_connections 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 -- 172.21.15.28:0/602265880 wait complete. 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 librados: starting msgr at 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.020+0000 7f8401ba0700 1 librados: starting objecter 2024-03-30T14:54:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 Processor -- start 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 -- start start 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 librados: setting wanted keys 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 librados: calling monclient init 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83fc110d80 con 0x7f83fc103190 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83fc102ae0 con 0x7f83fc10eab0 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f8401ba0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83fc102d10 con 0x7f83fc10afa0 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83fb7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f83fc103190 0x7f83fc100df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57846/0 (socket says 172.21.15.28:57846) 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83fb7fe700 1 -- 172.21.15.28:0/1559074653 learned_addr learned my addr 172.21.15.28:0/1559074653 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83faffd700 1 --1- 172.21.15.28:0/1559074653 >> v1:172.21.15.28:6790/0 conn(0x7f83fc10afa0 0x7f83fc101500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57668/0 (socket says 172.21.15.28:57668) 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter ms_handle_connect 0x7f83fc103190 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter ms_handle_connect 0x7f83fc10afa0 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter ms_handle_connect 0x7f83fc10eab0 2024-03-30T14:54:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603750042 0 0) 0x7f83fc102d10 con 0x7f83fc10afa0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e40031f0 con 0x7f83fc10afa0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029937676 0 0) 0x7f83fc110d80 con 0x7f83fc103190 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.021+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83fc102d10 con 0x7f83fc103190 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591164922 0 0) 0x7f83fc102ae0 con 0x7f83fc10eab0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83fc110d80 con 0x7f83fc10eab0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 850697025 0 0) 0x7f83e40031f0 con 0x7f83fc10afa0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83fc102ae0 con 0x7f83fc10afa0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1407935097 0 0) 0x7f83fc102d10 con 0x7f83fc103190 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83e40031f0 con 0x7f83fc103190 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83f00018e0 con 0x7f83fc10afa0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83ec004600 con 0x7f83fc103190 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1830817698 0 0) 0x7f83fc110d80 con 0x7f83fc10eab0 2024-03-30T14:54:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83fc102d10 con 0x7f83fc10eab0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83f4001af0 con 0x7f83fc10eab0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 508533421 0 0) 0x7f83fc102ae0 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 >> v1:172.21.15.134:6789/0 conn(0x7f83fc10eab0 legacy=0x7f83fc1a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 >> v1:172.21.15.28:6789/0 conn(0x7f83fc103190 legacy=0x7f83fc100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83fc1aca80 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83fc1a9a70 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f8401ba0700 10 client.34325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f83fc1a9f30 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.022+0000 7f8401ba0700 1 librados: init done 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.023+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f83f0003b20 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.023+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83f0005bf0 con 0x7f83fc10afa0 2024-03-30T14:54:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.023+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e8005120 con 0x7f83fc10afa0 2024-03-30T14:54:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f83f0005bf0 con 0x7f83fc10afa0 2024-03-30T14:54:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(65..65 src has 14..65) ==== 4081+0+0 (unknown 1242353675 0 0) 0x7f83f00883d0 con 0x7f83fc10afa0 2024-03-30T14:54:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 10 client.34325.objecter ms_dispatch 0x7f83fc001550 osd_map(65..65 src has 14..65) 2024-03-30T14:54:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 3 client.34325.objecter handle_osd_map got epochs [65,65] > 0 2024-03-30T14:54:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 3 client.34325.objecter handle_osd_map decoding full epoch 65 2024-03-30T14:54:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 20 client.34325.objecter dump_active .. 0 homeless 2024-03-30T14:54:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.024+0000 7f83f8ff9700 10 client.34325.objecter ms_handle_connect 0x7f83e4070390 2024-03-30T14:54:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.029+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f83f0056610 con 0x7f83fc10afa0 2024-03-30T14:54:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.230+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f83e8005c20 con 0x7f83fc10afa0 2024-03-30T14:54:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.231+0000 7f83f8ff9700 1 -- 172.21.15.28:0/1559074653 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) ==== 93+0+10 (unknown 3287015660 0 2170607528) 0x7f83f0055f10 con 0x7f83fc10afa0 2024-03-30T14:54:29.232 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.232+0000 7f8401ba0700 10 librados: watch_flush enter 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.232+0000 7f8401ba0700 10 librados: watch_flush exit 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.232+0000 7f8401ba0700 20 client.34325.objecter shutdown clearing up homeless session... 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.232+0000 7f8401ba0700 10 client.34325.objecter successfully canceled tick 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 >> v1:172.21.15.28:6800/3720192247 conn(0x7f83e4070390 legacy=0x7f83e4072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 >> v1:172.21.15.28:6790/0 conn(0x7f83fc10afa0 legacy=0x7f83fc101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 shutdown_connections 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 >> 172.21.15.28:0/1559074653 conn(0x7f83fc0fc9e0 msgr2=0x7f83fc0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 shutdown_connections 2024-03-30T14:54:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 -- 172.21.15.28:0/1559074653 wait complete. 2024-03-30T14:54:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:29.233+0000 7f8401ba0700 1 librados: shutdown 2024-03-30T14:54:29.555 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1559074653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T14:54:29.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1559074653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T14:54:29.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1559074653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T14:54:29.908 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi028 2024-03-30T14:54:29.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply rgw foo.a --placement '1;smithi028=foo.a' 2024-03-30T14:54:30.160 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:30.637 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:30 smithi134 ceph-mon[33714]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T14:54:30.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:30 smithi028 ceph-mon[29876]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T14:54:30.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:30 smithi028 ceph-mon[41614]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-30T14:54:31.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- 172.21.15.134:0/1952018471 >> v1:172.21.15.134:6789/0 conn(0x7f7a1010a1a0 legacy=0x7f7a1010c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:31.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- 172.21.15.134:0/1952018471 shutdown_connections 2024-03-30T14:54:31.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- 172.21.15.134:0/1952018471 >> 172.21.15.134:0/1952018471 conn(0x7f7a100fc500 msgr2=0x7f7a100fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- 172.21.15.134:0/1952018471 shutdown_connections 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- 172.21.15.134:0/1952018471 wait complete. 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 librados: starting msgr at 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 librados: starting objecter 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 Processor -- start 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 -- start start 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 librados: setting wanted keys 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.724+0000 7f7a1635c700 1 librados: calling monclient init 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a1635c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a101a49d0 con 0x7f7a1010a1a0 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a1635c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a101a5c10 con 0x7f7a10102b50 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a1635c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a101a6e50 con 0x7f7a10106660 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0ffff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7a10102b50 0x7f7a101023d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58974/0 (socket says 172.21.15.134:58974) 2024-03-30T14:54:31.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0ffff700 1 -- 172.21.15.134:0/3100526496 learned_addr learned my addr 172.21.15.134:0/3100526496 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a10102b50 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a10106660 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a1010a1a0 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978814237 0 0) 0x7f7a101a5c10 con 0x7f7a10102b50 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f80035b0 con 0x7f7a10102b50 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 662585816 0 0) 0x7f7a101a6e50 con 0x7f7a10106660 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a101a5c10 con 0x7f7a10106660 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1514828243 0 0) 0x7f7a101a49d0 con 0x7f7a1010a1a0 2024-03-30T14:54:31.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a101a6e50 con 0x7f7a1010a1a0 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1562100854 0 0) 0x7f79f80035b0 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a101a49d0 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.725+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a000018b0 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2779433206 0 0) 0x7f7a101a6e50 con 0x7f7a1010a1a0 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79f80035b0 con 0x7f7a1010a1a0 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4236138704 0 0) 0x7f7a101a49d0 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 >> v1:172.21.15.28:6790/0 conn(0x7f7a10106660 legacy=0x7f7a1019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 >> v1:172.21.15.28:6789/0 conn(0x7f7a1010a1a0 legacy=0x7f7a101a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a101a8090 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a101a7080 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a1635c700 10 client.24355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a101a75c0 con 0x7f7a10102b50 2024-03-30T14:54:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a1635c700 1 librados: init done 2024-03-30T14:54:31.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7a00001ca0 con 0x7f7a10102b50 2024-03-30T14:54:31.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.726+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a00005d00 con 0x7f7a10102b50 2024-03-30T14:54:31.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.727+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79fc005150 con 0x7f7a10102b50 2024-03-30T14:54:31.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.727+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f7a000039d0 con 0x7f7a10102b50 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(66..66 src has 14..66) ==== 4081+0+0 (unknown 3995989769 0 0) 0x7f7a00088510 con 0x7f7a10102b50 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 10 client.24355.objecter ms_dispatch 0x7f7a10001550 osd_map(66..66 src has 14..66) 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 3 client.24355.objecter handle_osd_map got epochs [66,66] > 0 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 3 client.24355.objecter handle_osd_map decoding full epoch 66 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 20 client.24355.objecter dump_active .. 0 homeless 2024-03-30T14:54:31.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.728+0000 7f7a0d7fa700 10 client.24355.objecter ms_handle_connect 0x7f79f8070000 2024-03-30T14:54:31.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.732+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7a00056830 con 0x7f7a10102b50 2024-03-30T14:54:31.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.939+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}) -- 0x7f79fc000bc0 con 0x7f79f8070000 2024-03-30T14:54:31.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.946+0000 7f7a0d7fa700 1 -- 172.21.15.134:0/3100526496 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (unknown 0 0 1123153589) 0x7f79fc000bc0 con 0x7f79f8070000 2024-03-30T14:54:31.947 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled rgw.foo.a update... 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 10 librados: watch_flush enter 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 10 librados: watch_flush exit 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 20 client.24355.objecter shutdown clearing up homeless session... 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 10 client.24355.objecter successfully canceled tick 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 >> v1:172.21.15.28:6800/3720192247 conn(0x7f79f8070000 legacy=0x7f79f80724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 >> v1:172.21.15.134:6789/0 conn(0x7f7a10102b50 legacy=0x7f7a101023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 shutdown_connections 2024-03-30T14:54:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 >> 172.21.15.134:0/3100526496 conn(0x7f7a100fc500 msgr2=0x7f7a100fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 shutdown_connections 2024-03-30T14:54:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 -- 172.21.15.134:0/3100526496 wait complete. 2024-03-30T14:54:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:31.948+0000 7f7a1635c700 1 librados: shutdown 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:32.547 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:32 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:32.574 DEBUG:teuthology.orchestra.run.smithi028:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@rgw.foo.a.service 2024-03-30T14:54:32.576 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi134 2024-03-30T14:54:32.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool create datapool 3 3 replicated 2024-03-30T14:54:32.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:54:32.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: osdmap e66: 8 total, 8 up, 8 in 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:32.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:32 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:32.814 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:33.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='client.24355 v1:172.21.15.134:0/3100526496' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:33.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:33.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='client.24355 v1:172.21.15.134:0/3100526496' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:33.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:33.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:54:33.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:54:33.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:33 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:33.627 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='client.24355 v1:172.21.15.134:0/3100526496' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:33.627 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:33.627 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:33.628 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:33 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.345+0000 7f7b1902b700 1 -- 172.21.15.134:0/3981562949 >> v1:172.21.15.134:6789/0 conn(0x7f7b14100330 legacy=0x7f7b14100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.345+0000 7f7b1902b700 1 -- 172.21.15.134:0/3981562949 shutdown_connections 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.345+0000 7f7b1902b700 1 -- 172.21.15.134:0/3981562949 >> 172.21.15.134:0/3981562949 conn(0x7f7b140fbcb0 msgr2=0x7f7b140fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 -- 172.21.15.134:0/3981562949 shutdown_connections 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 -- 172.21.15.134:0/3981562949 wait complete. 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 librados: starting msgr at 2024-03-30T14:54:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 librados: starting objecter 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 Processor -- start 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 -- start start 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 librados: setting wanted keys 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.346+0000 7f7b1902b700 1 librados: calling monclient init 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7b1902b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b141a49d0 con 0x7f7b1410a1a0 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7b1902b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b141a5c10 con 0x7f7b14106540 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7b1902b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b141a6e50 con 0x7f7b14100330 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7b1259c700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f7b14106540 0x7f7b1419f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34366/0 (socket says 172.21.15.134:34366) 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7b1259c700 1 -- 172.21.15.134:0/3499785024 learned_addr learned my addr 172.21.15.134:0/3499785024 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter ms_handle_connect 0x7f7b14106540 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter ms_handle_connect 0x7f7b1410a1a0 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:34.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter ms_handle_connect 0x7f7b14100330 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639854252 0 0) 0x7f7b141a5c10 con 0x7f7b14106540 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7afc0035b0 con 0x7f7b14106540 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 335006136 0 0) 0x7f7b141a49d0 con 0x7f7b1410a1a0 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b141a5c10 con 0x7f7b1410a1a0 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497140225 0 0) 0x7f7b141a6e50 con 0x7f7b14100330 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b141a49d0 con 0x7f7b14100330 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4200190465 0 0) 0x7f7afc0035b0 con 0x7f7b14106540 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b141a6e50 con 0x7f7b14106540 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7b080018b0 con 0x7f7b14106540 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1660991755 0 0) 0x7f7b141a5c10 con 0x7f7b1410a1a0 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7afc0035b0 con 0x7f7b1410a1a0 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7b0c001b70 con 0x7f7b1410a1a0 2024-03-30T14:54:34.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2819084615 0 0) 0x7f7b141a49d0 con 0x7f7b14100330 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b141a5c10 con 0x7f7b14100330 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7b04001430 con 0x7f7b14100330 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4180834094 0 0) 0x7f7b141a6e50 con 0x7f7b14106540 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 >> v1:172.21.15.28:6790/0 conn(0x7f7b14100330 legacy=0x7f7b14102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 >> v1:172.21.15.28:6789/0 conn(0x7f7b1410a1a0 legacy=0x7f7b141a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.347+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b141a8090 con 0x7f7b14106540 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b141a4c00 con 0x7f7b14106540 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7b1902b700 10 client.24361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b141a5140 con 0x7f7b14106540 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7b1902b700 1 librados: init done 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7b08003b40 con 0x7f7b14106540 2024-03-30T14:54:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.348+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7b08005cc0 con 0x7f7b14106540 2024-03-30T14:54:34.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b00005120 con 0x7f7b14106540 2024-03-30T14:54:34.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7f7b080036a0 con 0x7f7b14106540 2024-03-30T14:54:34.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(66..66 src has 14..66) ==== 4081+0+0 (unknown 3995989769 0 0) 0x7f7b08088490 con 0x7f7b14106540 2024-03-30T14:54:34.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7afbfff700 10 client.24361.objecter ms_dispatch 0x7f7b14001550 osd_map(66..66 src has 14..66) 2024-03-30T14:54:34.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7afbfff700 3 client.24361.objecter handle_osd_map got epochs [66,66] > 0 2024-03-30T14:54:34.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.349+0000 7f7afbfff700 3 client.24361.objecter handle_osd_map decoding full epoch 66 2024-03-30T14:54:34.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.350+0000 7f7afbfff700 20 client.24361.objecter dump_active .. 0 homeless 2024-03-30T14:54:34.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.350+0000 7f7afbfff700 10 client.24361.objecter ms_handle_connect 0x7f7afc070360 2024-03-30T14:54:34.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.354+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7b080567b0 con 0x7f7b14106540 2024-03-30T14:54:34.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:34.548+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f7b00005410 con 0x7f7b14106540 2024-03-30T14:54:34.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:34 smithi028 ceph-mon[29876]: Deploying daemon rgw.foo.a on smithi028 2024-03-30T14:54:34.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:34 smithi028 ceph-mon[29876]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:54:34.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:34 smithi028 ceph-mon[41614]: Deploying daemon rgw.foo.a on smithi028 2024-03-30T14:54:34.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:34 smithi028 ceph-mon[41614]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:54:34.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:34 smithi134 ceph-mon[33714]: Deploying daemon rgw.foo.a on smithi028 2024-03-30T14:54:34.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:34 smithi134 ceph-mon[33714]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-30T14:54:34.972 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:34 smithi028 systemd[1]: Starting Ceph rgw.foo.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:54:35.300 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 podman[70114]: 2024-03-30 14:54:34.92864976 +0000 UTC m=+0.017978165 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:54:35.300 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 podman[70114]: 2024-03-30 14:54:35.048163543 +0000 UTC m=+0.137491932 container create 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-30T14:54:35.300 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 podman[70114]: 2024-03-30 14:54:35.267024065 +0000 UTC m=+0.356352458 container init 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, ceph=True) 2024-03-30T14:54:35.300 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 podman[70114]: 2024-03-30 14:54:35.272449584 +0000 UTC m=+0.361777975 container start 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-30T14:54:35.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.305+0000 7f7afbfff700 1 -- 172.21.15.134:0/3499785024 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v67) ==== 160+0+0 (unknown 3174374627 0 0) 0x7f7b080560b0 con 0x7f7b14106540 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 10 librados: watch_flush enter 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 10 librados: watch_flush exit 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 20 client.24361.objecter shutdown clearing up homeless session... 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 10 client.24361.objecter successfully canceled tick 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 >> v1:172.21.15.28:6800/3720192247 conn(0x7f7afc070360 legacy=0x7f7afc072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 >> v1:172.21.15.134:6789/0 conn(0x7f7b14106540 legacy=0x7f7b1419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:35.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 shutdown_connections 2024-03-30T14:54:35.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 >> 172.21.15.134:0/3499785024 conn(0x7f7b140fbcb0 msgr2=0x7f7b140fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:35.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 shutdown_connections 2024-03-30T14:54:35.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 -- 172.21.15.134:0/3499785024 wait complete. 2024-03-30T14:54:35.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:35.308+0000 7f7b1902b700 1 librados: shutdown 2024-03-30T14:54:35.310 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'datapool' created 2024-03-30T14:54:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/3499785024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:35 smithi134 ceph-mon[33714]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/3499785024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:35 smithi028 ceph-mon[41614]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/3499785024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:35 smithi028 ceph-mon[29876]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-30T14:54:35.723 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 bash[70114]: 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 2024-03-30T14:54:35.723 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 14:54:35 smithi028 systemd[1]: Started Ceph rgw.foo.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:54:35.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- rbd pool init datapool 2024-03-30T14:54:36.194 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:36.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T14:54:36.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T14:54:36.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: pgmap v196: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T14:54:36.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.554 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.554 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.554 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.554 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:36 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: pgmap v196: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: osdmap e67: 8 total, 8 up, 8 in 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: pgmap v196: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:36.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:36 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4086439972' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:37 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:54:37.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4086439972' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: osdmap e68: 8 total, 8 up, 8 in 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4086439972' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-30T14:54:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:37 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:54:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: Checking dashboard <-> RGW credentials 2024-03-30T14:54:38.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: pgmap v199: 36 pgs: 6 creating+peering, 2 active+clean, 28 unknown; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:38.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/137058518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:38.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:38 smithi134 ceph-mon[33714]: from='client.24400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: Checking dashboard <-> RGW credentials 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: pgmap v199: 36 pgs: 6 creating+peering, 2 active+clean, 28 unknown; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/137058518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[29876]: from='client.24400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: osdmap e69: 8 total, 8 up, 8 in 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: Checking dashboard <-> RGW credentials 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: pgmap v199: 36 pgs: 6 creating+peering, 2 active+clean, 28 unknown; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:54:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:38.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/137058518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:38.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:38 smithi028 ceph-mon[41614]: from='client.24400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-30T14:54:39.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:39 smithi134 ceph-mon[33714]: from='client.24400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T14:54:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:39 smithi134 ceph-mon[33714]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T14:54:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:39 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[29876]: from='client.24400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T14:54:39.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[29876]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T14:54:39.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[41614]: from='client.24400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-30T14:54:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[41614]: osdmap e70: 8 total, 8 up, 8 in 2024-03-30T14:54:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:39.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:39 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-30T14:54:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:40 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:40 smithi134 ceph-mon[33714]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T14:54:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:40 smithi134 ceph-mon[33714]: pgmap v202: 68 pgs: 1 creating+activating, 11 creating+peering, 16 active+clean, 40 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:54:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[29876]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T14:54:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[29876]: pgmap v202: 68 pgs: 1 creating+activating, 11 creating+peering, 16 active+clean, 40 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:54:40.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-30T14:54:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[41614]: osdmap e71: 8 total, 8 up, 8 in 2024-03-30T14:54:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:40 smithi028 ceph-mon[41614]: pgmap v202: 68 pgs: 1 creating+activating, 11 creating+peering, 16 active+clean, 40 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:54:41.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.134 --placement '1;smithi134=iscsi.a' 2024-03-30T14:54:41.321 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:41.349 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: Cluster is now healthy 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.601 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:41 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: Cluster is now healthy 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: Cluster is now healthy 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: osdmap e72: 8 total, 8 up, 8 in 2024-03-30T14:54:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:41.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:41 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-30T14:54:42.478 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.478 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.478 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:54:42.478 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: pgmap v205: 100 pgs: 1 creating+activating, 20 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-30T14:54:42.479 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T14:54:42.479 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:42.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:54:42.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: pgmap v205: 100 pgs: 1 creating+activating, 20 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: osdmap e73: 8 total, 8 up, 8 in 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: pgmap v205: 100 pgs: 1 creating+activating, 20 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: osdmap e74: 8 total, 8 up, 8 in 2024-03-30T14:54:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:42.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 -- 172.21.15.134:0/1666161842 >> v1:172.21.15.134:6789/0 conn(0x7efc48105350 legacy=0x7efc48107740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 -- 172.21.15.134:0/1666161842 shutdown_connections 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 -- 172.21.15.134:0/1666161842 >> 172.21.15.134:0/1666161842 conn(0x7efc480fc4d0 msgr2=0x7efc480fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 -- 172.21.15.134:0/1666161842 shutdown_connections 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 -- 172.21.15.134:0/1666161842 wait complete. 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 librados: starting msgr at 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.951+0000 7efc4c95a700 1 librados: starting objecter 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 Processor -- start 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 -- start start 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 librados: setting wanted keys 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 librados: calling monclient init 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc481970e0 con 0x7efc48105350 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc481a28d0 con 0x7efc48108fe0 2024-03-30T14:54:42.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc4c95a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc481a3ab0 con 0x7efc48100f30 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc46d9d700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7efc48108fe0 0x7efc481a01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:36136/0 (socket says 172.21.15.134:36136) 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc46d9d700 1 -- 172.21.15.134:0/4122842432 learned_addr learned my addr 172.21.15.134:0/4122842432 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter ms_handle_connect 0x7efc48108fe0 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter ms_handle_connect 0x7efc48100f30 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter ms_handle_connect 0x7efc48105350 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590231047 0 0) 0x7efc481a28d0 con 0x7efc48108fe0 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc300035b0 con 0x7efc48108fe0 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 383407734 0 0) 0x7efc481a3ab0 con 0x7efc48100f30 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc481a28d0 con 0x7efc48100f30 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2056484089 0 0) 0x7efc481970e0 con 0x7efc48105350 2024-03-30T14:54:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc481a3ab0 con 0x7efc48105350 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4081511855 0 0) 0x7efc300035b0 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.952+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc481970e0 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc40001c90 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 200270798 0 0) 0x7efc481a28d0 con 0x7efc48100f30 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc300035b0 con 0x7efc48100f30 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3907906386 0 0) 0x7efc481a3ab0 con 0x7efc48105350 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc481a28d0 con 0x7efc48105350 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc3c001430 con 0x7efc48105350 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc380018b0 con 0x7efc48100f30 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 280982147 0 0) 0x7efc481970e0 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 >> v1:172.21.15.28:6790/0 conn(0x7efc48100f30 legacy=0x7efc48196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 >> v1:172.21.15.28:6789/0 conn(0x7efc48105350 legacy=0x7efc4819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc481a4cd0 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc481a3ce0 con 0x7efc48108fe0 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc4c95a700 10 client.24421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efc481a4160 con 0x7efc48108fe0 2024-03-30T14:54:42.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc4c95a700 1 librados: init done 2024-03-30T14:54:42.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7efc40003ec0 con 0x7efc48108fe0 2024-03-30T14:54:42.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.953+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc40006090 con 0x7efc48108fe0 2024-03-30T14:54:42.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.954+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc28005150 con 0x7efc48108fe0 2024-03-30T14:54:42.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.954+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7efc40003a20 con 0x7efc48108fe0 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(74..74 src has 14..74) ==== 5915+0+0 (unknown 328557178 0 0) 0x7efc40089050 con 0x7efc48108fe0 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 10 client.24421.objecter ms_dispatch 0x7efc48001550 osd_map(74..74 src has 14..74) 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 3 client.24421.objecter handle_osd_map got epochs [74,74] > 0 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 3 client.24421.objecter handle_osd_map decoding full epoch 74 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 20 client.24421.objecter dump_active .. 0 homeless 2024-03-30T14:54:42.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.955+0000 7efc377fe700 10 client.24421.objecter ms_handle_connect 0x7efc300703e0 2024-03-30T14:54:42.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:42.959+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efc40056b10 con 0x7efc48108fe0 2024-03-30T14:54:43.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.167+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.134", "placement": "1;smithi134=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7efc28000c90 con 0x7efc300703e0 2024-03-30T14:54:43.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.174+0000 7efc377fe700 1 -- 172.21.15.134:0/4122842432 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (unknown 0 0 803663096) 0x7efc28000c90 con 0x7efc300703e0 2024-03-30T14:54:43.175 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled iscsi.datapool update... 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 10 librados: watch_flush enter 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 10 librados: watch_flush exit 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 20 client.24421.objecter shutdown clearing up homeless session... 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 10 client.24421.objecter successfully canceled tick 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 >> v1:172.21.15.28:6800/3720192247 conn(0x7efc300703e0 legacy=0x7efc300728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 >> v1:172.21.15.134:6789/0 conn(0x7efc48108fe0 legacy=0x7efc481a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 shutdown_connections 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 >> 172.21.15.134:0/4122842432 conn(0x7efc480fc4d0 msgr2=0x7efc480fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 shutdown_connections 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 -- 172.21.15.134:0/4122842432 wait complete. 2024-03-30T14:54:43.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:43.176+0000 7efc4c95a700 1 librados: shutdown 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-30T14:54:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: osdmap e75: 8 total, 8 up, 8 in 2024-03-30T14:54:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2248569927' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-30T14:54:43.815 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-30T14:54:43.815 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:54:43.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T14:54:43.851 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:54:43.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-30T14:54:43.882 DEBUG:teuthology.orchestra.run.smithi134:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service 2024-03-30T14:54:43.924 INFO:tasks.cephadm:Adding prometheus.a on smithi134 2024-03-30T14:54:43.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi134=a' 2024-03-30T14:54:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: from='client.24421 v1:172.21.15.134:0/4122842432' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.134", "placement": "1;smithi134=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: Saving service iscsi.datapool spec with placement smithi134=iscsi.a;count:1 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: pgmap v208: 132 pgs: 20 creating+peering, 79 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.622 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:44 smithi134 ceph-mon[33714]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T14:54:44.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: from='client.24421 v1:172.21.15.134:0/4122842432' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.134", "placement": "1;smithi134=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:44.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: Saving service iscsi.datapool spec with placement smithi134=iscsi.a;count:1 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: pgmap v208: 132 pgs: 20 creating+peering, 79 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[29876]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: from='client.24421 v1:172.21.15.134:0/4122842432' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.134", "placement": "1;smithi134=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: Saving service iscsi.datapool spec with placement smithi134=iscsi.a;count:1 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: pgmap v208: 132 pgs: 20 creating+peering, 79 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: from='client.24394 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2375168254' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-30T14:54:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:44 smithi028 ceph-mon[41614]: osdmap e76: 8 total, 8 up, 8 in 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: Deploying daemon iscsi.iscsi.a on smithi134 2024-03-30T14:54:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:45 smithi134 ceph-mon[33714]: pgmap v210: 132 pgs: 15 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 3.5 KiB/s wr, 44 op/s 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.915+0000 7fccede2f700 1 -- 172.21.15.134:0/2923505440 >> v1:172.21.15.134:6789/0 conn(0x7fcce810a1c0 legacy=0x7fcce810c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.915+0000 7fccede2f700 1 -- 172.21.15.134:0/2923505440 shutdown_connections 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.915+0000 7fccede2f700 1 -- 172.21.15.134:0/2923505440 >> 172.21.15.134:0/2923505440 conn(0x7fcce80fbcb0 msgr2=0x7fcce80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.915+0000 7fccede2f700 1 -- 172.21.15.134:0/2923505440 shutdown_connections 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.915+0000 7fccede2f700 1 -- 172.21.15.134:0/2923505440 wait complete. 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 librados: starting msgr at 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 librados: starting objecter 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 Processor -- start 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 -- start start 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 librados: setting wanted keys 2024-03-30T14:54:45.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 librados: calling monclient init 2024-03-30T14:54:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcce8109d70 con 0x7fcce810a1c0 2024-03-30T14:54:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcce81a5ca0 con 0x7fcce8106620 2024-03-30T14:54:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fccede2f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcce81a6e80 con 0x7fcce8103650 2024-03-30T14:54:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fcce6ffd700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcce8106620 0x7fcce819fe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:36160/0 (socket says 172.21.15.134:36160) 2024-03-30T14:54:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fcce6ffd700 1 -- 172.21.15.134:0/1280013101 learned_addr learned my addr 172.21.15.134:0/1280013101 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fcce4ff9700 10 client.?.objecter ms_handle_connect 0x7fcce8106620 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.916+0000 7fcce4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 10 client.?.objecter ms_handle_connect 0x7fcce8103650 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 10 client.?.objecter ms_handle_connect 0x7fcce810a1c0 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653820801 0 0) 0x7fcce81a5ca0 con 0x7fcce8106620 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd00035b0 con 0x7fcce8106620 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608112472 0 0) 0x7fcce81a6e80 con 0x7fcce8103650 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcce81a5ca0 con 0x7fcce8103650 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298667567 0 0) 0x7fcce8109d70 con 0x7fcce810a1c0 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcce81a6e80 con 0x7fcce810a1c0 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115327499 0 0) 0x7fccd00035b0 con 0x7fcce8106620 2024-03-30T14:54:45.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcce8109d70 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccd8001930 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4292650629 0 0) 0x7fcce81a5ca0 con 0x7fcce8103650 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccd00035b0 con 0x7fcce8103650 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccdc0017e0 con 0x7fcce8103650 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3626279421 0 0) 0x7fcce8109d70 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 >> v1:172.21.15.28:6790/0 conn(0x7fcce8103650 legacy=0x7fcce8109220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 >> v1:172.21.15.28:6789/0 conn(0x7fcce810a1c0 legacy=0x7fcce81a3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce81a80a0 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcce81a70b0 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fccede2f700 10 client.24442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcce81a75f0 con 0x7fcce8106620 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fccede2f700 1 librados: init done 2024-03-30T14:54:45.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fccd8003b40 con 0x7fcce8106620 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.917+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccd8005d30 con 0x7fcce8106620 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.918+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd4005150 con 0x7fcce8106620 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7fccd80036a0 con 0x7fcce8106620 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(76..76 src has 14..76) ==== 5942+0+0 (unknown 1854185047 0 0) 0x7fccd8088c30 con 0x7fcce8106620 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 10 client.24442.objecter ms_dispatch 0x7fcce8001550 osd_map(76..76 src has 14..76) 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 3 client.24442.objecter handle_osd_map got epochs [76,76] > 0 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 3 client.24442.objecter handle_osd_map decoding full epoch 76 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.919+0000 7fcce4ff9700 20 client.24442.objecter dump_active .. 0 homeless 2024-03-30T14:54:45.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.920+0000 7fcce4ff9700 10 client.24442.objecter ms_handle_connect 0x7fccd0070030 2024-03-30T14:54:45.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:45.924+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fccd8056810 con 0x7fcce8106620 2024-03-30T14:54:45.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: Deploying daemon iscsi.iscsi.a on smithi134 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[29876]: pgmap v210: 132 pgs: 15 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 3.5 KiB/s wr, 44 op/s 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-30T14:54:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:45.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: Deploying daemon iscsi.iscsi.a on smithi134 2024-03-30T14:54:45.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:45 smithi028 ceph-mon[41614]: pgmap v210: 132 pgs: 15 creating+peering, 99 active+clean, 18 unknown; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 3.5 KiB/s wr, 44 op/s 2024-03-30T14:54:46.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.143+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}) -- 0x7fccd4000bc0 con 0x7fccd0070030 2024-03-30T14:54:46.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.151+0000 7fcce4ff9700 1 -- 172.21.15.134:0/1280013101 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (unknown 0 0 1342662408) 0x7fccd4000bc0 con 0x7fccd0070030 2024-03-30T14:54:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled prometheus update... 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 10 librados: watch_flush enter 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 10 librados: watch_flush exit 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 20 client.24442.objecter shutdown clearing up homeless session... 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 10 client.24442.objecter successfully canceled tick 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 >> v1:172.21.15.28:6800/3720192247 conn(0x7fccd0070030 legacy=0x7fccd00724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 >> v1:172.21.15.134:6789/0 conn(0x7fcce8106620 legacy=0x7fcce819fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 shutdown_connections 2024-03-30T14:54:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 >> 172.21.15.134:0/1280013101 conn(0x7fcce80fbcb0 msgr2=0x7fcce810cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 shutdown_connections 2024-03-30T14:54:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 -- 172.21.15.134:0/1280013101 wait complete. 2024-03-30T14:54:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:46.153+0000 7fccede2f700 1 librados: shutdown 2024-03-30T14:54:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:47 smithi134 ceph-mon[33714]: from='client.24442 v1:172.21.15.134:0/1280013101' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:47 smithi134 ceph-mon[33714]: Saving service prometheus spec with placement smithi134=a;count:1 2024-03-30T14:54:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:47 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:47 smithi134 ceph-mon[33714]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T14:54:47.228 DEBUG:teuthology.orchestra.run.smithi134:prometheus.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@prometheus.a.service 2024-03-30T14:54:47.231 INFO:tasks.cephadm:Adding node-exporter.a on smithi028 2024-03-30T14:54:47.231 INFO:tasks.cephadm:Adding node-exporter.b on smithi134 2024-03-30T14:54:47.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi028=a;smithi134=b' 2024-03-30T14:54:47.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[29876]: from='client.24442 v1:172.21.15.134:0/1280013101' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:47.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[29876]: Saving service prometheus spec with placement smithi134=a;count:1 2024-03-30T14:54:47.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[29876]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T14:54:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[41614]: from='client.24442 v1:172.21.15.134:0/1280013101' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[41614]: Saving service prometheus spec with placement smithi134=a;count:1 2024-03-30T14:54:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:47 smithi028 ceph-mon[41614]: osdmap e77: 8 total, 8 up, 8 in 2024-03-30T14:54:47.692 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:48.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:48 smithi134 ceph-mon[33714]: pgmap v212: 132 pgs: 6 creating+peering, 126 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 6.0 KiB/s wr, 156 op/s 2024-03-30T14:54:48.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:48 smithi134 systemd[1]: Starting Ceph iscsi.iscsi.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:54:48.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:48 smithi028 ceph-mon[29876]: pgmap v212: 132 pgs: 6 creating+peering, 126 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 6.0 KiB/s wr, 156 op/s 2024-03-30T14:54:48.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:48 smithi028 ceph-mon[41614]: pgmap v212: 132 pgs: 6 creating+peering, 126 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 6.0 KiB/s wr, 156 op/s 2024-03-30T14:54:49.349 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 podman[62598]: 2024-03-30 14:54:49.12419491 +0000 UTC m=+0.248266543 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T14:54:49.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 podman[62598]: 2024-03-30 14:54:49.307973542 +0000 UTC m=+0.432045147 container create 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-30T14:54:49.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 -- 172.21.15.134:0/1746498485 >> v1:172.21.15.134:6789/0 conn(0x7fbdb01008b0 legacy=0x7fbdb0100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 -- 172.21.15.134:0/1746498485 shutdown_connections 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 -- 172.21.15.134:0/1746498485 >> 172.21.15.134:0/1746498485 conn(0x7fbdb00fc4d0 msgr2=0x7fbdb00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 -- 172.21.15.134:0/1746498485 shutdown_connections 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 -- 172.21.15.134:0/1746498485 wait complete. 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 librados: starting msgr at 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.348+0000 7fbdb7450700 1 librados: starting objecter 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 Processor -- start 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 -- start start 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 librados: setting wanted keys 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 librados: calling monclient init 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbdb019a480 con 0x7fbdb01008b0 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbdb01a5c20 con 0x7fbdb01023f0 2024-03-30T14:54:49.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb7450700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbdb01a6e00 con 0x7fbdb010c610 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb49eb700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbdb01023f0 0x7fbdb019fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:36166/0 (socket says 172.21.15.134:36166) 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbdb49eb700 1 -- 172.21.15.134:0/2442868428 learned_addr learned my addr 172.21.15.134:0/2442868428 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbda67fc700 10 client.?.objecter ms_handle_connect 0x7fbdb01023f0 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.349+0000 7fbda67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 10 client.?.objecter ms_handle_connect 0x7fbdb010c610 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247683374 0 0) 0x7fbdb01a5c20 con 0x7fbdb01023f0 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd9c0035b0 con 0x7fbdb01023f0 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810992465 0 0) 0x7fbdb01a6e00 con 0x7fbdb010c610 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdb01a5c20 con 0x7fbdb010c610 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 10 client.?.objecter ms_handle_connect 0x7fbdb01008b0 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3681619878 0 0) 0x7fbd9c0035b0 con 0x7fbdb01023f0 2024-03-30T14:54:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdb01a6e00 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbda0001930 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197841122 0 0) 0x7fbdb019a480 con 0x7fbdb01008b0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd9c0035b0 con 0x7fbdb01008b0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3430200694 0 0) 0x7fbdb01a5c20 con 0x7fbdb010c610 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdb019a480 con 0x7fbdb010c610 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbda8001a60 con 0x7fbdb010c610 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1327277340 0 0) 0x7fbdb01a6e00 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 >> v1:172.21.15.28:6790/0 conn(0x7fbdb010c610 legacy=0x7fbdb01a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 >> v1:172.21.15.28:6789/0 conn(0x7fbdb01008b0 legacy=0x7fbdb0199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.350+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb01a8020 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb01a7030 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbdb7450700 10 client.24448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdb01a7570 con 0x7fbdb01023f0 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbdb7450700 1 librados: init done 2024-03-30T14:54:49.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbda0001cc0 con 0x7fbdb01023f0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.351+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbda0005dc0 con 0x7fbdb01023f0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.352+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd94005150 con 0x7fbdb01023f0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.352+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 15) ==== 91739+0+0 (unknown 3815050452 0 0) 0x7fbda0004160 con 0x7fbdb01023f0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.353+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(77..77 src has 14..77) ==== 5942+0+0 (unknown 1497819381 0 0) 0x7fbda0088e80 con 0x7fbdb01023f0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.353+0000 7fbda67fc700 10 client.24448.objecter ms_dispatch 0x7fbdb0001550 osd_map(77..77 src has 14..77) 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.353+0000 7fbda67fc700 3 client.24448.objecter handle_osd_map got epochs [77,77] > 0 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.353+0000 7fbda67fc700 3 client.24448.objecter handle_osd_map decoding full epoch 77 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.353+0000 7fbda67fc700 20 client.24448.objecter dump_active .. 0 homeless 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.354+0000 7fbda67fc700 10 client.24448.objecter ms_handle_connect 0x7fbd9c070290 2024-03-30T14:54:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.358+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbda0056920 con 0x7fbdb01023f0 2024-03-30T14:54:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.580+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi134=b", "target": ["mon-mgr", ""]}) -- 0x7fbd94000bc0 con 0x7fbd9c070290 2024-03-30T14:54:49.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.588+0000 7fbda67fc700 1 -- 172.21.15.134:0/2442868428 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (unknown 0 0 240551134) 0x7fbd94000bc0 con 0x7fbd9c070290 2024-03-30T14:54:49.589 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled node-exporter update... 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 10 librados: watch_flush enter 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 10 librados: watch_flush exit 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 20 client.24448.objecter shutdown clearing up homeless session... 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 10 client.24448.objecter successfully canceled tick 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 >> v1:172.21.15.28:6800/3720192247 conn(0x7fbd9c070290 legacy=0x7fbd9c072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 >> v1:172.21.15.134:6789/0 conn(0x7fbdb01023f0 legacy=0x7fbdb019fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 shutdown_connections 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 >> 172.21.15.134:0/2442868428 conn(0x7fbdb00fc4d0 msgr2=0x7fbdb00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 shutdown_connections 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 -- 172.21.15.134:0/2442868428 wait complete. 2024-03-30T14:54:49.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:49.591+0000 7fbdb7450700 1 librados: shutdown 2024-03-30T14:54:49.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:49 smithi134 ceph-mon[33714]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.8 KiB/s wr, 175 op/s 2024-03-30T14:54:49.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:49 smithi028 ceph-mon[41614]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.8 KiB/s wr, 175 op/s 2024-03-30T14:54:49.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:49 smithi028 ceph-mon[29876]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.8 KiB/s wr, 175 op/s 2024-03-30T14:54:49.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 podman[62598]: 2024-03-30 14:54:49.612166432 +0000 UTC m=+0.736238051 container init 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T14:54:49.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 podman[62598]: 2024-03-30 14:54:49.617904325 +0000 UTC m=+0.741975943 container start 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-03-30T14:54:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Started the configuration object watcher 2024-03-30T14:54:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Checking for config object changes every 1s 2024-03-30T14:54:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Processing osd blocklist entries for this node 2024-03-30T14:54:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 bash[62598]: 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 2024-03-30T14:54:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:49 smithi134 systemd[1]: Started Ceph iscsi.iscsi.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:54:50.329 DEBUG:teuthology.orchestra.run.smithi028:node-exporter.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.a.service 2024-03-30T14:54:50.333 DEBUG:teuthology.orchestra.run.smithi134:node-exporter.b> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.b.service 2024-03-30T14:54:50.335 INFO:tasks.cephadm:Adding alertmanager.a on smithi028 2024-03-30T14:54:50.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi028=a' 2024-03-30T14:54:50.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Reading the configuration object to update local LIO configuration 2024-03-30T14:54:50.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Configuration does not have an entry for this host(smithi134.front.sepia.ceph.com) - nothing to define to LIO 2024-03-30T14:54:50.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-30T14:54:50.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='client.24448 v1:172.21.15.134:0/2442868428' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi134=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: Saving service node-exporter spec with placement smithi028=a;smithi134=b;count:2 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/41723654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T14:54:50.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='client.24448 v1:172.21.15.134:0/2442868428' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi134=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:50.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: Saving service node-exporter spec with placement smithi028=a;smithi134=b;count:2 2024-03-30T14:54:50.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T14:54:50.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:50.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/41723654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T14:54:50.863 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='client.24448 v1:172.21.15.134:0/2442868428' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi134=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:50.863 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: Saving service node-exporter spec with placement smithi028=a;smithi134=b;count:2 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:54:50.864 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/41723654' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T14:54:50.908 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:52.309 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:52 smithi134 ceph-mon[33714]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-30T14:54:52.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:52 smithi028 ceph-mon[29876]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-30T14:54:52.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:52 smithi028 ceph-mon[41614]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 139 op/s 2024-03-30T14:54:52.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.922+0000 7f03c50c3700 1 -- 172.21.15.134:0/3887113308 >> v1:172.21.15.134:6789/0 conn(0x7f03c0105350 legacy=0x7f03c0107740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.922+0000 7f03c50c3700 1 -- 172.21.15.134:0/3887113308 shutdown_connections 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.922+0000 7f03c50c3700 1 -- 172.21.15.134:0/3887113308 >> 172.21.15.134:0/3887113308 conn(0x7f03c00fc4d0 msgr2=0x7f03c00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.922+0000 7f03c50c3700 1 -- 172.21.15.134:0/3887113308 shutdown_connections 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.922+0000 7f03c50c3700 1 -- 172.21.15.134:0/3887113308 wait complete. 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 librados: starting msgr at 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 librados: starting objecter 2024-03-30T14:54:52.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 Processor -- start 2024-03-30T14:54:52.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 -- start start 2024-03-30T14:54:52.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 librados: setting wanted keys 2024-03-30T14:54:52.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 librados: calling monclient init 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03c0197100 con 0x7f03c0105350 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03c01a28f0 con 0x7f03c0108fe0 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.923+0000 7f03c50c3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03c01a3ad0 con 0x7f03c0100f30 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03bf59e700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f03c0108fe0 0x7f03c01a01c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49466/0 (socket says 172.21.15.134:49466) 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03bf59e700 1 -- 172.21.15.134:0/2648116772 learned_addr learned my addr 172.21.15.134:0/2648116772 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter ms_handle_connect 0x7f03c0108fe0 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter ms_handle_connect 0x7f03c0105350 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter ms_handle_connect 0x7f03c0100f30 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031767464 0 0) 0x7f03c01a28f0 con 0x7f03c0108fe0 2024-03-30T14:54:52.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03a80035b0 con 0x7f03c0108fe0 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935615599 0 0) 0x7f03c01a3ad0 con 0x7f03c0100f30 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c01a28f0 con 0x7f03c0100f30 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884311986 0 0) 0x7f03c0197100 con 0x7f03c0105350 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c01a3ad0 con 0x7f03c0105350 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2726141703 0 0) 0x7f03a80035b0 con 0x7f03c0108fe0 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03c0197100 con 0x7f03c0108fe0 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03b8001c30 con 0x7f03c0108fe0 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3517240667 0 0) 0x7f03c01a3ad0 con 0x7f03c0105350 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03a80035b0 con 0x7f03c0105350 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3071854910 0 0) 0x7f03c01a28f0 con 0x7f03c0100f30 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03c01a3ad0 con 0x7f03c0100f30 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03b4001430 con 0x7f03c0105350 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03b0001870 con 0x7f03c0100f30 2024-03-30T14:54:52.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.924+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 560769765 0 0) 0x7f03c0197100 con 0x7f03c0108fe0 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 >> v1:172.21.15.28:6790/0 conn(0x7f03c0100f30 legacy=0x7f03c01965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 >> v1:172.21.15.28:6789/0 conn(0x7f03c0105350 legacy=0x7f03c019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c01a4cf0 con 0x7f03c0108fe0 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03c01a1940 con 0x7f03c0108fe0 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03c50c3700 10 client.24484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f03b8003ec0 con 0x7f03c0108fe0 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f03c01a1e80 con 0x7f03c0108fe0 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03c50c3700 1 librados: init done 2024-03-30T14:54:52.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.925+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03b80060c0 con 0x7f03c0108fe0 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.926+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c010e950 con 0x7f03c0108fe0 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 16) ==== 91790+0+0 (unknown 4216023963 0 0) 0x7f03b8006400 con 0x7f03c0108fe0 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(77..77 src has 14..77) ==== 5942+0+0 (unknown 1497819381 0 0) 0x7f03b80892c0 con 0x7f03c0108fe0 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 10 client.24484.objecter ms_dispatch 0x7f03c0001550 osd_map(77..77 src has 14..77) 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 3 client.24484.objecter handle_osd_map got epochs [77,77] > 0 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 3 client.24484.objecter handle_osd_map decoding full epoch 77 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.927+0000 7f03a7fff700 20 client.24484.objecter dump_active .. 0 homeless 2024-03-30T14:54:52.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.928+0000 7f03a7fff700 10 client.24484.objecter ms_handle_connect 0x7f03a80703d0 2024-03-30T14:54:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:52.932+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f03b80060c0 con 0x7f03c0108fe0 2024-03-30T14:54:53.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.171+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}) -- 0x7f03c01a21e0 con 0x7f03a80703d0 2024-03-30T14:54:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.178+0000 7f03a7fff700 1 -- 172.21.15.134:0/2648116772 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (unknown 0 0 1850065467) 0x7f03c01a21e0 con 0x7f03a80703d0 2024-03-30T14:54:53.179 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled alertmanager update... 2024-03-30T14:54:53.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 10 librados: watch_flush enter 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 10 librados: watch_flush exit 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 20 client.24484.objecter shutdown clearing up homeless session... 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 10 client.24484.objecter successfully canceled tick 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 >> v1:172.21.15.28:6800/3720192247 conn(0x7f03a80703d0 legacy=0x7f03a8072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 >> v1:172.21.15.134:6789/0 conn(0x7f03c0108fe0 legacy=0x7f03c01a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 shutdown_connections 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.180+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 >> 172.21.15.134:0/2648116772 conn(0x7f03c00fc4d0 msgr2=0x7f03c010b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:53.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.181+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 shutdown_connections 2024-03-30T14:54:53.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.181+0000 7f03c50c3700 1 -- 172.21.15.134:0/2648116772 wait complete. 2024-03-30T14:54:53.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:53.181+0000 7f03c50c3700 1 librados: shutdown 2024-03-30T14:54:53.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:53 smithi134 ceph-mon[33714]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T14:54:53.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:53 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[41614]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T14:54:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[29876]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-30T14:54:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:53 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:53.964 DEBUG:teuthology.orchestra.run.smithi028:alertmanager.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@alertmanager.a.service 2024-03-30T14:54:53.967 INFO:tasks.cephadm:Adding grafana.a on smithi134 2024-03-30T14:54:53.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph orch apply grafana '1;smithi134=a' 2024-03-30T14:54:54.537 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:54:54.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:54 smithi134 ceph-mon[33714]: from='client.24484 v1:172.21.15.134:0/2648116772' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:54.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:54 smithi134 ceph-mon[33714]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-30T14:54:54.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:54 smithi134 ceph-mon[33714]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-30T14:54:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[29876]: from='client.24484 v1:172.21.15.134:0/2648116772' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[29876]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-30T14:54:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[29876]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-30T14:54:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[41614]: from='client.24484 v1:172.21.15.134:0/2648116772' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[41614]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-03-30T14:54:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:54 smithi028 ceph-mon[41614]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 4.0 KiB/s wr, 125 op/s 2024-03-30T14:54:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: Deploying daemon prometheus.a on smithi134 2024-03-30T14:54:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:55 smithi134 ceph-mon[33714]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 98 op/s 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: Deploying daemon prometheus.a on smithi134 2024-03-30T14:54:55.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[29876]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 98 op/s 2024-03-30T14:54:55.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:54:55.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:54:55.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:55.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: Deploying daemon prometheus.a on smithi134 2024-03-30T14:54:55.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:55 smithi028 ceph-mon[41614]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 2.7 KiB/s wr, 98 op/s 2024-03-30T14:54:56.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- 172.21.15.134:0/3700700087 >> v1:172.21.15.134:6789/0 conn(0x7fa94c1053b0 legacy=0x7fa94c107800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:56.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- 172.21.15.134:0/3700700087 shutdown_connections 2024-03-30T14:54:56.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- 172.21.15.134:0/3700700087 >> 172.21.15.134:0/3700700087 conn(0x7fa94c0fc520 msgr2=0x7fa94c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:56.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- 172.21.15.134:0/3700700087 shutdown_connections 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- 172.21.15.134:0/3700700087 wait complete. 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 librados: starting msgr at 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 librados: starting objecter 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 Processor -- start 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- start start 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 librados: setting wanted keys 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 librados: calling monclient init 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa94c197e60 con 0x7fa94c1090f0 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa94c1a3600 con 0x7fa94c1053b0 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa953452700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa94c1a47e0 con 0x7fa94c100f80 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa9509ed700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fa94c1053b0 0x7fa94c19d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49476/0 (socket says 172.21.15.134:49476) 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.161+0000 7fa9509ed700 1 -- 172.21.15.134:0/682913829 learned_addr learned my addr 172.21.15.134:0/682913829 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:54:56.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter ms_handle_connect 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380284859 0 0) 0x7fa94c1a3600 con 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9380035b0 con 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter ms_handle_connect 0x7fa94c100f80 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter ms_handle_connect 0x7fa94c1090f0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198544715 0 0) 0x7fa94c1a47e0 con 0x7fa94c100f80 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa94c1a3600 con 0x7fa94c100f80 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4018204897 0 0) 0x7fa9380035b0 con 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa94c1a47e0 con 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa93c004780 con 0x7fa94c1053b0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334691926 0 0) 0x7fa94c197e60 con 0x7fa94c1090f0 2024-03-30T14:54:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9380035b0 con 0x7fa94c1090f0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3969987332 0 0) 0x7fa94c1a47e0 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 >> v1:172.21.15.28:6790/0 conn(0x7fa94c100f80 legacy=0x7fa94c197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 >> v1:172.21.15.28:6789/0 conn(0x7fa94c1090f0 legacy=0x7fa94c1a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1a5a00 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1a4a10 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa953452700 10 client.24490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a4f50 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa953452700 1 librados: init done 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa93c003700 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.162+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa93c0051d0 con 0x7fa94c1053b0 2024-03-30T14:54:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.163+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa930005150 con 0x7fa94c1053b0 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 16) ==== 91790+0+0 (unknown 4216023963 0 0) 0x7fa93c005390 con 0x7fa94c1053b0 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(77..77 src has 14..77) ==== 5942+0+0 (unknown 1497819381 0 0) 0x7fa93c0891a0 con 0x7fa94c1053b0 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 10 client.24490.objecter ms_dispatch 0x7fa94c001550 osd_map(77..77 src has 14..77) 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 3 client.24490.objecter handle_osd_map got epochs [77,77] > 0 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 3 client.24490.objecter handle_osd_map decoding full epoch 77 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.164+0000 7fa9427fc700 20 client.24490.objecter dump_active .. 0 homeless 2024-03-30T14:54:56.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.165+0000 7fa9427fc700 10 client.24490.objecter ms_handle_connect 0x7fa938070460 2024-03-30T14:54:56.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.169+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa93c056ba0 con 0x7fa94c1053b0 2024-03-30T14:54:56.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.376+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 --> v1:172.21.15.28:6800/3720192247 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}) -- 0x7fa930000bc0 con 0x7fa938070460 2024-03-30T14:54:56.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.384+0000 7fa9427fc700 1 -- 172.21.15.134:0/682913829 <== mgr.14152 v1:172.21.15.28:6800/3720192247 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (unknown 0 0 664801700) 0x7fa930000bc0 con 0x7fa938070460 2024-03-30T14:54:56.385 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled grafana update... 2024-03-30T14:54:56.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 10 librados: watch_flush enter 2024-03-30T14:54:56.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 10 librados: watch_flush exit 2024-03-30T14:54:56.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 20 client.24490.objecter shutdown clearing up homeless session... 2024-03-30T14:54:56.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 10 client.24490.objecter successfully canceled tick 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 >> v1:172.21.15.28:6800/3720192247 conn(0x7fa938070460 legacy=0x7fa938072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 >> v1:172.21.15.134:6789/0 conn(0x7fa94c1053b0 legacy=0x7fa94c19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 shutdown_connections 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 >> 172.21.15.134:0/682913829 conn(0x7fa94c0fc520 msgr2=0x7fa94c108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 shutdown_connections 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 -- 172.21.15.134:0/682913829 wait complete. 2024-03-30T14:54:56.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:54:56.386+0000 7fa953452700 1 librados: shutdown 2024-03-30T14:54:57.290 DEBUG:teuthology.orchestra.run.smithi134:grafana.a> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@grafana.a.service 2024-03-30T14:54:57.292 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T14:54:57.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T14:54:57.557 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='client.24490 v1:172.21.15.134:0/682913829' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: Saving service grafana spec with placement smithi134=a;count:1 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 KiB/s wr, 56 op/s 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='client.24490 v1:172.21.15.134:0/682913829' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: Saving service grafana spec with placement smithi134=a;count:1 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 KiB/s wr, 56 op/s 2024-03-30T14:54:57.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:57 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='client.24490 v1:172.21.15.134:0/682913829' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: Saving service grafana spec with placement smithi134=a;count:1 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 KiB/s wr, 56 op/s 2024-03-30T14:54:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:57 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:54:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:58 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:54:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:58 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:54:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:58 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-30T14:54:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:58 smithi134 ceph-mon[33714]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T14:54:58.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[29876]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-30T14:54:58.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:58 smithi028 ceph-mon[41614]: osdmap e78: 8 total, 8 up, 8 in 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.383+0000 7f31f1ec7700 1 -- 172.21.15.28:0/124904251 >> v1:172.21.15.28:6790/0 conn(0x7f31ec100310 legacy=0x7f31ec100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.383+0000 7f31f1ec7700 1 -- 172.21.15.28:0/124904251 shutdown_connections 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.383+0000 7f31f1ec7700 1 -- 172.21.15.28:0/124904251 >> 172.21.15.28:0/124904251 conn(0x7f31ec0fbcb0 msgr2=0x7f31ec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.383+0000 7f31f1ec7700 1 -- 172.21.15.28:0/124904251 shutdown_connections 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.383+0000 7f31f1ec7700 1 -- 172.21.15.28:0/124904251 wait complete. 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 librados: starting msgr at 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 librados: starting objecter 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 Processor -- start 2024-03-30T14:54:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 -- start start 2024-03-30T14:54:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 librados: setting wanted keys 2024-03-30T14:54:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 librados: calling monclient init 2024-03-30T14:54:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31ec1a4700 con 0x7f31ec10a180 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31ec1a5940 con 0x7f31ec106520 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.384+0000 7f31f1ec7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31ec1a6b80 con 0x7f31ec100310 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31eb7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f31ec100310 0x7f31ec102b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42346/0 (socket says 172.21.15.28:42346) 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31eb7fe700 1 -- 172.21.15.28:0/177102951 learned_addr learned my addr 172.21.15.28:0/177102951 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter ms_handle_connect 0x7f31ec100310 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter ms_handle_connect 0x7f31ec10a180 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter ms_handle_connect 0x7f31ec106520 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847651762 0 0) 0x7f31ec1a6b80 con 0x7f31ec100310 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31d8004450 con 0x7f31ec100310 2024-03-30T14:54:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944942823 0 0) 0x7f31ec1a5940 con 0x7f31ec106520 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ec1a6b80 con 0x7f31ec106520 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316189907 0 0) 0x7f31ec1a4700 con 0x7f31ec10a180 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ec1a5940 con 0x7f31ec10a180 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1711772437 0 0) 0x7f31d8004450 con 0x7f31ec100310 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31ec1a4700 con 0x7f31ec100310 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f31dc0038e0 con 0x7f31ec100310 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2643609479 0 0) 0x7f31ec1a6b80 con 0x7f31ec106520 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31d8004450 con 0x7f31ec106520 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f31d4001840 con 0x7f31ec106520 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1952835711 0 0) 0x7f31ec1a5940 con 0x7f31ec10a180 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31ec1a6b80 con 0x7f31ec10a180 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f31e4001af0 con 0x7f31ec10a180 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 994761204 0 0) 0x7f31ec1a4700 con 0x7f31ec100310 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 >> v1:172.21.15.134:6789/0 conn(0x7f31ec106520 legacy=0x7f31ec19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 >> v1:172.21.15.28:6789/0 conn(0x7f31ec10a180 legacy=0x7f31ec1a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec1a7dc0 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31ec1a4930 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31f1ec7700 10 client.34466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f31ec1a4e70 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.385+0000 7f31f1ec7700 1 librados: init done 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.386+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f31dc001c20 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.386+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f31dc004f00 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.386+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc005150 con 0x7f31ec100310 2024-03-30T14:54:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.387+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 16) ==== 91790+0+0 (unknown 4216023963 0 0) 0x7f31dc001780 con 0x7f31ec100310 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.387+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(79..79 src has 14..79) ==== 6001+0+0 (unknown 1147852664 0 0) 0x7f31dc088fd0 con 0x7f31ec100310 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.387+0000 7f31e97fa700 10 client.34466.objecter ms_dispatch 0x7f31ec001550 osd_map(79..79 src has 14..79) 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.387+0000 7f31e97fa700 3 client.34466.objecter handle_osd_map got epochs [79,79] > 0 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.387+0000 7f31e97fa700 3 client.34466.objecter handle_osd_map decoding full epoch 79 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.388+0000 7f31e97fa700 20 client.34466.objecter dump_active .. 0 homeless 2024-03-30T14:54:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.388+0000 7f31e97fa700 10 client.34466.objecter ms_handle_connect 0x7f31d8070440 2024-03-30T14:54:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.392+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f31dc056990 con 0x7f31ec100310 2024-03-30T14:54:59.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:59 smithi134 ceph-mon[33714]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T14:54:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:54:59 smithi134 ceph-mon[33714]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-30T14:54:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.684+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f31cc005440 con 0x7f31ec100310 2024-03-30T14:54:59.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:59 smithi028 ceph-mon[29876]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T14:54:59.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:54:59 smithi028 ceph-mon[29876]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-30T14:54:59.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:59 smithi028 ceph-mon[41614]: osdmap e79: 8 total, 8 up, 8 in 2024-03-30T14:54:59.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:54:59 smithi028 ceph-mon[41614]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-30T14:54:59.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.690+0000 7f31e97fa700 1 -- 172.21.15.28:0/177102951 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (unknown 57899324 0 4276245278) 0x7f31dc056290 con 0x7f31ec100310 2024-03-30T14:54:59.691 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-30T14:54:59.691 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDDJwhmcK/sKBAAJtY4fyAssSOMpEXiysLO1Q== 2024-03-30T14:54:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 10 librados: watch_flush enter 2024-03-30T14:54:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 10 librados: watch_flush exit 2024-03-30T14:54:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 20 client.34466.objecter shutdown clearing up homeless session... 2024-03-30T14:54:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 10 client.34466.objecter successfully canceled tick 2024-03-30T14:54:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 >> v1:172.21.15.28:6800/3720192247 conn(0x7f31d8070440 legacy=0x7f31d8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 >> v1:172.21.15.28:6790/0 conn(0x7f31ec100310 legacy=0x7f31ec102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 shutdown_connections 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 >> 172.21.15.28:0/177102951 conn(0x7f31ec0fbcb0 msgr2=0x7f31ec0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 shutdown_connections 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 -- 172.21.15.28:0/177102951 wait complete. 2024-03-30T14:54:59.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:54:59.692+0000 7f31f1ec7700 1 librados: shutdown 2024-03-30T14:55:00.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:54:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:00.511 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:55:00.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T14:55:00.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T14:55:00.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/177102951' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:00 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:00.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/177102951' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: osdmap e80: 8 total, 8 up, 8 in 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/177102951' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: from='client.34466 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: from='client.34466 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:00 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:02.093 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:55:02.593 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:02 smithi134 ceph-mon[33714]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T14:55:02.593 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:02 smithi134 ceph-mon[33714]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 116 B/s, 2 objects/s recovering 2024-03-30T14:55:02.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:02 smithi028 ceph-mon[29876]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T14:55:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:02 smithi028 ceph-mon[29876]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 116 B/s, 2 objects/s recovering 2024-03-30T14:55:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:02 smithi028 ceph-mon[41614]: osdmap e81: 8 total, 8 up, 8 in 2024-03-30T14:55:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:02 smithi028 ceph-mon[41614]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 116 B/s, 2 objects/s recovering 2024-03-30T14:55:03.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:03 smithi134 ceph-mon[33714]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:03 smithi028 ceph-mon[29876]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:03 smithi028 ceph-mon[41614]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:05.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.882+0000 7f29f6604700 1 -- 172.21.15.134:0/3186914759 >> v1:172.21.15.134:6789/0 conn(0x7f29f010a180 legacy=0x7f29f010c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:05.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.882+0000 7f29f6604700 1 -- 172.21.15.134:0/3186914759 shutdown_connections 2024-03-30T14:55:05.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- 172.21.15.134:0/3186914759 >> 172.21.15.134:0/3186914759 conn(0x7f29f00fd5f0 msgr2=0x7f29f00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:05.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- 172.21.15.134:0/3186914759 shutdown_connections 2024-03-30T14:55:05.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- 172.21.15.134:0/3186914759 wait complete. 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 librados: starting msgr at 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 librados: starting objecter 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 Processor -- start 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- start start 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 librados: setting wanted keys 2024-03-30T14:55:05.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 librados: calling monclient init 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29f0198260 con 0x7f29f0102010 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29f01a3a50 con 0x7f29f0106440 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.883+0000 7f29f6604700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29f01a4c30 con 0x7f29f010a180 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ef7fe700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f29f0106440 0x7f29f019dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:37118/0 (socket says 172.21.15.134:37118) 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ef7fe700 1 -- 172.21.15.134:0/33255300 learned_addr learned my addr 172.21.15.134:0/33255300 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter ms_handle_connect 0x7f29f0106440 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter ms_handle_connect 0x7f29f0102010 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter ms_handle_connect 0x7f29f010a180 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267048223 0 0) 0x7f29f01a3a50 con 0x7f29f0106440 2024-03-30T14:55:05.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d80035b0 con 0x7f29f0106440 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592124332 0 0) 0x7f29f0198260 con 0x7f29f0102010 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f01a3a50 con 0x7f29f0102010 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3383525592 0 0) 0x7f29f01a4c30 con 0x7f29f010a180 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f0198260 con 0x7f29f010a180 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4224630678 0 0) 0x7f29d80035b0 con 0x7f29f0106440 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29f01a4c30 con 0x7f29f0106440 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29e40018b0 con 0x7f29f0106440 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3756006185 0 0) 0x7f29f01a3a50 con 0x7f29f0102010 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29d80035b0 con 0x7f29f0102010 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29e0001870 con 0x7f29f0102010 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2682648134 0 0) 0x7f29f0198260 con 0x7f29f010a180 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29f01a3a50 con 0x7f29f010a180 2024-03-30T14:55:05.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29e80017b0 con 0x7f29f010a180 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 717591043 0 0) 0x7f29f01a4c30 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 >> v1:172.21.15.28:6790/0 conn(0x7f29f010a180 legacy=0x7f29f01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 >> v1:172.21.15.28:6789/0 conn(0x7f29f0102010 legacy=0x7f29f0197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f01a5e50 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29f01a2aa0 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29f6604700 10 client.24502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29f01a2fe0 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.884+0000 7f29f6604700 1 librados: init done 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.885+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f29e4003b20 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.885+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29e4005cb0 con 0x7f29f0106440 2024-03-30T14:55:05.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.885+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29dc005150 con 0x7f29f0106440 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 16) ==== 91790+0+0 (unknown 4216023963 0 0) 0x7f29e4003680 con 0x7f29f0106440 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(82..82 src has 14..82) ==== 6001+0+0 (unknown 3251494184 0 0) 0x7f29e4088e20 con 0x7f29f0106440 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 10 client.24502.objecter ms_dispatch 0x7f29f0001550 osd_map(82..82 src has 14..82) 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 3 client.24502.objecter handle_osd_map got epochs [82,82] > 0 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 3 client.24502.objecter handle_osd_map decoding full epoch 82 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.886+0000 7f29ed7fa700 20 client.24502.objecter dump_active .. 0 homeless 2024-03-30T14:55:05.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.887+0000 7f29ed7fa700 10 client.24502.objecter ms_handle_connect 0x7f29d8070440 2024-03-30T14:55:05.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:05.891+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f29e40567e0 con 0x7f29f0106440 2024-03-30T14:55:06.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.170+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f29dc005440 con 0x7f29f0106440 2024-03-30T14:55:06.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.176+0000 7f29ed7fa700 1 -- 172.21.15.134:0/33255300 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (unknown 3508017507 0 446034535) 0x7f29e40560e0 con 0x7f29f0106440 2024-03-30T14:55:06.177 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-03-30T14:55:06.177 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQDKJwhmTklEChAA5vefrvMYqMFGggpYa/BsIQ== 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 10 librados: watch_flush enter 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 10 librados: watch_flush exit 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 20 client.24502.objecter shutdown clearing up homeless session... 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 10 client.24502.objecter successfully canceled tick 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 >> v1:172.21.15.28:6800/3720192247 conn(0x7f29d8070440 legacy=0x7f29d8072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 >> v1:172.21.15.134:6789/0 conn(0x7f29f0106440 legacy=0x7f29f019dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 shutdown_connections 2024-03-30T14:55:06.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 >> 172.21.15.134:0/33255300 conn(0x7f29f00fd5f0 msgr2=0x7f29f0108fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:06.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 shutdown_connections 2024-03-30T14:55:06.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 -- 172.21.15.134:0/33255300 wait complete. 2024-03-30T14:55:06.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-30T14:55:06.178+0000 7f29f6604700 1 librados: shutdown 2024-03-30T14:55:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:06 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:06 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:06 smithi134 ceph-mon[33714]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T14:55:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:06 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T14:55:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:06 smithi134 ceph-mon[33714]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:06.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:06.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:06.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[29876]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T14:55:06.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[29876]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[41614]: osdmap e82: 8 total, 8 up, 8 in 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T14:55:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:06 smithi028 ceph-mon[41614]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 96 B/s, 3 objects/s recovering 2024-03-30T14:55:06.961 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-30T14:55:06.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-30T14:55:06.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T14:55:07.003 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T14:55:07.003 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T14:55:07.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.134:0/33255300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:07.220 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:07 smithi134 ceph-mon[33714]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T14:55:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.134:0/33255300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:07.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[29876]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: osdmap e83: 8 total, 8 up, 8 in 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.134:0/33255300' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T14:55:07.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:07 smithi028 ceph-mon[41614]: osdmap e84: 8 total, 8 up, 8 in 2024-03-30T14:55:08.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-mon[33714]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T14:55:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:08 smithi028 ceph-mon[29876]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T14:55:08.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:08 smithi028 ceph-mon[41614]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T14:55:08.668 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 systemd[1]: Starting Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:55:08.919 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 podman[64414]: 2024-03-30 14:55:08.57216556 +0000 UTC m=+0.018976058 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T14:55:08.919 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 podman[64414]: 2024-03-30 14:55:08.676597951 +0000 UTC m=+0.123408426 container create 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 podman[64414]: 2024-03-30 14:55:08.886876996 +0000 UTC m=+0.333687482 container init 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 podman[64414]: 2024-03-30 14:55:08.892082563 +0000 UTC m=+0.338893035 container start 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.915Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.915Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.915Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi134 (none))" 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.915Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.915Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.916Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.917Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.918Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T14:55:08.920 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.918Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T14:55:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/4102410237 >> v1:172.21.15.28:6789/0 conn(0x7fb0a810aec0 legacy=0x7fb0a810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/4102410237 shutdown_connections 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/4102410237 >> 172.21.15.28:0/4102410237 conn(0x7fb0a8077370 msgr2=0x7fb0a80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/4102410237 shutdown_connections 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/4102410237 wait complete. 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 librados: starting msgr at 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 librados: starting objecter 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.071+0000 7fb0ad7d5700 1 Processor -- start 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 -- start start 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 librados: setting wanted keys 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 librados: calling monclient init 2024-03-30T14:55:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0a81a7110 con 0x7fb0a810aec0 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0a81a8330 con 0x7fb0a810ea60 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0ad7d5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0a81a9570 con 0x7fb0a8100740 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0a6ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb0a8100740 0x7fb0a81075a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:33356/0 (socket says 172.21.15.28:33356) 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb0a6ffd700 1 -- 172.21.15.28:0/3993772074 learned_addr learned my addr 172.21.15.28:0/3993772074 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter ms_handle_connect 0x7fb0a8100740 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter ms_handle_connect 0x7fb0a810aec0 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter ms_handle_connect 0x7fb0a810ea60 2024-03-30T14:55:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980230549 0 0) 0x7fb0a81a9570 con 0x7fb0a8100740 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0940031f0 con 0x7fb0a8100740 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061886837 0 0) 0x7fb0a81a7110 con 0x7fb0a810aec0 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a81a9570 con 0x7fb0a810aec0 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955451972 0 0) 0x7fb0a81a8330 con 0x7fb0a810ea60 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a81a7110 con 0x7fb0a810ea60 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.072+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2835690916 0 0) 0x7fb0940031f0 con 0x7fb0a8100740 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0a81a8330 con 0x7fb0a8100740 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0a00012d0 con 0x7fb0a8100740 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1903422428 0 0) 0x7fb0a81a7110 con 0x7fb0a810ea60 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0940031f0 con 0x7fb0a810ea60 2024-03-30T14:55:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb098001c30 con 0x7fb0a810ea60 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2058671724 0 0) 0x7fb0a81a9570 con 0x7fb0a810aec0 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0a81a7110 con 0x7fb0a810aec0 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0900044b0 con 0x7fb0a810aec0 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1494249674 0 0) 0x7fb0a81a8330 con 0x7fb0a8100740 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 >> v1:172.21.15.134:6789/0 conn(0x7fb0a810ea60 legacy=0x7fb0a81a57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 >> v1:172.21.15.28:6789/0 conn(0x7fb0a810aec0 legacy=0x7fb0a81a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a81aa7b0 con 0x7fb0a8100740 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a81a7340 con 0x7fb0a8100740 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb0ad7d5700 10 client.34478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0a81a7800 con 0x7fb0a8100740 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb0ad7d5700 1 librados: init done 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb0a0001650 con 0x7fb0a8100740 2024-03-30T14:55:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.073+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0a0005d40 con 0x7fb0a8100740 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.074+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb088005120 con 0x7fb0a8100740 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.074+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 16) ==== 91790+0+0 (unknown 4216023963 0 0) 0x7fb0a001c3e0 con 0x7fb0a8100740 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(84..84 src has 14..84) ==== 6001+0+0 (unknown 3225961487 0 0) 0x7fb0a0088e70 con 0x7fb0a8100740 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 10 client.34478.objecter ms_dispatch 0x7fb0a8001550 osd_map(84..84 src has 14..84) 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 3 client.34478.objecter handle_osd_map got epochs [84,84] > 0 2024-03-30T14:55:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 3 client.34478.objecter handle_osd_map decoding full epoch 84 2024-03-30T14:55:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 20 client.34478.objecter dump_active .. 0 homeless 2024-03-30T14:55:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.075+0000 7fb09ffff700 10 client.34478.objecter ms_handle_connect 0x7fb0940702b0 2024-03-30T14:55:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.079+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb0a0056770 con 0x7fb0a8100740 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.920Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.920Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.704µs 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.920Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.920Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.920Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.634µs wal_replay_duration=202.974µs wbl_replay_duration=103ns total_replay_duration=238.449µs 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.922Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.922Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.922Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T14:55:09.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.938Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.777629ms db_storage=677ns remote_storage=1.065µs web_handler=365ns query_engine=969ns scrape=369.173µs scrape_sd=63.168µs notify=703ns notify_sd=1.27µs rules=16.165224ms tracing=5.796µs 2024-03-30T14:55:09.181 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.938Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T14:55:09.181 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:55:08.938Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T14:55:09.181 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 bash[64414]: 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab 2024-03-30T14:55:09.181 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:55:08 smithi134 systemd[1]: Started Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:55:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.342+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb088005410 con 0x7fb0a8100740 2024-03-30T14:55:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.346+0000 7fb09ffff700 1 -- 172.21.15.28:0/3993772074 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) ==== 74+0+175381 (unknown 3479712753 0 3421006127) 0x7fb0a0056070 con 0x7fb0a8100740 2024-03-30T14:55:09.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:55:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 10 librados: watch_flush enter 2024-03-30T14:55:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 10 librados: watch_flush exit 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 20 client.34478.objecter shutdown clearing up homeless session... 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 10 client.34478.objecter successfully canceled tick 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 >> v1:172.21.15.28:6800/3720192247 conn(0x7fb0940702b0 legacy=0x7fb094072770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 >> v1:172.21.15.28:6790/0 conn(0x7fb0a8100740 legacy=0x7fb0a81075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 shutdown_connections 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 >> 172.21.15.28:0/3993772074 conn(0x7fb0a8077370 msgr2=0x7fb0a81116b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 shutdown_connections 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 -- 172.21.15.28:0/3993772074 wait complete. 2024-03-30T14:55:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:09.348+0000 7fb0ad7d5700 1 librados: shutdown 2024-03-30T14:55:10.057 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6800","nonce":3720192247}]},"active_addr":"172.21.15.28:6800/3720192247","active_change":"2024-03-30T14:49:57.293849+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":406580277}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":4222366292}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2199995094}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":580061692}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":545344739}]}]} 2024-03-30T14:55:10.061 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T14:55:10.061 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T14:55:10.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:55:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T14:55:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3993772074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T14:55:10.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-mon[33714]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:10.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: ignoring --setuser ceph since I am not root 2024-03-30T14:55:10.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: ignoring --setgroup ceph since I am not root 2024-03-30T14:55:10.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:10.331 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:10.453 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setuser ceph since I am not root 2024-03-30T14:55:10.453 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ignoring --setgroup ceph since I am not root 2024-03-30T14:55:10.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3993772074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[41614]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3993772074' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T14:55:10.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-mon[29876]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:10.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:10.698+0000 7f6c4daf8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:55:10.929 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:10.650+0000 7effd61b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T14:55:10.930 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:10.744+0000 7effd61b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:55:11.087 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:10 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:10.798+0000 7f6c4daf8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T14:55:11.387 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-mon[33714]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T14:55:11.388 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-mon[33714]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T14:55:11.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-mon[29876]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T14:55:11.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-mon[29876]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T14:55:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-mon[41614]: from='mgr.14152 v1:172.21.15.28:0/290037309' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-30T14:55:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-mon[41614]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-30T14:55:11.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:11.386+0000 7effd61b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:55:11.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:11.479+0000 7effd61b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:55:11.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:11.622+0000 7effd61b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:55:11.758 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:11.476+0000 7f6c4daf8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T14:55:11.758 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:11.572+0000 7f6c4daf8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T14:55:11.758 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:11.721+0000 7f6c4daf8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T14:55:12.068 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:11.719+0000 7effd61b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:55:12.068 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:11.802+0000 7effd61b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:55:12.068 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:12.067+0000 7effd61b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:55:12.109 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:11.822+0000 7f6c4daf8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T14:55:12.109 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:11.914+0000 7f6c4daf8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T14:55:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 -- 172.21.15.28:0/471140421 >> v1:172.21.15.28:6790/0 conn(0x7f10c4101130 legacy=0x7f10c4101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 -- 172.21.15.28:0/471140421 shutdown_connections 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 -- 172.21.15.28:0/471140421 >> 172.21.15.28:0/471140421 conn(0x7f10c40fc9e0 msgr2=0x7f10c40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 -- 172.21.15.28:0/471140421 shutdown_connections 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 -- 172.21.15.28:0/471140421 wait complete. 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 librados: starting msgr at 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.108+0000 7f10ca5f5700 1 librados: starting objecter 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 Processor -- start 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 -- start start 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 librados: setting wanted keys 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 librados: calling monclient init 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10c41a92a0 con 0x7f10c410d960 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10c41aa4c0 con 0x7f10c4109dc0 2024-03-30T14:55:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10ca5f5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10c41ab700 con 0x7f10c4101130 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c37fe700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f10c4109dc0 0x7f10c41a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:58890/0 (socket says 172.21.15.28:58890) 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c37fe700 1 -- 172.21.15.28:0/65141971 learned_addr learned my addr 172.21.15.28:0/65141971 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c3fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f10c4101130 0x7f10c41062b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:33376/0 (socket says 172.21.15.28:33376) 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c17fa700 10 client.?.objecter ms_handle_connect 0x7f10c4109dc0 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c17fa700 10 client.?.objecter ms_handle_connect 0x7f10c4101130 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.109+0000 7f10c17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 10 client.?.objecter ms_handle_connect 0x7f10c410d960 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838602876 0 0) 0x7f10c41aa4c0 con 0x7f10c4109dc0 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10ac0031f0 con 0x7f10c4109dc0 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511225855 0 0) 0x7f10c41ab700 con 0x7f10c4101130 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10c41aa4c0 con 0x7f10c4101130 2024-03-30T14:55:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559982875 0 0) 0x7f10c41a92a0 con 0x7f10c410d960 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10c41ab700 con 0x7f10c410d960 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1338444727 0 0) 0x7f10ac0031f0 con 0x7f10c4109dc0 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10c41a92a0 con 0x7f10c4109dc0 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f10b8001930 con 0x7f10c4109dc0 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3956322515 0 0) 0x7f10c41aa4c0 con 0x7f10c4101130 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10ac0031f0 con 0x7f10c4101130 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f10b4004620 con 0x7f10c4101130 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4113981046 0 0) 0x7f10c41a92a0 con 0x7f10c4109dc0 2024-03-30T14:55:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.28:6790/0 conn(0x7f10c4101130 legacy=0x7f10c41062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.28:6789/0 conn(0x7f10c410d960 legacy=0x7f10c41a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c41ac940 con 0x7f10c4109dc0 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10c41ab930 con 0x7f10c4109dc0 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10ca5f5700 10 client.24523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f10c41abdf0 con 0x7f10c4109dc0 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10ca5f5700 1 librados: init done 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.110+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f10b8003b40 con 0x7f10c4109dc0 2024-03-30T14:55:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.111+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f10b8005c90 con 0x7f10c4109dc0 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.112+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b0005120 con 0x7f10c4109dc0 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.112+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 17) ==== 91804+0+0 (unknown 526730084 0 0) 0x7f10b801c4e0 con 0x7f10c4109dc0 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c3fff700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.28:6800/3720192247 conn(0x7f10ac0703d0 legacy=0x7f10ac072890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/3720192247 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(85..85 src has 14..85) ==== 6001+0+0 (unknown 3193694550 0 0) 0x7f10b8089100 con 0x7f10c4109dc0 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c17fa700 10 client.24523.objecter ms_dispatch 0x7f10c4001550 osd_map(85..85 src has 14..85) 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c17fa700 3 client.24523.objecter handle_osd_map got epochs [85,85] > 0 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c17fa700 3 client.24523.objecter handle_osd_map decoding full epoch 85 2024-03-30T14:55:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.113+0000 7f10c17fa700 20 client.24523.objecter dump_active .. 0 homeless 2024-03-30T14:55:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.117+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f10b8056a00 con 0x7f10c4109dc0 2024-03-30T14:55:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.314+0000 7f10c3fff700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.28:6800/3720192247 conn(0x7f10ac0703d0 legacy=0x7f10ac072890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/3720192247 2024-03-30T14:55:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.323+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10b0005410 con 0x7f10c4109dc0 2024-03-30T14:55:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.325+0000 7f10c17fa700 1 -- 172.21.15.28:0/65141971 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+20686 (unknown 1508052635 0 3786425258) 0x7f10b8056300 con 0x7f10c4109dc0 2024-03-30T14:55:12.326 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:55:12.326 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:48:49.804304+0000","modified":"2024-03-30T14:55:11.232755+0000","last_up_change":"2024-03-30T14:54:22.403545+0000","last_in_change":"2024-03-30T14:54:05.032145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:52:17.629184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T14:54:34.798924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T14:54:35.563381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T14:54:37.775241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T14:54:39.594196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T14:54:41.660853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb350905-2b09-408a-9adc-4f2af2766479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3394323575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3394323575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3394323575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3394323575}]},"public_addr":"172.21.15.28:6801/3394323575","cluster_addr":"172.21.15.28:6802/3394323575","heartbeat_back_addr":"172.21.15.28:6804/3394323575","heartbeat_front_addr":"172.21.15.28:6803/3394323575","state":["exists","up"]},{"osd":1,"uuid":"76bfea8d-68c7-448e-bb99-d3d003567520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1850313547}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1850313547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1850313547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1850313547}]},"public_addr":"172.21.15.28:6805/1850313547","cluster_addr":"172.21.15.28:6806/1850313547","heartbeat_back_addr":"172.21.15.28:6808/1850313547","heartbeat_front_addr":"172.21.15.28:6807/1850313547","state":["exists","up"]},{"osd":2,"uuid":"ad24917c-f5cb-4ca8-8a52-91d2256ff21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":3451325043}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":3451325043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":3451325043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":3451325043}]},"public_addr":"172.21.15.28:6809/3451325043","cluster_addr":"172.21.15.28:6810/3451325043","heartbeat_back_addr":"172.21.15.28:6812/3451325043","heartbeat_front_addr":"172.21.15.28:6811/3451325043","state":["exists","up"]},{"osd":3,"uuid":"4305caf1-0023-438f-8520-87814130b457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1506157250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1506157250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1506157250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1506157250}]},"public_addr":"172.21.15.28:6813/1506157250","cluster_addr":"172.21.15.28:6814/1506157250","heartbeat_back_addr":"172.21.15.28:6816/1506157250","heartbeat_front_addr":"172.21.15.28:6815/1506157250","state":["exists","up"]},{"osd":4,"uuid":"8180ec45-5260-4cdc-855c-be5302b1465c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6800","nonce":3063775514}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6801","nonce":3063775514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6803","nonce":3063775514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6802","nonce":3063775514}]},"public_addr":"172.21.15.134:6800/3063775514","cluster_addr":"172.21.15.134:6801/3063775514","heartbeat_back_addr":"172.21.15.134:6803/3063775514","heartbeat_front_addr":"172.21.15.134:6802/3063775514","state":["exists","up"]},{"osd":5,"uuid":"265992a1-46bc-4621-b9db-2b5ba1ee5e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6804","nonce":3668151179}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6805","nonce":3668151179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6807","nonce":3668151179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6806","nonce":3668151179}]},"public_addr":"172.21.15.134:6804/3668151179","cluster_addr":"172.21.15.134:6805/3668151179","heartbeat_back_addr":"172.21.15.134:6807/3668151179","heartbeat_front_addr":"172.21.15.134:6806/3668151179","state":["exists","up"]},{"osd":6,"uuid":"5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6808","nonce":2135224194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6809","nonce":2135224194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6811","nonce":2135224194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6810","nonce":2135224194}]},"public_addr":"172.21.15.134:6808/2135224194","cluster_addr":"172.21.15.134:6809/2135224194","heartbeat_back_addr":"172.21.15.134:6811/2135224194","heartbeat_front_addr":"172.21.15.134:6810/2135224194","state":["exists","up"]},{"osd":7,"uuid":"76f25ab9-af2f-4f45-afe3-13ece781d29c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6812","nonce":3758190298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6813","nonce":3758190298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6815","nonce":3758190298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6814","nonce":3758190298}]},"public_addr":"172.21.15.134:6812/3758190298","cluster_addr":"172.21.15.134:6813/3758190298","heartbeat_back_addr":"172.21.15.134:6815/3758190298","heartbeat_front_addr":"172.21.15.134:6814/3758190298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:59.189754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:31.216170+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:39.827889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:55:05.029867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:09.901960+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:37.321258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:55:05.294758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:21.111874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2354623613":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/2838206829":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/1249955842":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1446120238":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/429160846":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/997592180":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/2767265670":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1590506004":"2024-03-31T14:49:57.293695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:55:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 10 librados: watch_flush enter 2024-03-30T14:55:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 10 librados: watch_flush exit 2024-03-30T14:55:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 20 client.24523.objecter shutdown clearing up homeless session... 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 10 client.24523.objecter successfully canceled tick 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.28:6800/3720192247 conn(0x7f10ac0703d0 legacy=0x7f10ac072890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 >> v1:172.21.15.134:6789/0 conn(0x7f10c4109dc0 legacy=0x7f10c41a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 shutdown_connections 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.327+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 >> 172.21.15.28:0/65141971 conn(0x7f10c40fc9e0 msgr2=0x7f10c4106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.328+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 shutdown_connections 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.328+0000 7f10ca5f5700 1 -- 172.21.15.28:0/65141971 wait complete. 2024-03-30T14:55:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:12.328+0000 7f10ca5f5700 1 librados: shutdown 2024-03-30T14:55:12.362 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-mon[33714]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T14:55:12.362 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:12.179+0000 7effd61b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:55:12.362 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:12.265+0000 7effd61b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:55:12.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-mon[29876]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T14:55:12.400 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:12.190+0000 7f6c4daf8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T14:55:12.400 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:12.304+0000 7f6c4daf8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T14:55:12.400 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:12.398+0000 7f6c4daf8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T14:55:12.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-mon[41614]: osdmap e85: 8 total, 8 up, 8 in 2024-03-30T14:55:12.679 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:12.360+0000 7effd61b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:55:12.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:12.537+0000 7effd61b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:55:12.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:12.500+0000 7f6c4daf8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T14:55:12.972 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:12 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:12.687+0000 7f6c4daf8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T14:55:13.052 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T14:55:13.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T14:55:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:13.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/65141971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:13.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/65141971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:13.496 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/65141971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:13.929 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:13.495+0000 7effd61b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:55:13.930 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:13.627+0000 7effd61b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:55:14.214 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:13.729+0000 7f6c4daf8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T14:55:14.215 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:13.870+0000 7f6c4daf8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T14:55:14.215 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:14.213+0000 7f6c4daf8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:55:14.251 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:13.964+0000 7effd61b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T14:55:14.251 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:14.055+0000 7effd61b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:55:14.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:14.307+0000 7f6c4daf8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T14:55:14.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:14.250+0000 7effd61b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:55:14.680 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:14.435+0000 7effd61b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:55:14.793 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:14.509+0000 7f6c4daf8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:55:14.793 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:14.707+0000 7f6c4daf8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T14:55:14.986 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:14.721+0000 7effd61b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:55:15.083 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:15.011+0000 7f6c4daf8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T14:55:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.192+0000 7f71ca1e4700 1 -- 172.21.15.28:0/260246081 >> v1:172.21.15.28:6790/0 conn(0x7f71c410ea90 legacy=0x7f71c4110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.192+0000 7f71ca1e4700 1 -- 172.21.15.28:0/260246081 shutdown_connections 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.192+0000 7f71ca1e4700 1 -- 172.21.15.28:0/260246081 >> 172.21.15.28:0/260246081 conn(0x7f71c40fdb10 msgr2=0x7f71c40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.192+0000 7f71ca1e4700 1 -- 172.21.15.28:0/260246081 shutdown_connections 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.192+0000 7f71ca1e4700 1 -- 172.21.15.28:0/260246081 wait complete. 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 librados: starting msgr at 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 librados: starting objecter 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 Processor -- start 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 -- start start 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 librados: setting wanted keys 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 librados: calling monclient init 2024-03-30T14:55:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71c4076080 con 0x7f71c4102260 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71c4076290 con 0x7f71c410aef0 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.193+0000 7f71ca1e4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71c40764a0 con 0x7f71c410ea90 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c37fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71c4102260 0x7f71c4074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35620/0 (socket says 172.21.15.28:35620) 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c37fe700 1 -- 172.21.15.28:0/2851743394 learned_addr learned my addr 172.21.15.28:0/2851743394 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter ms_handle_connect 0x7f71c410aef0 2024-03-30T14:55:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter ms_handle_connect 0x7f71c4102260 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter ms_handle_connect 0x7f71c410ea90 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 169708764 0 0) 0x7f71c4076290 con 0x7f71c410aef0 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71ac0039c0 con 0x7f71c410aef0 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548501982 0 0) 0x7f71c4076080 con 0x7f71c4102260 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c4076290 con 0x7f71c4102260 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599899043 0 0) 0x7f71c4115f10 con 0x7f71c410ea90 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c4076080 con 0x7f71c410ea90 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 949412932 0 0) 0x7f71ac0039c0 con 0x7f71c410aef0 2024-03-30T14:55:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71c4115f10 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71b8001890 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3772255799 0 0) 0x7f71c4076290 con 0x7f71c4102260 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71ac0039c0 con 0x7f71c4102260 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71b4001870 con 0x7f71c4102260 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1728513964 0 0) 0x7f71c4115f10 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.28:6790/0 conn(0x7f71c410ea90 legacy=0x7f71c41a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.28:6789/0 conn(0x7f71c4102260 legacy=0x7f71c4074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c41acad0 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71c41a9aa0 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.194+0000 7f71ca1e4700 10 client.24529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.195+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71c41a9f60 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.195+0000 7f71ca1e4700 1 librados: init done 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.195+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f71b8003b20 con 0x7f71c410aef0 2024-03-30T14:55:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.195+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71b8004910 con 0x7f71c410aef0 2024-03-30T14:55:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.196+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b0005120 con 0x7f71c410aef0 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.196+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 17) ==== 91804+0+0 (unknown 526730084 0 0) 0x7f71b8003680 con 0x7f71c410aef0 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c37fe700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.28:6800/3720192247 conn(0x7f71ac070450 legacy=0x7f71ac072910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/3720192247 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(85..85 src has 14..85) ==== 6001+0+0 (unknown 3193694550 0 0) 0x7f71b8087c50 con 0x7f71c410aef0 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c0ff9700 10 client.24529.objecter ms_dispatch 0x7f71c4001550 osd_map(85..85 src has 14..85) 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c0ff9700 3 client.24529.objecter handle_osd_map got epochs [85,85] > 0 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c0ff9700 3 client.24529.objecter handle_osd_map decoding full epoch 85 2024-03-30T14:55:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.197+0000 7f71c0ff9700 20 client.24529.objecter dump_active .. 0 homeless 2024-03-30T14:55:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.201+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f71b8055550 con 0x7f71c410aef0 2024-03-30T14:55:15.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.398+0000 7f71c37fe700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.28:6800/3720192247 conn(0x7f71ac070450 legacy=0x7f71ac072910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.28:6800/3720192247 2024-03-30T14:55:15.399 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:15.280+0000 7f6c4daf8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:55:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.411+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71b0005410 con 0x7f71c410aef0 2024-03-30T14:55:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.413+0000 7f71c0ff9700 1 -- 172.21.15.28:0/2851743394 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+20686 (unknown 1508052635 0 3786425258) 0x7f71b8054e50 con 0x7f71c410aef0 2024-03-30T14:55:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:55:15.415 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:48:49.804304+0000","modified":"2024-03-30T14:55:11.232755+0000","last_up_change":"2024-03-30T14:54:22.403545+0000","last_in_change":"2024-03-30T14:54:05.032145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:52:17.629184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T14:54:34.798924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T14:54:35.563381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T14:54:37.775241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T14:54:39.594196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T14:54:41.660853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb350905-2b09-408a-9adc-4f2af2766479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":3394323575}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":3394323575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":3394323575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":3394323575}]},"public_addr":"172.21.15.28:6801/3394323575","cluster_addr":"172.21.15.28:6802/3394323575","heartbeat_back_addr":"172.21.15.28:6804/3394323575","heartbeat_front_addr":"172.21.15.28:6803/3394323575","state":["exists","up"]},{"osd":1,"uuid":"76bfea8d-68c7-448e-bb99-d3d003567520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1850313547}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1850313547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1850313547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1850313547}]},"public_addr":"172.21.15.28:6805/1850313547","cluster_addr":"172.21.15.28:6806/1850313547","heartbeat_back_addr":"172.21.15.28:6808/1850313547","heartbeat_front_addr":"172.21.15.28:6807/1850313547","state":["exists","up"]},{"osd":2,"uuid":"ad24917c-f5cb-4ca8-8a52-91d2256ff21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":3451325043}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":3451325043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":3451325043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":3451325043}]},"public_addr":"172.21.15.28:6809/3451325043","cluster_addr":"172.21.15.28:6810/3451325043","heartbeat_back_addr":"172.21.15.28:6812/3451325043","heartbeat_front_addr":"172.21.15.28:6811/3451325043","state":["exists","up"]},{"osd":3,"uuid":"4305caf1-0023-438f-8520-87814130b457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1506157250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1506157250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1506157250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1506157250}]},"public_addr":"172.21.15.28:6813/1506157250","cluster_addr":"172.21.15.28:6814/1506157250","heartbeat_back_addr":"172.21.15.28:6816/1506157250","heartbeat_front_addr":"172.21.15.28:6815/1506157250","state":["exists","up"]},{"osd":4,"uuid":"8180ec45-5260-4cdc-855c-be5302b1465c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6800","nonce":3063775514}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6801","nonce":3063775514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6803","nonce":3063775514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6802","nonce":3063775514}]},"public_addr":"172.21.15.134:6800/3063775514","cluster_addr":"172.21.15.134:6801/3063775514","heartbeat_back_addr":"172.21.15.134:6803/3063775514","heartbeat_front_addr":"172.21.15.134:6802/3063775514","state":["exists","up"]},{"osd":5,"uuid":"265992a1-46bc-4621-b9db-2b5ba1ee5e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6804","nonce":3668151179}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6805","nonce":3668151179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6807","nonce":3668151179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6806","nonce":3668151179}]},"public_addr":"172.21.15.134:6804/3668151179","cluster_addr":"172.21.15.134:6805/3668151179","heartbeat_back_addr":"172.21.15.134:6807/3668151179","heartbeat_front_addr":"172.21.15.134:6806/3668151179","state":["exists","up"]},{"osd":6,"uuid":"5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6808","nonce":2135224194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6809","nonce":2135224194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6811","nonce":2135224194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6810","nonce":2135224194}]},"public_addr":"172.21.15.134:6808/2135224194","cluster_addr":"172.21.15.134:6809/2135224194","heartbeat_back_addr":"172.21.15.134:6811/2135224194","heartbeat_front_addr":"172.21.15.134:6810/2135224194","state":["exists","up"]},{"osd":7,"uuid":"76f25ab9-af2f-4f45-afe3-13ece781d29c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6812","nonce":3758190298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6813","nonce":3758190298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6815","nonce":3758190298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6814","nonce":3758190298}]},"public_addr":"172.21.15.134:6812/3758190298","cluster_addr":"172.21.15.134:6813/3758190298","heartbeat_back_addr":"172.21.15.134:6815/3758190298","heartbeat_front_addr":"172.21.15.134:6814/3758190298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:59.189754+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:31.216170+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:39.827889+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:55:05.029867+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:09.901960+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:37.321258+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:55:05.294758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T14:54:21.111874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2354623613":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/2838206829":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/1249955842":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1446120238":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/429160846":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/997592180":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/2767265670":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1590506004":"2024-03-31T14:49:57.293695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 10 librados: watch_flush enter 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 10 librados: watch_flush exit 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 20 client.24529.objecter shutdown clearing up homeless session... 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 10 client.24529.objecter successfully canceled tick 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.28:6800/3720192247 conn(0x7f71ac070450 legacy=0x7f71ac072910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.416+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 >> v1:172.21.15.134:6789/0 conn(0x7f71c410aef0 legacy=0x7f71c4074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.417+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 shutdown_connections 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.417+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 >> 172.21.15.28:0/2851743394 conn(0x7f71c40fdb10 msgr2=0x7f71c4111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.417+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 shutdown_connections 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.417+0000 7f71ca1e4700 1 -- 172.21.15.28:0/2851743394 wait complete. 2024-03-30T14:55:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:15.417+0000 7f71ca1e4700 1 librados: shutdown 2024-03-30T14:55:15.429 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:14.985+0000 7effd61b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T14:55:15.429 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:15.222+0000 7effd61b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:55:15.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2851743394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:15.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2851743394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:15.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:15 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:15.535+0000 7f6c4daf8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T14:55:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2851743394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T14:55:16.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T14:55:16.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T14:55:16.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T14:55:16.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T14:55:16.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T14:55:16.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T14:55:16.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T14:55:16.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T14:55:16.430 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:16.028+0000 7effd61b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:55:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.644 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:16.900 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:16.582+0000 7f6c4daf8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T14:55:16.914 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:16.598+0000 7effd61b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:55:16.915 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:16.696+0000 7effd61b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: Standby manager daemon x restarted 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: Standby manager daemon x started 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:55:17.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[29876]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: Standby manager daemon x restarted 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: Standby manager daemon x started 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:55:17.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:16 smithi028 ceph-mon[41614]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: Standby manager daemon x restarted 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: Standby manager daemon x started 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-mon[33714]: from='mgr.? v1:172.21.15.134:0/3971823640' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:16.913+0000 7effd61b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:55:17.180 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: [30/Mar/2024:14:55:16] ENGINE Bus STARTING 2024-03-30T14:55:17.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: CherryPy Checker: 2024-03-30T14:55:17.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: The Application mounted at '' has an empty config. 2024-03-30T14:55:17.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: 2024-03-30T14:55:17.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: [30/Mar/2024:14:55:17] ENGINE Serving on http://:::9283 2024-03-30T14:55:17.181 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x[35810]: [30/Mar/2024:14:55:17] ENGINE Bus STARTED 2024-03-30T14:55:17.466 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:17.225+0000 7f6c4daf8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T14:55:17.466 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:17.328+0000 7f6c4daf8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T14:55:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:17.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:17.561+0000 7f6c4daf8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T14:55:17.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:17] ENGINE Bus STARTING 2024-03-30T14:55:18.044 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: CherryPy Checker: 2024-03-30T14:55:18.044 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: The Application mounted at '' has an empty config. 2024-03-30T14:55:18.044 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T14:55:18.044 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:17] ENGINE Serving on http://:::9283 2024-03-30T14:55:18.044 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:17] ENGINE Bus STARTED 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: Active manager daemon y restarted 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: Activating manager daemon y 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: mgrmap e19: y(active, starting, since 0.00816619s), standbys: x 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:55:18.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: Manager daemon y is now available 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:18.047 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:18.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T14:55:18.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: Active manager daemon y restarted 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: Activating manager daemon y 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: mgrmap e19: y(active, starting, since 0.00816619s), standbys: x 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:55:18.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: Manager daemon y is now available 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:18.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:18.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:17 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:18.398 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:55:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: Active manager daemon y restarted 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: Activating manager daemon y 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: osdmap e86: 8 total, 8 up, 8 in 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: mgrmap e19: y(active, starting, since 0.00816619s), standbys: x 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T14:55:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T14:55:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: Manager daemon y is now available 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:18.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:17 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: [30/Mar/2024:14:55:18] ENGINE Bus STARTING 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: mgrmap e20: y(active, since 1.01084s), standbys: x 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: [30/Mar/2024:14:55:18] ENGINE Serving on https://172.21.15.28:7150 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: [30/Mar/2024:14:55:18] ENGINE Serving on http://172.21.15.28:8765 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[41614]: [30/Mar/2024:14:55:18] ENGINE Bus STARTED 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: [30/Mar/2024:14:55:18] ENGINE Bus STARTING 2024-03-30T14:55:19.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: mgrmap e20: y(active, since 1.01084s), standbys: x 2024-03-30T14:55:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: [30/Mar/2024:14:55:18] ENGINE Serving on https://172.21.15.28:7150 2024-03-30T14:55:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: [30/Mar/2024:14:55:18] ENGINE Serving on http://172.21.15.28:8765 2024-03-30T14:55:19.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:19 smithi028 ceph-mon[29876]: [30/Mar/2024:14:55:18] ENGINE Bus STARTED 2024-03-30T14:55:19.904 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: [30/Mar/2024:14:55:18] ENGINE Bus STARTING 2024-03-30T14:55:19.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: mgrmap e20: y(active, since 1.01084s), standbys: x 2024-03-30T14:55:19.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:19.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: [30/Mar/2024:14:55:18] ENGINE Serving on https://172.21.15.28:7150 2024-03-30T14:55:19.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: [30/Mar/2024:14:55:18] ENGINE Serving on http://172.21.15.28:8765 2024-03-30T14:55:19.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-mon[33714]: [30/Mar/2024:14:55:18] ENGINE Bus STARTED 2024-03-30T14:55:20.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:20.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[29876]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:20.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[29876]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T14:55:20.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[41614]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[41614]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T14:55:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:20 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:20.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:20 smithi134 ceph-mon[33714]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:20 smithi134 ceph-mon[33714]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-30T14:55:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:20 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[41614]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:21 smithi028 ceph-mon[29876]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T14:55:22.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:21 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:21 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:21 smithi134 ceph-mon[33714]: osdmap e87: 8 total, 8 up, 8 in 2024-03-30T14:55:22.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 3760M 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[41614]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:22.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 3760M 2024-03-30T14:55:22.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:22.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:22 smithi028 ceph-mon[29876]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 3760M 2024-03-30T14:55:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:22 smithi134 ceph-mon[33714]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.575+0000 7fbba2e0b700 1 -- 172.21.15.28:0/386102136 >> v1:172.21.15.28:6789/0 conn(0x7fbb9c103200 legacy=0x7fbb9c1035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.575+0000 7fbba2e0b700 1 -- 172.21.15.28:0/386102136 shutdown_connections 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.575+0000 7fbba2e0b700 1 -- 172.21.15.28:0/386102136 >> 172.21.15.28:0/386102136 conn(0x7fbb9c0fc9d0 msgr2=0x7fbb9c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.575+0000 7fbba2e0b700 1 -- 172.21.15.28:0/386102136 shutdown_connections 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.575+0000 7fbba2e0b700 1 -- 172.21.15.28:0/386102136 wait complete. 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 librados: starting msgr at 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 librados: starting objecter 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 Processor -- start 2024-03-30T14:55:23.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 -- start start 2024-03-30T14:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 librados: setting wanted keys 2024-03-30T14:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 librados: calling monclient init 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb9c1a6eb0 con 0x7fbb9c10eb20 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb9c1a80d0 con 0x7fbb9c10b010 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba2e0b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb9c1a9310 con 0x7fbb9c103200 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba0ba7700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbb9c103200 0x7fbb9c10a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57220/0 (socket says 172.21.15.28:57220) 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.576+0000 7fbba0ba7700 1 -- 172.21.15.28:0/1204334587 learned_addr learned my addr 172.21.15.28:0/1204334587 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter ms_handle_connect 0x7fbb9c103200 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter ms_handle_connect 0x7fbb9c10eb20 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter ms_handle_connect 0x7fbb9c10b010 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773382219 0 0) 0x7fbb9c1a9310 con 0x7fbb9c103200 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb8c003aa0 con 0x7fbb9c103200 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3491995826 0 0) 0x7fbb9c1a6eb0 con 0x7fbb9c10eb20 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb9c1a9310 con 0x7fbb9c10eb20 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2132438794 0 0) 0x7fbb9c1a80d0 con 0x7fbb9c10b010 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb9c1a6eb0 con 0x7fbb9c10b010 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2654042227 0 0) 0x7fbb8c003aa0 con 0x7fbb9c103200 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb9c1a80d0 con 0x7fbb9c103200 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbb90001970 con 0x7fbb9c103200 2024-03-30T14:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3429083608 0 0) 0x7fbb9c1a9310 con 0x7fbb9c10eb20 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb8c003aa0 con 0x7fbb9c10eb20 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1020210232 0 0) 0x7fbb9c1a6eb0 con 0x7fbb9c10b010 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb9c1a9310 con 0x7fbb9c10b010 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbb94001db0 con 0x7fbb9c10eb20 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbb880017b0 con 0x7fbb9c10b010 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3912250746 0 0) 0x7fbb9c1a80d0 con 0x7fbb9c103200 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 >> v1:172.21.15.134:6789/0 conn(0x7fbb9c10b010 legacy=0x7fbb9c1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 >> v1:172.21.15.28:6789/0 conn(0x7fbb9c10eb20 legacy=0x7fbb9c1a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c1aa550 con 0x7fbb9c103200 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.577+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c1a70e0 con 0x7fbb9c103200 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbb90003b20 con 0x7fbb9c103200 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbb900050c0 con 0x7fbb9c103200 2024-03-30T14:55:23.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 10 client.34529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb9c1a75a0 con 0x7fbb9c103200 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 1 librados: init done 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 10 client.34529.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 15 client.34529.objecter _session_command_op_assign -1 1 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 20 client.34529.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 10 client.34529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.578+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbb9c06add0 con 0x7fbb9c103200 2024-03-30T14:55:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.579+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbb90003680 con 0x7fbb9c103200 2024-03-30T14:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.579+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(87..87 src has 14..87) ==== 6259+0+0 (unknown 3687459424 0 0) 0x7fbb90089030 con 0x7fbb9c103200 2024-03-30T14:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.579+0000 7fbb99ffb700 10 client.34529.objecter ms_dispatch 0x7fbb9c001550 osd_map(87..87 src has 14..87) 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.579+0000 7fbb99ffb700 3 client.34529.objecter handle_osd_map got epochs [87,87] > 0 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.579+0000 7fbb99ffb700 3 client.34529.objecter handle_osd_map decoding full epoch 87 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 10 client.34529.objecter checking command 1 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter _get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 15 client.34529.objecter _session_command_op_remove -1 1 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter _get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 15 client.34529.objecter _session_command_op_assign 5 1 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 4 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 10 client.34529.objecter _send_command 1 2024-03-30T14:55:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.134:6804/3668151179 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb90001bb0 con 0x7fbb8c07ab90 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 20 client.34529.objecter dump_active .. 0 homeless 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (unknown 3495163948 0 0) 0x7fbb90089500 con 0x7fbb9c103200 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 10 client.34529.objecter ms_handle_connect 0x7fbb8c0704b0 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.580+0000 7fbb99ffb700 10 client.34529.objecter ms_handle_connect 0x7fbb8c07ab90 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.581+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbb90001bb0 con 0x7fbb8c07ab90 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.581+0000 7fbb99ffb700 10 client.34529.objecter ms_dispatch 0x7fbb9c001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.581+0000 7fbb99ffb700 10 client.34529.objecter _finish_command 1 = system:0 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.581+0000 7fbb99ffb700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 4 2024-03-30T14:55:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.581+0000 7fbb99ffb700 15 client.34529.objecter _session_command_op_remove 5 1 2024-03-30T14:55:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 10 client.34529.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 15 client.34529.objecter _session_command_op_assign -1 2 2024-03-30T14:55:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 20 client.34529.objecter _get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 20 client.34529.objecter _get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 15 client.34529.objecter _session_command_op_remove -1 2 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 20 client.34529.objecter get_session s=0x7fbb8c07a4e0 osd=5 3 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 15 client.34529.objecter _session_command_op_assign 5 2 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 4 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 10 client.34529.objecter _send_command 2 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.597+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 --> v1:172.21.15.134:6804/3668151179 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb9c068b30 con 0x7fbb8c07ab90 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbb99ffb700 1 -- 172.21.15.28:0/1204334587 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1769387110) 0x7fbb9c068b30 con 0x7fbb8c07ab90 2024-03-30T14:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbb99ffb700 10 client.34529.objecter ms_dispatch 0x7fbb9c001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:23.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbb99ffb700 10 client.34529.objecter _finish_command 2 = system:0 2024-03-30T14:55:23.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbb99ffb700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 4 2024-03-30T14:55:23.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbb99ffb700 15 client.34529.objecter _session_command_op_remove 5 2 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbba2e0b700 10 librados: watch_flush enter 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.598+0000 7fbba2e0b700 10 librados: watch_flush exit 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 10 client.34529.objecter close_session for osd.5 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 >> v1:172.21.15.134:6804/3668151179 conn(0x7fbb8c07ab90 legacy=0x7fbb8c07d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 20 client.34529.objecter put_session s=0x7fbb8c07a4e0 osd=5 1 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 20 client.34529.objecter shutdown clearing up homeless session... 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 10 client.34529.objecter successfully canceled tick 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbb8c0704b0 legacy=0x7fbb8c072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 >> v1:172.21.15.28:6790/0 conn(0x7fbb9c103200 legacy=0x7fbb9c10a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba13a8700 1 -- 172.21.15.28:0/1204334587 reap_dead start 2024-03-30T14:55:23.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 shutdown_connections 2024-03-30T14:55:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 >> 172.21.15.28:0/1204334587 conn(0x7fbb9c0fc9d0 msgr2=0x7fbb9c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 shutdown_connections 2024-03-30T14:55:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 -- 172.21.15.28:0/1204334587 wait complete. 2024-03-30T14:55:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.599+0000 7fbba2e0b700 1 librados: shutdown 2024-03-30T14:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- 172.21.15.28:0/2442245768 >> v1:172.21.15.28:6790/0 conn(0x7fd780109df0 legacy=0x7fd78010c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- 172.21.15.28:0/2442245768 shutdown_connections 2024-03-30T14:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- 172.21.15.28:0/2442245768 >> 172.21.15.28:0/2442245768 conn(0x7fd7800fca10 msgr2=0x7fd7800fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- 172.21.15.28:0/2442245768 shutdown_connections 2024-03-30T14:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- 172.21.15.28:0/2442245768 wait complete. 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 librados: starting msgr at 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 librados: starting objecter 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 Processor -- start 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 -- start start 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 librados: setting wanted keys 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.810+0000 7fd7885ab700 1 librados: calling monclient init 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd7885ab700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd7801a9410 con 0x7fd780101160 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd7885ab700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd7801aa630 con 0x7fd78010d990 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd7885ab700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd7801ab870 con 0x7fd780109df0 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd786347700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd780101160 0x7fd7801064e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43370/0 (socket says 172.21.15.28:43370) 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd786347700 1 -- 172.21.15.28:0/3931230510 learned_addr learned my addr 172.21.15.28:0/3931230510 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter ms_handle_connect 0x7fd780109df0 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter ms_handle_connect 0x7fd780101160 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter ms_handle_connect 0x7fd78010d990 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1294010871 0 0) 0x7fd7801ab870 con 0x7fd780109df0 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7680031f0 con 0x7fd780109df0 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220464078 0 0) 0x7fd7801a9410 con 0x7fd780101160 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7801ab870 con 0x7fd780101160 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008185585 0 0) 0x7fd7801aa630 con 0x7fd78010d990 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7801a9410 con 0x7fd78010d990 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1837105472 0 0) 0x7fd7680031f0 con 0x7fd780109df0 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7801aa630 con 0x7fd780109df0 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd774004470 con 0x7fd780109df0 2024-03-30T14:55:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1817882202 0 0) 0x7fd7801ab870 con 0x7fd780101160 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7680031f0 con 0x7fd780101160 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd7780017f0 con 0x7fd780101160 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3991079413 0 0) 0x7fd7801a9410 con 0x7fd78010d990 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7801ab870 con 0x7fd78010d990 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd7700015b0 con 0x7fd78010d990 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3606177483 0 0) 0x7fd7801aa630 con 0x7fd780109df0 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 >> v1:172.21.15.134:6789/0 conn(0x7fd78010d990 legacy=0x7fd7801a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 >> v1:172.21.15.28:6789/0 conn(0x7fd780101160 legacy=0x7fd7801064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7801acab0 con 0x7fd780109df0 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.811+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7801a9640 con 0x7fd780109df0 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 10 client.34535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7801a9b00 con 0x7fd780109df0 2024-03-30T14:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 1 librados: init done 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd774001ab0 con 0x7fd780109df0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd774004d60 con 0x7fd780109df0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 10 client.34535.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 15 client.34535.objecter _session_command_op_assign -1 1 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 20 client.34535.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 10 client.34535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.812+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd76c002950 con 0x7fd780109df0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.813+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd77401b620 con 0x7fd780109df0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.813+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(87..87 src has 14..87) ==== 6259+0+0 (unknown 3687459424 0 0) 0x7fd7740880b0 con 0x7fd780109df0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.813+0000 7fd77f7fe700 10 client.34535.objecter ms_dispatch 0x7fd780001550 osd_map(87..87 src has 14..87) 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.813+0000 7fd77f7fe700 3 client.34535.objecter handle_osd_map got epochs [87,87] > 0 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.813+0000 7fd77f7fe700 3 client.34535.objecter handle_osd_map decoding full epoch 87 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 10 client.34535.objecter checking command 1 2024-03-30T14:55:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter _get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 15 client.34535.objecter _session_command_op_remove -1 1 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter _get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 15 client.34535.objecter _session_command_op_assign 6 1 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 4 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 10 client.34535.objecter _send_command 1 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.134:6808/2135224194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd774001080 con 0x7fd76807abc0 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 20 client.34535.objecter dump_active .. 0 homeless 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 10 client.34535.objecter ms_handle_connect 0x7fd7680704e0 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.814+0000 7fd77f7fe700 10 client.34535.objecter ms_handle_connect 0x7fd76807abc0 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.815+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd774001080 con 0x7fd76807abc0 2024-03-30T14:55:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.815+0000 7fd77f7fe700 10 client.34535.objecter ms_dispatch 0x7fd780001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.815+0000 7fd77f7fe700 10 client.34535.objecter _finish_command 1 = system:0 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.815+0000 7fd77f7fe700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 4 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.815+0000 7fd77f7fe700 15 client.34535.objecter _session_command_op_remove 6 1 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.816+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (unknown 3495163948 0 0) 0x7fd774053600 con 0x7fd780109df0 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 10 client.34535.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 15 client.34535.objecter _session_command_op_assign -1 2 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 20 client.34535.objecter _get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 20 client.34535.objecter _get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 15 client.34535.objecter _session_command_op_remove -1 2 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 20 client.34535.objecter get_session s=0x7fd76807a510 osd=6 3 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 15 client.34535.objecter _session_command_op_assign 6 2 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 4 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 10 client.34535.objecter _send_command 2 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 --> v1:172.21.15.134:6808/2135224194 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd76c001220 con 0x7fd76807abc0 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd77f7fe700 1 -- 172.21.15.28:0/3931230510 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 138119153) 0x7fd76c001220 con 0x7fd76807abc0 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd77f7fe700 10 client.34535.objecter ms_dispatch 0x7fd780001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd77f7fe700 10 client.34535.objecter _finish_command 2 = system:0 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd77f7fe700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 4 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.833+0000 7fd77f7fe700 15 client.34535.objecter _session_command_op_remove 6 2 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 10 librados: watch_flush enter 2024-03-30T14:55:23.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 10 librados: watch_flush exit 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 10 client.34535.objecter close_session for osd.6 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 >> v1:172.21.15.134:6808/2135224194 conn(0x7fd76807abc0 legacy=0x7fd76807d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 20 client.34535.objecter put_session s=0x7fd76807a510 osd=6 1 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 20 client.34535.objecter shutdown clearing up homeless session... 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 10 client.34535.objecter successfully canceled tick 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd7680704e0 legacy=0x7fd7680729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 >> v1:172.21.15.28:6790/0 conn(0x7fd780109df0 legacy=0x7fd7801a4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd786b48700 1 -- 172.21.15.28:0/3931230510 reap_dead start 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 shutdown_connections 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 >> 172.21.15.28:0/3931230510 conn(0x7fd7800fca10 msgr2=0x7fd780110600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 shutdown_connections 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 -- 172.21.15.28:0/3931230510 wait complete. 2024-03-30T14:55:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:23.834+0000 7fd7885ab700 1 librados: shutdown 2024-03-30T14:55:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:24 smithi134 ceph-mon[33714]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:25.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:24 smithi028 ceph-mon[41614]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:25.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:24 smithi028 ceph-mon[29876]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:55:26.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:25 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:26.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:25 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:26.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:25 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:26.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:25 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:25 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:25 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 -- 172.21.15.28:0/334157135 >> v1:172.21.15.28:6789/0 conn(0x7fdd5c10aec0 legacy=0x7fdd5c10d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 -- 172.21.15.28:0/334157135 shutdown_connections 2024-03-30T14:55:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 -- 172.21.15.28:0/334157135 >> 172.21.15.28:0/334157135 conn(0x7fdd5c0fdb00 msgr2=0x7fdd5c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 -- 172.21.15.28:0/334157135 shutdown_connections 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 -- 172.21.15.28:0/334157135 wait complete. 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 librados: starting msgr at 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.181+0000 7fdd610e0700 1 librados: starting objecter 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 Processor -- start 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 -- start start 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 librados: setting wanted keys 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 librados: calling monclient init 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd5c1a9330 con 0x7fdd5c102230 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd5c1aa550 con 0x7fdd5c10aec0 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd610e0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd5c1ab790 con 0x7fdd5c10ea60 2024-03-30T14:55:26.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd5ad9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd5c102230 0x7fdd5c1073d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43386/0 (socket says 172.21.15.28:43386) 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd5b59e700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdd5c10ea60 0x7fdd5c1a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57262/0 (socket says 172.21.15.28:57262) 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd5ad9d700 1 -- 172.21.15.28:0/3935424904 learned_addr learned my addr 172.21.15.28:0/3935424904 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c102230 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c10ea60 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259733926 0 0) 0x7fdd5c1a9330 con 0x7fdd5c102230 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd480031f0 con 0x7fdd5c102230 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2389041436 0 0) 0x7fdd5c1ab790 con 0x7fdd5c10ea60 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.182+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd5c1a9330 con 0x7fdd5c10ea60 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 10 client.?.objecter ms_handle_connect 0x7fdd5c10aec0 2024-03-30T14:55:26.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2198672526 0 0) 0x7fdd480031f0 con 0x7fdd5c102230 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd5c1ab790 con 0x7fdd5c102230 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1847658587 0 0) 0x7fdd5c1a9330 con 0x7fdd5c10ea60 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd480031f0 con 0x7fdd5c10ea60 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd440019b0 con 0x7fdd5c102230 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618458413 0 0) 0x7fdd5c1aa550 con 0x7fdd5c10aec0 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd5c1a9330 con 0x7fdd5c10aec0 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd54001b30 con 0x7fdd5c10ea60 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2325245474 0 0) 0x7fdd5c1ab790 con 0x7fdd5c102230 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 >> v1:172.21.15.28:6790/0 conn(0x7fdd5c10ea60 legacy=0x7fdd5c1a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 >> v1:172.21.15.134:6789/0 conn(0x7fdd5c10aec0 legacy=0x7fdd5c1a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c1ac9d0 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5c1ab9c0 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd610e0700 10 client.14652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd5c1abe80 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd610e0700 1 librados: init done 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdd44001c40 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.183+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd44005cc0 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd610e0700 10 client.14652.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd610e0700 15 client.14652.objecter _session_command_op_assign -1 1 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd610e0700 20 client.14652.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd610e0700 10 client.14652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdd3c002920 con 0x7fdd5c102230 2024-03-30T14:55:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.184+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdd4401c500 con 0x7fdd5c102230 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(88..88 src has 14..88) ==== 6259+0+0 (unknown 3158528336 0 0) 0x7fdd440890b0 con 0x7fdd5c102230 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 10 client.14652.objecter ms_dispatch 0x7fdd5c001550 osd_map(88..88 src has 14..88) 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 3 client.14652.objecter handle_osd_map got epochs [88,88] > 0 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 3 client.14652.objecter handle_osd_map decoding full epoch 88 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 10 client.14652.objecter checking command 1 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter _get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 15 client.14652.objecter _session_command_op_remove -1 1 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter _get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 15 client.14652.objecter _session_command_op_assign 1 1 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 4 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 10 client.14652.objecter _send_command 1 2024-03-30T14:55:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6805/1850313547 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd4807e980 con 0x7fdd4807ac70 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 20 client.14652.objecter dump_active .. 0 homeless 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (unknown 2353666835 0 0) 0x7fdd44089500 con 0x7fdd5c102230 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 10 client.14652.objecter ms_handle_connect 0x7fdd48070490 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.185+0000 7fdd53fff700 10 client.14652.objecter ms_handle_connect 0x7fdd4807ac70 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.186+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdd4807e980 con 0x7fdd4807ac70 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.186+0000 7fdd53fff700 10 client.14652.objecter ms_dispatch 0x7fdd5c001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.186+0000 7fdd53fff700 10 client.14652.objecter _finish_command 1 = system:0 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.186+0000 7fdd53fff700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 4 2024-03-30T14:55:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.186+0000 7fdd53fff700 15 client.14652.objecter _session_command_op_remove 1 1 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 10 client.14652.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 15 client.14652.objecter _session_command_op_assign -1 2 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 20 client.14652.objecter _get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 20 client.14652.objecter _get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 15 client.14652.objecter _session_command_op_remove -1 2 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 20 client.14652.objecter get_session s=0x7fdd4807a5c0 osd=1 3 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 15 client.14652.objecter _session_command_op_assign 1 2 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 4 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 10 client.14652.objecter _send_command 2 2024-03-30T14:55:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.202+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdd3c0011f0 con 0x7fdd4807ac70 2024-03-30T14:55:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd53fff700 1 -- 172.21.15.28:0/3935424904 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3707702002) 0x7fdd3c0011f0 con 0x7fdd4807ac70 2024-03-30T14:55:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd53fff700 10 client.14652.objecter ms_dispatch 0x7fdd5c001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd53fff700 10 client.14652.objecter _finish_command 2 = system:0 2024-03-30T14:55:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd53fff700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 4 2024-03-30T14:55:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd53fff700 15 client.14652.objecter _session_command_op_remove 1 2 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 10 librados: watch_flush enter 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 10 librados: watch_flush exit 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 10 client.14652.objecter close_session for osd.1 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 >> v1:172.21.15.28:6805/1850313547 conn(0x7fdd4807ac70 legacy=0x7fdd4807d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 20 client.14652.objecter put_session s=0x7fdd4807a5c0 osd=1 1 2024-03-30T14:55:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 20 client.14652.objecter shutdown clearing up homeless session... 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 10 client.14652.objecter successfully canceled tick 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdd48070490 legacy=0x7fdd48072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 >> v1:172.21.15.28:6789/0 conn(0x7fdd5c102230 legacy=0x7fdd5c1073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd5b59e700 1 -- 172.21.15.28:0/3935424904 reap_dead start 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 shutdown_connections 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 >> 172.21.15.28:0/3935424904 conn(0x7fdd5c0fdb00 msgr2=0x7fdd5c111bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.203+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 shutdown_connections 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.204+0000 7fdd610e0700 1 -- 172.21.15.28:0/3935424904 wait complete. 2024-03-30T14:55:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:26.204+0000 7fdd610e0700 1 librados: shutdown 2024-03-30T14:55:26.964 INFO:teuthology.orchestra.run.smithi028.stdout:223338299411 2024-03-30T14:55:26.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T14:55:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:26 smithi134 ceph-mon[33714]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T14:55:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:26 smithi134 ceph-mon[33714]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T14:55:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:26 smithi134 ceph-mon[33714]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T14:55:27.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[41614]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T14:55:27.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[41614]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T14:55:27.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[41614]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T14:55:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[29876]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-30T14:55:27.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[29876]: osdmap e88: 8 total, 8 up, 8 in 2024-03-30T14:55:27.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:26 smithi028 ceph-mon[29876]: osdmap e89: 8 total, 8 up, 8 in 2024-03-30T14:55:28.575 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:55:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:55:28.605 INFO:teuthology.orchestra.run.smithi028.stdout:176093659160 2024-03-30T14:55:28.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T14:55:29.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:28 smithi028 ceph-mon[29876]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-30T14:55:29.129 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:28 smithi028 ceph-mon[41614]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-30T14:55:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:28 smithi134 ceph-mon[33714]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-30T14:55:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- 172.21.15.28:0/1947001686 >> v1:172.21.15.28:6790/0 conn(0x7fafe8100ac0 legacy=0x7fafe8100ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- 172.21.15.28:0/1947001686 shutdown_connections 2024-03-30T14:55:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- 172.21.15.28:0/1947001686 >> 172.21.15.28:0/1947001686 conn(0x7fafe80761e0 msgr2=0x7fafe80765f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- 172.21.15.28:0/1947001686 shutdown_connections 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- 172.21.15.28:0/1947001686 wait complete. 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 librados: starting msgr at 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 librados: starting objecter 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 Processor -- start 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- start start 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 librados: setting wanted keys 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 librados: calling monclient init 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe81a70c0 con 0x7fafe810eac0 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe81a82e0 con 0x7fafe810af50 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.814+0000 7fafef0dd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fafe81a9520 con 0x7fafe8100ac0 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafece79700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fafe8100ac0 0x7fafe81074b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41690/0 (socket says 172.21.15.28:41690) 2024-03-30T14:55:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafece79700 1 -- 172.21.15.28:0/3716587634 learned_addr learned my addr 172.21.15.28:0/3716587634 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter ms_handle_connect 0x7fafe810eac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter ms_handle_connect 0x7fafe8100ac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764007388 0 0) 0x7fafe81a9520 con 0x7fafe8100ac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd8003aa0 con 0x7fafe8100ac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter ms_handle_connect 0x7fafe810af50 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632419595 0 0) 0x7fafe81a70c0 con 0x7fafe810eac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe81a9520 con 0x7fafe810eac0 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2284821614 0 0) 0x7fafe81a82e0 con 0x7fafe810af50 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe81a70c0 con 0x7fafe810af50 2024-03-30T14:55:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1829781625 0 0) 0x7fafd8003aa0 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafe81a82e0 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fafdc001550 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2029278265 0 0) 0x7fafe81a9520 con 0x7fafe810eac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafd8003aa0 con 0x7fafe810eac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fafe0001b40 con 0x7fafe810eac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3548836475 0 0) 0x7fafe81a70c0 con 0x7fafe810af50 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafe81a9520 con 0x7fafe810af50 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fafd4001830 con 0x7fafe810af50 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1020201259 0 0) 0x7fafe81a82e0 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 >> v1:172.21.15.134:6789/0 conn(0x7fafe810af50 legacy=0x7fafe81a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 >> v1:172.21.15.28:6789/0 conn(0x7fafe810eac0 legacy=0x7fafe81a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe81aa760 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fafe81a9750 con 0x7fafe8100ac0 2024-03-30T14:55:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fafdc003ad0 con 0x7fafe8100ac0 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.815+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fafdc005070 con 0x7fafe8100ac0 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 10 client.34553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fafe81a9c10 con 0x7fafe8100ac0 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 1 librados: init done 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 10 client.34553.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 15 client.34553.objecter _session_command_op_assign -1 1 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 20 client.34553.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 10 client.34553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.816+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fafe806a130 con 0x7fafe8100ac0 2024-03-30T14:55:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.817+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fafdc001780 con 0x7fafe8100ac0 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.817+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7fafdc089210 con 0x7fafe8100ac0 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.817+0000 7fafe5ffb700 10 client.34553.objecter ms_dispatch 0x7fafe8001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.817+0000 7fafe5ffb700 3 client.34553.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.817+0000 7fafe5ffb700 3 client.34553.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 10 client.34553.objecter checking command 1 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter _get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 15 client.34553.objecter _session_command_op_remove -1 1 2024-03-30T14:55:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter _get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 15 client.34553.objecter _session_command_op_assign 4 1 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 4 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 10 client.34553.objecter _send_command 1 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.134:6800/3063775514 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafd807e8e0 con 0x7fafd807abd0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 20 client.34553.objecter dump_active .. 0 homeless 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (unknown 2118522605 0 0) 0x7fafdc089680 con 0x7fafe8100ac0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 10 client.34553.objecter ms_handle_connect 0x7fafd80704f0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.818+0000 7fafe5ffb700 10 client.34553.objecter ms_handle_connect 0x7fafd807abd0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.819+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd807e8e0 con 0x7fafd807abd0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.819+0000 7fafe5ffb700 10 client.34553.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.819+0000 7fafe5ffb700 10 client.34553.objecter _finish_command 1 = system:0 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.819+0000 7fafe5ffb700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 4 2024-03-30T14:55:29.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.819+0000 7fafe5ffb700 15 client.34553.objecter _session_command_op_remove 4 1 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 client.34553.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 15 client.34553.objecter _session_command_op_assign -1 2 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter _get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter _get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 15 client.34553.objecter _session_command_op_remove -1 2 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter get_session s=0x7fafd807a520 osd=4 3 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 15 client.34553.objecter _session_command_op_assign 4 2 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 4 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 client.34553.objecter _send_command 2 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 --> v1:172.21.15.134:6800/3063775514 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fafe8068b30 con 0x7fafd807abd0 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafe5ffb700 1 -- 172.21.15.28:0/3716587634 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3051186307) 0x7fafe8068b30 con 0x7fafd807abd0 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafe5ffb700 10 client.34553.objecter ms_dispatch 0x7fafe8001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafe5ffb700 10 client.34553.objecter _finish_command 2 = system:0 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafe5ffb700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 4 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafe5ffb700 15 client.34553.objecter _session_command_op_remove 4 2 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 librados: watch_flush enter 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 librados: watch_flush exit 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 client.34553.objecter close_session for osd.4 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 >> v1:172.21.15.134:6800/3063775514 conn(0x7fafd807abd0 legacy=0x7fafd807d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter put_session s=0x7fafd807a520 osd=4 1 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 20 client.34553.objecter shutdown clearing up homeless session... 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.836+0000 7fafef0dd700 10 client.34553.objecter successfully canceled tick 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 >> v1:172.21.15.28:6800/3011431093 conn(0x7fafd80704f0 legacy=0x7fafd80729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 >> v1:172.21.15.28:6790/0 conn(0x7fafe8100ac0 legacy=0x7fafe81074b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafed67a700 1 -- 172.21.15.28:0/3716587634 reap_dead start 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 shutdown_connections 2024-03-30T14:55:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 >> 172.21.15.28:0/3716587634 conn(0x7fafe80761e0 msgr2=0x7fafe8112270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 shutdown_connections 2024-03-30T14:55:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 -- 172.21.15.28:0/3716587634 wait complete. 2024-03-30T14:55:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.837+0000 7fafef0dd700 1 librados: shutdown 2024-03-30T14:55:30.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:29 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.696+0000 7f226c127700 1 -- 172.21.15.28:0/1001249722 >> v1:172.21.15.28:6790/0 conn(0x7f226410aef0 legacy=0x7f226410d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.697+0000 7f226c127700 1 -- 172.21.15.28:0/1001249722 shutdown_connections 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.697+0000 7f226c127700 1 -- 172.21.15.28:0/1001249722 >> 172.21.15.28:0/1001249722 conn(0x7f22640fdb10 msgr2=0x7f22640fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.697+0000 7f226c127700 1 -- 172.21.15.28:0/1001249722 shutdown_connections 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.697+0000 7f226c127700 1 -- 172.21.15.28:0/1001249722 wait complete. 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 librados: starting msgr at 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 librados: starting objecter 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 Processor -- start 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 -- start start 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 librados: setting wanted keys 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 librados: calling monclient init 2024-03-30T14:55:30.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22641a93a0 con 0x7f226410ea90 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22641aa5c0 con 0x7f226410aef0 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.698+0000 7f226c127700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22641ab800 con 0x7f2264102260 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2269ec3700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2264102260 0x7f22641075c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41678/0 (socket says 172.21.15.28:41678) 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2269ec3700 1 -- 172.21.15.28:0/549467011 learned_addr learned my addr 172.21.15.28:0/549467011 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter ms_handle_connect 0x7f2264102260 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter ms_handle_connect 0x7f226410ea90 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter ms_handle_connect 0x7f226410aef0 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815500099 0 0) 0x7f22641a93a0 con 0x7f226410ea90 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22500039c0 con 0x7f226410ea90 2024-03-30T14:55:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373455485 0 0) 0x7f22641ab800 con 0x7f2264102260 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22641a93a0 con 0x7f2264102260 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3582193897 0 0) 0x7f22641aa5c0 con 0x7f226410aef0 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22641ab800 con 0x7f226410aef0 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2411945348 0 0) 0x7f22500039c0 con 0x7f226410ea90 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22641aa5c0 con 0x7f226410ea90 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f225c001c30 con 0x7f226410ea90 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 498208841 0 0) 0x7f22641a93a0 con 0x7f2264102260 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22500039c0 con 0x7f2264102260 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2258001890 con 0x7f2264102260 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 644084426 0 0) 0x7f22641ab800 con 0x7f226410aef0 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22641a93a0 con 0x7f226410aef0 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2548520503 0 0) 0x7f22641aa5c0 con 0x7f226410ea90 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 >> v1:172.21.15.28:6790/0 conn(0x7f2264102260 legacy=0x7f22641075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 >> v1:172.21.15.134:6789/0 conn(0x7f226410aef0 legacy=0x7f22641a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22641aca40 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.699+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22641a95d0 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22641a9a90 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 1 librados: init done 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f225c003f40 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f225c004ea0 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 10 client.14658.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 15 client.14658.objecter _session_command_op_assign -1 1 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 20 client.14658.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.700+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2248002920 con 0x7f226410ea90 2024-03-30T14:55:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.701+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f225c01b520 con 0x7f226410ea90 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f225c088080 con 0x7f226410ea90 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 10 client.14658.objecter ms_dispatch 0x7f2264001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 3 client.14658.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 3 client.14658.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 10 client.14658.objecter checking command 1 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter _get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 15 client.14658.objecter _session_command_op_remove -1 1 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter _get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 15 client.14658.objecter _session_command_op_assign 3 1 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 4 2024-03-30T14:55:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 10 client.14658.objecter _send_command 1 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6813/1506157250 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f225007e990 con 0x7f225007ac80 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 20 client.14658.objecter dump_active .. 0 homeless 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (unknown 2118522605 0 0) 0x7f225c0884f0 con 0x7f226410ea90 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 10 client.14658.objecter ms_handle_connect 0x7f2250070410 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.702+0000 7f2256ffd700 10 client.14658.objecter ms_handle_connect 0x7f225007ac80 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.703+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f225007e990 con 0x7f225007ac80 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.703+0000 7f2256ffd700 10 client.14658.objecter ms_dispatch 0x7f2264001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.703+0000 7f2256ffd700 10 client.14658.objecter _finish_command 1 = system:0 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.703+0000 7f2256ffd700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 4 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.703+0000 7f2256ffd700 15 client.14658.objecter _session_command_op_remove 3 1 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 10 client.14658.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 15 client.14658.objecter _session_command_op_assign -1 2 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 20 client.14658.objecter _get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 20 client.14658.objecter _get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 15 client.14658.objecter _session_command_op_remove -1 2 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 20 client.14658.objecter get_session s=0x7f225007a5d0 osd=3 3 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 15 client.14658.objecter _session_command_op_assign 3 2 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 4 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 10 client.14658.objecter _send_command 2 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.737+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 --> v1:172.21.15.28:6813/1506157250 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22480011f0 con 0x7f225007ac80 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f2256ffd700 1 -- 172.21.15.28:0/549467011 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 2426792065) 0x7f22480011f0 con 0x7f225007ac80 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f2256ffd700 10 client.14658.objecter ms_dispatch 0x7f2264001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f2256ffd700 10 client.14658.objecter _finish_command 2 = system:0 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f2256ffd700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 4 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f2256ffd700 15 client.14658.objecter _session_command_op_remove 3 2 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 10 librados: watch_flush enter 2024-03-30T14:55:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 10 librados: watch_flush exit 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 10 client.14658.objecter close_session for osd.3 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 >> v1:172.21.15.28:6813/1506157250 conn(0x7f225007ac80 legacy=0x7f225007d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 20 client.14658.objecter put_session s=0x7f225007a5d0 osd=3 1 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 20 client.14658.objecter shutdown clearing up homeless session... 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 10 client.14658.objecter successfully canceled tick 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2250070410 legacy=0x7f22500728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 >> v1:172.21.15.28:6789/0 conn(0x7f226410ea90 legacy=0x7f22641a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226a6c4700 1 -- 172.21.15.28:0/549467011 reap_dead start 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 shutdown_connections 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 >> 172.21.15.28:0/549467011 conn(0x7f22640fdb10 msgr2=0x7f2264111c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.738+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 shutdown_connections 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.739+0000 7f226c127700 1 -- 172.21.15.28:0/549467011 wait complete. 2024-03-30T14:55:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:29.739+0000 7f226c127700 1 librados: shutdown 2024-03-30T14:55:31.027 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:30 smithi028 ceph-mon[41614]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T14:55:31.027 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:30 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:31.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:30 smithi028 ceph-mon[29876]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T14:55:31.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:30 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:31.054 INFO:teuthology.orchestra.run.smithi028.stdout:115964117028 2024-03-30T14:55:31.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T14:55:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:30 smithi134 ceph-mon[33714]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-30T14:55:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:30 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 -- 172.21.15.28:0/2495905828 >> v1:172.21.15.28:6790/0 conn(0x7f9238103190 legacy=0x7f9238103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 -- 172.21.15.28:0/2495905828 shutdown_connections 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 -- 172.21.15.28:0/2495905828 >> 172.21.15.28:0/2495905828 conn(0x7f92380fc9e0 msgr2=0x7f92380fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 -- 172.21.15.28:0/2495905828 shutdown_connections 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 -- 172.21.15.28:0/2495905828 wait complete. 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 librados: starting msgr at 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 librados: starting objecter 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.260+0000 7f923de1b700 1 Processor -- start 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 -- start start 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 librados: setting wanted keys 2024-03-30T14:55:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 librados: calling monclient init 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92381a91a0 con 0x7f9238103190 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92381aa3c0 con 0x7f923810afa0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f923de1b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92381ab5e0 con 0x7f923810eab0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f92377fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9238103190 0x7f923810a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48422/0 (socket says 172.21.15.28:48422) 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f92377fe700 1 -- 172.21.15.28:0/1671489532 learned_addr learned my addr 172.21.15.28:0/1671489532 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter ms_handle_connect 0x7f923810eab0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter ms_handle_connect 0x7f9238103190 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter ms_handle_connect 0x7f923810afa0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071099418 0 0) 0x7f92381ab5e0 con 0x7f923810eab0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92240031f0 con 0x7f923810eab0 2024-03-30T14:55:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764073554 0 0) 0x7f92381a91a0 con 0x7f9238103190 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92381ab5e0 con 0x7f9238103190 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331349061 0 0) 0x7f92381aa3c0 con 0x7f923810afa0 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92381a91a0 con 0x7f923810afa0 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 839121735 0 0) 0x7f92240031f0 con 0x7f923810eab0 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92381aa3c0 con 0x7f923810eab0 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.261+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f92300017f0 con 0x7f923810eab0 2024-03-30T14:55:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3442560859 0 0) 0x7f92381ab5e0 con 0x7f9238103190 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92240031f0 con 0x7f9238103190 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f92280016f0 con 0x7f9238103190 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 830832695 0 0) 0x7f92381a91a0 con 0x7f923810afa0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92381ab5e0 con 0x7f923810afa0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9220001870 con 0x7f923810afa0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2356458227 0 0) 0x7f92381aa3c0 con 0x7f923810eab0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 >> v1:172.21.15.134:6789/0 conn(0x7f923810afa0 legacy=0x7f92381a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 >> v1:172.21.15.28:6789/0 conn(0x7f9238103190 legacy=0x7f923810a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92381ac840 con 0x7f923810eab0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f923de1b700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92381a93d0 con 0x7f923810eab0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f923de1b700 10 client.34559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f923de1b700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f92381a9890 con 0x7f923810eab0 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f923de1b700 1 librados: init done 2024-03-30T14:55:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9230003b20 con 0x7f923810eab0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f92300049f0 con 0x7f923810eab0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f922e7fc700 10 client.34559.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f922e7fc700 15 client.34559.objecter _session_command_op_assign -1 1 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f922e7fc700 20 client.34559.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.262+0000 7f922e7fc700 10 client.34559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.263+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f921c002920 con 0x7f923810eab0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.263+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9230003680 con 0x7f923810eab0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f9230087bc0 con 0x7f923810eab0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 10 client.34559.objecter ms_dispatch 0x7f9238001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 3 client.34559.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 3 client.34559.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 10 client.34559.objecter checking command 1 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter _get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 15 client.34559.objecter _session_command_op_remove -1 1 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter _get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 15 client.34559.objecter _session_command_op_assign 7 1 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 4 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 10 client.34559.objecter _send_command 1 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.134:6812/3758190298 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f922407e820 con 0x7f922407ab10 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 20 client.34559.objecter dump_active .. 0 homeless 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (unknown 2118522605 0 0) 0x7f9230088060 con 0x7f923810eab0 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.264+0000 7f9234ff9700 10 client.34559.objecter ms_handle_connect 0x7f9224070430 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 10 client.34559.objecter ms_handle_connect 0x7f922407ab10 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f922407e820 con 0x7f922407ab10 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 10 client.34559.objecter ms_dispatch 0x7f9238001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 10 client.34559.objecter _finish_command 1 = system:0 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 4 2024-03-30T14:55:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.265+0000 7f9234ff9700 15 client.34559.objecter _session_command_op_remove 7 1 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 client.34559.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 15 client.34559.objecter _session_command_op_assign -1 2 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter _get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter _get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 15 client.34559.objecter _session_command_op_remove -1 2 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter get_session s=0x7f922407a460 osd=7 3 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 15 client.34559.objecter _session_command_op_assign 7 2 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 4 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 client.34559.objecter _send_command 2 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 --> v1:172.21.15.134:6812/3758190298 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f921c0011f0 con 0x7f922407ab10 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9234ff9700 1 -- 172.21.15.28:0/1671489532 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3797944582) 0x7f921c0011f0 con 0x7f922407ab10 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9234ff9700 10 client.34559.objecter ms_dispatch 0x7f9238001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9234ff9700 10 client.34559.objecter _finish_command 2 = system:0 2024-03-30T14:55:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9234ff9700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 4 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9234ff9700 15 client.34559.objecter _session_command_op_remove 7 2 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 librados: watch_flush enter 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 librados: watch_flush exit 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 client.34559.objecter close_session for osd.7 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 >> v1:172.21.15.134:6812/3758190298 conn(0x7f922407ab10 legacy=0x7f922407cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter put_session s=0x7f922407a460 osd=7 1 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 20 client.34559.objecter shutdown clearing up homeless session... 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 10 client.34559.objecter successfully canceled tick 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9224070430 legacy=0x7f92240728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 >> v1:172.21.15.28:6790/0 conn(0x7f923810eab0 legacy=0x7f92381a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.282+0000 7f9237fff700 1 -- 172.21.15.28:0/1671489532 reap_dead start 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.283+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 shutdown_connections 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.283+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 >> 172.21.15.28:0/1671489532 conn(0x7f92380fc9e0 msgr2=0x7f92380fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.283+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 shutdown_connections 2024-03-30T14:55:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.283+0000 7f922e7fc700 1 -- 172.21.15.28:0/1671489532 wait complete. 2024-03-30T14:55:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.283+0000 7f922e7fc700 1 librados: shutdown 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.919+0000 7fc5a6264700 1 -- 172.21.15.28:0/676937652 >> v1:172.21.15.28:6789/0 conn(0x7fc5a0102250 legacy=0x7fc5a0102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.919+0000 7fc5a6264700 1 -- 172.21.15.28:0/676937652 shutdown_connections 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.919+0000 7fc5a6264700 1 -- 172.21.15.28:0/676937652 >> 172.21.15.28:0/676937652 conn(0x7fc5a00fdb20 msgr2=0x7fc5a00fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.919+0000 7fc5a6264700 1 -- 172.21.15.28:0/676937652 shutdown_connections 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.919+0000 7fc5a6264700 1 -- 172.21.15.28:0/676937652 wait complete. 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 librados: starting msgr at 2024-03-30T14:55:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 librados: starting objecter 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 Processor -- start 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 -- start start 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 librados: setting wanted keys 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 librados: calling monclient init 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a00760e0 con 0x7fc5a0102250 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a00762f0 con 0x7fc5a010ea80 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc5a6264700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a01aba50 con 0x7fc5a010aee0 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc59f7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc5a010aee0 0x7fc5a0074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41726/0 (socket says 172.21.15.28:41726) 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc59f7fe700 1 -- 172.21.15.28:0/2291224970 learned_addr learned my addr 172.21.15.28:0/2291224970 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.920+0000 7fc59ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc5a0102250 0x7fc5a0074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48434/0 (socket says 172.21.15.28:48434) 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter ms_handle_connect 0x7fc5a010aee0 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter ms_handle_connect 0x7fc5a0102250 2024-03-30T14:55:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter ms_handle_connect 0x7fc5a010ea80 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3507563146 0 0) 0x7fc5a00760e0 con 0x7fc5a0102250 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5880035b0 con 0x7fc5a0102250 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092206146 0 0) 0x7fc5a01aba50 con 0x7fc5a010aee0 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a00760e0 con 0x7fc5a010aee0 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3449521033 0 0) 0x7fc5a00762f0 con 0x7fc5a010ea80 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a01aba50 con 0x7fc5a010ea80 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2732590087 0 0) 0x7fc5880035b0 con 0x7fc5a0102250 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a00762f0 con 0x7fc5a0102250 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc590000f90 con 0x7fc5a0102250 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459422001 0 0) 0x7fc5a01aba50 con 0x7fc5a010ea80 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5880035b0 con 0x7fc5a010ea80 2024-03-30T14:55:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc598001c30 con 0x7fc5a010ea80 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 908341420 0 0) 0x7fc5a00760e0 con 0x7fc5a010aee0 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a01aba50 con 0x7fc5a010aee0 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc594001970 con 0x7fc5a010aee0 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4175313060 0 0) 0x7fc5a00762f0 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 >> v1:172.21.15.28:6790/0 conn(0x7fc5a010aee0 legacy=0x7fc5a0074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 >> v1:172.21.15.134:6789/0 conn(0x7fc5a010ea80 legacy=0x7fc5a01a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a01acc30 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.921+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a01abc80 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 10 client.14676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5a01ac140 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 1 librados: init done 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc590001130 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc590004c90 con 0x7fc5a0102250 2024-03-30T14:55:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 10 client.14676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 15 client.14676.objecter _session_command_op_assign -1 1 2024-03-30T14:55:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 20 client.14676.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 10 client.14676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.922+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc58c002920 con 0x7fc5a0102250 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.923+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc590004c90 con 0x7fc5a0102250 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.923+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7fc590088e90 con 0x7fc5a0102250 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.923+0000 7fc59d7fa700 10 client.14676.objecter ms_dispatch 0x7fc5a0001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.923+0000 7fc59d7fa700 3 client.14676.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.923+0000 7fc59d7fa700 3 client.14676.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 10 client.14676.objecter checking command 1 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter _get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 15 client.14676.objecter _session_command_op_remove -1 1 2024-03-30T14:55:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter _get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 15 client.14676.objecter _session_command_op_assign 0 1 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 4 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 10 client.14676.objecter _send_command 1 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5900056a0 con 0x7fc58807aad0 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 20 client.14676.objecter dump_active .. 0 homeless 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (unknown 2118522605 0 0) 0x7fc590089270 con 0x7fc5a0102250 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 10 client.14676.objecter ms_handle_connect 0x7fc588070420 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.924+0000 7fc59d7fa700 10 client.14676.objecter ms_handle_connect 0x7fc58807aad0 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.925+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc5900056a0 con 0x7fc58807aad0 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.925+0000 7fc59d7fa700 10 client.14676.objecter ms_dispatch 0x7fc5a0001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.925+0000 7fc59d7fa700 10 client.14676.objecter _finish_command 1 = system:0 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.925+0000 7fc59d7fa700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 4 2024-03-30T14:55:31.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.925+0000 7fc59d7fa700 15 client.14676.objecter _session_command_op_remove 0 1 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 10 client.14676.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 15 client.14676.objecter _session_command_op_assign -1 2 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 20 client.14676.objecter _get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 20 client.14676.objecter _get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 15 client.14676.objecter _session_command_op_remove -1 2 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 20 client.14676.objecter get_session s=0x7fc58807a420 osd=0 3 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 15 client.14676.objecter _session_command_op_assign 0 2 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 4 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 10 client.14676.objecter _send_command 2 2024-03-30T14:55:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc58c0011f0 con 0x7fc58807aad0 2024-03-30T14:55:31.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc59d7fa700 1 -- 172.21.15.28:0/2291224970 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1002017476) 0x7fc58c0011f0 con 0x7fc58807aad0 2024-03-30T14:55:31.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc59d7fa700 10 client.14676.objecter ms_dispatch 0x7fc5a0001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:31.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc59d7fa700 10 client.14676.objecter _finish_command 2 = system:0 2024-03-30T14:55:31.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc59d7fa700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 4 2024-03-30T14:55:31.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.941+0000 7fc59d7fa700 15 client.14676.objecter _session_command_op_remove 0 2 2024-03-30T14:55:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 10 librados: watch_flush enter 2024-03-30T14:55:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 10 librados: watch_flush exit 2024-03-30T14:55:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 10 client.14676.objecter close_session for osd.0 2024-03-30T14:55:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 >> v1:172.21.15.28:6801/3394323575 conn(0x7fc58807aad0 legacy=0x7fc58807cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 20 client.14676.objecter put_session s=0x7fc58807a420 osd=0 1 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 20 client.14676.objecter shutdown clearing up homeless session... 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 10 client.14676.objecter successfully canceled tick 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc588070420 legacy=0x7fc5880728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 >> v1:172.21.15.28:6789/0 conn(0x7fc5a0102250 legacy=0x7fc5a0074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a4801700 1 -- 172.21.15.28:0/2291224970 reap_dead start 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 shutdown_connections 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 >> 172.21.15.28:0/2291224970 conn(0x7fc5a00fdb20 msgr2=0x7fc5a0111f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 shutdown_connections 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 -- 172.21.15.28:0/2291224970 wait complete. 2024-03-30T14:55:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:31.942+0000 7fc5a6264700 1 librados: shutdown 2024-03-30T14:55:32.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:32 smithi028 ceph-mon[29876]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-30T14:55:32.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:32 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:32 smithi028 ceph-mon[41614]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-30T14:55:32.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:32 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:32.977 INFO:teuthology.orchestra.run.smithi028.stdout:261993005071 2024-03-30T14:55:32.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T14:55:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:32 smithi134 ceph-mon[33714]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-30T14:55:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:32 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:33.528 INFO:teuthology.orchestra.run.smithi028.stdout:137438953502 2024-03-30T14:55:33.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T14:55:33.844 INFO:teuthology.orchestra.run.smithi028.stdout:34359738418 2024-03-30T14:55:33.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T14:55:34.657 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-03-30T14:55:34.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.194+0000 7f71be213700 1 -- 172.21.15.28:0/2033380281 >> v1:172.21.15.28:6790/0 conn(0x7f71b81031a0 legacy=0x7f71b8103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.194+0000 7f71be213700 1 -- 172.21.15.28:0/2033380281 shutdown_connections 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.194+0000 7f71be213700 1 -- 172.21.15.28:0/2033380281 >> 172.21.15.28:0/2033380281 conn(0x7f71b80fc9f0 msgr2=0x7f71b80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.194+0000 7f71be213700 1 -- 172.21.15.28:0/2033380281 shutdown_connections 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.194+0000 7f71be213700 1 -- 172.21.15.28:0/2033380281 wait complete. 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.195+0000 7f71be213700 1 librados: starting msgr at 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.195+0000 7f71be213700 1 librados: starting objecter 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.195+0000 7f71be213700 1 Processor -- start 2024-03-30T14:55:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 -- start start 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 librados: setting wanted keys 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 librados: calling monclient init 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b81a93f0 con 0x7f71b81031a0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b81aa610 con 0x7f71b810eac0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71be213700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71b81ab850 con 0x7f71b810afb0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b77fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71b81031a0 0x7f71b810a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48454/0 (socket says 172.21.15.28:48454) 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b77fe700 1 -- 172.21.15.28:0/2320989476 learned_addr learned my addr 172.21.15.28:0/2320989476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b6ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f71b810afb0 0x7f71b81a42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41738/0 (socket says 172.21.15.28:41738) 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter ms_handle_connect 0x7f71b810afb0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter ms_handle_connect 0x7f71b81031a0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395780001 0 0) 0x7f71b81ab850 con 0x7f71b810afb0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71a00031f0 con 0x7f71b810afb0 2024-03-30T14:55:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter ms_handle_connect 0x7f71b810eac0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2820086597 0 0) 0x7f71b81a93f0 con 0x7f71b81031a0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.196+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b81ab850 con 0x7f71b81031a0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987371783 0 0) 0x7f71b81aa610 con 0x7f71b810eac0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b81a93f0 con 0x7f71b810eac0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3039681252 0 0) 0x7f71a00031f0 con 0x7f71b810afb0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b81aa610 con 0x7f71b810afb0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71ac0018b0 con 0x7f71b810afb0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 61350158 0 0) 0x7f71b81ab850 con 0x7f71b81031a0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71a00031f0 con 0x7f71b81031a0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71a8004500 con 0x7f71b81031a0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2086275832 0 0) 0x7f71b81a93f0 con 0x7f71b810eac0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b81ab850 con 0x7f71b810eac0 2024-03-30T14:55:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71b00013f0 con 0x7f71b810eac0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4187296989 0 0) 0x7f71a00031f0 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 >> v1:172.21.15.28:6790/0 conn(0x7f71b810afb0 legacy=0x7f71b81a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 >> v1:172.21.15.134:6789/0 conn(0x7f71b810eac0 legacy=0x7f71b81a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b81aca90 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71b81a9620 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71be213700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71b81a9ae0 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71be213700 1 librados: init done 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f71a8004780 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.197+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71a8004cb0 con 0x7f71b81031a0 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.198+0000 7f71be213700 10 client.14682.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.198+0000 7f71be213700 15 client.14682.objecter _session_command_op_assign -1 1 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.198+0000 7f71be213700 20 client.14682.objecter _get_session osd=-1 returning homeless 2024-03-30T14:55:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.198+0000 7f71be213700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.198+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f71a4002920 con 0x7f71b81031a0 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f71a8004e50 con 0x7f71b81031a0 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f71a8088fe0 con 0x7f71b81031a0 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 10 client.14682.objecter ms_dispatch 0x7f71b8001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 3 client.14682.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 3 client.14682.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 10 client.14682.objecter checking command 1 2024-03-30T14:55:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter _get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 15 client.14682.objecter _session_command_op_remove -1 1 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter _get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 15 client.14682.objecter _session_command_op_assign 2 1 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 4 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 10 client.14682.objecter _send_command 1 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71a8001080 con 0x7f71a007aae0 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 20 client.14682.objecter dump_active .. 0 homeless 2024-03-30T14:55:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (unknown 2118522605 0 0) 0x7f71a8089450 con 0x7f71b81031a0 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.199+0000 7f71b4ff9700 10 client.14682.objecter ms_handle_connect 0x7f71a0070400 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 10 client.14682.objecter ms_handle_connect 0x7f71a007aae0 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71a8001080 con 0x7f71a007aae0 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 10 client.14682.objecter ms_dispatch 0x7f71b8001550 command_reply(tid 1: 0 ) 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 10 client.14682.objecter _finish_command 1 = system:0 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 4 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.200+0000 7f71b4ff9700 15 client.14682.objecter _session_command_op_remove 2 1 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 10 client.14682.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 15 client.14682.objecter _session_command_op_assign -1 2 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 20 client.14682.objecter _get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 20 client.14682.objecter _get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 15 client.14682.objecter _session_command_op_remove -1 2 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 20 client.14682.objecter get_session s=0x7f71a007a430 osd=2 3 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 15 client.14682.objecter _session_command_op_assign 2 2 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 4 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 10 client.14682.objecter _send_command 2 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.216+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f71a40011f0 con 0x7f71a007aae0 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b4ff9700 1 -- 172.21.15.28:0/2320989476 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1016321414) 0x7f71a40011f0 con 0x7f71a007aae0 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b4ff9700 10 client.14682.objecter ms_dispatch 0x7f71b8001550 command_reply(tid 2: 0 ) 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b4ff9700 10 client.14682.objecter _finish_command 2 = system:0 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b4ff9700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 4 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b4ff9700 15 client.14682.objecter _session_command_op_remove 2 2 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 10 librados: watch_flush enter 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 10 librados: watch_flush exit 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 10 client.14682.objecter close_session for osd.2 2024-03-30T14:55:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 >> v1:172.21.15.28:6809/3451325043 conn(0x7f71a007aae0 legacy=0x7f71a007cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 20 client.14682.objecter put_session s=0x7f71a007a430 osd=2 1 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 20 client.14682.objecter shutdown clearing up homeless session... 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 10 client.14682.objecter successfully canceled tick 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 >> v1:172.21.15.28:6800/3011431093 conn(0x7f71a0070400 legacy=0x7f71a00728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 >> v1:172.21.15.28:6789/0 conn(0x7f71b81031a0 legacy=0x7f71b810a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71b7fff700 1 -- 172.21.15.28:0/2320989476 reap_dead start 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 shutdown_connections 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 >> 172.21.15.28:0/2320989476 conn(0x7f71b80fc9f0 msgr2=0x7f71b8110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 shutdown_connections 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 -- 172.21.15.28:0/2320989476 wait complete. 2024-03-30T14:55:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:33.217+0000 7f71be213700 1 librados: shutdown 2024-03-30T14:55:34.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:34 smithi028 ceph-mon[29876]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:34 smithi028 ceph-mon[41614]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:34 smithi134 ceph-mon[33714]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:35.398 INFO:teuthology.orchestra.run.smithi028.stdout:77309411368 2024-03-30T14:55:35.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T14:55:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:37.030 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:36 smithi028 ceph-mon[41614]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:55:37.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:36 smithi028 ceph-mon[29876]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:55:37.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:36 smithi134 ceph-mon[33714]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:55:38.532 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:55:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[29876]: 3.c deep-scrub starts 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[29876]: 3.c deep-scrub ok 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[29876]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[41614]: 3.c deep-scrub starts 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[41614]: 3.c deep-scrub ok 2024-03-30T14:55:39.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:38 smithi028 ceph-mon[41614]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-30T14:55:39.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:38 smithi134 ceph-mon[33714]: 3.c deep-scrub starts 2024-03-30T14:55:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:38 smithi134 ceph-mon[33714]: 3.c deep-scrub ok 2024-03-30T14:55:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:38 smithi134 ceph-mon[33714]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-30T14:55:39.459 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:39.460 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:39.461 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.086+0000 7f13f7c2d700 1 -- 172.21.15.28:0/3849459566 >> v1:172.21.15.28:6789/0 conn(0x7f13f010aef0 legacy=0x7f13f010d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.086+0000 7f13f7c2d700 1 -- 172.21.15.28:0/3849459566 shutdown_connections 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.086+0000 7f13f7c2d700 1 -- 172.21.15.28:0/3849459566 >> 172.21.15.28:0/3849459566 conn(0x7f13f00fdb30 msgr2=0x7f13f00fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.086+0000 7f13f7c2d700 1 -- 172.21.15.28:0/3849459566 shutdown_connections 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.086+0000 7f13f7c2d700 1 -- 172.21.15.28:0/3849459566 wait complete. 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 librados: starting msgr at 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 librados: starting objecter 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 Processor -- start 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 -- start start 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 librados: setting wanted keys 2024-03-30T14:55:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 librados: calling monclient init 2024-03-30T14:55:40.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13f01a70e0 con 0x7f13f010ea90 2024-03-30T14:55:40.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13f01a8300 con 0x7f13f0102260 2024-03-30T14:55:40.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f7c2d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13f01a9540 con 0x7f13f010aef0 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f51c8700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f13f010aef0 0x7f13f01a1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56762/0 (socket says 172.21.15.28:56762) 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13f51c8700 1 -- 172.21.15.28:0/1238923930 learned_addr learned my addr 172.21.15.28:0/1238923930 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter ms_handle_connect 0x7f13f010ea90 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter ms_handle_connect 0x7f13f0102260 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter ms_handle_connect 0x7f13f010aef0 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.087+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4080036947 0 0) 0x7f13f01a70e0 con 0x7f13f010ea90 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d80031f0 con 0x7f13f010ea90 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244570969 0 0) 0x7f13f01a8300 con 0x7f13f0102260 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f01a70e0 con 0x7f13f0102260 2024-03-30T14:55:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897837160 0 0) 0x7f13f01a9540 con 0x7f13f010aef0 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f01a8300 con 0x7f13f010aef0 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3212309489 0 0) 0x7f13f01a70e0 con 0x7f13f0102260 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13f01a9540 con 0x7f13f0102260 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2022984526 0 0) 0x7f13d80031f0 con 0x7f13f010ea90 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13f01a70e0 con 0x7f13f010ea90 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f13e0001bf0 con 0x7f13f0102260 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 532956588 0 0) 0x7f13f01a8300 con 0x7f13f010aef0 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13d80031f0 con 0x7f13f010aef0 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f13e40044b0 con 0x7f13f010aef0 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f13e80010f0 con 0x7f13f010ea90 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 451481417 0 0) 0x7f13f01a9540 con 0x7f13f0102260 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 >> v1:172.21.15.28:6790/0 conn(0x7f13f010aef0 legacy=0x7f13f01a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 >> v1:172.21.15.28:6789/0 conn(0x7f13f010ea90 legacy=0x7f13f01a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f01aa780 con 0x7f13f0102260 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13f01a8530 con 0x7f13f0102260 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13f7c2d700 10 client.24607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f13f01a89f0 con 0x7f13f0102260 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13f7c2d700 1 librados: init done 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f13e0003ec0 con 0x7f13f0102260 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.088+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f13e0004e40 con 0x7f13f0102260 2024-03-30T14:55:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.089+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13dc005120 con 0x7f13f0102260 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.090+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f13e0003a20 con 0x7f13f0102260 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f13e0088000 con 0x7f13f0102260 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 10 client.24607.objecter ms_dispatch 0x7f13f0001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 3 client.24607.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 3 client.24607.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 20 client.24607.objecter dump_active .. 0 homeless 2024-03-30T14:55:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.091+0000 7f13eeffd700 10 client.24607.objecter ms_handle_connect 0x7f13d80704d0 2024-03-30T14:55:40.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.094+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f13e0055960 con 0x7f13f0102260 2024-03-30T14:55:40.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:39 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.305+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f13dc005410 con 0x7f13f0102260 2024-03-30T14:55:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.306+0000 7f13eeffd700 1 -- 172.21.15.28:0/1238923930 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 1793367863) 0x7f13e00593b0 con 0x7f13f0102260 2024-03-30T14:55:40.307 INFO:teuthology.orchestra.run.smithi028.stdout:261993005073 2024-03-30T14:55:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 10 librados: watch_flush enter 2024-03-30T14:55:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 10 librados: watch_flush exit 2024-03-30T14:55:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 20 client.24607.objecter shutdown clearing up homeless session... 2024-03-30T14:55:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 10 client.24607.objecter successfully canceled tick 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 >> v1:172.21.15.28:6800/3011431093 conn(0x7f13d80704d0 legacy=0x7f13d8072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 >> v1:172.21.15.134:6789/0 conn(0x7f13f0102260 legacy=0x7f13f0107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 shutdown_connections 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 >> 172.21.15.28:0/1238923930 conn(0x7f13f00fdb30 msgr2=0x7f13f0111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 shutdown_connections 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 -- 172.21.15.28:0/1238923930 wait complete. 2024-03-30T14:55:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.308+0000 7f13f7c2d700 1 librados: shutdown 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.865+0000 7f2e93eb5700 1 -- 172.21.15.28:0/4033594943 >> v1:172.21.15.28:6790/0 conn(0x7f2e8c10b020 legacy=0x7f2e8c10d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.865+0000 7f2e93eb5700 1 -- 172.21.15.28:0/4033594943 shutdown_connections 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.865+0000 7f2e93eb5700 1 -- 172.21.15.28:0/4033594943 >> 172.21.15.28:0/4033594943 conn(0x7f2e8c0fc9e0 msgr2=0x7f2e8c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.865+0000 7f2e93eb5700 1 -- 172.21.15.28:0/4033594943 shutdown_connections 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.865+0000 7f2e93eb5700 1 -- 172.21.15.28:0/4033594943 wait complete. 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 librados: starting msgr at 2024-03-30T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 librados: starting objecter 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 Processor -- start 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 -- start start 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 librados: setting wanted keys 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 librados: calling monclient init 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e8c10abe0 con 0x7f2e8c103210 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e8c10adf0 con 0x7f2e8c10eb30 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e93eb5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e8c1a9640 con 0x7f2e8c10b020 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e91450700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2e8c10b020 0x7f2e8c108140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56780/0 (socket says 172.21.15.28:56780) 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e91450700 1 -- 172.21.15.28:0/521743447 learned_addr learned my addr 172.21.15.28:0/521743447 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e7effd700 10 client.?.objecter ms_handle_connect 0x7f2e8c103210 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e7effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e7effd700 10 client.?.objecter ms_handle_connect 0x7f2e8c10b020 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.866+0000 7f2e7effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 10 client.?.objecter ms_handle_connect 0x7f2e8c10eb30 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416576332 0 0) 0x7f2e8c10abe0 con 0x7f2e8c103210 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e780031f0 con 0x7f2e8c103210 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803099130 0 0) 0x7f2e8c1a9640 con 0x7f2e8c10b020 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e8c10abe0 con 0x7f2e8c10b020 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3481150805 0 0) 0x7f2e8c10adf0 con 0x7f2e8c10eb30 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e8c1a9640 con 0x7f2e8c10eb30 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 249733586 0 0) 0x7f2e780031f0 con 0x7f2e8c103210 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e8c10adf0 con 0x7f2e8c103210 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e88001830 con 0x7f2e8c103210 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1601072134 0 0) 0x7f2e8c10abe0 con 0x7f2e8c10b020 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e780031f0 con 0x7f2e8c10b020 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e800046a0 con 0x7f2e8c10b020 2024-03-30T14:55:40.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3705221587 0 0) 0x7f2e8c1a9640 con 0x7f2e8c10eb30 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e8c10abe0 con 0x7f2e8c10eb30 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e840013f0 con 0x7f2e8c10eb30 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3724536265 0 0) 0x7f2e8c10adf0 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 >> v1:172.21.15.28:6790/0 conn(0x7f2e8c10b020 legacy=0x7f2e8c108140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 >> v1:172.21.15.134:6789/0 conn(0x7f2e8c10eb30 legacy=0x7f2e8c1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c1aa880 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c1a9870 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e93eb5700 10 client.14694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2e8c1a9d30 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e93eb5700 1 librados: init done 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2e88003b20 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.867+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e88004aa0 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.869+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e70005120 con 0x7f2e8c103210 2024-03-30T14:55:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.869+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2e8801b3c0 con 0x7f2e8c103210 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.869+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f2e88087fe0 con 0x7f2e8c103210 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.870+0000 7f2e7effd700 10 client.14694.objecter ms_dispatch 0x7f2e8c001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.870+0000 7f2e7effd700 3 client.14694.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.870+0000 7f2e7effd700 3 client.14694.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.870+0000 7f2e7effd700 20 client.14694.objecter dump_active .. 0 homeless 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.870+0000 7f2e7effd700 10 client.14694.objecter ms_handle_connect 0x7f2e78070440 2024-03-30T14:55:40.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:40.874+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2e880565d0 con 0x7f2e8c103210 2024-03-30T14:55:41.021 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:41.021 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1238923930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1238923930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:40 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.089+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2e70005410 con 0x7f2e8c103210 2024-03-30T14:55:41.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.089+0000 7f2e7effd700 1 -- 172.21.15.28:0/521743447 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 599417965) 0x7f2e8805a020 con 0x7f2e8c103210 2024-03-30T14:55:41.090 INFO:teuthology.orchestra.run.smithi028.stdout:115964117030 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 10 librados: watch_flush enter 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 10 librados: watch_flush exit 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 20 client.14694.objecter shutdown clearing up homeless session... 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 10 client.14694.objecter successfully canceled tick 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2e78070440 legacy=0x7f2e78072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 >> v1:172.21.15.28:6789/0 conn(0x7f2e8c103210 legacy=0x7f2e8c107a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 shutdown_connections 2024-03-30T14:55:41.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 >> 172.21.15.28:0/521743447 conn(0x7f2e8c0fc9e0 msgr2=0x7f2e8c0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:41.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 shutdown_connections 2024-03-30T14:55:41.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 -- 172.21.15.28:0/521743447 wait complete. 2024-03-30T14:55:41.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:41.091+0000 7f2e93eb5700 1 librados: shutdown 2024-03-30T14:55:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T14:55:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T14:55:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1238923930' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T14:55:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T14:55:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T14:55:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:40 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:41.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T14:55:41.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T14:55:41.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/521743447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:55:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T14:55:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T14:55:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/521743447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:55:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:41 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T14:55:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:41 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T14:55:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/521743447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T14:55:42.181 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117028 got 115964117030 for osd.3 2024-03-30T14:55:42.181 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.361+0000 7f3ee6737700 1 -- 172.21.15.28:0/320998772 >> v1:172.21.15.28:6790/0 conn(0x7f3ee0109e10 legacy=0x7f3ee010c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.361+0000 7f3ee6737700 1 -- 172.21.15.28:0/320998772 shutdown_connections 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.361+0000 7f3ee6737700 1 -- 172.21.15.28:0/320998772 >> 172.21.15.28:0/320998772 conn(0x7f3ee00fca30 msgr2=0x7f3ee00fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.361+0000 7f3ee6737700 1 -- 172.21.15.28:0/320998772 shutdown_connections 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.361+0000 7f3ee6737700 1 -- 172.21.15.28:0/320998772 wait complete. 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 librados: starting msgr at 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 librados: starting objecter 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 Processor -- start 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 -- start start 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 librados: setting wanted keys 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 librados: calling monclient init 2024-03-30T14:55:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ee01067d0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ee01069e0 con 0x7f3ee0101180 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3ee6737700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ee01ab990 con 0x7f3ee0109e10 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3edf7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3ee0109e10 0x7f3ee006ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56800/0 (socket says 172.21.15.28:56800) 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.362+0000 7f3edf7fe700 1 -- 172.21.15.28:0/4215071242 learned_addr learned my addr 172.21.15.28:0/4215071242 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter ms_handle_connect 0x7f3ee0109e10 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter ms_handle_connect 0x7f3ee010d9b0 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter ms_handle_connect 0x7f3ee0101180 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508658732 0 0) 0x7f3ee01ab990 con 0x7f3ee0109e10 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ec80035b0 con 0x7f3ee0109e10 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536409390 0 0) 0x7f3ee01069e0 con 0x7f3ee0101180 2024-03-30T14:55:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee01ab990 con 0x7f3ee0101180 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4210519740 0 0) 0x7f3ee01067d0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee01069e0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4048733958 0 0) 0x7f3ee01ab990 con 0x7f3ee0101180 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ee01067d0 con 0x7f3ee0101180 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3ed40018b0 con 0x7f3ee0101180 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2178183418 0 0) 0x7f3ec80035b0 con 0x7f3ee0109e10 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ee01ab990 con 0x7f3ee0109e10 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3ed00015d0 con 0x7f3ee0109e10 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2314129377 0 0) 0x7f3ee01069e0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ec80035b0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3ed8001ae0 con 0x7f3ee010d9b0 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.363+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2203212839 0 0) 0x7f3ee01ab990 con 0x7f3ee0109e10 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 >> v1:172.21.15.134:6789/0 conn(0x7f3ee0101180 legacy=0x7f3ee006a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 >> v1:172.21.15.28:6789/0 conn(0x7f3ee010d9b0 legacy=0x7f3ee01a8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee01acb70 con 0x7f3ee0109e10 2024-03-30T14:55:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee01abbc0 con 0x7f3ee0109e10 2024-03-30T14:55:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3ee6737700 10 client.34589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ee01ac080 con 0x7f3ee0109e10 2024-03-30T14:55:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3ee6737700 1 librados: init done 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3ed0003760 con 0x7f3ee0109e10 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.364+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3ed0004e00 con 0x7f3ee0109e10 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.365+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3ed0004de0 con 0x7f3ee0109e10 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.365+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ecc005120 con 0x7f3ee0109e10 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(89..89 src has 14..89) ==== 6259+0+0 (unknown 1254913199 0 0) 0x7f3ed0088f30 con 0x7f3ee0109e10 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 10 client.34589.objecter ms_dispatch 0x7f3ee0001550 osd_map(89..89 src has 14..89) 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 3 client.34589.objecter handle_osd_map got epochs [89,89] > 0 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 3 client.34589.objecter handle_osd_map decoding full epoch 89 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 20 client.34589.objecter dump_active .. 0 homeless 2024-03-30T14:55:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.366+0000 7f3edd7fa700 10 client.34589.objecter ms_handle_connect 0x7f3ec8070530 2024-03-30T14:55:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.371+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3ed0057810 con 0x7f3ee0109e10 2024-03-30T14:55:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.586+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3ecc005410 con 0x7f3ee0109e10 2024-03-30T14:55:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.586+0000 7f3edd7fa700 1 -- 172.21.15.28:0/4215071242 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 935790178) 0x7f3ed0087370 con 0x7f3ee0109e10 2024-03-30T14:55:42.587 INFO:teuthology.orchestra.run.smithi028.stdout:223338299414 2024-03-30T14:55:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 10 librados: watch_flush enter 2024-03-30T14:55:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 10 librados: watch_flush exit 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 20 client.34589.objecter shutdown clearing up homeless session... 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 10 client.34589.objecter successfully canceled tick 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3ec8070530 legacy=0x7f3ec80729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 >> v1:172.21.15.28:6790/0 conn(0x7f3ee0109e10 legacy=0x7f3ee006ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 shutdown_connections 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 >> 172.21.15.28:0/4215071242 conn(0x7f3ee00fca30 msgr2=0x7f3ee0110600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 shutdown_connections 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.589+0000 7f3ee6737700 1 -- 172.21.15.28:0/4215071242 wait complete. 2024-03-30T14:55:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:42.590+0000 7f3ee6737700 1 librados: shutdown 2024-03-30T14:55:42.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005071 got 261993005073 for osd.7 2024-03-30T14:55:42.752 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:43.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.5 deep-scrub starts 2024-03-30T14:55:43.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.5 deep-scrub ok 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4215071242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.5 deep-scrub starts 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.5 deep-scrub ok 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T14:55:43.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T14:55:43.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T14:55:43.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4215071242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.5 deep-scrub starts 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.5 deep-scrub ok 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T14:55:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4215071242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T14:55:43.375 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299411 got 223338299414 for osd.6 2024-03-30T14:55:43.375 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T14:55:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T14:55:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 3761M 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:55:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: 5.1b scrub starts 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: 5.1b scrub ok 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 3761M 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: 5.1b scrub starts 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: 5.1b scrub ok 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 3761M 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: 5.1b scrub starts 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi134:/etc/ceph/ceph.conf 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: 5.1b scrub ok 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T14:55:44.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: Updating smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:44.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: Deploying daemon alertmanager.a on smithi028 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:45 smithi134 ceph-mon[33714]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T14:55:45.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: Deploying daemon alertmanager.a on smithi028 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[29876]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: Updating smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: Deploying daemon alertmanager.a on smithi028 2024-03-30T14:55:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:45 smithi028 ceph-mon[41614]: osdmap e90: 8 total, 8 up, 8 in 2024-03-30T14:55:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:46 smithi134 ceph-mon[33714]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:46 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T14:55:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:46 smithi134 ceph-mon[33714]: 3.b scrub ok 2024-03-30T14:55:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:46 smithi134 ceph-mon[33714]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T14:55:46.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[29876]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[29876]: 3.b scrub ok 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[29876]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[41614]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[41614]: 3.b scrub ok 2024-03-30T14:55:46.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:46 smithi028 ceph-mon[41614]: osdmap e91: 8 total, 8 up, 8 in 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 5.c deep-scrub starts 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 5.c deep-scrub ok 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T14:55:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:47 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T14:55:47.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 5.c deep-scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 5.c deep-scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 5.c deep-scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 5.c deep-scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T14:55:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T14:55:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T14:55:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T14:55:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:47 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T14:55:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 -- 172.21.15.28:0/3459550634 >> v1:172.21.15.28:6789/0 conn(0x7f30e0109dc0 legacy=0x7f30e010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 -- 172.21.15.28:0/3459550634 shutdown_connections 2024-03-30T14:55:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 -- 172.21.15.28:0/3459550634 >> 172.21.15.28:0/3459550634 conn(0x7f30e00fc9e0 msgr2=0x7f30e00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 -- 172.21.15.28:0/3459550634 shutdown_connections 2024-03-30T14:55:48.367 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:55:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 -- 172.21.15.28:0/3459550634 wait complete. 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 librados: starting msgr at 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 librados: starting objecter 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.365+0000 7f30e639a700 1 Processor -- start 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 -- start start 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 librados: setting wanted keys 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 librados: calling monclient init 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e01a9280 con 0x7f30e010d960 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e01aa4c0 con 0x7f30e0101130 2024-03-30T14:55:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30e639a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e01ab700 con 0x7f30e0109dc0 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30df7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f30e0109dc0 0x7f30e01a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56834/0 (socket says 172.21.15.28:56834) 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30df7fe700 1 -- 172.21.15.28:0/2452303767 learned_addr learned my addr 172.21.15.28:0/2452303767 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30dd7fa700 10 client.?.objecter ms_handle_connect 0x7f30e0109dc0 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30dd7fa700 10 client.?.objecter ms_handle_connect 0x7f30e010d960 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.366+0000 7f30dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105247000 0 0) 0x7f30e01ab700 con 0x7f30e0109dc0 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30c80031f0 con 0x7f30e0109dc0 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 10 client.?.objecter ms_handle_connect 0x7f30e0101130 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490296142 0 0) 0x7f30e01a9280 con 0x7f30e010d960 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e01ab700 con 0x7f30e010d960 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925537605 0 0) 0x7f30e01aa4c0 con 0x7f30e0101130 2024-03-30T14:55:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e01a9280 con 0x7f30e0101130 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2754839189 0 0) 0x7f30c80031f0 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e01aa4c0 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30d4004550 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3713023502 0 0) 0x7f30e01a9280 con 0x7f30e0101130 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30c80031f0 con 0x7f30e0101130 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30d00018b0 con 0x7f30e0101130 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2102322705 0 0) 0x7f30e01aa4c0 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 >> v1:172.21.15.134:6789/0 conn(0x7f30e0101130 legacy=0x7f30e0106360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 >> v1:172.21.15.28:6789/0 conn(0x7f30e010d960 legacy=0x7f30e01a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e01ac940 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30e01a94b0 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30e639a700 10 client.34601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f30e01a9970 con 0x7f30e0109dc0 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.367+0000 7f30e639a700 1 librados: init done 2024-03-30T14:55:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.368+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f30d4001ab0 con 0x7f30e0109dc0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.368+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30d4004f90 con 0x7f30e0109dc0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.369+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc005120 con 0x7f30e0109dc0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.369+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f30d4004f70 con 0x7f30e0109dc0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(91..91 src has 14..91) ==== 6259+0+0 (unknown 2721991072 0 0) 0x7f30d4088e40 con 0x7f30e0109dc0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 10 client.34601.objecter ms_dispatch 0x7f30e0001550 osd_map(91..91 src has 14..91) 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 3 client.34601.objecter handle_osd_map got epochs [91,91] > 0 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 3 client.34601.objecter handle_osd_map decoding full epoch 91 2024-03-30T14:55:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 20 client.34601.objecter dump_active .. 0 homeless 2024-03-30T14:55:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.370+0000 7f30dd7fa700 10 client.34601.objecter ms_handle_connect 0x7f30c80701f0 2024-03-30T14:55:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.374+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f30d405eb10 con 0x7f30e0109dc0 2024-03-30T14:55:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.292+0000 7fef020c2700 1 -- 172.21.15.28:0/1249765670 >> v1:172.21.15.28:6789/0 conn(0x7feefc10aed0 legacy=0x7feefc10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.292+0000 7fef020c2700 1 -- 172.21.15.28:0/1249765670 shutdown_connections 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.292+0000 7fef020c2700 1 -- 172.21.15.28:0/1249765670 >> 172.21.15.28:0/1249765670 conn(0x7feefc0fdaf0 msgr2=0x7feefc0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.292+0000 7fef020c2700 1 -- 172.21.15.28:0/1249765670 shutdown_connections 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.292+0000 7fef020c2700 1 -- 172.21.15.28:0/1249765670 wait complete. 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 librados: starting msgr at 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 librados: starting objecter 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 Processor -- start 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 -- start start 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 librados: setting wanted keys 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 librados: calling monclient init 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a6f30 con 0x7feefc10ea70 2024-03-30T14:55:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.293+0000 7fef020c2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a8150 con 0x7feefc10aed0 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7fef020c2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feefc1a9390 con 0x7feefc102240 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feefb7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7feefc102240 0x7feefc1073c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56820/0 (socket says 172.21.15.28:56820) 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feefb7fe700 1 -- 172.21.15.28:0/2091386671 learned_addr learned my addr 172.21.15.28:0/2091386671 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter ms_handle_connect 0x7feefc102240 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter ms_handle_connect 0x7feefc10ea70 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988287012 0 0) 0x7feefc1a9390 con 0x7feefc102240 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feee40031f0 con 0x7feefc102240 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter ms_handle_connect 0x7feefc10aed0 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:48.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694866156 0 0) 0x7feefc1a6f30 con 0x7feefc10ea70 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feefc1a9390 con 0x7feefc10ea70 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251682048 0 0) 0x7feefc1a8150 con 0x7feefc10aed0 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feefc1a6f30 con 0x7feefc10aed0 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2436768297 0 0) 0x7feee40031f0 con 0x7feefc102240 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feefc1a8150 con 0x7feefc102240 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feeec001a30 con 0x7feefc102240 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1517882304 0 0) 0x7feefc1a9390 con 0x7feefc10ea70 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feee40031f0 con 0x7feefc10ea70 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feef4001b30 con 0x7feefc10ea70 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1600920690 0 0) 0x7feefc1a8150 con 0x7feefc102240 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 >> v1:172.21.15.134:6789/0 conn(0x7feefc10aed0 legacy=0x7feefc1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 >> v1:172.21.15.28:6789/0 conn(0x7feefc10ea70 legacy=0x7feefc1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feefc1aa5d0 con 0x7feefc102240 2024-03-30T14:55:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.294+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feefc1a7160 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.295+0000 7fef020c2700 10 client.34595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.295+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feefc1a7620 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.295+0000 7fef020c2700 1 librados: init done 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.295+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7feeec001bd0 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.295+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feeec005cd0 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.296+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee8005120 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.296+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7feeec0040a0 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(91..91 src has 14..91) ==== 6259+0+0 (unknown 2721991072 0 0) 0x7feeec088dc0 con 0x7feefc102240 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 10 client.34595.objecter ms_dispatch 0x7feefc001550 osd_map(91..91 src has 14..91) 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 3 client.34595.objecter handle_osd_map got epochs [91,91] > 0 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 3 client.34595.objecter handle_osd_map decoding full epoch 91 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 20 client.34595.objecter dump_active .. 0 homeless 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.297+0000 7feef8ff9700 10 client.34595.objecter ms_handle_connect 0x7feee40702d0 2024-03-30T14:55:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.301+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7feeec056860 con 0x7feefc102240 2024-03-30T14:55:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.509+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7feee8005410 con 0x7feefc102240 2024-03-30T14:55:48.515 INFO:teuthology.orchestra.run.smithi028.stdout:77309411371 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.509+0000 7feef8ff9700 1 -- 172.21.15.28:0/2091386671 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3833507532) 0x7feeec05a2b0 con 0x7feefc102240 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 10 librados: watch_flush enter 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 10 librados: watch_flush exit 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 20 client.34595.objecter shutdown clearing up homeless session... 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 10 client.34595.objecter successfully canceled tick 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 >> v1:172.21.15.28:6800/3011431093 conn(0x7feee40702d0 legacy=0x7feee4072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 >> v1:172.21.15.28:6790/0 conn(0x7feefc102240 legacy=0x7feefc1073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 shutdown_connections 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.511+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 >> 172.21.15.28:0/2091386671 conn(0x7feefc0fdaf0 msgr2=0x7feefc10ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.512+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 shutdown_connections 2024-03-30T14:55:48.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.512+0000 7fef020c2700 1 -- 172.21.15.28:0/2091386671 wait complete. 2024-03-30T14:55:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.512+0000 7fef020c2700 1 librados: shutdown 2024-03-30T14:55:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.587+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f30cc005410 con 0x7f30e0109dc0 2024-03-30T14:55:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.587+0000 7f30dd7fa700 1 -- 172.21.15.28:0/2452303767 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (unknown 1823589948 0 2157769881) 0x7f30d4087250 con 0x7f30e0109dc0 2024-03-30T14:55:48.589 INFO:teuthology.orchestra.run.smithi028.stdout:137438953505 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.589+0000 7f30e639a700 10 librados: watch_flush enter 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.589+0000 7f30e639a700 10 librados: watch_flush exit 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.589+0000 7f30e639a700 20 client.34601.objecter shutdown clearing up homeless session... 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.589+0000 7f30e639a700 10 client.34601.objecter successfully canceled tick 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 >> v1:172.21.15.28:6800/3011431093 conn(0x7f30c80701f0 legacy=0x7f30c80726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 >> v1:172.21.15.28:6790/0 conn(0x7f30e0109dc0 legacy=0x7f30e01a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 shutdown_connections 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 >> 172.21.15.28:0/2452303767 conn(0x7f30e00fc9e0 msgr2=0x7f30e006a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 shutdown_connections 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 -- 172.21.15.28:0/2452303767 wait complete. 2024-03-30T14:55:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:48.590+0000 7f30e639a700 1 librados: shutdown 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T14:55:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:48 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T14:55:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T14:55:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T14:55:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T14:55:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:48 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T14:55:49.340 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953505 for osd.4 2024-03-30T14:55:49.340 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2091386671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2452303767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T14:55:49.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2091386671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2452303767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T14:55:49.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:49 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2091386671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2452303767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T14:55:49.703 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T14:55:49.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T14:55:49.704 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T14:55:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.780+0000 7f02ac5f6700 1 -- 172.21.15.28:0/1842872629 >> v1:172.21.15.28:6790/0 conn(0x7f02a410aef0 legacy=0x7f02a410d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.780+0000 7f02ac5f6700 1 -- 172.21.15.28:0/1842872629 shutdown_connections 2024-03-30T14:55:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.780+0000 7f02ac5f6700 1 -- 172.21.15.28:0/1842872629 >> 172.21.15.28:0/1842872629 conn(0x7f02a40fdb30 msgr2=0x7f02a40fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.780+0000 7f02ac5f6700 1 -- 172.21.15.28:0/1842872629 shutdown_connections 2024-03-30T14:55:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.780+0000 7f02ac5f6700 1 -- 172.21.15.28:0/1842872629 wait complete. 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 librados: starting msgr at 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 librados: starting objecter 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 Processor -- start 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 -- start start 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 librados: setting wanted keys 2024-03-30T14:55:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 librados: calling monclient init 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a4107760 con 0x7f02a4102260 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a4107970 con 0x7f02a410ea90 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02ac5f6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02a4107b80 con 0x7f02a410aef0 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a9b91700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f02a410aef0 0x7f02a4074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57194/0 (socket says 172.21.15.28:57194) 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a9b91700 1 -- 172.21.15.28:0/715406191 learned_addr learned my addr 172.21.15.28:0/715406191 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter ms_handle_connect 0x7f02a410aef0 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter ms_handle_connect 0x7f02a4102260 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter ms_handle_connect 0x7f02a410ea90 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076895342 0 0) 0x7f02a4107b80 con 0x7f02a410aef0 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f028c0031f0 con 0x7f02a410aef0 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2739524235 0 0) 0x7f02a4107760 con 0x7f02a4102260 2024-03-30T14:55:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a4107b80 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.781+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 325980541 0 0) 0x7f02a4107970 con 0x7f02a410ea90 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a4107760 con 0x7f02a410ea90 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1689487572 0 0) 0x7f02a4107b80 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02a4107970 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0294001c60 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1985574248 0 0) 0x7f028c0031f0 con 0x7f02a410aef0 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02a4107b80 con 0x7f02a410aef0 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0298004500 con 0x7f02a410aef0 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3580541604 0 0) 0x7f02a4107760 con 0x7f02a410ea90 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f028c0031f0 con 0x7f02a410ea90 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f029c0017b0 con 0x7f02a410ea90 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3812776480 0 0) 0x7f02a4107970 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 >> v1:172.21.15.28:6790/0 conn(0x7f02a410aef0 legacy=0x7f02a4074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 >> v1:172.21.15.134:6789/0 conn(0x7f02a410ea90 legacy=0x7f02a41a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a41acad0 con 0x7f02a4102260 2024-03-30T14:55:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02a41a9aa0 con 0x7f02a4102260 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02ac5f6700 10 client.14718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02a41a9f60 con 0x7f02a4102260 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02ac5f6700 1 librados: init done 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0294003ea0 con 0x7f02a4102260 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.782+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0294004d20 con 0x7f02a4102260 2024-03-30T14:55:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290005120 con 0x7f02a4102260 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f029401b560 con 0x7f02a4102260 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(91..91 src has 14..91) ==== 6259+0+0 (unknown 2721991072 0 0) 0x7f02940880c0 con 0x7f02a4102260 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02a37fe700 10 client.14718.objecter ms_dispatch 0x7f02a4001550 osd_map(91..91 src has 14..91) 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02a37fe700 3 client.14718.objecter handle_osd_map got epochs [91,91] > 0 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.784+0000 7f02a37fe700 3 client.14718.objecter handle_osd_map decoding full epoch 91 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.785+0000 7f02a37fe700 20 client.14718.objecter dump_active .. 0 homeless 2024-03-30T14:55:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.785+0000 7f02a37fe700 10 client.14718.objecter ms_handle_connect 0x7f028c0704d0 2024-03-30T14:55:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.789+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0294055a20 con 0x7f02a4102260 2024-03-30T14:55:49.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.998+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0290005410 con 0x7f02a4102260 2024-03-30T14:55:49.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:49.999+0000 7f02a37fe700 1 -- 172.21.15.28:0/715406191 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 1650871258) 0x7f0294059470 con 0x7f02a4102260 2024-03-30T14:55:50.000 INFO:teuthology.orchestra.run.smithi028.stdout:176093659164 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.000+0000 7f02ac5f6700 10 librados: watch_flush enter 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.000+0000 7f02ac5f6700 10 librados: watch_flush exit 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.000+0000 7f02ac5f6700 20 client.14718.objecter shutdown clearing up homeless session... 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.000+0000 7f02ac5f6700 10 client.14718.objecter successfully canceled tick 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.000+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 >> v1:172.21.15.28:6800/3011431093 conn(0x7f028c0704d0 legacy=0x7f028c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 >> v1:172.21.15.28:6789/0 conn(0x7f02a4102260 legacy=0x7f02a4074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 shutdown_connections 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 >> 172.21.15.28:0/715406191 conn(0x7f02a40fdb30 msgr2=0x7f02a41116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 shutdown_connections 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 -- 172.21.15.28:0/715406191 wait complete. 2024-03-30T14:55:50.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.001+0000 7f02ac5f6700 1 librados: shutdown 2024-03-30T14:55:50.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:49 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:55:50.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-03-30T14:55:50.421 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T14:55:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/715406191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/715406191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:55:50.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T14:55:50.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:50 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/715406191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T14:55:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:50 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.657+0000 7f2928936700 1 -- 172.21.15.28:0/2773648928 >> v1:172.21.15.28:6790/0 conn(0x7f292410eb10 legacy=0x7f2924110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 -- 172.21.15.28:0/2773648928 shutdown_connections 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 -- 172.21.15.28:0/2773648928 >> 172.21.15.28:0/2773648928 conn(0x7f29240fc9e0 msgr2=0x7f29240fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 -- 172.21.15.28:0/2773648928 shutdown_connections 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 -- 172.21.15.28:0/2773648928 wait complete. 2024-03-30T14:55:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 librados: starting msgr at 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.658+0000 7f2928936700 1 librados: starting objecter 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 Processor -- start 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 -- start start 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 librados: setting wanted keys 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 librados: calling monclient init 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29241b5fb0 con 0x7f292410eb10 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29241b71d0 con 0x7f292410b000 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f2928936700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29241b83f0 con 0x7f29241031f0 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f292259c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f29241031f0 0x7f292410a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57204/0 (socket says 172.21.15.28:57204) 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f292259c700 1 -- 172.21.15.28:0/1257759645 learned_addr learned my addr 172.21.15.28:0/1257759645 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter ms_handle_connect 0x7f29241031f0 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter ms_handle_connect 0x7f292410eb10 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter ms_handle_connect 0x7f292410b000 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890435865 0 0) 0x7f29241b83f0 con 0x7f29241031f0 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f290c0031f0 con 0x7f29241031f0 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431658361 0 0) 0x7f29241b5fb0 con 0x7f292410eb10 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29241b83f0 con 0x7f292410eb10 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662090905 0 0) 0x7f29241b71d0 con 0x7f292410b000 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.659+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29241b5fb0 con 0x7f292410b000 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3239510320 0 0) 0x7f290c0031f0 con 0x7f29241031f0 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29241b71d0 con 0x7f29241031f0 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3814563503 0 0) 0x7f29241b83f0 con 0x7f292410eb10 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f290c0031f0 con 0x7f292410eb10 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2914001930 con 0x7f29241031f0 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f291c004880 con 0x7f292410eb10 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1330650157 0 0) 0x7f29241b5fb0 con 0x7f292410b000 2024-03-30T14:55:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29241b83f0 con 0x7f292410b000 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2918001890 con 0x7f292410b000 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1222386854 0 0) 0x7f29241b71d0 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 >> v1:172.21.15.134:6789/0 conn(0x7f292410b000 legacy=0x7f292419fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 >> v1:172.21.15.28:6789/0 conn(0x7f292410eb10 legacy=0x7f29241b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29241b9630 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29241b61e0 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f2928936700 10 client.34613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f29241b66a0 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f2928936700 1 librados: init done 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2914001cb0 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.660+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2914005d30 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.661+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2904005120 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f291401c570 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(91..91 src has 14..91) ==== 6259+0+0 (unknown 2721991072 0 0) 0x7f29140890f0 con 0x7f29241031f0 2024-03-30T14:55:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 10 client.34613.objecter ms_dispatch 0x7f2924001550 osd_map(91..91 src has 14..91) 2024-03-30T14:55:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 3 client.34613.objecter handle_osd_map got epochs [91,91] > 0 2024-03-30T14:55:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 3 client.34613.objecter handle_osd_map decoding full epoch 91 2024-03-30T14:55:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 20 client.34613.objecter dump_active .. 0 homeless 2024-03-30T14:55:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.662+0000 7f29137fe700 10 client.34613.objecter ms_handle_connect 0x7f290c0704d0 2024-03-30T14:55:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.671+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2914056ab0 con 0x7f29241031f0 2024-03-30T14:55:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.885+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2904005410 con 0x7f29241031f0 2024-03-30T14:55:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.886+0000 7f29137fe700 1 -- 172.21.15.28:0/1257759645 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1068569428) 0x7f291405a500 con 0x7f29241031f0 2024-03-30T14:55:50.887 INFO:teuthology.orchestra.run.smithi028.stdout:34359738422 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 10 librados: watch_flush enter 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 10 librados: watch_flush exit 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 20 client.34613.objecter shutdown clearing up homeless session... 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 10 client.34613.objecter successfully canceled tick 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 >> v1:172.21.15.28:6800/3011431093 conn(0x7f290c0704d0 legacy=0x7f290c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 >> v1:172.21.15.28:6790/0 conn(0x7f29241031f0 legacy=0x7f292410a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 shutdown_connections 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 >> 172.21.15.28:0/1257759645 conn(0x7f29240fc9e0 msgr2=0x7f2924110f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 shutdown_connections 2024-03-30T14:55:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 -- 172.21.15.28:0/1257759645 wait complete. 2024-03-30T14:55:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:50.888+0000 7f2928936700 1 librados: shutdown 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 2.1 deep-scrub starts 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 2.1 deep-scrub ok 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 5.f deep-scrub starts 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 5.f deep-scrub ok 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T14:55:51.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1257759645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 2.1 deep-scrub starts 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 2.1 deep-scrub ok 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 5.f deep-scrub starts 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 5.f deep-scrub ok 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T14:55:51.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1257759645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 2.1 deep-scrub starts 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 2.1 deep-scrub ok 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 5.f deep-scrub starts 2024-03-30T14:55:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 5.f deep-scrub ok 2024-03-30T14:55:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T14:55:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T14:55:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1257759645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T14:55:52.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[41614]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[29876]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T14:55:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:52 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T14:55:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:52 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T14:55:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:52 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T14:55:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:52 smithi134 ceph-mon[33714]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:52 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T14:55:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:52 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T14:55:53.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659160 got 176093659164 for osd.5 2024-03-30T14:55:53.531 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T14:55:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T14:55:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:53 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T14:55:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:53 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T14:55:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:53 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T14:55:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:53 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T14:55:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:53 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T14:55:54.368 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738422 for osd.0 2024-03-30T14:55:54.368 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T14:55:54.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T14:55:54.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:54 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: osdmap e92: 8 total, 8 up, 8 in 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T14:55:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:54 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T14:55:55.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:55:55.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:55 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:55:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:55 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T14:55:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:55 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T14:55:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:55 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:55:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:55 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T14:55:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T14:55:56.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:56 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T14:55:56.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: osdmap e93: 8 total, 8 up, 8 in 2024-03-30T14:55:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:55:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T14:55:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T14:55:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T14:55:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:56 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T14:55:57.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4039661198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T14:55:57.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4039661198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 -- 172.21.15.28:0/1972320195 >> v1:172.21.15.28:6789/0 conn(0x7ffa78101130 legacy=0x7ffa78101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 -- 172.21.15.28:0/1972320195 shutdown_connections 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 -- 172.21.15.28:0/1972320195 >> 172.21.15.28:0/1972320195 conn(0x7ffa780fc9e0 msgr2=0x7ffa780fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 -- 172.21.15.28:0/1972320195 shutdown_connections 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 -- 172.21.15.28:0/1972320195 wait complete. 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 librados: starting msgr at 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.776+0000 7ffa800e2700 1 librados: starting objecter 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 Processor -- start 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 -- start start 2024-03-30T14:55:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 librados: setting wanted keys 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 librados: calling monclient init 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa781a9310 con 0x7ffa78109dc0 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa781aa530 con 0x7ffa78101130 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa800e2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa781ab770 con 0x7ffa7810d960 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa7e67f700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ffa7810d960 0x7ffa781a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57214/0 (socket says 172.21.15.28:57214) 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa7d67d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffa78109dc0 0x7ffa781a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45342/0 (socket says 172.21.15.28:45342) 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa7e67f700 1 -- 172.21.15.28:0/4039661198 learned_addr learned my addr 172.21.15.28:0/4039661198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter ms_handle_connect 0x7ffa78109dc0 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter ms_handle_connect 0x7ffa7810d960 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122899198 0 0) 0x7ffa781a9310 con 0x7ffa78109dc0 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa600031f0 con 0x7ffa78109dc0 2024-03-30T14:55:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter ms_handle_connect 0x7ffa78101130 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520709475 0 0) 0x7ffa781ab770 con 0x7ffa7810d960 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa781a9310 con 0x7ffa7810d960 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861330826 0 0) 0x7ffa781aa530 con 0x7ffa78101130 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa781ab770 con 0x7ffa78101130 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4228789621 0 0) 0x7ffa600031f0 con 0x7ffa78109dc0 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa781aa530 con 0x7ffa78109dc0 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.777+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffa6c0018b0 con 0x7ffa78109dc0 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2705704827 0 0) 0x7ffa781a9310 con 0x7ffa7810d960 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa600031f0 con 0x7ffa7810d960 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffa680015b0 con 0x7ffa7810d960 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1013910845 0 0) 0x7ffa781aa530 con 0x7ffa78109dc0 2024-03-30T14:55:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 >> v1:172.21.15.28:6790/0 conn(0x7ffa7810d960 legacy=0x7ffa781a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 >> v1:172.21.15.134:6789/0 conn(0x7ffa78101130 legacy=0x7ffa78106330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa781ac9b0 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa781ab9a0 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa800e2700 10 client.14730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa781abe60 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa800e2700 1 librados: init done 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ffa6c003b40 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.778+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffa6c005d10 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.779+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa64005150 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.779+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ffa6c01c550 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(93..93 src has 43..93) ==== 6259+0+0 (unknown 1478577302 0 0) 0x7ffa6c0890b0 con 0x7ffa78109dc0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 10 client.14730.objecter ms_dispatch 0x7ffa78001550 osd_map(93..93 src has 43..93) 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 3 client.14730.objecter handle_osd_map got epochs [93,93] > 0 2024-03-30T14:55:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 3 client.14730.objecter handle_osd_map decoding full epoch 93 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 20 client.14730.objecter dump_active .. 0 homeless 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.780+0000 7ffa76ffd700 10 client.14730.objecter ms_handle_connect 0x7ffa600704c0 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.784+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ffa6c056a10 con 0x7ffa78109dc0 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.989+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ffa64005440 con 0x7ffa78109dc0 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.989+0000 7ffa76ffd700 1 -- 172.21.15.28:0/4039661198 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 2008330095) 0x7ffa6c05a460 con 0x7ffa78109dc0 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 10 librados: watch_flush enter 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 10 librados: watch_flush exit 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 20 client.14730.objecter shutdown clearing up homeless session... 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 10 client.14730.objecter successfully canceled tick 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 >> v1:172.21.15.28:6800/3011431093 conn(0x7ffa600704c0 legacy=0x7ffa60072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 >> v1:172.21.15.28:6789/0 conn(0x7ffa78109dc0 legacy=0x7ffa781a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 shutdown_connections 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 >> 172.21.15.28:0/4039661198 conn(0x7ffa780fc9e0 msgr2=0x7ffa78110a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 shutdown_connections 2024-03-30T14:55:57.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 -- 172.21.15.28:0/4039661198 wait complete. 2024-03-30T14:55:57.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:55:56.992+0000 7ffa800e2700 1 librados: shutdown 2024-03-30T14:55:57.928 INFO:teuthology.orchestra.run.smithi028.stdout:55834574900 2024-03-30T14:55:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:57 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T14:55:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:57 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T14:55:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:57 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T14:55:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:57 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T14:55:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4039661198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T14:55:58.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:55:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T14:55:58.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:55:58 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T14:55:58.553 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574900 for osd.1 2024-03-30T14:55:58.554 DEBUG:teuthology.parallel:result is None 2024-03-30T14:55:58.554 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T14:55:58.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:55:58.853 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:58 smithi028 podman[79154]: 2024-03-30 14:55:58.769258983 +0000 UTC m=+0.132607305 volume create 37bc9403c0d1922dd7f9ba012d0beee3390c524f7840a5a0a7ef1a52d33ed867 2024-03-30T14:55:58.853 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:58 smithi028 podman[79154]: 2024-03-30 14:55:58.852886649 +0000 UTC m=+0.216234969 container create 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T14:55:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T14:55:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T14:55:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T14:55:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:55:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T14:55:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T14:55:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T14:55:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:55:58 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T14:55:59.473 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 podman[79154]: 2024-03-30 14:55:59.180573548 +0000 UTC m=+0.543921878 container init 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:59.473 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 podman[79154]: 2024-03-30 14:55:59.185673821 +0000 UTC m=+0.549022141 container start 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:55:59.473 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.197Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T14:55:59.473 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.197Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T14:55:59.473 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.199Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-30T14:55:59.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.200Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T14:55:59.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.223Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T14:55:59.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.223Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T14:55:59.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.226Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T14:55:59.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:55:59.226Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T14:55:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:55:59.850 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 bash[79154]: 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd 2024-03-30T14:55:59.850 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:55:59 smithi028 systemd[1]: Started Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:55:59.850 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE Bus STOPPING 2024-03-30T14:55:59.850 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:55:59.851 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE Bus STOPPED 2024-03-30T14:55:59.851 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE Bus STARTING 2024-03-30T14:56:00.222 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE Serving on http://:::9283 2024-03-30T14:56:00.223 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:55:59 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:55:59] ENGINE Bus STARTED 2024-03-30T14:56:00.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:55:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: Deploying daemon grafana.a on smithi134 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T14:56:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:00 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: Deploying daemon grafana.a on smithi134 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T14:56:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: Deploying daemon grafana.a on smithi134 2024-03-30T14:56:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:00.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T14:56:00.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:00 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T14:56:01.410 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:56:01.201Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000521039s 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.623+0000 7fa9efa55700 1 -- 172.21.15.28:0/2818531861 >> v1:172.21.15.28:6789/0 conn(0x7fa9e810ea70 legacy=0x7fa9e8110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.623+0000 7fa9efa55700 1 -- 172.21.15.28:0/2818531861 shutdown_connections 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.623+0000 7fa9efa55700 1 -- 172.21.15.28:0/2818531861 >> 172.21.15.28:0/2818531861 conn(0x7fa9e80fdaf0 msgr2=0x7fa9e80fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.623+0000 7fa9efa55700 1 -- 172.21.15.28:0/2818531861 shutdown_connections 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.623+0000 7fa9efa55700 1 -- 172.21.15.28:0/2818531861 wait complete. 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 librados: starting msgr at 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 librados: starting objecter 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 Processor -- start 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 -- start start 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 librados: setting wanted keys 2024-03-30T14:56:01.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 librados: calling monclient init 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9e81a9340 con 0x7fa9e810aed0 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9e81aa560 con 0x7fa9e810ea70 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9efa55700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9e81ab7a0 con 0x7fa9e8102240 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9ed7f1700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa9e8102240 0x7fa9e8075f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50916/0 (socket says 172.21.15.28:50916) 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9ed7f1700 1 -- 172.21.15.28:0/2920006626 learned_addr learned my addr 172.21.15.28:0/2920006626 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter ms_handle_connect 0x7fa9e8102240 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter ms_handle_connect 0x7fa9e810aed0 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter ms_handle_connect 0x7fa9e810ea70 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032450718 0 0) 0x7fa9e81ab7a0 con 0x7fa9e8102240 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9d00039c0 con 0x7fa9e8102240 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019058342 0 0) 0x7fa9e81a9340 con 0x7fa9e810aed0 2024-03-30T14:56:01.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e81ab7a0 con 0x7fa9e810aed0 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.624+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173566983 0 0) 0x7fa9e81aa560 con 0x7fa9e810ea70 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e81a9340 con 0x7fa9e810ea70 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 254241984 0 0) 0x7fa9d00039c0 con 0x7fa9e8102240 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9e81aa560 con 0x7fa9e8102240 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa9e0001db0 con 0x7fa9e8102240 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1494301179 0 0) 0x7fa9e81ab7a0 con 0x7fa9e810aed0 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9d00039c0 con 0x7fa9e810aed0 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2416590107 0 0) 0x7fa9e81a9340 con 0x7fa9e810ea70 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9e81ab7a0 con 0x7fa9e810ea70 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa9dc0017f0 con 0x7fa9e810aed0 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa9d8001980 con 0x7fa9e810ea70 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1717590626 0 0) 0x7fa9e81aa560 con 0x7fa9e8102240 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 >> v1:172.21.15.134:6789/0 conn(0x7fa9e810ea70 legacy=0x7fa9e81a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 >> v1:172.21.15.28:6789/0 conn(0x7fa9e810aed0 legacy=0x7fa9e81a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e81ac9e0 con 0x7fa9e8102240 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e81a9570 con 0x7fa9e8102240 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9efa55700 10 client.34625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9e81a9a30 con 0x7fa9e8102240 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9efa55700 1 librados: init done 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa9e00020a0 con 0x7fa9e8102240 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.625+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa9e0006130 con 0x7fa9e8102240 2024-03-30T14:56:01.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.626+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d4005120 con 0x7fa9e8102240 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa9e0003cb0 con 0x7fa9e8102240 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(93..93 src has 43..93) ==== 6259+0+0 (unknown 1478577302 0 0) 0x7fa9e00892f0 con 0x7fa9e8102240 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 10 client.34625.objecter ms_dispatch 0x7fa9e8001550 osd_map(93..93 src has 43..93) 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 3 client.34625.objecter handle_osd_map got epochs [93,93] > 0 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 3 client.34625.objecter handle_osd_map decoding full epoch 93 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 20 client.34625.objecter dump_active .. 0 homeless 2024-03-30T14:56:01.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.627+0000 7fa9e67fc700 10 client.34625.objecter ms_handle_connect 0x7fa9d0070510 2024-03-30T14:56:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.632+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa9e0006130 con 0x7fa9e8102240 2024-03-30T14:56:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T14:56:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T14:56:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:01 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T14:56:01.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.832+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9d4000b90 con 0x7fa9d0070510 2024-03-30T14:56:01.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.841+0000 7fa9e67fc700 1 -- 172.21.15.28:0/2920006626 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+345403 (unknown 2965378022 0 2448880789) 0x7fa9d4000b90 con 0x7fa9d0070510 2024-03-30T14:56:01.848 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 10 librados: watch_flush enter 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 10 librados: watch_flush exit 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 20 client.34625.objecter shutdown clearing up homeless session... 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 10 client.34625.objecter successfully canceled tick 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa9d0070510 legacy=0x7fa9d00729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.844+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 >> v1:172.21.15.28:6790/0 conn(0x7fa9e8102240 legacy=0x7fa9e8075f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.845+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 shutdown_connections 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.845+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 >> 172.21.15.28:0/2920006626 conn(0x7fa9e80fdaf0 msgr2=0x7fa9e8111f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.845+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 shutdown_connections 2024-03-30T14:56:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.845+0000 7fa9efa55700 1 -- 172.21.15.28:0/2920006626 wait complete. 2024-03-30T14:56:01.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:01.845+0000 7fa9efa55700 1 librados: shutdown 2024-03-30T14:56:01.852 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T14:56:01.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:01 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T14:56:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:01 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T14:56:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:01 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T14:56:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:01 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T14:56:02.754 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-30T14:56:01.586013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1008,"num_read_kb":854,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249428,"kb_used_data":8316,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711788,"statfs":{"total":767960285184,"available":767704870912,"internally_reserved":0,"allocated":8515584,"data_stored":4726698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.140440"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689208+0000","last_change":"2024-03-30T14:54:43.351532+0000","last_active":"2024-03-30T14:55:55.689208+0000","last_peered":"2024-03-30T14:55:55.689208+0000","last_clean":"2024-03-30T14:55:55.689208+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T14:55:55.689208+0000","last_undegraded":"2024-03-30T14:55:55.689208+0000","last_fullsized":"2024-03-30T14:55:55.689208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.847289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688920+0000","last_change":"2024-03-30T14:54:41.350461+0000","last_active":"2024-03-30T14:55:55.688920+0000","last_peered":"2024-03-30T14:55:55.688920+0000","last_clean":"2024-03-30T14:55:55.688920+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T14:55:55.688920+0000","last_undegraded":"2024-03-30T14:55:55.688920+0000","last_fullsized":"2024-03-30T14:55:55.688920+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.461361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454677+0000","last_change":"2024-03-30T14:55:46.590443+0000","last_active":"2024-03-30T14:55:55.454677+0000","last_peered":"2024-03-30T14:55:55.454677+0000","last_clean":"2024-03-30T14:55:55.454677+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T14:55:55.454677+0000","last_undegraded":"2024-03-30T14:55:55.454677+0000","last_fullsized":"2024-03-30T14:55:55.454677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.590360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.590360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.726057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688646+0000","last_change":"2024-03-30T14:55:40.318444+0000","last_active":"2024-03-30T14:55:55.688646+0000","last_peered":"2024-03-30T14:55:55.688646+0000","last_clean":"2024-03-30T14:55:55.688646+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T14:55:55.688646+0000","last_undegraded":"2024-03-30T14:55:55.688646+0000","last_fullsized":"2024-03-30T14:55:55.688646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:40.318295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.318295+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:45.878339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455375+0000","last_change":"2024-03-30T14:54:43.359234+0000","last_active":"2024-03-30T14:55:55.455375+0000","last_peered":"2024-03-30T14:55:55.455375+0000","last_clean":"2024-03-30T14:55:55.455375+0000","last_became_active":"2024-03-30T14:54:43.358170+0000","last_became_peered":"2024-03-30T14:54:43.358170+0000","last_unstale":"2024-03-30T14:55:55.455375+0000","last_undegraded":"2024-03-30T14:55:55.455375+0000","last_fullsized":"2024-03-30T14:55:55.455375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.637785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688547+0000","last_change":"2024-03-30T14:54:41.350763+0000","last_active":"2024-03-30T14:55:55.688547+0000","last_peered":"2024-03-30T14:55:55.688547+0000","last_clean":"2024-03-30T14:55:55.688547+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T14:55:55.688547+0000","last_undegraded":"2024-03-30T14:55:55.688547+0000","last_fullsized":"2024-03-30T14:55:55.688547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.735595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456587+0000","last_change":"2024-03-30T14:55:41.765389+0000","last_active":"2024-03-30T14:55:55.456587+0000","last_peered":"2024-03-30T14:55:55.456587+0000","last_clean":"2024-03-30T14:55:55.456587+0000","last_became_active":"2024-03-30T14:54:37.474114+0000","last_became_peered":"2024-03-30T14:54:37.474114+0000","last_unstale":"2024-03-30T14:55:55.456587+0000","last_undegraded":"2024-03-30T14:55:55.456587+0000","last_fullsized":"2024-03-30T14:55:55.456587+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.765306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.765306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:43.228077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"76'15","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456124+0000","last_change":"2024-03-30T14:54:39.333923+0000","last_active":"2024-03-30T14:55:55.456124+0000","last_peered":"2024-03-30T14:55:55.456124+0000","last_clean":"2024-03-30T14:55:55.456124+0000","last_became_active":"2024-03-30T14:54:39.333512+0000","last_became_peered":"2024-03-30T14:54:39.333512+0000","last_unstale":"2024-03-30T14:55:55.456124+0000","last_undegraded":"2024-03-30T14:55:55.456124+0000","last_fullsized":"2024-03-30T14:55:55.456124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.530539+0000","last_change":"2024-03-30T14:55:57.530576+0000","last_active":"2024-03-30T14:55:57.530539+0000","last_peered":"2024-03-30T14:55:57.530539+0000","last_clean":"2024-03-30T14:55:57.530539+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T14:55:57.530539+0000","last_undegraded":"2024-03-30T14:55:57.530539+0000","last_fullsized":"2024-03-30T14:55:57.530539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.959060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688952+0000","last_change":"2024-03-30T14:55:43.421378+0000","last_active":"2024-03-30T14:55:55.688952+0000","last_peered":"2024-03-30T14:55:55.688952+0000","last_clean":"2024-03-30T14:55:55.688952+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T14:55:55.688952+0000","last_undegraded":"2024-03-30T14:55:55.688952+0000","last_fullsized":"2024-03-30T14:55:55.688952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:43.421270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:43.421270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.069402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691260+0000","last_change":"2024-03-30T14:55:46.387117+0000","last_active":"2024-03-30T14:55:55.691260+0000","last_peered":"2024-03-30T14:55:55.691260+0000","last_clean":"2024-03-30T14:55:55.691260+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T14:55:55.691260+0000","last_undegraded":"2024-03-30T14:55:55.691260+0000","last_fullsized":"2024-03-30T14:55:55.691260+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.387017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.387017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.976653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689501+0000","last_change":"2024-03-30T14:54:39.329805+0000","last_active":"2024-03-30T14:55:55.689501+0000","last_peered":"2024-03-30T14:55:55.689501+0000","last_clean":"2024-03-30T14:55:55.689501+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T14:55:55.689501+0000","last_undegraded":"2024-03-30T14:55:55.689501+0000","last_fullsized":"2024-03-30T14:55:55.689501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.308058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691252+0000","last_change":"2024-03-30T14:55:49.394236+0000","last_active":"2024-03-30T14:55:55.691252+0000","last_peered":"2024-03-30T14:55:55.691252+0000","last_clean":"2024-03-30T14:55:55.691252+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T14:55:55.691252+0000","last_undegraded":"2024-03-30T14:55:55.691252+0000","last_fullsized":"2024-03-30T14:55:55.691252+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.394138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.394138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.971320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689068+0000","last_change":"2024-03-30T14:54:41.352116+0000","last_active":"2024-03-30T14:55:55.689068+0000","last_peered":"2024-03-30T14:55:55.689068+0000","last_clean":"2024-03-30T14:55:55.689068+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T14:55:55.689068+0000","last_undegraded":"2024-03-30T14:55:55.689068+0000","last_fullsized":"2024-03-30T14:55:55.689068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.338455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691370+0000","last_change":"2024-03-30T14:54:37.331434+0000","last_active":"2024-03-30T14:55:55.691370+0000","last_peered":"2024-03-30T14:55:55.691370+0000","last_clean":"2024-03-30T14:55:55.691370+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T14:55:55.691370+0000","last_undegraded":"2024-03-30T14:55:55.691370+0000","last_fullsized":"2024-03-30T14:55:55.691370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689408+0000","last_change":"2024-03-30T14:54:58.404856+0000","last_active":"2024-03-30T14:55:55.689408+0000","last_peered":"2024-03-30T14:55:55.689408+0000","last_clean":"2024-03-30T14:55:55.689408+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T14:55:55.689408+0000","last_undegraded":"2024-03-30T14:55:55.689408+0000","last_fullsized":"2024-03-30T14:55:55.689408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.073767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690553+0000","last_change":"2024-03-30T14:54:43.349041+0000","last_active":"2024-03-30T14:55:55.690553+0000","last_peered":"2024-03-30T14:55:55.690553+0000","last_clean":"2024-03-30T14:55:55.690553+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T14:55:55.690553+0000","last_undegraded":"2024-03-30T14:55:55.690553+0000","last_fullsized":"2024-03-30T14:55:55.690553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.764286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689511+0000","last_change":"2024-03-30T14:54:41.347727+0000","last_active":"2024-03-30T14:55:55.689511+0000","last_peered":"2024-03-30T14:55:55.689511+0000","last_clean":"2024-03-30T14:55:55.689511+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T14:55:55.689511+0000","last_undegraded":"2024-03-30T14:55:55.689511+0000","last_fullsized":"2024-03-30T14:55:55.689511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.353560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453918+0000","last_change":"2024-03-30T14:55:42.710206+0000","last_active":"2024-03-30T14:55:55.453918+0000","last_peered":"2024-03-30T14:55:55.453918+0000","last_clean":"2024-03-30T14:55:55.453918+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T14:55:55.453918+0000","last_undegraded":"2024-03-30T14:55:55.453918+0000","last_fullsized":"2024-03-30T14:55:55.453918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.710118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.710118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.005945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"76'15","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688833+0000","last_change":"2024-03-30T14:55:39.803863+0000","last_active":"2024-03-30T14:55:55.688833+0000","last_peered":"2024-03-30T14:55:55.688833+0000","last_clean":"2024-03-30T14:55:55.688833+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T14:55:55.688833+0000","last_undegraded":"2024-03-30T14:55:55.688833+0000","last_fullsized":"2024-03-30T14:55:55.688833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:55:39.803752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:39.803752+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.649577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456435+0000","last_change":"2024-03-30T14:54:43.354706+0000","last_active":"2024-03-30T14:55:55.456435+0000","last_peered":"2024-03-30T14:55:55.456435+0000","last_clean":"2024-03-30T14:55:55.456435+0000","last_became_active":"2024-03-30T14:54:43.354397+0000","last_became_peered":"2024-03-30T14:54:43.354397+0000","last_unstale":"2024-03-30T14:55:55.456435+0000","last_undegraded":"2024-03-30T14:55:55.456435+0000","last_fullsized":"2024-03-30T14:55:55.456435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.574805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689778+0000","last_change":"2024-03-30T14:54:41.348874+0000","last_active":"2024-03-30T14:55:55.689778+0000","last_peered":"2024-03-30T14:55:55.689778+0000","last_clean":"2024-03-30T14:55:55.689778+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T14:55:55.689778+0000","last_undegraded":"2024-03-30T14:55:55.689778+0000","last_fullsized":"2024-03-30T14:55:55.689778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.203011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689612+0000","last_change":"2024-03-30T14:54:37.474896+0000","last_active":"2024-03-30T14:55:55.689612+0000","last_peered":"2024-03-30T14:55:55.689612+0000","last_clean":"2024-03-30T14:55:55.689612+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T14:55:55.689612+0000","last_undegraded":"2024-03-30T14:55:55.689612+0000","last_fullsized":"2024-03-30T14:55:55.689612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456337+0000","last_change":"2024-03-30T14:54:39.339362+0000","last_active":"2024-03-30T14:55:55.456337+0000","last_peered":"2024-03-30T14:55:55.456337+0000","last_clean":"2024-03-30T14:55:55.456337+0000","last_became_active":"2024-03-30T14:54:39.338927+0000","last_became_peered":"2024-03-30T14:54:39.338927+0000","last_unstale":"2024-03-30T14:55:55.456337+0000","last_undegraded":"2024-03-30T14:55:55.456337+0000","last_fullsized":"2024-03-30T14:55:55.456337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"76'1","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689295+0000","last_change":"2024-03-30T14:54:43.354908+0000","last_active":"2024-03-30T14:55:55.689295+0000","last_peered":"2024-03-30T14:55:55.689295+0000","last_clean":"2024-03-30T14:55:55.689295+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T14:55:55.689295+0000","last_undegraded":"2024-03-30T14:55:55.689295+0000","last_fullsized":"2024-03-30T14:55:55.689295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.852744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:58.120389+0000","last_change":"2024-03-30T14:55:58.120425+0000","last_active":"2024-03-30T14:55:58.120389+0000","last_peered":"2024-03-30T14:55:58.120389+0000","last_clean":"2024-03-30T14:55:58.120389+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T14:55:58.120389+0000","last_undegraded":"2024-03-30T14:55:58.120389+0000","last_fullsized":"2024-03-30T14:55:58.120389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:58.120314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:58.120314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.131781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689019+0000","last_change":"2024-03-30T14:54:37.324901+0000","last_active":"2024-03-30T14:55:55.689019+0000","last_peered":"2024-03-30T14:55:55.689019+0000","last_clean":"2024-03-30T14:55:55.689019+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T14:55:55.689019+0000","last_undegraded":"2024-03-30T14:55:55.689019+0000","last_fullsized":"2024-03-30T14:55:55.689019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.138775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"76'10","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454724+0000","last_change":"2024-03-30T14:54:39.642246+0000","last_active":"2024-03-30T14:55:55.454724+0000","last_peered":"2024-03-30T14:55:55.454724+0000","last_clean":"2024-03-30T14:55:55.454724+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T14:55:55.454724+0000","last_undegraded":"2024-03-30T14:55:55.454724+0000","last_fullsized":"2024-03-30T14:55:55.454724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.644108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689676+0000","last_change":"2024-03-30T14:55:49.280406+0000","last_active":"2024-03-30T14:55:55.689676+0000","last_peered":"2024-03-30T14:55:55.689676+0000","last_clean":"2024-03-30T14:55:55.689676+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T14:55:55.689676+0000","last_undegraded":"2024-03-30T14:55:55.689676+0000","last_fullsized":"2024-03-30T14:55:55.689676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.280327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.280327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.497970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454883+0000","last_change":"2024-03-30T14:54:41.347270+0000","last_active":"2024-03-30T14:55:55.454883+0000","last_peered":"2024-03-30T14:55:55.454883+0000","last_clean":"2024-03-30T14:55:55.454883+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T14:55:55.454883+0000","last_undegraded":"2024-03-30T14:55:55.454883+0000","last_fullsized":"2024-03-30T14:55:55.454883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.840620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455463+0000","last_change":"2024-03-30T14:54:37.321125+0000","last_active":"2024-03-30T14:55:55.455463+0000","last_peered":"2024-03-30T14:55:55.455463+0000","last_clean":"2024-03-30T14:55:55.455463+0000","last_became_active":"2024-03-30T14:54:37.320726+0000","last_became_peered":"2024-03-30T14:54:37.320726+0000","last_unstale":"2024-03-30T14:55:55.455463+0000","last_undegraded":"2024-03-30T14:55:55.455463+0000","last_fullsized":"2024-03-30T14:55:55.455463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.061192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'11","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583631+0000","last_change":"2024-03-30T14:54:39.338805+0000","last_active":"2024-03-30T14:55:55.583631+0000","last_peered":"2024-03-30T14:55:55.583631+0000","last_clean":"2024-03-30T14:55:55.583631+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T14:55:55.583631+0000","last_undegraded":"2024-03-30T14:55:55.583631+0000","last_fullsized":"2024-03-30T14:55:55.583631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.870023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688948+0000","last_change":"2024-03-30T14:54:43.352814+0000","last_active":"2024-03-30T14:55:55.688948+0000","last_peered":"2024-03-30T14:55:55.688948+0000","last_clean":"2024-03-30T14:55:55.688948+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T14:55:55.688948+0000","last_undegraded":"2024-03-30T14:55:55.688948+0000","last_fullsized":"2024-03-30T14:55:55.688948+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.775606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.582874+0000","last_change":"2024-03-30T14:55:53.203651+0000","last_active":"2024-03-30T14:55:55.582874+0000","last_peered":"2024-03-30T14:55:55.582874+0000","last_clean":"2024-03-30T14:55:55.582874+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T14:55:55.582874+0000","last_undegraded":"2024-03-30T14:55:55.582874+0000","last_fullsized":"2024-03-30T14:55:55.582874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.203533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.203533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:56.173167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454668+0000","last_change":"2024-03-30T14:54:37.324115+0000","last_active":"2024-03-30T14:55:55.454668+0000","last_peered":"2024-03-30T14:55:55.454668+0000","last_clean":"2024-03-30T14:55:55.454668+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T14:55:55.454668+0000","last_undegraded":"2024-03-30T14:55:55.454668+0000","last_fullsized":"2024-03-30T14:55:55.454668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455583+0000","last_change":"2024-03-30T14:54:39.338811+0000","last_active":"2024-03-30T14:55:55.455583+0000","last_peered":"2024-03-30T14:55:55.455583+0000","last_clean":"2024-03-30T14:55:55.455583+0000","last_became_active":"2024-03-30T14:54:39.338462+0000","last_became_peered":"2024-03-30T14:54:39.338462+0000","last_unstale":"2024-03-30T14:55:55.455583+0000","last_undegraded":"2024-03-30T14:55:55.455583+0000","last_fullsized":"2024-03-30T14:55:55.455583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455118+0000","last_change":"2024-03-30T14:54:43.361222+0000","last_active":"2024-03-30T14:55:55.455118+0000","last_peered":"2024-03-30T14:55:55.455118+0000","last_clean":"2024-03-30T14:55:55.455118+0000","last_became_active":"2024-03-30T14:54:43.360837+0000","last_became_peered":"2024-03-30T14:54:43.360837+0000","last_unstale":"2024-03-30T14:55:55.455118+0000","last_undegraded":"2024-03-30T14:55:55.455118+0000","last_fullsized":"2024-03-30T14:55:55.455118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.092716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691130+0000","last_change":"2024-03-30T14:54:41.354167+0000","last_active":"2024-03-30T14:55:55.691130+0000","last_peered":"2024-03-30T14:55:55.691130+0000","last_clean":"2024-03-30T14:55:55.691130+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T14:55:55.691130+0000","last_undegraded":"2024-03-30T14:55:55.691130+0000","last_fullsized":"2024-03-30T14:55:55.691130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:08.473023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689253+0000","last_change":"2024-03-30T14:54:37.331035+0000","last_active":"2024-03-30T14:55:55.689253+0000","last_peered":"2024-03-30T14:55:55.689253+0000","last_clean":"2024-03-30T14:55:55.689253+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T14:55:55.689253+0000","last_undegraded":"2024-03-30T14:55:55.689253+0000","last_fullsized":"2024-03-30T14:55:55.689253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.994637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456121+0000","last_change":"2024-03-30T14:54:39.641424+0000","last_active":"2024-03-30T14:55:55.456121+0000","last_peered":"2024-03-30T14:55:55.456121+0000","last_clean":"2024-03-30T14:55:55.456121+0000","last_became_active":"2024-03-30T14:54:39.641142+0000","last_became_peered":"2024-03-30T14:54:39.641142+0000","last_unstale":"2024-03-30T14:55:55.456121+0000","last_undegraded":"2024-03-30T14:55:55.456121+0000","last_fullsized":"2024-03-30T14:55:55.456121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453011+0000","last_change":"2024-03-30T14:54:43.352784+0000","last_active":"2024-03-30T14:55:55.453011+0000","last_peered":"2024-03-30T14:55:55.453011+0000","last_clean":"2024-03-30T14:55:55.453011+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T14:55:55.453011+0000","last_undegraded":"2024-03-30T14:55:55.453011+0000","last_fullsized":"2024-03-30T14:55:55.453011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.948790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455487+0000","last_change":"2024-03-30T14:54:41.347572+0000","last_active":"2024-03-30T14:55:55.455487+0000","last_peered":"2024-03-30T14:55:55.455487+0000","last_clean":"2024-03-30T14:55:55.455487+0000","last_became_active":"2024-03-30T14:54:41.347229+0000","last_became_peered":"2024-03-30T14:54:41.347229+0000","last_unstale":"2024-03-30T14:55:55.455487+0000","last_undegraded":"2024-03-30T14:55:55.455487+0000","last_fullsized":"2024-03-30T14:55:55.455487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.472022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"69'1","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689252+0000","last_change":"2024-03-30T14:54:37.330051+0000","last_active":"2024-03-30T14:55:55.689252+0000","last_peered":"2024-03-30T14:55:55.689252+0000","last_clean":"2024-03-30T14:55:55.689252+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T14:55:55.689252+0000","last_undegraded":"2024-03-30T14:55:55.689252+0000","last_fullsized":"2024-03-30T14:55:55.689252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689794+0000","last_change":"2024-03-30T14:54:39.339910+0000","last_active":"2024-03-30T14:55:55.689794+0000","last_peered":"2024-03-30T14:55:55.689794+0000","last_clean":"2024-03-30T14:55:55.689794+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T14:55:55.689794+0000","last_undegraded":"2024-03-30T14:55:55.689794+0000","last_fullsized":"2024-03-30T14:55:55.689794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.626451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455873+0000","last_change":"2024-03-30T14:54:43.360542+0000","last_active":"2024-03-30T14:55:55.455873+0000","last_peered":"2024-03-30T14:55:55.455873+0000","last_clean":"2024-03-30T14:55:55.455873+0000","last_became_active":"2024-03-30T14:54:43.360215+0000","last_became_peered":"2024-03-30T14:54:43.360215+0000","last_unstale":"2024-03-30T14:55:55.455873+0000","last_undegraded":"2024-03-30T14:55:55.455873+0000","last_fullsized":"2024-03-30T14:55:55.455873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.894321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689567+0000","last_change":"2024-03-30T14:55:53.260188+0000","last_active":"2024-03-30T14:55:55.689567+0000","last_peered":"2024-03-30T14:55:55.689567+0000","last_clean":"2024-03-30T14:55:55.689567+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T14:55:55.689567+0000","last_undegraded":"2024-03-30T14:55:55.689567+0000","last_fullsized":"2024-03-30T14:55:55.689567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.260101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.260101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.452479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690060+0000","last_change":"2024-03-30T14:54:37.330248+0000","last_active":"2024-03-30T14:55:55.690060+0000","last_peered":"2024-03-30T14:55:55.690060+0000","last_clean":"2024-03-30T14:55:55.690060+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T14:55:55.690060+0000","last_undegraded":"2024-03-30T14:55:55.690060+0000","last_fullsized":"2024-03-30T14:55:55.690060+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"76'11","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689582+0000","last_change":"2024-03-30T14:54:39.334527+0000","last_active":"2024-03-30T14:55:55.689582+0000","last_peered":"2024-03-30T14:55:55.689582+0000","last_clean":"2024-03-30T14:55:55.689582+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T14:55:55.689582+0000","last_undegraded":"2024-03-30T14:55:55.689582+0000","last_fullsized":"2024-03-30T14:55:55.689582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.307222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"76'1","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454391+0000","last_change":"2024-03-30T14:54:43.354400+0000","last_active":"2024-03-30T14:55:55.454391+0000","last_peered":"2024-03-30T14:55:55.454391+0000","last_clean":"2024-03-30T14:55:55.454391+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T14:55:55.454391+0000","last_undegraded":"2024-03-30T14:55:55.454391+0000","last_fullsized":"2024-03-30T14:55:55.454391+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.205644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691112+0000","last_change":"2024-03-30T14:54:41.350464+0000","last_active":"2024-03-30T14:55:55.691112+0000","last_peered":"2024-03-30T14:55:55.691112+0000","last_clean":"2024-03-30T14:55:55.691112+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T14:55:55.691112+0000","last_undegraded":"2024-03-30T14:55:55.691112+0000","last_fullsized":"2024-03-30T14:55:55.691112+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.886706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690443+0000","last_change":"2024-03-30T14:54:37.474723+0000","last_active":"2024-03-30T14:55:55.690443+0000","last_peered":"2024-03-30T14:55:55.690443+0000","last_clean":"2024-03-30T14:55:55.690443+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T14:55:55.690443+0000","last_undegraded":"2024-03-30T14:55:55.690443+0000","last_fullsized":"2024-03-30T14:55:55.690443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.251496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455998+0000","last_change":"2024-03-30T14:55:53.742586+0000","last_active":"2024-03-30T14:55:55.455998+0000","last_peered":"2024-03-30T14:55:55.455998+0000","last_clean":"2024-03-30T14:55:55.455998+0000","last_became_active":"2024-03-30T14:54:39.642132+0000","last_became_peered":"2024-03-30T14:54:39.642132+0000","last_unstale":"2024-03-30T14:55:55.455998+0000","last_undegraded":"2024-03-30T14:55:55.455998+0000","last_fullsized":"2024-03-30T14:55:55.455998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:53.742492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.742492+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.865684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690251+0000","last_change":"2024-03-30T14:54:43.349809+0000","last_active":"2024-03-30T14:55:55.690251+0000","last_peered":"2024-03-30T14:55:55.690251+0000","last_clean":"2024-03-30T14:55:55.690251+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T14:55:55.690251+0000","last_undegraded":"2024-03-30T14:55:55.690251+0000","last_fullsized":"2024-03-30T14:55:55.690251+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.367821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"76'8","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456301+0000","last_change":"2024-03-30T14:54:41.346770+0000","last_active":"2024-03-30T14:55:55.456301+0000","last_peered":"2024-03-30T14:55:55.456301+0000","last_clean":"2024-03-30T14:55:55.456301+0000","last_became_active":"2024-03-30T14:54:41.346418+0000","last_became_peered":"2024-03-30T14:54:41.346418+0000","last_unstale":"2024-03-30T14:55:55.456301+0000","last_undegraded":"2024-03-30T14:55:55.456301+0000","last_fullsized":"2024-03-30T14:55:55.456301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453645+0000","last_change":"2024-03-30T14:54:37.325180+0000","last_active":"2024-03-30T14:55:55.453645+0000","last_peered":"2024-03-30T14:55:55.453645+0000","last_clean":"2024-03-30T14:55:55.453645+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T14:55:55.453645+0000","last_undegraded":"2024-03-30T14:55:55.453645+0000","last_fullsized":"2024-03-30T14:55:55.453645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691552+0000","last_change":"2024-03-30T14:54:39.644472+0000","last_active":"2024-03-30T14:55:55.691552+0000","last_peered":"2024-03-30T14:55:55.691552+0000","last_clean":"2024-03-30T14:55:55.691552+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T14:55:55.691552+0000","last_undegraded":"2024-03-30T14:55:55.691552+0000","last_fullsized":"2024-03-30T14:55:55.691552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.992331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689079+0000","last_change":"2024-03-30T14:55:48.721601+0000","last_active":"2024-03-30T14:55:55.689079+0000","last_peered":"2024-03-30T14:55:55.689079+0000","last_clean":"2024-03-30T14:55:55.689079+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T14:55:55.689079+0000","last_undegraded":"2024-03-30T14:55:55.689079+0000","last_fullsized":"2024-03-30T14:55:55.689079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.721477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.721477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:58.996126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456733+0000","last_change":"2024-03-30T14:54:41.345899+0000","last_active":"2024-03-30T14:55:55.456733+0000","last_peered":"2024-03-30T14:55:55.456733+0000","last_clean":"2024-03-30T14:55:55.456733+0000","last_became_active":"2024-03-30T14:54:41.345543+0000","last_became_peered":"2024-03-30T14:54:41.345543+0000","last_unstale":"2024-03-30T14:55:55.456733+0000","last_undegraded":"2024-03-30T14:55:55.456733+0000","last_fullsized":"2024-03-30T14:55:55.456733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689695+0000","last_change":"2024-03-30T14:55:48.068870+0000","last_active":"2024-03-30T14:55:55.689695+0000","last_peered":"2024-03-30T14:55:55.689695+0000","last_clean":"2024-03-30T14:55:55.689695+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T14:55:55.689695+0000","last_undegraded":"2024-03-30T14:55:55.689695+0000","last_fullsized":"2024-03-30T14:55:55.689695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.068767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.068767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:16.799841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454788+0000","last_change":"2024-03-30T14:54:39.642222+0000","last_active":"2024-03-30T14:55:55.454788+0000","last_peered":"2024-03-30T14:55:55.454788+0000","last_clean":"2024-03-30T14:55:55.454788+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T14:55:55.454788+0000","last_undegraded":"2024-03-30T14:55:55.454788+0000","last_fullsized":"2024-03-30T14:55:55.454788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.598823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689141+0000","last_change":"2024-03-30T14:54:43.357877+0000","last_active":"2024-03-30T14:55:55.689141+0000","last_peered":"2024-03-30T14:55:55.689141+0000","last_clean":"2024-03-30T14:55:55.689141+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T14:55:55.689141+0000","last_undegraded":"2024-03-30T14:55:55.689141+0000","last_fullsized":"2024-03-30T14:55:55.689141+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.019353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689345+0000","last_change":"2024-03-30T14:54:41.352746+0000","last_active":"2024-03-30T14:55:55.689345+0000","last_peered":"2024-03-30T14:55:55.689345+0000","last_clean":"2024-03-30T14:55:55.689345+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T14:55:55.689345+0000","last_undegraded":"2024-03-30T14:55:55.689345+0000","last_fullsized":"2024-03-30T14:55:55.689345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.582844+0000","last_change":"2024-03-30T14:55:50.191458+0000","last_active":"2024-03-30T14:55:55.582844+0000","last_peered":"2024-03-30T14:55:55.582844+0000","last_clean":"2024-03-30T14:55:55.582844+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T14:55:55.582844+0000","last_undegraded":"2024-03-30T14:55:55.582844+0000","last_fullsized":"2024-03-30T14:55:55.582844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.191353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.191353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:00.197326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456795+0000","last_change":"2024-03-30T14:55:48.827002+0000","last_active":"2024-03-30T14:55:55.456795+0000","last_peered":"2024-03-30T14:55:55.456795+0000","last_clean":"2024-03-30T14:55:55.456795+0000","last_became_active":"2024-03-30T14:54:39.643439+0000","last_became_peered":"2024-03-30T14:54:39.643439+0000","last_unstale":"2024-03-30T14:55:55.456795+0000","last_undegraded":"2024-03-30T14:55:55.456795+0000","last_fullsized":"2024-03-30T14:55:55.456795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:55:48.826912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.826912+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.602167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691673+0000","last_change":"2024-03-30T14:54:43.352897+0000","last_active":"2024-03-30T14:55:55.691673+0000","last_peered":"2024-03-30T14:55:55.691673+0000","last_clean":"2024-03-30T14:55:55.691673+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:55:55.691673+0000","last_undegraded":"2024-03-30T14:55:55.691673+0000","last_fullsized":"2024-03-30T14:55:55.691673+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.730639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690180+0000","last_change":"2024-03-30T14:54:37.330956+0000","last_active":"2024-03-30T14:55:55.690180+0000","last_peered":"2024-03-30T14:55:55.690180+0000","last_clean":"2024-03-30T14:55:55.690180+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T14:55:55.690180+0000","last_undegraded":"2024-03-30T14:55:55.690180+0000","last_fullsized":"2024-03-30T14:55:55.690180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.101820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688552+0000","last_change":"2024-03-30T14:54:39.643618+0000","last_active":"2024-03-30T14:55:55.688552+0000","last_peered":"2024-03-30T14:55:55.688552+0000","last_clean":"2024-03-30T14:55:55.688552+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T14:55:55.688552+0000","last_undegraded":"2024-03-30T14:55:55.688552+0000","last_fullsized":"2024-03-30T14:55:55.688552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.788995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689986+0000","last_change":"2024-03-30T14:54:41.352889+0000","last_active":"2024-03-30T14:55:55.689986+0000","last_peered":"2024-03-30T14:55:55.689986+0000","last_clean":"2024-03-30T14:55:55.689986+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T14:55:55.689986+0000","last_undegraded":"2024-03-30T14:55:55.689986+0000","last_fullsized":"2024-03-30T14:55:55.689986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.905986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455233+0000","last_change":"2024-03-30T14:54:43.358959+0000","last_active":"2024-03-30T14:55:55.455233+0000","last_peered":"2024-03-30T14:55:55.455233+0000","last_clean":"2024-03-30T14:55:55.455233+0000","last_became_active":"2024-03-30T14:54:43.358552+0000","last_became_peered":"2024-03-30T14:54:43.358552+0000","last_unstale":"2024-03-30T14:55:55.455233+0000","last_undegraded":"2024-03-30T14:55:55.455233+0000","last_fullsized":"2024-03-30T14:55:55.455233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.302782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689652+0000","last_change":"2024-03-30T14:54:37.328695+0000","last_active":"2024-03-30T14:55:55.689652+0000","last_peered":"2024-03-30T14:55:55.689652+0000","last_clean":"2024-03-30T14:55:55.689652+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T14:55:55.689652+0000","last_undegraded":"2024-03-30T14:55:55.689652+0000","last_fullsized":"2024-03-30T14:55:55.689652+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.320517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.431740+0000","last_change":"2024-03-30T14:55:56.431786+0000","last_active":"2024-03-30T14:55:56.431740+0000","last_peered":"2024-03-30T14:55:56.431740+0000","last_clean":"2024-03-30T14:55:56.431740+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T14:55:56.431740+0000","last_undegraded":"2024-03-30T14:55:56.431740+0000","last_fullsized":"2024-03-30T14:55:56.431740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:56.431647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.431647+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.459883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688280+0000","last_change":"2024-03-30T14:54:41.350489+0000","last_active":"2024-03-30T14:55:55.688280+0000","last_peered":"2024-03-30T14:55:55.688280+0000","last_clean":"2024-03-30T14:55:55.688280+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T14:55:55.688280+0000","last_undegraded":"2024-03-30T14:55:55.688280+0000","last_fullsized":"2024-03-30T14:55:55.688280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.343336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688763+0000","last_change":"2024-03-30T14:54:43.356146+0000","last_active":"2024-03-30T14:55:55.688763+0000","last_peered":"2024-03-30T14:55:55.688763+0000","last_clean":"2024-03-30T14:55:55.688763+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T14:55:55.688763+0000","last_undegraded":"2024-03-30T14:55:55.688763+0000","last_fullsized":"2024-03-30T14:55:55.688763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.912258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688621+0000","last_change":"2024-03-30T14:54:41.341454+0000","last_active":"2024-03-30T14:55:55.688621+0000","last_peered":"2024-03-30T14:55:55.688621+0000","last_clean":"2024-03-30T14:55:55.688621+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T14:55:55.688621+0000","last_undegraded":"2024-03-30T14:55:55.688621+0000","last_fullsized":"2024-03-30T14:55:55.688621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.239639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689516+0000","last_change":"2024-03-30T14:54:37.329066+0000","last_active":"2024-03-30T14:55:55.689516+0000","last_peered":"2024-03-30T14:55:55.689516+0000","last_clean":"2024-03-30T14:55:55.689516+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T14:55:55.689516+0000","last_undegraded":"2024-03-30T14:55:55.689516+0000","last_fullsized":"2024-03-30T14:55:55.689516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.221906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":118,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583247+0000","last_change":"2024-03-30T14:55:52.189892+0000","last_active":"2024-03-30T14:55:55.583247+0000","last_peered":"2024-03-30T14:55:55.583247+0000","last_clean":"2024-03-30T14:55:55.583247+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T14:55:55.583247+0000","last_undegraded":"2024-03-30T14:55:55.583247+0000","last_fullsized":"2024-03-30T14:55:55.583247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:55:52.189742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.189742+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.639688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453705+0000","last_change":"2024-03-30T14:54:43.353107+0000","last_active":"2024-03-30T14:55:55.453705+0000","last_peered":"2024-03-30T14:55:55.453705+0000","last_clean":"2024-03-30T14:55:55.453705+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T14:55:55.453705+0000","last_undegraded":"2024-03-30T14:55:55.453705+0000","last_fullsized":"2024-03-30T14:55:55.453705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.679600+0000","last_change":"2024-03-30T14:55:56.679624+0000","last_active":"2024-03-30T14:55:56.679600+0000","last_peered":"2024-03-30T14:55:56.679600+0000","last_clean":"2024-03-30T14:55:56.679600+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T14:55:56.679600+0000","last_undegraded":"2024-03-30T14:55:56.679600+0000","last_fullsized":"2024-03-30T14:55:56.679600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.679531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.679531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.763850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691482+0000","last_change":"2024-03-30T14:55:37.491888+0000","last_active":"2024-03-30T14:55:55.691482+0000","last_peered":"2024-03-30T14:55:55.691482+0000","last_clean":"2024-03-30T14:55:55.691482+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T14:55:55.691482+0000","last_undegraded":"2024-03-30T14:55:55.691482+0000","last_fullsized":"2024-03-30T14:55:55.691482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:55:37.491750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.952763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453703+0000","last_change":"2024-03-30T14:55:48.585645+0000","last_active":"2024-03-30T14:55:55.453703+0000","last_peered":"2024-03-30T14:55:55.453703+0000","last_clean":"2024-03-30T14:55:55.453703+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T14:55:55.453703+0000","last_undegraded":"2024-03-30T14:55:55.453703+0000","last_fullsized":"2024-03-30T14:55:55.453703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:48.585557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.585557+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.155340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456441+0000","last_change":"2024-03-30T14:54:43.356748+0000","last_active":"2024-03-30T14:55:55.456441+0000","last_peered":"2024-03-30T14:55:55.456441+0000","last_clean":"2024-03-30T14:55:55.456441+0000","last_became_active":"2024-03-30T14:54:43.356453+0000","last_became_peered":"2024-03-30T14:54:43.356453+0000","last_unstale":"2024-03-30T14:55:55.456441+0000","last_undegraded":"2024-03-30T14:55:55.456441+0000","last_fullsized":"2024-03-30T14:55:55.456441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":68,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691401+0000","last_change":"2024-03-30T14:54:40.343883+0000","last_active":"2024-03-30T14:55:55.691401+0000","last_peered":"2024-03-30T14:55:55.691401+0000","last_clean":"2024-03-30T14:55:55.691401+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T14:55:55.691401+0000","last_undegraded":"2024-03-30T14:55:55.691401+0000","last_fullsized":"2024-03-30T14:55:55.691401+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.689686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689244+0000","last_change":"2024-03-30T14:54:37.326038+0000","last_active":"2024-03-30T14:55:55.689244+0000","last_peered":"2024-03-30T14:55:55.689244+0000","last_clean":"2024-03-30T14:55:55.689244+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T14:55:55.689244+0000","last_undegraded":"2024-03-30T14:55:55.689244+0000","last_fullsized":"2024-03-30T14:55:55.689244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691070+0000","last_change":"2024-03-30T14:54:58.412647+0000","last_active":"2024-03-30T14:55:55.691070+0000","last_peered":"2024-03-30T14:55:55.691070+0000","last_clean":"2024-03-30T14:55:55.691070+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T14:55:55.691070+0000","last_undegraded":"2024-03-30T14:55:55.691070+0000","last_fullsized":"2024-03-30T14:55:55.691070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454357+0000","last_change":"2024-03-30T14:54:41.349882+0000","last_active":"2024-03-30T14:55:55.454357+0000","last_peered":"2024-03-30T14:55:55.454357+0000","last_clean":"2024-03-30T14:55:55.454357+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T14:55:55.454357+0000","last_undegraded":"2024-03-30T14:55:55.454357+0000","last_fullsized":"2024-03-30T14:55:55.454357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.007208+0000","last_change":"2024-03-30T14:55:56.007233+0000","last_active":"2024-03-30T14:55:56.007208+0000","last_peered":"2024-03-30T14:55:56.007208+0000","last_clean":"2024-03-30T14:55:56.007208+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T14:55:56.007208+0000","last_undegraded":"2024-03-30T14:55:56.007208+0000","last_fullsized":"2024-03-30T14:55:56.007208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.007133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.007133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.546510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689643+0000","last_change":"2024-03-30T14:54:37.475240+0000","last_active":"2024-03-30T14:55:55.689643+0000","last_peered":"2024-03-30T14:55:55.689643+0000","last_clean":"2024-03-30T14:55:55.689643+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T14:55:55.689643+0000","last_undegraded":"2024-03-30T14:55:55.689643+0000","last_fullsized":"2024-03-30T14:55:55.689643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688318+0000","last_change":"2024-03-30T14:55:49.700155+0000","last_active":"2024-03-30T14:55:55.688318+0000","last_peered":"2024-03-30T14:55:55.688318+0000","last_clean":"2024-03-30T14:55:55.688318+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T14:55:55.688318+0000","last_undegraded":"2024-03-30T14:55:55.688318+0000","last_fullsized":"2024-03-30T14:55:55.688318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.700022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.949979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"76'13","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690016+0000","last_change":"2024-03-30T14:54:39.337585+0000","last_active":"2024-03-30T14:55:55.690016+0000","last_peered":"2024-03-30T14:55:55.690016+0000","last_clean":"2024-03-30T14:55:55.690016+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T14:55:55.690016+0000","last_undegraded":"2024-03-30T14:55:55.690016+0000","last_fullsized":"2024-03-30T14:55:55.690016+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.693768+0000","last_change":"2024-03-30T14:55:57.693792+0000","last_active":"2024-03-30T14:55:57.693768+0000","last_peered":"2024-03-30T14:55:57.693768+0000","last_clean":"2024-03-30T14:55:57.693768+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T14:55:57.693768+0000","last_undegraded":"2024-03-30T14:55:57.693768+0000","last_fullsized":"2024-03-30T14:55:57.693768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.693703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.693703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.637711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689570+0000","last_change":"2024-03-30T14:54:43.356807+0000","last_active":"2024-03-30T14:55:55.689570+0000","last_peered":"2024-03-30T14:55:55.689570+0000","last_clean":"2024-03-30T14:55:55.689570+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T14:55:55.689570+0000","last_undegraded":"2024-03-30T14:55:55.689570+0000","last_fullsized":"2024-03-30T14:55:55.689570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:08.744451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690973+0000","last_change":"2024-03-30T14:55:05.131524+0000","last_active":"2024-03-30T14:55:55.690973+0000","last_peered":"2024-03-30T14:55:55.690973+0000","last_clean":"2024-03-30T14:55:55.690973+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T14:55:55.690973+0000","last_undegraded":"2024-03-30T14:55:55.690973+0000","last_fullsized":"2024-03-30T14:55:55.690973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-30T14:55:05.131440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:55:05.131440+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:33.848087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454922+0000","last_change":"2024-03-30T14:54:37.324739+0000","last_active":"2024-03-30T14:55:55.454922+0000","last_peered":"2024-03-30T14:55:55.454922+0000","last_clean":"2024-03-30T14:55:55.454922+0000","last_became_active":"2024-03-30T14:54:37.324423+0000","last_became_peered":"2024-03-30T14:54:37.324423+0000","last_unstale":"2024-03-30T14:55:55.454922+0000","last_undegraded":"2024-03-30T14:55:55.454922+0000","last_fullsized":"2024-03-30T14:55:55.454922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.830680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583532+0000","last_change":"2024-03-30T14:54:39.643201+0000","last_active":"2024-03-30T14:55:55.583532+0000","last_peered":"2024-03-30T14:55:55.583532+0000","last_clean":"2024-03-30T14:55:55.583532+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T14:55:55.583532+0000","last_undegraded":"2024-03-30T14:55:55.583532+0000","last_fullsized":"2024-03-30T14:55:55.583532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.482165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690544+0000","last_change":"2024-03-30T14:54:41.350344+0000","last_active":"2024-03-30T14:55:55.690544+0000","last_peered":"2024-03-30T14:55:55.690544+0000","last_clean":"2024-03-30T14:55:55.690544+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T14:55:55.690544+0000","last_undegraded":"2024-03-30T14:55:55.690544+0000","last_fullsized":"2024-03-30T14:55:55.690544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.814152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688709+0000","last_change":"2024-03-30T14:54:43.357715+0000","last_active":"2024-03-30T14:55:55.688709+0000","last_peered":"2024-03-30T14:55:55.688709+0000","last_clean":"2024-03-30T14:55:55.688709+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T14:55:55.688709+0000","last_undegraded":"2024-03-30T14:55:55.688709+0000","last_fullsized":"2024-03-30T14:55:55.688709+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.152983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453356+0000","last_change":"2024-03-30T14:54:37.326510+0000","last_active":"2024-03-30T14:55:55.453356+0000","last_peered":"2024-03-30T14:55:55.453356+0000","last_clean":"2024-03-30T14:55:55.453356+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T14:55:55.453356+0000","last_undegraded":"2024-03-30T14:55:55.453356+0000","last_fullsized":"2024-03-30T14:55:55.453356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.302833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.988557+0000","last_change":"2024-03-30T14:54:40.338153+0000","last_active":"2024-03-30T14:55:56.988557+0000","last_peered":"2024-03-30T14:55:56.988557+0000","last_clean":"2024-03-30T14:55:56.988557+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T14:55:56.988557+0000","last_undegraded":"2024-03-30T14:55:56.988557+0000","last_fullsized":"2024-03-30T14:55:56.988557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454509+0000","last_change":"2024-03-30T14:55:52.556360+0000","last_active":"2024-03-30T14:55:55.454509+0000","last_peered":"2024-03-30T14:55:55.454509+0000","last_clean":"2024-03-30T14:55:55.454509+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T14:55:55.454509+0000","last_undegraded":"2024-03-30T14:55:55.454509+0000","last_fullsized":"2024-03-30T14:55:55.454509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:52.556237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.556237+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.659172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.530391+0000","last_change":"2024-03-30T14:55:57.530418+0000","last_active":"2024-03-30T14:55:57.530391+0000","last_peered":"2024-03-30T14:55:57.530391+0000","last_clean":"2024-03-30T14:55:57.530391+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T14:55:57.530391+0000","last_undegraded":"2024-03-30T14:55:57.530391+0000","last_fullsized":"2024-03-30T14:55:57.530391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:01.784600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688787+0000","last_change":"2024-03-30T14:54:43.353079+0000","last_active":"2024-03-30T14:55:55.688787+0000","last_peered":"2024-03-30T14:55:55.688787+0000","last_clean":"2024-03-30T14:55:55.688787+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T14:55:55.688787+0000","last_undegraded":"2024-03-30T14:55:55.688787+0000","last_fullsized":"2024-03-30T14:55:55.688787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688979+0000","last_change":"2024-03-30T14:55:42.232642+0000","last_active":"2024-03-30T14:55:55.688979+0000","last_peered":"2024-03-30T14:55:55.688979+0000","last_clean":"2024-03-30T14:55:55.688979+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T14:55:55.688979+0000","last_undegraded":"2024-03-30T14:55:55.688979+0000","last_fullsized":"2024-03-30T14:55:55.688979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.232525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.232525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:48.072521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"76'19","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453942+0000","last_change":"2024-03-30T14:54:39.642101+0000","last_active":"2024-03-30T14:55:55.453942+0000","last_peered":"2024-03-30T14:55:55.453942+0000","last_clean":"2024-03-30T14:55:55.453942+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T14:55:55.453942+0000","last_undegraded":"2024-03-30T14:55:55.453942+0000","last_fullsized":"2024-03-30T14:55:55.453942+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.046560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583332+0000","last_change":"2024-03-30T14:54:41.345132+0000","last_active":"2024-03-30T14:55:55.583332+0000","last_peered":"2024-03-30T14:55:55.583332+0000","last_clean":"2024-03-30T14:55:55.583332+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T14:55:55.583332+0000","last_undegraded":"2024-03-30T14:55:55.583332+0000","last_fullsized":"2024-03-30T14:55:55.583332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.169133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453022+0000","last_change":"2024-03-30T14:54:43.355213+0000","last_active":"2024-03-30T14:55:55.453022+0000","last_peered":"2024-03-30T14:55:55.453022+0000","last_clean":"2024-03-30T14:55:55.453022+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T14:55:55.453022+0000","last_undegraded":"2024-03-30T14:55:55.453022+0000","last_fullsized":"2024-03-30T14:55:55.453022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.304491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691328+0000","last_change":"2024-03-30T14:55:41.454468+0000","last_active":"2024-03-30T14:55:55.691328+0000","last_peered":"2024-03-30T14:55:55.691328+0000","last_clean":"2024-03-30T14:55:55.691328+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T14:55:55.691328+0000","last_undegraded":"2024-03-30T14:55:55.691328+0000","last_fullsized":"2024-03-30T14:55:55.691328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.454365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:04.526213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"76'10","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689681+0000","last_change":"2024-03-30T14:54:39.338666+0000","last_active":"2024-03-30T14:55:55.689681+0000","last_peered":"2024-03-30T14:55:55.689681+0000","last_clean":"2024-03-30T14:55:55.689681+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T14:55:55.689681+0000","last_undegraded":"2024-03-30T14:55:55.689681+0000","last_fullsized":"2024-03-30T14:55:55.689681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453815+0000","last_change":"2024-03-30T14:55:50.585912+0000","last_active":"2024-03-30T14:55:55.453815+0000","last_peered":"2024-03-30T14:55:55.453815+0000","last_clean":"2024-03-30T14:55:55.453815+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T14:55:55.453815+0000","last_undegraded":"2024-03-30T14:55:55.453815+0000","last_fullsized":"2024-03-30T14:55:55.453815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:50.585773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.585773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.447085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690700+0000","last_change":"2024-03-30T14:55:49.029645+0000","last_active":"2024-03-30T14:55:55.690700+0000","last_peered":"2024-03-30T14:55:55.690700+0000","last_clean":"2024-03-30T14:55:55.690700+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T14:55:55.690700+0000","last_undegraded":"2024-03-30T14:55:55.690700+0000","last_fullsized":"2024-03-30T14:55:55.690700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.029548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.029548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:52.389450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"69'1","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454618+0000","last_change":"2024-03-30T14:54:37.326159+0000","last_active":"2024-03-30T14:55:55.454618+0000","last_peered":"2024-03-30T14:55:55.454618+0000","last_clean":"2024-03-30T14:55:55.454618+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T14:55:55.454618+0000","last_undegraded":"2024-03-30T14:55:55.454618+0000","last_fullsized":"2024-03-30T14:55:55.454618+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.929704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690207+0000","last_change":"2024-03-30T14:55:46.303554+0000","last_active":"2024-03-30T14:55:55.690207+0000","last_peered":"2024-03-30T14:55:55.690207+0000","last_clean":"2024-03-30T14:55:55.690207+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T14:55:55.690207+0000","last_undegraded":"2024-03-30T14:55:55.690207+0000","last_fullsized":"2024-03-30T14:55:55.690207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:55:46.303412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.303412+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.425234+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"76'8","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.692425+0000","last_change":"2024-03-30T14:55:55.692425+0000","last_active":"2024-03-30T14:55:55.692425+0000","last_peered":"2024-03-30T14:55:55.692425+0000","last_clean":"2024-03-30T14:55:55.692425+0000","last_became_active":"2024-03-30T14:54:41.353489+0000","last_became_peered":"2024-03-30T14:54:41.353489+0000","last_unstale":"2024-03-30T14:55:55.692425+0000","last_undegraded":"2024-03-30T14:55:55.692425+0000","last_fullsized":"2024-03-30T14:55:55.692425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:55.692261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:55.692261+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.231384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689326+0000","last_change":"2024-03-30T14:54:43.352748+0000","last_active":"2024-03-30T14:55:55.689326+0000","last_peered":"2024-03-30T14:55:55.689326+0000","last_clean":"2024-03-30T14:55:55.689326+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T14:55:55.689326+0000","last_undegraded":"2024-03-30T14:55:55.689326+0000","last_fullsized":"2024-03-30T14:55:55.689326+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.488227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"69'1","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456566+0000","last_change":"2024-03-30T14:54:37.323745+0000","last_active":"2024-03-30T14:55:55.456566+0000","last_peered":"2024-03-30T14:55:55.456566+0000","last_clean":"2024-03-30T14:55:55.456566+0000","last_became_active":"2024-03-30T14:54:37.323384+0000","last_became_peered":"2024-03-30T14:54:37.323384+0000","last_unstale":"2024-03-30T14:55:55.456566+0000","last_undegraded":"2024-03-30T14:55:55.456566+0000","last_fullsized":"2024-03-30T14:55:55.456566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.422909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"76'18","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455711+0000","last_change":"2024-03-30T14:54:39.642940+0000","last_active":"2024-03-30T14:55:55.455711+0000","last_peered":"2024-03-30T14:55:55.455711+0000","last_clean":"2024-03-30T14:55:55.455711+0000","last_became_active":"2024-03-30T14:54:39.642450+0000","last_became_peered":"2024-03-30T14:54:39.642450+0000","last_unstale":"2024-03-30T14:55:55.455711+0000","last_undegraded":"2024-03-30T14:55:55.455711+0000","last_fullsized":"2024-03-30T14:55:55.455711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.321890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688654+0000","last_change":"2024-03-30T14:55:54.381596+0000","last_active":"2024-03-30T14:55:55.688654+0000","last_peered":"2024-03-30T14:55:55.688654+0000","last_clean":"2024-03-30T14:55:55.688654+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T14:55:55.688654+0000","last_undegraded":"2024-03-30T14:55:55.688654+0000","last_fullsized":"2024-03-30T14:55:55.688654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:54.381448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:54.381448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.160867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690908+0000","last_change":"2024-03-30T14:54:43.351363+0000","last_active":"2024-03-30T14:55:55.690908+0000","last_peered":"2024-03-30T14:55:55.690908+0000","last_clean":"2024-03-30T14:55:55.690908+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T14:55:55.690908+0000","last_undegraded":"2024-03-30T14:55:55.690908+0000","last_fullsized":"2024-03-30T14:55:55.690908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.981871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690783+0000","last_change":"2024-03-30T14:54:41.354505+0000","last_active":"2024-03-30T14:55:55.690783+0000","last_peered":"2024-03-30T14:55:55.690783+0000","last_clean":"2024-03-30T14:55:55.690783+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T14:55:55.690783+0000","last_undegraded":"2024-03-30T14:55:55.690783+0000","last_fullsized":"2024-03-30T14:55:55.690783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455608+0000","last_change":"2024-03-30T14:54:37.323396+0000","last_active":"2024-03-30T14:55:55.455608+0000","last_peered":"2024-03-30T14:55:55.455608+0000","last_clean":"2024-03-30T14:55:55.455608+0000","last_became_active":"2024-03-30T14:54:37.323084+0000","last_became_peered":"2024-03-30T14:54:37.323084+0000","last_unstale":"2024-03-30T14:55:55.455608+0000","last_undegraded":"2024-03-30T14:55:55.455608+0000","last_fullsized":"2024-03-30T14:55:55.455608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.069394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"76'15","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690013+0000","last_change":"2024-03-30T14:54:39.338356+0000","last_active":"2024-03-30T14:55:55.690013+0000","last_peered":"2024-03-30T14:55:55.690013+0000","last_clean":"2024-03-30T14:55:55.690013+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T14:55:55.690013+0000","last_undegraded":"2024-03-30T14:55:55.690013+0000","last_fullsized":"2024-03-30T14:55:55.690013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454718+0000","last_change":"2024-03-30T14:54:43.360313+0000","last_active":"2024-03-30T14:55:55.454718+0000","last_peered":"2024-03-30T14:55:55.454718+0000","last_clean":"2024-03-30T14:55:55.454718+0000","last_became_active":"2024-03-30T14:54:43.360002+0000","last_became_peered":"2024-03-30T14:54:43.360002+0000","last_unstale":"2024-03-30T14:55:55.454718+0000","last_undegraded":"2024-03-30T14:55:55.454718+0000","last_fullsized":"2024-03-30T14:55:55.454718+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691111+0000","last_change":"2024-03-30T14:55:50.439135+0000","last_active":"2024-03-30T14:55:55.691111+0000","last_peered":"2024-03-30T14:55:55.691111+0000","last_clean":"2024-03-30T14:55:55.691111+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T14:55:55.691111+0000","last_undegraded":"2024-03-30T14:55:55.691111+0000","last_fullsized":"2024-03-30T14:55:55.691111+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.439008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.333757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690008+0000","last_change":"2024-03-30T14:55:47.298871+0000","last_active":"2024-03-30T14:55:55.690008+0000","last_peered":"2024-03-30T14:55:55.690008+0000","last_clean":"2024-03-30T14:55:55.690008+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T14:55:55.690008+0000","last_undegraded":"2024-03-30T14:55:55.690008+0000","last_fullsized":"2024-03-30T14:55:55.690008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:47.298753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:47.298753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.038171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"76'11","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690699+0000","last_change":"2024-03-30T14:54:39.335270+0000","last_active":"2024-03-30T14:55:55.690699+0000","last_peered":"2024-03-30T14:55:55.690699+0000","last_clean":"2024-03-30T14:55:55.690699+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T14:55:55.690699+0000","last_undegraded":"2024-03-30T14:55:55.690699+0000","last_fullsized":"2024-03-30T14:55:55.690699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689186+0000","last_change":"2024-03-30T14:54:43.353034+0000","last_active":"2024-03-30T14:55:55.689186+0000","last_peered":"2024-03-30T14:55:55.689186+0000","last_clean":"2024-03-30T14:55:55.689186+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T14:55:55.689186+0000","last_undegraded":"2024-03-30T14:55:55.689186+0000","last_fullsized":"2024-03-30T14:55:55.689186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:10.751689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689162+0000","last_change":"2024-03-30T14:55:46.232558+0000","last_active":"2024-03-30T14:55:55.689162+0000","last_peered":"2024-03-30T14:55:55.689162+0000","last_clean":"2024-03-30T14:55:55.689162+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T14:55:55.689162+0000","last_undegraded":"2024-03-30T14:55:55.689162+0000","last_fullsized":"2024-03-30T14:55:55.689162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.232465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.933698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583019+0000","last_change":"2024-03-30T14:54:37.326067+0000","last_active":"2024-03-30T14:55:55.583019+0000","last_peered":"2024-03-30T14:55:55.583019+0000","last_clean":"2024-03-30T14:55:55.583019+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T14:55:55.583019+0000","last_undegraded":"2024-03-30T14:55:55.583019+0000","last_fullsized":"2024-03-30T14:55:55.583019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.706959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690602+0000","last_change":"2024-03-30T14:55:50.322071+0000","last_active":"2024-03-30T14:55:55.690602+0000","last_peered":"2024-03-30T14:55:55.690602+0000","last_clean":"2024-03-30T14:55:55.690602+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T14:55:55.690602+0000","last_undegraded":"2024-03-30T14:55:55.690602+0000","last_fullsized":"2024-03-30T14:55:55.690602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T14:55:50.321974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.321974+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:54.493589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690810+0000","last_change":"2024-03-30T14:54:43.352969+0000","last_active":"2024-03-30T14:55:55.690810+0000","last_peered":"2024-03-30T14:55:55.690810+0000","last_clean":"2024-03-30T14:55:55.690810+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:55:55.690810+0000","last_undegraded":"2024-03-30T14:55:55.690810+0000","last_fullsized":"2024-03-30T14:55:55.690810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688595+0000","last_change":"2024-03-30T14:55:40.800056+0000","last_active":"2024-03-30T14:55:55.688595+0000","last_peered":"2024-03-30T14:55:55.688595+0000","last_clean":"2024-03-30T14:55:55.688595+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T14:55:55.688595+0000","last_undegraded":"2024-03-30T14:55:55.688595+0000","last_fullsized":"2024-03-30T14:55:55.688595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:40.799969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.799969+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:57.300469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455594+0000","last_change":"2024-03-30T14:55:45.783270+0000","last_active":"2024-03-30T14:55:55.455594+0000","last_peered":"2024-03-30T14:55:55.455594+0000","last_clean":"2024-03-30T14:55:55.455594+0000","last_became_active":"2024-03-30T14:54:37.327905+0000","last_became_peered":"2024-03-30T14:54:37.327905+0000","last_unstale":"2024-03-30T14:55:55.455594+0000","last_undegraded":"2024-03-30T14:55:55.455594+0000","last_fullsized":"2024-03-30T14:55:55.455594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:45.783178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:45.783178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.797807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689879+0000","last_change":"2024-03-30T14:55:51.283107+0000","last_active":"2024-03-30T14:55:55.689879+0000","last_peered":"2024-03-30T14:55:55.689879+0000","last_clean":"2024-03-30T14:55:55.689879+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T14:55:55.689879+0000","last_undegraded":"2024-03-30T14:55:55.689879+0000","last_fullsized":"2024-03-30T14:55:55.689879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:51.282990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:51.282990+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.280452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005077,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712564,"statfs":{"total":95995035648,"available":95961665536,"internally_reserved":0,"allocated":1441792,"data_stored":960977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299418,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32560,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712592,"statfs":{"total":95995035648,"available":95961694208,"internally_reserved":0,"allocated":1409024,"data_stored":958639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":856064,"data_stored":369478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":960321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":819200,"data_stored":368606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":864256,"data_stored":370448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":847872,"data_stored":369274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":835584,"data_stored":368955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:56:02.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: from='client.34625 v1:172.21.15.28:0/2920006626' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T14:56:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:02 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:02.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:02.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T14:56:02.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: from='client.34625 v1:172.21.15.28:0/2920006626' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: from='client.34625 v1:172.21.15.28:0/2920006626' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T14:56:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:02 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:03 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T14:56:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:03 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T14:56:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:03 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:03 smithi134 ceph-mon[33714]: 3.15 deep-scrub starts 2024-03-30T14:56:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:03 smithi134 ceph-mon[33714]: 3.15 deep-scrub ok 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[29876]: 3.15 deep-scrub starts 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[29876]: 3.15 deep-scrub ok 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[41614]: 3.15 deep-scrub starts 2024-03-30T14:56:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:03 smithi028 ceph-mon[41614]: 3.15 deep-scrub ok 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 3.0 deep-scrub starts 2024-03-30T14:56:04.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[41614]: 3.0 deep-scrub ok 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 3.0 deep-scrub starts 2024-03-30T14:56:04.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:04 smithi028 ceph-mon[29876]: 3.0 deep-scrub ok 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 3.0 deep-scrub starts 2024-03-30T14:56:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:04 smithi134 ceph-mon[33714]: 3.0 deep-scrub ok 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 -- 172.21.15.28:0/1378797249 >> v1:172.21.15.28:6790/0 conn(0x7f088810aec0 legacy=0x7f088810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 -- 172.21.15.28:0/1378797249 shutdown_connections 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 -- 172.21.15.28:0/1378797249 >> 172.21.15.28:0/1378797249 conn(0x7f08880fdb00 msgr2=0x7f08880fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 -- 172.21.15.28:0/1378797249 shutdown_connections 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 -- 172.21.15.28:0/1378797249 wait complete. 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 librados: starting msgr at 2024-03-30T14:56:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.966+0000 7f088fcf2700 1 librados: starting objecter 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 Processor -- start 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 -- start start 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 librados: setting wanted keys 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 librados: calling monclient init 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08881078b0 con 0x7f0888102230 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0888107ac0 con 0x7f088810aec0 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088fcf2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08881a9640 con 0x7f088810ea60 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088e28f700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f088810ea60 0x7f08881a5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50932/0 (socket says 172.21.15.28:50932) 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088e28f700 1 -- 172.21.15.28:0/2134160545 learned_addr learned my addr 172.21.15.28:0/2134160545 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f088da8e700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0888102230 0x7f0888104700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53394/0 (socket says 172.21.15.28:53394) 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter ms_handle_connect 0x7f088810ea60 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter ms_handle_connect 0x7f0888102230 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter ms_handle_connect 0x7f088810aec0 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797636819 0 0) 0x7f08881a9640 con 0x7f088810ea60 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08740031f0 con 0x7f088810ea60 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824438564 0 0) 0x7f08881078b0 con 0x7f0888102230 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08881a9640 con 0x7f0888102230 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563049393 0 0) 0x7f0888107ac0 con 0x7f088810aec0 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08881078b0 con 0x7f088810aec0 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1277960541 0 0) 0x7f08740031f0 con 0x7f088810ea60 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0888107ac0 con 0x7f088810ea60 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0880001c90 con 0x7f088810ea60 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2279484147 0 0) 0x7f08881a9640 con 0x7f0888102230 2024-03-30T14:56:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08740031f0 con 0x7f0888102230 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.967+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4216500264 0 0) 0x7f08881078b0 con 0x7f088810aec0 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08881a9640 con 0x7f088810aec0 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f08840012d0 con 0x7f0888102230 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0878004470 con 0x7f088810aec0 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1469781610 0 0) 0x7f0888107ac0 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 >> v1:172.21.15.134:6789/0 conn(0x7f088810aec0 legacy=0x7f0888104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 >> v1:172.21.15.28:6789/0 conn(0x7f0888102230 legacy=0x7f0888104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08881aa860 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08881a9870 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f088fcf2700 10 client.34631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f08881a9d30 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f088fcf2700 1 librados: init done 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0880003ec0 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.968+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0880006090 con 0x7f088810ea60 2024-03-30T14:56:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.969+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f086c005120 con 0x7f088810ea60 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.969+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0880003a20 con 0x7f088810ea60 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(93..93 src has 43..93) ==== 6259+0+0 (unknown 1478577302 0 0) 0x7f08800892b0 con 0x7f088810ea60 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 10 client.34631.objecter ms_dispatch 0x7f0888001550 osd_map(93..93 src has 43..93) 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 3 client.34631.objecter handle_osd_map got epochs [93,93] > 0 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 3 client.34631.objecter handle_osd_map decoding full epoch 93 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 20 client.34631.objecter dump_active .. 0 homeless 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.970+0000 7f087effd700 10 client.34631.objecter ms_handle_connect 0x7f0874070460 2024-03-30T14:56:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:04.975+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0880006090 con 0x7f088810ea60 2024-03-30T14:56:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.172+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f086c000b90 con 0x7f0874070460 2024-03-30T14:56:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.181+0000 7f087effd700 1 -- 172.21.15.28:0/2134160545 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+345704 (unknown 2965378022 0 981830950) 0x7f086c000b90 con 0x7f0874070460 2024-03-30T14:56:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.184+0000 7f088fcf2700 10 librados: watch_flush enter 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.184+0000 7f088fcf2700 10 librados: watch_flush exit 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.184+0000 7f088fcf2700 20 client.34631.objecter shutdown clearing up homeless session... 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.184+0000 7f088fcf2700 10 client.34631.objecter successfully canceled tick 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0874070460 legacy=0x7f0874072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 >> v1:172.21.15.28:6790/0 conn(0x7f088810ea60 legacy=0x7f08881a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 shutdown_connections 2024-03-30T14:56:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 >> 172.21.15.28:0/2134160545 conn(0x7f08880fdb00 msgr2=0x7f0888111f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:05.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 shutdown_connections 2024-03-30T14:56:05.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 -- 172.21.15.28:0/2134160545 wait complete. 2024-03-30T14:56:05.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:05.185+0000 7f088fcf2700 1 librados: shutdown 2024-03-30T14:56:05.190 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T14:56:05.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:05 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T14:56:05.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:05 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T14:56:05.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:05 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T14:56:05.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:05 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T14:56:05.839 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-30T14:56:03.586302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249408,"kb_used_data":8316,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711808,"statfs":{"total":767960285184,"available":767704891392,"internally_reserved":0,"allocated":8515584,"data_stored":4726698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002421"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689208+0000","last_change":"2024-03-30T14:54:43.351532+0000","last_active":"2024-03-30T14:55:55.689208+0000","last_peered":"2024-03-30T14:55:55.689208+0000","last_clean":"2024-03-30T14:55:55.689208+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T14:55:55.689208+0000","last_undegraded":"2024-03-30T14:55:55.689208+0000","last_fullsized":"2024-03-30T14:55:55.689208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.847289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688920+0000","last_change":"2024-03-30T14:54:41.350461+0000","last_active":"2024-03-30T14:55:55.688920+0000","last_peered":"2024-03-30T14:55:55.688920+0000","last_clean":"2024-03-30T14:55:55.688920+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T14:55:55.688920+0000","last_undegraded":"2024-03-30T14:55:55.688920+0000","last_fullsized":"2024-03-30T14:55:55.688920+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.461361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454677+0000","last_change":"2024-03-30T14:55:46.590443+0000","last_active":"2024-03-30T14:55:55.454677+0000","last_peered":"2024-03-30T14:55:55.454677+0000","last_clean":"2024-03-30T14:55:55.454677+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T14:55:55.454677+0000","last_undegraded":"2024-03-30T14:55:55.454677+0000","last_fullsized":"2024-03-30T14:55:55.454677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.590360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.590360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.726057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688646+0000","last_change":"2024-03-30T14:55:40.318444+0000","last_active":"2024-03-30T14:55:55.688646+0000","last_peered":"2024-03-30T14:55:55.688646+0000","last_clean":"2024-03-30T14:55:55.688646+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T14:55:55.688646+0000","last_undegraded":"2024-03-30T14:55:55.688646+0000","last_fullsized":"2024-03-30T14:55:55.688646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:40.318295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.318295+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:45.878339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455375+0000","last_change":"2024-03-30T14:54:43.359234+0000","last_active":"2024-03-30T14:55:55.455375+0000","last_peered":"2024-03-30T14:55:55.455375+0000","last_clean":"2024-03-30T14:55:55.455375+0000","last_became_active":"2024-03-30T14:54:43.358170+0000","last_became_peered":"2024-03-30T14:54:43.358170+0000","last_unstale":"2024-03-30T14:55:55.455375+0000","last_undegraded":"2024-03-30T14:55:55.455375+0000","last_fullsized":"2024-03-30T14:55:55.455375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.637785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688547+0000","last_change":"2024-03-30T14:54:41.350763+0000","last_active":"2024-03-30T14:55:55.688547+0000","last_peered":"2024-03-30T14:55:55.688547+0000","last_clean":"2024-03-30T14:55:55.688547+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T14:55:55.688547+0000","last_undegraded":"2024-03-30T14:55:55.688547+0000","last_fullsized":"2024-03-30T14:55:55.688547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.735595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456587+0000","last_change":"2024-03-30T14:55:41.765389+0000","last_active":"2024-03-30T14:55:55.456587+0000","last_peered":"2024-03-30T14:55:55.456587+0000","last_clean":"2024-03-30T14:55:55.456587+0000","last_became_active":"2024-03-30T14:54:37.474114+0000","last_became_peered":"2024-03-30T14:54:37.474114+0000","last_unstale":"2024-03-30T14:55:55.456587+0000","last_undegraded":"2024-03-30T14:55:55.456587+0000","last_fullsized":"2024-03-30T14:55:55.456587+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.765306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.765306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:43.228077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"76'15","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456124+0000","last_change":"2024-03-30T14:54:39.333923+0000","last_active":"2024-03-30T14:55:55.456124+0000","last_peered":"2024-03-30T14:55:55.456124+0000","last_clean":"2024-03-30T14:55:55.456124+0000","last_became_active":"2024-03-30T14:54:39.333512+0000","last_became_peered":"2024-03-30T14:54:39.333512+0000","last_unstale":"2024-03-30T14:55:55.456124+0000","last_undegraded":"2024-03-30T14:55:55.456124+0000","last_fullsized":"2024-03-30T14:55:55.456124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.530539+0000","last_change":"2024-03-30T14:55:57.530576+0000","last_active":"2024-03-30T14:55:57.530539+0000","last_peered":"2024-03-30T14:55:57.530539+0000","last_clean":"2024-03-30T14:55:57.530539+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T14:55:57.530539+0000","last_undegraded":"2024-03-30T14:55:57.530539+0000","last_fullsized":"2024-03-30T14:55:57.530539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.959060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688952+0000","last_change":"2024-03-30T14:55:43.421378+0000","last_active":"2024-03-30T14:55:55.688952+0000","last_peered":"2024-03-30T14:55:55.688952+0000","last_clean":"2024-03-30T14:55:55.688952+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T14:55:55.688952+0000","last_undegraded":"2024-03-30T14:55:55.688952+0000","last_fullsized":"2024-03-30T14:55:55.688952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:43.421270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:43.421270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.069402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691260+0000","last_change":"2024-03-30T14:55:46.387117+0000","last_active":"2024-03-30T14:55:55.691260+0000","last_peered":"2024-03-30T14:55:55.691260+0000","last_clean":"2024-03-30T14:55:55.691260+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T14:55:55.691260+0000","last_undegraded":"2024-03-30T14:55:55.691260+0000","last_fullsized":"2024-03-30T14:55:55.691260+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.387017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.387017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.976653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689501+0000","last_change":"2024-03-30T14:54:39.329805+0000","last_active":"2024-03-30T14:55:55.689501+0000","last_peered":"2024-03-30T14:55:55.689501+0000","last_clean":"2024-03-30T14:55:55.689501+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T14:55:55.689501+0000","last_undegraded":"2024-03-30T14:55:55.689501+0000","last_fullsized":"2024-03-30T14:55:55.689501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.308058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691252+0000","last_change":"2024-03-30T14:55:49.394236+0000","last_active":"2024-03-30T14:55:55.691252+0000","last_peered":"2024-03-30T14:55:55.691252+0000","last_clean":"2024-03-30T14:55:55.691252+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T14:55:55.691252+0000","last_undegraded":"2024-03-30T14:55:55.691252+0000","last_fullsized":"2024-03-30T14:55:55.691252+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.394138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.394138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.971320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689068+0000","last_change":"2024-03-30T14:54:41.352116+0000","last_active":"2024-03-30T14:55:55.689068+0000","last_peered":"2024-03-30T14:55:55.689068+0000","last_clean":"2024-03-30T14:55:55.689068+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T14:55:55.689068+0000","last_undegraded":"2024-03-30T14:55:55.689068+0000","last_fullsized":"2024-03-30T14:55:55.689068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.338455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:59.608058+0000","last_change":"2024-03-30T14:55:59.608083+0000","last_active":"2024-03-30T14:55:59.608058+0000","last_peered":"2024-03-30T14:55:59.608058+0000","last_clean":"2024-03-30T14:55:59.608058+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T14:55:59.608058+0000","last_undegraded":"2024-03-30T14:55:59.608058+0000","last_fullsized":"2024-03-30T14:55:59.608058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.607983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.607983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.338389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689408+0000","last_change":"2024-03-30T14:54:58.404856+0000","last_active":"2024-03-30T14:55:55.689408+0000","last_peered":"2024-03-30T14:55:55.689408+0000","last_clean":"2024-03-30T14:55:55.689408+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T14:55:55.689408+0000","last_undegraded":"2024-03-30T14:55:55.689408+0000","last_fullsized":"2024-03-30T14:55:55.689408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:58.073767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690553+0000","last_change":"2024-03-30T14:54:43.349041+0000","last_active":"2024-03-30T14:55:55.690553+0000","last_peered":"2024-03-30T14:55:55.690553+0000","last_clean":"2024-03-30T14:55:55.690553+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T14:55:55.690553+0000","last_undegraded":"2024-03-30T14:55:55.690553+0000","last_fullsized":"2024-03-30T14:55:55.690553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.764286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689511+0000","last_change":"2024-03-30T14:54:41.347727+0000","last_active":"2024-03-30T14:55:55.689511+0000","last_peered":"2024-03-30T14:55:55.689511+0000","last_clean":"2024-03-30T14:55:55.689511+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T14:55:55.689511+0000","last_undegraded":"2024-03-30T14:55:55.689511+0000","last_fullsized":"2024-03-30T14:55:55.689511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.353560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453918+0000","last_change":"2024-03-30T14:55:42.710206+0000","last_active":"2024-03-30T14:55:55.453918+0000","last_peered":"2024-03-30T14:55:55.453918+0000","last_clean":"2024-03-30T14:55:55.453918+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T14:55:55.453918+0000","last_undegraded":"2024-03-30T14:55:55.453918+0000","last_fullsized":"2024-03-30T14:55:55.453918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.710118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.710118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.005945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"76'15","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688833+0000","last_change":"2024-03-30T14:55:39.803863+0000","last_active":"2024-03-30T14:55:55.688833+0000","last_peered":"2024-03-30T14:55:55.688833+0000","last_clean":"2024-03-30T14:55:55.688833+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T14:55:55.688833+0000","last_undegraded":"2024-03-30T14:55:55.688833+0000","last_fullsized":"2024-03-30T14:55:55.688833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:55:39.803752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:39.803752+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.649577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456435+0000","last_change":"2024-03-30T14:54:43.354706+0000","last_active":"2024-03-30T14:55:55.456435+0000","last_peered":"2024-03-30T14:55:55.456435+0000","last_clean":"2024-03-30T14:55:55.456435+0000","last_became_active":"2024-03-30T14:54:43.354397+0000","last_became_peered":"2024-03-30T14:54:43.354397+0000","last_unstale":"2024-03-30T14:55:55.456435+0000","last_undegraded":"2024-03-30T14:55:55.456435+0000","last_fullsized":"2024-03-30T14:55:55.456435+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.574805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689778+0000","last_change":"2024-03-30T14:54:41.348874+0000","last_active":"2024-03-30T14:55:55.689778+0000","last_peered":"2024-03-30T14:55:55.689778+0000","last_clean":"2024-03-30T14:55:55.689778+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T14:55:55.689778+0000","last_undegraded":"2024-03-30T14:55:55.689778+0000","last_fullsized":"2024-03-30T14:55:55.689778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.203011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689612+0000","last_change":"2024-03-30T14:54:37.474896+0000","last_active":"2024-03-30T14:55:55.689612+0000","last_peered":"2024-03-30T14:55:55.689612+0000","last_clean":"2024-03-30T14:55:55.689612+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T14:55:55.689612+0000","last_undegraded":"2024-03-30T14:55:55.689612+0000","last_fullsized":"2024-03-30T14:55:55.689612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:01.665580+0000","last_change":"2024-03-30T14:56:01.665638+0000","last_active":"2024-03-30T14:56:01.665580+0000","last_peered":"2024-03-30T14:56:01.665580+0000","last_clean":"2024-03-30T14:56:01.665580+0000","last_became_active":"2024-03-30T14:54:39.338927+0000","last_became_peered":"2024-03-30T14:54:39.338927+0000","last_unstale":"2024-03-30T14:56:01.665580+0000","last_undegraded":"2024-03-30T14:56:01.665580+0000","last_fullsized":"2024-03-30T14:56:01.665580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:56:01.665505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.665505+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.704128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"76'1","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689295+0000","last_change":"2024-03-30T14:54:43.354908+0000","last_active":"2024-03-30T14:55:55.689295+0000","last_peered":"2024-03-30T14:55:55.689295+0000","last_clean":"2024-03-30T14:55:55.689295+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T14:55:55.689295+0000","last_undegraded":"2024-03-30T14:55:55.689295+0000","last_fullsized":"2024-03-30T14:55:55.689295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.852744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:58.120425+0000","last_change":"2024-03-30T14:55:58.120425+0000","last_active":"2024-03-30T14:55:58.120425+0000","last_peered":"2024-03-30T14:55:58.120425+0000","last_clean":"2024-03-30T14:55:58.120425+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T14:55:58.120425+0000","last_undegraded":"2024-03-30T14:55:58.120425+0000","last_fullsized":"2024-03-30T14:55:58.120425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:58.120314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:58.120314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.131781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689019+0000","last_change":"2024-03-30T14:54:37.324901+0000","last_active":"2024-03-30T14:55:55.689019+0000","last_peered":"2024-03-30T14:55:55.689019+0000","last_clean":"2024-03-30T14:55:55.689019+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T14:55:55.689019+0000","last_undegraded":"2024-03-30T14:55:55.689019+0000","last_fullsized":"2024-03-30T14:55:55.689019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.138775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"76'10","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454724+0000","last_change":"2024-03-30T14:54:39.642246+0000","last_active":"2024-03-30T14:55:55.454724+0000","last_peered":"2024-03-30T14:55:55.454724+0000","last_clean":"2024-03-30T14:55:55.454724+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T14:55:55.454724+0000","last_undegraded":"2024-03-30T14:55:55.454724+0000","last_fullsized":"2024-03-30T14:55:55.454724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.644108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689676+0000","last_change":"2024-03-30T14:55:49.280406+0000","last_active":"2024-03-30T14:55:55.689676+0000","last_peered":"2024-03-30T14:55:55.689676+0000","last_clean":"2024-03-30T14:55:55.689676+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T14:55:55.689676+0000","last_undegraded":"2024-03-30T14:55:55.689676+0000","last_fullsized":"2024-03-30T14:55:55.689676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.280327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.280327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.497970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454883+0000","last_change":"2024-03-30T14:54:41.347270+0000","last_active":"2024-03-30T14:55:55.454883+0000","last_peered":"2024-03-30T14:55:55.454883+0000","last_clean":"2024-03-30T14:55:55.454883+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T14:55:55.454883+0000","last_undegraded":"2024-03-30T14:55:55.454883+0000","last_fullsized":"2024-03-30T14:55:55.454883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.840620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455463+0000","last_change":"2024-03-30T14:54:37.321125+0000","last_active":"2024-03-30T14:55:55.455463+0000","last_peered":"2024-03-30T14:55:55.455463+0000","last_clean":"2024-03-30T14:55:55.455463+0000","last_became_active":"2024-03-30T14:54:37.320726+0000","last_became_peered":"2024-03-30T14:54:37.320726+0000","last_unstale":"2024-03-30T14:55:55.455463+0000","last_undegraded":"2024-03-30T14:55:55.455463+0000","last_fullsized":"2024-03-30T14:55:55.455463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.061192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"76'11","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583631+0000","last_change":"2024-03-30T14:54:39.338805+0000","last_active":"2024-03-30T14:55:55.583631+0000","last_peered":"2024-03-30T14:55:55.583631+0000","last_clean":"2024-03-30T14:55:55.583631+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T14:55:55.583631+0000","last_undegraded":"2024-03-30T14:55:55.583631+0000","last_fullsized":"2024-03-30T14:55:55.583631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.870023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688948+0000","last_change":"2024-03-30T14:54:43.352814+0000","last_active":"2024-03-30T14:55:55.688948+0000","last_peered":"2024-03-30T14:55:55.688948+0000","last_clean":"2024-03-30T14:55:55.688948+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T14:55:55.688948+0000","last_undegraded":"2024-03-30T14:55:55.688948+0000","last_fullsized":"2024-03-30T14:55:55.688948+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.775606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.582874+0000","last_change":"2024-03-30T14:55:53.203651+0000","last_active":"2024-03-30T14:55:55.582874+0000","last_peered":"2024-03-30T14:55:55.582874+0000","last_clean":"2024-03-30T14:55:55.582874+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T14:55:55.582874+0000","last_undegraded":"2024-03-30T14:55:55.582874+0000","last_fullsized":"2024-03-30T14:55:55.582874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.203533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.203533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:56.173167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454668+0000","last_change":"2024-03-30T14:54:37.324115+0000","last_active":"2024-03-30T14:55:55.454668+0000","last_peered":"2024-03-30T14:55:55.454668+0000","last_clean":"2024-03-30T14:55:55.454668+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T14:55:55.454668+0000","last_undegraded":"2024-03-30T14:55:55.454668+0000","last_fullsized":"2024-03-30T14:55:55.454668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455583+0000","last_change":"2024-03-30T14:54:39.338811+0000","last_active":"2024-03-30T14:55:55.455583+0000","last_peered":"2024-03-30T14:55:55.455583+0000","last_clean":"2024-03-30T14:55:55.455583+0000","last_became_active":"2024-03-30T14:54:39.338462+0000","last_became_peered":"2024-03-30T14:54:39.338462+0000","last_unstale":"2024-03-30T14:55:55.455583+0000","last_undegraded":"2024-03-30T14:55:55.455583+0000","last_fullsized":"2024-03-30T14:55:55.455583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455118+0000","last_change":"2024-03-30T14:54:43.361222+0000","last_active":"2024-03-30T14:55:55.455118+0000","last_peered":"2024-03-30T14:55:55.455118+0000","last_clean":"2024-03-30T14:55:55.455118+0000","last_became_active":"2024-03-30T14:54:43.360837+0000","last_became_peered":"2024-03-30T14:54:43.360837+0000","last_unstale":"2024-03-30T14:55:55.455118+0000","last_undegraded":"2024-03-30T14:55:55.455118+0000","last_fullsized":"2024-03-30T14:55:55.455118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.092716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691130+0000","last_change":"2024-03-30T14:54:41.354167+0000","last_active":"2024-03-30T14:55:55.691130+0000","last_peered":"2024-03-30T14:55:55.691130+0000","last_clean":"2024-03-30T14:55:55.691130+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T14:55:55.691130+0000","last_undegraded":"2024-03-30T14:55:55.691130+0000","last_fullsized":"2024-03-30T14:55:55.691130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:08.473023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689253+0000","last_change":"2024-03-30T14:54:37.331035+0000","last_active":"2024-03-30T14:55:55.689253+0000","last_peered":"2024-03-30T14:55:55.689253+0000","last_clean":"2024-03-30T14:55:55.689253+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T14:55:55.689253+0000","last_undegraded":"2024-03-30T14:55:55.689253+0000","last_fullsized":"2024-03-30T14:55:55.689253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.994637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456121+0000","last_change":"2024-03-30T14:54:39.641424+0000","last_active":"2024-03-30T14:55:55.456121+0000","last_peered":"2024-03-30T14:55:55.456121+0000","last_clean":"2024-03-30T14:55:55.456121+0000","last_became_active":"2024-03-30T14:54:39.641142+0000","last_became_peered":"2024-03-30T14:54:39.641142+0000","last_unstale":"2024-03-30T14:55:55.456121+0000","last_undegraded":"2024-03-30T14:55:55.456121+0000","last_fullsized":"2024-03-30T14:55:55.456121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453011+0000","last_change":"2024-03-30T14:54:43.352784+0000","last_active":"2024-03-30T14:55:55.453011+0000","last_peered":"2024-03-30T14:55:55.453011+0000","last_clean":"2024-03-30T14:55:55.453011+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T14:55:55.453011+0000","last_undegraded":"2024-03-30T14:55:55.453011+0000","last_fullsized":"2024-03-30T14:55:55.453011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.948790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455487+0000","last_change":"2024-03-30T14:54:41.347572+0000","last_active":"2024-03-30T14:55:55.455487+0000","last_peered":"2024-03-30T14:55:55.455487+0000","last_clean":"2024-03-30T14:55:55.455487+0000","last_became_active":"2024-03-30T14:54:41.347229+0000","last_became_peered":"2024-03-30T14:54:41.347229+0000","last_unstale":"2024-03-30T14:55:55.455487+0000","last_undegraded":"2024-03-30T14:55:55.455487+0000","last_fullsized":"2024-03-30T14:55:55.455487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.472022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:03.093322+0000","last_change":"2024-03-30T14:56:03.093369+0000","last_active":"2024-03-30T14:56:03.093322+0000","last_peered":"2024-03-30T14:56:03.093322+0000","last_clean":"2024-03-30T14:56:03.093322+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T14:56:03.093322+0000","last_undegraded":"2024-03-30T14:56:03.093322+0000","last_fullsized":"2024-03-30T14:56:03.093322+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:56:03.093218+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.894705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689794+0000","last_change":"2024-03-30T14:54:39.339910+0000","last_active":"2024-03-30T14:55:55.689794+0000","last_peered":"2024-03-30T14:55:55.689794+0000","last_clean":"2024-03-30T14:55:55.689794+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T14:55:55.689794+0000","last_undegraded":"2024-03-30T14:55:55.689794+0000","last_fullsized":"2024-03-30T14:55:55.689794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.626451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455873+0000","last_change":"2024-03-30T14:54:43.360542+0000","last_active":"2024-03-30T14:55:55.455873+0000","last_peered":"2024-03-30T14:55:55.455873+0000","last_clean":"2024-03-30T14:55:55.455873+0000","last_became_active":"2024-03-30T14:54:43.360215+0000","last_became_peered":"2024-03-30T14:54:43.360215+0000","last_unstale":"2024-03-30T14:55:55.455873+0000","last_undegraded":"2024-03-30T14:55:55.455873+0000","last_fullsized":"2024-03-30T14:55:55.455873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.894321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689567+0000","last_change":"2024-03-30T14:55:53.260188+0000","last_active":"2024-03-30T14:55:55.689567+0000","last_peered":"2024-03-30T14:55:55.689567+0000","last_clean":"2024-03-30T14:55:55.689567+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T14:55:55.689567+0000","last_undegraded":"2024-03-30T14:55:55.689567+0000","last_fullsized":"2024-03-30T14:55:55.689567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.260101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.260101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.452479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:00.482929+0000","last_change":"2024-03-30T14:56:00.482961+0000","last_active":"2024-03-30T14:56:00.482929+0000","last_peered":"2024-03-30T14:56:00.482929+0000","last_clean":"2024-03-30T14:56:00.482929+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T14:56:00.482929+0000","last_undegraded":"2024-03-30T14:56:00.482929+0000","last_fullsized":"2024-03-30T14:56:00.482929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:00.482861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.482861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.363058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"76'11","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689582+0000","last_change":"2024-03-30T14:54:39.334527+0000","last_active":"2024-03-30T14:55:55.689582+0000","last_peered":"2024-03-30T14:55:55.689582+0000","last_clean":"2024-03-30T14:55:55.689582+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T14:55:55.689582+0000","last_undegraded":"2024-03-30T14:55:55.689582+0000","last_fullsized":"2024-03-30T14:55:55.689582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.307222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"76'1","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454391+0000","last_change":"2024-03-30T14:54:43.354400+0000","last_active":"2024-03-30T14:55:55.454391+0000","last_peered":"2024-03-30T14:55:55.454391+0000","last_clean":"2024-03-30T14:55:55.454391+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T14:55:55.454391+0000","last_undegraded":"2024-03-30T14:55:55.454391+0000","last_fullsized":"2024-03-30T14:55:55.454391+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.205644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.529330+0000","last_change":"2024-03-30T14:54:41.350464+0000","last_active":"2024-03-30T14:55:57.529330+0000","last_peered":"2024-03-30T14:55:57.529330+0000","last_clean":"2024-03-30T14:55:57.529330+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T14:55:57.529330+0000","last_undegraded":"2024-03-30T14:55:57.529330+0000","last_fullsized":"2024-03-30T14:55:57.529330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690443+0000","last_change":"2024-03-30T14:54:37.474723+0000","last_active":"2024-03-30T14:55:55.690443+0000","last_peered":"2024-03-30T14:55:55.690443+0000","last_clean":"2024-03-30T14:55:55.690443+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T14:55:55.690443+0000","last_undegraded":"2024-03-30T14:55:55.690443+0000","last_fullsized":"2024-03-30T14:55:55.690443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.251496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455998+0000","last_change":"2024-03-30T14:55:53.742586+0000","last_active":"2024-03-30T14:55:55.455998+0000","last_peered":"2024-03-30T14:55:55.455998+0000","last_clean":"2024-03-30T14:55:55.455998+0000","last_became_active":"2024-03-30T14:54:39.642132+0000","last_became_peered":"2024-03-30T14:54:39.642132+0000","last_unstale":"2024-03-30T14:55:55.455998+0000","last_undegraded":"2024-03-30T14:55:55.455998+0000","last_fullsized":"2024-03-30T14:55:55.455998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:53.742492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.742492+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.865684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690251+0000","last_change":"2024-03-30T14:54:43.349809+0000","last_active":"2024-03-30T14:55:55.690251+0000","last_peered":"2024-03-30T14:55:55.690251+0000","last_clean":"2024-03-30T14:55:55.690251+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T14:55:55.690251+0000","last_undegraded":"2024-03-30T14:55:55.690251+0000","last_fullsized":"2024-03-30T14:55:55.690251+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.367821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"76'8","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528740+0000","last_change":"2024-03-30T14:54:41.346770+0000","last_active":"2024-03-30T14:55:57.528740+0000","last_peered":"2024-03-30T14:55:57.528740+0000","last_clean":"2024-03-30T14:55:57.528740+0000","last_became_active":"2024-03-30T14:54:41.346418+0000","last_became_peered":"2024-03-30T14:54:41.346418+0000","last_unstale":"2024-03-30T14:55:57.528740+0000","last_undegraded":"2024-03-30T14:55:57.528740+0000","last_fullsized":"2024-03-30T14:55:57.528740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.389636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:02.515777+0000","last_change":"2024-03-30T14:56:02.515803+0000","last_active":"2024-03-30T14:56:02.515777+0000","last_peered":"2024-03-30T14:56:02.515777+0000","last_clean":"2024-03-30T14:56:02.515777+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T14:56:02.515777+0000","last_undegraded":"2024-03-30T14:56:02.515777+0000","last_fullsized":"2024-03-30T14:56:02.515777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.515709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.515709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.607511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691552+0000","last_change":"2024-03-30T14:54:39.644472+0000","last_active":"2024-03-30T14:55:55.691552+0000","last_peered":"2024-03-30T14:55:55.691552+0000","last_clean":"2024-03-30T14:55:55.691552+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T14:55:55.691552+0000","last_undegraded":"2024-03-30T14:55:55.691552+0000","last_fullsized":"2024-03-30T14:55:55.691552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.992331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689079+0000","last_change":"2024-03-30T14:55:48.721601+0000","last_active":"2024-03-30T14:55:55.689079+0000","last_peered":"2024-03-30T14:55:55.689079+0000","last_clean":"2024-03-30T14:55:55.689079+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T14:55:55.689079+0000","last_undegraded":"2024-03-30T14:55:55.689079+0000","last_fullsized":"2024-03-30T14:55:55.689079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.721477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.721477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:58.996126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456733+0000","last_change":"2024-03-30T14:54:41.345899+0000","last_active":"2024-03-30T14:55:55.456733+0000","last_peered":"2024-03-30T14:55:55.456733+0000","last_clean":"2024-03-30T14:55:55.456733+0000","last_became_active":"2024-03-30T14:54:41.345543+0000","last_became_peered":"2024-03-30T14:54:41.345543+0000","last_unstale":"2024-03-30T14:55:55.456733+0000","last_undegraded":"2024-03-30T14:55:55.456733+0000","last_fullsized":"2024-03-30T14:55:55.456733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689695+0000","last_change":"2024-03-30T14:55:48.068870+0000","last_active":"2024-03-30T14:55:55.689695+0000","last_peered":"2024-03-30T14:55:55.689695+0000","last_clean":"2024-03-30T14:55:55.689695+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T14:55:55.689695+0000","last_undegraded":"2024-03-30T14:55:55.689695+0000","last_fullsized":"2024-03-30T14:55:55.689695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.068767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.068767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:16.799841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454788+0000","last_change":"2024-03-30T14:54:39.642222+0000","last_active":"2024-03-30T14:55:55.454788+0000","last_peered":"2024-03-30T14:55:55.454788+0000","last_clean":"2024-03-30T14:55:55.454788+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T14:55:55.454788+0000","last_undegraded":"2024-03-30T14:55:55.454788+0000","last_fullsized":"2024-03-30T14:55:55.454788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.598823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689141+0000","last_change":"2024-03-30T14:54:43.357877+0000","last_active":"2024-03-30T14:55:55.689141+0000","last_peered":"2024-03-30T14:55:55.689141+0000","last_clean":"2024-03-30T14:55:55.689141+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T14:55:55.689141+0000","last_undegraded":"2024-03-30T14:55:55.689141+0000","last_fullsized":"2024-03-30T14:55:55.689141+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.019353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689345+0000","last_change":"2024-03-30T14:54:41.352746+0000","last_active":"2024-03-30T14:55:55.689345+0000","last_peered":"2024-03-30T14:55:55.689345+0000","last_clean":"2024-03-30T14:55:55.689345+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T14:55:55.689345+0000","last_undegraded":"2024-03-30T14:55:55.689345+0000","last_fullsized":"2024-03-30T14:55:55.689345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.582844+0000","last_change":"2024-03-30T14:55:50.191458+0000","last_active":"2024-03-30T14:55:55.582844+0000","last_peered":"2024-03-30T14:55:55.582844+0000","last_clean":"2024-03-30T14:55:55.582844+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T14:55:55.582844+0000","last_undegraded":"2024-03-30T14:55:55.582844+0000","last_fullsized":"2024-03-30T14:55:55.582844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.191353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.191353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:00.197326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456795+0000","last_change":"2024-03-30T14:55:48.827002+0000","last_active":"2024-03-30T14:55:55.456795+0000","last_peered":"2024-03-30T14:55:55.456795+0000","last_clean":"2024-03-30T14:55:55.456795+0000","last_became_active":"2024-03-30T14:54:39.643439+0000","last_became_peered":"2024-03-30T14:54:39.643439+0000","last_unstale":"2024-03-30T14:55:55.456795+0000","last_undegraded":"2024-03-30T14:55:55.456795+0000","last_fullsized":"2024-03-30T14:55:55.456795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:55:48.826912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.826912+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.602167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691673+0000","last_change":"2024-03-30T14:54:43.352897+0000","last_active":"2024-03-30T14:55:55.691673+0000","last_peered":"2024-03-30T14:55:55.691673+0000","last_clean":"2024-03-30T14:55:55.691673+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:55:55.691673+0000","last_undegraded":"2024-03-30T14:55:55.691673+0000","last_fullsized":"2024-03-30T14:55:55.691673+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:55.730639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690180+0000","last_change":"2024-03-30T14:54:37.330956+0000","last_active":"2024-03-30T14:55:55.690180+0000","last_peered":"2024-03-30T14:55:55.690180+0000","last_clean":"2024-03-30T14:55:55.690180+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T14:55:55.690180+0000","last_undegraded":"2024-03-30T14:55:55.690180+0000","last_fullsized":"2024-03-30T14:55:55.690180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.101820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688552+0000","last_change":"2024-03-30T14:54:39.643618+0000","last_active":"2024-03-30T14:55:55.688552+0000","last_peered":"2024-03-30T14:55:55.688552+0000","last_clean":"2024-03-30T14:55:55.688552+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T14:55:55.688552+0000","last_undegraded":"2024-03-30T14:55:55.688552+0000","last_fullsized":"2024-03-30T14:55:55.688552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:06.788995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528959+0000","last_change":"2024-03-30T14:54:41.352889+0000","last_active":"2024-03-30T14:55:57.528959+0000","last_peered":"2024-03-30T14:55:57.528959+0000","last_clean":"2024-03-30T14:55:57.528959+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T14:55:57.528959+0000","last_undegraded":"2024-03-30T14:55:57.528959+0000","last_fullsized":"2024-03-30T14:55:57.528959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.905986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455233+0000","last_change":"2024-03-30T14:54:43.358959+0000","last_active":"2024-03-30T14:55:55.455233+0000","last_peered":"2024-03-30T14:55:55.455233+0000","last_clean":"2024-03-30T14:55:55.455233+0000","last_became_active":"2024-03-30T14:54:43.358552+0000","last_became_peered":"2024-03-30T14:54:43.358552+0000","last_unstale":"2024-03-30T14:55:55.455233+0000","last_undegraded":"2024-03-30T14:55:55.455233+0000","last_fullsized":"2024-03-30T14:55:55.455233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.302782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689652+0000","last_change":"2024-03-30T14:54:37.328695+0000","last_active":"2024-03-30T14:55:55.689652+0000","last_peered":"2024-03-30T14:55:55.689652+0000","last_clean":"2024-03-30T14:55:55.689652+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T14:55:55.689652+0000","last_undegraded":"2024-03-30T14:55:55.689652+0000","last_fullsized":"2024-03-30T14:55:55.689652+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.320517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.431740+0000","last_change":"2024-03-30T14:55:56.431786+0000","last_active":"2024-03-30T14:55:56.431740+0000","last_peered":"2024-03-30T14:55:56.431740+0000","last_clean":"2024-03-30T14:55:56.431740+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T14:55:56.431740+0000","last_undegraded":"2024-03-30T14:55:56.431740+0000","last_fullsized":"2024-03-30T14:55:56.431740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:56.431647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.431647+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.459883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688280+0000","last_change":"2024-03-30T14:54:41.350489+0000","last_active":"2024-03-30T14:55:55.688280+0000","last_peered":"2024-03-30T14:55:55.688280+0000","last_clean":"2024-03-30T14:55:55.688280+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T14:55:55.688280+0000","last_undegraded":"2024-03-30T14:55:55.688280+0000","last_fullsized":"2024-03-30T14:55:55.688280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.343336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688763+0000","last_change":"2024-03-30T14:54:43.356146+0000","last_active":"2024-03-30T14:55:55.688763+0000","last_peered":"2024-03-30T14:55:55.688763+0000","last_clean":"2024-03-30T14:55:55.688763+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T14:55:55.688763+0000","last_undegraded":"2024-03-30T14:55:55.688763+0000","last_fullsized":"2024-03-30T14:55:55.688763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.912258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:02.650742+0000","last_change":"2024-03-30T14:56:02.650786+0000","last_active":"2024-03-30T14:56:02.650742+0000","last_peered":"2024-03-30T14:56:02.650742+0000","last_clean":"2024-03-30T14:56:02.650742+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T14:56:02.650742+0000","last_undegraded":"2024-03-30T14:56:02.650742+0000","last_fullsized":"2024-03-30T14:56:02.650742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.650642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.650642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:08.689802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689516+0000","last_change":"2024-03-30T14:54:37.329066+0000","last_active":"2024-03-30T14:55:55.689516+0000","last_peered":"2024-03-30T14:55:55.689516+0000","last_clean":"2024-03-30T14:55:55.689516+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T14:55:55.689516+0000","last_undegraded":"2024-03-30T14:55:55.689516+0000","last_fullsized":"2024-03-30T14:55:55.689516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.221906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":118,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583247+0000","last_change":"2024-03-30T14:55:52.189892+0000","last_active":"2024-03-30T14:55:55.583247+0000","last_peered":"2024-03-30T14:55:55.583247+0000","last_clean":"2024-03-30T14:55:55.583247+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T14:55:55.583247+0000","last_undegraded":"2024-03-30T14:55:55.583247+0000","last_fullsized":"2024-03-30T14:55:55.583247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:55:52.189742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.189742+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.639688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453705+0000","last_change":"2024-03-30T14:54:43.353107+0000","last_active":"2024-03-30T14:55:55.453705+0000","last_peered":"2024-03-30T14:55:55.453705+0000","last_clean":"2024-03-30T14:55:55.453705+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T14:55:55.453705+0000","last_undegraded":"2024-03-30T14:55:55.453705+0000","last_fullsized":"2024-03-30T14:55:55.453705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:01.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.679600+0000","last_change":"2024-03-30T14:55:56.679624+0000","last_active":"2024-03-30T14:55:56.679600+0000","last_peered":"2024-03-30T14:55:56.679600+0000","last_clean":"2024-03-30T14:55:56.679600+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T14:55:56.679600+0000","last_undegraded":"2024-03-30T14:55:56.679600+0000","last_fullsized":"2024-03-30T14:55:56.679600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.679531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.679531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.763850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691482+0000","last_change":"2024-03-30T14:55:37.491888+0000","last_active":"2024-03-30T14:55:55.691482+0000","last_peered":"2024-03-30T14:55:55.691482+0000","last_clean":"2024-03-30T14:55:55.691482+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T14:55:55.691482+0000","last_undegraded":"2024-03-30T14:55:55.691482+0000","last_fullsized":"2024-03-30T14:55:55.691482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:55:37.491750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.952763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453703+0000","last_change":"2024-03-30T14:55:48.585645+0000","last_active":"2024-03-30T14:55:55.453703+0000","last_peered":"2024-03-30T14:55:55.453703+0000","last_clean":"2024-03-30T14:55:55.453703+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T14:55:55.453703+0000","last_undegraded":"2024-03-30T14:55:55.453703+0000","last_fullsized":"2024-03-30T14:55:55.453703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:48.585557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.585557+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.155340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:59.683220+0000","last_change":"2024-03-30T14:55:59.683242+0000","last_active":"2024-03-30T14:55:59.683220+0000","last_peered":"2024-03-30T14:55:59.683220+0000","last_clean":"2024-03-30T14:55:59.683220+0000","last_became_active":"2024-03-30T14:54:43.356453+0000","last_became_peered":"2024-03-30T14:54:43.356453+0000","last_unstale":"2024-03-30T14:55:59.683220+0000","last_undegraded":"2024-03-30T14:55:59.683220+0000","last_fullsized":"2024-03-30T14:55:59.683220+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.683158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.683158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.228244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":68,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691401+0000","last_change":"2024-03-30T14:54:40.343883+0000","last_active":"2024-03-30T14:55:55.691401+0000","last_peered":"2024-03-30T14:55:55.691401+0000","last_clean":"2024-03-30T14:55:55.691401+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T14:55:55.691401+0000","last_undegraded":"2024-03-30T14:55:55.691401+0000","last_fullsized":"2024-03-30T14:55:55.691401+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.689686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689244+0000","last_change":"2024-03-30T14:54:37.326038+0000","last_active":"2024-03-30T14:55:55.689244+0000","last_peered":"2024-03-30T14:55:55.689244+0000","last_clean":"2024-03-30T14:55:55.689244+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T14:55:55.689244+0000","last_undegraded":"2024-03-30T14:55:55.689244+0000","last_fullsized":"2024-03-30T14:55:55.689244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.529280+0000","last_change":"2024-03-30T14:54:58.412647+0000","last_active":"2024-03-30T14:55:57.529280+0000","last_peered":"2024-03-30T14:55:57.529280+0000","last_clean":"2024-03-30T14:55:57.529280+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T14:55:57.529280+0000","last_undegraded":"2024-03-30T14:55:57.529280+0000","last_fullsized":"2024-03-30T14:55:57.529280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454357+0000","last_change":"2024-03-30T14:54:41.349882+0000","last_active":"2024-03-30T14:55:55.454357+0000","last_peered":"2024-03-30T14:55:55.454357+0000","last_clean":"2024-03-30T14:55:55.454357+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T14:55:55.454357+0000","last_undegraded":"2024-03-30T14:55:55.454357+0000","last_fullsized":"2024-03-30T14:55:55.454357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:56.007208+0000","last_change":"2024-03-30T14:55:56.007233+0000","last_active":"2024-03-30T14:55:56.007208+0000","last_peered":"2024-03-30T14:55:56.007208+0000","last_clean":"2024-03-30T14:55:56.007208+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T14:55:56.007208+0000","last_undegraded":"2024-03-30T14:55:56.007208+0000","last_fullsized":"2024-03-30T14:55:56.007208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.007133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.007133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.546510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689643+0000","last_change":"2024-03-30T14:54:37.475240+0000","last_active":"2024-03-30T14:55:55.689643+0000","last_peered":"2024-03-30T14:55:55.689643+0000","last_clean":"2024-03-30T14:55:55.689643+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T14:55:55.689643+0000","last_undegraded":"2024-03-30T14:55:55.689643+0000","last_fullsized":"2024-03-30T14:55:55.689643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688318+0000","last_change":"2024-03-30T14:55:49.700155+0000","last_active":"2024-03-30T14:55:55.688318+0000","last_peered":"2024-03-30T14:55:55.688318+0000","last_clean":"2024-03-30T14:55:55.688318+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T14:55:55.688318+0000","last_undegraded":"2024-03-30T14:55:55.688318+0000","last_fullsized":"2024-03-30T14:55:55.688318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.700022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.949979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"76'13","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:00.305477+0000","last_change":"2024-03-30T14:56:00.305512+0000","last_active":"2024-03-30T14:56:00.305477+0000","last_peered":"2024-03-30T14:56:00.305477+0000","last_clean":"2024-03-30T14:56:00.305477+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T14:56:00.305477+0000","last_undegraded":"2024-03-30T14:56:00.305477+0000","last_fullsized":"2024-03-30T14:56:00.305477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'13","last_scrub_stamp":"2024-03-30T14:56:00.305404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.305404+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.155134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.693768+0000","last_change":"2024-03-30T14:55:57.693792+0000","last_active":"2024-03-30T14:55:57.693768+0000","last_peered":"2024-03-30T14:55:57.693768+0000","last_clean":"2024-03-30T14:55:57.693768+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T14:55:57.693768+0000","last_undegraded":"2024-03-30T14:55:57.693768+0000","last_fullsized":"2024-03-30T14:55:57.693768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.693703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.693703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.637711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689570+0000","last_change":"2024-03-30T14:54:43.356807+0000","last_active":"2024-03-30T14:55:55.689570+0000","last_peered":"2024-03-30T14:55:55.689570+0000","last_clean":"2024-03-30T14:55:55.689570+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T14:55:55.689570+0000","last_undegraded":"2024-03-30T14:55:55.689570+0000","last_fullsized":"2024-03-30T14:55:55.689570+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:08.744451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690973+0000","last_change":"2024-03-30T14:55:05.131524+0000","last_active":"2024-03-30T14:55:55.690973+0000","last_peered":"2024-03-30T14:55:55.690973+0000","last_clean":"2024-03-30T14:55:55.690973+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T14:55:55.690973+0000","last_undegraded":"2024-03-30T14:55:55.690973+0000","last_fullsized":"2024-03-30T14:55:55.690973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-30T14:55:05.131440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:55:05.131440+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:33.848087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454922+0000","last_change":"2024-03-30T14:54:37.324739+0000","last_active":"2024-03-30T14:55:55.454922+0000","last_peered":"2024-03-30T14:55:55.454922+0000","last_clean":"2024-03-30T14:55:55.454922+0000","last_became_active":"2024-03-30T14:54:37.324423+0000","last_became_peered":"2024-03-30T14:54:37.324423+0000","last_unstale":"2024-03-30T14:55:55.454922+0000","last_undegraded":"2024-03-30T14:55:55.454922+0000","last_fullsized":"2024-03-30T14:55:55.454922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.830680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528972+0000","last_change":"2024-03-30T14:54:39.643201+0000","last_active":"2024-03-30T14:55:57.528972+0000","last_peered":"2024-03-30T14:55:57.528972+0000","last_clean":"2024-03-30T14:55:57.528972+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T14:55:57.528972+0000","last_undegraded":"2024-03-30T14:55:57.528972+0000","last_fullsized":"2024-03-30T14:55:57.528972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:03.482165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690544+0000","last_change":"2024-03-30T14:54:41.350344+0000","last_active":"2024-03-30T14:55:55.690544+0000","last_peered":"2024-03-30T14:55:55.690544+0000","last_clean":"2024-03-30T14:55:55.690544+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T14:55:55.690544+0000","last_undegraded":"2024-03-30T14:55:55.690544+0000","last_fullsized":"2024-03-30T14:55:55.690544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.814152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688709+0000","last_change":"2024-03-30T14:54:43.357715+0000","last_active":"2024-03-30T14:55:55.688709+0000","last_peered":"2024-03-30T14:55:55.688709+0000","last_clean":"2024-03-30T14:55:55.688709+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T14:55:55.688709+0000","last_undegraded":"2024-03-30T14:55:55.688709+0000","last_fullsized":"2024-03-30T14:55:55.688709+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:07.152983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453356+0000","last_change":"2024-03-30T14:54:37.326510+0000","last_active":"2024-03-30T14:55:55.453356+0000","last_peered":"2024-03-30T14:55:55.453356+0000","last_clean":"2024-03-30T14:55:55.453356+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T14:55:55.453356+0000","last_undegraded":"2024-03-30T14:55:55.453356+0000","last_fullsized":"2024-03-30T14:55:55.453356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.302833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:01.995348+0000","last_change":"2024-03-30T14:54:40.338153+0000","last_active":"2024-03-30T14:56:01.995348+0000","last_peered":"2024-03-30T14:56:01.995348+0000","last_clean":"2024-03-30T14:56:01.995348+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T14:56:01.995348+0000","last_undegraded":"2024-03-30T14:56:01.995348+0000","last_fullsized":"2024-03-30T14:56:01.995348+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454509+0000","last_change":"2024-03-30T14:55:52.556360+0000","last_active":"2024-03-30T14:55:55.454509+0000","last_peered":"2024-03-30T14:55:55.454509+0000","last_clean":"2024-03-30T14:55:55.454509+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T14:55:55.454509+0000","last_undegraded":"2024-03-30T14:55:55.454509+0000","last_fullsized":"2024-03-30T14:55:55.454509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:52.556237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.556237+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.659172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.530391+0000","last_change":"2024-03-30T14:55:57.530418+0000","last_active":"2024-03-30T14:55:57.530391+0000","last_peered":"2024-03-30T14:55:57.530391+0000","last_clean":"2024-03-30T14:55:57.530391+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T14:55:57.530391+0000","last_undegraded":"2024-03-30T14:55:57.530391+0000","last_fullsized":"2024-03-30T14:55:57.530391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:01.784600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:56:01.265146+0000","last_change":"2024-03-30T14:56:01.265168+0000","last_active":"2024-03-30T14:56:01.265146+0000","last_peered":"2024-03-30T14:56:01.265146+0000","last_clean":"2024-03-30T14:56:01.265146+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T14:56:01.265146+0000","last_undegraded":"2024-03-30T14:56:01.265146+0000","last_fullsized":"2024-03-30T14:56:01.265146+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:01.265084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.265084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.573818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688979+0000","last_change":"2024-03-30T14:55:42.232642+0000","last_active":"2024-03-30T14:55:55.688979+0000","last_peered":"2024-03-30T14:55:55.688979+0000","last_clean":"2024-03-30T14:55:55.688979+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T14:55:55.688979+0000","last_undegraded":"2024-03-30T14:55:55.688979+0000","last_fullsized":"2024-03-30T14:55:55.688979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.232525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.232525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:48.072521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"76'19","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453942+0000","last_change":"2024-03-30T14:54:39.642101+0000","last_active":"2024-03-30T14:55:55.453942+0000","last_peered":"2024-03-30T14:55:55.453942+0000","last_clean":"2024-03-30T14:55:55.453942+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T14:55:55.453942+0000","last_undegraded":"2024-03-30T14:55:55.453942+0000","last_fullsized":"2024-03-30T14:55:55.453942+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:57.046560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583332+0000","last_change":"2024-03-30T14:54:41.345132+0000","last_active":"2024-03-30T14:55:55.583332+0000","last_peered":"2024-03-30T14:55:55.583332+0000","last_clean":"2024-03-30T14:55:55.583332+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T14:55:55.583332+0000","last_undegraded":"2024-03-30T14:55:55.583332+0000","last_fullsized":"2024-03-30T14:55:55.583332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:02.169133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.453022+0000","last_change":"2024-03-30T14:54:43.355213+0000","last_active":"2024-03-30T14:55:55.453022+0000","last_peered":"2024-03-30T14:55:55.453022+0000","last_clean":"2024-03-30T14:55:55.453022+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T14:55:55.453022+0000","last_undegraded":"2024-03-30T14:55:55.453022+0000","last_fullsized":"2024-03-30T14:55:55.453022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.304491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691328+0000","last_change":"2024-03-30T14:55:41.454468+0000","last_active":"2024-03-30T14:55:55.691328+0000","last_peered":"2024-03-30T14:55:55.691328+0000","last_clean":"2024-03-30T14:55:55.691328+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T14:55:55.691328+0000","last_undegraded":"2024-03-30T14:55:55.691328+0000","last_fullsized":"2024-03-30T14:55:55.691328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.454365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:04.526213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"76'10","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689681+0000","last_change":"2024-03-30T14:54:39.338666+0000","last_active":"2024-03-30T14:55:55.689681+0000","last_peered":"2024-03-30T14:55:55.689681+0000","last_clean":"2024-03-30T14:55:55.689681+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T14:55:55.689681+0000","last_undegraded":"2024-03-30T14:55:55.689681+0000","last_fullsized":"2024-03-30T14:55:55.689681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528402+0000","last_change":"2024-03-30T14:55:50.585912+0000","last_active":"2024-03-30T14:55:57.528402+0000","last_peered":"2024-03-30T14:55:57.528402+0000","last_clean":"2024-03-30T14:55:57.528402+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T14:55:57.528402+0000","last_undegraded":"2024-03-30T14:55:57.528402+0000","last_fullsized":"2024-03-30T14:55:57.528402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:50.585773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.585773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.447085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690700+0000","last_change":"2024-03-30T14:55:49.029645+0000","last_active":"2024-03-30T14:55:55.690700+0000","last_peered":"2024-03-30T14:55:55.690700+0000","last_clean":"2024-03-30T14:55:55.690700+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T14:55:55.690700+0000","last_undegraded":"2024-03-30T14:55:55.690700+0000","last_fullsized":"2024-03-30T14:55:55.690700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.029548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.029548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:52.389450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"69'1","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454618+0000","last_change":"2024-03-30T14:54:37.326159+0000","last_active":"2024-03-30T14:55:55.454618+0000","last_peered":"2024-03-30T14:55:55.454618+0000","last_clean":"2024-03-30T14:55:55.454618+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T14:55:55.454618+0000","last_undegraded":"2024-03-30T14:55:55.454618+0000","last_fullsized":"2024-03-30T14:55:55.454618+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:59.929704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690207+0000","last_change":"2024-03-30T14:55:46.303554+0000","last_active":"2024-03-30T14:55:55.690207+0000","last_peered":"2024-03-30T14:55:55.690207+0000","last_clean":"2024-03-30T14:55:55.690207+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T14:55:55.690207+0000","last_undegraded":"2024-03-30T14:55:55.690207+0000","last_fullsized":"2024-03-30T14:55:55.690207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:55:46.303412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.303412+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.425234+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"76'8","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528465+0000","last_change":"2024-03-30T14:55:55.692425+0000","last_active":"2024-03-30T14:55:57.528465+0000","last_peered":"2024-03-30T14:55:57.528465+0000","last_clean":"2024-03-30T14:55:57.528465+0000","last_became_active":"2024-03-30T14:54:41.353489+0000","last_became_peered":"2024-03-30T14:54:41.353489+0000","last_unstale":"2024-03-30T14:55:57.528465+0000","last_undegraded":"2024-03-30T14:55:57.528465+0000","last_fullsized":"2024-03-30T14:55:57.528465+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:55.692261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:55.692261+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.231384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689326+0000","last_change":"2024-03-30T14:54:43.352748+0000","last_active":"2024-03-30T14:55:55.689326+0000","last_peered":"2024-03-30T14:55:55.689326+0000","last_clean":"2024-03-30T14:55:55.689326+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T14:55:55.689326+0000","last_undegraded":"2024-03-30T14:55:55.689326+0000","last_fullsized":"2024-03-30T14:55:55.689326+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.488227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"69'1","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.456566+0000","last_change":"2024-03-30T14:54:37.323745+0000","last_active":"2024-03-30T14:55:55.456566+0000","last_peered":"2024-03-30T14:55:55.456566+0000","last_clean":"2024-03-30T14:55:55.456566+0000","last_became_active":"2024-03-30T14:54:37.323384+0000","last_became_peered":"2024-03-30T14:54:37.323384+0000","last_unstale":"2024-03-30T14:55:55.456566+0000","last_undegraded":"2024-03-30T14:55:55.456566+0000","last_fullsized":"2024-03-30T14:55:55.456566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:04.422909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"76'18","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455711+0000","last_change":"2024-03-30T14:54:39.642940+0000","last_active":"2024-03-30T14:55:55.455711+0000","last_peered":"2024-03-30T14:55:55.455711+0000","last_clean":"2024-03-30T14:55:55.455711+0000","last_became_active":"2024-03-30T14:54:39.642450+0000","last_became_peered":"2024-03-30T14:54:39.642450+0000","last_unstale":"2024-03-30T14:55:55.455711+0000","last_undegraded":"2024-03-30T14:55:55.455711+0000","last_fullsized":"2024-03-30T14:55:55.455711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:55:56.321890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.688654+0000","last_change":"2024-03-30T14:55:54.381596+0000","last_active":"2024-03-30T14:55:55.688654+0000","last_peered":"2024-03-30T14:55:55.688654+0000","last_clean":"2024-03-30T14:55:55.688654+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T14:55:55.688654+0000","last_undegraded":"2024-03-30T14:55:55.688654+0000","last_fullsized":"2024-03-30T14:55:55.688654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:54.381448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:54.381448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.160867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690908+0000","last_change":"2024-03-30T14:54:43.351363+0000","last_active":"2024-03-30T14:55:55.690908+0000","last_peered":"2024-03-30T14:55:55.690908+0000","last_clean":"2024-03-30T14:55:55.690908+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T14:55:55.690908+0000","last_undegraded":"2024-03-30T14:55:55.690908+0000","last_fullsized":"2024-03-30T14:55:55.690908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.981871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528959+0000","last_change":"2024-03-30T14:54:41.354505+0000","last_active":"2024-03-30T14:55:57.528959+0000","last_peered":"2024-03-30T14:55:57.528959+0000","last_clean":"2024-03-30T14:55:57.528959+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T14:55:57.528959+0000","last_undegraded":"2024-03-30T14:55:57.528959+0000","last_fullsized":"2024-03-30T14:55:57.528959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455608+0000","last_change":"2024-03-30T14:54:37.323396+0000","last_active":"2024-03-30T14:55:55.455608+0000","last_peered":"2024-03-30T14:55:55.455608+0000","last_clean":"2024-03-30T14:55:55.455608+0000","last_became_active":"2024-03-30T14:54:37.323084+0000","last_became_peered":"2024-03-30T14:54:37.323084+0000","last_unstale":"2024-03-30T14:55:55.455608+0000","last_undegraded":"2024-03-30T14:55:55.455608+0000","last_fullsized":"2024-03-30T14:55:55.455608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:00.069394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"76'15","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690013+0000","last_change":"2024-03-30T14:54:39.338356+0000","last_active":"2024-03-30T14:55:55.690013+0000","last_peered":"2024-03-30T14:55:55.690013+0000","last_clean":"2024-03-30T14:55:55.690013+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T14:55:55.690013+0000","last_undegraded":"2024-03-30T14:55:55.690013+0000","last_fullsized":"2024-03-30T14:55:55.690013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.454718+0000","last_change":"2024-03-30T14:54:43.360313+0000","last_active":"2024-03-30T14:55:55.454718+0000","last_peered":"2024-03-30T14:55:55.454718+0000","last_clean":"2024-03-30T14:55:55.454718+0000","last_became_active":"2024-03-30T14:54:43.360002+0000","last_became_peered":"2024-03-30T14:54:43.360002+0000","last_unstale":"2024-03-30T14:55:55.454718+0000","last_undegraded":"2024-03-30T14:55:55.454718+0000","last_fullsized":"2024-03-30T14:55:55.454718+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.691111+0000","last_change":"2024-03-30T14:55:50.439135+0000","last_active":"2024-03-30T14:55:55.691111+0000","last_peered":"2024-03-30T14:55:55.691111+0000","last_clean":"2024-03-30T14:55:55.691111+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T14:55:55.691111+0000","last_undegraded":"2024-03-30T14:55:55.691111+0000","last_fullsized":"2024-03-30T14:55:55.691111+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.439008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.333757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690008+0000","last_change":"2024-03-30T14:55:47.298871+0000","last_active":"2024-03-30T14:55:55.690008+0000","last_peered":"2024-03-30T14:55:55.690008+0000","last_clean":"2024-03-30T14:55:55.690008+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T14:55:55.690008+0000","last_undegraded":"2024-03-30T14:55:55.690008+0000","last_fullsized":"2024-03-30T14:55:55.690008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:47.298753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:47.298753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.038171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"76'11","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690699+0000","last_change":"2024-03-30T14:54:39.335270+0000","last_active":"2024-03-30T14:55:55.690699+0000","last_peered":"2024-03-30T14:55:55.690699+0000","last_clean":"2024-03-30T14:55:55.690699+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T14:55:55.690699+0000","last_undegraded":"2024-03-30T14:55:55.690699+0000","last_fullsized":"2024-03-30T14:55:55.690699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689186+0000","last_change":"2024-03-30T14:54:43.353034+0000","last_active":"2024-03-30T14:55:55.689186+0000","last_peered":"2024-03-30T14:55:55.689186+0000","last_clean":"2024-03-30T14:55:55.689186+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T14:55:55.689186+0000","last_undegraded":"2024-03-30T14:55:55.689186+0000","last_fullsized":"2024-03-30T14:55:55.689186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:10.751689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689162+0000","last_change":"2024-03-30T14:55:46.232558+0000","last_active":"2024-03-30T14:55:55.689162+0000","last_peered":"2024-03-30T14:55:55.689162+0000","last_clean":"2024-03-30T14:55:55.689162+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T14:55:55.689162+0000","last_undegraded":"2024-03-30T14:55:55.689162+0000","last_fullsized":"2024-03-30T14:55:55.689162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.232465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.933698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.583019+0000","last_change":"2024-03-30T14:54:37.326067+0000","last_active":"2024-03-30T14:55:55.583019+0000","last_peered":"2024-03-30T14:55:55.583019+0000","last_clean":"2024-03-30T14:55:55.583019+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T14:55:55.583019+0000","last_undegraded":"2024-03-30T14:55:55.583019+0000","last_fullsized":"2024-03-30T14:55:55.583019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:05.706959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690602+0000","last_change":"2024-03-30T14:55:50.322071+0000","last_active":"2024-03-30T14:55:55.690602+0000","last_peered":"2024-03-30T14:55:55.690602+0000","last_clean":"2024-03-30T14:55:55.690602+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T14:55:55.690602+0000","last_undegraded":"2024-03-30T14:55:55.690602+0000","last_fullsized":"2024-03-30T14:55:55.690602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T14:55:50.321974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.321974+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:54.493589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.690810+0000","last_change":"2024-03-30T14:54:43.352969+0000","last_active":"2024-03-30T14:55:55.690810+0000","last_peered":"2024-03-30T14:55:55.690810+0000","last_clean":"2024-03-30T14:55:55.690810+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:55:55.690810+0000","last_undegraded":"2024-03-30T14:55:55.690810+0000","last_fullsized":"2024-03-30T14:55:55.690810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":82,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:57.528460+0000","last_change":"2024-03-30T14:55:40.800056+0000","last_active":"2024-03-30T14:55:57.528460+0000","last_peered":"2024-03-30T14:55:57.528460+0000","last_clean":"2024-03-30T14:55:57.528460+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T14:55:57.528460+0000","last_undegraded":"2024-03-30T14:55:57.528460+0000","last_fullsized":"2024-03-30T14:55:57.528460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:40.799969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.799969+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:57.300469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.455594+0000","last_change":"2024-03-30T14:55:45.783270+0000","last_active":"2024-03-30T14:55:55.455594+0000","last_peered":"2024-03-30T14:55:55.455594+0000","last_clean":"2024-03-30T14:55:55.455594+0000","last_became_active":"2024-03-30T14:54:37.327905+0000","last_became_peered":"2024-03-30T14:54:37.327905+0000","last_unstale":"2024-03-30T14:55:55.455594+0000","last_undegraded":"2024-03-30T14:55:55.455594+0000","last_fullsized":"2024-03-30T14:55:55.455594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:45.783178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:45.783178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.797807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-30T14:55:55.689879+0000","last_change":"2024-03-30T14:55:51.283107+0000","last_active":"2024-03-30T14:55:55.689879+0000","last_peered":"2024-03-30T14:55:55.689879+0000","last_clean":"2024-03-30T14:55:55.689879+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T14:55:55.689879+0000","last_undegraded":"2024-03-30T14:55:55.689879+0000","last_fullsized":"2024-03-30T14:55:55.689879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:51.282990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:51.282990+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.280452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005078,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32584,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712568,"statfs":{"total":95995035648,"available":95961669632,"internally_reserved":0,"allocated":1441792,"data_stored":960977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299419,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32552,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712600,"statfs":{"total":95995035648,"available":95961702400,"internally_reserved":0,"allocated":1409024,"data_stored":958639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":856064,"data_stored":369478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1441792,"data_stored":960321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713172,"statfs":{"total":95995035648,"available":95962288128,"internally_reserved":0,"allocated":819200,"data_stored":368606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":864256,"data_stored":370448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":847872,"data_stored":369274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":368955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:56:05.845 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T14:56:05.845 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T14:56:05.845 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T14:56:05.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T14:56:05.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:05 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T14:56:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:05 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T14:56:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:06 smithi134 ceph-mon[33714]: from='client.34631 v1:172.21.15.28:0/2134160545' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:06 smithi134 ceph-mon[33714]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-30T14:56:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:06 smithi134 ceph-mon[33714]: 4.19 deep-scrub starts 2024-03-30T14:56:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:06 smithi134 ceph-mon[33714]: 4.19 deep-scrub ok 2024-03-30T14:56:06.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[41614]: from='client.34631 v1:172.21.15.28:0/2134160545' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:06.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[41614]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[41614]: 4.19 deep-scrub starts 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[41614]: 4.19 deep-scrub ok 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[29876]: from='client.34631 v1:172.21.15.28:0/2134160545' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[29876]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[29876]: 4.19 deep-scrub starts 2024-03-30T14:56:06.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:06 smithi028 ceph-mon[29876]: 4.19 deep-scrub ok 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 5.8 deep-scrub starts 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 5.8 deep-scrub ok 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T14:56:07.765 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T14:56:07.766 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:07 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 5.8 deep-scrub starts 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 5.8 deep-scrub ok 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T14:56:07.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 5.8 deep-scrub starts 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 5.8 deep-scrub ok 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T14:56:07.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T14:56:07.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:07 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T14:56:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- 172.21.15.28:0/65257440 >> v1:172.21.15.28:6789/0 conn(0x7f242c0745e0 legacy=0x7f242c0749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- 172.21.15.28:0/65257440 shutdown_connections 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- 172.21.15.28:0/65257440 >> 172.21.15.28:0/65257440 conn(0x7f242c0fdaf0 msgr2=0x7f242c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- 172.21.15.28:0/65257440 shutdown_connections 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- 172.21.15.28:0/65257440 wait complete. 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 librados: starting msgr at 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 librados: starting objecter 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 Processor -- start 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 -- start start 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 librados: setting wanted keys 2024-03-30T14:56:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.880+0000 7f24322f8700 1 librados: calling monclient init 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24322f8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f242c114e40 con 0x7f242c1151e0 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24322f8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f242c1aec50 con 0x7f242c0750d0 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24322f8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f242c1afe30 con 0x7f242c0745e0 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f242bfff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f242c0745e0 0x7f242c111ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50954/0 (socket says 172.21.15.28:50954) 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f242bfff700 1 -- 172.21.15.28:0/3002930625 learned_addr learned my addr 172.21.15.28:0/3002930625 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter ms_handle_connect 0x7f242c0745e0 2024-03-30T14:56:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter ms_handle_connect 0x7f242c1151e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 777014250 0 0) 0x7f242c1afe30 con 0x7f242c0745e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24140031f0 con 0x7f242c0745e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245749355 0 0) 0x7f242c114e40 con 0x7f242c1151e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f242c1afe30 con 0x7f242c1151e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter ms_handle_connect 0x7f242c0750d0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193226306 0 0) 0x7f242c1aec50 con 0x7f242c0750d0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.881+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f242c114e40 con 0x7f242c0750d0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3757386648 0 0) 0x7f24140031f0 con 0x7f242c0745e0 2024-03-30T14:56:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f242c1aec50 con 0x7f242c0745e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f241c0015d0 con 0x7f242c0745e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 715726919 0 0) 0x7f242c1afe30 con 0x7f242c1151e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24140031f0 con 0x7f242c1151e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2424001bb0 con 0x7f242c1151e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 262759700 0 0) 0x7f242c114e40 con 0x7f242c0750d0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f242c1afe30 con 0x7f242c0750d0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2420001870 con 0x7f242c0750d0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 174176185 0 0) 0x7f242c1aec50 con 0x7f242c0745e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 >> v1:172.21.15.134:6789/0 conn(0x7f242c0750d0 legacy=0x7f242c1123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 >> v1:172.21.15.28:6789/0 conn(0x7f242c1151e0 legacy=0x7f242c1ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c1b1010 con 0x7f242c0745e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f242c1adca0 con 0x7f242c0745e0 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24322f8700 10 client.34637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f242c1ae160 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24322f8700 1 librados: init done 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f241c003680 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.882+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f241c004de0 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.883+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2418005120 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f241c001b90 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(93..93 src has 43..93) ==== 6259+0+0 (unknown 1478577302 0 0) 0x7f241c088dd0 con 0x7f242c0745e0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 10 client.34637.objecter ms_dispatch 0x7f242c001550 osd_map(93..93 src has 43..93) 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 3 client.34637.objecter handle_osd_map got epochs [93,93] > 0 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 3 client.34637.objecter handle_osd_map decoding full epoch 93 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 20 client.34637.objecter dump_active .. 0 homeless 2024-03-30T14:56:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.884+0000 7f24297fa700 10 client.34637.objecter ms_handle_connect 0x7f2414070480 2024-03-30T14:56:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:07.889+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f241c056870 con 0x7f242c0745e0 2024-03-30T14:56:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.137+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2418005410 con 0x7f242c0745e0 2024-03-30T14:56:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.137+0000 7f24297fa700 1 -- 172.21.15.28:0/3002930625 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f241c05a2c0 con 0x7f242c0745e0 2024-03-30T14:56:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:56:08.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 10 librados: watch_flush enter 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 10 librados: watch_flush exit 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 20 client.34637.objecter shutdown clearing up homeless session... 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 10 client.34637.objecter successfully canceled tick 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2414070480 legacy=0x7f2414072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 >> v1:172.21.15.28:6790/0 conn(0x7f242c0745e0 legacy=0x7f242c111ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 shutdown_connections 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 >> 172.21.15.28:0/3002930625 conn(0x7f242c0fdaf0 msgr2=0x7f242c0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 shutdown_connections 2024-03-30T14:56:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.139+0000 7f24322f8700 1 -- 172.21.15.28:0/3002930625 wait complete. 2024-03-30T14:56:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:08.140+0000 7f24322f8700 1 librados: shutdown 2024-03-30T14:56:08.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:56:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:56:08.830 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T14:56:08.830 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T14:56:08.830 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:08.830 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T14:56:08.830 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T14:56:08.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T14:56:08.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T14:56:08.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3002930625' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3002930625' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T14:56:08.892 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T14:56:08.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3002930625' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T14:56:08.922 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T14:56:08.922 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T14:56:08.922 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-30T14:56:08.934 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-30T14:56:08.934 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e50bbcdd1630c7760b708f3a5b4d38a121e66cb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': '5af0400050a123dc31f105425174f106e057276a'}} 2024-03-30T14:56:08.934 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-30T14:56:08.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-30T14:56:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:09.472 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:09 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:56:09.203Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003281545s 2024-03-30T14:56:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:09 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T14:56:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:09 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T14:56:09.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:09 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T14:56:09.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:09 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T14:56:09.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:09 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T14:56:09.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:09 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T14:56:10.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:56:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:10.355 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:10 smithi134 systemd[1]: Starting Ceph grafana.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:10.674 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:10 smithi134 podman[66520]: 2024-03-30 14:56:10.257801333 +0000 UTC m=+0.018674110 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 4.e deep-scrub starts 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 4.e deep-scrub ok 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T14:56:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:10.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T14:56:10.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:10 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T14:56:10.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 4.e deep-scrub starts 2024-03-30T14:56:10.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 4.e deep-scrub ok 2024-03-30T14:56:10.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T14:56:10.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 4.e deep-scrub starts 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 4.e deep-scrub ok 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:10.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T14:56:10.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T14:56:10.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:10.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T14:56:10.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:10 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2336731239 >> v1:172.21.15.28:6790/0 conn(0x7fa324109dd0 legacy=0x7fa32410c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2336731239 shutdown_connections 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2336731239 >> 172.21.15.28:0/2336731239 conn(0x7fa3240fc9f0 msgr2=0x7fa3240fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2336731239 shutdown_connections 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2336731239 wait complete. 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 librados: starting msgr at 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.964+0000 7fa32b1e0700 1 librados: starting objecter 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 Processor -- start 2024-03-30T14:56:10.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 -- start start 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 librados: setting wanted keys 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 librados: calling monclient init 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa3241a7540 con 0x7fa324101140 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa3241a8720 con 0x7fa324109dd0 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa32b1e0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa3241a9900 con 0x7fa32410d970 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa328f7c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa324101140 0x7fa3241048b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54704/0 (socket says 172.21.15.28:54704) 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa328f7c700 1 -- 172.21.15.28:0/2516294588 learned_addr learned my addr 172.21.15.28:0/2516294588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter ms_handle_connect 0x7fa32410d970 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter ms_handle_connect 0x7fa324101140 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330078542 0 0) 0x7fa3241a9900 con 0x7fa32410d970 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa30c0031f0 con 0x7fa32410d970 2024-03-30T14:56:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter ms_handle_connect 0x7fa324109dd0 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965842242 0 0) 0x7fa3241a7540 con 0x7fa324101140 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3241a9900 con 0x7fa324101140 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3947034619 0 0) 0x7fa3241a8720 con 0x7fa324109dd0 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3241a7540 con 0x7fa324109dd0 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2853721382 0 0) 0x7fa30c0031f0 con 0x7fa32410d970 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3241a8720 con 0x7fa32410d970 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa3140017f0 con 0x7fa32410d970 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.965+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2456720507 0 0) 0x7fa3241a9900 con 0x7fa324101140 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa30c0031f0 con 0x7fa324101140 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa31c001c10 con 0x7fa324101140 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 78374187 0 0) 0x7fa3241a8720 con 0x7fa32410d970 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 >> v1:172.21.15.134:6789/0 conn(0x7fa324109dd0 legacy=0x7fa324104fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 >> v1:172.21.15.28:6789/0 conn(0x7fa324101140 legacy=0x7fa3241048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3241aaae0 con 0x7fa32410d970 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3241a7770 con 0x7fa32410d970 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa314003b20 con 0x7fa32410d970 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa314004a70 con 0x7fa32410d970 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa32b1e0700 10 client.34643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3241a7c30 con 0x7fa32410d970 2024-03-30T14:56:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.966+0000 7fa32b1e0700 1 librados: init done 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.967+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa324102620 con 0x7fa32410d970 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.967+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa314003680 con 0x7fa32410d970 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(93..93 src has 43..93) ==== 6259+0+0 (unknown 1478577302 0 0) 0x7fa314087c20 con 0x7fa32410d970 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 10 client.34643.objecter ms_dispatch 0x7fa324001550 osd_map(93..93 src has 43..93) 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 3 client.34643.objecter handle_osd_map got epochs [93,93] > 0 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 3 client.34643.objecter handle_osd_map decoding full epoch 93 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 20 client.34643.objecter dump_active .. 0 homeless 2024-03-30T14:56:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.968+0000 7fa321ffb700 10 client.34643.objecter ms_handle_connect 0x7fa30c070110 2024-03-30T14:56:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:10.972+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa3140556c0 con 0x7fa32410d970 2024-03-30T14:56:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.175+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 --> v1:172.21.15.28:6790/0 -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7fa3241137b0 con 0x7fa32410d970 2024-03-30T14:56:11.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.179+0000 7fa321ffb700 1 -- 172.21.15.28:0/2516294588 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v30)=0 v30) ==== 136+0+0 (unknown 4113108377 0 0) 0x7fa314059110 con 0x7fa32410d970 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.181+0000 7fa32b1e0700 10 librados: watch_flush enter 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 10 librados: watch_flush exit 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 20 client.34643.objecter shutdown clearing up homeless session... 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 10 client.34643.objecter successfully canceled tick 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa30c070110 legacy=0x7fa30c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 >> v1:172.21.15.28:6790/0 conn(0x7fa32410d970 legacy=0x7fa3241a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 shutdown_connections 2024-03-30T14:56:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 >> 172.21.15.28:0/2516294588 conn(0x7fa3240fc9f0 msgr2=0x7fa32406a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 shutdown_connections 2024-03-30T14:56:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 -- 172.21.15.28:0/2516294588 wait complete. 2024-03-30T14:56:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:11.182+0000 7fa32b1e0700 1 librados: shutdown 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE Bus STOPPING 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE Bus STOPPED 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE Bus STARTING 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE Serving on http://:::9283 2024-03-30T14:56:11.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:11] ENGINE Bus STARTED 2024-03-30T14:56:11.642 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 podman[66520]: 2024-03-30 14:56:11.314555869 +0000 UTC m=+1.075428664 container create 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:56:11.642 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 podman[66520]: 2024-03-30 14:56:11.533425297 +0000 UTC m=+1.294298064 container init 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:56:11.642 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 podman[66520]: 2024-03-30 14:56:11.539440424 +0000 UTC m=+1.300313180 container start 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T14:56:11.894 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-mon[33714]: 4.0 deep-scrub starts 2024-03-30T14:56:11.894 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-mon[33714]: 4.0 deep-scrub ok 2024-03-30T14:56:11.894 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-mon[33714]: from='client.34643 ' entity='client.admin' 2024-03-30T14:56:11.894 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:56:11.894 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:56:11.895 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 bash[66520]: 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb 2024-03-30T14:56:11.895 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 systemd[1]: Started Ceph grafana.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:11.895 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696380074Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-30T14:56:11.895 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.69664286Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-30T14:56:11.895 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696658922Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.69666478Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.69666975Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.6966747Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.69667954Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696684302Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.69668919Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696694062Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696699365Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696703349Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696707118Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696711003Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696715044Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-30T14:56:11.896 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696719228Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696723593Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696727609Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=settings t=2024-03-30T14:56:11.696731441Z level=info msg="App mode production" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore t=2024-03-30T14:56:11.696787703Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore t=2024-03-30T14:56:11.696803616Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.697879067Z level=info msg="Starting DB migrations" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.697949271Z level=info msg="Executing migration" id="create migration_log table" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.698542213Z level=info msg="Executing migration" id="create user table" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.698973693Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.699452063Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.69986104Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.700520645Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.700921474Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-30T14:56:11.897 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.701669247Z level=info msg="Executing migration" id="create user table v2" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.702096541Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.702507253Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.702945024Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.70329271Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.703642384Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.704086203Z level=info msg="Executing migration" id="Update user table charset" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.704382465Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.704768167Z level=info msg="Executing migration" id="Add missing user data" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.705114472Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.705528344Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.705883175Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.706304278Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.707742762Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-30T14:56:11.898 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.708180699Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.708628923Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.709030906Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.709450098Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.709817422Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.710112511Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.710516108Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.710879045Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.711267294Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.711641186Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.712500049Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.712877496Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-30T14:56:11.899 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.713296244Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-30T14:56:11.900 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.713679192Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-30T14:56:11.900 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.714029634Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.714443823Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.714732567Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.71502658Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.715345492Z level=info msg="Executing migration" id="create star table" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.715670267Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.716071423Z level=info msg="Executing migration" id="create org table v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.716462468Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.716841831Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.717206006Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.717575818Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.717958393Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.718328982Z level=info msg="Executing migration" id="Update org table charset" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.719182232Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-30T14:56:11.901 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.719668892Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.719966867Z level=info msg="Executing migration" id="create dashboard table" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.72034626Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.720781361Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.721762664Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.72249613Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.723455779Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.724190766Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.726999196Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-30T14:56:11.902 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.729738463Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.732302902Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.735055824Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.737843835Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.740588591Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.743605892Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.746372393Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.748933013Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.751759365Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.754339937Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.757059501Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.759670355Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.762451158Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.764913643Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.767636613Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-30T14:56:11.903 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.768196379Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.770232397Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.772992167Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.775708856Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.778312507Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.7807477Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.783498395Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.786281165Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.788707937Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.791302072Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.793661292Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.796297666Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.798930129Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.80162396Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-30T14:56:11.904 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.804186199Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.806729874Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.809413923Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.81190454Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.81445701Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.817080858Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.818015651Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.818690165Z level=info msg="Executing migration" id="create data_source table" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.819525728Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.820040972Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.820488214Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.821306066Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.821708074Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.822896728Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.82382684Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-30T14:56:11.905 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.824488927Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.825407255Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.826076916Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.826884295Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.827679631Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.828431324Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.828750193Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.829328083Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.830116117Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.830923129Z level=info msg="Executing migration" id="Add uid column" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.831857393Z level=info msg="Executing migration" id="Update uid value" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.832610071Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.833411175Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.833838743Z level=info msg="Executing migration" id="create api_key table" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.834241869Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-30T14:56:11.906 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.835142599Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.836103284Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.836966111Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.837697385Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.838583905Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.841382497Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.842687728Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.845517037Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.84838574Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.851156802Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.851700023Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.854602516Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.855013886Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.857404012Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-30T14:56:11.907 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.860301721Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.862924873Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.865857014Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.868545134Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.871117577Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.871491026Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.874287658Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.874751263Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.877301167Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.879746324Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.880211683Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.882836023Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.885603937Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.888022116Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-30T14:56:11.908 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.888637689Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-30T14:56:11.909 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.89128519Z level=info msg="Executing migration" id="create quota table v1" 2024-03-30T14:56:11.909 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.891689682Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-30T14:56:11.909 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.894082206Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-30T14:56:11.909 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.896599303Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-30T14:56:11.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[29876]: 4.0 deep-scrub starts 2024-03-30T14:56:11.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[29876]: 4.0 deep-scrub ok 2024-03-30T14:56:11.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[29876]: from='client.34643 ' entity='client.admin' 2024-03-30T14:56:11.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:56:11.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:56:11.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[41614]: 4.0 deep-scrub starts 2024-03-30T14:56:11.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[41614]: 4.0 deep-scrub ok 2024-03-30T14:56:11.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[41614]: from='client.34643 ' entity='client.admin' 2024-03-30T14:56:11.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:56:11.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:11 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:56:11.993 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-30T14:56:11.993 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T14:56:11.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.899263552Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.901956522Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.904443274Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.907141648Z level=info msg="Executing migration" id="create session table" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.907559973Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.910211139Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-30T14:56:12.146 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.910507526Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.912793649Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.91358411Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.913861303Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.914644567Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.915485158Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.916338898Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.916745928Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.917547493Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.918229484Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.918556765Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.919425233Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-30T14:56:12.147 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.920251155Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.921161206Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.921911275Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.922882238Z level=info msg="Executing migration" id="create alert table v1" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.923491257Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.924348826Z level=info msg="Executing migration" id="add index alert state" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.927967883Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.928861799Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.929411116Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.930317828Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.931243009Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.933041834Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-30T14:56:12.149 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.9334322Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.934314747Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.93522725Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.936003053Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.936407046Z level=info msg="Executing migration" id="Add column is_default" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.937276414Z level=info msg="Executing migration" id="Add column frequency" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.938088425Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.938866711Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.939602187Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.940452905Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.94143201Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.942114019Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.942505842Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.945349027Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-30T14:56:12.150 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.948129766Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.950693611Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.951159642Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.954010177Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.957039445Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.959797465Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.960274682Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.963007811Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.965717372Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.96854294Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.969118962Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.971908507Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.974739624Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.977559523Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-30T14:56:12.151 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.978177488Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.980946968Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.98348838Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.986082955Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.986545641Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.989235707Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.990099725Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.991904734Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.994683082Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.995127545Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:11.997863589Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.000510082Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.003095923Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.005798456Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-30T14:56:12.152 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.00851851Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.011200146Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.013560289Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.013965201Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.016440656Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.017219884Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.01808092Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.018550375Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.019349252Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.020170777Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.020638699Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.021501864Z level=info msg="Executing migration" id="Move region to single row" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.022112587Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.02257666Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-30T14:56:12.153 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.023337413Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.023804797Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.024227859Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.025208878Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.025818316Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.026646618Z level=info msg="Executing migration" id="create test_data table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.027531536Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.028331006Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.028811268Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.029641369Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.030529718Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.03137438Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.031772001Z level=info msg="Executing migration" id="create team table" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.032433265Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-30T14:56:12.154 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.033255175Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.034229982Z level=info msg="Executing migration" id="create team member table" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.034749711Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.035650375Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.036510176Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.037310948Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.038148757Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.038995287Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.039818576Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.040668022Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.041115617Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.041851086Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.042287213Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.043058879Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-30T14:56:12.155 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.043462198Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.043866458Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.044279138Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.04466469Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.044950531Z level=info msg="Executing migration" id="create tag table" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.045340694Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.045764115Z level=info msg="Executing migration" id="create login attempt table" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.046160256Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.046580787Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.04700008Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.049321786Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.049713656Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.050150143Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.050469702Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.050805052Z level=info msg="Executing migration" id="create user auth table" 2024-03-30T14:56:12.156 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.051222152Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.051643683Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.051907625Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.05278257Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.053644021Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.054533194Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.055436476Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.055892909Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.056787263Z level=info msg="Executing migration" id="create server_lock table" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.057208577Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.057640389Z level=info msg="Executing migration" id="create user auth token table" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.058065786Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.058496425Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.058919441Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-30T14:56:12.157 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.059353868Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.060319145Z level=info msg="Executing migration" id="create cache_data table" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.060731287Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.061177989Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.061601158Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.062331825Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.062644098Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.062919298Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-30T14:56:12.158 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.063389502Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.064188982Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.064623849Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.064890314Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.065340677Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.065742207Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.066224391Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.066640377Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.067508692Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.067927625Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.068227811Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-30T14:56:12.159 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.068588561Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.068980896Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.069387696Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.069634068Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.070071069Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.070476716Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.070887261Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.071309021Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.072176673Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.072576184Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.072951121Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.077355316Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.081729905Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.082163855Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.082572413Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-30T14:56:12.160 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.083531573Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.083950452Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.086894119Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.089412527Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.092177469Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.094665645Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.097564502Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.100263645Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.103011514Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.105361126Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.107934017Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.110679203Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.113385481Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.116228122Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-30T14:56:12.161 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.118772835Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.121485472Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.124170179Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.12698802Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.129460211Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.132203939Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.132548928Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.135127715Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.13780632Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.140400785Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.14298446Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.145626299Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-30T14:56:12.162 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.14821669Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-30T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:12.396 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.150712795Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-30T14:56:12.396 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.153299741Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-30T14:56:12.396 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.155959667Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.158703184Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.161200647Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.163892283Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.166255071Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.169071568Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.171608913Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.174249021Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.17673319Z level=info msg="Executing migration" id="create alert_image table" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.177497773Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.178255896Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.179175439Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-30T14:56:12.397 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.179881985Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.180339844Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.180423932Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.180697458Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.181483973Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.181919514Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.182054777Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.182172004Z level=info msg="alerts found to migrate" alerts=0 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.182558833Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.184098275Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.184721294Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.185516841Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.186374523Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-30T14:56:12.398 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.187111517Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.187914297Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.188768534Z level=info msg="Executing migration" id="create data_keys table" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.189597809Z level=info msg="Executing migration" id="create secrets table" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.190552766Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.196143343Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.197215498Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.198042068Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-30T14:56:12.399 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.2036535Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.20913716Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.210107914Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.210859611Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.211820475Z level=info msg="Executing migration" id="create permission table" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.212584772Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-30T14:56:12.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.213602908Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.214336679Z level=info msg="Executing migration" id="create role table" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.215144332Z level=info msg="Executing migration" id="add column display_name" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.216217895Z level=info msg="Executing migration" id="add column group_name" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.217275874Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.218131265Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.219161639Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.219861077Z level=info msg="Executing migration" id="create team role table" 2024-03-30T14:56:12.401 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.220865727Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.221608977Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.224444713Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.22726538Z level=info msg="Executing migration" id="create user role table" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.229847326Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.232854168Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.235445594Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.238315114Z level=info msg="Executing migration" id="create builtin role table" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.241217123Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.244053032Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.246661851Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.249462591Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.252132141Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-30T14:56:12.402 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.255038234Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.257709648Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.260394543Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.263232419Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.266034086Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.268849914Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.271493838Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.274019495Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.276734854Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.279383394Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.281923869Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.284573169Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.287328858Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-30T14:56:12.403 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.290138005Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.292790006Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.295480778Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.298023458Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.300704889Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.303393085Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.305958652Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.308526694Z level=info msg="Executing migration" id="add correlation config column" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.311247697Z level=info msg="Executing migration" id="create entity_events table" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.31407364Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.316611492Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.316704884Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.317545775Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T14:56:12.404 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.317786363Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.31824229Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.319098574Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.320058942Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.320827584Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.321603436Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.322440744Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.323300428Z level=info msg="Executing migration" id="Drop public config table" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.324218115Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-30T14:56:12.405 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.324933569Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.32591848Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.326585253Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.32755766Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.331348286Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.332539182Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.333725427Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.334593516Z level=info msg="Executing migration" id="add share column" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.335826767Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.336819005Z level=info msg="Executing migration" id="create file table" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.3375367Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.33848661Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.339279038Z level=info msg="Executing migration" id="create file_meta table" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.340224511Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-30T14:56:12.406 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.340988097Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.34183638Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.342694635Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.343613845Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.344100229Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.345320089Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.346173412Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.34710996Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.347839947Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.348837111Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.349602079Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.350919559Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-30T14:56:12.407 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.353631996Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.356483084Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.369377601Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.372093017Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.374670286Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.379003258Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.381960265Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.384584584Z level=info msg="Executing migration" id="create folder table" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.387268406Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.39020835Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.392834112Z level=info msg="Executing migration" id="Update folder title length" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.395828532Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=migrator t=2024-03-30T14:56:12.398452179Z level=info msg="migrations completed" performed=485 skipped=0 duration=700.529441ms 2024-03-30T14:56:12.408 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore t=2024-03-30T14:56:12.398888287Z level=info msg="Created default organization" 2024-03-30T14:56:12.680 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=plugin.finder t=2024-03-30T14:56:12.434146353Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-30T14:56:12.680 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=secrets t=2024-03-30T14:56:12.434235279Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-30T14:56:12.680 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=query_data t=2024-03-30T14:56:12.436924481Z level=info msg="Query Service initialization" 2024-03-30T14:56:12.680 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=live.push_http t=2024-03-30T14:56:12.440913111Z level=info msg="Live Push Gateway initialization" 2024-03-30T14:56:12.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: Deploying daemon node-exporter.a on smithi028 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 4.1f deep-scrub starts 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 4.1f deep-scrub ok 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T14:56:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: Deploying daemon node-exporter.a on smithi028 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 4.1f deep-scrub starts 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 4.1f deep-scrub ok 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T14:56:12.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:12 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: Deploying daemon node-exporter.a on smithi028 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 4.1f deep-scrub starts 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 4.1f deep-scrub ok 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: osdmap e94: 8 total, 8 up, 8 in 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T14:56:13.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:12 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=infra.usagestats.collector t=2024-03-30T14:56:13.044611803Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-30T14:56:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=provisioning.datasources t=2024-03-30T14:56:13.045744448Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-30T14:56:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=provisioning.datasources t=2024-03-30T14:56:13.056659507Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-30T14:56:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=provisioning.alerting t=2024-03-30T14:56:13.067554767Z level=info msg="starting to provision alerting" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=provisioning.alerting t=2024-03-30T14:56:13.067564135Z level=info msg="finished to provision alerting" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=ngalert.state.manager t=2024-03-30T14:56:13.067745772Z level=info msg="Warming state cache for startup" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=ngalert.state.manager t=2024-03-30T14:56:13.067895449Z level=info msg="State cache has been initialized" states=0 duration=149.723µs 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=ticker t=2024-03-30T14:56:13.067959246Z level=info msg=starting first_tick=2024-03-30T14:56:20Z 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=ngalert.multiorg.alertmanager t=2024-03-30T14:56:13.068112132Z level=info msg="starting MultiOrg Alertmanager" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=grafanaStorageLogger t=2024-03-30T14:56:13.068206222Z level=info msg="storage starting" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=http.server t=2024-03-30T14:56:13.073172575Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore.transactions t=2024-03-30T14:56:13.083660493Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore.transactions t=2024-03-30T14:56:13.094099359Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-30T14:56:13.431 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore.transactions t=2024-03-30T14:56:13.105154739Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T14:56:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T14:56:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T14:56:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T14:56:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T14:56:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T14:56:13.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T14:56:13.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T14:56:13.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:13 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T14:56:14.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.108+0000 7ffabb1c5700 1 -- 172.21.15.28:0/1901574900 >> v1:172.21.15.28:6789/0 conn(0x7ffab4109dd0 legacy=0x7ffab410c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:14.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.108+0000 7ffabb1c5700 1 -- 172.21.15.28:0/1901574900 shutdown_connections 2024-03-30T14:56:14.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.108+0000 7ffabb1c5700 1 -- 172.21.15.28:0/1901574900 >> 172.21.15.28:0/1901574900 conn(0x7ffab40fc9f0 msgr2=0x7ffab40fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.108+0000 7ffabb1c5700 1 -- 172.21.15.28:0/1901574900 shutdown_connections 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.108+0000 7ffabb1c5700 1 -- 172.21.15.28:0/1901574900 wait complete. 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 librados: starting msgr at 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 librados: starting objecter 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 Processor -- start 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 -- start start 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 librados: setting wanted keys 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 librados: calling monclient init 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffab41a9310 con 0x7ffab4101140 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffab41aa530 con 0x7ffab4109dd0 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffabb1c5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffab41ab770 con 0x7ffab410d970 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab8f61700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffab4101140 0x7ffab41063b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54710/0 (socket says 172.21.15.28:54710) 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab8f61700 1 -- 172.21.15.28:0/3084720398 learned_addr learned my addr 172.21.15.28:0/3084720398 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:14.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter ms_handle_connect 0x7ffab4101140 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter ms_handle_connect 0x7ffab410d970 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter ms_handle_connect 0x7ffab4109dd0 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516464594 0 0) 0x7ffab41a9310 con 0x7ffab4101140 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffaa40039c0 con 0x7ffab4101140 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109763514 0 0) 0x7ffab41ab770 con 0x7ffab410d970 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.109+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffab41a9310 con 0x7ffab410d970 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700532627 0 0) 0x7ffab41aa530 con 0x7ffab4109dd0 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffab41ab770 con 0x7ffab4109dd0 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 851368252 0 0) 0x7ffaa40039c0 con 0x7ffab4101140 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffab41aa530 con 0x7ffab4101140 2024-03-30T14:56:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffaa8001930 con 0x7ffab4101140 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3051818198 0 0) 0x7ffab41a9310 con 0x7ffab410d970 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffaa40039c0 con 0x7ffab410d970 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffaac001470 con 0x7ffab410d970 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1942133645 0 0) 0x7ffab41ab770 con 0x7ffab4109dd0 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffab41a9310 con 0x7ffab4109dd0 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffaa0001a00 con 0x7ffab4109dd0 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3333127884 0 0) 0x7ffab41aa530 con 0x7ffab4101140 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 >> v1:172.21.15.28:6790/0 conn(0x7ffab410d970 legacy=0x7ffab41a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 >> v1:172.21.15.134:6789/0 conn(0x7ffab4109dd0 legacy=0x7ffab41a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab41ac9b0 con 0x7ffab4101140 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffab41ab9a0 con 0x7ffab4101140 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ffaa8003b40 con 0x7ffab4101140 2024-03-30T14:56:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffaa8005d90 con 0x7ffab4101140 2024-03-30T14:56:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffabb1c5700 10 client.14760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffab41abe60 con 0x7ffab4101140 2024-03-30T14:56:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.110+0000 7ffabb1c5700 1 librados: init done 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.111+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab4102620 con 0x7ffab4101140 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.111+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ffaa801c670 con 0x7ffab4101140 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(94..94 src has 43..94) ==== 6259+0+0 (unknown 1180456038 0 0) 0x7ffaa8089100 con 0x7ffab4101140 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 10 client.14760.objecter ms_dispatch 0x7ffab4001550 osd_map(94..94 src has 43..94) 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 3 client.14760.objecter handle_osd_map got epochs [94,94] > 0 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 3 client.14760.objecter handle_osd_map decoding full epoch 94 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 20 client.14760.objecter dump_active .. 0 homeless 2024-03-30T14:56:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.112+0000 7ffab1ffb700 10 client.14760.objecter ms_handle_connect 0x7ffaa4070530 2024-03-30T14:56:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.117+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ffaa8056ac0 con 0x7ffab4101140 2024-03-30T14:56:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T14:56:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T14:56:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T14:56:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:13 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T14:56:14.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.329+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffab4103980 con 0x7ffaa4070530 2024-03-30T14:56:14.330 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:14 smithi028 bash[80459]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-30T14:56:14.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.338+0000 7ffab1ffb700 1 -- 172.21.15.28:0/3084720398 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+344220 (unknown 2965378022 0 2528724892) 0x7ffab4103980 con 0x7ffaa4070530 2024-03-30T14:56:14.343 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.341+0000 7ffabb1c5700 10 librados: watch_flush enter 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.341+0000 7ffabb1c5700 10 librados: watch_flush exit 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.341+0000 7ffabb1c5700 20 client.14760.objecter shutdown clearing up homeless session... 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.341+0000 7ffabb1c5700 10 client.14760.objecter successfully canceled tick 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 >> v1:172.21.15.28:6800/3011431093 conn(0x7ffaa4070530 legacy=0x7ffaa40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 >> v1:172.21.15.28:6789/0 conn(0x7ffab4101140 legacy=0x7ffab41063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 shutdown_connections 2024-03-30T14:56:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 >> 172.21.15.28:0/3084720398 conn(0x7ffab40fc9f0 msgr2=0x7ffab4110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 shutdown_connections 2024-03-30T14:56:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 -- 172.21.15.28:0/3084720398 wait complete. 2024-03-30T14:56:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:14.342+0000 7ffabb1c5700 1 librados: shutdown 2024-03-30T14:56:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T14:56:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[29876]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T14:56:14.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T14:56:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[41614]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T14:56:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:14 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T14:56:14.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:14 smithi028 bash[80459]: Getting image source signatures 2024-03-30T14:56:14.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:14 smithi028 bash[80459]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-30T14:56:14.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:14 smithi028 bash[80459]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T14:56:14.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:14 smithi028 bash[80459]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-30T14:56:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:14 smithi134 ceph-mon[33714]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:14 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T14:56:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:14 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T14:56:15.353 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-03-30T14:56:13.588429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":627,"ondisk_log_size":627,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249532,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711684,"statfs":{"total":767960285184,"available":767704764416,"internally_reserved":0,"allocated":7630848,"data_stored":4433298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002126"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185349+0000","last_change":"2024-03-30T14:54:43.351532+0000","last_active":"2024-03-30T14:56:12.185349+0000","last_peered":"2024-03-30T14:56:12.185349+0000","last_clean":"2024-03-30T14:56:12.185349+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T14:56:12.185349+0000","last_undegraded":"2024-03-30T14:56:12.185349+0000","last_fullsized":"2024-03-30T14:56:12.185349+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541194+0000","last_change":"2024-03-30T14:54:41.350461+0000","last_active":"2024-03-30T14:56:12.541194+0000","last_peered":"2024-03-30T14:56:12.541194+0000","last_clean":"2024-03-30T14:56:12.541194+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T14:56:12.541194+0000","last_undegraded":"2024-03-30T14:56:12.541194+0000","last_fullsized":"2024-03-30T14:56:12.541194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186510+0000","last_change":"2024-03-30T14:55:46.590443+0000","last_active":"2024-03-30T14:56:12.186510+0000","last_peered":"2024-03-30T14:56:12.186510+0000","last_clean":"2024-03-30T14:56:12.186510+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T14:56:12.186510+0000","last_undegraded":"2024-03-30T14:56:12.186510+0000","last_fullsized":"2024-03-30T14:56:12.186510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.590360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.590360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.726057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185232+0000","last_change":"2024-03-30T14:55:40.318444+0000","last_active":"2024-03-30T14:56:12.185232+0000","last_peered":"2024-03-30T14:56:12.185232+0000","last_clean":"2024-03-30T14:56:12.185232+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T14:56:12.185232+0000","last_undegraded":"2024-03-30T14:56:12.185232+0000","last_fullsized":"2024-03-30T14:56:12.185232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:40.318295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.318295+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:45.878339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368092+0000","last_change":"2024-03-30T14:54:43.359234+0000","last_active":"2024-03-30T14:56:12.368092+0000","last_peered":"2024-03-30T14:56:12.368092+0000","last_clean":"2024-03-30T14:56:12.368092+0000","last_became_active":"2024-03-30T14:54:43.358170+0000","last_became_peered":"2024-03-30T14:54:43.358170+0000","last_unstale":"2024-03-30T14:56:12.368092+0000","last_undegraded":"2024-03-30T14:56:12.368092+0000","last_fullsized":"2024-03-30T14:56:12.368092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185232+0000","last_change":"2024-03-30T14:54:41.350763+0000","last_active":"2024-03-30T14:56:12.185232+0000","last_peered":"2024-03-30T14:56:12.185232+0000","last_clean":"2024-03-30T14:56:12.185232+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T14:56:12.185232+0000","last_undegraded":"2024-03-30T14:56:12.185232+0000","last_fullsized":"2024-03-30T14:56:12.185232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":69,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368597+0000","last_change":"2024-03-30T14:55:41.765389+0000","last_active":"2024-03-30T14:56:12.368597+0000","last_peered":"2024-03-30T14:56:12.368597+0000","last_clean":"2024-03-30T14:56:12.368597+0000","last_became_active":"2024-03-30T14:54:37.474114+0000","last_became_peered":"2024-03-30T14:54:37.474114+0000","last_unstale":"2024-03-30T14:56:12.368597+0000","last_undegraded":"2024-03-30T14:56:12.368597+0000","last_fullsized":"2024-03-30T14:56:12.368597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.765306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.765306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:43.228077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368115+0000","last_change":"2024-03-30T14:56:05.629550+0000","last_active":"2024-03-30T14:56:12.368115+0000","last_peered":"2024-03-30T14:56:12.368115+0000","last_clean":"2024-03-30T14:56:12.368115+0000","last_became_active":"2024-03-30T14:54:39.333512+0000","last_became_peered":"2024-03-30T14:54:39.333512+0000","last_unstale":"2024-03-30T14:56:12.368115+0000","last_undegraded":"2024-03-30T14:56:12.368115+0000","last_fullsized":"2024-03-30T14:56:12.368115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:56:05.629449+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:27.238183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186676+0000","last_change":"2024-03-30T14:55:57.530576+0000","last_active":"2024-03-30T14:56:12.186676+0000","last_peered":"2024-03-30T14:56:12.186676+0000","last_clean":"2024-03-30T14:56:12.186676+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T14:56:12.186676+0000","last_undegraded":"2024-03-30T14:56:12.186676+0000","last_fullsized":"2024-03-30T14:56:12.186676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.959060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372743+0000","last_change":"2024-03-30T14:55:43.421378+0000","last_active":"2024-03-30T14:56:12.372743+0000","last_peered":"2024-03-30T14:56:12.372743+0000","last_clean":"2024-03-30T14:56:12.372743+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T14:56:12.372743+0000","last_undegraded":"2024-03-30T14:56:12.372743+0000","last_fullsized":"2024-03-30T14:56:12.372743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:43.421270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:43.421270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.069402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372957+0000","last_change":"2024-03-30T14:55:46.387117+0000","last_active":"2024-03-30T14:56:12.372957+0000","last_peered":"2024-03-30T14:56:12.372957+0000","last_clean":"2024-03-30T14:56:12.372957+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T14:56:12.372957+0000","last_undegraded":"2024-03-30T14:56:12.372957+0000","last_fullsized":"2024-03-30T14:56:12.372957+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.387017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.387017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.976653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185507+0000","last_change":"2024-03-30T14:54:39.329805+0000","last_active":"2024-03-30T14:56:12.185507+0000","last_peered":"2024-03-30T14:56:12.185507+0000","last_clean":"2024-03-30T14:56:12.185507+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T14:56:12.185507+0000","last_undegraded":"2024-03-30T14:56:12.185507+0000","last_fullsized":"2024-03-30T14:56:12.185507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372521+0000","last_change":"2024-03-30T14:55:49.394236+0000","last_active":"2024-03-30T14:56:12.372521+0000","last_peered":"2024-03-30T14:56:12.372521+0000","last_clean":"2024-03-30T14:56:12.372521+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T14:56:12.372521+0000","last_undegraded":"2024-03-30T14:56:12.372521+0000","last_fullsized":"2024-03-30T14:56:12.372521+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.394138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.394138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.971320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531993+0000","last_change":"2024-03-30T14:56:09.083644+0000","last_active":"2024-03-30T14:56:12.531993+0000","last_peered":"2024-03-30T14:56:12.531993+0000","last_clean":"2024-03-30T14:56:12.531993+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T14:56:12.531993+0000","last_undegraded":"2024-03-30T14:56:12.531993+0000","last_fullsized":"2024-03-30T14:56:12.531993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.083567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.083567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:36.418282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373039+0000","last_change":"2024-03-30T14:55:59.608083+0000","last_active":"2024-03-30T14:56:12.373039+0000","last_peered":"2024-03-30T14:56:12.373039+0000","last_clean":"2024-03-30T14:56:12.373039+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T14:56:12.373039+0000","last_undegraded":"2024-03-30T14:56:12.373039+0000","last_fullsized":"2024-03-30T14:56:12.373039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.607983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.607983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.338389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185683+0000","last_change":"2024-03-30T14:54:58.404856+0000","last_active":"2024-03-30T14:56:12.185683+0000","last_peered":"2024-03-30T14:56:12.185683+0000","last_clean":"2024-03-30T14:56:12.185683+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T14:56:12.185683+0000","last_undegraded":"2024-03-30T14:56:12.185683+0000","last_fullsized":"2024-03-30T14:56:12.185683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186044+0000","last_change":"2024-03-30T14:54:43.349041+0000","last_active":"2024-03-30T14:56:12.186044+0000","last_peered":"2024-03-30T14:56:12.186044+0000","last_clean":"2024-03-30T14:56:12.186044+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T14:56:12.186044+0000","last_undegraded":"2024-03-30T14:56:12.186044+0000","last_fullsized":"2024-03-30T14:56:12.186044+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541470+0000","last_change":"2024-03-30T14:54:41.347727+0000","last_active":"2024-03-30T14:56:12.541470+0000","last_peered":"2024-03-30T14:56:12.541470+0000","last_clean":"2024-03-30T14:56:12.541470+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T14:56:12.541470+0000","last_undegraded":"2024-03-30T14:56:12.541470+0000","last_fullsized":"2024-03-30T14:56:12.541470+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185927+0000","last_change":"2024-03-30T14:55:42.710206+0000","last_active":"2024-03-30T14:56:12.185927+0000","last_peered":"2024-03-30T14:56:12.185927+0000","last_clean":"2024-03-30T14:56:12.185927+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T14:56:12.185927+0000","last_undegraded":"2024-03-30T14:56:12.185927+0000","last_fullsized":"2024-03-30T14:56:12.185927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.710118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.710118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.005945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"76'15","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531674+0000","last_change":"2024-03-30T14:55:39.803863+0000","last_active":"2024-03-30T14:56:12.531674+0000","last_peered":"2024-03-30T14:56:12.531674+0000","last_clean":"2024-03-30T14:56:12.531674+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T14:56:12.531674+0000","last_undegraded":"2024-03-30T14:56:12.531674+0000","last_fullsized":"2024-03-30T14:56:12.531674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:55:39.803752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:39.803752+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.649577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368454+0000","last_change":"2024-03-30T14:56:11.760211+0000","last_active":"2024-03-30T14:56:12.368454+0000","last_peered":"2024-03-30T14:56:12.368454+0000","last_clean":"2024-03-30T14:56:12.368454+0000","last_became_active":"2024-03-30T14:54:43.354397+0000","last_became_peered":"2024-03-30T14:54:43.354397+0000","last_unstale":"2024-03-30T14:56:12.368454+0000","last_undegraded":"2024-03-30T14:56:12.368454+0000","last_fullsized":"2024-03-30T14:56:12.368454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:11.760085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:11.760085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:36.986177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185748+0000","last_change":"2024-03-30T14:54:41.348874+0000","last_active":"2024-03-30T14:56:12.185748+0000","last_peered":"2024-03-30T14:56:12.185748+0000","last_clean":"2024-03-30T14:56:12.185748+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T14:56:12.185748+0000","last_undegraded":"2024-03-30T14:56:12.185748+0000","last_fullsized":"2024-03-30T14:56:12.185748+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185565+0000","last_change":"2024-03-30T14:54:37.474896+0000","last_active":"2024-03-30T14:56:12.185565+0000","last_peered":"2024-03-30T14:56:12.185565+0000","last_clean":"2024-03-30T14:56:12.185565+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T14:56:12.185565+0000","last_undegraded":"2024-03-30T14:56:12.185565+0000","last_fullsized":"2024-03-30T14:56:12.185565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368292+0000","last_change":"2024-03-30T14:56:01.665638+0000","last_active":"2024-03-30T14:56:12.368292+0000","last_peered":"2024-03-30T14:56:12.368292+0000","last_clean":"2024-03-30T14:56:12.368292+0000","last_became_active":"2024-03-30T14:54:39.338927+0000","last_became_peered":"2024-03-30T14:54:39.338927+0000","last_unstale":"2024-03-30T14:56:12.368292+0000","last_undegraded":"2024-03-30T14:56:12.368292+0000","last_fullsized":"2024-03-30T14:56:12.368292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:56:01.665505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.665505+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.704128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"76'1","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532417+0000","last_change":"2024-03-30T14:54:43.354908+0000","last_active":"2024-03-30T14:56:12.532417+0000","last_peered":"2024-03-30T14:56:12.532417+0000","last_clean":"2024-03-30T14:56:12.532417+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T14:56:12.532417+0000","last_undegraded":"2024-03-30T14:56:12.532417+0000","last_fullsized":"2024-03-30T14:56:12.532417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532778+0000","last_change":"2024-03-30T14:55:58.120425+0000","last_active":"2024-03-30T14:56:12.532778+0000","last_peered":"2024-03-30T14:56:12.532778+0000","last_clean":"2024-03-30T14:56:12.532778+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T14:56:12.532778+0000","last_undegraded":"2024-03-30T14:56:12.532778+0000","last_fullsized":"2024-03-30T14:56:12.532778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:58.120314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:58.120314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.131781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541294+0000","last_change":"2024-03-30T14:54:37.324901+0000","last_active":"2024-03-30T14:56:12.541294+0000","last_peered":"2024-03-30T14:56:12.541294+0000","last_clean":"2024-03-30T14:56:12.541294+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T14:56:12.541294+0000","last_undegraded":"2024-03-30T14:56:12.541294+0000","last_fullsized":"2024-03-30T14:56:12.541294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"76'10","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186883+0000","last_change":"2024-03-30T14:54:39.642246+0000","last_active":"2024-03-30T14:56:12.186883+0000","last_peered":"2024-03-30T14:56:12.186883+0000","last_clean":"2024-03-30T14:56:12.186883+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T14:56:12.186883+0000","last_undegraded":"2024-03-30T14:56:12.186883+0000","last_fullsized":"2024-03-30T14:56:12.186883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541634+0000","last_change":"2024-03-30T14:55:49.280406+0000","last_active":"2024-03-30T14:56:12.541634+0000","last_peered":"2024-03-30T14:56:12.541634+0000","last_clean":"2024-03-30T14:56:12.541634+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T14:56:12.541634+0000","last_undegraded":"2024-03-30T14:56:12.541634+0000","last_fullsized":"2024-03-30T14:56:12.541634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.280327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.280327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.497970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.187124+0000","last_change":"2024-03-30T14:54:41.347270+0000","last_active":"2024-03-30T14:56:12.187124+0000","last_peered":"2024-03-30T14:56:12.187124+0000","last_clean":"2024-03-30T14:56:12.187124+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T14:56:12.187124+0000","last_undegraded":"2024-03-30T14:56:12.187124+0000","last_fullsized":"2024-03-30T14:56:12.187124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367852+0000","last_change":"2024-03-30T14:54:37.321125+0000","last_active":"2024-03-30T14:56:12.367852+0000","last_peered":"2024-03-30T14:56:12.367852+0000","last_clean":"2024-03-30T14:56:12.367852+0000","last_became_active":"2024-03-30T14:54:37.320726+0000","last_became_peered":"2024-03-30T14:54:37.320726+0000","last_unstale":"2024-03-30T14:56:12.367852+0000","last_undegraded":"2024-03-30T14:56:12.367852+0000","last_fullsized":"2024-03-30T14:56:12.367852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532626+0000","last_change":"2024-03-30T14:56:12.032133+0000","last_active":"2024-03-30T14:56:12.532626+0000","last_peered":"2024-03-30T14:56:12.532626+0000","last_clean":"2024-03-30T14:56:12.532626+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T14:56:12.532626+0000","last_undegraded":"2024-03-30T14:56:12.532626+0000","last_fullsized":"2024-03-30T14:56:12.532626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.032038+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:37.869855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531933+0000","last_change":"2024-03-30T14:54:43.352814+0000","last_active":"2024-03-30T14:56:12.531933+0000","last_peered":"2024-03-30T14:56:12.531933+0000","last_clean":"2024-03-30T14:56:12.531933+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T14:56:12.531933+0000","last_undegraded":"2024-03-30T14:56:12.531933+0000","last_fullsized":"2024-03-30T14:56:12.531933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531572+0000","last_change":"2024-03-30T14:55:53.203651+0000","last_active":"2024-03-30T14:56:12.531572+0000","last_peered":"2024-03-30T14:56:12.531572+0000","last_clean":"2024-03-30T14:56:12.531572+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T14:56:12.531572+0000","last_undegraded":"2024-03-30T14:56:12.531572+0000","last_fullsized":"2024-03-30T14:56:12.531572+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.203533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.203533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:56.173167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.374402+0000","last_change":"2024-03-30T14:56:12.374423+0000","last_active":"2024-03-30T14:56:12.374402+0000","last_peered":"2024-03-30T14:56:12.374402+0000","last_clean":"2024-03-30T14:56:12.374402+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T14:56:12.374402+0000","last_undegraded":"2024-03-30T14:56:12.374402+0000","last_fullsized":"2024-03-30T14:56:12.374402+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:12.374338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.374338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.427721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367866+0000","last_change":"2024-03-30T14:56:03.643626+0000","last_active":"2024-03-30T14:56:12.367866+0000","last_peered":"2024-03-30T14:56:12.367866+0000","last_clean":"2024-03-30T14:56:12.367866+0000","last_became_active":"2024-03-30T14:54:39.338462+0000","last_became_peered":"2024-03-30T14:54:39.338462+0000","last_unstale":"2024-03-30T14:56:12.367866+0000","last_undegraded":"2024-03-30T14:56:12.367866+0000","last_fullsized":"2024-03-30T14:56:12.367866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:56:03.643494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:03.643494+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.936270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367977+0000","last_change":"2024-03-30T14:54:43.361222+0000","last_active":"2024-03-30T14:56:12.367977+0000","last_peered":"2024-03-30T14:56:12.367977+0000","last_clean":"2024-03-30T14:56:12.367977+0000","last_became_active":"2024-03-30T14:54:43.360837+0000","last_became_peered":"2024-03-30T14:54:43.360837+0000","last_unstale":"2024-03-30T14:56:12.367977+0000","last_undegraded":"2024-03-30T14:56:12.367977+0000","last_fullsized":"2024-03-30T14:56:12.367977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532329+0000","last_change":"2024-03-30T14:54:41.354167+0000","last_active":"2024-03-30T14:56:12.532329+0000","last_peered":"2024-03-30T14:56:12.532329+0000","last_clean":"2024-03-30T14:56:12.532329+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T14:56:12.532329+0000","last_undegraded":"2024-03-30T14:56:12.532329+0000","last_fullsized":"2024-03-30T14:56:12.532329+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372973+0000","last_change":"2024-03-30T14:54:37.331035+0000","last_active":"2024-03-30T14:56:12.372973+0000","last_peered":"2024-03-30T14:56:12.372973+0000","last_clean":"2024-03-30T14:56:12.372973+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T14:56:12.372973+0000","last_undegraded":"2024-03-30T14:56:12.372973+0000","last_fullsized":"2024-03-30T14:56:12.372973+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368094+0000","last_change":"2024-03-30T14:56:07.672900+0000","last_active":"2024-03-30T14:56:12.368094+0000","last_peered":"2024-03-30T14:56:12.368094+0000","last_clean":"2024-03-30T14:56:12.368094+0000","last_became_active":"2024-03-30T14:54:39.641142+0000","last_became_peered":"2024-03-30T14:54:39.641142+0000","last_unstale":"2024-03-30T14:56:12.368094+0000","last_undegraded":"2024-03-30T14:56:12.368094+0000","last_fullsized":"2024-03-30T14:56:12.368094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:56:07.672785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.672785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.755665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:13.401189+0000","last_change":"2024-03-30T14:56:13.401210+0000","last_active":"2024-03-30T14:56:13.401189+0000","last_peered":"2024-03-30T14:56:13.401189+0000","last_clean":"2024-03-30T14:56:13.401189+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T14:56:13.401189+0000","last_undegraded":"2024-03-30T14:56:13.401189+0000","last_fullsized":"2024-03-30T14:56:13.401189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:13.401132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:13.401132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.048637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368293+0000","last_change":"2024-03-30T14:54:41.347572+0000","last_active":"2024-03-30T14:56:12.368293+0000","last_peered":"2024-03-30T14:56:12.368293+0000","last_clean":"2024-03-30T14:56:12.368293+0000","last_became_active":"2024-03-30T14:54:41.347229+0000","last_became_peered":"2024-03-30T14:54:41.347229+0000","last_unstale":"2024-03-30T14:56:12.368293+0000","last_undegraded":"2024-03-30T14:56:12.368293+0000","last_fullsized":"2024-03-30T14:56:12.368293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532844+0000","last_change":"2024-03-30T14:56:03.093369+0000","last_active":"2024-03-30T14:56:12.532844+0000","last_peered":"2024-03-30T14:56:12.532844+0000","last_clean":"2024-03-30T14:56:12.532844+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T14:56:12.532844+0000","last_undegraded":"2024-03-30T14:56:12.532844+0000","last_fullsized":"2024-03-30T14:56:12.532844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:56:03.093218+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.894705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541665+0000","last_change":"2024-03-30T14:54:39.339910+0000","last_active":"2024-03-30T14:56:12.541665+0000","last_peered":"2024-03-30T14:56:12.541665+0000","last_clean":"2024-03-30T14:56:12.541665+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T14:56:12.541665+0000","last_undegraded":"2024-03-30T14:56:12.541665+0000","last_fullsized":"2024-03-30T14:56:12.541665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367824+0000","last_change":"2024-03-30T14:54:43.360542+0000","last_active":"2024-03-30T14:56:12.367824+0000","last_peered":"2024-03-30T14:56:12.367824+0000","last_clean":"2024-03-30T14:56:12.367824+0000","last_became_active":"2024-03-30T14:54:43.360215+0000","last_became_peered":"2024-03-30T14:54:43.360215+0000","last_unstale":"2024-03-30T14:56:12.367824+0000","last_undegraded":"2024-03-30T14:56:12.367824+0000","last_fullsized":"2024-03-30T14:56:12.367824+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541466+0000","last_change":"2024-03-30T14:55:53.260188+0000","last_active":"2024-03-30T14:56:12.541466+0000","last_peered":"2024-03-30T14:56:12.541466+0000","last_clean":"2024-03-30T14:56:12.541466+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T14:56:12.541466+0000","last_undegraded":"2024-03-30T14:56:12.541466+0000","last_fullsized":"2024-03-30T14:56:12.541466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.260101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.260101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.452479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185911+0000","last_change":"2024-03-30T14:56:00.482961+0000","last_active":"2024-03-30T14:56:12.185911+0000","last_peered":"2024-03-30T14:56:12.185911+0000","last_clean":"2024-03-30T14:56:12.185911+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T14:56:12.185911+0000","last_undegraded":"2024-03-30T14:56:12.185911+0000","last_fullsized":"2024-03-30T14:56:12.185911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:00.482861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.482861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.363058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"76'11","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185782+0000","last_change":"2024-03-30T14:54:39.334527+0000","last_active":"2024-03-30T14:56:12.185782+0000","last_peered":"2024-03-30T14:56:12.185782+0000","last_clean":"2024-03-30T14:56:12.185782+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T14:56:12.185782+0000","last_undegraded":"2024-03-30T14:56:12.185782+0000","last_fullsized":"2024-03-30T14:56:12.185782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"76'1","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186614+0000","last_change":"2024-03-30T14:54:43.354400+0000","last_active":"2024-03-30T14:56:12.186614+0000","last_peered":"2024-03-30T14:56:12.186614+0000","last_clean":"2024-03-30T14:56:12.186614+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T14:56:12.186614+0000","last_undegraded":"2024-03-30T14:56:12.186614+0000","last_fullsized":"2024-03-30T14:56:12.186614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.205644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373038+0000","last_change":"2024-03-30T14:54:41.350464+0000","last_active":"2024-03-30T14:56:12.373038+0000","last_peered":"2024-03-30T14:56:12.373038+0000","last_clean":"2024-03-30T14:56:12.373038+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T14:56:12.373038+0000","last_undegraded":"2024-03-30T14:56:12.373038+0000","last_fullsized":"2024-03-30T14:56:12.373038+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533384+0000","last_change":"2024-03-30T14:54:37.474723+0000","last_active":"2024-03-30T14:56:12.533384+0000","last_peered":"2024-03-30T14:56:12.533384+0000","last_clean":"2024-03-30T14:56:12.533384+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T14:56:12.533384+0000","last_undegraded":"2024-03-30T14:56:12.533384+0000","last_fullsized":"2024-03-30T14:56:12.533384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368098+0000","last_change":"2024-03-30T14:55:53.742586+0000","last_active":"2024-03-30T14:56:12.368098+0000","last_peered":"2024-03-30T14:56:12.368098+0000","last_clean":"2024-03-30T14:56:12.368098+0000","last_became_active":"2024-03-30T14:54:39.642132+0000","last_became_peered":"2024-03-30T14:54:39.642132+0000","last_unstale":"2024-03-30T14:56:12.368098+0000","last_undegraded":"2024-03-30T14:56:12.368098+0000","last_fullsized":"2024-03-30T14:56:12.368098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:53.742492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.742492+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.865684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186115+0000","last_change":"2024-03-30T14:54:43.349809+0000","last_active":"2024-03-30T14:56:12.186115+0000","last_peered":"2024-03-30T14:56:12.186115+0000","last_clean":"2024-03-30T14:56:12.186115+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T14:56:12.186115+0000","last_undegraded":"2024-03-30T14:56:12.186115+0000","last_fullsized":"2024-03-30T14:56:12.186115+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"76'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368434+0000","last_change":"2024-03-30T14:54:41.346770+0000","last_active":"2024-03-30T14:56:12.368434+0000","last_peered":"2024-03-30T14:56:12.368434+0000","last_clean":"2024-03-30T14:56:12.368434+0000","last_became_active":"2024-03-30T14:54:41.346418+0000","last_became_peered":"2024-03-30T14:54:41.346418+0000","last_unstale":"2024-03-30T14:56:12.368434+0000","last_undegraded":"2024-03-30T14:56:12.368434+0000","last_fullsized":"2024-03-30T14:56:12.368434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185158+0000","last_change":"2024-03-30T14:56:02.515803+0000","last_active":"2024-03-30T14:56:12.185158+0000","last_peered":"2024-03-30T14:56:12.185158+0000","last_clean":"2024-03-30T14:56:12.185158+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T14:56:12.185158+0000","last_undegraded":"2024-03-30T14:56:12.185158+0000","last_fullsized":"2024-03-30T14:56:12.185158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.515709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.515709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.607511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373154+0000","last_change":"2024-03-30T14:54:39.644472+0000","last_active":"2024-03-30T14:56:12.373154+0000","last_peered":"2024-03-30T14:56:12.373154+0000","last_clean":"2024-03-30T14:56:12.373154+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T14:56:12.373154+0000","last_undegraded":"2024-03-30T14:56:12.373154+0000","last_fullsized":"2024-03-30T14:56:12.373154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531860+0000","last_change":"2024-03-30T14:55:48.721601+0000","last_active":"2024-03-30T14:56:12.531860+0000","last_peered":"2024-03-30T14:56:12.531860+0000","last_clean":"2024-03-30T14:56:12.531860+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T14:56:12.531860+0000","last_undegraded":"2024-03-30T14:56:12.531860+0000","last_fullsized":"2024-03-30T14:56:12.531860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.721477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.721477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:58.996126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368734+0000","last_change":"2024-03-30T14:56:09.706800+0000","last_active":"2024-03-30T14:56:12.368734+0000","last_peered":"2024-03-30T14:56:12.368734+0000","last_clean":"2024-03-30T14:56:12.368734+0000","last_became_active":"2024-03-30T14:54:41.345543+0000","last_became_peered":"2024-03-30T14:54:41.345543+0000","last_unstale":"2024-03-30T14:56:12.368734+0000","last_undegraded":"2024-03-30T14:56:12.368734+0000","last_fullsized":"2024-03-30T14:56:12.368734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.706719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.706719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:30.142928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532049+0000","last_change":"2024-03-30T14:55:48.068870+0000","last_active":"2024-03-30T14:56:12.532049+0000","last_peered":"2024-03-30T14:56:12.532049+0000","last_clean":"2024-03-30T14:56:12.532049+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T14:56:12.532049+0000","last_undegraded":"2024-03-30T14:56:12.532049+0000","last_fullsized":"2024-03-30T14:56:12.532049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.068767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.068767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:16.799841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186976+0000","last_change":"2024-03-30T14:54:39.642222+0000","last_active":"2024-03-30T14:56:12.186976+0000","last_peered":"2024-03-30T14:56:12.186976+0000","last_clean":"2024-03-30T14:56:12.186976+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T14:56:12.186976+0000","last_undegraded":"2024-03-30T14:56:12.186976+0000","last_fullsized":"2024-03-30T14:56:12.186976+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532737+0000","last_change":"2024-03-30T14:54:43.357877+0000","last_active":"2024-03-30T14:56:12.532737+0000","last_peered":"2024-03-30T14:56:12.532737+0000","last_clean":"2024-03-30T14:56:12.532737+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T14:56:12.532737+0000","last_undegraded":"2024-03-30T14:56:12.532737+0000","last_fullsized":"2024-03-30T14:56:12.532737+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373083+0000","last_change":"2024-03-30T14:56:09.561375+0000","last_active":"2024-03-30T14:56:12.373083+0000","last_peered":"2024-03-30T14:56:12.373083+0000","last_clean":"2024-03-30T14:56:12.373083+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T14:56:12.373083+0000","last_undegraded":"2024-03-30T14:56:12.373083+0000","last_fullsized":"2024-03-30T14:56:12.373083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.561303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.561303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.644838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531520+0000","last_change":"2024-03-30T14:55:50.191458+0000","last_active":"2024-03-30T14:56:12.531520+0000","last_peered":"2024-03-30T14:56:12.531520+0000","last_clean":"2024-03-30T14:56:12.531520+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T14:56:12.531520+0000","last_undegraded":"2024-03-30T14:56:12.531520+0000","last_fullsized":"2024-03-30T14:56:12.531520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.191353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.191353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:00.197326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368757+0000","last_change":"2024-03-30T14:55:48.827002+0000","last_active":"2024-03-30T14:56:12.368757+0000","last_peered":"2024-03-30T14:56:12.368757+0000","last_clean":"2024-03-30T14:56:12.368757+0000","last_became_active":"2024-03-30T14:54:39.643439+0000","last_became_peered":"2024-03-30T14:54:39.643439+0000","last_unstale":"2024-03-30T14:56:12.368757+0000","last_undegraded":"2024-03-30T14:56:12.368757+0000","last_fullsized":"2024-03-30T14:56:12.368757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:55:48.826912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.826912+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.602167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372779+0000","last_change":"2024-03-30T14:54:43.352897+0000","last_active":"2024-03-30T14:56:12.372779+0000","last_peered":"2024-03-30T14:56:12.372779+0000","last_clean":"2024-03-30T14:56:12.372779+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:56:12.372779+0000","last_undegraded":"2024-03-30T14:56:12.372779+0000","last_fullsized":"2024-03-30T14:56:12.372779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533285+0000","last_change":"2024-03-30T14:54:37.330956+0000","last_active":"2024-03-30T14:56:12.533285+0000","last_peered":"2024-03-30T14:56:12.533285+0000","last_clean":"2024-03-30T14:56:12.533285+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T14:56:12.533285+0000","last_undegraded":"2024-03-30T14:56:12.533285+0000","last_fullsized":"2024-03-30T14:56:12.533285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372542+0000","last_change":"2024-03-30T14:54:39.643618+0000","last_active":"2024-03-30T14:56:12.372542+0000","last_peered":"2024-03-30T14:56:12.372542+0000","last_clean":"2024-03-30T14:56:12.372542+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T14:56:12.372542+0000","last_undegraded":"2024-03-30T14:56:12.372542+0000","last_fullsized":"2024-03-30T14:56:12.372542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532343+0000","last_change":"2024-03-30T14:54:41.352889+0000","last_active":"2024-03-30T14:56:12.532343+0000","last_peered":"2024-03-30T14:56:12.532343+0000","last_clean":"2024-03-30T14:56:12.532343+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T14:56:12.532343+0000","last_undegraded":"2024-03-30T14:56:12.532343+0000","last_fullsized":"2024-03-30T14:56:12.532343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367969+0000","last_change":"2024-03-30T14:54:43.358959+0000","last_active":"2024-03-30T14:56:12.367969+0000","last_peered":"2024-03-30T14:56:12.367969+0000","last_clean":"2024-03-30T14:56:12.367969+0000","last_became_active":"2024-03-30T14:54:43.358552+0000","last_became_peered":"2024-03-30T14:54:43.358552+0000","last_unstale":"2024-03-30T14:56:12.367969+0000","last_undegraded":"2024-03-30T14:56:12.367969+0000","last_fullsized":"2024-03-30T14:56:12.367969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532909+0000","last_change":"2024-03-30T14:54:37.328695+0000","last_active":"2024-03-30T14:56:12.532909+0000","last_peered":"2024-03-30T14:56:12.532909+0000","last_clean":"2024-03-30T14:56:12.532909+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T14:56:12.532909+0000","last_undegraded":"2024-03-30T14:56:12.532909+0000","last_fullsized":"2024-03-30T14:56:12.532909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185529+0000","last_change":"2024-03-30T14:55:56.431786+0000","last_active":"2024-03-30T14:56:12.185529+0000","last_peered":"2024-03-30T14:56:12.185529+0000","last_clean":"2024-03-30T14:56:12.185529+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T14:56:12.185529+0000","last_undegraded":"2024-03-30T14:56:12.185529+0000","last_fullsized":"2024-03-30T14:56:12.185529+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:56.431647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.431647+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.459883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531467+0000","last_change":"2024-03-30T14:56:06.652426+0000","last_active":"2024-03-30T14:56:12.531467+0000","last_peered":"2024-03-30T14:56:12.531467+0000","last_clean":"2024-03-30T14:56:12.531467+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T14:56:12.531467+0000","last_undegraded":"2024-03-30T14:56:12.531467+0000","last_fullsized":"2024-03-30T14:56:12.531467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:56:06.652343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.926692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531924+0000","last_change":"2024-03-30T14:54:43.356146+0000","last_active":"2024-03-30T14:56:12.531924+0000","last_peered":"2024-03-30T14:56:12.531924+0000","last_clean":"2024-03-30T14:56:12.531924+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T14:56:12.531924+0000","last_undegraded":"2024-03-30T14:56:12.531924+0000","last_fullsized":"2024-03-30T14:56:12.531924+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531826+0000","last_change":"2024-03-30T14:56:02.650786+0000","last_active":"2024-03-30T14:56:12.531826+0000","last_peered":"2024-03-30T14:56:12.531826+0000","last_clean":"2024-03-30T14:56:12.531826+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T14:56:12.531826+0000","last_undegraded":"2024-03-30T14:56:12.531826+0000","last_fullsized":"2024-03-30T14:56:12.531826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.650642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.650642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:08.689802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533083+0000","last_change":"2024-03-30T14:54:37.329066+0000","last_active":"2024-03-30T14:56:12.533083+0000","last_peered":"2024-03-30T14:56:12.533083+0000","last_clean":"2024-03-30T14:56:12.533083+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T14:56:12.533083+0000","last_undegraded":"2024-03-30T14:56:12.533083+0000","last_fullsized":"2024-03-30T14:56:12.533083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":120,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532312+0000","last_change":"2024-03-30T14:55:52.189892+0000","last_active":"2024-03-30T14:56:12.532312+0000","last_peered":"2024-03-30T14:56:12.532312+0000","last_clean":"2024-03-30T14:56:12.532312+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T14:56:12.532312+0000","last_undegraded":"2024-03-30T14:56:12.532312+0000","last_fullsized":"2024-03-30T14:56:12.532312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:55:52.189742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.189742+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.639688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.184884+0000","last_change":"2024-03-30T14:54:43.353107+0000","last_active":"2024-03-30T14:56:12.184884+0000","last_peered":"2024-03-30T14:56:12.184884+0000","last_clean":"2024-03-30T14:56:12.184884+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T14:56:12.184884+0000","last_undegraded":"2024-03-30T14:56:12.184884+0000","last_fullsized":"2024-03-30T14:56:12.184884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531612+0000","last_change":"2024-03-30T14:55:56.679624+0000","last_active":"2024-03-30T14:56:12.531612+0000","last_peered":"2024-03-30T14:56:12.531612+0000","last_clean":"2024-03-30T14:56:12.531612+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T14:56:12.531612+0000","last_undegraded":"2024-03-30T14:56:12.531612+0000","last_fullsized":"2024-03-30T14:56:12.531612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.679531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.679531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.763850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372989+0000","last_change":"2024-03-30T14:55:37.491888+0000","last_active":"2024-03-30T14:56:12.372989+0000","last_peered":"2024-03-30T14:56:12.372989+0000","last_clean":"2024-03-30T14:56:12.372989+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T14:56:12.372989+0000","last_undegraded":"2024-03-30T14:56:12.372989+0000","last_fullsized":"2024-03-30T14:56:12.372989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:55:37.491750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.952763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185730+0000","last_change":"2024-03-30T14:55:48.585645+0000","last_active":"2024-03-30T14:56:12.185730+0000","last_peered":"2024-03-30T14:56:12.185730+0000","last_clean":"2024-03-30T14:56:12.185730+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T14:56:12.185730+0000","last_undegraded":"2024-03-30T14:56:12.185730+0000","last_fullsized":"2024-03-30T14:56:12.185730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:48.585557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.585557+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.155340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368486+0000","last_change":"2024-03-30T14:55:59.683242+0000","last_active":"2024-03-30T14:56:12.368486+0000","last_peered":"2024-03-30T14:56:12.368486+0000","last_clean":"2024-03-30T14:56:12.368486+0000","last_became_active":"2024-03-30T14:54:43.356453+0000","last_became_peered":"2024-03-30T14:54:43.356453+0000","last_unstale":"2024-03-30T14:56:12.368486+0000","last_undegraded":"2024-03-30T14:56:12.368486+0000","last_fullsized":"2024-03-30T14:56:12.368486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.683158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.683158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.228244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372651+0000","last_change":"2024-03-30T14:54:40.343883+0000","last_active":"2024-03-30T14:56:12.372651+0000","last_peered":"2024-03-30T14:56:12.372651+0000","last_clean":"2024-03-30T14:56:12.372651+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T14:56:12.372651+0000","last_undegraded":"2024-03-30T14:56:12.372651+0000","last_fullsized":"2024-03-30T14:56:12.372651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185590+0000","last_change":"2024-03-30T14:56:11.414503+0000","last_active":"2024-03-30T14:56:12.185590+0000","last_peered":"2024-03-30T14:56:12.185590+0000","last_clean":"2024-03-30T14:56:12.185590+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T14:56:12.185590+0000","last_undegraded":"2024-03-30T14:56:12.185590+0000","last_fullsized":"2024-03-30T14:56:12.185590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:11.414361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:11.414361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:12.225419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372772+0000","last_change":"2024-03-30T14:56:04.590554+0000","last_active":"2024-03-30T14:56:12.372772+0000","last_peered":"2024-03-30T14:56:12.372772+0000","last_clean":"2024-03-30T14:56:12.372772+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T14:56:12.372772+0000","last_undegraded":"2024-03-30T14:56:12.372772+0000","last_fullsized":"2024-03-30T14:56:12.372772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:56:04.590400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.590400+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.294040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.187265+0000","last_change":"2024-03-30T14:56:10.441560+0000","last_active":"2024-03-30T14:56:12.187265+0000","last_peered":"2024-03-30T14:56:12.187265+0000","last_clean":"2024-03-30T14:56:12.187265+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T14:56:12.187265+0000","last_undegraded":"2024-03-30T14:56:12.187265+0000","last_fullsized":"2024-03-30T14:56:12.187265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:10.441422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:10.441422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.928811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533159+0000","last_change":"2024-03-30T14:55:56.007233+0000","last_active":"2024-03-30T14:56:12.533159+0000","last_peered":"2024-03-30T14:56:12.533159+0000","last_clean":"2024-03-30T14:56:12.533159+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T14:56:12.533159+0000","last_undegraded":"2024-03-30T14:56:12.533159+0000","last_fullsized":"2024-03-30T14:56:12.533159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.007133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.007133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.546510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541352+0000","last_change":"2024-03-30T14:56:04.301702+0000","last_active":"2024-03-30T14:56:12.541352+0000","last_peered":"2024-03-30T14:56:12.541352+0000","last_clean":"2024-03-30T14:56:12.541352+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T14:56:12.541352+0000","last_undegraded":"2024-03-30T14:56:12.541352+0000","last_fullsized":"2024-03-30T14:56:12.541352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.301590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:33.597560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531356+0000","last_change":"2024-03-30T14:55:49.700155+0000","last_active":"2024-03-30T14:56:12.531356+0000","last_peered":"2024-03-30T14:56:12.531356+0000","last_clean":"2024-03-30T14:56:12.531356+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T14:56:12.531356+0000","last_undegraded":"2024-03-30T14:56:12.531356+0000","last_fullsized":"2024-03-30T14:56:12.531356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.700022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.949979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"76'13","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541915+0000","last_change":"2024-03-30T14:56:00.305512+0000","last_active":"2024-03-30T14:56:12.541915+0000","last_peered":"2024-03-30T14:56:12.541915+0000","last_clean":"2024-03-30T14:56:12.541915+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T14:56:12.541915+0000","last_undegraded":"2024-03-30T14:56:12.541915+0000","last_fullsized":"2024-03-30T14:56:12.541915+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'13","last_scrub_stamp":"2024-03-30T14:56:00.305404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.305404+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.155134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531988+0000","last_change":"2024-03-30T14:55:57.693792+0000","last_active":"2024-03-30T14:56:12.531988+0000","last_peered":"2024-03-30T14:56:12.531988+0000","last_clean":"2024-03-30T14:56:12.531988+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T14:56:12.531988+0000","last_undegraded":"2024-03-30T14:56:12.531988+0000","last_fullsized":"2024-03-30T14:56:12.531988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.693703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.693703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.637711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373035+0000","last_change":"2024-03-30T14:54:43.356807+0000","last_active":"2024-03-30T14:56:12.373035+0000","last_peered":"2024-03-30T14:56:12.373035+0000","last_clean":"2024-03-30T14:56:12.373035+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T14:56:12.373035+0000","last_undegraded":"2024-03-30T14:56:12.373035+0000","last_fullsized":"2024-03-30T14:56:12.373035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532048+0000","last_change":"2024-03-30T14:55:05.131524+0000","last_active":"2024-03-30T14:56:12.532048+0000","last_peered":"2024-03-30T14:56:12.532048+0000","last_clean":"2024-03-30T14:56:12.532048+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T14:56:12.532048+0000","last_undegraded":"2024-03-30T14:56:12.532048+0000","last_fullsized":"2024-03-30T14:56:12.532048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-30T14:55:05.131440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:55:05.131440+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:33.848087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368102+0000","last_change":"2024-03-30T14:54:37.324739+0000","last_active":"2024-03-30T14:56:12.368102+0000","last_peered":"2024-03-30T14:56:12.368102+0000","last_clean":"2024-03-30T14:56:12.368102+0000","last_became_active":"2024-03-30T14:54:37.324423+0000","last_became_peered":"2024-03-30T14:54:37.324423+0000","last_unstale":"2024-03-30T14:56:12.368102+0000","last_undegraded":"2024-03-30T14:56:12.368102+0000","last_fullsized":"2024-03-30T14:56:12.368102+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533339+0000","last_change":"2024-03-30T14:56:07.057582+0000","last_active":"2024-03-30T14:56:12.533339+0000","last_peered":"2024-03-30T14:56:12.533339+0000","last_clean":"2024-03-30T14:56:12.533339+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T14:56:12.533339+0000","last_undegraded":"2024-03-30T14:56:12.533339+0000","last_fullsized":"2024-03-30T14:56:12.533339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T14:56:07.057479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.057479+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.119135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532574+0000","last_change":"2024-03-30T14:54:41.350344+0000","last_active":"2024-03-30T14:56:12.532574+0000","last_peered":"2024-03-30T14:56:12.532574+0000","last_clean":"2024-03-30T14:56:12.532574+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T14:56:12.532574+0000","last_undegraded":"2024-03-30T14:56:12.532574+0000","last_fullsized":"2024-03-30T14:56:12.532574+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541278+0000","last_change":"2024-03-30T14:54:43.357715+0000","last_active":"2024-03-30T14:56:12.541278+0000","last_peered":"2024-03-30T14:56:12.541278+0000","last_clean":"2024-03-30T14:56:12.541278+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T14:56:12.541278+0000","last_undegraded":"2024-03-30T14:56:12.541278+0000","last_fullsized":"2024-03-30T14:56:12.541278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185134+0000","last_change":"2024-03-30T14:54:37.326510+0000","last_active":"2024-03-30T14:56:12.185134+0000","last_peered":"2024-03-30T14:56:12.185134+0000","last_clean":"2024-03-30T14:56:12.185134+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T14:56:12.185134+0000","last_undegraded":"2024-03-30T14:56:12.185134+0000","last_fullsized":"2024-03-30T14:56:12.185134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531792+0000","last_change":"2024-03-30T14:56:07.116284+0000","last_active":"2024-03-30T14:56:12.531792+0000","last_peered":"2024-03-30T14:56:12.531792+0000","last_clean":"2024-03-30T14:56:12.531792+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T14:56:12.531792+0000","last_undegraded":"2024-03-30T14:56:12.531792+0000","last_fullsized":"2024-03-30T14:56:12.531792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:56:07.116206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.116206+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.531800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186713+0000","last_change":"2024-03-30T14:55:52.556360+0000","last_active":"2024-03-30T14:56:12.186713+0000","last_peered":"2024-03-30T14:56:12.186713+0000","last_clean":"2024-03-30T14:56:12.186713+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T14:56:12.186713+0000","last_undegraded":"2024-03-30T14:56:12.186713+0000","last_fullsized":"2024-03-30T14:56:12.186713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:52.556237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.556237+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.659172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373276+0000","last_change":"2024-03-30T14:55:57.530418+0000","last_active":"2024-03-30T14:56:12.373276+0000","last_peered":"2024-03-30T14:56:12.373276+0000","last_clean":"2024-03-30T14:56:12.373276+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T14:56:12.373276+0000","last_undegraded":"2024-03-30T14:56:12.373276+0000","last_fullsized":"2024-03-30T14:56:12.373276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:01.784600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541083+0000","last_change":"2024-03-30T14:56:01.265168+0000","last_active":"2024-03-30T14:56:12.541083+0000","last_peered":"2024-03-30T14:56:12.541083+0000","last_clean":"2024-03-30T14:56:12.541083+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T14:56:12.541083+0000","last_undegraded":"2024-03-30T14:56:12.541083+0000","last_fullsized":"2024-03-30T14:56:12.541083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:01.265084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.265084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.573818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541417+0000","last_change":"2024-03-30T14:55:42.232642+0000","last_active":"2024-03-30T14:56:12.541417+0000","last_peered":"2024-03-30T14:56:12.541417+0000","last_clean":"2024-03-30T14:56:12.541417+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T14:56:12.541417+0000","last_undegraded":"2024-03-30T14:56:12.541417+0000","last_fullsized":"2024-03-30T14:56:12.541417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.232525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.232525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:48.072521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"76'19","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186009+0000","last_change":"2024-03-30T14:54:39.642101+0000","last_active":"2024-03-30T14:56:12.186009+0000","last_peered":"2024-03-30T14:56:12.186009+0000","last_clean":"2024-03-30T14:56:12.186009+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T14:56:12.186009+0000","last_undegraded":"2024-03-30T14:56:12.186009+0000","last_fullsized":"2024-03-30T14:56:12.186009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532335+0000","last_change":"2024-03-30T14:56:04.074991+0000","last_active":"2024-03-30T14:56:12.532335+0000","last_peered":"2024-03-30T14:56:12.532335+0000","last_clean":"2024-03-30T14:56:12.532335+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T14:56:12.532335+0000","last_undegraded":"2024-03-30T14:56:12.532335+0000","last_fullsized":"2024-03-30T14:56:12.532335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:04.074894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.074894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:24.283425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.184687+0000","last_change":"2024-03-30T14:54:43.355213+0000","last_active":"2024-03-30T14:56:12.184687+0000","last_peered":"2024-03-30T14:56:12.184687+0000","last_clean":"2024-03-30T14:56:12.184687+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T14:56:12.184687+0000","last_undegraded":"2024-03-30T14:56:12.184687+0000","last_fullsized":"2024-03-30T14:56:12.184687+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373058+0000","last_change":"2024-03-30T14:55:41.454468+0000","last_active":"2024-03-30T14:56:12.373058+0000","last_peered":"2024-03-30T14:56:12.373058+0000","last_clean":"2024-03-30T14:56:12.373058+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T14:56:12.373058+0000","last_undegraded":"2024-03-30T14:56:12.373058+0000","last_fullsized":"2024-03-30T14:56:12.373058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.454365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:04.526213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"76'10","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541561+0000","last_change":"2024-03-30T14:54:39.338666+0000","last_active":"2024-03-30T14:56:12.541561+0000","last_peered":"2024-03-30T14:56:12.541561+0000","last_clean":"2024-03-30T14:56:12.541561+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T14:56:12.541561+0000","last_undegraded":"2024-03-30T14:56:12.541561+0000","last_fullsized":"2024-03-30T14:56:12.541561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185637+0000","last_change":"2024-03-30T14:55:50.585912+0000","last_active":"2024-03-30T14:56:12.185637+0000","last_peered":"2024-03-30T14:56:12.185637+0000","last_clean":"2024-03-30T14:56:12.185637+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T14:56:12.185637+0000","last_undegraded":"2024-03-30T14:56:12.185637+0000","last_fullsized":"2024-03-30T14:56:12.185637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:50.585773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.585773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.447085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532306+0000","last_change":"2024-03-30T14:55:49.029645+0000","last_active":"2024-03-30T14:56:12.532306+0000","last_peered":"2024-03-30T14:56:12.532306+0000","last_clean":"2024-03-30T14:56:12.532306+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T14:56:12.532306+0000","last_undegraded":"2024-03-30T14:56:12.532306+0000","last_fullsized":"2024-03-30T14:56:12.532306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.029548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.029548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:52.389450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"69'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186815+0000","last_change":"2024-03-30T14:54:37.326159+0000","last_active":"2024-03-30T14:56:12.186815+0000","last_peered":"2024-03-30T14:56:12.186815+0000","last_clean":"2024-03-30T14:56:12.186815+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T14:56:12.186815+0000","last_undegraded":"2024-03-30T14:56:12.186815+0000","last_fullsized":"2024-03-30T14:56:12.186815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185412+0000","last_change":"2024-03-30T14:55:46.303554+0000","last_active":"2024-03-30T14:56:12.185412+0000","last_peered":"2024-03-30T14:56:12.185412+0000","last_clean":"2024-03-30T14:56:12.185412+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T14:56:12.185412+0000","last_undegraded":"2024-03-30T14:56:12.185412+0000","last_fullsized":"2024-03-30T14:56:12.185412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:55:46.303412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.303412+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.425234+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"76'8","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368603+0000","last_change":"2024-03-30T14:55:55.692425+0000","last_active":"2024-03-30T14:56:12.368603+0000","last_peered":"2024-03-30T14:56:12.368603+0000","last_clean":"2024-03-30T14:56:12.368603+0000","last_became_active":"2024-03-30T14:54:41.353489+0000","last_became_peered":"2024-03-30T14:54:41.353489+0000","last_unstale":"2024-03-30T14:56:12.368603+0000","last_undegraded":"2024-03-30T14:56:12.368603+0000","last_fullsized":"2024-03-30T14:56:12.368603+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:55.692261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:55.692261+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.231384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185375+0000","last_change":"2024-03-30T14:54:43.352748+0000","last_active":"2024-03-30T14:56:12.185375+0000","last_peered":"2024-03-30T14:56:12.185375+0000","last_clean":"2024-03-30T14:56:12.185375+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T14:56:12.185375+0000","last_undegraded":"2024-03-30T14:56:12.185375+0000","last_fullsized":"2024-03-30T14:56:12.185375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"69'1","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368560+0000","last_change":"2024-03-30T14:54:37.323745+0000","last_active":"2024-03-30T14:56:12.368560+0000","last_peered":"2024-03-30T14:56:12.368560+0000","last_clean":"2024-03-30T14:56:12.368560+0000","last_became_active":"2024-03-30T14:54:37.323384+0000","last_became_peered":"2024-03-30T14:54:37.323384+0000","last_unstale":"2024-03-30T14:56:12.368560+0000","last_undegraded":"2024-03-30T14:56:12.368560+0000","last_fullsized":"2024-03-30T14:56:12.368560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368546+0000","last_change":"2024-03-30T14:56:10.715437+0000","last_active":"2024-03-30T14:56:12.368546+0000","last_peered":"2024-03-30T14:56:12.368546+0000","last_clean":"2024-03-30T14:56:12.368546+0000","last_became_active":"2024-03-30T14:54:39.642450+0000","last_became_peered":"2024-03-30T14:54:39.642450+0000","last_unstale":"2024-03-30T14:56:12.368546+0000","last_undegraded":"2024-03-30T14:56:12.368546+0000","last_fullsized":"2024-03-30T14:56:12.368546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:56:10.715316+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:26.611400+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185259+0000","last_change":"2024-03-30T14:55:54.381596+0000","last_active":"2024-03-30T14:56:12.185259+0000","last_peered":"2024-03-30T14:56:12.185259+0000","last_clean":"2024-03-30T14:56:12.185259+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T14:56:12.185259+0000","last_undegraded":"2024-03-30T14:56:12.185259+0000","last_fullsized":"2024-03-30T14:56:12.185259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:54.381448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:54.381448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.160867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372869+0000","last_change":"2024-03-30T14:54:43.351363+0000","last_active":"2024-03-30T14:56:12.372869+0000","last_peered":"2024-03-30T14:56:12.372869+0000","last_clean":"2024-03-30T14:56:12.372869+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T14:56:12.372869+0000","last_undegraded":"2024-03-30T14:56:12.372869+0000","last_fullsized":"2024-03-30T14:56:12.372869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186308+0000","last_change":"2024-03-30T14:56:07.479522+0000","last_active":"2024-03-30T14:56:12.186308+0000","last_peered":"2024-03-30T14:56:12.186308+0000","last_clean":"2024-03-30T14:56:12.186308+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T14:56:12.186308+0000","last_undegraded":"2024-03-30T14:56:12.186308+0000","last_fullsized":"2024-03-30T14:56:12.186308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:56:07.479430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.479430+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.427750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368275+0000","last_change":"2024-03-30T14:54:37.323396+0000","last_active":"2024-03-30T14:56:12.368275+0000","last_peered":"2024-03-30T14:56:12.368275+0000","last_clean":"2024-03-30T14:56:12.368275+0000","last_became_active":"2024-03-30T14:54:37.323084+0000","last_became_peered":"2024-03-30T14:54:37.323084+0000","last_unstale":"2024-03-30T14:56:12.368275+0000","last_undegraded":"2024-03-30T14:56:12.368275+0000","last_fullsized":"2024-03-30T14:56:12.368275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"76'15","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541864+0000","last_change":"2024-03-30T14:54:39.338356+0000","last_active":"2024-03-30T14:56:12.541864+0000","last_peered":"2024-03-30T14:56:12.541864+0000","last_clean":"2024-03-30T14:56:12.541864+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T14:56:12.541864+0000","last_undegraded":"2024-03-30T14:56:12.541864+0000","last_fullsized":"2024-03-30T14:56:12.541864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368223+0000","last_change":"2024-03-30T14:56:06.675463+0000","last_active":"2024-03-30T14:56:12.368223+0000","last_peered":"2024-03-30T14:56:12.368223+0000","last_clean":"2024-03-30T14:56:12.368223+0000","last_became_active":"2024-03-30T14:54:43.360002+0000","last_became_peered":"2024-03-30T14:54:43.360002+0000","last_unstale":"2024-03-30T14:56:12.368223+0000","last_undegraded":"2024-03-30T14:56:12.368223+0000","last_fullsized":"2024-03-30T14:56:12.368223+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:06.675378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:06.675378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.466275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373248+0000","last_change":"2024-03-30T14:55:50.439135+0000","last_active":"2024-03-30T14:56:12.373248+0000","last_peered":"2024-03-30T14:56:12.373248+0000","last_clean":"2024-03-30T14:56:12.373248+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T14:56:12.373248+0000","last_undegraded":"2024-03-30T14:56:12.373248+0000","last_fullsized":"2024-03-30T14:56:12.373248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.439008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.333757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185330+0000","last_change":"2024-03-30T14:55:47.298871+0000","last_active":"2024-03-30T14:56:12.185330+0000","last_peered":"2024-03-30T14:56:12.185330+0000","last_clean":"2024-03-30T14:56:12.185330+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T14:56:12.185330+0000","last_undegraded":"2024-03-30T14:56:12.185330+0000","last_fullsized":"2024-03-30T14:56:12.185330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:47.298753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:47.298753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.038171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186223+0000","last_change":"2024-03-30T14:56:09.473519+0000","last_active":"2024-03-30T14:56:12.186223+0000","last_peered":"2024-03-30T14:56:12.186223+0000","last_clean":"2024-03-30T14:56:12.186223+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T14:56:12.186223+0000","last_undegraded":"2024-03-30T14:56:12.186223+0000","last_fullsized":"2024-03-30T14:56:12.186223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.473429+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.395740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.622534+0000","last_change":"2024-03-30T14:56:12.622556+0000","last_active":"2024-03-30T14:56:12.622534+0000","last_peered":"2024-03-30T14:56:12.622534+0000","last_clean":"2024-03-30T14:56:12.622534+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T14:56:12.622534+0000","last_undegraded":"2024-03-30T14:56:12.622534+0000","last_fullsized":"2024-03-30T14:56:12.622534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:12.622465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.622465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:20.190586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541515+0000","last_change":"2024-03-30T14:55:46.232558+0000","last_active":"2024-03-30T14:56:12.541515+0000","last_peered":"2024-03-30T14:56:12.541515+0000","last_clean":"2024-03-30T14:56:12.541515+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T14:56:12.541515+0000","last_undegraded":"2024-03-30T14:56:12.541515+0000","last_fullsized":"2024-03-30T14:56:12.541515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.232465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.933698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532117+0000","last_change":"2024-03-30T14:56:08.068026+0000","last_active":"2024-03-30T14:56:12.532117+0000","last_peered":"2024-03-30T14:56:12.532117+0000","last_clean":"2024-03-30T14:56:12.532117+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T14:56:12.532117+0000","last_undegraded":"2024-03-30T14:56:12.532117+0000","last_fullsized":"2024-03-30T14:56:12.532117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:08.067949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:08.067949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.341927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186074+0000","last_change":"2024-03-30T14:55:50.322071+0000","last_active":"2024-03-30T14:56:12.186074+0000","last_peered":"2024-03-30T14:56:12.186074+0000","last_clean":"2024-03-30T14:56:12.186074+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T14:56:12.186074+0000","last_undegraded":"2024-03-30T14:56:12.186074+0000","last_fullsized":"2024-03-30T14:56:12.186074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T14:55:50.321974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.321974+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:54.493589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186326+0000","last_change":"2024-03-30T14:54:43.352969+0000","last_active":"2024-03-30T14:56:12.186326+0000","last_peered":"2024-03-30T14:56:12.186326+0000","last_clean":"2024-03-30T14:56:12.186326+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:56:12.186326+0000","last_undegraded":"2024-03-30T14:56:12.186326+0000","last_fullsized":"2024-03-30T14:56:12.186326+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531780+0000","last_change":"2024-03-30T14:55:40.800056+0000","last_active":"2024-03-30T14:56:12.531780+0000","last_peered":"2024-03-30T14:56:12.531780+0000","last_clean":"2024-03-30T14:56:12.531780+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T14:56:12.531780+0000","last_undegraded":"2024-03-30T14:56:12.531780+0000","last_fullsized":"2024-03-30T14:56:12.531780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:40.799969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.799969+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:57.300469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":69,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368332+0000","last_change":"2024-03-30T14:55:45.783270+0000","last_active":"2024-03-30T14:56:12.368332+0000","last_peered":"2024-03-30T14:56:12.368332+0000","last_clean":"2024-03-30T14:56:12.368332+0000","last_became_active":"2024-03-30T14:54:37.327905+0000","last_became_peered":"2024-03-30T14:54:37.327905+0000","last_unstale":"2024-03-30T14:56:12.368332+0000","last_undegraded":"2024-03-30T14:56:12.368332+0000","last_fullsized":"2024-03-30T14:56:12.368332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:45.783178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:45.783178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.797807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185819+0000","last_change":"2024-03-30T14:55:51.283107+0000","last_active":"2024-03-30T14:56:12.185819+0000","last_peered":"2024-03-30T14:56:12.185819+0000","last_clean":"2024-03-30T14:56:12.185819+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T14:56:12.185819+0000","last_undegraded":"2024-03-30T14:56:12.185819+0000","last_fullsized":"2024-03-30T14:56:12.185819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:51.282990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:51.282990+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.280452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005080,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1331200,"data_stored":924302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299421,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32568,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712584,"statfs":{"total":95995035648,"available":95961686016,"internally_reserved":0,"allocated":1298432,"data_stored":921964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":745472,"data_stored":332803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1331200,"data_stored":923646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":708608,"data_stored":331931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":753664,"data_stored":333773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":737280,"data_stored":332599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":724992,"data_stored":332280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:56:15.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T14:56:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:15.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: from='client.14760 v1:172.21.15.28:0/3084720398' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:15.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T14:56:15.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 4.f deep-scrub starts 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 4.f deep-scrub ok 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: from='client.14760 v1:172.21.15.28:0/3084720398' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 4.f deep-scrub starts 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 4.f deep-scrub ok 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T14:56:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:15 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: from='client.14760 v1:172.21.15.28:0/3084720398' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 4.f deep-scrub starts 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 4.f deep-scrub ok 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T14:56:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:15 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T14:56:16.682 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:16 smithi028 bash[80459]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T14:56:16.972 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:16 smithi028 bash[80459]: Writing manifest to image destination 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[29876]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T14:56:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:16 smithi028 ceph-mon[41614]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:16 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T14:56:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:16 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T14:56:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:16 smithi134 ceph-mon[33714]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:17.474 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:17 smithi028 podman[80459]: 2024-03-30 14:56:17.113608778 +0000 UTC m=+2.992165402 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T14:56:17.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T14:56:17.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T14:56:17.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:17 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:17 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T14:56:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:17 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T14:56:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:17 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T14:56:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:17 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T14:56:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:17 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:18.200 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 podman[80459]: 2024-03-30 14:56:18.092511673 +0000 UTC m=+3.971068293 container create 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:18.528 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:56:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 podman[80459]: 2024-03-30 14:56:18.411574771 +0000 UTC m=+4.290131384 container init 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 podman[80459]: 2024-03-30 14:56:18.417363997 +0000 UTC m=+4.295920601 container start 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.422Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.422Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.423Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.423Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T14:56:18.529 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.423Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.423Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T14:56:18.530 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T14:56:18.531 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T14:56:18.532 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.424Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.425Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a[80693]: ts=2024-03-30T14:56:18.425Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T14:56:18.533 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 bash[80459]: 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b 2024-03-30T14:56:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/3140688681 >> v1:172.21.15.28:6790/0 conn(0x7f6ca0110c60 legacy=0x7f6ca0113120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/3140688681 shutdown_connections 2024-03-30T14:56:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/3140688681 >> 172.21.15.28:0/3140688681 conn(0x7f6ca00fc9d0 msgr2=0x7f6ca00fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/3140688681 shutdown_connections 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/3140688681 wait complete. 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 librados: starting msgr at 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.402+0000 7f6ca5cb0700 1 librados: starting objecter 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 Processor -- start 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 -- start start 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 librados: setting wanted keys 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 librados: calling monclient init 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ca01a9360 con 0x7f6ca0100d30 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ca01aa580 con 0x7f6ca0101910 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6ca5cb0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ca01ab7c0 con 0x7f6ca0110c60 2024-03-30T14:56:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9f7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6ca0100d30 0x7f6ca010f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54720/0 (socket says 172.21.15.28:54720) 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9f7fe700 1 -- 172.21.15.28:0/879696043 learned_addr learned my addr 172.21.15.28:0/879696043 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9ffff700 1 --1- 172.21.15.28:0/879696043 >> v1:172.21.15.28:6790/0 conn(0x7f6ca0110c60 0x7f6ca01a7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48922/0 (socket says 172.21.15.28:48922) 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter ms_handle_connect 0x7f6ca0100d30 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter ms_handle_connect 0x7f6ca0110c60 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982854977 0 0) 0x7f6ca01a9360 con 0x7f6ca0100d30 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c88003aa0 con 0x7f6ca0100d30 2024-03-30T14:56:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter ms_handle_connect 0x7f6ca0101910 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653581874 0 0) 0x7f6ca01ab7c0 con 0x7f6ca0110c60 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.403+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca01a9360 con 0x7f6ca0110c60 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086578990 0 0) 0x7f6ca01aa580 con 0x7f6ca0101910 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca01ab7c0 con 0x7f6ca0101910 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 575996111 0 0) 0x7f6c88003aa0 con 0x7f6ca0100d30 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ca01aa580 con 0x7f6ca0100d30 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c90001930 con 0x7f6ca0100d30 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 715533147 0 0) 0x7f6ca01a9360 con 0x7f6ca0110c60 2024-03-30T14:56:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c88003aa0 con 0x7f6ca0110c60 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c98004980 con 0x7f6ca0110c60 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1043964524 0 0) 0x7f6ca01ab7c0 con 0x7f6ca0101910 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ca01a9360 con 0x7f6ca0101910 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c94001a30 con 0x7f6ca0101910 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3874745412 0 0) 0x7f6ca01aa580 con 0x7f6ca0100d30 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 >> v1:172.21.15.28:6790/0 conn(0x7f6ca0110c60 legacy=0x7f6ca01a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 >> v1:172.21.15.134:6789/0 conn(0x7f6ca0101910 legacy=0x7f6ca01a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca01aca00 con 0x7f6ca0100d30 2024-03-30T14:56:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01ab9f0 con 0x7f6ca0100d30 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6ca5cb0700 10 client.14766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ca01abeb0 con 0x7f6ca0100d30 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6ca5cb0700 1 librados: init done 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6c90001d00 con 0x7f6ca0100d30 2024-03-30T14:56:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.404+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c90005d70 con 0x7f6ca0100d30 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.405+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c005120 con 0x7f6ca0100d30 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.405+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6c9001c410 con 0x7f6ca0100d30 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(94..94 src has 43..94) ==== 6259+0+0 (unknown 1180456038 0 0) 0x7f6c90088e20 con 0x7f6ca0100d30 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 10 client.14766.objecter ms_dispatch 0x7f6ca0001550 osd_map(94..94 src has 43..94) 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 3 client.14766.objecter handle_osd_map got epochs [94,94] > 0 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 3 client.14766.objecter handle_osd_map decoding full epoch 94 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 20 client.14766.objecter dump_active .. 0 homeless 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.406+0000 7f6c9cff9700 10 client.14766.objecter ms_handle_connect 0x7f6c88070380 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.410+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6c900568c0 con 0x7f6ca0100d30 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.622+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c8c000b90 con 0x7f6c88070380 2024-03-30T14:56:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.631+0000 7f6c9cff9700 1 -- 172.21.15.28:0/879696043 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+344220 (unknown 2965378022 0 2861410566) 0x7f6c8c000b90 con 0x7f6c88070380 2024-03-30T14:56:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 10 librados: watch_flush enter 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 10 librados: watch_flush exit 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 20 client.14766.objecter shutdown clearing up homeless session... 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 10 client.14766.objecter successfully canceled tick 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6c88070380 legacy=0x7f6c88072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 >> v1:172.21.15.28:6789/0 conn(0x7f6ca0100d30 legacy=0x7f6ca010f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 shutdown_connections 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 >> 172.21.15.28:0/879696043 conn(0x7f6ca00fc9d0 msgr2=0x7f6ca00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 shutdown_connections 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 -- 172.21.15.28:0/879696043 wait complete. 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:18.674+0000 7f6ca5cb0700 1 librados: shutdown 2024-03-30T14:56:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T14:56:18.972 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 14:56:18 smithi028 systemd[1]: Started Ceph node-exporter.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:19.405 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-30T14:56:17.589383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":627,"ondisk_log_size":627,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249532,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711684,"statfs":{"total":767960285184,"available":767704764416,"internally_reserved":0,"allocated":7630848,"data_stored":4433298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002180"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185349+0000","last_change":"2024-03-30T14:54:43.351532+0000","last_active":"2024-03-30T14:56:12.185349+0000","last_peered":"2024-03-30T14:56:12.185349+0000","last_clean":"2024-03-30T14:56:12.185349+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T14:56:12.185349+0000","last_undegraded":"2024-03-30T14:56:12.185349+0000","last_fullsized":"2024-03-30T14:56:12.185349+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541194+0000","last_change":"2024-03-30T14:54:41.350461+0000","last_active":"2024-03-30T14:56:12.541194+0000","last_peered":"2024-03-30T14:56:12.541194+0000","last_clean":"2024-03-30T14:56:12.541194+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T14:56:12.541194+0000","last_undegraded":"2024-03-30T14:56:12.541194+0000","last_fullsized":"2024-03-30T14:56:12.541194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186510+0000","last_change":"2024-03-30T14:55:46.590443+0000","last_active":"2024-03-30T14:56:12.186510+0000","last_peered":"2024-03-30T14:56:12.186510+0000","last_clean":"2024-03-30T14:56:12.186510+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T14:56:12.186510+0000","last_undegraded":"2024-03-30T14:56:12.186510+0000","last_fullsized":"2024-03-30T14:56:12.186510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.590360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.590360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.726057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185232+0000","last_change":"2024-03-30T14:55:40.318444+0000","last_active":"2024-03-30T14:56:12.185232+0000","last_peered":"2024-03-30T14:56:12.185232+0000","last_clean":"2024-03-30T14:56:12.185232+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T14:56:12.185232+0000","last_undegraded":"2024-03-30T14:56:12.185232+0000","last_fullsized":"2024-03-30T14:56:12.185232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:40.318295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.318295+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:45.878339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368092+0000","last_change":"2024-03-30T14:54:43.359234+0000","last_active":"2024-03-30T14:56:12.368092+0000","last_peered":"2024-03-30T14:56:12.368092+0000","last_clean":"2024-03-30T14:56:12.368092+0000","last_became_active":"2024-03-30T14:54:43.358170+0000","last_became_peered":"2024-03-30T14:54:43.358170+0000","last_unstale":"2024-03-30T14:56:12.368092+0000","last_undegraded":"2024-03-30T14:56:12.368092+0000","last_fullsized":"2024-03-30T14:56:12.368092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185232+0000","last_change":"2024-03-30T14:54:41.350763+0000","last_active":"2024-03-30T14:56:12.185232+0000","last_peered":"2024-03-30T14:56:12.185232+0000","last_clean":"2024-03-30T14:56:12.185232+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T14:56:12.185232+0000","last_undegraded":"2024-03-30T14:56:12.185232+0000","last_fullsized":"2024-03-30T14:56:12.185232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":69,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368597+0000","last_change":"2024-03-30T14:55:41.765389+0000","last_active":"2024-03-30T14:56:12.368597+0000","last_peered":"2024-03-30T14:56:12.368597+0000","last_clean":"2024-03-30T14:56:12.368597+0000","last_became_active":"2024-03-30T14:54:37.474114+0000","last_became_peered":"2024-03-30T14:54:37.474114+0000","last_unstale":"2024-03-30T14:56:12.368597+0000","last_undegraded":"2024-03-30T14:56:12.368597+0000","last_fullsized":"2024-03-30T14:56:12.368597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.765306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.765306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:43.228077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368115+0000","last_change":"2024-03-30T14:56:05.629550+0000","last_active":"2024-03-30T14:56:12.368115+0000","last_peered":"2024-03-30T14:56:12.368115+0000","last_clean":"2024-03-30T14:56:12.368115+0000","last_became_active":"2024-03-30T14:54:39.333512+0000","last_became_peered":"2024-03-30T14:54:39.333512+0000","last_unstale":"2024-03-30T14:56:12.368115+0000","last_undegraded":"2024-03-30T14:56:12.368115+0000","last_fullsized":"2024-03-30T14:56:12.368115+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:56:05.629449+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:27.238183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186676+0000","last_change":"2024-03-30T14:55:57.530576+0000","last_active":"2024-03-30T14:56:12.186676+0000","last_peered":"2024-03-30T14:56:12.186676+0000","last_clean":"2024-03-30T14:56:12.186676+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T14:56:12.186676+0000","last_undegraded":"2024-03-30T14:56:12.186676+0000","last_fullsized":"2024-03-30T14:56:12.186676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.959060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372743+0000","last_change":"2024-03-30T14:55:43.421378+0000","last_active":"2024-03-30T14:56:12.372743+0000","last_peered":"2024-03-30T14:56:12.372743+0000","last_clean":"2024-03-30T14:56:12.372743+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T14:56:12.372743+0000","last_undegraded":"2024-03-30T14:56:12.372743+0000","last_fullsized":"2024-03-30T14:56:12.372743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:43.421270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:43.421270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.069402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372957+0000","last_change":"2024-03-30T14:55:46.387117+0000","last_active":"2024-03-30T14:56:12.372957+0000","last_peered":"2024-03-30T14:56:12.372957+0000","last_clean":"2024-03-30T14:56:12.372957+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T14:56:12.372957+0000","last_undegraded":"2024-03-30T14:56:12.372957+0000","last_fullsized":"2024-03-30T14:56:12.372957+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.387017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.387017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.976653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185507+0000","last_change":"2024-03-30T14:54:39.329805+0000","last_active":"2024-03-30T14:56:12.185507+0000","last_peered":"2024-03-30T14:56:12.185507+0000","last_clean":"2024-03-30T14:56:12.185507+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T14:56:12.185507+0000","last_undegraded":"2024-03-30T14:56:12.185507+0000","last_fullsized":"2024-03-30T14:56:12.185507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":55,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372521+0000","last_change":"2024-03-30T14:55:49.394236+0000","last_active":"2024-03-30T14:56:12.372521+0000","last_peered":"2024-03-30T14:56:12.372521+0000","last_clean":"2024-03-30T14:56:12.372521+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T14:56:12.372521+0000","last_undegraded":"2024-03-30T14:56:12.372521+0000","last_fullsized":"2024-03-30T14:56:12.372521+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.394138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.394138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.971320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531993+0000","last_change":"2024-03-30T14:56:09.083644+0000","last_active":"2024-03-30T14:56:12.531993+0000","last_peered":"2024-03-30T14:56:12.531993+0000","last_clean":"2024-03-30T14:56:12.531993+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T14:56:12.531993+0000","last_undegraded":"2024-03-30T14:56:12.531993+0000","last_fullsized":"2024-03-30T14:56:12.531993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.083567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.083567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:36.418282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373039+0000","last_change":"2024-03-30T14:55:59.608083+0000","last_active":"2024-03-30T14:56:12.373039+0000","last_peered":"2024-03-30T14:56:12.373039+0000","last_clean":"2024-03-30T14:56:12.373039+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T14:56:12.373039+0000","last_undegraded":"2024-03-30T14:56:12.373039+0000","last_fullsized":"2024-03-30T14:56:12.373039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.607983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.607983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.338389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185683+0000","last_change":"2024-03-30T14:54:58.404856+0000","last_active":"2024-03-30T14:56:12.185683+0000","last_peered":"2024-03-30T14:56:12.185683+0000","last_clean":"2024-03-30T14:56:12.185683+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T14:56:12.185683+0000","last_undegraded":"2024-03-30T14:56:12.185683+0000","last_fullsized":"2024-03-30T14:56:12.185683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186044+0000","last_change":"2024-03-30T14:54:43.349041+0000","last_active":"2024-03-30T14:56:12.186044+0000","last_peered":"2024-03-30T14:56:12.186044+0000","last_clean":"2024-03-30T14:56:12.186044+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T14:56:12.186044+0000","last_undegraded":"2024-03-30T14:56:12.186044+0000","last_fullsized":"2024-03-30T14:56:12.186044+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541470+0000","last_change":"2024-03-30T14:54:41.347727+0000","last_active":"2024-03-30T14:56:12.541470+0000","last_peered":"2024-03-30T14:56:12.541470+0000","last_clean":"2024-03-30T14:56:12.541470+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T14:56:12.541470+0000","last_undegraded":"2024-03-30T14:56:12.541470+0000","last_fullsized":"2024-03-30T14:56:12.541470+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185927+0000","last_change":"2024-03-30T14:55:42.710206+0000","last_active":"2024-03-30T14:56:12.185927+0000","last_peered":"2024-03-30T14:56:12.185927+0000","last_clean":"2024-03-30T14:56:12.185927+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T14:56:12.185927+0000","last_undegraded":"2024-03-30T14:56:12.185927+0000","last_fullsized":"2024-03-30T14:56:12.185927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.710118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.710118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.005945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"76'15","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531674+0000","last_change":"2024-03-30T14:55:39.803863+0000","last_active":"2024-03-30T14:56:12.531674+0000","last_peered":"2024-03-30T14:56:12.531674+0000","last_clean":"2024-03-30T14:56:12.531674+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T14:56:12.531674+0000","last_undegraded":"2024-03-30T14:56:12.531674+0000","last_fullsized":"2024-03-30T14:56:12.531674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:55:39.803752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:39.803752+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.649577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368454+0000","last_change":"2024-03-30T14:56:11.760211+0000","last_active":"2024-03-30T14:56:12.368454+0000","last_peered":"2024-03-30T14:56:12.368454+0000","last_clean":"2024-03-30T14:56:12.368454+0000","last_became_active":"2024-03-30T14:54:43.354397+0000","last_became_peered":"2024-03-30T14:54:43.354397+0000","last_unstale":"2024-03-30T14:56:12.368454+0000","last_undegraded":"2024-03-30T14:56:12.368454+0000","last_fullsized":"2024-03-30T14:56:12.368454+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:11.760085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:11.760085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:36.986177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185748+0000","last_change":"2024-03-30T14:54:41.348874+0000","last_active":"2024-03-30T14:56:12.185748+0000","last_peered":"2024-03-30T14:56:12.185748+0000","last_clean":"2024-03-30T14:56:12.185748+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T14:56:12.185748+0000","last_undegraded":"2024-03-30T14:56:12.185748+0000","last_fullsized":"2024-03-30T14:56:12.185748+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185565+0000","last_change":"2024-03-30T14:54:37.474896+0000","last_active":"2024-03-30T14:56:12.185565+0000","last_peered":"2024-03-30T14:56:12.185565+0000","last_clean":"2024-03-30T14:56:12.185565+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T14:56:12.185565+0000","last_undegraded":"2024-03-30T14:56:12.185565+0000","last_fullsized":"2024-03-30T14:56:12.185565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368292+0000","last_change":"2024-03-30T14:56:01.665638+0000","last_active":"2024-03-30T14:56:12.368292+0000","last_peered":"2024-03-30T14:56:12.368292+0000","last_clean":"2024-03-30T14:56:12.368292+0000","last_became_active":"2024-03-30T14:54:39.338927+0000","last_became_peered":"2024-03-30T14:54:39.338927+0000","last_unstale":"2024-03-30T14:56:12.368292+0000","last_undegraded":"2024-03-30T14:56:12.368292+0000","last_fullsized":"2024-03-30T14:56:12.368292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:56:01.665505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.665505+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.704128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"76'1","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532417+0000","last_change":"2024-03-30T14:54:43.354908+0000","last_active":"2024-03-30T14:56:12.532417+0000","last_peered":"2024-03-30T14:56:12.532417+0000","last_clean":"2024-03-30T14:56:12.532417+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T14:56:12.532417+0000","last_undegraded":"2024-03-30T14:56:12.532417+0000","last_fullsized":"2024-03-30T14:56:12.532417+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532778+0000","last_change":"2024-03-30T14:55:58.120425+0000","last_active":"2024-03-30T14:56:12.532778+0000","last_peered":"2024-03-30T14:56:12.532778+0000","last_clean":"2024-03-30T14:56:12.532778+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T14:56:12.532778+0000","last_undegraded":"2024-03-30T14:56:12.532778+0000","last_fullsized":"2024-03-30T14:56:12.532778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:58.120314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:58.120314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:11.131781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541294+0000","last_change":"2024-03-30T14:54:37.324901+0000","last_active":"2024-03-30T14:56:12.541294+0000","last_peered":"2024-03-30T14:56:12.541294+0000","last_clean":"2024-03-30T14:56:12.541294+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T14:56:12.541294+0000","last_undegraded":"2024-03-30T14:56:12.541294+0000","last_fullsized":"2024-03-30T14:56:12.541294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"76'10","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186883+0000","last_change":"2024-03-30T14:54:39.642246+0000","last_active":"2024-03-30T14:56:12.186883+0000","last_peered":"2024-03-30T14:56:12.186883+0000","last_clean":"2024-03-30T14:56:12.186883+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T14:56:12.186883+0000","last_undegraded":"2024-03-30T14:56:12.186883+0000","last_fullsized":"2024-03-30T14:56:12.186883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541634+0000","last_change":"2024-03-30T14:55:49.280406+0000","last_active":"2024-03-30T14:56:12.541634+0000","last_peered":"2024-03-30T14:56:12.541634+0000","last_clean":"2024-03-30T14:56:12.541634+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T14:56:12.541634+0000","last_undegraded":"2024-03-30T14:56:12.541634+0000","last_fullsized":"2024-03-30T14:56:12.541634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.280327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.280327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.497970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.187124+0000","last_change":"2024-03-30T14:54:41.347270+0000","last_active":"2024-03-30T14:56:12.187124+0000","last_peered":"2024-03-30T14:56:12.187124+0000","last_clean":"2024-03-30T14:56:12.187124+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T14:56:12.187124+0000","last_undegraded":"2024-03-30T14:56:12.187124+0000","last_fullsized":"2024-03-30T14:56:12.187124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367852+0000","last_change":"2024-03-30T14:54:37.321125+0000","last_active":"2024-03-30T14:56:12.367852+0000","last_peered":"2024-03-30T14:56:12.367852+0000","last_clean":"2024-03-30T14:56:12.367852+0000","last_became_active":"2024-03-30T14:54:37.320726+0000","last_became_peered":"2024-03-30T14:54:37.320726+0000","last_unstale":"2024-03-30T14:56:12.367852+0000","last_undegraded":"2024-03-30T14:56:12.367852+0000","last_fullsized":"2024-03-30T14:56:12.367852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532626+0000","last_change":"2024-03-30T14:56:12.032133+0000","last_active":"2024-03-30T14:56:12.532626+0000","last_peered":"2024-03-30T14:56:12.532626+0000","last_clean":"2024-03-30T14:56:12.532626+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T14:56:12.532626+0000","last_undegraded":"2024-03-30T14:56:12.532626+0000","last_fullsized":"2024-03-30T14:56:12.532626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.032038+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:37.869855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531933+0000","last_change":"2024-03-30T14:54:43.352814+0000","last_active":"2024-03-30T14:56:12.531933+0000","last_peered":"2024-03-30T14:56:12.531933+0000","last_clean":"2024-03-30T14:56:12.531933+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T14:56:12.531933+0000","last_undegraded":"2024-03-30T14:56:12.531933+0000","last_fullsized":"2024-03-30T14:56:12.531933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531572+0000","last_change":"2024-03-30T14:55:53.203651+0000","last_active":"2024-03-30T14:56:12.531572+0000","last_peered":"2024-03-30T14:56:12.531572+0000","last_clean":"2024-03-30T14:56:12.531572+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T14:56:12.531572+0000","last_undegraded":"2024-03-30T14:56:12.531572+0000","last_fullsized":"2024-03-30T14:56:12.531572+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.203533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.203533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:56.173167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.374402+0000","last_change":"2024-03-30T14:56:12.374423+0000","last_active":"2024-03-30T14:56:12.374402+0000","last_peered":"2024-03-30T14:56:12.374402+0000","last_clean":"2024-03-30T14:56:12.374402+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T14:56:12.374402+0000","last_undegraded":"2024-03-30T14:56:12.374402+0000","last_fullsized":"2024-03-30T14:56:12.374402+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:12.374338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.374338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.427721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367866+0000","last_change":"2024-03-30T14:56:03.643626+0000","last_active":"2024-03-30T14:56:12.367866+0000","last_peered":"2024-03-30T14:56:12.367866+0000","last_clean":"2024-03-30T14:56:12.367866+0000","last_became_active":"2024-03-30T14:54:39.338462+0000","last_became_peered":"2024-03-30T14:54:39.338462+0000","last_unstale":"2024-03-30T14:56:12.367866+0000","last_undegraded":"2024-03-30T14:56:12.367866+0000","last_fullsized":"2024-03-30T14:56:12.367866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:56:03.643494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:03.643494+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.936270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367977+0000","last_change":"2024-03-30T14:54:43.361222+0000","last_active":"2024-03-30T14:56:12.367977+0000","last_peered":"2024-03-30T14:56:12.367977+0000","last_clean":"2024-03-30T14:56:12.367977+0000","last_became_active":"2024-03-30T14:54:43.360837+0000","last_became_peered":"2024-03-30T14:54:43.360837+0000","last_unstale":"2024-03-30T14:56:12.367977+0000","last_undegraded":"2024-03-30T14:56:12.367977+0000","last_fullsized":"2024-03-30T14:56:12.367977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532329+0000","last_change":"2024-03-30T14:54:41.354167+0000","last_active":"2024-03-30T14:56:12.532329+0000","last_peered":"2024-03-30T14:56:12.532329+0000","last_clean":"2024-03-30T14:56:12.532329+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T14:56:12.532329+0000","last_undegraded":"2024-03-30T14:56:12.532329+0000","last_fullsized":"2024-03-30T14:56:12.532329+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372973+0000","last_change":"2024-03-30T14:54:37.331035+0000","last_active":"2024-03-30T14:56:12.372973+0000","last_peered":"2024-03-30T14:56:12.372973+0000","last_clean":"2024-03-30T14:56:12.372973+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T14:56:12.372973+0000","last_undegraded":"2024-03-30T14:56:12.372973+0000","last_fullsized":"2024-03-30T14:56:12.372973+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368094+0000","last_change":"2024-03-30T14:56:07.672900+0000","last_active":"2024-03-30T14:56:12.368094+0000","last_peered":"2024-03-30T14:56:12.368094+0000","last_clean":"2024-03-30T14:56:12.368094+0000","last_became_active":"2024-03-30T14:54:39.641142+0000","last_became_peered":"2024-03-30T14:54:39.641142+0000","last_unstale":"2024-03-30T14:56:12.368094+0000","last_undegraded":"2024-03-30T14:56:12.368094+0000","last_fullsized":"2024-03-30T14:56:12.368094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:56:07.672785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.672785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.755665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:13.401189+0000","last_change":"2024-03-30T14:56:13.401210+0000","last_active":"2024-03-30T14:56:13.401189+0000","last_peered":"2024-03-30T14:56:13.401189+0000","last_clean":"2024-03-30T14:56:13.401189+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T14:56:13.401189+0000","last_undegraded":"2024-03-30T14:56:13.401189+0000","last_fullsized":"2024-03-30T14:56:13.401189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:13.401132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:13.401132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.048637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368293+0000","last_change":"2024-03-30T14:54:41.347572+0000","last_active":"2024-03-30T14:56:12.368293+0000","last_peered":"2024-03-30T14:56:12.368293+0000","last_clean":"2024-03-30T14:56:12.368293+0000","last_became_active":"2024-03-30T14:54:41.347229+0000","last_became_peered":"2024-03-30T14:54:41.347229+0000","last_unstale":"2024-03-30T14:56:12.368293+0000","last_undegraded":"2024-03-30T14:56:12.368293+0000","last_fullsized":"2024-03-30T14:56:12.368293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532844+0000","last_change":"2024-03-30T14:56:03.093369+0000","last_active":"2024-03-30T14:56:12.532844+0000","last_peered":"2024-03-30T14:56:12.532844+0000","last_clean":"2024-03-30T14:56:12.532844+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T14:56:12.532844+0000","last_undegraded":"2024-03-30T14:56:12.532844+0000","last_fullsized":"2024-03-30T14:56:12.532844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:56:03.093218+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:07.894705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541665+0000","last_change":"2024-03-30T14:54:39.339910+0000","last_active":"2024-03-30T14:56:12.541665+0000","last_peered":"2024-03-30T14:56:12.541665+0000","last_clean":"2024-03-30T14:56:12.541665+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T14:56:12.541665+0000","last_undegraded":"2024-03-30T14:56:12.541665+0000","last_fullsized":"2024-03-30T14:56:12.541665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367824+0000","last_change":"2024-03-30T14:54:43.360542+0000","last_active":"2024-03-30T14:56:12.367824+0000","last_peered":"2024-03-30T14:56:12.367824+0000","last_clean":"2024-03-30T14:56:12.367824+0000","last_became_active":"2024-03-30T14:54:43.360215+0000","last_became_peered":"2024-03-30T14:54:43.360215+0000","last_unstale":"2024-03-30T14:56:12.367824+0000","last_undegraded":"2024-03-30T14:56:12.367824+0000","last_fullsized":"2024-03-30T14:56:12.367824+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541466+0000","last_change":"2024-03-30T14:55:53.260188+0000","last_active":"2024-03-30T14:56:12.541466+0000","last_peered":"2024-03-30T14:56:12.541466+0000","last_clean":"2024-03-30T14:56:12.541466+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T14:56:12.541466+0000","last_undegraded":"2024-03-30T14:56:12.541466+0000","last_fullsized":"2024-03-30T14:56:12.541466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:53.260101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.260101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.452479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185911+0000","last_change":"2024-03-30T14:56:00.482961+0000","last_active":"2024-03-30T14:56:12.185911+0000","last_peered":"2024-03-30T14:56:12.185911+0000","last_clean":"2024-03-30T14:56:12.185911+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T14:56:12.185911+0000","last_undegraded":"2024-03-30T14:56:12.185911+0000","last_fullsized":"2024-03-30T14:56:12.185911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:00.482861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.482861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.363058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"76'11","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185782+0000","last_change":"2024-03-30T14:54:39.334527+0000","last_active":"2024-03-30T14:56:12.185782+0000","last_peered":"2024-03-30T14:56:12.185782+0000","last_clean":"2024-03-30T14:56:12.185782+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T14:56:12.185782+0000","last_undegraded":"2024-03-30T14:56:12.185782+0000","last_fullsized":"2024-03-30T14:56:12.185782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"76'1","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186614+0000","last_change":"2024-03-30T14:54:43.354400+0000","last_active":"2024-03-30T14:56:12.186614+0000","last_peered":"2024-03-30T14:56:12.186614+0000","last_clean":"2024-03-30T14:56:12.186614+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T14:56:12.186614+0000","last_undegraded":"2024-03-30T14:56:12.186614+0000","last_fullsized":"2024-03-30T14:56:12.186614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:12.205644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373038+0000","last_change":"2024-03-30T14:54:41.350464+0000","last_active":"2024-03-30T14:56:12.373038+0000","last_peered":"2024-03-30T14:56:12.373038+0000","last_clean":"2024-03-30T14:56:12.373038+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T14:56:12.373038+0000","last_undegraded":"2024-03-30T14:56:12.373038+0000","last_fullsized":"2024-03-30T14:56:12.373038+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533384+0000","last_change":"2024-03-30T14:54:37.474723+0000","last_active":"2024-03-30T14:56:12.533384+0000","last_peered":"2024-03-30T14:56:12.533384+0000","last_clean":"2024-03-30T14:56:12.533384+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T14:56:12.533384+0000","last_undegraded":"2024-03-30T14:56:12.533384+0000","last_fullsized":"2024-03-30T14:56:12.533384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368098+0000","last_change":"2024-03-30T14:55:53.742586+0000","last_active":"2024-03-30T14:56:12.368098+0000","last_peered":"2024-03-30T14:56:12.368098+0000","last_clean":"2024-03-30T14:56:12.368098+0000","last_became_active":"2024-03-30T14:54:39.642132+0000","last_became_peered":"2024-03-30T14:54:39.642132+0000","last_unstale":"2024-03-30T14:56:12.368098+0000","last_undegraded":"2024-03-30T14:56:12.368098+0000","last_fullsized":"2024-03-30T14:56:12.368098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:53.742492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:53.742492+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.865684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186115+0000","last_change":"2024-03-30T14:54:43.349809+0000","last_active":"2024-03-30T14:56:12.186115+0000","last_peered":"2024-03-30T14:56:12.186115+0000","last_clean":"2024-03-30T14:56:12.186115+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T14:56:12.186115+0000","last_undegraded":"2024-03-30T14:56:12.186115+0000","last_fullsized":"2024-03-30T14:56:12.186115+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"76'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368434+0000","last_change":"2024-03-30T14:54:41.346770+0000","last_active":"2024-03-30T14:56:12.368434+0000","last_peered":"2024-03-30T14:56:12.368434+0000","last_clean":"2024-03-30T14:56:12.368434+0000","last_became_active":"2024-03-30T14:54:41.346418+0000","last_became_peered":"2024-03-30T14:54:41.346418+0000","last_unstale":"2024-03-30T14:56:12.368434+0000","last_undegraded":"2024-03-30T14:56:12.368434+0000","last_fullsized":"2024-03-30T14:56:12.368434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185158+0000","last_change":"2024-03-30T14:56:02.515803+0000","last_active":"2024-03-30T14:56:12.185158+0000","last_peered":"2024-03-30T14:56:12.185158+0000","last_clean":"2024-03-30T14:56:12.185158+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T14:56:12.185158+0000","last_undegraded":"2024-03-30T14:56:12.185158+0000","last_fullsized":"2024-03-30T14:56:12.185158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.515709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.515709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.607511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373154+0000","last_change":"2024-03-30T14:54:39.644472+0000","last_active":"2024-03-30T14:56:12.373154+0000","last_peered":"2024-03-30T14:56:12.373154+0000","last_clean":"2024-03-30T14:56:12.373154+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T14:56:12.373154+0000","last_undegraded":"2024-03-30T14:56:12.373154+0000","last_fullsized":"2024-03-30T14:56:12.373154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531860+0000","last_change":"2024-03-30T14:55:48.721601+0000","last_active":"2024-03-30T14:56:12.531860+0000","last_peered":"2024-03-30T14:56:12.531860+0000","last_clean":"2024-03-30T14:56:12.531860+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T14:56:12.531860+0000","last_undegraded":"2024-03-30T14:56:12.531860+0000","last_fullsized":"2024-03-30T14:56:12.531860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.721477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.721477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:58.996126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368734+0000","last_change":"2024-03-30T14:56:09.706800+0000","last_active":"2024-03-30T14:56:12.368734+0000","last_peered":"2024-03-30T14:56:12.368734+0000","last_clean":"2024-03-30T14:56:12.368734+0000","last_became_active":"2024-03-30T14:54:41.345543+0000","last_became_peered":"2024-03-30T14:54:41.345543+0000","last_unstale":"2024-03-30T14:56:12.368734+0000","last_undegraded":"2024-03-30T14:56:12.368734+0000","last_fullsized":"2024-03-30T14:56:12.368734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.706719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.706719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:30.142928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532049+0000","last_change":"2024-03-30T14:55:48.068870+0000","last_active":"2024-03-30T14:56:12.532049+0000","last_peered":"2024-03-30T14:56:12.532049+0000","last_clean":"2024-03-30T14:56:12.532049+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T14:56:12.532049+0000","last_undegraded":"2024-03-30T14:56:12.532049+0000","last_fullsized":"2024-03-30T14:56:12.532049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:48.068767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.068767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:16.799841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186976+0000","last_change":"2024-03-30T14:54:39.642222+0000","last_active":"2024-03-30T14:56:12.186976+0000","last_peered":"2024-03-30T14:56:12.186976+0000","last_clean":"2024-03-30T14:56:12.186976+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T14:56:12.186976+0000","last_undegraded":"2024-03-30T14:56:12.186976+0000","last_fullsized":"2024-03-30T14:56:12.186976+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532737+0000","last_change":"2024-03-30T14:54:43.357877+0000","last_active":"2024-03-30T14:56:12.532737+0000","last_peered":"2024-03-30T14:56:12.532737+0000","last_clean":"2024-03-30T14:56:12.532737+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T14:56:12.532737+0000","last_undegraded":"2024-03-30T14:56:12.532737+0000","last_fullsized":"2024-03-30T14:56:12.532737+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373083+0000","last_change":"2024-03-30T14:56:09.561375+0000","last_active":"2024-03-30T14:56:12.373083+0000","last_peered":"2024-03-30T14:56:12.373083+0000","last_clean":"2024-03-30T14:56:12.373083+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T14:56:12.373083+0000","last_undegraded":"2024-03-30T14:56:12.373083+0000","last_fullsized":"2024-03-30T14:56:12.373083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:09.561303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.561303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.644838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531520+0000","last_change":"2024-03-30T14:55:50.191458+0000","last_active":"2024-03-30T14:56:12.531520+0000","last_peered":"2024-03-30T14:56:12.531520+0000","last_clean":"2024-03-30T14:56:12.531520+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T14:56:12.531520+0000","last_undegraded":"2024-03-30T14:56:12.531520+0000","last_fullsized":"2024-03-30T14:56:12.531520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.191353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.191353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:00.197326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368757+0000","last_change":"2024-03-30T14:55:48.827002+0000","last_active":"2024-03-30T14:56:12.368757+0000","last_peered":"2024-03-30T14:56:12.368757+0000","last_clean":"2024-03-30T14:56:12.368757+0000","last_became_active":"2024-03-30T14:54:39.643439+0000","last_became_peered":"2024-03-30T14:54:39.643439+0000","last_unstale":"2024-03-30T14:56:12.368757+0000","last_undegraded":"2024-03-30T14:56:12.368757+0000","last_fullsized":"2024-03-30T14:56:12.368757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:55:48.826912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.826912+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.602167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372779+0000","last_change":"2024-03-30T14:54:43.352897+0000","last_active":"2024-03-30T14:56:12.372779+0000","last_peered":"2024-03-30T14:56:12.372779+0000","last_clean":"2024-03-30T14:56:12.372779+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:56:12.372779+0000","last_undegraded":"2024-03-30T14:56:12.372779+0000","last_fullsized":"2024-03-30T14:56:12.372779+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533285+0000","last_change":"2024-03-30T14:54:37.330956+0000","last_active":"2024-03-30T14:56:12.533285+0000","last_peered":"2024-03-30T14:56:12.533285+0000","last_clean":"2024-03-30T14:56:12.533285+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T14:56:12.533285+0000","last_undegraded":"2024-03-30T14:56:12.533285+0000","last_fullsized":"2024-03-30T14:56:12.533285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372542+0000","last_change":"2024-03-30T14:54:39.643618+0000","last_active":"2024-03-30T14:56:12.372542+0000","last_peered":"2024-03-30T14:56:12.372542+0000","last_clean":"2024-03-30T14:56:12.372542+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T14:56:12.372542+0000","last_undegraded":"2024-03-30T14:56:12.372542+0000","last_fullsized":"2024-03-30T14:56:12.372542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532343+0000","last_change":"2024-03-30T14:54:41.352889+0000","last_active":"2024-03-30T14:56:12.532343+0000","last_peered":"2024-03-30T14:56:12.532343+0000","last_clean":"2024-03-30T14:56:12.532343+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T14:56:12.532343+0000","last_undegraded":"2024-03-30T14:56:12.532343+0000","last_fullsized":"2024-03-30T14:56:12.532343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.367969+0000","last_change":"2024-03-30T14:54:43.358959+0000","last_active":"2024-03-30T14:56:12.367969+0000","last_peered":"2024-03-30T14:56:12.367969+0000","last_clean":"2024-03-30T14:56:12.367969+0000","last_became_active":"2024-03-30T14:54:43.358552+0000","last_became_peered":"2024-03-30T14:54:43.358552+0000","last_unstale":"2024-03-30T14:56:12.367969+0000","last_undegraded":"2024-03-30T14:56:12.367969+0000","last_fullsized":"2024-03-30T14:56:12.367969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532909+0000","last_change":"2024-03-30T14:54:37.328695+0000","last_active":"2024-03-30T14:56:12.532909+0000","last_peered":"2024-03-30T14:56:12.532909+0000","last_clean":"2024-03-30T14:56:12.532909+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T14:56:12.532909+0000","last_undegraded":"2024-03-30T14:56:12.532909+0000","last_fullsized":"2024-03-30T14:56:12.532909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185529+0000","last_change":"2024-03-30T14:55:56.431786+0000","last_active":"2024-03-30T14:56:12.185529+0000","last_peered":"2024-03-30T14:56:12.185529+0000","last_clean":"2024-03-30T14:56:12.185529+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T14:56:12.185529+0000","last_undegraded":"2024-03-30T14:56:12.185529+0000","last_fullsized":"2024-03-30T14:56:12.185529+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:56.431647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.431647+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.459883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531467+0000","last_change":"2024-03-30T14:56:06.652426+0000","last_active":"2024-03-30T14:56:12.531467+0000","last_peered":"2024-03-30T14:56:12.531467+0000","last_clean":"2024-03-30T14:56:12.531467+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T14:56:12.531467+0000","last_undegraded":"2024-03-30T14:56:12.531467+0000","last_fullsized":"2024-03-30T14:56:12.531467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:56:06.652343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.926692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531924+0000","last_change":"2024-03-30T14:54:43.356146+0000","last_active":"2024-03-30T14:56:12.531924+0000","last_peered":"2024-03-30T14:56:12.531924+0000","last_clean":"2024-03-30T14:56:12.531924+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T14:56:12.531924+0000","last_undegraded":"2024-03-30T14:56:12.531924+0000","last_fullsized":"2024-03-30T14:56:12.531924+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531826+0000","last_change":"2024-03-30T14:56:02.650786+0000","last_active":"2024-03-30T14:56:12.531826+0000","last_peered":"2024-03-30T14:56:12.531826+0000","last_clean":"2024-03-30T14:56:12.531826+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T14:56:12.531826+0000","last_undegraded":"2024-03-30T14:56:12.531826+0000","last_fullsized":"2024-03-30T14:56:12.531826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:02.650642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:02.650642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:08.689802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533083+0000","last_change":"2024-03-30T14:54:37.329066+0000","last_active":"2024-03-30T14:56:12.533083+0000","last_peered":"2024-03-30T14:56:12.533083+0000","last_clean":"2024-03-30T14:56:12.533083+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T14:56:12.533083+0000","last_undegraded":"2024-03-30T14:56:12.533083+0000","last_fullsized":"2024-03-30T14:56:12.533083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":120,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532312+0000","last_change":"2024-03-30T14:55:52.189892+0000","last_active":"2024-03-30T14:56:12.532312+0000","last_peered":"2024-03-30T14:56:12.532312+0000","last_clean":"2024-03-30T14:56:12.532312+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T14:56:12.532312+0000","last_undegraded":"2024-03-30T14:56:12.532312+0000","last_fullsized":"2024-03-30T14:56:12.532312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:55:52.189742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.189742+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.639688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.184884+0000","last_change":"2024-03-30T14:54:43.353107+0000","last_active":"2024-03-30T14:56:12.184884+0000","last_peered":"2024-03-30T14:56:12.184884+0000","last_clean":"2024-03-30T14:56:12.184884+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T14:56:12.184884+0000","last_undegraded":"2024-03-30T14:56:12.184884+0000","last_fullsized":"2024-03-30T14:56:12.184884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531612+0000","last_change":"2024-03-30T14:55:56.679624+0000","last_active":"2024-03-30T14:56:12.531612+0000","last_peered":"2024-03-30T14:56:12.531612+0000","last_clean":"2024-03-30T14:56:12.531612+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T14:56:12.531612+0000","last_undegraded":"2024-03-30T14:56:12.531612+0000","last_fullsized":"2024-03-30T14:56:12.531612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.679531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.679531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.763850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372989+0000","last_change":"2024-03-30T14:55:37.491888+0000","last_active":"2024-03-30T14:56:12.372989+0000","last_peered":"2024-03-30T14:56:12.372989+0000","last_clean":"2024-03-30T14:56:12.372989+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T14:56:12.372989+0000","last_undegraded":"2024-03-30T14:56:12.372989+0000","last_fullsized":"2024-03-30T14:56:12.372989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:55:37.491750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.952763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185730+0000","last_change":"2024-03-30T14:55:48.585645+0000","last_active":"2024-03-30T14:56:12.185730+0000","last_peered":"2024-03-30T14:56:12.185730+0000","last_clean":"2024-03-30T14:56:12.185730+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T14:56:12.185730+0000","last_undegraded":"2024-03-30T14:56:12.185730+0000","last_fullsized":"2024-03-30T14:56:12.185730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:55:48.585557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:48.585557+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.155340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368486+0000","last_change":"2024-03-30T14:55:59.683242+0000","last_active":"2024-03-30T14:56:12.368486+0000","last_peered":"2024-03-30T14:56:12.368486+0000","last_clean":"2024-03-30T14:56:12.368486+0000","last_became_active":"2024-03-30T14:54:43.356453+0000","last_became_peered":"2024-03-30T14:54:43.356453+0000","last_unstale":"2024-03-30T14:56:12.368486+0000","last_undegraded":"2024-03-30T14:56:12.368486+0000","last_fullsized":"2024-03-30T14:56:12.368486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:59.683158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:59.683158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.228244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372651+0000","last_change":"2024-03-30T14:54:40.343883+0000","last_active":"2024-03-30T14:56:12.372651+0000","last_peered":"2024-03-30T14:56:12.372651+0000","last_clean":"2024-03-30T14:56:12.372651+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T14:56:12.372651+0000","last_undegraded":"2024-03-30T14:56:12.372651+0000","last_fullsized":"2024-03-30T14:56:12.372651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:54:35.298668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185590+0000","last_change":"2024-03-30T14:56:11.414503+0000","last_active":"2024-03-30T14:56:12.185590+0000","last_peered":"2024-03-30T14:56:12.185590+0000","last_clean":"2024-03-30T14:56:12.185590+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T14:56:12.185590+0000","last_undegraded":"2024-03-30T14:56:12.185590+0000","last_fullsized":"2024-03-30T14:56:12.185590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:11.414361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:11.414361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:12.225419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372772+0000","last_change":"2024-03-30T14:56:04.590554+0000","last_active":"2024-03-30T14:56:12.372772+0000","last_peered":"2024-03-30T14:56:12.372772+0000","last_clean":"2024-03-30T14:56:12.372772+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T14:56:12.372772+0000","last_undegraded":"2024-03-30T14:56:12.372772+0000","last_fullsized":"2024-03-30T14:56:12.372772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:56:04.590400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.590400+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.294040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.187265+0000","last_change":"2024-03-30T14:56:10.441560+0000","last_active":"2024-03-30T14:56:12.187265+0000","last_peered":"2024-03-30T14:56:12.187265+0000","last_clean":"2024-03-30T14:56:12.187265+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T14:56:12.187265+0000","last_undegraded":"2024-03-30T14:56:12.187265+0000","last_fullsized":"2024-03-30T14:56:12.187265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:10.441422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:10.441422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.928811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533159+0000","last_change":"2024-03-30T14:55:56.007233+0000","last_active":"2024-03-30T14:56:12.533159+0000","last_peered":"2024-03-30T14:56:12.533159+0000","last_clean":"2024-03-30T14:56:12.533159+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T14:56:12.533159+0000","last_undegraded":"2024-03-30T14:56:12.533159+0000","last_fullsized":"2024-03-30T14:56:12.533159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:56.007133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:56.007133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.546510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541352+0000","last_change":"2024-03-30T14:56:04.301702+0000","last_active":"2024-03-30T14:56:12.541352+0000","last_peered":"2024-03-30T14:56:12.541352+0000","last_clean":"2024-03-30T14:56:12.541352+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T14:56:12.541352+0000","last_undegraded":"2024-03-30T14:56:12.541352+0000","last_fullsized":"2024-03-30T14:56:12.541352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.301590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:33.597560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531356+0000","last_change":"2024-03-30T14:55:49.700155+0000","last_active":"2024-03-30T14:56:12.531356+0000","last_peered":"2024-03-30T14:56:12.531356+0000","last_clean":"2024-03-30T14:56:12.531356+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T14:56:12.531356+0000","last_undegraded":"2024-03-30T14:56:12.531356+0000","last_fullsized":"2024-03-30T14:56:12.531356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.700022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.949979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"76'13","reported_seq":91,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541915+0000","last_change":"2024-03-30T14:56:00.305512+0000","last_active":"2024-03-30T14:56:12.541915+0000","last_peered":"2024-03-30T14:56:12.541915+0000","last_clean":"2024-03-30T14:56:12.541915+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T14:56:12.541915+0000","last_undegraded":"2024-03-30T14:56:12.541915+0000","last_fullsized":"2024-03-30T14:56:12.541915+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'13","last_scrub_stamp":"2024-03-30T14:56:00.305404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:00.305404+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.155134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531988+0000","last_change":"2024-03-30T14:55:57.693792+0000","last_active":"2024-03-30T14:56:12.531988+0000","last_peered":"2024-03-30T14:56:12.531988+0000","last_clean":"2024-03-30T14:56:12.531988+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T14:56:12.531988+0000","last_undegraded":"2024-03-30T14:56:12.531988+0000","last_fullsized":"2024-03-30T14:56:12.531988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.693703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.693703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.637711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373035+0000","last_change":"2024-03-30T14:54:43.356807+0000","last_active":"2024-03-30T14:56:12.373035+0000","last_peered":"2024-03-30T14:56:12.373035+0000","last_clean":"2024-03-30T14:56:12.373035+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T14:56:12.373035+0000","last_undegraded":"2024-03-30T14:56:12.373035+0000","last_fullsized":"2024-03-30T14:56:12.373035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532048+0000","last_change":"2024-03-30T14:55:05.131524+0000","last_active":"2024-03-30T14:56:12.532048+0000","last_peered":"2024-03-30T14:56:12.532048+0000","last_clean":"2024-03-30T14:56:12.532048+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T14:56:12.532048+0000","last_undegraded":"2024-03-30T14:56:12.532048+0000","last_fullsized":"2024-03-30T14:56:12.532048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'117","last_scrub_stamp":"2024-03-30T14:55:05.131440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:55:05.131440+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:33.848087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368102+0000","last_change":"2024-03-30T14:54:37.324739+0000","last_active":"2024-03-30T14:56:12.368102+0000","last_peered":"2024-03-30T14:56:12.368102+0000","last_clean":"2024-03-30T14:56:12.368102+0000","last_became_active":"2024-03-30T14:54:37.324423+0000","last_became_peered":"2024-03-30T14:54:37.324423+0000","last_unstale":"2024-03-30T14:56:12.368102+0000","last_undegraded":"2024-03-30T14:56:12.368102+0000","last_fullsized":"2024-03-30T14:56:12.368102+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.533339+0000","last_change":"2024-03-30T14:56:07.057582+0000","last_active":"2024-03-30T14:56:12.533339+0000","last_peered":"2024-03-30T14:56:12.533339+0000","last_clean":"2024-03-30T14:56:12.533339+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T14:56:12.533339+0000","last_undegraded":"2024-03-30T14:56:12.533339+0000","last_fullsized":"2024-03-30T14:56:12.533339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T14:56:07.057479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.057479+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:09.119135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532574+0000","last_change":"2024-03-30T14:54:41.350344+0000","last_active":"2024-03-30T14:56:12.532574+0000","last_peered":"2024-03-30T14:56:12.532574+0000","last_clean":"2024-03-30T14:56:12.532574+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T14:56:12.532574+0000","last_undegraded":"2024-03-30T14:56:12.532574+0000","last_fullsized":"2024-03-30T14:56:12.532574+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:54:40.325599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541278+0000","last_change":"2024-03-30T14:54:43.357715+0000","last_active":"2024-03-30T14:56:12.541278+0000","last_peered":"2024-03-30T14:56:12.541278+0000","last_clean":"2024-03-30T14:56:12.541278+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T14:56:12.541278+0000","last_undegraded":"2024-03-30T14:56:12.541278+0000","last_fullsized":"2024-03-30T14:56:12.541278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185134+0000","last_change":"2024-03-30T14:54:37.326510+0000","last_active":"2024-03-30T14:56:12.185134+0000","last_peered":"2024-03-30T14:56:12.185134+0000","last_clean":"2024-03-30T14:56:12.185134+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T14:56:12.185134+0000","last_undegraded":"2024-03-30T14:56:12.185134+0000","last_fullsized":"2024-03-30T14:56:12.185134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531792+0000","last_change":"2024-03-30T14:56:07.116284+0000","last_active":"2024-03-30T14:56:12.531792+0000","last_peered":"2024-03-30T14:56:12.531792+0000","last_clean":"2024-03-30T14:56:12.531792+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T14:56:12.531792+0000","last_undegraded":"2024-03-30T14:56:12.531792+0000","last_fullsized":"2024-03-30T14:56:12.531792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:56:07.116206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.116206+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.531800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186713+0000","last_change":"2024-03-30T14:55:52.556360+0000","last_active":"2024-03-30T14:56:12.186713+0000","last_peered":"2024-03-30T14:56:12.186713+0000","last_clean":"2024-03-30T14:56:12.186713+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T14:56:12.186713+0000","last_undegraded":"2024-03-30T14:56:12.186713+0000","last_fullsized":"2024-03-30T14:56:12.186713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:55:52.556237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:52.556237+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:18.659172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373276+0000","last_change":"2024-03-30T14:55:57.530418+0000","last_active":"2024-03-30T14:56:12.373276+0000","last_peered":"2024-03-30T14:56:12.373276+0000","last_clean":"2024-03-30T14:56:12.373276+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T14:56:12.373276+0000","last_undegraded":"2024-03-30T14:56:12.373276+0000","last_fullsized":"2024-03-30T14:56:12.373276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:57.530310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:57.530310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:01.784600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541083+0000","last_change":"2024-03-30T14:56:01.265168+0000","last_active":"2024-03-30T14:56:12.541083+0000","last_peered":"2024-03-30T14:56:12.541083+0000","last_clean":"2024-03-30T14:56:12.541083+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T14:56:12.541083+0000","last_undegraded":"2024-03-30T14:56:12.541083+0000","last_fullsized":"2024-03-30T14:56:12.541083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:01.265084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:01.265084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:14.573818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541417+0000","last_change":"2024-03-30T14:55:42.232642+0000","last_active":"2024-03-30T14:56:12.541417+0000","last_peered":"2024-03-30T14:56:12.541417+0000","last_clean":"2024-03-30T14:56:12.541417+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T14:56:12.541417+0000","last_undegraded":"2024-03-30T14:56:12.541417+0000","last_fullsized":"2024-03-30T14:56:12.541417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:42.232525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:42.232525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:48.072521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"76'19","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186009+0000","last_change":"2024-03-30T14:54:39.642101+0000","last_active":"2024-03-30T14:56:12.186009+0000","last_peered":"2024-03-30T14:56:12.186009+0000","last_clean":"2024-03-30T14:56:12.186009+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T14:56:12.186009+0000","last_undegraded":"2024-03-30T14:56:12.186009+0000","last_fullsized":"2024-03-30T14:56:12.186009+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532335+0000","last_change":"2024-03-30T14:56:04.074991+0000","last_active":"2024-03-30T14:56:12.532335+0000","last_peered":"2024-03-30T14:56:12.532335+0000","last_clean":"2024-03-30T14:56:12.532335+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T14:56:12.532335+0000","last_undegraded":"2024-03-30T14:56:12.532335+0000","last_fullsized":"2024-03-30T14:56:12.532335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:04.074894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:04.074894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:24.283425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.184687+0000","last_change":"2024-03-30T14:54:43.355213+0000","last_active":"2024-03-30T14:56:12.184687+0000","last_peered":"2024-03-30T14:56:12.184687+0000","last_clean":"2024-03-30T14:56:12.184687+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T14:56:12.184687+0000","last_undegraded":"2024-03-30T14:56:12.184687+0000","last_fullsized":"2024-03-30T14:56:12.184687+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373058+0000","last_change":"2024-03-30T14:55:41.454468+0000","last_active":"2024-03-30T14:56:12.373058+0000","last_peered":"2024-03-30T14:56:12.373058+0000","last_clean":"2024-03-30T14:56:12.373058+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T14:56:12.373058+0000","last_undegraded":"2024-03-30T14:56:12.373058+0000","last_fullsized":"2024-03-30T14:56:12.373058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:55:41.454365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:04.526213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"76'10","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541561+0000","last_change":"2024-03-30T14:54:39.338666+0000","last_active":"2024-03-30T14:56:12.541561+0000","last_peered":"2024-03-30T14:56:12.541561+0000","last_clean":"2024-03-30T14:56:12.541561+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T14:56:12.541561+0000","last_undegraded":"2024-03-30T14:56:12.541561+0000","last_fullsized":"2024-03-30T14:56:12.541561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185637+0000","last_change":"2024-03-30T14:55:50.585912+0000","last_active":"2024-03-30T14:56:12.185637+0000","last_peered":"2024-03-30T14:56:12.185637+0000","last_clean":"2024-03-30T14:56:12.185637+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T14:56:12.185637+0000","last_undegraded":"2024-03-30T14:56:12.185637+0000","last_fullsized":"2024-03-30T14:56:12.185637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:50.585773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.585773+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.447085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532306+0000","last_change":"2024-03-30T14:55:49.029645+0000","last_active":"2024-03-30T14:56:12.532306+0000","last_peered":"2024-03-30T14:56:12.532306+0000","last_clean":"2024-03-30T14:56:12.532306+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T14:56:12.532306+0000","last_undegraded":"2024-03-30T14:56:12.532306+0000","last_fullsized":"2024-03-30T14:56:12.532306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:49.029548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:55:49.029548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:52.389450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"69'1","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186815+0000","last_change":"2024-03-30T14:54:37.326159+0000","last_active":"2024-03-30T14:56:12.186815+0000","last_peered":"2024-03-30T14:56:12.186815+0000","last_clean":"2024-03-30T14:56:12.186815+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T14:56:12.186815+0000","last_undegraded":"2024-03-30T14:56:12.186815+0000","last_fullsized":"2024-03-30T14:56:12.186815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185412+0000","last_change":"2024-03-30T14:55:46.303554+0000","last_active":"2024-03-30T14:56:12.185412+0000","last_peered":"2024-03-30T14:56:12.185412+0000","last_clean":"2024-03-30T14:56:12.185412+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T14:56:12.185412+0000","last_undegraded":"2024-03-30T14:56:12.185412+0000","last_fullsized":"2024-03-30T14:56:12.185412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:55:46.303412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.303412+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:55.425234+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"76'8","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368603+0000","last_change":"2024-03-30T14:55:55.692425+0000","last_active":"2024-03-30T14:56:12.368603+0000","last_peered":"2024-03-30T14:56:12.368603+0000","last_clean":"2024-03-30T14:56:12.368603+0000","last_became_active":"2024-03-30T14:54:41.353489+0000","last_became_peered":"2024-03-30T14:54:41.353489+0000","last_unstale":"2024-03-30T14:56:12.368603+0000","last_undegraded":"2024-03-30T14:56:12.368603+0000","last_fullsized":"2024-03-30T14:56:12.368603+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:55.692261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:55.692261+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.231384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185375+0000","last_change":"2024-03-30T14:54:43.352748+0000","last_active":"2024-03-30T14:56:12.185375+0000","last_peered":"2024-03-30T14:56:12.185375+0000","last_clean":"2024-03-30T14:56:12.185375+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T14:56:12.185375+0000","last_undegraded":"2024-03-30T14:56:12.185375+0000","last_fullsized":"2024-03-30T14:56:12.185375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"69'1","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368560+0000","last_change":"2024-03-30T14:54:37.323745+0000","last_active":"2024-03-30T14:56:12.368560+0000","last_peered":"2024-03-30T14:56:12.368560+0000","last_clean":"2024-03-30T14:56:12.368560+0000","last_became_active":"2024-03-30T14:54:37.323384+0000","last_became_peered":"2024-03-30T14:54:37.323384+0000","last_unstale":"2024-03-30T14:56:12.368560+0000","last_undegraded":"2024-03-30T14:56:12.368560+0000","last_fullsized":"2024-03-30T14:56:12.368560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368546+0000","last_change":"2024-03-30T14:56:10.715437+0000","last_active":"2024-03-30T14:56:12.368546+0000","last_peered":"2024-03-30T14:56:12.368546+0000","last_clean":"2024-03-30T14:56:12.368546+0000","last_became_active":"2024-03-30T14:54:39.642450+0000","last_became_peered":"2024-03-30T14:54:39.642450+0000","last_unstale":"2024-03-30T14:56:12.368546+0000","last_undegraded":"2024-03-30T14:56:12.368546+0000","last_fullsized":"2024-03-30T14:56:12.368546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:56:10.715316+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:26.611400+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185259+0000","last_change":"2024-03-30T14:55:54.381596+0000","last_active":"2024-03-30T14:56:12.185259+0000","last_peered":"2024-03-30T14:56:12.185259+0000","last_clean":"2024-03-30T14:56:12.185259+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T14:56:12.185259+0000","last_undegraded":"2024-03-30T14:56:12.185259+0000","last_fullsized":"2024-03-30T14:56:12.185259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:54.381448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:54.381448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:19.160867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.372869+0000","last_change":"2024-03-30T14:54:43.351363+0000","last_active":"2024-03-30T14:56:12.372869+0000","last_peered":"2024-03-30T14:56:12.372869+0000","last_clean":"2024-03-30T14:56:12.372869+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T14:56:12.372869+0000","last_undegraded":"2024-03-30T14:56:12.372869+0000","last_fullsized":"2024-03-30T14:56:12.372869+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186308+0000","last_change":"2024-03-30T14:56:07.479522+0000","last_active":"2024-03-30T14:56:12.186308+0000","last_peered":"2024-03-30T14:56:12.186308+0000","last_clean":"2024-03-30T14:56:12.186308+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T14:56:12.186308+0000","last_undegraded":"2024-03-30T14:56:12.186308+0000","last_fullsized":"2024-03-30T14:56:12.186308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:56:07.479430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:56:07.479430+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.427750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368275+0000","last_change":"2024-03-30T14:54:37.323396+0000","last_active":"2024-03-30T14:56:12.368275+0000","last_peered":"2024-03-30T14:56:12.368275+0000","last_clean":"2024-03-30T14:56:12.368275+0000","last_became_active":"2024-03-30T14:54:37.323084+0000","last_became_peered":"2024-03-30T14:54:37.323084+0000","last_unstale":"2024-03-30T14:56:12.368275+0000","last_undegraded":"2024-03-30T14:56:12.368275+0000","last_fullsized":"2024-03-30T14:56:12.368275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:54:36.304294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"76'15","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541864+0000","last_change":"2024-03-30T14:54:39.338356+0000","last_active":"2024-03-30T14:56:12.541864+0000","last_peered":"2024-03-30T14:56:12.541864+0000","last_clean":"2024-03-30T14:56:12.541864+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T14:56:12.541864+0000","last_undegraded":"2024-03-30T14:56:12.541864+0000","last_fullsized":"2024-03-30T14:56:12.541864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:54:38.312097+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368223+0000","last_change":"2024-03-30T14:56:06.675463+0000","last_active":"2024-03-30T14:56:12.368223+0000","last_peered":"2024-03-30T14:56:12.368223+0000","last_clean":"2024-03-30T14:56:12.368223+0000","last_became_active":"2024-03-30T14:54:43.360002+0000","last_became_peered":"2024-03-30T14:54:43.360002+0000","last_unstale":"2024-03-30T14:56:12.368223+0000","last_undegraded":"2024-03-30T14:56:12.368223+0000","last_fullsized":"2024-03-30T14:56:12.368223+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:06.675378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:06.675378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:23.466275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.373248+0000","last_change":"2024-03-30T14:55:50.439135+0000","last_active":"2024-03-30T14:56:12.373248+0000","last_peered":"2024-03-30T14:56:12.373248+0000","last_clean":"2024-03-30T14:56:12.373248+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T14:56:12.373248+0000","last_undegraded":"2024-03-30T14:56:12.373248+0000","last_fullsized":"2024-03-30T14:56:12.373248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.439008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:15.333757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185330+0000","last_change":"2024-03-30T14:55:47.298871+0000","last_active":"2024-03-30T14:56:12.185330+0000","last_peered":"2024-03-30T14:56:12.185330+0000","last_clean":"2024-03-30T14:56:12.185330+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T14:56:12.185330+0000","last_undegraded":"2024-03-30T14:56:12.185330+0000","last_fullsized":"2024-03-30T14:56:12.185330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:47.298753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:47.298753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:03.038171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186223+0000","last_change":"2024-03-30T14:56:09.473519+0000","last_active":"2024-03-30T14:56:12.186223+0000","last_peered":"2024-03-30T14:56:12.186223+0000","last_clean":"2024-03-30T14:56:12.186223+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T14:56:12.186223+0000","last_undegraded":"2024-03-30T14:56:12.186223+0000","last_fullsized":"2024-03-30T14:56:12.186223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:56:09.473429+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:13.395740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.622534+0000","last_change":"2024-03-30T14:56:12.622556+0000","last_active":"2024-03-30T14:56:12.622534+0000","last_peered":"2024-03-30T14:56:12.622534+0000","last_clean":"2024-03-30T14:56:12.622534+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T14:56:12.622534+0000","last_undegraded":"2024-03-30T14:56:12.622534+0000","last_fullsized":"2024-03-30T14:56:12.622534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:12.622465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:56:12.622465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:20.190586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.541515+0000","last_change":"2024-03-30T14:55:46.232558+0000","last_active":"2024-03-30T14:56:12.541515+0000","last_peered":"2024-03-30T14:56:12.541515+0000","last_clean":"2024-03-30T14:56:12.541515+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T14:56:12.541515+0000","last_undegraded":"2024-03-30T14:56:12.541515+0000","last_fullsized":"2024-03-30T14:56:12.541515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:55:46.232465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:06.933698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.532117+0000","last_change":"2024-03-30T14:56:08.068026+0000","last_active":"2024-03-30T14:56:12.532117+0000","last_peered":"2024-03-30T14:56:12.532117+0000","last_clean":"2024-03-30T14:56:12.532117+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T14:56:12.532117+0000","last_undegraded":"2024-03-30T14:56:12.532117+0000","last_fullsized":"2024-03-30T14:56:12.532117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:56:08.067949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:56:08.067949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:29.341927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186074+0000","last_change":"2024-03-30T14:55:50.322071+0000","last_active":"2024-03-30T14:56:12.186074+0000","last_peered":"2024-03-30T14:56:12.186074+0000","last_clean":"2024-03-30T14:56:12.186074+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T14:56:12.186074+0000","last_undegraded":"2024-03-30T14:56:12.186074+0000","last_fullsized":"2024-03-30T14:56:12.186074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T14:55:50.321974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:50.321974+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:54.493589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.186326+0000","last_change":"2024-03-30T14:54:43.352969+0000","last_active":"2024-03-30T14:56:12.186326+0000","last_peered":"2024-03-30T14:56:12.186326+0000","last_clean":"2024-03-30T14:56:12.186326+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T14:56:12.186326+0000","last_undegraded":"2024-03-30T14:56:12.186326+0000","last_fullsized":"2024-03-30T14:56:12.186326+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:54:42.332090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.531780+0000","last_change":"2024-03-30T14:55:40.800056+0000","last_active":"2024-03-30T14:56:12.531780+0000","last_peered":"2024-03-30T14:56:12.531780+0000","last_clean":"2024-03-30T14:56:12.531780+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T14:56:12.531780+0000","last_undegraded":"2024-03-30T14:56:12.531780+0000","last_fullsized":"2024-03-30T14:56:12.531780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:55:40.799969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:55:40.799969+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:56:57.300469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":69,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.368332+0000","last_change":"2024-03-30T14:55:45.783270+0000","last_active":"2024-03-30T14:56:12.368332+0000","last_peered":"2024-03-30T14:56:12.368332+0000","last_clean":"2024-03-30T14:56:12.368332+0000","last_became_active":"2024-03-30T14:54:37.327905+0000","last_became_peered":"2024-03-30T14:54:37.327905+0000","last_unstale":"2024-03-30T14:56:12.368332+0000","last_undegraded":"2024-03-30T14:56:12.368332+0000","last_fullsized":"2024-03-30T14:56:12.368332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:55:45.783178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:55:45.783178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:05.797807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T14:56:12.185819+0000","last_change":"2024-03-30T14:55:51.283107+0000","last_active":"2024-03-30T14:56:12.185819+0000","last_peered":"2024-03-30T14:56:12.185819+0000","last_clean":"2024-03-30T14:56:12.185819+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T14:56:12.185819+0000","last_undegraded":"2024-03-30T14:56:12.185819+0000","last_fullsized":"2024-03-30T14:56:12.185819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:55:51.282990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:55:51.282990+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T14:57:10.280452+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005080,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1331200,"data_stored":924302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299421,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32568,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712584,"statfs":{"total":95995035648,"available":95961686016,"internally_reserved":0,"allocated":1298432,"data_stored":921964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":745472,"data_stored":332803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1331200,"data_stored":923646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":708608,"data_stored":331931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":753664,"data_stored":333773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":737280,"data_stored":332599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":724992,"data_stored":332280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T14:56:19.410 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T14:56:19.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump 2024-03-30T14:56:19.582 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:19 smithi134 systemd[1]: Starting Ceph node-exporter.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:19.658 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:19 smithi134 ceph-mon[33714]: Deploying daemon node-exporter.b on smithi134 2024-03-30T14:56:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:19 smithi134 ceph-mon[33714]: from='client.14766 v1:172.21.15.28:0/879696043' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:19 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:56:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:19 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:56:19.930 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:19 smithi134 bash[66814]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[41614]: Deploying daemon node-exporter.b on smithi134 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[41614]: from='client.14766 v1:172.21.15.28:0/879696043' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[29876]: Deploying daemon node-exporter.b on smithi134 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[29876]: from='client.14766 v1:172.21.15.28:0/879696043' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T14:56:19.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:56:19.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:19 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:56:20.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:56:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:20.680 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:20 smithi134 bash[66814]: Getting image source signatures 2024-03-30T14:56:20.680 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:20 smithi134 bash[66814]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-30T14:56:20.680 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:20 smithi134 bash[66814]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-30T14:56:20.680 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:20 smithi134 bash[66814]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:21.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T14:56:21.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:20 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: osdmap e95: 8 total, 8 up, 8 in 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T14:56:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T14:56:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:20 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T14:56:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.723+0000 7fe1c51a5700 1 -- 172.21.15.28:0/583926881 >> v1:172.21.15.28:6789/0 conn(0x7fe1c010eb50 legacy=0x7fe1c0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 -- 172.21.15.28:0/583926881 shutdown_connections 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 -- 172.21.15.28:0/583926881 >> 172.21.15.28:0/583926881 conn(0x7fe1c00fc9e0 msgr2=0x7fe1c00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 -- 172.21.15.28:0/583926881 shutdown_connections 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 -- 172.21.15.28:0/583926881 wait complete. 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 librados: starting msgr at 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 librados: starting objecter 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 Processor -- start 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 -- start start 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 librados: setting wanted keys 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.724+0000 7fe1c51a5700 1 librados: calling monclient init 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1c51a5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1c01a6f80 con 0x7fe1c0103230 2024-03-30T14:56:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1c51a5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1c01a81a0 con 0x7fe1c010b040 2024-03-30T14:56:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1c51a5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1c01a93e0 con 0x7fe1c010eb50 2024-03-30T14:56:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1bed9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1c0103230 0x7fe1c010a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43384/0 (socket says 172.21.15.28:43384) 2024-03-30T14:56:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1bed9d700 1 -- 172.21.15.28:0/1571312101 learned_addr learned my addr 172.21.15.28:0/1571312101 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter ms_handle_connect 0x7fe1c010eb50 2024-03-30T14:56:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter ms_handle_connect 0x7fe1c0103230 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter ms_handle_connect 0x7fe1c010b040 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696580548 0 0) 0x7fe1c01a93e0 con 0x7fe1c010eb50 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ac0031f0 con 0x7fe1c010eb50 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133820838 0 0) 0x7fe1c01a6f80 con 0x7fe1c0103230 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c01a93e0 con 0x7fe1c0103230 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502616772 0 0) 0x7fe1c01a81a0 con 0x7fe1c010b040 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c01a6f80 con 0x7fe1c010b040 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3896661954 0 0) 0x7fe1c01a93e0 con 0x7fe1c0103230 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1c01a81a0 con 0x7fe1c0103230 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4291295218 0 0) 0x7fe1ac0031f0 con 0x7fe1c010eb50 2024-03-30T14:56:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.725+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1c01a93e0 con 0x7fe1c010eb50 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe1b00018b0 con 0x7fe1c0103230 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2658181083 0 0) 0x7fe1c01a6f80 con 0x7fe1c010b040 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1ac0031f0 con 0x7fe1c010b040 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe1b8001d60 con 0x7fe1c010eb50 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe1a80017f0 con 0x7fe1c010b040 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1431516890 0 0) 0x7fe1c01a81a0 con 0x7fe1c0103230 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 >> v1:172.21.15.28:6790/0 conn(0x7fe1c010eb50 legacy=0x7fe1c01a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 >> v1:172.21.15.134:6789/0 conn(0x7fe1c010b040 legacy=0x7fe1c01a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c01aa620 con 0x7fe1c0103230 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c01a9610 con 0x7fe1c0103230 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1c51a5700 10 client.14772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1c01a9ad0 con 0x7fe1c0103230 2024-03-30T14:56:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1c51a5700 1 librados: init done 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe1b0001c50 con 0x7fe1c0103230 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.726+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe1b0005cf0 con 0x7fe1c0103230 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.727+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a0005120 con 0x7fe1c0103230 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.727+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe1b0003950 con 0x7fe1c0103230 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(95..95 src has 43..95) ==== 6259+0+0 (unknown 2966647193 0 0) 0x7fe1b0088ee0 con 0x7fe1c0103230 2024-03-30T14:56:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 10 client.14772.objecter ms_dispatch 0x7fe1c0001550 osd_map(95..95 src has 43..95) 2024-03-30T14:56:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 3 client.14772.objecter handle_osd_map got epochs [95,95] > 0 2024-03-30T14:56:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 3 client.14772.objecter handle_osd_map decoding full epoch 95 2024-03-30T14:56:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 20 client.14772.objecter dump_active .. 0 homeless 2024-03-30T14:56:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.728+0000 7fe1b7fff700 10 client.14772.objecter ms_handle_connect 0x7fe1ac070580 2024-03-30T14:56:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.732+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe1b0056840 con 0x7fe1c0103230 2024-03-30T14:56:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.936+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fe1a0005c20 con 0x7fe1c0103230 2024-03-30T14:56:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.937+0000 7fe1b7fff700 1 -- 172.21.15.28:0/1571312101 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+4427 (unknown 2799817303 0 903780462) 0x7fe1b005a290 con 0x7fe1c0103230 2024-03-30T14:56:21.938 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 95 2024-03-30T14:56:21.938 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-03-30T14:48:49.804304+0000 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-03-30T14:56:19.720782+0000 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 18 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 72 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-30T14:56:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 8 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/3394323575 v1:172.21.15.28:6802/3394323575 exists,up eb350905-2b09-408a-9adc-4f2af2766479 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1850313547 v1:172.21.15.28:6806/1850313547 exists,up 76bfea8d-68c7-448e-bb99-d3d003567520 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/3451325043 v1:172.21.15.28:6810/3451325043 exists,up ad24917c-f5cb-4ca8-8a52-91d2256ff21c 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 27 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/1506157250 v1:172.21.15.28:6814/1506157250 exists,up 4305caf1-0023-438f-8520-87814130b457 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 32 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6800/3063775514 v1:172.21.15.134:6801/3063775514 exists,up 8180ec45-5260-4cdc-855c-be5302b1465c 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 41 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6804/3668151179 v1:172.21.15.134:6805/3668151179 exists,up 265992a1-46bc-4621-b9db-2b5ba1ee5e45 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 52 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6808/2135224194 v1:172.21.15.134:6809/2135224194 exists,up 5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d 2024-03-30T14:56:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 61 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6812/3758190298 v1:172.21.15.134:6813/3758190298 exists,up 76f25ab9-af2f-4f45-afe3-13ece781d29c 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.4 [1,5] 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.a [1,2] 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1b [1,2] 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2199995094 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/545344739 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4222366292 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/406580277 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2354623613 expires 2024-03-31T14:49:57.293695+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2838206829 expires 2024-03-31T14:49:29.238923+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1249955842 expires 2024-03-31T14:49:57.293695+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3720192247 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1446120238 expires 2024-03-31T14:49:29.238923+0000 2024-03-30T14:56:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/429160846 expires 2024-03-31T14:49:29.238923+0000 2024-03-30T14:56:21.942 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/580061692 expires 2024-03-31T14:55:17.563475+0000 2024-03-30T14:56:21.942 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/997592180 expires 2024-03-31T14:49:29.238923+0000 2024-03-30T14:56:21.942 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2767265670 expires 2024-03-31T14:49:57.293695+0000 2024-03-30T14:56:21.942 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1590506004 expires 2024-03-31T14:49:57.293695+0000 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 10 librados: watch_flush enter 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 10 librados: watch_flush exit 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 20 client.14772.objecter shutdown clearing up homeless session... 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 10 client.14772.objecter successfully canceled tick 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe1ac070580 legacy=0x7fe1ac072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 >> v1:172.21.15.28:6789/0 conn(0x7fe1c0103230 legacy=0x7fe1c010a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 shutdown_connections 2024-03-30T14:56:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.939+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 >> 172.21.15.28:0/1571312101 conn(0x7fe1c00fc9e0 msgr2=0x7fe1c00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.940+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 shutdown_connections 2024-03-30T14:56:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.940+0000 7fe1c51a5700 1 -- 172.21.15.28:0/1571312101 wait complete. 2024-03-30T14:56:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:21.940+0000 7fe1c51a5700 1 librados: shutdown 2024-03-30T14:56:22.180 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:21 smithi134 bash[66814]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-30T14:56:22.180 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:21 smithi134 bash[66814]: Writing manifest to image destination 2024-03-30T14:56:22.598 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 podman[66814]: 2024-03-30 14:56:22.196842519 +0000 UTC m=+2.515793565 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-30T14:56:22.598 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 podman[66814]: 2024-03-30 14:56:22.303850571 +0000 UTC m=+2.622801616 container create 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:56:22.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T14:56:22.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T14:56:22.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:56:22.598 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1571312101' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:56:22.599 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T14:56:22.599 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1571312101' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T14:56:22.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1571312101' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T14:56:22.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:22 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T14:56:22.713 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/3394323575 v1:172.21.15.28:6802/3394323575 exists,up eb350905-2b09-408a-9adc-4f2af2766479', 'osd.1 up in weight 1 up_from 13 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1850313547 v1:172.21.15.28:6806/1850313547 exists,up 76bfea8d-68c7-448e-bb99-d3d003567520', 'osd.2 up in weight 1 up_from 18 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/3451325043 v1:172.21.15.28:6810/3451325043 exists,up ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'osd.3 up in weight 1 up_from 27 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/1506157250 v1:172.21.15.28:6814/1506157250 exists,up 4305caf1-0023-438f-8520-87814130b457', 'osd.4 up in weight 1 up_from 32 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6800/3063775514 v1:172.21.15.134:6801/3063775514 exists,up 8180ec45-5260-4cdc-855c-be5302b1465c', 'osd.5 up in weight 1 up_from 41 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6804/3668151179 v1:172.21.15.134:6805/3668151179 exists,up 265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'osd.6 up in weight 1 up_from 52 up_thru 78 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6808/2135224194 v1:172.21.15.134:6809/2135224194 exists,up 5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'osd.7 up in weight 1 up_from 61 up_thru 74 down_at 0 last_clean_interval [0,0) v1:172.21.15.134:6812/3758190298 v1:172.21.15.134:6813/3758190298 exists,up 76f25ab9-af2f-4f45-afe3-13ece781d29c'] 2024-03-30T14:56:22.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon mon.a config show 2024-03-30T14:56:22.930 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 podman[66814]: 2024-03-30 14:56:22.597750842 +0000 UTC m=+2.916701884 container init 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:56:22.930 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 podman[66814]: 2024-03-30 14:56:22.604133745 +0000 UTC m=+2.923084788 container start 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.608Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.608Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.608Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.608Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=arp 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-30T14:56:22.931 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=edac 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-30T14:56:22.932 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=os 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-30T14:56:22.933 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=stat 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-30T14:56:22.934 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=time 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=uname 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.609Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.610Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b[66862]: ts=2024-03-30T14:56:22.610Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 bash[66814]: 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a 2024-03-30T14:56:22.935 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 14:56:22 smithi134 systemd[1]: Started Ceph node-exporter.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:23.372 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T14:56:23.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:23.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:24.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T14:56:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T14:56:24.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[29876]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T14:56:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T14:56:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:24 smithi028 ceph-mon[41614]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:24.590 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:24 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T14:56:24.590 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:24 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T14:56:24.590 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:24 smithi134 ceph-mon[33714]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:25.057 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:56:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mon.a", 2024-03-30T14:56:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "ceph", 2024-03-30T14:56:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket_mode": "", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_client_required": "cephx, none", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_cluster_required": "cephx", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_debug": "true", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_required": "cephx", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_supported": "", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio": "true", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_poll_ms": "250", 2024-03-30T14:56:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_reap_max": "16", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_async_discard": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_block_size": "4096", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_inflight_ios": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_enable_discard": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry": "3", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash": "0", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_hipri": "false", 2024-03-30T14:56:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_type": "", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_alloc_size": "1048576", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_allocator": "hybrid", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_buffered_io": "true", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_for_zeros": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_compact_log_sync": "false", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-30T14:56:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_log_runway": "4194304", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_prefetch": "1048576", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_flush_size": "524288", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_log_runway": "1048576", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery": "false", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_sync_write": "false", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-30T14:56:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocation_from_file": "true", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocator": "hybrid", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_create": "true", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_create": "false", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_path": "", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_size": "0", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_path": "", 2024-03-30T14:56:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_preallocate_file": "false", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_size": "107374182400", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_create": "false", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_path": "", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_size": "100663296", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs": "true", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune": "true", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-30T14:56:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size": "0", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_type": "2q", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_clone_cow": "true", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-30T14:56:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_mode": "none", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_csum_type": "crc32c", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_freelist": "false", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-30T14:56:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_small_allocations": "0", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_read": "true", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_write": "false", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-30T14:56:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fail_eio": "false", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-30T14:56:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-30T14:56:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_ignore_data_csum": "false", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_op_age": "5.000000", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_alloc_size": "0", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size": "0", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_deferred_txc": "32", 2024-03-30T14:56:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size": "0", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_nid_prealloc": "1024", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_retry_disk_reads": "3", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cf": "true", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-30T14:56:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_mem": "512", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_tracing": "false", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-30T14:56:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_zero_block_detection": "false", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-30T14:56:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_signatures": "false", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_version": "2", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_signatures": "false", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_version": "2", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_signatures": "false", 2024-03-30T14:56:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_version": "2", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_sign_messages": "true", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "chdir": "", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_acl_type": "", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_asio_thread_count": "2", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_mid": "0.750000", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_size": "16384", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_caps_release_delay": "5", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_check_pool_perm": "true", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_force_sync_read": "false", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_getattr_caps": "false", 2024-03-30T14:56:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_tick_delay": "0", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_remount": "false", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_dirsize_rbytes": "true", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_force_lazyio": "false", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_fs": "", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_release_failure": "false", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_inline_size": "4096", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mds_namespace": "", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_metadata": "", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_gid": "-1", 2024-03-30T14:56:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_timeout": "300", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_uid": "-1", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mountpoint": "/", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_notify_timeout": "10", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc": "true", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty": "104857600", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_objects": "1000", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_size": "209715200", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_target_dirty": "8388608", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_permissions": "true", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota": "true", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota_df": "true", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_bytes": "0", 2024-03-30T14:56:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_periods": "4", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_min": "131072", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_reconnect_stale": "false", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_shutdown_timeout": "30", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_snapdir": ".snap", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_tick_interval": "1", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_trace": "", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_try_dentry_invalidate": "false", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_faked_inos": "false", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_random_mds": "false", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog": "false", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_port": "12201", 2024-03-30T14:56:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_monitors": "default=true", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog": "false", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_level": "info", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "-", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network": "", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network_interface": "", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_isal": "false", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_level": "5", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_winsize": "-15", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zstd_level": "1", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_op_num_threads": "6", 2024-03-30T14:56:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_seastar_cpu_cores": "", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook": "", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook_timeout": "10", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "daemonize": "false", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "daos_pool": "tank", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_alienstore": "0/5", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_allow_any_pool_priority": "false", 2024-03-30T14:56:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok_assert_abort": "false", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asserts_on_shutdown": "true", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_auth": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bdev": "1/3", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluefs": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluestore": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_buffer": "0/1", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ceph_exporter": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephfs_mirror": "0/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephsqlite": "0/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_client": "0/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_compressor": "1/5", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_context": "0/1", 2024-03-30T14:56:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crimson_interrupt": "0/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crush": "1/1", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crypto": "1/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cyanstore": "0/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_deliberately_leak_memory": "false", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_disable_randomized_ping": "false", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_dpdk": "1/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_eventtrace": "1/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filer": "0/1", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filestore": "1/3", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_finisher": "1/1", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_fuse": "1/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeat_testing_span": "0", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeatmap": "1/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-30T14:56:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_javaclient": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journal": "1/3", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journaler": "0/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_kstore": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_lockdep": "0/1", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mclock": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_balancer": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_locker": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log_expire": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_migrator": "1/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_quiesce": "3/5", 2024-03-30T14:56:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_memstore": "1/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgr": "2/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgrc": "20/20", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mon": "20/20", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_monc": "0/10", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ms": "1/1", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_none": "0/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objclass": "0/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objectcacher": "0/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objecter": "0/1", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_optracker": "0/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_osd": "1/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_paxos": "20/20", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_perfcounter": "1/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_prioritycache": "1/5", 2024-03-30T14:56:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rados": "0/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd": "0/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_mirror": "0/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_pwl": "0/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_replay": "0/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_refs": "0/0", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_reserver": "1/1", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_access": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_datacache": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_dbstore": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_flight": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_sync": "1/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rocksdb": "4/5", 2024-03-30T14:56:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_backref": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cache": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cleaner": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_device": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_epm": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_journal": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_lba": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_odata": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_omap": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_onode": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_t": "0/5", 2024-03-30T14:56:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_tm": "0/5", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_striper": "0/1", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_test": "0/5", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_throttle": "1/1", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_timer": "0/1", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_tp": "0/5", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_trace": "1/5", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "device_failure_prediction_mode": "none", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_graylog": "false", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_journald": "false", 2024-03-30T14:56:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_stderr": "true", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_syslog": "false", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event_tracing": "false", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_addr": "0.0.0.0", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_http_port": "9926", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_prio_limit": "5", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sort_metrics": "true", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_stats_period": "5", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fake_statfs_for_testing": "0", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "fatal_signal_handlers": "true", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_purge_ops": "10", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_truncate_ops": "128", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_apply_finisher_threads": "1", 2024-03-30T14:56:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_blackhole": "false", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_clone_range": "true", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_snap": "true", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_caller_concurrency": "10", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_collect_device_partition_information": "true", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_commit_timeout": "600.000000", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_inject_read_err": "false", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_omap_check": "false", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_verify_split": "false", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_dump_file": "", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fadvise": "true", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fail_eio": "true", 2024-03-30T14:56:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_shards": "16", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_size": "128", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap": "false", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap_threshold": "4096", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_inject_stall": "0", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_parallel": "false", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_trailing": "false", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_writeahead": "false", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_kill_at": "0", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-30T14:56:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs": "0", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size": "0", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_merge_threshold": "-10", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_odsync_write": "false", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend": "rocksdb", 2024-03-30T14:56:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend_path": "", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_timeout": "60", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_threads": "2", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_punch_hole": "false", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-30T14:56:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_ops": "50", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_seek_data_hole": "false", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc": "false", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_splice": "false", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_multiple": "2", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_rand_factor": "20", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_update_to": "1000", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-30T14:56:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_enable": "true", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_xfs_extsize": "false", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_zfs_snap": "false", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "fio_dir": "/tmp/fio", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_allow_other": "true", 2024-03-30T14:56:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_atomic_o_trunc": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_big_writes": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_debug": "false", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_default_permissions": "false", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_disable_pagecache": "false", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_max_write": "0", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_multithreaded": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_require_active_mds": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_set_user_groups": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_move": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_read": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_write": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-30T14:56:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_use_invalidate_cb": "true", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_target_name": "ceph", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_file": "", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_inject_failure": "0", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_interval": "5", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "host": "smithi028", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-30T14:56:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "inject_early_sigterm": "false", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_agent_port": "6799", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_tracing_enable": "false", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_aio": "true", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_align_min_size": "65536", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_align": "true", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_size": "4096", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_dio": "true", 2024-03-30T14:56:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_discard": "false", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_force_aio": "false", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_ignore_corruption": "false", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_bytes": "10485760", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_entries": "100", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_replay_from": "0", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_write_header_frequency": "0", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_zero_on_create": "false", 2024-03-30T14:56:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prefetch_periods": "10", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prezero_periods": "5", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_write_head_interval": "15", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "keyfile": "", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_backend": "rocksdb", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_default_stripe_size": "65536", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount": "false", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_bytes": "67108864", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_ops": "512", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_nid_prealloc": "1024", 2024-03-30T14:56:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_onode_map_size": "1024", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_submit_transaction": "false", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_transaction": "false", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "librados_thread_count": "2", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep": "false", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep_force_backtrace": "false", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_coarse_timestamps": "true", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_flush_on_exit": "false", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_host": "127.0.0.1", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_port": "12201", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_new": "1000", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_recent": "10000", 2024-03-30T14:56:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stderr_prefix": "", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stop_at_utilization": "0.970000", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": "true", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_graylog": "false", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_journald": "false", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_stderr": "false", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_syslog": "false", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max_rotating_auth_attempts": "10", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_action_on_write_error": "1", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_alternate_name_max": "8192", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_asio_thread_count": "2", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_export_pin": "true", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_dirs": "true", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-30T14:56:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_interval": "5", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_interval": "10", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max": "-1", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max_until": "-1", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_merge_size": "50", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_midchunk": "0.300000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_start": "0.200000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_minchunk": "0.001000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_mode": "0", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_max": "1.200000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_min": "0.800000", 2024-03-30T14:56:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_overload_epochs": "2", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_bits": "3", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_size": "10000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_target_decay": "10.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_grace": "15.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_interval": "4.000000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_mid": "0.700000", 2024-03-30T14:56:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_delay": "0", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_sleep": "200", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_splitauth": "true", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_quiesce_threshold": "524288", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_release_free_interval": "10", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_reservation": "0.050000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_interval": "1", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_threshold": "262144", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_prealloc_inos": "1000", 2024-03-30T14:56:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_connect_bootstrapping": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_damage_table_max_entries": "10000", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_auth_pins": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_frag": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_scatterstat": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_subtrees": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_decay_halflife": "5.000000", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_default_dir_hash": "2", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_defer_session_stale": "true", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_deny_all_reconnect": "false", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_keys_per_op": "16384", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_commit_size": "10", 2024-03-30T14:56:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_entries": "0", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_prefetch": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_on_map": "false", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_early_reply": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enable_op_tracker": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enforce_unique_name": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random": "true", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-30T14:56:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_summarize_threshold": "10", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_health_dummy": "false", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_migrator_session_race": "false", 2024-03-30T14:56:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_join_fs": "", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_journal_format": "1", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_create_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_export_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_import_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_expire_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_replay_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_link_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_mdstable_at": "0", 2024-03-30T14:56:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_openc_at": "0", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_rename_at": "0", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_shutdown_at": "0", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_event_large_threshold": "524288", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_events_per_segment": "1024", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_events": "-1", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_segments": "128", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_pause": "false", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_segment_size": "0", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-30T14:56:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_threshold": "128", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_warn_factor": "2.000000", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_caps_per_client": "1048576", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_flushes": "100000", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_requests": "100000", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_export_size": "20971520", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_file_recover": "32", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_files": "64", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops": "8192", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_snaps_per_dir": "100", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_metrics_update_interval": "2", 2024-03-30T14:56:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_per_client": "100", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_working_set": "10000", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_numa_node": "-1", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_complaint_time": "30.000000", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_duration": "600", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_size": "20", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_size": "20", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_log_threshold": "5", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_grace": "15", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_interval": "5", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-30T14:56:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_caps": "30000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_threshold": "262144", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_interval": "1.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_gid": "0", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_uid": "0", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-30T14:56:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_shutdown_check": "0", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_skip_ino": "0", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_max_uid": "4294967294", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_min_uid": "0", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_rstat": "false", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_standby_replay_damaged": "false", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_symlink_recovery": "true", 2024-03-30T14:56:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_exports": "0", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_fragments": "0", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_tick_interval": "5.000000", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_valgrind_exit": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_backtrace": "1", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_scatter": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_sessions": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool_debug": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_device_bytes": "1073741824", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_set": "false", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_size": "65536", 2024-03-30T14:56:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_bytes": "134217728", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_messages": "512", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_disabled_modules": "", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_num_change": "128", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_bytes": "134217728", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_messages": "128", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_bytes": "134217728", 2024-03-30T14:56:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_messages": "128", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_bytes": "536870912", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_messages": "8192", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_pool": "true", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_standby_modules": "true", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_period": "5", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_threshold": "5", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_tick_period": "2", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_delete": "true", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_size_one": "false", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_auth_validate_all_caps": "true", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-30T14:56:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_bytes": "104857600", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_directed_command_retry": "2", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_parallel": "3", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_log_interval": "1.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_interval": "10.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_target_rank": "-1", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-30T14:56:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file_level": "debug", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_file": "true", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_journald": "true", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_bootstrap": "false", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_start": "false", 2024-03-30T14:56:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_trim": "true", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cpu_threads": "4", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_crush_min_required_version": "hammer", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_daemon_bytes": "419430400", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_crit": "5", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_warn": "5", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_size_warn": "16106127360", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-30T14:56:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_json": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_transactions": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_extra_checks": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_reef": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_squid": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_mkfs_grace": "60", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_uptime_grace": "60", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_default_strategy": "1", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_timeout": "5.000000", 2024-03-30T14:56:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_divisor": "2", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_enable_op_tracker": "true", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_fake_pool_delete": "false", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_force_quorum_join": "false", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_globalid_prealloc": "10000", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_detail_to_clog": "true", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_log_update_period": "5", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_max_detail": "50", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog": "true", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_interval": "600", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-30T14:56:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host": "172.21.15.28:6789/0", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host_override": "", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_initial_members": "", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease": "5.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_full_interval": "50", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max": "10000", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max_summary": "50", 2024-03-30T14:56:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_epochs": "500", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_osd": "10000", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pg_per_osd": "10000", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pool_pg_num": "65536", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_force_trim_to": "0", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_skip_sanity": "false", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_autotune": "true", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_target": "2147483648", 2024-03-30T14:56:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_beacon_grace": "30", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_digest_period": "5", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_inactive_grace": "60", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_min_osdmap_epochs": "50", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_complaint_time": "30", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_duration": "600", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_size": "20", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_size": "20", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_log_threshold": "5", 2024-03-30T14:56:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_in": "false", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size": "500", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_interval": "600", 2024-03-30T14:56:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_force_trim_to": "0", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_down_reporters": "2", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-30T14:56:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_report_timeout": "900", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-30T14:56:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_stuck_threshold": "60", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_probe_timeout": "2.000000", 2024-03-30T14:56:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_change": "0.050000", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_osds": "4", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_interval": "86400", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_max_keys": "100", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_timeout": "300", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_session_timeout": "300", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_smart_report_timeout": "5", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stat_smooth_intervals": "6", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_min_size": "2", 2024-03-30T14:56:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_size": "4", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_debug": "false", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_provider_kill_at": "0", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_requester_kill_at": "0", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_timeout": "60.000000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_target_pg_per_osd": "100", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_tick_interval": "5", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_interval": "300.000000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-30T14:56:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_use_min_delay_socket": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_older_version_delay": "604800", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_misplaced": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_older_version": "true", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-30T14:56:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_app": "false", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": "", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-30T14:56:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_tracing_enabled": "false", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_op_threads": "3", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_cm": "false", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_device_name": "", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_dscp": "96", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_local_gid": "", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_port_num": "1", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-30T14:56:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_sl": "3", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_support_srq": "true", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_type": "ib", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_reap_threshold": "5", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_before_connect": "false", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv4": "true", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv6": "false", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr1": "true", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr2": "false", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_max": "7568", 2024-03-30T14:56:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_min": "6800", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_count": "3", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_delay": "5", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_client": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mds": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mgr": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mon": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_osd": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_client_mode": "crc secure", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_mode": "crc secure", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_type": "", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_compress_secure": "false", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_idle_timeout": "900", 2024-03-30T14:56:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_ready_timeout": "10", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_data": "true", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_header": "true", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bad_msg": "false", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bug": "true", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_old_message": "true", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_skipped_message": "false", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_coremask": "0xF", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_enable_tso": "true", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-30T14:56:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hugepages": "", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_lro": "true", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_memory_channel": "4", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_pmd": "", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_port_id": "0", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_on_send": "false", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_initial_backoff": "0.200000", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_max": "1.000000", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-30T14:56:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_type": "", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_network_congestion": "0", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_socket_failures": "0", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_learn_addr_from_peer": "true", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_accept_failures": "4", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_backoff": "15.000000", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_client_mode": "secure crc", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_service_mode": "secure crc", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_min_size": "1024", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_mode": "none", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-30T14:56:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_min_cost": "65536", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_public_type": "", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_service_mode": "crc secure", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_listen_backlog": "512", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_nodelay": "true", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_rcvbuf": "0", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_type": "async+posix", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "no_config_file": "false", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_completion_locks_per_session": "32", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_ops": "1024", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-30T14:56:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_tick_interval": "5.000000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_timeout": "10.000000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_blackhole": "false", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "openssl_engine_opts": "", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_delay_time": "5.000000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_hist_halflife": "1000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_low_ops": "2", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_ops": "4", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_slop": "0.020000", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-30T14:56:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_asio_thread_count": "2", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_async_recovery_min_cost": "100", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_max": "512", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_min": "64", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_degraded": "false", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_peering": "false", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_unfound": "true", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_beacon_report_interval": "300", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_duration": "30", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_max_block_size": "67108864", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blkin_trace_all": "false", 2024-03-30T14:56:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_for_log_corruption": "false", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_update_on_start": "true", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_cap": "256", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_size_cap": "524288000", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_op_priority": "63", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_watch_timeout": "30", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_command_max_records": "256", 2024-03-30T14:56:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_compact_on_start": "false", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_on_start": "true", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_weight_set": "true", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_feed_pullee": "-1", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-30T14:56:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_misdirected_ops": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_acting_change": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_purge_strays": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_op_order": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_shutdown": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_trim_objects": "false", 2024-03-30T14:56:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_snaps": "false", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_keys": "1024", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_stride": "524288", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_notify_timeout": "30", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep": "0.000000", 2024-03-30T14:56:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_discard_disconnected_ops": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_enable_op_tracker": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_info": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-30T14:56:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_function_tracing": "false", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_grace": "20", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_interval": "6", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_peers": "10", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_size": "2000", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_stale": "600", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_max_size": "100000", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_min_size": "1000", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-30T14:56:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_size": "5120", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_kill_backfill_at": "0", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_cache_size": "50", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_dedup": "true", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max": "40", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_share_max_epochs": "40", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_name_len": "100", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_size": "0", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_backfills": "1", 2024-03-30T14:56:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_count": "5", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_period": "600", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_name_len": "2048", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_namespace_len": "256", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_size": "134217728", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_blocked_by": "16", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_log_entries": "10000", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pgls": "1024", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_cost": "8388608", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_objects": "10", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_scrubs": "3", 2024-03-30T14:56:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_trimming_pgs": "2", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_op_reply_len": "64", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_size": "90", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_profile": "balanced", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-30T14:56:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_base": "805306368", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_min": "134217728", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-30T14:56:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target": "4294967296", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_autotune": "false", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_pg_log_entries": "250", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_recovery_priority": "0", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_interval": "5", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_cache_shards": "32", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_op_tracker_shard": "32", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_auto_affinity": "true", 2024-03-30T14:56:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_node": "-1", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_prefer_iface": "true", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objecter_finishers": "1", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore": "bluestore", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_fuse": "false", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_tracing": "false", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_complaint_time": "30.000000", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_duration": "600", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_size": "20", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_size": "20", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_log_threshold": "5", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards": "0", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_hdd": "5", 2024-03-30T14:56:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_ssd": "8", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_min_cost": "65536", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue_cut_off": "high", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_timeout": "15", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_open_classes_on_start": "true", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_os_flags": "0", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_peering_op_priority": "255", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_cost": "1048576", 2024-03-30T14:56:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_priority": "5", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_max": "10000", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_min": "100", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-30T14:56:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-30T14:56:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crimson": "false", 2024-03-30T14:56:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-30T14:56:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-30T14:56:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flags": "0", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_min_size": "0", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_num": "32", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pgp_num": "0", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_ratio": "70", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_size": "3", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_type": "replicated", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-30T14:56:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_push_per_object_cost": "1000", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap": "true", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_cost": "20971520", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active": "0", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-30T14:56:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_single_start": "1", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_priority": "3", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_priority": "5", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep": "0.000000", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-30T14:56:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_repair_during_recovery": "false", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_requested_scrub_priority": "120", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair": "false", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_hour": "0", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_week_day": "0", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_max": "25", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_min": "5", 2024-03-30T14:56:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_cost": "52428800", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_during_recovery": "false", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_hour": "0", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_week_day": "0", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_event_cost": "4096", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_invalid_stats": "true", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-30T14:56:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_preemptions": "5", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_priority": "5", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_sleep": "0.000000", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_data_digest": "false", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_smart_report_timeout": "5", 2024-03-30T14:56:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_cost": "1048576", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_priority": "5", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_transaction_size": "30", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-30T14:56:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tracing": "false", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_use_stale_snap": "false", 2024-03-30T14:56:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "osdc_blkin_trace_all": "false", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_kill_at": "0", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_max_join_drift": "10", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min": "500", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min_wait": "0.050000", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_propose_interval": "1.000000", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max": "500", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_min": "10", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_stash_full_interval": "25", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_max": "500", 2024-03-30T14:56:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_min": "250", 2024-03-30T14:56:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "perf": "true", 2024-03-30T14:56:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "pid_file": "", 2024-03-30T14:56:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-30T14:56:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "-", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrv": "", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "public_bind_addr": "-", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network": "", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network_interface": "", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_enabled": "false", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_session_max_number": "256", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_mon_op_timeout": "0", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_osd_op_timeout": "0", 2024-03-30T14:56:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_tracing": "false", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_atime_update_interval": "60", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_parent_reads": "false", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_snap_reads": "false", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blkin_trace_all": "false", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache": "true", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_policy": "writearound", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_size": "33554432", 2024-03-30T14:56:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_clone_copy_on_read": "false", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_compression_hint": "none", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_concurrent_management_ops": "10", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_clone_format": "auto", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_data_pool": "", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_features": "61", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_format": "2", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_map_options": "", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_order": "22", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_pool": "rbd", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_count": "0", 2024-03-30T14:56:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_unit": "0", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_enable_alloc_hint": "true", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler": "simple", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-30T14:56:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_order": "24", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_pool": "", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_splay_width": "4", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_parent_reads": "false", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_snap_reads": "false", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-30T14:56:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-30T14:56:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mtime_update_interval": "60", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_non_blocking_aio": "true", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_thread_timeout": "60", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_threads": "1", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_parent_cache_enabled": "false", 2024-03-30T14:56:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_plugins": "", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_limit": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_exclude_ops": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_limit": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-30T14:56:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_read_from_replica_policy": "default", 2024-03-30T14:56:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_skip_partial_discard": "true", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_tracing": "false", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_names": "true", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_pool": "true", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_base_url": "", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_log_level": "", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_acl_grants_max_num": "100", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_admin_entry": "admin", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-30T14:56:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_backend_store": "rados", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_barbican_url": "", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_beast_enable_async": "true", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache": "false", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_enabled": "true", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_expiry_interval": "900", 2024-03-30T14:56:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_lru_size": "10000", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_config_store": "rados", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_content_length_compat": "false", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress": "true", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cors_rules_max_num": "100", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cross_domain_policy": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_addr": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-30T14:56:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_password": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_username": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_require_ssl": "true", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-30T14:56:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_addr": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_auth": "token", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_namespace": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_prefix": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_token_file": "", 2024-03-30T14:56:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_buffersize": "524288", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_time": "300", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-30T14:56:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_port": "6379", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_changes_size": "1000", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_num_shards": "128", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_window": "30", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-30T14:56:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-30T14:56:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_name": "", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_s3website_name": "", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dynamic_resharding": "true", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_gc_threads": "true", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_lc_threads": "true", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_ops_log": "false", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_quota_threads": "true", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_static_website": "false", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_usage_log": "false", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enforce_swift_acls": "true", 2024-03-30T14:56:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_exit_timeout_secs": "120", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_expose_bucket": "false", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_extended_http_attrs": "", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_filter": "none", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontends": "beast port=7480", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred": "50", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_objs": "32", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-30T14:56:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_period": "3600", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_init_timeout": "300", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_connection_idle": "300", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_domain": "", 2024-03-30T14:56:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password_path": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_project": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_tenant": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token_path": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_user": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_api_version": "2", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_domain": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_password": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_project": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_user": "", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-30T14:56:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_url": "", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_debug_interval": "-1", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_lock_max_time": "90", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_objs": "32", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_rules": "1000", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_worker": "3", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_thread_delay": "0", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-30T14:56:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_dnattr": "uid", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchfilter": "", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_http_headers": "", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name_utc": "false", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-30T14:56:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_name_len": "0", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_size": "0", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_chunk_size": "4194304", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_listing_results": "1000", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_notify_retries": "10", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_param_size": "1048576", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_size": "5368709120", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_slo_entries": "1000", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_log_max_shards": "64", 2024-03-30T14:56:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mp_lock_max_time": "600", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_max_gc": "300", 2024-03-30T14:56:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_async_rados_threads": "32", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_control_oids": "8", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_numa_node": "-1", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_chunk_size": "100", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_gc_interval": "600", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-30T14:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_timeout": "600", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_tracing": "false", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_token": "", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_url": "", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_verify_ssl": "true", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_rados": "false", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_socket_path": "", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-30T14:56:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval": "2.000000", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lanes": "3", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_partitions": "3", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_continue": "true", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-30T14:56:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_tracing": "false", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm": "", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_id": "", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region": "", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_request_uri": "", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_batch_size": "64", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-30T14:56:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_max_aio": "128", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_num_logs": "16", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_thread_interval": "600", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_resolve_cname": "false", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_run_sync_thread": "true", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_scheduler_type": "throttler", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_script_uri": "", 2024-03-30T14:56:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_service_provider_name": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_id": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_secret": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_entry": "sts", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_key": "sts", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_min_session_duration": "900", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_token_introspection_url": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_account_in_url": "false", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_entry": "auth", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_url": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_custom_header": "", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_need_stats": "true", 2024-03-30T14:56:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_tenant_name": "", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_token_expiration": "86400", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url": "", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url_prefix": "swift", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_lease_period": "120", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-30T14:56:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_thread_pool_size": "512", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_comment": "", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_createby": "", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_encoding": "", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_flag": "false", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_origin": "", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-30T14:56:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_tracker": "", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_trust_forwarded_https": "false", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_shards": "32", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_user_shards": "1", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_use_opa_authz": "false", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache": "false", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_max_buckets": "1000", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_policies_max_num": "100", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-30T14:56:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_unique_email": "true", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_verify_ssl": "true", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone": "", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_id": "", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup": "", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_id": "", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_auto_session_clear": "true", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_block_size": "4096", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-30T14:56:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_size": "536870912", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_index_type": "binary_search", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-30T14:56:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_partition_filters": "false", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_perf": "false", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "run_dir": "/var/run/ceph", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_block_create": "true", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cache_lru_size": "67108864", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_max_object_size": "16777216", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_device_size": "53687091200", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_capacity": "16", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-30T14:56:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_main_device_type": "SSD", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_segment_size": "67108864", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "set_keepcaps": "false", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "setgroup": "ceph", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser": "ceph", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser_match_path": "", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-30T14:56:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "thp": "false", 2024-03-30T14:56:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_default_timeout": "60", 2024-03-30T14:56:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-30T14:56:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "throttler_perf_counter": "true" 2024-03-30T14:56:25.206 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[29876]: 4.3 deep-scrub starts 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[29876]: 4.3 deep-scrub ok 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[41614]: 4.3 deep-scrub starts 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[41614]: 4.3 deep-scrub ok 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T14:56:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:25 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T14:56:25.522 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:25 smithi134 ceph-mon[33714]: 4.3 deep-scrub starts 2024-03-30T14:56:25.523 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:25 smithi134 ceph-mon[33714]: 4.3 deep-scrub ok 2024-03-30T14:56:25.523 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:25 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T14:56:25.523 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:25 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T14:56:26.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:26.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T14:56:26.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:26 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T14:56:26.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T14:56:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:26 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T14:56:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:27 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T14:56:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:27 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T14:56:27.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:27 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T14:56:27.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:27 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T14:56:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:27 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T14:56:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:27 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T14:56:28.367 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:56:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:56:28.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 5.1e scrub starts 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 5.1e scrub ok 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 6.12 deep-scrub starts 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[41614]: 6.12 deep-scrub ok 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 5.1e scrub starts 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 5.1e scrub ok 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 6.12 deep-scrub starts 2024-03-30T14:56:28.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:28 smithi028 ceph-mon[29876]: 6.12 deep-scrub ok 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 5.1e scrub starts 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 5.1e scrub ok 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 6.12 deep-scrub starts 2024-03-30T14:56:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:28 smithi134 ceph-mon[33714]: 6.12 deep-scrub ok 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.262+0000 7f1eb4da6700 1 -- 172.21.15.28:0/3331746111 >> v1:172.21.15.28:6790/0 conn(0x7f1eb0109dd0 legacy=0x7f1eb010c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.262+0000 7f1eb4da6700 1 -- 172.21.15.28:0/3331746111 shutdown_connections 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.262+0000 7f1eb4da6700 1 -- 172.21.15.28:0/3331746111 >> 172.21.15.28:0/3331746111 conn(0x7f1eb00fc9f0 msgr2=0x7f1eb00fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.262+0000 7f1eb4da6700 1 -- 172.21.15.28:0/3331746111 shutdown_connections 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.262+0000 7f1eb4da6700 1 -- 172.21.15.28:0/3331746111 wait complete. 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 librados: starting msgr at 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 librados: starting objecter 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 Processor -- start 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 -- start start 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 librados: setting wanted keys 2024-03-30T14:56:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 librados: calling monclient init 2024-03-30T14:56:29.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1eb01a92e0 con 0x7f1eb0101140 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1eb01aa500 con 0x7f1eb0109dd0 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eb4da6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1eb01ab740 con 0x7f1eb010d970 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eae59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1eb0101140 0x7f1eb010f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47088/0 (socket says 172.21.15.28:47088) 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1eae59c700 1 -- 172.21.15.28:0/826709670 learned_addr learned my addr 172.21.15.28:0/826709670 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter ms_handle_connect 0x7f1eb010d970 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter ms_handle_connect 0x7f1eb0101140 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter ms_handle_connect 0x7f1eb0109dd0 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.263+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141350925 0 0) 0x7f1eb01ab740 con 0x7f1eb010d970 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e9c003aa0 con 0x7f1eb010d970 2024-03-30T14:56:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71989110 0 0) 0x7f1eb01a92e0 con 0x7f1eb0101140 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb01ab740 con 0x7f1eb0101140 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593729085 0 0) 0x7f1eb01aa500 con 0x7f1eb0109dd0 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb01a92e0 con 0x7f1eb0109dd0 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1513663750 0 0) 0x7f1e9c003aa0 con 0x7f1eb010d970 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb01aa500 con 0x7f1eb010d970 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea8001b70 con 0x7f1eb010d970 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1210198475 0 0) 0x7f1eb01ab740 con 0x7f1eb0101140 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e9c003aa0 con 0x7f1eb0101140 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1e98001890 con 0x7f1eb0101140 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 673492923 0 0) 0x7f1eb01a92e0 con 0x7f1eb0109dd0 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb01ab740 con 0x7f1eb0109dd0 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0003860 con 0x7f1eb0109dd0 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2009094249 0 0) 0x7f1eb01aa500 con 0x7f1eb010d970 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.134:6789/0 conn(0x7f1eb0109dd0 legacy=0x7f1eb01a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6789/0 conn(0x7f1eb0101140 legacy=0x7f1eb010f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb01ac980 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb01a9510 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1eb4da6700 10 client.34667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1eb01a99d0 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1eb4da6700 1 librados: init done 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1ea8003ea0 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.264+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea8004e00 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.265+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1e90005140 con 0x7f1eb010d970 2024-03-30T14:56:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1ea801b4a0 con 0x7f1eb010d970 2024-03-30T14:56:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(95..95 src has 43..95) ==== 6259+0+0 (unknown 2966647193 0 0) 0x7f1ea8087fc0 con 0x7f1eb010d970 2024-03-30T14:56:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 10 client.34667.objecter ms_dispatch 0x7f1eb0001550 osd_map(95..95 src has 43..95) 2024-03-30T14:56:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 3 client.34667.objecter handle_osd_map got epochs [95,95] > 0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 3 client.34667.objecter handle_osd_map decoding full epoch 95 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 20 client.34667.objecter dump_active .. 0 homeless 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1127 (unknown 409899127 0 633015442) 0x7f1ea808a440 con 0x7f1eb010d970 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e9c070470 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.266+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e90004dc0 con 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669330582 0 0) 0x7f1e90004dc0 con 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb01aa500 con 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3616363787 0 0) 0x7f1eb01aa500 con 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e90004dc0 con 0x7f1e900027a0 2024-03-30T14:56:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0001890 con 0x7f1e900027a0 2024-03-30T14:56:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1058515524 0 0) 0x7f1e90004dc0 con 0x7f1e900027a0 2024-03-30T14:56:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f1e9000a740 con 0x7f1e900027a0 2024-03-30T14:56:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.267+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1ea0001b80 con 0x7f1e900027a0 2024-03-30T14:56:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.268+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6789/0 conn(0x7f1e900027a0 legacy=0x7f1e900056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.268+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6789/0 conn(0x7f1e900027a0 legacy=0x7f1e900056f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e9000af20 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293753609 0 0) 0x7f1e9000af20 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e90004dc0 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1331808047 0 0) 0x7f1e90004dc0 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e9000af20 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0003f50 con 0x7f1e9000aae0 2024-03-30T14:56:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 830749425 0 0) 0x7f1e9000af20 con 0x7f1e9000aae0 2024-03-30T14:56:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.275+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f1e9000c180 con 0x7f1e9000aae0 2024-03-30T14:56:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.0 v1:172.21.15.28:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f1ea0003db0 con 0x7f1e9000aae0 2024-03-30T14:56:29.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6789/0 conn(0x7f1e9000aae0 legacy=0x7f1e90008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6789/0 conn(0x7f1e9000aae0 legacy=0x7f1e90008e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.278 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e90002310 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.277+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765659894 0 0) 0x7f1e90002310 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e9000af20 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2653144815 0 0) 0x7f1e9000af20 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e90002310 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0004de0 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3025426391 0 0) 0x7f1e90002310 con 0x7f1e90001f00 2024-03-30T14:56:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f1e90012ce0 con 0x7f1e90001f00 2024-03-30T14:56:29.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 5 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1ea0007480 con 0x7f1e90001f00 2024-03-30T14:56:29.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.134:6789/0 conn(0x7f1e90001f00 legacy=0x7f1e9000b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.134:6789/0 conn(0x7f1e90001f00 legacy=0x7f1e9000b8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.278+0000 7f1eaed9d700 1 -- 172.21.15.28:0/826709670 reap_dead start 2024-03-30T14:56:29.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.285+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e900134a0 con 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700597995 0 0) 0x7f1e900134a0 con 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e90002310 con 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1443268207 0 0) 0x7f1e90002310 con 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e900134a0 con 0x7f1e90013070 2024-03-30T14:56:29.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0005390 con 0x7f1e90013070 2024-03-30T14:56:29.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 853545291 0 0) 0x7f1e900134a0 con 0x7f1e90013070 2024-03-30T14:56:29.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.286+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.134:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f1e90014680 con 0x7f1e90013070 2024-03-30T14:56:29.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.288+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.1 v1:172.21.15.134:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f1ea0006020 con 0x7f1e90013070 2024-03-30T14:56:29.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.288+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.134:6789/0 conn(0x7f1e90013070 legacy=0x7f1e900113c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.288+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.134:6789/0 conn(0x7f1e90013070 legacy=0x7f1e900113c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.290 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-03-30T14:56:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e90014380 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685353667 0 0) 0x7f1e90014380 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e900134a0 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1961767708 0 0) 0x7f1e900134a0 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e90014380 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0005940 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 352604455 0 0) 0x7f1e90014380 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.289+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f1e9001b1e0 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.290+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 5 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1ea0005b20 con 0x7f1e90013790 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.290+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6790/0 conn(0x7f1e90013790 legacy=0x7f1e90013bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.290+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6790/0 conn(0x7f1e90013790 legacy=0x7f1e90013bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f1e90019070 con 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 10 client.34667.objecter ms_handle_connect 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 10 client.34667.objecter resend_mon_ops 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114295567 0 0) 0x7f1e90019070 con 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e90014380 con 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2195616538 0 0) 0x7f1e90014380 con 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e90019070 con 0x7f1e9001b3c0 2024-03-30T14:56:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1ea0006570 con 0x7f1e9001b3c0 2024-03-30T14:56:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1254335909 0 0) 0x7f1e90019070 con 0x7f1e9001b3c0 2024-03-30T14:56:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.297+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 --> v1:172.21.15.28:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f1e9001a2a0 con 0x7f1e9001b3c0 2024-03-30T14:56:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 <== mon.2 v1:172.21.15.28:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f1ea0007630 con 0x7f1e9001b3c0 2024-03-30T14:56:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6790/0 conn(0x7f1e9001b3c0 legacy=0x7f1e900178b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1ea77fe700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6790/0 conn(0x7f1e9001b3c0 legacy=0x7f1e900178b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T14:56:29.300 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 10 librados: watch_flush enter 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 10 librados: watch_flush exit 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 20 client.34667.objecter shutdown clearing up homeless session... 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 10 client.34667.objecter successfully canceled tick 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1e9c070470 legacy=0x7f1e9c072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 >> v1:172.21.15.28:6790/0 conn(0x7f1eb010d970 legacy=0x7f1eb01a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eaed9d700 1 -- 172.21.15.28:0/826709670 reap_dead start 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 shutdown_connections 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 >> 172.21.15.28:0/826709670 conn(0x7f1eb00fc9f0 msgr2=0x7f1eb01105c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 shutdown_connections 2024-03-30T14:56:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 -- 172.21.15.28:0/826709670 wait complete. 2024-03-30T14:56:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:29.299+0000 7f1eb4da6700 1 librados: shutdown 2024-03-30T14:56:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:56:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:56:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 5.19 deep-scrub starts 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 5.19 deep-scrub ok 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 3.6 scrub starts 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 3.6 scrub ok 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T14:56:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/826709670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:56:29.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 5.19 deep-scrub starts 2024-03-30T14:56:29.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 5.19 deep-scrub ok 2024-03-30T14:56:29.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 3.6 scrub starts 2024-03-30T14:56:29.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 3.6 scrub ok 2024-03-30T14:56:29.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/826709670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 5.19 deep-scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 5.19 deep-scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 3.6 scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 3.6 scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T14:56:29.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/826709670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T14:56:30.067 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T14:56:30.081 INFO:tasks.workunit:Pulling workunits from ref 5af0400050a123dc31f105425174f106e057276a 2024-03-30T14:56:30.081 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T14:56:30.081 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T14:56:30.082 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-30T14:56:30.082 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T14:56:30.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:56:30.082 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-30T14:56:30.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-30T14:56:30.083 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-30T14:56:30.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:30.083 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-30T14:56:30.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:56:30.084 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-30T14:56:30.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T14:56:30.084 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-30T14:56:30.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:56:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-30T14:56:30.103 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T14:56:30.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:30.165 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-30T14:56:30.166 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-30T14:56:30.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:30.225 INFO:tasks.workunit:timeout=3h 2024-03-30T14:56:30.225 INFO:tasks.workunit:cleanup=True 2024-03-30T14:56:30.226 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5af0400050a123dc31f105425174f106e057276a 2024-03-30T14:56:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:30.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:30.287 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T14:56:30.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:30.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:30.380 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:56:29 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:30.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:30.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:30.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T14:56:30.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T14:56:30.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:30 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T14:56:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:30 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T14:56:30.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:30.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:30.890 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:30.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:30.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:31.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:31.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:31.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:31.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:31.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:31.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:31.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:31.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:31.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:31.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:31.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: 3.8 deep-scrub starts 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: 3.8 deep-scrub ok 2024-03-30T14:56:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: 3.8 deep-scrub starts 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: 3.8 deep-scrub ok 2024-03-30T14:56:31.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T14:56:31.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:31 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T14:56:31.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:31.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:31.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: Reconfiguring daemon alertmanager.a on smithi028 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: 3.8 deep-scrub starts 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: 3.8 deep-scrub ok 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T14:56:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:31 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T14:56:32.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:32.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:32.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:32.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:32.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:32.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:32.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:32.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:32.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:32.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:32.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:32.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:32.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T14:56:32.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T14:56:32.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[41614]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-30T14:56:32.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T14:56:32.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T14:56:32.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:32 smithi028 ceph-mon[29876]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-30T14:56:32.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:32.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:32.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:32 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T14:56:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:32 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T14:56:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:32 smithi134 ceph-mon[33714]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-30T14:56:32.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:32.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:33.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:33.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:33.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:33.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:33.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:33.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:33.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:33.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:33.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T14:56:33.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T14:56:33.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T14:56:33.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:33.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T14:56:33.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:33 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T14:56:33.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:33.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:33.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:33.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:33.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:33 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T14:56:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:33 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T14:56:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:33 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:33 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T14:56:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:33 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T14:56:33.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:33.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:34.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:34.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:34.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:34.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:34.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:34.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:34.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:34.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:34.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:34.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 5.1d deep-scrub starts 2024-03-30T14:56:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 5.1d deep-scrub ok 2024-03-30T14:56:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:56:34.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[41614]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 5.1d deep-scrub starts 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 5.1d deep-scrub ok 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:56:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T14:56:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T14:56:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T14:56:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T14:56:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T14:56:34.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:34 smithi028 ceph-mon[29876]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:34.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:34.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 5.1d deep-scrub starts 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 5.1d deep-scrub ok 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T14:56:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T14:56:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: osdmap e96: 8 total, 8 up, 8 in 2024-03-30T14:56:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T14:56:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T14:56:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:34 smithi134 ceph-mon[33714]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:34.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:34.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:35.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:35.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:35.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:35.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:35.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:35.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:35.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:35.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:35.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:35.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:35 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T14:56:35.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:35 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T14:56:35.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:35 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T14:56:35.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:35 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T14:56:35.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:35.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:35.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:35.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:35 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T14:56:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:35 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 -- 172.21.15.28:0/1718681983 >> v1:172.21.15.28:6789/0 conn(0x7f1454101160 legacy=0x7f1454101540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 -- 172.21.15.28:0/1718681983 shutdown_connections 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 -- 172.21.15.28:0/1718681983 >> 172.21.15.28:0/1718681983 conn(0x7f14540fca30 msgr2=0x7f14540fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 -- 172.21.15.28:0/1718681983 shutdown_connections 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 -- 172.21.15.28:0/1718681983 wait complete. 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 librados: starting msgr at 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 librados: starting objecter 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.942+0000 7f1459ebf700 1 Processor -- start 2024-03-30T14:56:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 -- start start 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 librados: setting wanted keys 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 librados: calling monclient init 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f145406b2b0 con 0x7f1454101160 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f145406b4c0 con 0x7f145410d990 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1459ebf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f14541a95c0 con 0x7f1454109df0 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1452ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1454109df0 0x7f1454105dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38966/0 (socket says 172.21.15.28:38966) 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1452ffd700 1 -- 172.21.15.28:0/3026110686 learned_addr learned my addr 172.21.15.28:0/3026110686 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter ms_handle_connect 0x7f1454109df0 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter ms_handle_connect 0x7f1454101160 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter ms_handle_connect 0x7f145410d990 2024-03-30T14:56:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523071040 0 0) 0x7f14541a95c0 con 0x7f1454109df0 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f143c0031f0 con 0x7f1454109df0 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025045835 0 0) 0x7f145406b2b0 con 0x7f1454101160 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14541a95c0 con 0x7f1454101160 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465575025 0 0) 0x7f145406b4c0 con 0x7f145410d990 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f145406b2b0 con 0x7f145410d990 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2094399842 0 0) 0x7f143c0031f0 con 0x7f1454109df0 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.943+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f145406b4c0 con 0x7f1454109df0 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1448001230 con 0x7f1454109df0 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4044639344 0 0) 0x7f14541a95c0 con 0x7f1454101160 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f143c0031f0 con 0x7f1454101160 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1547363767 0 0) 0x7f145406b2b0 con 0x7f145410d990 2024-03-30T14:56:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14541a95c0 con 0x7f145410d990 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f14440043b0 con 0x7f1454101160 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f144c001af0 con 0x7f145410d990 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2163126052 0 0) 0x7f145406b4c0 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 >> v1:172.21.15.134:6789/0 conn(0x7f145410d990 legacy=0x7f14541a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 >> v1:172.21.15.28:6789/0 conn(0x7f1454101160 legacy=0x7f14541056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14541aa820 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14541a97f0 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1459ebf700 10 client.34679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f14541a9cb0 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1459ebf700 1 librados: init done 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1448003b20 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.944+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1448005c60 con 0x7f1454109df0 2024-03-30T14:56:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1459ebf700 10 client.34679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1459ebf700 15 client.34679.objecter _session_command_op_assign -1 1 2024-03-30T14:56:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1459ebf700 20 client.34679.objecter _get_session osd=-1 returning homeless 2024-03-30T14:56:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1459ebf700 10 client.34679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1440002920 con 0x7f1454109df0 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.945+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1448003680 con 0x7f1454109df0 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(96..96 src has 43..96) ==== 6259+0+0 (unknown 896004952 0 0) 0x7f14480890a0 con 0x7f1454109df0 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 10 client.34679.objecter ms_dispatch 0x7f1454001550 osd_map(96..96 src has 43..96) 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 3 client.34679.objecter handle_osd_map got epochs [96,96] > 0 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 3 client.34679.objecter handle_osd_map decoding full epoch 96 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 10 client.34679.objecter checking command 1 2024-03-30T14:56:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter _get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 15 client.34679.objecter _session_command_op_remove -1 1 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter _get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 15 client.34679.objecter _session_command_op_assign 0 1 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 4 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 10 client.34679.objecter _send_command 1 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1444001080 con 0x7f143c07f0c0 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 20 client.34679.objecter dump_active .. 0 homeless 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (unknown 1524946726 0 0) 0x7f14480894e0 con 0x7f1454109df0 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 10 client.34679.objecter ms_handle_connect 0x7f143c074960 2024-03-30T14:56:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.946+0000 7f1450ff9700 10 client.34679.objecter ms_handle_connect 0x7f143c07f0c0 2024-03-30T14:56:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.947+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1444001080 con 0x7f143c07f0c0 2024-03-30T14:56:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.947+0000 7f1450ff9700 10 client.34679.objecter ms_dispatch 0x7f1454001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.947+0000 7f1450ff9700 10 client.34679.objecter _finish_command 1 = system:0 2024-03-30T14:56:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.947+0000 7f1450ff9700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 4 2024-03-30T14:56:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.947+0000 7f1450ff9700 15 client.34679.objecter _session_command_op_remove 0 1 2024-03-30T14:56:35.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 10 client.34679.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:56:35.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 15 client.34679.objecter _session_command_op_assign -1 2 2024-03-30T14:56:35.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 20 client.34679.objecter _get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 20 client.34679.objecter _get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 15 client.34679.objecter _session_command_op_remove -1 2 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 20 client.34679.objecter get_session s=0x7f143c07ea10 osd=0 3 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 15 client.34679.objecter _session_command_op_assign 0 2 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 4 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 10 client.34679.objecter _send_command 2 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.966+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f14400011f0 con 0x7f143c07f0c0 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1450ff9700 1 -- 172.21.15.28:0/3026110686 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f14400011f0 con 0x7f143c07f0c0 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1450ff9700 10 client.34679.objecter ms_dispatch 0x7f1454001550 command_reply(tid 2: 0 ) 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1450ff9700 10 client.34679.objecter _finish_command 2 = system:0 2024-03-30T14:56:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1450ff9700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 4 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1450ff9700 15 client.34679.objecter _session_command_op_remove 0 2 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 10 librados: watch_flush enter 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 10 librados: watch_flush exit 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 10 client.34679.objecter close_session for osd.0 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 >> v1:172.21.15.28:6801/3394323575 conn(0x7f143c07f0c0 legacy=0x7f143c081560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 20 client.34679.objecter put_session s=0x7f143c07ea10 osd=0 1 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 20 client.34679.objecter shutdown clearing up homeless session... 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 10 client.34679.objecter successfully canceled tick 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 >> v1:172.21.15.28:6800/3011431093 conn(0x7f143c074960 legacy=0x7f143c076e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 >> v1:172.21.15.28:6790/0 conn(0x7f1454109df0 legacy=0x7f1454105dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1453fff700 1 -- 172.21.15.28:0/3026110686 reap_dead start 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 shutdown_connections 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 >> 172.21.15.28:0/3026110686 conn(0x7f14540fca30 msgr2=0x7f1454110b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 shutdown_connections 2024-03-30T14:56:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 -- 172.21.15.28:0/3026110686 wait complete. 2024-03-30T14:56:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:35.967+0000 7f1459ebf700 1 librados: shutdown 2024-03-30T14:56:35.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:36.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:36.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:36.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:36.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:36.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:36.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:36.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:36.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:36.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:36.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:36.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:36.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.4 deep-scrub starts 2024-03-30T14:56:36.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.4 deep-scrub ok 2024-03-30T14:56:36.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.16 deep-scrub starts 2024-03-30T14:56:36.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.16 deep-scrub ok 2024-03-30T14:56:36.745 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.4 deep-scrub starts 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.4 deep-scrub ok 2024-03-30T14:56:36.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.16 deep-scrub starts 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.16 deep-scrub ok 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T14:56:36.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:36 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T14:56:36.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:36.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:36.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.4 deep-scrub starts 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.4 deep-scrub ok 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.16 deep-scrub starts 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.16 deep-scrub ok 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T14:56:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T14:56:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T14:56:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T14:56:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:36 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T14:56:36.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:36.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:37.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:37.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:37.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:37.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:37.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:37.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:37.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:37.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:37.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:37 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T14:56:37.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:37 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T14:56:37.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:37 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T14:56:37.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:37 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T14:56:37.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:37.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:37.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:37.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:37 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T14:56:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:37 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T14:56:37.953 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:56:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:56:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T14:56:37.954 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:56:38.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:38.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:38.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:38.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.979+0000 7f83b5500700 1 -- 172.21.15.28:0/993849144 >> v1:172.21.15.28:6790/0 conn(0x7f83b0109dc0 legacy=0x7f83b010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.979+0000 7f83b5500700 1 -- 172.21.15.28:0/993849144 shutdown_connections 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.979+0000 7f83b5500700 1 -- 172.21.15.28:0/993849144 >> 172.21.15.28:0/993849144 conn(0x7f83b00fc9e0 msgr2=0x7f83b00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.979+0000 7f83b5500700 1 -- 172.21.15.28:0/993849144 shutdown_connections 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.979+0000 7f83b5500700 1 -- 172.21.15.28:0/993849144 wait complete. 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 librados: starting msgr at 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 librados: starting objecter 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 Processor -- start 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 -- start start 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 librados: setting wanted keys 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 librados: calling monclient init 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83b006a9a0 con 0x7f83b0109dc0 2024-03-30T14:56:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83b01a8540 con 0x7f83b0101130 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83b5500700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f83b01a9720 con 0x7f83b010d960 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83ae7fc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f83b0109dc0 0x7f83b0105e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47150/0 (socket says 172.21.15.28:47150) 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83af7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f83b010d960 0x7f83b01a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38980/0 (socket says 172.21.15.28:38980) 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f83ae7fc700 1 -- 172.21.15.28:0/2907090354 learned_addr learned my addr 172.21.15.28:0/2907090354 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter ms_handle_connect 0x7f83b0109dc0 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter ms_handle_connect 0x7f83b010d960 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396997976 0 0) 0x7f83b006a9a0 con 0x7f83b0109dc0 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83980031f0 con 0x7f83b0109dc0 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter ms_handle_connect 0x7f83b0101130 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313624886 0 0) 0x7f83b01a9720 con 0x7f83b010d960 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.980+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83b006a9a0 con 0x7f83b010d960 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1926251508 0 0) 0x7f83b01a8540 con 0x7f83b0101130 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83b01a9720 con 0x7f83b0101130 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2473422088 0 0) 0x7f83980031f0 con 0x7f83b0109dc0 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83b01a8540 con 0x7f83b0109dc0 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83a40045f0 con 0x7f83b0109dc0 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3226311629 0 0) 0x7f83b006a9a0 con 0x7f83b010d960 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83980031f0 con 0x7f83b010d960 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83a80014b0 con 0x7f83b010d960 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 279879781 0 0) 0x7f83b01a9720 con 0x7f83b0101130 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83b006a9a0 con 0x7f83b0101130 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83a0001870 con 0x7f83b0101130 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4054686031 0 0) 0x7f83b01a8540 con 0x7f83b0109dc0 2024-03-30T14:56:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6790/0 conn(0x7f83b010d960 legacy=0x7f83b01a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.134:6789/0 conn(0x7f83b0101130 legacy=0x7f83b01056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b01aa900 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83b01a9950 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f83b5500700 10 client.14796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83b01a9e10 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f83b5500700 1 librados: init done 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f83a4001c10 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.981+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f83a4004d50 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.982+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f839c005100 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f83a4001770 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(96..96 src has 43..96) ==== 6259+0+0 (unknown 896004952 0 0) 0x7f83a4088d40 con 0x7f83b0109dc0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 osd_map(96..96 src has 43..96) 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 3 client.14796.objecter handle_osd_map got epochs [96,96] > 0 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 3 client.14796.objecter handle_osd_map decoding full epoch 96 2024-03-30T14:56:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 20 client.14796.objecter dump_active .. 0 homeless 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) ==== 54+0+16 (unknown 2433924269 0 2420000192) 0x7f83a40891a0 con 0x7f83b0109dc0 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f83980704c0 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 10 client.14796.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 1 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:56:38.310 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 1 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 0 1 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 4 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 10 client.14796.objecter _send_command 1 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.983+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f839c004b50 con 0x7f839c004730 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c004730 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c004b50 con 0x7f839c004730 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 10 client.14796.objecter _finish_command 1 = system:0 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 4 2024-03-30T14:56:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:37.984+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 0 1 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 10 client.14796.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 2 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 2 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0053e0 osd=0 3 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 0 2 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 4 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 10 client.14796.objecter _send_command 2 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.001+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c0079f0 con 0x7f839c004730 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c0079f0 con 0x7f839c004730 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f8397fff700 10 client.14796.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 4 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 0 2 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 10 client.14796.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 1 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 4 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 10 client.14796.objecter _send_command 3 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.003+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f839c002180 con 0x7f839c004280 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c004280 2024-03-30T14:56:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c002180 con 0x7f839c004280 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 3: 0 ) 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 10 client.14796.objecter _finish_command 3 = system:0 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.004+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 1 3 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 10 client.14796.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c004110 osd=1 3 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 1 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 10 client.14796.objecter _send_command 4 2024-03-30T14:56:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.021+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c001600 con 0x7f839c004280 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c001600 con 0x7f839c004280 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 10 client.14796.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 4 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 1 4 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 10 client.14796.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 5 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 5 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 2 5 2024-03-30T14:56:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 4 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 10 client.14796.objecter _send_command 5 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f839c0110e0 con 0x7f839c00d5d0 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.023+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c00d5d0 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.024+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c0110e0 con 0x7f839c00d5d0 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.024+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 5: 0 ) 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.024+0000 7f8397fff700 10 client.14796.objecter _finish_command 5 = system:0 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.024+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 4 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.024+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 2 5 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 10 client.14796.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 6 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 6 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c008370 osd=2 3 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 2 6 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 4 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 10 client.14796.objecter _send_command 6 2024-03-30T14:56:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.040+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c007eb0 con 0x7f839c00d5d0 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c007eb0 con 0x7f839c00d5d0 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 10 client.14796.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 4 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 2 6 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 10 client.14796.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 7 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 7 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 3 7 2024-03-30T14:56:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 4 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 10 client.14796.objecter _send_command 7 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f839c015f20 con 0x7f839c012410 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c012410 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c015f20 con 0x7f839c012410 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 7: 0 ) 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 10 client.14796.objecter _finish_command 7 = system:0 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 4 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.043+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 3 7 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 10 client.14796.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 8 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 8 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c00cc20 osd=3 3 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 3 8 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 4 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 10 client.14796.objecter _send_command 8 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.060+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c00c700 con 0x7f839c012410 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c00c700 con 0x7f839c012410 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 10 client.14796.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 4 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 3 8 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 10 client.14796.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 9 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 9 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 4 9 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 4 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 10 client.14796.objecter _send_command 9 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f839c01ee00 con 0x7f839c011f70 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.062+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c011f70 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.063+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c01ee00 con 0x7f839c011f70 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.063+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 9: 0 ) 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.063+0000 7f8397fff700 10 client.14796.objecter _finish_command 9 = system:0 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.063+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 4 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.063+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 4 9 2024-03-30T14:56:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 10 client.14796.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 10 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 10 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c00c700 osd=4 3 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 4 10 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 4 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 10 client.14796.objecter _send_command 10 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.080+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c0207b0 con 0x7f839c011f70 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.081+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c0207b0 con 0x7f839c011f70 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.081+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.081+0000 7f8397fff700 10 client.14796.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.081+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 4 2024-03-30T14:56:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.081+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 4 10 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 10 client.14796.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 11 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 11 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 5 11 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 4 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 10 client.14796.objecter _send_command 11 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f839c021350 con 0x7f839c020ed0 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.082+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c020ed0 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.084+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c021350 con 0x7f839c020ed0 2024-03-30T14:56:38.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.084+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 11: 0 ) 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.084+0000 7f8397fff700 10 client.14796.objecter _finish_command 11 = system:0 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.084+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 4 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.084+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 5 11 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 10 client.14796.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 12 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 12 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0207b0 osd=5 3 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 5 12 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 4 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 10 client.14796.objecter _send_command 12 2024-03-30T14:56:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.101+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c021580 con 0x7f839c020ed0 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c021580 con 0x7f839c020ed0 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 10 client.14796.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 4 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 5 12 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 10 client.14796.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 13 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 13 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 6 13 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 4 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 10 client.14796.objecter _send_command 13 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f839c0279d0 con 0x7f839c023ec0 2024-03-30T14:56:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.104+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c023ec0 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.105+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c0279d0 con 0x7f839c023ec0 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.105+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 13: 0 ) 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.105+0000 7f8397fff700 10 client.14796.objecter _finish_command 13 = system:0 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.105+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 4 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.105+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 6 13 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 10 client.14796.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 14 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 14 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c021580 osd=6 3 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 6 14 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 4 2024-03-30T14:56:38.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 10 client.14796.objecter _send_command 14 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.121+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c0279d0 con 0x7f839c023ec0 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c0279d0 con 0x7f839c023ec0 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 10 client.14796.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 4 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 6 14 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 10 client.14796.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 15 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 15 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 7 15 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 4 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 10 client.14796.objecter _send_command 15 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f839c02bd70 con 0x7f839c028050 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.124+0000 7f8397fff700 10 client.14796.objecter ms_handle_connect 0x7f839c028050 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.125+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f839c02bd70 con 0x7f839c028050 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.125+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 15: 0 ) 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.125+0000 7f8397fff700 10 client.14796.objecter _finish_command 15 = system:0 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.125+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 4 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.125+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 7 15 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 10 client.14796.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign -1 16 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 20 client.14796.objecter _get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 15 client.14796.objecter _session_command_op_remove -1 16 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 20 client.14796.objecter get_session s=0x7f839c0279d0 osd=7 3 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 15 client.14796.objecter _session_command_op_assign 7 16 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 4 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 10 client.14796.objecter _send_command 16 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.143+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f839c02bd70 con 0x7f839c028050 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f8397fff700 1 -- 172.21.15.28:0/2907090354 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f839c02bd70 con 0x7f839c028050 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f8397fff700 10 client.14796.objecter ms_dispatch 0x7f83b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f8397fff700 10 client.14796.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f8397fff700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 4 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f8397fff700 15 client.14796.objecter _session_command_op_remove 7 16 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 librados: watch_flush enter 2024-03-30T14:56:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 librados: watch_flush exit 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.0 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6801/3394323575 conn(0x7f839c004730 legacy=0x7f839c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0053e0 osd=0 1 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.1 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6805/1850313547 conn(0x7f839c004280 legacy=0x7f839c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c004110 osd=1 1 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.2 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6809/3451325043 conn(0x7f839c00d5d0 legacy=0x7f839c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c008370 osd=2 1 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.3 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6813/1506157250 conn(0x7f839c012410 legacy=0x7f839c014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00cc20 osd=3 1 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.4 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.134:6800/3063775514 conn(0x7f839c011f70 legacy=0x7f839c01d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c00c700 osd=4 1 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.5 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.134:6804/3668151179 conn(0x7f839c020ed0 legacy=0x7f839c01efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0207b0 osd=5 1 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.6 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.134:6808/2135224194 conn(0x7f839c023ec0 legacy=0x7f839c0262b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c021580 osd=6 1 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter close_session for osd.7 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.134:6812/3758190298 conn(0x7f839c028050 legacy=0x7f839c02a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter put_session s=0x7f839c0279d0 osd=7 1 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 20 client.14796.objecter shutdown clearing up homeless session... 2024-03-30T14:56:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 10 client.14796.objecter successfully canceled tick 2024-03-30T14:56:38.332 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:56:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6800/3011431093 conn(0x7f83980704c0 legacy=0x7f8398072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> v1:172.21.15.28:6789/0 conn(0x7f83b0109dc0 legacy=0x7f83b0105e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83af7fe700 1 -- 172.21.15.28:0/2907090354 reap_dead start 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83af7fe700 1 -- 172.21.15.28:0/2907090354 reap_dead start 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83af7fe700 1 -- 172.21.15.28:0/2907090354 reap_dead start 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83af7fe700 1 -- 172.21.15.28:0/2907090354 reap_dead start 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83af7fe700 1 -- 172.21.15.28:0/2907090354 reap_dead start 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 shutdown_connections 2024-03-30T14:56:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 >> 172.21.15.28:0/2907090354 conn(0x7f83b00fc9e0 msgr2=0x7f83b0110e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 shutdown_connections 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 -- 172.21.15.28:0/2907090354 wait complete. 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.146+0000 7f83b5500700 1 librados: shutdown 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:56:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:56:38.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:38.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:38.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: 3.7 deep-scrub starts 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: 3.7 deep-scrub ok 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2907090354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:38.592 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: 3.7 deep-scrub starts 2024-03-30T14:56:38.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: 3.7 deep-scrub ok 2024-03-30T14:56:38.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2907090354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:38.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T14:56:38.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:38 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T14:56:38.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:38.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 -- 172.21.15.28:0/4097493753 >> v1:172.21.15.28:6789/0 conn(0x7fc31410d960 legacy=0x7fc31410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 -- 172.21.15.28:0/4097493753 shutdown_connections 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 -- 172.21.15.28:0/4097493753 >> 172.21.15.28:0/4097493753 conn(0x7fc3140fc9e0 msgr2=0x7fc3140fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 -- 172.21.15.28:0/4097493753 shutdown_connections 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 -- 172.21.15.28:0/4097493753 wait complete. 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 librados: starting msgr at 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.695+0000 7fc31c035700 1 librados: starting objecter 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 Processor -- start 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 -- start start 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 librados: setting wanted keys 2024-03-30T14:56:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 librados: calling monclient init 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3141067c0 con 0x7fc31410d960 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3141069d0 con 0x7fc314109dc0 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31c035700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3141a8540 con 0x7fc314101130 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31a5d2700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc31410d960 0x7fc3141a4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39424/0 (socket says 172.21.15.28:39424) 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc31a5d2700 1 -- 172.21.15.28:0/2492981416 learned_addr learned my addr 172.21.15.28:0/2492981416 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc319dd1700 1 --1- 172.21.15.28:0/2492981416 >> v1:172.21.15.28:6790/0 conn(0x7fc314101130 0x7fc314103610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54514/0 (socket says 172.21.15.28:54514) 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter ms_handle_connect 0x7fc31410d960 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter ms_handle_connect 0x7fc314101130 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter ms_handle_connect 0x7fc314109dc0 2024-03-30T14:56:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726973824 0 0) 0x7fc3141067c0 con 0x7fc31410d960 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2fc0031f0 con 0x7fc31410d960 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992471492 0 0) 0x7fc3141a8540 con 0x7fc314101130 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3141067c0 con 0x7fc314101130 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980753205 0 0) 0x7fc3141069d0 con 0x7fc314109dc0 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3141a8540 con 0x7fc314109dc0 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.696+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3194943250 0 0) 0x7fc2fc0031f0 con 0x7fc31410d960 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3141069d0 con 0x7fc31410d960 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc30c0045e0 con 0x7fc31410d960 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2993551636 0 0) 0x7fc3141067c0 con 0x7fc314101130 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2fc0031f0 con 0x7fc314101130 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc304001bb0 con 0x7fc314101130 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2300340430 0 0) 0x7fc3141a8540 con 0x7fc314109dc0 2024-03-30T14:56:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3141067c0 con 0x7fc314109dc0 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc308001a30 con 0x7fc314109dc0 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1112564403 0 0) 0x7fc3141069d0 con 0x7fc31410d960 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 >> v1:172.21.15.28:6790/0 conn(0x7fc314101130 legacy=0x7fc314103610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 >> v1:172.21.15.134:6789/0 conn(0x7fc314109dc0 legacy=0x7fc314103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3141a9760 con 0x7fc31410d960 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3141a8770 con 0x7fc31410d960 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc31c035700 10 client.14802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3141a8c30 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc31c035700 1 librados: init done 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc30c001b70 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.697+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc30c004e20 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.698+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc300005120 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc30c003780 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(96..96 src has 43..96) ==== 6259+0+0 (unknown 896004952 0 0) 0x7fc30c088d60 con 0x7fc31410d960 2024-03-30T14:56:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 10 client.14802.objecter ms_dispatch 0x7fc314001550 osd_map(96..96 src has 43..96) 2024-03-30T14:56:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 3 client.14802.objecter handle_osd_map got epochs [96,96] > 0 2024-03-30T14:56:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 3 client.14802.objecter handle_osd_map decoding full epoch 96 2024-03-30T14:56:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 20 client.14802.objecter dump_active .. 0 homeless 2024-03-30T14:56:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.699+0000 7fc312ffd700 10 client.14802.objecter ms_handle_connect 0x7fc2fc0703e0 2024-03-30T14:56:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.704+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc30c056800 con 0x7fc31410d960 2024-03-30T14:56:38.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:38.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: 3.7 deep-scrub starts 2024-03-30T14:56:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: 3.7 deep-scrub ok 2024-03-30T14:56:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2907090354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T14:56:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:38 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T14:56:38.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:38.920+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fc300005410 con 0x7fc31410d960 2024-03-30T14:56:38.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:38.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:39.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:39.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:39.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:39.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:39.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:39.302 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:39.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:39.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.506+0000 7fc312ffd700 1 -- 172.21.15.28:0/2492981416 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v97) ==== 87+0+0 (unknown 72982671 0 0) 0x7fc30c05a250 con 0x7fc31410d960 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 10 librados: watch_flush enter 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 10 librados: watch_flush exit 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 20 client.14802.objecter shutdown clearing up homeless session... 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 10 client.14802.objecter successfully canceled tick 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc2fc0703e0 legacy=0x7fc2fc0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 >> v1:172.21.15.28:6789/0 conn(0x7fc31410d960 legacy=0x7fc3141a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 shutdown_connections 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 >> 172.21.15.28:0/2492981416 conn(0x7fc3140fc9e0 msgr2=0x7fc3141106b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.510+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 shutdown_connections 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.511+0000 7fc31c035700 1 -- 172.21.15.28:0/2492981416 wait complete. 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:39.511+0000 7fc31c035700 1 librados: shutdown 2024-03-30T14:56:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T14:56:39.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:39.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:39.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:39.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:39.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:56:39.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T14:56:39.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T14:56:39.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:56:39.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T14:56:39.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:39 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T14:56:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:39.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:39.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:56:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:39 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T14:56:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:39 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T14:56:39.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:39.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:40.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:40.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T14:56:40.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:40.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:40.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:40.309 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:40.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:40.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:56:39 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:40.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:40.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:40.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:40.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:40.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:40.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:40.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:40 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:56:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:40 smithi134 ceph-mon[33714]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T14:56:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:40 smithi134 ceph-mon[33714]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:40 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:40.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:40.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[41614]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[41614]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2492981416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[29876]: osdmap e97: 8 total, 8 up, 8 in 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[29876]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:40 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:41.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:41.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:41.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:56:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:41.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:41.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:41.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:41.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:41.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:41.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-30T14:56:41.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:41.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:41.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:41.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:41.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:41.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:41.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:41.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:41.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:42.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:42.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:42.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:42.223 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:42 smithi028 systemd[1]: Stopping Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:42.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:42.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:42.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:42.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:42 smithi028 ceph-mon[29876]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:42.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:42 smithi028 ceph-mon[41614]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:42.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:42.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:42 smithi134 ceph-mon[33714]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:56:42.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:42.701 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:42.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:42.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:42.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:42.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:42.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:42.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:43.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:43.102 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:42 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[79196]: ts=2024-03-30T14:56:42.795Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:56:43.102 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:42 smithi028 podman[83868]: 2024-03-30 14:56:42.81645657 +0000 UTC m=+0.145181200 container died 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:43.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:43.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:43.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:43.360 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:43 smithi028 podman[83868]: 2024-03-30 14:56:43.2088725 +0000 UTC m=+0.537597111 container cleanup 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:43.360 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:43 smithi028 bash[83868]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T14:56:43.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:43.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:43.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:43.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:43.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:43.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:43.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:43.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:44.076 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:43 smithi028 podman[83931]: 2024-03-30 14:56:43.771474502 +0000 UTC m=+0.950037972 container remove 61efd02d4e4d658dacd50d908081e316924753fd1f6cd1829c018809d2f08ebd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:44.076 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:43 smithi028 podman[83931]: 2024-03-30 14:56:43.871976442 +0000 UTC m=+1.050539843 volume remove 37bc9403c0d1922dd7f9ba012d0beee3390c524f7840a5a0a7ef1a52d33ed867 2024-03-30T14:56:44.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:44.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:44.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:44.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:44.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:44.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:44.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:44.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:44.649 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:44 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T14:56:44.649 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:44 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:44.649 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:44 smithi028 systemd[1]: Starting Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:44.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:56:44.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:44.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:44.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:44.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:44 smithi134 ceph-mon[33714]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:44.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:44 smithi028 ceph-mon[29876]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:44.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:44 smithi028 ceph-mon[41614]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:45.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:45.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:45.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:45.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:45.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:45.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:45.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:45.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:45.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:45.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:45.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:45.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:46.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:46.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:46.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:46.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:46.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:46.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:46.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:46.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:46.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:46.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:46.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:46.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:46.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[41614]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[41614]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:56:46.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:56:46.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[29876]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:46.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:46 smithi028 ceph-mon[29876]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T14:56:46.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:46.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:46 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:56:46.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:46 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:56:46.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:46 smithi134 ceph-mon[33714]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:46 smithi134 ceph-mon[33714]: osdmap e98: 8 total, 8 up, 8 in 2024-03-30T14:56:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:46.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:46.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:47.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:47.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:47.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:47.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- 172.21.15.28:0/1011861263 >> v1:172.21.15.28:6790/0 conn(0x7f6f7c10eab0 legacy=0x7f6f7c110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- 172.21.15.28:0/1011861263 shutdown_connections 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- 172.21.15.28:0/1011861263 >> 172.21.15.28:0/1011861263 conn(0x7f6f7c0fdb10 msgr2=0x7f6f7c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- 172.21.15.28:0/1011861263 shutdown_connections 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- 172.21.15.28:0/1011861263 wait complete. 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 librados: starting msgr at 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 librados: starting objecter 2024-03-30T14:56:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 Processor -- start 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 -- start start 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 librados: setting wanted keys 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.279+0000 7f6f814f6700 1 librados: calling monclient init 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f814f6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f7c107a30 con 0x7f6f7c10af10 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f814f6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f7c1a8680 con 0x7f6f7c102280 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f814f6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f7c1a9860 con 0x7f6f7c10eab0 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f7b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6f7c10eab0 0x7f6f7c1a5f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54540/0 (socket says 172.21.15.28:54540) 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f7a7fc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6f7c10af10 0x7f6f7c104f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39448/0 (socket says 172.21.15.28:39448) 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f7b7fe700 1 -- 172.21.15.28:0/3914688889 learned_addr learned my addr 172.21.15.28:0/3914688889 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter ms_handle_connect 0x7f6f7c10eab0 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter ms_handle_connect 0x7f6f7c10af10 2024-03-30T14:56:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter ms_handle_connect 0x7f6f7c102280 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3781625116 0 0) 0x7f6f7c107a30 con 0x7f6f7c10af10 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f680035b0 con 0x7f6f7c10af10 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639747795 0 0) 0x7f6f7c1a9860 con 0x7f6f7c10eab0 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f7c107a30 con 0x7f6f7c10eab0 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091347167 0 0) 0x7f6f7c1a8680 con 0x7f6f7c102280 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f7c1a9860 con 0x7f6f7c102280 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4056222399 0 0) 0x7f6f680035b0 con 0x7f6f7c10af10 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f7c1a8680 con 0x7f6f7c10af10 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6f64001830 con 0x7f6f7c10af10 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3020092505 0 0) 0x7f6f7c107a30 con 0x7f6f7c10eab0 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f680035b0 con 0x7f6f7c10eab0 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 311396487 0 0) 0x7f6f7c1a9860 con 0x7f6f7c102280 2024-03-30T14:56:47.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f7c107a30 con 0x7f6f7c102280 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6f74004810 con 0x7f6f7c10eab0 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.280+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6f6c001850 con 0x7f6f7c102280 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4058913605 0 0) 0x7f6f7c1a8680 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6790/0 conn(0x7f6f7c10eab0 legacy=0x7f6f7c1a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.134:6789/0 conn(0x7f6f7c102280 legacy=0x7f6f7c104880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7c1aaa40 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7c1a9a90 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f814f6700 10 client.14808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f7c1a9f50 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f814f6700 1 librados: init done 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6f64003b40 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6f64005b50 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.281+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6f5c005130 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.282+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6f640036a0 con 0x7f6f7c10af10 2024-03-30T14:56:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(98..98 src has 43..98) ==== 6259+0+0 (unknown 3717582423 0 0) 0x7f6f64088e20 con 0x7f6f7c10af10 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 osd_map(98..98 src has 43..98) 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 3 client.14808.objecter handle_osd_map got epochs [98,98] > 0 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 3 client.14808.objecter handle_osd_map decoding full epoch 98 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 20 client.14808.objecter dump_active .. 0 homeless 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v98) ==== 54+0+16 (unknown 2227819007 0 2420000192) 0x7f6f6408b2b0 con 0x7f6f7c10af10 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f68070490 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 10 client.14808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 1 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 1 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 0 1 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 4 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 10 client.14808.objecter _send_command 1 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c004ba0 con 0x7f6f5c004760 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.283+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c004760 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.284+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c004ba0 con 0x7f6f5c004760 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.284+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.284+0000 7f6f73fff700 10 client.14808.objecter _finish_command 1 = system:0 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.284+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 4 2024-03-30T14:56:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.284+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 0 1 2024-03-30T14:56:47.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 10 client.14808.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 2 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 2 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c005410 osd=0 3 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 0 2 2024-03-30T14:56:47.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 4 2024-03-30T14:56:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 10 client.14808.objecter _send_command 2 2024-03-30T14:56:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.301+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c007a20 con 0x7f6f5c004760 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c007a20 con 0x7f6f5c004760 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 10 client.14808.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 4 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 0 2 2024-03-30T14:56:47.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 10 client.14808.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 3 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 3 2024-03-30T14:56:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 1 3 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 4 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 10 client.14808.objecter _send_command 3 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c0022d0 con 0x7f6f5c004340 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.303+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c004340 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.304+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c0022d0 con 0x7f6f5c004340 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.304+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 3: 0 ) 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.304+0000 7f6f73fff700 10 client.14808.objecter _finish_command 3 = system:0 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.304+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 4 2024-03-30T14:56:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.304+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 1 3 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 10 client.14808.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 4 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 4 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c004160 osd=1 3 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 1 4 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 4 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 10 client.14808.objecter _send_command 4 2024-03-30T14:56:47.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.321+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c001750 con 0x7f6f5c004340 2024-03-30T14:56:47.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c001750 con 0x7f6f5c004340 2024-03-30T14:56:47.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f73fff700 10 client.14808.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 4 2024-03-30T14:56:47.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 1 4 2024-03-30T14:56:47.324 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:56:47.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 10 client.14808.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 5 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 5 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 2 5 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 4 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 10 client.14808.objecter _send_command 5 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.322+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c0111a0 con 0x7f6f5c00d600 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c00d600 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c0111a0 con 0x7f6f5c00d600 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 5: 0 ) 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 10 client.14808.objecter _finish_command 5 = system:0 2024-03-30T14:56:47.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 4 2024-03-30T14:56:47.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.323+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 2 5 2024-03-30T14:56:47.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 10 client.14808.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 6 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 6 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c0019b0 osd=2 3 2024-03-30T14:56:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 2 6 2024-03-30T14:56:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 4 2024-03-30T14:56:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 10 client.14808.objecter _send_command 6 2024-03-30T14:56:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.341+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c008130 con 0x7f6f5c00d600 2024-03-30T14:56:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c008130 con 0x7f6f5c00d600 2024-03-30T14:56:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 10 client.14808.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 4 2024-03-30T14:56:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 2 6 2024-03-30T14:56:47.344 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 10 client.14808.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 7 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 7 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 3 7 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 4 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 10 client.14808.objecter _send_command 7 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c014920 con 0x7f6f5c0124d0 2024-03-30T14:56:47.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.343+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c0124d0 2024-03-30T14:56:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.344+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c014920 con 0x7f6f5c0124d0 2024-03-30T14:56:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.344+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 7: 0 ) 2024-03-30T14:56:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.344+0000 7f6f73fff700 10 client.14808.objecter _finish_command 7 = system:0 2024-03-30T14:56:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.344+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 4 2024-03-30T14:56:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.344+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 3 7 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 10 client.14808.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 8 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 8 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c0083c0 osd=3 3 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 3 8 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 4 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 10 client.14808.objecter _send_command 8 2024-03-30T14:56:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.361+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c01a920 con 0x7f6f5c0124d0 2024-03-30T14:56:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c01a920 con 0x7f6f5c0124d0 2024-03-30T14:56:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f73fff700 10 client.14808.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 4 2024-03-30T14:56:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 3 8 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 10 client.14808.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 9 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 9 2024-03-30T14:56:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 4 9 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 4 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 10 client.14808.objecter _send_command 9 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.362+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c01ecf0 con 0x7f6f5c01afa0 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.363+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c01afa0 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.364+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c01ecf0 con 0x7f6f5c01afa0 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.364+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 9: 0 ) 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.364+0000 7f6f73fff700 10 client.14808.objecter _finish_command 9 = system:0 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.364+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 4 2024-03-30T14:56:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.364+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 4 9 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 10 client.14808.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 10 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 10 2024-03-30T14:56:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c01a920 osd=4 3 2024-03-30T14:56:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 4 10 2024-03-30T14:56:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 4 2024-03-30T14:56:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 10 client.14808.objecter _send_command 10 2024-03-30T14:56:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.380+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c01eed0 con 0x7f6f5c01afa0 2024-03-30T14:56:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c01eed0 con 0x7f6f5c01afa0 2024-03-30T14:56:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f73fff700 10 client.14808.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 4 2024-03-30T14:56:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 4 10 2024-03-30T14:56:47.384 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:56:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 10 client.14808.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 11 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 11 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 5 11 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 4 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 10 client.14808.objecter _send_command 11 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.382+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c0201d0 con 0x7f6f5c01f5f0 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.383+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c01f5f0 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.384+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c0201d0 con 0x7f6f5c01f5f0 2024-03-30T14:56:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.384+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 11: 0 ) 2024-03-30T14:56:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.384+0000 7f6f73fff700 10 client.14808.objecter _finish_command 11 = system:0 2024-03-30T14:56:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.384+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 4 2024-03-30T14:56:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.384+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 5 11 2024-03-30T14:56:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 10 client.14808.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 12 2024-03-30T14:56:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 12 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c01eed0 osd=5 3 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 5 12 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 4 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 10 client.14808.objecter _send_command 12 2024-03-30T14:56:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.400+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c020450 con 0x7f6f5c01f5f0 2024-03-30T14:56:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.402+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c020450 con 0x7f6f5c01f5f0 2024-03-30T14:56:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.402+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.402+0000 7f6f73fff700 10 client.14808.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.402+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 4 2024-03-30T14:56:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.402+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 5 12 2024-03-30T14:56:47.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 10 client.14808.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 13 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 13 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 6 13 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 4 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 10 client.14808.objecter _send_command 13 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c027720 con 0x7f6f5c023bd0 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.403+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c023bd0 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.404+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c027720 con 0x7f6f5c023bd0 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.404+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 13: 0 ) 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.404+0000 7f6f73fff700 10 client.14808.objecter _finish_command 13 = system:0 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.404+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 4 2024-03-30T14:56:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.404+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 6 13 2024-03-30T14:56:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 10 client.14808.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 14 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 14 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c020450 osd=6 3 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 6 14 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 4 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 10 client.14808.objecter _send_command 14 2024-03-30T14:56:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.421+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c027900 con 0x7f6f5c023bd0 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c027900 con 0x7f6f5c023bd0 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 10 client.14808.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 4 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 6 14 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:56:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 10 client.14808.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 15 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 15 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 7 15 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 4 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 10 client.14808.objecter _send_command 15 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6f5c028bc0 con 0x7f6f5c028020 2024-03-30T14:56:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.423+0000 7f6f73fff700 10 client.14808.objecter ms_handle_connect 0x7f6f5c028020 2024-03-30T14:56:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.424+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6f5c028bc0 con 0x7f6f5c028020 2024-03-30T14:56:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.424+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 15: 0 ) 2024-03-30T14:56:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.424+0000 7f6f73fff700 10 client.14808.objecter _finish_command 15 = system:0 2024-03-30T14:56:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.424+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 4 2024-03-30T14:56:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.424+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 7 15 2024-03-30T14:56:47.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 10 client.14808.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign -1 16 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 20 client.14808.objecter _get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_remove -1 16 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 20 client.14808.objecter get_session s=0x7f6f5c027900 osd=7 3 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 15 client.14808.objecter _session_command_op_assign 7 16 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 4 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 10 client.14808.objecter _send_command 16 2024-03-30T14:56:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.442+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f5c028bc0 con 0x7f6f5c028020 2024-03-30T14:56:47.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f73fff700 1 -- 172.21.15.28:0/3914688889 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6f5c028bc0 con 0x7f6f5c028020 2024-03-30T14:56:47.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f73fff700 10 client.14808.objecter ms_dispatch 0x7f6f7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:47.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f73fff700 10 client.14808.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f73fff700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 4 2024-03-30T14:56:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f73fff700 15 client.14808.objecter _session_command_op_remove 7 16 2024-03-30T14:56:47.445 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 librados: watch_flush enter 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 librados: watch_flush exit 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.0 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6f5c004760 legacy=0x7f6f5c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c005410 osd=0 1 2024-03-30T14:56:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.1 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6f5c004340 legacy=0x7f6f5c00ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c004160 osd=1 1 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.2 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6f5c00d600 legacy=0x7f6f5c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0019b0 osd=2 1 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.3 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f7b7fe700 1 -- 172.21.15.28:0/3914688889 reap_dead start 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6f5c0124d0 legacy=0x7f6f5c019200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c0083c0 osd=3 1 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.4 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6f5c01afa0 legacy=0x7f6f5c01d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01a920 osd=4 1 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.5 2024-03-30T14:56:47.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.134:6804/3668151179 conn(0x7f6f5c01f5f0 legacy=0x7f6f5c01fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c01eed0 osd=5 1 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.6 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.134:6808/2135224194 conn(0x7f6f5c023bd0 legacy=0x7f6f5c025fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c020450 osd=6 1 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter close_session for osd.7 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6f5c028020 legacy=0x7f6f5c028480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter put_session s=0x7f6f5c027900 osd=7 1 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 20 client.14808.objecter shutdown clearing up homeless session... 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 10 client.14808.objecter successfully canceled tick 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f7b7fe700 1 -- 172.21.15.28:0/3914688889 reap_dead start 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6f68070490 legacy=0x7f6f68072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> v1:172.21.15.28:6789/0 conn(0x7f6f7c10af10 legacy=0x7f6f7c104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 shutdown_connections 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.444+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 >> 172.21.15.28:0/3914688889 conn(0x7f6f7c0fdb10 msgr2=0x7f6f7c10a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.445+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 shutdown_connections 2024-03-30T14:56:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.445+0000 7f6f814f6700 1 -- 172.21.15.28:0/3914688889 wait complete. 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.445+0000 7f6f814f6700 1 librados: shutdown 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:56:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:56:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:47.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:47.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:47.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.532+0000 7f73e69b2700 1 -- 172.21.15.28:0/1086053924 >> v1:172.21.15.28:6789/0 conn(0x7f73e0101120 legacy=0x7f73e0101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.532+0000 7f73e69b2700 1 -- 172.21.15.28:0/1086053924 shutdown_connections 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.532+0000 7f73e69b2700 1 -- 172.21.15.28:0/1086053924 >> 172.21.15.28:0/1086053924 conn(0x7f73e00fc9f0 msgr2=0x7f73e00fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.532+0000 7f73e69b2700 1 -- 172.21.15.28:0/1086053924 shutdown_connections 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.532+0000 7f73e69b2700 1 -- 172.21.15.28:0/1086053924 wait complete. 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 librados: starting msgr at 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 librados: starting objecter 2024-03-30T14:56:47.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 Processor -- start 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 -- start start 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 librados: setting wanted keys 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 librados: calling monclient init 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73e01a92b0 con 0x7f73e010d950 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73e01aa4d0 con 0x7f73e0101120 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e69b2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73e01ab710 con 0x7f73e0109db0 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e4f4f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73e010d950 0x7f73e01a7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39464/0 (socket says 172.21.15.28:39464) 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73e4f4f700 1 -- 172.21.15.28:0/1832966612 learned_addr learned my addr 172.21.15.28:0/1832966612 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73df7fe700 1 --1- 172.21.15.28:0/1832966612 >> v1:172.21.15.28:6790/0 conn(0x7f73e0109db0 0x7f73e01a41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54554/0 (socket says 172.21.15.28:54554) 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73dd7fa700 10 client.?.objecter ms_handle_connect 0x7f73e010d950 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73dd7fa700 10 client.?.objecter ms_handle_connect 0x7f73e0109db0 2024-03-30T14:56:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.533+0000 7f73dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 10 client.?.objecter ms_handle_connect 0x7f73e0101120 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798268839 0 0) 0x7f73e01ab710 con 0x7f73e0109db0 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73c80031f0 con 0x7f73e0109db0 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1222212117 0 0) 0x7f73e01aa4d0 con 0x7f73e0101120 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e01ab710 con 0x7f73e0101120 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087786911 0 0) 0x7f73e01a92b0 con 0x7f73e010d950 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e01aa4d0 con 0x7f73e010d950 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1160002510 0 0) 0x7f73c80031f0 con 0x7f73e0109db0 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73e01a92b0 con 0x7f73e0109db0 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f73d4001a10 con 0x7f73e0109db0 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3041697062 0 0) 0x7f73e01ab710 con 0x7f73e0101120 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73c80031f0 con 0x7f73e0101120 2024-03-30T14:56:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f73d0001370 con 0x7f73e0101120 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 269145191 0 0) 0x7f73e01aa4d0 con 0x7f73e010d950 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73e01ab710 con 0x7f73e010d950 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f73d8001bf0 con 0x7f73e010d950 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1384691690 0 0) 0x7f73e01a92b0 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 >> v1:172.21.15.134:6789/0 conn(0x7f73e0101120 legacy=0x7f73e01062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 >> v1:172.21.15.28:6789/0 conn(0x7f73e010d950 legacy=0x7f73e01a7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e01ac950 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73e01ab940 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73e69b2700 10 client.34703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f73e01abe00 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73e69b2700 1 librados: init done 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f73d4001bb0 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.534+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f73d4005c30 con 0x7f73e0109db0 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.535+0000 7f73e69b2700 10 client.34703.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:47.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.535+0000 7f73e69b2700 15 client.34703.objecter _session_command_op_assign -1 1 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.535+0000 7f73e69b2700 20 client.34703.objecter _get_session osd=-1 returning homeless 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.535+0000 7f73e69b2700 10 client.34703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.535+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f73cc002920 con 0x7f73e0109db0 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.536+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f73d4004250 con 0x7f73e0109db0 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.536+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(98..98 src has 43..98) ==== 6259+0+0 (unknown 3717582423 0 0) 0x7f73d4088f00 con 0x7f73e0109db0 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.536+0000 7f73dd7fa700 10 client.34703.objecter ms_dispatch 0x7f73e0001550 osd_map(98..98 src has 43..98) 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.536+0000 7f73dd7fa700 3 client.34703.objecter handle_osd_map got epochs [98,98] > 0 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.536+0000 7f73dd7fa700 3 client.34703.objecter handle_osd_map decoding full epoch 98 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 10 client.34703.objecter checking command 1 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter _get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 15 client.34703.objecter _session_command_op_remove -1 1 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter _get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 15 client.34703.objecter _session_command_op_assign 1 1 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 4 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 10 client.34703.objecter _send_command 1 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6805/1850313547 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73c807e7d0 con 0x7f73c807aac0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 20 client.34703.objecter dump_active .. 0 homeless 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=98) ==== 24+0+0 (unknown 3138435627 0 0) 0x7f73d4089360 con 0x7f73e0109db0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 10 client.34703.objecter ms_handle_connect 0x7f73c80703e0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.537+0000 7f73dd7fa700 10 client.34703.objecter ms_handle_connect 0x7f73c807aac0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.538+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f73c807e7d0 con 0x7f73c807aac0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.538+0000 7f73dd7fa700 10 client.34703.objecter ms_dispatch 0x7f73e0001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.538+0000 7f73dd7fa700 10 client.34703.objecter _finish_command 1 = system:0 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.538+0000 7f73dd7fa700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 4 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.538+0000 7f73dd7fa700 15 client.34703.objecter _session_command_op_remove 1 1 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 client.34703.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:56:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 15 client.34703.objecter _session_command_op_assign -1 2 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter _get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter _get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 15 client.34703.objecter _session_command_op_remove -1 2 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter get_session s=0x7f73c807a410 osd=1 3 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 15 client.34703.objecter _session_command_op_assign 1 2 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 4 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 client.34703.objecter _send_command 2 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f73cc0011f0 con 0x7f73c807aac0 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73dd7fa700 1 -- 172.21.15.28:0/1832966612 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f73cc0011f0 con 0x7f73c807aac0 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73dd7fa700 10 client.34703.objecter ms_dispatch 0x7f73e0001550 command_reply(tid 2: 0 ) 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73dd7fa700 10 client.34703.objecter _finish_command 2 = system:0 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73dd7fa700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 4 2024-03-30T14:56:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73dd7fa700 15 client.34703.objecter _session_command_op_remove 1 2 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 librados: watch_flush enter 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 librados: watch_flush exit 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 client.34703.objecter close_session for osd.1 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 >> v1:172.21.15.28:6805/1850313547 conn(0x7f73c807aac0 legacy=0x7f73c807cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter put_session s=0x7f73c807a410 osd=1 1 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 20 client.34703.objecter shutdown clearing up homeless session... 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 10 client.34703.objecter successfully canceled tick 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 >> v1:172.21.15.28:6800/3011431093 conn(0x7f73c80703e0 legacy=0x7f73c80728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 >> v1:172.21.15.28:6790/0 conn(0x7f73e0109db0 legacy=0x7f73e01a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e4f4f700 1 -- 172.21.15.28:0/1832966612 reap_dead start 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 shutdown_connections 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 >> 172.21.15.28:0/1832966612 conn(0x7f73e00fc9f0 msgr2=0x7f73e006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 shutdown_connections 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 -- 172.21.15.28:0/1832966612 wait complete. 2024-03-30T14:56:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:47.557+0000 7f73e69b2700 1 librados: shutdown 2024-03-30T14:56:47.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:47.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:47.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3914688889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:47.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3914688889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:47.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:47.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3914688889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:47.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:47.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:48.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:48.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:48.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:48.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:48.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:48.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:48.446 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:56:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:56:48.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:48.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:48.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:48.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:48.722 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:48 smithi028 podman[84691]: 2024-03-30 14:56:48.486034182 +0000 UTC m=+0.021915793 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-30T14:56:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:48.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:48.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:48.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:48.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:48.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:49.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:48 smithi134 ceph-mon[33714]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-30T14:56:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:48 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:48 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:49.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:49.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[29876]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-30T14:56:49.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:49.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:49.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[41614]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-03-30T14:56:49.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:49.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:48 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:56:49.223 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:48 smithi028 podman[84691]: 2024-03-30 14:56:48.836740385 +0000 UTC m=+0.372621964 volume create 2668a34bb0e9cd1d369d70e8bf3b2bb34dd8d10c21a1f8771fa0df26fbc2e609 2024-03-30T14:56:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:49.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:49.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-30T14:56:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:49.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:49.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:49.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:49.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:49.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:49.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:49.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:49.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:49.886 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 podman[84691]: 2024-03-30 14:56:49.701718024 +0000 UTC m=+1.237599644 container create 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:49.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:50.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:50.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:50.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:50.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:50.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:50.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:56:49 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:56:50.470 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 podman[84691]: 2024-03-30 14:56:49.979100027 +0000 UTC m=+1.514981621 container init 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 podman[84691]: 2024-03-30 14:56:49.984361787 +0000 UTC m=+1.520243369 container start 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:49.997Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:49.997Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:49.998Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:49.999Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:50.022Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:50.023Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:50.026Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:50.026Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 bash[84691]: 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 2024-03-30T14:56:50.471 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:50 smithi028 systemd[1]: Started Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:50.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:50.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:50.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:50.854 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:56:50.855 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:56:50.855 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T14:56:50.855 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T14:56:50.855 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:56:50.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:50.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:50.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:51.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:51.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:51.218 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:51 smithi134 ceph-mon[33714]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:51.218 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:51.219 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.219 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.219 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:51 smithi134 ceph-mon[33714]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T14:56:51.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:51.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[29876]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[29876]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T14:56:51.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[41614]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:51.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:56:51.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:51.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:51 smithi028 ceph-mon[41614]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-30T14:56:51.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:51.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:51.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:51.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:51.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:51.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:51.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:51.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:51.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:51.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:52.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:52.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:52.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:52.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:52.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:52.347 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:52 smithi134 ceph-mon[33714]: Reconfiguring daemon prometheus.a on smithi134 2024-03-30T14:56:52.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:52.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:52.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:52 smithi028 ceph-mon[29876]: Reconfiguring daemon prometheus.a on smithi134 2024-03-30T14:56:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:52 smithi028 ceph-mon[41614]: Reconfiguring daemon prometheus.a on smithi134 2024-03-30T14:56:52.474 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:56:52 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:56:52.000Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000564686s 2024-03-30T14:56:52.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T14:56:52.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:52.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:52.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:52.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:56:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:52.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:52.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:52.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:52.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:53.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:53.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:53.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:53.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:53.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:53.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:53.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:53.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:53 smithi028 ceph-mon[41614]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:53.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:53 smithi028 ceph-mon[29876]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:53.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:53.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-mon[33714]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:53.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:53.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:53.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:53.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:53.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.789+0000 7effac857700 1 -- 172.21.15.28:0/3423028347 >> v1:172.21.15.28:6789/0 conn(0x7effa810ea80 legacy=0x7effa8110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 -- 172.21.15.28:0/3423028347 shutdown_connections 2024-03-30T14:56:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 -- 172.21.15.28:0/3423028347 >> 172.21.15.28:0/3423028347 conn(0x7effa80761c0 msgr2=0x7effa80765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 -- 172.21.15.28:0/3423028347 shutdown_connections 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 -- 172.21.15.28:0/3423028347 wait complete. 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 librados: starting msgr at 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 librados: starting objecter 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.790+0000 7effac857700 1 Processor -- start 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 -- start start 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 librados: setting wanted keys 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 librados: calling monclient init 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7effa81a7080 con 0x7effa8100a90 2024-03-30T14:56:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7effa81a82a0 con 0x7effa810af10 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effac857700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7effa81a94e0 con 0x7effa810ea80 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effa659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7effa8100a90 0x7effa81075d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60040/0 (socket says 172.21.15.28:60040) 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effa6d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7effa810ea80 0x7effa81a5740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47754/0 (socket says 172.21.15.28:47754) 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7effa659c700 1 -- 172.21.15.28:0/3663625274 learned_addr learned my addr 172.21.15.28:0/3663625274 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter ms_handle_connect 0x7effa810ea80 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter ms_handle_connect 0x7effa8100a90 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter ms_handle_connect 0x7effa810af10 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710108494 0 0) 0x7effa81a94e0 con 0x7effa810ea80 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff940039c0 con 0x7effa810ea80 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755326500 0 0) 0x7effa81a82a0 con 0x7effa810af10 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effa81a94e0 con 0x7effa810af10 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039967995 0 0) 0x7effa81a7080 con 0x7effa8100a90 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effa81a82a0 con 0x7effa8100a90 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2714159225 0 0) 0x7effa81a94e0 con 0x7effa810af10 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7effa81a7080 con 0x7effa810af10 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.791+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2065453982 0 0) 0x7eff940039c0 con 0x7effa810ea80 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7effa81a94e0 con 0x7effa810ea80 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4164218130 0 0) 0x7effa81a82a0 con 0x7effa8100a90 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff940039c0 con 0x7effa8100a90 2024-03-30T14:56:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7eff90001870 con 0x7effa810af10 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7effa00046b0 con 0x7effa810ea80 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7eff98001890 con 0x7effa8100a90 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 748943589 0 0) 0x7effa81a7080 con 0x7effa810af10 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 >> v1:172.21.15.28:6790/0 conn(0x7effa810ea80 legacy=0x7effa81a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 >> v1:172.21.15.28:6789/0 conn(0x7effa8100a90 legacy=0x7effa81075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa81aa720 con 0x7effa810af10 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7eff90003b40 con 0x7effa810af10 2024-03-30T14:56:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7eff90004ae0 con 0x7effa810af10 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7effa81a84d0 con 0x7effa810af10 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7effac857700 10 client.24739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7effa81a8990 con 0x7effa810af10 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.792+0000 7effac857700 1 librados: init done 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.793+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa806a200 con 0x7effa810af10 2024-03-30T14:56:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7eff900036a0 con 0x7effa810af10 2024-03-30T14:56:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(98..98 src has 43..98) ==== 6259+0+0 (unknown 3717582423 0 0) 0x7eff90087dd0 con 0x7effa810af10 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 10 client.24739.objecter ms_dispatch 0x7effa8001550 osd_map(98..98 src has 43..98) 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 3 client.24739.objecter handle_osd_map got epochs [98,98] > 0 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 3 client.24739.objecter handle_osd_map decoding full epoch 98 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 20 client.24739.objecter dump_active .. 0 homeless 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.794+0000 7eff9f7fe700 10 client.24739.objecter ms_handle_connect 0x7eff94074960 2024-03-30T14:56:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:53.799+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7eff90055730 con 0x7effa810af10 2024-03-30T14:56:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:53.853 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 systemd[1]: Stopping Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:53.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:53.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:54.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.022+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7effa8105bb0 con 0x7effa810af10 2024-03-30T14:56:54.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:54.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.852Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T14:56:54.118 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T14:56:54.119 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T14:56:54.119 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T14:56:54.119 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T14:56:54.119 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[64428]: ts=2024-03-30T14:56:53.853Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T14:56:54.119 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:53 smithi134 podman[69705]: 2024-03-30 14:56:53.863224146 +0000 UTC m=+0.123545516 container died 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.122+0000 7eff9f7fe700 1 -- 172.21.15.28:0/3663625274 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v99) ==== 97+0+0 (unknown 4046948202 0 0) 0x7eff90004ac0 con 0x7effa810af10 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 10 librados: watch_flush enter 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 10 librados: watch_flush exit 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 20 client.24739.objecter shutdown clearing up homeless session... 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 10 client.24739.objecter successfully canceled tick 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 >> v1:172.21.15.28:6800/3011431093 conn(0x7eff94074960 legacy=0x7eff94076e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.124+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 >> v1:172.21.15.134:6789/0 conn(0x7effa810af10 legacy=0x7effa81a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.125+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 shutdown_connections 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.125+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 >> 172.21.15.28:0/3663625274 conn(0x7effa80761c0 msgr2=0x7effa8111f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.125+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 shutdown_connections 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.125+0000 7effac857700 1 -- 172.21.15.28:0/3663625274 wait complete. 2024-03-30T14:56:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:54.125+0000 7effac857700 1 librados: shutdown 2024-03-30T14:56:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T14:56:54.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:54.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:54.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:54.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:54.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:54.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3663625274' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:54 smithi134 ceph-mon[33714]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.431 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 podman[69705]: 2024-03-30 14:56:54.275253845 +0000 UTC m=+0.535575202 container cleanup 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:54.431 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 bash[69705]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T14:56:54.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:54.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3663625274' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:54 smithi028 ceph-mon[41614]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3663625274' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:54 smithi028 ceph-mon[29876]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:56:54.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:54.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:54.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:54.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:54.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:54.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:54.904 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 podman[69724]: 2024-03-30 14:56:54.48479766 +0000 UTC m=+0.624032297 container remove 0fa82c3dba068b537c4737e482b5ebf8cb82d70c61c1335a89e23512fd2654ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:54.904 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-03-30T14:56:54.904 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 systemd[1]: Stopped Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:54.904 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@prometheus.a.service: Consumed 1.153s CPU time. 2024-03-30T14:56:54.904 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 systemd[1]: Starting Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T14:56:54.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:54.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:55.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:55.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:55.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[41614]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:55.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:55.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[41614]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:56:55.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[41614]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T14:56:55.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[29876]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:55.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:55.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[29876]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:56:55.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-mon[29876]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T14:56:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-mon[33714]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:56:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-mon[33714]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:56:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-mon[33714]: osdmap e99: 8 total, 8 up, 8 in 2024-03-30T14:56:55.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 podman[69874]: 2024-03-30 14:56:54.808098469 +0000 UTC m=+0.019432314 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-30T14:56:55.180 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:54 smithi134 podman[69874]: 2024-03-30 14:56:54.930319239 +0000 UTC m=+0.141653074 container create 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:55.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:55.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:55.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:55.390 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPING 2024-03-30T14:56:55.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:55.469 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:55.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:55.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:55.680 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 podman[69874]: 2024-03-30 14:56:55.188551368 +0000 UTC m=+0.399885203 container init 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 podman[69874]: 2024-03-30 14:56:55.194145951 +0000 UTC m=+0.405479789 container start 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.222Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.222Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.222Z caller=main.go:570 level=info host_details="(Linux 5.14.0-434.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024 x86_64 smithi134 (none))" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.222Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.222Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.223Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.224Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.224Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.224Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-30T14:56:55.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.272µs 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.227Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.24µs wal_replay_duration=578.528µs wbl_replay_duration=108ns total_replay_duration=632.668µs 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.228Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.228Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.228Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.246Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.15092ms db_storage=1.052µs remote_storage=993ns web_handler=349ns query_engine=482ns scrape=385.782µs scrape_sd=77.678µs notify=17.703µs notify_sd=18.34µs rules=17.431688ms tracing=6.724µs 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.246Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-30T14:56:55.682 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T14:56:55.246Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-30T14:56:55.683 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 bash[69874]: 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 2024-03-30T14:56:55.683 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 14:56:55 smithi134 systemd[1]: Started Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T14:56:55.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:55.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPED 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTING 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Serving on http://:::9283 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTED 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPING 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:56:55.722 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPED 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTING 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Serving on http://:::9283 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTED 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPING 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STOPPED 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTING 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Serving on http://:::9283 2024-03-30T14:56:55.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:56:55 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:14:56:55] ENGINE Bus STARTED 2024-03-30T14:56:55.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:55.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:55.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:56.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:56.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:56.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:56.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:56.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:56.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[41614]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:56.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:56 smithi028 ceph-mon[29876]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:56:56.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:56.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:56:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:56 smithi134 ceph-mon[33714]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T14:56:56.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.728 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:56.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:56.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:56.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:56.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:56:56.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:56.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:57.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.171+0000 7f88dd7e7700 1 -- 172.21.15.28:0/4032195275 >> v1:172.21.15.28:6789/0 conn(0x7f88d810d950 legacy=0x7f88d810fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.171+0000 7f88dd7e7700 1 -- 172.21.15.28:0/4032195275 shutdown_connections 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.171+0000 7f88dd7e7700 1 -- 172.21.15.28:0/4032195275 >> 172.21.15.28:0/4032195275 conn(0x7f88d8077370 msgr2=0x7f88d80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.171+0000 7f88dd7e7700 1 -- 172.21.15.28:0/4032195275 shutdown_connections 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.171+0000 7f88dd7e7700 1 -- 172.21.15.28:0/4032195275 wait complete. 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 librados: starting msgr at 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 librados: starting objecter 2024-03-30T14:56:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 Processor -- start 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 -- start start 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 librados: setting wanted keys 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 librados: calling monclient init 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d81a5f50 con 0x7f88d810d950 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d81a7170 con 0x7f88d8105f00 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.172+0000 7f88dd7e7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88d81a83b0 con 0x7f88d8109d80 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88d67fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f88d8109d80 0x7f88d81a0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47766/0 (socket says 172.21.15.28:47766) 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88d67fc700 1 -- 172.21.15.28:0/3978002449 learned_addr learned my addr 172.21.15.28:0/3978002449 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter ms_handle_connect 0x7f88d8105f00 2024-03-30T14:56:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter ms_handle_connect 0x7f88d8109d80 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter ms_handle_connect 0x7f88d810d950 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1630908778 0 0) 0x7f88d81a7170 con 0x7f88d8105f00 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88c00031f0 con 0x7f88d8105f00 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88512664 0 0) 0x7f88d81a83b0 con 0x7f88d8109d80 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d81a7170 con 0x7f88d8109d80 2024-03-30T14:56:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119766443 0 0) 0x7f88d81a5f50 con 0x7f88d810d950 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d81a83b0 con 0x7f88d810d950 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2210010136 0 0) 0x7f88c00031f0 con 0x7f88d8105f00 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88d81a5f50 con 0x7f88d8105f00 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88c8001990 con 0x7f88d8105f00 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1564873217 0 0) 0x7f88d81a83b0 con 0x7f88d810d950 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88c00031f0 con 0x7f88d810d950 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88d0001d60 con 0x7f88d810d950 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3119243991 0 0) 0x7f88d81a7170 con 0x7f88d8109d80 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88d81a83b0 con 0x7f88d8109d80 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4204308970 0 0) 0x7f88d81a5f50 con 0x7f88d8105f00 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 >> v1:172.21.15.28:6790/0 conn(0x7f88d8109d80 legacy=0x7f88d81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 >> v1:172.21.15.28:6789/0 conn(0x7f88d810d950 legacy=0x7f88d81a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d81a95f0 con 0x7f88d8105f00 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.173+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88d81a85e0 con 0x7f88d8105f00 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 10 client.24745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f88d81a8b20 con 0x7f88d8105f00 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 1 librados: init done 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f88c8003b20 con 0x7f88d8105f00 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88c80049e0 con 0x7f88d8105f00 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 10 client.24745.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 15 client.24745.objecter _session_command_op_assign -1 1 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 20 client.24745.objecter _get_session osd=-1 returning homeless 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 10 client.24745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.174+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f88d8102810 con 0x7f88d8105f00 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.175+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f88c8003680 con 0x7f88d8105f00 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(99..99 src has 43..99) ==== 6259+0+0 (unknown 723184552 0 0) 0x7f88c8087ca0 con 0x7f88d8105f00 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 10 client.24745.objecter ms_dispatch 0x7f88d8001550 osd_map(99..99 src has 43..99) 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 3 client.24745.objecter handle_osd_map got epochs [99,99] > 0 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 3 client.24745.objecter handle_osd_map decoding full epoch 99 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 10 client.24745.objecter checking command 1 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter _get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 15 client.24745.objecter _session_command_op_remove -1 1 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter _get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 15 client.24745.objecter _session_command_op_assign 2 1 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 4 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 10 client.24745.objecter _send_command 1 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6809/3451325043 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88c0080d30 con 0x7f88c007d020 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 20 client.24745.objecter dump_active .. 0 homeless 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (unknown 1226618837 0 0) 0x7f88c8088080 con 0x7f88d8105f00 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 10 client.24745.objecter ms_handle_connect 0x7f88c00726b0 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.176+0000 7f88bffff700 10 client.24745.objecter ms_handle_connect 0x7f88c007d020 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.177+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88c0080d30 con 0x7f88c007d020 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.177+0000 7f88bffff700 10 client.24745.objecter ms_dispatch 0x7f88d8001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.177+0000 7f88bffff700 10 client.24745.objecter _finish_command 1 = system:0 2024-03-30T14:56:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.177+0000 7f88bffff700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 4 2024-03-30T14:56:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.177+0000 7f88bffff700 15 client.24745.objecter _session_command_op_remove 2 1 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 10 client.24745.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 15 client.24745.objecter _session_command_op_assign -1 2 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 20 client.24745.objecter _get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 20 client.24745.objecter _get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 15 client.24745.objecter _session_command_op_remove -1 2 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 20 client.24745.objecter get_session s=0x7f88c007c970 osd=2 3 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 15 client.24745.objecter _session_command_op_assign 2 2 2024-03-30T14:56:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 4 2024-03-30T14:56:57.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 10 client.24745.objecter _send_command 2 2024-03-30T14:56:57.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 --> v1:172.21.15.28:6809/3451325043 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f88d8068b30 con 0x7f88c007d020 2024-03-30T14:56:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88bffff700 1 -- 172.21.15.28:0/3978002449 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f88d8068b30 con 0x7f88c007d020 2024-03-30T14:56:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88bffff700 10 client.24745.objecter ms_dispatch 0x7f88d8001550 command_reply(tid 2: 0 ) 2024-03-30T14:56:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88bffff700 10 client.24745.objecter _finish_command 2 = system:0 2024-03-30T14:56:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88bffff700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 4 2024-03-30T14:56:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.199+0000 7f88bffff700 15 client.24745.objecter _session_command_op_remove 2 2 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 10 librados: watch_flush enter 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 10 librados: watch_flush exit 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 10 client.24745.objecter close_session for osd.2 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 >> v1:172.21.15.28:6809/3451325043 conn(0x7f88c007d020 legacy=0x7f88c007f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 20 client.24745.objecter put_session s=0x7f88c007c970 osd=2 1 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 20 client.24745.objecter shutdown clearing up homeless session... 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 10 client.24745.objecter successfully canceled tick 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 >> v1:172.21.15.28:6800/3011431093 conn(0x7f88c00726b0 legacy=0x7f88c0074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 >> v1:172.21.15.134:6789/0 conn(0x7f88d8105f00 legacy=0x7f88d810d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88d77fe700 1 -- 172.21.15.28:0/3978002449 reap_dead start 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 shutdown_connections 2024-03-30T14:56:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 >> 172.21.15.28:0/3978002449 conn(0x7f88d8077370 msgr2=0x7f88d8110b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:57.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 shutdown_connections 2024-03-30T14:56:57.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 -- 172.21.15.28:0/3978002449 wait complete. 2024-03-30T14:56:57.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:57.200+0000 7f88dd7e7700 1 librados: shutdown 2024-03-30T14:56:57.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:56:57.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:56:57.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:57.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:57.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:57.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:57.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:57.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:57.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:57.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:57.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:57.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:56:57.993 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:57.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:56:58.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:58.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:58.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:58.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-30T14:56:58.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:58.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:58.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:58.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:56:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:58.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:56:58.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:56:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:58.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:56:58.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:56:58.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:58.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:56:58.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:58 smithi028 ceph-mon[29876]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:58.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:58 smithi028 ceph-mon[41614]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:58.916 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:58 smithi134 ceph-mon[33714]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:56:58.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:59.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:59.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:59.198 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:56:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.198+0000 7ff738280700 1 -- 172.21.15.28:0/2445513178 >> v1:172.21.15.28:6789/0 conn(0x7ff730110ca0 legacy=0x7ff730113140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.198+0000 7ff738280700 1 -- 172.21.15.28:0/2445513178 shutdown_connections 2024-03-30T14:56:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.198+0000 7ff738280700 1 -- 172.21.15.28:0/2445513178 >> 172.21.15.28:0/2445513178 conn(0x7ff7300ffd20 msgr2=0x7ff730102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.198+0000 7ff738280700 1 -- 172.21.15.28:0/2445513178 shutdown_connections 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.198+0000 7ff738280700 1 -- 172.21.15.28:0/2445513178 wait complete. 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 librados: starting msgr at 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 librados: starting objecter 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 Processor -- start 2024-03-30T14:56:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 -- start start 2024-03-30T14:56:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 librados: setting wanted keys 2024-03-30T14:56:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.199+0000 7ff738280700 1 librados: calling monclient init 2024-03-30T14:56:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff738280700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff730109a00 con 0x7ff73010d100 2024-03-30T14:56:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff73601c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff730104470 0x7ff730106c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54078/0 (socket says 172.21.15.28:54078) 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff73601c700 1 -- 172.21.15.28:0/643842946 learned_addr learned my addr 172.21.15.28:0/643842946 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter ms_handle_connect 0x7ff730104470 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter ms_handle_connect 0x7ff730110ca0 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter ms_handle_connect 0x7ff73010d100 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff738280700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff730109c10 con 0x7ff730110ca0 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff738280700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7301ae030 con 0x7ff730104470 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.200+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393159327 0 0) 0x7ff730109c10 con 0x7ff730110ca0 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff71c0035b0 con 0x7ff730110ca0 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647377909 0 0) 0x7ff7301ae030 con 0x7ff730104470 2024-03-30T14:56:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff730109c10 con 0x7ff730104470 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 432988602 0 0) 0x7ff730109a00 con 0x7ff73010d100 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7301ae030 con 0x7ff73010d100 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3220557728 0 0) 0x7ff71c0035b0 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff730109a00 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff7280017b0 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1401971198 0 0) 0x7ff730109c10 con 0x7ff730104470 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff71c0035b0 con 0x7ff730104470 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2513265305 0 0) 0x7ff730109a00 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6790/0 conn(0x7ff730104470 legacy=0x7ff730106c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6789/0 conn(0x7ff73010d100 legacy=0x7ff730107320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7301af210 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff738280700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7301ae260 con 0x7ff730110ca0 2024-03-30T14:56:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff738280700 10 client.24751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:56:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff738280700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7301ae720 con 0x7ff730110ca0 2024-03-30T14:56:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff738280700 1 librados: init done 2024-03-30T14:56:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.201+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff728001ef0 con 0x7ff730110ca0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.202+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff728005400 con 0x7ff730110ca0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.203+0000 7ff738280700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff714005100 con 0x7ff730110ca0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.203+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff72801bc40 con 0x7ff730110ca0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(99..99 src has 43..99) ==== 6259+0+0 (unknown 723184552 0 0) 0x7ff7280880e0 con 0x7ff730110ca0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 osd_map(99..99 src has 43..99) 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 3 client.24751.objecter handle_osd_map got epochs [99,99] > 0 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 3 client.24751.objecter handle_osd_map decoding full epoch 99 2024-03-30T14:56:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 20 client.24751.objecter dump_active .. 0 homeless 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v99) ==== 54+0+16 (unknown 2384750422 0 2420000192) 0x7ff72808a550 con 0x7ff730110ca0 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.204+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff71c0703b0 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 10 client.24751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 1 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 1 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 0 1 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 4 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 10 client.24751.objecter _send_command 1 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff718006d40 con 0x7ff718002fd0 2024-03-30T14:56:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.205+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff718002fd0 2024-03-30T14:56:59.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.206+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff718006d40 con 0x7ff718002fd0 2024-03-30T14:56:59.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.206+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 1: 0 ) 2024-03-30T14:56:59.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.206+0000 7ff7277fe700 10 client.24751.objecter _finish_command 1 = system:0 2024-03-30T14:56:59.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.206+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 4 2024-03-30T14:56:59.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.206+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 0 1 2024-03-30T14:56:59.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 10 client.24751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 2 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 2 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180028e0 osd=0 3 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 0 2 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 4 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 10 client.24751.objecter _send_command 2 2024-03-30T14:56:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.235+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff718001240 con 0x7ff718002fd0 2024-03-30T14:56:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff718001240 con 0x7ff718002fd0 2024-03-30T14:56:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 10 client.24751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 4 2024-03-30T14:56:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 0 2 2024-03-30T14:56:59.238 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 10 client.24751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 1 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 4 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 10 client.24751.objecter _send_command 3 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff718008cc0 con 0x7ff718008100 2024-03-30T14:56:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.237+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff718008100 2024-03-30T14:56:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.238+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff718008cc0 con 0x7ff718008100 2024-03-30T14:56:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.238+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 3: 0 ) 2024-03-30T14:56:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.238+0000 7ff7277fe700 10 client.24751.objecter _finish_command 3 = system:0 2024-03-30T14:56:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.238+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 4 2024-03-30T14:56:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.238+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 1 3 2024-03-30T14:56:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:59.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:59.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 10 client.24751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 4 2024-03-30T14:56:59.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 4 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718001f50 osd=1 3 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 1 4 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 4 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 10 client.24751.objecter _send_command 4 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.255+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7180096e0 con 0x7ff718008100 2024-03-30T14:56:59.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7180096e0 con 0x7ff718008100 2024-03-30T14:56:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7277fe700 10 client.24751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 4 2024-03-30T14:56:59.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 1 4 2024-03-30T14:56:59.259 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 10 client.24751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 5 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 5 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 2 5 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 4 2024-03-30T14:56:59.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 10 client.24751.objecter _send_command 5 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.257+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff718010b20 con 0x7ff71800ce70 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff71800ce70 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff718010b20 con 0x7ff71800ce70 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 5: 0 ) 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 10 client.24751.objecter _finish_command 5 = system:0 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 4 2024-03-30T14:56:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.258+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 2 5 2024-03-30T14:56:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 10 client.24751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 6 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 6 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718001cc0 osd=2 3 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 2 6 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 4 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 10 client.24751.objecter _send_command 6 2024-03-30T14:56:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.275+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff718009210 con 0x7ff71800ce70 2024-03-30T14:56:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff718009210 con 0x7ff71800ce70 2024-03-30T14:56:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7277fe700 10 client.24751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 4 2024-03-30T14:56:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 2 6 2024-03-30T14:56:59.279 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 10 client.24751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 7 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 7 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 3 7 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 4 2024-03-30T14:56:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 10 client.24751.objecter _send_command 7 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.277+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff718007f50 con 0x7ff718010d00 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff718010d00 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff718007f50 con 0x7ff718010d00 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 7: 0 ) 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 10 client.24751.objecter _finish_command 7 = system:0 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 4 2024-03-30T14:56:59.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.278+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 3 7 2024-03-30T14:56:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 10 client.24751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 8 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 8 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff718009420 osd=3 3 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 3 8 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 4 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 10 client.24751.objecter _send_command 8 2024-03-30T14:56:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.295+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff718012130 con 0x7ff718010d00 2024-03-30T14:56:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff718012130 con 0x7ff718010d00 2024-03-30T14:56:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7277fe700 10 client.24751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 4 2024-03-30T14:56:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 3 8 2024-03-30T14:56:59.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:56:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 10 client.24751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 9 2024-03-30T14:56:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 9 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 4 9 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 4 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 10 client.24751.objecter _send_command 9 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.297+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff71801a320 con 0x7ff7180074e0 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff7180074e0 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff71801a320 con 0x7ff7180074e0 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 9: 0 ) 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 10 client.24751.objecter _finish_command 9 = system:0 2024-03-30T14:56:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 4 2024-03-30T14:56:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.298+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 4 9 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 10 client.24751.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 10 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 10 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180132b0 osd=4 3 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 4 10 2024-03-30T14:56:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 4 2024-03-30T14:56:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 10 client.24751.objecter _send_command 10 2024-03-30T14:56:59.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.315+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff718007290 con 0x7ff7180074e0 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff718007290 con 0x7ff7180074e0 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 10 client.24751.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 4 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 4 10 2024-03-30T14:56:59.318 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:56:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 10 client.24751.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 11 2024-03-30T14:56:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 11 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 5 11 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 4 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 10 client.24751.objecter _send_command 11 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff71801ba40 con 0x7ff71801c6c0 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.317+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff71801c6c0 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.318+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff71801ba40 con 0x7ff71801c6c0 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.318+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 11: 0 ) 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.318+0000 7ff7277fe700 10 client.24751.objecter _finish_command 11 = system:0 2024-03-30T14:56:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.318+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 4 2024-03-30T14:56:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.318+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 5 11 2024-03-30T14:56:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.336+0000 7ff7257fa700 10 client.24751.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.336+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 12 2024-03-30T14:56:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.336+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.336+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 12 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180072d0 osd=5 3 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 5 12 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 4 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 10 client.24751.objecter _send_command 12 2024-03-30T14:56:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.337+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff71801c470 con 0x7ff71801c6c0 2024-03-30T14:56:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.338+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff71801c470 con 0x7ff71801c6c0 2024-03-30T14:56:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.338+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.338+0000 7ff7277fe700 10 client.24751.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.338+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 4 2024-03-30T14:56:59.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.338+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 5 12 2024-03-30T14:56:59.340 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:56:59.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 10 client.24751.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 13 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 13 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 6 13 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 4 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 10 client.24751.objecter _send_command 13 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff71801c4f0 con 0x7ff71801f3b0 2024-03-30T14:56:59.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.339+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff71801f3b0 2024-03-30T14:56:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.340+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff71801c4f0 con 0x7ff71801f3b0 2024-03-30T14:56:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.340+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 13: 0 ) 2024-03-30T14:56:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.340+0000 7ff7277fe700 10 client.24751.objecter _finish_command 13 = system:0 2024-03-30T14:56:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.340+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 4 2024-03-30T14:56:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.340+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 6 13 2024-03-30T14:56:59.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 10 client.24751.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 14 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 14 2024-03-30T14:56:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff71801c210 osd=6 3 2024-03-30T14:56:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 6 14 2024-03-30T14:56:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 4 2024-03-30T14:56:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 10 client.24751.objecter _send_command 14 2024-03-30T14:56:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.375+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7180118d0 con 0x7ff71801f3b0 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7180118d0 con 0x7ff71801f3b0 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7277fe700 10 client.24751.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 4 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 6 14 2024-03-30T14:56:59.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 10 client.24751.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 15 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 15 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 7 15 2024-03-30T14:56:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 4 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 10 client.24751.objecter _send_command 15 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.377+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff718012450 con 0x7ff718011ff0 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 10 client.24751.objecter ms_handle_connect 0x7ff718011ff0 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff718012450 con 0x7ff718011ff0 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 15: 0 ) 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 10 client.24751.objecter _finish_command 15 = system:0 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 4 2024-03-30T14:56:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.378+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 7 15 2024-03-30T14:56:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 10 client.24751.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign -1 16 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 20 client.24751.objecter _get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_remove -1 16 2024-03-30T14:56:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 20 client.24751.objecter get_session s=0x7ff7180118d0 osd=7 3 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 15 client.24751.objecter _session_command_op_assign 7 16 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 4 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 10 client.24751.objecter _send_command 16 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.397+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff718012450 con 0x7ff718011ff0 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.398+0000 7ff7277fe700 1 -- 172.21.15.28:0/643842946 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff718012450 con 0x7ff718011ff0 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.398+0000 7ff7277fe700 10 client.24751.objecter ms_dispatch 0x7ff730001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.398+0000 7ff7277fe700 10 client.24751.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.398+0000 7ff7277fe700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 4 2024-03-30T14:56:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.398+0000 7ff7277fe700 15 client.24751.objecter _session_command_op_remove 7 16 2024-03-30T14:56:59.400 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 librados: watch_flush enter 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 librados: watch_flush exit 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.0 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6801/3394323575 conn(0x7ff718002fd0 legacy=0x7ff718005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180028e0 osd=0 1 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.1 2024-03-30T14:56:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6805/1850313547 conn(0x7ff718008100 legacy=0x7ff7180085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001f50 osd=1 1 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.2 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6809/3451325043 conn(0x7ff71800ce70 legacy=0x7ff71800f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718001cc0 osd=2 1 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.3 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6813/1506157250 conn(0x7ff718010d00 legacy=0x7ff718011160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff718009420 osd=3 1 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.4 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff73681d700 1 -- 172.21.15.28:0/643842946 reap_dead start 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.134:6800/3063775514 conn(0x7ff7180074e0 legacy=0x7ff718018c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff73681d700 1 -- 172.21.15.28:0/643842946 reap_dead start 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180132b0 osd=4 1 2024-03-30T14:56:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.5 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.134:6804/3668151179 conn(0x7ff71801c6c0 legacy=0x7ff71801a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180072d0 osd=5 1 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.6 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.134:6808/2135224194 conn(0x7ff71801f3b0 legacy=0x7ff7180217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff71801c210 osd=6 1 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter close_session for osd.7 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.134:6812/3758190298 conn(0x7ff718011ff0 legacy=0x7ff718022ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter put_session s=0x7ff7180118d0 osd=7 1 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 20 client.24751.objecter shutdown clearing up homeless session... 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 10 client.24751.objecter successfully canceled tick 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff71c0703b0 legacy=0x7ff71c072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> v1:172.21.15.134:6789/0 conn(0x7ff730110ca0 legacy=0x7ff7301ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff73681d700 1 -- 172.21.15.28:0/643842946 reap_dead start 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff73681d700 1 -- 172.21.15.28:0/643842946 reap_dead start 2024-03-30T14:56:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 shutdown_connections 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 >> 172.21.15.28:0/643842946 conn(0x7ff7300ffd20 msgr2=0x7ff730113910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 shutdown_connections 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 -- 172.21.15.28:0/643842946 wait complete. 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:56:59.399+0000 7ff7257fa700 1 librados: shutdown 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:56:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:56:59.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:59.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:59.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:56:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/643842946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:59.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:56:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/643842946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:59.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:56:59.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:56:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:56:59.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:56:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:56:59.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:56:59.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:56:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:56:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/643842946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:56:59.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:00.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:00.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:00.222 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T14:57:00.003Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004199496s 2024-03-30T14:57:00.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:00.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:00.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:00.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:00.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:00.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:00.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:00.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:00.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:00.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:00.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:00.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:57:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:00.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:00 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:00.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:00.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:00.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:00.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: osdmap e100: 8 total, 8 up, 8 in 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:00.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:00 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:01.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:01.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:01.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:01.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:01.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:01.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:01.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:01.524 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:57:01.524 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:57:01.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:01.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:01.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:01.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:01.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:01.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:01.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:01.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:01.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:01.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:02.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:02.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:02.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:57:02.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:02.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:02.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:02.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:02 smithi028 ceph-mon[29876]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:02.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:02 smithi028 ceph-mon[41614]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:02.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:02.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:02.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T14:57:02.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:02 smithi134 ceph-mon[33714]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:02.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:02.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:02.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:02.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:02.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:03.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:03.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:03.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:03.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:03.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:03.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:03.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:03.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:03.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:03 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:03.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:03 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:03 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:03.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:03.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:03.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:03.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:03.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:04.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:04.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:04.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:04.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:04.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:04.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:04.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:04.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:04.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:04 smithi028 ceph-mon[29876]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:04.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:04 smithi028 ceph-mon[41614]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:04.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:04.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:04 smithi134 ceph-mon[33714]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:04.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:04.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:04.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:05.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:05.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:05.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.268+0000 7fef648c4700 1 -- 172.21.15.28:0/1183218511 >> v1:172.21.15.28:6790/0 conn(0x7fef6010aed0 legacy=0x7fef6010d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.268+0000 7fef648c4700 1 -- 172.21.15.28:0/1183218511 shutdown_connections 2024-03-30T14:57:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.268+0000 7fef648c4700 1 -- 172.21.15.28:0/1183218511 >> 172.21.15.28:0/1183218511 conn(0x7fef600fdaf0 msgr2=0x7fef600fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.268+0000 7fef648c4700 1 -- 172.21.15.28:0/1183218511 shutdown_connections 2024-03-30T14:57:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.268+0000 7fef648c4700 1 -- 172.21.15.28:0/1183218511 wait complete. 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 librados: starting msgr at 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 librados: starting objecter 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 Processor -- start 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 -- start start 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 librados: setting wanted keys 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.269+0000 7fef648c4700 1 librados: calling monclient init 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef648c4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef601a70a0 con 0x7fef60102240 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef648c4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef601a82c0 con 0x7fef6010aed0 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef648c4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fef601a9500 con 0x7fef6010ea70 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef5ed9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fef6010ea70 0x7fef601a5760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54100/0 (socket says 172.21.15.28:54100) 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef5e59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fef60102240 0x7fef60107540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42452/0 (socket says 172.21.15.28:42452) 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef5ed9d700 1 -- 172.21.15.28:0/3693826121 learned_addr learned my addr 172.21.15.28:0/3693826121 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter ms_handle_connect 0x7fef6010ea70 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter ms_handle_connect 0x7fef60102240 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter ms_handle_connect 0x7fef6010aed0 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736158071 0 0) 0x7fef601a70a0 con 0x7fef60102240 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef480039c0 con 0x7fef60102240 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048194132 0 0) 0x7fef601a9500 con 0x7fef6010ea70 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef601a70a0 con 0x7fef6010ea70 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734861785 0 0) 0x7fef601a82c0 con 0x7fef6010aed0 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef601a9500 con 0x7fef6010aed0 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 705080917 0 0) 0x7fef480039c0 con 0x7fef60102240 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef601a82c0 con 0x7fef60102240 2024-03-30T14:57:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fef500018b0 con 0x7fef60102240 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 806348676 0 0) 0x7fef601a70a0 con 0x7fef6010ea70 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef480039c0 con 0x7fef6010ea70 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fef58001c10 con 0x7fef6010ea70 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4079038076 0 0) 0x7fef601a9500 con 0x7fef6010aed0 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef601a70a0 con 0x7fef6010aed0 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.270+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fef54004470 con 0x7fef6010aed0 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3247845700 0 0) 0x7fef601a82c0 con 0x7fef60102240 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 >> v1:172.21.15.28:6790/0 conn(0x7fef6010ea70 legacy=0x7fef601a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 >> v1:172.21.15.134:6789/0 conn(0x7fef6010aed0 legacy=0x7fef601a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef601aa740 con 0x7fef60102240 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef601a72d0 con 0x7fef60102240 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 10 client.14838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef601a7790 con 0x7fef60102240 2024-03-30T14:57:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 1 librados: init done 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fef50003b40 con 0x7fef60102240 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fef50005c90 con 0x7fef60102240 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 10 client.14838.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 15 client.14838.objecter _session_command_op_assign -1 1 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 20 client.14838.objecter _get_session osd=-1 returning homeless 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 10 client.14838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.271+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fef40002920 con 0x7fef60102240 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.272+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fef50005f00 con 0x7fef60102240 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(100..100 src has 43..100) ==== 6259+0+0 (unknown 3782213559 0 0) 0x7fef50088d80 con 0x7fef60102240 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 10 client.14838.objecter ms_dispatch 0x7fef60001550 osd_map(100..100 src has 43..100) 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 3 client.14838.objecter handle_osd_map got epochs [100,100] > 0 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 3 client.14838.objecter handle_osd_map decoding full epoch 100 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 10 client.14838.objecter checking command 1 2024-03-30T14:57:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter _get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 15 client.14838.objecter _session_command_op_remove -1 1 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter _get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 15 client.14838.objecter _session_command_op_assign 3 1 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 4 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 10 client.14838.objecter _send_command 1 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6813/1506157250 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef4807ea70 con 0x7fef4807ad60 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 20 client.14838.objecter dump_active .. 0 homeless 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (unknown 2631957453 0 0) 0x7fef500891c0 con 0x7fef60102240 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 10 client.14838.objecter ms_handle_connect 0x7fef48070470 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.273+0000 7fef4f7fe700 10 client.14838.objecter ms_handle_connect 0x7fef4807ad60 2024-03-30T14:57:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.274+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fef4807ea70 con 0x7fef4807ad60 2024-03-30T14:57:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.274+0000 7fef4f7fe700 10 client.14838.objecter ms_dispatch 0x7fef60001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.274+0000 7fef4f7fe700 10 client.14838.objecter _finish_command 1 = system:0 2024-03-30T14:57:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.274+0000 7fef4f7fe700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 4 2024-03-30T14:57:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.274+0000 7fef4f7fe700 15 client.14838.objecter _session_command_op_remove 3 1 2024-03-30T14:57:05.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 10 client.14838.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 15 client.14838.objecter _session_command_op_assign -1 2 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 20 client.14838.objecter _get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 20 client.14838.objecter _get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 15 client.14838.objecter _session_command_op_remove -1 2 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 20 client.14838.objecter get_session s=0x7fef4807a6b0 osd=3 3 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 15 client.14838.objecter _session_command_op_assign 3 2 2024-03-30T14:57:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 4 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 10 client.14838.objecter _send_command 2 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 --> v1:172.21.15.28:6813/1506157250 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fef400011f0 con 0x7fef4807ad60 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef4f7fe700 1 -- 172.21.15.28:0/3693826121 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fef400011f0 con 0x7fef4807ad60 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef4f7fe700 10 client.14838.objecter ms_dispatch 0x7fef60001550 command_reply(tid 2: 0 ) 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef4f7fe700 10 client.14838.objecter _finish_command 2 = system:0 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef4f7fe700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 4 2024-03-30T14:57:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef4f7fe700 15 client.14838.objecter _session_command_op_remove 3 2 2024-03-30T14:57:05.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 10 librados: watch_flush enter 2024-03-30T14:57:05.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 10 librados: watch_flush exit 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 10 client.14838.objecter close_session for osd.3 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.315+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 >> v1:172.21.15.28:6813/1506157250 conn(0x7fef4807ad60 legacy=0x7fef4807d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 20 client.14838.objecter put_session s=0x7fef4807a6b0 osd=3 1 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 20 client.14838.objecter shutdown clearing up homeless session... 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 10 client.14838.objecter successfully canceled tick 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 >> v1:172.21.15.28:6800/3011431093 conn(0x7fef48070470 legacy=0x7fef48072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 >> v1:172.21.15.28:6789/0 conn(0x7fef60102240 legacy=0x7fef60107540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef5ed9d700 1 -- 172.21.15.28:0/3693826121 reap_dead start 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 shutdown_connections 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 >> 172.21.15.28:0/3693826121 conn(0x7fef600fdaf0 msgr2=0x7fef60111c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 shutdown_connections 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 -- 172.21.15.28:0/3693826121 wait complete. 2024-03-30T14:57:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.316+0000 7fef648c4700 1 librados: shutdown 2024-03-30T14:57:05.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:05.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.502+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/1718405983 >> v1:172.21.15.28:6790/0 conn(0x7f8bd410eab0 legacy=0x7f8bd4110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.502+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/1718405983 shutdown_connections 2024-03-30T14:57:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.502+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/1718405983 >> 172.21.15.28:0/1718405983 conn(0x7f8bd40fc9e0 msgr2=0x7f8bd40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.502+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/1718405983 shutdown_connections 2024-03-30T14:57:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.502+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/1718405983 wait complete. 2024-03-30T14:57:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 librados: starting msgr at 2024-03-30T14:57:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 librados: starting objecter 2024-03-30T14:57:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 Processor -- start 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 -- start start 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 librados: setting wanted keys 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 librados: calling monclient init 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bd40fec10 con 0x7f8bd410eab0 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bd410ac90 con 0x7f8bd4103190 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bdc4ff700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bd41a9640 con 0x7f8bd410afa0 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bd9a9a700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8bd410afa0 0x7f8bd41081b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54116/0 (socket says 172.21.15.28:54116) 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bd9a9a700 1 -- 172.21.15.28:0/4108041429 learned_addr learned my addr 172.21.15.28:0/4108041429 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f8bd410eab0 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bcb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f8bd410afa0 2024-03-30T14:57:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.503+0000 7f8bcb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f8bd4103190 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1924789838 0 0) 0x7f8bd40fec10 con 0x7f8bd410eab0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc00039c0 con 0x7f8bd410eab0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543376438 0 0) 0x7f8bd41a9640 con 0x7f8bd410afa0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd40fec10 con 0x7f8bd410afa0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195021064 0 0) 0x7f8bd410ac90 con 0x7f8bd4103190 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd41a9640 con 0x7f8bd4103190 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1229865738 0 0) 0x7f8bc00039c0 con 0x7f8bd410eab0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bd410ac90 con 0x7f8bd410eab0 2024-03-30T14:57:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8bcc001870 con 0x7f8bd410eab0 2024-03-30T14:57:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1837695013 0 0) 0x7f8bd40fec10 con 0x7f8bd410afa0 2024-03-30T14:57:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bc00039c0 con 0x7f8bd410afa0 2024-03-30T14:57:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8bc40017f0 con 0x7f8bd410afa0 2024-03-30T14:57:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4051075300 0 0) 0x7f8bd41a9640 con 0x7f8bd4103190 2024-03-30T14:57:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bd40fec10 con 0x7f8bd4103190 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8bd0001890 con 0x7f8bd4103190 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 165791986 0 0) 0x7f8bc00039c0 con 0x7f8bd410afa0 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 >> v1:172.21.15.134:6789/0 conn(0x7f8bd4103190 legacy=0x7f8bd4107aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 >> v1:172.21.15.28:6789/0 conn(0x7f8bd410eab0 legacy=0x7f8bd41a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd41aa820 con 0x7f8bd410afa0 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd41a9870 con 0x7f8bd410afa0 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bdc4ff700 10 client.34733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bd41a9d30 con 0x7f8bd410afa0 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.504+0000 7f8bdc4ff700 1 librados: init done 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.505+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8bc4003b40 con 0x7f8bd410afa0 2024-03-30T14:57:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.505+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8bc4005c30 con 0x7f8bd410afa0 2024-03-30T14:57:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.505+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb8005120 con 0x7f8bd410afa0 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.506+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8bc40036a0 con 0x7f8bd410afa0 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.506+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(100..100 src has 43..100) ==== 6259+0+0 (unknown 3782213559 0 0) 0x7f8bc4088ee0 con 0x7f8bd410afa0 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.506+0000 7f8bcb7fe700 10 client.34733.objecter ms_dispatch 0x7f8bd4001550 osd_map(100..100 src has 43..100) 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.506+0000 7f8bcb7fe700 3 client.34733.objecter handle_osd_map got epochs [100,100] > 0 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.506+0000 7f8bcb7fe700 3 client.34733.objecter handle_osd_map decoding full epoch 100 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.507+0000 7f8bcb7fe700 20 client.34733.objecter dump_active .. 0 homeless 2024-03-30T14:57:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.507+0000 7f8bcb7fe700 10 client.34733.objecter ms_handle_connect 0x7f8bc0070440 2024-03-30T14:57:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.515+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8bc40568a0 con 0x7f8bd410afa0 2024-03-30T14:57:05.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:05.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:05.765+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8bb8005410 con 0x7f8bd410afa0 2024-03-30T14:57:05.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:05.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:05.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:05.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:05.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:06.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:06.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:06.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:06.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:06.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:06.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:06.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-30T14:57:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:06.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.525+0000 7f8bcb7fe700 1 -- 172.21.15.28:0/4108041429 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v102) ==== 91+0+0 (unknown 3595958917 0 0) 0x7f8bc405a2f0 con 0x7f8bd410afa0 2024-03-30T14:57:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 10 librados: watch_flush enter 2024-03-30T14:57:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 10 librados: watch_flush exit 2024-03-30T14:57:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 20 client.34733.objecter shutdown clearing up homeless session... 2024-03-30T14:57:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 10 client.34733.objecter successfully canceled tick 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8bc0070440 legacy=0x7f8bc0072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 >> v1:172.21.15.28:6790/0 conn(0x7f8bd410afa0 legacy=0x7f8bd41081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 shutdown_connections 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 >> 172.21.15.28:0/4108041429 conn(0x7f8bd40fc9e0 msgr2=0x7f8bd4110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 shutdown_connections 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 -- 172.21.15.28:0/4108041429 wait complete. 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:06.528+0000 7f8bdc4ff700 1 librados: shutdown 2024-03-30T14:57:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T14:57:06.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:06.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:06.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4108041429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[29876]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:06.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4108041429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:06 smithi028 ceph-mon[41614]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:06.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: osdmap e101: 8 total, 8 up, 8 in 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4108041429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:06 smithi134 ceph-mon[33714]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:57:06.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:06.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:07.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:07.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:07.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:07.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:07.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:07.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:07.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:07.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:07.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:07.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:07.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[41614]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:57:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[41614]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:57:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[29876]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:57:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:07 smithi028 ceph-mon[29876]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:57:07.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:07.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:07.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:07.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:07 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:07 smithi134 ceph-mon[33714]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:57:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:07 smithi134 ceph-mon[33714]: osdmap e102: 8 total, 8 up, 8 in 2024-03-30T14:57:07.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:08.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:08.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:08.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:08.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:08.376 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:08.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:08.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:08.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:08.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:08.548 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:08.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:08.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:08.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:08.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:08.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T14:57:08.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T14:57:08.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T14:57:08.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[41614]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T14:57:08.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:08 smithi028 ceph-mon[29876]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-30T14:57:08.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:08 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T14:57:08.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:08 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T14:57:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:08 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T14:57:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:08 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T14:57:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:08 smithi134 ceph-mon[33714]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-30T14:57:08.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:08.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:08.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:09.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:09.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:09.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:09.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:09.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 -- 172.21.15.28:0/3133329488 >> v1:172.21.15.28:6789/0 conn(0x7f22a8109fe0 legacy=0x7f22a810c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 -- 172.21.15.28:0/3133329488 shutdown_connections 2024-03-30T14:57:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 -- 172.21.15.28:0/3133329488 >> 172.21.15.28:0/3133329488 conn(0x7f22a8077370 msgr2=0x7f22a80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 -- 172.21.15.28:0/3133329488 shutdown_connections 2024-03-30T14:57:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 -- 172.21.15.28:0/3133329488 wait complete. 2024-03-30T14:57:09.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 librados: starting msgr at 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.283+0000 7f22acfad700 1 librados: starting objecter 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 Processor -- start 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 -- start start 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 librados: setting wanted keys 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 librados: calling monclient init 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22a81a7370 con 0x7f22a8101870 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22a81a8590 con 0x7f22a8109fe0 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22acfad700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22a81a97d0 con 0x7f22a810db80 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22a6d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f22a810db80 0x7f22a81a5a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47600/0 (socket says 172.21.15.28:47600) 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f22a6d9d700 1 -- 172.21.15.28:0/2793913187 learned_addr learned my addr 172.21.15.28:0/2793913187 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter ms_handle_connect 0x7f22a810db80 2024-03-30T14:57:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter ms_handle_connect 0x7f22a8101870 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter ms_handle_connect 0x7f22a8109fe0 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623152243 0 0) 0x7f22a81a97d0 con 0x7f22a810db80 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22940031f0 con 0x7f22a810db80 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329994881 0 0) 0x7f22a81a7370 con 0x7f22a8101870 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.284+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a81a97d0 con 0x7f22a8101870 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784062484 0 0) 0x7f22a81a8590 con 0x7f22a8109fe0 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22a81a7370 con 0x7f22a8109fe0 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2012058926 0 0) 0x7f22940031f0 con 0x7f22a810db80 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a81a8590 con 0x7f22a810db80 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22a0001c90 con 0x7f22a810db80 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2755359003 0 0) 0x7f22a81a97d0 con 0x7f22a8101870 2024-03-30T14:57:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22940031f0 con 0x7f22a8101870 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22980017f0 con 0x7f22a8101870 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3991096250 0 0) 0x7f22a81a7370 con 0x7f22a8109fe0 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22a81a97d0 con 0x7f22a8109fe0 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2290004680 con 0x7f22a8109fe0 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1501787691 0 0) 0x7f22a81a8590 con 0x7f22a810db80 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.134:6789/0 conn(0x7f22a8109fe0 legacy=0x7f22a81a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6789/0 conn(0x7f22a8101870 legacy=0x7f22a8106720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a81aaa10 con 0x7f22a810db80 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22a81a75a0 con 0x7f22a810db80 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f22acfad700 10 client.34739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f22a81a7a60 con 0x7f22a810db80 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f22acfad700 1 librados: init done 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f22a0003ec0 con 0x7f22a810db80 2024-03-30T14:57:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.285+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22a0006010 con 0x7f22a810db80 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.286+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2288005100 con 0x7f22a810db80 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.287+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f22a0003a20 con 0x7f22a810db80 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.287+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(102..102 src has 43..102) ==== 6259+0+0 (unknown 161333944 0 0) 0x7f22a0089200 con 0x7f22a810db80 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.287+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 osd_map(102..102 src has 43..102) 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.287+0000 7f229f7fe700 3 client.34739.objecter handle_osd_map got epochs [102,102] > 0 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.287+0000 7f229f7fe700 3 client.34739.objecter handle_osd_map decoding full epoch 102 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f229f7fe700 20 client.34739.objecter dump_active .. 0 homeless 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) ==== 54+0+16 (unknown 2944338779 0 2420000192) 0x7f22a008b630 con 0x7f22a810db80 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f2294070460 2024-03-30T14:57:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 10 client.34739.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 1 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 1 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 0 1 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 4 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 10 client.34739.objecter _send_command 1 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2288004b70 con 0x7f2288004730 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.288+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f2288004730 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.289+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2288004b70 con 0x7f2288004730 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.289+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.289+0000 7f229f7fe700 10 client.34739.objecter _finish_command 1 = system:0 2024-03-30T14:57:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.289+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 4 2024-03-30T14:57:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.289+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 0 1 2024-03-30T14:57:09.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 10 client.34739.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 2 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 2 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f22880053e0 osd=0 3 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 0 2 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 4 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 10 client.34739.objecter _send_command 2 2024-03-30T14:57:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.306+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22880079f0 con 0x7f2288004730 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22880079f0 con 0x7f2288004730 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f229f7fe700 10 client.34739.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 4 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 0 2 2024-03-30T14:57:09.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:57:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 10 client.34739.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 3 2024-03-30T14:57:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 1 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 4 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 10 client.34739.objecter _send_command 3 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.309+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2288004430 con 0x7f2288002100 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f2288002100 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2288004430 con 0x7f2288002100 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 3: 0 ) 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 10 client.34739.objecter _finish_command 3 = system:0 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 4 2024-03-30T14:57:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.310+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 1 3 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 10 client.34739.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 4 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 4 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f22880040f0 osd=1 3 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 1 4 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 4 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 10 client.34739.objecter _send_command 4 2024-03-30T14:57:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.329+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22880016c0 con 0x7f2288002100 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22880016c0 con 0x7f2288002100 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 10 client.34739.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 4 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 1 4 2024-03-30T14:57:09.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 10 client.34739.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 5 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 5 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 2 5 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 4 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 10 client.34739.objecter _send_command 5 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f22880110e0 con 0x7f228800d5d0 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.331+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f228800d5d0 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.332+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22880110e0 con 0x7f228800d5d0 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.332+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 5: 0 ) 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.332+0000 7f229f7fe700 10 client.34739.objecter _finish_command 5 = system:0 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.332+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 4 2024-03-30T14:57:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.332+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 2 5 2024-03-30T14:57:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 10 client.34739.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 6 2024-03-30T14:57:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 6 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f2288008370 osd=2 3 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 2 6 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 4 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 10 client.34739.objecter _send_command 6 2024-03-30T14:57:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.349+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2288007ef0 con 0x7f228800d5d0 2024-03-30T14:57:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.351+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2288007ef0 con 0x7f228800d5d0 2024-03-30T14:57:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.351+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.351+0000 7f229f7fe700 10 client.34739.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.351+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 4 2024-03-30T14:57:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.351+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 2 6 2024-03-30T14:57:09.355 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:57:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 10 client.34739.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 7 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 7 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 3 7 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 4 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 10 client.34739.objecter _send_command 7 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f228800d3b0 con 0x7f2288012410 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.352+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f2288012410 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.353+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f228800d3b0 con 0x7f2288012410 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.353+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 7: 0 ) 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.353+0000 7f229f7fe700 10 client.34739.objecter _finish_command 7 = system:0 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.353+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 4 2024-03-30T14:57:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.353+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 3 7 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 10 client.34739.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 8 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 8 2024-03-30T14:57:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f2288008100 osd=3 3 2024-03-30T14:57:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 3 8 2024-03-30T14:57:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 4 2024-03-30T14:57:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 10 client.34739.objecter _send_command 8 2024-03-30T14:57:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.383+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22880137e0 con 0x7f2288012410 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22880137e0 con 0x7f2288012410 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f229f7fe700 10 client.34739.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 4 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 3 8 2024-03-30T14:57:09.387 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 10 client.34739.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 9 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 9 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 4 9 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 4 2024-03-30T14:57:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 10 client.34739.objecter _send_command 9 2024-03-30T14:57:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.385+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f22880139b0 con 0x7f2288017130 2024-03-30T14:57:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f2288017130 2024-03-30T14:57:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22880139b0 con 0x7f2288017130 2024-03-30T14:57:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 9: 0 ) 2024-03-30T14:57:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 10 client.34739.objecter _finish_command 9 = system:0 2024-03-30T14:57:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 4 2024-03-30T14:57:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.386+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 4 9 2024-03-30T14:57:09.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 10 client.34739.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 10 2024-03-30T14:57:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 10 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228800cc60 osd=4 3 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 4 10 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 4 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 10 client.34739.objecter _send_command 10 2024-03-30T14:57:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.405+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f228801c410 con 0x7f2288017130 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f228801c410 con 0x7f2288017130 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f229f7fe700 10 client.34739.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 4 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 4 10 2024-03-30T14:57:09.409 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 10 client.34739.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 11 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 11 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 5 11 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 4 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 10 client.34739.objecter _send_command 11 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.407+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f228801cfe0 con 0x7f228801cb30 2024-03-30T14:57:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.408+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f228801cb30 2024-03-30T14:57:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.409+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f228801cfe0 con 0x7f228801cb30 2024-03-30T14:57:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.409+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 11: 0 ) 2024-03-30T14:57:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.409+0000 7f229f7fe700 10 client.34739.objecter _finish_command 11 = system:0 2024-03-30T14:57:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.409+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 4 2024-03-30T14:57:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.409+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 5 11 2024-03-30T14:57:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 10 client.34739.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 12 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 12 2024-03-30T14:57:09.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801c410 osd=5 3 2024-03-30T14:57:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 5 12 2024-03-30T14:57:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 4 2024-03-30T14:57:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 10 client.34739.objecter _send_command 12 2024-03-30T14:57:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.441+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f228801cfe0 con 0x7f228801cb30 2024-03-30T14:57:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f228801cfe0 con 0x7f228801cb30 2024-03-30T14:57:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 10 client.34739.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 4 2024-03-30T14:57:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 5 12 2024-03-30T14:57:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 10 client.34739.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 13 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 13 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 6 13 2024-03-30T14:57:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 4 2024-03-30T14:57:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 10 client.34739.objecter _send_command 13 2024-03-30T14:57:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f228801d260 con 0x7f228801dcf0 2024-03-30T14:57:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.443+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f228801dcf0 2024-03-30T14:57:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.444+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f228801d260 con 0x7f228801dcf0 2024-03-30T14:57:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.444+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 13: 0 ) 2024-03-30T14:57:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.444+0000 7f229f7fe700 10 client.34739.objecter _finish_command 13 = system:0 2024-03-30T14:57:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.444+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 4 2024-03-30T14:57:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.444+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 6 13 2024-03-30T14:57:09.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 10 client.34739.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 14 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 14 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801cfe0 osd=6 3 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 6 14 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 4 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 10 client.34739.objecter _send_command 14 2024-03-30T14:57:09.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.463+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f228801d260 con 0x7f228801dcf0 2024-03-30T14:57:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.465+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f228801d260 con 0x7f228801dcf0 2024-03-30T14:57:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.465+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.468 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.465+0000 7f229f7fe700 10 client.34739.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.465+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 4 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.465+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 6 14 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 10 client.34739.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 15 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 15 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 7 15 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 4 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 10 client.34739.objecter _send_command 15 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f228802bdc0 con 0x7f22880282b0 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.466+0000 7f229f7fe700 10 client.34739.objecter ms_handle_connect 0x7f22880282b0 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.467+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f228802bdc0 con 0x7f22880282b0 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.467+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 15: 0 ) 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.467+0000 7f229f7fe700 10 client.34739.objecter _finish_command 15 = system:0 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.467+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 4 2024-03-30T14:57:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.467+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 7 15 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 10 client.34739.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign -1 16 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 20 client.34739.objecter _get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 15 client.34739.objecter _session_command_op_remove -1 16 2024-03-30T14:57:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 20 client.34739.objecter get_session s=0x7f228801c290 osd=7 3 2024-03-30T14:57:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 15 client.34739.objecter _session_command_op_assign 7 16 2024-03-30T14:57:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 4 2024-03-30T14:57:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 10 client.34739.objecter _send_command 16 2024-03-30T14:57:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.490+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f228802bfa0 con 0x7f22880282b0 2024-03-30T14:57:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f229f7fe700 1 -- 172.21.15.28:0/2793913187 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f228802bfa0 con 0x7f22880282b0 2024-03-30T14:57:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f229f7fe700 10 client.34739.objecter ms_dispatch 0x7f22a8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f229f7fe700 10 client.34739.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f229f7fe700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 4 2024-03-30T14:57:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f229f7fe700 15 client.34739.objecter _session_command_op_remove 7 16 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 librados: watch_flush enter 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 librados: watch_flush exit 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.0 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6801/3394323575 conn(0x7f2288004730 legacy=0x7f2288002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880053e0 osd=0 1 2024-03-30T14:57:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.1 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6805/1850313547 conn(0x7f2288002100 legacy=0x7f228800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f22880040f0 osd=1 1 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.2 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6809/3451325043 conn(0x7f228800d5d0 legacy=0x7f228800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008370 osd=2 1 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.3 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6813/1506157250 conn(0x7f2288012410 legacy=0x7f2288012890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f2288008100 osd=3 1 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.4 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.134:6800/3063775514 conn(0x7f2288017130 legacy=0x7f2288019520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22a6d9d700 1 -- 172.21.15.28:0/2793913187 reap_dead start 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228800cc60 osd=4 1 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.5 2024-03-30T14:57:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.134:6804/3668151179 conn(0x7f228801cb30 legacy=0x7f228801d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22a6d9d700 1 -- 172.21.15.28:0/2793913187 reap_dead start 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c410 osd=5 1 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.6 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.134:6808/2135224194 conn(0x7f228801dcf0 legacy=0x7f2288026b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801cfe0 osd=6 1 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter close_session for osd.7 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.134:6812/3758190298 conn(0x7f22880282b0 legacy=0x7f228802a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter put_session s=0x7f228801c290 osd=7 1 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 20 client.34739.objecter shutdown clearing up homeless session... 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22acfad700 10 client.34739.objecter successfully canceled tick 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.492+0000 7f22a6d9d700 1 -- 172.21.15.28:0/2793913187 reap_dead start 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2294070460 legacy=0x7f2294072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> v1:172.21.15.28:6790/0 conn(0x7f22a810db80 legacy=0x7f22a81a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 shutdown_connections 2024-03-30T14:57:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 >> 172.21.15.28:0/2793913187 conn(0x7f22a8077370 msgr2=0x7f22a8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 shutdown_connections 2024-03-30T14:57:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 -- 172.21.15.28:0/2793913187 wait complete. 2024-03-30T14:57:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:09.493+0000 7f22acfad700 1 librados: shutdown 2024-03-30T14:57:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:57:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:57:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:57:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:57:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:57:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:57:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:57:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:57:09.534 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:09.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:09.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2793913187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:09.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2793913187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:09.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:09.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:09.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:09.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:09.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2793913187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:09.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:10.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:10.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:10.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:10.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:10.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:10.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:10.429 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:10.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:10.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:57:10.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:10.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:10.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T14:57:10.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T14:57:10.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T14:57:10.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:10.655 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:10 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:10.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:10.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:10.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:10 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:10.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:10.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:10.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:11.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:11.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:11.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:11.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:11.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:11.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.514+0000 7f3822e79700 1 -- 172.21.15.28:0/847655551 >> v1:172.21.15.28:6790/0 conn(0x7f381c10aed0 legacy=0x7f381c10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.514+0000 7f3822e79700 1 -- 172.21.15.28:0/847655551 shutdown_connections 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.514+0000 7f3822e79700 1 -- 172.21.15.28:0/847655551 >> 172.21.15.28:0/847655551 conn(0x7f381c0fdaf0 msgr2=0x7f381c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.514+0000 7f3822e79700 1 -- 172.21.15.28:0/847655551 shutdown_connections 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.514+0000 7f3822e79700 1 -- 172.21.15.28:0/847655551 wait complete. 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 librados: starting msgr at 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 librados: starting objecter 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 Processor -- start 2024-03-30T14:57:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 -- start start 2024-03-30T14:57:11.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 librados: setting wanted keys 2024-03-30T14:57:11.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 librados: calling monclient init 2024-03-30T14:57:11.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f381c1a7180 con 0x7f381c10aed0 2024-03-30T14:57:11.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f381c1a83a0 con 0x7f381c10ea70 2024-03-30T14:57:11.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3822e79700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f381c1a95e0 con 0x7f381c102240 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3820c15700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f381c102240 0x7f381c107610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47604/0 (socket says 172.21.15.28:47604) 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f381bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f381c10aed0 0x7f381c1a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47230/0 (socket says 172.21.15.28:47230) 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.515+0000 7f3820c15700 1 -- 172.21.15.28:0/3128770832 learned_addr learned my addr 172.21.15.28:0/3128770832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 10 client.?.objecter ms_handle_connect 0x7f381c102240 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 10 client.?.objecter ms_handle_connect 0x7f381c10aed0 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 726981060 0 0) 0x7f381c1a95e0 con 0x7f381c102240 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f380c0031f0 con 0x7f381c102240 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447575903 0 0) 0x7f381c1a7180 con 0x7f381c10aed0 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f381c1a95e0 con 0x7f381c10aed0 2024-03-30T14:57:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1430639050 0 0) 0x7f380c0031f0 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f381c1a7180 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38100019b0 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1416048114 0 0) 0x7f381c1a95e0 con 0x7f381c10aed0 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f380c0031f0 con 0x7f381c10aed0 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3808004610 con 0x7f381c10aed0 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 302896918 0 0) 0x7f381c1a7180 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 >> v1:172.21.15.134:6789/0 conn(0x7f381c10ea70 legacy=0x7f381c1a5840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 >> v1:172.21.15.28:6789/0 conn(0x7f381c10aed0 legacy=0x7f381c1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1aa820 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.516+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f381c1a73b0 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 10 client.34745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f381c1a7870 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 1 librados: init done 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3810003b20 con 0x7f381c102240 2024-03-30T14:57:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3810005cd0 con 0x7f381c102240 2024-03-30T14:57:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 10 client.34745.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 15 client.34745.objecter _session_command_op_assign -1 1 2024-03-30T14:57:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 20 client.34745.objecter _get_session osd=-1 returning homeless 2024-03-30T14:57:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.517+0000 7f3822e79700 10 client.34745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.518+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3800002920 con 0x7f381c102240 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.519+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3810005fa0 con 0x7f381c102240 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(102..102 src has 43..102) ==== 6259+0+0 (unknown 161333944 0 0) 0x7f3810088f90 con 0x7f381c102240 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 10 client.34745.objecter ms_dispatch 0x7f381c001550 osd_map(102..102 src has 43..102) 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 3 client.34745.objecter handle_osd_map got epochs [102,102] > 0 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 3 client.34745.objecter handle_osd_map decoding full epoch 102 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 10 client.34745.objecter checking command 1 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter _get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 15 client.34745.objecter _session_command_op_remove -1 1 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter _get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 15 client.34745.objecter _session_command_op_assign 4 1 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 4 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 10 client.34745.objecter _send_command 1 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.134:6800/3063775514 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f380c07e990 con 0x7f380c07ac80 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 20 client.34745.objecter dump_active .. 0 homeless 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 10 client.34745.objecter ms_handle_connect 0x7f380c070390 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (unknown 2098468544 0 0) 0x7f38100893d0 con 0x7f381c102240 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.520+0000 7f3819ffb700 10 client.34745.objecter ms_handle_connect 0x7f380c07ac80 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.521+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f380c07e990 con 0x7f380c07ac80 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.521+0000 7f3819ffb700 10 client.34745.objecter ms_dispatch 0x7f381c001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.521+0000 7f3819ffb700 10 client.34745.objecter _finish_command 1 = system:0 2024-03-30T14:57:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.521+0000 7f3819ffb700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 4 2024-03-30T14:57:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.521+0000 7f3819ffb700 15 client.34745.objecter _session_command_op_remove 4 1 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T14:57:11.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T14:57:11.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T14:57:11.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:11 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T14:57:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 10 client.34745.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:57:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 15 client.34745.objecter _session_command_op_assign -1 2 2024-03-30T14:57:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 20 client.34745.objecter _get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 20 client.34745.objecter _get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 15 client.34745.objecter _session_command_op_remove -1 2 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 20 client.34745.objecter get_session s=0x7f380c07a5d0 osd=4 3 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 15 client.34745.objecter _session_command_op_assign 4 2 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 4 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 10 client.34745.objecter _send_command 2 2024-03-30T14:57:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.555+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 --> v1:172.21.15.134:6800/3063775514 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f38000011f0 con 0x7f380c07ac80 2024-03-30T14:57:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3819ffb700 1 -- 172.21.15.28:0/3128770832 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f38000011f0 con 0x7f380c07ac80 2024-03-30T14:57:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3819ffb700 10 client.34745.objecter ms_dispatch 0x7f381c001550 command_reply(tid 2: 0 ) 2024-03-30T14:57:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3819ffb700 10 client.34745.objecter _finish_command 2 = system:0 2024-03-30T14:57:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3819ffb700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 4 2024-03-30T14:57:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3819ffb700 15 client.34745.objecter _session_command_op_remove 4 2 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 10 librados: watch_flush enter 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 10 librados: watch_flush exit 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 10 client.34745.objecter close_session for osd.4 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 >> v1:172.21.15.134:6800/3063775514 conn(0x7f380c07ac80 legacy=0x7f380c07d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 20 client.34745.objecter put_session s=0x7f380c07a5d0 osd=4 1 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 20 client.34745.objecter shutdown clearing up homeless session... 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.556+0000 7f3822e79700 10 client.34745.objecter successfully canceled tick 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 >> v1:172.21.15.28:6800/3011431093 conn(0x7f380c070390 legacy=0x7f380c072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 >> v1:172.21.15.28:6790/0 conn(0x7f381c102240 legacy=0x7f381c107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3821416700 1 -- 172.21.15.28:0/3128770832 reap_dead start 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 shutdown_connections 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 >> 172.21.15.28:0/3128770832 conn(0x7f381c0fdaf0 msgr2=0x7f381c1116c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 shutdown_connections 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 -- 172.21.15.28:0/3128770832 wait complete. 2024-03-30T14:57:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:11.557+0000 7f3822e79700 1 librados: shutdown 2024-03-30T14:57:11.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:11.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:11.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:11.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:11.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:11.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:11 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T14:57:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:11 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T14:57:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:11 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T14:57:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:11 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T14:57:11.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:12.007 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:12.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:12.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:12.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:12.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:12.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:12.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:12.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:12.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:12.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:12.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:12 smithi134 ceph-mon[33714]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:12 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T14:57:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:12 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T14:57:12.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[41614]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T14:57:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T14:57:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[29876]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T14:57:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:12 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T14:57:13.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:13.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:13.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:13.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:13.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:13.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:57:13.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:13.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:13.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:13.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:13.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:13.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T14:57:13.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T14:57:13.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T14:57:13.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:13 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T14:57:13.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:13.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:13.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T14:57:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:13 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T14:57:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:13.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:14.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:14.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:14.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:14.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:14.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:14.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:14.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:14.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:14.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:14.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:14.682 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:14.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:14.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:14.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:14.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T14:57:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T14:57:14.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T14:57:14.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:14.857 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:14.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:14.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:14.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:15.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:15.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:15.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:15.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:15.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:15.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:15.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:15.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:15.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:15.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:15.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:15.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:15.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:15.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:15.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T14:57:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:15 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T14:57:15.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T14:57:15.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: osdmap e103: 8 total, 8 up, 8 in 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T14:57:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T14:57:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:15 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T14:57:15.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:16.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:16.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:16.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:16.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:16.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:16.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:16.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:16.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:16.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:16.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:16.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:16.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T14:57:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:16.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T14:57:16.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T14:57:16.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T14:57:16.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:16 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T14:57:16.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:16.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:16.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T14:57:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T14:57:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T14:57:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T14:57:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T14:57:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:16 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T14:57:16.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:16.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:17.036 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:17.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:17.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:17.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:17.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:17.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:17.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:17.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:17.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:17.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T14:57:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T14:57:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T14:57:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T14:57:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T14:57:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T14:57:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T14:57:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:17 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T14:57:17.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:17.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:17 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T14:57:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:17 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T14:57:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:17 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T14:57:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:17 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T14:57:17.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:17.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:17.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:18.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:18.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:18.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:18.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-30T14:57:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:18.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:18.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:18.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:18.549 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:18.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:18.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:18.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T14:57:18.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T14:57:18.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:18 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T14:57:18.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:18.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:18.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T14:57:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:18 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T14:57:19.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:19.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:19.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:19.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:19.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:19.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:19.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:19.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:19.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:19.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:19.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:19.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:19.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T14:57:19.848 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:57:19.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T14:57:19.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T14:57:19.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T14:57:19.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T14:57:19.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:19 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T14:57:19.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:19.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T14:57:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:19 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T14:57:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:19.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:19.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:20.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:20.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:20.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:20.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:20.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:20.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:20.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:20.711 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:20.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:20.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:57:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:57:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:20 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:20.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:57:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:20 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:21.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:21.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:21.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:21.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:21.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:21.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:21.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:21.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:21.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T14:57:21.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:21.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:21.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T14:57:21.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:21 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T14:57:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:21 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T14:57:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:21 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T14:57:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:21 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T14:57:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:21 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T14:57:21.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:22.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:22.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:22.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:22.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:22.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:22.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:22.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:22.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.370+0000 7f75eeffd700 1 -- 172.21.15.28:0/1457742814 <== mon.2 v1:172.21.15.28:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75e0004730 con 0x7f75f0100f80 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.370+0000 7f75f6419700 1 -- 172.21.15.28:0/1457742814 >> v1:172.21.15.28:6790/0 conn(0x7f75f0100f80 legacy=0x7f75f0101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.370+0000 7f75f6419700 1 -- 172.21.15.28:0/1457742814 shutdown_connections 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.370+0000 7f75f6419700 1 -- 172.21.15.28:0/1457742814 >> 172.21.15.28:0/1457742814 conn(0x7f75f00fc9e0 msgr2=0x7f75f00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.371+0000 7f75f6419700 1 -- 172.21.15.28:0/1457742814 shutdown_connections 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.371+0000 7f75f6419700 1 -- 172.21.15.28:0/1457742814 wait complete. 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 librados: starting msgr at 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 librados: starting objecter 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 Processor -- start 2024-03-30T14:57:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 -- start start 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 librados: setting wanted keys 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75f6419700 1 librados: calling monclient init 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75ef7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f75f0115290 0x7f75f01a84f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52254/0 (socket says 172.21.15.28:52254) 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75effff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f75f0106d80 0x7f75f0114b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44876/0 (socket says 172.21.15.28:44876) 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.372+0000 7f75ef7fe700 1 -- 172.21.15.28:0/2825840583 learned_addr learned my addr 172.21.15.28:0/2825840583 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter ms_handle_connect 0x7f75f0106d80 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f75f01adb40 con 0x7f75f0106d80 2024-03-30T14:57:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter ms_handle_connect 0x7f75f0115290 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter ms_handle_connect 0x7f75f01a9d40 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f75f01aed60 con 0x7f75f01a9d40 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f75f01add70 con 0x7f75f0115290 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516643353 0 0) 0x7f75f01adb40 con 0x7f75f0106d80 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75d80031f0 con 0x7f75f0106d80 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923719300 0 0) 0x7f75f01add70 con 0x7f75f0115290 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75f01adb40 con 0x7f75f0115290 2024-03-30T14:57:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223659819 0 0) 0x7f75f01aed60 con 0x7f75f01a9d40 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.373+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75f01add70 con 0x7f75f01a9d40 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3815895336 0 0) 0x7f75d80031f0 con 0x7f75f0106d80 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75f01aed60 con 0x7f75f0106d80 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1719806016 0 0) 0x7f75f01add70 con 0x7f75f01a9d40 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75d80031f0 con 0x7f75f01a9d40 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75e0004470 con 0x7f75f0106d80 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75e8001a60 con 0x7f75f01a9d40 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3075305921 0 0) 0x7f75f01adb40 con 0x7f75f0115290 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75f01add70 con 0x7f75f0115290 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75e4001910 con 0x7f75f0115290 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1756745723 0 0) 0x7f75f01aed60 con 0x7f75f0106d80 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 >> v1:172.21.15.28:6790/0 conn(0x7f75f0115290 legacy=0x7f75f01a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 >> v1:172.21.15.134:6789/0 conn(0x7f75f01a9d40 legacy=0x7f75f01ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f01adf80 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f75f01ae270 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75f6419700 10 client.14862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.374+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f75f01ae730 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.375+0000 7f75f6419700 1 librados: init done 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.375+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f75e0001d10 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.375+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75e0004e70 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.377+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc005120 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.378+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f75e0004e70 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.378+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(103..103 src has 53..103) ==== 6259+0+0 (unknown 3481079941 0 0) 0x7f75e0089040 con 0x7f75f0106d80 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.378+0000 7f75ed7fa700 10 client.14862.objecter ms_dispatch 0x7f75f0001550 osd_map(103..103 src has 53..103) 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.378+0000 7f75ed7fa700 3 client.14862.objecter handle_osd_map got epochs [103,103] > 0 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.378+0000 7f75ed7fa700 3 client.14862.objecter handle_osd_map decoding full epoch 103 2024-03-30T14:57:22.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.379+0000 7f75ed7fa700 20 client.14862.objecter dump_active .. 0 homeless 2024-03-30T14:57:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.379+0000 7f75ed7fa700 10 client.14862.objecter ms_handle_connect 0x7f75d8070480 2024-03-30T14:57:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.388+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f75e00569a0 con 0x7f75f0106d80 2024-03-30T14:57:22.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:22.779+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f75dc005410 con 0x7f75f0106d80 2024-03-30T14:57:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:22.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[29876]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[41614]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:22.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:22 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:22.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:22.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:22.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:22.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:22 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T14:57:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:22 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T14:57:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:22 smithi134 ceph-mon[33714]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:22 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:22 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:23.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:23.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:23.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:23.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:23.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:23.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:23.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:23.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:23.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.600+0000 7f75ed7fa700 1 -- 172.21.15.28:0/2825840583 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v105) ==== 101+0+0 (unknown 2951015359 0 0) 0x7f75e005a3f0 con 0x7f75f0106d80 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.603+0000 7f75f6419700 10 librados: watch_flush enter 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.603+0000 7f75f6419700 10 librados: watch_flush exit 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.603+0000 7f75f6419700 20 client.14862.objecter shutdown clearing up homeless session... 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.603+0000 7f75f6419700 10 client.14862.objecter successfully canceled tick 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.604+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 >> v1:172.21.15.28:6800/3011431093 conn(0x7f75d8070480 legacy=0x7f75d8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.604+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 >> v1:172.21.15.28:6789/0 conn(0x7f75f0106d80 legacy=0x7f75f0114b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:23.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.606+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 shutdown_connections 2024-03-30T14:57:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.606+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 >> 172.21.15.28:0/2825840583 conn(0x7f75f00fc9e0 msgr2=0x7f75f00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.607+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 shutdown_connections 2024-03-30T14:57:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.607+0000 7f75f6419700 1 -- 172.21.15.28:0/2825840583 wait complete. 2024-03-30T14:57:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:23.607+0000 7f75f6419700 1 librados: shutdown 2024-03-30T14:57:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T14:57:23.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:23.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T14:57:23.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T14:57:23.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:57:23.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T14:57:23.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T14:57:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T14:57:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T14:57:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T14:57:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T14:57:23.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:23 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T14:57:23.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:23.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: osdmap e104: 8 total, 8 up, 8 in 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T14:57:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T14:57:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T14:57:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T14:57:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:23 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T14:57:23.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:23.960 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:23.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:24.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:24.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:24.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:24.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:24.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:24.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:24.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:24.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:24.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: Cluster is now healthy 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: Cluster is now healthy 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:24.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:24 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:24.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:24.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:24.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:24.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:24.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:57:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: Cluster is now healthy 2024-03-30T14:57:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2825840583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: osdmap e105: 8 total, 8 up, 8 in 2024-03-30T14:57:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T14:57:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T14:57:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:24 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:24.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:25.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:25.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:25.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:25.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:25.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:25.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:25.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:25.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:25.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.686+0000 7f773b69b700 1 -- 172.21.15.28:0/3260547211 >> v1:172.21.15.28:6789/0 conn(0x7f773410aee0 legacy=0x7f773410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.686+0000 7f773b69b700 1 -- 172.21.15.28:0/3260547211 shutdown_connections 2024-03-30T14:57:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.686+0000 7f773b69b700 1 -- 172.21.15.28:0/3260547211 >> 172.21.15.28:0/3260547211 conn(0x7f77340fdae0 msgr2=0x7f77340fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.687+0000 7f773b69b700 1 -- 172.21.15.28:0/3260547211 shutdown_connections 2024-03-30T14:57:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.687+0000 7f773b69b700 1 -- 172.21.15.28:0/3260547211 wait complete. 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.687+0000 7f773b69b700 1 librados: starting msgr at 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.687+0000 7f773b69b700 1 librados: starting objecter 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.687+0000 7f773b69b700 1 Processor -- start 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 -- start start 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 librados: setting wanted keys 2024-03-30T14:57:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 librados: calling monclient init 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77341a93d0 con 0x7f7734102250 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77341aa5f0 con 0x7f773410aee0 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f773b69b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77341ab830 con 0x7f773410ea80 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f7739c38700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f773410ea80 0x7f77341a7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52312/0 (socket says 172.21.15.28:52312) 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 learned_addr learned my addr 172.21.15.28:0/1739648234 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f772a7fc700 10 client.?.objecter ms_handle_connect 0x7f773410ea80 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f772a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f772a7fc700 10 client.?.objecter ms_handle_connect 0x7f773410aee0 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f772a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.688+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949818773 0 0) 0x7f77341ab830 con 0x7f773410ea80 2024-03-30T14:57:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77200035b0 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 10 client.?.objecter ms_handle_connect 0x7f7734102250 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873245986 0 0) 0x7f77341a93d0 con 0x7f7734102250 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77341ab830 con 0x7f7734102250 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2909263042 0 0) 0x7f77200035b0 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77341a93d0 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.689+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f772c001b30 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1132541021 0 0) 0x7f77341ab830 con 0x7f7734102250 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77200035b0 con 0x7f7734102250 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2836664803 0 0) 0x7f77341a93d0 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.134:6789/0 conn(0x7f773410aee0 legacy=0x7f77341a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6789/0 conn(0x7f7734102250 legacy=0x7f7734107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77341aca70 con 0x7f773410ea80 2024-03-30T14:57:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.690+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f772c003ec0 con 0x7f773410ea80 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.691+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f772c005f60 con 0x7f773410ea80 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.691+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77341aa820 con 0x7f773410ea80 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.691+0000 7f773b69b700 10 client.34763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.691+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f77341aace0 con 0x7f773410ea80 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.691+0000 7f773b69b700 1 librados: init done 2024-03-30T14:57:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.692+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7734103730 con 0x7f773410ea80 2024-03-30T14:57:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.693+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f772c003a00 con 0x7f773410ea80 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(106..106 src has 53..106) ==== 6259+0+0 (unknown 1074157242 0 0) 0x7f772c0893e0 con 0x7f773410ea80 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 osd_map(106..106 src has 53..106) 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 3 client.34763.objecter handle_osd_map got epochs [106,106] > 0 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 3 client.34763.objecter handle_osd_map decoding full epoch 106 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 20 client.34763.objecter dump_active .. 0 homeless 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77200703b0 2024-03-30T14:57:25.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v106) ==== 54+0+16 (unknown 3550766263 0 2420000192) 0x7f772c056d40 con 0x7f773410ea80 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f773b69b700 10 client.34763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 1 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.694+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 1 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 0 1 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 4 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 10 client.34763.objecter _send_command 1 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7734075a70 con 0x7f77341aba00 2024-03-30T14:57:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.695+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77341aba00 2024-03-30T14:57:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.696+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7734075a70 con 0x7f77341aba00 2024-03-30T14:57:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.696+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.696+0000 7f772a7fc700 10 client.34763.objecter _finish_command 1 = system:0 2024-03-30T14:57:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.696+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 4 2024-03-30T14:57:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.696+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 0 1 2024-03-30T14:57:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 10 client.34763.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 2 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 2 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f7734068b30 osd=0 3 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 0 2 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 4 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 10 client.34763.objecter _send_command 2 2024-03-30T14:57:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.730+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341ac4f0 con 0x7f77341aba00 2024-03-30T14:57:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341ac4f0 con 0x7f77341aba00 2024-03-30T14:57:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f772a7fc700 10 client.34763.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 4 2024-03-30T14:57:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 0 2 2024-03-30T14:57:25.735 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 10 client.34763.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 1 3 2024-03-30T14:57:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 4 2024-03-30T14:57:25.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 10 client.34763.objecter _send_command 3 2024-03-30T14:57:25.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.733+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7734109950 con 0x7f7734109530 2024-03-30T14:57:25.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.734+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f7734109530 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 6.11 deep-scrub starts 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 6.11 deep-scrub ok 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T14:57:25.737 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[41614]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 6.11 deep-scrub starts 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 6.11 deep-scrub ok 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T14:57:25.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:25 smithi028 ceph-mon[29876]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:57:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.735+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7734109950 con 0x7f7734109530 2024-03-30T14:57:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.735+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 3: 0 ) 2024-03-30T14:57:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.735+0000 7f772a7fc700 10 client.34763.objecter _finish_command 3 = system:0 2024-03-30T14:57:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.735+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 4 2024-03-30T14:57:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.735+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 1 3 2024-03-30T14:57:25.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 10 client.34763.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 4 2024-03-30T14:57:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 4 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f7734108eb0 osd=1 3 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 1 4 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 4 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 10 client.34763.objecter _send_command 4 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.769+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7734109d80 con 0x7f7734109530 2024-03-30T14:57:25.771 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7734109d80 con 0x7f7734109530 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f772a7fc700 10 client.34763.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 4 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 1 4 2024-03-30T14:57:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 10 client.34763.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 5 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 5 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 2 5 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 4 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 10 client.34763.objecter _send_command 5 2024-03-30T14:57:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.771+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7734063360 con 0x7f77340ff390 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77340ff390 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7734063360 con 0x7f77340ff390 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 5: 0 ) 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 10 client.34763.objecter _finish_command 5 = system:0 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 4 2024-03-30T14:57:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.772+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 2 5 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.652+0000 7fb7577fe700 1 -- 172.21.15.28:0/1733703664 <== mon.0 v1:172.21.15.28:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb744004930 con 0x7fb758109500 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.653+0000 7fb75ed60700 1 -- 172.21.15.28:0/1733703664 >> v1:172.21.15.28:6789/0 conn(0x7fb758109500 legacy=0x7fb75810b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.653+0000 7fb75ed60700 1 -- 172.21.15.28:0/1733703664 shutdown_connections 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.653+0000 7fb75ed60700 1 -- 172.21.15.28:0/1733703664 >> 172.21.15.28:0/1733703664 conn(0x7fb758077370 msgr2=0x7fb7580fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.653+0000 7fb75ed60700 1 -- 172.21.15.28:0/1733703664 shutdown_connections 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.653+0000 7fb75ed60700 1 -- 172.21.15.28:0/1733703664 wait complete. 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.654+0000 7fb75ed60700 1 librados: starting msgr at 2024-03-30T14:57:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.654+0000 7fb75ed60700 1 librados: starting objecter 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.654+0000 7fb75ed60700 1 Processor -- start 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 -- start start 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 librados: setting wanted keys 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 librados: calling monclient init 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb75810cb10 con 0x7fb758109500 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb75810cd20 con 0x7fb75810d0d0 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb75ed60700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7581afe50 con 0x7fb758106650 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb757fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb758109500 0x7fb7581a8fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44898/0 (socket says 172.21.15.28:44898) 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb757fff700 1 -- 172.21.15.28:0/2115352205 learned_addr learned my addr 172.21.15.28:0/2115352205 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb755ffb700 10 client.?.objecter ms_handle_connect 0x7fb758106650 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb755ffb700 10 client.?.objecter ms_handle_connect 0x7fb758109500 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.655+0000 7fb755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 10 client.?.objecter ms_handle_connect 0x7fb75810d0d0 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377117873 0 0) 0x7fb75810cb10 con 0x7fb758109500 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb748003910 con 0x7fb758109500 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493343779 0 0) 0x7fb7581afe50 con 0x7fb758106650 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb75810cb10 con 0x7fb758106650 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902093973 0 0) 0x7fb75810cd20 con 0x7fb75810d0d0 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7581afe50 con 0x7fb75810d0d0 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 290818854 0 0) 0x7fb748003910 con 0x7fb758109500 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.656+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb75810cd20 con 0x7fb758109500 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 657553878 0 0) 0x7fb75810cb10 con 0x7fb758106650 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb748003910 con 0x7fb758106650 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb744001430 con 0x7fb758109500 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb74c001830 con 0x7fb758106650 2024-03-30T14:57:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 164082046 0 0) 0x7fb75810cd20 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 >> v1:172.21.15.28:6790/0 conn(0x7fb758106650 legacy=0x7fb75810c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 >> v1:172.21.15.134:6789/0 conn(0x7fb75810d0d0 legacy=0x7fb7581ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.657+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7581b1030 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.658+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7581b0080 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.658+0000 7fb75ed60700 10 client.14868.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.658+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7581b0540 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.658+0000 7fb75ed60700 1 librados: init done 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.659+0000 7fb75ed60700 10 client.14868.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.659+0000 7fb75ed60700 15 client.14868.objecter _session_command_op_assign -1 1 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.659+0000 7fb75ed60700 20 client.14868.objecter _get_session osd=-1 returning homeless 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.659+0000 7fb75ed60700 10 client.14868.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.659+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb758101d90 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.660+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb744001b70 con 0x7fb758109500 2024-03-30T14:57:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.661+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb744004f90 con 0x7fb758109500 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.661+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb744005210 con 0x7fb758109500 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(106..106 src has 53..106) ==== 6259+0+0 (unknown 1074157242 0 0) 0x7fb744089020 con 0x7fb758109500 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 10 client.14868.objecter ms_dispatch 0x7fb758001550 osd_map(106..106 src has 53..106) 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 3 client.14868.objecter handle_osd_map got epochs [106,106] > 0 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 3 client.14868.objecter handle_osd_map decoding full epoch 106 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 10 client.14868.objecter checking command 1 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter _get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 15 client.14868.objecter _session_command_op_remove -1 1 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter _get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 15 client.14868.objecter _session_command_op_assign 5 1 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 4 2024-03-30T14:57:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 10 client.14868.objecter _send_command 1 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.134:6804/3668151179 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb744001b70 con 0x7fb74807ac80 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 20 client.14868.objecter dump_active .. 0 homeless 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.662+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=106) ==== 24+0+0 (unknown 1205723201 0 0) 0x7fb744089400 con 0x7fb758109500 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.663+0000 7fb755ffb700 10 client.14868.objecter ms_handle_connect 0x7fb748070520 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.663+0000 7fb755ffb700 10 client.14868.objecter ms_handle_connect 0x7fb74807ac80 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.664+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb744001b70 con 0x7fb74807ac80 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.664+0000 7fb755ffb700 10 client.14868.objecter ms_dispatch 0x7fb758001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.664+0000 7fb755ffb700 10 client.14868.objecter _finish_command 1 = system:0 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.664+0000 7fb755ffb700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 4 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.664+0000 7fb755ffb700 15 client.14868.objecter _session_command_op_remove 5 1 2024-03-30T14:57:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 10 client.14868.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 15 client.14868.objecter _session_command_op_assign -1 2 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 20 client.14868.objecter _get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 20 client.14868.objecter _get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 15 client.14868.objecter _session_command_op_remove -1 2 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 20 client.14868.objecter get_session s=0x7fb74807a5d0 osd=5 3 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 15 client.14868.objecter _session_command_op_assign 5 2 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 4 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 10 client.14868.objecter _send_command 2 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.702+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 --> v1:172.21.15.134:6804/3668151179 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb758068b30 con 0x7fb74807ac80 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb755ffb700 1 -- 172.21.15.28:0/2115352205 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fb758068b30 con 0x7fb74807ac80 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb755ffb700 10 client.14868.objecter ms_dispatch 0x7fb758001550 command_reply(tid 2: 0 ) 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb755ffb700 10 client.14868.objecter _finish_command 2 = system:0 2024-03-30T14:57:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb755ffb700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 4 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb755ffb700 15 client.14868.objecter _session_command_op_remove 5 2 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 10 librados: watch_flush enter 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 10 librados: watch_flush exit 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 10 client.14868.objecter close_session for osd.5 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 >> v1:172.21.15.134:6804/3668151179 conn(0x7fb74807ac80 legacy=0x7fb74807d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 20 client.14868.objecter put_session s=0x7fb74807a5d0 osd=5 1 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 20 client.14868.objecter shutdown clearing up homeless session... 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 10 client.14868.objecter successfully canceled tick 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb748070520 legacy=0x7fb7480729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 >> v1:172.21.15.28:6789/0 conn(0x7fb758109500 legacy=0x7fb7581a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75d2fd700 1 -- 172.21.15.28:0/2115352205 reap_dead start 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.704+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 shutdown_connections 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.705+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 >> 172.21.15.28:0/2115352205 conn(0x7fb758077370 msgr2=0x7fb75810fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.705+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 shutdown_connections 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.705+0000 7fb75ed60700 1 -- 172.21.15.28:0/2115352205 wait complete. 2024-03-30T14:57:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.706+0000 7fb75ed60700 1 librados: shutdown 2024-03-30T14:57:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 10 client.34763.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 6 2024-03-30T14:57:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 6 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77340ff220 osd=2 3 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 2 6 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 4 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 10 client.34763.objecter _send_command 6 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.791+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341b55d0 con 0x7f77340ff390 2024-03-30T14:57:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.792+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341b55d0 con 0x7f77340ff390 2024-03-30T14:57:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.792+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.792+0000 7f772a7fc700 10 client.34763.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.792+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 4 2024-03-30T14:57:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.792+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 2 6 2024-03-30T14:57:25.794 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:57:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 10 client.34763.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 7 2024-03-30T14:57:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 7 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 3 7 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 4 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 10 client.34763.objecter _send_command 7 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f7734062a40 2024-03-30T14:57:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.793+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f77341b1a40 con 0x7f7734062a40 2024-03-30T14:57:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.794+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f77341b1a40 con 0x7f7734062a40 2024-03-30T14:57:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.794+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 7: 0 ) 2024-03-30T14:57:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.794+0000 7f772a7fc700 10 client.34763.objecter _finish_command 7 = system:0 2024-03-30T14:57:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.794+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 4 2024-03-30T14:57:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.794+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 3 7 2024-03-30T14:57:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:25.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 10 client.34763.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 8 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 8 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77340628d0 osd=3 3 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 3 8 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 4 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 10 client.34763.objecter _send_command 8 2024-03-30T14:57:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.829+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341b0f70 con 0x7f7734062a40 2024-03-30T14:57:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.831+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341b0f70 con 0x7f7734062a40 2024-03-30T14:57:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.831+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.831+0000 7f772a7fc700 10 client.34763.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.831+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 4 2024-03-30T14:57:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.831+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 3 8 2024-03-30T14:57:25.833 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 10 client.34763.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 9 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 9 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 4 9 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 4 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 10 client.34763.objecter _send_command 9 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f77341b0e90 con 0x7f77341b6460 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.832+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77341b6460 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.833+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f77341b0e90 con 0x7f77341b6460 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.833+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 9: 0 ) 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.833+0000 7f772a7fc700 10 client.34763.objecter _finish_command 9 = system:0 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.833+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 4 2024-03-30T14:57:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.833+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 4 9 2024-03-30T14:57:25.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 10 client.34763.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 10 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 10 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b0d20 osd=4 3 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 4 10 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 4 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 10 client.34763.objecter _send_command 10 2024-03-30T14:57:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.866+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341b0e90 con 0x7f77341b6460 2024-03-30T14:57:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.868+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341b0e90 con 0x7f77341b6460 2024-03-30T14:57:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.868+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.868+0000 7f772a7fc700 10 client.34763.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.868+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 4 2024-03-30T14:57:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.868+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 4 10 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 10 client.34763.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 11 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 11 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 5 11 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 4 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 10 client.34763.objecter _send_command 11 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f77341c2b10 con 0x7f77341bedf0 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.869+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77341bedf0 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.870+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f77341c2b10 con 0x7f77341bedf0 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.870+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 11: 0 ) 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.870+0000 7f772a7fc700 10 client.34763.objecter _finish_command 11 = system:0 2024-03-30T14:57:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.870+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 4 2024-03-30T14:57:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.870+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 5 11 2024-03-30T14:57:25.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 10 client.34763.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 12 2024-03-30T14:57:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 12 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b0e90 osd=5 3 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 5 12 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 4 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 10 client.34763.objecter _send_command 12 2024-03-30T14:57:25.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.904+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341b59d0 con 0x7f77341bedf0 2024-03-30T14:57:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.906+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341b59d0 con 0x7f77341bedf0 2024-03-30T14:57:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.906+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.906+0000 7f772a7fc700 10 client.34763.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.906+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 4 2024-03-30T14:57:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.906+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 5 12 2024-03-30T14:57:25.908 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:57:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 10 client.34763.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 13 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 13 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 6 13 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 4 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 10 client.34763.objecter _send_command 13 2024-03-30T14:57:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.907+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f77341c42a0 con 0x7f77341b5780 2024-03-30T14:57:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.908+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77341b5780 2024-03-30T14:57:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.909+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f77341c42a0 con 0x7f77341b5780 2024-03-30T14:57:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.909+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 13: 0 ) 2024-03-30T14:57:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.909+0000 7f772a7fc700 10 client.34763.objecter _finish_command 13 = system:0 2024-03-30T14:57:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.909+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 4 2024-03-30T14:57:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.909+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 6 13 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 6.11 deep-scrub starts 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 6.11 deep-scrub ok 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T14:57:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:25 smithi134 ceph-mon[33714]: osdmap e106: 8 total, 8 up, 8 in 2024-03-30T14:57:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:25.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 10 client.34763.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 14 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 14 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341b11e0 osd=6 3 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 6 14 2024-03-30T14:57:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 4 2024-03-30T14:57:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 10 client.34763.objecter _send_command 14 2024-03-30T14:57:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.943+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341c42a0 con 0x7f77341b5780 2024-03-30T14:57:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341c42a0 con 0x7f77341b5780 2024-03-30T14:57:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f772a7fc700 10 client.34763.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 4 2024-03-30T14:57:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 6 14 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 10 client.34763.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 15 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 15 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 7 15 2024-03-30T14:57:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 4 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 10 client.34763.objecter _send_command 15 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.945+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f77341cb4d0 con 0x7f77341badb0 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.946+0000 7f772a7fc700 10 client.34763.objecter ms_handle_connect 0x7f77341badb0 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.947+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f77341cb4d0 con 0x7f77341badb0 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.947+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 15: 0 ) 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.947+0000 7f772a7fc700 10 client.34763.objecter _finish_command 15 = system:0 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.947+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 4 2024-03-30T14:57:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.947+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 7 15 2024-03-30T14:57:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 10 client.34763.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign -1 16 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 20 client.34763.objecter _get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 15 client.34763.objecter _session_command_op_remove -1 16 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 20 client.34763.objecter get_session s=0x7f77341c5420 osd=7 3 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 15 client.34763.objecter _session_command_op_assign 7 16 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 4 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 10 client.34763.objecter _send_command 16 2024-03-30T14:57:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.980+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f77341cce80 con 0x7f77341badb0 2024-03-30T14:57:25.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.982+0000 7f772a7fc700 1 -- 172.21.15.28:0/1739648234 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f77341cce80 con 0x7f77341badb0 2024-03-30T14:57:25.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.982+0000 7f772a7fc700 10 client.34763.objecter ms_dispatch 0x7f7734001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:25.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.982+0000 7f772a7fc700 10 client.34763.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:25.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.982+0000 7f772a7fc700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 4 2024-03-30T14:57:25.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.982+0000 7f772a7fc700 15 client.34763.objecter _session_command_op_remove 7 16 2024-03-30T14:57:25.984 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 librados: watch_flush enter 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 librados: watch_flush exit 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.0 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6801/3394323575 conn(0x7f77341aba00 legacy=0x7f77341abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734068b30 osd=0 1 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.1 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6805/1850313547 conn(0x7f7734109530 legacy=0x7f773402c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f7734108eb0 osd=1 1 2024-03-30T14:57:25.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.2 2024-03-30T14:57:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6809/3451325043 conn(0x7f77340ff390 legacy=0x7f77341b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340ff220 osd=2 1 2024-03-30T14:57:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.3 2024-03-30T14:57:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6813/1506157250 conn(0x7f7734062a40 legacy=0x7f77341b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77340628d0 osd=3 1 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.4 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.134:6800/3063775514 conn(0x7f77341b6460 legacy=0x7f77341bd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0d20 osd=4 1 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.5 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.134:6804/3668151179 conn(0x7f77341bedf0 legacy=0x7f77341c1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b0e90 osd=5 1 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.6 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.134:6808/2135224194 conn(0x7f77341b5780 legacy=0x7f77341c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341b11e0 osd=6 1 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter close_session for osd.7 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.134:6812/3758190298 conn(0x7f77341badb0 legacy=0x7f77341c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter put_session s=0x7f77341c5420 osd=7 1 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 20 client.34763.objecter shutdown clearing up homeless session... 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 10 client.34763.objecter successfully canceled tick 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 reap_dead start 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 reap_dead start 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 reap_dead start 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 reap_dead start 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f7739c38700 1 -- 172.21.15.28:0/1739648234 reap_dead start 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.983+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6800/3011431093 conn(0x7f77200703b0 legacy=0x7f7720072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> v1:172.21.15.28:6790/0 conn(0x7f773410ea80 legacy=0x7f77341a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 shutdown_connections 2024-03-30T14:57:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 >> 172.21.15.28:0/1739648234 conn(0x7f77340fdae0 msgr2=0x7f7734111c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 shutdown_connections 2024-03-30T14:57:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 -- 172.21.15.28:0/1739648234 wait complete. 2024-03-30T14:57:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:25.984+0000 7f773b69b700 1 librados: shutdown 2024-03-30T14:57:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:57:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:57:26.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:26.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:26.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:26.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:26.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:26.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:26.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:26.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:26.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:26.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:26.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:26.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1739648234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: 4.7 deep-scrub starts 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[41614]: 4.7 deep-scrub ok 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1739648234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T14:57:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T14:57:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: 4.7 deep-scrub starts 2024-03-30T14:57:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:26 smithi028 ceph-mon[29876]: 4.7 deep-scrub ok 2024-03-30T14:57:26.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1739648234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: 4.7 deep-scrub starts 2024-03-30T14:57:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:26 smithi134 ceph-mon[33714]: 4.7 deep-scrub ok 2024-03-30T14:57:26.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:26.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:26.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:27.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:27.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:27.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:57:27.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:27.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:27.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:27.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-30T14:57:27.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:27.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:27.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:27.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:27.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:27 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T14:57:27.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:27 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T14:57:27.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:27 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T14:57:27.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:27 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T14:57:27.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:27.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:27.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:27.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:27 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T14:57:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:27 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T14:57:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:27.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:28.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:28.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:28.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:28.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:28.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:28.405 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:28.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:28.444 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:28.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:28.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:28.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[29876]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T14:57:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:28 smithi028 ceph-mon[41614]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:28.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:57:28.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:28.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:28.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:28.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:28.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:28 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T14:57:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:28 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T14:57:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:28 smithi134 ceph-mon[33714]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-30T14:57:29.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:29.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:29.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:29.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:29.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:29.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:29.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:29.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.1e deep-scrub starts 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.1e deep-scrub ok 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T14:57:29.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.1e deep-scrub starts 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.1e deep-scrub ok 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:29.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:29 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:29.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:29.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:29.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:29.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.1e deep-scrub starts 2024-03-30T14:57:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.1e deep-scrub ok 2024-03-30T14:57:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T14:57:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T14:57:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T14:57:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T14:57:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:29 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:29.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:29.975 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:29.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:30.077 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:30.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:30.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.236 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:30.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:30.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:30.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:30.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:57:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:57:30.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:30.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:30.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 6.1c deep-scrub starts 2024-03-30T14:57:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 6.1c deep-scrub ok 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T14:57:30.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 6.1c deep-scrub starts 2024-03-30T14:57:30.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 6.1c deep-scrub ok 2024-03-30T14:57:30.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:30.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T14:57:30.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:30 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T14:57:30.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:30.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:30.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:30.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:30.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T14:57:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T14:57:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: osdmap e107: 8 total, 8 up, 8 in 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 6.1c deep-scrub starts 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 6.1c deep-scrub ok 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T14:57:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:30 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T14:57:30.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:31.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:31.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:31.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:31.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:31.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:31.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:31.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:31.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:31.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T14:57:31.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T14:57:31.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T14:57:31.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:31 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T14:57:31.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:31 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T14:57:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:31 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T14:57:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:31 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T14:57:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:31 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T14:57:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:31.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:32.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:32.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:32.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:32.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:32.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:32.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:32.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:32.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:32.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:32.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T14:57:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:32.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:32.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:32.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T14:57:32.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T14:57:32.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T14:57:32.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:32.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T14:57:32.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:32 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T14:57:32.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:32.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:32.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.863+0000 7f5eb3d03700 1 -- 172.21.15.28:0/3652735174 >> v1:172.21.15.28:6789/0 conn(0x7f5ea4098990 legacy=0x7f5ea4098d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.863+0000 7f5eb3d03700 1 -- 172.21.15.28:0/3652735174 shutdown_connections 2024-03-30T14:57:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.863+0000 7f5eb3d03700 1 -- 172.21.15.28:0/3652735174 >> 172.21.15.28:0/3652735174 conn(0x7f5ea400a610 msgr2=0x7f5ea400aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.863+0000 7f5eb3d03700 1 -- 172.21.15.28:0/3652735174 shutdown_connections 2024-03-30T14:57:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.863+0000 7f5eb3d03700 1 -- 172.21.15.28:0/3652735174 wait complete. 2024-03-30T14:57:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 librados: starting msgr at 2024-03-30T14:57:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 librados: starting objecter 2024-03-30T14:57:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 Processor -- start 2024-03-30T14:57:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 -- start start 2024-03-30T14:57:32.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 librados: setting wanted keys 2024-03-30T14:57:32.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.864+0000 7f5eb3d03700 1 librados: calling monclient init 2024-03-30T14:57:32.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5eb1a9f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ea4098990 0x7f5ea4095270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41190/0 (socket says 172.21.15.28:41190) 2024-03-30T14:57:32.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5eb1a9f700 1 -- 172.21.15.28:0/1571404446 learned_addr learned my addr 172.21.15.28:0/1571404446 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ea4131b20 con 0x7f5ea4098990 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ea4142400 con 0x7f5ea409d970 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ea4143620 con 0x7f5ea40a1570 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f5ea4098990 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f5ea409d970 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter ms_handle_connect 0x7f5ea40a1570 2024-03-30T14:57:32.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592001577 0 0) 0x7f5ea4142400 con 0x7f5ea409d970 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e980035b0 con 0x7f5ea409d970 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584764113 0 0) 0x7f5ea4131b20 con 0x7f5ea4098990 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ea4142400 con 0x7f5ea4098990 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 440304204 0 0) 0x7f5ea4143620 con 0x7f5ea40a1570 2024-03-30T14:57:32.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.865+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ea4131b20 con 0x7f5ea40a1570 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3539171793 0 0) 0x7f5e980035b0 con 0x7f5ea409d970 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ea4143620 con 0x7f5ea409d970 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3131137569 0 0) 0x7f5ea4131b20 con 0x7f5ea40a1570 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e980035b0 con 0x7f5ea40a1570 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 520044733 0 0) 0x7f5ea4142400 con 0x7f5ea4098990 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ea4131b20 con 0x7f5ea4098990 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5e9c001990 con 0x7f5ea409d970 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5ea8001760 con 0x7f5ea40a1570 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.866+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5eac050e90 con 0x7f5ea4098990 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1336027219 0 0) 0x7f5ea4143620 con 0x7f5ea409d970 2024-03-30T14:57:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 >> v1:172.21.15.28:6790/0 conn(0x7f5ea40a1570 legacy=0x7f5ea413fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 >> v1:172.21.15.28:6789/0 conn(0x7f5ea4098990 legacy=0x7f5ea4095270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea4144880 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea4142630 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5eb3d03700 10 client.24796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ea4142af0 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5eb3d03700 1 librados: init done 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5e9c001c50 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.867+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5e9c004b90 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.868+0000 7f5eb3d03700 10 client.24796.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.868+0000 7f5eb3d03700 15 client.24796.objecter _session_command_op_assign -1 1 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5e9c01b210 con 0x7f5ea409d970 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5eb3d03700 20 client.24796.objecter _get_session osd=-1 returning homeless 2024-03-30T14:57:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5eb3d03700 10 client.24796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5e90002920 con 0x7f5ea409d970 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(107..107 src has 53..107) ==== 6259+0+0 (unknown 3062461253 0 0) 0x7f5e9c088ce0 con 0x7f5ea409d970 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5ea2ffd700 10 client.24796.objecter ms_dispatch 0x7f5eac001550 osd_map(107..107 src has 53..107) 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5ea2ffd700 3 client.24796.objecter handle_osd_map got epochs [107,107] > 0 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.869+0000 7f5ea2ffd700 3 client.24796.objecter handle_osd_map decoding full epoch 107 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 10 client.24796.objecter checking command 1 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter _get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 15 client.24796.objecter _session_command_op_remove -1 1 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter _get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 15 client.24796.objecter _session_command_op_assign 6 1 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 4 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 10 client.24796.objecter _send_command 1 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6808/2135224194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e9807e980 con 0x7f5e9807ac70 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 20 client.24796.objecter dump_active .. 0 homeless 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 10 client.24796.objecter ms_handle_connect 0x7f5e980703b0 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.870+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (unknown 3050433983 0 0) 0x7f5e9c055d10 con 0x7f5ea409d970 2024-03-30T14:57:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 10 client.24796.objecter ms_handle_connect 0x7f5e9807ac70 2024-03-30T14:57:32.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5e9807e980 con 0x7f5e9807ac70 2024-03-30T14:57:32.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 10 client.24796.objecter ms_dispatch 0x7f5eac001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:32.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 10 client.24796.objecter _finish_command 1 = system:0 2024-03-30T14:57:32.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 4 2024-03-30T14:57:32.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.871+0000 7f5ea2ffd700 15 client.24796.objecter _session_command_op_remove 6 1 2024-03-30T14:57:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 10 client.24796.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:57:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 15 client.24796.objecter _session_command_op_assign -1 2 2024-03-30T14:57:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 20 client.24796.objecter _get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 20 client.24796.objecter _get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 15 client.24796.objecter _session_command_op_remove -1 2 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 20 client.24796.objecter get_session s=0x7f5e9807a5c0 osd=6 3 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 15 client.24796.objecter _session_command_op_assign 6 2 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 4 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 10 client.24796.objecter _send_command 2 2024-03-30T14:57:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.910+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 --> v1:172.21.15.134:6808/2135224194 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f5e90001150 con 0x7f5e9807ac70 2024-03-30T14:57:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5ea2ffd700 1 -- 172.21.15.28:0/1571404446 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f5e90001150 con 0x7f5e9807ac70 2024-03-30T14:57:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5ea2ffd700 10 client.24796.objecter ms_dispatch 0x7f5eac001550 command_reply(tid 2: 0 ) 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5ea2ffd700 10 client.24796.objecter _finish_command 2 = system:0 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5ea2ffd700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 4 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5ea2ffd700 15 client.24796.objecter _session_command_op_remove 6 2 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 10 librados: watch_flush enter 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 10 librados: watch_flush exit 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 10 client.24796.objecter close_session for osd.6 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 >> v1:172.21.15.134:6808/2135224194 conn(0x7f5e9807ac70 legacy=0x7f5e9807d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 20 client.24796.objecter put_session s=0x7f5e9807a5c0 osd=6 1 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 20 client.24796.objecter shutdown clearing up homeless session... 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 10 client.24796.objecter successfully canceled tick 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5e980703b0 legacy=0x7f5e98072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.911+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 >> v1:172.21.15.134:6789/0 conn(0x7f5ea409d970 legacy=0x7f5ea413c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.912+0000 7f5eb22a0700 1 -- 172.21.15.28:0/1571404446 reap_dead start 2024-03-30T14:57:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.912+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 shutdown_connections 2024-03-30T14:57:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.912+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 >> 172.21.15.28:0/1571404446 conn(0x7f5ea400a610 msgr2=0x7f5ea40058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.912+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 shutdown_connections 2024-03-30T14:57:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.913+0000 7f5eb3d03700 1 -- 172.21.15.28:0/1571404446 wait complete. 2024-03-30T14:57:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:32.913+0000 7f5eb3d03700 1 librados: shutdown 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T14:57:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:32 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T14:57:32.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:33.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.038+0000 7f2ffceab700 1 -- 172.21.15.28:0/2279020562 >> v1:172.21.15.28:6789/0 conn(0x7f2ff810aff0 legacy=0x7f2ff810d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.039+0000 7f2ffceab700 1 -- 172.21.15.28:0/2279020562 shutdown_connections 2024-03-30T14:57:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.039+0000 7f2ffceab700 1 -- 172.21.15.28:0/2279020562 >> 172.21.15.28:0/2279020562 conn(0x7f2ff80fca30 msgr2=0x7f2ff80fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.039+0000 7f2ffceab700 1 -- 172.21.15.28:0/2279020562 shutdown_connections 2024-03-30T14:57:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.040+0000 7f2ffceab700 1 -- 172.21.15.28:0/2279020562 wait complete. 2024-03-30T14:57:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.040+0000 7f2ffceab700 1 librados: starting msgr at 2024-03-30T14:57:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.040+0000 7f2ffceab700 1 librados: starting objecter 2024-03-30T14:57:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.040+0000 7f2ffceab700 1 Processor -- start 2024-03-30T14:57:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ffceab700 1 -- start start 2024-03-30T14:57:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ffceab700 1 librados: setting wanted keys 2024-03-30T14:57:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ffceab700 1 librados: calling monclient init 2024-03-30T14:57:33.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ff6d9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2ff810eb00 0x7f2ff81a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41206/0 (socket says 172.21.15.28:41206) 2024-03-30T14:57:33.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ff6d9d700 1 -- 172.21.15.28:0/3184852428 learned_addr learned my addr 172.21.15.28:0/3184852428 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:33.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.041+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff81a7010 con 0x7f2ff810eb00 2024-03-30T14:57:33.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 10 client.?.objecter ms_handle_connect 0x7f2ff810eb00 2024-03-30T14:57:33.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:33.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff81a71e0 con 0x7f2ff810aff0 2024-03-30T14:57:33.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff81a73f0 con 0x7f2ff81031e0 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 10 client.?.objecter ms_handle_connect 0x7f2ff810aff0 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221260242 0 0) 0x7f2ff81a7010 con 0x7f2ff810eb00 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.042+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe00031f0 con 0x7f2ff810eb00 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 10 client.?.objecter ms_handle_connect 0x7f2ff81031e0 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 505738859 0 0) 0x7f2ff81a71e0 con 0x7f2ff810aff0 2024-03-30T14:57:33.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ff81a7010 con 0x7f2ff810aff0 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3741163234 0 0) 0x7f2fe00031f0 con 0x7f2ff810eb00 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ff81a71e0 con 0x7f2ff810eb00 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2ff0001bb0 con 0x7f2ff810eb00 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381128682 0 0) 0x7f2ff81a73f0 con 0x7f2ff81031e0 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe00031f0 con 0x7f2ff81031e0 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2956646616 0 0) 0x7f2ff81a7010 con 0x7f2ff810aff0 2024-03-30T14:57:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ff81a73f0 con 0x7f2ff810aff0 2024-03-30T14:57:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 607631448 0 0) 0x7f2ff81a71e0 con 0x7f2ff810eb00 2024-03-30T14:57:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 >> v1:172.21.15.28:6790/0 conn(0x7f2ff81031e0 legacy=0x7f2ff810a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 >> v1:172.21.15.134:6789/0 conn(0x7f2ff810aff0 legacy=0x7f2ff81a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.043+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff81a7600 con 0x7f2ff810eb00 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.044+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2ff0003ec0 con 0x7f2ff810eb00 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.044+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff81a78f0 con 0x7f2ff810eb00 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.044+0000 7f2ffceab700 10 client.14886.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.045+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ff81a7db0 con 0x7f2ff810eb00 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.045+0000 7f2ffceab700 1 librados: init done 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.045+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2ff0005f90 con 0x7f2ff810eb00 2024-03-30T14:57:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.046+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2ff001c900 con 0x7f2ff810eb00 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.046+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff806ae30 con 0x7f2ff810eb00 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.047+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(107..107 src has 53..107) ==== 6259+0+0 (unknown 3062461253 0 0) 0x7f2ff0087ba0 con 0x7f2ff810eb00 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.047+0000 7f2fe77fe700 10 client.14886.objecter ms_dispatch 0x7f2ff8001550 osd_map(107..107 src has 53..107) 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.047+0000 7f2fe77fe700 3 client.14886.objecter handle_osd_map got epochs [107,107] > 0 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.047+0000 7f2fe77fe700 3 client.14886.objecter handle_osd_map decoding full epoch 107 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.048+0000 7f2fe77fe700 20 client.14886.objecter dump_active .. 0 homeless 2024-03-30T14:57:33.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.048+0000 7f2fe77fe700 10 client.14886.objecter ms_handle_connect 0x7f2fe00705f0 2024-03-30T14:57:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.052+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2ff0003a20 con 0x7f2ff810eb00 2024-03-30T14:57:33.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:33.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:33.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.318+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f2ff8068b30 con 0x7f2ff810eb00 2024-03-30T14:57:33.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:33.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:33.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:33.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:33.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.773+0000 7f2fe77fe700 1 -- 172.21.15.28:0/3184852428 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v108) ==== 87+0+0 (unknown 1533436604 0 0) 0x7f2ff0056db0 con 0x7f2ff810eb00 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.775+0000 7f2ffceab700 10 librados: watch_flush enter 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.776+0000 7f2ffceab700 10 librados: watch_flush exit 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.776+0000 7f2ffceab700 20 client.14886.objecter shutdown clearing up homeless session... 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.776+0000 7f2ffceab700 10 client.14886.objecter successfully canceled tick 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.777+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2fe00705f0 legacy=0x7f2fe0072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.777+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 >> v1:172.21.15.28:6789/0 conn(0x7f2ff810eb00 legacy=0x7f2ff81a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.778+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 shutdown_connections 2024-03-30T14:57:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.778+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 >> 172.21.15.28:0/3184852428 conn(0x7f2ff80fca30 msgr2=0x7f2ff80fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.778+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 shutdown_connections 2024-03-30T14:57:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.778+0000 7f2ffceab700 1 -- 172.21.15.28:0/3184852428 wait complete. 2024-03-30T14:57:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:33.779+0000 7f2ffceab700 1 librados: shutdown 2024-03-30T14:57:33.780 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T14:57:33.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:33.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:33.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:33.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:33.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:33.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[41614]: 6.19 deep-scrub starts 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[41614]: 6.19 deep-scrub ok 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[29876]: 6.19 deep-scrub starts 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[29876]: 6.19 deep-scrub ok 2024-03-30T14:57:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:57:34.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:34.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-30T14:57:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:33 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:33 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:33 smithi134 ceph-mon[33714]: 6.19 deep-scrub starts 2024-03-30T14:57:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:33 smithi134 ceph-mon[33714]: 6.19 deep-scrub ok 2024-03-30T14:57:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:57:34.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:34.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:34.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:34.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:34.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:34.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:34.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:34.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:34.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:34.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T14:57:34.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T14:57:34.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:34 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T14:57:34.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:34.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:34.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:34.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:34.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:35.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:35.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3184852428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: osdmap e108: 8 total, 8 up, 8 in 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T14:57:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:34 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T14:57:35.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:35.244 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:35.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:35.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:35.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:35.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:35.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:35.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:35.705 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:35.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:35.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:35.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:35.892 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:35.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:35.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:35 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T14:57:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:35 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T14:57:35.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:36.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:36.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:36.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:36.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:35 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T14:57:36.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:36.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:36.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:36.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:36.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:36.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:36.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:36.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:36.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:36.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:36.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:36.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:36.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:36 smithi028 ceph-mon[41614]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:36.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:36 smithi028 ceph-mon[29876]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:37.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:37.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:37.179 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:36 smithi134 ceph-mon[33714]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:57:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:37.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:57:37.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:37.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:37.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:37.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:37.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.180+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1842799937 >> v1:172.21.15.28:6789/0 conn(0x7ff7ec10ea70 legacy=0x7ff7ec110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.180+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1842799937 shutdown_connections 2024-03-30T14:57:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.180+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1842799937 >> 172.21.15.28:0/1842799937 conn(0x7ff7ec0fdaf0 msgr2=0x7ff7ec0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.180+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1842799937 shutdown_connections 2024-03-30T14:57:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.180+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1842799937 wait complete. 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.181+0000 7ff7f46e6700 1 librados: starting msgr at 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.181+0000 7ff7f46e6700 1 librados: starting objecter 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f46e6700 1 Processor -- start 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f46e6700 1 -- start start 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f46e6700 1 librados: setting wanted keys 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f46e6700 1 librados: calling monclient init 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f1c81700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff7ec10aed0 0x7ff7ec1a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37144/0 (socket says 172.21.15.28:37144) 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f2c83700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff7ec10ea70 0x7ff7ec1a5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41230/0 (socket says 172.21.15.28:41230) 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f1c81700 1 -- 172.21.15.28:0/1345744874 learned_addr learned my addr 172.21.15.28:0/1345744874 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.182+0000 7ff7f2482700 1 --1- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6789/0 conn(0x7ff7ec102240 0x7ff7ec1075c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:55612/0 (socket says 172.21.15.28:55612) 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter ms_handle_connect 0x7ff7ec10aed0 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter ms_handle_connect 0x7ff7ec10ea70 2024-03-30T14:57:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter ms_handle_connect 0x7ff7ec102240 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7ec1a7150 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2821341011 0 0) 0x7ff7ec1a7150 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7ec1a7320 con 0x7ff7ec102240 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7ec1a7530 con 0x7ff7ec10aed0 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d80031f0 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4094383384 0 0) 0x7ff7d80031f0 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7ec1a7150 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff7e40048d0 con 0x7ff7ec10ea70 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196266673 0 0) 0x7ff7ec1a7320 con 0x7ff7ec102240 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d80031f0 con 0x7ff7ec102240 2024-03-30T14:57:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368473549 0 0) 0x7ff7ec1a7530 con 0x7ff7ec10aed0 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.183+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7ec1a7320 con 0x7ff7ec10aed0 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1781656485 0 0) 0x7ff7ec1a7150 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6790/0 conn(0x7ff7ec10aed0 legacy=0x7ff7ec1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6789/0 conn(0x7ff7ec102240 legacy=0x7ff7ec1075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec1a7760 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff7e4001ee0 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff7e40052b0 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.184+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec1a7a50 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.185+0000 7ff7f46e6700 10 client.14892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.185+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7ec1a7f10 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.185+0000 7ff7f46e6700 1 librados: init done 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.185+0000 7ff7f46e6700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff7ec103720 con 0x7ff7ec10ea70 2024-03-30T14:57:37.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff7e4005580 con 0x7ff7ec10ea70 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(108..108 src has 53..108) ==== 6259+0+0 (unknown 2096916314 0 0) 0x7ff7e4089520 con 0x7ff7ec10ea70 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 osd_map(108..108 src has 53..108) 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 3 client.14892.objecter handle_osd_map got epochs [108,108] > 0 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 3 client.14892.objecter handle_osd_map decoding full epoch 108 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 20 client.14892.objecter dump_active .. 0 homeless 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.187+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v108) ==== 54+0+16 (unknown 3989427009 0 2420000192) 0x7ff7e4056e80 con 0x7ff7ec10ea70 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d8070530 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 1 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 1 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 0 1 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 4 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 10 client.14892.objecter _send_command 1 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7d4006d40 con 0x7ff7d4002fd0 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.188+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4002fd0 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.189+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d4006d40 con 0x7ff7d4002fd0 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.189+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.189+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 1 = system:0 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.189+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 4 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.189+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 0 1 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 2 2024-03-30T14:57:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 2 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d40028e0 osd=0 3 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 0 2 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 4 2024-03-30T14:57:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 10 client.14892.objecter _send_command 2 2024-03-30T14:57:37.758 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:57:37.758 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:57:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.207+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d4001240 con 0x7ff7d4002fd0 2024-03-30T14:57:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.208+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d4001240 con 0x7ff7d4002fd0 2024-03-30T14:57:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.208+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.208+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.208+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 4 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.208+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 0 2 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 1 3 2024-03-30T14:57:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 4 2024-03-30T14:57:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 10 client.14892.objecter _send_command 3 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff7d4008cc0 con 0x7ff7d4008100 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.209+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4008100 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.210+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d4008cc0 con 0x7ff7d4008100 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.210+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 3: 0 ) 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.210+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 3 = system:0 2024-03-30T14:57:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.210+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.210+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 1 3 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4001f50 osd=1 3 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 1 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 10 client.14892.objecter _send_command 4 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.243+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d40096e0 con 0x7ff7d4008100 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d40096e0 con 0x7ff7d4008100 2024-03-30T14:57:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 4 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 1 4 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 5 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 5 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 2 5 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 4 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 10 client.14892.objecter _send_command 5 2024-03-30T14:57:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.246+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff7d400eb90 con 0x7ff7d400ce70 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d400ce70 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d400eb90 con 0x7ff7d400ce70 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 5: 0 ) 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 5 = system:0 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 4 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.247+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 2 5 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 6 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 6 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4001d20 osd=2 3 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 2 6 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 4 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 10 client.14892.objecter _send_command 6 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.281+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d40156a0 con 0x7ff7d400ce70 2024-03-30T14:57:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d40156a0 con 0x7ff7d400ce70 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 4 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 2 6 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 7 2024-03-30T14:57:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 7 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 3 7 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 4 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 10 client.14892.objecter _send_command 7 2024-03-30T14:57:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.284+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff7d4007d50 con 0x7ff7d4007750 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.285+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4007750 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.286+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d4007d50 con 0x7ff7d4007750 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.286+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 7: 0 ) 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.286+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 7 = system:0 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.286+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 4 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.286+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 3 7 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 8 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 8 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d400f100 osd=3 3 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 3 8 2024-03-30T14:57:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 4 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 10 client.14892.objecter _send_command 8 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.321+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d40071d0 con 0x7ff7d4007750 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d40071d0 con 0x7ff7d4007750 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 4 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 3 8 2024-03-30T14:57:37.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 9 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 9 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 4 9 2024-03-30T14:57:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 4 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 10 client.14892.objecter _send_command 9 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.324+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff7d40070f0 con 0x7ff7d4016530 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.325+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4016530 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.326+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d40070f0 con 0x7ff7d4016530 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.326+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 9: 0 ) 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.326+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 9 = system:0 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.326+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 4 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.326+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 4 9 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 10 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 10 2024-03-30T14:57:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4006f80 osd=4 3 2024-03-30T14:57:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 4 10 2024-03-30T14:57:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 4 2024-03-30T14:57:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 10 client.14892.objecter _send_command 10 2024-03-30T14:57:37.773 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.360+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d401e9b0 con 0x7ff7d4016530 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.362+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d401e9b0 con 0x7ff7d4016530 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.362+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.362+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.362+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 4 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.362+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 4 10 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 11 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 11 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 5 11 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 4 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 10 client.14892.objecter _send_command 11 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff7d4022cb0 con 0x7ff7d401f030 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.363+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d401f030 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.364+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d4022cb0 con 0x7ff7d401f030 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.364+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 11: 0 ) 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.364+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 11 = system:0 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.364+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 4 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.364+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 5 11 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 12 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 12 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d401e9b0 osd=5 3 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 5 12 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 4 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 10 client.14892.objecter _send_command 12 2024-03-30T14:57:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.398+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d4024660 con 0x7ff7d401f030 2024-03-30T14:57:37.777 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:57:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d4024660 con 0x7ff7d401f030 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 4 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 5 12 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 13 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 13 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 6 13 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 4 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 10 client.14892.objecter _send_command 13 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.400+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff7d4024620 con 0x7ff7d4024ef0 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.401+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4024ef0 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.402+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d4024620 con 0x7ff7d4024ef0 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.402+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 13: 0 ) 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.402+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 13 = system:0 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.402+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 4 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.402+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 6 13 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 14 2024-03-30T14:57:37.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 14 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d4024870 osd=6 3 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 6 14 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 4 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 10 client.14892.objecter _send_command 14 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.435+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d4024620 con 0x7ff7d4024ef0 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.437+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d4024620 con 0x7ff7d4024ef0 2024-03-30T14:57:37.781 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.437+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.437+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.437+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 4 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.437+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 6 14 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 15 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 15 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 7 15 2024-03-30T14:57:37.783 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:57:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 4 2024-03-30T14:57:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 10 client.14892.objecter _send_command 15 2024-03-30T14:57:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff7d402b8f0 con 0x7ff7d4027d70 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.438+0000 7ff7e37fe700 10 client.14892.objecter ms_handle_connect 0x7ff7d4027d70 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.439+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d402b8f0 con 0x7ff7d4027d70 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.439+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 15: 0 ) 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.439+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 15 = system:0 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.439+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 4 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.439+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 7 15 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 10 client.14892.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign -1 16 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 20 client.14892.objecter _get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_remove -1 16 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.472+0000 7ff7e17fa700 20 client.14892.objecter get_session s=0x7ff7d40253d0 osd=7 3 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.473+0000 7ff7e17fa700 15 client.14892.objecter _session_command_op_assign 7 16 2024-03-30T14:57:37.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.473+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 4 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.473+0000 7ff7e17fa700 10 client.14892.objecter _send_command 16 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.473+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff7d402bad0 con 0x7ff7d4027d70 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.474+0000 7ff7e37fe700 1 -- 172.21.15.28:0/1345744874 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff7d402bad0 con 0x7ff7d4027d70 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e37fe700 10 client.14892.objecter ms_dispatch 0x7ff7ec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e37fe700 10 client.14892.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e37fe700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 4 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e37fe700 15 client.14892.objecter _session_command_op_remove 7 16 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 librados: watch_flush enter 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 librados: watch_flush exit 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.0 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6801/3394323575 conn(0x7ff7d4002fd0 legacy=0x7ff7d4005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40028e0 osd=0 1 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.1 2024-03-30T14:57:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6805/1850313547 conn(0x7ff7d4008100 legacy=0x7ff7d40085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001f50 osd=1 1 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.2 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6809/3451325043 conn(0x7ff7d400ce70 legacy=0x7ff7d400d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4001d20 osd=2 1 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.3 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6813/1506157250 conn(0x7ff7d4007750 legacy=0x7ff7d40189e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d400f100 osd=3 1 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.4 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6800/3063775514 conn(0x7ff7d4016530 legacy=0x7ff7d401d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4006f80 osd=4 1 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.5 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6804/3668151179 conn(0x7ff7d401f030 legacy=0x7ff7d4021420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d401e9b0 osd=5 1 2024-03-30T14:57:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.6 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6808/2135224194 conn(0x7ff7d4024ef0 legacy=0x7ff7d4022e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d4024870 osd=6 1 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter close_session for osd.7 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.134:6812/3758190298 conn(0x7ff7d4027d70 legacy=0x7ff7d402a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter put_session s=0x7ff7d40253d0 osd=7 1 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 20 client.14892.objecter shutdown clearing up homeless session... 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 10 client.14892.objecter successfully canceled tick 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff7d8070530 legacy=0x7ff7d80729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7f2c83700 1 -- 172.21.15.28:0/1345744874 reap_dead start 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> v1:172.21.15.28:6789/0 conn(0x7ff7ec10ea70 legacy=0x7ff7ec1a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7f2c83700 1 -- 172.21.15.28:0/1345744874 reap_dead start 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7f2c83700 1 -- 172.21.15.28:0/1345744874 reap_dead start 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7f2c83700 1 -- 172.21.15.28:0/1345744874 reap_dead start 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.475+0000 7ff7f2c83700 1 -- 172.21.15.28:0/1345744874 reap_dead start 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.476+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 shutdown_connections 2024-03-30T14:57:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.476+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 >> 172.21.15.28:0/1345744874 conn(0x7ff7ec0fdaf0 msgr2=0x7ff7ec1006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.476+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 shutdown_connections 2024-03-30T14:57:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.476+0000 7ff7e17fa700 1 -- 172.21.15.28:0/1345744874 wait complete. 2024-03-30T14:57:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:37.476+0000 7ff7e17fa700 1 librados: shutdown 2024-03-30T14:57:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:57:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:57:37.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:37.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:37.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:37.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:37.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:38.055 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1345744874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:38.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1345744874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:38.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:38.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1345744874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:38.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:38.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:38.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:38.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:38.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:38.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:38.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:38.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:38.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:38.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:38.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:39.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.111 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:39.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:38 smithi134 ceph-mon[33714]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:39.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:38 smithi028 ceph-mon[41614]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:38 smithi028 ceph-mon[29876]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:39.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:39.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:39.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:39.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:39.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:39.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:39.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:39.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:39.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:39.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:39.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:39.919 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 94% (11786/12435) Updating files: 95% (11814/12435) Updating files: 96% (11938/12435) Updating files: 97% (12062/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-30T14:57:39.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:40.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:40.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:40.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:40.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:40.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:40.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:40.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:40.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.478+0000 7ff69cd94700 1 -- 172.21.15.28:0/3920670272 >> v1:172.21.15.28:6790/0 conn(0x7ff69810aef0 legacy=0x7ff69810d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 -- 172.21.15.28:0/3920670272 shutdown_connections 2024-03-30T14:57:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 -- 172.21.15.28:0/3920670272 >> 172.21.15.28:0/3920670272 conn(0x7ff6980fdb10 msgr2=0x7ff6980fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 -- 172.21.15.28:0/3920670272 shutdown_connections 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 -- 172.21.15.28:0/3920670272 wait complete. 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 librados: starting msgr at 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 librados: starting objecter 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 Processor -- start 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 -- start start 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 librados: setting wanted keys 2024-03-30T14:57:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.479+0000 7ff69cd94700 1 librados: calling monclient init 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff69cd94700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff698107950 con 0x7ff69810aef0 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff69cd94700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff698107b60 con 0x7ff698102260 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff69cd94700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6981a96b0 con 0x7ff69810ea90 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff696d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff69810ea90 0x7ff6981a5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34910/0 (socket says 172.21.15.28:34910) 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff695d9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff69810aef0 0x7ff698104eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52190/0 (socket says 172.21.15.28:52190) 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff696d9d700 1 -- 172.21.15.28:0/1634167911 learned_addr learned my addr 172.21.15.28:0/1634167911 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter ms_handle_connect 0x7ff69810aef0 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter ms_handle_connect 0x7ff69810ea90 2024-03-30T14:57:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter ms_handle_connect 0x7ff698102260 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962700877 0 0) 0x7ff698107b60 con 0x7ff698102260 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6840031f0 con 0x7ff698102260 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 530912737 0 0) 0x7ff698107950 con 0x7ff69810aef0 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff698107b60 con 0x7ff69810aef0 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652790122 0 0) 0x7ff6981a96b0 con 0x7ff69810ea90 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff698107950 con 0x7ff69810ea90 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3315228162 0 0) 0x7ff6840031f0 con 0x7ff698102260 2024-03-30T14:57:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.480+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6981a96b0 con 0x7ff698102260 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6880013d0 con 0x7ff698102260 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4136087851 0 0) 0x7ff698107b60 con 0x7ff69810aef0 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6840031f0 con 0x7ff69810aef0 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2767768279 0 0) 0x7ff698107950 con 0x7ff69810ea90 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff698107b60 con 0x7ff69810ea90 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff680004470 con 0x7ff69810aef0 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff690001bb0 con 0x7ff69810ea90 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2240350219 0 0) 0x7ff6981a96b0 con 0x7ff698102260 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 >> v1:172.21.15.28:6790/0 conn(0x7ff69810ea90 legacy=0x7ff6981a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 >> v1:172.21.15.28:6789/0 conn(0x7ff69810aef0 legacy=0x7ff698104eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6981aa8f0 con 0x7ff698102260 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6981a98e0 con 0x7ff698102260 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff69cd94700 10 client.24814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6981a9da0 con 0x7ff698102260 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff69cd94700 1 librados: init done 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff688001d30 con 0x7ff698102260 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.481+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6880049c0 con 0x7ff698102260 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.482+0000 7ff69cd94700 10 client.24814.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.482+0000 7ff69cd94700 15 client.24814.objecter _session_command_op_assign -1 1 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.482+0000 7ff69cd94700 20 client.24814.objecter _get_session osd=-1 returning homeless 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.482+0000 7ff69cd94700 10 client.24814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.482+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff678002920 con 0x7ff698102260 2024-03-30T14:57:40.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff688003930 con 0x7ff698102260 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(108..108 src has 53..108) ==== 6259+0+0 (unknown 2096916314 0 0) 0x7ff688087ce0 con 0x7ff698102260 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 10 client.24814.objecter ms_dispatch 0x7ff698001550 osd_map(108..108 src has 53..108) 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 3 client.24814.objecter handle_osd_map got epochs [108,108] > 0 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 3 client.24814.objecter handle_osd_map decoding full epoch 108 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 10 client.24814.objecter checking command 1 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter _get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 15 client.24814.objecter _session_command_op_remove -1 1 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter _get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 15 client.24814.objecter _session_command_op_assign 7 1 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 4 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 10 client.24814.objecter _send_command 1 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6812/3758190298 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff680001080 con 0x7ff68407ad90 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 20 client.24814.objecter dump_active .. 0 homeless 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.483+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=108) ==== 24+0+0 (unknown 1613570471 0 0) 0x7ff6880880c0 con 0x7ff698102260 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.484+0000 7ff68f7fe700 10 client.24814.objecter ms_handle_connect 0x7ff684070530 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.484+0000 7ff68f7fe700 10 client.24814.objecter ms_handle_connect 0x7ff68407ad90 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.485+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff680001080 con 0x7ff68407ad90 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.485+0000 7ff68f7fe700 10 client.24814.objecter ms_dispatch 0x7ff698001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.485+0000 7ff68f7fe700 10 client.24814.objecter _finish_command 1 = system:0 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.485+0000 7ff68f7fe700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 4 2024-03-30T14:57:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.485+0000 7ff68f7fe700 15 client.24814.objecter _session_command_op_remove 7 1 2024-03-30T14:57:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 10 client.24814.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T14:57:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 15 client.24814.objecter _session_command_op_assign -1 2 2024-03-30T14:57:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 20 client.24814.objecter _get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 20 client.24814.objecter _get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 15 client.24814.objecter _session_command_op_remove -1 2 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 20 client.24814.objecter get_session s=0x7ff68407a6e0 osd=7 3 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 15 client.24814.objecter _session_command_op_assign 7 2 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 4 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 10 client.24814.objecter _send_command 2 2024-03-30T14:57:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.504+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 --> v1:172.21.15.134:6812/3758190298 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7ff6780011f0 con 0x7ff68407ad90 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff68f7fe700 1 -- 172.21.15.28:0/1634167911 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7ff6780011f0 con 0x7ff68407ad90 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff68f7fe700 10 client.24814.objecter ms_dispatch 0x7ff698001550 command_reply(tid 2: 0 ) 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff68f7fe700 10 client.24814.objecter _finish_command 2 = system:0 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff68f7fe700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 4 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff68f7fe700 15 client.24814.objecter _session_command_op_remove 7 2 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 10 librados: watch_flush enter 2024-03-30T14:57:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 10 librados: watch_flush exit 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 10 client.24814.objecter close_session for osd.7 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 >> v1:172.21.15.134:6812/3758190298 conn(0x7ff68407ad90 legacy=0x7ff68407d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 20 client.24814.objecter put_session s=0x7ff68407a6e0 osd=7 1 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 20 client.24814.objecter shutdown clearing up homeless session... 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 10 client.24814.objecter successfully canceled tick 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff684070530 legacy=0x7ff6840729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 >> v1:172.21.15.134:6789/0 conn(0x7ff698102260 legacy=0x7ff6981047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff696d9d700 1 -- 172.21.15.28:0/1634167911 reap_dead start 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 shutdown_connections 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 >> 172.21.15.28:0/1634167911 conn(0x7ff6980fdb10 msgr2=0x7ff6981100e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 shutdown_connections 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 -- 172.21.15.28:0/1634167911 wait complete. 2024-03-30T14:57:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:40.505+0000 7ff69cd94700 1 librados: shutdown 2024-03-30T14:57:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:40.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:57:40.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.645 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:40.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:40.717 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '5af0400050a123dc31f105425174f106e057276a'. 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.718 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-03-30T14:57:40.719 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.719 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T14:57:40.719 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-30T14:57:40.719 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 5af0400050a Merge pull request #56547 from rhcs-dashboard/wip-65193-squid 2024-03-30T14:57:40.720 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T14:57:40.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:40.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:40.785 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T14:57:40.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:40 smithi028 ceph-mon[29876]: pgmap v96: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-30T14:57:40.787 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T14:57:40.787 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T14:57:40.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:40.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:40.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:41.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.035 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:41.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:41.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:40 smithi134 ceph-mon[33714]: pgmap v96: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-30T14:57:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:40 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:41.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:40 smithi028 ceph-mon[41614]: pgmap v96: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-30T14:57:41.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:40 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:41.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:40 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:41.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:41.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:57:41.371 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 321.97515652300001, 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93059181899999999, 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:57:41.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381208+0000", 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8050000000000001e-06 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.373 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381216+0000", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2020000000000004e-06 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381226+0000", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5489999999999995e-06 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381235+0000", 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1109999999999997e-06 2024-03-30T14:57:41.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381298+0000", 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2657999999999995e-05 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381306+0000", 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3629999999999994e-06 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310542+0000", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92923640699999999 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310695+0000", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015222900000000001 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310906+0000", 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021148200000000001 2024-03-30T14:57:41.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311551+0000", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064435399999999998 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311644+0000", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3702000000000005e-05 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311735+0000", 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0938000000000002e-05 2024-03-30T14:57:41.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311769+0000", 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3439999999999998e-05 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311798+0000", 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9578999999999999e-05 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.923930+0000", 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.432432646, 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058796459999999997, 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 112 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.923930+0000", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.923930+0000", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.923931+0000", 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.052e-06 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.923985+0000", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3814999999999999e-05 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.923998+0000", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2609000000000001e-05 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924006+0000", 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3489999999999992e-06 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.927690+0000", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003684321 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.927771+0000", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1044999999999994e-05 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.927906+0000", 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134172 2024-03-30T14:57:41.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929602+0000", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001695996 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929658+0000", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5977999999999998e-05 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929714+0000", 2024-03-30T14:57:41.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6765999999999998e-05 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929768+0000", 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4014000000000002e-05 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929810+0000", 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1529000000000002e-05 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.924018+0000", 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.43234509899997, 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006019941, 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 113 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.385 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924018+0000", 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924018+0000", 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924019+0000", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-07 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924024+0000", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0830000000000003e-06 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924030+0000", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6189999999999996e-06 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924037+0000", 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6080000000000001e-06 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928178+0000", 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041410570000000001 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928226+0000", 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8492000000000003e-05 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928322+0000", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5940000000000006e-05 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929823+0000", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015006430000000001 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929898+0000", 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4920999999999996e-05 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929955+0000", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7355000000000002e-05 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930001+0000", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5645000000000001e-05 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930038+0000", 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6777999999999999e-05 2024-03-30T14:57:41.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.924045+0000", 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.43231773700001, 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006146711, 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 114 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924045+0000", 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924045+0000", 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924046+0000", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2600000000000002e-07 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924049+0000", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3359999999999999e-06 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924055+0000", 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.417e-06 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924060+0000", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5269999999999998e-06 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928738+0000", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046781890000000001 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928813+0000", 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4574999999999997e-05 2024-03-30T14:57:41.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.928922+0000", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109116 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929829+0000", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00090671399999999998 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930094+0000", 2024-03-30T14:57:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000265087 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930126+0000", 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2209999999999998e-05 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930157+0000", 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1486999999999999e-05 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930192+0000", 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4427000000000001e-05 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.924070+0000", 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.43229332800001, 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062777980000000002, 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 115 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924070+0000", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924070+0000", 2024-03-30T14:57:41.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924070+0000", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7800000000000001e-07 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924127+0000", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6419999999999999e-05 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924136+0000", 2024-03-30T14:57:41.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5489999999999995e-06 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924140+0000", 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3440000000000003e-06 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929162+0000", 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050217229999999996 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929226+0000", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3620000000000004e-05 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.929360+0000", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134392 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930049+0000", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00068899 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.401 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930243+0000", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000193637 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930287+0000", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4422000000000003e-05 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930314+0000", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6749000000000001e-05 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930347+0000", 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3374000000000003e-05 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.924147+0000", 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.43221560400002, 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007444633, 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.403 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 116 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924147+0000", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924147+0000", 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924148+0000", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2399999999999998e-07 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924150+0000", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5359999999999999e-06 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924156+0000", 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.485e-06 2024-03-30T14:57:41.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924161+0000", 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8709999999999996e-06 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930392+0000", 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062312620000000004 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.406 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930478+0000", 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5578e-05 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930613+0000", 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000135203 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931331+0000", 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071871299999999995 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931367+0000", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5460000000000003e-05 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931495+0000", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012766399999999999 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931564+0000", 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8940000000000001e-05 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931592+0000", 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8396999999999999e-05 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:20.924167+0000", 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 320.43219587200002, 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076342440000000001, 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 117 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924167+0000", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.410 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924167+0000", 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924168+0000", 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7700000000000004e-07 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924173+0000", 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2870000000000004e-06 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.411 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924180+0000", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0539999999999997e-06 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.924186+0000", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.57e-06 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930880+0000", 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066948980000000003 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.930952+0000", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1903000000000006e-05 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931032+0000", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9993000000000005e-05 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931605+0000", 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057246700000000001 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931628+0000", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3589999999999999e-05 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931745+0000", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116311 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931778+0000", 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2885000000000001e-05 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.931801+0000", 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3708999999999999e-05 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.032735+0000", 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32362823, 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054629140000000001, 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.415 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 135 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032735+0000", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032735+0000", 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032736+0000", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5300000000000003e-07 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032746+0000", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0944000000000001e-05 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032756+0000", 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1889999999999996e-06 2024-03-30T14:57:41.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032765+0000", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9570000000000008e-06 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.036547+0000", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037820430000000001 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.036610+0000", 2024-03-30T14:57:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3295999999999998e-05 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.036730+0000", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119901 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.037616+0000", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00088656900000000003 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038061+0000", 2024-03-30T14:57:41.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044427300000000002 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038128+0000", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7460999999999997e-05 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038158+0000", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9920000000000002e-05 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038198+0000", 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9508000000000002e-05 2024-03-30T14:57:41.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.032776+0000", 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32358673700003, 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061747410000000001, 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 136 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032776+0000", 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032776+0000", 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032777+0000", 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0240000000000001e-06 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032784+0000", 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4790000000000001e-06 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032792+0000", 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8599999999999993e-06 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:41.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032798+0000", 2024-03-30T14:57:41.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1e-06 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.037122+0000", 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043247490000000001 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.037189+0000", 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6995999999999993e-05 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.037317+0000", 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000127138 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038023+0000", 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070658600000000004 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038082+0000", 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.906e-05 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038859+0000", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00077719799999999999 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038910+0000", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0890999999999998e-05 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038951+0000", 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.066e-05 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.032808+0000", 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32355485900001, 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075108249999999996, 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 137 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032808+0000", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032808+0000", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032809+0000", 2024-03-30T14:57:41.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3600000000000003e-07 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032870+0000", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0980999999999998e-05 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032878+0000", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1480000000000006e-06 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032884+0000", 2024-03-30T14:57:41.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2539999999999998e-06 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038243+0000", 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053591389999999997 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038321+0000", 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7867000000000005e-05 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.038438+0000", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116994 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040113+0000", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00167531 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040132+0000", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8176999999999999e-05 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.433 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040198+0000", 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6700999999999997e-05 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040270+0000", 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1732999999999999e-05 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040319+0000", 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8884999999999997e-05 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.032893+0000", 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32346944199998, 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075932700000000001, 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 138 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032893+0000", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032893+0000", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032894+0000", 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4799999999999997e-07 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032901+0000", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9920000000000001e-06 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032915+0000", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3427e-05 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032921+0000", 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2979999999999997e-06 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039000+0000", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006078717 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039075+0000", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5586000000000004e-05 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039202+0000", 2024-03-30T14:57:41.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012628800000000001 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040378+0000", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001176863 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040396+0000", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7793999999999999e-05 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040432+0000", 2024-03-30T14:57:41.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6233999999999999e-05 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040462+0000", 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9648000000000001e-05 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040487+0000", 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4675e-05 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.032928+0000", 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32343445399999, 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076952549999999998, 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 139 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032928+0000", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032928+0000", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032929+0000", 2024-03-30T14:57:41.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7599999999999997e-07 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032934+0000", 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8609999999999997e-06 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032941+0000", 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9009999999999997e-06 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.032946+0000", 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5539999999999997e-06 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039512+0000", 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065658519999999996 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039596+0000", 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3529000000000006e-05 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.039730+0000", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134413 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040530+0000", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00079966 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040542+0000", 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2587e-05 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040577+0000", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4857000000000002e-05 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040604+0000", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6236000000000001e-05 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040624+0000", 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0129e-05 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.033163+0000", 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32319993499999, 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0088615320000000001, 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033163+0000", 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033163+0000", 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033164+0000", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.234e-06 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033217+0000", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2972e-05 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033227+0000", 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0033e-05 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033234+0000", 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2609999999999996e-06 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040655+0000", 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074204450000000003 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040702+0000", 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6684999999999998e-05 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.040776+0000", 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4066999999999994e-05 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041575+0000", 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00079912799999999997 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041901+0000", 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032599199999999999 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041945+0000", 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4628e-05 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041993+0000", 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7982000000000002e-05 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.042025+0000", 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1105000000000001e-05 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.033246+0000", 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 318.32311728000002, 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089173250000000003, 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 141 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033246+0000", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033246+0000", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033246+0000", 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4000000000000001e-07 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033250+0000", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8879999999999999e-06 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033256+0000", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7320000000000003e-06 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.033261+0000", 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0690000000000001e-06 2024-03-30T14:57:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041133+0000", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078719610000000002 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041194+0000", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0690000000000003e-05 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041324+0000", 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012987200000000001 2024-03-30T14:57:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.041712+0000", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000388899 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.042063+0000", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035092599999999999 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.042111+0000", 2024-03-30T14:57:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7420999999999999e-05 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.042138+0000", 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7645999999999999e-05 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.042163+0000", 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4481000000000002e-05 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:23.581224+0000", 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.77513897, 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060216180000000003, 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 159 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581224+0000", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581224+0000", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581224+0000", 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5300000000000004e-07 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581227+0000", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9900000000000002e-06 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581232+0000", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6999999999999999e-06 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.581237+0000", 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4120000000000003e-06 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.586001+0000", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047645170000000002 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.586063+0000", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2050000000000004e-05 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.586200+0000", 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000136897 2024-03-30T14:57:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.586732+0000", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00053214699999999996 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.587021+0000", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000288953 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.587152+0000", 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013057499999999999 2024-03-30T14:57:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.587211+0000", 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9070999999999997e-05 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:23.587246+0000", 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4752999999999997e-05 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.465 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34346.0:4 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 184.024615066, 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14333030899999999, 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34346", 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4086439972", 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331750+0000", 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8610000000000001e-06 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331757+0000", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2699999999999999e-06 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331776+0000", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8924e-05 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331795+0000", 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8972999999999999e-05 2024-03-30T14:57:41.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331857+0000", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1649999999999994e-05 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331866+0000", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5149999999999995e-06 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474614+0000", 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14274793599999999 2024-03-30T14:57:41.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474650+0000", 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5559999999999998e-05 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475078+0000", 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042862 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34352.0:2 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 183.965235533, 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084430388999999995, 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34352", 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3269533966", 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391129+0000", 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.773e-06 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391135+0000", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9989999999999997e-06 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391143+0000", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1689999999999992e-06 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391151+0000", 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7300000000000005e-06 2024-03-30T14:57:41.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391213+0000", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1749999999999997e-05 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391220+0000", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2359999999999998e-06 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475387+0000", 2024-03-30T14:57:41.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084166640000000001 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475417+0000", 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9899e-05 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475558+0000", 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014119299999999999 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:39.322822+0000", 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 182.03354107499999, 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016139978999999999, 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322822+0000", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322822+0000", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322825+0000", 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.847e-06 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322837+0000", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1894e-05 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322865+0000", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8209e-05 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322878+0000", 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3123e-05 2024-03-30T14:57:41.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322969+0000", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1274999999999996e-05 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.322976+0000", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1590000000000004e-06 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.332080+0000", 2024-03-30T14:57:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091032969999999998 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.332090+0000", 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0115999999999999e-05 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.338833+0000", 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067434289999999996 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.338962+0000", 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012862999999999999 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24400.0:4 2.2 e71/67 2:5cea7035:::rbd_info:head v 71'1, mlcod=0'0)", 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:39.324218+0000", 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 182.03214449800001, 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070227299999999996, 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:57:41.481 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324218+0000", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324218+0000", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324220+0000", 2024-03-30T14:57:41.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1680000000000001e-06 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324230+0000", 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0540000000000001e-05 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324240+0000", 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9820000000000003e-06 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324250+0000", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9899999999999992e-06 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324316+0000", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5759000000000005e-05 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.324340+0000", 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3906000000000001e-05 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.331203+0000", 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068629920000000001 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.331241+0000", 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8393000000000003e-05 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14496.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-03-30T14:57:41.485 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:39.326830+0000", 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 182.02953260800001, 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012252806, 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14496", 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2375168254", 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:57:41.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326830+0000", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326830+0000", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326833+0000", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2859999999999998e-06 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:57:41.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326850+0000", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7020999999999998e-05 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326872+0000", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2022e-05 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.326884+0000", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2476999999999999e-05 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.332122+0000", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052375809999999998 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.332128+0000", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3810000000000004e-06 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.338995+0000", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068671130000000002 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:39.339083+0000", 2024-03-30T14:57:41.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7924999999999996e-05 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:57:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:57:41.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:41.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:41.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:41.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:41.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:41.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:41.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:41.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:41.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:41.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:42.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:42.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:42.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:42.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:42.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:42.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:42.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:42.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:42.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:42.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:42 smithi134 ceph-mon[33714]: pgmap v97: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:42.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:42 smithi028 ceph-mon[29876]: pgmap v97: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:42 smithi028 ceph-mon[41614]: pgmap v97: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:42.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:42.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:42.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:42.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:42.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:43.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:43.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:43.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:43.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:43.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:43.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:43.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:43.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T14:57:43.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:43.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:43.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:43.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:43.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:43.981 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T14:57:44.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:44.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:44.143 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T14:57:44.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:44.183 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T14:57:44.184 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T14:57:44.184 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T14:57:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:44.237 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T14:57:44.240 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-30T14:57:44.240 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T14:57:44.255 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-30T14:57:44.256 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-30T14:57:44.256 DEBUG:teuthology.orchestra.run.smithi028:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-30T14:57:44.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:44.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ parallel=1 2024-03-30T14:57:44.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' '' = --serial ']' 2024-03-30T14:57:44.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ crimson=0 2024-03-30T14:57:44.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' '' = --crimson ']' 2024-03-30T14:57:44.323 INFO:tasks.workunit.client.0.smithi028.stderr:+ color= 2024-03-30T14:57:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -t 1 ']' 2024-03-30T14:57:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-30T14:57:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-30T14:57:44.324 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:+ declare -A pids 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_aio 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_aio' 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_aio 2024-03-30T14:57:44.325 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:44.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:44.327 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_aio 2024-03-30T14:57:44.327 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92909 2024-03-30T14:57:44.327 INFO:tasks.workunit.client.0.smithi028.stdout:test api_aio on pid 92909 2024-03-30T14:57:44.330 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_aio on pid 92909' 2024-03-30T14:57:44.330 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92909 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_aio_pp 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_aio_pp' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_aio_pp 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.331 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.332 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_aio_pp 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92916 2024-03-30T14:57:44.333 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_aio_pp on pid 92916' 2024-03-30T14:57:44.334 INFO:tasks.workunit.client.0.smithi028.stdout:test api_aio_pp on pid 92916 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92916 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_io 2024-03-30T14:57:44.337 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_io' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_io 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.338 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.339 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.340 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.341 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.341 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.341 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.341 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_io 2024-03-30T14:57:44.341 INFO:tasks.workunit.client.0.smithi028.stdout:test api_io on pid 92930 2024-03-30T14:57:44.344 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92930 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_io on pid 92930' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92930 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_io_pp 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_io_pp' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_io_pp 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.345 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.346 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.347 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.348 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_io_pp 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92941 2024-03-30T14:57:44.349 INFO:tasks.workunit.client.0.smithi028.stdout:test api_io_pp on pid 92941 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_io_pp on pid 92941' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92941 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_asio 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_asio' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_asio 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.358 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.359 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.360 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.361 INFO:tasks.workunit.client.0.smithi028.stdout:test api_asio on pid 92956 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_asio 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92956 2024-03-30T14:57:44.367 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_asio on pid 92956' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92956 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_list 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_list' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_list 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.368 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.369 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.370 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.371 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.372 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.373 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92909/exe 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.374 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.375 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.375 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.375 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.375 INFO:tasks.workunit.client.0.smithi028.stdout:test api_list on pid 92975 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_list 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92975 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_list on pid 92975' 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92975 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_lock 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_lock' 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.381 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.382 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.383 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.384 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_aio: /' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.385 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.386 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_lock 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.387 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.388 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92916/exe 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_lock 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.389 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.390 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.390 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.390 INFO:tasks.workunit.client.0.smithi028.stdout:test api_lock on pid 92996 2024-03-30T14:57:44.391 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92996 2024-03-30T14:57:44.392 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.393 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_lock on pid 92996' 2024-03-30T14:57:44.393 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=92996 2024-03-30T14:57:44.393 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.393 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.393 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_lock_pp 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.394 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_lock_pp' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.395 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.396 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_lock_pp 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.397 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.398 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.399 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.400 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.401 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92930/exe 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_lock_pp 2024-03-30T14:57:44.402 INFO:tasks.workunit.client.0.smithi028.stdout:test api_lock_pp on pid 93023 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93023 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_lock_pp on pid 93023' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93023 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_misc 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_misc' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.405 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_misc 2024-03-30T14:57:44.406 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.407 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.408 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_io.log 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.409 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.410 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.411 INFO:tasks.workunit.client.0.smithi028.stdout:test api_misc on pid 93038 2024-03-30T14:57:44.413 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_misc 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93038 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_misc on pid 93038' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93038 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_io: /' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_misc_pp 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_misc_pp' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.414 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.415 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.416 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92956/exe 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_misc_pp 2024-03-30T14:57:44.417 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.418 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.419 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.420 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_misc_pp 2024-03-30T14:57:44.421 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.421 INFO:tasks.workunit.client.0.smithi028.stdout:test api_misc_pp on pid 93054 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93054 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_misc_pp on pid 93054' 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93054 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.427 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.428 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.429 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.430 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.431 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_asio: /' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92941/exe 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.432 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.433 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_tier_pp 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_tier_pp' 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.434 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.435 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92975/exe 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.436 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.437 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.438 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/92996/exe 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.439 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.440 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-30T14:57:44.441 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_lock: /' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.442 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.443 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_tier_pp 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.444 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.445 INFO:tasks.workunit.client.0.smithi028.stdout:test api_tier_pp on pid 93103 2024-03-30T14:57:44.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_tier_pp 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93103 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_tier_pp on pid 93103' 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93103 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.451 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.452 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.453 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.454 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93023/exe 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93038/exe 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_list.log 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.455 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_list: /' 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.456 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_misc: /' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_pool 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_pool' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.457 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.458 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.459 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_pool 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_pool 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-30T14:57:44.460 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93152 2024-03-30T14:57:44.461 INFO:tasks.workunit.client.0.smithi028.stdout:test api_pool on pid 93152 2024-03-30T14:57:44.463 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_pool on pid 93152' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93152 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_snapshots 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_snapshots' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.464 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_snapshots 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.467 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.468 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.469 INFO:tasks.workunit.client.0.smithi028.stdout:test api_snapshots on pid 93200 2024-03-30T14:57:44.472 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.472 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_snapshots 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93200 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_snapshots on pid 93200' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93200 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.473 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_snapshots_pp 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_snapshots_pp' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.474 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.475 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_snapshots_pp 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.479 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.480 INFO:tasks.workunit.client.0.smithi028.stdout:test api_snapshots_pp on pid 93222 2024-03-30T14:57:44.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_snapshots_pp 2024-03-30T14:57:44.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93222 2024-03-30T14:57:44.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_snapshots_pp on pid 93222' 2024-03-30T14:57:44.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93222 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_stat 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_stat' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.482 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.483 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_stat 2024-03-30T14:57:44.484 INFO:tasks.workunit.client.0.smithi028.stdout:test api_stat on pid 93239 2024-03-30T14:57:44.486 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_stat 2024-03-30T14:57:44.486 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93239 2024-03-30T14:57:44.486 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_stat on pid 93239' 2024-03-30T14:57:44.486 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93239 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93054/exe 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_stat_pp 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_stat_pp' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93103/exe 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.487 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.488 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.489 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.490 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.491 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_stat_pp 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.492 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.493 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-30T14:57:44.493 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-30T14:57:44.493 INFO:tasks.workunit.client.0.smithi028.stdout:test api_stat_pp on pid 93273 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_stat_pp 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93273 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_stat_pp on pid 93273' 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93273 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.496 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.497 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.498 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_watch_notify 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_watch_notify' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.499 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.500 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.501 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_watch_notify 2024-03-30T14:57:44.502 INFO:tasks.workunit.client.0.smithi028.stdout:test api_watch_notify on pid 93302 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.505 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_watch_notify 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93302 2024-03-30T14:57:44.506 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_watch_notify on pid 93302' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93302 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_watch_notify_pp 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_watch_notify_pp' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.507 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.508 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_watch_notify_pp 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.509 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.510 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.511 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.511 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.511 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_watch_notify_pp 2024-03-30T14:57:44.511 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93318 2024-03-30T14:57:44.511 INFO:tasks.workunit.client.0.smithi028.stdout:test api_watch_notify_pp on pid 93318 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_watch_notify_pp on pid 93318' 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93318 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_cmd 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_cmd' 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.517 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.518 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93152/exe 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.519 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.520 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_cmd 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.521 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.522 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.523 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.524 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.525 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.526 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.527 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.528 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.528 INFO:tasks.workunit.client.0.smithi028.stdout:test api_cmd on pid 93355 2024-03-30T14:57:44.531 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_cmd 2024-03-30T14:57:44.531 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.531 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93355 2024-03-30T14:57:44.531 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_cmd on pid 93355' 2024-03-30T14:57:44.531 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93355 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_pool: /' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_cmd_pp 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_cmd_pp' 2024-03-30T14:57:44.532 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-30T14:57:44.533 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.534 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.535 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.536 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.537 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93273/exe 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.538 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.539 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-30T14:57:44.540 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_cmd_pp 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93239/exe 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.541 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.542 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.543 INFO:tasks.workunit.client.0.smithi028.stdout:test api_cmd_pp on pid 93401 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_cmd_pp 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93401 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_cmd_pp on pid 93401' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93401 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_service 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_service' 2024-03-30T14:57:44.549 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.550 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.551 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_stat: /' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93302/exe 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.553 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.554 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.555 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.556 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.557 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.558 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_service 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93200/exe 2024-03-30T14:57:44.559 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.560 INFO:tasks.workunit.client.0.smithi028.stdout:test api_service on pid 93439 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_service 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93439 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_service on pid 93439' 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93439 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.561 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.562 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.563 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-30T14:57:44.564 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-30T14:57:44.571 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.572 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.573 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.574 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.575 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.576 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_service_pp 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.577 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_service_pp' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.578 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93222/exe 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.579 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.580 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_service_pp 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.581 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.582 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.583 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.584 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93318/exe 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.585 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.586 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.587 INFO:tasks.workunit.client.0.smithi028.stdout:test api_service_pp on pid 93555 2024-03-30T14:57:44.591 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_service_pp 2024-03-30T14:57:44.591 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93555 2024-03-30T14:57:44.591 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_service_pp on pid 93555' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93555 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_c_write_operations 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_c_write_operations' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.592 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.593 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93401/exe 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_c_write_operations 2024-03-30T14:57:44.594 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.595 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.596 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.596 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.596 INFO:tasks.workunit.client.0.smithi028.stdout:test api_c_write_operations on pid 93578 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_c_write_operations 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93578 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_c_write_operations on pid 93578' 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93578 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.602 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-30T14:57:44.603 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_c_read_operations 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_c_read_operations' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.604 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.605 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.606 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.607 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.608 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_c_read_operations 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.609 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.610 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.611 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.612 INFO:tasks.workunit.client.0.smithi028.stdout:test api_c_read_operations on pid 93636 2024-03-30T14:57:44.614 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_c_read_operations 2024-03-30T14:57:44.614 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93636 2024-03-30T14:57:44.614 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_c_read_operations on pid 93636' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93636 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.615 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93355/exe 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s api_cls_remote_reads 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' api_cls_remote_reads' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo api_cls_remote_reads 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.616 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.617 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.618 INFO:tasks.workunit.client.0.smithi028.stdout:test api_cls_remote_reads on pid 93650 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=api_cls_remote_reads 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93650 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test api_cls_remote_reads on pid 93650' 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93650 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.626 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.627 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_cmd: /' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s list_parallel 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' list_parallel' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.628 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.629 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.630 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.631 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.632 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.633 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.634 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.635 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.636 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo list_parallel 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.637 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.638 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.639 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93555/exe 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.640 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.641 INFO:tasks.workunit.client.0.smithi028.stdout:test list_parallel on pid 93692 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=list_parallel 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93692 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test list_parallel on pid 93692' 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93692 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.644 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.645 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s open_pools_parallel 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.646 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' open_pools_parallel' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.647 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo open_pools_parallel 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93439/exe 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.648 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.649 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.650 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.651 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.651 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.651 INFO:tasks.workunit.client.0.smithi028.stdout:test open_pools_parallel on pid 93727 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=open_pools_parallel 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93727 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test open_pools_parallel on pid 93727' 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93727 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.654 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_service: /' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.655 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s delete_pools_parallel 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' delete_pools_parallel' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.656 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.657 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo delete_pools_parallel 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.658 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_service.log 2024-03-30T14:57:44.659 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.659 INFO:tasks.workunit.client.0.smithi028.stdout:test delete_pools_parallel on pid 93744 2024-03-30T14:57:44.659 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93636/exe 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=delete_pools_parallel 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93744 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test delete_pools_parallel on pid 93744' 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93744 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.660 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93578/exe 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.661 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.662 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.663 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s cls 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' cls' 2024-03-30T14:57:44.664 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.665 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:44.668 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.668 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.668 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.668 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.668 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.669 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.670 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93650/exe 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.671 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.672 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.673 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.674 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.675 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo cls 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.676 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.677 INFO:tasks.workunit.client.0.smithi028.stdout:test cls on pid 93802 2024-03-30T14:57:44.681 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=cls 2024-03-30T14:57:44.681 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93802 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test cls on pid 93802' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93802 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93692/exe 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s cmd 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.682 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' cmd' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.683 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ list_parallel: /' 2024-03-30T14:57:44.684 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.685 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.686 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo cmd 2024-03-30T14:57:44.687 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.688 INFO:tasks.workunit.client.0.smithi028.stdout:test cmd on pid 93853 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=cmd 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93853 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test cmd on pid 93853' 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93853 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.693 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s handler_error 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.694 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.695 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.696 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.697 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' handler_error' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.698 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.699 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.700 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.701 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo handler_error 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.702 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.703 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.704 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=handler_error 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.705 INFO:tasks.workunit.client.0.smithi028.stdout:test handler_error on pid 93907 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93907 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test handler_error on pid 93907' 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=93907 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.709 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s io 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' io' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.710 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo io 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.711 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93802/exe 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_cls 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ cls: /' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93744/exe 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.715 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.717 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.718 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_cls.log 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.719 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93727/exe 2024-03-30T14:57:44.720 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.721 INFO:tasks.workunit.client.0.smithi028.stdout:test io on pid 94018 2024-03-30T14:57:44.724 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=io 2024-03-30T14:57:44.724 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94018 2024-03-30T14:57:44.724 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test io on pid 94018' 2024-03-30T14:57:44.724 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94018 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.725 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.726 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.727 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.728 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s ec_io 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.729 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' ec_io' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-30T14:57:44.730 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo ec_io 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.731 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.732 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.733 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93853/exe 2024-03-30T14:57:44.734 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/93907/exe 2024-03-30T14:57:44.735 INFO:tasks.workunit.client.0.smithi028.stdout:test ec_io on pid 94059 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=ec_io 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94059 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test ec_io on pid 94059' 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94059 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.738 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.739 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.740 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_cmd 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s list 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ cmd: /' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' list' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.741 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.742 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ handler_error: /' 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_handler_error 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo list 2024-03-30T14:57:44.743 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.744 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.744 INFO:tasks.workunit.client.0.smithi028.stdout:test list on pid 94090 2024-03-30T14:57:44.747 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=list 2024-03-30T14:57:44.747 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94090 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test list on pid 94090' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94090 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.748 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.749 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s ec_list 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.750 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' ec_list' 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo ec_list 2024-03-30T14:57:44.751 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.752 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.753 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94018/exe 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.754 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=ec_list 2024-03-30T14:57:44.755 INFO:tasks.workunit.client.0.smithi028.stdout:test ec_list on pid 94125 2024-03-30T14:57:44.757 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94125 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test ec_list on pid 94125' 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94125 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.758 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s misc 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' misc' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_io 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ io: /' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_io.log 2024-03-30T14:57:44.759 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo misc 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.760 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.761 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=misc 2024-03-30T14:57:44.762 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94159 2024-03-30T14:57:44.763 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test misc on pid 94159' 2024-03-30T14:57:44.764 INFO:tasks.workunit.client.0.smithi028.stdout:test misc on pid 94159 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94159 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s pool 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' pool' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.767 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo pool 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.768 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.769 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.770 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.771 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.772 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.773 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.774 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.774 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94090/exe 2024-03-30T14:57:44.774 INFO:tasks.workunit.client.0.smithi028.stdout:test pool on pid 94185 2024-03-30T14:57:44.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.778 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=pool 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94185 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test pool on pid 94185' 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94185 2024-03-30T14:57:44.779 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.780 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s read_operations 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.781 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.782 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94059/exe 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' read_operations' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_list.log 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ list: /' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_list 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo read_operations 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.783 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.784 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.785 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_ec_io 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.786 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ ec_io: /' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.787 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=read_operations 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94219 2024-03-30T14:57:44.788 INFO:tasks.workunit.client.0.smithi028.stdout:test read_operations on pid 94219 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test read_operations on pid 94219' 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94219 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.793 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s snapshots 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' snapshots' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.794 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.795 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo snapshots 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.796 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.797 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94125/exe 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.798 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=snapshots 2024-03-30T14:57:44.799 INFO:tasks.workunit.client.0.smithi028.stdout:test snapshots on pid 94232 2024-03-30T14:57:44.801 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94232 2024-03-30T14:57:44.801 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test snapshots on pid 94232' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94232 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.802 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.805 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s watch_notify 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' watch_notify' 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-30T14:57:44.806 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ ec_list: /' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_ec_list 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo watch_notify 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.807 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.808 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.809 INFO:tasks.workunit.client.0.smithi028.stdout:test watch_notify on pid 94246 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=watch_notify 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94246 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test watch_notify on pid 94246' 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94246 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-30T14:57:44.812 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ printf %25s write_operations 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.813 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+ r=' write_operations' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.814 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.815 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ awk '{print $1}' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.816 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.817 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.818 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.819 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ echo write_operations 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.820 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.821 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.821 INFO:tasks.workunit.client.0.smithi028.stdout:test write_operations on pid 94268 2024-03-30T14:57:44.833 INFO:tasks.workunit.client.0.smithi028.stderr:+ ff=write_operations 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94268 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ echo 'test write_operations on pid 94268' 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ pids[$f]=94268 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ ret=0 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93103 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93103 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.834 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+ . /etc/bashrc 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -z '' ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ BASHRCSOURCED=Y 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ shopt -q login_shell 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.835 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94159/exe 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.836 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.837 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.838 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.839 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:+++ umask 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.840 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ SHELL=/bin/bash 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94185/exe 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T14:57:44.841 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.842 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_misc 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ misc: /' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.843 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.844 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.845 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_pool 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_pool.log 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_misc.log 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.846 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94219/exe 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T14:57:44.847 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:+++ return 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export S_COLORS=auto 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:+++ S_COLORS=auto 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.848 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.849 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.850 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_read_operations 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ pool: /' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ read_operations: /' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94232/exe 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.851 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.852 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.853 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ snapshots: /' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_snapshots 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T14:57:44.854 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T14:57:44.855 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/locale 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.856 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94246/exe 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T14:57:44.857 INFO:tasks.workunit.client.0.smithi028.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:++++ LANG=en_US.UTF-8 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ LANG=en_US.UTF-8 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ unset LANG_backup config 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n '' ']' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' dumb = linux ']' 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.858 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/less.sh 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ '[' '' ']' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.859 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+++++ readlink /proc/94268/exe 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_watch_notify 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ watch_notify: /' 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:++++ basename /usr/bin/bash 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_declare='declare -f' 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+++ which_opt=-f 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export which_declare 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+++ export -f which 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset i 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:++ unset -f pathmunge 2024-03-30T14:57:44.860 INFO:tasks.workunit.client.0.smithi028.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ export PATH 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ unset rc 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ ceph_test_neorados_write_operations 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-30T14:57:44.861 INFO:tasks.workunit.client.0.smithi028.stderr:+ sed 's/^/ write_operations: /' 2024-03-30T14:57:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:44.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:44.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:44.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:44.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:45.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:45.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:45.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:45.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:45 smithi028 ceph-mon[29876]: pgmap v98: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:45.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:45 smithi028 ceph-mon[41614]: pgmap v98: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:45.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:45 smithi134 ceph-mon[33714]: pgmap v98: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:45.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:45.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:45.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:45.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:45.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:45.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:45.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:46.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:46.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:46.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:46.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:46.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:46.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:46.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1296722067' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/140818455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[29876]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1296722067' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/140818455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:46.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:46 smithi028 ceph-mon[41614]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:57:46.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:46.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:46.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1296722067' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/140818455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:46 smithi134 ceph-mon[33714]: osdmap e109: 8 total, 8 up, 8 in 2024-03-30T14:57:46.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:46.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:46.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:46.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:46.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:47.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:47.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:47.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:47.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:47.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:47.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:47.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:47.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:47.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T14:57:47.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:57:47.438 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T14:57:47.438 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-03-30T14:57:47.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 3 0 2024-03-30T14:57:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 -- 172.21.15.28:0/3061013116 >> v1:172.21.15.28:6790/0 conn(0x7efd5010ea70 legacy=0x7efd50110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 -- 172.21.15.28:0/3061013116 shutdown_connections 2024-03-30T14:57:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 -- 172.21.15.28:0/3061013116 >> 172.21.15.28:0/3061013116 conn(0x7efd500fdaf0 msgr2=0x7efd500fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 -- 172.21.15.28:0/3061013116 shutdown_connections 2024-03-30T14:57:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 -- 172.21.15.28:0/3061013116 wait complete. 2024-03-30T14:57:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 librados: starting msgr at 2024-03-30T14:57:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.487+0000 7efd57da5700 1 librados: starting objecter 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 Processor -- start 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 -- start start 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 librados: setting wanted keys 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 librados: calling monclient init 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd50076210 con 0x7efd5010aed0 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd50076420 con 0x7efd5010ea70 2024-03-30T14:57:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd57da5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efd501aba40 con 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd55b41700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7efd50102240 0x7efd50074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:35916/0 (socket says 172.21.15.28:35916) 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd55b41700 1 -- 172.21.15.28:0/2222633903 learned_addr learned my addr 172.21.15.28:0/2222633903 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd46ffd700 10 client.?.objecter ms_handle_connect 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.488+0000 7efd46ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504797867 0 0) 0x7efd501aba40 con 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd3c0035b0 con 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 10 client.?.objecter ms_handle_connect 0x7efd5010aed0 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 10 client.?.objecter ms_handle_connect 0x7efd5010ea70 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3716370741 0 0) 0x7efd3c0035b0 con 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd501aba40 con 0x7efd50102240 2024-03-30T14:57:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efd4c0018b0 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579636709 0 0) 0x7efd50076420 con 0x7efd5010ea70 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd3c0035b0 con 0x7efd5010ea70 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1315116530 0 0) 0x7efd501aba40 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 >> v1:172.21.15.134:6789/0 conn(0x7efd5010ea70 legacy=0x7efd501a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 >> v1:172.21.15.28:6789/0 conn(0x7efd5010aed0 legacy=0x7efd50074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd501acc60 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd501abc70 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd57da5700 10 client.35060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd501ac130 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.489+0000 7efd57da5700 1 librados: init done 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.490+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7efd4c003bc0 con 0x7efd50102240 2024-03-30T14:57:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.490+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efd4c005d30 con 0x7efd50102240 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd34005120 con 0x7efd50102240 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7efd4c003720 con 0x7efd50102240 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(110..110 src has 53..110) ==== 8833+0+0 (unknown 1277190360 0 0) 0x7efd4c089b40 con 0x7efd50102240 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd46ffd700 10 client.35060.objecter ms_dispatch 0x7efd50001550 osd_map(110..110 src has 53..110) 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd46ffd700 3 client.35060.objecter handle_osd_map got epochs [110,110] > 0 2024-03-30T14:57:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.491+0000 7efd46ffd700 3 client.35060.objecter handle_osd_map decoding full epoch 110 2024-03-30T14:57:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.492+0000 7efd46ffd700 20 client.35060.objecter dump_active .. 0 homeless 2024-03-30T14:57:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.492+0000 7efd46ffd700 10 client.35060.objecter ms_handle_connect 0x7efd3c070370 2024-03-30T14:57:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.496+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efd4c056ab0 con 0x7efd50102240 2024-03-30T14:57:47.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:47.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:47.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:47.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7efd34005410 con 0x7efd50102240 2024-03-30T14:57:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.741+0000 7f9bd1051700 1 -- 172.21.15.28:0/2151653372 >> v1:172.21.15.28:6790/0 conn(0x7f9bcc100d80 legacy=0x7f9bcc101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.741+0000 7f9bd1051700 1 -- 172.21.15.28:0/2151653372 shutdown_connections 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.741+0000 7f9bd1051700 1 -- 172.21.15.28:0/2151653372 >> 172.21.15.28:0/2151653372 conn(0x7f9bcc0fc9e0 msgr2=0x7f9bcc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7f9bd1051700 1 -- 172.21.15.28:0/2151653372 shutdown_connections 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7f9bd1051700 1 -- 172.21.15.28:0/2151653372 wait complete. 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7f9bd1051700 1 librados: starting msgr at 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7f9bd1051700 1 librados: starting objecter 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.742+0000 7f9bd1051700 1 Processor -- start 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 -- start start 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 librados: setting wanted keys 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 librados: calling monclient init 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bcc10f8f0 con 0x7f9bcc100d80 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bcc1aa840 con 0x7f9bcc110c80 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bd1051700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9bcc1aba80 con 0x7f9bcc101960 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bca59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9bcc101960 0x7f9bcc10cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:35930/0 (socket says 172.21.15.28:35930) 2024-03-30T14:57:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bca59c700 1 -- 172.21.15.28:0/1864998835 learned_addr learned my addr 172.21.15.28:0/1864998835 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter ms_handle_connect 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter ms_handle_connect 0x7f9bcc100d80 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter ms_handle_connect 0x7f9bcc110c80 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366331606 0 0) 0x7f9bcc1aba80 con 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bb40035b0 con 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319264639 0 0) 0x7f9bcc1aa840 con 0x7f9bcc110c80 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bcc1aba80 con 0x7f9bcc110c80 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3263986832 0 0) 0x7f9bb40035b0 con 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bcc1aa840 con 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.743+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9bc00018b0 con 0x7f9bcc101960 2024-03-30T14:57:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4013252058 0 0) 0x7f9bcc1aba80 con 0x7f9bcc110c80 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bb40035b0 con 0x7f9bcc110c80 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9bc4001b70 con 0x7f9bcc110c80 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3091653642 0 0) 0x7f9bcc1aa840 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.134:6789/0 conn(0x7f9bcc110c80 legacy=0x7f9bcc1a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6789/0 conn(0x7f9bcc100d80 legacy=0x7f9bcc10c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc1accc0 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bd1051700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc1a9890 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bd1051700 10 client.35066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bd1051700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9bcc1a9d50 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bd1051700 1 librados: init done 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9bc0003b40 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.744+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9bc0005c20 con 0x7f9bcc101960 2024-03-30T14:57:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.745+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9bcc06a440 con 0x7f9bcc101960 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.745+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9bc001c480 con 0x7f9bcc101960 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(110..110 src has 53..110) ==== 8833+0+0 (unknown 1277190360 0 0) 0x7f9bc0089b90 con 0x7f9bcc101960 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 osd_map(110..110 src has 53..110) 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 3 client.35066.objecter handle_osd_map got epochs [110,110] > 0 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 3 client.35066.objecter handle_osd_map decoding full epoch 110 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 20 client.35066.objecter dump_active .. 0 homeless 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v110) ==== 54+0+16 (unknown 4161977875 0 2420000192) 0x7f9bc0089fd0 con 0x7f9bcc101960 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bb4070340 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 1 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 1 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 0 1 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 4 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 1 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc068b30 con 0x7f9bcc1ab1e0 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.746+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1ab1e0 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.747+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc068b30 con 0x7f9bcc1ab1e0 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.747+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.747+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 1 = system:0 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.747+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 4 2024-03-30T14:57:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.747+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 0 1 2024-03-30T14:57:47.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 2 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 2 2024-03-30T14:57:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc117080 osd=0 3 2024-03-30T14:57:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 0 2 2024-03-30T14:57:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 4 2024-03-30T14:57:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 2 2024-03-30T14:57:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.775+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc10dc70 con 0x7f9bcc1ab1e0 2024-03-30T14:57:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc10dc70 con 0x7f9bcc1ab1e0 2024-03-30T14:57:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 4 2024-03-30T14:57:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 0 2 2024-03-30T14:57:47.780 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:57:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 3 2024-03-30T14:57:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 1 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 4 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 3 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.777+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1ab810 con 0x7f9bcc1ac870 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1ac870 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1ab810 con 0x7f9bcc1ac870 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 3: 0 ) 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 3 = system:0 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 4 2024-03-30T14:57:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.778+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 1 3 2024-03-30T14:57:47.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 4 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 4 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc112120 osd=1 3 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 1 4 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 4 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 4 2024-03-30T14:57:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.805+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1aac60 con 0x7f9bcc1ac870 2024-03-30T14:57:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.807+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1aac60 con 0x7f9bcc1ac870 2024-03-30T14:57:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.807+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.807+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.810 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.807+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 4 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.807+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 1 4 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 5 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 5 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 2 5 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 4 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 5 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1b6fd0 con 0x7f9bcc063360 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.808+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc063360 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.809+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1b6fd0 con 0x7f9bcc063360 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.809+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 5: 0 ) 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.809+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 5 = system:0 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.809+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 4 2024-03-30T14:57:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.809+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 2 5 2024-03-30T14:57:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 6 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 6 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1ab060 osd=2 3 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 2 6 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 4 2024-03-30T14:57:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 6 2024-03-30T14:57:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.837+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc0629f0 con 0x7f9bcc063360 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc0629f0 con 0x7f9bcc063360 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 4 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 2 6 2024-03-30T14:57:47.841 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 7 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 7 2024-03-30T14:57:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 3 7 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 4 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 7 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.839+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1b7cd0 con 0x7f9bcc1b7180 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1b7180 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1b7cd0 con 0x7f9bcc1b7180 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 7: 0 ) 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 7 = system:0 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 4 2024-03-30T14:57:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.840+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 3 7 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 8 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 8 2024-03-30T14:57:47.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc062c50 osd=3 3 2024-03-30T14:57:47.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 3 8 2024-03-30T14:57:47.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 4 2024-03-30T14:57:47.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 8 2024-03-30T14:57:47.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.869+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1b8650 con 0x7f9bcc1b7180 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1b8650 con 0x7f9bcc1b7180 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 4 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 3 8 2024-03-30T14:57:47.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 9 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 9 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 4 9 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 4 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 9 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.871+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1c0720 con 0x7f9bcc1bcb60 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.872+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1bcb60 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.873+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1c0720 con 0x7f9bcc1bcb60 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.873+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 9: 0 ) 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.873+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 9 = system:0 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.873+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 4 2024-03-30T14:57:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.873+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 4 9 2024-03-30T14:57:47.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 10 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 10 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1b8650 osd=4 3 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 4 10 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 4 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 10 2024-03-30T14:57:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.890+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1c0900 con 0x7f9bcc1bcb60 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1c0900 con 0x7f9bcc1bcb60 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 4 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 4 10 2024-03-30T14:57:47.894 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 11 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 11 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 5 11 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 4 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 11 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1c1c10 con 0x7f9bcc1c1020 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.892+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1c1020 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.893+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1c1c10 con 0x7f9bcc1c1020 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.893+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 11: 0 ) 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.893+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 11 = system:0 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.893+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 4 2024-03-30T14:57:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.893+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 5 11 2024-03-30T14:57:47.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 12 2024-03-30T14:57:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 12 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c0900 osd=5 3 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 5 12 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 4 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 12 2024-03-30T14:57:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.910+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1c1c10 con 0x7f9bcc1c1020 2024-03-30T14:57:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1c1c10 con 0x7f9bcc1c1020 2024-03-30T14:57:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 4 2024-03-30T14:57:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 5 12 2024-03-30T14:57:47.916 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 13 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 13 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 6 13 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 4 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 13 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.912+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1c9150 con 0x7f9bcc1c5600 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1c5600 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1c9150 con 0x7f9bcc1c5600 2024-03-30T14:57:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 13: 0 ) 2024-03-30T14:57:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 13 = system:0 2024-03-30T14:57:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 4 2024-03-30T14:57:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.913+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 6 13 2024-03-30T14:57:47.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:47.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 14 2024-03-30T14:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 14 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c1c10 osd=6 3 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 6 14 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 4 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 14 2024-03-30T14:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.930+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1c9110 con 0x7f9bcc1c5600 2024-03-30T14:57:47.934 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1c9110 con 0x7f9bcc1c5600 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 4 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 6 14 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 15 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 15 2024-03-30T14:57:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 7 15 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 4 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 15 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.932+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9bcc1ca5e0 con 0x7f9bcc1c98a0 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 10 client.35066.objecter ms_handle_connect 0x7f9bcc1c98a0 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9bcc1ca5e0 con 0x7f9bcc1c98a0 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 15: 0 ) 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 15 = system:0 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 4 2024-03-30T14:57:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.933+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 7 15 2024-03-30T14:57:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 10 client.35066.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:57:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign -1 16 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 20 client.35066.objecter _get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_remove -1 16 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 20 client.35066.objecter get_session s=0x7f9bcc1c9110 osd=7 3 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 15 client.35066.objecter _session_command_op_assign 7 16 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 4 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 10 client.35066.objecter _send_command 16 2024-03-30T14:57:47.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.950+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9bcc1ca5a0 con 0x7f9bcc1c98a0 2024-03-30T14:57:47.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb3fff700 1 -- 172.21.15.28:0/1864998835 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9bcc1ca5a0 con 0x7f9bcc1c98a0 2024-03-30T14:57:47.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb3fff700 10 client.35066.objecter ms_dispatch 0x7f9bcc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb3fff700 10 client.35066.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb3fff700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 4 2024-03-30T14:57:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb3fff700 15 client.35066.objecter _session_command_op_remove 7 16 2024-03-30T14:57:47.955 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:57:47.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 librados: watch_flush enter 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 librados: watch_flush exit 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.0 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6801/3394323575 conn(0x7f9bcc1ab1e0 legacy=0x7f9bcc1abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc117080 osd=0 1 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.1 2024-03-30T14:57:47.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6805/1850313547 conn(0x7f9bcc1ac870 legacy=0x7f9bcc1b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc112120 osd=1 1 2024-03-30T14:57:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.2 2024-03-30T14:57:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6809/3451325043 conn(0x7f9bcc063360 legacy=0x7f9bcc1b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1ab060 osd=2 1 2024-03-30T14:57:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.3 2024-03-30T14:57:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6813/1506157250 conn(0x7f9bcc1b7180 legacy=0x7f9bcc1b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc062c50 osd=3 1 2024-03-30T14:57:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.4 2024-03-30T14:57:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.134:6800/3063775514 conn(0x7f9bcc1bcb60 legacy=0x7f9bcc1bef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bcb59e700 1 -- 172.21.15.28:0/1864998835 reap_dead start 2024-03-30T14:57:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1b8650 osd=4 1 2024-03-30T14:57:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.5 2024-03-30T14:57:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.134:6804/3668151179 conn(0x7f9bcc1c1020 legacy=0x7f9bcc1c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bcb59e700 1 -- 172.21.15.28:0/1864998835 reap_dead start 2024-03-30T14:57:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c0900 osd=5 1 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.6 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.134:6808/2135224194 conn(0x7f9bcc1c5600 legacy=0x7f9bcc1c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c1c10 osd=6 1 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter close_session for osd.7 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.134:6812/3758190298 conn(0x7f9bcc1c98a0 legacy=0x7f9bcc1c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter put_session s=0x7f9bcc1c9110 osd=7 1 2024-03-30T14:57:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 20 client.35066.objecter shutdown clearing up homeless session... 2024-03-30T14:57:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bcb59e700 1 -- 172.21.15.28:0/1864998835 reap_dead start 2024-03-30T14:57:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.952+0000 7f9bb1ffb700 10 client.35066.objecter successfully canceled tick 2024-03-30T14:57:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9bb4070340 legacy=0x7f9bb4072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> v1:172.21.15.28:6790/0 conn(0x7f9bcc101960 legacy=0x7f9bcc10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 shutdown_connections 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 >> 172.21.15.28:0/1864998835 conn(0x7f9bcc0fc9e0 msgr2=0x7f9bcc113b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 shutdown_connections 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 -- 172.21.15.28:0/1864998835 wait complete. 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:47.953+0000 7f9bb1ffb700 1 librados: shutdown 2024-03-30T14:57:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:57:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:57:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:57:48.027 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:48.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:48.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:48.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:48.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:48.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:48.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:48.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:48.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:48.661 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:48.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:48.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:48.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:48.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:48.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:48.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:48.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:48.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:48.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:48.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:48 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:48.929 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:48 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:48 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:48.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:48.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:48.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:49.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:49.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:49.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:49.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:49.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:49.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:49.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:49.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:49.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:49.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:49.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:57:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:57:49.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:49.884 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:49.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:49.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:50.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:50.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:50.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:50.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:50.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:50.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:50.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: pgmap v100: 196 pgs: 64 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4209260695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4253613591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2605619531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2222633903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1864998835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: pgmap v100: 196 pgs: 64 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4209260695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4253613591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2605619531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2222633903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1864998835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:50 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:50.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T14:57:50.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:57:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: pgmap v100: 196 pgs: 64 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:57:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4209260695' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4253613591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2605619531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2222633903' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1864998835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: osdmap e110: 8 total, 8 up, 8 in 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.34820 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:50 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:50.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:50.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:50.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:50.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:50.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:50.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:50.910 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:50.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:51.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:51.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:51.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:51.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:51.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:57:51.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:51.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:51.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:51.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:51.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:51.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:51.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.654+0000 7ff506ed3700 1 -- 172.21.15.28:0/2189343460 >> v1:172.21.15.28:6790/0 conn(0x7ff500101130 legacy=0x7ff500101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.654+0000 7ff506ed3700 1 -- 172.21.15.28:0/2189343460 shutdown_connections 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.654+0000 7ff506ed3700 1 -- 172.21.15.28:0/2189343460 >> 172.21.15.28:0/2189343460 conn(0x7ff5000fc9e0 msgr2=0x7ff5000fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.654+0000 7ff506ed3700 1 -- 172.21.15.28:0/2189343460 shutdown_connections 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.654+0000 7ff506ed3700 1 -- 172.21.15.28:0/2189343460 wait complete. 2024-03-30T14:57:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 librados: starting msgr at 2024-03-30T14:57:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 librados: starting objecter 2024-03-30T14:57:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 Processor -- start 2024-03-30T14:57:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 -- start start 2024-03-30T14:57:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 librados: setting wanted keys 2024-03-30T14:57:51.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.655+0000 7ff506ed3700 1 librados: calling monclient init 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff506ed3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff500106920 con 0x7ff50010d960 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff504c6f700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff500101130 0x7ff5001046f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57730/0 (socket says 172.21.15.28:57730) 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff504c6f700 1 -- 172.21.15.28:0/2003249246 learned_addr learned my addr 172.21.15.28:0/2003249246 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5001a8540 con 0x7ff500109dc0 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5001a9720 con 0x7ff500101130 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter ms_handle_connect 0x7ff500101130 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896439743 0 0) 0x7ff5001a9720 con 0x7ff500101130 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4f00035b0 con 0x7ff500101130 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter ms_handle_connect 0x7ff500109dc0 2024-03-30T14:57:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter ms_handle_connect 0x7ff50010d960 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906040721 0 0) 0x7ff5001a8540 con 0x7ff500109dc0 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5001a9720 con 0x7ff500109dc0 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3809233116 0 0) 0x7ff4f00035b0 con 0x7ff500101130 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5001a8540 con 0x7ff500101130 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.656+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4f40015d0 con 0x7ff500101130 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2874751605 0 0) 0x7ff5001a9720 con 0x7ff500109dc0 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4f00035b0 con 0x7ff500109dc0 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4ec001870 con 0x7ff500109dc0 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3203548893 0 0) 0x7ff5001a8540 con 0x7ff500101130 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 >> v1:172.21.15.134:6789/0 conn(0x7ff500109dc0 legacy=0x7ff500104e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 >> v1:172.21.15.28:6789/0 conn(0x7ff50010d960 legacy=0x7ff5001a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5001aa900 con 0x7ff500101130 2024-03-30T14:57:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5001a8770 con 0x7ff500101130 2024-03-30T14:57:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff506ed3700 10 client.35189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5001a8c70 con 0x7ff500101130 2024-03-30T14:57:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff506ed3700 1 librados: init done 2024-03-30T14:57:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff4f4003c10 con 0x7ff500101130 2024-03-30T14:57:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.657+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4f40050f0 con 0x7ff500101130 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.658+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e4005120 con 0x7ff500101130 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff4f4003ec0 con 0x7ff500101130 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(112..112 src has 53..112) ==== 14761+0+0 (unknown 1927098168 0 0) 0x7ff4f408b210 con 0x7ff500101130 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 10 client.35189.objecter ms_dispatch 0x7ff500001550 osd_map(112..112 src has 53..112) 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 3 client.35189.objecter handle_osd_map got epochs [112,112] > 0 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 3 client.35189.objecter handle_osd_map decoding full epoch 112 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.660+0000 7ff4fdffb700 20 client.35189.objecter dump_active .. 0 homeless 2024-03-30T14:57:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.661+0000 7ff4fdffb700 10 client.35189.objecter ms_handle_connect 0x7ff4f0070240 2024-03-30T14:57:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.667+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff4f4001190 con 0x7ff500101130 2024-03-30T14:57:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:51.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:51.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:51.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:51.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:51.914+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 0.0} v 0) -- 0x7ff4e4005c20 con 0x7ff500101130 2024-03-30T14:57:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:51.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:51.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:52.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:52.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:52.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:52.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:52.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:52.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:52.490+0000 7efd575a4700 10 client.35060.objecter tick 2024-03-30T14:57:52.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:52.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:52.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:52.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:52.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:52.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:53.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:53.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:53.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:53.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:53.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:53.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:53.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:53.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:53.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:53.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:53.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:53.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:53.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:53.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [----------] Global test environment set-up. 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5072 ms) 2024-03-30T14:57:53.882 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ OK ] LibRadosService.RegisterLate (20 ms) 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: cluster: 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: id: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: health: HEALTH_WARN 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: noscrub flag(s) set 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: services: 2024-03-30T14:57:53.883 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: flags noscrub 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: data: 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pgs: 131 active+clean 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 1 active+clean+scrubbing 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: io: 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-30T14:57:53.884 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: cluster: 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: id: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: health: HEALTH_WARN 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: noscrub flag(s) set 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2 pool(s) do not have an application enabled 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: services: 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: flags noscrub 2024-03-30T14:57:53.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: data: 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pools: 8 pools, 196 pgs 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pgs: 32.653% pgs unknown 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 131 active+clean 2024-03-30T14:57:53.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 64 unknown 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 1 active+clean+scrubbing 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: io: 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: cluster: 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: id: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:57:53.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: health: HEALTH_WARN 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: noscrub flag(s) set 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2 pool(s) do not have an application enabled 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: services: 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-03-30T14:57:53.888 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T14:57:53.889 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T14:57:53.889 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: flags noscrub 2024-03-30T14:57:53.889 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-30T14:57:53.889 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T14:57:53.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.957+0000 7efd46ffd700 1 -- 172.21.15.28:0/2222633903 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v113) ==== 97+0+0 (unknown 3801776364 0 0) 0x7efd4c05a500 con 0x7efd50102240 2024-03-30T14:57:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.961+0000 7efd57da5700 10 librados: watch_flush enter 2024-03-30T14:57:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.962+0000 7efd57da5700 10 librados: watch_flush exit 2024-03-30T14:57:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.962+0000 7efd57da5700 20 client.35060.objecter shutdown clearing up homeless session... 2024-03-30T14:57:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.962+0000 7efd57da5700 10 client.35060.objecter successfully canceled tick 2024-03-30T14:57:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.962+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 >> v1:172.21.15.28:6800/3011431093 conn(0x7efd3c070370 legacy=0x7efd3c072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.963+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 >> v1:172.21.15.28:6790/0 conn(0x7efd50102240 legacy=0x7efd50074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:53.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.964+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 shutdown_connections 2024-03-30T14:57:53.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.964+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 >> 172.21.15.28:0/2222633903 conn(0x7efd500fdaf0 msgr2=0x7efd501116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:53.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.964+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 shutdown_connections 2024-03-30T14:57:53.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.964+0000 7efd57da5700 1 -- 172.21.15.28:0/2222633903 wait complete. 2024-03-30T14:57:53.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:53.964+0000 7efd57da5700 1 librados: shutdown 2024-03-30T14:57:53.966 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T14:57:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:53.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:53.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:54.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.111 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: pgmap v102: 356 pgs: 224 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/917910517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2268236432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2848446493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/13297926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4105744775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1207320814' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: pgmap v104: 644 pgs: 149 creating+peering, 341 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:54.160 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1747038784' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: pgmap v102: 356 pgs: 224 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/917910517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2268236432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2848446493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/13297926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4105744775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1207320814' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: pgmap v104: 644 pgs: 149 creating+peering, 341 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1747038784' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:54.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[41614]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:54.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:53 smithi028 ceph-mon[29876]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:54.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: pgmap v102: 356 pgs: 224 unknown, 131 active+clean, 1 active+clean+scrubbing; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/276436981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi028-92979-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi028-94210-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/917910517' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2268236432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2848446493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/13297926' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4105744775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1207320814' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: pgmap v104: 644 pgs: 149 creating+peering, 341 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1747038784' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: osdmap e111: 8 total, 8 up, 8 in 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.35060 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.24823 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:54.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:53 smithi134 ceph-mon[33714]: from='client.24832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:54.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:54.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.611+0000 7f74932a9700 1 -- 172.21.15.28:0/1895956996 >> v1:172.21.15.28:6790/0 conn(0x7f748c102240 legacy=0x7f748c102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.611+0000 7f74932a9700 1 -- 172.21.15.28:0/1895956996 shutdown_connections 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.611+0000 7f74932a9700 1 -- 172.21.15.28:0/1895956996 >> 172.21.15.28:0/1895956996 conn(0x7f748c0fdaf0 msgr2=0x7f748c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.611+0000 7f74932a9700 1 -- 172.21.15.28:0/1895956996 shutdown_connections 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.611+0000 7f74932a9700 1 -- 172.21.15.28:0/1895956996 wait complete. 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 librados: starting msgr at 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 librados: starting objecter 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 Processor -- start 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 -- start start 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 librados: setting wanted keys 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 librados: calling monclient init 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f748c076170 con 0x7f748c10ea70 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f748c076380 con 0x7f748c102240 2024-03-30T14:57:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74932a9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f748c1aba40 con 0x7f748c10aed0 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f7490844700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f748c10aed0 0x7f748c074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57812/0 (socket says 172.21.15.28:57812) 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f7490844700 1 -- 172.21.15.28:0/2688719994 learned_addr learned my addr 172.21.15.28:0/2688719994 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter ms_handle_connect 0x7f748c10aed0 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter ms_handle_connect 0x7f748c10ea70 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter ms_handle_connect 0x7f748c102240 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570679706 0 0) 0x7f748c1aba40 con 0x7f748c10aed0 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74780035b0 con 0x7f748c10aed0 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973723781 0 0) 0x7f748c076380 con 0x7f748c102240 2024-03-30T14:57:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.612+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f748c1aba40 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2060215629 0 0) 0x7f748c1aba40 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f748c076380 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7488001430 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1663446972 0 0) 0x7f74780035b0 con 0x7f748c10aed0 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f748c1aba40 con 0x7f748c10aed0 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f747c0018f0 con 0x7f748c10aed0 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 760173336 0 0) 0x7f748c076380 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6790/0 conn(0x7f748c10aed0 legacy=0x7f748c074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6789/0 conn(0x7f748c10ea70 legacy=0x7f748c1a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748c1acc20 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f748c1abc70 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74932a9700 10 client.25258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f748c1ac130 con 0x7f748c102240 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74932a9700 1 librados: init done 2024-03-30T14:57:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7488003700 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.613+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7488004de0 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.614+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7470005100 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.614+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7488001b70 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(113..113 src has 53..113) ==== 20035+0+0 (unknown 1053744259 0 0) 0x7f748808b530 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 osd_map(113..113 src has 53..113) 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 3 client.25258.objecter handle_osd_map got epochs [113,113] > 0 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 3 client.25258.objecter handle_osd_map decoding full epoch 113 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 20 client.25258.objecter dump_active .. 0 homeless 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f74780702e0 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v113) ==== 54+0+16 (unknown 891576468 0 2420000192) 0x7f748808d960 con 0x7f748c102240 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 10 client.25258.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 1 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 1 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 0 1 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 4 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 10 client.25258.objecter _send_command 1 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.615+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7470004b50 con 0x7f7470004730 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f7470004730 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7470004b50 con 0x7f7470004730 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 1: 0 ) 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 10 client.25258.objecter _finish_command 1 = system:0 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 4 2024-03-30T14:57:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.616+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 0 1 2024-03-30T14:57:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 10 client.25258.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 2 2024-03-30T14:57:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 2 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f74700053e0 osd=0 3 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 0 2 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 4 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 10 client.25258.objecter _send_command 2 2024-03-30T14:57:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.634+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7470007b00 con 0x7f7470004730 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.635+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7470007b00 con 0x7f7470004730 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.635+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.635+0000 7f74827fc700 10 client.25258.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.635+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 4 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.635+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 0 2 2024-03-30T14:57:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 10 client.25258.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 3 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 3 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 1 3 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 4 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 10 client.25258.objecter _send_command 3 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7470002200 con 0x7f74700042f0 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.636+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f74700042f0 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.637+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7470002200 con 0x7f74700042f0 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.637+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 3: 0 ) 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.637+0000 7f74827fc700 10 client.25258.objecter _finish_command 3 = system:0 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.637+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 4 2024-03-30T14:57:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.637+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 1 3 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 10 client.25258.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 4 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 4 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470004180 osd=1 3 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 1 4 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 4 2024-03-30T14:57:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 10 client.25258.objecter _send_command 4 2024-03-30T14:57:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.664+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7470001600 con 0x7f74700042f0 2024-03-30T14:57:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.665+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7470001600 con 0x7f74700042f0 2024-03-30T14:57:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.665+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.665+0000 7f74827fc700 10 client.25258.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.665+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 4 2024-03-30T14:57:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.665+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 1 4 2024-03-30T14:57:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:57:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 10 client.25258.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 5 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 5 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 2 5 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 4 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 10 client.25258.objecter _send_command 5 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f74700111f0 con 0x7f747000d6e0 2024-03-30T14:57:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.666+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f747000d6e0 2024-03-30T14:57:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.667+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f74700111f0 con 0x7f747000d6e0 2024-03-30T14:57:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.667+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 5: 0 ) 2024-03-30T14:57:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.667+0000 7f74827fc700 10 client.25258.objecter _finish_command 5 = system:0 2024-03-30T14:57:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.667+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 4 2024-03-30T14:57:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.667+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 2 5 2024-03-30T14:57:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 10 client.25258.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 6 2024-03-30T14:57:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 6 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470008480 osd=2 3 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 2 6 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 4 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 10 client.25258.objecter _send_command 6 2024-03-30T14:57:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.684+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7470007fc0 con 0x7f747000d6e0 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7470007fc0 con 0x7f747000d6e0 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74827fc700 10 client.25258.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 4 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 2 6 2024-03-30T14:57:54.687 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 10 client.25258.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 7 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 7 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 3 7 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 4 2024-03-30T14:57:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 10 client.25258.objecter _send_command 7 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.686+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7470016030 con 0x7f7470012520 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f7470012520 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7470016030 con 0x7f7470012520 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 7: 0 ) 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 10 client.25258.objecter _finish_command 7 = system:0 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 4 2024-03-30T14:57:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.687+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 3 7 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 10 client.25258.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 8 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 8 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f7470008250 osd=3 3 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 3 8 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 4 2024-03-30T14:57:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 10 client.25258.objecter _send_command 8 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.705+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f747000ca60 con 0x7f7470012520 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.706+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f747000ca60 con 0x7f7470012520 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.706+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.706+0000 7f74827fc700 10 client.25258.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.706+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 4 2024-03-30T14:57:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.706+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 3 8 2024-03-30T14:57:54.708 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:57:54.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 10 client.25258.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 9 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 9 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 4 9 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 4 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 10 client.25258.objecter _send_command 9 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f747001aa90 con 0x7f7470012080 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.707+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f7470012080 2024-03-30T14:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.708+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f747001aa90 con 0x7f7470012080 2024-03-30T14:57:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.708+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 9: 0 ) 2024-03-30T14:57:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.708+0000 7f74827fc700 10 client.25258.objecter _finish_command 9 = system:0 2024-03-30T14:57:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.708+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 4 2024-03-30T14:57:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.708+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 4 9 2024-03-30T14:57:54.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 10 client.25258.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 10 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 10 2024-03-30T14:57:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747000c810 osd=4 3 2024-03-30T14:57:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 4 10 2024-03-30T14:57:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 4 2024-03-30T14:57:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 10 client.25258.objecter _send_command 10 2024-03-30T14:57:54.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.725+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f747001aa90 con 0x7f7470012080 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f747001aa90 con 0x7f7470012080 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74827fc700 10 client.25258.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 4 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 4 10 2024-03-30T14:57:54.728 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:57:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 10 client.25258.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 11 2024-03-30T14:57:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 11 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 5 11 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 4 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 10 client.25258.objecter _send_command 11 2024-03-30T14:57:54.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.727+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f747001edf0 con 0x7f747001b110 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f747001b110 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f747001edf0 con 0x7f747001b110 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 11: 0 ) 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 10 client.25258.objecter _finish_command 11 = system:0 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 4 2024-03-30T14:57:54.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.728+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 5 11 2024-03-30T14:57:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 10 client.25258.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 12 2024-03-30T14:57:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 12 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747001aa90 osd=5 3 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 5 12 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 4 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 10 client.25258.objecter _send_command 12 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.745+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f747001efd0 con 0x7f747001b110 2024-03-30T14:57:54.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f747001efd0 con 0x7f747001b110 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74827fc700 10 client.25258.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 4 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 5 12 2024-03-30T14:57:54.748 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:57:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 10 client.25258.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 13 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 13 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 6 13 2024-03-30T14:57:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 4 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 10 client.25258.objecter _send_command 13 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.747+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f74700202e0 con 0x7f747001f6f0 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f747001f6f0 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f74700202e0 con 0x7f747001f6f0 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 13: 0 ) 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 10 client.25258.objecter _finish_command 13 = system:0 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 4 2024-03-30T14:57:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.748+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 6 13 2024-03-30T14:57:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 10 client.25258.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 14 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 14 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f747001efd0 osd=6 3 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 6 14 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 4 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 10 client.25258.objecter _send_command 14 2024-03-30T14:57:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.766+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f74700202e0 con 0x7f747001f6f0 2024-03-30T14:57:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.767+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f74700202e0 con 0x7f747001f6f0 2024-03-30T14:57:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.767+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.767+0000 7f74827fc700 10 client.25258.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.767+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 4 2024-03-30T14:57:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.767+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 6 14 2024-03-30T14:57:54.769 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 10 client.25258.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 15 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 15 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 7 15 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 4 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 10 client.25258.objecter _send_command 15 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7470027a10 con 0x7f7470023cd0 2024-03-30T14:57:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.768+0000 7f74827fc700 10 client.25258.objecter ms_handle_connect 0x7f7470023cd0 2024-03-30T14:57:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.769+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7470027a10 con 0x7f7470023cd0 2024-03-30T14:57:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.769+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 15: 0 ) 2024-03-30T14:57:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.769+0000 7f74827fc700 10 client.25258.objecter _finish_command 15 = system:0 2024-03-30T14:57:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.769+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 4 2024-03-30T14:57:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.769+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 7 15 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 10 client.25258.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign -1 16 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 20 client.25258.objecter _get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 15 client.25258.objecter _session_command_op_remove -1 16 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 20 client.25258.objecter get_session s=0x7f74700202e0 osd=7 3 2024-03-30T14:57:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 15 client.25258.objecter _session_command_op_assign 7 16 2024-03-30T14:57:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 4 2024-03-30T14:57:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 10 client.25258.objecter _send_command 16 2024-03-30T14:57:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.786+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f747000cd50 con 0x7f7470023cd0 2024-03-30T14:57:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74827fc700 1 -- 172.21.15.28:0/2688719994 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f747000cd50 con 0x7f7470023cd0 2024-03-30T14:57:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74827fc700 10 client.25258.objecter ms_dispatch 0x7f748c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:57:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74827fc700 10 client.25258.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:57:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74827fc700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 4 2024-03-30T14:57:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74827fc700 15 client.25258.objecter _session_command_op_remove 7 16 2024-03-30T14:57:54.789 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 librados: watch_flush enter 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 librados: watch_flush exit 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.0 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6801/3394323575 conn(0x7f7470004730 legacy=0x7f7470002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700053e0 osd=0 1 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.1 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6805/1850313547 conn(0x7f74700042f0 legacy=0x7f747000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470004180 osd=1 1 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.2 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6809/3451325043 conn(0x7f747000d6e0 legacy=0x7f747000fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008480 osd=2 1 2024-03-30T14:57:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.3 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6813/1506157250 conn(0x7f7470012520 legacy=0x7f7470014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f7470008250 osd=3 1 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.4 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.134:6800/3063775514 conn(0x7f7470012080 legacy=0x7f7470019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747000c810 osd=4 1 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.5 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.134:6804/3668151179 conn(0x7f747001b110 legacy=0x7f747001d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001aa90 osd=5 1 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.6 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.134:6808/2135224194 conn(0x7f747001f6f0 legacy=0x7f747001fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f747001efd0 osd=6 1 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter close_session for osd.7 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.134:6812/3758190298 conn(0x7f7470023cd0 legacy=0x7f7470026120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter put_session s=0x7f74700202e0 osd=7 1 2024-03-30T14:57:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 20 client.25258.objecter shutdown clearing up homeless session... 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 10 client.25258.objecter successfully canceled tick 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f7491846700 1 -- 172.21.15.28:0/2688719994 reap_dead start 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f7491846700 1 -- 172.21.15.28:0/2688719994 reap_dead start 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f7491846700 1 -- 172.21.15.28:0/2688719994 reap_dead start 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f7491846700 1 -- 172.21.15.28:0/2688719994 reap_dead start 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f7491846700 1 -- 172.21.15.28:0/2688719994 reap_dead start 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.28:6800/3011431093 conn(0x7f74780702e0 legacy=0x7f74780727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> v1:172.21.15.134:6789/0 conn(0x7f748c102240 legacy=0x7f748c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 shutdown_connections 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.788+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 >> 172.21.15.28:0/2688719994 conn(0x7f748c0fdaf0 msgr2=0x7f748c1116c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.789+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 shutdown_connections 2024-03-30T14:57:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.789+0000 7f74932a9700 1 -- 172.21.15.28:0/2688719994 wait complete. 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.789+0000 7f74932a9700 1 librados: shutdown 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:57:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:57:54.822 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:57:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:57:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T14:57:54.822 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T14:57:54.822 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:57:54.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:54.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.959+0000 7ff4fdffb700 1 -- 172.21.15.28:0/2003249246 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]=0 set osd.3 primary-affinity to 0 (802) v114) ==== 129+0+0 (unknown 3815223863 0 0) 0x7ff4f4056a50 con 0x7ff500101130 2024-03-30T14:57:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.963+0000 7ff506ed3700 10 librados: watch_flush enter 2024-03-30T14:57:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.964+0000 7ff506ed3700 10 librados: watch_flush exit 2024-03-30T14:57:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.964+0000 7ff506ed3700 20 client.35189.objecter shutdown clearing up homeless session... 2024-03-30T14:57:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.964+0000 7ff506ed3700 10 client.35189.objecter successfully canceled tick 2024-03-30T14:57:54.967 INFO:tasks.workunit.client.0.smithi028.stdout: api_ser handler_error: Running main() from gmock_main.cc 2024-03-30T14:57:54.967 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [----------] Global test environment set-up. 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (10226 ms) 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [----------] 1 test from neocls_handler_error (10226 ms total) 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: 2024-03-30T14:57:54.968 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [----------] Global test environment tear-down 2024-03-30T14:57:54.969 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [==========] 1 test from 1 test suite ran. (10226 ms total) 2024-03-30T14:57:54.969 INFO:tasks.workunit.client.0.smithi028.stdout: handler_error: [ PASSED ] 1 test. 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.966+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff4f0070240 legacy=0x7ff4f0072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 >> v1:172.21.15.28:6790/0 conn(0x7ff500101130 legacy=0x7ff5001046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 shutdown_connections 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 >> 172.21.15.28:0/2003249246 conn(0x7ff5000fc9e0 msgr2=0x7ff5000ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 shutdown_connections 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 -- 172.21.15.28:0/2003249246 wait complete. 2024-03-30T14:57:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:57:54.967+0000 7ff506ed3700 1 librados: shutdown 2024-03-30T14:57:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 0 (802) 2024-03-30T14:57:54.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:55.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:55.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:55.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1522502118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1241392178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1333607810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1012171239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1422378701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: pgmap v106: 836 pgs: 149 creating+peering, 533 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1359122880' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2003249246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: pgmap v107: 836 pgs: 588 creating+peering, 81 unknown, 167 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 140 B/s wr, 1 op/s 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3785422690' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2019932805' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1135746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:57:55.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24907 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2688719994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:57:55.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1522502118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1241392178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1333607810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1012171239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1422378701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: pgmap v106: 836 pgs: 149 creating+peering, 533 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1359122880' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2003249246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: pgmap v107: 836 pgs: 588 creating+peering, 81 unknown, 167 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 140 B/s wr, 1 op/s 2024-03-30T14:57:55.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3785422690' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2019932805' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1135746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:57:55.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24907 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2688719994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:55.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [----------] Global test environment set-up. 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (7 ms) 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2 ms) 2024-03-30T14:57:55.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (5 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (5 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-30T14:57:55.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [----------] 12 tests from AsioRados (30 ms total) 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [----------] Global test environment tear-down 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (10846 ms total) 2024-03-30T14:57:55.263 INFO:tasks.workunit.client.0.smithi028.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-30T14:57:55.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:55.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:55.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:57:55.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:57:55.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi028-93525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.34820 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi028-93069-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1522502118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1241392178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1333607810' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1012171239' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1422378701' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: pgmap v106: 836 pgs: 149 creating+peering, 533 unknown, 154 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1359122880' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2003249246' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: pgmap v107: 836 pgs: 588 creating+peering, 81 unknown, 167 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 140 B/s wr, 1 op/s 2024-03-30T14:57:55.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3785422690' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: osdmap e112: 8 total, 8 up, 8 in 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.35189 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.35060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93654-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24823 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi028-92999-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi028-94235-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:55.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi028-93032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2019932805' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1135746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: osdmap e113: 8 total, 8 up, 8 in 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24892 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24907 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.24964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:55.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2688719994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:57:55.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:55.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:55.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:55.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:55.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T14:57:55.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:55.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:56.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:56.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:56.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:56.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:56.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:56.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:56.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:56.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:56.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/517891268' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[41614]: from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:57:56.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/517891268' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:56 smithi028 ceph-mon[29876]: from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi028-94210-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:56.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:56.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.35189 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi028-93253-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi028-93116-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi028-93086-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi028-93081-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24892 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi028-93390-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24907 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi028-93417-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24961 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi028-93750-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi028-93408-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi028-93441-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.24964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93580-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:56.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: osdmap e114: 8 total, 8 up, 8 in 2024-03-30T14:57:56.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/517891268' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:57:56.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:56 smithi134 ceph-mon[33714]: from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:56.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (74 ms) 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (59 ms) 2024-03-30T14:57:56.461 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (11750 ms) 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (11883 ms total) 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (11884 ms total) 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-30T14:57:56.462 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11815 ms) 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11816 ms total) 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11816 ms total) 2024-03-30T14:57:56.463 INFO:tasks.workunit.client.0.smithi028.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-30T14:57:56.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:56.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] Global test environment set-up. 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.ListObjects (1358 ms) 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (45 ms) 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo1 2024-03-30T14:57:56.634 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo2 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo3 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo4 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo5 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo1 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo6,foo7 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo7 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo6 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo4 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo5 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns2:foo7 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns2:foo6 2024-03-30T14:57:56.635 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo1 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo1 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo2 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo3 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (81 ms) 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 1 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 10 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 13 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 7 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 14 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 0 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 15 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 11 0 2024-03-30T14:57:56.636 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 5 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 8 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 6 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 3 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 4 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 12 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 9 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 2 0 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (94 ms) 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: x cursor=MIN 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=1 cursor=17:02547ec2:::1:head 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=10 cursor=17:52ea6a34:::10:head 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=13 cursor=17:566253c9:::13:head 2024-03-30T14:57:56.637 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=7 cursor=17:5c6b0b28:::7:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=14 cursor=17:62a1935d:::14:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=0 cursor=17:6cac518f:::0:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=15 cursor=17:863748b0:::15:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=11 cursor=17:89d3ae78:::11:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=5 cursor=17:b29083e3:::5:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=8 cursor=17:bd63b0f1:::8:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=6 cursor=17:c4fdafeb:::6:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=3 cursor=17:cfc208b3:::3:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=4 cursor=17:d83876eb:::4:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=12 cursor=17:de5d7c5f:::12:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=9 cursor=17:e960b815:::9:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > oid=2 cursor=17:f905c69b:::2:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=1 cursor=17:02547ec2:::1:head 2024-03-30T14:57:56.638 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:02547ec2:::1:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:02547ec2:::1:head -> 1 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=10 cursor=17:52ea6a34:::10:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:52ea6a34:::10:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:52ea6a34:::10:head -> 10 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=13 cursor=17:566253c9:::13:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:566253c9:::13:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:566253c9:::13:head -> 13 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=7 cursor=17:5c6b0b28:::7:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:5c6b0b28:::7:head 2024-03-30T14:57:56.639 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:5c6b0b28:::7:head -> 7 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=14 cursor=17:62a1935d:::14:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:62a1935d:::14:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:62a1935d:::14:head -> 14 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=0 cursor=17:6cac518f:::0:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:6cac518f:::0:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:6cac518f:::0:head -> 0 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=15 cursor=17:863748b0:::15:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:863748b0:::15:head 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:863748b0:::15:head -> 15 2024-03-30T14:57:56.709 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=11 cursor=17:89d3ae78:::11:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:89d3ae78:::11:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:89d3ae78:::11:head -> 11 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=5 cursor=17:b29083e3:::5:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:b29083e3:::5:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:b29083e3:::5:head -> 5 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=8 cursor=17:bd63b0f1:::8:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:bd63b0f1:::8:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:bd63b0f1:::8:head -> 8 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=6 cursor=17:c4fdafeb:::6:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:c4fdafeb:::6:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:c4fdafeb:::6:head -> 6 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=3 cursor=17:cfc208b3:::3:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:cfc208b3:::3:head 2024-03-30T14:57:56.710 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:cfc208b3:::3:head -> 3 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=4 cursor=17:d83876eb:::4:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:d83876eb:::4:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:d83876eb:::4:head -> 4 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=12 cursor=17:de5d7c5f:::12:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:de5d7c5f:::12:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:de5d7c5f:::12:head -> 12 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=9 cursor=17:e960b815:::9:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:e960b815:::9:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:e960b815:::9:head -> 9 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : oid=2 cursor=17:f905c69b:::2:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:f905c69b:::2:head 2024-03-30T14:57:56.711 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:f905c69b:::2:head -> 2 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:f905c69b:::2:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:f905c69b:::2:head expected=17:f905c69b:::2:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:f905c69b:::2:head -> 2 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=2 expected=2 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:c4fdafeb:::6:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:c4fdafeb:::6:head expected=17:c4fdafeb:::6:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:c4fdafeb:::6:head -> 6 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=6 expected=6 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:bd63b0f1:::8:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:bd63b0f1:::8:head expected=17:bd63b0f1:::8:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:bd63b0f1:::8:head -> 8 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=8 expected=8 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:02547ec2:::1:head 2024-03-30T14:57:56.712 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:02547ec2:::1:head expected=17:02547ec2:::1:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:02547ec2:::1:head -> 1 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=1 expected=1 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:b29083e3:::5:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:b29083e3:::5:head expected=17:b29083e3:::5:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:b29083e3:::5:head -> 5 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=5 expected=5 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:89d3ae78:::11:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:89d3ae78:::11:head expected=17:89d3ae78:::11:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:89d3ae78:::11:head -> 11 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=11 expected=11 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:52ea6a34:::10:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:52ea6a34:::10:head expected=17:52ea6a34:::10:head 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:52ea6a34:::10:head -> 10 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=10 expected=10 2024-03-30T14:57:56.713 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:e960b815:::9:head 2024-03-30T14:57:56.714 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:e960b815:::9:head expected=17:e960b815:::9:head 2024-03-30T14:57:56.714 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:e960b815:::9:head -> 9 2024-03-30T14:57:56.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:56.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:56.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=9 list_parallel: process_1_[93827]: starting. 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: creating pool ceph_test_rados_list_parallel.smithi028-93806 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: created object 0... 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: created object 25... 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: created object 49... 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: finishing. 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_1_[93827]: shutting down. 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: starting. 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: listing objects. 2024-03-30T14:57:56.863 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: listed object 0... 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: listed object 25... 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: saw 50 objects 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_2_[93832]: shutting down. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: starting. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: creating pool ceph_test_rados_list_parallel.smithi028-93806 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: created object 0... 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: created object 25... 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: created object 49... 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: finishing. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_3_[96749]: shutting down. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: starting. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: listing objects. 2024-03-30T14:57:56.864 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: listed object 0... 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: listed object 25... 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: saw 43 objects 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_4_[96750]: shutting down. 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: starting. 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: removed 25 objects... 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: removed half of the objects 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: removed 50 objects... 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: removed 50 objects 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_5_[96751]: shutting down. 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: starting. 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: creating pool ceph_test_rados_list_parallel.smithi028-93806 2024-03-30T14:57:56.865 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: created object 0... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: created object 25... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: created object 49... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: finishing. 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_6_[96906]: shutting down. 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: starting. 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: listing objects. 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: listed object 0... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: listed object 25... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: listed object 50... 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: saw 56 objects 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_7_[96907]: shutting down. 2024-03-30T14:57:56.866 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: starting. 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: added 25 objects... 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: added half of the objects 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: added 50 objects... 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: added 50 objects 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_8_[96908]: shutting down. 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.867 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:56.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:56.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:56.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:57:57.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: starting. 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: creating pool ceph_test_rados_list_parallel.smithi028-93806 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: created object 0... 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: created object 25... 2024-03-30T14:57:57.060 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: created object 49... 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: finishing. 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_9_[96976]: shutting down. 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: starting. 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: listing objects. 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: listed object 0... 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: listed object 25... 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: listed object 50... 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: listed object 75... 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: saw 98 objects 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_10_[96977]: shutting down. 2024-03-30T14:57:57.061 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: starting. 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: removed 25 objects... 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: removed half of the objects 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: removed 50 objects... 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: removed 50 objects 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_13_[96980]: shutting down. 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.062 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: starting. 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: added 25 objects... 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: added half of the objects 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: added 50 objects... 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: added 50 objects 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_11_[96978]: shutting down. 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: starting. 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: added 25 objects... 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: added half of the objects 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: added 50 objects... 2024-03-30T14:57:57.063 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: added 50 objects 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_12_[96979]: shutting down. 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: starting. 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: creating pool ceph_test_rados_list_parallel.smithi028-93806 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: created object 0... 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: created object 25... 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: created object 49... 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: finishing. 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_14_[97056]: shutting down. 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.064 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.065 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.065 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:57.065 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: starting. 2024-03-30T14:57:57.065 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listing objects. 2024-03-30T14:57:57.065 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 0... 2024-03-30T14:57:57.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:57.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:57.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:57.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:57.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:57.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:57.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: pgmap v110: 1164 pgs: 514 creating+peering, 505 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-30T14:57:57.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2917608872' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:57.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:57:57.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: pgmap v110: 1164 pgs: 514 creating+peering, 505 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2917608872' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:57:57.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:57.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:57.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:57 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: pgmap v110: 1164 pgs: 514 creating+peering, 505 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 170 B/s wr, 0 op/s 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2917608872' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: osdmap e115: 8 total, 8 up, 8 in 2024-03-30T14:57:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:57 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:57.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:57.698 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:57.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:57.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:57.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:57.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:58.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:58.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:58.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:58.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:57:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:57:58.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:57:58.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:57:58.456 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 2 api_cmd: Running main() from gmock_main.cc 2024-03-30T14:57:58.456 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (77 ms) 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (66 ms) 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (12610 ms) 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:56.126168+0000 mon.a [INF] from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi028-94235-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:56.126244+0000 mon.a [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi028-92979-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:56.456991+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.457 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:56.756467+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.058492+0000 mon.a [INF] from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.328987+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.329130+0000 mon.a [INF] from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.334446+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.335567+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.336183+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.374319+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.374616+0000 client.admin [INF] onexx 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.389069+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.458 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.622680+0000 mon.a [INF] from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:57:58.479 INFO:tasks.workunit.client.0.smithi028.stdout: 5... 2024-03-30T14:57:58.479 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 50... 2024-03-30T14:57:58.479 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 75... 2024-03-30T14:57:58.479 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 100... 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: listed object 125... 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: saw 150 objects 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_15_[97057]: shutting down. 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: starting. 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: added 25 objects... 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: added half of the objects 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: added 50 objects... 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: added 50 objects 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: process_16_[97058]: shutting down. 2024-03-30T14:57:58.480 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.481 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.481 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.481 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.481 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******************************* 2024-03-30T14:57:58.482 INFO:tasks.workunit.client.0.smithi028.stdout: list_parallel: ******* SUCCESS ********** 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (278 ms) 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-30T14:57:58.484 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (12 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (9 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (15 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (3 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (4 ms) 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-30T14:57:58.485 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (8 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (22 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (975 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (20 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (21 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (14 ms) 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-30T14:57:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-30T14:57:58.487 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (1399 ms total) 2024-03-30T14:57:58.487 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: 2024-03-30T14:57:58.487 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-30T14:57:58.487 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (13855 ms total) 2024-03-30T14:57:58.487 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-30T14:57:58.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:58.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:58.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:58.725 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:57:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: onexx 2024-03-30T14:57:58.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: onexx 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:58 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:58.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:58.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:58.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi028-93540-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:57:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4290739773' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: onexx 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: osdmap e116: 8 total, 8 up, 8 in 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:58 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:58.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:58.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:58.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:58.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:57:59.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:57:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:57:59.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:59.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:59.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:59.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:59.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:57:59.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:57:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:57:59.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:57:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:57:59.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:57:59.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:57:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:59.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: pgmap v113: 980 pgs: 367 creating+peering, 468 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: twoxx 2024-03-30T14:57:59.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:57:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[29876]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: pgmap v113: 980 pgs: 367 creating+peering, 468 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T14:57:59.786 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: twoxx 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:57:59.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:57:59.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:57:59.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:59.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:57:59 smithi028 ceph-mon[41614]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:57:59.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:57:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:57:59.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:57:59.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:57:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: pgmap v113: 980 pgs: 367 creating+peering, 468 unknown, 145 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:57:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:57:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi028-94210-1"}]': finished 2024-03-30T14:57:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: twoxx 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:57:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:57:59.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:57:59 smithi134 ceph-mon[33714]: osdmap e117: 8 total, 8 up, 8 in 2024-03-30T14:57:59.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:00.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:00.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:00.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:00.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:00.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:00.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:00.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:00.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:00.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:00.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:00.601 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:58:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T14:58:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T14:58:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T14:58:00.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24958.0:2 29.1a e114/113 29:5c6776db:::0.obj:head v 114'1, mlcod=0'0)", 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.155298+0000", 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4426948069999996, 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020667207999999999, 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155298+0000", 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155298+0000", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155298+0000", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3699999999999999e-07 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155306+0000", 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2269999999999993e-06 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155312+0000", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2059999999999998e-06 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.155337+0000", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5389e-05 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174580+0000", 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019243519000000001 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174598+0000", 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7688000000000001e-05 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175933+0000", 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013351560000000001 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175965+0000", 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1785999999999997e-05 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(38.0 epoch 114/113 rep_tid 111 entries 114'1 (0'0) error 38:0384ab8e:::oid:head by client.25045.0:2 0.000000 -27 [r=-27+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.187866+0000", 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4101266319999999, 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012081349, 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:00.606 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187866+0000", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187866+0000", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187867+0000", 2024-03-30T14:58:00.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1699999999999998e-07 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187876+0000", 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2159999999999995e-06 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187889+0000", 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3689000000000001e-05 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187903+0000", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3803e-05 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.196477+0000", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.008573677 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199947+0000", 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034704470000000002 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.609 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24892.0:2 22.10 e114/111 22:0c63179d:test-rados-api-smithi028-93390-2::foo:head v 114'1, mlcod=0'0)", 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.187913+0000", 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4100800079999996, 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012067718999999999, 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187913+0000", 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187913+0000", 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187913+0000", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.08e-07 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187919+0000", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5709999999999997e-06 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187924+0000", 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9180000000000002e-06 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.187928+0000", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3950000000000003e-06 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.196575+0000", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086467620000000005 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.196594+0000", 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9123e-05 2024-03-30T14:58:00.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199950+0000", 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003356192 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199980+0000", 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0349999999999999e-05 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25081.0:2 37.6 37:602f83fe:::foo:head [create] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.197524+0000", 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4004686060000004, 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092626624, 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25081", 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/317566051", 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197526+0000", 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.86e-06 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197531+0000", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2379999999999997e-06 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197542+0000", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0648e-05 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197552+0000", 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0217000000000001e-05 2024-03-30T14:58:00.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199081+0000", 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001528908 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199091+0000", 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6239999999999997e-06 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246480+0000", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.047389642 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246590+0000", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109821 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246710+0000", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120436 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256246+0000", 2024-03-30T14:58:00.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095358879999999993 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267473+0000", 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011226707000000001 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290057+0000", 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022584090000000001 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290109+0000", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2052000000000002e-05 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290151+0000", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1492999999999997e-05 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25084.0:2 40.6 e114/113 40:602f83fe:::foo:head v 114'1, mlcod=0'0)", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.197680+0000", 2024-03-30T14:58:00.621 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.4003127160000002, 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017917499, 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197680+0000", 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197680+0000", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197680+0000", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8400000000000005e-07 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197691+0000", 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0910999999999999e-05 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197697+0000", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9660000000000001e-06 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197705+0000", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3969999999999997e-06 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.198890+0000", 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001185523 2024-03-30T14:58:00.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.198909+0000", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8444000000000002e-05 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.215567+0000", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016658731 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.215597+0000", 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0043e-05 2024-03-30T14:58:00.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:3 11.1a e114/110 11:5e465472:test-rados-api-smithi028-93257-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head v 114'1, mlcod=0'0)", 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.216781+0000", 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3812115689999995, 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0087670990000000004, 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216781+0000", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216781+0000", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216781+0000", 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8500000000000002e-07 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216802+0000", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0293e-05 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216807+0000", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6760000000000004e-06 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.216814+0000", 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.472e-06 2024-03-30T14:58:00.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.217882+0000", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010681 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.217911+0000", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8974000000000001e-05 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.225507+0000", 2024-03-30T14:58:00.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075964179999999997 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.225548+0000", 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0880999999999997e-05 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24892.0:5 22.12 22:4a393722:test-rados-api-smithi028-93390-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.233288+0000", 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3647048210000001, 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.057181524, 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24892", 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1207320814", 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233289+0000", 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.091e-06 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233295+0000", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8749999999999997e-06 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233306+0000", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0744000000000001e-05 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233313+0000", 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6140000000000003e-06 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233359+0000", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6215999999999999e-05 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233366+0000", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0679999999999999e-06 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242316+0000", 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0089497279999999 2024-03-30T14:58:00.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242442+0000", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012600999999999999 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242608+0000", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000165451 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259414+0000", 2024-03-30T14:58:00.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016806394999999998 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259426+0000", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2312e-05 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290387+0000", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030960560000000002 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290434+0000", 2024-03-30T14:58:00.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6702000000000001e-05 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290469+0000", 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5757999999999999e-05 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:4 23.2 23:4029b033:test-rados-api-smithi028-93408-2::nonexistent:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.240287+0000", 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3577056289999998, 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99567974800000003, 2024-03-30T14:58:00.637 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240288+0000", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.212e-06 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240293+0000", 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1719999999999998e-06 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240304+0000", 2024-03-30T14:58:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0149000000000001e-05 2024-03-30T14:58:00.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240312+0000", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1480000000000006e-06 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241020+0000", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070865900000000005 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241028+0000", 2024-03-30T14:58:00.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9510000000000006e-06 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235868+0000", 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99483949599999999 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235967+0000", 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8961000000000003e-05 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:7 11.18 11:1ea51813:test-rados-api-smithi028-93257-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.245958+0000", 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3520344279999996, 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071106750000000003, 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245958+0000", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245958+0000", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245960+0000", 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2669999999999999e-06 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245966+0000", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1190000000000002e-06 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245979+0000", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3038e-05 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.245987+0000", 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4780000000000008e-06 2024-03-30T14:58:00.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.246060+0000", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2949999999999998e-05 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.246220+0000", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015960599999999999 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.246401+0000", 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018118499999999999 2024-03-30T14:58:00.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247147+0000", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00074653899999999995 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247273+0000", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012541599999999999 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.252970+0000", 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056968310000000003 2024-03-30T14:58:00.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.253028+0000", 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8439000000000001e-05 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.253069+0000", 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0806999999999997e-05 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24964.0:5 30.2 30:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=28b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.247506+0000", 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3504871500000002, 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099908429999999993, 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24964", 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2019932805", 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5 2024-03-30T14:58:00.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247506+0000", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247506+0000", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247507+0000", 2024-03-30T14:58:00.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.344e-06 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247512+0000", 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2689999999999999e-06 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247520+0000", 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0530000000000007e-06 2024-03-30T14:58:00.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247528+0000", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5140000000000004e-06 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247581+0000", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3514000000000003e-05 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.247646+0000", 2024-03-30T14:58:00.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5042000000000005e-05 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,4", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256149+0000", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085027899999999997 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256909+0000", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00075959400000000005 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256972+0000", 2024-03-30T14:58:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3826999999999997e-05 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.257411+0000", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043827200000000002 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.257458+0000", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7077999999999999e-05 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.257496+0000", 2024-03-30T14:58:00.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8546e-05 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24940.0:9 26.7 26:e046d0fd:test-rados-api-smithi028-93750-4::testobj:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.249168+0000", 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.3488250749999997, 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074770729999999999, 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24940", 2024-03-30T14:58:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249168+0000", 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249168+0000", 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249169+0000", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4399999999999998e-07 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249173+0000", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9529999999999998e-06 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249181+0000", 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6669999999999996e-06 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249189+0000", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0849999999999997e-06 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249241+0000", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1557999999999997e-05 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249314+0000", 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3462999999999998e-05 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.249405+0000", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0916999999999997e-05 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.250235+0000", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00082981000000000005 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.251653+0000", 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014178439999999999 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256506+0000", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048534340000000002 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256561+0000", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5195000000000003e-05 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.256645+0000", 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3202999999999996e-05 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.661 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(14.1d epoch 114/111 rep_tid 87 entries 114'2 (0'0) error 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head by client.24853.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.217314+0000", 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.380678917, 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026357647000000001, 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217314+0000", 2024-03-30T14:58:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217314+0000", 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217314+0000", 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7e-07 2024-03-30T14:58:00.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217323+0000", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0960000000000007e-06 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217339+0000", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6180000000000001e-05 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.217356+0000", 2024-03-30T14:58:00.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6201999999999998e-05 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.227822+0000", 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010466197999999999 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.243671+0000", 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015849500999999998 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25048.0:2 46.0s0 46:00000000::::head [pgnls start_epoch 116] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.425725+0000", 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.1722673100000001, 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2275935200000001, 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25048", 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:58:00.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T14:58:00.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425727+0000", 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8560000000000001e-06 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425733+0000", 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6740000000000002e-06 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425742+0000", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9740000000000008e-06 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425750+0000", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6539999999999999e-06 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425815+0000", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5493999999999993e-05 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:58:00.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425818+0000", 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.706e-06 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651104+0000", 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2252865639999999 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651143+0000", 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8489999999999999e-05 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.653319+0000", 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021761079999999999 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2064 11.18 11:1888b8ad:test-rados-api-smithi028-93257-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.140535+0000", 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.457458098, 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085813029999999998, 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2064 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T14:58:00.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140536+0000", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.023e-06 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140540+0000", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5650000000000003e-06 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140549+0000", 2024-03-30T14:58:00.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.789e-06 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140555+0000", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.947e-06 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140611+0000", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5859999999999997e-05 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140733+0000", 2024-03-30T14:58:00.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121965 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.204443+0000", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063710599000000007 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.206091+0000", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001648084 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226061+0000", 2024-03-30T14:58:00.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019969810000000001 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226257+0000", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019557099999999999 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226301+0000", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4688000000000003e-05 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226348+0000", 2024-03-30T14:58:00.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6128999999999998e-05 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2065 11.18 11:1888b8ad:test-rados-api-smithi028-93257-16::big.copy2:head [read 0~25165836] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.226514+0000", 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.3714785869999999, 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018423089, 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.677 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2065 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226514+0000", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226514+0000", 2024-03-30T14:58:00.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226516+0000", 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6449999999999999e-06 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226521+0000", 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8500000000000002e-06 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226530+0000", 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5519999999999993e-06 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226536+0000", 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1890000000000004e-06 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226591+0000", 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4616e-05 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226684+0000", 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2942000000000006e-05 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.244937+0000", 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018253294999999999 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2070 11.1e e116/110 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head v 116'5, mlcod=115'3)", 2024-03-30T14:58:00.681 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.381901+0000", 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.2160915160000001, 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040999507999999997, 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.381901+0000", 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.381901+0000", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.381904+0000", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7180000000000001e-06 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.406109+0000", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024205476 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.406132+0000", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2240999999999999e-05 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.406144+0000", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1936999999999999e-05 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.406231+0000", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7095000000000006e-05 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.406277+0000", 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6291000000000001e-05 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.422823+0000", 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016546478999999999 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.422901+0000", 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7270999999999999e-05 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2073 11.1e e116/110 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head v 116'8, mlcod=116'6)", 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.463112+0000", 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.1348808940000001, 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.121009771, 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T14:58:00.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463113+0000", 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5859999999999999e-06 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471534+0000", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084207299999999995 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471550+0000", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5531000000000001e-05 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471557+0000", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5140000000000004e-06 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485095+0000", 2024-03-30T14:58:00.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013537693 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485121+0000", 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6554999999999999e-05 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584083+0000", 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.098961412999999998 2024-03-30T14:58:00.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584122+0000", 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8748999999999998e-05 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34880.0:5 13.b e117/110 13:d4d78c7e:test-rados-api-smithi028-93525-3::foo:head v 117'1, mlcod=0'0)", 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.619342+0000", 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.978650915, 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082191580000000007, 2024-03-30T14:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619342+0000", 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619342+0000", 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619342+0000", 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7599999999999997e-07 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619353+0000", 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1241e-05 2024-03-30T14:58:00.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619360+0000", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2670000000000003e-06 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619387+0000", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7311e-05 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619565+0000", 2024-03-30T14:58:00.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017804799999999999 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.619585+0000", 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9409999999999999e-05 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.627523+0000", 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079386390000000008 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.627561+0000", 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7765999999999997e-05 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2074 11.15 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.630356+0000", 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.967636339, 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.095916847999999999, 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2074 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630358+0000", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.623e-06 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666706+0000", 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036348502999999997 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666726+0000", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9771000000000001e-05 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666738+0000", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2181000000000001e-05 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669591+0000", 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028525410000000001 2024-03-30T14:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669724+0000", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013316700000000001 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669879+0000", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015516999999999999 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.723327+0000", 2024-03-30T14:58:00.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053447407000000002 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.724496+0000", 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001169123 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726138+0000", 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016417020000000001 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726191+0000", 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3597000000000002e-05 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726273+0000", 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2063000000000004e-05 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2076 11.15 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.729323+0000", 2024-03-30T14:58:00.701 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.86866987, 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015101603, 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2076 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729323+0000", 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729323+0000", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729326+0000", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9840000000000001e-06 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729913+0000", 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00058682600000000004 2024-03-30T14:58:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729929+0000", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6739999999999999e-05 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.729942+0000", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2886e-05 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.730026+0000", 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3350999999999995e-05 2024-03-30T14:58:00.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.730124+0000", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8870999999999996e-05 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.734487+0000", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043629890000000003 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.740675+0000", 2024-03-30T14:58:00.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0061876489999999999 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.743134+0000", 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024592659999999999 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.744347+0000", 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001212858 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:00.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.744388+0000", 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0305999999999997e-05 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.744424+0000", 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6876999999999999e-05 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:00.707 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1176672499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4259934820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4132935590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:00.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:00.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1176672499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4259934820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4132935590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:00.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:00 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:00.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:00.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:00.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:00.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:00.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:01.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1176672499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4259934820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4132935590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: osdmap e118: 8 total, 8 up, 8 in 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:01.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:00 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:01.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:01.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:01.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:01.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:01.626 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.623216+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.626 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.624229+0000 mon.a [INF] from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T14:58:01.626 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.890316+0000 mon.a [INF] from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.890920+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.892582+0000 mon.a [INF] from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.893069+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:57.893489+0000 mon.a [INF] from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.354888+0000 mon.a [INF] from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.621495+0000 mon.a [INF] from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.622130+0000 mon.a [INF] from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.626797+0000 mon.a [INF] from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.627220+0000 mon.a [INF] from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.627384+0000 mon.a [INF] from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.640105+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.640367+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/1176672499' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.627 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.642174+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/4259934820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:01.773 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.642421+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/4132935590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.773 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.713930+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.773 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.718038+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:01.773 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.919108+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.773 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.919468+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.920249+0000 mon.a [INF] from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.920816+0000 mon.a [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:57:59.921768+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.440586+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.704967+0000 mon.a [INF] from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.711649+0000 mon.a [INF] from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.711879+0000 mon.a [INF] from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.712301+0000 mon.a [INF] from='client.25354 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.712561+0000 mon.a [INF] from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.712792+0000 mon.a [INF] from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.774 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.713007+0000 mon.a [INF] from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:01.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:01.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:01.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.4 deep-scrub starts 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.4 deep-scrub ok 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.3 deep-scrub starts 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.3 deep-scrub ok 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.6 deep-scrub starts 2024-03-30T14:58:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.6 deep-scrub ok 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.7 deep-scrub starts 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.0 deep-scrub starts 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: 11.0 deep-scrub ok 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: pgmap v115: 756 pgs: 134 creating+peering, 15 creating+activating, 1 active+clean+snaptrim, 56 unknown, 550 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 31 MiB/s wr, 483 op/s 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.25354 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.4 deep-scrub starts 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.4 deep-scrub ok 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.3 deep-scrub starts 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.3 deep-scrub ok 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.6 deep-scrub starts 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.6 deep-scrub ok 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.7 deep-scrub starts 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.0 deep-scrub starts 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: 11.0 deep-scrub ok 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: pgmap v115: 756 pgs: 134 creating+peering, 15 creating+activating, 1 active+clean+snaptrim, 56 unknown, 550 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 31 MiB/s wr, 483 op/s 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:01.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.25354 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:01.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:58:01.978 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:01.978 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:01 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:02.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:02.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T14:58:02.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:02.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.4 deep-scrub starts 2024-03-30T14:58:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.4 deep-scrub ok 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.3 deep-scrub starts 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.3 deep-scrub ok 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.6 deep-scrub starts 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.6 deep-scrub ok 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.7 deep-scrub starts 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.0 deep-scrub starts 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: 11.0 deep-scrub ok 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: pgmap v115: 756 pgs: 134 creating+peering, 15 creating+activating, 1 active+clean+snaptrim, 56 unknown, 550 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 31 MiB/s wr, 483 op/s 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.25354 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.25450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.25357 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi028-94210-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: osdmap e119: 8 total, 8 up, 8 in 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:01 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:02.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:02.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:02.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:02.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:02.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:02.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:02.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:02.669 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T14:58:02.669 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T14:58:02.670 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-30T14:58:02.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-30T14:58:02.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:02.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.7 deep-scrub ok 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.5 deep-scrub starts 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.5 deep-scrub ok 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.1 deep-scrub starts 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.1 deep-scrub ok 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.7 deep-scrub ok 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.5 deep-scrub starts 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.5 deep-scrub ok 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.1 deep-scrub starts 2024-03-30T14:58:02.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.1 deep-scrub ok 2024-03-30T14:58:02.780 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.713098+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:02.780 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.719414+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.780 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.722207+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:02.780 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:00.989159+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.247024+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.247206+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.505550+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.505958+0000 mon.a [INF] from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.506479+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.626368+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.626534+0000 client.admin [INF] threexx 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.771587+0000 mon.a [INF] from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.777147+0000 mon.a [INF] from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.777380+0000 mon.a [INF] from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779175+0000 mon.a [INF] from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779615+0000 mon.a [INF] from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:02.781 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779710+0000 mon.a [INF] from='client.25354 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779784+0000 mon.a [INF] from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779840+0000 mon.a [INF] from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.779903+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.784880+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.786755+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.788562+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.792750+0000 mon.c [INF] from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.799908+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:01.803635+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.358822+0000 mon.a [INF] from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.359449+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.360995+0000 mon.a [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.361354+0000 mon.a [INF] from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:02.782 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.361476+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:02.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.363701+0000 mon.a [INF] from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smith cmd: Running main() from gmock_main.cc 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [----------] Global test environment set-up. 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (10568 ms) 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2038 ms) 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-30T14:58:02.801 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (5459 ms) 2024-03-30T14:58:02.802 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [----------] 3 tests from NeoRadosCmd (18065 ms total) 2024-03-30T14:58:02.802 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: 2024-03-30T14:58:02.802 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [----------] Global test environment tear-down 2024-03-30T14:58:02.802 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [==========] 3 tests from 1 test suite ran. (18065 ms total) 2024-03-30T14:58:02.802 INFO:tasks.workunit.client.0.smithi028.stdout: cmd: [ PASSED ] 3 tests. 2024-03-30T14:58:02.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:02.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:03.010 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-30T14:58:03.010 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-30T14:58:03.010 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-30T14:58:03.010 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: seed 93069 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (3805 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (5 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (1116 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1091 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7 ms) 2024-03-30T14:58:03.011 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (4 ms) 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-30T14:58:03.012 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (9 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (8 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (12 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-03-30T14:58:03.013 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (6103 ms total) 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1190 ms) 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (23 ms) 2024-03-30T14:58:03.014 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-30T14:58:03.015 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-30T14:58:03.015 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-30T14:58:03.015 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-03-30T14:58:03.015 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-30T14:58:03.015 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2024-03-30T14:58:03.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:03.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:03.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.8 deep-scrub starts 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.8 deep-scrub ok 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: pgmap v118: 884 pgs: 134 creating+peering, 15 creating+activating, 312 unknown, 423 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 508 op/s 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: threexx 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:03.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25354 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.8 deep-scrub starts 2024-03-30T14:58:03.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.8 deep-scrub ok 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: pgmap v118: 884 pgs: 134 creating+peering, 15 creating+activating, 312 unknown, 423 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 508 op/s 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: threexx 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25354 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:03.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.9 deep-scrub starts 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: 11.9 deep-scrub ok 2024-03-30T14:58:03.105 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.9 deep-scrub starts 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: 11.9 deep-scrub ok 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:02 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN cls: Running main() from gmock_main.cc 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [----------] Global test environment set-up. 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [ OK ] NeoRadosCls.DNE (11764 ms) 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-30T14:58:03.135 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (6661 ms) 2024-03-30T14:58:03.136 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [----------] 2 tests from NeoRadosCls (18425 ms total) 2024-03-30T14:58:03.136 INFO:tasks.workunit.client.0.smithi028.stdout: cls: 2024-03-30T14:58:03.136 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [----------] Global test environment tear-down 2024-03-30T14:58:03.136 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [==========] 2 tests from 1 test suite ran. (18425 ms total) 2024-03-30T14:58:03.136 INFO:tasks.workunit.client.0.smithi028.stdout: cls: [ PASSED ] 2 tests. 2024-03-30T14:58:03.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.7 deep-scrub ok 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.5 deep-scrub starts 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.5 deep-scrub ok 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.1 deep-scrub starts 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.1 deep-scrub ok 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.8 deep-scrub starts 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.8 deep-scrub ok 2024-03-30T14:58:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: pgmap v118: 884 pgs: 134 creating+peering, 15 creating+activating, 312 unknown, 423 active+clean; 145 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 34 MiB/s wr, 508 op/s 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: threexx 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi028-93069-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25354 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi028-92979-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25357 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:03.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: osdmap e120: 8 total, 8 up, 8 in 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.9 deep-scrub starts 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: 11.9 deep-scrub ok 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:03.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:02 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:03.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:03.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:03.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:03.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 -- 172.21.15.28:0/371973649 >> v1:172.21.15.28:6790/0 conn(0x7fa84c102260 legacy=0x7fa84c102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 -- 172.21.15.28:0/371973649 shutdown_connections 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 -- 172.21.15.28:0/371973649 >> 172.21.15.28:0/371973649 conn(0x7fa84c0fdb10 msgr2=0x7fa84c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 -- 172.21.15.28:0/371973649 shutdown_connections 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 -- 172.21.15.28:0/371973649 wait complete. 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 librados: starting msgr at 2024-03-30T14:58:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.456+0000 7fa850a18700 1 librados: starting objecter 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 Processor -- start 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 -- start start 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 librados: setting wanted keys 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 librados: calling monclient init 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa84c1a9560 con 0x7fa84c10aef0 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa84c1aa780 con 0x7fa84c102260 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa850a18700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa84c1ab9c0 con 0x7fa84c10ea90 2024-03-30T14:58:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa84ad9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa84c10ea90 0x7fa84c1a7c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47738/0 (socket says 172.21.15.28:47738) 2024-03-30T14:58:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa84ad9d700 1 -- 172.21.15.28:0/2459087569 learned_addr learned my addr 172.21.15.28:0/2459087569 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter ms_handle_connect 0x7fa84c10ea90 2024-03-30T14:58:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter ms_handle_connect 0x7fa84c102260 2024-03-30T14:58:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter ms_handle_connect 0x7fa84c10aef0 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 750439404 0 0) 0x7fa84c1ab9c0 con 0x7fa84c10ea90 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8340035b0 con 0x7fa84c10ea90 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914985829 0 0) 0x7fa84c1aa780 con 0x7fa84c102260 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa84c1ab9c0 con 0x7fa84c102260 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2999063342 0 0) 0x7fa8340035b0 con 0x7fa84c10ea90 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa84c1aa780 con 0x7fa84c10ea90 2024-03-30T14:58:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.457+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa844001c50 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2024793741 0 0) 0x7fa84c1ab9c0 con 0x7fa84c102260 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8340035b0 con 0x7fa84c102260 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa83c001430 con 0x7fa84c102260 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3665594015 0 0) 0x7fa84c1aa780 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.134:6789/0 conn(0x7fa84c102260 legacy=0x7fa84c075f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6789/0 conn(0x7fa84c10aef0 legacy=0x7fa84c1a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c1acc00 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa84c1aa9b0 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa850a18700 10 client.35498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa84c1aae70 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa850a18700 1 librados: init done 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa844003ec0 con 0x7fa84c10ea90 2024-03-30T14:58:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.458+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa844006090 con 0x7fa84c10ea90 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.459+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa82c005100 con 0x7fa84c10ea90 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.459+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa844003a20 con 0x7fa84c10ea90 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(121..121 src has 53..121) ==== 17184+0+0 (unknown 3257677250 0 0) 0x7fa84408bf60 con 0x7fa84c10ea90 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 osd_map(121..121 src has 53..121) 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 3 client.35498.objecter handle_osd_map got epochs [121,121] > 0 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 3 client.35498.objecter handle_osd_map decoding full epoch 121 2024-03-30T14:58:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 20 client.35498.objecter dump_active .. 0 homeless 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.460+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v121) ==== 54+0+16 (unknown 1649006044 0 2420000192) 0x7fa84408e3c0 con 0x7fa84c10ea90 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 10 client.35498.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 1 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 1 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 0 1 2024-03-30T14:58:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 4 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 10 client.35498.objecter _send_command 1 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa82c004b50 con 0x7fa82c004730 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa8340702e0 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.461+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c004730 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.462+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c004b50 con 0x7fa82c004730 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.462+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.462+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 1 = system:0 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.462+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 4 2024-03-30T14:58:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.462+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 0 1 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 10 client.35498.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 2 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 2 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c0053e0 osd=0 3 2024-03-30T14:58:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 0 2 2024-03-30T14:58:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 4 2024-03-30T14:58:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 10 client.35498.objecter _send_command 2 2024-03-30T14:58:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.490+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c007b00 con 0x7fa82c004730 2024-03-30T14:58:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.491+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c007b00 con 0x7fa82c004730 2024-03-30T14:58:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 4 2024-03-30T14:58:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 0 2 2024-03-30T14:58:03.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 10 client.35498.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 1 3 2024-03-30T14:58:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 4 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 10 client.35498.objecter _send_command 3 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa82c002200 con 0x7fa82c0042f0 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.492+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c0042f0 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.493+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c002200 con 0x7fa82c0042f0 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.493+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 3: 0 ) 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.493+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 3 = system:0 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.493+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 4 2024-03-30T14:58:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.493+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 1 3 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 10 client.35498.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 4 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 4 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c004180 osd=1 3 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 1 4 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 4 2024-03-30T14:58:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 10 client.35498.objecter _send_command 4 2024-03-30T14:58:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.510+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c001640 con 0x7fa82c0042f0 2024-03-30T14:58:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c001640 con 0x7fa82c0042f0 2024-03-30T14:58:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 4 2024-03-30T14:58:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 1 4 2024-03-30T14:58:03.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:58:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 10 client.35498.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 5 2024-03-30T14:58:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 5 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 2 5 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 4 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 10 client.35498.objecter _send_command 5 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.512+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa82c0111f0 con 0x7fa82c00d6e0 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c00d6e0 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c0111f0 con 0x7fa82c00d6e0 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 5: 0 ) 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 5 = system:0 2024-03-30T14:58:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 4 2024-03-30T14:58:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.513+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 2 5 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 10 client.35498.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 6 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 6 2024-03-30T14:58:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c008480 osd=2 3 2024-03-30T14:58:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 2 6 2024-03-30T14:58:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 4 2024-03-30T14:58:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 10 client.35498.objecter _send_command 6 2024-03-30T14:58:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.530+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c007f60 con 0x7fa82c00d6e0 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c007f60 con 0x7fa82c00d6e0 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 4 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 2 6 2024-03-30T14:58:03.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:58:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 10 client.35498.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 7 2024-03-30T14:58:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 7 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 3 7 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 4 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 10 client.35498.objecter _send_command 7 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa82c00d480 con 0x7fa82c012520 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.532+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c012520 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.533+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c00d480 con 0x7fa82c012520 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.533+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 7: 0 ) 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.533+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 7 = system:0 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.533+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 4 2024-03-30T14:58:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.533+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 3 7 2024-03-30T14:58:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 10 client.35498.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 8 2024-03-30T14:58:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 8 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c00cd30 osd=3 3 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 3 8 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 4 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 10 client.35498.objecter _send_command 8 2024-03-30T14:58:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.550+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c00c870 con 0x7fa82c012520 2024-03-30T14:58:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.552+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c00c870 con 0x7fa82c012520 2024-03-30T14:58:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.552+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.552+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.552+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 4 2024-03-30T14:58:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.552+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 3 8 2024-03-30T14:58:03.554 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:58:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 10 client.35498.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 9 2024-03-30T14:58:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 9 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 4 9 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 4 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 10 client.35498.objecter _send_command 9 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa82c01a9b0 con 0x7fa82c012080 2024-03-30T14:58:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.553+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c012080 2024-03-30T14:58:03.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.554+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c01a9b0 con 0x7fa82c012080 2024-03-30T14:58:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.554+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 9: 0 ) 2024-03-30T14:58:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.554+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 9 = system:0 2024-03-30T14:58:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.554+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 4 2024-03-30T14:58:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.554+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 4 9 2024-03-30T14:58:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 10 client.35498.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 10 2024-03-30T14:58:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 10 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c00c870 osd=4 3 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 4 10 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 4 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 10 client.35498.objecter _send_command 10 2024-03-30T14:58:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.570+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c01a9b0 con 0x7fa82c012080 2024-03-30T14:58:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c01a9b0 con 0x7fa82c012080 2024-03-30T14:58:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 4 2024-03-30T14:58:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 4 10 2024-03-30T14:58:03.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:58:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 10 client.35498.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 11 2024-03-30T14:58:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 11 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 5 11 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 4 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 10 client.35498.objecter _send_command 11 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.572+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa82c01ed10 con 0x7fa82c01b030 2024-03-30T14:58:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.573+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c01b030 2024-03-30T14:58:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.574+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c01ed10 con 0x7fa82c01b030 2024-03-30T14:58:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.574+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 11: 0 ) 2024-03-30T14:58:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.574+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 11 = system:0 2024-03-30T14:58:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.574+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 4 2024-03-30T14:58:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.574+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 5 11 2024-03-30T14:58:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:03.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 10 client.35498.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 12 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 12 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c01a9b0 osd=5 3 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 5 12 2024-03-30T14:58:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 4 2024-03-30T14:58:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 10 client.35498.objecter _send_command 12 2024-03-30T14:58:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.591+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c01eef0 con 0x7fa82c01b030 2024-03-30T14:58:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c01eef0 con 0x7fa82c01b030 2024-03-30T14:58:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 4 2024-03-30T14:58:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 5 12 2024-03-30T14:58:03.595 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:58:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 10 client.35498.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 13 2024-03-30T14:58:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 13 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 6 13 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 4 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 10 client.35498.objecter _send_command 13 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa82c020220 con 0x7fa82c01f610 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.593+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c01f610 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.594+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c020220 con 0x7fa82c01f610 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.594+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 13: 0 ) 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.594+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 13 = system:0 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.594+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 4 2024-03-30T14:58:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.594+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 6 13 2024-03-30T14:58:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 10 client.35498.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 14 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 14 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c01eef0 osd=6 3 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 6 14 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 4 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 10 client.35498.objecter _send_command 14 2024-03-30T14:58:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.611+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c020220 con 0x7fa82c01f610 2024-03-30T14:58:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.613+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c020220 con 0x7fa82c01f610 2024-03-30T14:58:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.613+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.613+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.613+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 4 2024-03-30T14:58:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.613+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 6 14 2024-03-30T14:58:03.615 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 10 client.35498.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 15 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 15 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 7 15 2024-03-30T14:58:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 4 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 10 client.35498.objecter _send_command 15 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa82c027700 con 0x7fa82c023bf0 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.614+0000 7fa83b7fe700 10 client.35498.objecter ms_handle_connect 0x7fa82c023bf0 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.615+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa82c027700 con 0x7fa82c023bf0 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.615+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 15: 0 ) 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.615+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 15 = system:0 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.615+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 4 2024-03-30T14:58:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.615+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 7 15 2024-03-30T14:58:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 10 client.35498.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign -1 16 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 20 client.35498.objecter _get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 15 client.35498.objecter _session_command_op_remove -1 16 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 20 client.35498.objecter get_session s=0x7fa82c0204a0 osd=7 3 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 15 client.35498.objecter _session_command_op_assign 7 16 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 4 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 10 client.35498.objecter _send_command 16 2024-03-30T14:58:03.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.632+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa82c00c680 con 0x7fa82c023bf0 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa83b7fe700 1 -- 172.21.15.28:0/2459087569 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa82c00c680 con 0x7fa82c023bf0 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa83b7fe700 10 client.35498.objecter ms_dispatch 0x7fa84c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa83b7fe700 10 client.35498.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa83b7fe700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 4 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa83b7fe700 15 client.35498.objecter _session_command_op_remove 7 16 2024-03-30T14:58:03.635 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 librados: watch_flush enter 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 librados: watch_flush exit 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.0 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6801/3394323575 conn(0x7fa82c004730 legacy=0x7fa82c002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0053e0 osd=0 1 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.1 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6805/1850313547 conn(0x7fa82c0042f0 legacy=0x7fa82c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c004180 osd=1 1 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.2 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6809/3451325043 conn(0x7fa82c00d6e0 legacy=0x7fa82c00fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c008480 osd=2 1 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.3 2024-03-30T14:58:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6813/1506157250 conn(0x7fa82c012520 legacy=0x7fa82c014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00cd30 osd=3 1 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.4 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.134:6800/3063775514 conn(0x7fa82c012080 legacy=0x7fa82c0171b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa84ad9d700 1 -- 172.21.15.28:0/2459087569 reap_dead start 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa84ad9d700 1 -- 172.21.15.28:0/2459087569 reap_dead start 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c00c870 osd=4 1 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.5 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.134:6804/3668151179 conn(0x7fa82c01b030 legacy=0x7fa82c01d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01a9b0 osd=5 1 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 10 client.35498.objecter close_session for osd.6 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.634+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.134:6808/2135224194 conn(0x7fa82c01f610 legacy=0x7fa82c01fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c01eef0 osd=6 1 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 10 client.35498.objecter close_session for osd.7 2024-03-30T14:58:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.134:6812/3758190298 conn(0x7fa82c023bf0 legacy=0x7fa82c025fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 20 client.35498.objecter put_session s=0x7fa82c0204a0 osd=7 1 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 20 client.35498.objecter shutdown clearing up homeless session... 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 10 client.35498.objecter successfully canceled tick 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa8340702e0 legacy=0x7fa8340727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> v1:172.21.15.28:6790/0 conn(0x7fa84c10ea90 legacy=0x7fa84c1a7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa84ad9d700 1 -- 172.21.15.28:0/2459087569 reap_dead start 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 shutdown_connections 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 >> 172.21.15.28:0/2459087569 conn(0x7fa84c0fdb10 msgr2=0x7fa84c1116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 shutdown_connections 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 -- 172.21.15.28:0/2459087569 wait complete. 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:03.635+0000 7fa850a18700 1 librados: shutdown 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:58:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:58:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:58:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:58:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:58:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:58:03.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:03.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:03.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:03.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:03.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:04.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:04.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T14:58:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: fourxx 2024-03-30T14:58:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: fourxx 2024-03-30T14:58:04.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:04.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:04.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:04.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:04.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:58:04.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2459087569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:04.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2459087569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:04.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:04.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:04.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:04.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:04.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3991931878' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: fourxx 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi028-93408-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi028-93086-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-4", "overlaypool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi028-92999-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3576801120' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4069139007' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1942494314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: osdmap e121: 8 total, 8 up, 8 in 2024-03-30T14:58:04.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:04.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2459087569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:04.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:04.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:04.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:04.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:04.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:04.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:04.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:04.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:04.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:05.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:05.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:05.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:05.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:05.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: starting. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: creating pool ceph_test_rados_open_pools_parallel.smithi028-94032 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: created object 0... 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: created object 25... 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: created object 49... 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: finishing. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_1_[94088]: shutting down. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_2_[94089]: starting. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_2_[94089]: rados_pool_create. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_2_[94089]: rados_ioctx_create. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_2_[94089]: shutting down. 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: ******************************* 2024-03-30T14:58:05.279 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: starting. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: creating pool ceph_test_rados_open_pools_parallel.smithi028-94032 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: created object 0... 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: created object 25... 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: created object 49... 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: finishing. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_3_[97452]: shutting down. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: ******************************* 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_4_[97453]: starting. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_4_[97453]: rados_pool_create. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_4_[97453]: rados_ioctx_create. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: process_4_[97453]: shutting down. 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: ******************************* 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: ******************************* 2024-03-30T14:58:05.280 INFO:tasks.workunit.client.0.smithi028.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-30T14:58:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:05.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:05.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:05.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:05.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:05.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:05.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:05.819 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-30T14:58:05.819 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-30T14:58:05.819 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: handle_notify cookie 93834992382096 notify_id 489626271744 notifier_gid 25084 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (11635 ms) 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Trying... 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: handle_notify cookie 93835005080800 notify_id 506806140928 notifier_gid 35294 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Waiting for 3.000000000s 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Timed out. 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Flushing... 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: Flushed... 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9360 ms) 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (20995 ms total) 2024-03-30T14:58:05.820 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: 2024-03-30T14:58:05.821 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-30T14:58:05.821 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (20995 ms total) 2024-03-30T14:58:05.821 INFO:tasks.workunit.client.0.smithi028.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-30T14:58:05.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:05.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:05.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:05.952 INFO:tasks.workunit.client.0.smithi028.stdout:i028-92999-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.363944+0000 mon.a [INF] from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi028-94235-1"}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.367398+0000 mon.a [INF] from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.367565+0000 mon.a [INF] from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi028-93032-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.367853+0000 mon.b [INF] from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: got: 2024-03-30T14:58:02.368053+0000 mon.a [INF] from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8613 ms) 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (21367 ms total) 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (21368 ms total) 2024-03-30T14:58:05.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-30T14:58:06.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:06.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:06.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:06.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:06.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:05 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:06.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:05 smithi134 ceph-mon[33714]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T14:58:06.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:05 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:06.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[29876]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[41614]: from='client.35390 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-30T14:58:06.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:05 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi028-94210-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:06.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:06.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:06.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:06.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:06.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:06.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:06.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:06.827 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:06.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:06.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:06.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:07.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:07.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: pgmap v121: 780 pgs: 45 creating+peering, 89 unknown, 646 active+clean; 145 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s wr, 9 op/s 2024-03-30T14:58:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]': finished 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: 11.2 deep-scrub starts 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: 11.2 deep-scrub ok 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3749196455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: pgmap v123: 748 pgs: 64 creating+peering, 104 unknown, 580 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 561 KiB/s wr, 2 op/s 2024-03-30T14:58:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2265876563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2942889660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:07.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:58:07.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.35492 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:07.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: pgmap v121: 780 pgs: 45 creating+peering, 89 unknown, 646 active+clean; 145 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s wr, 9 op/s 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]': finished 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: 11.2 deep-scrub starts 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: 11.2 deep-scrub ok 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3749196455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: pgmap v123: 748 pgs: 64 creating+peering, 104 unknown, 580 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 561 KiB/s wr, 2 op/s 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: pgmap v121: 780 pgs: 45 creating+peering, 89 unknown, 646 active+clean; 145 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s wr, 9 op/s 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi028-94235-1"}]': finished 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi028-93540-1"}]': finished 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-4-cache", "mode": "writeback"}]': finished 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi028-92999-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: 11.2 deep-scrub starts 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: 11.2 deep-scrub ok 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3749196455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: pgmap v123: 748 pgs: 64 creating+peering, 104 unknown, 580 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 561 KiB/s wr, 2 op/s 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: osdmap e122: 8 total, 8 up, 8 in 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:07.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2265876563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2942889660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.35492 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi028-93390-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:07.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi028-93086-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi028-93116-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi028-93032-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi028-93408-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2265876563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2942889660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: osdmap e123: 8 total, 8 up, 8 in 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.35492 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:07.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:07.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:07.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:07.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:07.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:07.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:07.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:07.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:07.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:07.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:07.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:07.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:07.885 INFO:tasks.workunit.client.0.smithi028.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-30T14:58:07.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-30T14:58:07.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-30T14:58:07.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (4 ms) 2024-03-30T14:58:07.885 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-30T14:58:07.886 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (6 ms) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1283 ms total) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (23490 ms total) 2024-03-30T14:58:07.887 INFO:tasks.workunit.client.0.smithi028.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-30T14:58:07.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:07.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:07.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:08.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:08.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:08.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:08.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:08.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.25561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2"}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:08.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.25561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:08.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.25561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi028-94235-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:08.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-4"}]': finished 2024-03-30T14:58:08.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:08.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi028-92979-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:08.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: osdmap e124: 8 total, 8 up, 8 in 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2389145280' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/658745307' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:08.228 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]: dispatch 2024-03-30T14:58:08.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:08.229 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:08.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:08.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:08.520 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:08.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:08.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:08.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:08.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:08.821 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:08.821 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3787664845 >> v1:172.21.15.28:6790/0 conn(0x7f6a84075100 legacy=0x7f6a8410f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3787664845 shutdown_connections 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3787664845 >> 172.21.15.28:0/3787664845 conn(0x7f6a840fdaf0 msgr2=0x7f6a840fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3787664845 shutdown_connections 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3787664845 wait complete. 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.840+0000 7f6a8aaf1700 1 librados: starting msgr at 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 librados: starting objecter 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 Processor -- start 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 -- start start 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 librados: setting wanted keys 2024-03-30T14:58:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 librados: calling monclient init 2024-03-30T14:58:08.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a841a9c10 con 0x7f6a84074610 2024-03-30T14:58:08.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8908e700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6a841a6100 0x7f6a841a84f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40188/0 (socket says 172.21.15.28:40188) 2024-03-30T14:58:08.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8908e700 1 -- 172.21.15.28:0/3185449337 learned_addr learned my addr 172.21.15.28:0/3185449337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:08.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter ms_handle_connect 0x7f6a841a6100 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a841aadf0 con 0x7f6a84110c90 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter ms_handle_connect 0x7f6a84074610 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a841a9de0 con 0x7f6a841a6100 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter ms_handle_connect 0x7f6a84110c90 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.841+0000 7f6a81ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065342320 0 0) 0x7f6a841a9de0 con 0x7f6a841a6100 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a740035b0 con 0x7f6a841a6100 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904799533 0 0) 0x7f6a841aadf0 con 0x7f6a84110c90 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a841a9de0 con 0x7f6a84110c90 2024-03-30T14:58:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2639697053 0 0) 0x7f6a841a9de0 con 0x7f6a84110c90 2024-03-30T14:58:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a841aadf0 con 0x7f6a84110c90 2024-03-30T14:58:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a70001370 con 0x7f6a84110c90 2024-03-30T14:58:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1676848260 0 0) 0x7f6a740035b0 con 0x7f6a841a6100 2024-03-30T14:58:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a841a9de0 con 0x7f6a841a6100 2024-03-30T14:58:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a7c001c10 con 0x7f6a841a6100 2024-03-30T14:58:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1790796685 0 0) 0x7f6a841aadf0 con 0x7f6a84110c90 2024-03-30T14:58:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 >> v1:172.21.15.28:6790/0 conn(0x7f6a841a6100 legacy=0x7f6a841a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 >> v1:172.21.15.28:6789/0 conn(0x7f6a84074610 legacy=0x7f6a8410c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.842+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a841a9ff0 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a841aa2e0 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a8aaf1700 10 client.25630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a841aa7e0 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a8aaf1700 1 librados: init done 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6a70001510 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.843+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a70004ed0 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.844+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a84075cc0 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.844+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6a70005070 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(125..125 src has 53..125) ==== 19249+0+0 (unknown 2575540358 0 0) 0x7f6a7008b300 con 0x7f6a84110c90 2024-03-30T14:58:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 10 client.25630.objecter ms_dispatch 0x7f6a84001550 osd_map(125..125 src has 53..125) 2024-03-30T14:58:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 3 client.25630.objecter handle_osd_map got epochs [125,125] > 0 2024-03-30T14:58:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 3 client.25630.objecter handle_osd_map decoding full epoch 125 2024-03-30T14:58:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 20 client.25630.objecter dump_active .. 0 homeless 2024-03-30T14:58:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.845+0000 7f6a81ffb700 10 client.25630.objecter ms_handle_connect 0x7f6a74070270 2024-03-30T14:58:08.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:08.850+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6a70056720 con 0x7f6a84110c90 2024-03-30T14:58:08.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:08.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:08.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:08.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:08.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:09.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: pgmap v126: 884 pgs: 32 creating+peering, 304 unknown, 548 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 569 KiB/s wr, 2 op/s 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.25561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:09.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: pgmap v126: 884 pgs: 32 creating+peering, 304 unknown, 548 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 569 KiB/s wr, 2 op/s 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.25561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:09 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:09.143+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f6a8410dd10 con 0x7f6a84110c90 2024-03-30T14:58:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:09.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:09.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:09.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:09.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: pgmap v126: 884 pgs: 32 creating+peering, 304 unknown, 548 active+clean; 145 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 569 KiB/s wr, 2 op/s 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.25561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-4", "tierpool": "test-rados-api-smithi028-93253-4-cache"}]': finished 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35396 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi028-93069-22"}]': finished 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi028-93115-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-30T14:58:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: osdmap e125: 8 total, 8 up, 8 in 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:09.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:09 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:09.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:09.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:09.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:09.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:09.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:09.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:10.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:10.052 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: Running main() from gmock_main.cc 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [----------] Global test environment set-up. 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-30T14:58:10.104 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (13683 ms) 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (11641 ms) 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (25324 ms total) 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [----------] Global test environment tear-down 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (25324 ms total) 2024-03-30T14:58:10.105 INFO:tasks.workunit.client.0.smithi028.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-30T14:58:10.108 INFO:tasks.workunit.client.0.smithi028.stdout: list: Running main() from gmock_main.cc 2024-03-30T14:58:10.108 INFO:tasks.workunit.client.0.smithi028.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-30T14:58:10.108 INFO:tasks.workunit.client.0.smithi028.stdout: list: [----------] Global test environment set-up. 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [----------] 3 tests from NeoradosList 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ OK ] NeoradosList.ListObjects (13690 ms) 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ OK ] NeoradosList.ListObjectsNS (6807 ms) 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4834 ms) 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [----------] 3 tests from NeoradosList (25331 ms total) 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [----------] Global test environment tear-down 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [==========] 3 tests from 1 test suite ran. (25331 ms total) 2024-03-30T14:58:10.109 INFO:tasks.workunit.client.0.smithi028.stdout: list: [ PASSED ] 3 tests. 2024-03-30T14:58:10.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:10.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:10.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]': finished 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3185449337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:10.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]': finished 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3185449337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:58:10.154 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.155 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:10 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:10.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:10.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:10.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:10.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi028-94235-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93540-6", "pg_num": 4}]': finished 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087697659' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3185449337' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: osdmap e126: 8 total, 8 up, 8 in 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:10.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:10 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:58:10.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:10.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:10.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:10.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:10.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:10.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:10.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.913+0000 7f6a81ffb700 1 -- 172.21.15.28:0/3185449337 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v128) ==== 91+0+0 (unknown 1379125034 0 0) 0x7f6a7005a170 con 0x7f6a84110c90 2024-03-30T14:58:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.916+0000 7f6a8aaf1700 10 librados: watch_flush enter 2024-03-30T14:58:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.917+0000 7f6a8aaf1700 10 librados: watch_flush exit 2024-03-30T14:58:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.917+0000 7f6a8aaf1700 20 client.25630.objecter shutdown clearing up homeless session... 2024-03-30T14:58:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.917+0000 7f6a8aaf1700 10 client.25630.objecter successfully canceled tick 2024-03-30T14:58:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.918+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6a74070270 legacy=0x7f6a74072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.918+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 >> v1:172.21.15.134:6789/0 conn(0x7f6a84110c90 legacy=0x7f6a8410cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.918+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 shutdown_connections 2024-03-30T14:58:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.918+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 >> 172.21.15.28:0/3185449337 conn(0x7f6a840fdaf0 msgr2=0x7f6a841110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.919+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 shutdown_connections 2024-03-30T14:58:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.919+0000 7f6a8aaf1700 1 -- 172.21.15.28:0/3185449337 wait complete. 2024-03-30T14:58:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:10.919+0000 7f6a8aaf1700 1 librados: shutdown 2024-03-30T14:58:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T14:58:10.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:10.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:11.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:11.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:11.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:11.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:11.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: pgmap v129: 672 pgs: 32 creating+peering, 1 active+clean+snaptrim, 76 unknown, 563 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2024-03-30T14:58:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:11.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051384692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2614257680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1698161059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[41614]: from='client.25624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: pgmap v129: 672 pgs: 32 creating+peering, 1 active+clean+snaptrim, 76 unknown, 563 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:11.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051384692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2614257680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1698161059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:11 smithi028 ceph-mon[29876]: from='client.25624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:11.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:11.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:11.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: pgmap v129: 672 pgs: 32 creating+peering, 1 active+clean+snaptrim, 76 unknown, 563 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25459 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi028-94210-2"}]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/312481782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3771858176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828471784' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/777281174' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051384692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2614257680' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: osdmap e127: 8 total, 8 up, 8 in 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]: dispatch 2024-03-30T14:58:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi028-93417-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: osdmap e128: 8 total, 8 up, 8 in 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1698161059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:11 smithi134 ceph-mon[33714]: from='client.25624 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:11.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:11.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:11.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:11.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:11.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: Running main() from gmock_main.cc 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] Global test environment set-up. 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStat.Stat (1272 ms) 2024-03-30T14:58:11.944 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (55 ms) 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStat.StatNS (33 ms) 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (5 ms) 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] 5 tests from LibRadosStat (1365 ms total) 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-30T14:58:11.945 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2211 ms) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (14 ms) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1 ms) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2226 ms total) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [----------] Global test environment tear-down 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (27431 ms total) 2024-03-30T14:58:11.946 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: Running main() from gmock_main.cc 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [----------] Global test environment set-up. 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (1303 ms) 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-30T14:58:11.947 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: no timeout :/ 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: no timeout :/ 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: no timeout :/ 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: no timeout :/ 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: no timeout :/ 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (29 ms) 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.Checksum (2 ms) 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-30T14:58:11.948 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3 ms) 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (1011 ms) 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-30T14:58:11.949 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.TruncTest (6 ms) 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (10 ms) 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (6 ms) 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-30T14:58:11.950 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.RmXattr (18 ms) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [----------] 14 tests from LibRadosIo (2408 ms total) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (2216 ms) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-30T14:58:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4 ms) 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (2 ms) 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-30T14:58:11.952 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-30T14:58:11.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-03-30T14:58:11.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-30T14:58:11.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-30T14:58:11.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [----------] 10 tests from LibRadosIoEC (2262 ms total) 2024-03-30T14:58:11.953 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [- api_stat_pp: Running main() from gmock_main.cc 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: seed 93390 2024-03-30T14:58:11.955 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (267 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (1064 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (15 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (1359 ms total) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (2199 ms) 2024-03-30T14:58:11.956 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2 ms) 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (15 ms) 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (9 ms) 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2225 ms total) 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (27447 ms total) 2024-03-30T14:58:11.957 INFO:tasks.workunit.client.0.smithi028.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-30T14:58:11.962 INFO:tasks.workunit.client.0.smithi028.stdout:---------] Global test environment tear-down 2024-03-30T14:58:11.962 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [==========] 24 tests from 2 test suites ran. (27569 ms total) 2024-03-30T14:58:11.962 INFO:tasks.workunit.client.0.smithi028.stdout: api_io: [ PASSED ] 24 tests. 2024-03-30T14:58:11.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:12.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:12.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:12.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:58:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:12.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:12.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T14:58:12.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:12 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:12.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:12.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:12.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25654 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25666 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35414 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi028-93408-7"}]': finished 2024-03-30T14:58:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi028-93032-16"}]': finished 2024-03-30T14:58:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35432 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi028-93390-7"}]': finished 2024-03-30T14:58:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25525 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi028-93115-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:58:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "overlaypool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25624 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25654 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi028-92979-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25666 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: osdmap e129: 8 total, 8 up, 8 in 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:12.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]: dispatch 2024-03-30T14:58:12.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:12.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:12 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:12.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:12.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:12.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:12.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:13.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:13.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:13.082 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:13.082 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:13.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:13.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:13.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:13.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:13.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:13.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:13.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:13.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:58:13.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:13.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T14:58:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:13.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:13.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:14.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.179+0000 7ff72c450700 1 -- 172.21.15.28:0/2530322646 >> v1:172.21.15.28:6790/0 conn(0x7ff724110c60 legacy=0x7ff72410ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.181+0000 7ff72c450700 1 -- 172.21.15.28:0/2530322646 shutdown_connections 2024-03-30T14:58:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.181+0000 7ff72c450700 1 -- 172.21.15.28:0/2530322646 >> 172.21.15.28:0/2530322646 conn(0x7ff72406d0f0 msgr2=0x7ff72406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.182+0000 7ff72c450700 1 -- 172.21.15.28:0/2530322646 shutdown_connections 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.185+0000 7ff72c450700 1 -- 172.21.15.28:0/2530322646 wait complete. 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 librados: starting msgr at 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 librados: starting objecter 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 Processor -- start 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 -- start start 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 librados: setting wanted keys 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 librados: calling monclient init 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff72410df40 con 0x7ff72411da40 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7241b7460 con 0x7ff724110790 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.186+0000 7ff72c450700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff7241b8640 con 0x7ff724110c60 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff7299eb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff724110c60 0x7ff72410b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42926/0 (socket says 172.21.15.28:42926) 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff72a1ec700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff724110790 0x7ff72410ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:39494/0 (socket says 172.21.15.28:39494) 2024-03-30T14:58:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff7299eb700 1 -- 172.21.15.28:0/1882399255 learned_addr learned my addr 172.21.15.28:0/1882399255 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff71b7fe700 10 client.?.objecter ms_handle_connect 0x7ff724110c60 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff71b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff71b7fe700 10 client.?.objecter ms_handle_connect 0x7ff724110790 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.187+0000 7ff71b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.190+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4197271833 0 0) 0x7ff7241b7460 con 0x7ff724110790 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.190+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7100035b0 con 0x7ff724110790 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.190+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708707814 0 0) 0x7ff7241b8640 con 0x7ff724110c60 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.190+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7241b7460 con 0x7ff724110c60 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 10 client.?.objecter ms_handle_connect 0x7ff72411da40 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1692228735 0 0) 0x7ff7241b7460 con 0x7ff724110c60 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7241b8640 con 0x7ff724110c60 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1759893958 0 0) 0x7ff7100035b0 con 0x7ff724110790 2024-03-30T14:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7241b7460 con 0x7ff724110790 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff71c004830 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.193+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff7200017b0 con 0x7ff724110790 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 690140905 0 0) 0x7ff7241b8640 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 >> v1:172.21.15.134:6789/0 conn(0x7ff724110790 legacy=0x7ff72410ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 >> v1:172.21.15.28:6789/0 conn(0x7ff72411da40 legacy=0x7ff7241b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7241b9820 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff72c450700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7241b7690 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff72c450700 10 client.36395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff72c450700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7241b7b50 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.194+0000 7ff72c450700 1 librados: init done 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff71c001e60 con 0x7ff724110c60 2024-03-30T14:58:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff71c005200 con 0x7ff724110c60 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff72c450700 10 client.36395.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff72c450700 15 client.36395.objecter _session_command_op_assign -1 1 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff72c450700 20 client.36395.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff72c450700 10 client.36395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.195+0000 7ff72c450700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff72406e280 con 0x7ff724110c60 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.198+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff71c0053c0 con 0x7ff724110c60 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.198+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(130..130 src has 53..130) ==== 14499+0+0 (unknown 2456466862 0 0) 0x7ff71c08b4d0 con 0x7ff724110c60 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.198+0000 7ff71b7fe700 10 client.36395.objecter ms_dispatch 0x7ff724001550 osd_map(130..130 src has 53..130) 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.198+0000 7ff71b7fe700 3 client.36395.objecter handle_osd_map got epochs [130,130] > 0 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.198+0000 7ff71b7fe700 3 client.36395.objecter handle_osd_map decoding full epoch 130 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 10 client.36395.objecter checking command 1 2024-03-30T14:58:14.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter _get_session s=0x7ff71008af10 osd=0 3 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter put_session s=0x7ff71008af10 osd=0 3 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 15 client.36395.objecter _session_command_op_remove -1 1 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter _get_session s=0x7ff71008af10 osd=0 3 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter get_session s=0x7ff71008af10 osd=0 3 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 15 client.36395.objecter _session_command_op_assign 0 1 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter put_session s=0x7ff71008af10 osd=0 4 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 10 client.36395.objecter _send_command 1 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff71008f310 con 0x7ff71008b5c0 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 20 client.36395.objecter dump_active .. 0 homeless 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.199+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (unknown 199063836 0 0) 0x7ff71c08b8a0 con 0x7ff724110c60 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.200+0000 7ff71b7fe700 10 client.36395.objecter ms_handle_connect 0x7ff710070460 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.200+0000 7ff71b7fe700 10 client.36395.objecter ms_handle_connect 0x7ff71008b5c0 2024-03-30T14:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.201+0000 7ff71b7fe700 1 -- 172.21.15.28:0/1882399255 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff71008f310 con 0x7ff71008b5c0 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.201+0000 7ff71b7fe700 10 client.36395.objecter ms_dispatch 0x7ff724001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.201+0000 7ff71b7fe700 10 client.36395.objecter _finish_command 1 = system:0 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.201+0000 7ff71b7fe700 20 client.36395.objecter put_session s=0x7ff71008af10 osd=0 4 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:13.201+0000 7ff71b7fe700 15 client.36395.objecter _session_command_op_remove 0 1 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:14.036 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:14.037 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:14.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: pgmap v132: 776 pgs: 1 active+clean+snaptrim, 276 unknown, 499 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2024-03-30T14:58:14.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]': finished 2024-03-30T14:58:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: pgmap v132: 776 pgs: 1 active+clean+snaptrim, 276 unknown, 499 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T14:58:14.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:14.042 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]': finished 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.043 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.045 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:13 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: Running main() from gmock_main.cc 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] Global test environment set-up. 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-30T14:58:14.081 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (293 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.LockShared (5 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1108 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1057 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2024-03-30T14:58:14.082 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] 8 tests from LibRadosLock (2481 ms total) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (2190 ms) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (11 ms) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1109 ms) 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-30T14:58:14.083 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1095 ms) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (13 ms) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (5 ms) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (7 ms) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (4433 ms total) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [----------] Global test environment tear-down 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (29663 ms total) 2024-03-30T14:58:14.084 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-30T14:58:14.099 INFO:tasks.workunit.client.0.smithi028.stdout: pool: Running main() from gmock_main.cc 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [----------] Global test environment set-up. 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolList (10496 ms) 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3166 ms) 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4649 ms) 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5898 ms) 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-30T14:58:14.100 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2910 ms) 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2165 ms) 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [----------] 6 tests from NeoRadosPools (29285 ms total) 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [----------] Global test environment tear-down 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [==========] 6 tests from 1 test suite ran. (29285 ms total) 2024-03-30T14:58:14.101 INFO:tasks.workunit.client.0.smithi028.stdout: pool: [ PASSED ] 6 tests. 2024-03-30T14:58:14.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: seed 93116 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (290 ms) 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-30T14:58:14.130 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (3 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1100 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1051 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (15 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2471 ms total) 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: 2024-03-30T14:58:14.131 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (2187 ms) 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (22 ms) 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1114 ms) 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1085 ms) 2024-03-30T14:58:14.132 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (12 ms) 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (5 ms) 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (6 ms) 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (4435 ms total) 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: 2024-03-30T14:58:14.133 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-30T14:58:14.134 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (29713 ms total) 2024-03-30T14:58:14.134 INFO:tasks.workunit.client.0.smithi028.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-30T14:58:14.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: pgmap v132: 776 pgs: 1 active+clean+snaptrim, 276 unknown, 499 active+clean; 145 MiB data, 763 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2024-03-30T14:58:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T14:58:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T14:58:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93540-6", "mode": "writeback"}]': finished 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3286821987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3476545792' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2560162111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: osdmap e130: 8 total, 8 up, 8 in 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]: dispatch 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]: dispatch 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:14.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:13 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]: dispatch 2024-03-30T14:58:14.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:14.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:14.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:14.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:14.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:14.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:14.793 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:14.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-30T14:58:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:14.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:14.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:14.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:15.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:15.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:15.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:15.237 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-30T14:58:15.237 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-30T14:58:15.237 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-30T14:58:15.237 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5033 ms) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (42 ms) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20030 ms) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: attempt 0 of 20 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5459 ms) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30564 ms total) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30565 ms total) 2024-03-30T14:58:15.238 INFO:tasks.workunit.client.0.smithi028.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (287 ms) 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94340249387568 err -107 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1039 ms) 2024-03-30T14:58:15.246 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94340250106752 err -107 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (18 ms) 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239041 cookie 94340250106752 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (10 ms) 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239041 cookie 94340250106752 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (11 ms) 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239041 cookie 94340250106752 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (11 ms) 2024-03-30T14:58:15.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239042 cookie 94340250106752 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239042 cookie 94340250164512 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (12 ms) 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239041 cookie 94340250106752 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 493921239042 cookie 94340250106752 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3012 ms) 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94340250106752 err -107 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_cb from 24907 notify_id 523986010117 cookie 94340250106752 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-30T14:58:15.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94340250106752 err -107 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10412 ms total) 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1189 ms) 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1189 ms total) 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (30731 ms total) 2024-03-30T14:58:15.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-30T14:58:15.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:15.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:15.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T14:58:15.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: 3.b scrub ok 2024-03-30T14:58:15.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:15.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1826953898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1786221605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: 3.b scrub ok 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1826953898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1786221605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:15.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: 3.b scrub ok 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.35450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi028-93086-10"}]': finished 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.35504 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi028-94235-2"}]': finished 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi028-93116-10"}]': finished 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: osdmap e131: 8 total, 8 up, 8 in 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1425420443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]: dispatch 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1826953898' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1786221605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:15.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:15.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:15.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:15.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:15.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:15.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:15.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:15.986 INFO:tasks.workunit.client.0.smithi028.stdout:vice: 2024-03-30T14:58:15.986 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: data: 2024-03-30T14:58:15.986 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pools: 8 pools, 196 pgs 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pgs: 32.653% pgs unknown 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 131 active+clean 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 64 unknown 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 1 active+clean+scrubbing 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: io: 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: cluster: 2024-03-30T14:58:15.987 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: id: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: health: HEALTH_WARN 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 18 pool(s) do not have an application enabled 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: services: 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: flags noscrub,nodeep-scrub 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: data: 2024-03-30T14:58:15.988 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pools: 28 pools, 836 pgs 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: objects: 199 objects, 582 KiB 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: usage: 247 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: pgs: 9.689% pgs unknown 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 70.335% pgs not active 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 588 creating+peering 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 167 active+clean 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 81 unknown 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: io: 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: client: 1.4 KiB/s rd, 140 B/s wr, 1 op/s rd, 0 op/s wr 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6210 ms) 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-30T14:58:15.989 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ OK ] LibRadosService.Status (20032 ms) 2024-03-30T14:58:15.990 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [----------] 4 tests from LibRadosService (31334 ms total) 2024-03-30T14:58:15.990 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: 2024-03-30T14:58:15.990 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [----------] Global test environment tear-down 2024-03-30T14:58:15.990 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31334 ms total) 2024-03-30T14:58:15.990 INFO:tasks.workunit.client.0.smithi028.stdout: api_service: [ PASSED ] 4 tests. 2024-03-30T14:58:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:16.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:16.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:16.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:16.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:16.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:16.326 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-30T14:58:16.326 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T14:58:16.326 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-30T14:58:16.326 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-30T14:58:16.326 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2402 ms) 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (4 ms) 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2406 ms total) 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (158 ms) 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5135 ms) 2024-03-30T14:58:16.327 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (16 ms) 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (30 ms) 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637069872 notify_id 558345748482 notifier_gid 35510 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (13 ms) 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637090976 notify_id 558345748483 notifier_gid 35510 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (14 ms) 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637094048 notify_id 558345748484 notifier_gid 35510 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (21 ms) 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637091984 notify_id 558345748487 notifier_gid 35510 2024-03-30T14:58:16.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (15 ms) 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637094448 notify_id 558345748480 notifier_gid 35510 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (24 ms) 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637090976 notify_id 558345748488 notifier_gid 35510 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (9 ms) 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: trying... 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637090976 notify_id 558345748489 notifier_gid 35510 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: timed out 2024-03-30T14:58:16.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: flushing 2024-03-30T14:58:16.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:16.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:16.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:16.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:16.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:16.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:16.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:16.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:16.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:17.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:17.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:17.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: pgmap v135: 712 pgs: 64 creating+peering, 1 active+clean+snaptrim, 91 unknown, 556 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 8 op/s 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2201312551' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:17.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: pool 'PoolQuotaPP_smithi028-92999-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: pgmap v135: 712 pgs: 64 creating+peering, 1 active+clean+snaptrim, 91 unknown, 556 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 8 op/s 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:17.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2201312551' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: pool 'PoolQuotaPP_smithi028-92999-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:58:17.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:16 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:17.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:17.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:17.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: pgmap v135: 712 pgs: 64 creating+peering, 1 active+clean+snaptrim, 91 unknown, 556 active+clean; 145 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.2 KiB/s wr, 8 op/s 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi028-93417-12"}]': finished 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: osdmap e132: 8 total, 8 up, 8 in 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.25696 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2201312551' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: pool 'PoolQuotaPP_smithi028-92999-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-30T14:58:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:16 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:17.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:17.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:17.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:17.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:17.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:17.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:18.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:18.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:18.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:18.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:18.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:18.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:18.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:18.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:18.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:18.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:18.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:18.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:18.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.134:0/1080188093' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: pgmap v138: 616 pgs: 64 creating+peering, 160 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:18.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1530822932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1222023213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3216563079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.27412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T14:58:18.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:18.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.27481 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.27442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:18.989 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T14:58:18.989 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T14:58:18.989 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.134:0/1080188093' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: pgmap v138: 616 pgs: 64 creating+peering, 160 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1530822932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:18.990 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1222023213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3216563079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.27412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.27481 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.27442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:18.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:19.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:19.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:19.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T14:58:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T14:58:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.134:0/1080188093' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-30T14:58:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T14:58:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: pgmap v138: 616 pgs: 64 creating+peering, 160 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.25696 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi028-93441-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1530822932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: osdmap e133: 8 total, 8 up, 8 in 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi028-94235-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: osdmap e134: 8 total, 8 up, 8 in 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1222023213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3216563079' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.27412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.27481 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.27442 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:19.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:19.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:19.468 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:19.468 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:19.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:19.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:19.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:19.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:19.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:19.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:19.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:19.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:20.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:20.120 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:20.120 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: pgmap v141: 744 pgs: 32 creating+peering, 320 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:20.120 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.27412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.120 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.27481 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.120 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.27442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1931842706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.121 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.122 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:20 smithi134 ceph-mon[33714]: from='client.40931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:20.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:20.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:20.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:20.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T14:58:20.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:20.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:20.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:20.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: pgmap v141: 744 pgs: 32 creating+peering, 320 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.27412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.27481 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.27442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1931842706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[29876]: from='client.40931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: pgmap v141: 744 pgs: 32 creating+peering, 320 unknown, 392 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.27412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.27481 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi028-92979-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.27442 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi028-93525-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:20.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: osdmap e135: 8 total, 8 up, 8 in 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi028-94235-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: osdmap e136: 8 total, 8 up, 8 in 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1931842706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:20.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:20 smithi028 ceph-mon[41614]: from='client.40931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.523+0000 7fab12309700 1 -- 172.21.15.28:0/3419997015 >> v1:172.21.15.134:6789/0 conn(0x7fab0c106d80 legacy=0x7fab0c113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.524+0000 7fab12309700 1 -- 172.21.15.28:0/3419997015 shutdown_connections 2024-03-30T14:58:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.524+0000 7fab12309700 1 -- 172.21.15.28:0/3419997015 >> 172.21.15.28:0/3419997015 conn(0x7fab0c0fc9e0 msgr2=0x7fab0c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.526+0000 7fab12309700 1 -- 172.21.15.28:0/3419997015 shutdown_connections 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.526+0000 7fab12309700 1 -- 172.21.15.28:0/3419997015 wait complete. 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 librados: starting msgr at 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 librados: starting objecter 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 Processor -- start 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 -- start start 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 librados: setting wanted keys 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.527+0000 7fab12309700 1 librados: calling monclient init 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.528+0000 7fab12309700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab0c1b62f0 con 0x7fab0c115290 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.528+0000 7fab12309700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab0c1b74d0 con 0x7fab0c100f80 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.528+0000 7fab12309700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fab0c1b8710 con 0x7fab0c1b24e0 2024-03-30T14:58:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.528+0000 7fab0bfff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fab0c100f80 0x7fab0c114ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:45794/0 (socket says 172.21.15.28:45794) 2024-03-30T14:58:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.528+0000 7fab0bfff700 1 -- 172.21.15.28:0/768504986 learned_addr learned my addr 172.21.15.28:0/768504986 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 10 client.?.objecter ms_handle_connect 0x7fab0c100f80 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 10 client.?.objecter ms_handle_connect 0x7fab0c1b24e0 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443139 0 0) 0x7fab0c1b74d0 con 0x7fab0c100f80 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaf40031f0 con 0x7fab0c100f80 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331055652 0 0) 0x7fab0c1b8710 con 0x7fab0c1b24e0 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab0c1b74d0 con 0x7fab0c1b24e0 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 529406814 0 0) 0x7faaf40031f0 con 0x7fab0c100f80 2024-03-30T14:58:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab0c1b8710 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.531+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faafc001830 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2240452993 0 0) 0x7fab0c1b8710 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6790/0 conn(0x7fab0c1b24e0 legacy=0x7fab0c1b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6789/0 conn(0x7fab0c115290 legacy=0x7fab0c072cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c1b9950 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab12309700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c1b88e0 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab12309700 10 client.33469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab12309700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab0c1b8da0 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab12309700 1 librados: init done 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.532+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7faafc003b40 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.533+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faafc005cb0 con 0x7fab0c100f80 2024-03-30T14:58:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.533+0000 7fab12309700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7faaf8005100 con 0x7fab0c100f80 2024-03-30T14:58:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.535+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7faafc0036a0 con 0x7fab0c100f80 2024-03-30T14:58:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(137..137 src has 53..137) ==== 14700+0+0 (unknown 4167883058 0 0) 0x7faafc08b1a0 con 0x7fab0c100f80 2024-03-30T14:58:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 osd_map(137..137 src has 53..137) 2024-03-30T14:58:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 3 client.33469.objecter handle_osd_map got epochs [137,137] > 0 2024-03-30T14:58:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 3 client.33469.objecter handle_osd_map decoding full epoch 137 2024-03-30T14:58:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 10 client.33469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=138}) -- 0x7fab0c1b8710 con 0x7fab0c100f80 2024-03-30T14:58:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 20 client.33469.objecter dump_active .. 0 homeless 2024-03-30T14:58:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.536+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v137) ==== 54+0+16 (unknown 1090232970 0 2420000192) 0x7faafc08d5c0 con 0x7fab0c100f80 2024-03-30T14:58:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.540+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf4070220 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 1 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 1 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 0 1 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 4 2024-03-30T14:58:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 10 client.33469.objecter _send_command 1 2024-03-30T14:58:20.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.545+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaf8002310 con 0x7faaf8005af0 2024-03-30T14:58:20.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.546+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf8005af0 2024-03-30T14:58:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.549+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf8002310 con 0x7faaf8005af0 2024-03-30T14:58:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.549+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.549+0000 7fab097fa700 10 client.33469.objecter _finish_command 1 = system:0 2024-03-30T14:58:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.549+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 4 2024-03-30T14:58:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.549+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 0 1 2024-03-30T14:58:20.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:20.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 2 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 2 2024-03-30T14:58:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf80053e0 osd=0 3 2024-03-30T14:58:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 0 2 2024-03-30T14:58:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 4 2024-03-30T14:58:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 10 client.33469.objecter _send_command 2 2024-03-30T14:58:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.594+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf80018e0 con 0x7faaf8005af0 2024-03-30T14:58:20.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.599+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf80018e0 con 0x7faaf8005af0 2024-03-30T14:58:20.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.599+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.599+0000 7fab097fa700 10 client.33469.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.599+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 4 2024-03-30T14:58:20.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.599+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 0 2 2024-03-30T14:58:20.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 3 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 3 2024-03-30T14:58:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 1 3 2024-03-30T14:58:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 4 2024-03-30T14:58:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 10 client.33469.objecter _send_command 3 2024-03-30T14:58:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.604+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7faaf8004f30 con 0x7faaf800a780 2024-03-30T14:58:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.608+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf800a780 2024-03-30T14:58:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.613+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf8004f30 con 0x7faaf800a780 2024-03-30T14:58:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.613+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 3: 0 ) 2024-03-30T14:58:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.613+0000 7fab097fa700 10 client.33469.objecter _finish_command 3 = system:0 2024-03-30T14:58:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.613+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 4 2024-03-30T14:58:20.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.613+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 1 3 2024-03-30T14:58:20.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 4 2024-03-30T14:58:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 4 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8004700 osd=1 3 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 1 4 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 4 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 10 client.33469.objecter _send_command 4 2024-03-30T14:58:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.661+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf80042a0 con 0x7faaf800a780 2024-03-30T14:58:20.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf80042a0 con 0x7faaf800a780 2024-03-30T14:58:20.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7fab097fa700 10 client.33469.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 4 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 1 4 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 5 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.664+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 5 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 2 5 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 4 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 10 client.33469.objecter _send_command 5 2024-03-30T14:58:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.665+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7faaf800ff80 con 0x7faaf800f410 2024-03-30T14:58:20.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.668+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf800f410 2024-03-30T14:58:20.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.671+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf800ff80 con 0x7faaf800f410 2024-03-30T14:58:20.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.671+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 5: 0 ) 2024-03-30T14:58:20.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.671+0000 7fab097fa700 10 client.33469.objecter _finish_command 5 = system:0 2024-03-30T14:58:20.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.671+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 4 2024-03-30T14:58:20.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.671+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 2 5 2024-03-30T14:58:20.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:20.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 6 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 6 2024-03-30T14:58:20.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8004500 osd=2 3 2024-03-30T14:58:20.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 2 6 2024-03-30T14:58:20.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.707+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 4 2024-03-30T14:58:20.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.708+0000 7faaf2ffd700 10 client.33469.objecter _send_command 6 2024-03-30T14:58:20.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.708+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf8010980 con 0x7faaf800f410 2024-03-30T14:58:20.718 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:58:20.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.713+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf8010980 con 0x7faaf800f410 2024-03-30T14:58:20.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.714+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.714+0000 7fab097fa700 10 client.33469.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.714+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 4 2024-03-30T14:58:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.714+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 2 6 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 7 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 7 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 3 7 2024-03-30T14:58:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 4 2024-03-30T14:58:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 10 client.33469.objecter _send_command 7 2024-03-30T14:58:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7faaf8017cc0 con 0x7faaf80141b0 2024-03-30T14:58:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.718+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf80141b0 2024-03-30T14:58:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.720+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf8017cc0 con 0x7faaf80141b0 2024-03-30T14:58:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.720+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 7: 0 ) 2024-03-30T14:58:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.720+0000 7fab097fa700 10 client.33469.objecter _finish_command 7 = system:0 2024-03-30T14:58:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.720+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 4 2024-03-30T14:58:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.720+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 3 7 2024-03-30T14:58:20.766 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:58:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 8 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 8 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8009d50 osd=3 3 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 3 8 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 4 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 10 client.33469.objecter _send_command 8 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.760+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf8010520 con 0x7faaf80141b0 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf8010520 con 0x7faaf80141b0 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7fab097fa700 10 client.33469.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 4 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 3 8 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 9 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.763+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 9 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 4 9 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 4 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 10 client.33469.objecter _send_command 9 2024-03-30T14:58:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7faaf800f130 con 0x7faaf80102d0 2024-03-30T14:58:20.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.764+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf80102d0 2024-03-30T14:58:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.765+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf800f130 con 0x7faaf80102d0 2024-03-30T14:58:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.765+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 9: 0 ) 2024-03-30T14:58:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.765+0000 7fab097fa700 10 client.33469.objecter _finish_command 9 = system:0 2024-03-30T14:58:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.765+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 4 2024-03-30T14:58:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.765+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 4 9 2024-03-30T14:58:20.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 10 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 10 2024-03-30T14:58:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf801a630 osd=4 3 2024-03-30T14:58:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 4 10 2024-03-30T14:58:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 4 2024-03-30T14:58:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 10 client.33469.objecter _send_command 10 2024-03-30T14:58:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.800+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf800f130 con 0x7faaf80102d0 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf800f130 con 0x7faaf80102d0 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7fab097fa700 10 client.33469.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 4 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 4 10 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 11 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 11 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.803+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 5 11 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.804+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 4 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.804+0000 7faaf2ffd700 10 client.33469.objecter _send_command 11 2024-03-30T14:58:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.804+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7faaf8020900 con 0x7faaf801cd20 2024-03-30T14:58:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.810+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf801cd20 2024-03-30T14:58:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.811+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf8020900 con 0x7faaf801cd20 2024-03-30T14:58:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.811+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 11: 0 ) 2024-03-30T14:58:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.811+0000 7fab097fa700 10 client.33469.objecter _finish_command 11 = system:0 2024-03-30T14:58:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.811+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 4 2024-03-30T14:58:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.811+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 5 11 2024-03-30T14:58:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 12 2024-03-30T14:58:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 12 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf800f130 osd=5 3 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 5 12 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 4 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 10 client.33469.objecter _send_command 12 2024-03-30T14:58:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.850+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf800e4e0 con 0x7faaf801cd20 2024-03-30T14:58:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.852+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf800e4e0 con 0x7faaf801cd20 2024-03-30T14:58:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.852+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.852+0000 7fab097fa700 10 client.33469.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.852+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 4 2024-03-30T14:58:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.852+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 5 12 2024-03-30T14:58:20.859 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 13 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 13 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 6 13 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 4 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 10 client.33469.objecter _send_command 13 2024-03-30T14:58:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.858+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7faaf80220c0 con 0x7faaf800e290 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.859+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf800e290 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.860+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fab0400a1d0 con 0x7faaf800e290 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.860+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 13: 0 ) 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.860+0000 7fab097fa700 10 client.33469.objecter _finish_command 13 = system:0 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.860+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 4 2024-03-30T14:58:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.860+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 6 13 2024-03-30T14:58:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 14 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 14 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8001be0 osd=6 3 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 6 14 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 4 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 10 client.33469.objecter _send_command 14 2024-03-30T14:58:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.908+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf8018a70 con 0x7faaf800e290 2024-03-30T14:58:20.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:58:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.914+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf8018a70 con 0x7faaf800e290 2024-03-30T14:58:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.914+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.914+0000 7fab097fa700 10 client.33469.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.914+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 4 2024-03-30T14:58:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.914+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 6 14 2024-03-30T14:58:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 15 2024-03-30T14:58:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 15 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 7 15 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 4 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 10 client.33469.objecter _send_command 15 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.916+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7faaf80294d0 con 0x7faaf80259c0 2024-03-30T14:58:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.919+0000 7fab097fa700 10 client.33469.objecter ms_handle_connect 0x7faaf80259c0 2024-03-30T14:58:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.923+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faaf80294d0 con 0x7faaf80259c0 2024-03-30T14:58:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.923+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 15: 0 ) 2024-03-30T14:58:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.923+0000 7fab097fa700 10 client.33469.objecter _finish_command 15 = system:0 2024-03-30T14:58:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.923+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 4 2024-03-30T14:58:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.923+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 7 15 2024-03-30T14:58:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 10 client.33469.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign -1 16 2024-03-30T14:58:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 20 client.33469.objecter _get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_remove -1 16 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 20 client.33469.objecter get_session s=0x7faaf8018a70 osd=7 3 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 15 client.33469.objecter _session_command_op_assign 7 16 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 4 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 10 client.33469.objecter _send_command 16 2024-03-30T14:58:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.961+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7faaf800e6f0 con 0x7faaf80259c0 2024-03-30T14:58:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.965+0000 7fab097fa700 1 -- 172.21.15.28:0/768504986 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faaf800e6f0 con 0x7faaf80259c0 2024-03-30T14:58:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.965+0000 7fab097fa700 10 client.33469.objecter ms_dispatch 0x7fab0c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.965+0000 7fab097fa700 10 client.33469.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.965+0000 7fab097fa700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 4 2024-03-30T14:58:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.965+0000 7fab097fa700 15 client.33469.objecter _session_command_op_remove 7 16 2024-03-30T14:58:20.969 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 librados: watch_flush enter 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 librados: watch_flush exit 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.0 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6801/3394323575 conn(0x7faaf8005af0 legacy=0x7faaf8007ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf80053e0 osd=0 1 2024-03-30T14:58:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.1 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6805/1850313547 conn(0x7faaf800a780 legacy=0x7faaf800cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004700 osd=1 1 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.2 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6809/3451325043 conn(0x7faaf800f410 legacy=0x7faaf800f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8004500 osd=2 1 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.969+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.3 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6813/1506157250 conn(0x7faaf80141b0 legacy=0x7faaf80165a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8009d50 osd=3 1 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.4 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7fab108a6700 1 -- 172.21.15.28:0/768504986 reap_dead start 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7fab108a6700 1 -- 172.21.15.28:0/768504986 reap_dead start 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.134:6800/3063775514 conn(0x7faaf80102d0 legacy=0x7faaf8018e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf801a630 osd=4 1 2024-03-30T14:58:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.5 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.134:6804/3668151179 conn(0x7faaf801cd20 legacy=0x7faaf801f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf800f130 osd=5 1 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.6 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.134:6808/2135224194 conn(0x7faaf800e290 legacy=0x7faaf8020900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8001be0 osd=6 1 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 10 client.33469.objecter close_session for osd.7 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.134:6812/3758190298 conn(0x7faaf80259c0 legacy=0x7faaf8027db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter put_session s=0x7faaf8018a70 osd=7 1 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 20 client.33469.objecter shutdown clearing up homeless session... 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.970+0000 7faaf2ffd700 10 client.33469.objecter successfully canceled tick 2024-03-30T14:58:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.972+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.28:6800/3011431093 conn(0x7faaf4070220 legacy=0x7faaf40726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.972+0000 7fab108a6700 1 -- 172.21.15.28:0/768504986 reap_dead start 2024-03-30T14:58:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.972+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> v1:172.21.15.134:6789/0 conn(0x7fab0c100f80 legacy=0x7fab0c114ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.975+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 shutdown_connections 2024-03-30T14:58:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.975+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 >> 172.21.15.28:0/768504986 conn(0x7fab0c0fc9e0 msgr2=0x7fab0c117680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.977+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 shutdown_connections 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.977+0000 7faaf2ffd700 1 -- 172.21.15.28:0/768504986 wait complete. 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:20.977+0000 7faaf2ffd700 1 librados: shutdown 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:58:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:58:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:58:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:58:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:58:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:58:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:21.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:21.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:21.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 5.1b scrub starts 2024-03-30T14:58:21.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 5.1b scrub ok 2024-03-30T14:58:21.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: pgmap v144: 624 pgs: 32 creating+peering, 40 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.40931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:21.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4179484271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/768504986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.32200 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/680296349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:21.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='client.32242 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 5.1b scrub starts 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 5.1b scrub ok 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: pgmap v144: 624 pgs: 32 creating+peering, 40 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T14:58:21.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.40931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4179484271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/768504986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.32200 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/680296349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='client.32242 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:21 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:21.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:21.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:21.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:21.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 5.1b scrub starts 2024-03-30T14:58:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 5.1b scrub ok 2024-03-30T14:58:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T14:58:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T14:58:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: pgmap v144: 624 pgs: 32 creating+peering, 40 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.40931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: osdmap e137: 8 total, 8 up, 8 in 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4179484271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T14:58:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/768504986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: osdmap e138: 8 total, 8 up, 8 in 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.32200 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/680296349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='client.32242 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:21.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:21 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:21.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:21.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.524+0000 7f43c8028700 1 -- 172.21.15.28:0/3151910296 >> v1:172.21.15.134:6789/0 conn(0x7f43c010e560 legacy=0x7f43c010e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:21.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.530+0000 7f43c8028700 1 -- 172.21.15.28:0/3151910296 shutdown_connections 2024-03-30T14:58:21.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.530+0000 7f43c8028700 1 -- 172.21.15.28:0/3151910296 >> 172.21.15.28:0/3151910296 conn(0x7f43c006d0f0 msgr2=0x7f43c006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:21.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.533+0000 7f43c8028700 1 -- 172.21.15.28:0/3151910296 shutdown_connections 2024-03-30T14:58:21.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:21.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:21.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.534+0000 7f43c8028700 1 -- 172.21.15.28:0/3151910296 wait complete. 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 librados: starting msgr at 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 librados: starting objecter 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 Processor -- start 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 -- start start 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 librados: setting wanted keys 2024-03-30T14:58:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 librados: calling monclient init 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43c010def0 con 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43c013abf0 con 0x7f43c007a070 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.535+0000 7f43c8028700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43c013bdd0 con 0x7f43c0076490 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43c65c5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f43c010e560 0x7f43c01384c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33482/0 (socket says 172.21.15.28:33482) 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43c65c5700 1 -- 172.21.15.28:0/440858253 learned_addr learned my addr 172.21.15.28:0/440858253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43b6ffd700 10 client.?.objecter ms_handle_connect 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43b6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263840692 0 0) 0x7f43c010def0 con 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.537+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43ac003810 con 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3742235162 0 0) 0x7f43ac003810 con 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43c010def0 con 0x7f43c010e560 2024-03-30T14:58:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f43b0004010 con 0x7f43c010e560 2024-03-30T14:58:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2255701269 0 0) 0x7f43c010def0 con 0x7f43c010e560 2024-03-30T14:58:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 >> v1:172.21.15.28:6790/0 conn(0x7f43c0076490 legacy=0x7f43c0084d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 >> v1:172.21.15.134:6789/0 conn(0x7f43c007a070 legacy=0x7f43c010d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c013cfb0 con 0x7f43c010e560 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.538+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43c013d240 con 0x7f43c010e560 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.539+0000 7f43c8028700 10 client.18885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.539+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f43c013d700 con 0x7f43c010e560 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.539+0000 7f43c8028700 1 librados: init done 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.540+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f43b0001370 con 0x7f43c010e560 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.540+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f43b0005d10 con 0x7f43c010e560 2024-03-30T14:58:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.540+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c006e280 con 0x7f43c010e560 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.541+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f43b0005f30 con 0x7f43c010e560 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.541+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(138..138 src has 53..138) ==== 15397+0+0 (unknown 2600772135 0 0) 0x7f43b008c380 con 0x7f43c010e560 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 10 client.18885.objecter ms_dispatch 0x7f43c0001550 osd_map(138..138 src has 53..138) 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map got epochs [138,138] > 0 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map decoding full epoch 138 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 10 client.18885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=139}) -- 0x7f43c010def0 con 0x7f43c010e560 2024-03-30T14:58:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.542+0000 7f43b6ffd700 20 client.18885.objecter dump_active .. 0 homeless 2024-03-30T14:58:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.543+0000 7f43b6ffd700 10 client.18885.objecter ms_handle_connect 0x7f43ac0703d0 2024-03-30T14:58:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.558+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f43b0057930 con 0x7f43c010e560 2024-03-30T14:58:21.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:21.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:21.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:21.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 10 ==== osd_map(139..139 src has 53..139) ==== 2851+0+0 (unknown 570623570 0 0) 0x7f43b0055530 con 0x7f43c010e560 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 10 client.18885.objecter ms_dispatch 0x7f43c0001550 osd_map(139..139 src has 53..139) 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map got epochs [139,139] > 138 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map decoding incremental epoch 139 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 10 client.18885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=140}) -- 0x7f43c013abf0 con 0x7f43c010e560 2024-03-30T14:58:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:21.998+0000 7f43b6ffd700 20 client.18885.objecter dump_active .. 0 homeless 2024-03-30T14:58:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:22.012+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f43c0084460 con 0x7f43c010e560 2024-03-30T14:58:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:22.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:22.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:22.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:22.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:22.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:22.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:22.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:22.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:22.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:22.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:22.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:22.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:22.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:23.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:23.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:23.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:23.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.32200 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.32242 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.32200 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.32242 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.32200 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.32242 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi028-92979-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: osdmap e139: 8 total, 8 up, 8 in 2024-03-30T14:58:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.246+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v140) ==== 101+0+0 (unknown 1217501619 0 0) 0x7f43b005b380 con 0x7f43c010e560 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.248+0000 7f43b6ffd700 1 -- 172.21.15.28:0/440858253 <== mon.0 v1:172.21.15.28:6789/0 12 ==== osd_map(140..140 src has 53..140) ==== 2457+0+0 (unknown 25122296 0 0) 0x7f43b0088380 con 0x7f43c010e560 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.248+0000 7f43b6ffd700 10 client.18885.objecter ms_dispatch 0x7f43c0001550 osd_map(140..140 src has 53..140) 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.248+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map got epochs [140,140] > 139 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.248+0000 7f43b6ffd700 3 client.18885.objecter handle_osd_map decoding incremental epoch 140 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.248+0000 7f43b6ffd700 20 client.18885.objecter dump_active .. 0 homeless 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.249+0000 7f43c8028700 10 librados: watch_flush enter 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.249+0000 7f43c8028700 10 librados: watch_flush exit 2024-03-30T14:58:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.249+0000 7f43c8028700 20 client.18885.objecter shutdown clearing up homeless session... 2024-03-30T14:58:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.249+0000 7f43c8028700 10 client.18885.objecter successfully canceled tick 2024-03-30T14:58:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.253+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 >> v1:172.21.15.28:6800/3011431093 conn(0x7f43ac0703d0 legacy=0x7f43ac072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.253+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 >> v1:172.21.15.28:6789/0 conn(0x7f43c010e560 legacy=0x7f43c01384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:23.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.257+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 shutdown_connections 2024-03-30T14:58:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.257+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 >> 172.21.15.28:0/440858253 conn(0x7f43c006d0f0 msgr2=0x7f43c0074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.257+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 shutdown_connections 2024-03-30T14:58:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.257+0000 7f43c8028700 1 -- 172.21.15.28:0/440858253 wait complete. 2024-03-30T14:58:23.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:23.257+0000 7f43c8028700 1 librados: shutdown 2024-03-30T14:58:23.282 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T14:58:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:23.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:23.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:23.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:23.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:23.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:23.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:24.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:24.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: pgmap v147: 784 pgs: 32 creating+peering, 200 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T14:58:24.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[29876]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:58:24.216 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: pgmap v147: 784 pgs: 32 creating+peering, 200 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T14:58:24.217 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:24.218 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:23 smithi028 ceph-mon[41614]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:58:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:24.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:24.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: pgmap v147: 784 pgs: 32 creating+peering, 200 unknown, 552 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 2 op/s 2024-03-30T14:58:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T14:58:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T14:58:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]: dispatch 2024-03-30T14:58:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T14:58:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/440858253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-7"}]': finished 2024-03-30T14:58:24.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:23 smithi134 ceph-mon[33714]: osdmap e140: 8 total, 8 up, 8 in 2024-03-30T14:58:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:24.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:24.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:24.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:24.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:24.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:24.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:25.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:25.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:25.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.091+0000 7f3e2e08c700 1 -- 172.21.15.28:0/3340706694 >> v1:172.21.15.28:6790/0 conn(0x7f3e281090f0 legacy=0x7f3e2810b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.093+0000 7f3e2e08c700 1 -- 172.21.15.28:0/3340706694 shutdown_connections 2024-03-30T14:58:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.093+0000 7f3e2e08c700 1 -- 172.21.15.28:0/3340706694 >> 172.21.15.28:0/3340706694 conn(0x7f3e280772c0 msgr2=0x7f3e280776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.093+0000 7f3e2e08c700 1 -- 172.21.15.28:0/3340706694 shutdown_connections 2024-03-30T14:58:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.095+0000 7f3e2e08c700 1 -- 172.21.15.28:0/3340706694 wait complete. 2024-03-30T14:58:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 librados: starting msgr at 2024-03-30T14:58:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 librados: starting objecter 2024-03-30T14:58:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 Processor -- start 2024-03-30T14:58:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 -- start start 2024-03-30T14:58:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 librados: setting wanted keys 2024-03-30T14:58:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.096+0000 7f3e2e08c700 1 librados: calling monclient init 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e2e08c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3e281ad630 con 0x7f3e2810cc90 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e2e08c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3e281ae850 con 0x7f3e28100460 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e2e08c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3e281afa90 con 0x7f3e281090f0 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e27fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3e2810cc90 0x7f3e281abd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43842/0 (socket says 172.21.15.28:43842) 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e27fff700 1 -- 172.21.15.28:0/1170151920 learned_addr learned my addr 172.21.15.28:0/1170151920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e24ff9700 10 client.?.objecter ms_handle_connect 0x7f3e2810cc90 2024-03-30T14:58:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.097+0000 7f3e24ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.098+0000 7f3e24ff9700 10 client.?.objecter ms_handle_connect 0x7f3e28100460 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.098+0000 7f3e24ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.098+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355384592 0 0) 0x7f3e281ae850 con 0x7f3e28100460 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.098+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e100035b0 con 0x7f3e28100460 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 10 client.?.objecter ms_handle_connect 0x7f3e281090f0 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172699468 0 0) 0x7f3e281afa90 con 0x7f3e281090f0 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e281ae850 con 0x7f3e281090f0 2024-03-30T14:58:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2438770913 0 0) 0x7f3e100035b0 con 0x7f3e28100460 2024-03-30T14:58:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e281afa90 con 0x7f3e28100460 2024-03-30T14:58:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.099+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3e1c001830 con 0x7f3e28100460 2024-03-30T14:58:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.100+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 267394478 0 0) 0x7f3e281ae850 con 0x7f3e281090f0 2024-03-30T14:58:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.100+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e100035b0 con 0x7f3e281090f0 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.100+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3e18004410 con 0x7f3e281090f0 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 907403506 0 0) 0x7f3e281afa90 con 0x7f3e28100460 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 >> v1:172.21.15.28:6790/0 conn(0x7f3e281090f0 legacy=0x7f3e28102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 >> v1:172.21.15.28:6789/0 conn(0x7f3e2810cc90 legacy=0x7f3e281abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e281b0cd0 con 0x7f3e28100460 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e2e08c700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e281aea80 con 0x7f3e28100460 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e2e08c700 10 client.37534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e2e08c700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e281aef40 con 0x7f3e28100460 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.101+0000 7f3e2e08c700 1 librados: init done 2024-03-30T14:58:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.102+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3e1c003b20 con 0x7f3e28100460 2024-03-30T14:58:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.102+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3e1c004ac0 con 0x7f3e28100460 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.103+0000 7f3e2e08c700 10 client.37534.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.103+0000 7f3e2e08c700 15 client.37534.objecter _session_command_op_assign -1 1 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.103+0000 7f3e2e08c700 20 client.37534.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.103+0000 7f3e2e08c700 10 client.37534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.103+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3e1c01b300 con 0x7f3e28100460 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.104+0000 7f3e2e08c700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3e14002920 con 0x7f3e28100460 2024-03-30T14:58:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.104+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(141..141 src has 53..141) ==== 12365+0+0 (unknown 72798548 0 0) 0x7f3e1c08a590 con 0x7f3e28100460 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.104+0000 7f3e24ff9700 10 client.37534.objecter ms_dispatch 0x7f3e28001550 osd_map(141..141 src has 53..141) 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.104+0000 7f3e24ff9700 3 client.37534.objecter handle_osd_map got epochs [141,141] > 0 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.104+0000 7f3e24ff9700 3 client.37534.objecter handle_osd_map decoding full epoch 141 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 10 client.37534.objecter checking command 1 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter _get_session s=0x7f3e10086590 osd=1 3 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter put_session s=0x7f3e10086590 osd=1 3 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 15 client.37534.objecter _session_command_op_remove -1 1 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter _get_session s=0x7f3e10086590 osd=1 3 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter get_session s=0x7f3e10086590 osd=1 3 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 15 client.37534.objecter _session_command_op_assign 1 1 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter put_session s=0x7f3e10086590 osd=1 4 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 10 client.37534.objecter _send_command 1 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 --> v1:172.21.15.28:6805/1850313547 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e1008aa10 con 0x7f3e10086c80 2024-03-30T14:58:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 20 client.37534.objecter dump_active .. 0 homeless 2024-03-30T14:58:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=141) ==== 24+0+0 (unknown 1472502307 0 0) 0x7f3e1c08a9a0 con 0x7f3e28100460 2024-03-30T14:58:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 10 client.37534.objecter ms_handle_connect 0x7f3e10070350 2024-03-30T14:58:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.105+0000 7f3e24ff9700 10 client.37534.objecter ms_handle_connect 0x7f3e10086c80 2024-03-30T14:58:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.108+0000 7f3e24ff9700 1 -- 172.21.15.28:0/1170151920 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3e1008aa10 con 0x7f3e10086c80 2024-03-30T14:58:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.108+0000 7f3e24ff9700 10 client.37534.objecter ms_dispatch 0x7f3e28001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.108+0000 7f3e24ff9700 10 client.37534.objecter _finish_command 1 = system:0 2024-03-30T14:58:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.108+0000 7f3e24ff9700 20 client.37534.objecter put_session s=0x7f3e10086590 osd=1 4 2024-03-30T14:58:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:25.108+0000 7f3e24ff9700 15 client.37534.objecter _session_command_op_remove 1 1 2024-03-30T14:58:25.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:25.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:25.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:25.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:25.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: pgmap v150: 688 pgs: 64 creating+peering, 1 active+clean+snaptrim, 83 unknown, 540 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: 3.d deep-scrub starts 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: 3.d deep-scrub ok 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2482478745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.35977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:25.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:25.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:25.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:25.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:25.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:25.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:26.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: pgmap v150: 688 pgs: 64 creating+peering, 1 active+clean+snaptrim, 83 unknown, 540 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: 3.d deep-scrub starts 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: 3.d deep-scrub ok 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2482478745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.35977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: pgmap v150: 688 pgs: 64 creating+peering, 1 active+clean+snaptrim, 83 unknown, 540 active+clean; 145 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: 3.d deep-scrub starts 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: 3.d deep-scrub ok 2024-03-30T14:58:26.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: osdmap e141: 8 total, 8 up, 8 in 2024-03-30T14:58:26.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2482478745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.35977 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:26.197 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:26.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-30T14:58:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:26.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:26.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:58:26.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:26.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:26.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:26.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:58:26.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.35977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/778975440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[29876]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:58:26.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.35977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/778975440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:26.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:26 smithi028 ceph-mon[41614]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:26.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:27.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T14:58:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T14:58:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]: dispatch 2024-03-30T14:58:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.35977 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3323927705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/778975440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:26 smithi134 ceph-mon[33714]: osdmap e142: 8 total, 8 up, 8 in 2024-03-30T14:58:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:27.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:27.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:27.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:27.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:27.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:27.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:27.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:27.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:27.863 INFO:tasks.workunit.client.0.smithi028.stdout: a ec_list: Running main() from gmock_main.cc 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [----------] Global test environment set-up. 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (19169 ms) 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (10108 ms) 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-30T14:58:27.864 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (13776 ms) 2024-03-30T14:58:27.865 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [----------] 3 tests from NeoradosECList (43053 ms total) 2024-03-30T14:58:27.865 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: 2024-03-30T14:58:27.865 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [----------] Global test environment tear-down 2024-03-30T14:58:27.865 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (43054 ms total) 2024-03-30T14:58:27.865 INFO:tasks.workunit.client.0.smithi028.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-30T14:58:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:27.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:27.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:28.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:28.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T14:58:28.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.6 deep-scrub starts 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.6 deep-scrub ok 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T14:58:28.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: pgmap v153: 648 pgs: 64 creating+peering, 1 active+clean+snaptrim, 160 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.46211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:28.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]': finished 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/516799775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2275516611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.46235 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.36979 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T14:58:28.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.6 deep-scrub starts 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.6 deep-scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: pgmap v153: 648 pgs: 64 creating+peering, 1 active+clean+snaptrim, 160 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.46211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]': finished 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/516799775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2275516611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.46235 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.36979 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T14:58:28.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:27 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T14:58:28.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T14:58:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.6 deep-scrub starts 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.6 deep-scrub ok 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: pgmap v153: 648 pgs: 64 creating+peering, 1 active+clean+snaptrim, 160 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T14:58:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.46211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4"}]': finished 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: osdmap e143: 8 total, 8 up, 8 in 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/516799775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2275516611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]: dispatch 2024-03-30T14:58:28.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.46235 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.36979 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T14:58:28.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:27 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T14:58:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:28.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:28.347 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:28.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:28.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:28.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:28.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:29.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:29.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:29.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:29.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:29.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:29.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.46211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.46235 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.36979 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.46211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:29.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.46235 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.36979 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.221 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.46211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.27400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi028-94235-3"}]': finished 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi028-93540-4", "tierpool": "test-rados-api-smithi028-93540-6"}]': finished 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.46235 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93757-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.36979 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi028-92979-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: osdmap e144: 8 total, 8 up, 8 in 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1362653445' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(14.1d epoch 114/111 rep_tid 87 entries 114'2 (0'0) error 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head by client.24853.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.173901+0000", 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.278388698000001, 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023963393999999999, 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173902+0000", 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7300000000000002e-07 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173911+0000", 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8720000000000008e-06 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173929+0000", 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8666000000000001e-05 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173946+0000", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6724000000000001e-05 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.193026+0000", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019079608000000001 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.197865+0000", 2024-03-30T14:58:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048391509999999999 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(15.1 epoch 114/111 rep_tid 80 entries 114'2 (0'0) error 15:8386110d:test-rados-api-smithi028-93086-4::foo:head by client.24856.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T14:58:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.207370+0000", 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.244919834000001, 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047719461999999997, 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T14:58:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207371+0000", 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4999999999999998e-07 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207383+0000", 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2333e-05 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207399+0000", 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6235999999999998e-05 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207417+0000", 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7303999999999999e-05 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.247826+0000", 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040409395000000001 2024-03-30T14:58:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.255090+0000", 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072638440000000002 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:14 11.a e115/114 11:56696d74:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head v 115'1, mlcod=0'0)", 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.259589+0000", 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.192700989000002, 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017594875999999999, 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259589+0000", 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259589+0000", 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259589+0000", 2024-03-30T14:58:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.17e-07 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259597+0000", 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9470000000000003e-06 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259602+0000", 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3560000000000003e-06 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259608+0000", 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7519999999999999e-06 2024-03-30T14:58:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.260347+0000", 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073834200000000001 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.260365+0000", 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8284999999999999e-05 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.277146+0000", 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016780997999999998 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.277184+0000", 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7978999999999997e-05 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34820.0:20 9.8 e115/114 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head v 115'1, mlcod=0'0)", 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.281149+0000", 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.171140760999997, 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021284207999999999, 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281150+0000", 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0800000000000001e-07 2024-03-30T14:58:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281157+0000", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5959999999999998e-06 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281163+0000", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4249999999999997e-06 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281170+0000", 2024-03-30T14:58:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2110000000000001e-06 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282755+0000", 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001584577 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282773+0000", 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8474e-05 2024-03-30T14:58:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302395+0000", 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019622245999999999 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302434+0000", 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8371000000000002e-05 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24832.0:15 10.1b e115/114 10:db53fca8:test-rados-api-smithi028-93032-8::foo:head v 115'2, mlcod=0'0)", 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.291026+0000", 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.161263957999999, 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01785983, 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291026+0000", 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291026+0000", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291027+0000", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0100000000000004e-07 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291033+0000", 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.631e-06 2024-03-30T14:58:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291040+0000", 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4889999999999998e-06 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.291047+0000", 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9480000000000002e-06 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294814+0000", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037671580000000001 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294835+0000", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1367999999999999e-05 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.308831+0000", 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013995623 2024-03-30T14:58:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.308886+0000", 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4911999999999999e-05 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24877.0:3 18.16 e115/114 18:6ceb5ccc:test-rados-api-smithi028-93253-2::foo:head v 115'2, mlcod=0'0)", 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.294560+0000", 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.157730469999997, 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014209472000000001, 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294560+0000", 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294560+0000", 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294560+0000", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8599999999999999e-07 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294568+0000", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6539999999999999e-06 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294573+0000", 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.395e-06 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294578+0000", 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7299999999999996e-06 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294923+0000", 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034484499999999998 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294940+0000", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7615e-05 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.308727+0000", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013786667000000001 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.308769+0000", 2024-03-30T14:58:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2179999999999999e-05 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24901.0:9 23.6 e115/114 23:602f83fe:::foo:head v 115'1, mlcod=0'0)", 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.294903+0000", 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.157387008999997, 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014112403000000001, 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294903+0000", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294903+0000", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294904+0000", 2024-03-30T14:58:29.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8599999999999998e-07 2024-03-30T14:58:29.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294908+0000", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5190000000000003e-06 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294913+0000", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2089999999999996e-06 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294920+0000", 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3500000000000002e-06 2024-03-30T14:58:29.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295629+0000", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070916499999999999 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295648+0000", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8975e-05 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.308772+0000", 2024-03-30T14:58:29.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013124277 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.309016+0000", 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000243422 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24853.0:11 14.15 e115/111 14:ada94957:test-rados-api-smithi028-93116-5::foo:head v 115'3, mlcod=115'1)", 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.372768+0000", 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 32.079521778999997, 2024-03-30T14:58:29.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038572226000000001, 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372769+0000", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0950000000000001e-06 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372783+0000", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3463e-05 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372792+0000", 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.448e-06 2024-03-30T14:58:29.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372802+0000", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7890000000000006e-06 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387040+0000", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014237757 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387062+0000", 2024-03-30T14:58:29.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1857000000000001e-05 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411293+0000", 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024231447 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411341+0000", 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7370000000000002e-05 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32242.0:2 120.6 e139/138 120:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.080886+0000", 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.3714040690000004, 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029704504999999999, 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8999999999999998e-07 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080897+0000", 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0572e-05 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080903+0000", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4659999999999996e-06 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080913+0000", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.431e-06 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097177+0000", 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016263732999999999 2024-03-30T14:58:29.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097197+0000", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0520999999999999e-05 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110549+0000", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013351633 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110591+0000", 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1859e-05 2024-03-30T14:58:29.503 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(115.6s1 139/136 ECSubWrite(tid=562, reqid=client.27400.0:256, at_version=139'28, trim_to=0'0, roll_forward_to=138'27))", 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.083614+0000", 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.3686763480000002, 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026921001, 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7399999999999999e-07 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083633+0000", 2024-03-30T14:58:29.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8479999999999999e-05 2024-03-30T14:58:29.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.506 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.506 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083641+0000", 2024-03-30T14:58:29.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7190000000000007e-06 2024-03-30T14:58:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:29.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083652+0000", 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0902e-05 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097409+0000", 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013756496 2024-03-30T14:58:29.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097418+0000", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110508+0000", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013089735999999999 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110535+0000", 2024-03-30T14:58:29.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7146000000000001e-05 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32239.0:2 119.1b e139/137 119:d96a01e0:::testobj:head v 139'1, mlcod=0'0)", 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.110557+0000", 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 7.3417326919999999, 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024229506000000001, 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110558+0000", 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3299999999999998e-07 2024-03-30T14:58:29.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110568+0000", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0029e-05 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110575+0000", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3660000000000003e-06 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110585+0000", 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3400000000000004e-06 2024-03-30T14:58:29.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111135+0000", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055055900000000005 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111154+0000", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9269000000000001e-05 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134746+0000", 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023591918999999999 2024-03-30T14:58:29.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134787+0000", 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0691000000000001e-05 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:12 104.1c e141/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:head v 141'1, mlcod=0'0)", 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:24.404245+0000", 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.0480453000000001, 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041302041999999997, 2024-03-30T14:58:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404246+0000", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5799999999999999e-06 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404257+0000", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193999999999999e-05 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404267+0000", 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0215e-05 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404276+0000", 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4520000000000003e-06 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433295+0000", 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029018773000000001 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433316+0000", 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0636999999999999e-05 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445508+0000", 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012192305000000001 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445547+0000", 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8886e-05 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.8 epoch 142/140 rep_tid 1005 entries 142'13 (0'0) error 123:1179ae53:::test_441:head by client.44099.0:448 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 142'11)", 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:26.818611+0000", 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.6336793470000002, 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038477480000000001, 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T14:58:29.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818612+0000", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.37e-07 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818622+0000", 2024-03-30T14:58:29.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0477e-05 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851833+0000", 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033211076999999999 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851867+0000", 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3874999999999997e-05 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851967+0000", 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100182 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.857088+0000", 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051211319999999996 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.1f epoch 143/140 rep_tid 611 entries 143'25 (0'0) error 123:fbee941d:::test_708:head by client.44099.0:715 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 143'23)", 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.835765+0000", 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.616524911, 2024-03-30T14:58:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.114938996, 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835768+0000", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.892e-06 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835782+0000", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3456e-05 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835803+0000", 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.101e-05 2024-03-30T14:58:29.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835820+0000", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-05 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.935436+0000", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099616275000000004 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.950704+0000", 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015268273000000001 2024-03-30T14:58:29.525 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1491 104.1c e144/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:5 v 144'6, mlcod=143'3)", 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.902235+0000", 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.5500547730000001, 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022363271000000001, 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902236+0000", 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5999999999999999e-07 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902245+0000", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9339999999999995e-06 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902253+0000", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6200000000000005e-06 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902263+0000", 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1749999999999994e-06 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903266+0000", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010038880000000001 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903291+0000", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4907999999999999e-05 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924561+0000", 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021269507 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924599+0000", 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7778999999999999e-05 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.36958.0:2 131.0 e144/143 131:0384ab8e:::oid:head v 144'1, mlcod=0'0)", 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.974896+0000", 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.477393827, 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018604233000000001, 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974896+0000", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974896+0000", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974897+0000", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0100000000000004e-07 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974904+0000", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1799999999999999e-06 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974911+0000", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7070000000000003e-06 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.974919+0000", 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4120000000000001e-06 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.981648+0000", 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067291470000000004 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.990246+0000", 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0085973669999999999 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.992270+0000", 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020240919999999999 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.993501+0000", 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001230627 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1533 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.918864+0000", 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.53342621300000004, 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046703298999999997, 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.535 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918865+0000", 2024-03-30T14:58:29.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3909999999999999e-06 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955732+0000", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036866290000000003 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955761+0000", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9051e-05 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955774+0000", 2024-03-30T14:58:29.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3448999999999999e-05 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955856+0000", 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1716000000000002e-05 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955881+0000", 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5361e-05 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965519+0000", 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096378399999999999 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965567+0000", 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8201000000000001e-05 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1535 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T14:58:29.539 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.958473+0000", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.49381666200000002, 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055252190999999999, 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T14:58:29.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958476+0000", 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2699999999999999e-06 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004606+0000", 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046129950000000003 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004628+0000", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2775000000000001e-05 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004638+0000", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3659999999999998e-06 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004730+0000", 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2026000000000003e-05 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004753+0000", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2898e-05 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013677+0000", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089245069999999999 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013726+0000", 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8399000000000002e-05 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1536 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.007695+0000", 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.44459515799999999, 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029420031999999999, 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T14:58:29.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007696+0000", 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4410000000000001e-06 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024530+0000", 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016833292 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024541+0000", 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1619e-05 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024550+0000", 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2840000000000006e-06 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024603+0000", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3566999999999997e-05 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024628+0000", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4326999999999999e-05 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037063+0000", 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012435493000000001 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037115+0000", 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2009000000000003e-05 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.3 epoch 144/143 rep_tid 1544 entries 144'5 (0'0) error 130:c0c9adc9:::test_234:head by client.46235.0:241 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 144'3)", 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.136999+0000", 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.31529151100000002, 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031151346999999999, 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:58:29.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:58:29.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137001+0000", 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1840000000000002e-06 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137017+0000", 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:58:29.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150695+0000", 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013677962 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150725+0000", 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.042e-05 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150878+0000", 2024-03-30T14:58:29.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015338299999999999 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.168150+0000", 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017271531 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:58:29.552 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:58:29.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:29.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:29.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:29.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:29.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:29.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:30.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:30.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: pgmap v155: 744 pgs: 64 creating+peering, 1 active+clean+snaptrim, 256 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 934 B/s wr, 3 op/s 2024-03-30T14:58:30.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1179536722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.49064 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 11.0 deep-scrub starts 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: 11.0 deep-scrub ok 2024-03-30T14:58:30.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: pgmap v155: 744 pgs: 64 creating+peering, 1 active+clean+snaptrim, 256 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 934 B/s wr, 3 op/s 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1179536722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.49064 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 11.0 deep-scrub starts 2024-03-30T14:58:30.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: 11.0 deep-scrub ok 2024-03-30T14:58:30.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout:pi_watch_notify_pp: flushed 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3010 ms) 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: trying... 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637090912 notify_id 571230650369 notifier_gid 35510 2024-03-30T14:58:30.167 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: timed out 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: flushing 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: flushed 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3021 ms) 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: List watches 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify2 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637093024 notify_id 584115552261 notifier_gid 35510 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify2 done 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: watch_check 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: unwatch2 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: flushing 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: done 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3033 ms) 2024-03-30T14:58:30.168 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: List watches 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify2 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: handle_notify cookie 94572637093024 notify_id 597000454145 notifier_gid 35510 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: notify2 done 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: watch_check 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: unwatch2 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: flushing 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: done 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3031 ms) 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17531 ms total) 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (45602 ms total) 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: Running main() from gmock_main.cc 2024-03-30T14:58:30.169 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [----------] Global test environment set-up. 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (11633 ms) 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (6670 ms) 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (5048 ms) 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3744 ms) 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (4039 ms) 2024-03-30T14:58:30.170 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (6095 ms) 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (8105 ms) 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (45334 ms total) 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [----------] Global test environment tear-down 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (45334 ms total) 2024-03-30T14:58:30.171 INFO:tasks.workunit.client.0.smithi028.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-30T14:58:30.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:30.241 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:30.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:30.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: pgmap v155: 744 pgs: 64 creating+peering, 1 active+clean+snaptrim, 256 unknown, 423 active+clean; 145 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 934 B/s wr, 3 op/s 2024-03-30T14:58:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T14:58:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.35510 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi028-93540-6", "pool2": "test-rados-api-smithi028-93540-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: osdmap e145: 8 total, 8 up, 8 in 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1179536722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.49064 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:30.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 11.0 deep-scrub starts 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: 11.0 deep-scrub ok 2024-03-30T14:58:30.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:30.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:30.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:30.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:30.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:30.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:30.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:31.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:31.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:31.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:31.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:31.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:31.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:31.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:31.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.3 deep-scrub starts 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.3 deep-scrub ok 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: pgmap v158: 772 pgs: 96 unknown, 676 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1005 B/s wr, 4 op/s 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.2 deep-scrub starts 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.2 deep-scrub ok 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.4 deep-scrub starts 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.4 deep-scrub ok 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.7 deep-scrub starts 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.49064 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: 11.7 deep-scrub ok 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:31.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.3 deep-scrub starts 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.3 deep-scrub ok 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: pgmap v158: 772 pgs: 96 unknown, 676 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1005 B/s wr, 4 op/s 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.2 deep-scrub starts 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.2 deep-scrub ok 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.4 deep-scrub starts 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.4 deep-scrub ok 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.7 deep-scrub starts 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:31.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.49064 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: 11.7 deep-scrub ok 2024-03-30T14:58:31.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:31.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:31.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-30T14:58:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.3 deep-scrub starts 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.3 deep-scrub ok 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: pgmap v158: 772 pgs: 96 unknown, 676 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1005 B/s wr, 4 op/s 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.2 deep-scrub starts 2024-03-30T14:58:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.2 deep-scrub ok 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.4 deep-scrub starts 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.4 deep-scrub ok 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.7 deep-scrub starts 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.49064 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi028-92999-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: osdmap e146: 8 total, 8 up, 8 in 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]: dispatch 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: 11.7 deep-scrub ok 2024-03-30T14:58:31.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:31.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:31.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:31.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:31.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:32.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:32.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:32.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:32.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:32.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:32.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:32.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:32.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:32.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:32.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:32.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 11.1 deep-scrub starts 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 11.1 deep-scrub ok 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 11.5 deep-scrub starts 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 11.5 deep-scrub ok 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T14:58:32.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]': finished 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/553852123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.42271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: pgmap v161: 580 pgs: 96 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 op/s 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.42271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/425654988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1777674724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[29876]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 11.1 deep-scrub starts 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 11.1 deep-scrub ok 2024-03-30T14:58:32.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 11.5 deep-scrub starts 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 11.5 deep-scrub ok 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]': finished 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:58:32.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/553852123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.42271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: pgmap v161: 580 pgs: 96 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 op/s 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.42271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/425654988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1777674724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:32 smithi028 ceph-mon[41614]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:58:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:32.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 11.1 deep-scrub starts 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 11.1 deep-scrub ok 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 11.5 deep-scrub starts 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 11.5 deep-scrub ok 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T14:58:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-9", "mode": "writeback"}]': finished 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: osdmap e147: 8 total, 8 up, 8 in 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/553852123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.42271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: pgmap v161: 580 pgs: 96 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 op/s 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:32.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.42271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi028-92979-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:32.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/425654988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1777674724' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:32.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:32 smithi134 ceph-mon[33714]: osdmap e148: 8 total, 8 up, 8 in 2024-03-30T14:58:33.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.051 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:33.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:33.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-30T14:58:33.213 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11838 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6657 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (5057 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (5916 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4903 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (5384 ms) 2024-03-30T14:58:33.214 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8795 ms) 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (48551 ms total) 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (48551 ms total) 2024-03-30T14:58:33.215 INFO:tasks.workunit.client.0.smithi028.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-30T14:58:33.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:33.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:33.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:33.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:33.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:33.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:33.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:33.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:33.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:33.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:33.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:34.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:34.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 11.8 deep-scrub starts 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 11.8 deep-scrub ok 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T14:58:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.43423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.43429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.43423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.43429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:33 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:34.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 11.8 deep-scrub starts 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 11.8 deep-scrub ok 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.43423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.43429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.43423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.43429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 11.8 deep-scrub starts 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 11.8 deep-scrub ok 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T14:58:34.230 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.43423 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.43429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.43423 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.43429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi028-92999-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: osdmap e149: 8 total, 8 up, 8 in 2024-03-30T14:58:34.231 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:33 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]: dispatch 2024-03-30T14:58:34.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:34.425 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:34.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:34.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:34.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:34.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:35.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:35.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:35.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[41614]: 11.9 deep-scrub starts 2024-03-30T14:58:35.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[41614]: 11.9 deep-scrub ok 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[41614]: pgmap v164: 516 pgs: 25 creating+peering, 27 unknown, 464 active+clean; 193 MiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s wr, 0 op/s 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[29876]: 11.9 deep-scrub starts 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[29876]: 11.9 deep-scrub ok 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[29876]: pgmap v164: 516 pgs: 25 creating+peering, 27 unknown, 464 active+clean; 193 MiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s wr, 0 op/s 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:35.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:35 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:35 smithi134 ceph-mon[33714]: 11.9 deep-scrub starts 2024-03-30T14:58:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:35 smithi134 ceph-mon[33714]: 11.9 deep-scrub ok 2024-03-30T14:58:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:35 smithi134 ceph-mon[33714]: pgmap v164: 516 pgs: 25 creating+peering, 27 unknown, 464 active+clean; 193 MiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s wr, 0 op/s 2024-03-30T14:58:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:35 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:35.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:35.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:35.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:35.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:35.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:35.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:35.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.757+0000 7f3abeebe700 1 -- 172.21.15.28:0/1641080935 >> v1:172.21.15.134:6789/0 conn(0x7f3ab806d9a0 legacy=0x7f3ab806dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.757+0000 7f3ab77fe700 1 -- 172.21.15.28:0/1641080935 <== mon.1 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3aa8001470 con 0x7f3ab806d9a0 2024-03-30T14:58:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.758+0000 7f3abeebe700 1 -- 172.21.15.28:0/1641080935 shutdown_connections 2024-03-30T14:58:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.758+0000 7f3abeebe700 1 -- 172.21.15.28:0/1641080935 >> 172.21.15.28:0/1641080935 conn(0x7f3ab806ce80 msgr2=0x7f3ab810b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.759+0000 7f3abeebe700 1 -- 172.21.15.28:0/1641080935 shutdown_connections 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.759+0000 7f3abeebe700 1 -- 172.21.15.28:0/1641080935 wait complete. 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 librados: starting msgr at 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 librados: starting objecter 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 Processor -- start 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 -- start start 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 librados: setting wanted keys 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 librados: calling monclient init 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ab81b6430 con 0x7f3ab806e490 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ab81b7610 con 0x7f3ab806d9a0 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.760+0000 7f3abeebe700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3ab81b87f0 con 0x7f3ab811d8f0 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab7fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ab806e490 0x7f3ab811ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37288/0 (socket says 172.21.15.28:37288) 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab7fff700 1 -- 172.21.15.28:0/2421482678 learned_addr learned my addr 172.21.15.28:0/2421482678 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter ms_handle_connect 0x7f3ab806e490 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter ms_handle_connect 0x7f3ab806d9a0 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter ms_handle_connect 0x7f3ab811d8f0 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.761+0000 7f3ab5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 995376654 0 0) 0x7f3ab81b7610 con 0x7f3ab806d9a0 2024-03-30T14:58:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3aa00032d0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365925213 0 0) 0x7f3ab81b87f0 con 0x7f3ab811d8f0 2024-03-30T14:58:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ab81b7610 con 0x7f3ab811d8f0 2024-03-30T14:58:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1744115401 0 0) 0x7f3aa00032d0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ab81b87f0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.762+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3aa80018c0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2226626930 0 0) 0x7f3ab81b7610 con 0x7f3ab811d8f0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3aa00032d0 con 0x7f3ab811d8f0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3ab0001c30 con 0x7f3ab811d8f0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1719489207 0 0) 0x7f3ab81b87f0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6790/0 conn(0x7f3ab811d8f0 legacy=0x7f3ab81b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6789/0 conn(0x7f3ab806e490 legacy=0x7f3ab811ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab81b9a10 con 0x7f3ab806d9a0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.763+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab81b7840 con 0x7f3ab806d9a0 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.764+0000 7f3abeebe700 10 client.48004.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.764+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ab81b7d00 con 0x7f3ab806d9a0 2024-03-30T14:58:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.764+0000 7f3abeebe700 1 librados: init done 2024-03-30T14:58:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.764+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3aa8003760 con 0x7f3ab806d9a0 2024-03-30T14:58:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.765+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3aa8004cf0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.765+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3aa4005100 con 0x7f3ab806d9a0 2024-03-30T14:58:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.766+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3aa801b5b0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(151..151 src has 53..151) ==== 11555+0+0 (unknown 383038465 0 0) 0x7f3aa8089540 con 0x7f3ab806d9a0 2024-03-30T14:58:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 osd_map(151..151 src has 53..151) 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 3 client.48004.objecter handle_osd_map got epochs [151,151] > 0 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 3 client.48004.objecter handle_osd_map decoding full epoch 151 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 20 client.48004.objecter dump_active .. 0 homeless 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v151) ==== 54+0+16 (unknown 2267073188 0 2420000192) 0x7f3aa80559f0 con 0x7f3ab806d9a0 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.767+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa0070370 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 10 client.48004.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 1 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 1 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 0 1 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 4 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 10 client.48004.objecter _send_command 1 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.769+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3aa4004b50 con 0x7f3aa4004730 2024-03-30T14:58:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.770+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa4004730 2024-03-30T14:58:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.771+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa4004b50 con 0x7f3aa4004730 2024-03-30T14:58:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.771+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.771+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 1 = system:0 2024-03-30T14:58:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.772+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 4 2024-03-30T14:58:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.772+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 0 1 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 10 client.48004.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 2 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 2 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40053e0 osd=0 3 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 0 2 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 4 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 10 client.48004.objecter _send_command 2 2024-03-30T14:58:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.824+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa4007b00 con 0x7f3aa4004730 2024-03-30T14:58:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.828+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa4007b00 con 0x7f3aa4004730 2024-03-30T14:58:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.828+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.828+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.828+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 4 2024-03-30T14:58:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.828+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 0 2 2024-03-30T14:58:35.830 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 10 client.48004.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 3 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 3 2024-03-30T14:58:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 1 3 2024-03-30T14:58:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 4 2024-03-30T14:58:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 10 client.48004.objecter _send_command 3 2024-03-30T14:58:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.832+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3aa4002200 con 0x7f3aa40042f0 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.835+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa40042f0 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.836+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa4002200 con 0x7f3aa40042f0 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.836+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 3: 0 ) 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.836+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 3 = system:0 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.836+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 4 2024-03-30T14:58:35.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.836+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 1 3 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 10 client.48004.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 4 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 4 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa4004180 osd=1 3 2024-03-30T14:58:35.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 1 4 2024-03-30T14:58:35.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 4 2024-03-30T14:58:35.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 10 client.48004.objecter _send_command 4 2024-03-30T14:58:35.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.881+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa4001600 con 0x7f3aa40042f0 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.884+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa4001600 con 0x7f3aa40042f0 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.884+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.884+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.884+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 4 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.884+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 1 4 2024-03-30T14:58:35.886 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:58:35.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 10 client.48004.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:35.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 5 2024-03-30T14:58:35.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 5 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 2 5 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 4 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 10 client.48004.objecter _send_command 5 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3aa40111f0 con 0x7f3aa400d6e0 2024-03-30T14:58:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.889+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa400d6e0 2024-03-30T14:58:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.891+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa40111f0 con 0x7f3aa400d6e0 2024-03-30T14:58:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.891+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 5: 0 ) 2024-03-30T14:58:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.891+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 5 = system:0 2024-03-30T14:58:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.891+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 4 2024-03-30T14:58:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.891+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 2 5 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 10 client.48004.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 6 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 6 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa4008480 osd=2 3 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 2 6 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 4 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 10 client.48004.objecter _send_command 6 2024-03-30T14:58:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.929+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa4007f60 con 0x7f3aa400d6e0 2024-03-30T14:58:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.932+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa4007f60 con 0x7f3aa400d6e0 2024-03-30T14:58:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.932+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.932+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.932+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 4 2024-03-30T14:58:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.932+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 2 6 2024-03-30T14:58:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:58:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 10 client.48004.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 7 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 7 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 3 7 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 4 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 10 client.48004.objecter _send_command 7 2024-03-30T14:58:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.933+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3aa4016030 con 0x7f3aa4012520 2024-03-30T14:58:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa4012520 2024-03-30T14:58:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa4016030 con 0x7f3aa4012520 2024-03-30T14:58:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 7: 0 ) 2024-03-30T14:58:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 7 = system:0 2024-03-30T14:58:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 4 2024-03-30T14:58:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.941+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 3 7 2024-03-30T14:58:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 10 client.48004.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 8 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 8 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa400cd30 osd=3 3 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 3 8 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 4 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 10 client.48004.objecter _send_command 8 2024-03-30T14:58:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.977+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa400c870 con 0x7f3aa4012520 2024-03-30T14:58:35.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:58:35.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa400c870 con 0x7f3aa4012520 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 4 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 3 8 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 10 client.48004.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 9 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:35.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:35.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:35.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 9 2024-03-30T14:58:35.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:35.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 4 9 2024-03-30T14:58:35.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 4 2024-03-30T14:58:35.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 10 client.48004.objecter _send_command 9 2024-03-30T14:58:35.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.984+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3aa400c850 con 0x7f3aa4012080 2024-03-30T14:58:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:35.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.986+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa4012080 2024-03-30T14:58:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.988+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa400c850 con 0x7f3aa4012080 2024-03-30T14:58:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.988+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 9: 0 ) 2024-03-30T14:58:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.988+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 9 = system:0 2024-03-30T14:58:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.988+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 4 2024-03-30T14:58:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:35.988+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 4 9 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 10 client.48004.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 10 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 10 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa400c620 osd=4 3 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 4 10 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 4 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 10 client.48004.objecter _send_command 10 2024-03-30T14:58:36.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.023+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa400cb80 con 0x7f3aa4012080 2024-03-30T14:58:36.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.025+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa400cb80 con 0x7f3aa4012080 2024-03-30T14:58:36.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.025+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:36.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.025+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:36.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.026+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 4 2024-03-30T14:58:36.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.026+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 4 10 2024-03-30T14:58:36.029 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 10 client.48004.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 11 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 11 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 5 11 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 4 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 10 client.48004.objecter _send_command 11 2024-03-30T14:58:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.028+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3aa40118e0 con 0x7f3aa4016d10 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.031+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa4016d10 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.032+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa40118e0 con 0x7f3aa4016d10 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.032+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 11: 0 ) 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.032+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 11 = system:0 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.032+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 4 2024-03-30T14:58:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.032+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 5 11 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 10 client.48004.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 12 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 12 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40113a0 osd=5 3 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 5 12 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 4 2024-03-30T14:58:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 10 client.48004.objecter _send_command 12 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.070+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa401f2c0 con 0x7f3aa4016d10 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa401f2c0 con 0x7f3aa4016d10 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 4 2024-03-30T14:58:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 5 12 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 10 client.48004.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 13 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 13 2024-03-30T14:58:36.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 6 13 2024-03-30T14:58:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 4 2024-03-30T14:58:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 10 client.48004.objecter _send_command 13 2024-03-30T14:58:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.072+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3aa4023510 con 0x7f3aa401f7d0 2024-03-30T14:58:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa401f7d0 2024-03-30T14:58:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa4023510 con 0x7f3aa401f7d0 2024-03-30T14:58:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 13: 0 ) 2024-03-30T14:58:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 13 = system:0 2024-03-30T14:58:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 4 2024-03-30T14:58:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.078+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 6 13 2024-03-30T14:58:36.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 10 client.48004.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 14 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 14 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40118e0 osd=6 3 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 6 14 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 4 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 10 client.48004.objecter _send_command 14 2024-03-30T14:58:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.113+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa40236f0 con 0x7f3aa401f7d0 2024-03-30T14:58:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.116+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa40236f0 con 0x7f3aa401f7d0 2024-03-30T14:58:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.116+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.116+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.116+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 4 2024-03-30T14:58:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.116+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 6 14 2024-03-30T14:58:36.118 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:58:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 10 client.48004.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 15 2024-03-30T14:58:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 15 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 7 15 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 4 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 10 client.48004.objecter _send_command 15 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.117+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3aa4024a20 con 0x7f3aa4023e10 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.118+0000 7f3ab5ffb700 10 client.48004.objecter ms_handle_connect 0x7f3aa4023e10 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.119+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3aa4024a20 con 0x7f3aa4023e10 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.119+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 15: 0 ) 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.119+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 15 = system:0 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.119+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 4 2024-03-30T14:58:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.119+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 7 15 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 10 client.48004.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign -1 16 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 20 client.48004.objecter _get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_remove -1 16 2024-03-30T14:58:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 20 client.48004.objecter get_session s=0x7f3aa40236f0 osd=7 3 2024-03-30T14:58:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 15 client.48004.objecter _session_command_op_assign 7 16 2024-03-30T14:58:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 4 2024-03-30T14:58:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 10 client.48004.objecter _send_command 16 2024-03-30T14:58:36.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.159+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3aa4024a20 con 0x7f3aa4023e10 2024-03-30T14:58:36.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:58:36.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.162+0000 7f3ab5ffb700 1 -- 172.21.15.28:0/2421482678 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3aa4024a20 con 0x7f3aa4023e10 2024-03-30T14:58:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.162+0000 7f3ab5ffb700 10 client.48004.objecter ms_dispatch 0x7f3ab8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.162+0000 7f3ab5ffb700 10 client.48004.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.162+0000 7f3ab5ffb700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 4 2024-03-30T14:58:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.162+0000 7f3ab5ffb700 15 client.48004.objecter _session_command_op_remove 7 16 2024-03-30T14:58:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.164+0000 7f3abeebe700 10 librados: watch_flush enter 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.165+0000 7f3abeebe700 10 librados: watch_flush exit 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.165+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.0 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.165+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6801/3394323575 conn(0x7f3aa4004730 legacy=0x7f3aa4002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.165+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40053e0 osd=0 1 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.1 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6805/1850313547 conn(0x7f3aa40042f0 legacy=0x7f3aa400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4004180 osd=1 1 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.2 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6809/3451325043 conn(0x7f3aa400d6e0 legacy=0x7f3aa400fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abd45b700 1 -- 172.21.15.28:0/2421482678 reap_dead start 2024-03-30T14:58:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa4008480 osd=2 1 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.3 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6813/1506157250 conn(0x7f3aa4012520 legacy=0x7f3aa4014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400cd30 osd=3 1 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.4 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.134:6800/3063775514 conn(0x7f3aa4012080 legacy=0x7f3aa40191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa400c620 osd=4 1 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.5 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.134:6804/3668151179 conn(0x7f3aa4016d10 legacy=0x7f3aa401a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40113a0 osd=5 1 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.6 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.166+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.134:6808/2135224194 conn(0x7f3aa401f7d0 legacy=0x7f3aa4021c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40118e0 osd=6 1 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 10 client.48004.objecter close_session for osd.7 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.134:6812/3758190298 conn(0x7f3aa4023e10 legacy=0x7f3aa4024270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abd45b700 1 -- 172.21.15.28:0/2421482678 reap_dead start 2024-03-30T14:58:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 20 client.48004.objecter put_session s=0x7f3aa40236f0 osd=7 1 2024-03-30T14:58:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 20 client.48004.objecter shutdown clearing up homeless session... 2024-03-30T14:58:36.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 10 client.48004.objecter successfully canceled tick 2024-03-30T14:58:36.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3aa0070370 legacy=0x7f3aa0072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.167+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> v1:172.21.15.134:6789/0 conn(0x7f3ab806d9a0 legacy=0x7f3ab811a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.168+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 shutdown_connections 2024-03-30T14:58:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.168+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 >> 172.21.15.28:0/2421482678 conn(0x7f3ab806ce80 msgr2=0x7f3ab8070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.169+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 shutdown_connections 2024-03-30T14:58:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.172+0000 7f3abeebe700 1 -- 172.21.15.28:0/2421482678 wait complete. 2024-03-30T14:58:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:36.172+0000 7f3abeebe700 1 librados: shutdown 2024-03-30T14:58:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:58:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:58:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:58:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:58:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:58:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:58:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:58:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:58:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:36.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:36.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:36.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T14:58:36.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T14:58:36.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:58:36.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1162406823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:36 smithi134 ceph-mon[33714]: from='client.45382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:36.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T14:58:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T14:58:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1162406823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[29876]: from='client.45382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T14:58:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-9"}]': finished 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: osdmap e150: 8 total, 8 up, 8 in 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: osdmap e151: 8 total, 8 up, 8 in 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1162406823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:36 smithi028 ceph-mon[41614]: from='client.45382 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:36.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:36.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:36.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:36.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:36.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:36.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:36.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:36.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:36.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:37.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:37.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:37.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:37.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.369+0000 7fe8cbf46700 1 -- 172.21.15.28:0/1425989431 >> v1:172.21.15.28:6790/0 conn(0x7fe8c410d960 legacy=0x7fe8c410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.369+0000 7fe8cbf46700 1 -- 172.21.15.28:0/1425989431 shutdown_connections 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.369+0000 7fe8cbf46700 1 -- 172.21.15.28:0/1425989431 >> 172.21.15.28:0/1425989431 conn(0x7fe8c40fc9e0 msgr2=0x7fe8c40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.369+0000 7fe8cbf46700 1 -- 172.21.15.28:0/1425989431 shutdown_connections 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.369+0000 7fe8cbf46700 1 -- 172.21.15.28:0/1425989431 wait complete. 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 librados: starting msgr at 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 librados: starting objecter 2024-03-30T14:58:37.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 Processor -- start 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 -- start start 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 librados: setting wanted keys 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 librados: calling monclient init 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8c41a7530 con 0x7fe8c4101130 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8c41a8710 con 0x7fe8c410d960 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8cbf46700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8c41a98f0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8c94e1700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe8c4109dc0 0x7fe8c4106000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45928/0 (socket says 172.21.15.28:45928) 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8c94e1700 1 -- 172.21.15.28:0/2933837618 learned_addr learned my addr 172.21.15.28:0/2933837618 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter ms_handle_connect 0x7fe8c4109dc0 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter ms_handle_connect 0x7fe8c4101130 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter ms_handle_connect 0x7fe8c410d960 2024-03-30T14:58:37.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.370+0000 7fe8b6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225035367 0 0) 0x7fe8c41a98f0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8b00035b0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304529032 0 0) 0x7fe8c41a8710 con 0x7fe8c410d960 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8c41a98f0 con 0x7fe8c410d960 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1915841728 0 0) 0x7fe8b00035b0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8c41a8710 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe8b8001970 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3779486770 0 0) 0x7fe8c41a98f0 con 0x7fe8c410d960 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8b00035b0 con 0x7fe8c410d960 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe8bc0017b0 con 0x7fe8c410d960 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3689791688 0 0) 0x7fe8c41a8710 con 0x7fe8c4109dc0 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 >> v1:172.21.15.134:6789/0 conn(0x7fe8c410d960 legacy=0x7fe8c41a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 >> v1:172.21.15.28:6789/0 conn(0x7fe8c4101130 legacy=0x7fe8c41058f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c41aaad0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8cbf46700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c41a7760 con 0x7fe8c4109dc0 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8cbf46700 10 client.56798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8cbf46700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8c41a7c20 con 0x7fe8c4109dc0 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8cbf46700 1 librados: init done 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe8b8001cc0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.371+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe8b8005dc0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.372+0000 7fe8cbf46700 10 client.56798.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.372+0000 7fe8cbf46700 15 client.56798.objecter _session_command_op_assign -1 1 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.372+0000 7fe8cbf46700 20 client.56798.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.372+0000 7fe8cbf46700 10 client.56798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.372+0000 7fe8cbf46700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe8a8002920 con 0x7fe8c4109dc0 2024-03-30T14:58:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe8b8003950 con 0x7fe8c4109dc0 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(153..153 src has 53..153) ==== 12017+0+0 (unknown 4102015266 0 0) 0x7fe8b808a800 con 0x7fe8c4109dc0 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 10 client.56798.objecter ms_dispatch 0x7fe8c4001550 osd_map(153..153 src has 53..153) 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 3 client.56798.objecter handle_osd_map got epochs [153,153] > 0 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 3 client.56798.objecter handle_osd_map decoding full epoch 153 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 10 client.56798.objecter checking command 1 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter _get_session s=0x7fe8b0086aa0 osd=2 3 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter put_session s=0x7fe8b0086aa0 osd=2 3 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 15 client.56798.objecter _session_command_op_remove -1 1 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter _get_session s=0x7fe8b0086aa0 osd=2 3 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter get_session s=0x7fe8b0086aa0 osd=2 3 2024-03-30T14:58:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 15 client.56798.objecter _session_command_op_assign 2 1 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter put_session s=0x7fe8b0086aa0 osd=2 4 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 10 client.56798.objecter _send_command 1 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 --> v1:172.21.15.28:6809/3451325043 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8b80042c0 con 0x7fe8b0087150 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 20 client.56798.objecter dump_active .. 0 homeless 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 10 client.56798.objecter ms_handle_connect 0x7fe8b0070330 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.373+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=153) ==== 24+0+0 (unknown 2280005239 0 0) 0x7fe8b808abe0 con 0x7fe8c4109dc0 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.374+0000 7fe8b6ffd700 10 client.56798.objecter ms_handle_connect 0x7fe8b0087150 2024-03-30T14:58:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.374+0000 7fe8b6ffd700 1 -- 172.21.15.28:0/2933837618 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe8b80042c0 con 0x7fe8b0087150 2024-03-30T14:58:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.375+0000 7fe8b6ffd700 10 client.56798.objecter ms_dispatch 0x7fe8c4001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.375+0000 7fe8b6ffd700 10 client.56798.objecter _finish_command 1 = system:0 2024-03-30T14:58:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.375+0000 7fe8b6ffd700 20 client.56798.objecter put_session s=0x7fe8b0086aa0 osd=2 4 2024-03-30T14:58:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.375+0000 7fe8b6ffd700 15 client.56798.objecter _session_command_op_remove 2 1 2024-03-30T14:58:37.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:37.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: 11.6 deep-scrub starts 2024-03-30T14:58:37.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: 11.6 deep-scrub ok 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: pgmap v167: 548 pgs: 128 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 4 op/s 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2421482678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:37.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.45382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3415225450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[29876]: from='client.55709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: 11.6 deep-scrub starts 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: 11.6 deep-scrub ok 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: pgmap v167: 548 pgs: 128 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 4 op/s 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2421482678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:37.410 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:37.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:37.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.45382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:37.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:58:37.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3415225450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.412 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.412 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:37 smithi028 ceph-mon[41614]: from='client.55709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:37.509 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:37.509 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:37.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: 11.6 deep-scrub starts 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: 11.6 deep-scrub ok 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: pgmap v167: 548 pgs: 128 unknown, 420 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 4 op/s 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2421482678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:37.511 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.45382 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi028-92979-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: osdmap e152: 8 total, 8 up, 8 in 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3415225450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.512 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:37 smithi134 ceph-mon[33714]: from='client.55709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:37.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:37.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:37.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 -- 172.21.15.28:0/1229599531 >> v1:172.21.15.28:6789/0 conn(0x7f6270102230 legacy=0x7f6270102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 -- 172.21.15.28:0/1229599531 shutdown_connections 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 -- 172.21.15.28:0/1229599531 >> 172.21.15.28:0/1229599531 conn(0x7f62700fdb00 msgr2=0x7f62700fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 -- 172.21.15.28:0/1229599531 shutdown_connections 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 -- 172.21.15.28:0/1229599531 wait complete. 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 librados: starting msgr at 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.653+0000 7f6277f22700 1 librados: starting objecter 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 Processor -- start 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 -- start start 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 librados: setting wanted keys 2024-03-30T14:58:37.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 librados: calling monclient init 2024-03-30T14:58:37.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62701a7140 con 0x7f627010aec0 2024-03-30T14:58:37.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62701a8360 con 0x7f6270102230 2024-03-30T14:58:37.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f6277f22700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62701a95a0 con 0x7f627010ea60 2024-03-30T14:58:37.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f62764bf700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f627010ea60 0x7f62701a5800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45956/0 (socket says 172.21.15.28:45956) 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f62754bd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f627010aec0 0x7f62701a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f62764bf700 1 -- 172.21.15.28:0/2702575564 learned_addr learned my addr 172.21.15.28:0/2702575564 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f627010aec0 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f627010ea60 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f6270102230 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413969304 0 0) 0x7f62701a7140 con 0x7f627010aec0 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62580031f0 con 0x7f627010aec0 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494924431 0 0) 0x7f62701a95a0 con 0x7f627010ea60 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62701a7140 con 0x7f627010ea60 2024-03-30T14:58:37.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882428240 0 0) 0x7f62701a8360 con 0x7f6270102230 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.654+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62701a95a0 con 0x7f6270102230 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1803992790 0 0) 0x7f62580031f0 con 0x7f627010aec0 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62701a8360 con 0x7f627010aec0 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2009243793 0 0) 0x7f62701a7140 con 0x7f627010ea60 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62580031f0 con 0x7f627010ea60 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4184570961 0 0) 0x7f62701a95a0 con 0x7f6270102230 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62701a7140 con 0x7f6270102230 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62640018b0 con 0x7f627010aec0 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62680013d0 con 0x7f627010ea60 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6260004730 con 0x7f6270102230 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4293407022 0 0) 0x7f62701a8360 con 0x7f627010aec0 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 >> v1:172.21.15.28:6790/0 conn(0x7f627010ea60 legacy=0x7f62701a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 >> v1:172.21.15.134:6789/0 conn(0x7f6270102230 legacy=0x7f62701075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62701aa7e0 con 0x7f627010aec0 2024-03-30T14:58:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62701a97d0 con 0x7f627010aec0 2024-03-30T14:58:37.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f6277f22700 10 client.23760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:37.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62701a9c90 con 0x7f627010aec0 2024-03-30T14:58:37.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.655+0000 7f6277f22700 1 librados: init done 2024-03-30T14:58:37.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.656+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6264003b40 con 0x7f627010aec0 2024-03-30T14:58:37.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.656+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6264005ca0 con 0x7f627010aec0 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.657+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625c005150 con 0x7f627010aec0 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.657+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f626401c4e0 con 0x7f627010aec0 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.657+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(153..153 src has 53..153) ==== 12017+0+0 (unknown 4102015266 0 0) 0x7f626408a600 con 0x7f627010aec0 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.657+0000 7f626effd700 10 client.23760.objecter ms_dispatch 0x7f6270001550 osd_map(153..153 src has 53..153) 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.658+0000 7f626effd700 3 client.23760.objecter handle_osd_map got epochs [153,153] > 0 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.658+0000 7f626effd700 3 client.23760.objecter handle_osd_map decoding full epoch 153 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.658+0000 7f626effd700 20 client.23760.objecter dump_active .. 0 homeless 2024-03-30T14:58:37.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.658+0000 7f626effd700 10 client.23760.objecter ms_handle_connect 0x7f6258070490 2024-03-30T14:58:37.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.663+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f626405ecf0 con 0x7f627010aec0 2024-03-30T14:58:37.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:37.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:37.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:37.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:37.879+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f625c005440 con 0x7f627010aec0 2024-03-30T14:58:37.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:37.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:37.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:38.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:38.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:38.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.161+0000 7f626effd700 1 -- 172.21.15.28:0/2702575564 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v154) ==== 87+0+0 (unknown 536053850 0 0) 0x7f6264087350 con 0x7f627010aec0 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.164+0000 7f6277f22700 10 librados: watch_flush enter 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.164+0000 7f6277f22700 10 librados: watch_flush exit 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.164+0000 7f6277f22700 20 client.23760.objecter shutdown clearing up homeless session... 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.164+0000 7f6277f22700 10 client.23760.objecter successfully canceled tick 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.169+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6258070490 legacy=0x7f6258072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.169+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 >> v1:172.21.15.28:6789/0 conn(0x7f627010aec0 legacy=0x7f62701a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:38.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.169+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 shutdown_connections 2024-03-30T14:58:38.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.169+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 >> 172.21.15.28:0/2702575564 conn(0x7f62700fdb00 msgr2=0x7f62701116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.181+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 shutdown_connections 2024-03-30T14:58:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.181+0000 7f6277f22700 1 -- 172.21.15.28:0/2702575564 wait complete. 2024-03-30T14:58:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:38.181+0000 7f6277f22700 1 librados: shutdown 2024-03-30T14:58:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T14:58:38.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-30T14:58:38.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] Global test environment set-up. 2024-03-30T14:58:38.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-30T14:58:38.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-30T14:58:38.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:57:44.484+0000 7f449bc37780 -1 failed for service _ceph-mon._tcp 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:57:44.484+0000 7f449bc37780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (52 ms) 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2017 ms) 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2069 ms total) 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:58:38.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: started 0x558c2feb2660 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: started 0x558c301d2440 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: started 2 aios 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: waiting 0x558c2feb2660 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: waiting 0x558c301d2440 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: done. 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (13151 ms) 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (13151 ms total) 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-30T14:58:38.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.Exec (111 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.Applications (7840 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-30T14:58:38.203 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (24872 ms) 2024-03-30T14:58:38.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] 8 tests from LibRadosMisc (32830 ms total) 2024-03-30T14:58:38.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: 2024-03-30T14:58:38.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [----------] Global test environment tear-down 2024-03-30T14:58:38.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (53742 ms total) 2024-03-30T14:58:38.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-30T14:58:38.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:38.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T14:58:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:38.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:38.321 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:38.351 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:38.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-30T14:58:38.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:38.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:38.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:38.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:38.550 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:38.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:38.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:38.672 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.55709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.673 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.674 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:38 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.55709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:58:38.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.55709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi028-92999-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: osdmap e153: 8 total, 8 up, 8 in 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:38.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93339-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2702575564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: osdmap e154: 8 total, 8 up, 8 in 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:38 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:38.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:38.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:38.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:38.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:38.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:39.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:39.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:39.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:39.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:58:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:39.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:39.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:39.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:39.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:39.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:39.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:39.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:39.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:39.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:39.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:39.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:39.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:39.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:40.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:40.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:40.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:40.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:40.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: pgmap v170: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:40.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: 4.1c deep-scrub starts 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: 4.1c deep-scrub ok 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: pgmap v170: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: 4.1c deep-scrub starts 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: 4.1c deep-scrub ok 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.204 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:40.205 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:40.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:40.278 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:40.279 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:40.279 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: pgmap v170: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: 4.1c deep-scrub starts 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: 4.1c deep-scrub ok 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:40.279 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:40.280 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:40.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:40.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:40.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:40.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:40.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:40.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:40.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:40.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:40.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T14:58:40.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:40.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:41.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:41.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:41.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:58:41.134 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:41.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:41.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3935361328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/474739575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: pgmap v173: 548 pgs: 128 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:58:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:41.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:41.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3935361328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/474739575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: pgmap v173: 548 pgs: 128 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:41.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3935361328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/474739575' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: pgmap v173: 548 pgs: 128 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: osdmap e155: 8 total, 8 up, 8 in 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi028-93441-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:41.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi028-92979-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache", "force_nonempty":""}]': finished 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi028-92999-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: osdmap e156: 8 total, 8 up, 8 in 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]: dispatch 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:41.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:41.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:41.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:41.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:41.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:41.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:41.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:41.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:41.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:41.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:41.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:41.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:42.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:42.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:42.201 INFO:tasks.workunit.client.0.smithi028.stdout: misc: Running main() from gmock_main.cc 2024-03-30T14:58:42.201 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-30T14:58:42.201 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [----------] Global test environment set-up. 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.Version (10499 ms) 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (3170 ms) 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.LongName (6802 ms) 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3741 ms) 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (5060 ms) 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-30T14:58:42.202 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4929 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.Exec (5371 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (5771 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3047 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3295 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1685 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4011 ms) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [----------] 12 tests from NeoRadosMisc (57381 ms total) 2024-03-30T14:58:42.203 INFO:tasks.workunit.client.0.smithi028.stdout: misc: 2024-03-30T14:58:42.204 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [----------] Global test environment tear-down 2024-03-30T14:58:42.204 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [==========] 12 tests from 1 test suite ran. (57381 ms total) 2024-03-30T14:58:42.204 INFO:tasks.workunit.client.0.smithi028.stdout: misc: [ PASSED ] 12 tests. 2024-03-30T14:58:42.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:42.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:42.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:42.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]': finished 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]': finished 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:42.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:58:42.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-11", "mode": "writeback"}]': finished 2024-03-30T14:58:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93339-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: osdmap e157: 8 total, 8 up, 8 in 2024-03-30T14:58:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1843271946' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]: dispatch 2024-03-30T14:58:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:42.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:42.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:42.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:42.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:42.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:42.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:42.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:42.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:42.974 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:42.974 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:42.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:43.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:43.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:43.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:43.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: pgmap v176: 556 pgs: 168 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]': finished 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4198757132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1426906321' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.56855 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: pgmap v176: 556 pgs: 168 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]': finished 2024-03-30T14:58:43.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4198757132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1426906321' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.56855 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:43 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:43.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: pgmap v176: 556 pgs: 168 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.48631 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93339-10", "tierpool":"test-rados-api-smithi028-93339-10-cache"}]': finished 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: osdmap e158: 8 total, 8 up, 8 in 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4198757132' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1426906321' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.56855 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:43 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:43.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:43.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 -- 172.21.15.28:0/738466253 >> v1:172.21.15.28:6789/0 conn(0x7f8844105f00 legacy=0x7f88441062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 -- 172.21.15.28:0/738466253 shutdown_connections 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 -- 172.21.15.28:0/738466253 >> 172.21.15.28:0/738466253 conn(0x7f8844077370 msgr2=0x7f88440fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 -- 172.21.15.28:0/738466253 shutdown_connections 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 -- 172.21.15.28:0/738466253 wait complete. 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 librados: starting msgr at 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 librados: starting objecter 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.579+0000 7f8849c5a700 1 Processor -- start 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 -- start start 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 librados: setting wanted keys 2024-03-30T14:58:43.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 librados: calling monclient init 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88441a6f70 con 0x7f884410d950 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88441a8150 con 0x7f8844105f00 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8849c5a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88441a9390 con 0x7f8844109d80 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8842ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8844109d80 0x7f88441a1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44096/0 (socket says 172.21.15.28:44096) 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8843fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f884410d950 0x7f88441a5600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60368/0 (socket says 172.21.15.28:60368) 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8842ffd700 1 -- 172.21.15.28:0/489145286 learned_addr learned my addr 172.21.15.28:0/489145286 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter ms_handle_connect 0x7f8844109d80 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter ms_handle_connect 0x7f884410d950 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter ms_handle_connect 0x7f8844105f00 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219490682 0 0) 0x7f88441a9390 con 0x7f8844109d80 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882c003aa0 con 0x7f8844109d80 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917901409 0 0) 0x7f88441a8150 con 0x7f8844105f00 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88441a9390 con 0x7f8844105f00 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514821183 0 0) 0x7f88441a6f70 con 0x7f884410d950 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.580+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88441a8150 con 0x7f884410d950 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2060111034 0 0) 0x7f88441a9390 con 0x7f8844105f00 2024-03-30T14:58:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88441a6f70 con 0x7f8844105f00 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999987280 0 0) 0x7f882c003aa0 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88441a9390 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8834004610 con 0x7f8844105f00 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8838001960 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3927406607 0 0) 0x7f88441a8150 con 0x7f884410d950 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f882c003aa0 con 0x7f884410d950 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f883c001b30 con 0x7f884410d950 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2082066927 0 0) 0x7f88441a9390 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 >> v1:172.21.15.134:6789/0 conn(0x7f8844105f00 legacy=0x7f884410d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 >> v1:172.21.15.28:6789/0 conn(0x7f884410d950 legacy=0x7f88441a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88441aa5d0 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8849c5a700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88441a71a0 con 0x7f8844109d80 2024-03-30T14:58:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8849c5a700 10 client.56867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8849c5a700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88441a7660 con 0x7f8844109d80 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8849c5a700 1 librados: init done 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8838003b20 con 0x7f8844109d80 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.581+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8838005bf0 con 0x7f8844109d80 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.582+0000 7f8849c5a700 10 client.56867.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.582+0000 7f8849c5a700 15 client.56867.objecter _session_command_op_assign -1 1 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.582+0000 7f8849c5a700 20 client.56867.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.582+0000 7f8849c5a700 10 client.56867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.582+0000 7f8849c5a700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8830002950 con 0x7f8844109d80 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f883801c4f0 con 0x7f8844109d80 2024-03-30T14:58:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(159..159 src has 53..159) ==== 12237+0+0 (unknown 1677320547 0 0) 0x7f883808a530 con 0x7f8844109d80 2024-03-30T14:58:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 10 client.56867.objecter ms_dispatch 0x7f8844001550 osd_map(159..159 src has 53..159) 2024-03-30T14:58:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 3 client.56867.objecter handle_osd_map got epochs [159,159] > 0 2024-03-30T14:58:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 3 client.56867.objecter handle_osd_map decoding full epoch 159 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 10 client.56867.objecter checking command 1 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter _get_session s=0x7f882c085d90 osd=3 3 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter put_session s=0x7f882c085d90 osd=3 3 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 15 client.56867.objecter _session_command_op_remove -1 1 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter _get_session s=0x7f882c085d90 osd=3 3 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter get_session s=0x7f882c085d90 osd=3 3 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 15 client.56867.objecter _session_command_op_assign 3 1 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter put_session s=0x7f882c085d90 osd=3 4 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 10 client.56867.objecter _send_command 1 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 --> v1:172.21.15.28:6813/1506157250 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f882c08a1d0 con 0x7f882c086440 2024-03-30T14:58:43.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 20 client.56867.objecter dump_active .. 0 homeless 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=159) ==== 24+0+0 (unknown 2685853585 0 0) 0x7f883808a940 con 0x7f8844109d80 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.583+0000 7f8840ff9700 10 client.56867.objecter ms_handle_connect 0x7f882c0704a0 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 10 client.56867.objecter ms_handle_connect 0x7f882c086440 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 1 -- 172.21.15.28:0/489145286 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f882c08a1d0 con 0x7f882c086440 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 10 client.56867.objecter ms_dispatch 0x7f8844001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 10 client.56867.objecter _finish_command 1 = system:0 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 20 client.56867.objecter put_session s=0x7f882c085d90 osd=3 4 2024-03-30T14:58:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:43.584+0000 7f8840ff9700 15 client.56867.objecter _session_command_op_remove 3 1 2024-03-30T14:58:43.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:43.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:43.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:43.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:43.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:44.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:44.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T14:58:44.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:44.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:44.191 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: Running main() from gmock_main.cc 2024-03-30T14:58:44.191 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-30T14:58:44.191 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [----------] Global test environment set-up. 2024-03-30T14:58:44.191 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolList (11978 ms) 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6660 ms) 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5896 ms) 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4088 ms) 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-30T14:58:44.192 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4924 ms) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6392 ms) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6919 ms) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5151 ms) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7697 ms) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [----------] 9 tests from LibRadosPools (59705 ms total) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [----------] Global test environment tear-down 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (59705 ms total) 2024-03-30T14:58:44.193 INFO:tasks.workunit.client.0.smithi028.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-30T14:58:44.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.56855 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:44.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:58:44.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:58:44.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:58:44.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:44.226 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.56855 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:58:44.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:44.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:44.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:44.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:44.345 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:44.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:44.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.56855 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi028-92979-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: osdmap e159: 8 total, 8 up, 8 in 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]: dispatch 2024-03-30T14:58:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:58:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:58:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:44.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:44.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:44.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:44.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:44.728 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:44.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-30T14:58:44.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:44.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:44.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:44.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:44.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:45.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:45.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:45.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:45.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:45.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:45.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:45.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[41614]: pgmap v179: 524 pgs: 20 creating+peering, 42 unknown, 462 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 258 KiB/s wr, 9 op/s 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[41614]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[29876]: pgmap v179: 524 pgs: 20 creating+peering, 42 unknown, 462 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 258 KiB/s wr, 9 op/s 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:45.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:45.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[29876]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:58:45.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:45.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:45.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:45.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:45.669 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:45 smithi134 ceph-mon[33714]: pgmap v179: 524 pgs: 20 creating+peering, 42 unknown, 462 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 258 KiB/s wr, 9 op/s 2024-03-30T14:58:45.669 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:45 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:58:45.669 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-11"}]': finished 2024-03-30T14:58:45.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:45 smithi134 ceph-mon[33714]: osdmap e160: 8 total, 8 up, 8 in 2024-03-30T14:58:45.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:45.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:45.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:45.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.466+0000 7f408fe56700 1 -- 172.21.15.28:0/12094480 >> v1:172.21.15.28:6790/0 conn(0x7f4088109dc0 legacy=0x7f408810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.466+0000 7f408fe56700 1 -- 172.21.15.28:0/12094480 shutdown_connections 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.466+0000 7f408fe56700 1 -- 172.21.15.28:0/12094480 >> 172.21.15.28:0/12094480 conn(0x7f40880fc9e0 msgr2=0x7f40880fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.466+0000 7f408fe56700 1 -- 172.21.15.28:0/12094480 shutdown_connections 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.466+0000 7f408fe56700 1 -- 172.21.15.28:0/12094480 wait complete. 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 librados: starting msgr at 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 librados: starting objecter 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 Processor -- start 2024-03-30T14:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 -- start start 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 librados: setting wanted keys 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 librados: calling monclient init 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40881a7530 con 0x7f4088101130 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40881a8710 con 0x7f4088109dc0 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408fe56700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40881a98f0 con 0x7f408810d960 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408dbf2700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4088101130 0x7f4088104870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60452/0 (socket says 172.21.15.28:60452) 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.467+0000 7f408dbf2700 1 -- 172.21.15.28:0/353839802 learned_addr learned my addr 172.21.15.28:0/353839802 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter ms_handle_connect 0x7f4088109dc0 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter ms_handle_connect 0x7f4088101130 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter ms_handle_connect 0x7f408810d960 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906641519 0 0) 0x7f40881a8710 con 0x7f4088109dc0 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40740035b0 con 0x7f4088109dc0 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021705531 0 0) 0x7f40881a98f0 con 0x7f408810d960 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40881a8710 con 0x7f408810d960 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300254672 0 0) 0x7f40881a7530 con 0x7f4088101130 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40881a98f0 con 0x7f4088101130 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 463365201 0 0) 0x7f40740035b0 con 0x7f4088109dc0 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40881a7530 con 0x7f4088109dc0 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f40780019d0 con 0x7f4088109dc0 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3621412238 0 0) 0x7f40881a98f0 con 0x7f4088101130 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40740035b0 con 0x7f4088101130 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f40840017b0 con 0x7f4088101130 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2151639889 0 0) 0x7f40881a8710 con 0x7f408810d960 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40881a98f0 con 0x7f408810d960 2024-03-30T14:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4080001a20 con 0x7f408810d960 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3775730033 0 0) 0x7f40740035b0 con 0x7f4088101130 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6790/0 conn(0x7f408810d960 legacy=0x7f40881a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.134:6789/0 conn(0x7f4088109dc0 legacy=0x7f4088104f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40881aaad0 con 0x7f4088101130 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40881a7760 con 0x7f4088101130 2024-03-30T14:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f408fe56700 10 client.23829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40881a7c20 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.468+0000 7f408fe56700 1 librados: init done 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.469+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4084001d20 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.469+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4084005b10 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.470+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f406c005100 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.470+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f40840036a0 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(161..161 src has 53..161) ==== 11385+0+0 (unknown 3530124416 0 0) 0x7f408408a1a0 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 osd_map(161..161 src has 53..161) 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 3 client.23829.objecter handle_osd_map got epochs [161,161] > 0 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 3 client.23829.objecter handle_osd_map decoding full epoch 161 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 20 client.23829.objecter dump_active .. 0 homeless 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f40740705d0 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v161) ==== 54+0+16 (unknown 1338815507 0 2420000192) 0x7f408408c600 con 0x7f4088101130 2024-03-30T14:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 10 client.23829.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 1 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 1 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 0 1 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 4 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 10 client.23829.objecter _send_command 1 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.471+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f406c004b50 con 0x7f406c004730 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.472+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c004730 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.473+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c004b50 con 0x7f406c004730 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.473+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.473+0000 7f407effd700 10 client.23829.objecter _finish_command 1 = system:0 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.473+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 4 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.473+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 0 1 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 10 client.23829.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 2 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 2 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c0053e0 osd=0 3 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 0 2 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 4 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 10 client.23829.objecter _send_command 2 2024-03-30T14:58:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.490+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c007b00 con 0x7f406c004730 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c007b00 con 0x7f406c004730 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 10 client.23829.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 4 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 0 2 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 10 client.23829.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 3 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 3 2024-03-30T14:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 1 3 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 4 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 10 client.23829.objecter _send_command 3 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f406c002200 con 0x7f406c0042f0 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.493+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c0042f0 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.494+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c002200 con 0x7f406c0042f0 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.494+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 3: 0 ) 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.494+0000 7f407effd700 10 client.23829.objecter _finish_command 3 = system:0 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.494+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 4 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.494+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 1 3 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 10 client.23829.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 4 2024-03-30T14:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 4 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c004180 osd=1 3 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 1 4 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 4 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 10 client.23829.objecter _send_command 4 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.512+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c001600 con 0x7f406c0042f0 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:58:45.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.513+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c001600 con 0x7f406c0042f0 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.513+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.513+0000 7f407effd700 10 client.23829.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.513+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 4 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.513+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 1 4 2024-03-30T14:58:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 10 client.23829.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 5 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 5 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 2 5 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 4 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 10 client.23829.objecter _send_command 5 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f406c0111f0 con 0x7f406c00d6e0 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.514+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c00d6e0 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.515+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c0111f0 con 0x7f406c00d6e0 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.515+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 5: 0 ) 2024-03-30T14:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.515+0000 7f407effd700 10 client.23829.objecter _finish_command 5 = system:0 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.515+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 4 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.515+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 2 5 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 10 client.23829.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 6 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 6 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c008480 osd=2 3 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 2 6 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 4 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 10 client.23829.objecter _send_command 6 2024-03-30T14:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.531+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c007f60 con 0x7f406c00d6e0 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c007f60 con 0x7f406c00d6e0 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f407effd700 10 client.23829.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 4 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 2 6 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 10 client.23829.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 7 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 7 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 3 7 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 4 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 10 client.23829.objecter _send_command 7 2024-03-30T14:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.533+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f406c016030 con 0x7f406c012520 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.534+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c012520 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.535+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c016030 con 0x7f406c012520 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.535+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 7: 0 ) 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.535+0000 7f407effd700 10 client.23829.objecter _finish_command 7 = system:0 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.535+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 4 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.535+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 3 7 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 10 client.23829.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 8 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 8 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00cd30 osd=3 3 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 3 8 2024-03-30T14:58:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 4 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 10 client.23829.objecter _send_command 8 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.554+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c00c7b0 con 0x7f406c012520 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c00c7b0 con 0x7f406c012520 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 10 client.23829.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 4 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 3 8 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 10 client.23829.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 9 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 9 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 4 9 2024-03-30T14:58:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 4 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 10 client.23829.objecter _send_command 9 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f406c00c6d0 con 0x7f406c012080 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.556+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c012080 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.557+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c00c6d0 con 0x7f406c012080 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.557+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 9: 0 ) 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.557+0000 7f407effd700 10 client.23829.objecter _finish_command 9 = system:0 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.557+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 4 2024-03-30T14:58:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.557+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 4 9 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 10 client.23829.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 10 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 10 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00c560 osd=4 3 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 4 10 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 4 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 10 client.23829.objecter _send_command 10 2024-03-30T14:58:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.575+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c00c6d0 con 0x7f406c012080 2024-03-30T14:58:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c00c6d0 con 0x7f406c012080 2024-03-30T14:58:45.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:58:45.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:58:45.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:58:45.912 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 10 client.23829.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 4 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 4 10 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 10 client.23829.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 11 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 11 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 5 11 2024-03-30T14:58:45.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 4 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 10 client.23829.objecter _send_command 11 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f406c023160 con 0x7f406c01f5b0 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.577+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c01f5b0 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.578+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c023160 con 0x7f406c01f5b0 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.578+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 11: 0 ) 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.578+0000 7f407effd700 10 client.23829.objecter _finish_command 11 = system:0 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.578+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 4 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.578+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 5 11 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 10 client.23829.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 12 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 12 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c00c6d0 osd=5 3 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 5 12 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 4 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 10 client.23829.objecter _send_command 12 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.595+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c024b10 con 0x7f406c01f5b0 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.596+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c024b10 con 0x7f406c01f5b0 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.596+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.596+0000 7f407effd700 10 client.23829.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.596+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 4 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.596+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 5 12 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 10 client.23829.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 13 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 13 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 6 13 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 4 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 10 client.23829.objecter _send_command 13 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f406c0257a0 con 0x7f406c025230 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.597+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c025230 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.598+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c0257a0 con 0x7f406c025230 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.598+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 13: 0 ) 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.598+0000 7f407effd700 10 client.23829.objecter _finish_command 13 = system:0 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.598+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 4 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.598+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 6 13 2024-03-30T14:58:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 10 client.23829.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 14 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 14 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c024b10 osd=6 3 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 6 14 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 4 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 10 client.23829.objecter _send_command 14 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.614+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c0257a0 con 0x7f406c025230 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c0257a0 con 0x7f406c025230 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f407effd700 10 client.23829.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 4 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 6 14 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 10 client.23829.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 15 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 15 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 7 15 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 4 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 10 client.23829.objecter _send_command 15 2024-03-30T14:58:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.616+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f406c02baf0 con 0x7f406c027d40 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.617+0000 7f407effd700 10 client.23829.objecter ms_handle_connect 0x7f406c027d40 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.618+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f406c02baf0 con 0x7f406c027d40 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.618+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 15: 0 ) 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.618+0000 7f407effd700 10 client.23829.objecter _finish_command 15 = system:0 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.618+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 4 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.618+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 7 15 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 10 client.23829.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign -1 16 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 20 client.23829.objecter _get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 15 client.23829.objecter _session_command_op_remove -1 16 2024-03-30T14:58:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 20 client.23829.objecter get_session s=0x7f406c024950 osd=7 3 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 15 client.23829.objecter _session_command_op_assign 7 16 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 4 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 10 client.23829.objecter _send_command 16 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.635+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406c00ca20 con 0x7f406c027d40 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f407effd700 1 -- 172.21.15.28:0/353839802 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f406c00ca20 con 0x7f406c027d40 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f407effd700 10 client.23829.objecter ms_dispatch 0x7f4088001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f407effd700 10 client.23829.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f407effd700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 4 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f407effd700 15 client.23829.objecter _session_command_op_remove 7 16 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 librados: watch_flush enter 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 librados: watch_flush exit 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.0 2024-03-30T14:58:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6801/3394323575 conn(0x7f406c004730 legacy=0x7f406c002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c0053e0 osd=0 1 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.1 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6805/1850313547 conn(0x7f406c0042f0 legacy=0x7f406c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c004180 osd=1 1 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.2 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6809/3451325043 conn(0x7f406c00d6e0 legacy=0x7f406c00fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c008480 osd=2 1 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408e3f3700 1 -- 172.21.15.28:0/353839802 reap_dead start 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.3 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6813/1506157250 conn(0x7f406c012520 legacy=0x7f406c014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00cd30 osd=3 1 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.4 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.134:6800/3063775514 conn(0x7f406c012080 legacy=0x7f406c019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c560 osd=4 1 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.5 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.134:6804/3668151179 conn(0x7f406c01f5b0 legacy=0x7f406c0219a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c00c6d0 osd=5 1 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.6 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.134:6808/2135224194 conn(0x7f406c025230 legacy=0x7f406c023160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024b10 osd=6 1 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter close_session for osd.7 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.134:6812/3758190298 conn(0x7f406c027d40 legacy=0x7f406c02a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter put_session s=0x7f406c024950 osd=7 1 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 20 client.23829.objecter shutdown clearing up homeless session... 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 10 client.23829.objecter successfully canceled tick 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408e3f3700 1 -- 172.21.15.28:0/353839802 reap_dead start 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6800/3011431093 conn(0x7f40740705d0 legacy=0x7f4074072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> v1:172.21.15.28:6789/0 conn(0x7f4088101130 legacy=0x7f4088104870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 shutdown_connections 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 >> 172.21.15.28:0/353839802 conn(0x7f40880fc9e0 msgr2=0x7f4088110840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 shutdown_connections 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 -- 172.21.15.28:0/353839802 wait complete. 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:45.637+0000 7f408fe56700 1 librados: shutdown 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:58:45.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:45.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:46.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:46.161 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:58:46.161 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T14:58:46.161 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T14:58:46.161 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:58:46.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:46.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:46.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:46.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:46.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:46.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:46.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:46.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1234052616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3339570482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.48934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/353839802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: pgmap v182: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 3 op/s 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:46.819 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.820 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.48934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.820 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:58:46.820 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.820 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:46.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T14:58:46.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:46.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:46.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:46.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1234052616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3339570482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.48934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/353839802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: pgmap v182: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 3 op/s 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.48934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: osdmap e161: 8 total, 8 up, 8 in 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1234052616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3339570482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.48934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/353839802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: pgmap v182: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 3 op/s 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi028-92999-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.48934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi028-92979-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: osdmap e162: 8 total, 8 up, 8 in 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:47.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:47.083 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:47.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:47.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:47.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:47.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:47.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:47.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:47.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:47.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:47.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:47.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:47.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 -- 172.21.15.28:0/1506888794 >> v1:172.21.15.28:6789/0 conn(0x7ff984103230 legacy=0x7ff984103610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 -- 172.21.15.28:0/1506888794 shutdown_connections 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 -- 172.21.15.28:0/1506888794 >> 172.21.15.28:0/1506888794 conn(0x7ff9840fc9e0 msgr2=0x7ff9840fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 -- 172.21.15.28:0/1506888794 shutdown_connections 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 -- 172.21.15.28:0/1506888794 wait complete. 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 librados: starting msgr at 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 librados: starting objecter 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.817+0000 7ff9894e7700 1 Processor -- start 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 -- start start 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 librados: setting wanted keys 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 librados: calling monclient init 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff98410aba0 con 0x7ff984103230 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff98410adb0 con 0x7ff98410b040 2024-03-30T14:58:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff9894e7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff9841a9640 con 0x7ff98410eb50 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff982ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff984103230 0x7ff9841079f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60510/0 (socket says 172.21.15.28:60510) 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff982ffd700 1 -- 172.21.15.28:0/3868049447 learned_addr learned my addr 172.21.15.28:0/3868049447 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter ms_handle_connect 0x7ff98410eb50 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter ms_handle_connect 0x7ff984103230 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022580891 0 0) 0x7ff9841a9640 con 0x7ff98410eb50 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9700031f0 con 0x7ff98410eb50 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182043799 0 0) 0x7ff98410aba0 con 0x7ff984103230 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9841a9640 con 0x7ff984103230 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter ms_handle_connect 0x7ff98410b040 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477781628 0 0) 0x7ff98410adb0 con 0x7ff98410b040 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff98410aba0 con 0x7ff98410b040 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.818+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638609717 0 0) 0x7ff9700031f0 con 0x7ff98410eb50 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff98410adb0 con 0x7ff98410eb50 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff9740018b0 con 0x7ff98410eb50 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2962944348 0 0) 0x7ff9841a9640 con 0x7ff984103230 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9700031f0 con 0x7ff984103230 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff97c001d30 con 0x7ff984103230 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1288985581 0 0) 0x7ff98410aba0 con 0x7ff98410b040 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9841a9640 con 0x7ff98410b040 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff96c001af0 con 0x7ff98410b040 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1780955285 0 0) 0x7ff98410adb0 con 0x7ff98410eb50 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 >> v1:172.21.15.134:6789/0 conn(0x7ff98410b040 legacy=0x7ff984108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 >> v1:172.21.15.28:6789/0 conn(0x7ff984103230 legacy=0x7ff9841079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9841aa820 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9841a9870 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff9894e7700 10 client.56912.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9841a9d30 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff9894e7700 1 librados: init done 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff974003b40 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.819+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff9740049f0 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.820+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98406a400 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.820+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff9740036a0 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(163..163 src has 53..163) ==== 11423+0+0 (unknown 1623760771 0 0) 0x7ff974089120 con 0x7ff98410eb50 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 10 client.56912.objecter ms_dispatch 0x7ff984001550 osd_map(163..163 src has 53..163) 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 3 client.56912.objecter handle_osd_map got epochs [163,163] > 0 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 3 client.56912.objecter handle_osd_map decoding full epoch 163 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 20 client.56912.objecter dump_active .. 0 homeless 2024-03-30T14:58:47.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.821+0000 7ff97bfff700 10 client.56912.objecter ms_handle_connect 0x7ff9700703e0 2024-03-30T14:58:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:47.826+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff974055670 con 0x7ff98410eb50 2024-03-30T14:58:47.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:47.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:47.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:47.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:47.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[41614]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:58:47.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:47.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:47.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:47 smithi028 ceph-mon[29876]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:58:47.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.041+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff984068b30 con 0x7ff98410eb50 2024-03-30T14:58:48.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:48.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.175+0000 7ff97bfff700 1 -- 172.21.15.28:0/3868049447 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v164) ==== 97+0+0 (unknown 3832115197 0 0) 0x7ff9740590c0 con 0x7ff98410eb50 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 10 librados: watch_flush enter 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 10 librados: watch_flush exit 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 20 client.56912.objecter shutdown clearing up homeless session... 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 10 client.56912.objecter successfully canceled tick 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff9700703e0 legacy=0x7ff9700728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.178+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 >> v1:172.21.15.28:6790/0 conn(0x7ff98410eb50 legacy=0x7ff9841a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.179+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 shutdown_connections 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.179+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 >> 172.21.15.28:0/3868049447 conn(0x7ff9840fc9e0 msgr2=0x7ff9840fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.179+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 shutdown_connections 2024-03-30T14:58:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.179+0000 7ff9894e7700 1 -- 172.21.15.28:0/3868049447 wait complete. 2024-03-30T14:58:48.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:48.179+0000 7ff9894e7700 1 librados: shutdown 2024-03-30T14:58:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T14:58:48.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:48.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:48.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:47 smithi134 ceph-mon[33714]: osdmap e163: 8 total, 8 up, 8 in 2024-03-30T14:58:48.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:48.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:48.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:48.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:48.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:58:48.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:48.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:48.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:48.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:48.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:48.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: pgmap v185: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:48.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:48.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3868049447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3212247843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.48958 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/299876133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[41614]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: pgmap v185: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3868049447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3212247843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.48958 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/299876133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:48 smithi028 ceph-mon[29876]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:48.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:49.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:49.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:49.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:49.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: pgmap v185: 460 pgs: 1 active+clean+snaptrim, 128 unknown, 331 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3868049447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.56912 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.56912 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: osdmap e164: 8 total, 8 up, 8 in 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3212247843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:49.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.48958 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:49.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/299876133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:49.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:48 smithi134 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:49.207 INFO:tasks.workunit.client.0.smithi028.stdout: io: Running main() from gmock_main.cc 2024-03-30T14:58:49.207 INFO:tasks.workunit.client.0.smithi028.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-30T14:58:49.207 INFO:tasks.workunit.client.0.smithi028.stdout: io: [----------] Global test environment set-up. 2024-03-30T14:58:49.207 INFO:tasks.workunit.client.0.smithi028.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-30T14:58:49.207 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.Limits (11690 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (5332 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.ReadOp (5084 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.SparseRead (3515 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3678 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4949 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (5362 ms) 2024-03-30T14:58:49.208 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (5791 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2993 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.Trunc (4013 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.Remove (3118 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2881 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.RmXattr (3022 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3006 ms) 2024-03-30T14:58:49.209 INFO:tasks.workunit.client.0.smithi028.stdout: io: [----------] 14 tests from NeoRadosIo (64434 ms total) 2024-03-30T14:58:49.210 INFO:tasks.workunit.client.0.smithi028.stdout: io: 2024-03-30T14:58:49.210 INFO:tasks.workunit.client.0.smithi028.stdout: io: [----------] Global test environment tear-down 2024-03-30T14:58:49.210 INFO:tasks.workunit.client.0.smithi028.stdout: io: [==========] 14 tests from 1 test suite ran. (64435 ms total) 2024-03-30T14:58:49.210 INFO:tasks.workunit.client.0.smithi028.stdout: io: [ PASSED ] 14 tests. 2024-03-30T14:58:49.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:49.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:49.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:49.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:49.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:49.506 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:58:49.506 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T14:58:49.506 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T14:58:49.506 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T14:58:49.506 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:58:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:49.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:49.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:49.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:49.754 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:49.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:49.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:49.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[41614]: from='client.48958 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[41614]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[41614]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[29876]: from='client.48958 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[29876]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:49.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:49 smithi028 ceph-mon[29876]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:58:49.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:49.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:50.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:50.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:50.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:50.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:49 smithi134 ceph-mon[33714]: from='client.48958 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi028-92999-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:50.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:49 smithi134 ceph-mon[33714]: from='client.48955 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi028-92979-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:50.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:49 smithi134 ceph-mon[33714]: osdmap e165: 8 total, 8 up, 8 in 2024-03-30T14:58:50.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:50.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T14:58:50.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:50.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:50.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:50.411 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:50.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:58:50.432 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- 172.21.15.28:0/3578215889 >> v1:172.21.15.28:6790/0 conn(0x7f1f2810aec0 legacy=0x7f1f2810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- 172.21.15.28:0/3578215889 shutdown_connections 2024-03-30T14:58:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- 172.21.15.28:0/3578215889 >> 172.21.15.28:0/3578215889 conn(0x7f1f28077370 msgr2=0x7f1f280fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- 172.21.15.28:0/3578215889 shutdown_connections 2024-03-30T14:58:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- 172.21.15.28:0/3578215889 wait complete. 2024-03-30T14:58:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 librados: starting msgr at 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 librados: starting objecter 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 Processor -- start 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 -- start start 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 librados: setting wanted keys 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.458+0000 7f1f2e26d700 1 librados: calling monclient init 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f2e26d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1f28107980 con 0x7f1f28100740 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f2e26d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1f281a8630 con 0x7f1f2810ea60 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f2e26d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1f281a9810 con 0x7f1f2810aec0 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f27fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f28100740 0x7f1f281047d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34050/0 (socket says 172.21.15.28:34050) 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f27fff700 1 -- 172.21.15.28:0/400332906 learned_addr learned my addr 172.21.15.28:0/400332906 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter ms_handle_connect 0x7f1f2810aec0 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter ms_handle_connect 0x7f1f28100740 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter ms_handle_connect 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111210134 0 0) 0x7f1f281a9810 con 0x7f1f2810aec0 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f100035b0 con 0x7f1f2810aec0 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571318241 0 0) 0x7f1f281a8630 con 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.459+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f281a9810 con 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1388407457 0 0) 0x7f1f281a9810 con 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f281a8630 con 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1f20001bb0 con 0x7f1f2810ea60 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3866313575 0 0) 0x7f1f100035b0 con 0x7f1f2810aec0 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f281a9810 con 0x7f1f2810aec0 2024-03-30T14:58:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1f1c001830 con 0x7f1f2810aec0 2024-03-30T14:58:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1901668819 0 0) 0x7f1f281a8630 con 0x7f1f2810ea60 2024-03-30T14:58:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 >> v1:172.21.15.28:6790/0 conn(0x7f1f2810aec0 legacy=0x7f1f28104ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 >> v1:172.21.15.28:6789/0 conn(0x7f1f28100740 legacy=0x7f1f281047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f281aa9f0 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f2e26d700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1f281a7680 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f2e26d700 10 client.48979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f2e26d700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1f281a7b40 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f2e26d700 1 librados: init done 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1f20003ec0 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.460+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1f20004de0 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.461+0000 7f1f2e26d700 10 client.48979.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.461+0000 7f1f2e26d700 15 client.48979.objecter _session_command_op_assign -1 1 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.461+0000 7f1f2e26d700 20 client.48979.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.461+0000 7f1f2e26d700 10 client.48979.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.461+0000 7f1f2e26d700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1f14002920 con 0x7f1f2810ea60 2024-03-30T14:58:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.462+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1f200060b0 con 0x7f1f2810ea60 2024-03-30T14:58:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(166..166 src has 53..166) ==== 10611+0+0 (unknown 968256842 0 0) 0x7f1f200892f0 con 0x7f1f2810ea60 2024-03-30T14:58:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 10 client.48979.objecter ms_dispatch 0x7f1f28001550 osd_map(166..166 src has 53..166) 2024-03-30T14:58:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 3 client.48979.objecter handle_osd_map got epochs [166,166] > 0 2024-03-30T14:58:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 3 client.48979.objecter handle_osd_map decoding full epoch 166 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 10 client.48979.objecter checking command 1 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter _get_session s=0x7f1f10082d80 osd=4 3 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter put_session s=0x7f1f10082d80 osd=4 3 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 15 client.48979.objecter _session_command_op_remove -1 1 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter _get_session s=0x7f1f10082d80 osd=4 3 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter get_session s=0x7f1f10082d80 osd=4 3 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 15 client.48979.objecter _session_command_op_assign 4 1 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter put_session s=0x7f1f10082d80 osd=4 4 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 10 client.48979.objecter _send_command 1 2024-03-30T14:58:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 --> v1:172.21.15.134:6800/3063775514 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f1008b590 con 0x7f1f10083430 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 20 client.48979.objecter dump_active .. 0 homeless 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=166) ==== 24+0+0 (unknown 3784672393 0 0) 0x7f1f20089700 con 0x7f1f2810ea60 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 10 client.48979.objecter ms_handle_connect 0x7f1f10070380 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.463+0000 7f1f257fa700 10 client.48979.objecter ms_handle_connect 0x7f1f10083430 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.464+0000 7f1f257fa700 1 -- 172.21.15.28:0/400332906 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1f1008b590 con 0x7f1f10083430 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.464+0000 7f1f257fa700 10 client.48979.objecter ms_dispatch 0x7f1f28001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.464+0000 7f1f257fa700 10 client.48979.objecter _finish_command 1 = system:0 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.464+0000 7f1f257fa700 20 client.48979.objecter put_session s=0x7f1f10082d80 osd=4 4 2024-03-30T14:58:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:50.464+0000 7f1f257fa700 15 client.48979.objecter _session_command_op_remove 4 1 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:50.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:50.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:50.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:50.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:50.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:50.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: pgmap v188: 492 pgs: 64 creating+peering, 32 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 55 KiB/s wr, 57 op/s 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:50.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:50 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:50.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:50.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: pgmap v188: 492 pgs: 64 creating+peering, 32 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 55 KiB/s wr, 57 op/s 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: pgmap v188: 492 pgs: 64 creating+peering, 32 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 55 KiB/s wr, 57 op/s 2024-03-30T14:58:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:58:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: osdmap e166: 8 total, 8 up, 8 in 2024-03-30T14:58:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:50 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:51.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:51.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:51.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: Running main() from gmock_main.cc 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [----------] Global test environment set-up. 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-30T14:58:51.190 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (13659 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (6805 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3768 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (5068 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (4905 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (4265 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (4615 ms) 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-30T14:58:51.191 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3458 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3967 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2854 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (4003 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3018 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2986 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3005 ms) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (66376 ms total) 2024-03-30T14:58:51.192 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: 2024-03-30T14:58:51.193 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [----------] Global test environment tear-down 2024-03-30T14:58:51.193 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (66376 ms total) 2024-03-30T14:58:51.193 INFO:tasks.workunit.client.0.smithi028.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-30T14:58:51.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:51.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:51.381 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:51.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-30T14:58:51.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:51.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:51.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:51.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:51.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:51.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:58:51.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3327579476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2790375915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:51.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:51.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:52.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:52.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:52.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:52.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3327579476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2790375915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: osdmap e167: 8 total, 8 up, 8 in 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3327579476' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1328251834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2790375915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:58:52.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:917 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94326471858816 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:54:57.389505+0000", 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 234.83498151800001, 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0143281449999999, 2024-03-30T14:58:52.227 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 917 2024-03-30T14:58:52.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389505+0000", 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389505+0000", 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389508+0000", 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9900000000000002e-06 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389519+0000", 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0617e-05 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389536+0000", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7238e-05 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389550+0000", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3613e-05 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.396980+0000", 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074302750000000001 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.396990+0000", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6569999999999992e-06 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.400791+0000", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0038010900000001 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.400953+0000", 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016236500000000001 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402085+0000", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011316469999999999 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402190+0000", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010480999999999999 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402593+0000", 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040285000000000001 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402647+0000", 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4255000000000001e-05 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403834+0000", 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011867379999999999 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:924 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94326471858816 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:54:57.517418+0000", 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 234.70706846900001, 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.88591363700000003, 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 924 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517418+0000", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517418+0000", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517420+0000", 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0920000000000001e-06 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517425+0000", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.835e-06 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517437+0000", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1479e-05 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517448+0000", 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0969e-05 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.238 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.238 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517523+0000", 2024-03-30T14:58:52.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4740000000000006e-05 2024-03-30T14:58:52.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517534+0000", 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1192999999999999e-05 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.401224+0000", 2024-03-30T14:58:52.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.88369055500000004 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.401274+0000", 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9820000000000001e-05 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402290+0000", 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010158739999999999 2024-03-30T14:58:52.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402387+0000", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7085999999999998e-05 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403039+0000", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00065200800000000004 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403122+0000", 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2751999999999996e-05 2024-03-30T14:58:52.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403332+0000", 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021023400000000001 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.242 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25069.0:3 35.f 35:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:58:52.243 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.200377+0000", 2024-03-30T14:58:52.243 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.024109996, 2024-03-30T14:58:52.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0835292080000001, 2024-03-30T14:58:52.243 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25069", 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3135164836", 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200379+0000", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6729999999999999e-06 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200387+0000", 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1799999999999996e-06 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200395+0000", 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3850000000000002e-06 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200404+0000", 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6219999999999998e-06 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200468+0000", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4142999999999999e-05 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200475+0000", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4580000000000004e-06 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259289+0000", 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0588138250000001 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259412+0000", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012236699999999999 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,6", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.260458+0000", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010460790000000001 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.261342+0000", 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00088464100000000003 2024-03-30T14:58:52.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.268237+0000", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068943329999999999 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283840+0000", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015603258 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283874+0000", 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3736999999999999e-05 2024-03-30T14:58:52.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283906+0000", 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2506999999999999e-05 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25048.0:1 46.6s0 46:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:56.132808+0000", 2024-03-30T14:58:52.250 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 56.091678676000001, 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2874541669999999, 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25048", 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132810+0000", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0250000000000001e-06 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132815+0000", 2024-03-30T14:58:52.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9869999999999999e-06 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132823+0000", 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0560000000000005e-06 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132833+0000", 2024-03-30T14:58:52.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5200000000000003e-06 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145608+0000", 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012775376999999999 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145613+0000", 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5079999999999999e-06 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.254 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397079+0000", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2514661540000001 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397194+0000", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115407 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397531+0000", 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000336745 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.402316+0000", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047853660000000001 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420194+0000", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017877613000000001 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420262+0000", 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8409000000000002e-05 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.256 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34886.0:1 50.0s0 50:0c828f53:test-rados-api-smithi028-93540-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.337241+0000", 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 54.887246468000001, 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3929710050000002, 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34886", 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4069139007", 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T14:58:52.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.258 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337242+0000", 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.792e-06 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337249+0000", 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9430000000000003e-06 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337260+0000", 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0535e-05 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337269+0000", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8009999999999993e-06 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342443+0000", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051742710000000003 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342445+0000", 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.446e-06 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718228+0000", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3757831299999999 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718356+0000", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012730400000000001 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719072+0000", 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071628200000000005 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.720051+0000", 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00097890299999999993 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730124+0000", 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010073343 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730212+0000", 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7255000000000004e-05 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2059 11.7 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head [writefull 0~25165836] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.945782+0000", 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 54.278705369999997, 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.086346471999999994, 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.263 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2059 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945784+0000", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0310000000000001e-06 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969470+0000", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023686577 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969496+0000", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6210000000000001e-05 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969507+0000", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0723000000000001e-05 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969596+0000", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8653999999999995e-05 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969758+0000", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016222 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969914+0000", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015554699999999999 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030515+0000", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060601742 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030889+0000", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037352300000000001 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032049+0000", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011604639999999999 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032088+0000", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8621000000000002e-05 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032128+0000", 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0160000000000002e-05 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2061 11.7 11:e5291d64:test-rados-api-smithi028-93257-16::big.copy:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.033297+0000", 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 54.191189825000002, 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062226268000000001, 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.269 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2061 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033297+0000", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033297+0000", 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033298+0000", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2600000000000001e-07 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033302+0000", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2939999999999999e-06 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033310+0000", 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9160000000000001e-06 2024-03-30T14:58:52.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033318+0000", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2740000000000002e-06 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033350+0000", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1976e-05 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.033422+0000", 2024-03-30T14:58:52.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2729000000000001e-05 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.079860+0000", 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.046437337000000002 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.089954+0000", 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010094786 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.095176+0000", 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005221669 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.095460+0000", 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028380300000000001 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.095496+0000", 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5760000000000003e-05 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.095523+0000", 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7798e-05 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2069 11.7 11:e17e9421:test-rados-api-smithi028-93257-17::big:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.274133+0000", 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 53.950354453999999, 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.081199162000000005, 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.275 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2069 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.274133+0000", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.274133+0000", 2024-03-30T14:58:52.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.274135+0000", 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1959999999999998e-06 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.297785+0000", 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023650082999999999 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.297799+0000", 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4194e-05 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.297809+0000", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0436e-05 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.297862+0000", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2802e-05 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.297980+0000", 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117278 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298095+0000", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115542 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.341305+0000", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.043210070000000003 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355224+0000", 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013918783000000001 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355245+0000", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0893000000000001e-05 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355286+0000", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1298000000000003e-05 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355332+0000", 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5587000000000002e-05 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.281 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:5 13.b 13:d4d78c7e:test-rados-api-smithi028-93525-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.472376+0000", 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 53.752110737999999, 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15821444200000001, 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T14:58:52.282 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 5 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472378+0000", 2024-03-30T14:58:52.283 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2000000000000001e-06 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472386+0000", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8760000000000005e-06 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472397+0000", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0900000000000001e-05 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472409+0000", 2024-03-30T14:58:52.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1379999999999999e-05 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472488+0000", 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9796999999999995e-05 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472495+0000", 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5749999999999997e-06 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517331+0000", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044836187 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517472+0000", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014135100000000001 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.521687+0000", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042141230000000002 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.534881+0000", 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01319443 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.553213+0000", 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018331719 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630475+0000", 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.077262443 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630530+0000", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4910000000000001e-05 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630591+0000", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0550999999999997e-05 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25483.0:2 63.1b 63:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.989489+0000", 2024-03-30T14:58:52.288 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 51.234998433000001, 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9747056280000002, 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25483", 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1101472932", 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989490+0000", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5880000000000001e-06 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989498+0000", 2024-03-30T14:58:52.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.396e-06 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989506+0000", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8459999999999997e-06 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989518+0000", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1309e-05 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989578+0000", 2024-03-30T14:58:52.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0393000000000003e-05 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989589+0000", 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0829000000000001e-05 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914500+0000", 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9249114629999999 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914631+0000", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013069899999999999 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914772+0000", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014078499999999999 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.917517+0000", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027451939999999998 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962191+0000", 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044673933999999998 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964110+0000", 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019193019999999999 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964150+0000", 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0212000000000001e-05 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964194+0000", 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3677999999999998e-05 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35456.0:2 69.0 69:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.789274+0000", 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 50.435212903, 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729640450000001, 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.295 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35456", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3389323125", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789276+0000", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.317e-06 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789282+0000", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0510000000000002e-06 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789297+0000", 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4766999999999999e-05 2024-03-30T14:58:52.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.298 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.298 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789307+0000", 2024-03-30T14:58:52.298 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8810000000000008e-06 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806392+0000", 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017084881 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806402+0000", 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9089999999999995e-06 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885233+0000", 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0788315669999999 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885345+0000", 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111907 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885469+0000", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012389899999999999 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.901898+0000", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016428991 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.902088+0000", 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000189459 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962138+0000", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060050234000000001 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962190+0000", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2432000000000002e-05 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962238+0000", 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7750000000000002e-05 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.302 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25489.0:3 64.f 64:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:02.914202+0000", 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 49.310285186999998, 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.084413231000000005, 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25489", 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2934336176", 2024-03-30T14:58:52.303 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914204+0000", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.905e-06 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914214+0000", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9399999999999997e-06 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914224+0000", 2024-03-30T14:58:52.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0190999999999999e-05 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914235+0000", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0686999999999999e-05 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914298+0000", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3063000000000003e-05 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914308+0000", 2024-03-30T14:58:52.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0642000000000001e-05 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995802+0000", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.081493828000000004 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995889+0000", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7015000000000006e-05 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995993+0000", 2024-03-30T14:58:52.307 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104177 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996545+0000", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055223100000000003 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996864+0000", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031808999999999999 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998524+0000", 2024-03-30T14:58:52.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016600040000000001 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998579+0000", 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5368000000000003e-05 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998615+0000", 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6090000000000002e-05 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:9 111.0 111:0487255f:test-rados-api-smithi028-93525-8::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e140)", 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:23.274964+0000", 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 28.949522662, 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053347366, 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 9 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274964+0000", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274964+0000", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274967+0000", 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2249999999999999e-06 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274974+0000", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1740000000000003e-06 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274987+0000", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3379e-05 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.274999+0000", 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1647e-05 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.280098+0000", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050996610000000001 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.280205+0000", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010664500000000001 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.280811+0000", 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00060607599999999999 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.286470+0000", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056586839999999998 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.291929+0000", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005459696 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.328240+0000", 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036310823999999998 2024-03-30T14:58:52.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.315 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.315 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.328279+0000", 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8625999999999999e-05 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.328312+0000", 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2728999999999998e-05 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49076.0:2 135.1b 135:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:29.173584+0000", 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 23.050903363, 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0617338199999999, 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49076", 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2505490190", 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173586+0000", 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9530000000000002e-06 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173594+0000", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4069999999999993e-06 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173615+0000", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0788000000000002e-05 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173632+0000", 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7589999999999999e-05 2024-03-30T14:58:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176962+0000", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033293039999999999 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176970+0000", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9889999999999992e-06 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226049+0000", 2024-03-30T14:58:52.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.049079015 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226159+0000", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110021 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226288+0000", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012942800000000001 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.229542+0000", 2024-03-30T14:58:52.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032538939999999998 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.231623+0000", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020808440000000001 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235241+0000", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036180539999999999 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235288+0000", 2024-03-30T14:58:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7382000000000001e-05 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235317+0000", 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9150999999999999e-05 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:21 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [write 0~128] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:30.172513+0000", 2024-03-30T14:58:52.325 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.051974123000001, 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.059672243999999999, 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T14:58:52.326 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.172513+0000", 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.172513+0000", 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.172515+0000", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4219999999999999e-06 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.172524+0000", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.568e-06 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.191018+0000", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018493809 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.191044+0000", 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6843999999999999e-05 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.217853+0000", 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.026808964000000001 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.217997+0000", 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014330199999999999 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.218158+0000", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016173000000000001 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.221089+0000", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002930944 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.225867+0000", 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047775630000000003 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.231844+0000", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059773539999999998 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.232020+0000", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017555399999999999 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.232185+0000", 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016519000000000001 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:23 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [write 0~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.346422+0000", 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 20.878064554000002, 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12244129300000001, 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 23 2024-03-30T14:58:52.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346424+0000", 2024-03-30T14:58:52.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9690000000000001e-06 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346432+0000", 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5979999999999999e-06 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346442+0000", 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0431000000000001e-05 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346453+0000", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0896e-05 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356731+0000", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010277319 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356836+0000", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105664 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:58:52.335 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.357052+0000", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021593299999999999 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.366074+0000", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090220760000000004 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.388374+0000", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022299229 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468758+0000", 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080384786999999999 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468818+0000", 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9493000000000001e-05 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468864+0000", 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5898e-05 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46235.0:781 130.18 130:1eb75461:::test_774:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e147)", 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.412773+0000", 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 20.811713549, 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.065519376000000004, 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46235", 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 781 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412773+0000", 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412773+0000", 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412775+0000", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6619999999999999e-06 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412782+0000", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5760000000000002e-06 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412792+0000", 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9720000000000004e-06 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412797+0000", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7200000000000003e-06 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412836+0000", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8838000000000003e-05 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.412913+0000", 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6882e-05 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.478293+0000", 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.065379725999999999 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:25 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [rollback 8] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.211894+0000", 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.012593126999999, 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12770631599999999, 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.342 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 25 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211896+0000", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0789999999999999e-06 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211903+0000", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2359999999999998e-06 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211912+0000", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4109999999999996e-06 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211923+0000", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1038e-05 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212228+0000", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030524600000000001 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212330+0000", 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102433 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212488+0000", 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015767099999999999 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.213555+0000", 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001067045 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.292128+0000", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.078572806999999995 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339484+0000", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.047356374999999999 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339548+0000", 2024-03-30T14:58:52.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4126000000000002e-05 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339600+0000", 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1848999999999998e-05 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.56873.0:10 165.0 165:0384ab8e:::oid:head [write 128~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:47.173414+0000", 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.0510732249999997, 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.039412282999999999, 2024-03-30T14:58:52.348 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.56873", 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2397599544", 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 10 2024-03-30T14:58:52.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173414+0000", 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173414+0000", 2024-03-30T14:58:52.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173416+0000", 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.729e-06 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173425+0000", 2024-03-30T14:58:52.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0409999999999998e-06 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173435+0000", 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0728e-05 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.173448+0000", 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2702e-05 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.177884+0000", 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0044360279999999998 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.177990+0000", 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105847 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.178164+0000", 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017441000000000001 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.186098+0000", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007934188 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.207890+0000", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.021792063 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.212725+0000", 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048344720000000002 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.212781+0000", 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6147999999999998e-05 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.212826+0000", 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4926999999999999e-05 2024-03-30T14:58:52.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.56873.0:17 165.0 165:0384ab8e:::oid:head [write 128~128] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:48.183973+0000", 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.0405143389999996, 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.029032643, 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.56873", 2024-03-30T14:58:52.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2397599544", 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 17 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.183973+0000", 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.183973+0000", 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.183976+0000", 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0680000000000001e-06 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T14:58:52.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.183985+0000", 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5990000000000008e-06 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.184003+0000", 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7620000000000001e-05 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T14:58:52.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.184015+0000", 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2198000000000001e-05 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.184110+0000", 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4839000000000005e-05 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.184117+0000", 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.328e-06 2024-03-30T14:58:52.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.198254+0000", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014136525 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.198349+0000", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5074999999999994e-05 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.200711+0000", 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002362598 2024-03-30T14:58:52.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.204347+0000", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036351790000000001 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.210960+0000", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0066132200000000004 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212902+0000", 2024-03-30T14:58:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001941978 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212959+0000", 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7377000000000003e-05 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.213005+0000", 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6038999999999997e-05 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T14:58:52.364 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T14:58:52.365 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:58:52.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:52.367 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:52.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:52.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:52.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:52.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:52.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:52.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 -- 172.21.15.28:0/715676541 >> v1:172.21.15.28:6789/0 conn(0x7fb768101130 legacy=0x7fb768101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 -- 172.21.15.28:0/715676541 shutdown_connections 2024-03-30T14:58:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 -- 172.21.15.28:0/715676541 >> 172.21.15.28:0/715676541 conn(0x7fb7680fc9e0 msgr2=0x7fb7680fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 -- 172.21.15.28:0/715676541 shutdown_connections 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 -- 172.21.15.28:0/715676541 wait complete. 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 librados: starting msgr at 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.764+0000 7fb76d143700 1 librados: starting objecter 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 Processor -- start 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 -- start start 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 librados: setting wanted keys 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 librados: calling monclient init 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7681a7150 con 0x7fb768101130 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7681a8370 con 0x7fb76810d960 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb76d143700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7681a95b0 con 0x7fb768109dc0 2024-03-30T14:58:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb766d9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb768101130 0x7fb768106500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34088/0 (socket says 172.21.15.28:34088) 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb766d9d700 1 -- 172.21.15.28:0/3851895655 learned_addr learned my addr 172.21.15.28:0/3851895655 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb74ffff700 10 client.?.objecter ms_handle_connect 0x7fb768101130 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb74ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb74ffff700 10 client.?.objecter ms_handle_connect 0x7fb768109dc0 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.765+0000 7fb74ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309592652 0 0) 0x7fb7681a7150 con 0x7fb768101130 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7500039c0 con 0x7fb768101130 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 10 client.?.objecter ms_handle_connect 0x7fb76810d960 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 846285647 0 0) 0x7fb7681a95b0 con 0x7fb768109dc0 2024-03-30T14:58:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7681a7150 con 0x7fb768109dc0 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835909421 0 0) 0x7fb7681a8370 con 0x7fb76810d960 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7681a95b0 con 0x7fb76810d960 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1889148040 0 0) 0x7fb7500039c0 con 0x7fb768101130 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7681a8370 con 0x7fb768101130 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb758001350 con 0x7fb768101130 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1700294610 0 0) 0x7fb7681a7150 con 0x7fb768109dc0 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7500039c0 con 0x7fb768109dc0 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb75c001890 con 0x7fb768109dc0 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4089786930 0 0) 0x7fb7681a95b0 con 0x7fb76810d960 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7681a7150 con 0x7fb76810d960 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb760001c30 con 0x7fb76810d960 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3823466574 0 0) 0x7fb7681a8370 con 0x7fb768101130 2024-03-30T14:58:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6790/0 conn(0x7fb768109dc0 legacy=0x7fb7681a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.134:6789/0 conn(0x7fb76810d960 legacy=0x7fb7681a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7681aa7f0 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7681a97e0 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb76d143700 10 client.23877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7681a9ca0 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb76d143700 1 librados: init done 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb758001d00 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.766+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb758004ee0 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.767+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb754005100 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.768+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb758005120 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(168..168 src has 53..168) ==== 10168+0+0 (unknown 3058254285 0 0) 0x7fb75808a100 con 0x7fb768101130 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 osd_map(168..168 src has 53..168) 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 3 client.23877.objecter handle_osd_map got epochs [168,168] > 0 2024-03-30T14:58:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 3 client.23877.objecter handle_osd_map decoding full epoch 168 2024-03-30T14:58:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 20 client.23877.objecter dump_active .. 0 homeless 2024-03-30T14:58:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.769+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb750070420 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.773+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v168) ==== 54+0+16 (unknown 307017714 0 2420000192) 0x7fb758056b40 con 0x7fb768101130 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 10 client.23877.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 1 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 1 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 0 1 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 4 2024-03-30T14:58:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 10 client.23877.objecter _send_command 1 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb754004b50 con 0x7fb754004730 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.774+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb754004730 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.775+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754004b50 con 0x7fb754004730 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.775+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.775+0000 7fb74ffff700 10 client.23877.objecter _finish_command 1 = system:0 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.775+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 4 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.775+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 0 1 2024-03-30T14:58:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 10 client.23877.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 2 2024-03-30T14:58:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 2 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540053e0 osd=0 3 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 0 2 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 4 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 10 client.23877.objecter _send_command 2 2024-03-30T14:58:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.792+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754007b00 con 0x7fb754004730 2024-03-30T14:58:52.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754007b00 con 0x7fb754004730 2024-03-30T14:58:52.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb74ffff700 10 client.23877.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 4 2024-03-30T14:58:52.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 0 2 2024-03-30T14:58:52.798 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: pgmap v191: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 KiB/s wr, 55 op/s 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:52.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:52.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: pgmap v191: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 KiB/s wr, 55 op/s 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:52.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:52.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:52 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 10 client.23877.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 1 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 4 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 10 client.23877.objecter _send_command 3 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.794+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb754002160 con 0x7fb7540042a0 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb7540042a0 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754002160 con 0x7fb7540042a0 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 3: 0 ) 2024-03-30T14:58:52.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 10 client.23877.objecter _finish_command 3 = system:0 2024-03-30T14:58:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 4 2024-03-30T14:58:52.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.796+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 1 3 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 10 client.23877.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 4 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 4 2024-03-30T14:58:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754004130 osd=1 3 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 1 4 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 4 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 10 client.23877.objecter _send_command 4 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.814+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754001600 con 0x7fb7540042a0 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.815+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754001600 con 0x7fb7540042a0 2024-03-30T14:58:52.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.815+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.815+0000 7fb74ffff700 10 client.23877.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.815+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 4 2024-03-30T14:58:52.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.815+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 1 4 2024-03-30T14:58:52.818 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:58:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 10 client.23877.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 5 2024-03-30T14:58:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 5 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 2 5 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 4 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 10 client.23877.objecter _send_command 5 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb754008c30 con 0x7fb754011af0 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.816+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb754011af0 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.817+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754008c30 con 0x7fb754011af0 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.817+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 5: 0 ) 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.817+0000 7fb74ffff700 10 client.23877.objecter _finish_command 5 = system:0 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.817+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 4 2024-03-30T14:58:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.817+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 2 5 2024-03-30T14:58:52.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 10 client.23877.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 6 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 6 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754008480 osd=2 3 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 2 6 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 4 2024-03-30T14:58:52.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 10 client.23877.objecter _send_command 6 2024-03-30T14:58:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.833+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754007fc0 con 0x7fb754011af0 2024-03-30T14:58:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754007fc0 con 0x7fb754011af0 2024-03-30T14:58:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 10 client.23877.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 4 2024-03-30T14:58:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 2 6 2024-03-30T14:58:52.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 10 client.23877.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 7 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 7 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 3 7 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 4 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 10 client.23877.objecter _send_command 7 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb754016170 con 0x7fb754015600 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.835+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb754015600 2024-03-30T14:58:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.836+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754016170 con 0x7fb754015600 2024-03-30T14:58:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.836+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 7: 0 ) 2024-03-30T14:58:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.836+0000 7fb74ffff700 10 client.23877.objecter _finish_command 7 = system:0 2024-03-30T14:58:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.836+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 4 2024-03-30T14:58:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.836+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 3 7 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 10 client.23877.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 8 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 8 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754008250 osd=3 3 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 3 8 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 4 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 10 client.23877.objecter _send_command 8 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.853+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754017ca0 con 0x7fb754015600 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.854+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754017ca0 con 0x7fb754015600 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.854+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.854+0000 7fb74ffff700 10 client.23877.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.854+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 4 2024-03-30T14:58:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.854+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 3 8 2024-03-30T14:58:52.856 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 10 client.23877.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 9 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 9 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 4 9 2024-03-30T14:58:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 4 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 10 client.23877.objecter _send_command 9 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb75401ebc0 con 0x7fb75401afc0 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.855+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb75401afc0 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.856+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb75401ebc0 con 0x7fb75401afc0 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.856+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 9: 0 ) 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.856+0000 7fb74ffff700 10 client.23877.objecter _finish_command 9 = system:0 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.856+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 4 2024-03-30T14:58:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.856+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 4 9 2024-03-30T14:58:52.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 10 client.23877.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 10 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 10 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754017ca0 osd=4 3 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 4 10 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 4 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 10 client.23877.objecter _send_command 10 2024-03-30T14:58:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.872+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754020f90 con 0x7fb75401afc0 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.874+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754020f90 con 0x7fb75401afc0 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.874+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.874+0000 7fb74ffff700 10 client.23877.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.874+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 4 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.874+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 4 10 2024-03-30T14:58:52.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 10 client.23877.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 11 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 11 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 5 11 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 4 2024-03-30T14:58:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 10 client.23877.objecter _send_command 11 2024-03-30T14:58:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb7540202e0 con 0x7fb754020d40 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.875+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb754020d40 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.876+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb7540202e0 con 0x7fb754020d40 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.876+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 11: 0 ) 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.876+0000 7fb74ffff700 10 client.23877.objecter _finish_command 11 = system:0 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.876+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 4 2024-03-30T14:58:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.876+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 5 11 2024-03-30T14:58:52.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 10 client.23877.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 12 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 12 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540213f0 osd=5 3 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 5 12 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 4 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 10 client.23877.objecter _send_command 12 2024-03-30T14:58:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.893+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb7540202e0 con 0x7fb754020d40 2024-03-30T14:58:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb7540202e0 con 0x7fb754020d40 2024-03-30T14:58:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb74ffff700 10 client.23877.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 4 2024-03-30T14:58:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 5 12 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 10 client.23877.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 13 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 13 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 6 13 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 4 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 10 client.23877.objecter _send_command 13 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.895+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb754017ab0 con 0x7fb754023c80 2024-03-30T14:58:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.896+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb754023c80 2024-03-30T14:58:52.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.897+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754017ab0 con 0x7fb754023c80 2024-03-30T14:58:52.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.897+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 13: 0 ) 2024-03-30T14:58:52.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.897+0000 7fb74ffff700 10 client.23877.objecter _finish_command 13 = system:0 2024-03-30T14:58:52.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.897+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 4 2024-03-30T14:58:52.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.897+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 6 13 2024-03-30T14:58:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:52.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 10 client.23877.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 14 2024-03-30T14:58:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 14 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb7540202e0 osd=6 3 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 6 14 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 4 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 10 client.23877.objecter _send_command 14 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.913+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754028f90 con 0x7fb754023c80 2024-03-30T14:58:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.915+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754028f90 con 0x7fb754023c80 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.915+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.915+0000 7fb74ffff700 10 client.23877.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.915+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 4 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.915+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 6 14 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:58:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 10 client.23877.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 15 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 15 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 7 15 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 4 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 10 client.23877.objecter _send_command 15 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb754029b20 con 0x7fb7540296b0 2024-03-30T14:58:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.916+0000 7fb74ffff700 10 client.23877.objecter ms_handle_connect 0x7fb7540296b0 2024-03-30T14:58:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.917+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb754029b20 con 0x7fb7540296b0 2024-03-30T14:58:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.917+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 15: 0 ) 2024-03-30T14:58:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.917+0000 7fb74ffff700 10 client.23877.objecter _finish_command 15 = system:0 2024-03-30T14:58:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.917+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 4 2024-03-30T14:58:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.917+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 7 15 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.933+0000 7fb76d143700 10 client.23877.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign -1 16 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 20 client.23877.objecter _get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 15 client.23877.objecter _session_command_op_remove -1 16 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 20 client.23877.objecter get_session s=0x7fb754028f90 osd=7 3 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 15 client.23877.objecter _session_command_op_assign 7 16 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 4 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 10 client.23877.objecter _send_command 16 2024-03-30T14:58:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.934+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb754029b20 con 0x7fb7540296b0 2024-03-30T14:58:52.936 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:58:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.935+0000 7fb74ffff700 1 -- 172.21.15.28:0/3851895655 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb754029b20 con 0x7fb7540296b0 2024-03-30T14:58:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.935+0000 7fb74ffff700 10 client.23877.objecter ms_dispatch 0x7fb768001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:58:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.935+0000 7fb74ffff700 10 client.23877.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:58:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.935+0000 7fb74ffff700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 4 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.935+0000 7fb74ffff700 15 client.23877.objecter _session_command_op_remove 7 16 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 librados: watch_flush enter 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 librados: watch_flush exit 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.0 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6801/3394323575 conn(0x7fb754004730 legacy=0x7fb754002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540053e0 osd=0 1 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.1 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6805/1850313547 conn(0x7fb7540042a0 legacy=0x7fb75400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754004130 osd=1 1 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.2 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6809/3451325043 conn(0x7fb754011af0 legacy=0x7fb754013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008480 osd=2 1 2024-03-30T14:58:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.3 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6813/1506157250 conn(0x7fb754015600 legacy=0x7fb754015a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754008250 osd=3 1 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.4 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.134:6800/3063775514 conn(0x7fb75401afc0 legacy=0x7fb75401d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754017ca0 osd=4 1 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.5 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.134:6804/3668151179 conn(0x7fb754020d40 legacy=0x7fb75401ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540213f0 osd=5 1 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.6 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76759e700 1 -- 172.21.15.28:0/3851895655 reap_dead start 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.134:6808/2135224194 conn(0x7fb754023c80 legacy=0x7fb754026070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76759e700 1 -- 172.21.15.28:0/3851895655 reap_dead start 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76759e700 1 -- 172.21.15.28:0/3851895655 reap_dead start 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76759e700 1 -- 172.21.15.28:0/3851895655 reap_dead start 2024-03-30T14:58:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb7540202e0 osd=6 1 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter close_session for osd.7 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.134:6812/3758190298 conn(0x7fb7540296b0 legacy=0x7fb7540277c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter put_session s=0x7fb754028f90 osd=7 1 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 20 client.23877.objecter shutdown clearing up homeless session... 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 10 client.23877.objecter successfully canceled tick 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb750070420 legacy=0x7fb7500728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> v1:172.21.15.28:6789/0 conn(0x7fb768101130 legacy=0x7fb768106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 shutdown_connections 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 >> 172.21.15.28:0/3851895655 conn(0x7fb7680fc9e0 msgr2=0x7fb76806a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 shutdown_connections 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 -- 172.21.15.28:0/3851895655 wait complete. 2024-03-30T14:58:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:52.936+0000 7fb76d143700 1 librados: shutdown 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:58:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:58:53.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:53.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: pgmap v191: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 KiB/s wr, 55 op/s 2024-03-30T14:58:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi028-92999-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi028-93441-10"}]': finished 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi028-92979-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: osdmap e168: 8 total, 8 up, 8 in 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:52 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:58:53.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:53.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:53.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T14:58:53.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:53.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:53.577 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:53.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:53.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:53.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:53.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:53.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:54.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:54.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:54.036 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:54.036 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3851895655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.037 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:54.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:54.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:54.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3851895655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:54.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3851895655' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:58:54.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: osdmap e169: 8 total, 8 up, 8 in 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:58:54.160 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]: dispatch 2024-03-30T14:58:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:54.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:54.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:54.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:54.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:54.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:54.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:54.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:54.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:54.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:54.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:54.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:54.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:54.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:54.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: pgmap v194: 388 pgs: 27 creating+peering, 5 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 45 KiB/s wr, 45 op/s 2024-03-30T14:58:54.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]': finished 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/784231600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3007875987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.56957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:54.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: pgmap v194: 388 pgs: 27 creating+peering, 5 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 45 KiB/s wr, 45 op/s 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:54.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]': finished 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/784231600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3007875987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.56957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:54.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:54.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:54.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:54.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:55.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:55.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: pgmap v194: 388 pgs: 27 creating+peering, 5 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 45 KiB/s wr, 45 op/s 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-13", "mode": "writeback"}]': finished 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: osdmap e170: 8 total, 8 up, 8 in 2024-03-30T14:58:55.068 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/784231600' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3007875987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.56957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:55.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:55.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:55.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:55.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:55.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:55.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:55.520 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:55.520 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:55.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:55.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:55.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:55.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:55.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:55.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:55.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:55.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.4 scrub starts 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.4 scrub ok 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.0 scrub starts 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.0 scrub ok 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.6 scrub starts 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.6 scrub ok 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:58:55.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: from='client.56957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.3 scrub starts 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[29876]: 168.3 scrub ok 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:55.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.4 scrub starts 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.4 scrub ok 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.0 scrub starts 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.0 scrub ok 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.6 scrub starts 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.6 scrub ok 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: from='client.56957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:58:55.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.3 scrub starts 2024-03-30T14:58:55.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:55 smithi028 ceph-mon[41614]: 168.3 scrub ok 2024-03-30T14:58:56.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:56.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:56.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.0"}]: dispatch 2024-03-30T14:58:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.1"}]: dispatch 2024-03-30T14:58:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.2"}]: dispatch 2024-03-30T14:58:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.3"}]: dispatch 2024-03-30T14:58:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.4"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.5"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.6"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.7"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.8"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "168.9"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.4 scrub starts 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.4 scrub ok 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.0 scrub starts 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.0 scrub ok 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.6 scrub starts 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.6 scrub ok 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi028-93441-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:58:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi028-92979-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: from='client.56957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi028-92999-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: osdmap e171: 8 total, 8 up, 8 in 2024-03-30T14:58:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.3 scrub starts 2024-03-30T14:58:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:55 smithi134 ceph-mon[33714]: 168.3 scrub ok 2024-03-30T14:58:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:56.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:56.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T14:58:56.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:56.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.334+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/932638792 >> v1:172.21.15.28:6790/0 conn(0x7f8a0410ea90 legacy=0x7f8a04110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.334+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/932638792 shutdown_connections 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.334+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/932638792 >> 172.21.15.28:0/932638792 conn(0x7f8a040fdb30 msgr2=0x7f8a040fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.334+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/932638792 shutdown_connections 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.334+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/932638792 wait complete. 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 librados: starting msgr at 2024-03-30T14:58:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 librados: starting objecter 2024-03-30T14:58:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 Processor -- start 2024-03-30T14:58:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 -- start start 2024-03-30T14:58:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 librados: setting wanted keys 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 librados: calling monclient init 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8a040761f0 con 0x7f8a0410ea90 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8a04076400 con 0x7f8a04102260 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.335+0000 7f8a0b9ad700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8a041aba60 con 0x7f8a0410aef0 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f8a09f4a700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8a0410ea90 0x7f8a041a8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34192/0 (socket says 172.21.15.28:34192) 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f8a09f4a700 1 -- 172.21.15.28:0/2882507325 learned_addr learned my addr 172.21.15.28:0/2882507325 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter ms_handle_connect 0x7f8a0410ea90 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter ms_handle_connect 0x7f8a0410aef0 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter ms_handle_connect 0x7f8a04102260 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630804911 0 0) 0x7f8a04076400 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89f00035b0 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766066667 0 0) 0x7f8a041aba60 con 0x7f8a0410aef0 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a04076400 con 0x7f8a0410aef0 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1830289081 0 0) 0x7f89f00035b0 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a041aba60 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8a000017e0 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1568091566 0 0) 0x7f8a04076400 con 0x7f8a0410aef0 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89f00035b0 con 0x7f8a0410aef0 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f89f4001830 con 0x7f8a0410aef0 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1531866594 0 0) 0x7f8a041aba60 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 >> v1:172.21.15.28:6790/0 conn(0x7f8a0410aef0 legacy=0x7f8a04074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 >> v1:172.21.15.28:6789/0 conn(0x7f8a0410ea90 legacy=0x7f8a041a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a041acc80 con 0x7f8a04102260 2024-03-30T14:58:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.336+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a041abc90 con 0x7f8a04102260 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 10 client.49027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8a00003b20 con 0x7f8a04102260 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a041ac150 con 0x7f8a04102260 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8a00004b40 con 0x7f8a04102260 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 1 librados: init done 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 10 client.49027.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 15 client.49027.objecter _session_command_op_assign -1 1 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 20 client.49027.objecter _get_session osd=-1 returning homeless 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 10 client.49027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.337+0000 7f8a0b9ad700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8a04103740 con 0x7f8a04102260 2024-03-30T14:58:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.338+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8a00003680 con 0x7f8a04102260 2024-03-30T14:58:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(172..172 src has 53..172) ==== 10696+0+0 (unknown 665001356 0 0) 0x7f8a000890f0 con 0x7f8a04102260 2024-03-30T14:58:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 10 client.49027.objecter ms_dispatch 0x7f8a04001550 osd_map(172..172 src has 53..172) 2024-03-30T14:58:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 3 client.49027.objecter handle_osd_map got epochs [172,172] > 0 2024-03-30T14:58:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 3 client.49027.objecter handle_osd_map decoding full epoch 172 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 10 client.49027.objecter checking command 1 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter _get_session s=0x7f89f0082510 osd=5 3 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter put_session s=0x7f89f0082510 osd=5 3 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 15 client.49027.objecter _session_command_op_remove -1 1 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter _get_session s=0x7f89f0082510 osd=5 3 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter get_session s=0x7f89f0082510 osd=5 3 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 15 client.49027.objecter _session_command_op_assign 5 1 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter put_session s=0x7f89f0082510 osd=5 4 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 10 client.49027.objecter _send_command 1 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 --> v1:172.21.15.134:6804/3668151179 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89f0086990 con 0x7f89f0082c00 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 20 client.49027.objecter dump_active .. 0 homeless 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 10 client.49027.objecter ms_handle_connect 0x7f89f0070300 2024-03-30T14:58:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (unknown 2307180195 0 0) 0x7f8a000894c0 con 0x7f8a04102260 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.339+0000 7f89fa7fc700 10 client.49027.objecter ms_handle_connect 0x7f89f0082c00 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.340+0000 7f89fa7fc700 1 -- 172.21.15.28:0/2882507325 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f89f0086990 con 0x7f89f0082c00 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.340+0000 7f89fa7fc700 10 client.49027.objecter ms_dispatch 0x7f8a04001550 command_reply(tid 1: 0 ) 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.340+0000 7f89fa7fc700 10 client.49027.objecter _finish_command 1 = system:0 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.340+0000 7f89fa7fc700 20 client.49027.objecter put_session s=0x7f89f0082510 osd=5 4 2024-03-30T14:58:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:56.340+0000 7f89fa7fc700 15 client.49027.objecter _session_command_op_remove 5 1 2024-03-30T14:58:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:58:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:58:56.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:56.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:56.481 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:56.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:56.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:56.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:56.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:56.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.1 scrub starts 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.1 scrub ok 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.8 scrub starts 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.8 scrub ok 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: pgmap v197: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 KiB/s wr, 71 op/s 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:56.860 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.5 scrub starts 2024-03-30T14:58:56.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.5 scrub ok 2024-03-30T14:58:56.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:58:56.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.9 scrub starts 2024-03-30T14:58:56.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:56 smithi134 ceph-mon[33714]: 168.9 scrub ok 2024-03-30T14:58:56.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:56.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:56.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.1 scrub starts 2024-03-30T14:58:56.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.1 scrub ok 2024-03-30T14:58:56.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.8 scrub starts 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.8 scrub ok 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: pgmap v197: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 KiB/s wr, 71 op/s 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.5 scrub starts 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.5 scrub ok 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.9 scrub starts 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[41614]: 168.9 scrub ok 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.1 scrub starts 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.1 scrub ok 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.8 scrub starts 2024-03-30T14:58:56.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.8 scrub ok 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: pgmap v197: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 KiB/s wr, 71 op/s 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.5 scrub starts 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.5 scrub ok 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: osdmap e172: 8 total, 8 up, 8 in 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.9 scrub starts 2024-03-30T14:58:56.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:56 smithi028 ceph-mon[29876]: 168.9 scrub ok 2024-03-30T14:58:57.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:57.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.071+0000 7f130bce1700 1 -- 172.21.15.28:0/847347414 >> v1:172.21.15.28:6790/0 conn(0x7f130410aee0 legacy=0x7f130410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.071+0000 7f130bce1700 1 -- 172.21.15.28:0/847347414 shutdown_connections 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.071+0000 7f130bce1700 1 -- 172.21.15.28:0/847347414 >> 172.21.15.28:0/847347414 conn(0x7f13040fdb20 msgr2=0x7f13040fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.071+0000 7f130bce1700 1 -- 172.21.15.28:0/847347414 shutdown_connections 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.071+0000 7f130bce1700 1 -- 172.21.15.28:0/847347414 wait complete. 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 librados: starting msgr at 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 librados: starting objecter 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 Processor -- start 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 -- start start 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 librados: setting wanted keys 2024-03-30T14:58:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 librados: calling monclient init 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13041a9370 con 0x7f130410ea80 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13041aa590 con 0x7f130410aee0 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130bce1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f13041ab7d0 con 0x7f1304102250 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f1309a7d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1304102250 0x7f1304107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52118/0 (socket says 172.21.15.28:52118) 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f1309a7d700 1 -- 172.21.15.28:0/3907542289 learned_addr learned my addr 172.21.15.28:0/3907542289 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f130a27e700 1 --1- 172.21.15.28:0/3907542289 >> v1:172.21.15.28:6789/0 conn(0x7f130410ea80 0x7f13041a7a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34202/0 (socket says 172.21.15.28:34202) 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter ms_handle_connect 0x7f130410ea80 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter ms_handle_connect 0x7f1304102250 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter ms_handle_connect 0x7f130410aee0 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T14:58:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26308318 0 0) 0x7f13041ab7d0 con 0x7f1304102250 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f00031f0 con 0x7f1304102250 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.072+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 347926527 0 0) 0x7f13041a9370 con 0x7f130410ea80 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13041ab7d0 con 0x7f130410ea80 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639876529 0 0) 0x7f13041aa590 con 0x7f130410aee0 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13041a9370 con 0x7f130410aee0 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1667824385 0 0) 0x7f12f00031f0 con 0x7f1304102250 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13041aa590 con 0x7f1304102250 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1300001930 con 0x7f1304102250 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 712201371 0 0) 0x7f13041a9370 con 0x7f130410aee0 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12f00031f0 con 0x7f130410aee0 2024-03-30T14:58:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2516747671 0 0) 0x7f13041ab7d0 con 0x7f130410ea80 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13041a9370 con 0x7f130410ea80 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12f4001840 con 0x7f130410aee0 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12fc0014b0 con 0x7f130410ea80 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2363743670 0 0) 0x7f13041aa590 con 0x7f1304102250 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 >> v1:172.21.15.134:6789/0 conn(0x7f130410aee0 legacy=0x7f13041a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 >> v1:172.21.15.28:6789/0 conn(0x7f130410ea80 legacy=0x7f13041a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13041aca10 con 0x7f1304102250 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13041aba00 con 0x7f1304102250 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f130bce1700 10 client.56984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f13041abec0 con 0x7f1304102250 2024-03-30T14:58:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f130bce1700 1 librados: init done 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1300001d00 con 0x7f1304102250 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.073+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1300005e40 con 0x7f1304102250 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.074+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e8005120 con 0x7f1304102250 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.075+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1300003950 con 0x7f1304102250 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.075+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(172..172 src has 53..172) ==== 10696+0+0 (unknown 665001356 0 0) 0x7f130008a020 con 0x7f1304102250 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.075+0000 7f12faffd700 10 client.56984.objecter ms_dispatch 0x7f1304001550 osd_map(172..172 src has 53..172) 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.075+0000 7f12faffd700 3 client.56984.objecter handle_osd_map got epochs [172,172] > 0 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.075+0000 7f12faffd700 3 client.56984.objecter handle_osd_map decoding full epoch 172 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.076+0000 7f12faffd700 20 client.56984.objecter dump_active .. 0 homeless 2024-03-30T14:58:57.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.076+0000 7f12faffd700 10 client.56984.objecter ms_handle_connect 0x7f12f00703e0 2024-03-30T14:58:57.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.080+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1300056830 con 0x7f1304102250 2024-03-30T14:58:57.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:57.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:57.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:57.322+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f12e8005410 con 0x7f1304102250 2024-03-30T14:58:57.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:57.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:57.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:57.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:57.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:58:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:57.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:58:57.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:58:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:58:57.658 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:58:57.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-30T14:58:57.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:57.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:57.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:57.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:57.903 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:57.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:57.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: 168.2 scrub starts 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: 168.2 scrub ok 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/336311746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3907542289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[29876]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: 168.2 scrub starts 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: 168.2 scrub ok 2024-03-30T14:58:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/336311746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3907542289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:57 smithi028 ceph-mon[41614]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:57.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:57.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:58.058 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: 168.2 scrub starts 2024-03-30T14:58:58.058 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: 168.2 scrub ok 2024-03-30T14:58:58.058 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: osdmap e173: 8 total, 8 up, 8 in 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/336311746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.56972 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3907542289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:58.059 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:57 smithi134 ceph-mon[33714]: from='client.56984 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:58:58.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:58.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.188+0000 7f12faffd700 1 -- 172.21.15.28:0/3907542289 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v174) ==== 91+0+0 (unknown 1783110574 0 0) 0x7f130005a280 con 0x7f1304102250 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.195+0000 7f130bce1700 10 librados: watch_flush enter 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 10 librados: watch_flush exit 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 20 client.56984.objecter shutdown clearing up homeless session... 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 10 client.56984.objecter successfully canceled tick 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 >> v1:172.21.15.28:6800/3011431093 conn(0x7f12f00703e0 legacy=0x7f12f00728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 >> v1:172.21.15.28:6790/0 conn(0x7f1304102250 legacy=0x7f1304107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 shutdown_connections 2024-03-30T14:58:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.197+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 >> 172.21.15.28:0/3907542289 conn(0x7f13040fdb20 msgr2=0x7f1304107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:58:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.202+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 shutdown_connections 2024-03-30T14:58:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.206+0000 7f130bce1700 1 -- 172.21.15.28:0/3907542289 wait complete. 2024-03-30T14:58:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:58:58.208+0000 7f130bce1700 1 librados: shutdown 2024-03-30T14:58:58.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T14:58:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:58.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:58.350 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:58.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:58.470 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:58:58.470 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:58:58.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:58:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:58:58.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:58:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:58:58.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:58.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:58.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:58:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:58.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:58:58.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:58.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:58.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:58.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.002 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:59.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:59.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:59.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: pgmap v200: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:58:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:58:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:58:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: 5.1e scrub starts 2024-03-30T14:58:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:58 smithi134 ceph-mon[33714]: 5.1e scrub ok 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: pgmap v200: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: 5.1e scrub starts 2024-03-30T14:58:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[41614]: 5.1e scrub ok 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: pgmap v200: 460 pgs: 72 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 24 op/s 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: from='client.56972 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi028-92999-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1802135113' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi028-92979-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: from='client.56984 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: osdmap e174: 8 total, 8 up, 8 in 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: 5.1e scrub starts 2024-03-30T14:58:59.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:58 smithi028 ceph-mon[29876]: 5.1e scrub ok 2024-03-30T14:58:59.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:58:59.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T14:58:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:59.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:58:59.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:58:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.381 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:59.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:58:59.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:59.493 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:58:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:59.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:59.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:58:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.634 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:58:59.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:58:59.702 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:58:59.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:58:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:58:59 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:58:59.883 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T14:58:59.883 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T14:58:59.883 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: osdmap e175: 8 total, 8 up, 8 in 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:58:59 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:58:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:58:59.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:58:59.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:58:59.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:00.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:00.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:00.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:00.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:00.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:00.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:00.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:00.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:00.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:00.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:00.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: pgmap v203: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/727746987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2"}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: pgmap v203: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:00.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/727746987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2"}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:00 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: pgmap v203: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:00.990 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: osdmap e176: 8 total, 8 up, 8 in 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/727746987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2"}]: dispatch 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.991 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:00 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:00.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:01.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:01.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:01.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:01.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:01.249 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:59:01.249 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:59:01.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:01.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.387+0000 7fba0d63f700 1 -- 172.21.15.28:0/2794570520 >> v1:172.21.15.28:6790/0 conn(0x7fba08107010 legacy=0x7fba081073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 -- 172.21.15.28:0/2794570520 shutdown_connections 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 -- 172.21.15.28:0/2794570520 >> 172.21.15.28:0/2794570520 conn(0x7fba080761d0 msgr2=0x7fba080765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 -- 172.21.15.28:0/2794570520 shutdown_connections 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 -- 172.21.15.28:0/2794570520 wait complete. 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 librados: starting msgr at 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 librados: starting objecter 2024-03-30T14:59:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 Processor -- start 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 -- start start 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 librados: setting wanted keys 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.388+0000 7fba0d63f700 1 librados: calling monclient init 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba0d63f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba081a70b0 con 0x7fba08107010 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba0d63f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba081a82d0 con 0x7fba0810eab0 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba0d63f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba081a9510 con 0x7fba0810aee0 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba067fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fba0810aee0 0x7fba081a1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34406/0 (socket says 172.21.15.28:34406) 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba06ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba08107010 0x7fba08100ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55990/0 (socket says 172.21.15.28:55990) 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba067fc700 1 -- 172.21.15.28:0/2301364588 learned_addr learned my addr 172.21.15.28:0/2301364588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter ms_handle_connect 0x7fba08107010 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter ms_handle_connect 0x7fba0810aee0 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter ms_handle_connect 0x7fba0810eab0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941205643 0 0) 0x7fba081a70b0 con 0x7fba08107010 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9f00031f0 con 0x7fba08107010 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16241213 0 0) 0x7fba081a9510 con 0x7fba0810aee0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba081a70b0 con 0x7fba0810aee0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 711108987 0 0) 0x7fba081a82d0 con 0x7fba0810eab0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba081a9510 con 0x7fba0810eab0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1777602404 0 0) 0x7fba081a70b0 con 0x7fba0810aee0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba081a82d0 con 0x7fba0810aee0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb9fc0017f0 con 0x7fba0810aee0 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2218719804 0 0) 0x7fb9f00031f0 con 0x7fba08107010 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba081a70b0 con 0x7fba08107010 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb9f8001510 con 0x7fba08107010 2024-03-30T14:59:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4181301100 0 0) 0x7fba081a9510 con 0x7fba0810eab0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9f00031f0 con 0x7fba0810eab0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fba00001bb0 con 0x7fba0810eab0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1788066823 0 0) 0x7fba081a82d0 con 0x7fba0810aee0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.134:6789/0 conn(0x7fba0810eab0 legacy=0x7fba081a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6789/0 conn(0x7fba08107010 legacy=0x7fba08100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba081aa750 con 0x7fba0810aee0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.389+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fba081a9740 con 0x7fba0810aee0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fba0d63f700 10 client.57008.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fba081a9c80 con 0x7fba0810aee0 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fba0d63f700 1 librados: init done 2024-03-30T14:59:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb9fc003b20 con 0x7fba0810aee0 2024-03-30T14:59:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb9fc005c10 con 0x7fba0810aee0 2024-03-30T14:59:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.390+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb9f4005100 con 0x7fba0810aee0 2024-03-30T14:59:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.391+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb9fc003680 con 0x7fba0810aee0 2024-03-30T14:59:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(177..177 src has 53..177) ==== 11494+0+0 (unknown 3670397908 0 0) 0x7fb9fc08a150 con 0x7fba0810aee0 2024-03-30T14:59:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 osd_map(177..177 src has 53..177) 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 3 client.57008.objecter handle_osd_map got epochs [177,177] > 0 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 3 client.57008.objecter handle_osd_map decoding full epoch 177 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 20 client.57008.objecter dump_active .. 0 homeless 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v177) ==== 54+0+16 (unknown 3776437891 0 2420000192) 0x7fb9fc08a580 con 0x7fba0810aee0 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f0070410 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 10 client.57008.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 1 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 1 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 0 1 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 4 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 10 client.57008.objecter _send_command 1 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9f4004b50 con 0x7fb9f4004730 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.392+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f4004730 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.393+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f4004b50 con 0x7fb9f4004730 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.393+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.393+0000 7fb9effff700 10 client.57008.objecter _finish_command 1 = system:0 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.393+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 4 2024-03-30T14:59:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.393+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 0 1 2024-03-30T14:59:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:01.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 10 client.57008.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 2 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 2 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40053e0 osd=0 3 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 0 2 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 4 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 10 client.57008.objecter _send_command 2 2024-03-30T14:59:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.410+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f400be00 con 0x7fb9f4004730 2024-03-30T14:59:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f400be00 con 0x7fb9f4004730 2024-03-30T14:59:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 10 client.57008.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 4 2024-03-30T14:59:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 0 2 2024-03-30T14:59:01.414 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 10 client.57008.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 3 2024-03-30T14:59:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 1 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 4 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 10 client.57008.objecter _send_command 3 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb9f40021c0 con 0x7fb9f40042c0 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.412+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f40042c0 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.413+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f40021c0 con 0x7fb9f40042c0 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.413+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.413+0000 7fb9effff700 10 client.57008.objecter _finish_command 3 = system:0 2024-03-30T14:59:01.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.413+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 4 2024-03-30T14:59:01.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.413+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 1 3 2024-03-30T14:59:01.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 10 client.57008.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 4 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 4 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4004150 osd=1 3 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 1 4 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 4 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 10 client.57008.objecter _send_command 4 2024-03-30T14:59:01.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.430+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f4001600 con 0x7fb9f40042c0 2024-03-30T14:59:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f4001600 con 0x7fb9f40042c0 2024-03-30T14:59:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 10 client.57008.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 4 2024-03-30T14:59:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 1 4 2024-03-30T14:59:01.435 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 10 client.57008.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 5 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 5 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 2 5 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 4 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 10 client.57008.objecter _send_command 5 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb9f40154f0 con 0x7fb9f40119e0 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.432+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f40119e0 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.433+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f40154f0 con 0x7fb9f40119e0 2024-03-30T14:59:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.433+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:01.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.433+0000 7fb9effff700 10 client.57008.objecter _finish_command 5 = system:0 2024-03-30T14:59:01.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.433+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 4 2024-03-30T14:59:01.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.433+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 2 5 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 10 client.57008.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 6 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 6 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f400c780 osd=2 3 2024-03-30T14:59:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 2 6 2024-03-30T14:59:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 4 2024-03-30T14:59:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 10 client.57008.objecter _send_command 6 2024-03-30T14:59:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.450+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f400c260 con 0x7fb9f40119e0 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f400c260 con 0x7fb9f40119e0 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fb9effff700 10 client.57008.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 4 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 2 6 2024-03-30T14:59:01.453 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 10 client.57008.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 7 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 7 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 3 7 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 4 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 10 client.57008.objecter _send_command 7 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.451+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb9f401a330 con 0x7fb9f4016820 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f4016820 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f401a330 con 0x7fb9f4016820 2024-03-30T14:59:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 10 client.57008.objecter _finish_command 7 = system:0 2024-03-30T14:59:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 4 2024-03-30T14:59:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.452+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 3 7 2024-03-30T14:59:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 10 client.57008.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 8 2024-03-30T14:59:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 8 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4011030 osd=3 3 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 3 8 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 4 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 10 client.57008.objecter _send_command 8 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.469+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f4010b10 con 0x7fb9f4016820 2024-03-30T14:59:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f4010b10 con 0x7fb9f4016820 2024-03-30T14:59:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 10 client.57008.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 4 2024-03-30T14:59:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 3 8 2024-03-30T14:59:01.472 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 10 client.57008.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 9 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 9 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 4 9 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 4 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 10 client.57008.objecter _send_command 9 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb9f4010e10 con 0x7fb9f4016380 2024-03-30T14:59:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.471+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f4016380 2024-03-30T14:59:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.472+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f4010e10 con 0x7fb9f4016380 2024-03-30T14:59:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.472+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.472+0000 7fb9effff700 10 client.57008.objecter _finish_command 9 = system:0 2024-03-30T14:59:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.472+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 4 2024-03-30T14:59:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.472+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 4 9 2024-03-30T14:59:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 10 client.57008.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 10 2024-03-30T14:59:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 10 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f4010b10 osd=4 3 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 4 10 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 4 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 10 client.57008.objecter _send_command 10 2024-03-30T14:59:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.489+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f401ed90 con 0x7fb9f4016380 2024-03-30T14:59:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f401ed90 con 0x7fb9f4016380 2024-03-30T14:59:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fb9effff700 10 client.57008.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 4 2024-03-30T14:59:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 4 10 2024-03-30T14:59:01.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 10 client.57008.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 11 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 11 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 5 11 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 4 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 10 client.57008.objecter _send_command 11 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.491+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb9f40210d0 con 0x7fb9f401f4b0 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.492+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f401f4b0 2024-03-30T14:59:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.493+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f40210d0 con 0x7fb9f401f4b0 2024-03-30T14:59:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.493+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.493+0000 7fb9effff700 10 client.57008.objecter _finish_command 11 = system:0 2024-03-30T14:59:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.493+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 4 2024-03-30T14:59:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.493+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 5 11 2024-03-30T14:59:01.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 10 client.57008.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 12 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 12 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f401ed90 osd=5 3 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 5 12 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 4 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 10 client.57008.objecter _send_command 12 2024-03-30T14:59:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.510+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f40210d0 con 0x7fb9f401f4b0 2024-03-30T14:59:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f40210d0 con 0x7fb9f401f4b0 2024-03-30T14:59:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 10 client.57008.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 4 2024-03-30T14:59:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 5 12 2024-03-30T14:59:01.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:01.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 10 client.57008.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 13 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 13 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 6 13 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 4 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 10 client.57008.objecter _send_command 13 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb9f4021500 con 0x7fb9f4023c40 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.512+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f4023c40 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.513+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f4021500 con 0x7fb9f4023c40 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.513+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:01.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.513+0000 7fb9effff700 10 client.57008.objecter _finish_command 13 = system:0 2024-03-30T14:59:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.513+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 4 2024-03-30T14:59:01.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.513+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 6 13 2024-03-30T14:59:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:01.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 10 client.57008.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 14 2024-03-30T14:59:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 14 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40210d0 osd=6 3 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 6 14 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 4 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 10 client.57008.objecter _send_command 14 2024-03-30T14:59:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.533+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f40277b0 con 0x7fb9f4023c40 2024-03-30T14:59:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f40277b0 con 0x7fb9f4023c40 2024-03-30T14:59:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fb9effff700 10 client.57008.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 4 2024-03-30T14:59:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 6 14 2024-03-30T14:59:01.537 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 10 client.57008.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 15 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 15 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 7 15 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 4 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 10 client.57008.objecter _send_command 15 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.535+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb9f4028a80 con 0x7fb9f4027ed0 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.536+0000 7fb9effff700 10 client.57008.objecter ms_handle_connect 0x7fb9f4027ed0 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.537+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb9f4028a80 con 0x7fb9f4027ed0 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.537+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.537+0000 7fb9effff700 10 client.57008.objecter _finish_command 15 = system:0 2024-03-30T14:59:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.537+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 4 2024-03-30T14:59:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.537+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 7 15 2024-03-30T14:59:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 10 client.57008.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign -1 16 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 20 client.57008.objecter _get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_remove -1 16 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 20 client.57008.objecter get_session s=0x7fb9f40277b0 osd=7 3 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 15 client.57008.objecter _session_command_op_assign 7 16 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 4 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 10 client.57008.objecter _send_command 16 2024-03-30T14:59:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.554+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9f4028a80 con 0x7fb9f4027ed0 2024-03-30T14:59:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.556+0000 7fb9effff700 1 -- 172.21.15.28:0/2301364588 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9f4028a80 con 0x7fb9f4027ed0 2024-03-30T14:59:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.556+0000 7fb9effff700 10 client.57008.objecter ms_dispatch 0x7fba08001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.556+0000 7fb9effff700 10 client.57008.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.556+0000 7fb9effff700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 4 2024-03-30T14:59:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.556+0000 7fb9effff700 15 client.57008.objecter _session_command_op_remove 7 16 2024-03-30T14:59:01.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 librados: watch_flush enter 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 librados: watch_flush exit 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.0 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6801/3394323575 conn(0x7fb9f4004730 legacy=0x7fb9f4002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40053e0 osd=0 1 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.1 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6805/1850313547 conn(0x7fb9f40042c0 legacy=0x7fb9f400f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4004150 osd=1 1 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.2 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6809/3451325043 conn(0x7fb9f40119e0 legacy=0x7fb9f4013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f400c780 osd=2 1 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.3 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6813/1506157250 conn(0x7fb9f4016820 legacy=0x7fb9f4018c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4011030 osd=3 1 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.4 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.134:6800/3063775514 conn(0x7fb9f4016380 legacy=0x7fb9f401d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba077fe700 1 -- 172.21.15.28:0/2301364588 reap_dead start 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f4010b10 osd=4 1 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba077fe700 1 -- 172.21.15.28:0/2301364588 reap_dead start 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.5 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.134:6804/3668151179 conn(0x7fb9f401f4b0 legacy=0x7fb9f401f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f401ed90 osd=5 1 2024-03-30T14:59:01.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.6 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.134:6808/2135224194 conn(0x7fb9f4023c40 legacy=0x7fb9f4026030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40210d0 osd=6 1 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter close_session for osd.7 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.134:6812/3758190298 conn(0x7fb9f4027ed0 legacy=0x7fb9f4028330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter put_session s=0x7fb9f40277b0 osd=7 1 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba077fe700 1 -- 172.21.15.28:0/2301364588 reap_dead start 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 20 client.57008.objecter shutdown clearing up homeless session... 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 10 client.57008.objecter successfully canceled tick 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb9f0070410 legacy=0x7fb9f00728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> v1:172.21.15.28:6790/0 conn(0x7fba0810aee0 legacy=0x7fba081a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 shutdown_connections 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 >> 172.21.15.28:0/2301364588 conn(0x7fba080761d0 msgr2=0x7fba08112080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:01.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 shutdown_connections 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 -- 172.21.15.28:0/2301364588 wait complete. 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:01.557+0000 7fba0d63f700 1 librados: shutdown 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:01.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:01.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:01.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:01.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:01.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:01.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.923 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.924 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.924 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: 168.7 scrub starts 2024-03-30T14:59:01.924 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: 168.7 scrub ok 2024-03-30T14:59:01.924 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2301364588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:01.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T14:59:01.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: 168.7 scrub starts 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: 168.7 scrub ok 2024-03-30T14:59:01.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2301364588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2495254587' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi028-92999-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi028-92979-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi028-93257-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: osdmap e177: 8 total, 8 up, 8 in 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-30T14:59:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: 168.7 scrub starts 2024-03-30T14:59:01.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: 168.7 scrub ok 2024-03-30T14:59:01.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2301364588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:02.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:02.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:02.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:02.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:02.186 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:59:02.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:02.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:02.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:59:02.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:02.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:02.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:02.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:02.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:02.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:02.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:02.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:02.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:02.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:02.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:02.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:03.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:03.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: pgmap v206: 460 pgs: 96 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:59:03.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: pgmap v206: 460 pgs: 96 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:02 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.186 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:03.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:03.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:03.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: pgmap v206: 460 pgs: 96 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-30T14:59:03.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: osdmap e178: 8 total, 8 up, 8 in 2024-03-30T14:59:03.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:02 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-30T14:59:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:03.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:03.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:03.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.738+0000 7f2b84f56700 1 -- 172.21.15.28:0/251521846 >> v1:172.21.15.28:6790/0 conn(0x7f2b80102230 legacy=0x7f2b80102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.738+0000 7f2b84f56700 1 -- 172.21.15.28:0/251521846 shutdown_connections 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.738+0000 7f2b84f56700 1 -- 172.21.15.28:0/251521846 >> 172.21.15.28:0/251521846 conn(0x7f2b800fdb00 msgr2=0x7f2b800fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.738+0000 7f2b84f56700 1 -- 172.21.15.28:0/251521846 shutdown_connections 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.738+0000 7f2b84f56700 1 -- 172.21.15.28:0/251521846 wait complete. 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 librados: starting msgr at 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 librados: starting objecter 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 Processor -- start 2024-03-30T14:59:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 -- start start 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 librados: setting wanted keys 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 librados: calling monclient init 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b801a94a0 con 0x7f2b8010aec0 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b801aa6c0 con 0x7f2b8010ea60 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b84f56700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b801ab900 con 0x7f2b80102230 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.739+0000 7f2b7e59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2b80102230 0x7f2b80075f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34488/0 (socket says 172.21.15.28:34488) 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b7e59c700 1 -- 172.21.15.28:0/1423889779 learned_addr learned my addr 172.21.15.28:0/1423889779 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter ms_handle_connect 0x7f2b8010aec0 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter ms_handle_connect 0x7f2b80102230 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter ms_handle_connect 0x7f2b8010ea60 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353550363 0 0) 0x7f2b801a94a0 con 0x7f2b8010aec0 2024-03-30T14:59:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b680031f0 con 0x7f2b8010aec0 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3652366861 0 0) 0x7f2b801ab900 con 0x7f2b80102230 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b801a94a0 con 0x7f2b80102230 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185041445 0 0) 0x7f2b801aa6c0 con 0x7f2b8010ea60 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b801ab900 con 0x7f2b8010ea60 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3907100619 0 0) 0x7f2b801a94a0 con 0x7f2b80102230 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b801aa6c0 con 0x7f2b80102230 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b74001430 con 0x7f2b80102230 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1001988504 0 0) 0x7f2b680031f0 con 0x7f2b8010aec0 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b801a94a0 con 0x7f2b8010aec0 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b70001800 con 0x7f2b8010aec0 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4024907573 0 0) 0x7f2b801ab900 con 0x7f2b8010ea60 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b680031f0 con 0x7f2b8010ea60 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b78001af0 con 0x7f2b8010ea60 2024-03-30T14:59:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2822186045 0 0) 0x7f2b801a94a0 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 >> v1:172.21.15.28:6790/0 conn(0x7f2b80102230 legacy=0x7f2b80075f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 >> v1:172.21.15.134:6789/0 conn(0x7f2b8010ea60 legacy=0x7f2b801a7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.740+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b801acb40 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b801abb30 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 10 client.23940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b801abff0 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 1 librados: init done 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2b70003b40 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b700049f0 con 0x7f2b8010aec0 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 10 client.23940.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 15 client.23940.objecter _session_command_op_assign -1 1 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 20 client.23940.objecter _get_session osd=-1 returning homeless 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 10 client.23940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.741+0000 7f2b84f56700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2b60002920 con 0x7f2b8010aec0 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.742+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2b700036a0 con 0x7f2b8010aec0 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(179..179 src has 53..179) ==== 11554+0+0 (unknown 3455265839 0 0) 0x7f2b70089090 con 0x7f2b8010aec0 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 10 client.23940.objecter ms_dispatch 0x7f2b80001550 osd_map(179..179 src has 53..179) 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 3 client.23940.objecter handle_osd_map got epochs [179,179] > 0 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 3 client.23940.objecter handle_osd_map decoding full epoch 179 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 10 client.23940.objecter checking command 1 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter _get_session s=0x7f2b680842f0 osd=6 3 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter put_session s=0x7f2b680842f0 osd=6 3 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 15 client.23940.objecter _session_command_op_remove -1 1 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter _get_session s=0x7f2b680842f0 osd=6 3 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter get_session s=0x7f2b680842f0 osd=6 3 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 15 client.23940.objecter _session_command_op_assign 6 1 2024-03-30T14:59:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter put_session s=0x7f2b680842f0 osd=6 4 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 10 client.23940.objecter _send_command 1 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 --> v1:172.21.15.134:6808/2135224194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b68088770 con 0x7f2b680849e0 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 20 client.23940.objecter dump_active .. 0 homeless 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.743+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=179) ==== 24+0+0 (unknown 3283815715 0 0) 0x7f2b70089470 con 0x7f2b8010aec0 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.744+0000 7f2b6f7fe700 10 client.23940.objecter ms_handle_connect 0x7f2b68070500 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.744+0000 7f2b6f7fe700 10 client.23940.objecter ms_handle_connect 0x7f2b680849e0 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.745+0000 7f2b6f7fe700 1 -- 172.21.15.28:0/1423889779 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2b68088770 con 0x7f2b680849e0 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.745+0000 7f2b6f7fe700 10 client.23940.objecter ms_dispatch 0x7f2b80001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.745+0000 7f2b6f7fe700 10 client.23940.objecter _finish_command 1 = system:0 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.745+0000 7f2b6f7fe700 20 client.23940.objecter put_session s=0x7f2b680842f0 osd=6 4 2024-03-30T14:59:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:03.745+0000 7f2b6f7fe700 15 client.23940.objecter _session_command_op_remove 6 1 2024-03-30T14:59:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:59:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:59:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:59:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:59:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:59:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:59:03.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:03.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:03.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:03.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:04.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:04.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T14:59:04.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T14:59:04.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T14:59:04.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T14:59:04.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/447404369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2993758800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T14:59:04.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 4.3 scrub starts 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: 4.3 scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T14:59:04.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/447404369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2993758800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 4.3 scrub starts 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: 4.3 scrub ok 2024-03-30T14:59:04.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:04.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:04.173 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T14:59:04.173 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T14:59:04.173 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T14:59:04.173 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: osdmap e179: 8 total, 8 up, 8 in 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/447404369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1734910647' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2993758800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.49066 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:04.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T14:59:04.175 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T14:59:04.175 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 4.3 scrub starts 2024-03-30T14:59:04.175 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: 4.3 scrub ok 2024-03-30T14:59:04.175 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:04.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:04.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:04.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:04.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:04.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:04.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:59:04.480 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:59:04.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-30T14:59:04.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:04.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:04.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:04.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:04.800 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:59:04.800 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T14:59:04.800 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T14:59:04.800 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:59:04.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:59:04.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:04.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:04.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:05.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:05.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:05.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T14:59:05.197 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: seed 93257 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (4957 ms) 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (19 ms) 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (1025 ms) 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (26 ms) 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-30T14:59:05.198 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (19 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-03-30T14:59:05.199 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1559 ms) 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (336 ms) 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: done waiting, doing copies 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: done waiting 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60942 ms) 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (60 ms) 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-30T14:59:05.200 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4926 ms) 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-30T14:59:05.201 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-30T14:59:05.202 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-30T14:59:05.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:05.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:05.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: pgmap v209: 460 pgs: 70 creating+peering, 26 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:59:05.428 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T14:59:05.429 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:05.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:05.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T14:59:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: pgmap v209: 460 pgs: 70 creating+peering, 26 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T14:59:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: pgmap v209: 460 pgs: 70 creating+peering, 26 unknown, 364 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi028-92979-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.34853 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi028-93257-1","app":"app1","key":"key1"}]': finished 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.49066 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi028-92999-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: osdmap e180: 8 total, 8 up, 8 in 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T14:59:05.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:05.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:05.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:05.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:05.734 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:59:05.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:05.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:05.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:05.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:05.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:06.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:06.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:06.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:06.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:06.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 22 tests from LibRadosMi api_snapshots: Running main() from gmock_main.cc 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (5756 ms) 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4550 ms) 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2902 ms) 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1932 ms) 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (15140 ms total) 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: 2024-03-30T14:59:06.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (6326 ms) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6901 ms) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (7382 ms) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (20609 ms total) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2873 ms) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2006 ms) 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-30T14:59:06.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1996 ms) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2006 ms) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8881 ms total) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4060 ms) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3958 ms) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8018 ms total) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (81676 ms total) 2024-03-30T14:59:06.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-30T14:59:06.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T14:59:06.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T14:59:06.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T14:59:06.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:06.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: osdmap e181: 8 total, 8 up, 8 in 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3482878893' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:06.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:06.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:06.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:06.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:06.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:06.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:06.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:06.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.840 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:06.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.925+0000 7f0731fc2700 1 -- 172.21.15.28:0/647802488 >> v1:172.21.15.28:6790/0 conn(0x7f072c102260 legacy=0x7f072c102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.925+0000 7f0731fc2700 1 -- 172.21.15.28:0/647802488 shutdown_connections 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.925+0000 7f0731fc2700 1 -- 172.21.15.28:0/647802488 >> 172.21.15.28:0/647802488 conn(0x7f072c0fdb10 msgr2=0x7f072c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.925+0000 7f0731fc2700 1 -- 172.21.15.28:0/647802488 shutdown_connections 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.925+0000 7f0731fc2700 1 -- 172.21.15.28:0/647802488 wait complete. 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 librados: starting msgr at 2024-03-30T14:59:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 librados: starting objecter 2024-03-30T14:59:06.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 Processor -- start 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 -- start start 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 librados: setting wanted keys 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.926+0000 7f0731fc2700 1 librados: calling monclient init 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0731fc2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f072c1079e0 con 0x7f072c10ea90 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0731fc2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f072c1a8660 con 0x7f072c10aef0 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0731fc2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f072c1a9840 con 0x7f072c102260 2024-03-30T14:59:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f072bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f072c10ea90 0x7f072c1a5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56204/0 (socket says 172.21.15.28:56204) 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f072b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f072c102260 0x7f072c104830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34578/0 (socket says 172.21.15.28:34578) 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f072bfff700 1 -- 172.21.15.28:0/1854429982 learned_addr learned my addr 172.21.15.28:0/1854429982 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter ms_handle_connect 0x7f072c10ea90 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter ms_handle_connect 0x7f072c102260 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240093854 0 0) 0x7f072c1a9840 con 0x7f072c102260 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07140035b0 con 0x7f072c102260 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter ms_handle_connect 0x7f072c10aef0 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94632571 0 0) 0x7f072c1079e0 con 0x7f072c10ea90 2024-03-30T14:59:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f072c1a9840 con 0x7f072c10ea90 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360399745 0 0) 0x7f072c1a8660 con 0x7f072c10aef0 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f072c1079e0 con 0x7f072c10aef0 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4155595726 0 0) 0x7f07140035b0 con 0x7f072c102260 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f072c1a8660 con 0x7f072c102260 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f071c0015d0 con 0x7f072c102260 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1888356642 0 0) 0x7f072c1a9840 con 0x7f072c10ea90 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07140035b0 con 0x7f072c10ea90 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0724001c30 con 0x7f072c10ea90 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1625946813 0 0) 0x7f072c1079e0 con 0x7f072c10aef0 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f072c1a9840 con 0x7f072c10aef0 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.927+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f07200017f0 con 0x7f072c10aef0 2024-03-30T14:59:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4190255748 0 0) 0x7f072c1a8660 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 >> v1:172.21.15.134:6789/0 conn(0x7f072c10aef0 legacy=0x7f072c104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 >> v1:172.21.15.28:6789/0 conn(0x7f072c10ea90 legacy=0x7f072c1a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c1aaa20 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f072c1a9a70 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0731fc2700 10 client.57065.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f072c1a9f30 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0731fc2700 1 librados: init done 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f071c0011c0 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.928+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f071c004e00 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.929+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0718005120 con 0x7f072c102260 2024-03-30T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.929+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f071c001b70 con 0x7f072c102260 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(182..182 src has 53..182) ==== 10536+0+0 (unknown 1135700982 0 0) 0x7f071c089fe0 con 0x7f072c102260 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 10 client.57065.objecter ms_dispatch 0x7f072c001550 osd_map(182..182 src has 53..182) 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 3 client.57065.objecter handle_osd_map got epochs [182,182] > 0 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 3 client.57065.objecter handle_osd_map decoding full epoch 182 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 20 client.57065.objecter dump_active .. 0 homeless 2024-03-30T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.930+0000 7f0728ff9700 10 client.57065.objecter ms_handle_connect 0x7f07140704f0 2024-03-30T14:59:06.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:06.934+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f071c0568b0 con 0x7f072c102260 2024-03-30T14:59:06.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:06.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:06.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:07.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:07.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.153+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0718005410 con 0x7f072c102260 2024-03-30T14:59:07.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:07.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.201+0000 7f0728ff9700 1 -- 172.21.15.28:0/1854429982 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v183) ==== 101+0+0 (unknown 2261882912 0 0) 0x7f071c05a300 con 0x7f072c102260 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.206+0000 7f0731fc2700 10 librados: watch_flush enter 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 10 librados: watch_flush exit 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 20 client.57065.objecter shutdown clearing up homeless session... 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 10 client.57065.objecter successfully canceled tick 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 >> v1:172.21.15.28:6800/3011431093 conn(0x7f07140704f0 legacy=0x7f07140729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 >> v1:172.21.15.28:6790/0 conn(0x7f072c102260 legacy=0x7f072c104830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 shutdown_connections 2024-03-30T14:59:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.207+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 >> 172.21.15.28:0/1854429982 conn(0x7f072c0fdb10 msgr2=0x7f072c1116e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.209+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 shutdown_connections 2024-03-30T14:59:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.210+0000 7f0731fc2700 1 -- 172.21.15.28:0/1854429982 wait complete. 2024-03-30T14:59:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:07.211+0000 7f0731fc2700 1 librados: shutdown 2024-03-30T14:59:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T14:59:07.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:07.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:07.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:07.374 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T14:59:07.374 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T14:59:07.374 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T14:59:07.375 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T14:59:07.375 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: pgmap v212: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:07.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1148433196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1854429982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:07 smithi134 ceph-mon[33714]: from='client.57065 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: pgmap v212: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1148433196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1854429982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[41614]: from='client.57065 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: pgmap v212: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:07.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.56945 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi028-93441-15"}]': finished 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/242447951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93525-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93257-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: osdmap e182: 8 total, 8 up, 8 in 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1148433196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.57050 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1854429982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:07 smithi028 ceph-mon[29876]: from='client.57065 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:07.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:07.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:07.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:07.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:07.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:07.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:07.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:07.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:07.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:07.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:08.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T14:59:08.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:08.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:08.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T14:59:08.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:08.362 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:08.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: from='client.57065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T14:59:08.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: from='client.57065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:59:08.364 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:08.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T14:59:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T14:59:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: from='client.57050 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92979-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3074877088' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: from='client.57065 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T14:59:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T14:59:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: osdmap e183: 8 total, 8 up, 8 in 2024-03-30T14:59:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:08.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:08.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:08.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:08.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:08.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:08.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:08.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:08.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:08.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:08.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:09.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 -- 172.21.15.28:0/2446148106 >> v1:172.21.15.28:6790/0 conn(0x7f8178101140 legacy=0x7f8178101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 -- 172.21.15.28:0/2446148106 shutdown_connections 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 -- 172.21.15.28:0/2446148106 >> 172.21.15.28:0/2446148106 conn(0x7f81780fc9d0 msgr2=0x7f81780fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 -- 172.21.15.28:0/2446148106 shutdown_connections 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 -- 172.21.15.28:0/2446148106 wait complete. 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 librados: starting msgr at 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.045+0000 7f817d275700 1 librados: starting objecter 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 Processor -- start 2024-03-30T14:59:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 -- start start 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 librados: setting wanted keys 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 librados: calling monclient init 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81781a9330 con 0x7f817810d970 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81781aa550 con 0x7f8178101140 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f817d275700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81781ab790 con 0x7f8178109dd0 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f81767fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8178109dd0 0x7f81781a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38334/0 (socket says 172.21.15.28:38334) 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f81767fc700 1 -- 172.21.15.28:0/3754830079 learned_addr learned my addr 172.21.15.28:0/3754830079 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter ms_handle_connect 0x7f817810d970 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter ms_handle_connect 0x7f8178109dd0 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter ms_handle_connect 0x7f8178101140 2024-03-30T14:59:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454610801 0 0) 0x7f81781a9330 con 0x7f817810d970 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81640031f0 con 0x7f817810d970 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198774039 0 0) 0x7f81781ab790 con 0x7f8178109dd0 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81781a9330 con 0x7f8178109dd0 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359663392 0 0) 0x7f81781aa550 con 0x7f8178101140 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.046+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81781ab790 con 0x7f8178101140 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4250546497 0 0) 0x7f81781a9330 con 0x7f8178109dd0 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81781aa550 con 0x7f8178109dd0 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8160001870 con 0x7f8178109dd0 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2555742483 0 0) 0x7f81781ab790 con 0x7f8178101140 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81781a9330 con 0x7f8178101140 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 754074074 0 0) 0x7f81640031f0 con 0x7f817810d970 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81781ab790 con 0x7f817810d970 2024-03-30T14:59:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8170004470 con 0x7f8178101140 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8168001bb0 con 0x7f817810d970 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3088777007 0 0) 0x7f81781aa550 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.134:6789/0 conn(0x7f8178101140 legacy=0x7f8178106440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6789/0 conn(0x7f817810d970 legacy=0x7f81781a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81781ac9d0 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f81781ab9c0 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f817d275700 10 client.57092.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f81781abe80 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f817d275700 1 librados: init done 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8160003b20 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.047+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8160005d10 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.048+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8158005100 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.048+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8160003680 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(184..184 src has 53..184) ==== 10341+0+0 (unknown 3498688420 0 0) 0x7f8160089f90 con 0x7f8178109dd0 2024-03-30T14:59:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 osd_map(184..184 src has 53..184) 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 3 client.57092.objecter handle_osd_map got epochs [184,184] > 0 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 3 client.57092.objecter handle_osd_map decoding full epoch 184 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 20 client.57092.objecter dump_active .. 0 homeless 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v184) ==== 54+0+16 (unknown 3164069218 0 2420000192) 0x7f816008a360 con 0x7f8178109dd0 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f81640703b0 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 10 client.57092.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 1 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 1 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 0 1 2024-03-30T14:59:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 4 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 10 client.57092.objecter _send_command 1 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8158004b50 con 0x7f8158004730 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.049+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f8158004730 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.050+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8158004b50 con 0x7f8158004730 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.050+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.050+0000 7f816ffff700 10 client.57092.objecter _finish_command 1 = system:0 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.050+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 4 2024-03-30T14:59:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.050+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 0 1 2024-03-30T14:59:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 10 client.57092.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 2 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 2 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f81580053e0 osd=0 3 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 0 2 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 4 2024-03-30T14:59:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 10 client.57092.objecter _send_command 2 2024-03-30T14:59:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.068+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81580079f0 con 0x7f8158004730 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81580079f0 con 0x7f8158004730 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 10 client.57092.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 4 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 0 2 2024-03-30T14:59:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 10 client.57092.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 3 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 3 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 1 3 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 4 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 10 client.57092.objecter _send_command 3 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f81580021c0 con 0x7f81580042c0 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.070+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f81580042c0 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.071+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81580021c0 con 0x7f81580042c0 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.071+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.071+0000 7f816ffff700 10 client.57092.objecter _finish_command 3 = system:0 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.071+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 4 2024-03-30T14:59:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.071+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 1 3 2024-03-30T14:59:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 10 client.57092.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 4 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 4 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158004150 osd=1 3 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 1 4 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 4 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 10 client.57092.objecter _send_command 4 2024-03-30T14:59:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.088+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8158001600 con 0x7f81580042c0 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8158001600 con 0x7f81580042c0 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f816ffff700 10 client.57092.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 4 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 1 4 2024-03-30T14:59:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 10 client.57092.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 5 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 5 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 2 5 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 4 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 10 client.57092.objecter _send_command 5 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.090+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f81580110e0 con 0x7f815800d5d0 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f815800d5d0 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81580110e0 con 0x7f815800d5d0 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 10 client.57092.objecter _finish_command 5 = system:0 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 4 2024-03-30T14:59:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.091+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 2 5 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 10 client.57092.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 6 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 6 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158008370 osd=2 3 2024-03-30T14:59:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 2 6 2024-03-30T14:59:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 4 2024-03-30T14:59:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 10 client.57092.objecter _send_command 6 2024-03-30T14:59:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.108+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8158007e50 con 0x7f815800d5d0 2024-03-30T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.109+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8158007e50 con 0x7f815800d5d0 2024-03-30T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.109+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.109+0000 7f816ffff700 10 client.57092.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.109+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 4 2024-03-30T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.109+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 2 6 2024-03-30T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 10 client.57092.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 7 2024-03-30T14:59:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 7 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 3 7 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 4 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 10 client.57092.objecter _send_command 7 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8158015f20 con 0x7f8158012410 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.110+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f8158012410 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.111+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8158015f20 con 0x7f8158012410 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.111+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.111+0000 7f816ffff700 10 client.57092.objecter _finish_command 7 = system:0 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.111+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 4 2024-03-30T14:59:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.111+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 3 7 2024-03-30T14:59:09.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 10 client.57092.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 8 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 8 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800cc20 osd=3 3 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 3 8 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 4 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 10 client.57092.objecter _send_command 8 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.127+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f815800c700 con 0x7f8158012410 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f815800c700 con 0x7f8158012410 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f816ffff700 10 client.57092.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 4 2024-03-30T14:59:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 3 8 2024-03-30T14:59:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 10 client.57092.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 9 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 9 2024-03-30T14:59:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 4 9 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 4 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 10 client.57092.objecter _send_command 9 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.129+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8158012160 con 0x7f815800c810 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.130+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f815800c810 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.131+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8158012160 con 0x7f815800c810 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.131+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.131+0000 7f816ffff700 10 client.57092.objecter _finish_command 9 = system:0 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.131+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 4 2024-03-30T14:59:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.131+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 4 9 2024-03-30T14:59:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:09.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 10 client.57092.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 10 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 10 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800c6a0 osd=4 3 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 4 10 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 4 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 10 client.57092.objecter _send_command 10 2024-03-30T14:59:09.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.147+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f815801a7d0 con 0x7f815800c810 2024-03-30T14:59:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f815801a7d0 con 0x7f815800c810 2024-03-30T14:59:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 10 client.57092.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 4 2024-03-30T14:59:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 4 10 2024-03-30T14:59:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 10 client.57092.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 11 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 11 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 5 11 2024-03-30T14:59:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 4 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 10 client.57092.objecter _send_command 11 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f815801ea00 con 0x7f815801ae50 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.149+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f815801ae50 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.150+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f815801ea00 con 0x7f815801ae50 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.150+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.150+0000 7f816ffff700 10 client.57092.objecter _finish_command 11 = system:0 2024-03-30T14:59:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.150+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 4 2024-03-30T14:59:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.150+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 5 11 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 10 client.57092.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 12 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 12 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815801a7d0 osd=5 3 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 5 12 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 4 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 10 client.57092.objecter _send_command 12 2024-03-30T14:59:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.167+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81580114e0 con 0x7f815801ae50 2024-03-30T14:59:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81580114e0 con 0x7f815801ae50 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f816ffff700 10 client.57092.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 4 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 5 12 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 10 client.57092.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 13 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 13 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 6 13 2024-03-30T14:59:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 4 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 10 client.57092.objecter _send_command 13 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.168+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f81580201c0 con 0x7f8158011290 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.169+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f8158011290 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.170+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8168006540 con 0x7f8158011290 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.170+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.170+0000 7f816ffff700 10 client.57092.objecter _finish_command 13 = system:0 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.170+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 4 2024-03-30T14:59:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.170+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 6 13 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 10 client.57092.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 14 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 14 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f815800c510 osd=6 3 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 6 14 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 4 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 10 client.57092.objecter _send_command 14 2024-03-30T14:59:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.186+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8158016cd0 con 0x7f8158011290 2024-03-30T14:59:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8158016cd0 con 0x7f8158011290 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f816ffff700 10 client.57092.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 4 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 6 14 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 10 client.57092.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 15 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 15 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 7 15 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 4 2024-03-30T14:59:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 10 client.57092.objecter _send_command 15 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.188+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f81580275d0 con 0x7f8158023ac0 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.189+0000 7f816ffff700 10 client.57092.objecter ms_handle_connect 0x7f8158023ac0 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.190+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81580275d0 con 0x7f8158023ac0 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.190+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.190+0000 7f816ffff700 10 client.57092.objecter _finish_command 15 = system:0 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.190+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 4 2024-03-30T14:59:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.190+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 7 15 2024-03-30T14:59:09.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 17.5 scrub starts 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 10 client.57092.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign -1 16 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 20 client.57092.objecter _get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 15 client.57092.objecter _session_command_op_remove -1 16 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 20 client.57092.objecter get_session s=0x7f8158016cd0 osd=7 3 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 15 client.57092.objecter _session_command_op_assign 7 16 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 4 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 10 client.57092.objecter _send_command 16 2024-03-30T14:59:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.214+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8158016170 con 0x7f8158023ac0 2024-03-30T14:59:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.225+0000 7f816ffff700 1 -- 172.21.15.28:0/3754830079 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8158016170 con 0x7f8158023ac0 2024-03-30T14:59:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.225+0000 7f816ffff700 10 client.57092.objecter ms_dispatch 0x7f8178001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.225+0000 7f816ffff700 10 client.57092.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.225+0000 7f816ffff700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 4 2024-03-30T14:59:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.225+0000 7f816ffff700 15 client.57092.objecter _session_command_op_remove 7 16 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 librados: watch_flush enter 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 librados: watch_flush exit 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.0 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6801/3394323575 conn(0x7f8158004730 legacy=0x7f8158002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f81580053e0 osd=0 1 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.1 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6805/1850313547 conn(0x7f81580042c0 legacy=0x7f815800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158004150 osd=1 1 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.2 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6809/3451325043 conn(0x7f815800d5d0 legacy=0x7f815800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f81777fe700 1 -- 172.21.15.28:0/3754830079 reap_dead start 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158008370 osd=2 1 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.3 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6813/1506157250 conn(0x7f8158012410 legacy=0x7f8158014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800cc20 osd=3 1 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.4 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.134:6800/3063775514 conn(0x7f815800c810 legacy=0x7f81580190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c6a0 osd=4 1 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.5 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.134:6804/3668151179 conn(0x7f815801ae50 legacy=0x7f815801d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815801a7d0 osd=5 1 2024-03-30T14:59:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.6 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.134:6808/2135224194 conn(0x7f8158011290 legacy=0x7f815801ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f815800c510 osd=6 1 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter close_session for osd.7 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.134:6812/3758190298 conn(0x7f8158023ac0 legacy=0x7f8158025eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f81777fe700 1 -- 172.21.15.28:0/3754830079 reap_dead start 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter put_session s=0x7f8158016cd0 osd=7 1 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 20 client.57092.objecter shutdown clearing up homeless session... 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 10 client.57092.objecter successfully canceled tick 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6800/3011431093 conn(0x7f81640703b0 legacy=0x7f8164072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.232+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> v1:172.21.15.28:6790/0 conn(0x7f8178109dd0 legacy=0x7f81781a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.233+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 shutdown_connections 2024-03-30T14:59:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.233+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 >> 172.21.15.28:0/3754830079 conn(0x7f81780fc9d0 msgr2=0x7f8178106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:09.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.236+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 shutdown_connections 2024-03-30T14:59:09.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.236+0000 7f817d275700 1 -- 172.21.15.28:0/3754830079 wait complete. 2024-03-30T14:59:09.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:09.236+0000 7f817d275700 1 librados: shutdown 2024-03-30T14:59:09.238 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:09.239 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:09.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:09.306 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 17.5 scrub starts 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 17.5 scrub ok 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: pgmap v215: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T14:59:09.307 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:09.308 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3754830079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:09.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:09.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 17.5 scrub starts 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 17.5 scrub ok 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: pgmap v215: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T14:59:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3754830079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 17.5 scrub ok 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: pgmap v215: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T14:59:09.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93257-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: osdmap e184: 8 total, 8 up, 8 in 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:09.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3754830079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:09.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:09.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:09.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:09.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:09.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:09.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:09.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:09.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:10.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:10.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:10.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:10.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:10.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:10.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:10.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:10.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:10.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:10.415 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T14:59:10.415 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1268068278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2255706229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.49129 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.57086 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: pgmap v218: 428 pgs: 8 creating+peering, 96 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:10.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1268068278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2255706229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.49129 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.57086 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: pgmap v218: 428 pgs: 8 creating+peering, 96 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:10.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:10 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:10.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:10.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:10.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:10.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: osdmap e185: 8 total, 8 up, 8 in 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1268068278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2255706229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.49129 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.57086 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T14:59:10.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: pgmap v218: 428 pgs: 8 creating+peering, 96 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:10 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:10.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:10.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:10.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:10.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:11.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 -- 172.21.15.28:0/230354045 >> v1:172.21.15.28:6789/0 conn(0x7fe100105ec0 legacy=0x7fe1001062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 -- 172.21.15.28:0/230354045 shutdown_connections 2024-03-30T14:59:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 -- 172.21.15.28:0/230354045 >> 172.21.15.28:0/230354045 conn(0x7fe100077370 msgr2=0x7fe1000fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 -- 172.21.15.28:0/230354045 shutdown_connections 2024-03-30T14:59:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 -- 172.21.15.28:0/230354045 wait complete. 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 librados: starting msgr at 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 librados: starting objecter 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.105+0000 7fe108579700 1 Processor -- start 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 -- start start 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 librados: setting wanted keys 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 librados: calling monclient init 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1001a7140 con 0x7fe100105ec0 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1001a8360 con 0x7fe100109d20 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe108579700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1001a95a0 con 0x7fe10010da20 2024-03-30T14:59:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe106b16700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe10010da20 0x7fe1001a5800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38400/0 (socket says 172.21.15.28:38400) 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe106b16700 1 -- 172.21.15.28:0/1316681448 learned_addr learned my addr 172.21.15.28:0/1316681448 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter ms_handle_connect 0x7fe10010da20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter ms_handle_connect 0x7fe100105ec0 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter ms_handle_connect 0x7fe100109d20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406206771 0 0) 0x7fe1001a95a0 con 0x7fe10010da20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ec0031f0 con 0x7fe10010da20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331595218 0 0) 0x7fe1001a7140 con 0x7fe100105ec0 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1001a95a0 con 0x7fe100105ec0 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764696061 0 0) 0x7fe1001a8360 con 0x7fe100109d20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.106+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1001a7140 con 0x7fe100109d20 2024-03-30T14:59:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2014367594 0 0) 0x7fe0ec0031f0 con 0x7fe10010da20 2024-03-30T14:59:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1001a8360 con 0x7fe10010da20 2024-03-30T14:59:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe0f8001c90 con 0x7fe10010da20 2024-03-30T14:59:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3544884796 0 0) 0x7fe1001a95a0 con 0x7fe100105ec0 2024-03-30T14:59:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0ec0031f0 con 0x7fe100105ec0 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1766996370 0 0) 0x7fe1001a7140 con 0x7fe100109d20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1001a95a0 con 0x7fe100109d20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe0fc001910 con 0x7fe100105ec0 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe0f00017f0 con 0x7fe100109d20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 337133705 0 0) 0x7fe1001a8360 con 0x7fe10010da20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 >> v1:172.21.15.134:6789/0 conn(0x7fe100109d20 legacy=0x7fe1001a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 >> v1:172.21.15.28:6789/0 conn(0x7fe100105ec0 legacy=0x7fe1000ff610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1001aa7e0 con 0x7fe10010da20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe108579700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1001a7370 con 0x7fe10010da20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe108579700 10 client.57110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe108579700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1001a7830 con 0x7fe10010da20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe108579700 1 librados: init done 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe0f8003ec0 con 0x7fe10010da20 2024-03-30T14:59:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.107+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe0f8006090 con 0x7fe10010da20 2024-03-30T14:59:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe108579700 10 client.57110.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe108579700 15 client.57110.objecter _session_command_op_assign -1 1 2024-03-30T14:59:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe108579700 20 client.57110.objecter _get_session osd=-1 returning homeless 2024-03-30T14:59:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe108579700 10 client.57110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe108579700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe0e4002920 con 0x7fe10010da20 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.108+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe0f801c730 con 0x7fe10010da20 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(186..186 src has 53..186) ==== 11424+0+0 (unknown 1414062844 0 0) 0x7fe0f808a650 con 0x7fe10010da20 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 10 client.57110.objecter ms_dispatch 0x7fe100001550 osd_map(186..186 src has 53..186) 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 3 client.57110.objecter handle_osd_map got epochs [186,186] > 0 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 3 client.57110.objecter handle_osd_map decoding full epoch 186 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 10 client.57110.objecter checking command 1 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter _get_session s=0x7fe0ec083ae0 osd=7 3 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter put_session s=0x7fe0ec083ae0 osd=7 3 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 15 client.57110.objecter _session_command_op_remove -1 1 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter _get_session s=0x7fe0ec083ae0 osd=7 3 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter get_session s=0x7fe0ec083ae0 osd=7 3 2024-03-30T14:59:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 15 client.57110.objecter _session_command_op_assign 7 1 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter put_session s=0x7fe0ec083ae0 osd=7 4 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 10 client.57110.objecter _send_command 1 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 --> v1:172.21.15.134:6812/3758190298 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0ec087ee0 con 0x7fe0ec084190 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 20 client.57110.objecter dump_active .. 0 homeless 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 10 client.57110.objecter ms_handle_connect 0x7fe0ec0703a0 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.109+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=186) ==== 24+0+0 (unknown 3092393978 0 0) 0x7fe0f808aa80 con 0x7fe10010da20 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 10 client.57110.objecter ms_handle_connect 0x7fe0ec084190 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 1 -- 172.21.15.28:0/1316681448 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe0ec087ee0 con 0x7fe0ec084190 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 10 client.57110.objecter ms_dispatch 0x7fe100001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 10 client.57110.objecter _finish_command 1 = system:0 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 20 client.57110.objecter put_session s=0x7fe0ec083ae0 osd=7 4 2024-03-30T14:59:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:11.110+0000 7fe0f77fe700 15 client.57110.objecter _session_command_op_remove 7 1 2024-03-30T14:59:11.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-03-30T14:59:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-03-30T14:59:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-03-30T14:59:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-03-30T14:59:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-03-30T14:59:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-03-30T14:59:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-03-30T14:59:11.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.171 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:11.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:11.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:11.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:11.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:11.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:11.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:11.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:11.802 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:11.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:11.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.49129 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.57086 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:11.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 4.2 deep-scrub starts 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 4.2 deep-scrub ok 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: 4.1e deep-scrub starts 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T14:59:11.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.49129 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.57086 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 4.2 deep-scrub starts 2024-03-30T14:59:11.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 4.2 deep-scrub ok 2024-03-30T14:59:11.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: 4.1e deep-scrub starts 2024-03-30T14:59:11.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T14:59:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.49129 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi028-92999-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.57086 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi028-92979-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: osdmap e186: 8 total, 8 up, 8 in 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 4.2 deep-scrub starts 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 4.2 deep-scrub ok 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: 4.1e deep-scrub starts 2024-03-30T14:59:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:11.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:12.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:12.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:12.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T14:59:12.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-30T14:59:12.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:12.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:12.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:12.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:12.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:12.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:12.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: 42.14 scrub starts 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: 42.14 scrub ok 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: 4.1e deep-scrub ok 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: 42.9 scrub starts 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: 42.9 scrub ok 2024-03-30T14:59:12.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: pgmap v221: 364 pgs: 8 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3094477690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.49162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/655449973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[29876]: from='client.49165 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: 42.14 scrub starts 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: 42.14 scrub ok 2024-03-30T14:59:12.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: 4.1e deep-scrub ok 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: 42.9 scrub starts 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: 42.9 scrub ok 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: pgmap v221: 364 pgs: 8 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3094477690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.49162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/655449973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.779 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:12 smithi028 ceph-mon[41614]: from='client.49165 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:12.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:12.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:59:12.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: 42.14 scrub starts 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: 42.14 scrub ok 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: 4.1e deep-scrub ok 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93257-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: osdmap e187: 8 total, 8 up, 8 in 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: 42.9 scrub starts 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: 42.9 scrub ok 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: pgmap v221: 364 pgs: 8 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi028-93525-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: osdmap e188: 8 total, 8 up, 8 in 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3094477690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.49162 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/655449973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:12 smithi134 ceph-mon[33714]: from='client.49165 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:12.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:12.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:13.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:13.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:13.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:13.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:13.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:13.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:13.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:13.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:13.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:59:13.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:13.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:13.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T14:59:13.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T14:59:13.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: from='client.49162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: from='client.49165 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:13.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: from='client.49162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: from='client.49165 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:13 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:13.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:13.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:13.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:13.912 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:59:13.912 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: from='client.49162 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi028-92999-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: from='client.49165 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi028-92979-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: osdmap e189: 8 total, 8 up, 8 in 2024-03-30T14:59:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:13 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:13.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:13.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:13.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:13.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:14.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:14.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:14.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:14.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:14.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:14.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T14:59:14.558 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:14.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:14.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:14.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: pgmap v224: 396 pgs: 18 unknown, 378 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:14.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: pgmap v224: 396 pgs: 18 unknown, 378 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:14.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:14 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:14.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:14.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:14.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:14.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:14.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:15.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:15.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: pgmap v224: 396 pgs: 18 unknown, 378 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: osdmap e190: 8 total, 8 up, 8 in 2024-03-30T14:59:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2599549866' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:14 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]: dispatch 2024-03-30T14:59:15.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:15.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:15.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:15.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:15.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:15.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:15.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:15.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:15.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:15.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:15.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:15.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:15.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:15.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 42.2 scrub starts 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 42.2 scrub ok 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 6.d deep-scrub starts 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 6.d deep-scrub ok 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T14:59:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 5.12 deep-scrub starts 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 5.12 deep-scrub ok 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1176583961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3126631070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.57134 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: from='client.57131 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T14:59:15.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 42.2 scrub starts 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 42.2 scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 6.d deep-scrub starts 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 6.d deep-scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 5.12 deep-scrub starts 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 5.12 deep-scrub ok 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:15.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1176583961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3126631070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.57134 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: from='client.57131 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T14:59:15.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:15 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T14:59:15.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:15.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:16.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:16.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 42.2 scrub starts 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 42.2 scrub ok 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 6.d deep-scrub starts 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 6.d deep-scrub ok 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 5.12 deep-scrub starts 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 5.12 deep-scrub ok 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93257-24"}]': finished 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: osdmap e191: 8 total, 8 up, 8 in 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1176583961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3126631070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.57134 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: from='client.57131 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T14:59:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:15 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T14:59:16.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:16.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:16.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:16.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:16.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:16.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:16.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:16.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:16.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:16.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:16.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:16.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:16.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: 17.17 scrub starts 2024-03-30T14:59:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: 17.17 scrub ok 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: pgmap v227: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: from='client.57134 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: from='client.57131 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: 17.17 scrub starts 2024-03-30T14:59:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: 17.17 scrub ok 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: pgmap v227: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: from='client.57134 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: from='client.57131 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T14:59:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:16 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T14:59:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:16.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:16.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:17.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:17.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: 17.17 scrub starts 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: 17.17 scrub ok 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: pgmap v227: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: from='client.57134 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi028-92979-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: from='client.57131 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi028-92999-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: osdmap e192: 8 total, 8 up, 8 in 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T14:59:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:16 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T14:59:17.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-30T14:59:17.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: seed 93525 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (8179 ms) 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3321 ms) 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3484 ms) 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2903 ms) 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4915 ms) 2024-03-30T14:59:17.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (22802 ms total) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5312 ms) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (7989 ms) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8237 ms) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3618 ms) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2059 ms) 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T14:59:17.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1967-g4e50bbcd/rpm/el9/BUILD/ceph-19.0.0-1967-g4e50bbcd/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi028-93525-7 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17967 ms) 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (45182 ms total) 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-30T14:59:17.209 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5059 ms) 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5068 ms total) 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3002 ms) 2024-03-30T14:59:17.210 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-30T14:59:17.211 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T14:59:17.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-03-30T14:59:17.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:17.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:17.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:17.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 -- 172.21.15.28:0/2552284131 >> v1:172.21.15.28:6790/0 conn(0x7f1880101130 legacy=0x7f1880101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 -- 172.21.15.28:0/2552284131 shutdown_connections 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 -- 172.21.15.28:0/2552284131 >> 172.21.15.28:0/2552284131 conn(0x7f18800fc9e0 msgr2=0x7f18800fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 -- 172.21.15.28:0/2552284131 shutdown_connections 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 -- 172.21.15.28:0/2552284131 wait complete. 2024-03-30T14:59:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 librados: starting msgr at 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.375+0000 7f1885d9a700 1 librados: starting objecter 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 Processor -- start 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 -- start start 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 librados: setting wanted keys 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 librados: calling monclient init 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18800fece0 con 0x7f188010d960 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18801aa930 con 0x7f1880101130 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f1885d9a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18801abb10 con 0x7f1880109dc0 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f187effd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1880109dc0 0x7f18800fd550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38562/0 (socket says 172.21.15.28:38562) 2024-03-30T14:59:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f187effd700 1 -- 172.21.15.28:0/2373071842 learned_addr learned my addr 172.21.15.28:0/2373071842 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f187cff9700 10 client.?.objecter ms_handle_connect 0x7f1880109dc0 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.376+0000 7f187cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 10 client.?.objecter ms_handle_connect 0x7f188010d960 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 10 client.?.objecter ms_handle_connect 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744374084 0 0) 0x7f18801abb10 con 0x7f1880109dc0 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18680035b0 con 0x7f1880109dc0 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036581841 0 0) 0x7f18801aa930 con 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18801abb10 con 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1485181367 0 0) 0x7f18801abb10 con 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18801aa930 con 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1870001430 con 0x7f1880101130 2024-03-30T14:59:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 61675814 0 0) 0x7f18680035b0 con 0x7f1880109dc0 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18801abb10 con 0x7f1880109dc0 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f18740018f0 con 0x7f1880109dc0 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2830181894 0 0) 0x7f18801aa930 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6790/0 conn(0x7f1880109dc0 legacy=0x7f18800fd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6789/0 conn(0x7f188010d960 legacy=0x7f18801a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18801accf0 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18801a9980 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f1885d9a700 10 client.49201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f18801a9e40 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f1885d9a700 1 librados: init done 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1870003700 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.377+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1870004de0 con 0x7f1880101130 2024-03-30T14:59:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.378+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f186c005100 con 0x7f1880101130 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.379+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f187001b6a0 con 0x7f1880101130 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(193..193 src has 53..193) ==== 10373+0+0 (unknown 317966980 0 0) 0x7f1870088f60 con 0x7f1880101130 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 osd_map(193..193 src has 53..193) 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 3 client.49201.objecter handle_osd_map got epochs [193,193] > 0 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 3 client.49201.objecter handle_osd_map decoding full epoch 193 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 20 client.49201.objecter dump_active .. 0 homeless 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v193) ==== 54+0+16 (unknown 2804109408 0 2420000192) 0x7f18700893a0 con 0x7f1880101130 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f18680702e0 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 10 client.49201.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 1 2024-03-30T14:59:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 1 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 0 1 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 4 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 10 client.49201.objecter _send_command 1 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f186c004b50 con 0x7f186c004730 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.380+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c004730 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.382+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c004b50 con 0x7f186c004730 2024-03-30T14:59:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.382+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.382+0000 7f187cff9700 10 client.49201.objecter _finish_command 1 = system:0 2024-03-30T14:59:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.382+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 4 2024-03-30T14:59:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.382+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 0 1 2024-03-30T14:59:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 10 client.49201.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 2 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 2 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c0053e0 osd=0 3 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 0 2 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 4 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 10 client.49201.objecter _send_command 2 2024-03-30T14:59:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.399+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c007b00 con 0x7f186c004730 2024-03-30T14:59:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c007b00 con 0x7f186c004730 2024-03-30T14:59:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f187cff9700 10 client.49201.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 4 2024-03-30T14:59:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 0 2 2024-03-30T14:59:17.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 10 client.49201.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 3 2024-03-30T14:59:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 1 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 4 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 10 client.49201.objecter _send_command 3 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.401+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f186c002200 con 0x7f186c0042f0 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c0042f0 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c002200 con 0x7f186c0042f0 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 10 client.49201.objecter _finish_command 3 = system:0 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 4 2024-03-30T14:59:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.402+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 1 3 2024-03-30T14:59:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 10 client.49201.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 4 2024-03-30T14:59:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 4 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c004180 osd=1 3 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 1 4 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 4 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 10 client.49201.objecter _send_command 4 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.420+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c001600 con 0x7f186c0042f0 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c001600 con 0x7f186c0042f0 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 10 client.49201.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 4 2024-03-30T14:59:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 1 4 2024-03-30T14:59:17.426 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 10 client.49201.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 5 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 5 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 2 5 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 4 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 10 client.49201.objecter _send_command 5 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f186c0111f0 con 0x7f186c00d6e0 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.422+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c00d6e0 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.423+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c0111f0 con 0x7f186c00d6e0 2024-03-30T14:59:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.423+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.423+0000 7f187cff9700 10 client.49201.objecter _finish_command 5 = system:0 2024-03-30T14:59:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.423+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 4 2024-03-30T14:59:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.423+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 2 5 2024-03-30T14:59:17.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 10 client.49201.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 6 2024-03-30T14:59:17.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 6 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c008480 osd=2 3 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 2 6 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 4 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 10 client.49201.objecter _send_command 6 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.441+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c007fc0 con 0x7f186c00d6e0 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c007fc0 con 0x7f186c00d6e0 2024-03-30T14:59:17.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f187cff9700 10 client.49201.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 4 2024-03-30T14:59:17.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 2 6 2024-03-30T14:59:17.444 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f1885d9a700 10 client.49201.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.442+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 7 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 7 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 3 7 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 4 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 10 client.49201.objecter _send_command 7 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f186c016030 con 0x7f186c012520 2024-03-30T14:59:17.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c012520 2024-03-30T14:59:17.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c016030 con 0x7f186c012520 2024-03-30T14:59:17.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:17.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 10 client.49201.objecter _finish_command 7 = system:0 2024-03-30T14:59:17.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 4 2024-03-30T14:59:17.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.443+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 3 7 2024-03-30T14:59:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 10 client.49201.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 8 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 8 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c008250 osd=3 3 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 3 8 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 4 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 10 client.49201.objecter _send_command 8 2024-03-30T14:59:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.460+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c00ca60 con 0x7f186c012520 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c00ca60 con 0x7f186c012520 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f187cff9700 10 client.49201.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 4 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 3 8 2024-03-30T14:59:17.463 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 10 client.49201.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 9 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 9 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 4 9 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 4 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 10 client.49201.objecter _send_command 9 2024-03-30T14:59:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.462+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f186c01ae70 con 0x7f186c017360 2024-03-30T14:59:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c017360 2024-03-30T14:59:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c01ae70 con 0x7f186c017360 2024-03-30T14:59:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 10 client.49201.objecter _finish_command 9 = system:0 2024-03-30T14:59:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 4 2024-03-30T14:59:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.463+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 4 9 2024-03-30T14:59:17.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 10 client.49201.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 10 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 10 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c011b70 osd=4 3 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 4 10 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 4 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 10 client.49201.objecter _send_command 10 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.482+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c0008d0 con 0x7f186c017360 2024-03-30T14:59:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c0008d0 con 0x7f186c017360 2024-03-30T14:59:17.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f187cff9700 10 client.49201.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 4 2024-03-30T14:59:17.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 4 10 2024-03-30T14:59:17.486 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 10 client.49201.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 11 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 11 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 5 11 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 4 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 10 client.49201.objecter _send_command 11 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.484+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f186c01ef60 con 0x7f186c01b380 2024-03-30T14:59:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c01b380 2024-03-30T14:59:17.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c01ef60 con 0x7f186c01b380 2024-03-30T14:59:17.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:17.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 10 client.49201.objecter _finish_command 11 = system:0 2024-03-30T14:59:17.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 4 2024-03-30T14:59:17.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.485+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 5 11 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 10 client.49201.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 12 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 12 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c0008d0 osd=5 3 2024-03-30T14:59:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 5 12 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 4 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 10 client.49201.objecter _send_command 12 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.506+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c01f170 con 0x7f186c01b380 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.508+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c01f170 con 0x7f186c01b380 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.508+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.508+0000 7f187cff9700 10 client.49201.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.508+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 4 2024-03-30T14:59:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.508+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 5 12 2024-03-30T14:59:17.510 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 10 client.49201.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 13 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 13 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 6 13 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 4 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 10 client.49201.objecter _send_command 13 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f186c020460 con 0x7f186c01f890 2024-03-30T14:59:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.509+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c01f890 2024-03-30T14:59:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:17.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.510+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c020460 con 0x7f186c01f890 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.510+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.510+0000 7f187cff9700 10 client.49201.objecter _finish_command 13 = system:0 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.510+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 4 2024-03-30T14:59:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.510+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 6 13 2024-03-30T14:59:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 10 client.49201.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 14 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 14 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c01f170 osd=6 3 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 6 14 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 4 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 10 client.49201.objecter _send_command 14 2024-03-30T14:59:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.527+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c020460 con 0x7f186c01f890 2024-03-30T14:59:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c020460 con 0x7f186c01f890 2024-03-30T14:59:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f187cff9700 10 client.49201.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 4 2024-03-30T14:59:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 6 14 2024-03-30T14:59:17.531 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 10 client.49201.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 15 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 15 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 7 15 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 4 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 10 client.49201.objecter _send_command 15 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.529+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f186c027980 con 0x7f186c023e70 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.530+0000 7f187cff9700 10 client.49201.objecter ms_handle_connect 0x7f186c023e70 2024-03-30T14:59:17.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.531+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f186c027980 con 0x7f186c023e70 2024-03-30T14:59:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.531+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.531+0000 7f187cff9700 10 client.49201.objecter _finish_command 15 = system:0 2024-03-30T14:59:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.531+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 4 2024-03-30T14:59:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.531+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 7 15 2024-03-30T14:59:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 10 client.49201.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign -1 16 2024-03-30T14:59:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 20 client.49201.objecter _get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_remove -1 16 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 20 client.49201.objecter get_session s=0x7f186c020460 osd=7 3 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 15 client.49201.objecter _session_command_op_assign 7 16 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 4 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 10 client.49201.objecter _send_command 16 2024-03-30T14:59:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.547+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f186c027980 con 0x7f186c023e70 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187cff9700 1 -- 172.21.15.28:0/2373071842 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f186c027980 con 0x7f186c023e70 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187cff9700 10 client.49201.objecter ms_dispatch 0x7f1880001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187cff9700 10 client.49201.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187cff9700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 4 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187cff9700 15 client.49201.objecter _session_command_op_remove 7 16 2024-03-30T14:59:17.550 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:17.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 librados: watch_flush enter 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 librados: watch_flush exit 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.0 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6801/3394323575 conn(0x7f186c004730 legacy=0x7f186c002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0053e0 osd=0 1 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.1 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6805/1850313547 conn(0x7f186c0042f0 legacy=0x7f186c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c004180 osd=1 1 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.2 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6809/3451325043 conn(0x7f186c00d6e0 legacy=0x7f186c00fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008480 osd=2 1 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.3 2024-03-30T14:59:17.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6813/1506157250 conn(0x7f186c012520 legacy=0x7f186c014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c008250 osd=3 1 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.4 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.134:6800/3063775514 conn(0x7f186c017360 legacy=0x7f186c019750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c011b70 osd=4 1 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.5 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.134:6804/3668151179 conn(0x7f186c01b380 legacy=0x7f186c01d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c0008d0 osd=5 1 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.6 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.134:6808/2135224194 conn(0x7f186c01f890 legacy=0x7f186c01fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c01f170 osd=6 1 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter close_session for osd.7 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.134:6812/3758190298 conn(0x7f186c023e70 legacy=0x7f186c026260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter put_session s=0x7f186c020460 osd=7 1 2024-03-30T14:59:17.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 20 client.49201.objecter shutdown clearing up homeless session... 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 10 client.49201.objecter successfully canceled tick 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187ffff700 1 -- 172.21.15.28:0/2373071842 reap_dead start 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187ffff700 1 -- 172.21.15.28:0/2373071842 reap_dead start 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187ffff700 1 -- 172.21.15.28:0/2373071842 reap_dead start 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187ffff700 1 -- 172.21.15.28:0/2373071842 reap_dead start 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f187ffff700 1 -- 172.21.15.28:0/2373071842 reap_dead start 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.28:6800/3011431093 conn(0x7f18680702e0 legacy=0x7f18680727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> v1:172.21.15.134:6789/0 conn(0x7f1880101130 legacy=0x7f18800fce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 shutdown_connections 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.549+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 >> 172.21.15.28:0/2373071842 conn(0x7f18800fc9e0 msgr2=0x7f188006a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.550+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 shutdown_connections 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.550+0000 7f1885d9a700 1 -- 172.21.15.28:0/2373071842 wait complete. 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:17.550+0000 7f1885d9a700 1 librados: shutdown 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:17.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:17.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:17.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:17.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:17.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:17.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:17.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:18.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:18.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:18.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:17 smithi134 ceph-mon[33714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:17 smithi134 ceph-mon[33714]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:59:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2373071842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:18.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[41614]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[41614]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2373071842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:18.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[29876]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:18.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/887399718' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi028-93257-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:18.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[29876]: osdmap e193: 8 total, 8 up, 8 in 2024-03-30T14:59:18.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2373071842' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:18.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:18.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:18.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:18.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:18.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:18.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:18.573 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:18.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:18.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:18.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:18.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 17.e scrub starts 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 17.e scrub ok 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: pgmap v230: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 42.1b deep-scrub starts 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: 42.1b deep-scrub ok 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:18.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1345990030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: from='client.57152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3281961480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[41614]: from='client.49189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 17.e scrub starts 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 17.e scrub ok 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: pgmap v230: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 42.1b deep-scrub starts 2024-03-30T14:59:18.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: 42.1b deep-scrub ok 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1345990030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: from='client.57152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3281961480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:18 smithi028 ceph-mon[29876]: from='client.49189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:18.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:18.907 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:18.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:19.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:19.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:19.083 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:59:19.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:59:19.083 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T14:59:19.083 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T14:59:19.083 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:59:19.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:19.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 17.e scrub starts 2024-03-30T14:59:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 17.e scrub ok 2024-03-30T14:59:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: pgmap v230: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 42.1b deep-scrub starts 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: 42.1b deep-scrub ok 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: osdmap e194: 8 total, 8 up, 8 in 2024-03-30T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1345990030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: from='client.57152 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3281961480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:19.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:18 smithi134 ceph-mon[33714]: from='client.49189 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:19.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:19.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:19.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:19.417 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:19.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:19.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:19.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:19.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:19.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:19.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:19.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T14:59:19.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:19.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:20.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:20.132 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T14:59:20.132 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T14:59:20.132 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:20.132 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T14:59:20.132 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T14:59:20.133 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: from='client.57152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.133 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: from='client.49189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.133 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:59:20.133 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3876186611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.133 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:19 smithi134 ceph-mon[33714]: from='client.57164 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:20.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: from='client.57152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: from='client.49189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3876186611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[29876]: from='client.57164 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: from='client.57152 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi028-92999-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: from='client.49189 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi028-92979-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: osdmap e195: 8 total, 8 up, 8 in 2024-03-30T14:59:20.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3876186611' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:19 smithi028 ceph-mon[41614]: from='client.57164 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:20.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:20.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:20.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:20.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:20.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:20.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:20.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:20.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[41614]: pgmap v233: 428 pgs: 32 unknown, 64 creating+peering, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:20.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:20.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:20.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[41614]: from='client.57164 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[41614]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:59:20.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[29876]: pgmap v233: 428 pgs: 32 unknown, 64 creating+peering, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:20.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:20.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:20.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[29876]: from='client.57164 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:20.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:20 smithi028 ceph-mon[29876]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:59:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:20.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:20.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:21.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:21.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:21.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-mon[33714]: pgmap v233: 428 pgs: 32 unknown, 64 creating+peering, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-mon[33714]: from='client.57164 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi028-93257-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:20 smithi134 ceph-mon[33714]: osdmap e196: 8 total, 8 up, 8 in 2024-03-30T14:59:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:21.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:21.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:21.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:21.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.472+0000 7f74e1277700 1 -- 172.21.15.28:0/178686145 >> v1:172.21.15.28:6790/0 conn(0x7f74dc10eab0 legacy=0x7f74dc110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.472+0000 7f74e1277700 1 -- 172.21.15.28:0/178686145 shutdown_connections 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.472+0000 7f74e1277700 1 -- 172.21.15.28:0/178686145 >> 172.21.15.28:0/178686145 conn(0x7f74dc0fc9e0 msgr2=0x7f74dc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.472+0000 7f74e1277700 1 -- 172.21.15.28:0/178686145 shutdown_connections 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.472+0000 7f74e1277700 1 -- 172.21.15.28:0/178686145 wait complete. 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 librados: starting msgr at 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 librados: starting objecter 2024-03-30T14:59:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 Processor -- start 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 -- start start 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 librados: setting wanted keys 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 librados: calling monclient init 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f74dc10aca0 con 0x7f74dc10eab0 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f74dc1a8630 con 0x7f74dc103190 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74e1277700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f74dc1a9810 con 0x7f74dc10afa0 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74da7fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f74dc10afa0 0x7f74dc108250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43080/0 (socket says 172.21.15.28:43080) 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.473+0000 7f74da7fc700 1 -- 172.21.15.28:0/1090509211 learned_addr learned my addr 172.21.15.28:0/1090509211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter ms_handle_connect 0x7f74dc10afa0 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter ms_handle_connect 0x7f74dc10eab0 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter ms_handle_connect 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614238376 0 0) 0x7f74dc1a9810 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74c80035b0 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354772133 0 0) 0x7f74dc1a8630 con 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74dc1a9810 con 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3334738585 0 0) 0x7f74c80035b0 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74dc1a8630 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f74cc001970 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 421268692 0 0) 0x7f74dc1a9810 con 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74c80035b0 con 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f74c40017e0 con 0x7f74dc103190 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2302584285 0 0) 0x7f74dc1a8630 con 0x7f74dc10afa0 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 >> v1:172.21.15.134:6789/0 conn(0x7f74dc103190 legacy=0x7f74dc107b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 >> v1:172.21.15.28:6789/0 conn(0x7f74dc10eab0 legacy=0x7f74dc1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc1aa9f0 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc1a7680 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74e1277700 10 client.57188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f74dc1a7b40 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74e1277700 1 librados: init done 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f74cc003df0 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.474+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f74cc005e70 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.475+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74bc005120 con 0x7f74dc10afa0 2024-03-30T14:59:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f74cc003950 con 0x7f74dc10afa0 2024-03-30T14:59:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(197..197 src has 53..197) ==== 10722+0+0 (unknown 1386565190 0 0) 0x7f74cc08a3c0 con 0x7f74dc10afa0 2024-03-30T14:59:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 10 client.57188.objecter ms_dispatch 0x7f74dc001550 osd_map(197..197 src has 53..197) 2024-03-30T14:59:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 3 client.57188.objecter handle_osd_map got epochs [197,197] > 0 2024-03-30T14:59:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 3 client.57188.objecter handle_osd_map decoding full epoch 197 2024-03-30T14:59:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 20 client.57188.objecter dump_active .. 0 homeless 2024-03-30T14:59:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.476+0000 7f74d3fff700 10 client.57188.objecter ms_handle_connect 0x7f74c8070330 2024-03-30T14:59:21.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.484+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f74cc056bd0 con 0x7f74dc10afa0 2024-03-30T14:59:21.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:21.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:21.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:21.717+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f74bc005410 con 0x7f74dc10afa0 2024-03-30T14:59:21.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:21.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:21.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:21.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:21.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 4.13 deep-scrub starts 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 4.13 deep-scrub ok 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:21.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4158703813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3329333735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.57176 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1090509211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:21.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[29876]: from='client.57188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 4.13 deep-scrub starts 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 4.13 deep-scrub ok 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4158703813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.983 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3329333735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.57176 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1090509211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:21.984 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:21 smithi028 ceph-mon[41614]: from='client.57188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:22.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:22.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 4.13 deep-scrub starts 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 4.13 deep-scrub ok 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T14:59:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: osdmap e197: 8 total, 8 up, 8 in 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4158703813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3329333735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.57173 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.57176 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1090509211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:21 smithi134 ceph-mon[33714]: from='client.57188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T14:59:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.183+0000 7f74d3fff700 1 -- 172.21.15.28:0/1090509211 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v198) ==== 87+0+0 (unknown 2496006685 0 0) 0x7f74cc05a620 con 0x7f74dc10afa0 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] Global test environment set-up. 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (16176 ms) 2024-03-30T14:59:22.190 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (7186 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (5939 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (7942 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (11135 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (48378 ms total) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4988 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6007 ms) 2024-03-30T14:59:22.191 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7988 ms) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6021 ms) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3986 ms) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi028-94321-11. 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: Waiting for snaps to purge. 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20001 ms) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48991 ms total) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [----------] Global test environment tear-down 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (97369 ms total) 2024-03-30T14:59:22.192 INFO:tasks.workunit.client.0.smithi028.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 10 librados: watch_flush enter 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 10 librados: watch_flush exit 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 20 client.57188.objecter shutdown clearing up homeless session... 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 10 client.57188.objecter successfully canceled tick 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 >> v1:172.21.15.28:6800/3011431093 conn(0x7f74c8070330 legacy=0x7f74c80727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 >> v1:172.21.15.28:6790/0 conn(0x7f74dc10afa0 legacy=0x7f74dc108250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 shutdown_connections 2024-03-30T14:59:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.187+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 >> 172.21.15.28:0/1090509211 conn(0x7f74dc0fc9e0 msgr2=0x7f74dc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.188+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 shutdown_connections 2024-03-30T14:59:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.190+0000 7f74e1277700 1 -- 172.21.15.28:0/1090509211 wait complete. 2024-03-30T14:59:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:22.191+0000 7f74e1277700 1 librados: shutdown 2024-03-30T14:59:22.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T14:59:22.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:22.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:22.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:22.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:22.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:22.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:22.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:59:22.617 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:22.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:22.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:22.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.869 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:22.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:22.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T14:59:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T14:59:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: pgmap v236: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.57176 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.57188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/354569125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[29876]: from='client.57182 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: pgmap v236: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:22.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.57176 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.57188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/354569125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:22.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:22 smithi028 ceph-mon[41614]: from='client.57182 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:22.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:22.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:23.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:23.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: pgmap v236: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T14:59:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.57173 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi028-92979-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.57176 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi028-92999-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.57188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: osdmap e198: 8 total, 8 up, 8 in 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/354569125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:22 smithi134 ceph-mon[33714]: from='client.57182 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:23.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:23.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:23.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:23.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:23.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:23.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:23.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:23.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:23.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:23.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:23.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: from='client.57182 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: from='client.57182 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.805 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:23 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:23.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:23.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:23.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:24.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:24.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: from='client.57182 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi028-93257-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: osdmap e199: 8 total, 8 up, 8 in 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2817065657' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:23 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]: dispatch 2024-03-30T14:59:24.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:24.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.398 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:24.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:24.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:24.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:24.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:24.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:24.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:24.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: pgmap v239: 324 pgs: 8 unknown, 316 active+clean; 600 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4034238058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: pgmap v239: 324 pgs: 8 unknown, 316 active+clean; 600 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:24.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4034238058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:24.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:24 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:25.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:25.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:25.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:25.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: pgmap v239: 324 pgs: 8 unknown, 316 active+clean; 600 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2 op/s 2024-03-30T14:59:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:59:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:59:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi028-93525-16"}]': finished 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: osdmap e200: 8 total, 8 up, 8 in 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4034238058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:24 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:25.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:25.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.313+0000 7f656e813700 1 -- 172.21.15.28:0/4146564925 >> v1:172.21.15.28:6790/0 conn(0x7f6568101130 legacy=0x7f6568101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.313+0000 7f656e813700 1 -- 172.21.15.28:0/4146564925 shutdown_connections 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.313+0000 7f656e813700 1 -- 172.21.15.28:0/4146564925 >> 172.21.15.28:0/4146564925 conn(0x7f65680fc9e0 msgr2=0x7f65680fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.313+0000 7f656e813700 1 -- 172.21.15.28:0/4146564925 shutdown_connections 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.313+0000 7f656e813700 1 -- 172.21.15.28:0/4146564925 wait complete. 2024-03-30T14:59:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 librados: starting msgr at 2024-03-30T14:59:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 librados: starting objecter 2024-03-30T14:59:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 Processor -- start 2024-03-30T14:59:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 -- start start 2024-03-30T14:59:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 librados: setting wanted keys 2024-03-30T14:59:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.314+0000 7f656e813700 1 librados: calling monclient init 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f656e813700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6568106640 con 0x7f6568109dc0 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f656e813700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6568106850 con 0x7f6568101130 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65677fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6568109dc0 0x7f656810e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49336/0 (socket says 172.21.15.28:49336) 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f656cdb0700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f656810d960 0x7f65681a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43174/0 (socket says 172.21.15.28:43174) 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65677fe700 1 -- 172.21.15.28:0/1815891230 learned_addr learned my addr 172.21.15.28:0/1815891230 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f6567fff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6568101130 0x7f656810dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:56212/0 (socket says 172.21.15.28:56212) 2024-03-30T14:59:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65681ab940 con 0x7f656810d960 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter ms_handle_connect 0x7f6568101130 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter ms_handle_connect 0x7f6568109dc0 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter ms_handle_connect 0x7f656810d960 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468282562 0 0) 0x7f6568106640 con 0x7f6568109dc0 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65500031f0 con 0x7f6568109dc0 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3137209662 0 0) 0x7f6568106850 con 0x7f6568101130 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6568106640 con 0x7f6568101130 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706877652 0 0) 0x7f65681ab940 con 0x7f656810d960 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.315+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6568106850 con 0x7f656810d960 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1023722785 0 0) 0x7f6568106640 con 0x7f6568101130 2024-03-30T14:59:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65681ab940 con 0x7f6568101130 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 448132439 0 0) 0x7f65500031f0 con 0x7f6568109dc0 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6568106640 con 0x7f6568109dc0 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6558004490 con 0x7f6568101130 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f655c0012d0 con 0x7f6568109dc0 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 42043871 0 0) 0x7f6568106850 con 0x7f656810d960 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65500031f0 con 0x7f656810d960 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6560001bb0 con 0x7f656810d960 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2643151453 0 0) 0x7f65681ab940 con 0x7f6568101130 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 >> v1:172.21.15.28:6790/0 conn(0x7f656810d960 legacy=0x7f65681a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 >> v1:172.21.15.28:6789/0 conn(0x7f6568109dc0 legacy=0x7f656810e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65681a9920 con 0x7f6568101130 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65681a9c10 con 0x7f6568101130 2024-03-30T14:59:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f656e813700 10 client.49258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65681aa0d0 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f656e813700 1 librados: init done 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6558003680 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.316+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6558004df0 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.317+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6554005120 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.317+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6558004dd0 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.318+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(201..201 src has 53..201) ==== 10255+0+0 (unknown 3912855293 0 0) 0x7f6558089e60 con 0x7f6568101130 2024-03-30T14:59:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.318+0000 7f65657fa700 10 client.49258.objecter ms_dispatch 0x7f6568001550 osd_map(201..201 src has 53..201) 2024-03-30T14:59:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.318+0000 7f65657fa700 3 client.49258.objecter handle_osd_map got epochs [201,201] > 0 2024-03-30T14:59:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.318+0000 7f65657fa700 3 client.49258.objecter handle_osd_map decoding full epoch 201 2024-03-30T14:59:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.318+0000 7f65657fa700 20 client.49258.objecter dump_active .. 0 homeless 2024-03-30T14:59:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.319+0000 7f65657fa700 10 client.49258.objecter ms_handle_connect 0x7f6550070400 2024-03-30T14:59:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.323+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6558056820 con 0x7f6568101130 2024-03-30T14:59:25.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:25.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24910.0:2 25.17 25:ecb6a661:test-rados-api-smithi028-93441-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966911+0000", 2024-03-30T14:59:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.291307623999998, 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3400814589999999, 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24910", 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1422378701", 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T14:59:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966913+0000", 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8819999999999999e-06 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966920+0000", 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0570000000000004e-06 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966930+0000", 2024-03-30T14:59:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0156000000000001e-05 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966940+0000", 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0147e-05 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083450+0000", 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.116509813 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083460+0000", 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.292378+0000", 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2089187990000001 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294004+0000", 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00162598 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294807+0000", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080235899999999999 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295802+0000", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000994834 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.299106+0000", 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003304202 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306871+0000", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077654550000000001 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306919+0000", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7447000000000002e-05 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306992+0000", 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3418999999999996e-05 2024-03-30T14:59:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24961.0:2 27.15 27:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.968658+0000", 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.289560609999995, 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23402463300000001, 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:59:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24961", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1241392178", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T14:59:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968659+0000", 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.73e-06 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968665+0000", 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8200000000000002e-06 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968676+0000", 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0528e-05 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968685+0000", 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.329e-06 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969119+0000", 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043351199999999999 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969125+0000", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2280000000000003e-06 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049864+0000", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080739092999999998 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049873+0000", 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7539999999999995e-06 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202601+0000", 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.152728379 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202682+0000", 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1260000000000005e-05 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35042.0:4 21.1f 21:ff7b1f36:::obj1:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.174931+0000", 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.083287132999999, 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0929989090000001, 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35042", 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/917910517", 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174933+0000", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.421e-06 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174937+0000", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4139999999999996e-06 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174945+0000", 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2090000000000005e-06 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174955+0000", 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5300000000000002e-06 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175014+0000", 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9110000000000002e-05 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175023+0000", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2920000000000001e-06 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267861+0000", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0928374670000001 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267930+0000", 2024-03-30T14:59:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9466000000000002e-05 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34820.0:20 9.8 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.214349+0000", 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.043869505000004, 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.089392117, 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34820", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4253613591", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T14:59:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214351+0000", 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.103e-06 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214358+0000", 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5320000000000001e-06 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214370+0000", 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1377e-05 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214379+0000", 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9090000000000006e-06 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214437+0000", 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8692e-05 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214446+0000", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8680000000000005e-06 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253200+0000", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.038753864 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253307+0000", 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010676299999999999 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253432+0000", 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124957 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.266525+0000", 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013092763 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.268732+0000", 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022073129999999998 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303646+0000", 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034913911999999998 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303698+0000", 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1986e-05 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303741+0000", 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3078000000000003e-05 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25075.0:2 39.1b 39:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.243275+0000", 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.014943428999999, 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0594717039999999, 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25075", 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3988199589", 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243277+0000", 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.796e-06 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243283+0000", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2740000000000004e-06 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243299+0000", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6532e-05 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243310+0000", 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0463e-05 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243383+0000", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3207000000000006e-05 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243392+0000", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8559999999999996e-06 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267447+0000", 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.024054757 2024-03-30T14:59:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267568+0000", 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012110199999999999 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267732+0000", 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016446500000000001 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280028+0000", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012295451000000001 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.284857+0000", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048288619999999997 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302648+0000", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017791609999999999 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302697+0000", 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9116999999999997e-05 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302747+0000", 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9212000000000002e-05 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24853.0:7 14.1d 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.164526+0000", 2024-03-30T14:59:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 89.093692584999999, 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.092228541999999997, 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24853", 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164528+0000", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.328e-06 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164538+0000", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0189e-05 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164550+0000", 2024-03-30T14:59:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1353e-05 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164560+0000", 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0828e-05 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167851+0000", 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032908939999999999 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167956+0000", 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010505699999999999 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256754+0000", 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088797893000000003 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24910.0:5 25.b e119/112 25:d1d7d6c3:test-rados-api-smithi028-93441-3::foo:head v 119'1, mlcod=0'0)", 2024-03-30T14:59:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.770100+0000", 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 84.488118279999995, 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019002787, 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770100+0000", 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770100+0000", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770100+0000", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2099999999999998e-07 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770109+0000", 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6680000000000007e-06 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770116+0000", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0060000000000003e-06 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770126+0000", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0103e-05 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770175+0000", 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9359e-05 2024-03-30T14:59:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.770216+0000", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0055999999999997e-05 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.789057+0000", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01884187 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.789103+0000", 2024-03-30T14:59:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5404000000000001e-05 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35405.0:2 62.13 62:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.989764+0000", 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 84.268454238000004, 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.977776094, 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35405", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1251877021", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T14:59:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989767+0000", 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.615e-06 2024-03-30T14:59:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989773+0000", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2430000000000002e-06 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989784+0000", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1043000000000001e-05 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989793+0000", 2024-03-30T14:59:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5279999999999992e-06 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511344+0000", 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.52155030499999999 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511352+0000", 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4479999999999994e-06 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949501+0000", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.43814895 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949615+0000", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114282 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949747+0000", 2024-03-30T14:59:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000131564 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956142+0000", 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063947639999999998 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956280+0000", 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000138496 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967445+0000", 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011165029999999999 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967488+0000", 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2397000000000001e-05 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967540+0000", 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2429000000000002e-05 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:483 115.4s0 115:2235e3a1:::482:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.258977+0000", 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 61.999241083999998, 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15220373500000001, 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T14:59:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 483 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258980+0000", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5359999999999999e-06 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258989+0000", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281859+0000", 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022870352 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281876+0000", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6492000000000001e-05 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338663+0000", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056787694 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338791+0000", 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012725400000000001 2024-03-30T14:59:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.339139+0000", 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034834299999999998 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.368274+0000", 2024-03-30T14:59:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029135306 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411112+0000", 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.042837274000000002 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411181+0000", 2024-03-30T14:59:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9336000000000002e-05 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:2 123.15 123:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.364324+0000", 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 61.893894261, 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2153437069999999, 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T14:59:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364326+0000", 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.435e-06 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364336+0000", 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6139999999999998e-06 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364348+0000", 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2194e-05 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364360+0000", 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1511e-05 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364419+0000", 2024-03-30T14:59:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8956999999999998e-05 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364429+0000", 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0026e-05 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538529+0000", 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.174100744 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538649+0000", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011977900000000001 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538787+0000", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000137458 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.541812+0000", 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030249740000000002 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.545225+0000", 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034129009999999999 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571213+0000", 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025988165000000001 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571265+0000", 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2385000000000001e-05 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.579668+0000", 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084025639999999995 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:715 123.1f 123:fbee941d:::test_708:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e143)", 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.834770+0000", 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 57.423447826, 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146558462, 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 715 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T14:59:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834773+0000", 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.57e-06 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834784+0000", 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1277999999999999e-05 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834803+0000", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8281999999999999e-05 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834813+0000", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.078e-05 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834910+0000", 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6772000000000001e-05 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835058+0000", 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148066 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.981329+0000", 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146270714 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:893 123.1f 123:fb160ca0:::test_886:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e144)", 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.946326+0000", 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.311891948000003, 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018882216, 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 893 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946326+0000", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946326+0000", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946329+0000", 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9610000000000001e-06 2024-03-30T14:59:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946341+0000", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1426999999999999e-05 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946365+0000", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3810000000000001e-05 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946376+0000", 2024-03-30T14:59:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1039999999999999e-05 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946444+0000", 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8133999999999995e-05 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.946570+0000", 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125998 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965209+0000", 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018638846000000001 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.2 epoch 146/143 rep_tid 752 entries 146'28 (0'0) error 130:40466eb8:::test_509:head by client.46235.0:516 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 145'26)", 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.234330+0000", 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.023888409999998, 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024038321000000001, 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T14:59:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T14:59:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.72e-07 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234339+0000", 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9590000000000001e-06 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234353+0000", 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3560000000000001e-05 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234368+0000", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5308e-05 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.241947+0000", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007578904 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.258368+0000", 2024-03-30T14:59:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016421318000000001 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.49064.0:2 134.1f 134:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.249689+0000", 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.008529514999999, 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016646265, 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.49064", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1179536722", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249689+0000", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249689+0000", 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249691+0000", 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1959999999999998e-06 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249700+0000", 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8189999999999998e-06 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249712+0000", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2296000000000001e-05 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.249724+0000", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2028000000000001e-05 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.253019+0000", 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032945660000000001 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.253135+0000", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116389 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,7", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.253275+0000", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000139481 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.261931+0000", 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086563009999999999 2024-03-30T14:59:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.266201+0000", 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042706159999999996 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.266240+0000", 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8522000000000001e-05 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.266292+0000", 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1835000000000001e-05 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.266335+0000", 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3216000000000001e-05 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.46235.0:517 130.5 130:a3243b6a:::test_510:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.270135+0000", 2024-03-30T14:59:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 54.988083033000002, 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023663910999999999, 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.46235", 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 517 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T14:59:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270137+0000", 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.356e-06 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270143+0000", 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5889999999999999e-06 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270151+0000", 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5000000000000002e-06 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270160+0000", 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6399999999999992e-06 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270522+0000", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036188600000000001 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270653+0000", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000130791 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.293799+0000", 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023146149000000001 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42667.0:2 138.0 138:0384ab8e:::oid:head [write 0~7,setxattr key1 (5) in=4b,truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.329605+0000", 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.928613353000003, 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93406830600000001, 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42667", 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2528295921", 2024-03-30T14:59:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329607+0000", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.874e-06 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329614+0000", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1169999999999998e-06 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329624+0000", 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0457e-05 2024-03-30T14:59:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329636+0000", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1382999999999999e-05 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350953+0000", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021317545 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350963+0000", 2024-03-30T14:59:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193e-05 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.249891+0000", 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89892724199999996 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250074+0000", 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018359100000000001 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,7", 2024-03-30T14:59:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250291+0000", 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021668499999999999 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252632+0000", 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023406579999999998 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257757+0000", 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051254320000000001 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263567+0000", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058095780000000001 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263628+0000", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1237999999999999e-05 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263673+0000", 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5312999999999998e-05 2024-03-30T14:59:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.46235.0:778 130.7 130:e3d206dd:::test_771:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.375510+0000", 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.882708053000002, 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018247001999999998, 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.46235", 2024-03-30T14:59:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 778 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375510+0000", 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375510+0000", 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375512+0000", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.677e-06 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375519+0000", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6730000000000003e-06 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375529+0000", 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0667e-05 2024-03-30T14:59:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375540+0000", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0635999999999999e-05 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375693+0000", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015360600000000001 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.375820+0000", 2024-03-30T14:59:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126434 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.393757+0000", 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017937308999999999 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42271.0:2 136.6 136:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:32.176958+0000", 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.081259949, 2024-03-30T14:59:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.090549882999999998, 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42271", 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/553852123", 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176961+0000", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-06 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176971+0000", 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9720000000000004e-06 2024-03-30T14:59:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176993+0000", 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2331e-05 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.177006+0000", 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3244999999999999e-05 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198821+0000", 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021814229000000001 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198830+0000", 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8740000000000001e-06 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250126+0000", 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.051296025000000002 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250242+0000", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116638 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250389+0000", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014667799999999999 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252485+0000", 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020958959999999999 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257681+0000", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005195987 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267377+0000", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096964649999999996 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267466+0000", 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8781000000000003e-05 2024-03-30T14:59:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267508+0000", 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2174e-05 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.42298.0:3 137.0 e148/147 137:0384ab8e:::oid:head v 148'2, mlcod=0'0)", 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:32.225913+0000", 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 53.032305317999999, 2024-03-30T14:59:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017866961000000001, 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225913+0000", 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225913+0000", 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225914+0000", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2200000000000002e-07 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225922+0000", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4719999999999995e-06 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225932+0000", 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6469999999999993e-06 2024-03-30T14:59:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.225941+0000", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3680000000000008e-06 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.228424+0000", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024829930000000002 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.228449+0000", 2024-03-30T14:59:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4712999999999999e-05 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.243747+0000", 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015298396000000001 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.243780+0000", 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2450000000000003e-05 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.43429.0:2 140.7 140:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.310951+0000", 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 51.947267580999998, 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.059809913999999999, 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.43429", 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1777674724", 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310953+0000", 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2840000000000001e-06 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310961+0000", 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7340000000000008e-06 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313256+0000", 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022953600000000002 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313269+0000", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2472e-05 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313347+0000", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8175000000000003e-05 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313356+0000", 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9029999999999993e-06 2024-03-30T14:59:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316618+0000", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003262587 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316732+0000", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011378 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316884+0000", 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015224100000000001 2024-03-30T14:59:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.332636+0000", 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015752226000000001 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.342429+0000", 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097926090000000007 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370655+0000", 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.028226161999999999 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370712+0000", 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6952e-05 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370761+0000", 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8429e-05 2024-03-30T14:59:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:25.574 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:59:25.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:25.547+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f6554005410 con 0x7f6568101130 2024-03-30T14:59:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:25.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:25.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:25.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:25.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:25.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:25.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1815891230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[41614]: from='client.49258 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1815891230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:25.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:25 smithi028 ceph-mon[29876]: from='client.49258 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:25.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:25.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:26.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3283617111' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi028-92979-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi028-92999-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: osdmap e201: 8 total, 8 up, 8 in 2024-03-30T14:59:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1815891230' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:25 smithi134 ceph-mon[33714]: from='client.49258 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T14:59:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.193+0000 7f65657fa700 1 -- 172.21.15.28:0/1815891230 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v202) ==== 113+0+0 (unknown 1749794882 0 0) 0x7f655805a270 con 0x7f6568101130 2024-03-30T14:59:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.196+0000 7f656e813700 10 librados: watch_flush enter 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 10 librados: watch_flush exit 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 20 client.49258.objecter shutdown clearing up homeless session... 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 10 client.49258.objecter successfully canceled tick 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6550070400 legacy=0x7f65500728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 >> v1:172.21.15.134:6789/0 conn(0x7f6568101130 legacy=0x7f656810dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 shutdown_connections 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 >> 172.21.15.28:0/1815891230 conn(0x7f65680fc9e0 msgr2=0x7f656806a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 shutdown_connections 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 -- 172.21.15.28:0/1815891230 wait complete. 2024-03-30T14:59:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:26.197+0000 7f656e813700 1 librados: shutdown 2024-03-30T14:59:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T14:59:26.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:26.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T14:59:26.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:26.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:26.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:26.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T14:59:26.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: Running main() from gmock_main.cc 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] Global test environment set-up. 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10148 ms) 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (2830 ms) 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (5781 ms) 2024-03-30T14:59:26.720 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (5907 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (5093 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4874 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (5383 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (5775 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3004 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (5029 ms) 2024-03-30T14:59:26.721 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3133 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2859 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2999 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3023 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.Flush (2982 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3004 ms) 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-30T14:59:26.722 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3008 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2987 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3013 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3017 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3282 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2695 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3009 ms) 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-30T14:59:26.723 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3014 ms) 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2980 ms) 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3529 ms) 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] 26 tests from LibRadosAio (102359 ms total) 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: 2024-03-30T14:59:26.724 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-30T14:59:26.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:26.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:26.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:26.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:27.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:27.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:59:27.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:27.124 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.49258 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:59:27.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:27.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.49258 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:59:27.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:59:27.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:27 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:27.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:27.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:27.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:27.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:27.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: pgmap v242: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T14:59:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.49258 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T14:59:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: osdmap e202: 8 total, 8 up, 8 in 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:27 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:27.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:27.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:27.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:27.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:27.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:27.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:27.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:27.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:28.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:28.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:28.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:28.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-03-30T14:59:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:28.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:28.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:28.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:28.349 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:59:28.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1866544138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.57227 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:59:28.351 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.352 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1866544138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.352 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.352 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.57227 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.352 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:28.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi028-93257-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi028-92979-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: osdmap e203: 8 total, 8 up, 8 in 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1866544138' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:28.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.57227 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:28.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:28.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:28.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:28.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:28.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:28.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:28.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:28.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:28.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:28.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:29.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:29.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:29.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:29.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: pgmap v245: 340 pgs: 48 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: from='client.57227 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: pgmap v245: 340 pgs: 48 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: from='client.57227 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:59:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:29.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:29.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:29.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:29.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:29.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:59:29.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:29.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:29.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:29.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: pgmap v245: 340 pgs: 48 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: from='client.57227 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi028-92999-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: osdmap e204: 8 total, 8 up, 8 in 2024-03-30T14:59:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:29.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:29.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:29.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:29.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:30.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:30.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:30.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:30.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:30.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 -- 172.21.15.28:0/958533733 >> v1:172.21.15.28:6789/0 conn(0x7f2a7c10afb0 legacy=0x7f2a7c10d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 -- 172.21.15.28:0/958533733 shutdown_connections 2024-03-30T14:59:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 -- 172.21.15.28:0/958533733 >> 172.21.15.28:0/958533733 conn(0x7f2a7c0fc9f0 msgr2=0x7f2a7c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 -- 172.21.15.28:0/958533733 shutdown_connections 2024-03-30T14:59:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 -- 172.21.15.28:0/958533733 wait complete. 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 librados: starting msgr at 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 librados: starting objecter 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.296+0000 7f2a81e05700 1 Processor -- start 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 -- start start 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 librados: setting wanted keys 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 librados: calling monclient init 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a7c1a9510 con 0x7f2a7c10eac0 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a7c1aa6f0 con 0x7f2a7c1031a0 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a81e05700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2a7c1ab8d0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a7affd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2a7c10afb0 0x7f2a7c1a46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55760/0 (socket says 172.21.15.28:55760) 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a7affd700 1 -- 172.21.15.28:0/500251240 learned_addr learned my addr 172.21.15.28:0/500251240 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 10 client.?.objecter ms_handle_connect 0x7f2a7c10afb0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 10 client.?.objecter ms_handle_connect 0x7f2a7c1031a0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298773158 0 0) 0x7f2a7c1aa6f0 con 0x7f2a7c1031a0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a640031f0 con 0x7f2a7c1031a0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.297+0000 7f2a78ff9700 10 client.?.objecter ms_handle_connect 0x7f2a7c10eac0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454385020 0 0) 0x7f2a7c1ab8d0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a7c1aa6f0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721484139 0 0) 0x7f2a7c1a9510 con 0x7f2a7c10eac0 2024-03-30T14:59:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a7c1ab8d0 con 0x7f2a7c10eac0 2024-03-30T14:59:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4161689568 0 0) 0x7f2a640031f0 con 0x7f2a7c1031a0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a7c1a9510 con 0x7f2a7c1031a0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2a6c001830 con 0x7f2a7c1031a0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4251672991 0 0) 0x7f2a7c1aa6f0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a640031f0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2a700046a0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2432157302 0 0) 0x7f2a7c1ab8d0 con 0x7f2a7c10eac0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a7c1aa6f0 con 0x7f2a7c10eac0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2a740013f0 con 0x7f2a7c10eac0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 53701131 0 0) 0x7f2a640031f0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.134:6789/0 conn(0x7f2a7c1031a0 legacy=0x7f2a7c19f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6789/0 conn(0x7f2a7c10eac0 legacy=0x7f2a7c1a7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c1acab0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c1aa920 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a81e05700 10 client.57239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.298+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a7c1aade0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.299+0000 7f2a81e05700 1 librados: init done 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.299+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2a70001b70 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.299+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2a70004e40 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.300+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2a68005100 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.300+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2a70004e40 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(206..206 src has 53..206) ==== 11058+0+0 (unknown 1147942477 0 0) 0x7f2a7008a2d0 con 0x7f2a7c10afb0 2024-03-30T14:59:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 osd_map(206..206 src has 53..206) 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 3 client.57239.objecter handle_osd_map got epochs [206,206] > 0 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 3 client.57239.objecter handle_osd_map decoding full epoch 206 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 20 client.57239.objecter dump_active .. 0 homeless 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v206) ==== 54+0+16 (unknown 3301893239 0 2420000192) 0x7f2a7008c730 con 0x7f2a7c10afb0 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.301+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a640703a0 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 10 client.57239.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 1 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 1 2024-03-30T14:59:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 0 1 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 4 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 10 client.57239.objecter _send_command 1 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a68004b50 con 0x7f2a68004730 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.302+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a68004730 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.303+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a68004b50 con 0x7f2a68004730 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.303+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.303+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 1 = system:0 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.303+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 4 2024-03-30T14:59:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.303+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 0 1 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 10 client.57239.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 2 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 2 2024-03-30T14:59:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a680053e0 osd=0 3 2024-03-30T14:59:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 0 2 2024-03-30T14:59:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 4 2024-03-30T14:59:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 10 client.57239.objecter _send_command 2 2024-03-30T14:59:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.322+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a68007b00 con 0x7f2a68004730 2024-03-30T14:59:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:30.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.323+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a68007b00 con 0x7f2a68004730 2024-03-30T14:59:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.323+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.323+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.323+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 4 2024-03-30T14:59:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.323+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 0 2 2024-03-30T14:59:30.327 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 10 client.57239.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 1 3 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 4 2024-03-30T14:59:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 10 client.57239.objecter _send_command 3 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2a68002200 con 0x7f2a680042f0 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.324+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a680042f0 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.325+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a68002200 con 0x7f2a680042f0 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.325+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.325+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 3 = system:0 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.325+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 4 2024-03-30T14:59:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.325+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 1 3 2024-03-30T14:59:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 10 client.57239.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 4 2024-03-30T14:59:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 4 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a68004180 osd=1 3 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 1 4 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 4 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 10 client.57239.objecter _send_command 4 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.342+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a68001660 con 0x7f2a680042f0 2024-03-30T14:59:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.343+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a68001660 con 0x7f2a680042f0 2024-03-30T14:59:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.343+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.343+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.343+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 4 2024-03-30T14:59:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.343+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 1 4 2024-03-30T14:59:30.346 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 10 client.57239.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 5 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 5 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 2 5 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 4 2024-03-30T14:59:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 10 client.57239.objecter _send_command 5 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2a680111f0 con 0x7f2a6800d6e0 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.344+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a6800d6e0 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.345+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a680111f0 con 0x7f2a6800d6e0 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.345+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.345+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 5 = system:0 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.345+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 4 2024-03-30T14:59:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.345+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 2 5 2024-03-30T14:59:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 10 client.57239.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 6 2024-03-30T14:59:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 6 2024-03-30T14:59:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a68008480 osd=2 3 2024-03-30T14:59:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 2 6 2024-03-30T14:59:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 4 2024-03-30T14:59:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 10 client.57239.objecter _send_command 6 2024-03-30T14:59:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.362+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a68007fc0 con 0x7f2a6800d6e0 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.363+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a68007fc0 con 0x7f2a6800d6e0 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.363+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.363+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.363+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 4 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.363+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 2 6 2024-03-30T14:59:30.370 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 10 client.57239.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 7 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 7 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 3 7 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 4 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 10 client.57239.objecter _send_command 7 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2a68016030 con 0x7f2a68012520 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.364+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a68012520 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.365+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a68016030 con 0x7f2a68012520 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.365+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.365+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 7 = system:0 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.365+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 4 2024-03-30T14:59:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.365+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 3 7 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 10 client.57239.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 8 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 8 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6800cd30 osd=3 3 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 3 8 2024-03-30T14:59:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 4 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 10 client.57239.objecter _send_command 8 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.381+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a6800c810 con 0x7f2a68012520 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a6800c810 con 0x7f2a68012520 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 4 2024-03-30T14:59:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 3 8 2024-03-30T14:59:30.386 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 10 client.57239.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 9 2024-03-30T14:59:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 9 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 4 9 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 4 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 10 client.57239.objecter _send_command 9 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.383+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2a6801ab60 con 0x7f2a68012080 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.384+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a68012080 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.385+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a6801ab60 con 0x7f2a68012080 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.385+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.385+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 9 = system:0 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.385+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 4 2024-03-30T14:59:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.385+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 4 9 2024-03-30T14:59:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 10 client.57239.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 10 2024-03-30T14:59:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 10 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6800c810 osd=4 3 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 4 10 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 4 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 10 client.57239.objecter _send_command 10 2024-03-30T14:59:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.401+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a6801c510 con 0x7f2a68012080 2024-03-30T14:59:30.405 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a6801c510 con 0x7f2a68012080 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 4 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 4 10 2024-03-30T14:59:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 10 client.57239.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 11 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 11 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 5 11 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 4 2024-03-30T14:59:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 10 client.57239.objecter _send_command 11 2024-03-30T14:59:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.403+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2a6801d0e0 con 0x7f2a6801cc30 2024-03-30T14:59:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.404+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a6801cc30 2024-03-30T14:59:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.405+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a6801d0e0 con 0x7f2a6801cc30 2024-03-30T14:59:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.405+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:30.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.405+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 11 = system:0 2024-03-30T14:59:30.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.405+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 4 2024-03-30T14:59:30.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.405+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 5 11 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 10 client.57239.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 12 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 12 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801c510 osd=5 3 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 5 12 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 4 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 10 client.57239.objecter _send_command 12 2024-03-30T14:59:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.423+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a6801d0e0 con 0x7f2a6801cc30 2024-03-30T14:59:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.424+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a6801d0e0 con 0x7f2a6801cc30 2024-03-30T14:59:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.424+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.424+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.424+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 4 2024-03-30T14:59:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.424+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 5 12 2024-03-30T14:59:30.428 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 10 client.57239.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 13 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 13 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 6 13 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 4 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 10 client.57239.objecter _send_command 13 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2a6801d4e0 con 0x7f2a6801fc20 2024-03-30T14:59:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.425+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a6801fc20 2024-03-30T14:59:30.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:30.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:30.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:59:30.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:30.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.426+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a6801d4e0 con 0x7f2a6801fc20 2024-03-30T14:59:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.426+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.426+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 13 = system:0 2024-03-30T14:59:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.426+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 4 2024-03-30T14:59:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.426+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 6 13 2024-03-30T14:59:30.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:30.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi028-92979-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: osdmap e205: 8 total, 8 up, 8 in 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]: dispatch 2024-03-30T14:59:30.434 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 10 client.57239.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 14 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 14 2024-03-30T14:59:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801d0e0 osd=6 3 2024-03-30T14:59:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 6 14 2024-03-30T14:59:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 4 2024-03-30T14:59:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 10 client.57239.objecter _send_command 14 2024-03-30T14:59:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.443+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a6801d4e0 con 0x7f2a6801fc20 2024-03-30T14:59:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a6801d4e0 con 0x7f2a6801fc20 2024-03-30T14:59:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 4 2024-03-30T14:59:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 6 14 2024-03-30T14:59:30.448 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 10 client.57239.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 15 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 15 2024-03-30T14:59:30.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 7 15 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 4 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 10 client.57239.objecter _send_command 15 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.445+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2a68024860 con 0x7f2a680254f0 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 10 client.57239.objecter ms_handle_connect 0x7f2a680254f0 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a68024860 con 0x7f2a680254f0 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:30.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 15 = system:0 2024-03-30T14:59:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 4 2024-03-30T14:59:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.446+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 7 15 2024-03-30T14:59:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:30.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 10 client.57239.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign -1 16 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 20 client.57239.objecter _get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_remove -1 16 2024-03-30T14:59:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 20 client.57239.objecter get_session s=0x7f2a6801d4e0 osd=7 3 2024-03-30T14:59:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 15 client.57239.objecter _session_command_op_assign 7 16 2024-03-30T14:59:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 4 2024-03-30T14:59:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 10 client.57239.objecter _send_command 16 2024-03-30T14:59:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.463+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2a68024860 con 0x7f2a680254f0 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a78ff9700 1 -- 172.21.15.28:0/500251240 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2a68024860 con 0x7f2a680254f0 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a78ff9700 10 client.57239.objecter ms_dispatch 0x7f2a7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a78ff9700 10 client.57239.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a78ff9700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 4 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a78ff9700 15 client.57239.objecter _session_command_op_remove 7 16 2024-03-30T14:59:30.467 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 librados: watch_flush enter 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 librados: watch_flush exit 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.0 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6801/3394323575 conn(0x7f2a68004730 legacy=0x7f2a68002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a680053e0 osd=0 1 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.1 2024-03-30T14:59:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6805/1850313547 conn(0x7f2a680042f0 legacy=0x7f2a6800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68004180 osd=1 1 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.2 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6809/3451325043 conn(0x7f2a6800d6e0 legacy=0x7f2a6800fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a68008480 osd=2 1 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.3 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6813/1506157250 conn(0x7f2a68012520 legacy=0x7f2a68014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800cd30 osd=3 1 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.4 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.134:6800/3063775514 conn(0x7f2a68012080 legacy=0x7f2a68019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a7bfff700 1 -- 172.21.15.28:0/500251240 reap_dead start 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a7bfff700 1 -- 172.21.15.28:0/500251240 reap_dead start 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6800c810 osd=4 1 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.5 2024-03-30T14:59:30.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.134:6804/3668151179 conn(0x7f2a6801cc30 legacy=0x7f2a6801ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801c510 osd=5 1 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.6 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.134:6808/2135224194 conn(0x7f2a6801fc20 legacy=0x7f2a68022010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d0e0 osd=6 1 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter close_session for osd.7 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a7bfff700 1 -- 172.21.15.28:0/500251240 reap_dead start 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.134:6812/3758190298 conn(0x7f2a680254f0 legacy=0x7f2a68025930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter put_session s=0x7f2a6801d4e0 osd=7 1 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 20 client.57239.objecter shutdown clearing up homeless session... 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 10 client.57239.objecter successfully canceled tick 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2a640703a0 legacy=0x7f2a64072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.465+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> v1:172.21.15.28:6790/0 conn(0x7f2a7c10afb0 legacy=0x7f2a7c1a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.466+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 shutdown_connections 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.466+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 >> 172.21.15.28:0/500251240 conn(0x7f2a7c0fc9f0 msgr2=0x7f2a7c0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.466+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 shutdown_connections 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.466+0000 7f2a81e05700 1 -- 172.21.15.28:0/500251240 wait complete. 2024-03-30T14:59:30.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:30.466+0000 7f2a81e05700 1 librados: shutdown 2024-03-30T14:59:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:30.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:30.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:30.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:30.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:30.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:30.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:30.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:30.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:31.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:31.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:31.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWritescPP (73904 ms total) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (153 ms) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (153 ms total) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (101 ms) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (2 ms) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (103 ms total) 2024-03-30T14:59:31.228 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (38 ms) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (45 ms total) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (36 ms) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (40 ms total) 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.229 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1068 ms) 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1068 ms total) 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (106744 ms total) 2024-03-30T14:59:31.230 INFO:tasks.workunit.client.0.smithi028.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-30T14:59:31.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:31.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:31.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: pgmap v248: 308 pgs: 4 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 8 unknown, 280 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/500251240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: pgmap v248: 308 pgs: 4 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 8 unknown, 280 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/500251240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:31.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:31.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:31.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:31.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: pgmap v248: 308 pgs: 4 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 8 unknown, 280 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-13"}]': finished 2024-03-30T14:59:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: osdmap e206: 8 total, 8 up, 8 in 2024-03-30T14:59:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]: dispatch 2024-03-30T14:59:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/500251240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:31.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:31.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:31.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:31.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:31.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:31.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:31.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.992+0000 7fe4ceac1700 1 -- 172.21.15.28:0/2825051067 >> v1:172.21.15.28:6789/0 conn(0x7fe4c810d960 legacy=0x7fe4c810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:31.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.992+0000 7fe4ceac1700 1 -- 172.21.15.28:0/2825051067 shutdown_connections 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 -- 172.21.15.28:0/2825051067 >> 172.21.15.28:0/2825051067 conn(0x7fe4c80fc9e0 msgr2=0x7fe4c80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 -- 172.21.15.28:0/2825051067 shutdown_connections 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 -- 172.21.15.28:0/2825051067 wait complete. 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 librados: starting msgr at 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 librados: starting objecter 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 Processor -- start 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 -- start start 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 librados: setting wanted keys 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.993+0000 7fe4ceac1700 1 librados: calling monclient init 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4ceac1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4c81a9320 con 0x7fe4c8101130 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4ceac1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4c81aa500 con 0x7fe4c810d960 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4ceac1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe4c81ab740 con 0x7fe4c8109dc0 2024-03-30T14:59:31.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4cc85d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe4c8101130 0x7fe4c8106380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42964/0 (socket says 172.21.15.28:42964) 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c7fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe4c8109dc0 0x7fe4c81a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55776/0 (socket says 172.21.15.28:55776) 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4cc85d700 1 -- 172.21.15.28:0/3550037053 learned_addr learned my addr 172.21.15.28:0/3550037053 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter ms_handle_connect 0x7fe4c8101130 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter ms_handle_connect 0x7fe4c8109dc0 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter ms_handle_connect 0x7fe4c810d960 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103997186 0 0) 0x7fe4c81a9320 con 0x7fe4c8101130 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4b80035b0 con 0x7fe4c8101130 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 36181640 0 0) 0x7fe4c81ab740 con 0x7fe4c8109dc0 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4c81a9320 con 0x7fe4c8109dc0 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931581087 0 0) 0x7fe4c81aa500 con 0x7fe4c810d960 2024-03-30T14:59:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4c81ab740 con 0x7fe4c810d960 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1351110206 0 0) 0x7fe4c81a9320 con 0x7fe4c8109dc0 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4c81aa500 con 0x7fe4c8109dc0 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe4b40018b0 con 0x7fe4c8109dc0 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2428108132 0 0) 0x7fe4b80035b0 con 0x7fe4c8101130 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4c81a9320 con 0x7fe4c8101130 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3646227789 0 0) 0x7fe4c81ab740 con 0x7fe4c810d960 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4b80035b0 con 0x7fe4c810d960 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe4bc0012d0 con 0x7fe4c8101130 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.994+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe4c0004800 con 0x7fe4c810d960 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 277242107 0 0) 0x7fe4c81aa500 con 0x7fe4c8109dc0 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 >> v1:172.21.15.134:6789/0 conn(0x7fe4c810d960 legacy=0x7fe4c81a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 >> v1:172.21.15.28:6789/0 conn(0x7fe4c8101130 legacy=0x7fe4c8106380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c81ac980 con 0x7fe4c8109dc0 2024-03-30T14:59:32.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c81ab970 con 0x7fe4c8109dc0 2024-03-30T14:59:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe4b4003b40 con 0x7fe4c8109dc0 2024-03-30T14:59:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe4b4005be0 con 0x7fe4c8109dc0 2024-03-30T14:59:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4ceac1700 10 client.57245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4c81abe30 con 0x7fe4c8109dc0 2024-03-30T14:59:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.995+0000 7fe4ceac1700 1 librados: init done 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.996+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c8102610 con 0x7fe4c8109dc0 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.996+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe4b40036a0 con 0x7fe4c8109dc0 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(207..207 src has 53..207) ==== 10164+0+0 (unknown 150079891 0 0) 0x7fe4b4089f30 con 0x7fe4c8109dc0 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 10 client.57245.objecter ms_dispatch 0x7fe4c8001550 osd_map(207..207 src has 53..207) 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 3 client.57245.objecter handle_osd_map got epochs [207,207] > 0 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 3 client.57245.objecter handle_osd_map decoding full epoch 207 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 20 client.57245.objecter dump_active .. 0 homeless 2024-03-30T14:59:32.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:31.997+0000 7fe4c5ffb700 10 client.57245.objecter ms_handle_connect 0x7fe4b80703d0 2024-03-30T14:59:32.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:32.002+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe4b4056970 con 0x7fe4c8109dc0 2024-03-30T14:59:32.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:32.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:32.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:32.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:32.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:32.244+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe4c8103950 con 0x7fe4c8109dc0 2024-03-30T14:59:32.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:32.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:32.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: pgmap v251: 300 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 32 unknown, 258 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: pgmap v251: 300 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 32 unknown, 258 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:32.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:32.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:59:32.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:32.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:32.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:32.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/556303511' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi028-93257-36"}]': finished 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/96785430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi028-92999-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: osdmap e207: 8 total, 8 up, 8 in 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: pgmap v251: 300 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 32 unknown, 258 active+clean; 595 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s wr, 0 op/s 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: osdmap e208: 8 total, 8 up, 8 in 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2535484916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]: dispatch 2024-03-30T14:59:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:32.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:32.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:32.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:32.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:32.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:32.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:33.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:33.115 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.185+0000 7fe4c5ffb700 1 -- 172.21.15.28:0/3550037053 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v209) ==== 97+0+0 (unknown 4185939409 0 0) 0x7fe4b405a3c0 con 0x7fe4c8109dc0 2024-03-30T14:59:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.188+0000 7fe4ceac1700 10 librados: watch_flush enter 2024-03-30T14:59:33.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 10 librados: watch_flush exit 2024-03-30T14:59:33.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 20 client.57245.objecter shutdown clearing up homeless session... 2024-03-30T14:59:33.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 10 client.57245.objecter successfully canceled tick 2024-03-30T14:59:33.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe4b80703d0 legacy=0x7fe4b8072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 >> v1:172.21.15.28:6790/0 conn(0x7fe4c8109dc0 legacy=0x7fe4c81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 shutdown_connections 2024-03-30T14:59:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.189+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 >> 172.21.15.28:0/3550037053 conn(0x7fe4c80fc9e0 msgr2=0x7fe4c8110ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.190+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 shutdown_connections 2024-03-30T14:59:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.190+0000 7fe4ceac1700 1 -- 172.21.15.28:0/3550037053 wait complete. 2024-03-30T14:59:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:33.191+0000 7fe4ceac1700 1 librados: shutdown 2024-03-30T14:59:33.194 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T14:59:33.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:33.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:33.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:33.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3550037053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.57245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.57245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3550037053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.57245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.57245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:59:33.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:33.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:33.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:33.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:33.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3550037053' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.57245 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.57224 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi028-92979-27"}]': finished 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.57245 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: osdmap e209: 8 total, 8 up, 8 in 2024-03-30T14:59:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:33.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:33.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:33.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:33.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:33.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:33.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:34.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:34.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:34.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:34.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:34.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:34.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:34.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: pgmap v254: 332 pgs: 21 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 20 unknown, 281 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: pgmap v254: 332 pgs: 21 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 20 unknown, 281 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:34 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.484 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:34.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:34.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:34.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:34.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: pgmap v254: 332 pgs: 21 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 20 unknown, 281 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/382496424' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi028-92999-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi028-92979-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: osdmap e210: 8 total, 8 up, 8 in 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:34 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:34.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:34.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:34.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:34.886 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:59:34.886 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:59:34.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:34.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:34.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:35.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:59:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:35.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:35.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:35.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:35.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:35.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:35.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:35.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:35.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:35.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:35.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:35.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:35.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:36.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:36.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:36.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:36.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:36.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:36.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:36.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:36.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:36.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:36.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:36.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:36.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: pgmap v257: 300 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:59:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2865872511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[41614]: from='client.57263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: pgmap v257: 300 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:59:36.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2865872511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:36 smithi028 ceph-mon[29876]: from='client.57263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:36.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T14:59:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:36.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:36.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: osdmap e211: 8 total, 8 up, 8 in 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: pgmap v257: 300 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi028-92979-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: osdmap e212: 8 total, 8 up, 8 in 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2865872511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]: dispatch 2024-03-30T14:59:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:36 smithi134 ceph-mon[33714]: from='client.57263 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:36.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:36.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:37.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:37.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.073+0000 7fc6079a8700 1 -- 172.21.15.28:0/4033258605 >> v1:172.21.15.28:6790/0 conn(0x7fc600106650 legacy=0x7fc600106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.073+0000 7fc6079a8700 1 -- 172.21.15.28:0/4033258605 shutdown_connections 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.073+0000 7fc6079a8700 1 -- 172.21.15.28:0/4033258605 >> 172.21.15.28:0/4033258605 conn(0x7fc600077370 msgr2=0x7fc6000fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.073+0000 7fc6079a8700 1 -- 172.21.15.28:0/4033258605 shutdown_connections 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.073+0000 7fc6079a8700 1 -- 172.21.15.28:0/4033258605 wait complete. 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 librados: starting msgr at 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 librados: starting objecter 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 Processor -- start 2024-03-30T14:59:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 -- start start 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 librados: setting wanted keys 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 librados: calling monclient init 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6001a92d0 con 0x7fc600109500 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6001aa510 con 0x7fc60010d0d0 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc6079a8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6001ab750 con 0x7fc600106650 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc605744700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc600106650 0x7fc6001098e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55852/0 (socket says 172.21.15.28:55852) 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc605744700 1 -- 172.21.15.28:0/3087389991 learned_addr learned my addr 172.21.15.28:0/3087389991 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter ms_handle_connect 0x7fc600109500 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter ms_handle_connect 0x7fc600106650 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2277580482 0 0) 0x7fc6001a92d0 con 0x7fc600109500 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5e8003c50 con 0x7fc600109500 2024-03-30T14:59:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter ms_handle_connect 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.074+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686849469 0 0) 0x7fc6001ab750 con 0x7fc600106650 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6001a92d0 con 0x7fc600106650 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244275482 0 0) 0x7fc6001aa510 con 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6001ab750 con 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4237017970 0 0) 0x7fc5e8003c50 con 0x7fc600109500 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6001aa510 con 0x7fc600109500 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5f40017c0 con 0x7fc600109500 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3012627882 0 0) 0x7fc6001a92d0 con 0x7fc600106650 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5e8003c50 con 0x7fc600106650 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5f8004470 con 0x7fc600106650 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4041332740 0 0) 0x7fc6001ab750 con 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6001a92d0 con 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5f0001c30 con 0x7fc60010d0d0 2024-03-30T14:59:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 579470723 0 0) 0x7fc6001aa510 con 0x7fc600109500 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 >> v1:172.21.15.28:6790/0 conn(0x7fc600106650 legacy=0x7fc6001098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 >> v1:172.21.15.134:6789/0 conn(0x7fc60010d0d0 legacy=0x7fc6001a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6001ac990 con 0x7fc600109500 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6001ab980 con 0x7fc600109500 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc6079a8700 10 client.24132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6001abe40 con 0x7fc600109500 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc6079a8700 1 librados: init done 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc5f4003b20 con 0x7fc600109500 2024-03-30T14:59:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.075+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5f4004a90 con 0x7fc600109500 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.076+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ec005120 con 0x7fc600109500 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.077+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc5f4003680 con 0x7fc600109500 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.077+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(212..212 src has 53..212) ==== 10941+0+0 (unknown 1857688070 0 0) 0x7fc5f4088ed0 con 0x7fc600109500 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.077+0000 7fc5fe7fc700 10 client.24132.objecter ms_dispatch 0x7fc600001550 osd_map(212..212 src has 53..212) 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.077+0000 7fc5fe7fc700 3 client.24132.objecter handle_osd_map got epochs [212,212] > 0 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.077+0000 7fc5fe7fc700 3 client.24132.objecter handle_osd_map decoding full epoch 212 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.078+0000 7fc5fe7fc700 20 client.24132.objecter dump_active .. 0 homeless 2024-03-30T14:59:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.078+0000 7fc5fe7fc700 10 client.24132.objecter ms_handle_connect 0x7fc5e8070390 2024-03-30T14:59:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.082+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc5f4055600 con 0x7fc600109500 2024-03-30T14:59:37.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:37.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:37.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:37.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:37.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:37.351+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7fc5ec005410 con 0x7fc600109500 2024-03-30T14:59:37.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:37.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:37.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:37.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:37.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:37.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]': finished 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='client.57263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:59:37.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]': finished 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='client.57263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:59:37.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:59:37.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:37.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-15", "mode": "writeback"}]': finished 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='client.57263 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi028-92999-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: osdmap e213: 8 total, 8 up, 8 in 2024-03-30T14:59:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T14:59:37.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:37.965 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:37.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:38.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:38.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:38.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T14:59:38.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.191+0000 7fc5fe7fc700 1 -- 172.21.15.28:0/3087389991 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v214) ==== 113+0+0 (unknown 1280806501 0 0) 0x7fc5f4059050 con 0x7fc600109500 2024-03-30T14:59:38.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 10 librados: watch_flush enter 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 10 librados: watch_flush exit 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 20 client.24132.objecter shutdown clearing up homeless session... 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 10 client.24132.objecter successfully canceled tick 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc5e8070390 legacy=0x7fc5e8072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.194+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 >> v1:172.21.15.28:6789/0 conn(0x7fc600109500 legacy=0x7fc600109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.195+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 shutdown_connections 2024-03-30T14:59:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.195+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 >> 172.21.15.28:0/3087389991 conn(0x7fc600077370 msgr2=0x7fc60010fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:38.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.195+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 shutdown_connections 2024-03-30T14:59:38.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.197+0000 7fc6079a8700 1 -- 172.21.15.28:0/3087389991 wait complete. 2024-03-30T14:59:38.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:38.197+0000 7fc6079a8700 1 librados: shutdown 2024-03-30T14:59:38.199 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T14:59:38.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:38.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:38.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:38.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:38.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:38.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:38.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:38.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:38.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:38.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:38.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:38.851 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: pgmap v260: 340 pgs: 4 active+clean+snaptrim, 40 unknown, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.852 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: pgmap v260: 340 pgs: 4 active+clean+snaptrim, 40 unknown, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: pgmap v260: 340 pgs: 4 active+clean+snaptrim, 40 unknown, 9 active+clean+snaptrim_wait, 287 active+clean; 610 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3087389991' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: osdmap e214: 8 total, 8 up, 8 in 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:38.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:38.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:39.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:39.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:39.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:39.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:39.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:39.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:39.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:39.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1240692661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.57275 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1240692661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.57275 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:39.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:39.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:39.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: osdmap e215: 8 total, 8 up, 8 in 2024-03-30T14:59:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1240692661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.57275 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1261945810' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]: dispatch 2024-03-30T14:59:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]: dispatch 2024-03-30T14:59:39.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:39.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:39.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:40.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-03-30T14:59:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:40.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:40.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:40.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:40.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:40.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:40.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:40.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:40.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:40.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:40.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:40.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:40.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 -- 172.21.15.28:0/1700978823 >> v1:172.21.15.28:6790/0 conn(0x7f04c410eab0 legacy=0x7f04c4110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 -- 172.21.15.28:0/1700978823 shutdown_connections 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 -- 172.21.15.28:0/1700978823 >> 172.21.15.28:0/1700978823 conn(0x7f04c40fc9e0 msgr2=0x7f04c40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 -- 172.21.15.28:0/1700978823 shutdown_connections 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 -- 172.21.15.28:0/1700978823 wait complete. 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 librados: starting msgr at 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 librados: starting objecter 2024-03-30T14:59:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.765+0000 7f04cb75b700 1 Processor -- start 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 -- start start 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 librados: setting wanted keys 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 librados: calling monclient init 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04c40fec80 con 0x7f04c410afa0 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04c4102aa0 con 0x7f04c4103190 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04cb75b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04c4102cd0 con 0x7f04c410eab0 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c8cf6700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f04c410afa0 0x7f04c4101500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35214/0 (socket says 172.21.15.28:35214) 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c8cf6700 1 -- 172.21.15.28:0/2193372987 learned_addr learned my addr 172.21.15.28:0/2193372987 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c9cf8700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f04c410eab0 0x7f04c41a82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56834/0 (socket says 172.21.15.28:56834) 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter ms_handle_connect 0x7f04c410eab0 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter ms_handle_connect 0x7f04c410afa0 2024-03-30T14:59:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter ms_handle_connect 0x7f04c4103190 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676670286 0 0) 0x7f04c4102cd0 con 0x7f04c410eab0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04ac0031f0 con 0x7f04c410eab0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314456285 0 0) 0x7f04c4102aa0 con 0x7f04c4103190 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c4102cd0 con 0x7f04c4103190 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382019723 0 0) 0x7f04c40fec80 con 0x7f04c410afa0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c4102aa0 con 0x7f04c410afa0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2264415319 0 0) 0x7f04ac0031f0 con 0x7f04c410eab0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04c40fec80 con 0x7f04c410eab0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f04b4004470 con 0x7f04c410eab0 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2192444597 0 0) 0x7f04c4102cd0 con 0x7f04c4103190 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04ac0031f0 con 0x7f04c4103190 2024-03-30T14:59:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f04bc001bb0 con 0x7f04c4103190 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2894947640 0 0) 0x7f04c4102aa0 con 0x7f04c410afa0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04c4102cd0 con 0x7f04c410afa0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.766+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f04b80013d0 con 0x7f04c410afa0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2634055572 0 0) 0x7f04c40fec80 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.134:6789/0 conn(0x7f04c4103190 legacy=0x7f04c4100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6789/0 conn(0x7f04c410afa0 legacy=0x7f04c4101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c41aca40 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04c41a9a70 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04cb75b700 10 client.57287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f04c41a9f30 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04cb75b700 1 librados: init done 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f04b4001ab0 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f04b4004ce0 con 0x7f04c410eab0 2024-03-30T14:59:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.767+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f04b0005100 con 0x7f04c410eab0 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.768+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f04b4004610 con 0x7f04c410eab0 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(216..216 src has 53..216) ==== 9661+0+0 (unknown 2704938066 0 0) 0x7f04b4089c90 con 0x7f04c410eab0 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 osd_map(216..216 src has 53..216) 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 3 client.57287.objecter handle_osd_map got epochs [216,216] > 0 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 3 client.57287.objecter handle_osd_map decoding full epoch 216 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 20 client.57287.objecter dump_active .. 0 homeless 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v216) ==== 54+0+16 (unknown 1417165073 0 2420000192) 0x7f04b408c060 con 0x7f04c410eab0 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04ac070430 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 10 client.57287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 1 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 1 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 0 1 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 4 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 10 client.57287.objecter _send_command 1 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.769+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04b0004b50 con 0x7f04b0004730 2024-03-30T14:59:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.770+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b0004730 2024-03-30T14:59:40.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: pgmap v263: 332 pgs: 4 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 278 active+clean; 618 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-30T14:59:40.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:40.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:40.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:40.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.57275 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: pgmap v263: 332 pgs: 4 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 278 active+clean; 618 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.57275 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:40 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.771+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b0004b50 con 0x7f04b0004730 2024-03-30T14:59:40.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.771+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:40.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.771+0000 7f04c27fc700 10 client.57287.objecter _finish_command 1 = system:0 2024-03-30T14:59:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.771+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 4 2024-03-30T14:59:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.771+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 0 1 2024-03-30T14:59:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 10 client.57287.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 2 2024-03-30T14:59:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 2 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b00053e0 osd=0 3 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 0 2 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 4 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 10 client.57287.objecter _send_command 2 2024-03-30T14:59:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.788+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b0007b00 con 0x7f04b0004730 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b0007b00 con 0x7f04b0004730 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 10 client.57287.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 4 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 0 2 2024-03-30T14:59:40.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 10 client.57287.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 3 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 3 2024-03-30T14:59:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 1 3 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 4 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 10 client.57287.objecter _send_command 3 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f04b0002200 con 0x7f04b00042f0 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.790+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b00042f0 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.791+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b0002200 con 0x7f04b00042f0 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.791+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.791+0000 7f04c27fc700 10 client.57287.objecter _finish_command 3 = system:0 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.791+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 4 2024-03-30T14:59:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.791+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 1 3 2024-03-30T14:59:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 10 client.57287.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 4 2024-03-30T14:59:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 4 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b0004180 osd=1 3 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 1 4 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 4 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 10 client.57287.objecter _send_command 4 2024-03-30T14:59:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.808+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b0001660 con 0x7f04b00042f0 2024-03-30T14:59:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b0001660 con 0x7f04b00042f0 2024-03-30T14:59:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 10 client.57287.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 4 2024-03-30T14:59:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 1 4 2024-03-30T14:59:40.812 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:40.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 10 client.57287.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 5 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 5 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 2 5 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 4 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 10 client.57287.objecter _send_command 5 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f04b00111f0 con 0x7f04b000d6e0 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.810+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b000d6e0 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.811+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b00111f0 con 0x7f04b000d6e0 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.811+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.811+0000 7f04c27fc700 10 client.57287.objecter _finish_command 5 = system:0 2024-03-30T14:59:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.811+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 4 2024-03-30T14:59:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.811+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 2 5 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 10 client.57287.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 6 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 6 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b0008480 osd=2 3 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 2 6 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 4 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 10 client.57287.objecter _send_command 6 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.828+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b0007fc0 con 0x7f04b000d6e0 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.829+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b0007fc0 con 0x7f04b000d6e0 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.829+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.829+0000 7f04c27fc700 10 client.57287.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.829+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 4 2024-03-30T14:59:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.829+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 2 6 2024-03-30T14:59:40.831 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 10 client.57287.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 7 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 7 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 3 7 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 4 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 10 client.57287.objecter _send_command 7 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f04b0016030 con 0x7f04b0012520 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b0012520 2024-03-30T14:59:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b0016030 con 0x7f04b0012520 2024-03-30T14:59:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 10 client.57287.objecter _finish_command 7 = system:0 2024-03-30T14:59:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 4 2024-03-30T14:59:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.830+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 3 7 2024-03-30T14:59:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 10 client.57287.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 8 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 8 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b000cd30 osd=3 3 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 3 8 2024-03-30T14:59:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 4 2024-03-30T14:59:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 10 client.57287.objecter _send_command 8 2024-03-30T14:59:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.847+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b000c870 con 0x7f04b0012520 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b000c870 con 0x7f04b0012520 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04c27fc700 10 client.57287.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 4 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 3 8 2024-03-30T14:59:40.851 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 10 client.57287.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 9 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 9 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 4 9 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 4 2024-03-30T14:59:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 10 client.57287.objecter _send_command 9 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.849+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f04b001a8e0 con 0x7f04b0012080 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.850+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b0012080 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.851+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b001a8e0 con 0x7f04b0012080 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.851+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.851+0000 7f04c27fc700 10 client.57287.objecter _finish_command 9 = system:0 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.851+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 4 2024-03-30T14:59:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.851+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 4 9 2024-03-30T14:59:40.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 10 client.57287.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 10 2024-03-30T14:59:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 10 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b000c870 osd=4 3 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 4 10 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 4 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 10 client.57287.objecter _send_command 10 2024-03-30T14:59:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.867+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b001a8e0 con 0x7f04b0012080 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b001a8e0 con 0x7f04b0012080 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04c27fc700 10 client.57287.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 4 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 4 10 2024-03-30T14:59:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 10 client.57287.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 11 2024-03-30T14:59:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 11 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 5 11 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 4 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 10 client.57287.objecter _send_command 11 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.869+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f04b001ecf0 con 0x7f04b001b040 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b001b040 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b001ecf0 con 0x7f04b001b040 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 10 client.57287.objecter _finish_command 11 = system:0 2024-03-30T14:59:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 4 2024-03-30T14:59:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.870+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 5 11 2024-03-30T14:59:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:40.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:40.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 10 client.57287.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 12 2024-03-30T14:59:40.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 12 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001a920 osd=5 3 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 5 12 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 4 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 10 client.57287.objecter _send_command 12 2024-03-30T14:59:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.887+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b001ecf0 con 0x7f04b001b040 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b001ecf0 con 0x7f04b001b040 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04c27fc700 10 client.57287.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 4 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 5 12 2024-03-30T14:59:40.891 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 10 client.57287.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 13 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 13 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 6 13 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 4 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 10 client.57287.objecter _send_command 13 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.889+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f04b001ff80 con 0x7f04b0020c10 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.890+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b0020c10 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.891+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b001ff80 con 0x7f04b0020c10 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.891+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.891+0000 7f04c27fc700 10 client.57287.objecter _finish_command 13 = system:0 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.891+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 4 2024-03-30T14:59:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.891+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 6 13 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 10 client.57287.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 14 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 14 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001ecf0 osd=6 3 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 6 14 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 4 2024-03-30T14:59:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 10 client.57287.objecter _send_command 14 2024-03-30T14:59:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.908+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b001ff80 con 0x7f04b0020c10 2024-03-30T14:59:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.909+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b001ff80 con 0x7f04b0020c10 2024-03-30T14:59:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.909+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.909+0000 7f04c27fc700 10 client.57287.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.909+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 4 2024-03-30T14:59:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.909+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 6 14 2024-03-30T14:59:40.911 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 10 client.57287.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 15 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 15 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 7 15 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 4 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 10 client.57287.objecter _send_command 15 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f04b00275f0 con 0x7f04b0020200 2024-03-30T14:59:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.910+0000 7f04c27fc700 10 client.57287.objecter ms_handle_connect 0x7f04b0020200 2024-03-30T14:59:40.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.911+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f04b00275f0 con 0x7f04b0020200 2024-03-30T14:59:40.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.911+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:40.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.911+0000 7f04c27fc700 10 client.57287.objecter _finish_command 15 = system:0 2024-03-30T14:59:40.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.911+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 4 2024-03-30T14:59:40.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.911+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 7 15 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: pgmap v263: 332 pgs: 4 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 278 active+clean; 618 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.57275 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi028-92999-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.49300 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi028-92979-28"}]': finished 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-15"}]': finished 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: osdmap e216: 8 total, 8 up, 8 in 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:40 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 10 client.57287.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign -1 16 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 20 client.57287.objecter _get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_remove -1 16 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 20 client.57287.objecter get_session s=0x7f04b001ff80 osd=7 3 2024-03-30T14:59:40.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 15 client.57287.objecter _session_command_op_assign 7 16 2024-03-30T14:59:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 4 2024-03-30T14:59:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 10 client.57287.objecter _send_command 16 2024-03-30T14:59:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.933+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04b00277d0 con 0x7f04b0020200 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04c27fc700 1 -- 172.21.15.28:0/2193372987 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f04b00277d0 con 0x7f04b0020200 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04c27fc700 10 client.57287.objecter ms_dispatch 0x7f04c4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04c27fc700 10 client.57287.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04c27fc700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 4 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04c27fc700 15 client.57287.objecter _session_command_op_remove 7 16 2024-03-30T14:59:40.936 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 librados: watch_flush enter 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 librados: watch_flush exit 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.0 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6801/3394323575 conn(0x7f04b0004730 legacy=0x7f04b0002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b00053e0 osd=0 1 2024-03-30T14:59:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.1 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6805/1850313547 conn(0x7f04b00042f0 legacy=0x7f04b000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0004180 osd=1 1 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.2 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6809/3451325043 conn(0x7f04b000d6e0 legacy=0x7f04b000fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b0008480 osd=2 1 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.3 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6813/1506157250 conn(0x7f04b0012520 legacy=0x7f04b0014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000cd30 osd=3 1 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.4 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.134:6800/3063775514 conn(0x7f04b0012080 legacy=0x7f04b00191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b000c870 osd=4 1 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.5 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.134:6804/3668151179 conn(0x7f04b001b040 legacy=0x7f04b001d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001a920 osd=5 1 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.6 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.134:6808/2135224194 conn(0x7f04b0020c10 legacy=0x7f04b0021070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ecf0 osd=6 1 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter close_session for osd.7 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.134:6812/3758190298 conn(0x7f04b0020200 legacy=0x7f04b0025dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter put_session s=0x7f04b001ff80 osd=7 1 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 20 client.57287.objecter shutdown clearing up homeless session... 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.935+0000 7f04cb75b700 10 client.57287.objecter successfully canceled tick 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04c9cf8700 1 -- 172.21.15.28:0/2193372987 reap_dead start 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04c9cf8700 1 -- 172.21.15.28:0/2193372987 reap_dead start 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6800/3011431093 conn(0x7f04ac070430 legacy=0x7f04ac0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04c9cf8700 1 -- 172.21.15.28:0/2193372987 reap_dead start 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04c9cf8700 1 -- 172.21.15.28:0/2193372987 reap_dead start 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04c9cf8700 1 -- 172.21.15.28:0/2193372987 reap_dead start 2024-03-30T14:59:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> v1:172.21.15.28:6790/0 conn(0x7f04c410eab0 legacy=0x7f04c41a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 shutdown_connections 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 >> 172.21.15.28:0/2193372987 conn(0x7f04c40fc9e0 msgr2=0x7f04c4110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 shutdown_connections 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 -- 172.21.15.28:0/2193372987 wait complete. 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:40.936+0000 7f04cb75b700 1 librados: shutdown 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:40.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.007 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:41.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:41.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:41.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:41.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:41.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:41.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:41.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:41.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:41.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:41.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:41.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:41.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:41.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:41.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:41.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2193372987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:59:41.986 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2193372987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:41.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:41.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:41.988 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:41 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:41.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:42.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:42.108 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:59:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T14:59:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T14:59:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T14:59:42.109 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:59:42.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:42.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2193372987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi028-92979-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: osdmap e217: 8 total, 8 up, 8 in 2024-03-30T14:59:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2199235111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]: dispatch 2024-03-30T14:59:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:41 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout:hotsECPP.SnapRemovePP (2022 ms) 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1997 ms) 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2004 ms) 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9025 ms total) 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-30T14:59:42.194 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4041 ms) 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4209 ms) 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3748 ms) 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11998 ms total) 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (117644 ms total) 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-30T14:59:42.195 INFO:tasks.workunit.client.0.smithi028.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T14:59:42.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:42.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:42.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:42.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:42.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:42.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:42.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:42.817 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: pgmap v266: 260 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:42.817 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:42.817 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:42.817 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3627795266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[41614]: from='client.57293 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: pgmap v266: 260 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:42.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:42.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:59:42.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3627795266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:42 smithi028 ceph-mon[29876]: from='client.57293 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:42.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T14:59:42.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:42.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:42.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:43.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:43.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:43.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:43.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: pgmap v266: 260 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.57212 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi028-93525-21"}]': finished 2024-03-30T14:59:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: osdmap e218: 8 total, 8 up, 8 in 2024-03-30T14:59:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3627795266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:42 smithi134 ceph-mon[33714]: from='client.57293 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:43.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:43.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:43.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:43.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:43.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:43.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:43.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:43.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:43.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:43.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:43.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:43.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:43.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.57293 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:43.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.57293 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:43.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:44.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:44.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:44.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:44.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi028-92979-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.57293 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi028-92999-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: osdmap e219: 8 total, 8 up, 8 in 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-30T14:59:44.204 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (10938 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (27982 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8075 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3964 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2847 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3131 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2879 ms) 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-30T14:59:44.205 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2997 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3024 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2971 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3012 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3006 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2984 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3015 ms) 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-30T14:59:44.206 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3019 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3281 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2693 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3023 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3002 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2980 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3529 ms) 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-30T14:59:44.207 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2490 ms) 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2977 ms) 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3266 ms) 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2749 ms) 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3003 ms) 2024-03-30T14:59:44.208 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-30T14:59:44.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:44.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:44.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:44.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:44.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:44.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:44.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:44.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:44.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:44.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.738+0000 7f21bf42b700 1 -- 172.21.15.28:0/4080530605 >> v1:172.21.15.28:6790/0 conn(0x7f21b8109dc0 legacy=0x7f21b810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.738+0000 7f21bf42b700 1 -- 172.21.15.28:0/4080530605 shutdown_connections 2024-03-30T14:59:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.738+0000 7f21bf42b700 1 -- 172.21.15.28:0/4080530605 >> 172.21.15.28:0/4080530605 conn(0x7f21b80fc9e0 msgr2=0x7f21b80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.738+0000 7f21bf42b700 1 -- 172.21.15.28:0/4080530605 shutdown_connections 2024-03-30T14:59:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 -- 172.21.15.28:0/4080530605 wait complete. 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 librados: starting msgr at 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 librados: starting objecter 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 Processor -- start 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 -- start start 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 librados: setting wanted keys 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.739+0000 7f21bf42b700 1 librados: calling monclient init 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bf42b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21b81a9140 con 0x7f21b810d960 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bf42b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21b81aa360 con 0x7f21b8101130 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bf42b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21b81ab580 con 0x7f21b8109dc0 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bc9c6700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f21b8109dc0 0x7f21b81a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56888/0 (socket says 172.21.15.28:56888) 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bc9c6700 1 -- 172.21.15.28:0/2589994825 learned_addr learned my addr 172.21.15.28:0/2589994825 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21bd9c8700 1 --1- 172.21.15.28:0/2589994825 >> v1:172.21.15.28:6789/0 conn(0x7f21b810d960 0x7f21b81a7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35260/0 (socket says 172.21.15.28:35260) 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter ms_handle_connect 0x7f21b8109dc0 2024-03-30T14:59:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter ms_handle_connect 0x7f21b810d960 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193112900 0 0) 0x7f21b81ab580 con 0x7f21b8109dc0 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21a40039c0 con 0x7f21b8109dc0 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter ms_handle_connect 0x7f21b8101130 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892197592 0 0) 0x7f21b81a9140 con 0x7f21b810d960 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21b81ab580 con 0x7f21b810d960 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417181579 0 0) 0x7f21b81aa360 con 0x7f21b8101130 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21b81a9140 con 0x7f21b8101130 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2392807636 0 0) 0x7f21a40039c0 con 0x7f21b8109dc0 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21b81aa360 con 0x7f21b8109dc0 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f21a8001810 con 0x7f21b8109dc0 2024-03-30T14:59:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3088379337 0 0) 0x7f21b81a9140 con 0x7f21b8101130 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21a40039c0 con 0x7f21b8101130 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.740+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f21b4001830 con 0x7f21b8101130 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2033978277 0 0) 0x7f21b81ab580 con 0x7f21b810d960 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21b81a9140 con 0x7f21b810d960 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f21b0001db0 con 0x7f21b810d960 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1844928588 0 0) 0x7f21b81aa360 con 0x7f21b8109dc0 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 >> v1:172.21.15.134:6789/0 conn(0x7f21b8101130 legacy=0x7f21b8106140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 >> v1:172.21.15.28:6789/0 conn(0x7f21b810d960 legacy=0x7f21b81a7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b81ac7c0 con 0x7f21b8109dc0 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21b81a9370 con 0x7f21b8109dc0 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21bf42b700 10 client.57305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f21b81a9830 con 0x7f21b8109dc0 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21bf42b700 1 librados: init done 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f21a8001ac0 con 0x7f21b8109dc0 2024-03-30T14:59:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.741+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f21a8004e50 con 0x7f21b8109dc0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.742+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219c005120 con 0x7f21b8109dc0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.742+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f21a8004e30 con 0x7f21b8109dc0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(220..220 src has 53..220) ==== 9648+0+0 (unknown 2939775186 0 0) 0x7f21a8089d70 con 0x7f21b8109dc0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 10 client.57305.objecter ms_dispatch 0x7f21b8001550 osd_map(220..220 src has 53..220) 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 3 client.57305.objecter handle_osd_map got epochs [220,220] > 0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 3 client.57305.objecter handle_osd_map decoding full epoch 220 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 20 client.57305.objecter dump_active .. 0 homeless 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.743+0000 7f21ae7fc700 10 client.57305.objecter ms_handle_connect 0x7f21a40702c0 2024-03-30T14:59:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.749+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f21a8056990 con 0x7f21b8109dc0 2024-03-30T14:59:44.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: pgmap v269: 332 pgs: 5 creating+peering, 4 active+clean+snaptrim, 25 unknown, 6 active+clean+snaptrim_wait, 292 active+clean; 586 KiB data, 531 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: pgmap v269: 332 pgs: 5 creating+peering, 4 active+clean+snaptrim, 25 unknown, 6 active+clean+snaptrim_wait, 292 active+clean; 586 KiB data, 531 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:44.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:59:44.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:44.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:44.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:44.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:44.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:44.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:44.966+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f219c005410 con 0x7f21b8109dc0 2024-03-30T14:59:45.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:45.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: pgmap v269: 332 pgs: 5 creating+peering, 4 active+clean+snaptrim, 25 unknown, 6 active+clean+snaptrim_wait, 292 active+clean; 586 KiB data, 531 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: osdmap e220: 8 total, 8 up, 8 in 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.195+0000 7f21ae7fc700 1 -- 172.21.15.28:0/2589994825 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v221) ==== 91+0+0 (unknown 2861383169 0 0) 0x7f21a805a3e0 con 0x7f21b8109dc0 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.198+0000 7f21bf42b700 10 librados: watch_flush enter 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 10 librados: watch_flush exit 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 20 client.57305.objecter shutdown clearing up homeless session... 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 10 client.57305.objecter successfully canceled tick 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 >> v1:172.21.15.28:6800/3011431093 conn(0x7f21a40702c0 legacy=0x7f21a4072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 >> v1:172.21.15.28:6790/0 conn(0x7f21b8109dc0 legacy=0x7f21b81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 shutdown_connections 2024-03-30T14:59:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 >> 172.21.15.28:0/2589994825 conn(0x7f21b80fc9e0 msgr2=0x7f21b80fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 shutdown_connections 2024-03-30T14:59:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 -- 172.21.15.28:0/2589994825 wait complete. 2024-03-30T14:59:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:45.199+0000 7f21bf42b700 1 librados: shutdown 2024-03-30T14:59:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T14:59:45.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:45.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:45.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:45.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:45.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:45.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:45.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:45.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:45.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:46.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2589994825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.57305 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.57305 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2309712691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[29876]: from='client.57299 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2589994825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.57305 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.57305 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.026 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2309712691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.027 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:45 smithi028 ceph-mon[41614]: from='client.57299 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:46.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:46.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2589994825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.57305 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.57305 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: osdmap e221: 8 total, 8 up, 8 in 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]: dispatch 2024-03-30T14:59:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2309712691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:45 smithi134 ceph-mon[33714]: from='client.57299 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:46.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:46.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:46.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:46.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:46.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:46.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:46.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:46.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:46.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:46.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:46.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:46.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:46.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: pgmap v272: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: 42.1a scrub starts 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: 42.1a scrub ok 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]': finished 2024-03-30T14:59:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.57299 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: pgmap v272: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:46.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: 42.1a scrub starts 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: 42.1a scrub ok 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]': finished 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.57299 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:46.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:47.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:47.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: pgmap v272: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: 42.1a scrub starts 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: 42.1a scrub ok 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-17", "mode": "writeback"}]': finished 2024-03-30T14:59:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.57299 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi028-92999-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: osdmap e222: 8 total, 8 up, 8 in 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4286010033' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]: dispatch 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T14:59:47.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:47.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:47.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:47.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:47.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:47.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:47.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.728 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:47.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:47.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:47.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:47.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:47.991 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:47.992 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.993 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:47.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:47.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:47 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:48.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T14:59:48.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.57281 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi028-92979-29"}]': finished 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: osdmap e223: 8 total, 8 up, 8 in 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:47 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:48.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:48.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:48.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:48.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:48.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:48.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:48.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:48.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:48.672 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:48.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.757 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:48.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:48.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:48.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:48.918 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T14:59:48.918 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T14:59:48.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T14:59:48.918 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T14:59:48.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 449.533040169, 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93059181899999999, 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T14:59:48.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381208+0000", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8050000000000001e-06 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381216+0000", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2020000000000004e-06 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381226+0000", 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5489999999999995e-06 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381235+0000", 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1109999999999997e-06 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381298+0000", 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2657999999999995e-05 2024-03-30T14:59:48.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381306+0000", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3629999999999994e-06 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310542+0000", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92923640699999999 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310695+0000", 2024-03-30T14:59:48.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015222900000000001 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310906+0000", 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021148200000000001 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311551+0000", 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064435399999999998 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311644+0000", 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3702000000000005e-05 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311735+0000", 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0938000000000002e-05 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311769+0000", 2024-03-30T14:59:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3439999999999998e-05 2024-03-30T14:59:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311798+0000", 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9578999999999999e-05 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34346.0:4 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 311.58249871200002, 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14333030899999999, 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34346", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4086439972", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331750+0000", 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8610000000000001e-06 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331757+0000", 2024-03-30T14:59:48.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2699999999999999e-06 2024-03-30T14:59:48.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331776+0000", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8924e-05 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331795+0000", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8972999999999999e-05 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331857+0000", 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1649999999999994e-05 2024-03-30T14:59:48.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331866+0000", 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5149999999999995e-06 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474614+0000", 2024-03-30T14:59:48.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14274793599999999 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474650+0000", 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5559999999999998e-05 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475078+0000", 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042862 2024-03-30T14:59:48.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34352.0:2 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 311.52311917899999, 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084430388999999995, 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.934 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34352", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3269533966", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391129+0000", 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.773e-06 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391135+0000", 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9989999999999997e-06 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391143+0000", 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1689999999999992e-06 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391151+0000", 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7300000000000005e-06 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391213+0000", 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1749999999999997e-05 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391220+0000", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2359999999999998e-06 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475387+0000", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084166640000000001 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475417+0000", 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9899e-05 2024-03-30T14:59:48.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475558+0000", 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014119299999999999 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24907.0:2 24.b 24:d3fd6930:test-rados-api-smithi028-93417-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966645+0000", 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.947601547, 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27101285899999999, 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24907", 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1522502118", 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.940 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966647+0000", 2024-03-30T14:59:48.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.165e-06 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966656+0000", 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4060000000000005e-06 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966667+0000", 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.118e-05 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966679+0000", 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.202e-05 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967482+0000", 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080304200000000001 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967489+0000", 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8129999999999998e-06 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093584+0000", 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12609579200000001 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093596+0000", 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1165000000000001e-05 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224181+0000", 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13058547400000001 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224307+0000", 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125637 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224451+0000", 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014380600000000001 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226643+0000", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021921750000000002 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227566+0000", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092316999999999996 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237564+0000", 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099978580000000001 2024-03-30T14:59:48.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237612+0000", 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8216000000000001e-05 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237658+0000", 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5939999999999997e-05 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.947 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:2 23.10 23:0f4c2fcb:test-rados-api-smithi028-93408-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.971291+0000", 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.942955743, 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26655647300000002, 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971294+0000", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.242e-06 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971305+0000", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0713999999999999e-05 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971327+0000", 2024-03-30T14:59:48.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2089000000000001e-05 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971340+0000", 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2721999999999999e-05 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971419+0000", 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9756e-05 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971426+0000", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5610000000000004e-06 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072900+0000", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.101473934 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072909+0000", 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5359999999999998e-06 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226905+0000", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153995785 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227020+0000", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011438300000000001 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227154+0000", 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013489 2024-03-30T14:59:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227892+0000", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073729500000000001 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.235325+0000", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007433762 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237751+0000", 2024-03-30T14:59:48.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024258859999999999 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237804+0000", 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2366e-05 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237847+0000", 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3551999999999999e-05 2024-03-30T14:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.957 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24940.0:2 26.5 26:a21fad64:test-rados-api-smithi028-93750-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.979441+0000", 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.934806017, 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25955751599999999, 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24940", 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979443+0000", 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8150000000000002e-06 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979453+0000", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3689999999999996e-06 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979464+0000", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1061e-05 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979475+0000", 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1506e-05 2024-03-30T14:59:48.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057216+0000", 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.077740857999999996 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057225+0000", 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210726+0000", 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153500514 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210858+0000", 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132724 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.211010+0000", 2024-03-30T14:59:48.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015139900000000001 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.215902+0000", 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048923489999999998 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.230960+0000", 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015057615 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238899+0000", 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079398179999999995 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238954+0000", 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4842000000000001e-05 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238998+0000", 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3739000000000003e-05 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34880.0:2 13.16 13:6e692731:test-rados-api-smithi028-93525-2::foo:head [write 0~128] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.147626+0000", 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.766620098, 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.148417239, 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34880", 2024-03-30T14:59:48.966 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T14:59:48.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: 101.8 scrub starts 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: 101.8 scrub ok 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: 17.6 scrub starts 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: 17.6 scrub ok 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: pgmap v275: 292 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:59:48.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2222872349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[41614]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: 101.8 scrub starts 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: 101.8 scrub ok 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: 17.6 scrub starts 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: 17.6 scrub ok 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: pgmap v275: 292 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2222872349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:48 smithi028 ceph-mon[29876]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:48.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.971 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147629+0000", 2024-03-30T14:59:48.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147636+0000", 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6860000000000006e-06 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147651+0000", 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5071e-05 2024-03-30T14:59:48.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147662+0000", 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1103e-05 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147747+0000", 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4765000000000006e-05 2024-03-30T14:59:48.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147757+0000", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4099999999999997e-06 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280327+0000", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.132570166 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280452+0000", 2024-03-30T14:59:48.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125182 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280591+0000", 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013946 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282110+0000", 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015181719999999999 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282802+0000", 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069276799999999996 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295940+0000", 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013137729000000001 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295998+0000", 2024-03-30T14:59:48.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.766e-05 2024-03-30T14:59:48.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.296044+0000", 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5887000000000003e-05 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35207.0:2 45.6 45:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.133917+0000", 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 112.780329509, 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151675741, 2024-03-30T14:59:48.979 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35207", 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/517891268", 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T14:59:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133919+0000", 2024-03-30T14:59:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.902e-06 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133926+0000", 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4460000000000004e-06 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133942+0000", 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5692000000000002e-05 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133954+0000", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.169e-05 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134023+0000", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8934000000000001e-05 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134029+0000", 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6760000000000001e-06 2024-03-30T14:59:48.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223013+0000", 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088983584000000004 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223022+0000", 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5620000000000004e-06 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264315+0000", 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041293273999999998 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264431+0000", 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116506 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264556+0000", 2024-03-30T14:59:48.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124237 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.279474+0000", 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014917977000000001 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280282+0000", 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080855300000000005 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285498+0000", 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052162550000000004 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285549+0000", 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0924999999999998e-05 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285593+0000", 2024-03-30T14:59:48.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3528000000000001e-05 2024-03-30T14:59:48.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.988 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2059 11.7 e116/110 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head v 116'1, mlcod=0'0)", 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.970108+0000", 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 110.94413863, 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061806397999999999, 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T14:59:48.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970112+0000", 2024-03-30T14:59:48.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0609999999999997e-06 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013285+0000", 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.04317315 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013310+0000", 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4615000000000001e-05 2024-03-30T14:59:48.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013320+0000", 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0107e-05 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013400+0000", 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0039999999999999e-05 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013425+0000", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5494e-05 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031865+0000", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01844001 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031914+0000", 2024-03-30T14:59:48.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8921000000000001e-05 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2069 11.7 e116/110 11:e17e9421:test-rados-api-smithi028-93257-17::big:head v 116'4, mlcod=116'2)", 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.298185+0000", 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 110.616061493, 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.042950631000000003, 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:48.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298185+0000", 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298185+0000", 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298186+0000", 2024-03-30T14:59:48.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2950000000000001e-06 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323173+0000", 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024987056000000001 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323201+0000", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7954e-05 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323215+0000", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3456e-05 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323280+0000", 2024-03-30T14:59:49.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5036999999999994e-05 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323303+0000", 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.285e-05 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.341074+0000", 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017771022000000001 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.341136+0000", 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1960999999999998e-05 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2074 11.15 e117/110 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head v 117'1, mlcod=0'0)", 2024-03-30T14:59:49.003 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.677443+0000", 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 110.236803213, 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044884778, 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677447+0000", 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.811e-06 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689208+0000", 2024-03-30T14:59:49.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011761303000000001 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689235+0000", 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6809e-05 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689248+0000", 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2918000000000001e-05 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689313+0000", 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4999999999999994e-05 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689337+0000", 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3526000000000001e-05 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722259+0000", 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032922117000000001 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722328+0000", 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9294000000000004e-05 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35456.0:2 69.0 e121/120 69:0384ab8e:::oid:head v 121'1, mlcod=0'0)", 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:02.921584+0000", 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.992662741, 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039818633999999999, 2024-03-30T14:59:49.008 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921584+0000", 2024-03-30T14:59:49.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921584+0000", 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921584+0000", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.72e-07 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921592+0000", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1899999999999995e-06 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921597+0000", 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8289999999999998e-06 2024-03-30T14:59:49.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921605+0000", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7570000000000004e-06 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.948172+0000", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026567153999999999 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.948190+0000", 2024-03-30T14:59:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8368000000000001e-05 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.959565+0000", 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011374604 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.961402+0000", 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00183746 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25483.0:2 63.1b e121/119 63:d96a01e0:::testobj:head v 121'1, mlcod=0'0)", 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:02.921797+0000", 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 105.99244913, 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039609870999999998, 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921797+0000", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921797+0000", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921798+0000", 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4200000000000002e-07 2024-03-30T14:59:49.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921801+0000", 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6799999999999999e-06 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921805+0000", 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1439999999999996e-06 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.921810+0000", 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4859999999999999e-06 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.933884+0000", 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012073677999999999 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.933900+0000", 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6773e-05 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.954427+0000", 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020526284999999998 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.961407+0000", 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0069805830000000003 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32236.0:2 121.0 121:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:21.014681+0000", 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 87.899565437999996, 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2733807989999999, 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32236", 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/520874703", 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:49.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014685+0000", 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4340000000000001e-06 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014695+0000", 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0513000000000001e-05 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014709+0000", 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3791000000000001e-05 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014721+0000", 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2165e-05 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075744+0000", 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061023012000000001 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075753+0000", 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2e-06 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162599+0000", 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0868460719999999 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162721+0000", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121662 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.171761+0000", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090404189999999992 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.176616+0000", 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004854934 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.272952+0000", 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.096335658000000005 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.287962+0000", 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015010364 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288013+0000", 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0917000000000001e-05 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288062+0000", 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8658000000000001e-05 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27442.0:16 111.7 111:e0acde30:test-rados-api-smithi028-93525-9::foo:head [write 128~128] snapc 5=[5,4] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T14:59:49.026 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.920475+0000", 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.993771081000006, 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040266582000000002, 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27442", 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 16 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920475+0000", 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920475+0000", 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920477+0000", 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5460000000000001e-06 2024-03-30T14:59:49.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920485+0000", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6420000000000007e-06 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920495+0000", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.893e-06 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920505+0000", 2024-03-30T14:59:49.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0297e-05 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920748+0000", 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024347600000000001 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.920860+0000", 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111518 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.921076+0000", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021597000000000001 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.927547+0000", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064713679999999999 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.940240+0000", 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012692601 2024-03-30T14:59:49.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.960647+0000", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020406942000000001 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.960700+0000", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3480000000000003e-05 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.960742+0000", 2024-03-30T14:59:49.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1848999999999999e-05 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1598 124.0 e147/140 124:0384ab8e:::oid:1 v 147'6, mlcod=144'3)", 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.394209+0000", 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 77.520037504000001, 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050952077999999998, 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.033 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394210+0000", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.102e-06 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394226+0000", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554e-05 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394235+0000", 2024-03-30T14:59:49.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394244+0000", 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2120000000000003e-06 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396824+0000", 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025802059999999998 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396852+0000", 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8252999999999999e-05 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445122+0000", 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048270362999999997 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445161+0000", 2024-03-30T14:59:49.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8736999999999999e-05 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:35 111.1 e149/134 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head v 149'7, mlcod=148'4)", 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.224950+0000", 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 75.689296193000004, 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043878153000000003, 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.038 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224951+0000", 2024-03-30T14:59:49.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.91e-07 2024-03-30T14:59:49.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224962+0000", 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0858e-05 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224967+0000", 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9699999999999996e-06 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224977+0000", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1079999999999999e-06 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235114+0000", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010137607999999999 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235139+0000", 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4437e-05 2024-03-30T14:59:49.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268792+0000", 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033653216 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268829+0000", 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6665000000000002e-05 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.043 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.48856.0:9 157.2s0 157:523311ec:test-rados-api-smithi028-93441-13::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.185690+0000", 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 62.728556685000001, 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053529637999999997, 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.48856", 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1328251834", 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9 2024-03-30T14:59:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185693+0000", 2024-03-30T14:59:49.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1939999999999998e-06 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185705+0000", 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1446e-05 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185727+0000", 2024-03-30T14:59:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2249e-05 2024-03-30T14:59:49.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185739+0000", 2024-03-30T14:59:49.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185814+0000", 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5013999999999997e-05 2024-03-30T14:59:49.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185822+0000", 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5329999999999997e-06 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198631+0000", 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012809375 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198727+0000", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5884000000000006e-05 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.199305+0000", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057763400000000005 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.211644+0000", 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012338905000000001 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239140+0000", 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027495908999999999 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239220+0000", 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9802999999999995e-05 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.56915.0:2 173.1b 173:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:49.185572+0000", 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 59.728674712999997, 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.049005913, 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.56915", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1224869641", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185574+0000", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9039999999999999e-06 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185581+0000", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7090000000000002e-06 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185595+0000", 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3257e-05 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185606+0000", 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1419e-05 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193710+0000", 2024-03-30T14:59:49.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081039819999999992 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193719+0000", 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1670000000000005e-06 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223661+0000", 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299414410000001 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223739+0000", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8771999999999997e-05 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223867+0000", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705999999999999 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.225806+0000", 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019391670000000001 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.227421+0000", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00161532 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234490+0000", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070693969999999998 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234537+0000", 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6709999999999998e-05 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234578+0000", 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0608000000000002e-05 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57176.0:2 205.19 205:9f74c5ce:::oid_OperateMtime_:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:22.192868+0000", 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.721378513000001, 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039557034999999997, 2024-03-30T14:59:49.061 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57176", 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3329333735", 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192868+0000", 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192868+0000", 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192870+0000", 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.945e-06 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T14:59:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192876+0000", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3729999999999998e-06 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192887+0000", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0648e-05 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192898+0000", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0648999999999999e-05 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192957+0000", 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8869000000000002e-05 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.192966+0000", 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3530000000000001e-06 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.223320+0000", 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030353721 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.223436+0000", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011629300000000001 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.225239+0000", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018029649999999999 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.230571+0000", 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053321599999999999 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.230984+0000", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00041264399999999999 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.232342+0000", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013582139999999999 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.232385+0000", 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2957999999999998e-05 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T14:59:49.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:22.232425+0000", 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0243000000000001e-05 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T14:59:49.068 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T14:59:49.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: 101.8 scrub starts 2024-03-30T14:59:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: 101.8 scrub ok 2024-03-30T14:59:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: 17.6 scrub starts 2024-03-30T14:59:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: 17.6 scrub ok 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: pgmap v275: 292 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 282 active+clean; 586 KiB data, 532 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-17"}]': finished 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi028-92979-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: osdmap e224: 8 total, 8 up, 8 in 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2222872349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:48 smithi134 ceph-mon[33714]: from='client.57314 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:49.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:49.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 -- 172.21.15.28:0/1559750536 >> v1:172.21.15.28:6790/0 conn(0x7ff6a8108130 legacy=0x7ff6a8108510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 -- 172.21.15.28:0/1559750536 shutdown_connections 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 -- 172.21.15.28:0/1559750536 >> 172.21.15.28:0/1559750536 conn(0x7ff6a8076230 msgr2=0x7ff6a8076640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 -- 172.21.15.28:0/1559750536 shutdown_connections 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 -- 172.21.15.28:0/1559750536 wait complete. 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 librados: starting msgr at 2024-03-30T14:59:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.216+0000 7ff6b02af700 1 librados: starting objecter 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 Processor -- start 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 -- start start 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 librados: setting wanted keys 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 librados: calling monclient init 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6a8107920 con 0x7ff6a8108130 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6a81a86b0 con 0x7ff6a810aea0 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6b02af700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6a81a9890 con 0x7ff6a810eaa0 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6ae84c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff6a810eaa0 0x7ff6a81a5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58578/0 (socket says 172.21.15.28:58578) 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6ae84c700 1 -- 172.21.15.28:0/1641802454 learned_addr learned my addr 172.21.15.28:0/1641802454 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff6ae04b700 1 --1- 172.21.15.28:0/1641802454 >> v1:172.21.15.28:6789/0 conn(0x7ff6a8108130 0x7ff6a8104770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56910/0 (socket says 172.21.15.28:56910) 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter ms_handle_connect 0x7ff6a810eaa0 2024-03-30T14:59:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter ms_handle_connect 0x7ff6a8108130 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter ms_handle_connect 0x7ff6a810aea0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845750383 0 0) 0x7ff6a81a9890 con 0x7ff6a810eaa0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6940035b0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170483572 0 0) 0x7ff6a81a86b0 con 0x7ff6a810aea0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a81a9890 con 0x7ff6a810aea0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3918826378 0 0) 0x7ff6940035b0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6a81a86b0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6a0001c50 con 0x7ff6a810eaa0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.217+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1581767855 0 0) 0x7ff6a81a9890 con 0x7ff6a810aea0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6940035b0 con 0x7ff6a810aea0 2024-03-30T14:59:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6a40017e0 con 0x7ff6a810aea0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 180234652 0 0) 0x7ff6a81a86b0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 >> v1:172.21.15.134:6789/0 conn(0x7ff6a810aea0 legacy=0x7ff6a8104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 >> v1:172.21.15.28:6789/0 conn(0x7ff6a8108130 legacy=0x7ff6a8104770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a81aaa70 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a81a7700 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff6b02af700 10 client.57323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6a81a7bc0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff6b02af700 1 librados: init done 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff6a0003ec0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.218+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6a0006090 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.219+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68c005120 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.219+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff6a0003a20 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(225..225 src has 53..225) ==== 9238+0+0 (unknown 3128692126 0 0) 0x7ff6a0089fc0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 10 client.57323.objecter ms_dispatch 0x7ff6a8001550 osd_map(225..225 src has 53..225) 2024-03-30T14:59:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 3 client.57323.objecter handle_osd_map got epochs [225,225] > 0 2024-03-30T14:59:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 3 client.57323.objecter handle_osd_map decoding full epoch 225 2024-03-30T14:59:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 20 client.57323.objecter dump_active .. 0 homeless 2024-03-30T14:59:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.220+0000 7ff69f7fe700 10 client.57323.objecter ms_handle_connect 0x7ff694070370 2024-03-30T14:59:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.224+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff6a0056da0 con 0x7ff6a810eaa0 2024-03-30T14:59:49.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:49.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:49.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:49.467+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7ff68c005410 con 0x7ff6a810eaa0 2024-03-30T14:59:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:49.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:49.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:49.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:49.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:49.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:49.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 101.5 scrub starts 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 101.5 scrub ok 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T14:59:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:59:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1641802454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:49 smithi134 ceph-mon[33714]: from='client.57323 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:49.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:49.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:49.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:50.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:50.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.200+0000 7ff69f7fe700 1 -- 172.21.15.28:0/1641802454 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v226) ==== 113+0+0 (unknown 585027023 0 0) 0x7ff6a005a7f0 con 0x7ff6a810eaa0 2024-03-30T14:59:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 10 librados: watch_flush enter 2024-03-30T14:59:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 10 librados: watch_flush exit 2024-03-30T14:59:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 20 client.57323.objecter shutdown clearing up homeless session... 2024-03-30T14:59:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 10 client.57323.objecter successfully canceled tick 2024-03-30T14:59:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff694070370 legacy=0x7ff694072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 >> v1:172.21.15.28:6790/0 conn(0x7ff6a810eaa0 legacy=0x7ff6a81a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 shutdown_connections 2024-03-30T14:59:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.204+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 >> 172.21.15.28:0/1641802454 conn(0x7ff6a8076230 msgr2=0x7ff6a8110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:50.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 101.5 scrub starts 2024-03-30T14:59:50.206 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 101.5 scrub ok 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1641802454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[41614]: from='client.57323 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:50.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 101.5 scrub starts 2024-03-30T14:59:50.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 101.5 scrub ok 2024-03-30T14:59:50.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T14:59:50.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T14:59:50.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:50.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T14:59:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T14:59:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: from='client.57314 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi028-92999-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: osdmap e225: 8 total, 8 up, 8 in 2024-03-30T14:59:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1641802454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:49 smithi028 ceph-mon[29876]: from='client.57323 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T14:59:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.205+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 shutdown_connections 2024-03-30T14:59:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.206+0000 7ff6b02af700 1 -- 172.21.15.28:0/1641802454 wait complete. 2024-03-30T14:59:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:50.207+0000 7ff6b02af700 1 librados: shutdown 2024-03-30T14:59:50.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T14:59:50.244 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:50.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T14:59:50.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:50.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:50.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T14:59:50.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:50.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:50.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:50.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:50.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:50.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:50.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:50.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: pgmap v278: 292 pgs: 32 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.57323 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: pgmap v278: 292 pgs: 32 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:50.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.57323 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T14:59:50.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:50 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T14:59:50.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:51.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-03-30T14:59:51.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: pgmap v278: 292 pgs: 32 creating+peering, 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T14:59:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T14:59:51.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi028-92979-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.57323 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: osdmap e226: 8 total, 8 up, 8 in 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T14:59:51.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:50 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T14:59:51.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:51.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:51.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:51.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:51.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:51.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:51.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:51.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:51.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:51.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:51.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/796261573' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='client.49366 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:51.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/796261573' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='client.49366 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:51.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:52.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:52.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:52.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:52.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: osdmap e227: 8 total, 8 up, 8 in 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/796261573' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='client.49366 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T14:59:52.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:52.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:52.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:52.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:52.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:52.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:52.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:52.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:52.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:52.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:52.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: pgmap v281: 332 pgs: 4 active+clean+snaptrim, 72 unknown, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.49366 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T14:59:52.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: pgmap v281: 332 pgs: 4 active+clean+snaptrim, 72 unknown, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.49366 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T14:59:52.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:52 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T14:59:53.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:53.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:53.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T14:59:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: pgmap v281: 332 pgs: 4 active+clean+snaptrim, 72 unknown, 6 active+clean+snaptrim_wait, 250 active+clean; 586 KiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.49366 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi028-92999-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: osdmap e228: 8 total, 8 up, 8 in 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T14:59:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:52 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T14:59:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:53.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:53.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:53.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:53.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:53.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:53.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:53.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:53.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.740+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2473123517 >> v1:172.21.15.28:6790/0 conn(0x7f7ac010d9c0 legacy=0x7f7ac010fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2473123517 shutdown_connections 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2473123517 >> 172.21.15.28:0/2473123517 conn(0x7f7ac00761c0 msgr2=0x7f7ac00765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2473123517 shutdown_connections 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2473123517 wait complete. 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 librados: starting msgr at 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 librados: starting objecter 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 Processor -- start 2024-03-30T14:59:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 -- start start 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 librados: setting wanted keys 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.741+0000 7f7ac65a2700 1 librados: calling monclient init 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ac0101660 con 0x7f7ac010d9c0 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ac01a85c0 con 0x7f7ac0101ba0 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ac01a97a0 con 0x7f7ac0109e20 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abf7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7ac0109e20 0x7f7ac00ffb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58642/0 (socket says 172.21.15.28:58642) 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abf7fe700 1 -- 172.21.15.28:0/2866558914 learned_addr learned my addr 172.21.15.28:0/2866558914 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter ms_handle_connect 0x7f7ac0109e20 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter ms_handle_connect 0x7f7ac010d9c0 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102363884 0 0) 0x7f7ac0101660 con 0x7f7ac010d9c0 2024-03-30T14:59:53.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa80031f0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1803780202 0 0) 0x7f7ac01a97a0 con 0x7f7ac0109e20 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ac0101660 con 0x7f7ac0109e20 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter ms_handle_connect 0x7f7ac0101ba0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1746093571 0 0) 0x7f7ac01a85c0 con 0x7f7ac0101ba0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ac01a97a0 con 0x7f7ac0101ba0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1566082437 0 0) 0x7f7aa80031f0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ac01a85c0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2269710915 0 0) 0x7f7ac0101660 con 0x7f7ac0109e20 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7aa80031f0 con 0x7f7ac0109e20 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ab80047f0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ab4001890 con 0x7f7ac0109e20 2024-03-30T14:59:53.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3709640655 0 0) 0x7f7ac01a85c0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6790/0 conn(0x7f7ac0109e20 legacy=0x7f7ac00ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.134:6789/0 conn(0x7f7ac0101ba0 legacy=0x7f7ac00ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac01aa980 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac01a7610 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 10 client.24192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ac01a7ad0 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.742+0000 7f7ac65a2700 1 librados: init done 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.743+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7ab8001e30 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.743+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ab8005000 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.743+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7aac005100 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.744+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7ab8003b70 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(229..229 src has 53..229) ==== 8918+0+0 (unknown 4005226730 0 0) 0x7f7ab8088c00 con 0x7f7ac010d9c0 2024-03-30T14:59:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 osd_map(229..229 src has 53..229) 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 3 client.24192.objecter handle_osd_map got epochs [229,229] > 0 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 3 client.24192.objecter handle_osd_map decoding full epoch 229 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 20 client.24192.objecter dump_active .. 0 homeless 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v229) ==== 54+0+16 (unknown 3570346261 0 2420000192) 0x7f7ab8089060 con 0x7f7ac010d9c0 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.745+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aa8070150 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 1 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 1 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 0 1 2024-03-30T14:59:53.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 4 2024-03-30T14:59:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 10 client.24192.objecter _send_command 1 2024-03-30T14:59:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7aac004b50 con 0x7f7aac004730 2024-03-30T14:59:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.746+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac004730 2024-03-30T14:59:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.747+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac004b50 con 0x7f7aac004730 2024-03-30T14:59:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.747+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 1: 0 ) 2024-03-30T14:59:53.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.747+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 1 = system:0 2024-03-30T14:59:53.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.747+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 4 2024-03-30T14:59:53.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.747+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 0 1 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 2 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 2 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac0053e0 osd=0 3 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 0 2 2024-03-30T14:59:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 4 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 10 client.24192.objecter _send_command 2 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.765+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac0079f0 con 0x7f7aac004730 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.766+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac0079f0 con 0x7f7aac004730 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.766+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.766+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.766+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 4 2024-03-30T14:59:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.766+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 0 2 2024-03-30T14:59:53.768 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T14:59:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 3 2024-03-30T14:59:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 1 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 4 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 10 client.24192.objecter _send_command 3 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7aac002180 con 0x7f7aac004280 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.767+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac004280 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.768+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac002180 con 0x7f7aac004280 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.768+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 3: 0 ) 2024-03-30T14:59:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.768+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 3 = system:0 2024-03-30T14:59:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.768+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 4 2024-03-30T14:59:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.768+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 1 3 2024-03-30T14:59:53.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 4 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 4 2024-03-30T14:59:53.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac004110 osd=1 3 2024-03-30T14:59:53.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 1 4 2024-03-30T14:59:53.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 4 2024-03-30T14:59:53.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 10 client.24192.objecter _send_command 4 2024-03-30T14:59:53.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.784+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac001600 con 0x7f7aac004280 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac001600 con 0x7f7aac004280 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 4 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 1 4 2024-03-30T14:59:53.788 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 5 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 5 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 2 5 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 4 2024-03-30T14:59:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 10 client.24192.objecter _send_command 5 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7aac0110e0 con 0x7f7aac00d5d0 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.786+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac00d5d0 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.787+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac0110e0 con 0x7f7aac00d5d0 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.787+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 5: 0 ) 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.787+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 5 = system:0 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.787+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 4 2024-03-30T14:59:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.787+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 2 5 2024-03-30T14:59:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 6 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 6 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac008370 osd=2 3 2024-03-30T14:59:53.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 2 6 2024-03-30T14:59:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 4 2024-03-30T14:59:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 10 client.24192.objecter _send_command 6 2024-03-30T14:59:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.804+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac007e50 con 0x7f7aac00d5d0 2024-03-30T14:59:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac007e50 con 0x7f7aac00d5d0 2024-03-30T14:59:53.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 4 2024-03-30T14:59:53.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 2 6 2024-03-30T14:59:53.808 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T14:59:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 7 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 7 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 3 7 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 4 2024-03-30T14:59:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 10 client.24192.objecter _send_command 7 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7aac015f20 con 0x7f7aac012410 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.806+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac012410 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.807+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac015f20 con 0x7f7aac012410 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.807+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 7: 0 ) 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.807+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 7 = system:0 2024-03-30T14:59:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.807+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 4 2024-03-30T14:59:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.807+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 3 7 2024-03-30T14:59:53.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 8 2024-03-30T14:59:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 8 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00cc20 osd=3 3 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 3 8 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 4 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 10 client.24192.objecter _send_command 8 2024-03-30T14:59:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.824+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac00c700 con 0x7f7aac012410 2024-03-30T14:59:53.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T14:59:53.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:53.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:53.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T14:59:53.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:53 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac00c700 con 0x7f7aac012410 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 4 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 3 8 2024-03-30T14:59:53.829 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T14:59:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 9 2024-03-30T14:59:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 9 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 4 9 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 4 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 10 client.24192.objecter _send_command 9 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7aac00c6e0 con 0x7f7aac011f70 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.826+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac011f70 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.827+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac00c6e0 con 0x7f7aac011f70 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.827+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 9: 0 ) 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.827+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 9 = system:0 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.827+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 4 2024-03-30T14:59:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.827+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 4 9 2024-03-30T14:59:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:53.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 10 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 10 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00c4b0 osd=4 3 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 4 10 2024-03-30T14:59:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 4 2024-03-30T14:59:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 10 client.24192.objecter _send_command 10 2024-03-30T14:59:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.844+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac0203b0 con 0x7f7aac011f70 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac0203b0 con 0x7f7aac011f70 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 4 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 4 10 2024-03-30T14:59:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 11 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 11 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 5 11 2024-03-30T14:59:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 4 2024-03-30T14:59:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 10 client.24192.objecter _send_command 11 2024-03-30T14:59:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.846+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7aac020f80 con 0x7f7aac020ad0 2024-03-30T14:59:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac020ad0 2024-03-30T14:59:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac020f80 con 0x7f7aac020ad0 2024-03-30T14:59:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 11: 0 ) 2024-03-30T14:59:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 11 = system:0 2024-03-30T14:59:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 4 2024-03-30T14:59:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.847+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 5 11 2024-03-30T14:59:53.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 12 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 12 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac00c740 osd=5 3 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 5 12 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 4 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 10 client.24192.objecter _send_command 12 2024-03-30T14:59:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.864+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac021160 con 0x7f7aac020ad0 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac021160 con 0x7f7aac020ad0 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 4 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 5 12 2024-03-30T14:59:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 13 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 13 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 6 13 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 4 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 10 client.24192.objecter _send_command 13 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.866+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7aac021310 con 0x7f7aac023aa0 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.867+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac023aa0 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.868+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac021310 con 0x7f7aac023aa0 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.868+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 13: 0 ) 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.868+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 13 = system:0 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.868+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 4 2024-03-30T14:59:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.868+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 6 13 2024-03-30T14:59:53.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 14 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 14 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac020f80 osd=6 3 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 6 14 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 4 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 10 client.24192.objecter _send_command 14 2024-03-30T14:59:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.884+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac0275b0 con 0x7f7aac023aa0 2024-03-30T14:59:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac0275b0 con 0x7f7aac023aa0 2024-03-30T14:59:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 4 2024-03-30T14:59:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 6 14 2024-03-30T14:59:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T14:59:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T14:59:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 15 2024-03-30T14:59:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 15 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 7 15 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 4 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 10 client.24192.objecter _send_command 15 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.886+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7aac029880 con 0x7f7aac027cd0 2024-03-30T14:59:53.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.887+0000 7f7abd7fa700 10 client.24192.objecter ms_handle_connect 0x7f7aac027cd0 2024-03-30T14:59:53.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.888+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7aac029880 con 0x7f7aac027cd0 2024-03-30T14:59:53.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.888+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 15: 0 ) 2024-03-30T14:59:53.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.888+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 15 = system:0 2024-03-30T14:59:53.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.888+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 4 2024-03-30T14:59:53.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.888+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 7 15 2024-03-30T14:59:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 10 client.24192.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T14:59:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign -1 16 2024-03-30T14:59:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 20 client.24192.objecter _get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_remove -1 16 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 20 client.24192.objecter get_session s=0x7f7aac0275b0 osd=7 3 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 15 client.24192.objecter _session_command_op_assign 7 16 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 4 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 10 client.24192.objecter _send_command 16 2024-03-30T14:59:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.905+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7aac029880 con 0x7f7aac027cd0 2024-03-30T14:59:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7abd7fa700 1 -- 172.21.15.28:0/2866558914 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7aac029880 con 0x7f7aac027cd0 2024-03-30T14:59:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7abd7fa700 10 client.24192.objecter ms_dispatch 0x7f7ac0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T14:59:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7abd7fa700 10 client.24192.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T14:59:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7abd7fa700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 4 2024-03-30T14:59:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7abd7fa700 15 client.24192.objecter _session_command_op_remove 7 16 2024-03-30T14:59:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 librados: watch_flush enter 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 librados: watch_flush exit 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.0 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6801/3394323575 conn(0x7f7aac004730 legacy=0x7f7aac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0053e0 osd=0 1 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.1 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6805/1850313547 conn(0x7f7aac004280 legacy=0x7f7aac00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac004110 osd=1 1 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.2 2024-03-30T14:59:53.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6809/3451325043 conn(0x7f7aac00d5d0 legacy=0x7f7aac00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac008370 osd=2 1 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.3 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6813/1506157250 conn(0x7f7aac012410 legacy=0x7f7aac014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00cc20 osd=3 1 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.4 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.907+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.134:6800/3063775514 conn(0x7f7aac011f70 legacy=0x7f7aac01d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c4b0 osd=4 1 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.5 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.134:6804/3668151179 conn(0x7f7aac020ad0 legacy=0x7f7aac01ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac00c740 osd=5 1 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.6 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.134:6808/2135224194 conn(0x7f7aac023aa0 legacy=0x7f7aac025e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac020f80 osd=6 1 2024-03-30T14:59:53.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 10 client.24192.objecter close_session for osd.7 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.134:6812/3758190298 conn(0x7f7aac027cd0 legacy=0x7f7aac028130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 20 client.24192.objecter put_session s=0x7f7aac0275b0 osd=7 1 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 20 client.24192.objecter shutdown clearing up homeless session... 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 10 client.24192.objecter successfully canceled tick 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac4b3f700 1 -- 172.21.15.28:0/2866558914 reap_dead start 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac4b3f700 1 -- 172.21.15.28:0/2866558914 reap_dead start 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac4b3f700 1 -- 172.21.15.28:0/2866558914 reap_dead start 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac4b3f700 1 -- 172.21.15.28:0/2866558914 reap_dead start 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac4b3f700 1 -- 172.21.15.28:0/2866558914 reap_dead start 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7aa8070150 legacy=0x7f7aa8072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> v1:172.21.15.28:6789/0 conn(0x7f7ac010d9c0 legacy=0x7f7ac01a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 shutdown_connections 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 >> 172.21.15.28:0/2866558914 conn(0x7f7ac00761c0 msgr2=0x7f7ac01111e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:53.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 shutdown_connections 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 -- 172.21.15.28:0/2866558914 wait complete. 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:53.908+0000 7f7ac65a2700 1 librados: shutdown 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T14:59:53.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T14:59:53.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:53.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:53.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:54.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.018+0000 7f37f9464700 1 -- 172.21.15.28:0/180433447 >> v1:172.21.15.28:6790/0 conn(0x7f37f4109e00 legacy=0x7f37f410c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.018+0000 7f37f9464700 1 -- 172.21.15.28:0/180433447 shutdown_connections 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.018+0000 7f37f9464700 1 -- 172.21.15.28:0/180433447 >> 172.21.15.28:0/180433447 conn(0x7f37f40fca00 msgr2=0x7f37f40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.018+0000 7f37f9464700 1 -- 172.21.15.28:0/180433447 shutdown_connections 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.018+0000 7f37f9464700 1 -- 172.21.15.28:0/180433447 wait complete. 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 librados: starting msgr at 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 librados: starting objecter 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 Processor -- start 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 -- start start 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 librados: setting wanted keys 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 librados: calling monclient init 2024-03-30T14:59:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37f41a9400 con 0x7f37f4101170 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37f41aa620 con 0x7f37f410d9a0 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.019+0000 7f37f9464700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37f41ab860 con 0x7f37f4109e00 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f2ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f37f4101170 0x7f37f4106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56982/0 (socket says 172.21.15.28:56982) 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f2ffd700 1 -- 172.21.15.28:0/925799244 learned_addr learned my addr 172.21.15.28:0/925799244 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f27fc700 1 --1- 172.21.15.28:0/925799244 >> v1:172.21.15.28:6790/0 conn(0x7f37f4109e00 0x7f37f41a4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58648/0 (socket says 172.21.15.28:58648) 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter ms_handle_connect 0x7f37f4101170 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter ms_handle_connect 0x7f37f4109e00 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter ms_handle_connect 0x7f37f410d9a0 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070140779 0 0) 0x7f37f41ab860 con 0x7f37f4109e00 2024-03-30T14:59:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc0031f0 con 0x7f37f4109e00 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489334778 0 0) 0x7f37f41a9400 con 0x7f37f4101170 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f41ab860 con 0x7f37f4101170 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379163750 0 0) 0x7f37f41aa620 con 0x7f37f410d9a0 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f41a9400 con 0x7f37f410d9a0 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1376689857 0 0) 0x7f37dc0031f0 con 0x7f37f4109e00 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37f41aa620 con 0x7f37f4109e00 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37e8004470 con 0x7f37f4109e00 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1376224505 0 0) 0x7f37f41ab860 con 0x7f37f4101170 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37dc0031f0 con 0x7f37f4101170 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2410179931 0 0) 0x7f37f41a9400 con 0x7f37f410d9a0 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37f41ab860 con 0x7f37f410d9a0 2024-03-30T14:59:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37e40018b0 con 0x7f37f4101170 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37ec0013f0 con 0x7f37f410d9a0 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 595199571 0 0) 0x7f37f41aa620 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 >> v1:172.21.15.134:6789/0 conn(0x7f37f410d9a0 legacy=0x7f37f41a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 >> v1:172.21.15.28:6789/0 conn(0x7f37f4101170 legacy=0x7f37f4106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f41acaa0 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37f41aba90 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f9464700 10 client.57347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f37f41abf50 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.020+0000 7f37f9464700 1 librados: init done 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.021+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f37e8001ab0 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.021+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37e8004de0 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.021+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e0005120 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.022+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f37e8004f80 con 0x7f37f4109e00 2024-03-30T14:59:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.022+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(229..229 src has 53..229) ==== 8918+0+0 (unknown 4005226730 0 0) 0x7f37e8089c00 con 0x7f37f4109e00 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.022+0000 7f37dbfff700 10 client.57347.objecter ms_dispatch 0x7f37f4001550 osd_map(229..229 src has 53..229) 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.022+0000 7f37dbfff700 3 client.57347.objecter handle_osd_map got epochs [229,229] > 0 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.022+0000 7f37dbfff700 3 client.57347.objecter handle_osd_map decoding full epoch 229 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.023+0000 7f37dbfff700 20 client.57347.objecter dump_active .. 0 homeless 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.023+0000 7f37dbfff700 10 client.57347.objecter ms_handle_connect 0x7f37dc0704e0 2024-03-30T14:59:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.027+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f37e8056b00 con 0x7f37f4109e00 2024-03-30T14:59:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:54.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: osdmap e229: 8 total, 8 up, 8 in 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1904812330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]: dispatch 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]: dispatch 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T14:59:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:53 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T14:59:54.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:54.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:54.265+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f37e0005410 con 0x7f37f4109e00 2024-03-30T14:59:54.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:54.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:54.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:54.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:54.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:54.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:54.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:54.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:54.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:54.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:54.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:54.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:54.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:55.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:55.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:55.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: pgmap v284: 292 pgs: 4 active+clean+snaptrim, 11 unknown, 4 active+clean+snaptrim_wait, 273 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2866558914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:55.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]': finished 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/925799244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[41614]: from='client.57347 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: pgmap v284: 292 pgs: 4 active+clean+snaptrim, 11 unknown, 4 active+clean+snaptrim_wait, 273 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2866558914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T14:59:55.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]': finished 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/925799244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:54 smithi028 ceph-mon[29876]: from='client.57347 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T14:59:55.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: pgmap v284: 292 pgs: 4 active+clean+snaptrim, 11 unknown, 4 active+clean+snaptrim_wait, 273 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2866558914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T14:59:55.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-19", "mode": "writeback"}]': finished 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.49354 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi028-92979-30"}]': finished 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: osdmap e230: 8 total, 8 up, 8 in 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.57335 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/925799244' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:54 smithi134 ceph-mon[33714]: from='client.57347 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T14:59:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.199+0000 7f37dbfff700 1 -- 172.21.15.28:0/925799244 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v231) ==== 101+0+0 (unknown 715160524 0 0) 0x7f37e805a550 con 0x7f37f4109e00 2024-03-30T14:59:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.203+0000 7f37f9464700 10 librados: watch_flush enter 2024-03-30T14:59:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 10 librados: watch_flush exit 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 20 client.57347.objecter shutdown clearing up homeless session... 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 10 client.57347.objecter successfully canceled tick 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 >> v1:172.21.15.28:6800/3011431093 conn(0x7f37dc0704e0 legacy=0x7f37dc0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 >> v1:172.21.15.28:6790/0 conn(0x7f37f4109e00 legacy=0x7f37f41a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 shutdown_connections 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.204+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 >> 172.21.15.28:0/925799244 conn(0x7f37f40fca00 msgr2=0x7f37f4110b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.205+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 shutdown_connections 2024-03-30T14:59:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.205+0000 7f37f9464700 1 -- 172.21.15.28:0/925799244 wait complete. 2024-03-30T14:59:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T14:59:55.205+0000 7f37f9464700 1 librados: shutdown 2024-03-30T14:59:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T14:59:55.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:55.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:55.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:55.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:55.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:55.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:55.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:55.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:55.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:55.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:55.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:55.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:55.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: 17.15 scrub starts 2024-03-30T14:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: 17.15 scrub ok 2024-03-30T14:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: 42.1 scrub starts 2024-03-30T14:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: 42.1 scrub ok 2024-03-30T14:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.57347 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:55.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{ 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "prefix": "osd pool set", 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "var": "eio", 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "val": "true" 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: }]: dispatch 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: from='client.57335 ' entity='client.admin' cmd=[{ 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "prefix": "osd pool set", 2024-03-30T14:59:55.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "var": "eio", 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: "val": "true" 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[29876]: }]: dispatch 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: 17.15 scrub starts 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: 17.15 scrub ok 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: 42.1 scrub starts 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: 42.1 scrub ok 2024-03-30T14:59:55.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.57347 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:55.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{ 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "prefix": "osd pool set", 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "var": "eio", 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "val": "true" 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: }]: dispatch 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: from='client.57335 ' entity='client.admin' cmd=[{ 2024-03-30T14:59:55.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "prefix": "osd pool set", 2024-03-30T14:59:55.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:55.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "var": "eio", 2024-03-30T14:59:55.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: "val": "true" 2024-03-30T14:59:55.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:55 smithi028 ceph-mon[41614]: }]: dispatch 2024-03-30T14:59:55.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:56.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:56.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:56.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: 17.15 scrub starts 2024-03-30T14:59:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: 17.15 scrub ok 2024-03-30T14:59:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: 42.1 scrub starts 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: 42.1 scrub ok 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.57335 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi028-92999-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi028-92979-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.57347 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: osdmap e231: 8 total, 8 up, 8 in 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]: dispatch 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2486297809' entity='client.admin' cmd=[{ 2024-03-30T14:59:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "var": "eio", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "val": "true" 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: }]: dispatch 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: from='client.57335 ' entity='client.admin' cmd=[{ 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "var": "eio", 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: "val": "true" 2024-03-30T14:59:56.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:55 smithi134 ceph-mon[33714]: }]: dispatch 2024-03-30T14:59:56.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:56.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:56.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:56.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:56.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:56.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:56.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:56.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:56.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:56.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:56.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:56.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: pgmap v287: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 284 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 17.1f scrub starts 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 17.1f scrub ok 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: from='client.57335 ' entity='client.admin' cmd='[{ 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: "prefix": "osd pool set", 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: "var": "eio", 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: "val": "true" 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: }]': finished 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[41614]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T14:59:56.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: pgmap v287: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 284 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 17.1f scrub starts 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 17.1f scrub ok 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: from='client.57335 ' entity='client.admin' cmd='[{ 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: "prefix": "osd pool set", 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: "var": "eio", 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: "val": "true" 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: }]': finished 2024-03-30T14:59:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:56 smithi028 ceph-mon[29876]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:59:57.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:57.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:57.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:57.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T14:59:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T14:59:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T14:59:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: pgmap v287: 324 pgs: 4 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 284 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 17.1f scrub starts 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 17.1f scrub ok 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-19"}]': finished 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: from='client.57335 ' entity='client.admin' cmd='[{ 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: "prefix": "osd pool set", 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: "pool": "PoolEIOFlag_smithi028-92999-33", 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: "var": "eio", 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: "val": "true" 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: }]': finished 2024-03-30T14:59:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:56 smithi134 ceph-mon[33714]: osdmap e232: 8 total, 8 up, 8 in 2024-03-30T14:59:57.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T14:59:57.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:57.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:57.373 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T14:59:57.373 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T14:59:57.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:57.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:57.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:57.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:57.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T14:59:57.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:57.782 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:57.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:57.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:57.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:57.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 42.10 scrub starts 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 42.10 scrub ok 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:58.011 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 6.19 deep-scrub starts 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: 6.19 deep-scrub ok 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[41614]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 42.10 scrub starts 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 42.10 scrub ok 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 6.19 deep-scrub starts 2024-03-30T14:59:58.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: 6.19 deep-scrub ok 2024-03-30T14:59:58.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T14:59:58.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:57 smithi028 ceph-mon[29876]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:59:58.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:58.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:58.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T14:59:58.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 42.10 scrub starts 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 42.10 scrub ok 2024-03-30T14:59:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 6.19 deep-scrub starts 2024-03-30T14:59:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: 6.19 deep-scrub ok 2024-03-30T14:59:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi028-92979-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T14:59:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:57 smithi134 ceph-mon[33714]: osdmap e233: 8 total, 8 up, 8 in 2024-03-30T14:59:58.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:58.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:58.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:58.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:58.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:14:59:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T14:59:58.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:58.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:58.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T14:59:58.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:58.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:58.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:58.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:58.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T14:59:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:58.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:58.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: pgmap v290: 268 pgs: 4 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 252 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: 42.0 scrub starts 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: 42.0 scrub ok 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:59:58.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1696588145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[41614]: from='client.57359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: pgmap v290: 268 pgs: 4 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 252 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: 42.0 scrub starts 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: 42.0 scrub ok 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:59:58.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1696588145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:58.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:58 smithi028 ceph-mon[29876]: from='client.57359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:59.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:59.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T14:59:59.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T14:59:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:59.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: pgmap v290: 268 pgs: 4 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 252 active+clean; 586 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: 42.0 scrub starts 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: 42.0 scrub ok 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T14:59:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: osdmap e234: 8 total, 8 up, 8 in 2024-03-30T14:59:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1696588145' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:58 smithi134 ceph-mon[33714]: from='client.57359 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T14:59:59.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T14:59:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:59.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T14:59:59.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:59.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:59.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T14:59:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:59.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T14:59:59.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T14:59:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:59.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T14:59:59.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T14:59:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T14:59:59.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T14:59:59.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.57359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T14:59:59.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.57359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T14:59:59.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 14:59:59 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T14:59:59.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T14:59:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T14:59:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T14:59:59.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:00.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:00.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:00.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:00.173 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.57359 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi028-92999-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: osdmap e235: 8 total, 8 up, 8 in 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T15:00:00.174 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 14:59:59 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T15:00:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:00.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:00.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.319 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:00.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:00.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:00.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:00.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- 172.21.15.28:0/2199210360 >> v1:172.21.15.28:6790/0 conn(0x7f8b98109e10 legacy=0x7f8b9810c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- 172.21.15.28:0/2199210360 shutdown_connections 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- 172.21.15.28:0/2199210360 >> 172.21.15.28:0/2199210360 conn(0x7f8b980fca30 msgr2=0x7f8b980fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- 172.21.15.28:0/2199210360 shutdown_connections 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- 172.21.15.28:0/2199210360 wait complete. 2024-03-30T15:00:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 librados: starting msgr at 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 librados: starting objecter 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 Processor -- start 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 -- start start 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 librados: setting wanted keys 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.504+0000 7f8b9e77a700 1 librados: calling monclient init 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b9e77a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b981a4f50 con 0x7f8b98101180 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b9e77a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b981a6170 con 0x7f8b98109e10 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b9e77a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b981a63a0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b97fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b98101180 0x7f8b98106360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45968/0 (socket says 172.21.15.28:45968) 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b97fff700 1 -- 172.21.15.28:0/233695796 learned_addr learned my addr 172.21.15.28:0/233695796 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 10 client.?.objecter ms_handle_connect 0x7f8b9810d9b0 2024-03-30T15:00:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187332639 0 0) 0x7f8b981a63a0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b800031f0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 10 client.?.objecter ms_handle_connect 0x7f8b98109e10 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.505+0000 7f8b957fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 759302651 0 0) 0x7f8b981a6170 con 0x7f8b98109e10 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b981a63a0 con 0x7f8b98109e10 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 10 client.?.objecter ms_handle_connect 0x7f8b98101180 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4218950054 0 0) 0x7f8b800031f0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b981a6170 con 0x7f8b9810d9b0 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b90001590 con 0x7f8b9810d9b0 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1722945199 0 0) 0x7f8b981a4f50 con 0x7f8b98101180 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b800031f0 con 0x7f8b98101180 2024-03-30T15:00:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3174980876 0 0) 0x7f8b981a63a0 con 0x7f8b98109e10 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b981a4f50 con 0x7f8b98109e10 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b8c004470 con 0x7f8b98109e10 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2485787136 0 0) 0x7f8b981a6170 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 >> v1:172.21.15.134:6789/0 conn(0x7f8b98109e10 legacy=0x7f8b9819fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 >> v1:172.21.15.28:6789/0 conn(0x7f8b98101180 legacy=0x7f8b98106360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b981a65d0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b981a5120 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b9e77a700 10 client.57371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b981a55e0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b9e77a700 1 librados: init done 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.506+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8b90003ea0 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.507+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b90004d30 con 0x7f8b9810d9b0 2024-03-30T15:00:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.507+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b84005120 con 0x7f8b9810d9b0 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8b9001b650 con 0x7f8b9810d9b0 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(236..236 src has 53..236) ==== 8923+0+0 (unknown 4225478408 0 0) 0x7f8b90088c10 con 0x7f8b9810d9b0 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 10 client.57371.objecter ms_dispatch 0x7f8b98001550 osd_map(236..236 src has 53..236) 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 3 client.57371.objecter handle_osd_map got epochs [236,236] > 0 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 3 client.57371.objecter handle_osd_map decoding full epoch 236 2024-03-30T15:00:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.508+0000 7f8b957fa700 20 client.57371.objecter dump_active .. 0 homeless 2024-03-30T15:00:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.509+0000 7f8b957fa700 10 client.57371.objecter ms_handle_connect 0x7f8b800705b0 2024-03-30T15:00:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.513+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8b90055b20 con 0x7f8b9810d9b0 2024-03-30T15:00:00.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:00.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:00.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:00.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:00.735+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f8b84005410 con 0x7f8b9810d9b0 2024-03-30T15:00:00.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:00.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:00.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:00.960 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:00.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:01.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:01.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:01.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: pgmap v293: 324 pgs: 64 creating+peering, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: application not enabled on pool 'RoundTripAppend_smithi028-92979-31' 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 17.19 scrub starts 2024-03-30T15:00:01.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 17.19 scrub ok 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:00:01.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/233695796' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[29876]: from='client.57371 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: pgmap v293: 324 pgs: 64 creating+peering, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: application not enabled on pool 'RoundTripAppend_smithi028-92979-31' 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:01.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 17.19 scrub starts 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 17.19 scrub ok 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:00:01.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:00:01.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/233695796' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:00 smithi028 ceph-mon[41614]: from='client.57371 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: pgmap v293: 324 pgs: 64 creating+peering, 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: application not enabled on pool 'RoundTripAppend_smithi028-92979-31' 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 17.19 scrub starts 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 17.19 scrub ok 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:00:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: osdmap e236: 8 total, 8 up, 8 in 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2879088186' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/233695796' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:00 smithi134 ceph-mon[33714]: from='client.57371 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:00:01.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:01.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.212+0000 7f8b957fa700 1 -- 172.21.15.28:0/233695796 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v237) ==== 113+0+0 (unknown 1429578716 0 0) 0x7f8b90059570 con 0x7f8b9810d9b0 2024-03-30T15:00:01.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:01.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.216+0000 7f8b9e77a700 10 librados: watch_flush enter 2024-03-30T15:00:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:01.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.218+0000 7f8b9e77a700 10 librados: watch_flush exit 2024-03-30T15:00:01.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.218+0000 7f8b9e77a700 20 client.57371.objecter shutdown clearing up homeless session... 2024-03-30T15:00:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.218+0000 7f8b9e77a700 10 client.57371.objecter successfully canceled tick 2024-03-30T15:00:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.218+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8b800705b0 legacy=0x7f8b80072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.218+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 >> v1:172.21.15.28:6790/0 conn(0x7f8b9810d9b0 legacy=0x7f8b981a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.221+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 shutdown_connections 2024-03-30T15:00:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.221+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 >> 172.21.15.28:0/233695796 conn(0x7f8b980fca30 msgr2=0x7f8b9806aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.221+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 shutdown_connections 2024-03-30T15:00:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.221+0000 7f8b9e77a700 1 -- 172.21.15.28:0/233695796 wait complete. 2024-03-30T15:00:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:01.221+0000 7f8b9e77a700 1 librados: shutdown 2024-03-30T15:00:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T15:00:01.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:01.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:01.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:01.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:01.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:01.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:01.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:01.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:01.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:01.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:01.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: 12.7 deep-scrub starts 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: 12.7 deep-scrub ok 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57371 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:01.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3409750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57365 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: 3.17 deep-scrub starts 2024-03-30T15:00:01.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[41614]: 3.17 deep-scrub ok 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: 12.7 deep-scrub starts 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: 12.7 deep-scrub ok 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57371 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3409750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57365 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: 3.17 deep-scrub starts 2024-03-30T15:00:01.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:01 smithi028 ceph-mon[29876]: 3.17 deep-scrub ok 2024-03-30T15:00:01.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:01.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:02.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: 12.7 deep-scrub starts 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: 12.7 deep-scrub ok 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57353 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi028-92979-31"}]': finished 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57371 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: osdmap e237: 8 total, 8 up, 8 in 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:02.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3409750943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57365 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: 3.17 deep-scrub starts 2024-03-30T15:00:02.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:01 smithi134 ceph-mon[33714]: 3.17 deep-scrub ok 2024-03-30T15:00:02.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:02.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:02.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:02.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:02.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:02.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:02.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:02.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:02.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:02.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:02.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:02.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: pgmap v296: 324 pgs: 32 creating+peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: 17.1c scrub starts 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: 17.1c scrub ok 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]': finished 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.57365 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: 6.11 deep-scrub starts 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[41614]: 6.11 deep-scrub ok 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: pgmap v296: 324 pgs: 32 creating+peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: 17.1c scrub starts 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: 17.1c scrub ok 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]': finished 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.57365 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:02.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: 6.11 deep-scrub starts 2024-03-30T15:00:02.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:02 smithi028 ceph-mon[29876]: 6.11 deep-scrub ok 2024-03-30T15:00:02.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-03-30T15:00:02.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:03.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:03.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:03.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: pgmap v296: 324 pgs: 32 creating+peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 253 active+clean; 586 KiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: 17.1c scrub starts 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: 17.1c scrub ok 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-21", "mode": "writeback"}]': finished 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.57365 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi028-92999-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi028-92979-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: osdmap e238: 8 total, 8 up, 8 in 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: 6.11 deep-scrub starts 2024-03-30T15:00:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:02 smithi134 ceph-mon[33714]: 6.11 deep-scrub ok 2024-03-30T15:00:03.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:03.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:03.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:03.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:03.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:03.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:03.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:03.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:04.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:04.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 6.b deep-scrub starts 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 6.b deep-scrub ok 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 101.3 scrub starts 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 101.3 scrub ok 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 42.c scrub starts 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: 42.c scrub ok 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:04.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 6.b deep-scrub starts 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 6.b deep-scrub ok 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 101.3 scrub starts 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 101.3 scrub ok 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 42.c scrub starts 2024-03-30T15:00:04.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: 42.c scrub ok 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T15:00:04.019 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:04.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:04.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:04.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 6.b deep-scrub starts 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 6.b deep-scrub ok 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 101.3 scrub starts 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 101.3 scrub ok 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 42.c scrub starts 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: 42.c scrub ok 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: osdmap e239: 8 total, 8 up, 8 in 2024-03-30T15:00:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]: dispatch 2024-03-30T15:00:04.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:04.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.468+0000 7fb1fb352700 1 -- 172.21.15.28:0/3728473667 >> v1:172.21.15.28:6790/0 conn(0x7fb1f4101130 legacy=0x7fb1f4101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.468+0000 7fb1fb352700 1 -- 172.21.15.28:0/3728473667 shutdown_connections 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 -- 172.21.15.28:0/3728473667 >> 172.21.15.28:0/3728473667 conn(0x7fb1f40fc9e0 msgr2=0x7fb1f40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 -- 172.21.15.28:0/3728473667 shutdown_connections 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 -- 172.21.15.28:0/3728473667 wait complete. 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 librados: starting msgr at 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 librados: starting objecter 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 Processor -- start 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 -- start start 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 librados: setting wanted keys 2024-03-30T15:00:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.469+0000 7fb1fb352700 1 librados: calling monclient init 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1fb352700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1f4106870 con 0x7fb1f4109dc0 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1fb352700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1f41a8540 con 0x7fb1f410d960 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1fb352700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb1f41a9720 con 0x7fb1f4101130 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1f90ee700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb1f4101130 0x7fb1f4104640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56340/0 (socket says 172.21.15.28:56340) 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1f90ee700 1 -- 172.21.15.28:0/2919604687 learned_addr learned my addr 172.21.15.28:0/2919604687 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1f88ed700 1 --1- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6789/0 conn(0x7fb1f4109dc0 0x7fb1f4104d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46040/0 (socket says 172.21.15.28:46040) 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter ms_handle_connect 0x7fb1f4101130 2024-03-30T15:00:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter ms_handle_connect 0x7fb1f4109dc0 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter ms_handle_connect 0x7fb1f410d960 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1826433535 0 0) 0x7fb1f41a9720 con 0x7fb1f4101130 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e00031f0 con 0x7fb1f4101130 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169192199 0 0) 0x7fb1f4106870 con 0x7fb1f4109dc0 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1f41a9720 con 0x7fb1f4109dc0 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695172882 0 0) 0x7fb1f41a8540 con 0x7fb1f410d960 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1f4106870 con 0x7fb1f410d960 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 381824528 0 0) 0x7fb1e00031f0 con 0x7fb1f4101130 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1f41a8540 con 0x7fb1f4101130 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb1f0004680 con 0x7fb1f4101130 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4108228825 0 0) 0x7fb1f4106870 con 0x7fb1f410d960 2024-03-30T15:00:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1e00031f0 con 0x7fb1f410d960 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb1ec001470 con 0x7fb1f410d960 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1131280336 0 0) 0x7fb1f41a8540 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.134:6789/0 conn(0x7fb1f410d960 legacy=0x7fb1f41a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.470+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6789/0 conn(0x7fb1f4109dc0 legacy=0x7fb1f4104d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f41aa900 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f41a7590 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1fb352700 10 client.57389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1f41a7a50 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1fb352700 1 librados: init done 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb1f0001af0 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb1f0004fa0 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.471+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb1d8005100 con 0x7fb1f4101130 2024-03-30T15:00:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.472+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb1f0004f80 con 0x7fb1f4101130 2024-03-30T15:00:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(240..240 src has 53..240) ==== 10019+0+0 (unknown 352102238 0 0) 0x7fb1f0089e90 con 0x7fb1f4101130 2024-03-30T15:00:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 osd_map(240..240 src has 53..240) 2024-03-30T15:00:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 3 client.57389.objecter handle_osd_map got epochs [240,240] > 0 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 3 client.57389.objecter handle_osd_map decoding full epoch 240 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 20 client.57389.objecter dump_active .. 0 homeless 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v240) ==== 54+0+16 (unknown 1531916168 0 2420000192) 0x7fb1f008c270 con 0x7fb1f4101130 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1e0070060 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 10 client.57389.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 1 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 1 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 0 1 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 4 2024-03-30T15:00:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 10 client.57389.objecter _send_command 1 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1d8004b50 con 0x7fb1d8004730 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.473+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d8004730 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.474+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d8004b50 con 0x7fb1d8004730 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.474+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.474+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 1 = system:0 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.474+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 4 2024-03-30T15:00:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.474+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 0 1 2024-03-30T15:00:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 10 client.57389.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 2 2024-03-30T15:00:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 2 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d80053e0 osd=0 3 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 0 2 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 4 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 10 client.57389.objecter _send_command 2 2024-03-30T15:00:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.491+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d8007b00 con 0x7fb1d8004730 2024-03-30T15:00:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d8007b00 con 0x7fb1d8004730 2024-03-30T15:00:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 4 2024-03-30T15:00:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 0 2 2024-03-30T15:00:04.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:04.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 10 client.57389.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 1 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 4 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 10 client.57389.objecter _send_command 3 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb1d8002200 con 0x7fb1d80042f0 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.493+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d80042f0 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.494+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d8002200 con 0x7fb1d80042f0 2024-03-30T15:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.494+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.494+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 3 = system:0 2024-03-30T15:00:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.494+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 4 2024-03-30T15:00:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.494+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 1 3 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 10 client.57389.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 4 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 4 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8004180 osd=1 3 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 1 4 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 4 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 10 client.57389.objecter _send_command 4 2024-03-30T15:00:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.511+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d8001600 con 0x7fb1d80042f0 2024-03-30T15:00:04.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d8001600 con 0x7fb1d80042f0 2024-03-30T15:00:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 4 2024-03-30T15:00:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 1 4 2024-03-30T15:00:04.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 10 client.57389.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 5 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 5 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 2 5 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 4 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 10 client.57389.objecter _send_command 5 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.512+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb1d8008c80 con 0x7fb1d800d6e0 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d800d6e0 2024-03-30T15:00:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d8008c80 con 0x7fb1d800d6e0 2024-03-30T15:00:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 5 = system:0 2024-03-30T15:00:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 4 2024-03-30T15:00:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.513+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 2 5 2024-03-30T15:00:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 10 client.57389.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 6 2024-03-30T15:00:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 6 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8008480 osd=2 3 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 2 6 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 4 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 10 client.57389.objecter _send_command 6 2024-03-30T15:00:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.530+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d800eab0 con 0x7fb1d800d6e0 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d800eab0 con 0x7fb1d800d6e0 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 4 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 2 6 2024-03-30T15:00:04.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 10 client.57389.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 7 2024-03-30T15:00:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 7 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 3 7 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 4 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 10 client.57389.objecter _send_command 7 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.532+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb1d801a3c0 con 0x7fb1d80167c0 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d80167c0 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d801a3c0 con 0x7fb1d80167c0 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 7 = system:0 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 4 2024-03-30T15:00:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.533+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 3 7 2024-03-30T15:00:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 10 client.57389.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 8 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 8 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8008270 osd=3 3 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 3 8 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 4 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 10 client.57389.objecter _send_command 8 2024-03-30T15:00:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.550+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d800e590 con 0x7fb1d80167c0 2024-03-30T15:00:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.552+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d800e590 con 0x7fb1d80167c0 2024-03-30T15:00:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.552+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.552+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.552+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 4 2024-03-30T15:00:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.552+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 3 8 2024-03-30T15:00:04.554 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 10 client.57389.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 9 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 9 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 4 9 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 4 2024-03-30T15:00:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 10 client.57389.objecter _send_command 9 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb1d800e4e0 con 0x7fb1d800d240 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.553+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d800d240 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.554+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d800e4e0 con 0x7fb1d800d240 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.554+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.554+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 9 = system:0 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.554+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 4 2024-03-30T15:00:04.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.554+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 4 9 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 10 client.57389.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 10 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 10 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d800e340 osd=4 3 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 4 10 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 4 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 10 client.57389.objecter _send_command 10 2024-03-30T15:00:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.571+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d801ef70 con 0x7fb1d800d240 2024-03-30T15:00:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.573+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d801ef70 con 0x7fb1d800d240 2024-03-30T15:00:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.573+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.573+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.573+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 4 2024-03-30T15:00:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.573+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 4 10 2024-03-30T15:00:04.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 10 client.57389.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 11 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 11 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 5 11 2024-03-30T15:00:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 4 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 10 client.57389.objecter _send_command 11 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb1d8021210 con 0x7fb1d801f690 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.574+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d801f690 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.575+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d8021210 con 0x7fb1d801f690 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.575+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.575+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 11 = system:0 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.575+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 4 2024-03-30T15:00:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.575+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 5 11 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 10 client.57389.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 12 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 12 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d800e540 osd=5 3 2024-03-30T15:00:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 5 12 2024-03-30T15:00:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 4 2024-03-30T15:00:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 10 client.57389.objecter _send_command 12 2024-03-30T15:00:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.592+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d8021490 con 0x7fb1d801f690 2024-03-30T15:00:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d8021490 con 0x7fb1d801f690 2024-03-30T15:00:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 4 2024-03-30T15:00:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 5 12 2024-03-30T15:00:04.596 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 10 client.57389.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 13 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 13 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 6 13 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 4 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 10 client.57389.objecter _send_command 13 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb1d80276c0 con 0x7fb1d8021490 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.594+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d8021490 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.595+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d80276c0 con 0x7fb1d8021490 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.595+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.595+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 13 = system:0 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.595+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 4 2024-03-30T15:00:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.595+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 6 13 2024-03-30T15:00:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 10 client.57389.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 14 2024-03-30T15:00:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 14 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8021210 osd=6 3 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 6 14 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 4 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 10 client.57389.objecter _send_command 14 2024-03-30T15:00:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.613+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d8029070 con 0x7fb1d8021490 2024-03-30T15:00:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d8029070 con 0x7fb1d8021490 2024-03-30T15:00:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 4 2024-03-30T15:00:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 6 14 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 10 client.57389.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 15 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 15 2024-03-30T15:00:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 7 15 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 4 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 10 client.57389.objecter _send_command 15 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.615+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb1d8029c60 con 0x7fb1d8029790 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.616+0000 7fb1ea7fc700 10 client.57389.objecter ms_handle_connect 0x7fb1d8029790 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.617+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb1d8029c60 con 0x7fb1d8029790 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.617+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.617+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 15 = system:0 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.617+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 4 2024-03-30T15:00:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.617+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 7 15 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 10 client.57389.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign -1 16 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 20 client.57389.objecter _get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_remove -1 16 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 20 client.57389.objecter get_session s=0x7fb1d8029070 osd=7 3 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 15 client.57389.objecter _session_command_op_assign 7 16 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 4 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 10 client.57389.objecter _send_command 16 2024-03-30T15:00:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.635+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb1d8029c60 con 0x7fb1d8029790 2024-03-30T15:00:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.636+0000 7fb1ea7fc700 1 -- 172.21.15.28:0/2919604687 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb1d8029c60 con 0x7fb1d8029790 2024-03-30T15:00:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.636+0000 7fb1ea7fc700 10 client.57389.objecter ms_dispatch 0x7fb1f4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.636+0000 7fb1ea7fc700 10 client.57389.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.636+0000 7fb1ea7fc700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 4 2024-03-30T15:00:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.636+0000 7fb1ea7fc700 15 client.57389.objecter _session_command_op_remove 7 16 2024-03-30T15:00:04.638 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 librados: watch_flush enter 2024-03-30T15:00:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 librados: watch_flush exit 2024-03-30T15:00:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.0 2024-03-30T15:00:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6801/3394323575 conn(0x7fb1d8004730 legacy=0x7fb1d8002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d80053e0 osd=0 1 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.1 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6805/1850313547 conn(0x7fb1d80042f0 legacy=0x7fb1d800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8004180 osd=1 1 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.2 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6809/3451325043 conn(0x7fb1d800d6e0 legacy=0x7fb1d800db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008480 osd=2 1 2024-03-30T15:00:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.3 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6813/1506157250 conn(0x7fb1d80167c0 legacy=0x7fb1d8018bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8008270 osd=3 1 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.4 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.134:6800/3063775514 conn(0x7fb1d800d240 legacy=0x7fb1d801a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e340 osd=4 1 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.5 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.134:6804/3668151179 conn(0x7fb1d801f690 legacy=0x7fb1d801faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d800e540 osd=5 1 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.6 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.134:6808/2135224194 conn(0x7fb1d8021490 legacy=0x7fb1d8025e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8021210 osd=6 1 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter close_session for osd.7 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.134:6812/3758190298 conn(0x7fb1d8029790 legacy=0x7fb1d80278a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1f98ef700 1 -- 172.21.15.28:0/2919604687 reap_dead start 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter put_session s=0x7fb1d8029070 osd=7 1 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 20 client.57389.objecter shutdown clearing up homeless session... 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 10 client.57389.objecter successfully canceled tick 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb1e0070060 legacy=0x7fb1e0072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1f98ef700 1 -- 172.21.15.28:0/2919604687 reap_dead start 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> v1:172.21.15.28:6790/0 conn(0x7fb1f4101130 legacy=0x7fb1f4104640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1f98ef700 1 -- 172.21.15.28:0/2919604687 reap_dead start 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1f98ef700 1 -- 172.21.15.28:0/2919604687 reap_dead start 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1f98ef700 1 -- 172.21.15.28:0/2919604687 reap_dead start 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 shutdown_connections 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 >> 172.21.15.28:0/2919604687 conn(0x7fb1f40fc9e0 msgr2=0x7fb1f4110e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 shutdown_connections 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 -- 172.21.15.28:0/2919604687 wait complete. 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.637+0000 7fb1fb352700 1 librados: shutdown 2024-03-30T15:00:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:04.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.676+0000 7f544b6e8700 1 -- 172.21.15.28:0/1807169454 >> v1:172.21.15.28:6789/0 conn(0x7f544410eb10 legacy=0x7f5444110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.676+0000 7f544b6e8700 1 -- 172.21.15.28:0/1807169454 shutdown_connections 2024-03-30T15:00:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.676+0000 7f544b6e8700 1 -- 172.21.15.28:0/1807169454 >> 172.21.15.28:0/1807169454 conn(0x7f54440fc9e0 msgr2=0x7f54440fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.676+0000 7f544b6e8700 1 -- 172.21.15.28:0/1807169454 shutdown_connections 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.676+0000 7f544b6e8700 1 -- 172.21.15.28:0/1807169454 wait complete. 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 librados: starting msgr at 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 librados: starting objecter 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 Processor -- start 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 -- start start 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 librados: setting wanted keys 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 librados: calling monclient init 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f5449484700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54441031f0 0x7f544410a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46052/0 (socket says 172.21.15.28:46052) 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f5449484700 1 -- 172.21.15.28:0/609619745 learned_addr learned my addr 172.21.15.28:0/609619745 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54441a7170 con 0x7f54441031f0 2024-03-30T15:00:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54441a8390 con 0x7f544410b000 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54441a95d0 con 0x7f544410eb10 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter ms_handle_connect 0x7f54441031f0 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter ms_handle_connect 0x7f544410eb10 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter ms_handle_connect 0x7f544410b000 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742512775 0 0) 0x7f54441a7170 con 0x7f54441031f0 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54300031f0 con 0x7f54441031f0 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2459921767 0 0) 0x7f54441a95d0 con 0x7f544410eb10 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.677+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54441a7170 con 0x7f544410eb10 2024-03-30T15:00:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169108563 0 0) 0x7f54441a8390 con 0x7f544410b000 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54441a95d0 con 0x7f544410b000 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2804013338 0 0) 0x7f54300031f0 con 0x7f54441031f0 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54441a8390 con 0x7f54441031f0 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5440001a30 con 0x7f54441031f0 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 979823763 0 0) 0x7f54441a7170 con 0x7f544410eb10 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54300031f0 con 0x7f544410eb10 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f543c001c50 con 0x7f544410eb10 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 170884400 0 0) 0x7f54441a95d0 con 0x7f544410b000 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54441a7170 con 0x7f544410b000 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1985443141 0 0) 0x7f54441a8390 con 0x7f54441031f0 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 >> v1:172.21.15.28:6790/0 conn(0x7f544410eb10 legacy=0x7f54441a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 >> v1:172.21.15.134:6789/0 conn(0x7f544410b000 legacy=0x7f54441a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54441aa810 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54441a9800 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f544b6e8700 10 client.24240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f54441a9cc0 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f544b6e8700 1 librados: init done 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5440001c90 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.678+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5440005c50 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.679+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5428005120 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.680+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f54400039b0 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.680+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(240..240 src has 53..240) ==== 10019+0+0 (unknown 352102238 0 0) 0x7f5440089c20 con 0x7f54441031f0 2024-03-30T15:00:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.680+0000 7f543a7fc700 10 client.24240.objecter ms_dispatch 0x7f5444001550 osd_map(240..240 src has 53..240) 2024-03-30T15:00:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.680+0000 7f543a7fc700 3 client.24240.objecter handle_osd_map got epochs [240,240] > 0 2024-03-30T15:00:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.680+0000 7f543a7fc700 3 client.24240.objecter handle_osd_map decoding full epoch 240 2024-03-30T15:00:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.681+0000 7f543a7fc700 20 client.24240.objecter dump_active .. 0 homeless 2024-03-30T15:00:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.681+0000 7f543a7fc700 10 client.24240.objecter ms_handle_connect 0x7f54300704d0 2024-03-30T15:00:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.685+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f54400566f0 con 0x7f54441031f0 2024-03-30T15:00:04.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:04.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:04.907+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5428005410 con 0x7f54441031f0 2024-03-30T15:00:04.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:00:04.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:00:04.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: pgmap v299: 292 pgs: 8 creating+peering, 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 278 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 17.12 scrub starts 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: 17.12 scrub ok 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1394453650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.57383 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:04.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2919604687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: pgmap v299: 292 pgs: 8 creating+peering, 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 278 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 17.12 scrub starts 2024-03-30T15:00:04.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: 17.12 scrub ok 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1394453650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.57383 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:04.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2919604687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:04.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:05.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:05.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:05.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:05.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: pgmap v299: 292 pgs: 8 creating+peering, 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 278 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 17.12 scrub starts 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: 17.12 scrub ok 2024-03-30T15:00:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi028-92979-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-21"}]': finished 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: osdmap e240: 8 total, 8 up, 8 in 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1394453650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.57383 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2919604687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:05.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.200+0000 7f543a7fc700 1 -- 172.21.15.28:0/609619745 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v241) ==== 87+0+0 (unknown 4178067545 0 0) 0x7f544005a140 con 0x7f54441031f0 2024-03-30T15:00:05.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.203+0000 7f544b6e8700 10 librados: watch_flush enter 2024-03-30T15:00:05.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.204+0000 7f544b6e8700 10 librados: watch_flush exit 2024-03-30T15:00:05.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.204+0000 7f544b6e8700 20 client.24240.objecter shutdown clearing up homeless session... 2024-03-30T15:00:05.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.204+0000 7f544b6e8700 10 client.24240.objecter successfully canceled tick 2024-03-30T15:00:05.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.205+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 >> v1:172.21.15.28:6800/3011431093 conn(0x7f54300704d0 legacy=0x7f5430072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:05.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.205+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 >> v1:172.21.15.28:6789/0 conn(0x7f54441031f0 legacy=0x7f544410a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:05.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.206+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 shutdown_connections 2024-03-30T15:00:05.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.206+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 >> 172.21.15.28:0/609619745 conn(0x7f54440fc9e0 msgr2=0x7f54440fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:05.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.206+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 shutdown_connections 2024-03-30T15:00:05.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.206+0000 7f544b6e8700 1 -- 172.21.15.28:0/609619745 wait complete. 2024-03-30T15:00:05.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:05.207+0000 7f544b6e8700 1 librados: shutdown 2024-03-30T15:00:05.211 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:00:05.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:05.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:05.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:05.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:05.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:05.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:05.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:05.672 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:00:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:00:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:00:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:00:05.672 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:00:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:05.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:05.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:05.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: 101.0 scrub starts 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: 101.0 scrub ok 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:05.862 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: from='client.57383 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[41614]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: 101.0 scrub starts 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: 101.0 scrub ok 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: from='client.57383 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:05.863 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:05.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:05 smithi028 ceph-mon[29876]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T15:00:05.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:05.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:05.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:06.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:06.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:06.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: 101.0 scrub starts 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: 101.0 scrub ok 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: from='client.57383 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/609619745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:05 smithi134 ceph-mon[33714]: osdmap e241: 8 total, 8 up, 8 in 2024-03-30T15:00:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:06.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:06.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:06.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:06.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:06.537 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:06.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:06.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:06.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:06.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:06.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:06.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:06.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:06.962 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:06.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:07.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: pgmap v302: 300 pgs: 2 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1223375427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='client.49435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:06 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:07.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: pgmap v302: 300 pgs: 2 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1223375427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='client.49435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: pgmap v302: 300 pgs: 2 active+clean+snaptrim, 40 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: osdmap e242: 8 total, 8 up, 8 in 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1223375427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='client.49435 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:07.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:06 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:07.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:07.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:07.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:00:07.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:00:07.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:07.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:07.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:07.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:07.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:07.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:07.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:07.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:07.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:07.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.49435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T15:00:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:07 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:08.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:08.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:08.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.49435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.49435 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: osdmap e243: 8 total, 8 up, 8 in 2024-03-30T15:00:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2501446069' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:07 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]: dispatch 2024-03-30T15:00:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:08.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:08.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:08.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:08.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:08.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:00:08.541 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:08.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:08.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:08.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:08.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:08.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: pgmap v305: 356 pgs: 2 active+clean+snaptrim, 96 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3034013289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.57407 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: pgmap v305: 356 pgs: 2 active+clean+snaptrim, 96 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3034013289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.57407 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:08.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:08 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:08.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:08.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 -- 172.21.15.28:0/3364027164 >> v1:172.21.15.28:6790/0 conn(0x7f1b34109dc0 legacy=0x7f1b3410c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 -- 172.21.15.28:0/3364027164 shutdown_connections 2024-03-30T15:00:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 -- 172.21.15.28:0/3364027164 >> 172.21.15.28:0/3364027164 conn(0x7f1b340fc9e0 msgr2=0x7f1b340fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 -- 172.21.15.28:0/3364027164 shutdown_connections 2024-03-30T15:00:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 -- 172.21.15.28:0/3364027164 wait complete. 2024-03-30T15:00:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 librados: starting msgr at 2024-03-30T15:00:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.880+0000 7f1b38a77700 1 librados: starting objecter 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 Processor -- start 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 -- start start 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 librados: setting wanted keys 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 librados: calling monclient init 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b3406b1c0 con 0x7f1b3410d960 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b3406b390 con 0x7f1b34109dc0 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b38a77700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b341a95b0 con 0x7f1b34101130 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b3259c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1b34101130 0x7f1b34105670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49696/0 (socket says 172.21.15.28:49696) 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b3259c700 1 -- 172.21.15.28:0/2759587358 learned_addr learned my addr 172.21.15.28:0/2759587358 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b32d9d700 1 --1- 172.21.15.28:0/2759587358 >> v1:172.21.15.28:6789/0 conn(0x7f1b3410d960 0x7f1b341a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38330/0 (socket says 172.21.15.28:38330) 2024-03-30T15:00:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter ms_handle_connect 0x7f1b34101130 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter ms_handle_connect 0x7f1b3410d960 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter ms_handle_connect 0x7f1b34109dc0 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.881+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036577431 0 0) 0x7f1b3406b390 con 0x7f1b34109dc0 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b1c0039c0 con 0x7f1b34109dc0 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110232305 0 0) 0x7f1b3406b1c0 con 0x7f1b3410d960 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b3406b390 con 0x7f1b3410d960 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784553032 0 0) 0x7f1b341a95b0 con 0x7f1b34101130 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b3406b1c0 con 0x7f1b34101130 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 241839246 0 0) 0x7f1b1c0039c0 con 0x7f1b34109dc0 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b341a95b0 con 0x7f1b34109dc0 2024-03-30T15:00:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b28004580 con 0x7f1b34109dc0 2024-03-30T15:00:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 21480560 0 0) 0x7f1b3406b390 con 0x7f1b3410d960 2024-03-30T15:00:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b1c0039c0 con 0x7f1b3410d960 2024-03-30T15:00:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3879052568 0 0) 0x7f1b3406b1c0 con 0x7f1b34101130 2024-03-30T15:00:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b3406b390 con 0x7f1b34101130 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b2c0015b0 con 0x7f1b3410d960 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b240017f0 con 0x7f1b34101130 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4178216408 0 0) 0x7f1b341a95b0 con 0x7f1b34109dc0 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 >> v1:172.21.15.28:6790/0 conn(0x7f1b34101130 legacy=0x7f1b34105670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 >> v1:172.21.15.28:6789/0 conn(0x7f1b3410d960 legacy=0x7f1b341a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b341aa7f0 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b341a97e0 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b38a77700 10 client.49453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b341a9ca0 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b38a77700 1 librados: init done 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1b28001030 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.882+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b28004b60 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.883+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b14005120 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.884+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1b2801b200 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.884+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(244..244 src has 53..244) ==== 9956+0+0 (unknown 2255683959 0 0) 0x7f1b28088ad0 con 0x7f1b34109dc0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.884+0000 7f1b237fe700 10 client.49453.objecter ms_dispatch 0x7f1b34001550 osd_map(244..244 src has 53..244) 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.884+0000 7f1b237fe700 3 client.49453.objecter handle_osd_map got epochs [244,244] > 0 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.884+0000 7f1b237fe700 3 client.49453.objecter handle_osd_map decoding full epoch 244 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.885+0000 7f1b237fe700 20 client.49453.objecter dump_active .. 0 homeless 2024-03-30T15:00:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.885+0000 7f1b237fe700 10 client.49453.objecter ms_handle_connect 0x7f1b1c0705b0 2024-03-30T15:00:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:08.893+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1b280555c0 con 0x7f1b34109dc0 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: pgmap v305: 356 pgs: 2 active+clean+snaptrim, 96 unknown, 4 active+clean+snaptrim_wait, 254 active+clean; 585 KiB data, 542 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.57377 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi028-92979-32"}]': finished 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: osdmap e244: 8 total, 8 up, 8 in 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3034013289' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.57407 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:08 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:08.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:09.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:09.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.115+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f1b14005410 con 0x7f1b34109dc0 2024-03-30T15:00:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:09.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.218+0000 7f1b237fe700 1 -- 172.21.15.28:0/2759587358 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v245) ==== 113+0+0 (unknown 3813719816 0 0) 0x7f1b28059010 con 0x7f1b34109dc0 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.224+0000 7f1b38a77700 10 librados: watch_flush enter 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.224+0000 7f1b38a77700 10 librados: watch_flush exit 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.224+0000 7f1b38a77700 20 client.49453.objecter shutdown clearing up homeless session... 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.224+0000 7f1b38a77700 10 client.49453.objecter successfully canceled tick 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.225+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1b1c0705b0 legacy=0x7f1b1c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.225+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 >> v1:172.21.15.134:6789/0 conn(0x7f1b34109dc0 legacy=0x7f1b34105d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.225+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 shutdown_connections 2024-03-30T15:00:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.225+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 >> 172.21.15.28:0/2759587358 conn(0x7f1b340fc9e0 msgr2=0x7f1b34110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:09.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.227+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 shutdown_connections 2024-03-30T15:00:09.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.227+0000 7f1b38a77700 1 -- 172.21.15.28:0/2759587358 wait complete. 2024-03-30T15:00:09.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:09.228+0000 7f1b38a77700 1 librados: shutdown 2024-03-30T15:00:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T15:00:09.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:09.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:09.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:09.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:09.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:09.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:09.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:09.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:09.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:09.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:09.892 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:09.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:09.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:09.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2759587358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:09.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:09.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.57407 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:09.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:09 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:09.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:10.016 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:10.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:10.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2759587358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.57407 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:10.179 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2759587358' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.57407 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi028-92999-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi028-92979-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: osdmap e245: 8 total, 8 up, 8 in 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:10.180 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:10.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:09 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:10.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:10.371 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:10.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:10.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:10.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:10.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:10.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:10.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:10.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:10.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:10.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-03-30T15:00:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:10.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:10.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:11.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:11.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:11.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:11.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: pgmap v308: 388 pgs: 32 creating+peering, 356 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T15:00:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: pgmap v308: 388 pgs: 32 creating+peering, 356 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T15:00:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: pgmap v308: 388 pgs: 32 creating+peering, 356 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: osdmap e246: 8 total, 8 up, 8 in 2024-03-30T15:00:11.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]: dispatch 2024-03-30T15:00:11.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:11.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:11.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.411 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:11.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:11.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:11.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:11.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:11.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:11.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:11.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:11.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]': finished 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:11.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]': finished 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:11 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:12.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:12.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:12.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi028-92979-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-23", "mode": "writeback"}]': finished 2024-03-30T15:00:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: osdmap e247: 8 total, 8 up, 8 in 2024-03-30T15:00:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:11 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:12.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:12.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:12.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:12.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:12.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:12.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.244+0000 7f9a2d59a700 1 -- 172.21.15.28:0/2306280817 <== mon.1 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a200046d0 con 0x7f9a301152a0 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.245+0000 7f9a34fbf700 1 -- 172.21.15.28:0/2306280817 >> v1:172.21.15.134:6789/0 conn(0x7f9a301152a0 legacy=0x7f9a30117690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.245+0000 7f9a34fbf700 1 -- 172.21.15.28:0/2306280817 shutdown_connections 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.245+0000 7f9a34fbf700 1 -- 172.21.15.28:0/2306280817 >> 172.21.15.28:0/2306280817 conn(0x7f9a300fc9d0 msgr2=0x7f9a300fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.245+0000 7f9a34fbf700 1 -- 172.21.15.28:0/2306280817 shutdown_connections 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.245+0000 7f9a34fbf700 1 -- 172.21.15.28:0/2306280817 wait complete. 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 librados: starting msgr at 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 librados: starting objecter 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 Processor -- start 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 -- start start 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 librados: setting wanted keys 2024-03-30T15:00:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.246+0000 7f9a34fbf700 1 librados: calling monclient init 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a34fbf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a301ad7a0 con 0x7f9a30100f90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a34fbf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a301ae9c0 con 0x7f9a30106d90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a34fbf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a301afc00 con 0x7f9a301152a0 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a2dd9b700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9a30106d90 0x7f9a301a8500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:40300/0 (socket says 172.21.15.28:40300) 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a2dd9b700 1 -- 172.21.15.28:0/575262628 learned_addr learned my addr 172.21.15.28:0/575262628 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter ms_handle_connect 0x7f9a30106d90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter ms_handle_connect 0x7f9a30100f90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941074474 0 0) 0x7f9a301ae9c0 con 0x7f9a30106d90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a180035b0 con 0x7f9a30106d90 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter ms_handle_connect 0x7f9a301152a0 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.247+0000 7f9a1f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3799078062 0 0) 0x7f9a180035b0 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a301ae9c0 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a240018e0 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928042495 0 0) 0x7f9a301afc00 con 0x7f9a301152a0 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a180035b0 con 0x7f9a301152a0 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4266099534 0 0) 0x7f9a301ae9c0 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6790/0 conn(0x7f9a301152a0 legacy=0x7f9a301abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6789/0 conn(0x7f9a30100f90 legacy=0x7f9a30114b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a301b0e40 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a34fbf700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a301ad9d0 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a34fbf700 10 client.49465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9a24001c40 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a34fbf700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a301ade90 con 0x7f9a30106d90 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a34fbf700 1 librados: init done 2024-03-30T15:00:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.248+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a24005db0 con 0x7f9a30106d90 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.249+0000 7f9a34fbf700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9a3006ae30 con 0x7f9a30106d90 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9a240070d0 con 0x7f9a30106d90 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(248..248 src has 53..248) ==== 9748+0+0 (unknown 2167712372 0 0) 0x7f9a24089f50 con 0x7f9a30106d90 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 osd_map(248..248 src has 53..248) 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 3 client.49465.objecter handle_osd_map got epochs [248,248] > 0 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 3 client.49465.objecter handle_osd_map decoding full epoch 248 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 20 client.49465.objecter dump_active .. 0 homeless 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v248) ==== 54+0+16 (unknown 203594432 0 2420000192) 0x7f9a240520d0 con 0x7f9a30106d90 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.250+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a18070450 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 1 2024-03-30T15:00:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 1 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 0 1 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 4 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 1 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a14006d40 con 0x7f9a14002fd0 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a14002fd0 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.251+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14006d40 con 0x7f9a14002fd0 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.252+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.252+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 1 = system:0 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.252+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 4 2024-03-30T15:00:12.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.252+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 0 1 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 2 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 2 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a140028e0 osd=0 3 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 0 2 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 4 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 2 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.277+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a14001240 con 0x7f9a14002fd0 2024-03-30T15:00:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a14001240 con 0x7f9a14002fd0 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 4 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 0 2 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 1 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 4 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 3 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9a14008cc0 con 0x7f9a14008100 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.279+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a14008100 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.280+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14008cc0 con 0x7f9a14008100 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.280+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.280+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 3 = system:0 2024-03-30T15:00:12.580 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.280+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 4 2024-03-30T15:00:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.280+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 1 3 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 4 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 4 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14001f50 osd=1 3 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 1 4 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 4 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 4 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.298+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a140096e0 con 0x7f9a14008100 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a140096e0 con 0x7f9a14008100 2024-03-30T15:00:12.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 4 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 1 4 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 5 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 5 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 2 5 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 4 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 5 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9a14010b00 con 0x7f9a1400ce70 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.300+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a1400ce70 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.301+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14010b00 con 0x7f9a1400ce70 2024-03-30T15:00:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.301+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.301+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 5 = system:0 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.301+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 4 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.301+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 2 5 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 6 2024-03-30T15:00:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 6 2024-03-30T15:00:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14001d20 osd=2 3 2024-03-30T15:00:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 2 6 2024-03-30T15:00:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 4 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 6 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.318+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a140091c0 con 0x7f9a1400ce70 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.319+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a140091c0 con 0x7f9a1400ce70 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.319+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.319+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.319+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 4 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.319+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 2 6 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 7 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 7 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 3 7 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 4 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 7 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9a14011820 con 0x7f9a14010cb0 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.320+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a14010cb0 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.321+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14011820 con 0x7f9a14010cb0 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.321+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.321+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 7 = system:0 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.321+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 4 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.321+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 3 7 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 8 2024-03-30T15:00:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 8 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14009460 osd=3 3 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 3 8 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 4 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 8 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.338+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a140121d0 con 0x7f9a14010cb0 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:12.590 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a140121d0 con 0x7f9a14010cb0 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 4 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 3 8 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 9 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 9 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 4 9 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 4 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 9 2024-03-30T15:00:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9a1401a2a0 con 0x7f9a140166c0 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.340+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a140166c0 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.341+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a1401a2a0 con 0x7f9a140166c0 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.341+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.341+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 9 = system:0 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.341+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 4 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.341+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 4 9 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 10 2024-03-30T15:00:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 10 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14013350 osd=4 3 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 4 10 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 4 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 10 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.358+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a1401b600 con 0x7f9a140166c0 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a1401b600 con 0x7f9a140166c0 2024-03-30T15:00:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 4 2024-03-30T15:00:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 4 10 2024-03-30T15:00:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 11 2024-03-30T15:00:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 11 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 5 11 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 4 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 11 2024-03-30T15:00:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9a1401c9f0 con 0x7f9a1401b810 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.360+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a1401b810 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.361+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a1401c9f0 con 0x7f9a1401b810 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.361+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.361+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 11 = system:0 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.361+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 4 2024-03-30T15:00:12.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.361+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 5 11 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 12 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 12 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a1401b600 osd=5 3 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 5 12 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 4 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 12 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.378+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a1401c9f0 con 0x7f9a1401b810 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a1401c9f0 con 0x7f9a1401b810 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 4 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 5 12 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 13 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 13 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 6 13 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 4 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 13 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9a14022e70 con 0x7f9a1401f360 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.380+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a1401f360 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.381+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14022e70 con 0x7f9a1401f360 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.381+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.381+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 13 = system:0 2024-03-30T15:00:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.381+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 4 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.381+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 6 13 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 14 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 14 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a1401c9f0 osd=6 3 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 6 14 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 4 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 14 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.398+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a14022e70 con 0x7f9a1401f360 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a14022e70 con 0x7f9a1401f360 2024-03-30T15:00:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 4 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 6 14 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 15 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 15 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 7 15 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 4 2024-03-30T15:00:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 15 2024-03-30T15:00:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9a14027240 con 0x7f9a14023590 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.400+0000 7f9a1f7fe700 10 client.49465.objecter ms_handle_connect 0x7f9a14023590 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.401+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9a14027240 con 0x7f9a14023590 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.401+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.401+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 15 = system:0 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.401+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 4 2024-03-30T15:00:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.401+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 7 15 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 10 client.49465.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign -1 16 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 20 client.49465.objecter _get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_remove -1 16 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 20 client.49465.objecter get_session s=0x7f9a14022e70 osd=7 3 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 15 client.49465.objecter _session_command_op_assign 7 16 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 4 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 10 client.49465.objecter _send_command 16 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.418+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a14027240 con 0x7f9a14023590 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.419+0000 7f9a1f7fe700 1 -- 172.21.15.28:0/575262628 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9a14027240 con 0x7f9a14023590 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.419+0000 7f9a1f7fe700 10 client.49465.objecter ms_dispatch 0x7f9a30001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.419+0000 7f9a1f7fe700 10 client.49465.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.419+0000 7f9a1f7fe700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 4 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.419+0000 7f9a1f7fe700 15 client.49465.objecter _session_command_op_remove 7 16 2024-03-30T15:00:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 librados: watch_flush enter 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 librados: watch_flush exit 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.0 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6801/3394323575 conn(0x7f9a14002fd0 legacy=0x7f9a14005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a140028e0 osd=0 1 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.1 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6805/1850313547 conn(0x7f9a14008100 legacy=0x7f9a140085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001f50 osd=1 1 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.2 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6809/3451325043 conn(0x7f9a1400ce70 legacy=0x7f9a1400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14001d20 osd=2 1 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.3 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6813/1506157250 conn(0x7f9a14010cb0 legacy=0x7f9a14011110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14009460 osd=3 1 2024-03-30T15:00:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.4 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.134:6800/3063775514 conn(0x7f9a140166c0 legacy=0x7f9a14018ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14013350 osd=4 1 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.5 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.134:6804/3668151179 conn(0x7f9a1401b810 legacy=0x7f9a14011a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401b600 osd=5 1 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.6 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.134:6808/2135224194 conn(0x7f9a1401f360 legacy=0x7f9a14021750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a1401c9f0 osd=6 1 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter close_session for osd.7 2024-03-30T15:00:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.134:6812/3758190298 conn(0x7f9a14023590 legacy=0x7f9a140259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter put_session s=0x7f9a14022e70 osd=7 1 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 20 client.49465.objecter shutdown clearing up homeless session... 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 10 client.49465.objecter successfully canceled tick 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9a18070450 legacy=0x7f9a18072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a2ed9d700 1 -- 172.21.15.28:0/575262628 reap_dead start 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> v1:172.21.15.134:6789/0 conn(0x7f9a30106d90 legacy=0x7f9a301a8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 shutdown_connections 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 >> 172.21.15.28:0/575262628 conn(0x7f9a300fc9d0 msgr2=0x7f9a300fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 shutdown_connections 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 -- 172.21.15.28:0/575262628 wait complete. 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:12.420+0000 7f9a1d7fa700 1 librados: shutdown 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:12.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:12.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:12.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:12.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:12.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:12.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:12.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: pgmap v311: 332 pgs: 8 unknown, 324 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/575262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: pgmap v311: 332 pgs: 8 unknown, 324 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T15:00:12.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/575262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:13.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:13.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:13.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: pgmap v311: 332 pgs: 8 unknown, 324 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: osdmap e248: 8 total, 8 up, 8 in 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/575262628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:13.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:13.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:13.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:13.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:13.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:13.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:13.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:13.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:13.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:13.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:13.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:13.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:14.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:14.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:14.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:14.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T15:00:14.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:13 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: osdmap e249: 8 total, 8 up, 8 in 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]: dispatch 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:13 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:14.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:14.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:14.310 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:00:14.310 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:00:14.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:14.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:14.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:14.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:14.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:14.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:14.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:14.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:14.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:14.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: pgmap v314: 324 pgs: 19 creating+peering, 13 unknown, 292 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: pgmap v314: 324 pgs: 19 creating+peering, 13 unknown, 292 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T15:00:14.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:14 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:14.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:14.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:00:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:14.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:15.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:15.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:15.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: pgmap v314: 324 pgs: 19 creating+peering, 13 unknown, 292 active+clean; 586 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-23"}]': finished 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3030476693' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1989601426' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: osdmap e250: 8 total, 8 up, 8 in 2024-03-30T15:00:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:14 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]: dispatch 2024-03-30T15:00:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:15.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:15.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:15.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:15.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:15.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:15.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:00:15.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:15.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:15.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:15.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:15.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:15.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:15.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:16.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:16.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:16.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3908620917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:16.090 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3908620917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.091 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:15 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.49447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi028-92979-33"}]': finished 2024-03-30T15:00:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: osdmap e251: 8 total, 8 up, 8 in 2024-03-30T15:00:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3908620917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:15 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:16.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:16.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:16.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:16.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:16.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:16.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:16.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:16.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: pgmap v317: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: pgmap v317: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:16.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:16 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:16.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:16.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:16.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:16.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:17.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:17.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: pgmap v317: 324 pgs: 32 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi028-92979-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: osdmap e252: 8 total, 8 up, 8 in 2024-03-30T15:00:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:16 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:17.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:17.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:17.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:17.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:17.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:17.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:17.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:17.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.687+0000 7ff17b35a700 1 -- 172.21.15.28:0/402726091 >> v1:172.21.15.28:6790/0 conn(0x7ff17410ea70 legacy=0x7ff174110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.687+0000 7ff17b35a700 1 -- 172.21.15.28:0/402726091 shutdown_connections 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.687+0000 7ff17b35a700 1 -- 172.21.15.28:0/402726091 >> 172.21.15.28:0/402726091 conn(0x7ff174077370 msgr2=0x7ff1740fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.687+0000 7ff17b35a700 1 -- 172.21.15.28:0/402726091 shutdown_connections 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.687+0000 7ff17b35a700 1 -- 172.21.15.28:0/402726091 wait complete. 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 librados: starting msgr at 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 librados: starting objecter 2024-03-30T15:00:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 Processor -- start 2024-03-30T15:00:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 -- start start 2024-03-30T15:00:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 librados: setting wanted keys 2024-03-30T15:00:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 librados: calling monclient init 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1741078e0 con 0x7ff17410ea70 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff174107af0 con 0x7ff17410af00 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff17b35a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1741a9690 con 0x7ff174100780 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff1790f6700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff174100780 0x7ff174104730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49822/0 (socket says 172.21.15.28:49822) 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff1790f6700 1 -- 172.21.15.28:0/2556252980 learned_addr learned my addr 172.21.15.28:0/2556252980 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff16a7fc700 10 client.?.objecter ms_handle_connect 0x7ff174100780 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.688+0000 7ff16a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 10 client.?.objecter ms_handle_connect 0x7ff17410ea70 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 10 client.?.objecter ms_handle_connect 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508799465 0 0) 0x7ff174107af0 con 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1600031f0 con 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130258390 0 0) 0x7ff1741078e0 con 0x7ff17410ea70 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff174107af0 con 0x7ff17410ea70 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101000261 0 0) 0x7ff1741a9690 con 0x7ff174100780 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1741078e0 con 0x7ff174100780 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3971667923 0 0) 0x7ff174107af0 con 0x7ff17410ea70 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1741a9690 con 0x7ff17410ea70 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3991506696 0 0) 0x7ff1600031f0 con 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff174107af0 con 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff16c0047f0 con 0x7ff17410ea70 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff170001250 con 0x7ff17410af00 2024-03-30T15:00:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4108140207 0 0) 0x7ff1741078e0 con 0x7ff174100780 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1600031f0 con 0x7ff174100780 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff164001870 con 0x7ff174100780 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 353650987 0 0) 0x7ff174107af0 con 0x7ff17410af00 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 >> v1:172.21.15.28:6790/0 conn(0x7ff174100780 legacy=0x7ff174104730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 >> v1:172.21.15.28:6789/0 conn(0x7ff17410ea70 legacy=0x7ff1741a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.689+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1741a7670 con 0x7ff17410af00 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1741a7960 con 0x7ff17410af00 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff17b35a700 10 client.49489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1741a7e20 con 0x7ff17410af00 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff17b35a700 1 librados: init done 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff170003b20 con 0x7ff17410af00 2024-03-30T15:00:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff170004a20 con 0x7ff17410af00 2024-03-30T15:00:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.690+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff158005120 con 0x7ff17410af00 2024-03-30T15:00:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.691+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff170003680 con 0x7ff17410af00 2024-03-30T15:00:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(253..253 src has 53..253) ==== 10392+0+0 (unknown 1605381990 0 0) 0x7ff170088dc0 con 0x7ff17410af00 2024-03-30T15:00:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 10 client.49489.objecter ms_dispatch 0x7ff174001550 osd_map(253..253 src has 53..253) 2024-03-30T15:00:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 3 client.49489.objecter handle_osd_map got epochs [253,253] > 0 2024-03-30T15:00:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 3 client.49489.objecter handle_osd_map decoding full epoch 253 2024-03-30T15:00:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 20 client.49489.objecter dump_active .. 0 homeless 2024-03-30T15:00:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.692+0000 7ff16a7fc700 10 client.49489.objecter ms_handle_connect 0x7ff1600703e0 2024-03-30T15:00:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.696+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff170055700 con 0x7ff17410af00 2024-03-30T15:00:17.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:17.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:17.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:17.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:17.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[41614]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T15:00:17.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:17.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:17.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:17.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:17.931+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff158005410 con 0x7ff17410af00 2024-03-30T15:00:17.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:18.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:18.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:18.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:17 smithi134 ceph-mon[33714]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T15:00:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.218+0000 7ff16a7fc700 1 -- 172.21.15.28:0/2556252980 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v254) ==== 97+0+0 (unknown 3111155299 0 0) 0x7ff170059150 con 0x7ff17410af00 2024-03-30T15:00:18.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:18.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:18.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[29876]: osdmap e253: 8 total, 8 up, 8 in 2024-03-30T15:00:18.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 10 librados: watch_flush enter 2024-03-30T15:00:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 10 librados: watch_flush exit 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 20 client.49489.objecter shutdown clearing up homeless session... 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 10 client.49489.objecter successfully canceled tick 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff1600703e0 legacy=0x7ff1600728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 >> v1:172.21.15.134:6789/0 conn(0x7ff17410af00 legacy=0x7ff174104e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 shutdown_connections 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 >> 172.21.15.28:0/2556252980 conn(0x7ff174077370 msgr2=0x7ff1740fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 shutdown_connections 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.222+0000 7ff17b35a700 1 -- 172.21.15.28:0/2556252980 wait complete. 2024-03-30T15:00:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:18.223+0000 7ff17b35a700 1 librados: shutdown 2024-03-30T15:00:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:00:18.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:18.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:18.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:18.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:18.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:18.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:18.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:18.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:18.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:18.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:18.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:18.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: pgmap v320: 388 pgs: 96 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:18.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2556252980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.49489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.49489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:18.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: pgmap v320: 388 pgs: 96 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2556252980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.49489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.49489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:18.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:19.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:19.060 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: pgmap v320: 388 pgs: 96 unknown, 1 active+clean+snaptrim, 32 creating+peering, 259 active+clean; 585 KiB data, 543 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2556252980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.49489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi028-92979-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1497402342' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.49489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: osdmap e254: 8 total, 8 up, 8 in 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:19.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:19.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:19.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:19.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:19.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:19.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:19.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- 172.21.15.28:0/215140840 >> v1:172.21.15.28:6789/0 conn(0x7ff26810ea80 legacy=0x7ff268110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- 172.21.15.28:0/215140840 shutdown_connections 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- 172.21.15.28:0/215140840 >> 172.21.15.28:0/215140840 conn(0x7ff2680fdb20 msgr2=0x7ff2680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- 172.21.15.28:0/215140840 shutdown_connections 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- 172.21.15.28:0/215140840 wait complete. 2024-03-30T15:00:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 librados: starting msgr at 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 librados: starting objecter 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 Processor -- start 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- start start 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 librados: setting wanted keys 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 librados: calling monclient init 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2681a9390 con 0x7ff26810aee0 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2681aa5b0 con 0x7ff268102250 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.489+0000 7ff26edba700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2681ab7f0 con 0x7ff26810ea80 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff267fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff26810aee0 0x7ff2681a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34956/0 (socket says 172.21.15.28:34956) 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff267fff700 1 -- 172.21.15.28:0/2559538859 learned_addr learned my addr 172.21.15.28:0/2559538859 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff26d357700 1 --1- 172.21.15.28:0/2559538859 >> v1:172.21.15.28:6790/0 conn(0x7ff26810ea80 0x7ff2681a7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47044/0 (socket says 172.21.15.28:47044) 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter ms_handle_connect 0x7ff26810ea80 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter ms_handle_connect 0x7ff26810aee0 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter ms_handle_connect 0x7ff268102250 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249079947 0 0) 0x7ff2681ab7f0 con 0x7ff26810ea80 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2580039c0 con 0x7ff26810ea80 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770604378 0 0) 0x7ff2681a9390 con 0x7ff26810aee0 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2681ab7f0 con 0x7ff26810aee0 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157573045 0 0) 0x7ff2681aa5b0 con 0x7ff268102250 2024-03-30T15:00:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2681a9390 con 0x7ff268102250 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3180881404 0 0) 0x7ff2580039c0 con 0x7ff26810ea80 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2681aa5b0 con 0x7ff26810ea80 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff260004780 con 0x7ff26810ea80 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1813458180 0 0) 0x7ff2681ab7f0 con 0x7ff26810aee0 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2580039c0 con 0x7ff26810aee0 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff254001960 con 0x7ff26810aee0 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 930927739 0 0) 0x7ff2681a9390 con 0x7ff268102250 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2681ab7f0 con 0x7ff268102250 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff25c001870 con 0x7ff268102250 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2869945150 0 0) 0x7ff2681aa5b0 con 0x7ff26810ea80 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 >> v1:172.21.15.134:6789/0 conn(0x7ff268102250 legacy=0x7ff268107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 >> v1:172.21.15.28:6789/0 conn(0x7ff26810aee0 legacy=0x7ff2681a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2681aca30 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.490+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2681a95c0 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff260001980 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff260005420 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff26edba700 10 client.57464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2681a9a80 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff26edba700 1 librados: init done 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.491+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff268103730 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.492+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff260005400 con 0x7ff26810ea80 2024-03-30T15:00:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(255..255 src has 53..255) ==== 11191+0+0 (unknown 3293668596 0 0) 0x7ff26008a630 con 0x7ff26810ea80 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 10 client.57464.objecter ms_dispatch 0x7ff268001550 osd_map(255..255 src has 53..255) 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 3 client.57464.objecter handle_osd_map got epochs [255,255] > 0 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 3 client.57464.objecter handle_osd_map decoding full epoch 255 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 20 client.57464.objecter dump_active .. 0 homeless 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.493+0000 7ff265ffb700 10 client.57464.objecter ms_handle_connect 0x7ff2580704f0 2024-03-30T15:00:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.497+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff260056c70 con 0x7ff26810ea80 2024-03-30T15:00:19.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:19.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:19.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:19.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:19.714+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7ff268075a20 con 0x7ff26810ea80 2024-03-30T15:00:19.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:19.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:19.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:19.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:19.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2559538859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:19 smithi134 ceph-mon[33714]: from='client.57464 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:20.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:20.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:20.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:20.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:20.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:20.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T15:00:20.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2559538859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[41614]: from='client.57464 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: osdmap e255: 8 total, 8 up, 8 in 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:20.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2559538859' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:20.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:19 smithi028 ceph-mon[29876]: from='client.57464 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:00:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:20.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.226+0000 7ff265ffb700 1 -- 172.21.15.28:0/2559538859 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v256) ==== 113+0+0 (unknown 1149068473 0 0) 0x7ff26005a6c0 con 0x7ff26810ea80 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.231+0000 7ff26edba700 10 librados: watch_flush enter 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 10 librados: watch_flush exit 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 20 client.57464.objecter shutdown clearing up homeless session... 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 10 client.57464.objecter successfully canceled tick 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff2580704f0 legacy=0x7ff2580729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 >> v1:172.21.15.28:6790/0 conn(0x7ff26810ea80 legacy=0x7ff2681a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 shutdown_connections 2024-03-30T15:00:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 >> 172.21.15.28:0/2559538859 conn(0x7ff2680fdb20 msgr2=0x7ff268111f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 shutdown_connections 2024-03-30T15:00:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 -- 172.21.15.28:0/2559538859 wait complete. 2024-03-30T15:00:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:20.232+0000 7ff26edba700 1 librados: shutdown 2024-03-30T15:00:20.235 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T15:00:20.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:20.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:20.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:20.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:20.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:20.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:20.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:20.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:20.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:20.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:20.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:20.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: pgmap v323: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.57464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:20.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: pgmap v323: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.57464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:20.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:20 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:20.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:20.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:20.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:00:21.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:21.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:21.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: pgmap v323: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2087737423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.57464 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: osdmap e256: 8 total, 8 up, 8 in 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]: dispatch 2024-03-30T15:00:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:20 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:21.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:21.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:21.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:21.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:21.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:21.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-03-30T15:00:21.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:21.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:21.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:21.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:21.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:21.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]': finished 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/434993017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.49507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:21.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]': finished 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/434993017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.49507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:21.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:21 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:21.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:22.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:22.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:22.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-25", "mode": "writeback"}]': finished 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: osdmap e257: 8 total, 8 up, 8 in 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3348620406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]: dispatch 2024-03-30T15:00:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/434993017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.49507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:22.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:21 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:22.217 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:00:22.217 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:00:22.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:22.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:22.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:22.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:22.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:22.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:22.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:00:22.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:22.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:22.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:23.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:23.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:23.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:23.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:23.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: pgmap v326: 452 pgs: 64 unknown, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.49507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.175 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: pgmap v326: 452 pgs: 64 unknown, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.49507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:22 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: pgmap v326: 452 pgs: 64 unknown, 388 active+clean; 586 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.57440 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi028-92979-34"}]': finished 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.49507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi028-92999-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: osdmap e258: 8 total, 8 up, 8 in 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:22 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:23.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:23.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:23.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:23.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:23.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:23.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:23.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:23.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:23.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:23.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:23.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:23.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:23.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:23.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:23.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:23.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T15:00:23.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:23.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:23 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:24.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:24.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-25"}]': finished 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi028-92979-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: osdmap e259: 8 total, 8 up, 8 in 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:23 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:24.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:24.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:24.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:24.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:24.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:24.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:24.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[41614]: pgmap v329: 420 pgs: 9 unknown, 411 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:24.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:24.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[41614]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T15:00:24.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/202140323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[29876]: pgmap v329: 420 pgs: 9 unknown, 411 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[29876]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T15:00:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/202140323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:24.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:24.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:25.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:25.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:25.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:24 smithi134 ceph-mon[33714]: pgmap v329: 420 pgs: 9 unknown, 411 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:24 smithi134 ceph-mon[33714]: osdmap e260: 8 total, 8 up, 8 in 2024-03-30T15:00:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/202140323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:25.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:25.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:25.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:25.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:25.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 -- 172.21.15.28:0/2292573817 >> v1:172.21.15.28:6789/0 conn(0x7f8cf010ea90 legacy=0x7f8cf0110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 -- 172.21.15.28:0/2292573817 shutdown_connections 2024-03-30T15:00:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 -- 172.21.15.28:0/2292573817 >> 172.21.15.28:0/2292573817 conn(0x7f8cf00fdb10 msgr2=0x7f8cf00fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 -- 172.21.15.28:0/2292573817 shutdown_connections 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 -- 172.21.15.28:0/2292573817 wait complete. 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 librados: starting msgr at 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.772+0000 7f8cf48ca700 1 librados: starting objecter 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 Processor -- start 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 -- start start 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 librados: setting wanted keys 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 librados: calling monclient init 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8cf01a9410 con 0x7f8cf010aef0 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8cf01aa630 con 0x7f8cf010ea90 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cf48ca700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8cf01ab870 con 0x7f8cf0102260 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cedd9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8cf010aef0 0x7f8cf01a42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35010/0 (socket says 172.21.15.28:35010) 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cee59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8cf0102260 0x7f8cf0107640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47092/0 (socket says 172.21.15.28:47092) 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cedd9b700 1 -- 172.21.15.28:0/202140323 learned_addr learned my addr 172.21.15.28:0/202140323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cf0102260 2024-03-30T15:00:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cf010aef0 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652980325 0 0) 0x7f8cf01ab870 con 0x7f8cf0102260 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cd80031f0 con 0x7f8cf0102260 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f8cf010ea90 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582273849 0 0) 0x7f8cf01a9410 con 0x7f8cf010aef0 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.773+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf01ab870 con 0x7f8cf010aef0 2024-03-30T15:00:25.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778483216 0 0) 0x7f8cf01aa630 con 0x7f8cf010ea90 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf01a9410 con 0x7f8cf010ea90 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 214349814 0 0) 0x7f8cd80031f0 con 0x7f8cf0102260 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cf01aa630 con 0x7f8cf0102260 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ce00018b0 con 0x7f8cf0102260 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3979701978 0 0) 0x7f8cf01ab870 con 0x7f8cf010aef0 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cd80031f0 con 0x7f8cf010aef0 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ce4001230 con 0x7f8cf010aef0 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2535730575 0 0) 0x7f8cf01aa630 con 0x7f8cf0102260 2024-03-30T15:00:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.134:6789/0 conn(0x7f8cf010ea90 legacy=0x7f8cf01a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6789/0 conn(0x7f8cf010aef0 legacy=0x7f8cf01a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf01acab0 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf01a9640 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cf48ca700 10 client.57482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8cf01a9b00 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cf48ca700 1 librados: init done 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8ce0003b40 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.774+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ce0005d10 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.775+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8cd0005130 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.776+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8ce001c550 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.776+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(260..260 src has 53..260) ==== 9980+0+0 (unknown 718420611 0 0) 0x7f8ce0089f10 con 0x7f8cf0102260 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.776+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 osd_map(260..260 src has 53..260) 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.776+0000 7f8cdf7fe700 3 client.57482.objecter handle_osd_map got epochs [260,260] > 0 2024-03-30T15:00:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.776+0000 7f8cdf7fe700 3 client.57482.objecter handle_osd_map decoding full epoch 260 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cdf7fe700 20 client.57482.objecter dump_active .. 0 homeless 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v260) ==== 54+0+16 (unknown 2605084944 0 2420000192) 0x7f8ce008a350 con 0x7f8cf0102260 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd8070340 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 1 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 1 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 0 1 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 4 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 10 client.57482.objecter _send_command 1 2024-03-30T15:00:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8cd0004b80 con 0x7f8cd0004760 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.777+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd0004760 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.778+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd0004b80 con 0x7f8cd0004760 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.778+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.778+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 1 = system:0 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.778+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 4 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.778+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 0 1 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 2 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 2 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0005410 osd=0 3 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 0 2 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 4 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 10 client.57482.objecter _send_command 2 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.806+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0007b30 con 0x7f8cd0004760 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0007b30 con 0x7f8cd0004760 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 4 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 0 2 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 3 2024-03-30T15:00:25.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 1 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 4 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 10 client.57482.objecter _send_command 3 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8cd0002190 con 0x7f8cd00042d0 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.808+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd00042d0 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.809+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd0002190 con 0x7f8cd00042d0 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.809+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:25.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.809+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 3 = system:0 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.809+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 4 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.809+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 1 3 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 4 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0004160 osd=1 3 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 1 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 10 client.57482.objecter _send_command 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.826+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0001630 con 0x7f8cd00042d0 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0001630 con 0x7f8cd00042d0 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 1 4 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 5 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 5 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 2 5 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 4 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 10 client.57482.objecter _send_command 5 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8cd0010fe0 con 0x7f8cd00089c0 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.828+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd00089c0 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.829+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd0010fe0 con 0x7f8cd00089c0 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.829+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.829+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 5 = system:0 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.829+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 4 2024-03-30T15:00:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.829+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 2 5 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 6 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 6 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0001a30 osd=2 3 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 2 6 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 4 2024-03-30T15:00:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 10 client.57482.objecter _send_command 6 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.845+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0008050 con 0x7f8cd00089c0 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0008050 con 0x7f8cd00089c0 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 4 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 2 6 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 7 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 7 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 3 7 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 4 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 10 client.57482.objecter _send_command 7 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.847+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8cd000d4c0 con 0x7f8cd00111c0 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd00111c0 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd000d4c0 con 0x7f8cd00111c0 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 7 = system:0 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 4 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.848+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 3 7 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 8 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 8 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0008090 osd=3 3 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 3 8 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 4 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 10 client.57482.objecter _send_command 8 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.865+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0012500 con 0x7f8cd00111c0 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0012500 con 0x7f8cd00111c0 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 4 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 3 8 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 9 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 9 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 4 9 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 4 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 10 client.57482.objecter _send_command 9 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.867+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8cd0013920 con 0x7f8cd0016ba0 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.868+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd0016ba0 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.869+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd0013920 con 0x7f8cd0016ba0 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.869+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.869+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 9 = system:0 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.869+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 4 2024-03-30T15:00:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.869+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 4 9 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 10 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 10 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0013680 osd=4 3 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 4 10 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 4 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 10 client.57482.objecter _send_command 10 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.885+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd001b830 con 0x7f8cd0016ba0 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.887+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd001b830 con 0x7f8cd0016ba0 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.887+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.887+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.887+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 4 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.887+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 4 10 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 11 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 11 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 5 11 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 4 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 10 client.57482.objecter _send_command 11 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8cd0013480 con 0x7f8cd001ba40 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.888+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd001ba40 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.889+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd0013480 con 0x7f8cd001ba40 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.889+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.889+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 11 = system:0 2024-03-30T15:00:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.889+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 4 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.889+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 5 11 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 12 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 12 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd001b830 osd=5 3 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 5 12 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 4 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 10 client.57482.objecter _send_command 12 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.905+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0013480 con 0x7f8cd001ba40 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0013480 con 0x7f8cd001ba40 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 4 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 5 12 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 13 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 13 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 6 13 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 4 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 10 client.57482.objecter _send_command 13 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8cd00230a0 con 0x7f8cd001f590 2024-03-30T15:00:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.908+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd001f590 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.909+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd00230a0 con 0x7f8cd001f590 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.909+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.909+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 13 = system:0 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.909+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 4 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.909+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 6 13 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 14 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 14 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd0013480 osd=6 3 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 6 14 2024-03-30T15:00:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 4 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 10 client.57482.objecter _send_command 14 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.928+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd00230a0 con 0x7f8cd001f590 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd00230a0 con 0x7f8cd001f590 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 4 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 6 14 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 15 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 15 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 7 15 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 4 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 10 client.57482.objecter _send_command 15 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.930+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8cd00273f0 con 0x7f8cd0023720 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.931+0000 7f8cdf7fe700 10 client.57482.objecter ms_handle_connect 0x7f8cd0023720 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.932+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8cd00273f0 con 0x7f8cd0023720 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.932+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.932+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 15 = system:0 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.932+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 4 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.932+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 7 15 2024-03-30T15:00:25.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 10 client.57482.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign -1 16 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 20 client.57482.objecter _get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_remove -1 16 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 20 client.57482.objecter get_session s=0x7f8cd00230a0 osd=7 3 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 15 client.57482.objecter _session_command_op_assign 7 16 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 4 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 10 client.57482.objecter _send_command 16 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.948+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8cd0028da0 con 0x7f8cd0023720 2024-03-30T15:00:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cdf7fe700 1 -- 172.21.15.28:0/202140323 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8cd0028da0 con 0x7f8cd0023720 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cdf7fe700 10 client.57482.objecter ms_dispatch 0x7f8cf0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cdf7fe700 10 client.57482.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cdf7fe700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 4 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cdf7fe700 15 client.57482.objecter _session_command_op_remove 7 16 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 librados: watch_flush enter 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 librados: watch_flush exit 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.0 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6801/3394323575 conn(0x7f8cd0004760 legacy=0x7f8cd0002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0005410 osd=0 1 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.1 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6805/1850313547 conn(0x7f8cd00042d0 legacy=0x7f8cd000ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0004160 osd=1 1 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.2 2024-03-30T15:00:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6809/3451325043 conn(0x7f8cd00089c0 legacy=0x7f8cd000f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0001a30 osd=2 1 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.3 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6813/1506157250 conn(0x7f8cd00111c0 legacy=0x7f8cd0011620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0008090 osd=3 1 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.4 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.134:6800/3063775514 conn(0x7f8cd0016ba0 legacy=0x7f8cd0018f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013680 osd=4 1 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.5 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.134:6804/3668151179 conn(0x7f8cd001ba40 legacy=0x7f8cd0011d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd001b830 osd=5 1 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.6 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.134:6808/2135224194 conn(0x7f8cd001f590 legacy=0x7f8cd0021980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd0013480 osd=6 1 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter close_session for osd.7 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.134:6812/3758190298 conn(0x7f8cd0023720 legacy=0x7f8cd0025bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter put_session s=0x7f8cd00230a0 osd=7 1 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 20 client.57482.objecter shutdown clearing up homeless session... 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 10 client.57482.objecter successfully canceled tick 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8cd8070340 legacy=0x7f8cd8072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8ceed9d700 1 -- 172.21.15.28:0/202140323 reap_dead start 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> v1:172.21.15.28:6790/0 conn(0x7f8cf0102260 legacy=0x7f8cf0107640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8ceed9d700 1 -- 172.21.15.28:0/202140323 reap_dead start 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8ceed9d700 1 -- 172.21.15.28:0/202140323 reap_dead start 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8ceed9d700 1 -- 172.21.15.28:0/202140323 reap_dead start 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8ceed9d700 1 -- 172.21.15.28:0/202140323 reap_dead start 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 shutdown_connections 2024-03-30T15:00:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 >> 172.21.15.28:0/202140323 conn(0x7f8cf00fdb10 msgr2=0x7f8cf00fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 shutdown_connections 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.951+0000 7f8cf48ca700 1 -- 172.21.15.28:0/202140323 wait complete. 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:24.952+0000 7f8cf48ca700 1 librados: shutdown 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:25.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:25.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:25.671 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:25.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:25.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:25.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:25.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:25.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:25.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:25.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:26.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[41614]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[29876]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:25 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:26.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:25 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi028-92979-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:25 smithi134 ceph-mon[33714]: osdmap e261: 8 total, 8 up, 8 in 2024-03-30T15:00:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:25 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:26.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:26.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:26.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:26.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:26.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:26.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:26.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:26.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:26.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:26.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:26.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:26.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:26.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:26.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:26.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[41614]: pgmap v332: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[41614]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T15:00:27.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[29876]: pgmap v332: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:27.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:27.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:27.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:27.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:26 smithi028 ceph-mon[29876]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T15:00:27.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:27.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:26 smithi134 ceph-mon[33714]: pgmap v332: 364 pgs: 40 unknown, 324 active+clean; 586 KiB data, 553 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:00:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:26 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:26 smithi134 ceph-mon[33714]: osdmap e262: 8 total, 8 up, 8 in 2024-03-30T15:00:27.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:27.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:27.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:27.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:27.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:27.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:27.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:27.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:27.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:27.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:27.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:27.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:27.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:28.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:28.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T15:00:28.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:28.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:28.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: osdmap e263: 8 total, 8 up, 8 in 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:28.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:28.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:28.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:28.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:28.414 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:28.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:28.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:28.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:00:28.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:28.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.723+0000 7f9831b47700 1 -- 172.21.15.28:0/4239947954 >> v1:172.21.15.28:6790/0 conn(0x7f982c102250 legacy=0x7f982c102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.723+0000 7f9831b47700 1 -- 172.21.15.28:0/4239947954 shutdown_connections 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.723+0000 7f9831b47700 1 -- 172.21.15.28:0/4239947954 >> 172.21.15.28:0/4239947954 conn(0x7f982c0fdb20 msgr2=0x7f982c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.723+0000 7f9831b47700 1 -- 172.21.15.28:0/4239947954 shutdown_connections 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.723+0000 7f9831b47700 1 -- 172.21.15.28:0/4239947954 wait complete. 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 librados: starting msgr at 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 librados: starting objecter 2024-03-30T15:00:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 Processor -- start 2024-03-30T15:00:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 -- start start 2024-03-30T15:00:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 librados: setting wanted keys 2024-03-30T15:00:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 librados: calling monclient init 2024-03-30T15:00:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f982c1a7160 con 0x7f982c102250 2024-03-30T15:00:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f982c1a8380 con 0x7f982c10aee0 2024-03-30T15:00:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f9831b47700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f982c1a95c0 con 0x7f982c10ea80 2024-03-30T15:00:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f982bfff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f982c10ea80 0x7f982c1a5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56578/0 (socket says 172.21.15.28:56578) 2024-03-30T15:00:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.724+0000 7f982bfff700 1 -- 172.21.15.28:0/3399691395 learned_addr learned my addr 172.21.15.28:0/3399691395 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter ms_handle_connect 0x7f982c102250 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter ms_handle_connect 0x7f982c10ea80 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter ms_handle_connect 0x7f982c10aee0 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973597962 0 0) 0x7f982c1a7160 con 0x7f982c102250 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98140031f0 con 0x7f982c102250 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314031346 0 0) 0x7f982c1a8380 con 0x7f982c10aee0 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f982c1a7160 con 0x7f982c10aee0 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296580196 0 0) 0x7f982c1a95c0 con 0x7f982c10ea80 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f982c1a8380 con 0x7f982c10ea80 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2736185594 0 0) 0x7f98140031f0 con 0x7f982c102250 2024-03-30T15:00:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f982c1a95c0 con 0x7f982c102250 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9820004500 con 0x7f982c102250 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4259055971 0 0) 0x7f982c1a7160 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98140031f0 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3874472842 0 0) 0x7f982c1a8380 con 0x7f982c10ea80 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f982c1a7160 con 0x7f982c10ea80 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f981c001d40 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9824001bb0 con 0x7f982c10ea80 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1458245727 0 0) 0x7f98140031f0 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 >> v1:172.21.15.28:6790/0 conn(0x7f982c10ea80 legacy=0x7f982c1a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 >> v1:172.21.15.28:6789/0 conn(0x7f982c102250 legacy=0x7f982c1075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c1aa800 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.725+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f982c1a97f0 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9831b47700 10 client.49528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f982c1a9cb0 con 0x7f982c10aee0 2024-03-30T15:00:28.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9831b47700 1 librados: init done 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f981c0014b0 con 0x7f982c10aee0 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f981c0048d0 con 0x7f982c10aee0 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.726+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9818005120 con 0x7f982c10aee0 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.727+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f981c003680 con 0x7f982c10aee0 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(264..264 src has 53..264) ==== 9283+0+0 (unknown 700245793 0 0) 0x7f981c088880 con 0x7f982c10aee0 2024-03-30T15:00:28.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 10 client.49528.objecter ms_dispatch 0x7f982c001550 osd_map(264..264 src has 53..264) 2024-03-30T15:00:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 3 client.49528.objecter handle_osd_map got epochs [264,264] > 0 2024-03-30T15:00:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 3 client.49528.objecter handle_osd_map decoding full epoch 264 2024-03-30T15:00:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 20 client.49528.objecter dump_active .. 0 homeless 2024-03-30T15:00:28.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.728+0000 7f9828ff9700 10 client.49528.objecter ms_handle_connect 0x7f9814070470 2024-03-30T15:00:28.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.732+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f981c055610 con 0x7f982c10aee0 2024-03-30T15:00:28.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:28.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:28.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:28.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:28.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:28.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: pgmap v335: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:28.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4018219597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[41614]: from='client.57488 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: pgmap v335: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:28.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4018219597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:28.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:28 smithi028 ceph-mon[29876]: from='client.57488 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:28.957+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f9818005410 con 0x7f982c10aee0 2024-03-30T15:00:29.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:29.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: pgmap v335: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: osdmap e264: 8 total, 8 up, 8 in 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4051625719' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4018219597' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:28 smithi134 ceph-mon[33714]: from='client.57488 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.231+0000 7f9828ff9700 1 -- 172.21.15.28:0/3399691395 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v265) ==== 91+0+0 (unknown 1503747274 0 0) 0x7f981c059060 con 0x7f982c10aee0 2024-03-30T15:00:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.234+0000 7f9831b47700 10 librados: watch_flush enter 2024-03-30T15:00:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.235+0000 7f9831b47700 10 librados: watch_flush exit 2024-03-30T15:00:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.235+0000 7f9831b47700 20 client.49528.objecter shutdown clearing up homeless session... 2024-03-30T15:00:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.235+0000 7f9831b47700 10 client.49528.objecter successfully canceled tick 2024-03-30T15:00:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.235+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9814070470 legacy=0x7f9814072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.235+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 >> v1:172.21.15.134:6789/0 conn(0x7f982c10aee0 legacy=0x7f982c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.236+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 shutdown_connections 2024-03-30T15:00:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.236+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 >> 172.21.15.28:0/3399691395 conn(0x7f982c0fdb20 msgr2=0x7f982c111c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.236+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 shutdown_connections 2024-03-30T15:00:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.236+0000 7f9831b47700 1 -- 172.21.15.28:0/3399691395 wait complete. 2024-03-30T15:00:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.236+0000 7f9831b47700 1 librados: shutdown 2024-03-30T15:00:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:00:29.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:29.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:29.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:29.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:29.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:29.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:29.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:29.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:29.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:29.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3399691395' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.49528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57488 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.49528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:29.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1641643656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[41614]: from='client.57500 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3399691395' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.49528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57488 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.49528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:29.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1641643656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:29.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:29 smithi028 ceph-mon[29876]: from='client.57500 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.174+0000 7ff4e008a700 1 -- 172.21.15.28:0/1047338117 >> v1:172.21.15.28:6790/0 conn(0x7ff4d8108110 legacy=0x7ff4d81084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.174+0000 7ff4e008a700 1 -- 172.21.15.28:0/1047338117 shutdown_connections 2024-03-30T15:00:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.174+0000 7ff4e008a700 1 -- 172.21.15.28:0/1047338117 >> 172.21.15.28:0/1047338117 conn(0x7ff4d80761d0 msgr2=0x7ff4d80765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.174+0000 7ff4e008a700 1 -- 172.21.15.28:0/1047338117 shutdown_connections 2024-03-30T15:00:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.174+0000 7ff4e008a700 1 -- 172.21.15.28:0/1047338117 wait complete. 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 librados: starting msgr at 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 librados: starting objecter 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 Processor -- start 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 -- start start 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 librados: setting wanted keys 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 librados: calling monclient init 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4d81078f0 con 0x7ff4d810ae80 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4d81a8690 con 0x7ff4d8108110 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4e008a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4d81a9870 con 0x7ff4d810ea80 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4dd625700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4d810ae80 0x7ff4d8104e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54866/0 (socket says 172.21.15.28:54866) 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4de627700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff4d810ea80 0x7ff4d81a5f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56588/0 (socket says 172.21.15.28:56588) 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4dd625700 1 -- 172.21.15.28:0/1641643656 learned_addr learned my addr 172.21.15.28:0/1641643656 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter ms_handle_connect 0x7ff4d810ea80 2024-03-30T15:00:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter ms_handle_connect 0x7ff4d810ae80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter ms_handle_connect 0x7ff4d8108110 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197758030 0 0) 0x7ff4d81a9870 con 0x7ff4d810ea80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c00031f0 con 0x7ff4d810ea80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651140013 0 0) 0x7ff4d81078f0 con 0x7ff4d810ae80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d81a9870 con 0x7ff4d810ae80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101281474 0 0) 0x7ff4d81a8690 con 0x7ff4d8108110 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.175+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d81078f0 con 0x7ff4d8108110 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 9022678 0 0) 0x7ff4c00031f0 con 0x7ff4d810ea80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4d81a8690 con 0x7ff4d810ea80 2024-03-30T15:00:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4d00017e0 con 0x7ff4d810ea80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3837750451 0 0) 0x7ff4d81a9870 con 0x7ff4d810ae80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4c00031f0 con 0x7ff4d810ae80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4cc0018b0 con 0x7ff4d810ae80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3149642821 0 0) 0x7ff4d81078f0 con 0x7ff4d8108110 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4d81a9870 con 0x7ff4d8108110 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4c80017b0 con 0x7ff4d8108110 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3736606696 0 0) 0x7ff4d81a8690 con 0x7ff4d810ea80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 >> v1:172.21.15.134:6789/0 conn(0x7ff4d8108110 legacy=0x7ff4d8104740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 >> v1:172.21.15.28:6789/0 conn(0x7ff4d810ae80 legacy=0x7ff4d8104e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d81aaa50 con 0x7ff4d810ea80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d81a76e0 con 0x7ff4d810ea80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4e008a700 10 client.57500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4d81a7ba0 con 0x7ff4d810ea80 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4e008a700 1 librados: init done 2024-03-30T15:00:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff4d0003b40 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.176+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4d0005bc0 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.177+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c4005120 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.177+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff4d00036a0 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(264..264 src has 53..264) ==== 9283+0+0 (unknown 700245793 0 0) 0x7ff4d0089910 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 10 client.57500.objecter ms_dispatch 0x7ff4d8001550 osd_map(264..264 src has 53..264) 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 3 client.57500.objecter handle_osd_map got epochs [264,264] > 0 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 3 client.57500.objecter handle_osd_map decoding full epoch 264 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 20 client.57500.objecter dump_active .. 0 homeless 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.178+0000 7ff4d6ffd700 10 client.57500.objecter ms_handle_connect 0x7ff4c0070530 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.182+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff4d00566c0 con 0x7ff4d810ea80 2024-03-30T15:00:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:29.426+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7ff4c4005410 con 0x7ff4d810ea80 2024-03-30T15:00:30.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:30.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:30.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:30.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3399691395' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.49528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57476 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi028-92979-35"}]': finished 2024-03-30T15:00:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57488 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi028-92999-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.49528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: osdmap e265: 8 total, 8 up, 8 in 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1641643656' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:29 smithi134 ceph-mon[33714]: from='client.57500 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:00:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.228+0000 7ff4d6ffd700 1 -- 172.21.15.28:0/1641643656 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v266) ==== 113+0+0 (unknown 202835233 0 0) 0x7ff4d005a110 con 0x7ff4d810ea80 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 10 librados: watch_flush enter 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 10 librados: watch_flush exit 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 20 client.57500.objecter shutdown clearing up homeless session... 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 10 client.57500.objecter successfully canceled tick 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff4c0070530 legacy=0x7ff4c00729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.232+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 >> v1:172.21.15.28:6790/0 conn(0x7ff4d810ea80 legacy=0x7ff4d81a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.233+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 shutdown_connections 2024-03-30T15:00:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.233+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 >> 172.21.15.28:0/1641643656 conn(0x7ff4d80761d0 msgr2=0x7ff4d8110f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.233+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 shutdown_connections 2024-03-30T15:00:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.233+0000 7ff4e008a700 1 -- 172.21.15.28:0/1641643656 wait complete. 2024-03-30T15:00:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:30.236+0000 7ff4e008a700 1 librados: shutdown 2024-03-30T15:00:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T15:00:30.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:30.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:30.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:30.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:30.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:30.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:30.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:30.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:30.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:30.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:30.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:31.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:31.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: 101.c scrub starts 2024-03-30T15:00:31.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: 101.c scrub ok 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: pgmap v338: 324 pgs: 32 creating+peering, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]': finished 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.57500 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T15:00:31.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: 101.c scrub starts 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: 101.c scrub ok 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: pgmap v338: 324 pgs: 32 creating+peering, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]': finished 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.57500 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T15:00:31.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:30 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T15:00:31.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: 101.c scrub starts 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: 101.c scrub ok 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: pgmap v338: 324 pgs: 32 creating+peering, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-27", "mode": "writeback"}]': finished 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi028-92979-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.57500 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: osdmap e266: 8 total, 8 up, 8 in 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T15:00:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:30 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T15:00:31.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:00:31.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:00:31.204 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:00:31.204 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:00:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:00:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:31.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:31.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:31.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:31.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:31.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:31.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: 101.b scrub starts 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: 101.b scrub ok 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:31.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: 101.b scrub starts 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: 101.b scrub ok 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:31.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:31.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:31.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:31.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:00:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:31.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:32.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:32.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: 101.b scrub starts 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: 101.b scrub ok 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: osdmap e267: 8 total, 8 up, 8 in 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:32.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:32.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:32.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:32.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:32.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-03-30T15:00:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:32.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:00:32.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:32.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:32.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:32.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:32.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:32.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:32.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:32.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.005 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:33.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:33.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:33.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: pgmap v341: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:00:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: 101.e scrub starts 2024-03-30T15:00:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: 101.e scrub ok 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T15:00:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:32 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: pgmap v341: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: 101.e scrub starts 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: 101.e scrub ok 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T15:00:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: pgmap v341: 324 pgs: 32 unknown, 292 active+clean; 586 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: 101.e scrub starts 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: 101.e scrub ok 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi028-92979-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1225934775' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi028-92999-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: osdmap e268: 8 total, 8 up, 8 in 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]: dispatch 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T15:00:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:32 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T15:00:33.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:33.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:33.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:33.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:33.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:33.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:33.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:33.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:33.708 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:33.709 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:33.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:33.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:33.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:33.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:33.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 42.3 scrub starts 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 42.3 scrub ok 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:33.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T15:00:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 42.3 scrub starts 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 42.3 scrub ok 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:33.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:33.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:33.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T15:00:33.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:33 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T15:00:34.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:34.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:34.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 42.3 scrub starts 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 42.3 scrub ok 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-27"}]': finished 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: osdmap e269: 8 total, 8 up, 8 in 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:34.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:34.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T15:00:34.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:33 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T15:00:34.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:34.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:34.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:34.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:34.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:34.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:34.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:34.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:35.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:35.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:35.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:35.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:35.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:35.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: pgmap v344: 300 pgs: 4 unknown, 296 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 42.d scrub starts 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 42.d scrub ok 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.185 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.186 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 17.f scrub starts 2024-03-30T15:00:35.186 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:34 smithi134 ceph-mon[33714]: 17.f scrub ok 2024-03-30T15:00:35.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:35.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:35.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: pgmap v344: 300 pgs: 4 unknown, 296 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:00:35.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 42.d scrub starts 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 42.d scrub ok 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 17.f scrub starts 2024-03-30T15:00:35.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[29876]: 17.f scrub ok 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: pgmap v344: 300 pgs: 4 unknown, 296 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 42.d scrub starts 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 42.d scrub ok 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: osdmap e270: 8 total, 8 up, 8 in 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:35.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:35.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 17.f scrub starts 2024-03-30T15:00:35.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:34 smithi028 ceph-mon[41614]: 17.f scrub ok 2024-03-30T15:00:35.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:35.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:35.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:35.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:35.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:35.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:35.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:35.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:35.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:35.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:35.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:35.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:35.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:35.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:36.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:36.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 17.d scrub starts 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 17.d scrub ok 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 17.16 scrub starts 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 17.16 scrub ok 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 101.2 scrub starts 2024-03-30T15:00:36.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: 101.2 scrub ok 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 17.d scrub starts 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 17.d scrub ok 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 17.16 scrub starts 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 17.16 scrub ok 2024-03-30T15:00:36.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 101.2 scrub starts 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: 101.2 scrub ok 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 17.d scrub starts 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 17.d scrub ok 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 17.16 scrub starts 2024-03-30T15:00:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 17.16 scrub ok 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/174647268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 101.2 scrub starts 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: 101.2 scrub ok 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: osdmap e271: 8 total, 8 up, 8 in 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]: dispatch 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:36.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:36.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:36.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 -- 172.21.15.28:0/627687207 >> v1:172.21.15.28:6790/0 conn(0x7fc32410d9b0 legacy=0x7fc32410fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 -- 172.21.15.28:0/627687207 shutdown_connections 2024-03-30T15:00:36.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 -- 172.21.15.28:0/627687207 >> 172.21.15.28:0/627687207 conn(0x7fc3240fca30 msgr2=0x7fc3240fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:36.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 -- 172.21.15.28:0/627687207 shutdown_connections 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 -- 172.21.15.28:0/627687207 wait complete. 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 librados: starting msgr at 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.306+0000 7fc32ab79700 1 librados: starting objecter 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 Processor -- start 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 -- start start 2024-03-30T15:00:36.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 librados: setting wanted keys 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 librados: calling monclient init 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3241a7120 con 0x7fc324109e10 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3241a8340 con 0x7fc324101180 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc32ab79700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3241a9580 con 0x7fc32410d9b0 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc323fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc324109e10 0x7fc3241a1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54942/0 (socket says 172.21.15.28:54942) 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc323fff700 1 -- 172.21.15.28:0/1657266544 learned_addr learned my addr 172.21.15.28:0/1657266544 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc329116700 1 --1- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6790/0 conn(0x7fc32410d9b0 0x7fc3241a57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56692/0 (socket says 172.21.15.28:56692) 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter ms_handle_connect 0x7fc32410d9b0 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter ms_handle_connect 0x7fc324109e10 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter ms_handle_connect 0x7fc324101180 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310136586 0 0) 0x7fc3241a9580 con 0x7fc32410d9b0 2024-03-30T15:00:36.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc30c0031f0 con 0x7fc32410d9b0 2024-03-30T15:00:36.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923171302 0 0) 0x7fc3241a8340 con 0x7fc324101180 2024-03-30T15:00:36.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3241a9580 con 0x7fc324101180 2024-03-30T15:00:36.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025041828 0 0) 0x7fc3241a7120 con 0x7fc324109e10 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.307+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3241a8340 con 0x7fc324109e10 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1099470856 0 0) 0x7fc30c0031f0 con 0x7fc32410d9b0 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3241a7120 con 0x7fc32410d9b0 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2618962428 0 0) 0x7fc3241a9580 con 0x7fc324101180 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc30c0031f0 con 0x7fc324101180 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1855235543 0 0) 0x7fc3241a8340 con 0x7fc324109e10 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3241a9580 con 0x7fc324109e10 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc31c004860 con 0x7fc32410d9b0 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc3140014d0 con 0x7fc324101180 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc318001230 con 0x7fc324109e10 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 129158312 0 0) 0x7fc3241a7120 con 0x7fc32410d9b0 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.134:6789/0 conn(0x7fc324101180 legacy=0x7fc3241064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6789/0 conn(0x7fc324109e10 legacy=0x7fc3241a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3241aa7c0 con 0x7fc32410d9b0 2024-03-30T15:00:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc32ab79700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3241a7350 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc31c001e70 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc31c005360 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc32ab79700 10 client.57530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc32ab79700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3241a7810 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.308+0000 7fc32ab79700 1 librados: init done 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.309+0000 7fc32ab79700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc324102660 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc31c0055e0 con 0x7fc32410d9b0 2024-03-30T15:00:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(272..272 src has 53..272) ==== 9712+0+0 (unknown 3740431593 0 0) 0x7fc31c08a0d0 con 0x7fc32410d9b0 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 osd_map(272..272 src has 53..272) 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 3 client.57530.objecter handle_osd_map got epochs [272,272] > 0 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 3 client.57530.objecter handle_osd_map decoding full epoch 272 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 20 client.57530.objecter dump_active .. 0 homeless 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.310+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v272) ==== 54+0+16 (unknown 506183972 0 2420000192) 0x7fc31c08a4b0 con 0x7fc32410d9b0 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 1 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 1 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 0 1 2024-03-30T15:00:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 4 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 10 client.57530.objecter _send_command 1 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc304006d40 con 0x7fc304002fd0 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc30c070510 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.311+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc304002fd0 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.312+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304006d40 con 0x7fc304002fd0 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.312+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.312+0000 7fc321ffb700 10 client.57530.objecter _finish_command 1 = system:0 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.312+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 4 2024-03-30T15:00:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.312+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 0 1 2024-03-30T15:00:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:36.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 2 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 2 2024-03-30T15:00:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040028e0 osd=0 3 2024-03-30T15:00:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 0 2 2024-03-30T15:00:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 4 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 10 client.57530.objecter _send_command 2 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.329+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc304001240 con 0x7fc304002fd0 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc304001240 con 0x7fc304002fd0 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 10 client.57530.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 4 2024-03-30T15:00:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 0 2 2024-03-30T15:00:36.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:36.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 1 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 4 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 10 client.57530.objecter _send_command 3 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc304008cc0 con 0x7fc304008100 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.331+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc304008100 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.332+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304008cc0 con 0x7fc304008100 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.332+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.332+0000 7fc321ffb700 10 client.57530.objecter _finish_command 3 = system:0 2024-03-30T15:00:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.332+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 4 2024-03-30T15:00:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.332+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 1 3 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 4 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 4 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304001f50 osd=1 3 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 1 4 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 4 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 10 client.57530.objecter _send_command 4 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.349+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3040096e0 con 0x7fc304008100 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.350+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3040096e0 con 0x7fc304008100 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.350+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.350+0000 7fc321ffb700 10 client.57530.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.350+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 4 2024-03-30T15:00:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.350+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 1 4 2024-03-30T15:00:36.353 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 5 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 5 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 2 5 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 4 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 10 client.57530.objecter _send_command 5 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc304010b20 con 0x7fc30400ce70 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.351+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc30400ce70 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.352+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304010b20 con 0x7fc30400ce70 2024-03-30T15:00:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.352+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.352+0000 7fc321ffb700 10 client.57530.objecter _finish_command 5 = system:0 2024-03-30T15:00:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.352+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 4 2024-03-30T15:00:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.352+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 2 5 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 6 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 6 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304001cc0 osd=2 3 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 2 6 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 4 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 10 client.57530.objecter _send_command 6 2024-03-30T15:00:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.368+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc304009210 con 0x7fc30400ce70 2024-03-30T15:00:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc304009210 con 0x7fc30400ce70 2024-03-30T15:00:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc321ffb700 10 client.57530.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 4 2024-03-30T15:00:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 2 6 2024-03-30T15:00:36.372 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:36.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 7 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 7 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 3 7 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 4 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 10 client.57530.objecter _send_command 7 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.370+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc304007f50 con 0x7fc304010d00 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.371+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc304010d00 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.371+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304007f50 con 0x7fc304010d00 2024-03-30T15:00:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.371+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.372+0000 7fc321ffb700 10 client.57530.objecter _finish_command 7 = system:0 2024-03-30T15:00:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.372+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 4 2024-03-30T15:00:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.372+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 3 7 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 8 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 8 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304009420 osd=3 3 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 3 8 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 4 2024-03-30T15:00:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 10 client.57530.objecter _send_command 8 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.388+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc304012130 con 0x7fc304010d00 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc304012130 con 0x7fc304010d00 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc321ffb700 10 client.57530.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 4 2024-03-30T15:00:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 3 8 2024-03-30T15:00:36.392 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 9 2024-03-30T15:00:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 9 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 4 9 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 4 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 10 client.57530.objecter _send_command 9 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.390+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc30401e710 con 0x7fc30401aaf0 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.391+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc30401aaf0 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.391+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc30401e710 con 0x7fc30401aaf0 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.392+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.392+0000 7fc321ffb700 10 client.57530.objecter _finish_command 9 = system:0 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.392+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 4 2024-03-30T15:00:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.392+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 4 9 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 10 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 10 2024-03-30T15:00:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040120f0 osd=4 3 2024-03-30T15:00:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 4 10 2024-03-30T15:00:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 4 2024-03-30T15:00:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 10 client.57530.objecter _send_command 10 2024-03-30T15:00:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.408+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3040200c0 con 0x7fc30401aaf0 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3040200c0 con 0x7fc30401aaf0 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 10 client.57530.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 4 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 4 10 2024-03-30T15:00:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:36.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 11 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 11 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 5 11 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 4 2024-03-30T15:00:36.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 10 client.57530.objecter _send_command 11 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc304020c40 con 0x7fc3040207e0 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.410+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc3040207e0 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.411+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304020c40 con 0x7fc3040207e0 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.411+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.411+0000 7fc321ffb700 10 client.57530.objecter _finish_command 11 = system:0 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.411+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 4 2024-03-30T15:00:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.411+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 5 11 2024-03-30T15:00:36.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 12 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 12 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc3040200c0 osd=5 3 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 5 12 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 4 2024-03-30T15:00:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 10 client.57530.objecter _send_command 12 2024-03-30T15:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.428+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc304020c40 con 0x7fc3040207e0 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc304020c40 con 0x7fc3040207e0 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 10 client.57530.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 4 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 5 12 2024-03-30T15:00:36.431 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 13 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 13 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 6 13 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 4 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 10 client.57530.objecter _send_command 13 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc304020fd0 con 0x7fc3040237d0 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.430+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc3040237d0 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.431+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc304020fd0 con 0x7fc3040237d0 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.431+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.431+0000 7fc321ffb700 10 client.57530.objecter _finish_command 13 = system:0 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.431+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 4 2024-03-30T15:00:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.431+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 6 13 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 14 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 14 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304020c40 osd=6 3 2024-03-30T15:00:36.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 6 14 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 4 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 10 client.57530.objecter _send_command 14 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.450+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3040272e0 con 0x7fc3040237d0 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc3040272e0 con 0x7fc3040237d0 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 10 client.57530.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 4 2024-03-30T15:00:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 6 14 2024-03-30T15:00:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:36.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:36.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 15 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 15 2024-03-30T15:00:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 7 15 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 4 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 10 client.57530.objecter _send_command 15 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc30402b490 con 0x7fc3040277f0 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.452+0000 7fc321ffb700 10 client.57530.objecter ms_handle_connect 0x7fc3040277f0 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.453+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc30402b490 con 0x7fc3040277f0 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.453+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.453+0000 7fc321ffb700 10 client.57530.objecter _finish_command 15 = system:0 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.453+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 4 2024-03-30T15:00:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.453+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 7 15 2024-03-30T15:00:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 10 client.57530.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign -1 16 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 20 client.57530.objecter _get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_remove -1 16 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 20 client.57530.objecter get_session s=0x7fc304007290 osd=7 3 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 15 client.57530.objecter _session_command_op_assign 7 16 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 4 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 10 client.57530.objecter _send_command 16 2024-03-30T15:00:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.476+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc30402ce40 con 0x7fc3040277f0 2024-03-30T15:00:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc321ffb700 1 -- 172.21.15.28:0/1657266544 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc30402ce40 con 0x7fc3040277f0 2024-03-30T15:00:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc321ffb700 10 client.57530.objecter ms_dispatch 0x7fc324001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc321ffb700 10 client.57530.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc321ffb700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 4 2024-03-30T15:00:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc321ffb700 15 client.57530.objecter _session_command_op_remove 7 16 2024-03-30T15:00:36.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.478+0000 7fc30b7fe700 10 librados: watch_flush enter 2024-03-30T15:00:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 librados: watch_flush exit 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.0 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6801/3394323575 conn(0x7fc304002fd0 legacy=0x7fc304005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040028e0 osd=0 1 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.1 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6805/1850313547 conn(0x7fc304008100 legacy=0x7fc3040085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001f50 osd=1 1 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.2 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6809/3451325043 conn(0x7fc30400ce70 legacy=0x7fc30400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc329116700 1 -- 172.21.15.28:0/1657266544 reap_dead start 2024-03-30T15:00:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304001cc0 osd=2 1 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.3 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6813/1506157250 conn(0x7fc304010d00 legacy=0x7fc304011160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304009420 osd=3 1 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.4 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.134:6800/3063775514 conn(0x7fc30401aaf0 legacy=0x7fc30401cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040120f0 osd=4 1 2024-03-30T15:00:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.5 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.134:6804/3668151179 conn(0x7fc3040207e0 legacy=0x7fc30401e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc3040200c0 osd=5 1 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.6 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.134:6808/2135224194 conn(0x7fc3040237d0 legacy=0x7fc304025bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304020c40 osd=6 1 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter close_session for osd.7 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.134:6812/3758190298 conn(0x7fc3040277f0 legacy=0x7fc304029be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter put_session s=0x7fc304007290 osd=7 1 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 20 client.57530.objecter shutdown clearing up homeless session... 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 10 client.57530.objecter successfully canceled tick 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc329116700 1 -- 172.21.15.28:0/1657266544 reap_dead start 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc30c070510 legacy=0x7fc30c0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> v1:172.21.15.28:6790/0 conn(0x7fc32410d9b0 legacy=0x7fc3241a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 shutdown_connections 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 >> 172.21.15.28:0/1657266544 conn(0x7fc3240fca30 msgr2=0x7fc324106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 shutdown_connections 2024-03-30T15:00:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 -- 172.21.15.28:0/1657266544 wait complete. 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:36.479+0000 7fc30b7fe700 1 librados: shutdown 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:36.486 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:36.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:36.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:36.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:36.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:36.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:36.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:36.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:00:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:00:36.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 101.7 scrub starts 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 101.7 scrub ok 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 101.1 scrub starts 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 101.1 scrub ok 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1657266544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 101.7 scrub starts 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 101.7 scrub ok 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 101.1 scrub starts 2024-03-30T15:00:36.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 101.1 scrub ok 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:36.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1657266544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:36.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:36.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:36 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:36.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:36.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:37.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:37.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:00:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 101.7 scrub starts 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 101.7 scrub ok 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 101.1 scrub starts 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 101.1 scrub ok 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.57506 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi028-92979-36"}]': finished 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: osdmap e272: 8 total, 8 up, 8 in 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1657266544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:37.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:36 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:37.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:37.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:37.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:37.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:37.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:37.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:37.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:37.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:37.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:37.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:37.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:37.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:37.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:37 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:37.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:38.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: osdmap e273: 8 total, 8 up, 8 in 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:38.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:37 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:38.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:38.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:38.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:38.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:38.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:38.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:38.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:38.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:38.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:38.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 -- 172.21.15.28:0/1617145948 >> v1:172.21.15.28:6789/0 conn(0x7fdac810d9f0 legacy=0x7fdac810fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 -- 172.21.15.28:0/1617145948 shutdown_connections 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 -- 172.21.15.28:0/1617145948 >> 172.21.15.28:0/1617145948 conn(0x7fdac8077370 msgr2=0x7fdac80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 -- 172.21.15.28:0/1617145948 shutdown_connections 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 -- 172.21.15.28:0/1617145948 wait complete. 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 librados: starting msgr at 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.668+0000 7fdaccad9700 1 librados: starting objecter 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 Processor -- start 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 -- start start 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 librados: setting wanted keys 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 librados: calling monclient init 2024-03-30T15:00:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdac8106800 con 0x7fdac8101790 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdac81a85d0 con 0x7fdac8109e50 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdaccad9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdac81a97b0 con 0x7fdac810d9f0 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdac6d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdac810d9f0 0x7fdac81a5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43268/0 (socket says 172.21.15.28:43268) 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdac659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdac8101790 0x7fdac81045d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41718/0 (socket says 172.21.15.28:41718) 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdac6d9d700 1 -- 172.21.15.28:0/1162434008 learned_addr learned my addr 172.21.15.28:0/1162434008 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter ms_handle_connect 0x7fdac8101790 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter ms_handle_connect 0x7fdac810d9f0 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341308633 0 0) 0x7fdac81a97b0 con 0x7fdac810d9f0 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab00031f0 con 0x7fdac810d9f0 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114272541 0 0) 0x7fdac8106800 con 0x7fdac8101790 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac81a97b0 con 0x7fdac8101790 2024-03-30T15:00:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter ms_handle_connect 0x7fdac8109e50 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.669+0000 7fdab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378217783 0 0) 0x7fdac81a85d0 con 0x7fdac8109e50 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac8106800 con 0x7fdac8109e50 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3543758284 0 0) 0x7fdab00031f0 con 0x7fdac810d9f0 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdac81a85d0 con 0x7fdac810d9f0 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdac0004730 con 0x7fdac810d9f0 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2090595240 0 0) 0x7fdac81a97b0 con 0x7fdac8101790 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdab00031f0 con 0x7fdac8101790 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdab8001960 con 0x7fdac8101790 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 242499967 0 0) 0x7fdac81a85d0 con 0x7fdac810d9f0 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 >> v1:172.21.15.134:6789/0 conn(0x7fdac8109e50 legacy=0x7fdac8104ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 >> v1:172.21.15.28:6789/0 conn(0x7fdac8101790 legacy=0x7fdac81045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac81aa990 con 0x7fdac810d9f0 2024-03-30T15:00:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdac81a99e0 con 0x7fdac810d9f0 2024-03-30T15:00:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdaccad9700 10 client.57536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdac81a9ea0 con 0x7fdac810d9f0 2024-03-30T15:00:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdaccad9700 1 librados: init done 2024-03-30T15:00:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdac00049f0 con 0x7fdac810d9f0 2024-03-30T15:00:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.670+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdac0005280 con 0x7fdac810d9f0 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.671+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac8102750 con 0x7fdac810d9f0 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.671+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdac0005420 con 0x7fdac810d9f0 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(274..274 src has 53..274) ==== 9715+0+0 (unknown 2778652822 0 0) 0x7fdac008a240 con 0x7fdac810d9f0 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 10 client.57536.objecter ms_dispatch 0x7fdac8001550 osd_map(274..274 src has 53..274) 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 3 client.57536.objecter handle_osd_map got epochs [274,274] > 0 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 3 client.57536.objecter handle_osd_map decoding full epoch 274 2024-03-30T15:00:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 20 client.57536.objecter dump_active .. 0 homeless 2024-03-30T15:00:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.672+0000 7fdab77fe700 10 client.57536.objecter ms_handle_connect 0x7fdab0070200 2024-03-30T15:00:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.677+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdac0056e40 con 0x7fdac810d9f0 2024-03-30T15:00:38.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:38.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:38.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:38.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:38.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:38.902+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fdac8105b00 con 0x7fdac810d9f0 2024-03-30T15:00:39.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: 17.2 scrub starts 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: 17.2 scrub ok 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: pgmap v350: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: 101.9 scrub starts 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: 101.9 scrub ok 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T15:00:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.220+0000 7fdab77fe700 1 -- 172.21.15.28:0/1162434008 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v275) ==== 113+0+0 (unknown 3689191711 0 0) 0x7fdac005a890 con 0x7fdac810d9f0 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: 17.2 scrub starts 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: 17.2 scrub ok 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: pgmap v350: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: 101.9 scrub starts 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: 101.9 scrub ok 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:39.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: 17.2 scrub starts 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: 17.2 scrub ok 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: pgmap v350: 300 pgs: 40 unknown, 260 active+clean; 586 KiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: 101.9 scrub starts 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: 101.9 scrub ok 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: osdmap e274: 8 total, 8 up, 8 in 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:39.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.224+0000 7fdaccad9700 10 librados: watch_flush enter 2024-03-30T15:00:39.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.224+0000 7fdaccad9700 10 librados: watch_flush exit 2024-03-30T15:00:39.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.224+0000 7fdaccad9700 20 client.57536.objecter shutdown clearing up homeless session... 2024-03-30T15:00:39.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.224+0000 7fdaccad9700 10 client.57536.objecter successfully canceled tick 2024-03-30T15:00:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.224+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdab0070200 legacy=0x7fdab00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.225+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 >> v1:172.21.15.28:6790/0 conn(0x7fdac810d9f0 legacy=0x7fdac81a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.225+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 shutdown_connections 2024-03-30T15:00:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.225+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 >> 172.21.15.28:0/1162434008 conn(0x7fdac8077370 msgr2=0x7fdac8100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.226+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 shutdown_connections 2024-03-30T15:00:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.226+0000 7fdaccad9700 1 -- 172.21.15.28:0/1162434008 wait complete. 2024-03-30T15:00:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:39.226+0000 7fdaccad9700 1 librados: shutdown 2024-03-30T15:00:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T15:00:39.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.290 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.417 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:39.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:39.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:39.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:39.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: 17.1b scrub starts 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: 17.1b scrub ok 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1162434008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.57536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.57536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T15:00:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: 17.1b scrub starts 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: 17.1b scrub ok 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1162434008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.57536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:39.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.57536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:39.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:39 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:40.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:40.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:40.136 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T15:00:40.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:00:40.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:40.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: 17.1b scrub starts 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: 17.1b scrub ok 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1162434008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.57536 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi028-92979-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.57536 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:00:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: osdmap e275: 8 total, 8 up, 8 in 2024-03-30T15:00:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/361707034' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:39 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]: dispatch 2024-03-30T15:00:40.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:40.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:40.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:40.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:40.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:40.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:40.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:40.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:40.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:40.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:40.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.823+0000 7fc0006aa700 1 -- 172.21.15.28:0/461035882 >> v1:172.21.15.28:6790/0 conn(0x7fbff8109dc0 legacy=0x7fbff810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.823+0000 7fc0006aa700 1 -- 172.21.15.28:0/461035882 shutdown_connections 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.823+0000 7fc0006aa700 1 -- 172.21.15.28:0/461035882 >> 172.21.15.28:0/461035882 conn(0x7fbff80fc9a0 msgr2=0x7fbff80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- 172.21.15.28:0/461035882 shutdown_connections 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- 172.21.15.28:0/461035882 wait complete. 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 librados: starting msgr at 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 librados: starting objecter 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 Processor -- start 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- start start 2024-03-30T15:00:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 librados: setting wanted keys 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 librados: calling monclient init 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbff81a6f00 con 0x7fbff8109dc0 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbff81a8120 con 0x7fbff8101130 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.824+0000 7fc0006aa700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbff81a9360 con 0x7fbff810d960 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbffec47700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbff810d960 0x7fbff81a55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43310/0 (socket says 172.21.15.28:43310) 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbffdc45700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbff8109dc0 0x7fbff81a1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41760/0 (socket says 172.21.15.28:41760) 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbffec47700 1 -- 172.21.15.28:0/3497068431 learned_addr learned my addr 172.21.15.28:0/3497068431 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter ms_handle_connect 0x7fbff8109dc0 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter ms_handle_connect 0x7fbff810d960 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter ms_handle_connect 0x7fbff8101130 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560084761 0 0) 0x7fbff81a8120 con 0x7fbff8101130 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe4003aa0 con 0x7fbff8101130 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290794999 0 0) 0x7fbff81a9360 con 0x7fbff810d960 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff81a8120 con 0x7fbff810d960 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473548371 0 0) 0x7fbff81a6f00 con 0x7fbff8109dc0 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff81a9360 con 0x7fbff8109dc0 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2190472941 0 0) 0x7fbfe4003aa0 con 0x7fbff8101130 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbff81a6f00 con 0x7fbff8101130 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbff4001870 con 0x7fbff8101130 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1556364720 0 0) 0x7fbff81a9360 con 0x7fbff8109dc0 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfe4003aa0 con 0x7fbff8109dc0 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbfe8004430 con 0x7fbff8109dc0 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2046705659 0 0) 0x7fbff81a8120 con 0x7fbff810d960 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbff81a9360 con 0x7fbff810d960 2024-03-30T15:00:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbff0001c10 con 0x7fbff810d960 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4048574056 0 0) 0x7fbff81a6f00 con 0x7fbff8101130 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 >> v1:172.21.15.28:6790/0 conn(0x7fbff810d960 legacy=0x7fbff81a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 >> v1:172.21.15.28:6789/0 conn(0x7fbff8109dc0 legacy=0x7fbff81a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.825+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff81aa5a0 con 0x7fbff8101130 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a8350 con 0x7fbff8101130 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fc0006aa700 10 client.49573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbff81a8810 con 0x7fbff8101130 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fc0006aa700 1 librados: init done 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbff4001c60 con 0x7fbff8101130 2024-03-30T15:00:40.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.826+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbff4004b80 con 0x7fbff8101130 2024-03-30T15:00:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.827+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfdc005120 con 0x7fbff8101130 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.827+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbff4003950 con 0x7fbff8101130 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(276..276 src has 53..276) ==== 9712+0+0 (unknown 2233897551 0 0) 0x7fbff4088ad0 con 0x7fbff8101130 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 10 client.49573.objecter ms_dispatch 0x7fbff8001550 osd_map(276..276 src has 53..276) 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 3 client.49573.objecter handle_osd_map got epochs [276,276] > 0 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 3 client.49573.objecter handle_osd_map decoding full epoch 276 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 20 client.49573.objecter dump_active .. 0 homeless 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.828+0000 7fbfef7fe700 10 client.49573.objecter ms_handle_connect 0x7fbfe40704d0 2024-03-30T15:00:40.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:40.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:40.833+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbff40556b0 con 0x7fbff8101130 2024-03-30T15:00:40.933 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:00:40.933 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:00:40.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:40.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:40.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: 42.18 scrub starts 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[29876]: 42.18 scrub ok 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:00:40.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: 42.18 scrub starts 2024-03-30T15:00:40.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:40 smithi028 ceph-mon[41614]: 42.18 scrub ok 2024-03-30T15:00:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.047+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fbfdc005410 con 0x7fbff8101130 2024-03-30T15:00:41.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:41.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:00:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:00:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:41.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.57518 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-46"}]': finished 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: osdmap e276: 8 total, 8 up, 8 in 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: 42.18 scrub starts 2024-03-30T15:00:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:40 smithi134 ceph-mon[33714]: 42.18 scrub ok 2024-03-30T15:00:41.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:41.192 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:41.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.226+0000 7fbfef7fe700 1 -- 172.21.15.28:0/3497068431 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v277) ==== 101+0+0 (unknown 1759445216 0 0) 0x7fbff4059100 con 0x7fbff8101130 2024-03-30T15:00:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 10 librados: watch_flush enter 2024-03-30T15:00:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 10 librados: watch_flush exit 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 20 client.49573.objecter shutdown clearing up homeless session... 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 10 client.49573.objecter successfully canceled tick 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbfe40704d0 legacy=0x7fbfe4072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 >> v1:172.21.15.134:6789/0 conn(0x7fbff8101130 legacy=0x7fbff806ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 shutdown_connections 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.229+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 >> 172.21.15.28:0/3497068431 conn(0x7fbff80fc9a0 msgr2=0x7fbff80fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.230+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 shutdown_connections 2024-03-30T15:00:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.230+0000 7fc0006aa700 1 -- 172.21.15.28:0/3497068431 wait complete. 2024-03-30T15:00:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:41.230+0000 7fc0006aa700 1 librados: shutdown 2024-03-30T15:00:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:00:41.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:41.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:41.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:41.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:41.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:41.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:41.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:41.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:41.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:41.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3497068431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.49573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]': finished 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.49573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:41.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: 17.1e scrub starts 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[29876]: 17.1e scrub ok 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3497068431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.49573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]': finished 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.49573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:41.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: 17.1e scrub starts 2024-03-30T15:00:41.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:41 smithi028 ceph-mon[41614]: 17.1e scrub ok 2024-03-30T15:00:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:41.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:42.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:42.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3497068431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.49573 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:00:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-29", "mode": "writeback"}]': finished 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi028-92999-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.49573 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: osdmap e277: 8 total, 8 up, 8 in 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: 17.1e scrub starts 2024-03-30T15:00:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:41 smithi134 ceph-mon[33714]: 17.1e scrub ok 2024-03-30T15:00:42.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:42.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:42.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:42.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:42.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:42.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:42.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:42.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:42.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:42.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:42.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:42.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:42.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 101.15 scrub starts 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 101.15 scrub ok 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: pgmap v356: 292 pgs: 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 17.a scrub starts 2024-03-30T15:00:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: 17.a scrub ok 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 101.15 scrub starts 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 101.15 scrub ok 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: pgmap v356: 292 pgs: 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:00:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 17.a scrub starts 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: 17.a scrub ok 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:42.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:42 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:42.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:43.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:43.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:43.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 101.15 scrub starts 2024-03-30T15:00:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 101.15 scrub ok 2024-03-30T15:00:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: pgmap v356: 292 pgs: 292 active+clean; 4.6 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-30T15:00:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 17.a scrub starts 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: 17.a scrub ok 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2296641682' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: osdmap e278: 8 total, 8 up, 8 in 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:42 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]: dispatch 2024-03-30T15:00:43.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:43.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:43.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:43.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:43.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:43.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:43.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:43.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:43.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:43.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:43.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:43.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:43.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:44.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:44.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:44.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 17.c scrub starts 2024-03-30T15:00:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 17.c scrub ok 2024-03-30T15:00:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 42.17 scrub starts 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 42.17 scrub ok 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:44.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 42.1c deep-scrub starts 2024-03-30T15:00:44.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: 42.1c deep-scrub ok 2024-03-30T15:00:44.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:43 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 17.c scrub starts 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 17.c scrub ok 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 42.17 scrub starts 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 42.17 scrub ok 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:00:44.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 42.1c deep-scrub starts 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: 42.1c deep-scrub ok 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 17.c scrub starts 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 17.c scrub ok 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 42.17 scrub starts 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 42.17 scrub ok 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi028-92999-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.57524 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi028-92979-37"}]': finished 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: osdmap e279: 8 total, 8 up, 8 in 2024-03-30T15:00:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 42.1c deep-scrub starts 2024-03-30T15:00:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: 42.1c deep-scrub ok 2024-03-30T15:00:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:43 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:00:44.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:44.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:44.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:44.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:44.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:44.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:44.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:44.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:44.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:44.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:44.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:44.922 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:44.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:45.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:45.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:45.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:45.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:00:45.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:00:45.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:45.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:45.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:45.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:45.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: pgmap v359: 300 pgs: 4 creating+peering, 4 unknown, 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:45.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:00:45.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:45.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:00:45.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: pgmap v359: 300 pgs: 4 creating+peering, 4 unknown, 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:45.432 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:00:45.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:45 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:00:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:45.459 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:45.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:45.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: pgmap v359: 300 pgs: 4 creating+peering, 4 unknown, 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi028-92979-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: osdmap e280: 8 total, 8 up, 8 in 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:45.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]: dispatch 2024-03-30T15:00:45.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:45.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:00:45.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:45 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:00:45.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:45.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:45.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:45.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:45.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.914+0000 7f02cce82700 1 -- 172.21.15.28:0/2039354533 >> v1:172.21.15.28:6790/0 conn(0x7f02c81031f0 legacy=0x7f02c81035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.914+0000 7f02cce82700 1 -- 172.21.15.28:0/2039354533 shutdown_connections 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.914+0000 7f02cce82700 1 -- 172.21.15.28:0/2039354533 >> 172.21.15.28:0/2039354533 conn(0x7f02c80fc9e0 msgr2=0x7f02c80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.914+0000 7f02cce82700 1 -- 172.21.15.28:0/2039354533 shutdown_connections 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.914+0000 7f02cce82700 1 -- 172.21.15.28:0/2039354533 wait complete. 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 librados: starting msgr at 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 librados: starting objecter 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 Processor -- start 2024-03-30T15:00:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 -- start start 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 librados: setting wanted keys 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 librados: calling monclient init 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02c810ac90 con 0x7f02c81031f0 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02c81a8650 con 0x7f02c810b000 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.915+0000 7f02cce82700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02c81a9830 con 0x7f02c810eb10 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02c6d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f02c810eb10 0x7f02c81a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43342/0 (socket says 172.21.15.28:43342) 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02c659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f02c81031f0 0x7f02c8107ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41788/0 (socket says 172.21.15.28:41788) 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02c6d9d700 1 -- 172.21.15.28:0/3625230000 learned_addr learned my addr 172.21.15.28:0/3625230000 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter ms_handle_connect 0x7f02c810eb10 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter ms_handle_connect 0x7f02c81031f0 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter ms_handle_connect 0x7f02c810b000 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592647043 0 0) 0x7f02c81a9830 con 0x7f02c810eb10 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b40031f0 con 0x7f02c810eb10 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207594930 0 0) 0x7f02c810ac90 con 0x7f02c81031f0 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02c81a9830 con 0x7f02c81031f0 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68455077 0 0) 0x7f02c81a8650 con 0x7f02c810b000 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02c810ac90 con 0x7f02c810b000 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1613002011 0 0) 0x7f02b40031f0 con 0x7f02c810eb10 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02c81a8650 con 0x7f02c810eb10 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f02b8001c30 con 0x7f02c810eb10 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3197666406 0 0) 0x7f02c81a9830 con 0x7f02c81031f0 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02b40031f0 con 0x7f02c81031f0 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f02c0001430 con 0x7f02c81031f0 2024-03-30T15:00:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3426180616 0 0) 0x7f02c810ac90 con 0x7f02c810b000 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02c81a9830 con 0x7f02c810b000 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f02b0001830 con 0x7f02c810b000 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2463981421 0 0) 0x7f02c81a8650 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.134:6789/0 conn(0x7f02c810b000 legacy=0x7f02c81081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6789/0 conn(0x7f02c81031f0 legacy=0x7f02c8107ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c81aaa10 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02c81a76a0 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02cce82700 10 client.57560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f02c81a7b60 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.916+0000 7f02cce82700 1 librados: init done 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.917+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f02b8003ec0 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.917+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f02b8005fb0 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.917+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f02a8005100 con 0x7f02c810eb10 2024-03-30T15:00:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f02b801c870 con 0x7f02c810eb10 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(281..281 src has 53..281) ==== 9755+0+0 (unknown 2294432103 0 0) 0x7f02b8089f90 con 0x7f02c810eb10 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 osd_map(281..281 src has 53..281) 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 3 client.57560.objecter handle_osd_map got epochs [281,281] > 0 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 3 client.57560.objecter handle_osd_map decoding full epoch 281 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 20 client.57560.objecter dump_active .. 0 homeless 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.918+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v281) ==== 54+0+16 (unknown 1135329989 0 2420000192) 0x7f02b808c3f0 con 0x7f02c810eb10 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02b40703d0 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 10 client.57560.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 1 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 1 2024-03-30T15:00:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 0 1 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 4 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 10 client.57560.objecter _send_command 1 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02a8004b50 con 0x7f02a8004730 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.919+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a8004730 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.920+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a8004b50 con 0x7f02a8004730 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.920+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.920+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 1 = system:0 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.920+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 4 2024-03-30T15:00:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.920+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 0 1 2024-03-30T15:00:45.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:45.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 10 client.57560.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 2 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 2 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a80053e0 osd=0 3 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 0 2 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 4 2024-03-30T15:00:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 10 client.57560.objecter _send_command 2 2024-03-30T15:00:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.937+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a800bf10 con 0x7f02a8004730 2024-03-30T15:00:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a800bf10 con 0x7f02a8004730 2024-03-30T15:00:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 4 2024-03-30T15:00:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 0 2 2024-03-30T15:00:45.941 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 10 client.57560.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 1 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 4 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 10 client.57560.objecter _send_command 3 2024-03-30T15:00:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f02a8002200 con 0x7f02a80042f0 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.939+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a80042f0 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.940+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a8002200 con 0x7f02a80042f0 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.940+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.940+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 3 = system:0 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.940+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 4 2024-03-30T15:00:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.940+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 1 3 2024-03-30T15:00:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 10 client.57560.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 4 2024-03-30T15:00:45.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 4 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.956+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8004180 osd=1 3 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.957+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 1 4 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.957+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 4 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.957+0000 7f02cce82700 10 client.57560.objecter _send_command 4 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.957+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a8001640 con 0x7f02a80042f0 2024-03-30T15:00:45.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a8001640 con 0x7f02a80042f0 2024-03-30T15:00:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 4 2024-03-30T15:00:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 1 4 2024-03-30T15:00:45.961 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 10 client.57560.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 5 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 5 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 2 5 2024-03-30T15:00:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 4 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 10 client.57560.objecter _send_command 5 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f02a8015600 con 0x7f02a8011af0 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.959+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a8011af0 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.960+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a8015600 con 0x7f02a8011af0 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.960+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.960+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 5 = system:0 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.960+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 4 2024-03-30T15:00:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.960+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 2 5 2024-03-30T15:00:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 10 client.57560.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 6 2024-03-30T15:00:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 6 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a800c890 osd=2 3 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 2 6 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 4 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 10 client.57560.objecter _send_command 6 2024-03-30T15:00:45.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.977+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a800c370 con 0x7f02a8011af0 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a800c370 con 0x7f02a8011af0 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 4 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 2 6 2024-03-30T15:00:45.981 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 10 client.57560.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 7 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 7 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 3 7 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 4 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 10 client.57560.objecter _send_command 7 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.978+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f02a80118f0 con 0x7f02a8016930 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a8016930 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a80118f0 con 0x7f02a8016930 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 7 = system:0 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 4 2024-03-30T15:00:45.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.979+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 3 7 2024-03-30T15:00:45.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 10 client.57560.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 8 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 8 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8011140 osd=3 3 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 3 8 2024-03-30T15:00:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 4 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 10 client.57560.objecter _send_command 8 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.996+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a8010c80 con 0x7f02a8016930 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a8010c80 con 0x7f02a8016930 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 4 2024-03-30T15:00:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 3 8 2024-03-30T15:00:46.001 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 10 client.57560.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 9 2024-03-30T15:00:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 9 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 4 9 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 4 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 10 client.57560.objecter _send_command 9 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.998+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f02a801edf0 con 0x7f02a8016490 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a8016490 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a801edf0 con 0x7f02a8016490 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 9 = system:0 2024-03-30T15:00:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 4 2024-03-30T15:00:46.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:45.999+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 4 9 2024-03-30T15:00:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 10 client.57560.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 10 2024-03-30T15:00:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 10 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8010c80 osd=4 3 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 4 10 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 4 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 10 client.57560.objecter _send_command 10 2024-03-30T15:00:46.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.016+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a801edf0 con 0x7f02a8016490 2024-03-30T15:00:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a801edf0 con 0x7f02a8016490 2024-03-30T15:00:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 4 2024-03-30T15:00:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 4 10 2024-03-30T15:00:46.021 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:46.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 10 client.57560.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 11 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 11 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 5 11 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 4 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 10 client.57560.objecter _send_command 11 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f02a8023170 con 0x7f02a801f470 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a801f470 2024-03-30T15:00:46.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a8023170 con 0x7f02a801f470 2024-03-30T15:00:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 11 = system:0 2024-03-30T15:00:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 4 2024-03-30T15:00:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.019+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 5 11 2024-03-30T15:00:46.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 10 client.57560.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:46.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 12 2024-03-30T15:00:46.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 12 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a801edf0 osd=5 3 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 5 12 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 4 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 10 client.57560.objecter _send_command 12 2024-03-30T15:00:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.036+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a8024b20 con 0x7f02a801f470 2024-03-30T15:00:46.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a8024b20 con 0x7f02a801f470 2024-03-30T15:00:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 4 2024-03-30T15:00:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 5 12 2024-03-30T15:00:46.041 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 10 client.57560.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 13 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 13 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 6 13 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 4 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 10 client.57560.objecter _send_command 13 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.038+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f02a80256a0 con 0x7f02a8025240 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.039+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a8025240 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.039+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a80256a0 con 0x7f02a8025240 2024-03-30T15:00:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.040+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.040+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 13 = system:0 2024-03-30T15:00:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.040+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 4 2024-03-30T15:00:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.040+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 6 13 2024-03-30T15:00:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 10 client.57560.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 14 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 14 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a8024b20 osd=6 3 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 6 14 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 4 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 10 client.57560.objecter _send_command 14 2024-03-30T15:00:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.057+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a80256a0 con 0x7f02a8025240 2024-03-30T15:00:46.060 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a80256a0 con 0x7f02a8025240 2024-03-30T15:00:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 4 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 6 14 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 10 client.57560.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 15 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 15 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 7 15 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 4 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 10 client.57560.objecter _send_command 15 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.058+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f02a802bb50 con 0x7f02a80258d0 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.059+0000 7f02bf7fe700 10 client.57560.objecter ms_handle_connect 0x7f02a80258d0 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.060+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f02a802bb50 con 0x7f02a80258d0 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.060+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.060+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 15 = system:0 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.060+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 4 2024-03-30T15:00:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.060+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 7 15 2024-03-30T15:00:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:46.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 10 client.57560.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign -1 16 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 20 client.57560.objecter _get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 15 client.57560.objecter _session_command_op_remove -1 16 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 20 client.57560.objecter get_session s=0x7f02a80256a0 osd=7 3 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 15 client.57560.objecter _session_command_op_assign 7 16 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 4 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 10 client.57560.objecter _send_command 16 2024-03-30T15:00:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.078+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f02a802bb50 con 0x7f02a80258d0 2024-03-30T15:00:46.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.079+0000 7f02bf7fe700 1 -- 172.21.15.28:0/3625230000 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f02a802bb50 con 0x7f02a80258d0 2024-03-30T15:00:46.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.079+0000 7f02bf7fe700 10 client.57560.objecter ms_dispatch 0x7f02c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:46.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.079+0000 7f02bf7fe700 10 client.57560.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:46.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.079+0000 7f02bf7fe700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 4 2024-03-30T15:00:46.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.079+0000 7f02bf7fe700 15 client.57560.objecter _session_command_op_remove 7 16 2024-03-30T15:00:46.082 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:46.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 librados: watch_flush enter 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 librados: watch_flush exit 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.0 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6801/3394323575 conn(0x7f02a8004730 legacy=0x7f02a8002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80053e0 osd=0 1 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.1 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6805/1850313547 conn(0x7f02a80042f0 legacy=0x7f02a800f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8004180 osd=1 1 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.2 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6809/3451325043 conn(0x7f02a8011af0 legacy=0x7f02a8013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a800c890 osd=2 1 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.3 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6813/1506157250 conn(0x7f02a8016930 legacy=0x7f02a8018d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8011140 osd=3 1 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.4 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.134:6800/3063775514 conn(0x7f02a8016490 legacy=0x7f02a801a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02c6d9d700 1 -- 172.21.15.28:0/3625230000 reap_dead start 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8010c80 osd=4 1 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.5 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.134:6804/3668151179 conn(0x7f02a801f470 legacy=0x7f02a80218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a801edf0 osd=5 1 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.6 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.134:6808/2135224194 conn(0x7f02a8025240 legacy=0x7f02a8023350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a8024b20 osd=6 1 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter close_session for osd.7 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.134:6812/3758190298 conn(0x7f02a80258d0 legacy=0x7f02a802a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02c6d9d700 1 -- 172.21.15.28:0/3625230000 reap_dead start 2024-03-30T15:00:46.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter put_session s=0x7f02a80256a0 osd=7 1 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 20 client.57560.objecter shutdown clearing up homeless session... 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 10 client.57560.objecter successfully canceled tick 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02c6d9d700 1 -- 172.21.15.28:0/3625230000 reap_dead start 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02c6d9d700 1 -- 172.21.15.28:0/3625230000 reap_dead start 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6800/3011431093 conn(0x7f02b40703d0 legacy=0x7f02b4072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> v1:172.21.15.28:6790/0 conn(0x7f02c810eb10 legacy=0x7f02c81a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 shutdown_connections 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 >> 172.21.15.28:0/3625230000 conn(0x7f02c80fc9e0 msgr2=0x7f02c80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 shutdown_connections 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 -- 172.21.15.28:0/3625230000 wait complete. 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.080+0000 7f02cce82700 1 librados: shutdown 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:46.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 -- 172.21.15.28:0/325789418 >> v1:172.21.15.28:6789/0 conn(0x7f5d4c10afa0 legacy=0x7f5d4c10d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 -- 172.21.15.28:0/325789418 shutdown_connections 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 -- 172.21.15.28:0/325789418 >> 172.21.15.28:0/325789418 conn(0x7f5d4c0fc9e0 msgr2=0x7f5d4c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 -- 172.21.15.28:0/325789418 shutdown_connections 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 -- 172.21.15.28:0/325789418 wait complete. 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 librados: starting msgr at 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.179+0000 7f5d51fb3700 1 librados: starting objecter 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 Processor -- start 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 -- start start 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 librados: setting wanted keys 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 librados: calling monclient init 2024-03-30T15:00:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d4c1a9110 con 0x7f5d4c10afa0 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d4c1aa330 con 0x7f5d4c10eab0 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d51fb3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d4c1ab570 con 0x7f5d4c103190 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d4b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5d4c103190 0x7f5d4c10a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43362/0 (socket says 172.21.15.28:43362) 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d4b7fe700 1 -- 172.21.15.28:0/2862065650 learned_addr learned my addr 172.21.15.28:0/2862065650 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter ms_handle_connect 0x7f5d4c103190 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter ms_handle_connect 0x7f5d4c10afa0 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter ms_handle_connect 0x7f5d4c10eab0 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108782828 0 0) 0x7f5d4c1ab570 con 0x7f5d4c103190 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d380039c0 con 0x7f5d4c103190 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1136369159 0 0) 0x7f5d4c1aa330 con 0x7f5d4c10eab0 2024-03-30T15:00:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.180+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c1ab570 con 0x7f5d4c10eab0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1928084020 0 0) 0x7f5d4c1a9110 con 0x7f5d4c10afa0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c1aa330 con 0x7f5d4c10afa0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1076963149 0 0) 0x7f5d4c1ab570 con 0x7f5d4c10eab0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d4c1a9110 con 0x7f5d4c10eab0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2253394240 0 0) 0x7f5d380039c0 con 0x7f5d4c103190 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d4c1ab570 con 0x7f5d4c103190 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d44001590 con 0x7f5d4c10eab0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d3c0019d0 con 0x7f5d4c103190 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3094680955 0 0) 0x7f5d4c1aa330 con 0x7f5d4c10afa0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d380039c0 con 0x7f5d4c10afa0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d34001710 con 0x7f5d4c10afa0 2024-03-30T15:00:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 204710468 0 0) 0x7f5d4c1a9110 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 >> v1:172.21.15.28:6790/0 conn(0x7f5d4c103190 legacy=0x7f5d4c10a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 >> v1:172.21.15.28:6789/0 conn(0x7f5d4c10afa0 legacy=0x7f5d4c1a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c1ac7b0 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c1aa560 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d51fb3700 10 client.49591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d4c1aaa20 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d51fb3700 1 librados: init done 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5d44003ea0 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.181+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d44004d90 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.182+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c005120 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.182+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5d4401b6b0 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(281..281 src has 53..281) ==== 9755+0+0 (unknown 2294432103 0 0) 0x7f5d44088d80 con 0x7f5d4c10eab0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 10 client.49591.objecter ms_dispatch 0x7f5d4c001550 osd_map(281..281 src has 53..281) 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 3 client.49591.objecter handle_osd_map got epochs [281,281] > 0 2024-03-30T15:00:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 3 client.49591.objecter handle_osd_map decoding full epoch 281 2024-03-30T15:00:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 20 client.49591.objecter dump_active .. 0 homeless 2024-03-30T15:00:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.183+0000 7f5d48ff9700 10 client.49591.objecter ms_handle_connect 0x7f5d380704a0 2024-03-30T15:00:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.188+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5d440b9270 con 0x7f5d4c10eab0 2024-03-30T15:00:46.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.411+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d2c000b90 con 0x7f5d380704a0 2024-03-30T15:00:46.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.431+0000 7f5d48ff9700 1 -- 172.21.15.28:0/2862065650 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+684366 (unknown 2965378022 0 1302483448) 0x7f5d2c000b90 con 0x7f5d380704a0 2024-03-30T15:00:46.443 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:00:46.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 10 librados: watch_flush enter 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 10 librados: watch_flush exit 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 20 client.49591.objecter shutdown clearing up homeless session... 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 10 client.49591.objecter successfully canceled tick 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5d380704a0 legacy=0x7f5d38072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.437+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 >> v1:172.21.15.134:6789/0 conn(0x7f5d4c10eab0 legacy=0x7f5d4c1a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.438+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 shutdown_connections 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.438+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 >> 172.21.15.28:0/2862065650 conn(0x7f5d4c0fc9e0 msgr2=0x7f5d4c110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.438+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 shutdown_connections 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.438+0000 7f5d51fb3700 1 -- 172.21.15.28:0/2862065650 wait complete. 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:46.438+0000 7f5d51fb3700 1 librados: shutdown 2024-03-30T15:00:46.450 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:00:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3625230000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3625230000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:46.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:46.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:46.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-29"}]': finished 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: osdmap e281: 8 total, 8 up, 8 in 2024-03-30T15:00:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3625230000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:46.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:46.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:46.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:46.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:47.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:47.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:47.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:47.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:47.244 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-03-30T15:00:46.225991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793869,"num_objects":334,"num_object_clones":1,"num_object_copies":1002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":1389,"num_read_kb":1141,"num_write":1261,"num_write_kb":11251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2566,"ondisk_log_size":2566,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":882,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":593780,"kb_used_data":56900,"kb_used_omap":12,"kb_used_meta":536691,"kb_avail":749367436,"statfs":{"total":767960285184,"available":767352254464,"internally_reserved":0,"allocated":58265600,"data_stored":40595283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":549572144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020406"},"pg_stats":[{"pgid":"254.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:46.220477+0000","last_change":"2024-03-30T15:00:46.220477+0000","last_active":"2024-03-30T15:00:46.220477+0000","last_peered":"2024-03-30T15:00:46.220477+0000","last_clean":"2024-03-30T15:00:46.220477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:46.220477+0000","last_undegraded":"2024-03-30T15:00:46.220477+0000","last_fullsized":"2024-03-30T15:00:46.220477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.220477+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.220477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":451,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233147+0000","last_change":"2024-03-30T15:00:43.233147+0000","last_active":"2024-03-30T15:00:43.233147+0000","last_peered":"2024-03-30T15:00:43.233147+0000","last_clean":"2024-03-30T15:00:43.233147+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:00:43.233147+0000","last_undegraded":"2024-03-30T15:00:43.233147+0000","last_fullsized":"2024-03-30T15:00:43.233147+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01065052,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1b","version":"229'85","reported_seq":482,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233337+0000","last_change":"2024-03-30T15:00:43.233337+0000","last_active":"2024-03-30T15:00:43.233337+0000","last_peered":"2024-03-30T15:00:43.233337+0000","last_clean":"2024-03-30T15:00:43.233337+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:00:43.233337+0000","last_undegraded":"2024-03-30T15:00:43.233337+0000","last_fullsized":"2024-03-30T15:00:43.233337+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025393439999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.18","version":"230'87","reported_seq":486,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222649+0000","last_change":"2024-03-30T15:00:43.224129+0000","last_active":"2024-03-30T15:00:43.222649+0000","last_peered":"2024-03-30T15:00:43.222649+0000","last_clean":"2024-03-30T15:00:43.222649+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:00:43.222649+0000","last_undegraded":"2024-03-30T15:00:43.222649+0000","last_fullsized":"2024-03-30T15:00:43.222649+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001687252,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.19","version":"204'102","reported_seq":503,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225470+0000","last_change":"2024-03-30T15:00:43.225813+0000","last_active":"2024-03-30T15:00:43.225470+0000","last_peered":"2024-03-30T15:00:43.225470+0000","last_clean":"2024-03-30T15:00:43.225470+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:00:43.225470+0000","last_undegraded":"2024-03-30T15:00:43.225470+0000","last_fullsized":"2024-03-30T15:00:43.225470+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001343522,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1e","version":"204'34","reported_seq":444,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229600+0000","last_change":"2024-03-30T15:00:43.231075+0000","last_active":"2024-03-30T15:00:43.229600+0000","last_peered":"2024-03-30T15:00:43.229600+0000","last_clean":"2024-03-30T15:00:43.229600+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:00:43.229600+0000","last_undegraded":"2024-03-30T15:00:43.229600+0000","last_fullsized":"2024-03-30T15:00:43.229600+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087880479999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1f","version":"279'62","reported_seq":521,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234913+0000","last_change":"2024-03-30T15:00:43.234913+0000","last_active":"2024-03-30T15:00:43.234913+0000","last_peered":"2024-03-30T15:00:43.234913+0000","last_clean":"2024-03-30T15:00:43.234913+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:00:43.234913+0000","last_undegraded":"2024-03-30T15:00:43.234913+0000","last_fullsized":"2024-03-30T15:00:43.234913+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011100334,"stat_sum":{"num_bytes":4194341,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1c","version":"222'34","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.235577+0000","last_change":"2024-03-30T15:00:43.235577+0000","last_active":"2024-03-30T15:00:43.235577+0000","last_peered":"2024-03-30T15:00:43.235577+0000","last_clean":"2024-03-30T15:00:43.235577+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:00:43.235577+0000","last_undegraded":"2024-03-30T15:00:43.235577+0000","last_fullsized":"2024-03-30T15:00:43.235577+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021958989999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1d","version":"204'51","reported_seq":455,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234880+0000","last_change":"2024-03-30T15:00:43.234880+0000","last_active":"2024-03-30T15:00:43.234880+0000","last_peered":"2024-03-30T15:00:43.234880+0000","last_clean":"2024-03-30T15:00:43.234880+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:00:43.234880+0000","last_undegraded":"2024-03-30T15:00:43.234880+0000","last_fullsized":"2024-03-30T15:00:43.234880+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095938800000000008,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.12","version":"235'39","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211992+0000","last_change":"2024-03-30T15:00:43.212808+0000","last_active":"2024-03-30T15:00:43.211992+0000","last_peered":"2024-03-30T15:00:43.211992+0000","last_clean":"2024-03-30T15:00:43.211992+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:00:43.211992+0000","last_undegraded":"2024-03-30T15:00:43.211992+0000","last_fullsized":"2024-03-30T15:00:43.211992+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034693419999999998,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.13","version":"237'34","reported_seq":444,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212906+0000","last_change":"2024-03-30T15:00:43.213650+0000","last_active":"2024-03-30T15:00:43.212906+0000","last_peered":"2024-03-30T15:00:43.212906+0000","last_clean":"2024-03-30T15:00:43.212906+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:00:43.212906+0000","last_undegraded":"2024-03-30T15:00:43.212906+0000","last_fullsized":"2024-03-30T15:00:43.212906+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041086289999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.10","version":"0'0","reported_seq":413,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225120+0000","last_change":"2024-03-30T15:00:43.225727+0000","last_active":"2024-03-30T15:00:43.225120+0000","last_peered":"2024-03-30T15:00:43.225120+0000","last_clean":"2024-03-30T15:00:43.225120+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:00:43.225120+0000","last_undegraded":"2024-03-30T15:00:43.225120+0000","last_fullsized":"2024-03-30T15:00:43.225120+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030227269999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.11","version":"204'17","reported_seq":430,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224256+0000","last_change":"2024-03-30T15:00:43.224756+0000","last_active":"2024-03-30T15:00:43.224256+0000","last_peered":"2024-03-30T15:00:43.224256+0000","last_clean":"2024-03-30T15:00:43.224256+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:00:43.224256+0000","last_undegraded":"2024-03-30T15:00:43.224256+0000","last_fullsized":"2024-03-30T15:00:43.224256+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020847700000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.16","version":"217'17","reported_seq":426,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.235456+0000","last_change":"2024-03-30T15:00:43.236048+0000","last_active":"2024-03-30T15:00:43.235456+0000","last_peered":"2024-03-30T15:00:43.235456+0000","last_clean":"2024-03-30T15:00:43.235456+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:00:43.235456+0000","last_undegraded":"2024-03-30T15:00:43.235456+0000","last_fullsized":"2024-03-30T15:00:43.235456+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013371539999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.17","version":"241'73","reported_seq":482,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216438+0000","last_change":"2024-03-30T15:00:43.217166+0000","last_active":"2024-03-30T15:00:43.216438+0000","last_peered":"2024-03-30T15:00:43.216438+0000","last_clean":"2024-03-30T15:00:43.216438+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:00:43.216438+0000","last_undegraded":"2024-03-30T15:00:43.216438+0000","last_fullsized":"2024-03-30T15:00:43.216438+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001827506,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.14","version":"233'51","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.213250+0000","last_change":"2024-03-30T15:00:43.214272+0000","last_active":"2024-03-30T15:00:43.213250+0000","last_peered":"2024-03-30T15:00:43.213250+0000","last_clean":"2024-03-30T15:00:43.213250+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:00:43.213250+0000","last_undegraded":"2024-03-30T15:00:43.213250+0000","last_fullsized":"2024-03-30T15:00:43.213250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00145184,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.15","version":"267'90","reported_seq":494,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230305+0000","last_change":"2024-03-30T15:00:43.231273+0000","last_active":"2024-03-30T15:00:43.230305+0000","last_peered":"2024-03-30T15:00:43.230305+0000","last_clean":"2024-03-30T15:00:43.230305+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:00:43.230305+0000","last_undegraded":"2024-03-30T15:00:43.230305+0000","last_fullsized":"2024-03-30T15:00:43.230305+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'90","last_scrub_stamp":"2024-03-30T15:00:41.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.653662+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.546074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0083710509999999991,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.a","version":"204'34","reported_seq":443,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227164+0000","last_change":"2024-03-30T15:00:43.227678+0000","last_active":"2024-03-30T15:00:43.227164+0000","last_peered":"2024-03-30T15:00:43.227164+0000","last_clean":"2024-03-30T15:00:43.227164+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:00:43.227164+0000","last_undegraded":"2024-03-30T15:00:43.227164+0000","last_fullsized":"2024-03-30T15:00:43.227164+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047107750000000004,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.b","version":"238'104","reported_seq":514,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.219542+0000","last_change":"2024-03-30T15:00:43.219728+0000","last_active":"2024-03-30T15:00:43.219542+0000","last_peered":"2024-03-30T15:00:43.219542+0000","last_clean":"2024-03-30T15:00:43.219542+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:00:43.219542+0000","last_undegraded":"2024-03-30T15:00:43.219542+0000","last_fullsized":"2024-03-30T15:00:43.219542+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:00:30.724310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.724310+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:37.476973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018219289999999999,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.8","version":"204'34","reported_seq":452,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226820+0000","last_change":"2024-03-30T15:00:43.228511+0000","last_active":"2024-03-30T15:00:43.226820+0000","last_peered":"2024-03-30T15:00:43.226820+0000","last_clean":"2024-03-30T15:00:43.226820+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:00:43.226820+0000","last_undegraded":"2024-03-30T15:00:43.226820+0000","last_fullsized":"2024-03-30T15:00:43.226820+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T14:59:47.247116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.247116+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.386865+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0019213380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.9","version":"254'37","reported_seq":457,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216860+0000","last_change":"2024-03-30T15:00:43.218761+0000","last_active":"2024-03-30T15:00:43.216860+0000","last_peered":"2024-03-30T15:00:43.216860+0000","last_clean":"2024-03-30T15:00:43.216860+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:00:43.216860+0000","last_undegraded":"2024-03-30T15:00:43.216860+0000","last_fullsized":"2024-03-30T15:00:43.216860+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:00:37.797402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.797402+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.144918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024570769999999998,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.e","version":"204'108","reported_seq":515,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.219332+0000","last_change":"2024-03-30T15:00:43.219557+0000","last_active":"2024-03-30T15:00:43.219332+0000","last_peered":"2024-03-30T15:00:43.219332+0000","last_clean":"2024-03-30T15:00:43.219332+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:00:43.219332+0000","last_undegraded":"2024-03-30T15:00:43.219332+0000","last_fullsized":"2024-03-30T15:00:43.219332+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:00:31.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:31.737537+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.754808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017474070000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.f","version":"248'56","reported_seq":468,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216724+0000","last_change":"2024-03-30T15:00:43.217112+0000","last_active":"2024-03-30T15:00:43.216724+0000","last_peered":"2024-03-30T15:00:43.216724+0000","last_clean":"2024-03-30T15:00:43.216724+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:00:43.216724+0000","last_undegraded":"2024-03-30T15:00:43.216724+0000","last_fullsized":"2024-03-30T15:00:43.216724+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00150314,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.c","version":"204'51","reported_seq":463,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229290+0000","last_change":"2024-03-30T15:00:43.232136+0000","last_active":"2024-03-30T15:00:43.229290+0000","last_peered":"2024-03-30T15:00:43.229290+0000","last_clean":"2024-03-30T15:00:43.229290+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:00:43.229290+0000","last_undegraded":"2024-03-30T15:00:43.229290+0000","last_fullsized":"2024-03-30T15:00:43.229290+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:00:29.555003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:29.555003+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:52.389904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074406610000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.d","version":"235'87","reported_seq":483,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234628+0000","last_change":"2024-03-30T15:00:43.234991+0000","last_active":"2024-03-30T15:00:43.234628+0000","last_peered":"2024-03-30T15:00:43.234628+0000","last_clean":"2024-03-30T15:00:43.234628+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:00:43.234628+0000","last_undegraded":"2024-03-30T15:00:43.234628+0000","last_fullsized":"2024-03-30T15:00:43.234628+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036646000000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.2","version":"204'85","reported_seq":491,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225821+0000","last_change":"2024-03-30T15:00:43.230968+0000","last_active":"2024-03-30T15:00:43.225821+0000","last_peered":"2024-03-30T15:00:43.225821+0000","last_clean":"2024-03-30T15:00:43.225821+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:00:43.225821+0000","last_undegraded":"2024-03-30T15:00:43.225821+0000","last_fullsized":"2024-03-30T15:00:43.225821+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'85","last_scrub_stamp":"2024-03-30T15:00:35.252370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.252370+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.684231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053428640000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.3","version":"248'39","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229490+0000","last_change":"2024-03-30T15:00:43.230248+0000","last_active":"2024-03-30T15:00:43.229490+0000","last_peered":"2024-03-30T15:00:43.229490+0000","last_clean":"2024-03-30T15:00:43.229490+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:00:43.229490+0000","last_undegraded":"2024-03-30T15:00:43.229490+0000","last_fullsized":"2024-03-30T15:00:43.229490+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:02.714228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.714228+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.613594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0084702249999999996,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.0","version":"248'39","reported_seq":457,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224166+0000","last_change":"2024-03-30T15:00:43.227563+0000","last_active":"2024-03-30T15:00:43.224166+0000","last_peered":"2024-03-30T15:00:43.224166+0000","last_clean":"2024-03-30T15:00:43.224166+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:00:43.224166+0000","last_undegraded":"2024-03-30T15:00:43.224166+0000","last_fullsized":"2024-03-30T15:00:43.224166+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:04.710962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.710962+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:28.195328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061256050000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.1","version":"238'103","reported_seq":517,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.219288+0000","last_change":"2024-03-30T15:00:43.219491+0000","last_active":"2024-03-30T15:00:43.219288+0000","last_peered":"2024-03-30T15:00:43.219288+0000","last_clean":"2024-03-30T15:00:43.219288+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:00:43.219288+0000","last_undegraded":"2024-03-30T15:00:43.219288+0000","last_fullsized":"2024-03-30T15:00:43.219288+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'103","last_scrub_stamp":"2024-03-30T15:00:35.786699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.786699+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.805956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022054919999999999,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.6","version":"204'22","reported_seq":437,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.235247+0000","last_change":"2024-03-30T15:00:43.235889+0000","last_active":"2024-03-30T15:00:43.235247+0000","last_peered":"2024-03-30T15:00:43.235247+0000","last_clean":"2024-03-30T15:00:43.235247+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:00:43.235247+0000","last_undegraded":"2024-03-30T15:00:43.235247+0000","last_fullsized":"2024-03-30T15:00:43.235247+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091356760000000006,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.7","version":"0'0","reported_seq":423,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233246+0000","last_change":"2024-03-30T15:00:43.234537+0000","last_active":"2024-03-30T15:00:43.233246+0000","last_peered":"2024-03-30T15:00:43.233246+0000","last_clean":"2024-03-30T15:00:43.233246+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:00:43.233246+0000","last_undegraded":"2024-03-30T15:00:43.233246+0000","last_fullsized":"2024-03-30T15:00:43.233246+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.644063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.644063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.017986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0083022960000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.4","version":"235'68","reported_seq":469,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.213751+0000","last_change":"2024-03-30T15:00:43.214361+0000","last_active":"2024-03-30T15:00:43.213751+0000","last_peered":"2024-03-30T15:00:43.213751+0000","last_clean":"2024-03-30T15:00:43.213751+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:00:43.213751+0000","last_undegraded":"2024-03-30T15:00:43.213751+0000","last_fullsized":"2024-03-30T15:00:43.213751+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001262364,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"101.5","version":"204'102","reported_seq":507,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232681+0000","last_change":"2024-03-30T15:00:43.233284+0000","last_active":"2024-03-30T15:00:43.232681+0000","last_peered":"2024-03-30T15:00:43.232681+0000","last_clean":"2024-03-30T15:00:43.232681+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:00:43.232681+0000","last_undegraded":"2024-03-30T15:00:43.232681+0000","last_fullsized":"2024-03-30T15:00:43.232681+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T14:59:48.278271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.278271+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:55.727532+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078062200000000002,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"b","length":"2"}]},{"pgid":"42.15","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228171+0000","last_change":"2024-03-30T14:57:56.252990+0000","last_active":"2024-03-30T15:00:43.228171+0000","last_peered":"2024-03-30T15:00:43.228171+0000","last_clean":"2024-03-30T15:00:43.228171+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:00:43.228171+0000","last_undegraded":"2024-03-30T15:00:43.228171+0000","last_fullsized":"2024-03-30T15:00:43.228171+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217092+0000","last_change":"2024-03-30T14:59:10.495352+0000","last_active":"2024-03-30T15:00:43.217092+0000","last_peered":"2024-03-30T15:00:43.217092+0000","last_clean":"2024-03-30T15:00:43.217092+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:00:43.217092+0000","last_undegraded":"2024-03-30T15:00:43.217092+0000","last_fullsized":"2024-03-30T15:00:43.217092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.495272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.495272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":347,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233262+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:00:43.233262+0000","last_peered":"2024-03-30T15:00:43.233262+0000","last_clean":"2024-03-30T15:00:43.233262+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:00:43.233262+0000","last_undegraded":"2024-03-30T15:00:43.233262+0000","last_fullsized":"2024-03-30T15:00:43.233262+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.937324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":340,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228816+0000","last_change":"2024-03-30T14:57:56.238237+0000","last_active":"2024-03-30T15:00:43.228816+0000","last_peered":"2024-03-30T15:00:43.228816+0000","last_clean":"2024-03-30T15:00:43.228816+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:00:43.228816+0000","last_undegraded":"2024-03-30T15:00:43.228816+0000","last_fullsized":"2024-03-30T15:00:43.228816+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":340,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209195+0000","last_change":"2024-03-30T14:57:56.261442+0000","last_active":"2024-03-30T15:00:43.209195+0000","last_peered":"2024-03-30T15:00:43.209195+0000","last_clean":"2024-03-30T15:00:43.209195+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:00:43.209195+0000","last_undegraded":"2024-03-30T15:00:43.209195+0000","last_fullsized":"2024-03-30T15:00:43.209195+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223756+0000","last_change":"2024-03-30T14:59:56.788394+0000","last_active":"2024-03-30T15:00:43.223756+0000","last_peered":"2024-03-30T15:00:43.223756+0000","last_clean":"2024-03-30T15:00:43.223756+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:00:43.223756+0000","last_undegraded":"2024-03-30T15:00:43.223756+0000","last_fullsized":"2024-03-30T15:00:43.223756+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.788278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.788278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:25.851592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231735+0000","last_change":"2024-03-30T14:57:56.244353+0000","last_active":"2024-03-30T15:00:43.231735+0000","last_peered":"2024-03-30T15:00:43.231735+0000","last_clean":"2024-03-30T15:00:43.231735+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:00:43.231735+0000","last_undegraded":"2024-03-30T15:00:43.231735+0000","last_fullsized":"2024-03-30T15:00:43.231735+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227558+0000","last_change":"2024-03-30T14:57:56.239220+0000","last_active":"2024-03-30T15:00:43.227558+0000","last_peered":"2024-03-30T15:00:43.227558+0000","last_clean":"2024-03-30T15:00:43.227558+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:00:43.227558+0000","last_undegraded":"2024-03-30T15:00:43.227558+0000","last_fullsized":"2024-03-30T15:00:43.227558+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225472+0000","last_change":"2024-03-30T14:57:56.240474+0000","last_active":"2024-03-30T15:00:43.225472+0000","last_peered":"2024-03-30T15:00:43.225472+0000","last_clean":"2024-03-30T15:00:43.225472+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:00:43.225472+0000","last_undegraded":"2024-03-30T15:00:43.225472+0000","last_fullsized":"2024-03-30T15:00:43.225472+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.330307+0000","last_change":"2024-03-30T15:00:43.330329+0000","last_active":"2024-03-30T15:00:43.330307+0000","last_peered":"2024-03-30T15:00:43.330307+0000","last_clean":"2024-03-30T15:00:43.330307+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:00:43.330307+0000","last_undegraded":"2024-03-30T15:00:43.330307+0000","last_fullsized":"2024-03-30T15:00:43.330307+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.330250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.998934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232950+0000","last_change":"2024-03-30T14:57:56.253494+0000","last_active":"2024-03-30T15:00:43.232950+0000","last_peered":"2024-03-30T15:00:43.232950+0000","last_clean":"2024-03-30T15:00:43.232950+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:00:43.232950+0000","last_undegraded":"2024-03-30T15:00:43.232950+0000","last_fullsized":"2024-03-30T15:00:43.232950+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222447+0000","last_change":"2024-03-30T14:59:52.803098+0000","last_active":"2024-03-30T15:00:43.222447+0000","last_peered":"2024-03-30T15:00:43.222447+0000","last_clean":"2024-03-30T15:00:43.222447+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:00:43.222447+0000","last_undegraded":"2024-03-30T15:00:43.222447+0000","last_fullsized":"2024-03-30T15:00:43.222447+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.803012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.803012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.827147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230115+0000","last_change":"2024-03-30T14:57:56.253984+0000","last_active":"2024-03-30T15:00:43.230115+0000","last_peered":"2024-03-30T15:00:43.230115+0000","last_clean":"2024-03-30T15:00:43.230115+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:00:43.230115+0000","last_undegraded":"2024-03-30T15:00:43.230115+0000","last_fullsized":"2024-03-30T15:00:43.230115+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223572+0000","last_change":"2024-03-30T15:00:40.305048+0000","last_active":"2024-03-30T15:00:43.223572+0000","last_peered":"2024-03-30T15:00:43.223572+0000","last_clean":"2024-03-30T15:00:43.223572+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:00:43.223572+0000","last_undegraded":"2024-03-30T15:00:43.223572+0000","last_fullsized":"2024-03-30T15:00:43.223572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:40.304903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:40.304903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.820755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222453+0000","last_change":"2024-03-30T14:59:17.774212+0000","last_active":"2024-03-30T15:00:43.222453+0000","last_peered":"2024-03-30T15:00:43.222453+0000","last_clean":"2024-03-30T15:00:43.222453+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:00:43.222453+0000","last_undegraded":"2024-03-30T15:00:43.222453+0000","last_fullsized":"2024-03-30T15:00:43.222453+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.774120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226269+0000","last_change":"2024-03-30T14:59:45.812010+0000","last_active":"2024-03-30T15:00:43.226269+0000","last_peered":"2024-03-30T15:00:43.226269+0000","last_clean":"2024-03-30T15:00:43.226269+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:00:43.226269+0000","last_undegraded":"2024-03-30T15:00:43.226269+0000","last_fullsized":"2024-03-30T15:00:43.226269+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:45.811929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:45.811929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.196406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231568+0000","last_change":"2024-03-30T14:57:56.248423+0000","last_active":"2024-03-30T15:00:43.231568+0000","last_peered":"2024-03-30T15:00:43.231568+0000","last_clean":"2024-03-30T15:00:43.231568+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:00:43.231568+0000","last_undegraded":"2024-03-30T15:00:43.231568+0000","last_fullsized":"2024-03-30T15:00:43.231568+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227353+0000","last_change":"2024-03-30T14:57:56.226093+0000","last_active":"2024-03-30T15:00:43.227353+0000","last_peered":"2024-03-30T15:00:43.227353+0000","last_clean":"2024-03-30T15:00:43.227353+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:00:43.227353+0000","last_undegraded":"2024-03-30T15:00:43.227353+0000","last_fullsized":"2024-03-30T15:00:43.227353+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233845+0000","last_change":"2024-03-30T14:57:56.276413+0000","last_active":"2024-03-30T15:00:43.233845+0000","last_peered":"2024-03-30T15:00:43.233845+0000","last_clean":"2024-03-30T15:00:43.233845+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:00:43.233845+0000","last_undegraded":"2024-03-30T15:00:43.233845+0000","last_fullsized":"2024-03-30T15:00:43.233845+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234747+0000","last_change":"2024-03-30T14:57:56.253085+0000","last_active":"2024-03-30T15:00:43.234747+0000","last_peered":"2024-03-30T15:00:43.234747+0000","last_clean":"2024-03-30T15:00:43.234747+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:00:43.234747+0000","last_undegraded":"2024-03-30T15:00:43.234747+0000","last_fullsized":"2024-03-30T15:00:43.234747+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224709+0000","last_change":"2024-03-30T14:59:55.177743+0000","last_active":"2024-03-30T15:00:43.224709+0000","last_peered":"2024-03-30T15:00:43.224709+0000","last_clean":"2024-03-30T15:00:43.224709+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:00:43.224709+0000","last_undegraded":"2024-03-30T15:00:43.224709+0000","last_fullsized":"2024-03-30T15:00:43.224709+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.177623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.177623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.770057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":347,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230027+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:00:43.230027+0000","last_peered":"2024-03-30T15:00:43.230027+0000","last_clean":"2024-03-30T15:00:43.230027+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:00:43.230027+0000","last_undegraded":"2024-03-30T15:00:43.230027+0000","last_fullsized":"2024-03-30T15:00:43.230027+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:00.729852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217725+0000","last_change":"2024-03-30T15:00:32.754787+0000","last_active":"2024-03-30T15:00:43.217725+0000","last_peered":"2024-03-30T15:00:43.217725+0000","last_clean":"2024-03-30T15:00:43.217725+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:00:43.217725+0000","last_undegraded":"2024-03-30T15:00:43.217725+0000","last_fullsized":"2024-03-30T15:00:43.217725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.754592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.754592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.958130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.221679+0000","last_change":"2024-03-30T14:59:13.804909+0000","last_active":"2024-03-30T15:00:43.221679+0000","last_peered":"2024-03-30T15:00:43.221679+0000","last_clean":"2024-03-30T15:00:43.221679+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:00:43.221679+0000","last_undegraded":"2024-03-30T15:00:43.221679+0000","last_fullsized":"2024-03-30T15:00:43.221679+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.804817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.804817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216868+0000","last_change":"2024-03-30T15:00:33.756912+0000","last_active":"2024-03-30T15:00:43.216868+0000","last_peered":"2024-03-30T15:00:43.216868+0000","last_clean":"2024-03-30T15:00:43.216868+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:00:43.216868+0000","last_undegraded":"2024-03-30T15:00:43.216868+0000","last_fullsized":"2024-03-30T15:00:43.216868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:33.756810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.756810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:33.994340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227901+0000","last_change":"2024-03-30T15:00:03.114650+0000","last_active":"2024-03-30T15:00:43.227901+0000","last_peered":"2024-03-30T15:00:43.227901+0000","last_clean":"2024-03-30T15:00:43.227901+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:00:43.227901+0000","last_undegraded":"2024-03-30T15:00:43.227901+0000","last_fullsized":"2024-03-30T15:00:43.227901+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.114576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.114576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:10.524652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234212+0000","last_change":"2024-03-30T14:57:56.260622+0000","last_active":"2024-03-30T15:00:43.234212+0000","last_peered":"2024-03-30T15:00:43.234212+0000","last_clean":"2024-03-30T15:00:43.234212+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:00:43.234212+0000","last_undegraded":"2024-03-30T15:00:43.234212+0000","last_fullsized":"2024-03-30T15:00:43.234212+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230217+0000","last_change":"2024-03-30T14:57:56.249378+0000","last_active":"2024-03-30T15:00:43.230217+0000","last_peered":"2024-03-30T15:00:43.230217+0000","last_clean":"2024-03-30T15:00:43.230217+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:00:43.230217+0000","last_undegraded":"2024-03-30T15:00:43.230217+0000","last_fullsized":"2024-03-30T15:00:43.230217+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215844+0000","last_change":"2024-03-30T14:59:11.544255+0000","last_active":"2024-03-30T15:00:43.215844+0000","last_peered":"2024-03-30T15:00:43.215844+0000","last_clean":"2024-03-30T15:00:43.215844+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:00:43.215844+0000","last_undegraded":"2024-03-30T15:00:43.215844+0000","last_fullsized":"2024-03-30T15:00:43.215844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:11.544174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:11.544174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225873+0000","last_change":"2024-03-30T14:57:56.271774+0000","last_active":"2024-03-30T15:00:43.225873+0000","last_peered":"2024-03-30T15:00:43.225873+0000","last_clean":"2024-03-30T15:00:43.225873+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:00:43.225873+0000","last_undegraded":"2024-03-30T15:00:43.225873+0000","last_fullsized":"2024-03-30T15:00:43.225873+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223962+0000","last_change":"2024-03-30T14:57:56.199752+0000","last_active":"2024-03-30T15:00:43.223962+0000","last_peered":"2024-03-30T15:00:43.223962+0000","last_clean":"2024-03-30T15:00:43.223962+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:00:43.223962+0000","last_undegraded":"2024-03-30T15:00:43.223962+0000","last_fullsized":"2024-03-30T15:00:43.223962+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230808+0000","last_change":"2024-03-30T14:57:56.228301+0000","last_active":"2024-03-30T15:00:43.230808+0000","last_peered":"2024-03-30T15:00:43.230808+0000","last_clean":"2024-03-30T15:00:43.230808+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:00:43.230808+0000","last_undegraded":"2024-03-30T15:00:43.230808+0000","last_fullsized":"2024-03-30T15:00:43.230808+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217243+0000","last_change":"2024-03-30T14:59:09.501246+0000","last_active":"2024-03-30T15:00:43.217243+0000","last_peered":"2024-03-30T15:00:43.217243+0000","last_clean":"2024-03-30T15:00:43.217243+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:00:43.217243+0000","last_undegraded":"2024-03-30T15:00:43.217243+0000","last_fullsized":"2024-03-30T15:00:43.217243+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.501169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.501169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227455+0000","last_change":"2024-03-30T14:58:15.280305+0000","last_active":"2024-03-30T15:00:43.227455+0000","last_peered":"2024-03-30T15:00:43.227455+0000","last_clean":"2024-03-30T15:00:43.227455+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:00:43.227455+0000","last_undegraded":"2024-03-30T15:00:43.227455+0000","last_fullsized":"2024-03-30T15:00:43.227455+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:15.280217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.280217+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":350,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211725+0000","last_change":"2024-03-30T14:58:13.189214+0000","last_active":"2024-03-30T15:00:43.211725+0000","last_peered":"2024-03-30T15:00:43.211725+0000","last_clean":"2024-03-30T15:00:43.211725+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:00:43.211725+0000","last_undegraded":"2024-03-30T15:00:43.211725+0000","last_fullsized":"2024-03-30T15:00:43.211725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.189081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.189081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":528,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223661+0000","last_change":"2024-03-30T14:58:31.806764+0000","last_active":"2024-03-30T15:00:43.223661+0000","last_peered":"2024-03-30T15:00:43.223661+0000","last_clean":"2024-03-30T15:00:43.223661+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:00:43.223661+0000","last_undegraded":"2024-03-30T15:00:43.223661+0000","last_fullsized":"2024-03-30T15:00:43.223661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:31.806606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.806606+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227598+0000","last_change":"2024-03-30T14:59:13.422572+0000","last_active":"2024-03-30T15:00:43.227598+0000","last_peered":"2024-03-30T15:00:43.227598+0000","last_clean":"2024-03-30T15:00:43.227598+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:43.227598+0000","last_undegraded":"2024-03-30T15:00:43.227598+0000","last_fullsized":"2024-03-30T15:00:43.227598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.422453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.422453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230661+0000","last_change":"2024-03-30T15:00:00.045570+0000","last_active":"2024-03-30T15:00:43.230661+0000","last_peered":"2024-03-30T15:00:43.230661+0000","last_clean":"2024-03-30T15:00:43.230661+0000","last_became_active":"2024-03-30T14:57:55.189173+0000","last_became_peered":"2024-03-30T14:57:55.189173+0000","last_unstale":"2024-03-30T15:00:43.230661+0000","last_undegraded":"2024-03-30T15:00:43.230661+0000","last_fullsized":"2024-03-30T15:00:43.230661+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.045488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.045488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.985958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217307+0000","last_change":"2024-03-30T15:00:04.553363+0000","last_active":"2024-03-30T15:00:43.217307+0000","last_peered":"2024-03-30T15:00:43.217307+0000","last_clean":"2024-03-30T15:00:43.217307+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:00:43.217307+0000","last_undegraded":"2024-03-30T15:00:43.217307+0000","last_fullsized":"2024-03-30T15:00:43.217307+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.553281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.553281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.976142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":479,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227193+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:00:43.227193+0000","last_peered":"2024-03-30T15:00:43.227193+0000","last_clean":"2024-03-30T15:00:43.227193+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:00:43.227193+0000","last_undegraded":"2024-03-30T15:00:43.227193+0000","last_fullsized":"2024-03-30T15:00:43.227193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:08.635027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":455,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216247+0000","last_change":"2024-03-30T14:59:18.646158+0000","last_active":"2024-03-30T15:00:43.216247+0000","last_peered":"2024-03-30T15:00:43.216247+0000","last_clean":"2024-03-30T15:00:43.216247+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:00:43.216247+0000","last_undegraded":"2024-03-30T15:00:43.216247+0000","last_fullsized":"2024-03-30T15:00:43.216247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:18.646076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:18.646076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":448,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228473+0000","last_change":"2024-03-30T14:59:09.325167+0000","last_active":"2024-03-30T15:00:43.228473+0000","last_peered":"2024-03-30T15:00:43.228473+0000","last_clean":"2024-03-30T15:00:43.228473+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:00:43.228473+0000","last_undegraded":"2024-03-30T15:00:43.228473+0000","last_fullsized":"2024-03-30T15:00:43.228473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226389+0000","last_change":"2024-03-30T14:59:07.831752+0000","last_active":"2024-03-30T15:00:43.226389+0000","last_peered":"2024-03-30T15:00:43.226389+0000","last_clean":"2024-03-30T15:00:43.226389+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:00:43.226389+0000","last_undegraded":"2024-03-30T15:00:43.226389+0000","last_fullsized":"2024-03-30T15:00:43.226389+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.831675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.831675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.18","version":"115'2","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211122+0000","last_change":"2024-03-30T14:57:54.060829+0000","last_active":"2024-03-30T15:00:43.211122+0000","last_peered":"2024-03-30T15:00:43.211122+0000","last_clean":"2024-03-30T15:00:43.211122+0000","last_became_active":"2024-03-30T14:57:54.060553+0000","last_became_peered":"2024-03-30T14:57:54.060553+0000","last_unstale":"2024-03-30T15:00:43.211122+0000","last_undegraded":"2024-03-30T15:00:43.211122+0000","last_fullsized":"2024-03-30T15:00:43.211122+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":354,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230527+0000","last_change":"2024-03-30T14:59:54.144187+0000","last_active":"2024-03-30T15:00:43.230527+0000","last_peered":"2024-03-30T15:00:43.230527+0000","last_clean":"2024-03-30T15:00:43.230527+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:00:43.230527+0000","last_undegraded":"2024-03-30T15:00:43.230527+0000","last_fullsized":"2024-03-30T15:00:43.230527+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:54.144084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:54.144084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:04.029603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227382+0000","last_change":"2024-03-30T14:58:20.234367+0000","last_active":"2024-03-30T15:00:43.227382+0000","last_peered":"2024-03-30T15:00:43.227382+0000","last_clean":"2024-03-30T15:00:43.227382+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:00:43.227382+0000","last_undegraded":"2024-03-30T15:00:43.227382+0000","last_fullsized":"2024-03-30T15:00:43.227382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:58:20.234281+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.234281+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223294+0000","last_change":"2024-03-30T14:57:29.596356+0000","last_active":"2024-03-30T15:00:43.223294+0000","last_peered":"2024-03-30T15:00:43.223294+0000","last_clean":"2024-03-30T15:00:43.223294+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:00:43.223294+0000","last_undegraded":"2024-03-30T15:00:43.223294+0000","last_fullsized":"2024-03-30T15:00:43.223294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:29.596249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:29.596249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233991+0000","last_change":"2024-03-30T14:58:23.099186+0000","last_active":"2024-03-30T15:00:43.233991+0000","last_peered":"2024-03-30T15:00:43.233991+0000","last_clean":"2024-03-30T15:00:43.233991+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:00:43.233991+0000","last_undegraded":"2024-03-30T15:00:43.233991+0000","last_fullsized":"2024-03-30T15:00:43.233991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.099104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.099104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217411+0000","last_change":"2024-03-30T14:58:11.247180+0000","last_active":"2024-03-30T15:00:43.217411+0000","last_peered":"2024-03-30T15:00:43.217411+0000","last_clean":"2024-03-30T15:00:43.217411+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:00:43.217411+0000","last_undegraded":"2024-03-30T15:00:43.217411+0000","last_fullsized":"2024-03-30T15:00:43.217411+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:11.247088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:11.247088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1b","version":"115'10","reported_seq":362,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224651+0000","last_change":"2024-03-30T15:00:38.719511+0000","last_active":"2024-03-30T15:00:43.224651+0000","last_peered":"2024-03-30T15:00:43.224651+0000","last_clean":"2024-03-30T15:00:43.224651+0000","last_became_active":"2024-03-30T14:57:55.179081+0000","last_became_peered":"2024-03-30T14:57:55.179081+0000","last_unstale":"2024-03-30T15:00:43.224651+0000","last_undegraded":"2024-03-30T15:00:43.224651+0000","last_fullsized":"2024-03-30T15:00:43.224651+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:38.719347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:38.719347+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.530355+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":354,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230764+0000","last_change":"2024-03-30T15:00:01.176797+0000","last_active":"2024-03-30T15:00:43.230764+0000","last_peered":"2024-03-30T15:00:43.230764+0000","last_clean":"2024-03-30T15:00:43.230764+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:00:43.230764+0000","last_undegraded":"2024-03-30T15:00:43.230764+0000","last_fullsized":"2024-03-30T15:00:43.230764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.176673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.080692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":476,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230869+0000","last_change":"2024-03-30T14:58:31.256279+0000","last_active":"2024-03-30T15:00:43.230869+0000","last_peered":"2024-03-30T15:00:43.230869+0000","last_clean":"2024-03-30T15:00:43.230869+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:00:43.230869+0000","last_undegraded":"2024-03-30T15:00:43.230869+0000","last_fullsized":"2024-03-30T15:00:43.230869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T14:58:31.256139+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.256139+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":350,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224773+0000","last_change":"2024-03-30T14:59:07.286916+0000","last_active":"2024-03-30T15:00:43.224773+0000","last_peered":"2024-03-30T15:00:43.224773+0000","last_clean":"2024-03-30T15:00:43.224773+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:00:43.224773+0000","last_undegraded":"2024-03-30T15:00:43.224773+0000","last_fullsized":"2024-03-30T15:00:43.224773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.286790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.286790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":521,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227490+0000","last_change":"2024-03-30T14:58:59.314094+0000","last_active":"2024-03-30T15:00:43.227490+0000","last_peered":"2024-03-30T15:00:43.227490+0000","last_clean":"2024-03-30T15:00:43.227490+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:00:43.227490+0000","last_undegraded":"2024-03-30T15:00:43.227490+0000","last_fullsized":"2024-03-30T15:00:43.227490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.314003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.314003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":434,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231860+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:00:43.231860+0000","last_peered":"2024-03-30T15:00:43.231860+0000","last_clean":"2024-03-30T15:00:43.231860+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:00:43.231860+0000","last_undegraded":"2024-03-30T15:00:43.231860+0000","last_fullsized":"2024-03-30T15:00:43.231860+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1a","version":"115'5","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209311+0000","last_change":"2024-03-30T14:57:54.064591+0000","last_active":"2024-03-30T15:00:43.209311+0000","last_peered":"2024-03-30T15:00:43.209311+0000","last_clean":"2024-03-30T15:00:43.209311+0000","last_became_active":"2024-03-30T14:57:54.063735+0000","last_became_peered":"2024-03-30T14:57:54.063735+0000","last_unstale":"2024-03-30T15:00:43.209311+0000","last_undegraded":"2024-03-30T15:00:43.209311+0000","last_fullsized":"2024-03-30T15:00:43.209311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226131+0000","last_change":"2024-03-30T14:59:53.801443+0000","last_active":"2024-03-30T15:00:43.226131+0000","last_peered":"2024-03-30T15:00:43.226131+0000","last_clean":"2024-03-30T15:00:43.226131+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:00:43.226131+0000","last_undegraded":"2024-03-30T15:00:43.226131+0000","last_fullsized":"2024-03-30T15:00:43.226131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.801332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.801332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.086851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223142+0000","last_change":"2024-03-30T14:58:21.250924+0000","last_active":"2024-03-30T15:00:43.223142+0000","last_peered":"2024-03-30T15:00:43.223142+0000","last_clean":"2024-03-30T15:00:43.223142+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:00:43.223142+0000","last_undegraded":"2024-03-30T15:00:43.223142+0000","last_fullsized":"2024-03-30T15:00:43.223142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:21.250827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.250827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227139+0000","last_change":"2024-03-30T14:58:19.963766+0000","last_active":"2024-03-30T15:00:43.227139+0000","last_peered":"2024-03-30T15:00:43.227139+0000","last_clean":"2024-03-30T15:00:43.227139+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:00:43.227139+0000","last_undegraded":"2024-03-30T15:00:43.227139+0000","last_fullsized":"2024-03-30T15:00:43.227139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:58:19.963665+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.963665+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":422,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233882+0000","last_change":"2024-03-30T14:59:10.014828+0000","last_active":"2024-03-30T15:00:43.233882+0000","last_peered":"2024-03-30T15:00:43.233882+0000","last_clean":"2024-03-30T15:00:43.233882+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:00:43.233882+0000","last_undegraded":"2024-03-30T15:00:43.233882+0000","last_fullsized":"2024-03-30T15:00:43.233882+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:10.014745+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.014745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223510+0000","last_change":"2024-03-30T14:59:02.261089+0000","last_active":"2024-03-30T15:00:43.223510+0000","last_peered":"2024-03-30T15:00:43.223510+0000","last_clean":"2024-03-30T15:00:43.223510+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:00:43.223510+0000","last_undegraded":"2024-03-30T15:00:43.223510+0000","last_fullsized":"2024-03-30T15:00:43.223510+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.260964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.260964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229571+0000","last_change":"2024-03-30T14:59:55.119990+0000","last_active":"2024-03-30T15:00:43.229571+0000","last_peered":"2024-03-30T15:00:43.229571+0000","last_clean":"2024-03-30T15:00:43.229571+0000","last_became_active":"2024-03-30T14:57:55.209239+0000","last_became_peered":"2024-03-30T14:57:55.209239+0000","last_unstale":"2024-03-30T15:00:43.229571+0000","last_undegraded":"2024-03-30T15:00:43.229571+0000","last_fullsized":"2024-03-30T15:00:43.229571+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.119871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.119871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:24.874887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.221667+0000","last_change":"2024-03-30T14:59:48.794962+0000","last_active":"2024-03-30T15:00:43.221667+0000","last_peered":"2024-03-30T15:00:43.221667+0000","last_clean":"2024-03-30T15:00:43.221667+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:00:43.221667+0000","last_undegraded":"2024-03-30T15:00:43.221667+0000","last_fullsized":"2024-03-30T15:00:43.221667+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:48.794824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.794824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.182803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":408,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224001+0000","last_change":"2024-03-30T14:59:49.053482+0000","last_active":"2024-03-30T15:00:43.224001+0000","last_peered":"2024-03-30T15:00:43.224001+0000","last_clean":"2024-03-30T15:00:43.224001+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:00:43.224001+0000","last_undegraded":"2024-03-30T15:00:43.224001+0000","last_fullsized":"2024-03-30T15:00:43.224001+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:49.053394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:49.053394+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.339042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":472,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223485+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:00:43.223485+0000","last_peered":"2024-03-30T15:00:43.223485+0000","last_clean":"2024-03-30T15:00:43.223485+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:00:43.223485+0000","last_undegraded":"2024-03-30T15:00:43.223485+0000","last_fullsized":"2024-03-30T15:00:43.223485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.455990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":450,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215767+0000","last_change":"2024-03-30T14:58:25.170349+0000","last_active":"2024-03-30T15:00:43.215767+0000","last_peered":"2024-03-30T15:00:43.215767+0000","last_clean":"2024-03-30T15:00:43.215767+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:00:43.215767+0000","last_undegraded":"2024-03-30T15:00:43.215767+0000","last_fullsized":"2024-03-30T15:00:43.215767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.170239+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":443,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230359+0000","last_change":"2024-03-30T14:59:56.291563+0000","last_active":"2024-03-30T15:00:43.230359+0000","last_peered":"2024-03-30T15:00:43.230359+0000","last_clean":"2024-03-30T15:00:43.230359+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:00:43.230359+0000","last_undegraded":"2024-03-30T15:00:43.230359+0000","last_fullsized":"2024-03-30T15:00:43.230359+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.291453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.291453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.028131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"115'2","reported_seq":342,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227666+0000","last_change":"2024-03-30T14:57:56.220489+0000","last_active":"2024-03-30T15:00:43.227666+0000","last_peered":"2024-03-30T15:00:43.227666+0000","last_clean":"2024-03-30T15:00:43.227666+0000","last_became_active":"2024-03-30T14:57:56.219719+0000","last_became_peered":"2024-03-30T14:57:56.219719+0000","last_unstale":"2024-03-30T15:00:43.227666+0000","last_undegraded":"2024-03-30T15:00:43.227666+0000","last_fullsized":"2024-03-30T15:00:43.227666+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.221890+0000","last_change":"2024-03-30T14:59:46.819021+0000","last_active":"2024-03-30T15:00:43.221890+0000","last_peered":"2024-03-30T15:00:43.221890+0000","last_clean":"2024-03-30T15:00:43.221890+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:00:43.221890+0000","last_undegraded":"2024-03-30T15:00:43.221890+0000","last_fullsized":"2024-03-30T15:00:43.221890+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:46.818941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.818941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:52.638110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":530,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211622+0000","last_change":"2024-03-30T15:00:30.490223+0000","last_active":"2024-03-30T15:00:43.211622+0000","last_peered":"2024-03-30T15:00:43.211622+0000","last_clean":"2024-03-30T15:00:43.211622+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:00:43.211622+0000","last_undegraded":"2024-03-30T15:00:43.211622+0000","last_fullsized":"2024-03-30T15:00:43.211622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:00:30.490104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.490104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.592628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":472,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228022+0000","last_change":"2024-03-30T14:59:10.283945+0000","last_active":"2024-03-30T15:00:43.228022+0000","last_peered":"2024-03-30T15:00:43.228022+0000","last_clean":"2024-03-30T15:00:43.228022+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:00:43.228022+0000","last_undegraded":"2024-03-30T15:00:43.228022+0000","last_fullsized":"2024-03-30T15:00:43.228022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.283859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232161+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:00:43.232161+0000","last_peered":"2024-03-30T15:00:43.232161+0000","last_clean":"2024-03-30T15:00:43.232161+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:00:43.232161+0000","last_undegraded":"2024-03-30T15:00:43.232161+0000","last_fullsized":"2024-03-30T15:00:43.232161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209010+0000","last_change":"2024-03-30T14:58:23.157209+0000","last_active":"2024-03-30T15:00:43.209010+0000","last_peered":"2024-03-30T15:00:43.209010+0000","last_clean":"2024-03-30T15:00:43.209010+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:00:43.209010+0000","last_undegraded":"2024-03-30T15:00:43.209010+0000","last_fullsized":"2024-03-30T15:00:43.209010+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.157095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.157095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"115'5","reported_seq":358,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217651+0000","last_change":"2024-03-30T14:59:15.591705+0000","last_active":"2024-03-30T15:00:43.217651+0000","last_peered":"2024-03-30T15:00:43.217651+0000","last_clean":"2024-03-30T15:00:43.217651+0000","last_became_active":"2024-03-30T14:57:55.239156+0000","last_became_peered":"2024-03-30T14:57:55.239156+0000","last_unstale":"2024-03-30T15:00:43.217651+0000","last_undegraded":"2024-03-30T15:00:43.217651+0000","last_fullsized":"2024-03-30T15:00:43.217651+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T14:59:15.591612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.591612+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223520+0000","last_change":"2024-03-30T14:59:19.788455+0000","last_active":"2024-03-30T15:00:43.223520+0000","last_peered":"2024-03-30T15:00:43.223520+0000","last_clean":"2024-03-30T15:00:43.223520+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:00:43.223520+0000","last_undegraded":"2024-03-30T15:00:43.223520+0000","last_fullsized":"2024-03-30T15:00:43.223520+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.788329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.788329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216548+0000","last_change":"2024-03-30T14:59:01.518493+0000","last_active":"2024-03-30T15:00:43.216548+0000","last_peered":"2024-03-30T15:00:43.216548+0000","last_clean":"2024-03-30T15:00:43.216548+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:00:43.216548+0000","last_undegraded":"2024-03-30T15:00:43.216548+0000","last_fullsized":"2024-03-30T15:00:43.216548+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.518414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.518414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":488,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212320+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:00:43.212320+0000","last_peered":"2024-03-30T15:00:43.212320+0000","last_clean":"2024-03-30T15:00:43.212320+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:00:43.212320+0000","last_undegraded":"2024-03-30T15:00:43.212320+0000","last_fullsized":"2024-03-30T15:00:43.212320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228383+0000","last_change":"2024-03-30T14:59:13.210437+0000","last_active":"2024-03-30T15:00:43.228383+0000","last_peered":"2024-03-30T15:00:43.228383+0000","last_clean":"2024-03-30T15:00:43.228383+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:00:43.228383+0000","last_undegraded":"2024-03-30T15:00:43.228383+0000","last_fullsized":"2024-03-30T15:00:43.228383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.210302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.210302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":444,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224888+0000","last_change":"2024-03-30T14:59:19.135083+0000","last_active":"2024-03-30T15:00:43.224888+0000","last_peered":"2024-03-30T15:00:43.224888+0000","last_clean":"2024-03-30T15:00:43.224888+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:00:43.224888+0000","last_undegraded":"2024-03-30T15:00:43.224888+0000","last_fullsized":"2024-03-30T15:00:43.224888+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.135006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.135006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"115'5","reported_seq":357,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.235221+0000","last_change":"2024-03-30T15:00:35.109378+0000","last_active":"2024-03-30T15:00:43.235221+0000","last_peered":"2024-03-30T15:00:43.235221+0000","last_clean":"2024-03-30T15:00:43.235221+0000","last_became_active":"2024-03-30T14:57:55.164637+0000","last_became_peered":"2024-03-30T14:57:55.164637+0000","last_unstale":"2024-03-30T15:00:43.235221+0000","last_undegraded":"2024-03-30T15:00:43.235221+0000","last_fullsized":"2024-03-30T15:00:43.235221+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.109233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.109233+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.628853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215668+0000","last_change":"2024-03-30T14:57:54.026635+0000","last_active":"2024-03-30T15:00:43.215668+0000","last_peered":"2024-03-30T15:00:43.215668+0000","last_clean":"2024-03-30T15:00:43.215668+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:00:43.215668+0000","last_undegraded":"2024-03-30T15:00:43.215668+0000","last_fullsized":"2024-03-30T15:00:43.215668+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234087+0000","last_change":"2024-03-30T14:59:15.207502+0000","last_active":"2024-03-30T15:00:43.234087+0000","last_peered":"2024-03-30T15:00:43.234087+0000","last_clean":"2024-03-30T15:00:43.234087+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:00:43.234087+0000","last_undegraded":"2024-03-30T15:00:43.234087+0000","last_fullsized":"2024-03-30T15:00:43.234087+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.207413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.207413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":470,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215560+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:00:43.215560+0000","last_peered":"2024-03-30T15:00:43.215560+0000","last_clean":"2024-03-30T15:00:43.215560+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:00:43.215560+0000","last_undegraded":"2024-03-30T15:00:43.215560+0000","last_fullsized":"2024-03-30T15:00:43.215560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:59.702850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":819,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226853+0000","last_change":"2024-03-30T14:58:21.947917+0000","last_active":"2024-03-30T15:00:43.226853+0000","last_peered":"2024-03-30T15:00:43.226853+0000","last_clean":"2024-03-30T15:00:43.226853+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:00:43.226853+0000","last_undegraded":"2024-03-30T15:00:43.226853+0000","last_fullsized":"2024-03-30T15:00:43.226853+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:58:21.947837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.947837+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":448,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208929+0000","last_change":"2024-03-30T14:59:05.612363+0000","last_active":"2024-03-30T15:00:43.208929+0000","last_peered":"2024-03-30T15:00:43.208929+0000","last_clean":"2024-03-30T15:00:43.208929+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:00:43.208929+0000","last_undegraded":"2024-03-30T15:00:43.208929+0000","last_fullsized":"2024-03-30T15:00:43.208929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.612262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.612262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228218+0000","last_change":"2024-03-30T14:58:20.292759+0000","last_active":"2024-03-30T15:00:43.228218+0000","last_peered":"2024-03-30T15:00:43.228218+0000","last_clean":"2024-03-30T15:00:43.228218+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:00:43.228218+0000","last_undegraded":"2024-03-30T15:00:43.228218+0000","last_fullsized":"2024-03-30T15:00:43.228218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:20.292655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.292655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'9","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228134+0000","last_change":"2024-03-30T14:57:56.274073+0000","last_active":"2024-03-30T15:00:43.228134+0000","last_peered":"2024-03-30T15:00:43.228134+0000","last_clean":"2024-03-30T15:00:43.228134+0000","last_became_active":"2024-03-30T14:57:56.272633+0000","last_became_peered":"2024-03-30T14:57:56.272633+0000","last_unstale":"2024-03-30T15:00:43.228134+0000","last_undegraded":"2024-03-30T15:00:43.228134+0000","last_fullsized":"2024-03-30T15:00:43.228134+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":344,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229794+0000","last_change":"2024-03-30T14:57:55.140517+0000","last_active":"2024-03-30T15:00:43.229794+0000","last_peered":"2024-03-30T15:00:43.229794+0000","last_clean":"2024-03-30T15:00:43.229794+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:00:43.229794+0000","last_undegraded":"2024-03-30T15:00:43.229794+0000","last_fullsized":"2024-03-30T15:00:43.229794+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231919+0000","last_change":"2024-03-30T14:58:35.074272+0000","last_active":"2024-03-30T15:00:43.231919+0000","last_peered":"2024-03-30T15:00:43.231919+0000","last_clean":"2024-03-30T15:00:43.231919+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:00:43.231919+0000","last_undegraded":"2024-03-30T15:00:43.231919+0000","last_fullsized":"2024-03-30T15:00:43.231919+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:35.074185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.074185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":555,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217569+0000","last_change":"2024-03-30T15:00:39.833443+0000","last_active":"2024-03-30T15:00:43.217569+0000","last_peered":"2024-03-30T15:00:43.217569+0000","last_clean":"2024-03-30T15:00:43.217569+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:00:43.217569+0000","last_undegraded":"2024-03-30T15:00:43.217569+0000","last_fullsized":"2024-03-30T15:00:43.217569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:00:39.833329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:39.833329+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.420557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231059+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:00:43.231059+0000","last_peered":"2024-03-30T15:00:43.231059+0000","last_clean":"2024-03-30T15:00:43.231059+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:00:43.231059+0000","last_undegraded":"2024-03-30T15:00:43.231059+0000","last_fullsized":"2024-03-30T15:00:43.231059+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":706,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226908+0000","last_change":"2024-03-30T14:59:14.976524+0000","last_active":"2024-03-30T15:00:43.226908+0000","last_peered":"2024-03-30T15:00:43.226908+0000","last_clean":"2024-03-30T15:00:43.226908+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:00:43.226908+0000","last_undegraded":"2024-03-30T15:00:43.226908+0000","last_fullsized":"2024-03-30T15:00:43.226908+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T14:59:14.976438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.976438+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":434,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233633+0000","last_change":"2024-03-30T14:59:05.166302+0000","last_active":"2024-03-30T15:00:43.233633+0000","last_peered":"2024-03-30T15:00:43.233633+0000","last_clean":"2024-03-30T15:00:43.233633+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:00:43.233633+0000","last_undegraded":"2024-03-30T15:00:43.233633+0000","last_fullsized":"2024-03-30T15:00:43.233633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.166200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.166200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":340,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226890+0000","last_change":"2024-03-30T14:57:56.273818+0000","last_active":"2024-03-30T15:00:43.226890+0000","last_peered":"2024-03-30T15:00:43.226890+0000","last_clean":"2024-03-30T15:00:43.226890+0000","last_became_active":"2024-03-30T14:57:56.273501+0000","last_became_peered":"2024-03-30T14:57:56.273501+0000","last_unstale":"2024-03-30T15:00:43.226890+0000","last_undegraded":"2024-03-30T15:00:43.226890+0000","last_fullsized":"2024-03-30T15:00:43.226890+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217137+0000","last_change":"2024-03-30T14:59:53.656204+0000","last_active":"2024-03-30T15:00:43.217137+0000","last_peered":"2024-03-30T15:00:43.217137+0000","last_clean":"2024-03-30T15:00:43.217137+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:00:43.217137+0000","last_undegraded":"2024-03-30T15:00:43.217137+0000","last_fullsized":"2024-03-30T15:00:43.217137+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.656111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.656111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:54.708968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224028+0000","last_change":"2024-03-30T14:59:03.841069+0000","last_active":"2024-03-30T15:00:43.224028+0000","last_peered":"2024-03-30T15:00:43.224028+0000","last_clean":"2024-03-30T15:00:43.224028+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:00:43.224028+0000","last_undegraded":"2024-03-30T15:00:43.224028+0000","last_fullsized":"2024-03-30T15:00:43.224028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.840945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.840945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":484,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230662+0000","last_change":"2024-03-30T14:59:51.039814+0000","last_active":"2024-03-30T15:00:43.230662+0000","last_peered":"2024-03-30T15:00:43.230662+0000","last_clean":"2024-03-30T15:00:43.230662+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:00:43.230662+0000","last_undegraded":"2024-03-30T15:00:43.230662+0000","last_fullsized":"2024-03-30T15:00:43.230662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T14:59:51.039715+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.039715+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.286012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.221829+0000","last_change":"2024-03-30T14:58:58.777774+0000","last_active":"2024-03-30T15:00:43.221829+0000","last_peered":"2024-03-30T15:00:43.221829+0000","last_clean":"2024-03-30T15:00:43.221829+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:00:43.221829+0000","last_undegraded":"2024-03-30T15:00:43.221829+0000","last_fullsized":"2024-03-30T15:00:43.221829+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.777653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":457,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224540+0000","last_change":"2024-03-30T14:59:22.144499+0000","last_active":"2024-03-30T15:00:43.224540+0000","last_peered":"2024-03-30T15:00:43.224540+0000","last_clean":"2024-03-30T15:00:43.224540+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:00:43.224540+0000","last_undegraded":"2024-03-30T15:00:43.224540+0000","last_fullsized":"2024-03-30T15:00:43.224540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:22.144389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:59:22.144389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":443,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232521+0000","last_change":"2024-03-30T14:59:55.270674+0000","last_active":"2024-03-30T15:00:43.232521+0000","last_peered":"2024-03-30T15:00:43.232521+0000","last_clean":"2024-03-30T15:00:43.232521+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:00:43.232521+0000","last_undegraded":"2024-03-30T15:00:43.232521+0000","last_fullsized":"2024-03-30T15:00:43.232521+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.270595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.270595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.865117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230998+0000","last_change":"2024-03-30T15:00:04.031901+0000","last_active":"2024-03-30T15:00:43.230998+0000","last_peered":"2024-03-30T15:00:43.230998+0000","last_clean":"2024-03-30T15:00:43.230998+0000","last_became_active":"2024-03-30T14:57:56.247871+0000","last_became_peered":"2024-03-30T14:57:56.247871+0000","last_unstale":"2024-03-30T15:00:43.230998+0000","last_undegraded":"2024-03-30T15:00:43.230998+0000","last_fullsized":"2024-03-30T15:00:43.230998+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.031690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.031690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:22.542186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.213294+0000","last_change":"2024-03-30T14:57:54.024875+0000","last_active":"2024-03-30T15:00:43.213294+0000","last_peered":"2024-03-30T15:00:43.213294+0000","last_clean":"2024-03-30T15:00:43.213294+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:00:43.213294+0000","last_undegraded":"2024-03-30T15:00:43.213294+0000","last_fullsized":"2024-03-30T15:00:43.213294+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212747+0000","last_change":"2024-03-30T14:57:24.232434+0000","last_active":"2024-03-30T15:00:43.212747+0000","last_peered":"2024-03-30T15:00:43.212747+0000","last_clean":"2024-03-30T15:00:43.212747+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:00:43.212747+0000","last_undegraded":"2024-03-30T15:00:43.212747+0000","last_fullsized":"2024-03-30T15:00:43.212747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:24.232315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:57:24.232315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":519,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233535+0000","last_change":"2024-03-30T14:58:22.340066+0000","last_active":"2024-03-30T15:00:43.233535+0000","last_peered":"2024-03-30T15:00:43.233535+0000","last_clean":"2024-03-30T15:00:43.233535+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:00:43.233535+0000","last_undegraded":"2024-03-30T15:00:43.233535+0000","last_fullsized":"2024-03-30T15:00:43.233535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:58:22.339962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.339962+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224298+0000","last_change":"2024-03-30T14:58:19.237610+0000","last_active":"2024-03-30T15:00:43.224298+0000","last_peered":"2024-03-30T15:00:43.224298+0000","last_clean":"2024-03-30T15:00:43.224298+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:00:43.224298+0000","last_undegraded":"2024-03-30T15:00:43.224298+0000","last_fullsized":"2024-03-30T15:00:43.224298+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.237530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.237530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":457,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231107+0000","last_change":"2024-03-30T14:58:36.096934+0000","last_active":"2024-03-30T15:00:43.231107+0000","last_peered":"2024-03-30T15:00:43.231107+0000","last_clean":"2024-03-30T15:00:43.231107+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:00:43.231107+0000","last_undegraded":"2024-03-30T15:00:43.231107+0000","last_fullsized":"2024-03-30T15:00:43.231107+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T14:58:36.096812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.096812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227052+0000","last_change":"2024-03-30T14:58:22.249707+0000","last_active":"2024-03-30T15:00:43.227052+0000","last_peered":"2024-03-30T15:00:43.227052+0000","last_clean":"2024-03-30T15:00:43.227052+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:00:43.227052+0000","last_undegraded":"2024-03-30T15:00:43.227052+0000","last_fullsized":"2024-03-30T15:00:43.227052+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.249628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.249628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.11","version":"115'7","reported_seq":347,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223695+0000","last_change":"2024-03-30T14:57:56.224803+0000","last_active":"2024-03-30T15:00:43.223695+0000","last_peered":"2024-03-30T15:00:43.223695+0000","last_clean":"2024-03-30T15:00:43.223695+0000","last_became_active":"2024-03-30T14:57:56.224505+0000","last_became_peered":"2024-03-30T14:57:56.224505+0000","last_unstale":"2024-03-30T15:00:43.223695+0000","last_undegraded":"2024-03-30T15:00:43.223695+0000","last_fullsized":"2024-03-30T15:00:43.223695+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211933+0000","last_change":"2024-03-30T14:57:54.046311+0000","last_active":"2024-03-30T15:00:43.211933+0000","last_peered":"2024-03-30T15:00:43.211933+0000","last_clean":"2024-03-30T15:00:43.211933+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:00:43.211933+0000","last_undegraded":"2024-03-30T15:00:43.211933+0000","last_fullsized":"2024-03-30T15:00:43.211933+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":468,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211817+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:00:43.211817+0000","last_peered":"2024-03-30T15:00:43.211817+0000","last_clean":"2024-03-30T15:00:43.211817+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:00:43.211817+0000","last_undegraded":"2024-03-30T15:00:43.211817+0000","last_fullsized":"2024-03-30T15:00:43.211817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:39.060252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234032+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:00:43.234032+0000","last_peered":"2024-03-30T15:00:43.234032+0000","last_clean":"2024-03-30T15:00:43.234032+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:00:43.234032+0000","last_undegraded":"2024-03-30T15:00:43.234032+0000","last_fullsized":"2024-03-30T15:00:43.234032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.989386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222062+0000","last_change":"2024-03-30T14:58:30.843293+0000","last_active":"2024-03-30T15:00:43.222062+0000","last_peered":"2024-03-30T15:00:43.222062+0000","last_clean":"2024-03-30T15:00:43.222062+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:00:43.222062+0000","last_undegraded":"2024-03-30T15:00:43.222062+0000","last_fullsized":"2024-03-30T15:00:43.222062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:30.843163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:30.843163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209154+0000","last_change":"2024-03-30T14:58:36.234502+0000","last_active":"2024-03-30T15:00:43.209154+0000","last_peered":"2024-03-30T15:00:43.209154+0000","last_clean":"2024-03-30T15:00:43.209154+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:00:43.209154+0000","last_undegraded":"2024-03-30T15:00:43.209154+0000","last_fullsized":"2024-03-30T15:00:43.209154+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:36.234425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.234425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231096+0000","last_change":"2024-03-30T15:00:41.329527+0000","last_active":"2024-03-30T15:00:43.231096+0000","last_peered":"2024-03-30T15:00:43.231096+0000","last_clean":"2024-03-30T15:00:43.231096+0000","last_became_active":"2024-03-30T14:57:54.085820+0000","last_became_peered":"2024-03-30T14:57:54.085820+0000","last_unstale":"2024-03-30T15:00:43.231096+0000","last_undegraded":"2024-03-30T15:00:43.231096+0000","last_fullsized":"2024-03-30T15:00:43.231096+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:41.329383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.329383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:54.482215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":344,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223885+0000","last_change":"2024-03-30T14:57:55.074016+0000","last_active":"2024-03-30T15:00:43.223885+0000","last_peered":"2024-03-30T15:00:43.223885+0000","last_clean":"2024-03-30T15:00:43.223885+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:00:43.223885+0000","last_undegraded":"2024-03-30T15:00:43.223885+0000","last_fullsized":"2024-03-30T15:00:43.223885+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":508,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216440+0000","last_change":"2024-03-30T14:59:00.517828+0000","last_active":"2024-03-30T15:00:43.216440+0000","last_peered":"2024-03-30T15:00:43.216440+0000","last_clean":"2024-03-30T15:00:43.216440+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:00:43.216440+0000","last_undegraded":"2024-03-30T15:00:43.216440+0000","last_fullsized":"2024-03-30T15:00:43.216440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:59:00.517731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:00.517731+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232320+0000","last_change":"2024-03-30T14:58:23.951589+0000","last_active":"2024-03-30T15:00:43.232320+0000","last_peered":"2024-03-30T15:00:43.232320+0000","last_clean":"2024-03-30T15:00:43.232320+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:00:43.232320+0000","last_undegraded":"2024-03-30T15:00:43.232320+0000","last_fullsized":"2024-03-30T15:00:43.232320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.951507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223354+0000","last_change":"2024-03-30T14:58:33.889037+0000","last_active":"2024-03-30T15:00:43.223354+0000","last_peered":"2024-03-30T15:00:43.223354+0000","last_clean":"2024-03-30T15:00:43.223354+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:00:43.223354+0000","last_undegraded":"2024-03-30T15:00:43.223354+0000","last_fullsized":"2024-03-30T15:00:43.223354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:33.888931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:33.888931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226732+0000","last_change":"2024-03-30T14:59:12.985700+0000","last_active":"2024-03-30T15:00:43.226732+0000","last_peered":"2024-03-30T15:00:43.226732+0000","last_clean":"2024-03-30T15:00:43.226732+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:00:43.226732+0000","last_undegraded":"2024-03-30T15:00:43.226732+0000","last_fullsized":"2024-03-30T15:00:43.226732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.985620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.985620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1f","version":"115'5","reported_seq":358,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217747+0000","last_change":"2024-03-30T14:59:55.698533+0000","last_active":"2024-03-30T15:00:43.217747+0000","last_peered":"2024-03-30T15:00:43.217747+0000","last_clean":"2024-03-30T15:00:43.217747+0000","last_became_active":"2024-03-30T14:57:55.189358+0000","last_became_peered":"2024-03-30T14:57:55.189358+0000","last_unstale":"2024-03-30T15:00:43.217747+0000","last_undegraded":"2024-03-30T15:00:43.217747+0000","last_fullsized":"2024-03-30T15:00:43.217747+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T14:59:55.698445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.698445+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.398571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":366,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216129+0000","last_change":"2024-03-30T14:59:51.656251+0000","last_active":"2024-03-30T15:00:43.216129+0000","last_peered":"2024-03-30T15:00:43.216129+0000","last_clean":"2024-03-30T15:00:43.216129+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:00:43.216129+0000","last_undegraded":"2024-03-30T15:00:43.216129+0000","last_fullsized":"2024-03-30T15:00:43.216129+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T14:59:51.656163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.656163+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.934763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224034+0000","last_change":"2024-03-30T14:59:08.794368+0000","last_active":"2024-03-30T15:00:43.224034+0000","last_peered":"2024-03-30T15:00:43.224034+0000","last_clean":"2024-03-30T15:00:43.224034+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:00:43.224034+0000","last_undegraded":"2024-03-30T15:00:43.224034+0000","last_fullsized":"2024-03-30T15:00:43.224034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.794288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.794288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":474,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223607+0000","last_change":"2024-03-30T14:58:25.223144+0000","last_active":"2024-03-30T15:00:43.223607+0000","last_peered":"2024-03-30T15:00:43.223607+0000","last_clean":"2024-03-30T15:00:43.223607+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:00:43.223607+0000","last_undegraded":"2024-03-30T15:00:43.223607+0000","last_fullsized":"2024-03-30T15:00:43.223607+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:25.223044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.223044+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230373+0000","last_change":"2024-03-30T14:58:25.984629+0000","last_active":"2024-03-30T15:00:43.230373+0000","last_peered":"2024-03-30T15:00:43.230373+0000","last_clean":"2024-03-30T15:00:43.230373+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:00:43.230373+0000","last_undegraded":"2024-03-30T15:00:43.230373+0000","last_fullsized":"2024-03-30T15:00:43.230373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.984552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.984552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230518+0000","last_change":"2024-03-30T15:00:02.270908+0000","last_active":"2024-03-30T15:00:43.230518+0000","last_peered":"2024-03-30T15:00:43.230518+0000","last_clean":"2024-03-30T15:00:43.230518+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:00:43.230518+0000","last_undegraded":"2024-03-30T15:00:43.230518+0000","last_fullsized":"2024-03-30T15:00:43.230518+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.270821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.039536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228230+0000","last_change":"2024-03-30T15:00:02.153599+0000","last_active":"2024-03-30T15:00:43.228230+0000","last_peered":"2024-03-30T15:00:43.228230+0000","last_clean":"2024-03-30T15:00:43.228230+0000","last_became_active":"2024-03-30T14:57:54.052219+0000","last_became_peered":"2024-03-30T14:57:54.052219+0000","last_unstale":"2024-03-30T15:00:43.228230+0000","last_undegraded":"2024-03-30T15:00:43.228230+0000","last_fullsized":"2024-03-30T15:00:43.228230+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.153490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.153490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:24.098704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223152+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:00:43.223152+0000","last_peered":"2024-03-30T15:00:43.223152+0000","last_clean":"2024-03-30T15:00:43.223152+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:00:43.223152+0000","last_undegraded":"2024-03-30T15:00:43.223152+0000","last_fullsized":"2024-03-30T15:00:43.223152+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.044750+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":521,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226520+0000","last_change":"2024-03-30T14:59:05.977968+0000","last_active":"2024-03-30T15:00:43.226520+0000","last_peered":"2024-03-30T15:00:43.226520+0000","last_clean":"2024-03-30T15:00:43.226520+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:00:43.226520+0000","last_undegraded":"2024-03-30T15:00:43.226520+0000","last_fullsized":"2024-03-30T15:00:43.226520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:05.977866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.977866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230130+0000","last_change":"2024-03-30T14:58:35.054537+0000","last_active":"2024-03-30T15:00:43.230130+0000","last_peered":"2024-03-30T15:00:43.230130+0000","last_clean":"2024-03-30T15:00:43.230130+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:00:43.230130+0000","last_undegraded":"2024-03-30T15:00:43.230130+0000","last_fullsized":"2024-03-30T15:00:43.230130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:58:35.054439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.054439+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":459,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233998+0000","last_change":"2024-03-30T14:58:24.985818+0000","last_active":"2024-03-30T15:00:43.233998+0000","last_peered":"2024-03-30T15:00:43.233998+0000","last_clean":"2024-03-30T15:00:43.233998+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:00:43.233998+0000","last_undegraded":"2024-03-30T15:00:43.233998+0000","last_fullsized":"2024-03-30T15:00:43.233998+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T14:58:24.985660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.985660+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":434,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231267+0000","last_change":"2024-03-30T14:58:27.124070+0000","last_active":"2024-03-30T15:00:43.231267+0000","last_peered":"2024-03-30T15:00:43.231267+0000","last_clean":"2024-03-30T15:00:43.231267+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:43.231267+0000","last_undegraded":"2024-03-30T15:00:43.231267+0000","last_fullsized":"2024-03-30T15:00:43.231267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.123941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.123941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"115'7","reported_seq":351,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232678+0000","last_change":"2024-03-30T14:57:55.143775+0000","last_active":"2024-03-30T15:00:43.232678+0000","last_peered":"2024-03-30T15:00:43.232678+0000","last_clean":"2024-03-30T15:00:43.232678+0000","last_became_active":"2024-03-30T14:57:55.139826+0000","last_became_peered":"2024-03-30T14:57:55.139826+0000","last_unstale":"2024-03-30T15:00:43.232678+0000","last_undegraded":"2024-03-30T15:00:43.232678+0000","last_fullsized":"2024-03-30T15:00:43.232678+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216883+0000","last_change":"2024-03-30T14:59:56.678123+0000","last_active":"2024-03-30T15:00:43.216883+0000","last_peered":"2024-03-30T15:00:43.216883+0000","last_clean":"2024-03-30T15:00:43.216883+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:00:43.216883+0000","last_undegraded":"2024-03-30T15:00:43.216883+0000","last_fullsized":"2024-03-30T15:00:43.216883+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.678045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.678045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:12.776149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":368,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234032+0000","last_change":"2024-03-30T14:57:56.275636+0000","last_active":"2024-03-30T15:00:43.234032+0000","last_peered":"2024-03-30T15:00:43.234032+0000","last_clean":"2024-03-30T15:00:43.234032+0000","last_became_active":"2024-03-30T14:57:56.273316+0000","last_became_peered":"2024-03-30T14:57:56.273316+0000","last_unstale":"2024-03-30T15:00:43.234032+0000","last_undegraded":"2024-03-30T15:00:43.234032+0000","last_fullsized":"2024-03-30T15:00:43.234032+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:57:30.919064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:57:30.919064+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":455,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216065+0000","last_change":"2024-03-30T14:58:22.261893+0000","last_active":"2024-03-30T15:00:43.216065+0000","last_peered":"2024-03-30T15:00:43.216065+0000","last_clean":"2024-03-30T15:00:43.216065+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:00:43.216065+0000","last_undegraded":"2024-03-30T15:00:43.216065+0000","last_fullsized":"2024-03-30T15:00:43.216065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.261803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.261803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.235037+0000","last_change":"2024-03-30T14:58:26.076218+0000","last_active":"2024-03-30T15:00:43.235037+0000","last_peered":"2024-03-30T15:00:43.235037+0000","last_clean":"2024-03-30T15:00:43.235037+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:00:43.235037+0000","last_undegraded":"2024-03-30T15:00:43.235037+0000","last_fullsized":"2024-03-30T15:00:43.235037+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:26.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:26.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229519+0000","last_change":"2024-03-30T14:59:51.262210+0000","last_active":"2024-03-30T15:00:43.229519+0000","last_peered":"2024-03-30T15:00:43.229519+0000","last_clean":"2024-03-30T15:00:43.229519+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:00:43.229519+0000","last_undegraded":"2024-03-30T15:00:43.229519+0000","last_fullsized":"2024-03-30T15:00:43.229519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:51.262127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.262127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.554991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224077+0000","last_change":"2024-03-30T15:00:37.677858+0000","last_active":"2024-03-30T15:00:43.224077+0000","last_peered":"2024-03-30T15:00:43.224077+0000","last_clean":"2024-03-30T15:00:43.224077+0000","last_became_active":"2024-03-30T14:57:55.139072+0000","last_became_peered":"2024-03-30T14:57:55.139072+0000","last_unstale":"2024-03-30T15:00:43.224077+0000","last_undegraded":"2024-03-30T15:00:43.224077+0000","last_fullsized":"2024-03-30T15:00:43.224077+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.677768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.677768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.094988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":344,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226899+0000","last_change":"2024-03-30T14:57:55.126527+0000","last_active":"2024-03-30T15:00:43.226899+0000","last_peered":"2024-03-30T15:00:43.226899+0000","last_clean":"2024-03-30T15:00:43.226899+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:00:43.226899+0000","last_undegraded":"2024-03-30T15:00:43.226899+0000","last_fullsized":"2024-03-30T15:00:43.226899+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":460,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215916+0000","last_change":"2024-03-30T14:59:15.487022+0000","last_active":"2024-03-30T15:00:43.215916+0000","last_peered":"2024-03-30T15:00:43.215916+0000","last_clean":"2024-03-30T15:00:43.215916+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:00:43.215916+0000","last_undegraded":"2024-03-30T15:00:43.215916+0000","last_fullsized":"2024-03-30T15:00:43.215916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:15.486892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.486892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":455,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226105+0000","last_change":"2024-03-30T15:00:37.049789+0000","last_active":"2024-03-30T15:00:43.226105+0000","last_peered":"2024-03-30T15:00:43.226105+0000","last_clean":"2024-03-30T15:00:43.226105+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:00:43.226105+0000","last_undegraded":"2024-03-30T15:00:43.226105+0000","last_fullsized":"2024-03-30T15:00:43.226105+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.049685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.049685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.710602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230968+0000","last_change":"2024-03-30T14:58:24.291134+0000","last_active":"2024-03-30T15:00:43.230968+0000","last_peered":"2024-03-30T15:00:43.230968+0000","last_clean":"2024-03-30T15:00:43.230968+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:00:43.230968+0000","last_undegraded":"2024-03-30T15:00:43.230968+0000","last_fullsized":"2024-03-30T15:00:43.230968+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:24.291008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.291008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222770+0000","last_change":"2024-03-30T14:59:02.809926+0000","last_active":"2024-03-30T15:00:43.222770+0000","last_peered":"2024-03-30T15:00:43.222770+0000","last_clean":"2024-03-30T15:00:43.222770+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:00:43.222770+0000","last_undegraded":"2024-03-30T15:00:43.222770+0000","last_fullsized":"2024-03-30T15:00:43.222770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.809814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.809814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3","version":"115'5","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224104+0000","last_change":"2024-03-30T14:57:55.091367+0000","last_active":"2024-03-30T15:00:43.224104+0000","last_peered":"2024-03-30T15:00:43.224104+0000","last_clean":"2024-03-30T15:00:43.224104+0000","last_became_active":"2024-03-30T14:57:55.089349+0000","last_became_peered":"2024-03-30T14:57:55.089349+0000","last_unstale":"2024-03-30T15:00:43.224104+0000","last_undegraded":"2024-03-30T15:00:43.224104+0000","last_fullsized":"2024-03-30T15:00:43.224104+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212651+0000","last_change":"2024-03-30T14:57:54.070389+0000","last_active":"2024-03-30T15:00:43.212651+0000","last_peered":"2024-03-30T15:00:43.212651+0000","last_clean":"2024-03-30T15:00:43.212651+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:00:43.212651+0000","last_undegraded":"2024-03-30T15:00:43.212651+0000","last_fullsized":"2024-03-30T15:00:43.212651+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232405+0000","last_change":"2024-03-30T14:58:31.097064+0000","last_active":"2024-03-30T15:00:43.232405+0000","last_peered":"2024-03-30T15:00:43.232405+0000","last_clean":"2024-03-30T15:00:43.232405+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:00:43.232405+0000","last_undegraded":"2024-03-30T15:00:43.232405+0000","last_fullsized":"2024-03-30T15:00:43.232405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:58:31.096968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.096968+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209499+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:00:43.209499+0000","last_peered":"2024-03-30T15:00:43.209499+0000","last_clean":"2024-03-30T15:00:43.209499+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:00:43.209499+0000","last_undegraded":"2024-03-30T15:00:43.209499+0000","last_fullsized":"2024-03-30T15:00:43.209499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.736897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":407,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231816+0000","last_change":"2024-03-30T14:59:53.269594+0000","last_active":"2024-03-30T15:00:43.231816+0000","last_peered":"2024-03-30T15:00:43.231816+0000","last_clean":"2024-03-30T15:00:43.231816+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:00:43.231816+0000","last_undegraded":"2024-03-30T15:00:43.231816+0000","last_fullsized":"2024-03-30T15:00:43.231816+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:53.269452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.269452+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.687457+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225687+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:00:43.225687+0000","last_peered":"2024-03-30T15:00:43.225687+0000","last_clean":"2024-03-30T15:00:43.225687+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:00:43.225687+0000","last_undegraded":"2024-03-30T15:00:43.225687+0000","last_fullsized":"2024-03-30T15:00:43.225687+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.363643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'5","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226070+0000","last_change":"2024-03-30T14:57:55.177168+0000","last_active":"2024-03-30T15:00:43.226070+0000","last_peered":"2024-03-30T15:00:43.226070+0000","last_clean":"2024-03-30T15:00:43.226070+0000","last_became_active":"2024-03-30T14:57:55.164327+0000","last_became_peered":"2024-03-30T14:57:55.164327+0000","last_unstale":"2024-03-30T15:00:43.226070+0000","last_undegraded":"2024-03-30T15:00:43.226070+0000","last_fullsized":"2024-03-30T15:00:43.226070+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.213193+0000","last_change":"2024-03-30T14:57:54.020464+0000","last_active":"2024-03-30T15:00:43.213193+0000","last_peered":"2024-03-30T15:00:43.213193+0000","last_clean":"2024-03-30T15:00:43.213193+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:00:43.213193+0000","last_undegraded":"2024-03-30T15:00:43.213193+0000","last_fullsized":"2024-03-30T15:00:43.213193+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":383,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228413+0000","last_change":"2024-03-30T14:58:13.210584+0000","last_active":"2024-03-30T15:00:43.228413+0000","last_peered":"2024-03-30T15:00:43.228413+0000","last_clean":"2024-03-30T15:00:43.228413+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:00:43.228413+0000","last_undegraded":"2024-03-30T15:00:43.228413+0000","last_fullsized":"2024-03-30T15:00:43.228413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:13.210426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.210426+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228467+0000","last_change":"2024-03-30T14:59:09.029236+0000","last_active":"2024-03-30T15:00:43.228467+0000","last_peered":"2024-03-30T15:00:43.228467+0000","last_clean":"2024-03-30T15:00:43.228467+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:00:43.228467+0000","last_undegraded":"2024-03-30T15:00:43.228467+0000","last_fullsized":"2024-03-30T15:00:43.228467+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.029159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.029159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":523,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.232062+0000","last_change":"2024-03-30T14:59:08.160662+0000","last_active":"2024-03-30T15:00:43.232062+0000","last_peered":"2024-03-30T15:00:43.232062+0000","last_clean":"2024-03-30T15:00:43.232062+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:00:43.232062+0000","last_undegraded":"2024-03-30T15:00:43.232062+0000","last_fullsized":"2024-03-30T15:00:43.232062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:08.160528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.160528+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215538+0000","last_change":"2024-03-30T14:59:16.481218+0000","last_active":"2024-03-30T15:00:43.215538+0000","last_peered":"2024-03-30T15:00:43.215538+0000","last_clean":"2024-03-30T15:00:43.215538+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:00:43.215538+0000","last_undegraded":"2024-03-30T15:00:43.215538+0000","last_fullsized":"2024-03-30T15:00:43.215538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:16.481128+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.481128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"115'5","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208758+0000","last_change":"2024-03-30T14:57:54.041959+0000","last_active":"2024-03-30T15:00:43.208758+0000","last_peered":"2024-03-30T15:00:43.208758+0000","last_clean":"2024-03-30T15:00:43.208758+0000","last_became_active":"2024-03-30T14:57:54.041685+0000","last_became_peered":"2024-03-30T14:57:54.041685+0000","last_unstale":"2024-03-30T15:00:43.208758+0000","last_undegraded":"2024-03-30T15:00:43.208758+0000","last_fullsized":"2024-03-30T15:00:43.208758+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224616+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:00:43.224616+0000","last_peered":"2024-03-30T15:00:43.224616+0000","last_clean":"2024-03-30T15:00:43.224616+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:00:43.224616+0000","last_undegraded":"2024-03-30T15:00:43.224616+0000","last_fullsized":"2024-03-30T15:00:43.224616+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.403334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224475+0000","last_change":"2024-03-30T14:59:20.453682+0000","last_active":"2024-03-30T15:00:43.224475+0000","last_peered":"2024-03-30T15:00:43.224475+0000","last_clean":"2024-03-30T15:00:43.224475+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:00:43.224475+0000","last_undegraded":"2024-03-30T15:00:43.224475+0000","last_fullsized":"2024-03-30T15:00:43.224475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.453594+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227134+0000","last_change":"2024-03-30T14:57:31.554095+0000","last_active":"2024-03-30T15:00:43.227134+0000","last_peered":"2024-03-30T15:00:43.227134+0000","last_clean":"2024-03-30T15:00:43.227134+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:00:43.227134+0000","last_undegraded":"2024-03-30T15:00:43.227134+0000","last_fullsized":"2024-03-30T15:00:43.227134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:31.554003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:31.554003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":448,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227911+0000","last_change":"2024-03-30T14:59:15.144954+0000","last_active":"2024-03-30T15:00:43.227911+0000","last_peered":"2024-03-30T15:00:43.227911+0000","last_clean":"2024-03-30T15:00:43.227911+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:00:43.227911+0000","last_undegraded":"2024-03-30T15:00:43.227911+0000","last_fullsized":"2024-03-30T15:00:43.227911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.144864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208959+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:00:43.208959+0000","last_peered":"2024-03-30T15:00:43.208959+0000","last_clean":"2024-03-30T15:00:43.208959+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:00:43.208959+0000","last_undegraded":"2024-03-30T15:00:43.208959+0000","last_fullsized":"2024-03-30T15:00:43.208959+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:29.384685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'9","reported_seq":362,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215736+0000","last_change":"2024-03-30T14:59:47.655366+0000","last_active":"2024-03-30T15:00:43.215736+0000","last_peered":"2024-03-30T15:00:43.215736+0000","last_clean":"2024-03-30T15:00:43.215736+0000","last_became_active":"2024-03-30T14:57:55.181249+0000","last_became_peered":"2024-03-30T14:57:55.181249+0000","last_unstale":"2024-03-30T15:00:43.215736+0000","last_undegraded":"2024-03-30T15:00:43.215736+0000","last_fullsized":"2024-03-30T15:00:43.215736+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9","last_scrub_stamp":"2024-03-30T14:59:47.655290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.655290+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.586327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223126+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:00:43.223126+0000","last_peered":"2024-03-30T15:00:43.223126+0000","last_clean":"2024-03-30T15:00:43.223126+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:00:43.223126+0000","last_undegraded":"2024-03-30T15:00:43.223126+0000","last_fullsized":"2024-03-30T15:00:43.223126+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.153282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":460,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230514+0000","last_change":"2024-03-30T14:59:08.292999+0000","last_active":"2024-03-30T15:00:43.230514+0000","last_peered":"2024-03-30T15:00:43.230514+0000","last_clean":"2024-03-30T15:00:43.230514+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:00:43.230514+0000","last_undegraded":"2024-03-30T15:00:43.230514+0000","last_fullsized":"2024-03-30T15:00:43.230514+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:08.292876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.292876+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229621+0000","last_change":"2024-03-30T14:58:12.927851+0000","last_active":"2024-03-30T15:00:43.229621+0000","last_peered":"2024-03-30T15:00:43.229621+0000","last_clean":"2024-03-30T15:00:43.229621+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:00:43.229621+0000","last_undegraded":"2024-03-30T15:00:43.229621+0000","last_fullsized":"2024-03-30T15:00:43.229621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:58:12.927762+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.927762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212023+0000","last_change":"2024-03-30T14:59:17.441377+0000","last_active":"2024-03-30T15:00:43.212023+0000","last_peered":"2024-03-30T15:00:43.212023+0000","last_clean":"2024-03-30T15:00:43.212023+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:00:43.212023+0000","last_undegraded":"2024-03-30T15:00:43.212023+0000","last_fullsized":"2024-03-30T15:00:43.212023+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.441289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.441289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208903+0000","last_change":"2024-03-30T14:58:18.177964+0000","last_active":"2024-03-30T15:00:43.208903+0000","last_peered":"2024-03-30T15:00:43.208903+0000","last_clean":"2024-03-30T15:00:43.208903+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:00:43.208903+0000","last_undegraded":"2024-03-30T15:00:43.208903+0000","last_fullsized":"2024-03-30T15:00:43.208903+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:18.177806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:18.177806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":347,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216248+0000","last_change":"2024-03-30T14:57:54.013168+0000","last_active":"2024-03-30T15:00:43.216248+0000","last_peered":"2024-03-30T15:00:43.216248+0000","last_clean":"2024-03-30T15:00:43.216248+0000","last_became_active":"2024-03-30T14:57:54.012861+0000","last_became_peered":"2024-03-30T14:57:54.012861+0000","last_unstale":"2024-03-30T15:00:43.216248+0000","last_undegraded":"2024-03-30T15:00:43.216248+0000","last_fullsized":"2024-03-30T15:00:43.216248+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208794+0000","last_change":"2024-03-30T14:57:54.070762+0000","last_active":"2024-03-30T15:00:43.208794+0000","last_peered":"2024-03-30T15:00:43.208794+0000","last_clean":"2024-03-30T15:00:43.208794+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:00:43.208794+0000","last_undegraded":"2024-03-30T15:00:43.208794+0000","last_fullsized":"2024-03-30T15:00:43.208794+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":386,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226514+0000","last_change":"2024-03-30T14:58:16.905699+0000","last_active":"2024-03-30T15:00:43.226514+0000","last_peered":"2024-03-30T15:00:43.226514+0000","last_clean":"2024-03-30T15:00:43.226514+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:00:43.226514+0000","last_undegraded":"2024-03-30T15:00:43.226514+0000","last_fullsized":"2024-03-30T15:00:43.226514+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:58:16.905604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:16.905604+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234920+0000","last_change":"2024-03-30T14:58:28.115261+0000","last_active":"2024-03-30T15:00:43.234920+0000","last_peered":"2024-03-30T15:00:43.234920+0000","last_clean":"2024-03-30T15:00:43.234920+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:00:43.234920+0000","last_undegraded":"2024-03-30T15:00:43.234920+0000","last_fullsized":"2024-03-30T15:00:43.234920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:28.115166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:28.115166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227111+0000","last_change":"2024-03-30T14:59:08.012564+0000","last_active":"2024-03-30T15:00:43.227111+0000","last_peered":"2024-03-30T15:00:43.227111+0000","last_clean":"2024-03-30T15:00:43.227111+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:00:43.227111+0000","last_undegraded":"2024-03-30T15:00:43.227111+0000","last_fullsized":"2024-03-30T15:00:43.227111+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.012470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.012470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209543+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:00:43.209543+0000","last_peered":"2024-03-30T15:00:43.209543+0000","last_clean":"2024-03-30T15:00:43.209543+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:00:43.209543+0000","last_undegraded":"2024-03-30T15:00:43.209543+0000","last_fullsized":"2024-03-30T15:00:43.209543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.714400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":347,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.213623+0000","last_change":"2024-03-30T14:57:54.019408+0000","last_active":"2024-03-30T15:00:43.213623+0000","last_peered":"2024-03-30T15:00:43.213623+0000","last_clean":"2024-03-30T15:00:43.213623+0000","last_became_active":"2024-03-30T14:57:54.014697+0000","last_became_peered":"2024-03-30T14:57:54.014697+0000","last_unstale":"2024-03-30T15:00:43.213623+0000","last_undegraded":"2024-03-30T15:00:43.213623+0000","last_fullsized":"2024-03-30T15:00:43.213623+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":354,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224658+0000","last_change":"2024-03-30T14:59:56.148641+0000","last_active":"2024-03-30T15:00:43.224658+0000","last_peered":"2024-03-30T15:00:43.224658+0000","last_clean":"2024-03-30T15:00:43.224658+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:00:43.224658+0000","last_undegraded":"2024-03-30T15:00:43.224658+0000","last_fullsized":"2024-03-30T15:00:43.224658+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.148529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.148529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.134818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":374,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224579+0000","last_change":"2024-03-30T14:58:17.304912+0000","last_active":"2024-03-30T15:00:43.224579+0000","last_peered":"2024-03-30T15:00:43.224579+0000","last_clean":"2024-03-30T15:00:43.224579+0000","last_became_active":"2024-03-30T14:57:56.224608+0000","last_became_peered":"2024-03-30T14:57:56.224608+0000","last_unstale":"2024-03-30T15:00:43.224579+0000","last_undegraded":"2024-03-30T15:00:43.224579+0000","last_fullsized":"2024-03-30T15:00:43.224579+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:58:17.304759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:17.304759+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216512+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:00:43.216512+0000","last_peered":"2024-03-30T15:00:43.216512+0000","last_clean":"2024-03-30T15:00:43.216512+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:00:43.216512+0000","last_undegraded":"2024-03-30T15:00:43.216512+0000","last_fullsized":"2024-03-30T15:00:43.216512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.891147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217900+0000","last_change":"2024-03-30T14:58:25.320605+0000","last_active":"2024-03-30T15:00:43.217900+0000","last_peered":"2024-03-30T15:00:43.217900+0000","last_clean":"2024-03-30T15:00:43.217900+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:00:43.217900+0000","last_undegraded":"2024-03-30T15:00:43.217900+0000","last_fullsized":"2024-03-30T15:00:43.217900+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.320527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.320527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227428+0000","last_change":"2024-03-30T14:58:58.957529+0000","last_active":"2024-03-30T15:00:43.227428+0000","last_peered":"2024-03-30T15:00:43.227428+0000","last_clean":"2024-03-30T15:00:43.227428+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:00:43.227428+0000","last_undegraded":"2024-03-30T15:00:43.227428+0000","last_fullsized":"2024-03-30T15:00:43.227428+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.957449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.957449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215974+0000","last_change":"2024-03-30T14:59:07.508558+0000","last_active":"2024-03-30T15:00:43.215974+0000","last_peered":"2024-03-30T15:00:43.215974+0000","last_clean":"2024-03-30T15:00:43.215974+0000","last_became_active":"2024-03-30T14:57:55.201639+0000","last_became_peered":"2024-03-30T14:57:55.201639+0000","last_unstale":"2024-03-30T15:00:43.215974+0000","last_undegraded":"2024-03-30T15:00:43.215974+0000","last_fullsized":"2024-03-30T15:00:43.215974+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.508480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.508480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217328+0000","last_change":"2024-03-30T15:00:03.553767+0000","last_active":"2024-03-30T15:00:43.217328+0000","last_peered":"2024-03-30T15:00:43.217328+0000","last_clean":"2024-03-30T15:00:43.217328+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:00:43.217328+0000","last_undegraded":"2024-03-30T15:00:43.217328+0000","last_fullsized":"2024-03-30T15:00:43.217328+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.553680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.553680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.545159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":475,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217399+0000","last_change":"2024-03-30T15:00:41.903846+0000","last_active":"2024-03-30T15:00:43.217399+0000","last_peered":"2024-03-30T15:00:43.217399+0000","last_clean":"2024-03-30T15:00:43.217399+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:00:43.217399+0000","last_undegraded":"2024-03-30T15:00:43.217399+0000","last_fullsized":"2024-03-30T15:00:43.217399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:00:41.903753+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.903753+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.208912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":350,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209186+0000","last_change":"2024-03-30T14:59:09.624243+0000","last_active":"2024-03-30T15:00:43.209186+0000","last_peered":"2024-03-30T15:00:43.209186+0000","last_clean":"2024-03-30T15:00:43.209186+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:00:43.209186+0000","last_undegraded":"2024-03-30T15:00:43.209186+0000","last_fullsized":"2024-03-30T15:00:43.209186+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.624114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.624114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.216081+0000","last_change":"2024-03-30T14:58:58.605917+0000","last_active":"2024-03-30T15:00:43.216081+0000","last_peered":"2024-03-30T15:00:43.216081+0000","last_clean":"2024-03-30T15:00:43.216081+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:00:43.216081+0000","last_undegraded":"2024-03-30T15:00:43.216081+0000","last_fullsized":"2024-03-30T15:00:43.216081+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.605792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.605792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":444,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.229035+0000","last_change":"2024-03-30T14:59:21.144421+0000","last_active":"2024-03-30T15:00:43.229035+0000","last_peered":"2024-03-30T15:00:43.229035+0000","last_clean":"2024-03-30T15:00:43.229035+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:00:43.229035+0000","last_undegraded":"2024-03-30T15:00:43.229035+0000","last_fullsized":"2024-03-30T15:00:43.229035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.144335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.144335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.a","version":"115'10","reported_seq":362,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225659+0000","last_change":"2024-03-30T15:00:42.061005+0000","last_active":"2024-03-30T15:00:43.225659+0000","last_peered":"2024-03-30T15:00:43.225659+0000","last_clean":"2024-03-30T15:00:43.225659+0000","last_became_active":"2024-03-30T14:57:55.103400+0000","last_became_peered":"2024-03-30T14:57:55.103400+0000","last_unstale":"2024-03-30T15:00:43.225659+0000","last_undegraded":"2024-03-30T15:00:43.225659+0000","last_fullsized":"2024-03-30T15:00:43.225659+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:42.060906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:42.060906+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.539739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217529+0000","last_change":"2024-03-30T14:59:14.560489+0000","last_active":"2024-03-30T15:00:43.217529+0000","last_peered":"2024-03-30T15:00:43.217529+0000","last_clean":"2024-03-30T15:00:43.217529+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:00:43.217529+0000","last_undegraded":"2024-03-30T15:00:43.217529+0000","last_fullsized":"2024-03-30T15:00:43.217529+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.560377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.560377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":465,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215830+0000","last_change":"2024-03-30T14:59:10.626455+0000","last_active":"2024-03-30T15:00:43.215830+0000","last_peered":"2024-03-30T15:00:43.215830+0000","last_clean":"2024-03-30T15:00:43.215830+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:00:43.215830+0000","last_undegraded":"2024-03-30T15:00:43.215830+0000","last_fullsized":"2024-03-30T15:00:43.215830+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.626365+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":449,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224997+0000","last_change":"2024-03-30T14:58:32.272039+0000","last_active":"2024-03-30T15:00:43.224997+0000","last_peered":"2024-03-30T15:00:43.224997+0000","last_clean":"2024-03-30T15:00:43.224997+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:00:43.224997+0000","last_undegraded":"2024-03-30T15:00:43.224997+0000","last_fullsized":"2024-03-30T15:00:43.224997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T14:58:32.271622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:32.271622+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":532,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217214+0000","last_change":"2024-03-30T14:58:59.472999+0000","last_active":"2024-03-30T15:00:43.217214+0000","last_peered":"2024-03-30T15:00:43.217214+0000","last_clean":"2024-03-30T15:00:43.217214+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:00:43.217214+0000","last_undegraded":"2024-03-30T15:00:43.217214+0000","last_fullsized":"2024-03-30T15:00:43.217214+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.472884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.472884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":445,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230218+0000","last_change":"2024-03-30T15:00:36.069861+0000","last_active":"2024-03-30T15:00:43.230218+0000","last_peered":"2024-03-30T15:00:43.230218+0000","last_clean":"2024-03-30T15:00:43.230218+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:00:43.230218+0000","last_undegraded":"2024-03-30T15:00:43.230218+0000","last_fullsized":"2024-03-30T15:00:43.230218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:00:36.069752+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:00:36.069752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:40.687480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"115'2","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212607+0000","last_change":"2024-03-30T14:57:54.018464+0000","last_active":"2024-03-30T15:00:43.212607+0000","last_peered":"2024-03-30T15:00:43.212607+0000","last_clean":"2024-03-30T15:00:43.212607+0000","last_became_active":"2024-03-30T14:57:54.018184+0000","last_became_peered":"2024-03-30T14:57:54.018184+0000","last_unstale":"2024-03-30T15:00:43.212607+0000","last_undegraded":"2024-03-30T15:00:43.212607+0000","last_fullsized":"2024-03-30T15:00:43.212607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":353,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217171+0000","last_change":"2024-03-30T14:59:21.597544+0000","last_active":"2024-03-30T15:00:43.217171+0000","last_peered":"2024-03-30T15:00:43.217171+0000","last_clean":"2024-03-30T15:00:43.217171+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:00:43.217171+0000","last_undegraded":"2024-03-30T15:00:43.217171+0000","last_fullsized":"2024-03-30T15:00:43.217171+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":391,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228664+0000","last_change":"2024-03-30T14:58:15.287344+0000","last_active":"2024-03-30T15:00:43.228664+0000","last_peered":"2024-03-30T15:00:43.228664+0000","last_clean":"2024-03-30T15:00:43.228664+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:00:43.228664+0000","last_undegraded":"2024-03-30T15:00:43.228664+0000","last_fullsized":"2024-03-30T15:00:43.228664+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:15.286572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.286572+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223679+0000","last_change":"2024-03-30T14:58:31.196084+0000","last_active":"2024-03-30T15:00:43.223679+0000","last_peered":"2024-03-30T15:00:43.223679+0000","last_clean":"2024-03-30T15:00:43.223679+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:00:43.223679+0000","last_undegraded":"2024-03-30T15:00:43.223679+0000","last_fullsized":"2024-03-30T15:00:43.223679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:31.196010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.196010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227499+0000","last_change":"2024-03-30T14:59:01.294973+0000","last_active":"2024-03-30T15:00:43.227499+0000","last_peered":"2024-03-30T15:00:43.227499+0000","last_clean":"2024-03-30T15:00:43.227499+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:00:43.227499+0000","last_undegraded":"2024-03-30T15:00:43.227499+0000","last_fullsized":"2024-03-30T15:00:43.227499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.294888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.294888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":349,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217542+0000","last_change":"2024-03-30T14:58:12.296903+0000","last_active":"2024-03-30T15:00:43.217542+0000","last_peered":"2024-03-30T15:00:43.217542+0000","last_clean":"2024-03-30T15:00:43.217542+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:00:43.217542+0000","last_undegraded":"2024-03-30T15:00:43.217542+0000","last_fullsized":"2024-03-30T15:00:43.217542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:12.296793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.296793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"115'2","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.226053+0000","last_change":"2024-03-30T14:57:55.109289+0000","last_active":"2024-03-30T15:00:43.226053+0000","last_peered":"2024-03-30T15:00:43.226053+0000","last_clean":"2024-03-30T15:00:43.226053+0000","last_became_active":"2024-03-30T14:57:55.108185+0000","last_became_peered":"2024-03-30T14:57:55.108185+0000","last_unstale":"2024-03-30T15:00:43.226053+0000","last_undegraded":"2024-03-30T15:00:43.226053+0000","last_fullsized":"2024-03-30T15:00:43.226053+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.234166+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:00:43.234166+0000","last_peered":"2024-03-30T15:00:43.234166+0000","last_clean":"2024-03-30T15:00:43.234166+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:00:43.234166+0000","last_undegraded":"2024-03-30T15:00:43.234166+0000","last_fullsized":"2024-03-30T15:00:43.234166+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.587803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":482,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.222264+0000","last_change":"2024-03-30T14:58:37.879183+0000","last_active":"2024-03-30T15:00:43.222264+0000","last_peered":"2024-03-30T15:00:43.222264+0000","last_clean":"2024-03-30T15:00:43.222264+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:00:43.222264+0000","last_undegraded":"2024-03-30T15:00:43.222264+0000","last_fullsized":"2024-03-30T15:00:43.222264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T14:58:37.879074+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.212163+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:00:43.212163+0000","last_peered":"2024-03-30T15:00:43.212163+0000","last_clean":"2024-03-30T15:00:43.212163+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:00:43.212163+0000","last_undegraded":"2024-03-30T15:00:43.212163+0000","last_fullsized":"2024-03-30T15:00:43.212163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:21.606179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228900+0000","last_change":"2024-03-30T14:58:19.314405+0000","last_active":"2024-03-30T15:00:43.228900+0000","last_peered":"2024-03-30T15:00:43.228900+0000","last_clean":"2024-03-30T15:00:43.228900+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:00:43.228900+0000","last_undegraded":"2024-03-30T15:00:43.228900+0000","last_fullsized":"2024-03-30T15:00:43.228900+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.314262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.314262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227082+0000","last_change":"2024-03-30T14:59:03.291569+0000","last_active":"2024-03-30T15:00:43.227082+0000","last_peered":"2024-03-30T15:00:43.227082+0000","last_clean":"2024-03-30T15:00:43.227082+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:00:43.227082+0000","last_undegraded":"2024-03-30T15:00:43.227082+0000","last_fullsized":"2024-03-30T15:00:43.227082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.291469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.291469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":341,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.208621+0000","last_change":"2024-03-30T14:57:56.258314+0000","last_active":"2024-03-30T15:00:43.208621+0000","last_peered":"2024-03-30T15:00:43.208621+0000","last_clean":"2024-03-30T15:00:43.208621+0000","last_became_active":"2024-03-30T14:57:56.257360+0000","last_became_peered":"2024-03-30T14:57:56.257360+0000","last_unstale":"2024-03-30T15:00:43.208621+0000","last_undegraded":"2024-03-30T15:00:43.208621+0000","last_fullsized":"2024-03-30T15:00:43.208621+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":354,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225688+0000","last_change":"2024-03-30T14:59:50.075598+0000","last_active":"2024-03-30T15:00:43.225688+0000","last_peered":"2024-03-30T15:00:43.225688+0000","last_clean":"2024-03-30T15:00:43.225688+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:00:43.225688+0000","last_undegraded":"2024-03-30T15:00:43.225688+0000","last_fullsized":"2024-03-30T15:00:43.225688+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.075512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.075512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.184144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":465,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224378+0000","last_change":"2024-03-30T14:59:46.413217+0000","last_active":"2024-03-30T15:00:43.224378+0000","last_peered":"2024-03-30T15:00:43.224378+0000","last_clean":"2024-03-30T15:00:43.224378+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T15:00:43.224378+0000","last_undegraded":"2024-03-30T15:00:43.224378+0000","last_fullsized":"2024-03-30T15:00:43.224378+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T14:59:46.413127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.413127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.902904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.233946+0000","last_change":"2024-03-30T14:58:15.115774+0000","last_active":"2024-03-30T15:00:43.233946+0000","last_peered":"2024-03-30T15:00:43.233946+0000","last_clean":"2024-03-30T15:00:43.233946+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:00:43.233946+0000","last_undegraded":"2024-03-30T15:00:43.233946+0000","last_fullsized":"2024-03-30T15:00:43.233946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:15.115694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.115694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227287+0000","last_change":"2024-03-30T14:58:29.060905+0000","last_active":"2024-03-30T15:00:43.227287+0000","last_peered":"2024-03-30T15:00:43.227287+0000","last_clean":"2024-03-30T15:00:43.227287+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:00:43.227287+0000","last_undegraded":"2024-03-30T15:00:43.227287+0000","last_fullsized":"2024-03-30T15:00:43.227287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:29.060822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:29.060822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230493+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:00:43.230493+0000","last_peered":"2024-03-30T15:00:43.230493+0000","last_clean":"2024-03-30T15:00:43.230493+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:00:43.230493+0000","last_undegraded":"2024-03-30T15:00:43.230493+0000","last_fullsized":"2024-03-30T15:00:43.230493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.177682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.225732+0000","last_change":"2024-03-30T14:59:16.765869+0000","last_active":"2024-03-30T15:00:43.225732+0000","last_peered":"2024-03-30T15:00:43.225732+0000","last_clean":"2024-03-30T15:00:43.225732+0000","last_became_active":"2024-03-30T14:57:54.012378+0000","last_became_peered":"2024-03-30T14:57:54.012378+0000","last_unstale":"2024-03-30T15:00:43.225732+0000","last_undegraded":"2024-03-30T15:00:43.225732+0000","last_fullsized":"2024-03-30T15:00:43.225732+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:16.765776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.765776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.227503+0000","last_change":"2024-03-30T15:00:00.172131+0000","last_active":"2024-03-30T15:00:43.227503+0000","last_peered":"2024-03-30T15:00:43.227503+0000","last_clean":"2024-03-30T15:00:43.227503+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:00:43.227503+0000","last_undegraded":"2024-03-30T15:00:43.227503+0000","last_fullsized":"2024-03-30T15:00:43.227503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.172053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.172053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:07.869357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":459,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223659+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:00:43.223659+0000","last_peered":"2024-03-30T15:00:43.223659+0000","last_clean":"2024-03-30T15:00:43.223659+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:00:43.223659+0000","last_undegraded":"2024-03-30T15:00:43.223659+0000","last_fullsized":"2024-03-30T15:00:43.223659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.313881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231968+0000","last_change":"2024-03-30T14:58:25.078864+0000","last_active":"2024-03-30T15:00:43.231968+0000","last_peered":"2024-03-30T15:00:43.231968+0000","last_clean":"2024-03-30T15:00:43.231968+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:00:43.231968+0000","last_undegraded":"2024-03-30T15:00:43.231968+0000","last_fullsized":"2024-03-30T15:00:43.231968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.078771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.078771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231502+0000","last_change":"2024-03-30T14:58:19.191376+0000","last_active":"2024-03-30T15:00:43.231502+0000","last_peered":"2024-03-30T15:00:43.231502+0000","last_clean":"2024-03-30T15:00:43.231502+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:00:43.231502+0000","last_undegraded":"2024-03-30T15:00:43.231502+0000","last_fullsized":"2024-03-30T15:00:43.231502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.191278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.191278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":444,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.209431+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:00:43.209431+0000","last_peered":"2024-03-30T15:00:43.209431+0000","last_clean":"2024-03-30T15:00:43.209431+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:00:43.209431+0000","last_undegraded":"2024-03-30T15:00:43.209431+0000","last_fullsized":"2024-03-30T15:00:43.209431+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.919742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":355,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.231422+0000","last_change":"2024-03-30T15:00:34.263519+0000","last_active":"2024-03-30T15:00:43.231422+0000","last_peered":"2024-03-30T15:00:43.231422+0000","last_clean":"2024-03-30T15:00:43.231422+0000","last_became_active":"2024-03-30T14:57:54.084290+0000","last_became_peered":"2024-03-30T14:57:54.084290+0000","last_unstale":"2024-03-30T15:00:43.231422+0000","last_undegraded":"2024-03-30T15:00:43.231422+0000","last_fullsized":"2024-03-30T15:00:43.231422+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.263392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.263392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.423638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":344,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.230374+0000","last_change":"2024-03-30T14:57:55.106979+0000","last_active":"2024-03-30T15:00:43.230374+0000","last_peered":"2024-03-30T15:00:43.230374+0000","last_clean":"2024-03-30T15:00:43.230374+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:00:43.230374+0000","last_undegraded":"2024-03-30T15:00:43.230374+0000","last_fullsized":"2024-03-30T15:00:43.230374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.221987+0000","last_change":"2024-03-30T14:59:59.784365+0000","last_active":"2024-03-30T15:00:43.221987+0000","last_peered":"2024-03-30T15:00:43.221987+0000","last_clean":"2024-03-30T15:00:43.221987+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:00:43.221987+0000","last_undegraded":"2024-03-30T15:00:43.221987+0000","last_fullsized":"2024-03-30T15:00:43.221987+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:59:59.784262+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.784262+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.754986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":350,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.217246+0000","last_change":"2024-03-30T14:59:06.484972+0000","last_active":"2024-03-30T15:00:43.217246+0000","last_peered":"2024-03-30T15:00:43.217246+0000","last_clean":"2024-03-30T15:00:43.217246+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:00:43.217246+0000","last_undegraded":"2024-03-30T15:00:43.217246+0000","last_fullsized":"2024-03-30T15:00:43.217246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:06.484877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T14:59:06.484877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":439,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223078+0000","last_change":"2024-03-30T14:59:12.396403+0000","last_active":"2024-03-30T15:00:43.223078+0000","last_peered":"2024-03-30T15:00:43.223078+0000","last_clean":"2024-03-30T15:00:43.223078+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:00:43.223078+0000","last_undegraded":"2024-03-30T15:00:43.223078+0000","last_fullsized":"2024-03-30T15:00:43.223078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.396321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":348,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228391+0000","last_change":"2024-03-30T14:59:01.955656+0000","last_active":"2024-03-30T15:00:43.228391+0000","last_peered":"2024-03-30T15:00:43.228391+0000","last_clean":"2024-03-30T15:00:43.228391+0000","last_became_active":"2024-03-30T14:57:56.272206+0000","last_became_peered":"2024-03-30T14:57:56.272206+0000","last_unstale":"2024-03-30T15:00:43.228391+0000","last_undegraded":"2024-03-30T15:00:43.228391+0000","last_fullsized":"2024-03-30T15:00:43.228391+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.955549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.955549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":352,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224637+0000","last_change":"2024-03-30T15:00:43.030756+0000","last_active":"2024-03-30T15:00:43.224637+0000","last_peered":"2024-03-30T15:00:43.224637+0000","last_clean":"2024-03-30T15:00:43.224637+0000","last_became_active":"2024-03-30T14:57:55.195272+0000","last_became_peered":"2024-03-30T14:57:55.195272+0000","last_unstale":"2024-03-30T15:00:43.224637+0000","last_undegraded":"2024-03-30T15:00:43.224637+0000","last_fullsized":"2024-03-30T15:00:43.224637+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.029978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.029978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.644314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":346,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.211033+0000","last_change":"2024-03-30T14:57:54.063914+0000","last_active":"2024-03-30T15:00:43.211033+0000","last_peered":"2024-03-30T15:00:43.211033+0000","last_clean":"2024-03-30T15:00:43.211033+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:00:43.211033+0000","last_undegraded":"2024-03-30T15:00:43.211033+0000","last_fullsized":"2024-03-30T15:00:43.211033+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223209+0000","last_change":"2024-03-30T14:59:57.415396+0000","last_active":"2024-03-30T15:00:43.223209+0000","last_peered":"2024-03-30T15:00:43.223209+0000","last_clean":"2024-03-30T15:00:43.223209+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T15:00:43.223209+0000","last_undegraded":"2024-03-30T15:00:43.223209+0000","last_fullsized":"2024-03-30T15:00:43.223209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.852042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":456,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.215750+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:00:43.215750+0000","last_peered":"2024-03-30T15:00:43.215750+0000","last_clean":"2024-03-30T15:00:43.215750+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:00:43.215750+0000","last_undegraded":"2024-03-30T15:00:43.215750+0000","last_fullsized":"2024-03-30T15:00:43.215750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.262876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.224852+0000","last_change":"2024-03-30T14:59:04.268770+0000","last_active":"2024-03-30T15:00:43.224852+0000","last_peered":"2024-03-30T15:00:43.224852+0000","last_clean":"2024-03-30T15:00:43.224852+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:00:43.224852+0000","last_undegraded":"2024-03-30T15:00:43.224852+0000","last_fullsized":"2024-03-30T15:00:43.224852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:04.268662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T14:59:04.268662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.223409+0000","last_change":"2024-03-30T14:58:27.174113+0000","last_active":"2024-03-30T15:00:43.223409+0000","last_peered":"2024-03-30T15:00:43.223409+0000","last_clean":"2024-03-30T15:00:43.223409+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:00:43.223409+0000","last_undegraded":"2024-03-30T15:00:43.223409+0000","last_fullsized":"2024-03-30T15:00:43.223409+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.174017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.174017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.d","version":"115'5","reported_seq":357,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-30T15:00:43.228367+0000","last_change":"2024-03-30T15:00:35.045624+0000","last_active":"2024-03-30T15:00:43.228367+0000","last_peered":"2024-03-30T15:00:43.228367+0000","last_clean":"2024-03-30T15:00:43.228367+0000","last_became_active":"2024-03-30T14:57:55.114299+0000","last_became_peered":"2024-03-30T14:57:55.114299+0000","last_unstale":"2024-03-30T15:00:43.228367+0000","last_undegraded":"2024-03-30T15:00:43.228367+0000","last_fullsized":"2024-03-30T15:00:43.228367+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.045519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.045519+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.009780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":4195498,"num_objects":119,"num_object_clones":1,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":95,"num_read_kb":1,"num_write":539,"num_write_kb":8725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14020608,"data_stored":12586494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1784,"ondisk_log_size":1784,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":359,"num_read_kb":354,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":254,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005134,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72716,"kb_used_data":5428,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672436,"statfs":{"total":95995035648,"available":95920574464,"internally_reserved":0,"allocated":5558272,"data_stored":3332448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299475,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76828,"kb_used_data":5380,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668324,"statfs":{"total":95995035648,"available":95916363776,"internally_reserved":0,"allocated":5509120,"data_stored":3341521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72120,"kb_used_data":4832,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673032,"statfs":{"total":95995035648,"available":95921184768,"internally_reserved":0,"allocated":4947968,"data_stored":2765047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54604,"kb_used_data":9532,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93690548,"statfs":{"total":95995035648,"available":95939121152,"internally_reserved":0,"allocated":9760768,"data_stored":7550135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80380,"kb_used_data":8940,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664772,"statfs":{"total":95995035648,"available":95912726528,"internally_reserved":0,"allocated":9154560,"data_stored":6958587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117091,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80460,"kb_used_data":9012,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664692,"statfs":{"total":95995035648,"available":95912644608,"internally_reserved":0,"allocated":9228288,"data_stored":6960845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953565,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80376,"kb_used_data":8928,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664776,"statfs":{"total":95995035648,"available":95912730624,"internally_reserved":0,"allocated":9142272,"data_stored":6935125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659224,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76296,"kb_used_data":4848,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668856,"statfs":{"total":95995035648,"available":95916908544,"internally_reserved":0,"allocated":4964352,"data_stored":2751575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4370432,"data_stored":4194729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4382720,"data_stored":4194768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":4194795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:00:47.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:00:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:47.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:47.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:47.348 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:00:47.349 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:00:47.349 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:00:47.349 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:00:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: pgmap v362: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T15:00:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: 12.19 deep-scrub starts 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: 12.19 deep-scrub ok 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T15:00:47.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.432 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:47.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: pgmap v362: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: 12.19 deep-scrub starts 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: 12.19 deep-scrub ok 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: pgmap v362: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi028-92979-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: osdmap e282: 8 total, 8 up, 8 in 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3573831665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:00:47.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]: dispatch 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: 12.19 deep-scrub starts 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: 12.19 deep-scrub ok 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.57542 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi028-92999-47"}]': finished 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: osdmap e283: 8 total, 8 up, 8 in 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:47.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:47.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:47.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:47.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:47.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:47.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:47.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:00:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:47.965 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:47.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:48.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:48.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:48.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:48.200 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:48.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:48.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:48.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:48.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:48.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:48.485 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.49591 v1:172.21.15.28:0/2862065650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: 17.1a scrub starts 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: 17.1a scrub ok 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: 12.1d scrub starts 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: 12.1d scrub ok 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:48.486 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T15:00:48.487 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.487 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.487 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.487 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:48.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.49591 v1:172.21.15.28:0/2862065650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: 17.1a scrub starts 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: 17.1a scrub ok 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:48.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: 12.1d scrub starts 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: 12.1d scrub ok 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:48.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.49591 v1:172.21.15.28:0/2862065650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: 17.1a scrub starts 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: 17.1a scrub ok 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: 12.1d scrub starts 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: 12.1d scrub ok 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi028-92999-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: osdmap e284: 8 total, 8 up, 8 in 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:48.706 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:48.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:48.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:48.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:48.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:48.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:48.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:48.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:48.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:49.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:49.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:49.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:49.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:49.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:49.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:49.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:49.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: pgmap v365: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1024 KiB/s wr, 0 op/s 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: 101.1f deep-scrub starts 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: 101.1f deep-scrub ok 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:49.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: pgmap v365: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1024 KiB/s wr, 0 op/s 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: 101.1f deep-scrub starts 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: 101.1f deep-scrub ok 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:49.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:49.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:49.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: pgmap v365: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1024 KiB/s wr, 0 op/s 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: 101.1f deep-scrub starts 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: 101.1f deep-scrub ok 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:49.657 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:49.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:49.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:49.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:49.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:49.884 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:00:49.884 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:00:49.884 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:00:49.884 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:00:49.884 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:00:49.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:00:49.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:50.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:50.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.314+0000 7fdc28bbd700 1 -- 172.21.15.28:0/1090179411 >> v1:172.21.15.28:6790/0 conn(0x7fdc24101140 legacy=0x7fdc24101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 -- 172.21.15.28:0/1090179411 shutdown_connections 2024-03-30T15:00:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 -- 172.21.15.28:0/1090179411 >> 172.21.15.28:0/1090179411 conn(0x7fdc240fc9f0 msgr2=0x7fdc240fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 -- 172.21.15.28:0/1090179411 shutdown_connections 2024-03-30T15:00:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 -- 172.21.15.28:0/1090179411 wait complete. 2024-03-30T15:00:50.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 librados: starting msgr at 2024-03-30T15:00:50.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.315+0000 7fdc28bbd700 1 librados: starting objecter 2024-03-30T15:00:50.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 Processor -- start 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 -- start start 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 librados: setting wanted keys 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 librados: calling monclient init 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc24106790 con 0x7fdc2410d970 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc241069a0 con 0x7fdc24101140 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc28bbd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdc241ab950 con 0x7fdc24109dd0 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc21d9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdc24109dd0 0x7fdc2406ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49516/0 (socket says 172.21.15.28:49516) 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc21d9b700 1 -- 172.21.15.28:0/4049064069 learned_addr learned my addr 172.21.15.28:0/4049064069 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc22d9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdc2410d970 0x7fdc241a8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42442/0 (socket says 172.21.15.28:42442) 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdc24109dd0 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdc2410d970 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdc24101140 2024-03-30T15:00:50.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355982474 0 0) 0x7fdc241ab950 con 0x7fdc24109dd0 2024-03-30T15:00:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc100031f0 con 0x7fdc24109dd0 2024-03-30T15:00:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065793468 0 0) 0x7fdc24106790 con 0x7fdc2410d970 2024-03-30T15:00:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.316+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc241ab950 con 0x7fdc2410d970 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833352327 0 0) 0x7fdc241069a0 con 0x7fdc24101140 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc24106790 con 0x7fdc24101140 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384295164 0 0) 0x7fdc100031f0 con 0x7fdc24109dd0 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc241069a0 con 0x7fdc24109dd0 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdc140018b0 con 0x7fdc24109dd0 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 641766042 0 0) 0x7fdc241ab950 con 0x7fdc2410d970 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc100031f0 con 0x7fdc2410d970 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdc1c001bb0 con 0x7fdc2410d970 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3839159966 0 0) 0x7fdc24106790 con 0x7fdc24101140 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc241ab950 con 0x7fdc24101140 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdc0c0015d0 con 0x7fdc24101140 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2286943620 0 0) 0x7fdc241069a0 con 0x7fdc24109dd0 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 >> v1:172.21.15.134:6789/0 conn(0x7fdc24101140 legacy=0x7fdc2406a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 >> v1:172.21.15.28:6789/0 conn(0x7fdc2410d970 legacy=0x7fdc241a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc241acb30 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc241abb80 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc28bbd700 10 client.57584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc241ac040 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc28bbd700 1 librados: init done 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.317+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdc14003b40 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.318+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdc14005c60 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.318+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc04005120 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.319+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdc140036a0 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.320+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(286..286 src has 53..286) ==== 9720+0+0 (unknown 2397305642 0 0) 0x7fdc14089ae0 con 0x7fdc24109dd0 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.320+0000 7fdc1b7fe700 10 client.57584.objecter ms_dispatch 0x7fdc24001550 osd_map(286..286 src has 53..286) 2024-03-30T15:00:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.320+0000 7fdc1b7fe700 3 client.57584.objecter handle_osd_map got epochs [286,286] > 0 2024-03-30T15:00:50.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.320+0000 7fdc1b7fe700 3 client.57584.objecter handle_osd_map decoding full epoch 286 2024-03-30T15:00:50.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.320+0000 7fdc1b7fe700 20 client.57584.objecter dump_active .. 0 homeless 2024-03-30T15:00:50.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.321+0000 7fdc1b7fe700 10 client.57584.objecter ms_handle_connect 0x7fdc100704c0 2024-03-30T15:00:50.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.328+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdc140566c0 con 0x7fdc24109dd0 2024-03-30T15:00:50.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:50.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:50.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:00:50.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:50.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:00:50.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:50.547+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fdc04005410 con 0x7fdc24109dd0 2024-03-30T15:00:50.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:50.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:50.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:50.811 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:50.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:50.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:50.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.1d deep-scrub starts 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.1d deep-scrub ok 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.8 deep-scrub starts 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.8 deep-scrub ok 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.3 scrub starts 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: 17.3 scrub ok 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.1d deep-scrub starts 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.1d deep-scrub ok 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.8 deep-scrub starts 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.8 deep-scrub ok 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.3 scrub starts 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: 17.3 scrub ok 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:50.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:50 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.1d deep-scrub starts 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.1d deep-scrub ok 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.8 deep-scrub starts 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.8 deep-scrub ok 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: osdmap e285: 8 total, 8 up, 8 in 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]: dispatch 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-30T15:00:50.906 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.3 scrub starts 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: 17.3 scrub ok 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi028-92999-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3107101299' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi028-92979-38"}]': finished 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: osdmap e286: 8 total, 8 up, 8 in 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:50.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:50 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:50.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:51.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:51.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:51.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.217+0000 7fdc1b7fe700 1 -- 172.21.15.28:0/4049064069 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v287) ==== 87+0+0 (unknown 1756738107 0 0) 0x7fdc1405a110 con 0x7fdc24109dd0 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.223+0000 7fdc28bbd700 10 librados: watch_flush enter 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.224+0000 7fdc28bbd700 10 librados: watch_flush exit 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.224+0000 7fdc28bbd700 20 client.57584.objecter shutdown clearing up homeless session... 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.224+0000 7fdc28bbd700 10 client.57584.objecter successfully canceled tick 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.226+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdc100704c0 legacy=0x7fdc10072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.226+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 >> v1:172.21.15.28:6790/0 conn(0x7fdc24109dd0 legacy=0x7fdc2406ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.226+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 shutdown_connections 2024-03-30T15:00:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.226+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 >> 172.21.15.28:0/4049064069 conn(0x7fdc240fc9f0 msgr2=0x7fdc241105c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:51.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.229+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 shutdown_connections 2024-03-30T15:00:51.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.230+0000 7fdc28bbd700 1 -- 172.21.15.28:0/4049064069 wait complete. 2024-03-30T15:00:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:51.230+0000 7fdc28bbd700 1 librados: shutdown 2024-03-30T15:00:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:00:51.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:51.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:51.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:51.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:51.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:51.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:51.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: 42.16 scrub starts 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: 42.16 scrub ok 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4049064069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.57584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]': finished 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.57584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: 42.16 scrub starts 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: 42.16 scrub ok 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4049064069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.57584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]': finished 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.57584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:51.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:51.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:51.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:51.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:51.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: 42.16 scrub starts 2024-03-30T15:00:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: 42.16 scrub ok 2024-03-30T15:00:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4049064069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.57584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-31", "mode": "writeback"}]': finished 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi028-92979-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.57584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:00:51.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: osdmap e287: 8 total, 8 up, 8 in 2024-03-30T15:00:51.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:51.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:51.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:00:52.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:52.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:52.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:52.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:52.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:52.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:52.502 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: 101.1e deep-scrub starts 2024-03-30T15:00:52.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: 101.1e deep-scrub ok 2024-03-30T15:00:52.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:52.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:52.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: pgmap v371: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:52.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:52.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T15:00:52.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:52.506 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:00:52.506 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24892.0:2 22.10 22:0c63179d:test-rados-api-smithi028-93390-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:00:52.507 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 177.44367323399999, 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16686387499999999, 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24892", 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1207320814", 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:00:52.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056000+0000", 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.897e-06 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056007+0000", 2024-03-30T15:00:52.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8290000000000002e-06 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056018+0000", 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1647999999999999e-05 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056029+0000", 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0702e-05 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056154+0000", 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012552099999999999 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056165+0000", 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0147e-05 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097196+0000", 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041031265999999997 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097299+0000", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010320200000000001 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097434+0000", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013449600000000001 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.104018+0000", 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.006584047 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.184696+0000", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080678643999999994 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222775+0000", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038078911999999999 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222823+0000", 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7997000000000001e-05 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222862+0000", 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8566999999999999e-05 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24832.0:14 10.1b 10:db53fca8:test-rados-api-smithi028-93032-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 177.200606975, 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99005185600000001, 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.517 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24832", 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2605619531", 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299065+0000", 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4160000000000001e-06 2024-03-30T15:00:52.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299071+0000", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8510000000000004e-06 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299081+0000", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8670000000000006e-06 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299087+0000", 2024-03-30T15:00:52.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7559999999999996e-06 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299131+0000", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4267999999999997e-05 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299138+0000", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2779999999999999e-06 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.199889+0000", 2024-03-30T15:00:52.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.90075121199999997 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.200039+0000", 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015004800000000001 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.200192+0000", 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015276299999999999 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.209842+0000", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0096505029999999995 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.220748+0000", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010905475 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289041+0000", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.068293870000000007 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289083+0000", 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1668000000000002e-05 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289116+0000", 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2880999999999999e-05 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35288.0:2 53.0 53:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 175.08752182200001, 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3211012919999998, 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35288", 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3734394375", 2024-03-30T15:00:52.525 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412151+0000", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5829999999999999e-06 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412159+0000", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7840000000000004e-06 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412168+0000", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6229999999999992e-06 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412175+0000", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7240000000000003e-06 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412220+0000", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5077999999999998e-05 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412226+0000", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8359999999999997e-06 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729625+0000", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3173991090000001 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729721+0000", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6324999999999997e-05 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:00:52.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729844+0000", 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012299199999999999 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733011+0000", 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031669369999999999 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733063+0000", 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1668000000000001e-05 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733112+0000", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9196999999999997e-05 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733163+0000", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1159000000000003e-05 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733251+0000", 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7276999999999998e-05 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.34853.0:2059 11.7 e116/110 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head v 116'1, mlcod=0'0)", 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 174.52957802899999, 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060642963000000001, 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970096+0000", 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3900000000000002e-06 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983905+0000", 2024-03-30T15:00:52.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013808289 2024-03-30T15:00:52.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983934+0000", 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8943999999999999e-05 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983958+0000", 2024-03-30T15:00:52.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4706e-05 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.984028+0000", 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9355000000000003e-05 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.984046+0000", 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8637000000000001e-05 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030684+0000", 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.046637985999999999 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030736+0000", 2024-03-30T15:00:52.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1656000000000001e-05 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.34853.0:2069 11.7 e116/110 11:e17e9421:test-rados-api-smithi028-93257-17::big:head v 116'4, mlcod=116'2)", 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 174.20146350499999, 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.056908669000000002, 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.538 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298210+0000", 2024-03-30T15:00:52.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.508e-06 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306447+0000", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0082367710000000004 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306462+0000", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.506e-05 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306473+0000", 2024-03-30T15:00:52.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0987e-05 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306529+0000", 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6245000000000001e-05 2024-03-30T15:00:52.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306548+0000", 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8896e-05 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355055+0000", 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.048506794999999998 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355116+0000", 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1407000000000003e-05 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35393.0:3 55.14 55:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:00:52.543 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 173.94532022000001, 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.176375602, 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35393", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3403500224", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:00:52.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554353+0000", 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.99e-06 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554359+0000", 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2380000000000002e-06 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554645+0000", 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028636500000000001 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554656+0000", 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0383e-05 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.555818+0000", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001162649 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.555833+0000", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5004999999999999e-05 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730652+0000", 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.174818852 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730726+0000", 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4120000000000002e-05 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.34853.0:2074 11.15 e117/110 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head v 117'1, mlcod=0'0)", 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.670514+0000", 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 173.82915732500001, 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053794176999999999, 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.548 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.670514+0000", 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.670514+0000", 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.670517+0000", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8459999999999999e-06 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.694080+0000", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023563041 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.694104+0000", 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3935999999999999e-05 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.694114+0000", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0073e-05 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.694174+0000", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0822000000000001e-05 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.694194+0000", 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9656e-05 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.724240+0000", 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030046255000000001 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.724308+0000", 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7547999999999998e-05 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25357.0:2 59.15 59:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:00:52.552 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 170.71072002599999, 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.083096852999999998, 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25357", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4132935590", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:00:52.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788953+0000", 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.125e-06 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788959+0000", 2024-03-30T15:00:52.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.285e-06 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788974+0000", 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4980999999999999e-05 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788986+0000", 2024-03-30T15:00:52.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1239e-05 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789049+0000", 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3813999999999996e-05 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789060+0000", 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0365e-05 2024-03-30T15:00:52.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.809307+0000", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020247586000000001 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.809316+0000", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.867e-06 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860035+0000", 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.050718621999999998 2024-03-30T15:00:52.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860156+0000", 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120742 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860291+0000", 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013580600000000001 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.862487+0000", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002195865 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.865606+0000", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031187010000000002 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.871952+0000", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0063455330000000004 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.872002+0000", 2024-03-30T15:00:52.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0482000000000002e-05 2024-03-30T15:00:52.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.872048+0000", 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5840000000000002e-05 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:10 104.a 104:503bb488:test-rados-api-smithi028-93441-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 150.444183581, 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.23059840000000001, 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.563 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 10 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055490+0000", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.289e-06 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055499+0000", 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8130000000000002e-06 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055511+0000", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1936000000000001e-05 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055521+0000", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0862e-05 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076707+0000", 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.021186060999999999 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076816+0000", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108422 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076977+0000", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160798 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.081552+0000", 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045756740000000001 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.118486+0000", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036933360999999998 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285988+0000", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16750200700000001 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286037+0000", 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9163999999999999e-05 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286086+0000", 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9013e-05 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46169.0:2 128.0 128:0384ab8e:::oid:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 146.91372813699999, 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.399107962, 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.569 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46169", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2731295739", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585945+0000", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7409999999999999e-06 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585952+0000", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1110000000000002e-06 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585961+0000", 2024-03-30T15:00:52.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6439999999999995e-06 2024-03-30T15:00:52.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585972+0000", 2024-03-30T15:00:52.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0444e-05 2024-03-30T15:00:52.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.586033+0000", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0677999999999997e-05 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.586041+0000", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7020000000000007e-06 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.960969+0000", 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3749282319999998 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.961084+0000", 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114992 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,6", 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.961221+0000", 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013679 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.962152+0000", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00093109599999999996 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.962664+0000", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00051144500000000002 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.984986+0000", 2024-03-30T15:00:52.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022322437000000001 2024-03-30T15:00:52.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.985022+0000", 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5281000000000003e-05 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.985051+0000", 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9369e-05 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2080 11.1b 11:df4c7475:test-rados-api-smithi028-93257-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:00:52.579 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 143.46186424499999, 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.156258491, 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:00:52.580 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2080 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037808+0000", 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.54e-07 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037811+0000", 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.737e-06 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037816+0000", 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5689999999999998e-06 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037820+0000", 2024-03-30T15:00:52.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0799999999999999e-06 2024-03-30T15:00:52.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037856+0000", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6032999999999999e-05 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037932+0000", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5718999999999998e-05 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.112269+0000", 2024-03-30T15:00:52.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.074337635999999999 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.123357+0000", 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.011087211 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.123994+0000", 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00063726099999999997 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.193965+0000", 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069971071999999995 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.194023+0000", 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7599999999999997e-05 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.194065+0000", 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2719000000000002e-05 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.588 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46235.0:777 130.f 130:f18d14f3:::test_770:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 141.191472624, 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058944257, 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.589 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46235", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 777 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:00:52.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308201+0000", 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3910000000000001e-06 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308208+0000", 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9299999999999997e-06 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308223+0000", 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4952e-05 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308231+0000", 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1880000000000002e-06 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308292+0000", 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1274999999999999e-05 2024-03-30T15:00:52.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308402+0000", 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109546 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.367143+0000", 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058740975000000001 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:28 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 0~128] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-30T15:00:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:00:52.595 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 140.32494468600001, 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.070825352999999994, 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 28 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174728+0000", 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.761e-06 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174743+0000", 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.485e-05 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174754+0000", 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1413999999999999e-05 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174765+0000", 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0715e-05 2024-03-30T15:00:52.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174834+0000", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8806999999999994e-05 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174967+0000", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013262199999999999 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.175110+0000", 2024-03-30T15:00:52.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014391499999999999 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.176336+0000", 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001225603 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.183259+0000", 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0069225720000000001 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245465+0000", 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062206836000000001 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245511+0000", 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5689999999999998e-05 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245552+0000", 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0568000000000002e-05 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:35 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 128~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 139.316484239, 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14093118900000001, 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.603 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 35 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183189+0000", 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7710000000000001e-06 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183197+0000", 2024-03-30T15:00:52.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.977e-06 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183208+0000", 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1868999999999999e-05 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183219+0000", 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1049e-05 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.191898+0000", 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0086782060000000008 2024-03-30T15:00:52.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.192015+0000", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011683 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.192223+0000", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000208383 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.199864+0000", 2024-03-30T15:00:52.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076411250000000003 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.199969+0000", 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104924 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324017+0000", 2024-03-30T15:00:52.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.124047831 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324072+0000", 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5185000000000001e-05 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324118+0000", 2024-03-30T15:00:52.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6038999999999997e-05 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.43456.0:2 141.1b 141:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 139.29774621999999, 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.141261546, 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.611 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.43456", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3977280399", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201927+0000", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201940+0000", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.344e-05 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201952+0000", 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.189e-05 2024-03-30T15:00:52.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201962+0000", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.628e-06 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.208727+0000", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067653949999999996 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.208737+0000", 2024-03-30T15:00:52.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0134e-05 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337524+0000", 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1287864860000001 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337650+0000", 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012597699999999999 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337783+0000", 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013337200000000001 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.340765+0000", 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029821169999999998 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.342269+0000", 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015038790000000001 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343072+0000", 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00080310199999999998 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343137+0000", 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4850000000000004e-05 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343186+0000", 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9258000000000002e-05 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.618 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:42 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 128~128] snapc 8=[8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 137.19300864900001, 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12159848600000001, 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 42 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306664+0000", 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.877e-06 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306674+0000", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3179999999999996e-06 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306685+0000", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1630999999999999e-05 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306696+0000", 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0427e-05 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.331819+0000", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025123125 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.331826+0000", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1829999999999997e-06 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337586+0000", 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057599330000000001 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337695+0000", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109527 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337897+0000", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020184399999999999 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.340104+0000", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022068370000000001 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.365863+0000", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025758547 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427962+0000", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062099013000000002 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428068+0000", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106545 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428261+0000", 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019267899999999999 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.48940.0:2 169.1b 169:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.295121114, 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0270853879999999, 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.625 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.48940", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/728758827", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:00:52.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204552+0000", 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204560+0000", 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4739999999999999e-06 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216581+0000", 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012021041 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216592+0000", 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1868000000000001e-05 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216715+0000", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012253599999999999 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216724+0000", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5960000000000004e-06 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224347+0000", 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0076235769999999 2024-03-30T15:00:52.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224466+0000", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118509 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224603+0000", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013684600000000001 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.225507+0000", 2024-03-30T15:00:52.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090434700000000001 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.226930+0000", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00142267 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231488+0000", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004558472 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231539+0000", 2024-03-30T15:00:52.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0834999999999997e-05 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231635+0000", 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6477999999999993e-05 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.48946.0:2 170.0 170:0384ab8e:::oid:head [append 0~128,setxattr attr1 (11) in=5b,setxattr attr2 (256) in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 125.29348020400001, 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012141285, 2024-03-30T15:00:52.632 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.48946", 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/417824189", 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206193+0000", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1280000000000002e-06 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206205+0000", 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1595e-05 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206215+0000", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0397000000000001e-05 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206223+0000", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7859999999999997e-06 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206739+0000", 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00051627499999999996 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206756+0000", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7142999999999998e-05 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212703+0000", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0059471760000001 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212819+0000", 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115915 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212988+0000", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016922899999999999 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.216595+0000", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036063169999999999 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.217141+0000", 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000545901 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218232+0000", 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010908669999999999 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218287+0000", 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5744e-05 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218332+0000", 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4811999999999997e-05 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24069.0:1 209.1s0 209:8f59135b:test-rados-api-smithi028-93257-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 85.294024424, 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0606089889999999, 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.639 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24069", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/556303511", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205648+0000", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5939999999999999e-06 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205655+0000", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3509999999999998e-06 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205664+0000", 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0880000000000001e-06 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205671+0000", 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2819999999999999e-06 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.209177+0000", 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035059549999999998 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.209181+0000", 2024-03-30T15:00:52.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7529999999999999e-06 2024-03-30T15:00:52.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.255633+0000", 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.046452484 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.255983+0000", 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000349656 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.256346+0000", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036334200000000001 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.256897+0000", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055132599999999999 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.266188+0000", 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0092902729999999999 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.266256+0000", 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7885000000000005e-05 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57506.0:1 249.6s0 249:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 20.284784058, 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0321323259999999, 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:00:52.646 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57506", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/174647268", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:00:52.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214890+0000", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5160000000000001e-06 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214899+0000", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1560000000000001e-06 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214912+0000", 2024-03-30T15:00:52.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3456e-05 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214925+0000", 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3271000000000001e-05 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.215120+0000", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019473900000000001 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.215125+0000", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3480000000000003e-06 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242234+0000", 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.027108057 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242343+0000", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109604 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242674+0000", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033041199999999998 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.243135+0000", 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000461108 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.246953+0000", 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038186940000000001 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.247019+0000", 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5964999999999996e-05 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:00:52.652 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:00:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:52.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: 101.1e deep-scrub starts 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: 101.1e deep-scrub ok 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: pgmap v371: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: 101.1e deep-scrub starts 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: 101.1e deep-scrub ok 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: pgmap v371: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 319 B/s, 4 objects/s recovering 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: osdmap e288: 8 total, 8 up, 8 in 2024-03-30T15:00:52.807 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]: dispatch 2024-03-30T15:00:52.807 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:52.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- 172.21.15.28:0/4167898083 >> v1:172.21.15.28:6790/0 conn(0x7f384810ea80 legacy=0x7f3848110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:52.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- 172.21.15.28:0/4167898083 shutdown_connections 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- 172.21.15.28:0/4167898083 >> 172.21.15.28:0/4167898083 conn(0x7f38480fdb20 msgr2=0x7f38480fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- 172.21.15.28:0/4167898083 shutdown_connections 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- 172.21.15.28:0/4167898083 wait complete. 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 librados: starting msgr at 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 librados: starting objecter 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 Processor -- start 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 -- start start 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 librados: setting wanted keys 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.816+0000 7f384e827700 1 librados: calling monclient init 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f384e827700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38481a9370 con 0x7f3848102250 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f384e827700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38481aa590 con 0x7f384810ea80 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f384e827700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38481ab7d0 con 0x7f384810aee0 2024-03-30T15:00:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f3847fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3848102250 0x7f3848107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42456/0 (socket says 172.21.15.28:42456) 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f3847fff700 1 -- 172.21.15.28:0/3943711483 learned_addr learned my addr 172.21.15.28:0/3943711483 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38477fe700 1 --1- 172.21.15.28:0/3943711483 >> v1:172.21.15.28:6790/0 conn(0x7f384810aee0 0x7f38481a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49530/0 (socket says 172.21.15.28:49530) 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter ms_handle_connect 0x7f384810aee0 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter ms_handle_connect 0x7f3848102250 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter ms_handle_connect 0x7f384810ea80 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266543466 0 0) 0x7f38481ab7d0 con 0x7f384810aee0 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38340031f0 con 0x7f384810aee0 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609942080 0 0) 0x7f38481aa590 con 0x7f384810ea80 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38481ab7d0 con 0x7f384810ea80 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934696517 0 0) 0x7f38481a9370 con 0x7f3848102250 2024-03-30T15:00:52.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38481aa590 con 0x7f3848102250 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3881228965 0 0) 0x7f38340031f0 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38481a9370 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.817+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 756190120 0 0) 0x7f38481ab7d0 con 0x7f384810ea80 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38340031f0 con 0x7f384810ea80 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38300018b0 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38400048c0 con 0x7f384810ea80 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 293060018 0 0) 0x7f38481aa590 con 0x7f3848102250 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38481ab7d0 con 0x7f3848102250 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3838001130 con 0x7f3848102250 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 429112870 0 0) 0x7f38481a9370 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 >> v1:172.21.15.134:6789/0 conn(0x7f384810ea80 legacy=0x7f38481a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 >> v1:172.21.15.28:6789/0 conn(0x7f3848102250 legacy=0x7f3848107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38481aca10 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f38481a95a0 con 0x7f384810aee0 2024-03-30T15:00:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f384e827700 10 client.57590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f38481a9a60 con 0x7f384810aee0 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f384e827700 1 librados: init done 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3830003b40 con 0x7f384810aee0 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.818+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3830005c70 con 0x7f384810aee0 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.819+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3828005120 con 0x7f384810aee0 2024-03-30T15:00:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.819+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f383001c570 con 0x7f384810aee0 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(288..288 src has 53..288) ==== 9739+0+0 (unknown 3350677622 0 0) 0x7f3830089e60 con 0x7f384810aee0 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 10 client.57590.objecter ms_dispatch 0x7f3848001550 osd_map(288..288 src has 53..288) 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 3 client.57590.objecter handle_osd_map got epochs [288,288] > 0 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 3 client.57590.objecter handle_osd_map decoding full epoch 288 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 20 client.57590.objecter dump_active .. 0 homeless 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.820+0000 7f38457fa700 10 client.57590.objecter ms_handle_connect 0x7f3834070480 2024-03-30T15:00:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:52.824+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3830056a40 con 0x7f384810aee0 2024-03-30T15:00:52.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:52.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:52.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:53.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.027+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3828000b90 con 0x7f3834070480 2024-03-30T15:00:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.046+0000 7f38457fa700 1 -- 172.21.15.28:0/3943711483 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+742590 (unknown 2965378022 0 430412179) 0x7f3828000b90 con 0x7f3834070480 2024-03-30T15:00:53.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:00:53.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.052+0000 7f384e827700 10 librados: watch_flush enter 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 10 librados: watch_flush exit 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 20 client.57590.objecter shutdown clearing up homeless session... 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 10 client.57590.objecter successfully canceled tick 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3834070480 legacy=0x7f3834072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 >> v1:172.21.15.28:6790/0 conn(0x7f384810aee0 legacy=0x7f38481a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 shutdown_connections 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 >> 172.21.15.28:0/3943711483 conn(0x7f38480fdb20 msgr2=0x7f3848107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 shutdown_connections 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 -- 172.21.15.28:0/3943711483 wait complete. 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:53.053+0000 7f384e827700 1 librados: shutdown 2024-03-30T15:00:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:00:53.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:53.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:53.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:00:53.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:53.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:53.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:53.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:53.488 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:53.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:53.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:53.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:53.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:53.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:53.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: from='client.57590 v1:172.21.15.28:0/3943711483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T15:00:53.898 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:53.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: from='client.57590 v1:172.21.15.28:0/3943711483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:53.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:53.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: from='client.57590 v1:172.21.15.28:0/3943711483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi028-92979-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-31"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: osdmap e289: 8 total, 8 up, 8 in 2024-03-30T15:00:53.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]: dispatch 2024-03-30T15:00:53.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-30T15:00:52.209983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793741,"num_objects":333,"num_object_clones":1,"num_object_copies":999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1394,"num_read_kb":1146,"num_write":1256,"num_write_kb":11248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1772834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2564,"ondisk_log_size":2564,"up":873,"acting":873,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":878,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":590488,"kb_used_data":53584,"kb_used_omap":12,"kb_used_meta":536691,"kb_avail":749370728,"statfs":{"total":767960285184,"available":767355625472,"internally_reserved":0,"allocated":54870016,"data_stored":36908835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":549572144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-128,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-5,"num_write_kb":-3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000507"},"pg_stats":[{"pgid":"255.1f","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.236503+0000","last_change":"2024-03-30T15:00:48.234543+0000","last_active":"2024-03-30T15:00:48.236503+0000","last_peered":"2024-03-30T15:00:48.236503+0000","last_clean":"2024-03-30T15:00:48.236503+0000","last_became_active":"2024-03-30T15:00:48.234239+0000","last_became_peered":"2024-03-30T15:00:48.234239+0000","last_unstale":"2024-03-30T15:00:48.236503+0000","last_undegraded":"2024-03-30T15:00:48.236503+0000","last_fullsized":"2024-03-30T15:00:48.236503+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.839765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.1e","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.240600+0000","last_change":"2024-03-30T15:00:48.237170+0000","last_active":"2024-03-30T15:00:48.240600+0000","last_peered":"2024-03-30T15:00:48.240600+0000","last_clean":"2024-03-30T15:00:48.240600+0000","last_became_active":"2024-03-30T15:00:48.236778+0000","last_became_peered":"2024-03-30T15:00:48.236778+0000","last_unstale":"2024-03-30T15:00:48.240600+0000","last_undegraded":"2024-03-30T15:00:48.240600+0000","last_fullsized":"2024-03-30T15:00:48.240600+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.1d","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228191+0000","last_change":"2024-03-30T15:00:48.227239+0000","last_active":"2024-03-30T15:00:48.228191+0000","last_peered":"2024-03-30T15:00:48.228191+0000","last_clean":"2024-03-30T15:00:48.228191+0000","last_became_active":"2024-03-30T15:00:48.226511+0000","last_became_peered":"2024-03-30T15:00:48.226511+0000","last_unstale":"2024-03-30T15:00:48.228191+0000","last_undegraded":"2024-03-30T15:00:48.228191+0000","last_fullsized":"2024-03-30T15:00:48.228191+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:52.937230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.1c","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.243885+0000","last_change":"2024-03-30T15:00:48.243431+0000","last_active":"2024-03-30T15:00:48.243885+0000","last_peered":"2024-03-30T15:00:48.243885+0000","last_clean":"2024-03-30T15:00:48.243885+0000","last_became_active":"2024-03-30T15:00:48.241695+0000","last_became_peered":"2024-03-30T15:00:48.241695+0000","last_unstale":"2024-03-30T15:00:48.243885+0000","last_undegraded":"2024-03-30T15:00:48.243885+0000","last_fullsized":"2024-03-30T15:00:48.243885+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.158821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.1b","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.243002+0000","last_change":"2024-03-30T15:00:48.242729+0000","last_active":"2024-03-30T15:00:48.243002+0000","last_peered":"2024-03-30T15:00:48.243002+0000","last_clean":"2024-03-30T15:00:48.243002+0000","last_became_active":"2024-03-30T15:00:48.242414+0000","last_became_peered":"2024-03-30T15:00:48.242414+0000","last_unstale":"2024-03-30T15:00:48.243002+0000","last_undegraded":"2024-03-30T15:00:48.243002+0000","last_fullsized":"2024-03-30T15:00:48.243002+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.910939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.1a","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.237533+0000","last_change":"2024-03-30T15:00:48.231338+0000","last_active":"2024-03-30T15:00:48.237533+0000","last_peered":"2024-03-30T15:00:48.237533+0000","last_clean":"2024-03-30T15:00:48.237533+0000","last_became_active":"2024-03-30T15:00:48.231022+0000","last_became_peered":"2024-03-30T15:00:48.231022+0000","last_unstale":"2024-03-30T15:00:48.237533+0000","last_undegraded":"2024-03-30T15:00:48.237533+0000","last_fullsized":"2024-03-30T15:00:48.237533+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:52.031027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.19","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.233342+0000","last_change":"2024-03-30T15:00:48.231245+0000","last_active":"2024-03-30T15:00:48.233342+0000","last_peered":"2024-03-30T15:00:48.233342+0000","last_clean":"2024-03-30T15:00:48.233342+0000","last_became_active":"2024-03-30T15:00:48.230875+0000","last_became_peered":"2024-03-30T15:00:48.230875+0000","last_unstale":"2024-03-30T15:00:48.233342+0000","last_undegraded":"2024-03-30T15:00:48.233342+0000","last_fullsized":"2024-03-30T15:00:48.233342+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:51.639575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.18","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238452+0000","last_change":"2024-03-30T15:00:48.237718+0000","last_active":"2024-03-30T15:00:48.238452+0000","last_peered":"2024-03-30T15:00:48.238452+0000","last_clean":"2024-03-30T15:00:48.238452+0000","last_became_active":"2024-03-30T15:00:48.237079+0000","last_became_peered":"2024-03-30T15:00:48.237079+0000","last_unstale":"2024-03-30T15:00:48.238452+0000","last_undegraded":"2024-03-30T15:00:48.238452+0000","last_fullsized":"2024-03-30T15:00:48.238452+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:08.376442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.17","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.243118+0000","last_change":"2024-03-30T15:00:48.240405+0000","last_active":"2024-03-30T15:00:48.243118+0000","last_peered":"2024-03-30T15:00:48.243118+0000","last_clean":"2024-03-30T15:00:48.243118+0000","last_became_active":"2024-03-30T15:00:48.240046+0000","last_became_peered":"2024-03-30T15:00:48.240046+0000","last_unstale":"2024-03-30T15:00:48.243118+0000","last_undegraded":"2024-03-30T15:00:48.243118+0000","last_fullsized":"2024-03-30T15:00:48.243118+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.848640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.16","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238416+0000","last_change":"2024-03-30T15:00:48.235956+0000","last_active":"2024-03-30T15:00:48.238416+0000","last_peered":"2024-03-30T15:00:48.238416+0000","last_clean":"2024-03-30T15:00:48.238416+0000","last_became_active":"2024-03-30T15:00:48.235665+0000","last_became_peered":"2024-03-30T15:00:48.235665+0000","last_unstale":"2024-03-30T15:00:48.238416+0000","last_undegraded":"2024-03-30T15:00:48.238416+0000","last_fullsized":"2024-03-30T15:00:48.238416+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.349898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.15","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.241642+0000","last_change":"2024-03-30T15:00:48.238712+0000","last_active":"2024-03-30T15:00:48.241642+0000","last_peered":"2024-03-30T15:00:48.241642+0000","last_clean":"2024-03-30T15:00:48.241642+0000","last_became_active":"2024-03-30T15:00:48.238396+0000","last_became_peered":"2024-03-30T15:00:48.238396+0000","last_unstale":"2024-03-30T15:00:48.241642+0000","last_undegraded":"2024-03-30T15:00:48.241642+0000","last_fullsized":"2024-03-30T15:00:48.241642+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.595661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.14","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.236254+0000","last_change":"2024-03-30T15:00:48.235451+0000","last_active":"2024-03-30T15:00:48.236254+0000","last_peered":"2024-03-30T15:00:48.236254+0000","last_clean":"2024-03-30T15:00:48.236254+0000","last_became_active":"2024-03-30T15:00:48.235132+0000","last_became_peered":"2024-03-30T15:00:48.235132+0000","last_unstale":"2024-03-30T15:00:48.236254+0000","last_undegraded":"2024-03-30T15:00:48.236254+0000","last_fullsized":"2024-03-30T15:00:48.236254+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:15.421458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.13","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.239239+0000","last_change":"2024-03-30T15:00:48.238256+0000","last_active":"2024-03-30T15:00:48.239239+0000","last_peered":"2024-03-30T15:00:48.239239+0000","last_clean":"2024-03-30T15:00:48.239239+0000","last_became_active":"2024-03-30T15:00:48.237958+0000","last_became_peered":"2024-03-30T15:00:48.237958+0000","last_unstale":"2024-03-30T15:00:48.239239+0000","last_undegraded":"2024-03-30T15:00:48.239239+0000","last_fullsized":"2024-03-30T15:00:48.239239+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.861490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.12","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.240365+0000","last_change":"2024-03-30T15:00:48.239031+0000","last_active":"2024-03-30T15:00:48.240365+0000","last_peered":"2024-03-30T15:00:48.240365+0000","last_clean":"2024-03-30T15:00:48.240365+0000","last_became_active":"2024-03-30T15:00:48.238754+0000","last_became_peered":"2024-03-30T15:00:48.238754+0000","last_unstale":"2024-03-30T15:00:48.240365+0000","last_undegraded":"2024-03-30T15:00:48.240365+0000","last_fullsized":"2024-03-30T15:00:48.240365+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:02.251194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.11","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.236907+0000","last_change":"2024-03-30T15:00:48.233081+0000","last_active":"2024-03-30T15:00:48.236907+0000","last_peered":"2024-03-30T15:00:48.236907+0000","last_clean":"2024-03-30T15:00:48.236907+0000","last_became_active":"2024-03-30T15:00:48.232797+0000","last_became_peered":"2024-03-30T15:00:48.232797+0000","last_unstale":"2024-03-30T15:00:48.236907+0000","last_undegraded":"2024-03-30T15:00:48.236907+0000","last_fullsized":"2024-03-30T15:00:48.236907+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:13.301514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.10","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.242590+0000","last_change":"2024-03-30T15:00:48.241337+0000","last_active":"2024-03-30T15:00:48.242590+0000","last_peered":"2024-03-30T15:00:48.242590+0000","last_clean":"2024-03-30T15:00:48.242590+0000","last_became_active":"2024-03-30T15:00:48.240717+0000","last_became_peered":"2024-03-30T15:00:48.240717+0000","last_unstale":"2024-03-30T15:00:48.242590+0000","last_undegraded":"2024-03-30T15:00:48.242590+0000","last_fullsized":"2024-03-30T15:00:48.242590+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.621566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.f","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.241818+0000","last_change":"2024-03-30T15:00:48.239567+0000","last_active":"2024-03-30T15:00:48.241818+0000","last_peered":"2024-03-30T15:00:48.241818+0000","last_clean":"2024-03-30T15:00:48.241818+0000","last_became_active":"2024-03-30T15:00:48.239196+0000","last_became_peered":"2024-03-30T15:00:48.239196+0000","last_unstale":"2024-03-30T15:00:48.241818+0000","last_undegraded":"2024-03-30T15:00:48.241818+0000","last_fullsized":"2024-03-30T15:00:48.241818+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:56.133819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.e","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238957+0000","last_change":"2024-03-30T15:00:48.237236+0000","last_active":"2024-03-30T15:00:48.238957+0000","last_peered":"2024-03-30T15:00:48.238957+0000","last_clean":"2024-03-30T15:00:48.238957+0000","last_became_active":"2024-03-30T15:00:48.236919+0000","last_became_peered":"2024-03-30T15:00:48.236919+0000","last_unstale":"2024-03-30T15:00:48.238957+0000","last_undegraded":"2024-03-30T15:00:48.238957+0000","last_fullsized":"2024-03-30T15:00:48.238957+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.552364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"255.d","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.243913+0000","last_change":"2024-03-30T15:00:48.243136+0000","last_active":"2024-03-30T15:00:48.243913+0000","last_peered":"2024-03-30T15:00:48.243913+0000","last_clean":"2024-03-30T15:00:48.243913+0000","last_became_active":"2024-03-30T15:00:48.242663+0000","last_became_peered":"2024-03-30T15:00:48.242663+0000","last_unstale":"2024-03-30T15:00:48.243913+0000","last_undegraded":"2024-03-30T15:00:48.243913+0000","last_fullsized":"2024-03-30T15:00:48.243913+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.757055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.c","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.240416+0000","last_change":"2024-03-30T15:00:48.237860+0000","last_active":"2024-03-30T15:00:48.240416+0000","last_peered":"2024-03-30T15:00:48.240416+0000","last_clean":"2024-03-30T15:00:48.240416+0000","last_became_active":"2024-03-30T15:00:48.237504+0000","last_became_peered":"2024-03-30T15:00:48.237504+0000","last_unstale":"2024-03-30T15:00:48.240416+0000","last_undegraded":"2024-03-30T15:00:48.240416+0000","last_fullsized":"2024-03-30T15:00:48.240416+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:55.064323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.b","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.235707+0000","last_change":"2024-03-30T15:00:48.225901+0000","last_active":"2024-03-30T15:00:48.235707+0000","last_peered":"2024-03-30T15:00:48.235707+0000","last_clean":"2024-03-30T15:00:48.235707+0000","last_became_active":"2024-03-30T15:00:48.224420+0000","last_became_peered":"2024-03-30T15:00:48.224420+0000","last_unstale":"2024-03-30T15:00:48.235707+0000","last_undegraded":"2024-03-30T15:00:48.235707+0000","last_fullsized":"2024-03-30T15:00:48.235707+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:54.681255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.a","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.237322+0000","last_change":"2024-03-30T15:00:48.236748+0000","last_active":"2024-03-30T15:00:48.237322+0000","last_peered":"2024-03-30T15:00:48.237322+0000","last_clean":"2024-03-30T15:00:48.237322+0000","last_became_active":"2024-03-30T15:00:48.236455+0000","last_became_peered":"2024-03-30T15:00:48.236455+0000","last_unstale":"2024-03-30T15:00:48.237322+0000","last_undegraded":"2024-03-30T15:00:48.237322+0000","last_fullsized":"2024-03-30T15:00:48.237322+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.018970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.9","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.234144+0000","last_change":"2024-03-30T15:00:48.233184+0000","last_active":"2024-03-30T15:00:48.234144+0000","last_peered":"2024-03-30T15:00:48.234144+0000","last_clean":"2024-03-30T15:00:48.234144+0000","last_became_active":"2024-03-30T15:00:48.232879+0000","last_became_peered":"2024-03-30T15:00:48.232879+0000","last_unstale":"2024-03-30T15:00:48.234144+0000","last_undegraded":"2024-03-30T15:00:48.234144+0000","last_fullsized":"2024-03-30T15:00:48.234144+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:10.928250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.8","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238919+0000","last_change":"2024-03-30T15:00:48.235983+0000","last_active":"2024-03-30T15:00:48.238919+0000","last_peered":"2024-03-30T15:00:48.238919+0000","last_clean":"2024-03-30T15:00:48.238919+0000","last_became_active":"2024-03-30T15:00:48.235495+0000","last_became_peered":"2024-03-30T15:00:48.235495+0000","last_unstale":"2024-03-30T15:00:48.238919+0000","last_undegraded":"2024-03-30T15:00:48.238919+0000","last_fullsized":"2024-03-30T15:00:48.238919+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.388984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"255.2","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.236949+0000","last_change":"2024-03-30T15:00:48.228375+0000","last_active":"2024-03-30T15:00:48.236949+0000","last_peered":"2024-03-30T15:00:48.236949+0000","last_clean":"2024-03-30T15:00:48.236949+0000","last_became_active":"2024-03-30T15:00:48.228083+0000","last_became_peered":"2024-03-30T15:00:48.228083+0000","last_unstale":"2024-03-30T15:00:48.236949+0000","last_undegraded":"2024-03-30T15:00:48.236949+0000","last_fullsized":"2024-03-30T15:00:48.236949+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.727644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.3","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238598+0000","last_change":"2024-03-30T15:00:48.236810+0000","last_active":"2024-03-30T15:00:48.238598+0000","last_peered":"2024-03-30T15:00:48.238598+0000","last_clean":"2024-03-30T15:00:48.238598+0000","last_became_active":"2024-03-30T15:00:48.236543+0000","last_became_peered":"2024-03-30T15:00:48.236543+0000","last_unstale":"2024-03-30T15:00:48.238598+0000","last_undegraded":"2024-03-30T15:00:48.238598+0000","last_fullsized":"2024-03-30T15:00:48.238598+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.548613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.0","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.243121+0000","last_change":"2024-03-30T15:00:48.241408+0000","last_active":"2024-03-30T15:00:48.243121+0000","last_peered":"2024-03-30T15:00:48.243121+0000","last_clean":"2024-03-30T15:00:48.243121+0000","last_became_active":"2024-03-30T15:00:48.241132+0000","last_became_peered":"2024-03-30T15:00:48.241132+0000","last_unstale":"2024-03-30T15:00:48.243121+0000","last_undegraded":"2024-03-30T15:00:48.243121+0000","last_fullsized":"2024-03-30T15:00:48.243121+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.599757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.1","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.241594+0000","last_change":"2024-03-30T15:00:48.240440+0000","last_active":"2024-03-30T15:00:48.241594+0000","last_peered":"2024-03-30T15:00:48.241594+0000","last_clean":"2024-03-30T15:00:48.241594+0000","last_became_active":"2024-03-30T15:00:48.240165+0000","last_became_peered":"2024-03-30T15:00:48.240165+0000","last_unstale":"2024-03-30T15:00:48.241594+0000","last_undegraded":"2024-03-30T15:00:48.241594+0000","last_fullsized":"2024-03-30T15:00:48.241594+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:11.753801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.6","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.242720+0000","last_change":"2024-03-30T15:00:48.239722+0000","last_active":"2024-03-30T15:00:48.242720+0000","last_peered":"2024-03-30T15:00:48.242720+0000","last_clean":"2024-03-30T15:00:48.242720+0000","last_became_active":"2024-03-30T15:00:48.239164+0000","last_became_peered":"2024-03-30T15:00:48.239164+0000","last_unstale":"2024-03-30T15:00:48.242720+0000","last_undegraded":"2024-03-30T15:00:48.242720+0000","last_fullsized":"2024-03-30T15:00:48.242720+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.386371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.7","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.241837+0000","last_change":"2024-03-30T15:00:48.241041+0000","last_active":"2024-03-30T15:00:48.241837+0000","last_peered":"2024-03-30T15:00:48.241837+0000","last_clean":"2024-03-30T15:00:48.241837+0000","last_became_active":"2024-03-30T15:00:48.240699+0000","last_became_peered":"2024-03-30T15:00:48.240699+0000","last_unstale":"2024-03-30T15:00:48.241837+0000","last_undegraded":"2024-03-30T15:00:48.241837+0000","last_fullsized":"2024-03-30T15:00:48.241837+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:13.073496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.4","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.242548+0000","last_change":"2024-03-30T15:00:48.241258+0000","last_active":"2024-03-30T15:00:48.242548+0000","last_peered":"2024-03-30T15:00:48.242548+0000","last_clean":"2024-03-30T15:00:48.242548+0000","last_became_active":"2024-03-30T15:00:48.240826+0000","last_became_peered":"2024-03-30T15:00:48.240826+0000","last_unstale":"2024-03-30T15:00:48.242548+0000","last_undegraded":"2024-03-30T15:00:48.242548+0000","last_fullsized":"2024-03-30T15:00:48.242548+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:58.407083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.5","version":"0'0","reported_seq":10,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.238988+0000","last_change":"2024-03-30T15:00:48.234160+0000","last_active":"2024-03-30T15:00:48.238988+0000","last_peered":"2024-03-30T15:00:48.238988+0000","last_clean":"2024-03-30T15:00:48.238988+0000","last_became_active":"2024-03-30T15:00:48.233674+0000","last_became_peered":"2024-03-30T15:00:48.233674+0000","last_unstale":"2024-03-30T15:00:48.238988+0000","last_undegraded":"2024-03-30T15:00:48.238988+0000","last_fullsized":"2024-03-30T15:00:48.238988+0000","mapping_epoch":283,"log_start":"0'0","ondisk_log_start":"0'0","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:47.204945+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.204945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:08.003241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":463,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215195+0000","last_change":"2024-03-30T15:00:43.233147+0000","last_active":"2024-03-30T15:00:48.215195+0000","last_peered":"2024-03-30T15:00:48.215195+0000","last_clean":"2024-03-30T15:00:48.215195+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:00:48.215195+0000","last_undegraded":"2024-03-30T15:00:48.215195+0000","last_fullsized":"2024-03-30T15:00:48.215195+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01065052,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"229'85","reported_seq":494,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221210+0000","last_change":"2024-03-30T15:00:43.233337+0000","last_active":"2024-03-30T15:00:48.221210+0000","last_peered":"2024-03-30T15:00:48.221210+0000","last_clean":"2024-03-30T15:00:48.221210+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:00:48.221210+0000","last_undegraded":"2024-03-30T15:00:48.221210+0000","last_fullsized":"2024-03-30T15:00:48.221210+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025393439999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"230'87","reported_seq":498,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217333+0000","last_change":"2024-03-30T15:00:43.224129+0000","last_active":"2024-03-30T15:00:48.217333+0000","last_peered":"2024-03-30T15:00:48.217333+0000","last_clean":"2024-03-30T15:00:48.217333+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:00:48.217333+0000","last_undegraded":"2024-03-30T15:00:48.217333+0000","last_fullsized":"2024-03-30T15:00:48.217333+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001687252,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"204'102","reported_seq":515,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216574+0000","last_change":"2024-03-30T15:00:43.225813+0000","last_active":"2024-03-30T15:00:48.216574+0000","last_peered":"2024-03-30T15:00:48.216574+0000","last_clean":"2024-03-30T15:00:48.216574+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:00:48.216574+0000","last_undegraded":"2024-03-30T15:00:48.216574+0000","last_fullsized":"2024-03-30T15:00:48.216574+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001343522,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"204'34","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.213486+0000","last_change":"2024-03-30T15:00:43.231075+0000","last_active":"2024-03-30T15:00:48.213486+0000","last_peered":"2024-03-30T15:00:48.213486+0000","last_clean":"2024-03-30T15:00:48.213486+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:00:48.213486+0000","last_undegraded":"2024-03-30T15:00:48.213486+0000","last_fullsized":"2024-03-30T15:00:48.213486+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087880479999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"283'65","reported_seq":544,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216322+0000","last_change":"2024-03-30T15:00:47.743389+0000","last_active":"2024-03-30T15:00:48.216322+0000","last_peered":"2024-03-30T15:00:48.216322+0000","last_clean":"2024-03-30T15:00:48.216322+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:00:48.216322+0000","last_undegraded":"2024-03-30T15:00:48.216322+0000","last_fullsized":"2024-03-30T15:00:48.216322+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'65","last_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_deep_scrub":"283'65","last_deep_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.743297+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:08.914882+0000","scrub_duration":10,"objects_trimmed":1,"snaptrim_duration":0.011100334,"stat_sum":{"num_bytes":4194341,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"222'34","reported_seq":452,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228358+0000","last_change":"2024-03-30T15:00:43.235577+0000","last_active":"2024-03-30T15:00:48.228358+0000","last_peered":"2024-03-30T15:00:48.228358+0000","last_clean":"2024-03-30T15:00:48.228358+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:00:48.228358+0000","last_undegraded":"2024-03-30T15:00:48.228358+0000","last_fullsized":"2024-03-30T15:00:48.228358+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021958989999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"204'51","reported_seq":467,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224304+0000","last_change":"2024-03-30T15:00:43.234880+0000","last_active":"2024-03-30T15:00:48.224304+0000","last_peered":"2024-03-30T15:00:48.224304+0000","last_clean":"2024-03-30T15:00:48.224304+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:00:48.224304+0000","last_undegraded":"2024-03-30T15:00:48.224304+0000","last_fullsized":"2024-03-30T15:00:48.224304+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095938800000000008,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"235'39","reported_seq":459,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216793+0000","last_change":"2024-03-30T15:00:43.212808+0000","last_active":"2024-03-30T15:00:48.216793+0000","last_peered":"2024-03-30T15:00:48.216793+0000","last_clean":"2024-03-30T15:00:48.216793+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:00:48.216793+0000","last_undegraded":"2024-03-30T15:00:48.216793+0000","last_fullsized":"2024-03-30T15:00:48.216793+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034693419999999998,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221099+0000","last_change":"2024-03-30T15:00:43.213650+0000","last_active":"2024-03-30T15:00:48.221099+0000","last_peered":"2024-03-30T15:00:48.221099+0000","last_clean":"2024-03-30T15:00:48.221099+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:00:48.221099+0000","last_undegraded":"2024-03-30T15:00:48.221099+0000","last_fullsized":"2024-03-30T15:00:48.221099+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041086289999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"0'0","reported_seq":425,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217435+0000","last_change":"2024-03-30T15:00:43.225727+0000","last_active":"2024-03-30T15:00:48.217435+0000","last_peered":"2024-03-30T15:00:48.217435+0000","last_clean":"2024-03-30T15:00:48.217435+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:00:48.217435+0000","last_undegraded":"2024-03-30T15:00:48.217435+0000","last_fullsized":"2024-03-30T15:00:48.217435+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030227269999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"204'17","reported_seq":442,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220486+0000","last_change":"2024-03-30T15:00:43.224756+0000","last_active":"2024-03-30T15:00:48.220486+0000","last_peered":"2024-03-30T15:00:48.220486+0000","last_clean":"2024-03-30T15:00:48.220486+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:00:48.220486+0000","last_undegraded":"2024-03-30T15:00:48.220486+0000","last_fullsized":"2024-03-30T15:00:48.220486+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020847700000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"217'17","reported_seq":438,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227159+0000","last_change":"2024-03-30T15:00:43.236048+0000","last_active":"2024-03-30T15:00:48.227159+0000","last_peered":"2024-03-30T15:00:48.227159+0000","last_clean":"2024-03-30T15:00:48.227159+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:00:48.227159+0000","last_undegraded":"2024-03-30T15:00:48.227159+0000","last_fullsized":"2024-03-30T15:00:48.227159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013371539999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"241'73","reported_seq":494,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220756+0000","last_change":"2024-03-30T15:00:43.217166+0000","last_active":"2024-03-30T15:00:48.220756+0000","last_peered":"2024-03-30T15:00:48.220756+0000","last_clean":"2024-03-30T15:00:48.220756+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:00:48.220756+0000","last_undegraded":"2024-03-30T15:00:48.220756+0000","last_fullsized":"2024-03-30T15:00:48.220756+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001827506,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"233'51","reported_seq":470,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217933+0000","last_change":"2024-03-30T15:00:43.214272+0000","last_active":"2024-03-30T15:00:48.217933+0000","last_peered":"2024-03-30T15:00:48.217933+0000","last_clean":"2024-03-30T15:00:48.217933+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:00:48.217933+0000","last_undegraded":"2024-03-30T15:00:48.217933+0000","last_fullsized":"2024-03-30T15:00:48.217933+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00145184,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"267'90","reported_seq":506,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211752+0000","last_change":"2024-03-30T15:00:43.231273+0000","last_active":"2024-03-30T15:00:48.211752+0000","last_peered":"2024-03-30T15:00:48.211752+0000","last_clean":"2024-03-30T15:00:48.211752+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:00:48.211752+0000","last_undegraded":"2024-03-30T15:00:48.211752+0000","last_fullsized":"2024-03-30T15:00:48.211752+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'90","last_scrub_stamp":"2024-03-30T15:00:41.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.653662+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:08.451324+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0083710509999999991,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"204'34","reported_seq":455,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222348+0000","last_change":"2024-03-30T15:00:43.227678+0000","last_active":"2024-03-30T15:00:48.222348+0000","last_peered":"2024-03-30T15:00:48.222348+0000","last_clean":"2024-03-30T15:00:48.222348+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:00:48.222348+0000","last_undegraded":"2024-03-30T15:00:48.222348+0000","last_fullsized":"2024-03-30T15:00:48.222348+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047107750000000004,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":526,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220991+0000","last_change":"2024-03-30T15:00:43.219728+0000","last_active":"2024-03-30T15:00:48.220991+0000","last_peered":"2024-03-30T15:00:48.220991+0000","last_clean":"2024-03-30T15:00:48.220991+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:00:48.220991+0000","last_undegraded":"2024-03-30T15:00:48.220991+0000","last_fullsized":"2024-03-30T15:00:48.220991+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:00:30.724310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.724310+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.727404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018219289999999999,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"204'34","reported_seq":464,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220799+0000","last_change":"2024-03-30T15:00:43.228511+0000","last_active":"2024-03-30T15:00:48.220799+0000","last_peered":"2024-03-30T15:00:48.220799+0000","last_clean":"2024-03-30T15:00:48.220799+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:00:48.220799+0000","last_undegraded":"2024-03-30T15:00:48.220799+0000","last_fullsized":"2024-03-30T15:00:48.220799+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T14:59:47.247116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.247116+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.179101+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0019213380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217774+0000","last_change":"2024-03-30T15:00:43.218761+0000","last_active":"2024-03-30T15:00:48.217774+0000","last_peered":"2024-03-30T15:00:48.217774+0000","last_clean":"2024-03-30T15:00:48.217774+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:00:48.217774+0000","last_undegraded":"2024-03-30T15:00:48.217774+0000","last_fullsized":"2024-03-30T15:00:48.217774+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:00:37.797402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.797402+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.762439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024570769999999998,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":527,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222703+0000","last_change":"2024-03-30T15:00:43.219557+0000","last_active":"2024-03-30T15:00:48.222703+0000","last_peered":"2024-03-30T15:00:48.222703+0000","last_clean":"2024-03-30T15:00:48.222703+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:00:48.222703+0000","last_undegraded":"2024-03-30T15:00:48.222703+0000","last_fullsized":"2024-03-30T15:00:48.222703+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:00:31.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:31.737537+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:40.721216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017474070000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"248'56","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221360+0000","last_change":"2024-03-30T15:00:43.217112+0000","last_active":"2024-03-30T15:00:48.221360+0000","last_peered":"2024-03-30T15:00:48.221360+0000","last_clean":"2024-03-30T15:00:48.221360+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:00:48.221360+0000","last_undegraded":"2024-03-30T15:00:48.221360+0000","last_fullsized":"2024-03-30T15:00:48.221360+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00150314,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"204'51","reported_seq":475,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227250+0000","last_change":"2024-03-30T15:00:43.232136+0000","last_active":"2024-03-30T15:00:48.227250+0000","last_peered":"2024-03-30T15:00:48.227250+0000","last_clean":"2024-03-30T15:00:48.227250+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:00:48.227250+0000","last_undegraded":"2024-03-30T15:00:48.227250+0000","last_fullsized":"2024-03-30T15:00:48.227250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:00:29.555003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:29.555003+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.118282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074406610000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"235'87","reported_seq":495,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.229983+0000","last_change":"2024-03-30T15:00:43.234991+0000","last_active":"2024-03-30T15:00:48.229983+0000","last_peered":"2024-03-30T15:00:48.229983+0000","last_clean":"2024-03-30T15:00:48.229983+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:00:48.229983+0000","last_undegraded":"2024-03-30T15:00:48.229983+0000","last_fullsized":"2024-03-30T15:00:48.229983+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036646000000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"204'85","reported_seq":503,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221350+0000","last_change":"2024-03-30T15:00:43.230968+0000","last_active":"2024-03-30T15:00:48.221350+0000","last_peered":"2024-03-30T15:00:48.221350+0000","last_clean":"2024-03-30T15:00:48.221350+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:00:48.221350+0000","last_undegraded":"2024-03-30T15:00:48.221350+0000","last_fullsized":"2024-03-30T15:00:48.221350+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'85","last_scrub_stamp":"2024-03-30T15:00:35.252370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.252370+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.108496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053428640000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211638+0000","last_change":"2024-03-30T15:00:43.230248+0000","last_active":"2024-03-30T15:00:48.211638+0000","last_peered":"2024-03-30T15:00:48.211638+0000","last_clean":"2024-03-30T15:00:48.211638+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:00:48.211638+0000","last_undegraded":"2024-03-30T15:00:48.211638+0000","last_fullsized":"2024-03-30T15:00:48.211638+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:02.714228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.714228+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.283062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0084702249999999996,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"248'39","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211103+0000","last_change":"2024-03-30T15:00:43.227563+0000","last_active":"2024-03-30T15:00:48.211103+0000","last_peered":"2024-03-30T15:00:48.211103+0000","last_clean":"2024-03-30T15:00:48.211103+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:00:48.211103+0000","last_undegraded":"2024-03-30T15:00:48.211103+0000","last_fullsized":"2024-03-30T15:00:48.211103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:04.710962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.710962+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:08.332527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061256050000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"238'103","reported_seq":529,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218320+0000","last_change":"2024-03-30T15:00:43.219491+0000","last_active":"2024-03-30T15:00:48.218320+0000","last_peered":"2024-03-30T15:00:48.218320+0000","last_clean":"2024-03-30T15:00:48.218320+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:00:48.218320+0000","last_undegraded":"2024-03-30T15:00:48.218320+0000","last_fullsized":"2024-03-30T15:00:48.218320+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'103","last_scrub_stamp":"2024-03-30T15:00:35.786699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.786699+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.175171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022054919999999999,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"204'22","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230552+0000","last_change":"2024-03-30T15:00:43.235889+0000","last_active":"2024-03-30T15:00:48.230552+0000","last_peered":"2024-03-30T15:00:48.230552+0000","last_clean":"2024-03-30T15:00:48.230552+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:00:48.230552+0000","last_undegraded":"2024-03-30T15:00:48.230552+0000","last_fullsized":"2024-03-30T15:00:48.230552+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091356760000000006,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"0'0","reported_seq":435,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219146+0000","last_change":"2024-03-30T15:00:43.234537+0000","last_active":"2024-03-30T15:00:48.219146+0000","last_peered":"2024-03-30T15:00:48.219146+0000","last_clean":"2024-03-30T15:00:48.219146+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:00:48.219146+0000","last_undegraded":"2024-03-30T15:00:48.219146+0000","last_fullsized":"2024-03-30T15:00:48.219146+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.644063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.644063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:37.806049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0083022960000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218035+0000","last_change":"2024-03-30T15:00:43.214361+0000","last_active":"2024-03-30T15:00:48.218035+0000","last_peered":"2024-03-30T15:00:48.218035+0000","last_clean":"2024-03-30T15:00:48.218035+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:00:48.218035+0000","last_undegraded":"2024-03-30T15:00:48.218035+0000","last_fullsized":"2024-03-30T15:00:48.218035+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001262364,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"204'102","reported_seq":519,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228120+0000","last_change":"2024-03-30T15:00:43.233284+0000","last_active":"2024-03-30T15:00:48.228120+0000","last_peered":"2024-03-30T15:00:48.228120+0000","last_clean":"2024-03-30T15:00:48.228120+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:00:48.228120+0000","last_undegraded":"2024-03-30T15:00:48.228120+0000","last_fullsized":"2024-03-30T15:00:48.228120+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T14:59:48.278271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.278271+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.225938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078062200000000002,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226029+0000","last_change":"2024-03-30T14:57:56.252990+0000","last_active":"2024-03-30T15:00:48.226029+0000","last_peered":"2024-03-30T15:00:48.226029+0000","last_clean":"2024-03-30T15:00:48.226029+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:00:48.226029+0000","last_undegraded":"2024-03-30T15:00:48.226029+0000","last_fullsized":"2024-03-30T15:00:48.226029+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219805+0000","last_change":"2024-03-30T14:59:10.495352+0000","last_active":"2024-03-30T15:00:48.219805+0000","last_peered":"2024-03-30T15:00:48.219805+0000","last_clean":"2024-03-30T15:00:48.219805+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:00:48.219805+0000","last_undegraded":"2024-03-30T15:00:48.219805+0000","last_fullsized":"2024-03-30T15:00:48.219805+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.495272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.495272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":357,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228102+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:00:48.228102+0000","last_peered":"2024-03-30T15:00:48.228102+0000","last_clean":"2024-03-30T15:00:48.228102+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:00:48.228102+0000","last_undegraded":"2024-03-30T15:00:48.228102+0000","last_fullsized":"2024-03-30T15:00:48.228102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.937324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":350,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228510+0000","last_change":"2024-03-30T14:57:56.238237+0000","last_active":"2024-03-30T15:00:48.228510+0000","last_peered":"2024-03-30T15:00:48.228510+0000","last_clean":"2024-03-30T15:00:48.228510+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:00:48.228510+0000","last_undegraded":"2024-03-30T15:00:48.228510+0000","last_fullsized":"2024-03-30T15:00:48.228510+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":350,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217323+0000","last_change":"2024-03-30T14:57:56.261442+0000","last_active":"2024-03-30T15:00:48.217323+0000","last_peered":"2024-03-30T15:00:48.217323+0000","last_clean":"2024-03-30T15:00:48.217323+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:00:48.217323+0000","last_undegraded":"2024-03-30T15:00:48.217323+0000","last_fullsized":"2024-03-30T15:00:48.217323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211123+0000","last_change":"2024-03-30T14:59:56.788394+0000","last_active":"2024-03-30T15:00:48.211123+0000","last_peered":"2024-03-30T15:00:48.211123+0000","last_clean":"2024-03-30T15:00:48.211123+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:00:48.211123+0000","last_undegraded":"2024-03-30T15:00:48.211123+0000","last_fullsized":"2024-03-30T15:00:48.211123+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.788278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.788278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:25.851592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223565+0000","last_change":"2024-03-30T14:57:56.244353+0000","last_active":"2024-03-30T15:00:48.223565+0000","last_peered":"2024-03-30T15:00:48.223565+0000","last_clean":"2024-03-30T15:00:48.223565+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:00:48.223565+0000","last_undegraded":"2024-03-30T15:00:48.223565+0000","last_fullsized":"2024-03-30T15:00:48.223565+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226294+0000","last_change":"2024-03-30T14:57:56.239220+0000","last_active":"2024-03-30T15:00:48.226294+0000","last_peered":"2024-03-30T15:00:48.226294+0000","last_clean":"2024-03-30T15:00:48.226294+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:00:48.226294+0000","last_undegraded":"2024-03-30T15:00:48.226294+0000","last_fullsized":"2024-03-30T15:00:48.226294+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224621+0000","last_change":"2024-03-30T14:57:56.240474+0000","last_active":"2024-03-30T15:00:48.224621+0000","last_peered":"2024-03-30T15:00:48.224621+0000","last_clean":"2024-03-30T15:00:48.224621+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:00:48.224621+0000","last_undegraded":"2024-03-30T15:00:48.224621+0000","last_fullsized":"2024-03-30T15:00:48.224621+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223563+0000","last_change":"2024-03-30T15:00:43.330329+0000","last_active":"2024-03-30T15:00:48.223563+0000","last_peered":"2024-03-30T15:00:48.223563+0000","last_clean":"2024-03-30T15:00:48.223563+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:00:48.223563+0000","last_undegraded":"2024-03-30T15:00:48.223563+0000","last_fullsized":"2024-03-30T15:00:48.223563+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.330250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.998934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227983+0000","last_change":"2024-03-30T14:57:56.253494+0000","last_active":"2024-03-30T15:00:48.227983+0000","last_peered":"2024-03-30T15:00:48.227983+0000","last_clean":"2024-03-30T15:00:48.227983+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:00:48.227983+0000","last_undegraded":"2024-03-30T15:00:48.227983+0000","last_fullsized":"2024-03-30T15:00:48.227983+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.213594+0000","last_change":"2024-03-30T14:59:52.803098+0000","last_active":"2024-03-30T15:00:48.213594+0000","last_peered":"2024-03-30T15:00:48.213594+0000","last_clean":"2024-03-30T15:00:48.213594+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:00:48.213594+0000","last_undegraded":"2024-03-30T15:00:48.213594+0000","last_fullsized":"2024-03-30T15:00:48.213594+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.803012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.803012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.827147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219246+0000","last_change":"2024-03-30T14:57:56.253984+0000","last_active":"2024-03-30T15:00:48.219246+0000","last_peered":"2024-03-30T15:00:48.219246+0000","last_clean":"2024-03-30T15:00:48.219246+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:00:48.219246+0000","last_undegraded":"2024-03-30T15:00:48.219246+0000","last_fullsized":"2024-03-30T15:00:48.219246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215497+0000","last_change":"2024-03-30T15:00:40.305048+0000","last_active":"2024-03-30T15:00:48.215497+0000","last_peered":"2024-03-30T15:00:48.215497+0000","last_clean":"2024-03-30T15:00:48.215497+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:00:48.215497+0000","last_undegraded":"2024-03-30T15:00:48.215497+0000","last_fullsized":"2024-03-30T15:00:48.215497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:40.304903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:40.304903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.820755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211701+0000","last_change":"2024-03-30T14:59:17.774212+0000","last_active":"2024-03-30T15:00:48.211701+0000","last_peered":"2024-03-30T15:00:48.211701+0000","last_clean":"2024-03-30T15:00:48.211701+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:00:48.211701+0000","last_undegraded":"2024-03-30T15:00:48.211701+0000","last_fullsized":"2024-03-30T15:00:48.211701+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.774120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215288+0000","last_change":"2024-03-30T14:59:45.812010+0000","last_active":"2024-03-30T15:00:48.215288+0000","last_peered":"2024-03-30T15:00:48.215288+0000","last_clean":"2024-03-30T15:00:48.215288+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:00:48.215288+0000","last_undegraded":"2024-03-30T15:00:48.215288+0000","last_fullsized":"2024-03-30T15:00:48.215288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:45.811929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:45.811929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.196406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230329+0000","last_change":"2024-03-30T14:57:56.248423+0000","last_active":"2024-03-30T15:00:48.230329+0000","last_peered":"2024-03-30T15:00:48.230329+0000","last_clean":"2024-03-30T15:00:48.230329+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:00:48.230329+0000","last_undegraded":"2024-03-30T15:00:48.230329+0000","last_fullsized":"2024-03-30T15:00:48.230329+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227643+0000","last_change":"2024-03-30T14:57:56.226093+0000","last_active":"2024-03-30T15:00:48.227643+0000","last_peered":"2024-03-30T15:00:48.227643+0000","last_clean":"2024-03-30T15:00:48.227643+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:00:48.227643+0000","last_undegraded":"2024-03-30T15:00:48.227643+0000","last_fullsized":"2024-03-30T15:00:48.227643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228312+0000","last_change":"2024-03-30T14:57:56.276413+0000","last_active":"2024-03-30T15:00:48.228312+0000","last_peered":"2024-03-30T15:00:48.228312+0000","last_clean":"2024-03-30T15:00:48.228312+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:00:48.228312+0000","last_undegraded":"2024-03-30T15:00:48.228312+0000","last_fullsized":"2024-03-30T15:00:48.228312+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227326+0000","last_change":"2024-03-30T14:57:56.253085+0000","last_active":"2024-03-30T15:00:48.227326+0000","last_peered":"2024-03-30T15:00:48.227326+0000","last_clean":"2024-03-30T15:00:48.227326+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:00:48.227326+0000","last_undegraded":"2024-03-30T15:00:48.227326+0000","last_fullsized":"2024-03-30T15:00:48.227326+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217348+0000","last_change":"2024-03-30T14:59:55.177743+0000","last_active":"2024-03-30T15:00:48.217348+0000","last_peered":"2024-03-30T15:00:48.217348+0000","last_clean":"2024-03-30T15:00:48.217348+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:00:48.217348+0000","last_undegraded":"2024-03-30T15:00:48.217348+0000","last_fullsized":"2024-03-30T15:00:48.217348+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.177623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.177623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.770057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":357,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219493+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:00:48.219493+0000","last_peered":"2024-03-30T15:00:48.219493+0000","last_clean":"2024-03-30T15:00:48.219493+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:00:48.219493+0000","last_undegraded":"2024-03-30T15:00:48.219493+0000","last_fullsized":"2024-03-30T15:00:48.219493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:00.729852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221139+0000","last_change":"2024-03-30T15:00:32.754787+0000","last_active":"2024-03-30T15:00:48.221139+0000","last_peered":"2024-03-30T15:00:48.221139+0000","last_clean":"2024-03-30T15:00:48.221139+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:00:48.221139+0000","last_undegraded":"2024-03-30T15:00:48.221139+0000","last_fullsized":"2024-03-30T15:00:48.221139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.754592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.754592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.958130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.214591+0000","last_change":"2024-03-30T14:59:13.804909+0000","last_active":"2024-03-30T15:00:48.214591+0000","last_peered":"2024-03-30T15:00:48.214591+0000","last_clean":"2024-03-30T15:00:48.214591+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:00:48.214591+0000","last_undegraded":"2024-03-30T15:00:48.214591+0000","last_fullsized":"2024-03-30T15:00:48.214591+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.804817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.804817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219667+0000","last_change":"2024-03-30T15:00:33.756912+0000","last_active":"2024-03-30T15:00:48.219667+0000","last_peered":"2024-03-30T15:00:48.219667+0000","last_clean":"2024-03-30T15:00:48.219667+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:00:48.219667+0000","last_undegraded":"2024-03-30T15:00:48.219667+0000","last_fullsized":"2024-03-30T15:00:48.219667+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:33.756810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.756810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:33.994340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223660+0000","last_change":"2024-03-30T15:00:03.114650+0000","last_active":"2024-03-30T15:00:48.223660+0000","last_peered":"2024-03-30T15:00:48.223660+0000","last_clean":"2024-03-30T15:00:48.223660+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:00:48.223660+0000","last_undegraded":"2024-03-30T15:00:48.223660+0000","last_fullsized":"2024-03-30T15:00:48.223660+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.114576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.114576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:10.524652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.232533+0000","last_change":"2024-03-30T14:57:56.260622+0000","last_active":"2024-03-30T15:00:48.232533+0000","last_peered":"2024-03-30T15:00:48.232533+0000","last_clean":"2024-03-30T15:00:48.232533+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:00:48.232533+0000","last_undegraded":"2024-03-30T15:00:48.232533+0000","last_fullsized":"2024-03-30T15:00:48.232533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227310+0000","last_change":"2024-03-30T14:57:56.249378+0000","last_active":"2024-03-30T15:00:48.227310+0000","last_peered":"2024-03-30T15:00:48.227310+0000","last_clean":"2024-03-30T15:00:48.227310+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:00:48.227310+0000","last_undegraded":"2024-03-30T15:00:48.227310+0000","last_fullsized":"2024-03-30T15:00:48.227310+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217875+0000","last_change":"2024-03-30T14:59:11.544255+0000","last_active":"2024-03-30T15:00:48.217875+0000","last_peered":"2024-03-30T15:00:48.217875+0000","last_clean":"2024-03-30T15:00:48.217875+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:00:48.217875+0000","last_undegraded":"2024-03-30T15:00:48.217875+0000","last_fullsized":"2024-03-30T15:00:48.217875+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:11.544174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:11.544174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220476+0000","last_change":"2024-03-30T14:57:56.271774+0000","last_active":"2024-03-30T15:00:48.220476+0000","last_peered":"2024-03-30T15:00:48.220476+0000","last_clean":"2024-03-30T15:00:48.220476+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:00:48.220476+0000","last_undegraded":"2024-03-30T15:00:48.220476+0000","last_fullsized":"2024-03-30T15:00:48.220476+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.233966+0000","last_change":"2024-03-30T14:57:56.199752+0000","last_active":"2024-03-30T15:00:48.233966+0000","last_peered":"2024-03-30T15:00:48.233966+0000","last_clean":"2024-03-30T15:00:48.233966+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:00:48.233966+0000","last_undegraded":"2024-03-30T15:00:48.233966+0000","last_fullsized":"2024-03-30T15:00:48.233966+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":349,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223560+0000","last_change":"2024-03-30T14:57:56.228301+0000","last_active":"2024-03-30T15:00:48.223560+0000","last_peered":"2024-03-30T15:00:48.223560+0000","last_clean":"2024-03-30T15:00:48.223560+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:00:48.223560+0000","last_undegraded":"2024-03-30T15:00:48.223560+0000","last_fullsized":"2024-03-30T15:00:48.223560+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219912+0000","last_change":"2024-03-30T14:59:09.501246+0000","last_active":"2024-03-30T15:00:48.219912+0000","last_peered":"2024-03-30T15:00:48.219912+0000","last_clean":"2024-03-30T15:00:48.219912+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:00:48.219912+0000","last_undegraded":"2024-03-30T15:00:48.219912+0000","last_fullsized":"2024-03-30T15:00:48.219912+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.501169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.501169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227730+0000","last_change":"2024-03-30T14:58:15.280305+0000","last_active":"2024-03-30T15:00:48.227730+0000","last_peered":"2024-03-30T15:00:48.227730+0000","last_clean":"2024-03-30T15:00:48.227730+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:00:48.227730+0000","last_undegraded":"2024-03-30T15:00:48.227730+0000","last_fullsized":"2024-03-30T15:00:48.227730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:15.280217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.280217+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221347+0000","last_change":"2024-03-30T14:58:13.189214+0000","last_active":"2024-03-30T15:00:48.221347+0000","last_peered":"2024-03-30T15:00:48.221347+0000","last_clean":"2024-03-30T15:00:48.221347+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:00:48.221347+0000","last_undegraded":"2024-03-30T15:00:48.221347+0000","last_fullsized":"2024-03-30T15:00:48.221347+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.189081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.189081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":539,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.214302+0000","last_change":"2024-03-30T14:58:31.806764+0000","last_active":"2024-03-30T15:00:48.214302+0000","last_peered":"2024-03-30T15:00:48.214302+0000","last_clean":"2024-03-30T15:00:48.214302+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:00:48.214302+0000","last_undegraded":"2024-03-30T15:00:48.214302+0000","last_fullsized":"2024-03-30T15:00:48.214302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:31.806606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.806606+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.231252+0000","last_change":"2024-03-30T14:59:13.422572+0000","last_active":"2024-03-30T15:00:48.231252+0000","last_peered":"2024-03-30T15:00:48.231252+0000","last_clean":"2024-03-30T15:00:48.231252+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:48.231252+0000","last_undegraded":"2024-03-30T15:00:48.231252+0000","last_fullsized":"2024-03-30T15:00:48.231252+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.422453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.422453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216033+0000","last_change":"2024-03-30T15:00:45.239550+0000","last_active":"2024-03-30T15:00:48.216033+0000","last_peered":"2024-03-30T15:00:48.216033+0000","last_clean":"2024-03-30T15:00:48.216033+0000","last_became_active":"2024-03-30T15:00:45.238835+0000","last_became_peered":"2024-03-30T15:00:45.238835+0000","last_unstale":"2024-03-30T15:00:48.216033+0000","last_undegraded":"2024-03-30T15:00:48.216033+0000","last_fullsized":"2024-03-30T15:00:48.216033+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.045488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.045488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:00.553997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219889+0000","last_change":"2024-03-30T15:00:04.553363+0000","last_active":"2024-03-30T15:00:48.219889+0000","last_peered":"2024-03-30T15:00:48.219889+0000","last_clean":"2024-03-30T15:00:48.219889+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:00:48.219889+0000","last_undegraded":"2024-03-30T15:00:48.219889+0000","last_fullsized":"2024-03-30T15:00:48.219889+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.553281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.553281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.976142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":489,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228311+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:00:48.228311+0000","last_peered":"2024-03-30T15:00:48.228311+0000","last_clean":"2024-03-30T15:00:48.228311+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:00:48.228311+0000","last_undegraded":"2024-03-30T15:00:48.228311+0000","last_fullsized":"2024-03-30T15:00:48.228311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:08.635027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":465,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217680+0000","last_change":"2024-03-30T14:59:18.646158+0000","last_active":"2024-03-30T15:00:48.217680+0000","last_peered":"2024-03-30T15:00:48.217680+0000","last_clean":"2024-03-30T15:00:48.217680+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:00:48.217680+0000","last_undegraded":"2024-03-30T15:00:48.217680+0000","last_fullsized":"2024-03-30T15:00:48.217680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:18.646076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:18.646076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":458,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224437+0000","last_change":"2024-03-30T14:59:09.325167+0000","last_active":"2024-03-30T15:00:48.224437+0000","last_peered":"2024-03-30T15:00:48.224437+0000","last_clean":"2024-03-30T15:00:48.224437+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:00:48.224437+0000","last_undegraded":"2024-03-30T15:00:48.224437+0000","last_fullsized":"2024-03-30T15:00:48.224437+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":452,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219296+0000","last_change":"2024-03-30T14:59:07.831752+0000","last_active":"2024-03-30T15:00:48.219296+0000","last_peered":"2024-03-30T15:00:48.219296+0000","last_clean":"2024-03-30T15:00:48.219296+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:00:48.219296+0000","last_undegraded":"2024-03-30T15:00:48.219296+0000","last_fullsized":"2024-03-30T15:00:48.219296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.831675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.831675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.18","version":"115'2","reported_seq":39,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222063+0000","last_change":"2024-03-30T15:00:45.254965+0000","last_active":"2024-03-30T15:00:48.222063+0000","last_peered":"2024-03-30T15:00:48.222063+0000","last_clean":"2024-03-30T15:00:48.222063+0000","last_became_active":"2024-03-30T15:00:45.247625+0000","last_became_peered":"2024-03-30T15:00:45.247625+0000","last_unstale":"2024-03-30T15:00:48.222063+0000","last_undegraded":"2024-03-30T15:00:48.222063+0000","last_fullsized":"2024-03-30T15:00:48.222063+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228910+0000","last_change":"2024-03-30T14:59:54.144187+0000","last_active":"2024-03-30T15:00:48.228910+0000","last_peered":"2024-03-30T15:00:48.228910+0000","last_clean":"2024-03-30T15:00:48.228910+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:00:48.228910+0000","last_undegraded":"2024-03-30T15:00:48.228910+0000","last_fullsized":"2024-03-30T15:00:48.228910+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:54.144084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:54.144084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:04.029603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.231057+0000","last_change":"2024-03-30T14:58:20.234367+0000","last_active":"2024-03-30T15:00:48.231057+0000","last_peered":"2024-03-30T15:00:48.231057+0000","last_clean":"2024-03-30T15:00:48.231057+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:00:48.231057+0000","last_undegraded":"2024-03-30T15:00:48.231057+0000","last_fullsized":"2024-03-30T15:00:48.231057+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:58:20.234281+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.234281+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220912+0000","last_change":"2024-03-30T14:57:29.596356+0000","last_active":"2024-03-30T15:00:48.220912+0000","last_peered":"2024-03-30T15:00:48.220912+0000","last_clean":"2024-03-30T15:00:48.220912+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:00:48.220912+0000","last_undegraded":"2024-03-30T15:00:48.220912+0000","last_fullsized":"2024-03-30T15:00:48.220912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:29.596249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:29.596249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230273+0000","last_change":"2024-03-30T14:58:23.099186+0000","last_active":"2024-03-30T15:00:48.230273+0000","last_peered":"2024-03-30T15:00:48.230273+0000","last_clean":"2024-03-30T15:00:48.230273+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:00:48.230273+0000","last_undegraded":"2024-03-30T15:00:48.230273+0000","last_fullsized":"2024-03-30T15:00:48.230273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.099104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.099104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":367,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220768+0000","last_change":"2024-03-30T15:00:44.895646+0000","last_active":"2024-03-30T15:00:48.220768+0000","last_peered":"2024-03-30T15:00:48.220768+0000","last_clean":"2024-03-30T15:00:48.220768+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:00:48.220768+0000","last_undegraded":"2024-03-30T15:00:48.220768+0000","last_fullsized":"2024-03-30T15:00:48.220768+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.895504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.895504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.201212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1b","version":"115'10","reported_seq":386,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.236045+0000","last_change":"2024-03-30T15:00:45.262813+0000","last_active":"2024-03-30T15:00:48.236045+0000","last_peered":"2024-03-30T15:00:48.236045+0000","last_clean":"2024-03-30T15:00:48.236045+0000","last_became_active":"2024-03-30T15:00:45.243932+0000","last_became_peered":"2024-03-30T15:00:45.243932+0000","last_unstale":"2024-03-30T15:00:48.236045+0000","last_undegraded":"2024-03-30T15:00:48.236045+0000","last_fullsized":"2024-03-30T15:00:48.236045+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:38.719347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:38.719347+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:01.401906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.232964+0000","last_change":"2024-03-30T15:00:01.176797+0000","last_active":"2024-03-30T15:00:48.232964+0000","last_peered":"2024-03-30T15:00:48.232964+0000","last_clean":"2024-03-30T15:00:48.232964+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:00:48.232964+0000","last_undegraded":"2024-03-30T15:00:48.232964+0000","last_fullsized":"2024-03-30T15:00:48.232964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.176673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.080692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":486,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.233063+0000","last_change":"2024-03-30T14:58:31.256279+0000","last_active":"2024-03-30T15:00:48.233063+0000","last_peered":"2024-03-30T15:00:48.233063+0000","last_clean":"2024-03-30T15:00:48.233063+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:00:48.233063+0000","last_undegraded":"2024-03-30T15:00:48.233063+0000","last_fullsized":"2024-03-30T15:00:48.233063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T14:58:31.256139+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.256139+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216302+0000","last_change":"2024-03-30T14:59:07.286916+0000","last_active":"2024-03-30T15:00:48.216302+0000","last_peered":"2024-03-30T15:00:48.216302+0000","last_clean":"2024-03-30T15:00:48.216302+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:00:48.216302+0000","last_undegraded":"2024-03-30T15:00:48.216302+0000","last_fullsized":"2024-03-30T15:00:48.216302+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.286790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.286790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":532,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.231156+0000","last_change":"2024-03-30T14:58:59.314094+0000","last_active":"2024-03-30T15:00:48.231156+0000","last_peered":"2024-03-30T15:00:48.231156+0000","last_clean":"2024-03-30T15:00:48.231156+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:00:48.231156+0000","last_undegraded":"2024-03-30T15:00:48.231156+0000","last_fullsized":"2024-03-30T15:00:48.231156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.314003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.314003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":444,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230616+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:00:48.230616+0000","last_peered":"2024-03-30T15:00:48.230616+0000","last_clean":"2024-03-30T15:00:48.230616+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:00:48.230616+0000","last_undegraded":"2024-03-30T15:00:48.230616+0000","last_fullsized":"2024-03-30T15:00:48.230616+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1a","version":"115'5","reported_seq":30,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225265+0000","last_change":"2024-03-30T15:00:47.055900+0000","last_active":"2024-03-30T15:00:48.225265+0000","last_peered":"2024-03-30T15:00:48.225265+0000","last_clean":"2024-03-30T15:00:48.225265+0000","last_became_active":"2024-03-30T15:00:45.243189+0000","last_became_peered":"2024-03-30T15:00:45.243189+0000","last_unstale":"2024-03-30T15:00:48.225265+0000","last_undegraded":"2024-03-30T15:00:48.225265+0000","last_fullsized":"2024-03-30T15:00:48.225265+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:47.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.055803+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:51.944246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211470+0000","last_change":"2024-03-30T14:59:53.801443+0000","last_active":"2024-03-30T15:00:48.211470+0000","last_peered":"2024-03-30T15:00:48.211470+0000","last_clean":"2024-03-30T15:00:48.211470+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:00:48.211470+0000","last_undegraded":"2024-03-30T15:00:48.211470+0000","last_fullsized":"2024-03-30T15:00:48.211470+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.801332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.801332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.086851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220796+0000","last_change":"2024-03-30T14:58:21.250924+0000","last_active":"2024-03-30T15:00:48.220796+0000","last_peered":"2024-03-30T15:00:48.220796+0000","last_clean":"2024-03-30T15:00:48.220796+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:00:48.220796+0000","last_undegraded":"2024-03-30T15:00:48.220796+0000","last_fullsized":"2024-03-30T15:00:48.220796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:21.250827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.250827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221921+0000","last_change":"2024-03-30T14:58:19.963766+0000","last_active":"2024-03-30T15:00:48.221921+0000","last_peered":"2024-03-30T15:00:48.221921+0000","last_clean":"2024-03-30T15:00:48.221921+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:00:48.221921+0000","last_undegraded":"2024-03-30T15:00:48.221921+0000","last_fullsized":"2024-03-30T15:00:48.221921+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:58:19.963665+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.963665+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":432,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228413+0000","last_change":"2024-03-30T14:59:10.014828+0000","last_active":"2024-03-30T15:00:48.228413+0000","last_peered":"2024-03-30T15:00:48.228413+0000","last_clean":"2024-03-30T15:00:48.228413+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:00:48.228413+0000","last_undegraded":"2024-03-30T15:00:48.228413+0000","last_fullsized":"2024-03-30T15:00:48.228413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:10.014745+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.014745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223339+0000","last_change":"2024-03-30T14:59:02.261089+0000","last_active":"2024-03-30T15:00:48.223339+0000","last_peered":"2024-03-30T15:00:48.223339+0000","last_clean":"2024-03-30T15:00:48.223339+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:00:48.223339+0000","last_undegraded":"2024-03-30T15:00:48.223339+0000","last_fullsized":"2024-03-30T15:00:48.223339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.260964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.260964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219407+0000","last_change":"2024-03-30T15:00:45.252550+0000","last_active":"2024-03-30T15:00:48.219407+0000","last_peered":"2024-03-30T15:00:48.219407+0000","last_clean":"2024-03-30T15:00:48.219407+0000","last_became_active":"2024-03-30T15:00:45.251660+0000","last_became_peered":"2024-03-30T15:00:45.251660+0000","last_unstale":"2024-03-30T15:00:48.219407+0000","last_undegraded":"2024-03-30T15:00:48.219407+0000","last_fullsized":"2024-03-30T15:00:48.219407+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.119871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.119871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:12.960492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":365,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211473+0000","last_change":"2024-03-30T14:59:48.794962+0000","last_active":"2024-03-30T15:00:48.211473+0000","last_peered":"2024-03-30T15:00:48.211473+0000","last_clean":"2024-03-30T15:00:48.211473+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:00:48.211473+0000","last_undegraded":"2024-03-30T15:00:48.211473+0000","last_fullsized":"2024-03-30T15:00:48.211473+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:48.794824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.794824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.182803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":419,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215819+0000","last_change":"2024-03-30T14:59:49.053482+0000","last_active":"2024-03-30T15:00:48.215819+0000","last_peered":"2024-03-30T15:00:48.215819+0000","last_clean":"2024-03-30T15:00:48.215819+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:00:48.215819+0000","last_undegraded":"2024-03-30T15:00:48.215819+0000","last_fullsized":"2024-03-30T15:00:48.215819+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:49.053394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:49.053394+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.339042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":482,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221044+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:00:48.221044+0000","last_peered":"2024-03-30T15:00:48.221044+0000","last_clean":"2024-03-30T15:00:48.221044+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:00:48.221044+0000","last_undegraded":"2024-03-30T15:00:48.221044+0000","last_fullsized":"2024-03-30T15:00:48.221044+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.455990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":460,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219495+0000","last_change":"2024-03-30T14:58:25.170349+0000","last_active":"2024-03-30T15:00:48.219495+0000","last_peered":"2024-03-30T15:00:48.219495+0000","last_clean":"2024-03-30T15:00:48.219495+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:00:48.219495+0000","last_undegraded":"2024-03-30T15:00:48.219495+0000","last_fullsized":"2024-03-30T15:00:48.219495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.170239+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":453,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226741+0000","last_change":"2024-03-30T14:59:56.291563+0000","last_active":"2024-03-30T15:00:48.226741+0000","last_peered":"2024-03-30T15:00:48.226741+0000","last_clean":"2024-03-30T15:00:48.226741+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:00:48.226741+0000","last_undegraded":"2024-03-30T15:00:48.226741+0000","last_fullsized":"2024-03-30T15:00:48.226741+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.291453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.291453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.028131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"115'2","reported_seq":21,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217762+0000","last_change":"2024-03-30T15:00:45.251189+0000","last_active":"2024-03-30T15:00:48.217762+0000","last_peered":"2024-03-30T15:00:48.217762+0000","last_clean":"2024-03-30T15:00:48.217762+0000","last_became_active":"2024-03-30T15:00:45.241122+0000","last_became_peered":"2024-03-30T15:00:45.241122+0000","last_unstale":"2024-03-30T15:00:48.217762+0000","last_undegraded":"2024-03-30T15:00:48.217762+0000","last_fullsized":"2024-03-30T15:00:48.217762+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":365,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.214744+0000","last_change":"2024-03-30T14:59:46.819021+0000","last_active":"2024-03-30T15:00:48.214744+0000","last_peered":"2024-03-30T15:00:48.214744+0000","last_clean":"2024-03-30T15:00:48.214744+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:00:48.214744+0000","last_undegraded":"2024-03-30T15:00:48.214744+0000","last_fullsized":"2024-03-30T15:00:48.214744+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:46.818941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.818941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:52.638110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":541,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221299+0000","last_change":"2024-03-30T15:00:30.490223+0000","last_active":"2024-03-30T15:00:48.221299+0000","last_peered":"2024-03-30T15:00:48.221299+0000","last_clean":"2024-03-30T15:00:48.221299+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:00:48.221299+0000","last_undegraded":"2024-03-30T15:00:48.221299+0000","last_fullsized":"2024-03-30T15:00:48.221299+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:00:30.490104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.490104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.592628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":482,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222217+0000","last_change":"2024-03-30T14:59:10.283945+0000","last_active":"2024-03-30T15:00:48.222217+0000","last_peered":"2024-03-30T15:00:48.222217+0000","last_clean":"2024-03-30T15:00:48.222217+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:00:48.222217+0000","last_undegraded":"2024-03-30T15:00:48.222217+0000","last_fullsized":"2024-03-30T15:00:48.222217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.283859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":464,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230861+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:00:48.230861+0000","last_peered":"2024-03-30T15:00:48.230861+0000","last_clean":"2024-03-30T15:00:48.230861+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:00:48.230861+0000","last_undegraded":"2024-03-30T15:00:48.230861+0000","last_fullsized":"2024-03-30T15:00:48.230861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":447,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219410+0000","last_change":"2024-03-30T14:58:23.157209+0000","last_active":"2024-03-30T15:00:48.219410+0000","last_peered":"2024-03-30T15:00:48.219410+0000","last_clean":"2024-03-30T15:00:48.219410+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:00:48.219410+0000","last_undegraded":"2024-03-30T15:00:48.219410+0000","last_fullsized":"2024-03-30T15:00:48.219410+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.157095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.157095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"115'5","reported_seq":40,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222991+0000","last_change":"2024-03-30T15:00:45.253584+0000","last_active":"2024-03-30T15:00:48.222991+0000","last_peered":"2024-03-30T15:00:48.222991+0000","last_clean":"2024-03-30T15:00:48.222991+0000","last_became_active":"2024-03-30T15:00:45.243133+0000","last_became_peered":"2024-03-30T15:00:45.243133+0000","last_unstale":"2024-03-30T15:00:48.222991+0000","last_undegraded":"2024-03-30T15:00:48.222991+0000","last_fullsized":"2024-03-30T15:00:48.222991+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T14:59:15.591612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.591612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.213110+0000","last_change":"2024-03-30T14:59:19.788455+0000","last_active":"2024-03-30T15:00:48.213110+0000","last_peered":"2024-03-30T15:00:48.213110+0000","last_clean":"2024-03-30T15:00:48.213110+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:00:48.213110+0000","last_undegraded":"2024-03-30T15:00:48.213110+0000","last_fullsized":"2024-03-30T15:00:48.213110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.788329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.788329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218053+0000","last_change":"2024-03-30T14:59:01.518493+0000","last_active":"2024-03-30T15:00:48.218053+0000","last_peered":"2024-03-30T15:00:48.218053+0000","last_clean":"2024-03-30T15:00:48.218053+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:00:48.218053+0000","last_undegraded":"2024-03-30T15:00:48.218053+0000","last_fullsized":"2024-03-30T15:00:48.218053+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.518414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.518414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":498,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221860+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:00:48.221860+0000","last_peered":"2024-03-30T15:00:48.221860+0000","last_clean":"2024-03-30T15:00:48.221860+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:00:48.221860+0000","last_undegraded":"2024-03-30T15:00:48.221860+0000","last_fullsized":"2024-03-30T15:00:48.221860+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":466,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224243+0000","last_change":"2024-03-30T14:59:13.210437+0000","last_active":"2024-03-30T15:00:48.224243+0000","last_peered":"2024-03-30T15:00:48.224243+0000","last_clean":"2024-03-30T15:00:48.224243+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:00:48.224243+0000","last_undegraded":"2024-03-30T15:00:48.224243+0000","last_fullsized":"2024-03-30T15:00:48.224243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.210302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.210302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":454,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217774+0000","last_change":"2024-03-30T14:59:19.135083+0000","last_active":"2024-03-30T15:00:48.217774+0000","last_peered":"2024-03-30T15:00:48.217774+0000","last_clean":"2024-03-30T15:00:48.217774+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:00:48.217774+0000","last_undegraded":"2024-03-30T15:00:48.217774+0000","last_fullsized":"2024-03-30T15:00:48.217774+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.135006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.135006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"115'5","reported_seq":22,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222211+0000","last_change":"2024-03-30T15:00:45.258419+0000","last_active":"2024-03-30T15:00:48.222211+0000","last_peered":"2024-03-30T15:00:48.222211+0000","last_clean":"2024-03-30T15:00:48.222211+0000","last_became_active":"2024-03-30T15:00:45.244278+0000","last_became_peered":"2024-03-30T15:00:45.244278+0000","last_unstale":"2024-03-30T15:00:48.222211+0000","last_undegraded":"2024-03-30T15:00:48.222211+0000","last_fullsized":"2024-03-30T15:00:48.222211+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.109233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.109233+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.992633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219401+0000","last_change":"2024-03-30T14:57:54.026635+0000","last_active":"2024-03-30T15:00:48.219401+0000","last_peered":"2024-03-30T15:00:48.219401+0000","last_clean":"2024-03-30T15:00:48.219401+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:00:48.219401+0000","last_undegraded":"2024-03-30T15:00:48.219401+0000","last_fullsized":"2024-03-30T15:00:48.219401+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230390+0000","last_change":"2024-03-30T14:59:15.207502+0000","last_active":"2024-03-30T15:00:48.230390+0000","last_peered":"2024-03-30T15:00:48.230390+0000","last_clean":"2024-03-30T15:00:48.230390+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:00:48.230390+0000","last_undegraded":"2024-03-30T15:00:48.230390+0000","last_fullsized":"2024-03-30T15:00:48.230390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.207413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.207413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219298+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:00:48.219298+0000","last_peered":"2024-03-30T15:00:48.219298+0000","last_clean":"2024-03-30T15:00:48.219298+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:00:48.219298+0000","last_undegraded":"2024-03-30T15:00:48.219298+0000","last_fullsized":"2024-03-30T15:00:48.219298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:59.702850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":834,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221482+0000","last_change":"2024-03-30T14:58:21.947917+0000","last_active":"2024-03-30T15:00:48.221482+0000","last_peered":"2024-03-30T15:00:48.221482+0000","last_clean":"2024-03-30T15:00:48.221482+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:00:48.221482+0000","last_undegraded":"2024-03-30T15:00:48.221482+0000","last_fullsized":"2024-03-30T15:00:48.221482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:58:21.947837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.947837+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":458,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216426+0000","last_change":"2024-03-30T14:59:05.612363+0000","last_active":"2024-03-30T15:00:48.216426+0000","last_peered":"2024-03-30T15:00:48.216426+0000","last_clean":"2024-03-30T15:00:48.216426+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:00:48.216426+0000","last_undegraded":"2024-03-30T15:00:48.216426+0000","last_fullsized":"2024-03-30T15:00:48.216426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.612262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.612262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223955+0000","last_change":"2024-03-30T14:58:20.292759+0000","last_active":"2024-03-30T15:00:48.223955+0000","last_peered":"2024-03-30T15:00:48.223955+0000","last_clean":"2024-03-30T15:00:48.223955+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:00:48.223955+0000","last_undegraded":"2024-03-30T15:00:48.223955+0000","last_fullsized":"2024-03-30T15:00:48.223955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:20.292655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.292655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'9","reported_seq":32,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224646+0000","last_change":"2024-03-30T15:00:45.264420+0000","last_active":"2024-03-30T15:00:48.224646+0000","last_peered":"2024-03-30T15:00:48.224646+0000","last_clean":"2024-03-30T15:00:48.224646+0000","last_became_active":"2024-03-30T15:00:45.242541+0000","last_became_peered":"2024-03-30T15:00:45.242541+0000","last_unstale":"2024-03-30T15:00:48.224646+0000","last_undegraded":"2024-03-30T15:00:48.224646+0000","last_fullsized":"2024-03-30T15:00:48.224646+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":354,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227980+0000","last_change":"2024-03-30T14:57:55.140517+0000","last_active":"2024-03-30T15:00:48.227980+0000","last_peered":"2024-03-30T15:00:48.227980+0000","last_clean":"2024-03-30T15:00:48.227980+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:00:48.227980+0000","last_undegraded":"2024-03-30T15:00:48.227980+0000","last_fullsized":"2024-03-30T15:00:48.227980+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227392+0000","last_change":"2024-03-30T14:58:35.074272+0000","last_active":"2024-03-30T15:00:48.227392+0000","last_peered":"2024-03-30T15:00:48.227392+0000","last_clean":"2024-03-30T15:00:48.227392+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:00:48.227392+0000","last_undegraded":"2024-03-30T15:00:48.227392+0000","last_fullsized":"2024-03-30T15:00:48.227392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:35.074185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.074185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":566,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220028+0000","last_change":"2024-03-30T15:00:39.833443+0000","last_active":"2024-03-30T15:00:48.220028+0000","last_peered":"2024-03-30T15:00:48.220028+0000","last_clean":"2024-03-30T15:00:48.220028+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:00:48.220028+0000","last_undegraded":"2024-03-30T15:00:48.220028+0000","last_fullsized":"2024-03-30T15:00:48.220028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:00:39.833329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:39.833329+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.420557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223826+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:00:48.223826+0000","last_peered":"2024-03-30T15:00:48.223826+0000","last_clean":"2024-03-30T15:00:48.223826+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:00:48.223826+0000","last_undegraded":"2024-03-30T15:00:48.223826+0000","last_fullsized":"2024-03-30T15:00:48.223826+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":716,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221613+0000","last_change":"2024-03-30T14:59:14.976524+0000","last_active":"2024-03-30T15:00:48.221613+0000","last_peered":"2024-03-30T15:00:48.221613+0000","last_clean":"2024-03-30T15:00:48.221613+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:00:48.221613+0000","last_undegraded":"2024-03-30T15:00:48.221613+0000","last_fullsized":"2024-03-30T15:00:48.221613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T14:59:14.976438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.976438+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":444,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.229786+0000","last_change":"2024-03-30T14:59:05.166302+0000","last_active":"2024-03-30T15:00:48.229786+0000","last_peered":"2024-03-30T15:00:48.229786+0000","last_clean":"2024-03-30T15:00:48.229786+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:00:48.229786+0000","last_undegraded":"2024-03-30T15:00:48.229786+0000","last_fullsized":"2024-03-30T15:00:48.229786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.166200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.166200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221529+0000","last_change":"2024-03-30T15:00:45.242039+0000","last_active":"2024-03-30T15:00:48.221529+0000","last_peered":"2024-03-30T15:00:48.221529+0000","last_clean":"2024-03-30T15:00:48.221529+0000","last_became_active":"2024-03-30T15:00:45.240866+0000","last_became_peered":"2024-03-30T15:00:45.240866+0000","last_unstale":"2024-03-30T15:00:48.221529+0000","last_undegraded":"2024-03-30T15:00:48.221529+0000","last_fullsized":"2024-03-30T15:00:48.221529+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219526+0000","last_change":"2024-03-30T14:59:53.656204+0000","last_active":"2024-03-30T15:00:48.219526+0000","last_peered":"2024-03-30T15:00:48.219526+0000","last_clean":"2024-03-30T15:00:48.219526+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:00:48.219526+0000","last_undegraded":"2024-03-30T15:00:48.219526+0000","last_fullsized":"2024-03-30T15:00:48.219526+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.656111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.656111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:54.708968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.213994+0000","last_change":"2024-03-30T14:59:03.841069+0000","last_active":"2024-03-30T15:00:48.213994+0000","last_peered":"2024-03-30T15:00:48.213994+0000","last_clean":"2024-03-30T15:00:48.213994+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:00:48.213994+0000","last_undegraded":"2024-03-30T15:00:48.213994+0000","last_fullsized":"2024-03-30T15:00:48.213994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.840945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.840945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":494,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.232829+0000","last_change":"2024-03-30T14:59:51.039814+0000","last_active":"2024-03-30T15:00:48.232829+0000","last_peered":"2024-03-30T15:00:48.232829+0000","last_clean":"2024-03-30T15:00:48.232829+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:00:48.232829+0000","last_undegraded":"2024-03-30T15:00:48.232829+0000","last_fullsized":"2024-03-30T15:00:48.232829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T14:59:51.039715+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.039715+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.286012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":474,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211739+0000","last_change":"2024-03-30T14:58:58.777774+0000","last_active":"2024-03-30T15:00:48.211739+0000","last_peered":"2024-03-30T15:00:48.211739+0000","last_clean":"2024-03-30T15:00:48.211739+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:00:48.211739+0000","last_undegraded":"2024-03-30T15:00:48.211739+0000","last_fullsized":"2024-03-30T15:00:48.211739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.777653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":467,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215979+0000","last_change":"2024-03-30T14:59:22.144499+0000","last_active":"2024-03-30T15:00:48.215979+0000","last_peered":"2024-03-30T15:00:48.215979+0000","last_clean":"2024-03-30T15:00:48.215979+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:00:48.215979+0000","last_undegraded":"2024-03-30T15:00:48.215979+0000","last_fullsized":"2024-03-30T15:00:48.215979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:22.144389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:59:22.144389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":453,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228231+0000","last_change":"2024-03-30T14:59:55.270674+0000","last_active":"2024-03-30T15:00:48.228231+0000","last_peered":"2024-03-30T15:00:48.228231+0000","last_clean":"2024-03-30T15:00:48.228231+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:00:48.228231+0000","last_undegraded":"2024-03-30T15:00:48.228231+0000","last_fullsized":"2024-03-30T15:00:48.228231+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.270595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.270595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.865117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221836+0000","last_change":"2024-03-30T15:00:45.248453+0000","last_active":"2024-03-30T15:00:48.221836+0000","last_peered":"2024-03-30T15:00:48.221836+0000","last_clean":"2024-03-30T15:00:48.221836+0000","last_became_active":"2024-03-30T15:00:45.248115+0000","last_became_peered":"2024-03-30T15:00:45.248115+0000","last_unstale":"2024-03-30T15:00:48.221836+0000","last_undegraded":"2024-03-30T15:00:48.221836+0000","last_fullsized":"2024-03-30T15:00:48.221836+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.031690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.031690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:25.102887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218408+0000","last_change":"2024-03-30T14:57:54.024875+0000","last_active":"2024-03-30T15:00:48.218408+0000","last_peered":"2024-03-30T15:00:48.218408+0000","last_clean":"2024-03-30T15:00:48.218408+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:00:48.218408+0000","last_undegraded":"2024-03-30T15:00:48.218408+0000","last_fullsized":"2024-03-30T15:00:48.218408+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":458,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218191+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:00:48.218191+0000","last_peered":"2024-03-30T15:00:48.218191+0000","last_clean":"2024-03-30T15:00:48.218191+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:00:48.218191+0000","last_undegraded":"2024-03-30T15:00:48.218191+0000","last_fullsized":"2024-03-30T15:00:48.218191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.197637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":530,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228459+0000","last_change":"2024-03-30T14:58:22.340066+0000","last_active":"2024-03-30T15:00:48.228459+0000","last_peered":"2024-03-30T15:00:48.228459+0000","last_clean":"2024-03-30T15:00:48.228459+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:00:48.228459+0000","last_undegraded":"2024-03-30T15:00:48.228459+0000","last_fullsized":"2024-03-30T15:00:48.228459+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:58:22.339962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.339962+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.234277+0000","last_change":"2024-03-30T14:58:19.237610+0000","last_active":"2024-03-30T15:00:48.234277+0000","last_peered":"2024-03-30T15:00:48.234277+0000","last_clean":"2024-03-30T15:00:48.234277+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:00:48.234277+0000","last_undegraded":"2024-03-30T15:00:48.234277+0000","last_fullsized":"2024-03-30T15:00:48.234277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.237530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.237530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":467,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223922+0000","last_change":"2024-03-30T14:58:36.096934+0000","last_active":"2024-03-30T15:00:48.223922+0000","last_peered":"2024-03-30T15:00:48.223922+0000","last_clean":"2024-03-30T15:00:48.223922+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:00:48.223922+0000","last_undegraded":"2024-03-30T15:00:48.223922+0000","last_fullsized":"2024-03-30T15:00:48.223922+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T14:58:36.096812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.096812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230744+0000","last_change":"2024-03-30T14:58:22.249707+0000","last_active":"2024-03-30T15:00:48.230744+0000","last_peered":"2024-03-30T15:00:48.230744+0000","last_clean":"2024-03-30T15:00:48.230744+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:00:48.230744+0000","last_undegraded":"2024-03-30T15:00:48.230744+0000","last_fullsized":"2024-03-30T15:00:48.230744+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.249628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.249628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.11","version":"115'7","reported_seq":23,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215943+0000","last_change":"2024-03-30T15:00:45.250504+0000","last_active":"2024-03-30T15:00:48.215943+0000","last_peered":"2024-03-30T15:00:48.215943+0000","last_clean":"2024-03-30T15:00:48.215943+0000","last_became_active":"2024-03-30T15:00:45.239363+0000","last_became_peered":"2024-03-30T15:00:45.239363+0000","last_unstale":"2024-03-30T15:00:48.215943+0000","last_undegraded":"2024-03-30T15:00:48.215943+0000","last_fullsized":"2024-03-30T15:00:48.215943+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221536+0000","last_change":"2024-03-30T14:57:54.046311+0000","last_active":"2024-03-30T15:00:48.221536+0000","last_peered":"2024-03-30T15:00:48.221536+0000","last_clean":"2024-03-30T15:00:48.221536+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:00:48.221536+0000","last_undegraded":"2024-03-30T15:00:48.221536+0000","last_fullsized":"2024-03-30T15:00:48.221536+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":478,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221443+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:00:48.221443+0000","last_peered":"2024-03-30T15:00:48.221443+0000","last_clean":"2024-03-30T15:00:48.221443+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:00:48.221443+0000","last_undegraded":"2024-03-30T15:00:48.221443+0000","last_fullsized":"2024-03-30T15:00:48.221443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:39.060252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":464,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228910+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:00:48.228910+0000","last_peered":"2024-03-30T15:00:48.228910+0000","last_clean":"2024-03-30T15:00:48.228910+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:00:48.228910+0000","last_undegraded":"2024-03-30T15:00:48.228910+0000","last_fullsized":"2024-03-30T15:00:48.228910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.989386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.214895+0000","last_change":"2024-03-30T14:58:30.843293+0000","last_active":"2024-03-30T15:00:48.214895+0000","last_peered":"2024-03-30T15:00:48.214895+0000","last_clean":"2024-03-30T15:00:48.214895+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:00:48.214895+0000","last_undegraded":"2024-03-30T15:00:48.214895+0000","last_fullsized":"2024-03-30T15:00:48.214895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:30.843163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:30.843163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217212+0000","last_change":"2024-03-30T14:58:36.234502+0000","last_active":"2024-03-30T15:00:48.217212+0000","last_peered":"2024-03-30T15:00:48.217212+0000","last_clean":"2024-03-30T15:00:48.217212+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:00:48.217212+0000","last_undegraded":"2024-03-30T15:00:48.217212+0000","last_fullsized":"2024-03-30T15:00:48.217212+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:36.234425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.234425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222041+0000","last_change":"2024-03-30T15:00:45.247477+0000","last_active":"2024-03-30T15:00:48.222041+0000","last_peered":"2024-03-30T15:00:48.222041+0000","last_clean":"2024-03-30T15:00:48.222041+0000","last_became_active":"2024-03-30T15:00:45.246276+0000","last_became_peered":"2024-03-30T15:00:45.246276+0000","last_unstale":"2024-03-30T15:00:48.222041+0000","last_undegraded":"2024-03-30T15:00:48.222041+0000","last_fullsized":"2024-03-30T15:00:48.222041+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:41.329383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.329383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:01.674890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":354,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223702+0000","last_change":"2024-03-30T14:57:55.074016+0000","last_active":"2024-03-30T15:00:48.223702+0000","last_peered":"2024-03-30T15:00:48.223702+0000","last_clean":"2024-03-30T15:00:48.223702+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:00:48.223702+0000","last_undegraded":"2024-03-30T15:00:48.223702+0000","last_fullsized":"2024-03-30T15:00:48.223702+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":518,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217958+0000","last_change":"2024-03-30T14:59:00.517828+0000","last_active":"2024-03-30T15:00:48.217958+0000","last_peered":"2024-03-30T15:00:48.217958+0000","last_clean":"2024-03-30T15:00:48.217958+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:00:48.217958+0000","last_undegraded":"2024-03-30T15:00:48.217958+0000","last_fullsized":"2024-03-30T15:00:48.217958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:59:00.517731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:00.517731+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228144+0000","last_change":"2024-03-30T14:58:23.951589+0000","last_active":"2024-03-30T15:00:48.228144+0000","last_peered":"2024-03-30T15:00:48.228144+0000","last_clean":"2024-03-30T15:00:48.228144+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:00:48.228144+0000","last_undegraded":"2024-03-30T15:00:48.228144+0000","last_fullsized":"2024-03-30T15:00:48.228144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.951507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.212112+0000","last_change":"2024-03-30T14:58:33.889037+0000","last_active":"2024-03-30T15:00:48.212112+0000","last_peered":"2024-03-30T15:00:48.212112+0000","last_clean":"2024-03-30T15:00:48.212112+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:00:48.212112+0000","last_undegraded":"2024-03-30T15:00:48.212112+0000","last_fullsized":"2024-03-30T15:00:48.212112+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:33.888931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:33.888931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220966+0000","last_change":"2024-03-30T14:59:12.985700+0000","last_active":"2024-03-30T15:00:48.220966+0000","last_peered":"2024-03-30T15:00:48.220966+0000","last_clean":"2024-03-30T15:00:48.220966+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:00:48.220966+0000","last_undegraded":"2024-03-30T15:00:48.220966+0000","last_fullsized":"2024-03-30T15:00:48.220966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.985620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.985620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":376,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217988+0000","last_change":"2024-03-30T14:59:51.656251+0000","last_active":"2024-03-30T15:00:48.217988+0000","last_peered":"2024-03-30T15:00:48.217988+0000","last_clean":"2024-03-30T15:00:48.217988+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:00:48.217988+0000","last_undegraded":"2024-03-30T15:00:48.217988+0000","last_fullsized":"2024-03-30T15:00:48.217988+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T14:59:51.656163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.656163+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.934763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211286+0000","last_change":"2024-03-30T14:59:08.794368+0000","last_active":"2024-03-30T15:00:48.211286+0000","last_peered":"2024-03-30T15:00:48.211286+0000","last_clean":"2024-03-30T15:00:48.211286+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:00:48.211286+0000","last_undegraded":"2024-03-30T15:00:48.211286+0000","last_fullsized":"2024-03-30T15:00:48.211286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.794288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.794288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":484,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221167+0000","last_change":"2024-03-30T14:58:25.223144+0000","last_active":"2024-03-30T15:00:48.221167+0000","last_peered":"2024-03-30T15:00:48.221167+0000","last_clean":"2024-03-30T15:00:48.221167+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:00:48.221167+0000","last_undegraded":"2024-03-30T15:00:48.221167+0000","last_fullsized":"2024-03-30T15:00:48.221167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:25.223044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.223044+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227304+0000","last_change":"2024-03-30T14:58:25.984629+0000","last_active":"2024-03-30T15:00:48.227304+0000","last_peered":"2024-03-30T15:00:48.227304+0000","last_clean":"2024-03-30T15:00:48.227304+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:00:48.227304+0000","last_undegraded":"2024-03-30T15:00:48.227304+0000","last_fullsized":"2024-03-30T15:00:48.227304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.984552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.984552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226881+0000","last_change":"2024-03-30T15:00:02.270908+0000","last_active":"2024-03-30T15:00:48.226881+0000","last_peered":"2024-03-30T15:00:48.226881+0000","last_clean":"2024-03-30T15:00:48.226881+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:00:48.226881+0000","last_undegraded":"2024-03-30T15:00:48.226881+0000","last_fullsized":"2024-03-30T15:00:48.226881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.270821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.039536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217875+0000","last_change":"2024-03-30T15:00:45.241964+0000","last_active":"2024-03-30T15:00:48.217875+0000","last_peered":"2024-03-30T15:00:48.217875+0000","last_clean":"2024-03-30T15:00:48.217875+0000","last_became_active":"2024-03-30T15:00:45.241487+0000","last_became_peered":"2024-03-30T15:00:45.241487+0000","last_unstale":"2024-03-30T15:00:48.217875+0000","last_undegraded":"2024-03-30T15:00:48.217875+0000","last_fullsized":"2024-03-30T15:00:48.217875+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.153490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.153490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:09.628092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":362,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217774+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:00:48.217774+0000","last_peered":"2024-03-30T15:00:48.217774+0000","last_clean":"2024-03-30T15:00:48.217774+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:00:48.217774+0000","last_undegraded":"2024-03-30T15:00:48.217774+0000","last_fullsized":"2024-03-30T15:00:48.217774+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.044750+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":532,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224644+0000","last_change":"2024-03-30T14:59:05.977968+0000","last_active":"2024-03-30T15:00:48.224644+0000","last_peered":"2024-03-30T15:00:48.224644+0000","last_clean":"2024-03-30T15:00:48.224644+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:00:48.224644+0000","last_undegraded":"2024-03-30T15:00:48.224644+0000","last_fullsized":"2024-03-30T15:00:48.224644+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:05.977866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.977866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219592+0000","last_change":"2024-03-30T14:58:35.054537+0000","last_active":"2024-03-30T15:00:48.219592+0000","last_peered":"2024-03-30T15:00:48.219592+0000","last_clean":"2024-03-30T15:00:48.219592+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:00:48.219592+0000","last_undegraded":"2024-03-30T15:00:48.219592+0000","last_fullsized":"2024-03-30T15:00:48.219592+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:58:35.054439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.054439+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228560+0000","last_change":"2024-03-30T14:58:24.985818+0000","last_active":"2024-03-30T15:00:48.228560+0000","last_peered":"2024-03-30T15:00:48.228560+0000","last_clean":"2024-03-30T15:00:48.228560+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:00:48.228560+0000","last_undegraded":"2024-03-30T15:00:48.228560+0000","last_fullsized":"2024-03-30T15:00:48.228560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T14:58:24.985660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.985660+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":444,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225227+0000","last_change":"2024-03-30T14:58:27.124070+0000","last_active":"2024-03-30T15:00:48.225227+0000","last_peered":"2024-03-30T15:00:48.225227+0000","last_clean":"2024-03-30T15:00:48.225227+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:48.225227+0000","last_undegraded":"2024-03-30T15:00:48.225227+0000","last_fullsized":"2024-03-30T15:00:48.225227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.123941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.123941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"115'7","reported_seq":28,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219500+0000","last_change":"2024-03-30T15:00:45.264797+0000","last_active":"2024-03-30T15:00:48.219500+0000","last_peered":"2024-03-30T15:00:48.219500+0000","last_clean":"2024-03-30T15:00:48.219500+0000","last_became_active":"2024-03-30T15:00:45.253237+0000","last_became_peered":"2024-03-30T15:00:45.253237+0000","last_unstale":"2024-03-30T15:00:48.219500+0000","last_undegraded":"2024-03-30T15:00:48.219500+0000","last_fullsized":"2024-03-30T15:00:48.219500+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219435+0000","last_change":"2024-03-30T14:59:56.678123+0000","last_active":"2024-03-30T15:00:48.219435+0000","last_peered":"2024-03-30T15:00:48.219435+0000","last_clean":"2024-03-30T15:00:48.219435+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:00:48.219435+0000","last_undegraded":"2024-03-30T15:00:48.219435+0000","last_fullsized":"2024-03-30T15:00:48.219435+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.678045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.678045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:12.776149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":378,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228655+0000","last_change":"2024-03-30T14:57:56.275636+0000","last_active":"2024-03-30T15:00:48.228655+0000","last_peered":"2024-03-30T15:00:48.228655+0000","last_clean":"2024-03-30T15:00:48.228655+0000","last_became_active":"2024-03-30T14:57:56.273316+0000","last_became_peered":"2024-03-30T14:57:56.273316+0000","last_unstale":"2024-03-30T15:00:48.228655+0000","last_undegraded":"2024-03-30T15:00:48.228655+0000","last_fullsized":"2024-03-30T15:00:48.228655+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:57:30.919064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:57:30.919064+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":465,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224024+0000","last_change":"2024-03-30T14:58:22.261893+0000","last_active":"2024-03-30T15:00:48.224024+0000","last_peered":"2024-03-30T15:00:48.224024+0000","last_clean":"2024-03-30T15:00:48.224024+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:00:48.224024+0000","last_undegraded":"2024-03-30T15:00:48.224024+0000","last_fullsized":"2024-03-30T15:00:48.224024+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.261803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.261803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227481+0000","last_change":"2024-03-30T14:58:26.076218+0000","last_active":"2024-03-30T15:00:48.227481+0000","last_peered":"2024-03-30T15:00:48.227481+0000","last_clean":"2024-03-30T15:00:48.227481+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:00:48.227481+0000","last_undegraded":"2024-03-30T15:00:48.227481+0000","last_fullsized":"2024-03-30T15:00:48.227481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:26.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:26.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226120+0000","last_change":"2024-03-30T14:59:51.262210+0000","last_active":"2024-03-30T15:00:48.226120+0000","last_peered":"2024-03-30T15:00:48.226120+0000","last_clean":"2024-03-30T15:00:48.226120+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:00:48.226120+0000","last_undegraded":"2024-03-30T15:00:48.226120+0000","last_fullsized":"2024-03-30T15:00:48.226120+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:51.262127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.262127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.554991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223815+0000","last_change":"2024-03-30T15:00:37.677858+0000","last_active":"2024-03-30T15:00:48.223815+0000","last_peered":"2024-03-30T15:00:48.223815+0000","last_clean":"2024-03-30T15:00:48.223815+0000","last_became_active":"2024-03-30T14:57:55.139072+0000","last_became_peered":"2024-03-30T14:57:55.139072+0000","last_unstale":"2024-03-30T15:00:48.223815+0000","last_undegraded":"2024-03-30T15:00:48.223815+0000","last_fullsized":"2024-03-30T15:00:48.223815+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.677768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.677768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.621733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":354,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230655+0000","last_change":"2024-03-30T14:57:55.126527+0000","last_active":"2024-03-30T15:00:48.230655+0000","last_peered":"2024-03-30T15:00:48.230655+0000","last_clean":"2024-03-30T15:00:48.230655+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:00:48.230655+0000","last_undegraded":"2024-03-30T15:00:48.230655+0000","last_fullsized":"2024-03-30T15:00:48.230655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":470,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219696+0000","last_change":"2024-03-30T14:59:15.487022+0000","last_active":"2024-03-30T15:00:48.219696+0000","last_peered":"2024-03-30T15:00:48.219696+0000","last_clean":"2024-03-30T15:00:48.219696+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:00:48.219696+0000","last_undegraded":"2024-03-30T15:00:48.219696+0000","last_fullsized":"2024-03-30T15:00:48.219696+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:15.486892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.486892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":465,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224262+0000","last_change":"2024-03-30T15:00:37.049789+0000","last_active":"2024-03-30T15:00:48.224262+0000","last_peered":"2024-03-30T15:00:48.224262+0000","last_clean":"2024-03-30T15:00:48.224262+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:00:48.224262+0000","last_undegraded":"2024-03-30T15:00:48.224262+0000","last_fullsized":"2024-03-30T15:00:48.224262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.049685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.049685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.710602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.233160+0000","last_change":"2024-03-30T14:58:24.291134+0000","last_active":"2024-03-30T15:00:48.233160+0000","last_peered":"2024-03-30T15:00:48.233160+0000","last_clean":"2024-03-30T15:00:48.233160+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:00:48.233160+0000","last_undegraded":"2024-03-30T15:00:48.233160+0000","last_fullsized":"2024-03-30T15:00:48.233160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:24.291008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.291008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":452,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216164+0000","last_change":"2024-03-30T14:59:02.809926+0000","last_active":"2024-03-30T15:00:48.216164+0000","last_peered":"2024-03-30T15:00:48.216164+0000","last_clean":"2024-03-30T15:00:48.216164+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:00:48.216164+0000","last_undegraded":"2024-03-30T15:00:48.216164+0000","last_fullsized":"2024-03-30T15:00:48.216164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.809814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.809814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3","version":"115'5","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.234061+0000","last_change":"2024-03-30T14:57:55.091367+0000","last_active":"2024-03-30T15:00:48.234061+0000","last_peered":"2024-03-30T15:00:48.234061+0000","last_clean":"2024-03-30T15:00:48.234061+0000","last_became_active":"2024-03-30T14:57:55.089349+0000","last_became_peered":"2024-03-30T14:57:55.089349+0000","last_unstale":"2024-03-30T15:00:48.234061+0000","last_undegraded":"2024-03-30T15:00:48.234061+0000","last_fullsized":"2024-03-30T15:00:48.234061+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218095+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:00:48.218095+0000","last_peered":"2024-03-30T15:00:48.218095+0000","last_clean":"2024-03-30T15:00:48.218095+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:00:48.218095+0000","last_undegraded":"2024-03-30T15:00:48.218095+0000","last_fullsized":"2024-03-30T15:00:48.218095+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.612522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":468,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.231065+0000","last_change":"2024-03-30T14:58:31.097064+0000","last_active":"2024-03-30T15:00:48.231065+0000","last_peered":"2024-03-30T15:00:48.231065+0000","last_clean":"2024-03-30T15:00:48.231065+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:00:48.231065+0000","last_undegraded":"2024-03-30T15:00:48.231065+0000","last_fullsized":"2024-03-30T15:00:48.231065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:58:31.096968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.096968+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":466,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217014+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:00:48.217014+0000","last_peered":"2024-03-30T15:00:48.217014+0000","last_clean":"2024-03-30T15:00:48.217014+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:00:48.217014+0000","last_undegraded":"2024-03-30T15:00:48.217014+0000","last_fullsized":"2024-03-30T15:00:48.217014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.736897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":418,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227186+0000","last_change":"2024-03-30T14:59:53.269594+0000","last_active":"2024-03-30T15:00:48.227186+0000","last_peered":"2024-03-30T15:00:48.227186+0000","last_clean":"2024-03-30T15:00:48.227186+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:00:48.227186+0000","last_undegraded":"2024-03-30T15:00:48.227186+0000","last_fullsized":"2024-03-30T15:00:48.227186+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:53.269452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.269452+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.687457+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.233632+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:00:48.233632+0000","last_peered":"2024-03-30T15:00:48.233632+0000","last_clean":"2024-03-30T15:00:48.233632+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:00:48.233632+0000","last_undegraded":"2024-03-30T15:00:48.233632+0000","last_fullsized":"2024-03-30T15:00:48.233632+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.363643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'5","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220703+0000","last_change":"2024-03-30T14:57:55.177168+0000","last_active":"2024-03-30T15:00:48.220703+0000","last_peered":"2024-03-30T15:00:48.220703+0000","last_clean":"2024-03-30T15:00:48.220703+0000","last_became_active":"2024-03-30T14:57:55.164327+0000","last_became_peered":"2024-03-30T14:57:55.164327+0000","last_unstale":"2024-03-30T15:00:48.220703+0000","last_undegraded":"2024-03-30T15:00:48.220703+0000","last_fullsized":"2024-03-30T15:00:48.220703+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218289+0000","last_change":"2024-03-30T14:57:54.020464+0000","last_active":"2024-03-30T15:00:48.218289+0000","last_peered":"2024-03-30T15:00:48.218289+0000","last_clean":"2024-03-30T15:00:48.218289+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:00:48.218289+0000","last_undegraded":"2024-03-30T15:00:48.218289+0000","last_fullsized":"2024-03-30T15:00:48.218289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":393,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224338+0000","last_change":"2024-03-30T14:58:13.210584+0000","last_active":"2024-03-30T15:00:48.224338+0000","last_peered":"2024-03-30T15:00:48.224338+0000","last_clean":"2024-03-30T15:00:48.224338+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:00:48.224338+0000","last_undegraded":"2024-03-30T15:00:48.224338+0000","last_fullsized":"2024-03-30T15:00:48.224338+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:13.210426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.210426+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226537+0000","last_change":"2024-03-30T14:59:09.029236+0000","last_active":"2024-03-30T15:00:48.226537+0000","last_peered":"2024-03-30T15:00:48.226537+0000","last_clean":"2024-03-30T15:00:48.226537+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:00:48.226537+0000","last_undegraded":"2024-03-30T15:00:48.226537+0000","last_fullsized":"2024-03-30T15:00:48.226537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.029159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.029159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":534,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230778+0000","last_change":"2024-03-30T14:59:08.160662+0000","last_active":"2024-03-30T15:00:48.230778+0000","last_peered":"2024-03-30T15:00:48.230778+0000","last_clean":"2024-03-30T15:00:48.230778+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:00:48.230778+0000","last_undegraded":"2024-03-30T15:00:48.230778+0000","last_fullsized":"2024-03-30T15:00:48.230778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:08.160528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.160528+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219189+0000","last_change":"2024-03-30T14:59:16.481218+0000","last_active":"2024-03-30T15:00:48.219189+0000","last_peered":"2024-03-30T15:00:48.219189+0000","last_clean":"2024-03-30T15:00:48.219189+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:00:48.219189+0000","last_undegraded":"2024-03-30T15:00:48.219189+0000","last_fullsized":"2024-03-30T15:00:48.219189+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:16.481128+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.481128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"115'5","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216170+0000","last_change":"2024-03-30T14:57:54.041959+0000","last_active":"2024-03-30T15:00:48.216170+0000","last_peered":"2024-03-30T15:00:48.216170+0000","last_clean":"2024-03-30T15:00:48.216170+0000","last_became_active":"2024-03-30T14:57:54.041685+0000","last_became_peered":"2024-03-30T14:57:54.041685+0000","last_unstale":"2024-03-30T15:00:48.216170+0000","last_undegraded":"2024-03-30T15:00:48.216170+0000","last_fullsized":"2024-03-30T15:00:48.216170+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":362,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.235835+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:00:48.235835+0000","last_peered":"2024-03-30T15:00:48.235835+0000","last_clean":"2024-03-30T15:00:48.235835+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:00:48.235835+0000","last_undegraded":"2024-03-30T15:00:48.235835+0000","last_fullsized":"2024-03-30T15:00:48.235835+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.403334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.235933+0000","last_change":"2024-03-30T14:59:20.453682+0000","last_active":"2024-03-30T15:00:48.235933+0000","last_peered":"2024-03-30T15:00:48.235933+0000","last_clean":"2024-03-30T15:00:48.235933+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:00:48.235933+0000","last_undegraded":"2024-03-30T15:00:48.235933+0000","last_fullsized":"2024-03-30T15:00:48.235933+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.453594+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227505+0000","last_change":"2024-03-30T14:57:31.554095+0000","last_active":"2024-03-30T15:00:48.227505+0000","last_peered":"2024-03-30T15:00:48.227505+0000","last_clean":"2024-03-30T15:00:48.227505+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:00:48.227505+0000","last_undegraded":"2024-03-30T15:00:48.227505+0000","last_fullsized":"2024-03-30T15:00:48.227505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:31.554003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:31.554003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":458,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222118+0000","last_change":"2024-03-30T14:59:15.144954+0000","last_active":"2024-03-30T15:00:48.222118+0000","last_peered":"2024-03-30T15:00:48.222118+0000","last_clean":"2024-03-30T15:00:48.222118+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:00:48.222118+0000","last_undegraded":"2024-03-30T15:00:48.222118+0000","last_fullsized":"2024-03-30T15:00:48.222118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.144864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216512+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:00:48.216512+0000","last_peered":"2024-03-30T15:00:48.216512+0000","last_clean":"2024-03-30T15:00:48.216512+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:00:48.216512+0000","last_undegraded":"2024-03-30T15:00:48.216512+0000","last_fullsized":"2024-03-30T15:00:48.216512+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:29.384685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'9","reported_seq":373,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222320+0000","last_change":"2024-03-30T14:59:47.655366+0000","last_active":"2024-03-30T15:00:48.222320+0000","last_peered":"2024-03-30T15:00:48.222320+0000","last_clean":"2024-03-30T15:00:48.222320+0000","last_became_active":"2024-03-30T14:57:55.181249+0000","last_became_peered":"2024-03-30T14:57:55.181249+0000","last_unstale":"2024-03-30T15:00:48.222320+0000","last_undegraded":"2024-03-30T15:00:48.222320+0000","last_fullsized":"2024-03-30T15:00:48.222320+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9","last_scrub_stamp":"2024-03-30T14:59:47.655290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.655290+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.416191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":362,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223139+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:00:48.223139+0000","last_peered":"2024-03-30T15:00:48.223139+0000","last_clean":"2024-03-30T15:00:48.223139+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:00:48.223139+0000","last_undegraded":"2024-03-30T15:00:48.223139+0000","last_fullsized":"2024-03-30T15:00:48.223139+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.153282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":470,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222326+0000","last_change":"2024-03-30T14:59:08.292999+0000","last_active":"2024-03-30T15:00:48.222326+0000","last_peered":"2024-03-30T15:00:48.222326+0000","last_clean":"2024-03-30T15:00:48.222326+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:00:48.222326+0000","last_undegraded":"2024-03-30T15:00:48.222326+0000","last_fullsized":"2024-03-30T15:00:48.222326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:08.292876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.292876+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227801+0000","last_change":"2024-03-30T14:58:12.927851+0000","last_active":"2024-03-30T15:00:48.227801+0000","last_peered":"2024-03-30T15:00:48.227801+0000","last_clean":"2024-03-30T15:00:48.227801+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:00:48.227801+0000","last_undegraded":"2024-03-30T15:00:48.227801+0000","last_fullsized":"2024-03-30T15:00:48.227801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:58:12.927762+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.927762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221618+0000","last_change":"2024-03-30T14:59:17.441377+0000","last_active":"2024-03-30T15:00:48.221618+0000","last_peered":"2024-03-30T15:00:48.221618+0000","last_clean":"2024-03-30T15:00:48.221618+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:00:48.221618+0000","last_undegraded":"2024-03-30T15:00:48.221618+0000","last_fullsized":"2024-03-30T15:00:48.221618+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.441289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.441289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219269+0000","last_change":"2024-03-30T14:58:18.177964+0000","last_active":"2024-03-30T15:00:48.219269+0000","last_peered":"2024-03-30T15:00:48.219269+0000","last_clean":"2024-03-30T15:00:48.219269+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:00:48.219269+0000","last_undegraded":"2024-03-30T15:00:48.219269+0000","last_fullsized":"2024-03-30T15:00:48.219269+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:18.177806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:18.177806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222765+0000","last_change":"2024-03-30T14:57:54.013168+0000","last_active":"2024-03-30T15:00:48.222765+0000","last_peered":"2024-03-30T15:00:48.222765+0000","last_clean":"2024-03-30T15:00:48.222765+0000","last_became_active":"2024-03-30T14:57:54.012861+0000","last_became_peered":"2024-03-30T14:57:54.012861+0000","last_unstale":"2024-03-30T15:00:48.222765+0000","last_undegraded":"2024-03-30T15:00:48.222765+0000","last_fullsized":"2024-03-30T15:00:48.222765+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216289+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:00:48.216289+0000","last_peered":"2024-03-30T15:00:48.216289+0000","last_clean":"2024-03-30T15:00:48.216289+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:00:48.216289+0000","last_undegraded":"2024-03-30T15:00:48.216289+0000","last_fullsized":"2024-03-30T15:00:48.216289+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.187292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":396,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224523+0000","last_change":"2024-03-30T14:58:16.905699+0000","last_active":"2024-03-30T15:00:48.224523+0000","last_peered":"2024-03-30T15:00:48.224523+0000","last_clean":"2024-03-30T15:00:48.224523+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:00:48.224523+0000","last_undegraded":"2024-03-30T15:00:48.224523+0000","last_fullsized":"2024-03-30T15:00:48.224523+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:58:16.905604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:16.905604+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227568+0000","last_change":"2024-03-30T14:58:28.115261+0000","last_active":"2024-03-30T15:00:48.227568+0000","last_peered":"2024-03-30T15:00:48.227568+0000","last_clean":"2024-03-30T15:00:48.227568+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:00:48.227568+0000","last_undegraded":"2024-03-30T15:00:48.227568+0000","last_fullsized":"2024-03-30T15:00:48.227568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:28.115166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:28.115166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221832+0000","last_change":"2024-03-30T14:59:08.012564+0000","last_active":"2024-03-30T15:00:48.221832+0000","last_peered":"2024-03-30T15:00:48.221832+0000","last_clean":"2024-03-30T15:00:48.221832+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:00:48.221832+0000","last_undegraded":"2024-03-30T15:00:48.221832+0000","last_fullsized":"2024-03-30T15:00:48.221832+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.012470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.012470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221190+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:00:48.221190+0000","last_peered":"2024-03-30T15:00:48.221190+0000","last_clean":"2024-03-30T15:00:48.221190+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:00:48.221190+0000","last_undegraded":"2024-03-30T15:00:48.221190+0000","last_fullsized":"2024-03-30T15:00:48.221190+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.714400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218565+0000","last_change":"2024-03-30T14:57:54.019408+0000","last_active":"2024-03-30T15:00:48.218565+0000","last_peered":"2024-03-30T15:00:48.218565+0000","last_clean":"2024-03-30T15:00:48.218565+0000","last_became_active":"2024-03-30T14:57:54.014697+0000","last_became_peered":"2024-03-30T14:57:54.014697+0000","last_unstale":"2024-03-30T15:00:48.218565+0000","last_undegraded":"2024-03-30T15:00:48.218565+0000","last_fullsized":"2024-03-30T15:00:48.218565+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216158+0000","last_change":"2024-03-30T14:59:56.148641+0000","last_active":"2024-03-30T15:00:48.216158+0000","last_peered":"2024-03-30T15:00:48.216158+0000","last_clean":"2024-03-30T15:00:48.216158+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:00:48.216158+0000","last_undegraded":"2024-03-30T15:00:48.216158+0000","last_fullsized":"2024-03-30T15:00:48.216158+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.148529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.148529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.134818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":384,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216066+0000","last_change":"2024-03-30T14:58:17.304912+0000","last_active":"2024-03-30T15:00:48.216066+0000","last_peered":"2024-03-30T15:00:48.216066+0000","last_clean":"2024-03-30T15:00:48.216066+0000","last_became_active":"2024-03-30T14:57:56.224608+0000","last_became_peered":"2024-03-30T14:57:56.224608+0000","last_unstale":"2024-03-30T15:00:48.216066+0000","last_undegraded":"2024-03-30T15:00:48.216066+0000","last_fullsized":"2024-03-30T15:00:48.216066+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:58:17.304759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:17.304759+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":466,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221654+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:00:48.221654+0000","last_peered":"2024-03-30T15:00:48.221654+0000","last_clean":"2024-03-30T15:00:48.221654+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:00:48.221654+0000","last_undegraded":"2024-03-30T15:00:48.221654+0000","last_fullsized":"2024-03-30T15:00:48.221654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.891147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218586+0000","last_change":"2024-03-30T14:58:25.320605+0000","last_active":"2024-03-30T15:00:48.218586+0000","last_peered":"2024-03-30T15:00:48.218586+0000","last_clean":"2024-03-30T15:00:48.218586+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:00:48.218586+0000","last_undegraded":"2024-03-30T15:00:48.218586+0000","last_fullsized":"2024-03-30T15:00:48.218586+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.320527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.320527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225947+0000","last_change":"2024-03-30T14:58:58.957529+0000","last_active":"2024-03-30T15:00:48.225947+0000","last_peered":"2024-03-30T15:00:48.225947+0000","last_clean":"2024-03-30T15:00:48.225947+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:00:48.225947+0000","last_undegraded":"2024-03-30T15:00:48.225947+0000","last_fullsized":"2024-03-30T15:00:48.225947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.957449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.957449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221818+0000","last_change":"2024-03-30T14:59:07.508558+0000","last_active":"2024-03-30T15:00:48.221818+0000","last_peered":"2024-03-30T15:00:48.221818+0000","last_clean":"2024-03-30T15:00:48.221818+0000","last_became_active":"2024-03-30T14:57:55.201639+0000","last_became_peered":"2024-03-30T14:57:55.201639+0000","last_unstale":"2024-03-30T15:00:48.221818+0000","last_undegraded":"2024-03-30T15:00:48.221818+0000","last_fullsized":"2024-03-30T15:00:48.221818+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.508480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.508480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219778+0000","last_change":"2024-03-30T15:00:03.553767+0000","last_active":"2024-03-30T15:00:48.219778+0000","last_peered":"2024-03-30T15:00:48.219778+0000","last_clean":"2024-03-30T15:00:48.219778+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:00:48.219778+0000","last_undegraded":"2024-03-30T15:00:48.219778+0000","last_fullsized":"2024-03-30T15:00:48.219778+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.553680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.553680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.545159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":485,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219877+0000","last_change":"2024-03-30T15:00:41.903846+0000","last_active":"2024-03-30T15:00:48.219877+0000","last_peered":"2024-03-30T15:00:48.219877+0000","last_clean":"2024-03-30T15:00:48.219877+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:00:48.219877+0000","last_undegraded":"2024-03-30T15:00:48.219877+0000","last_fullsized":"2024-03-30T15:00:48.219877+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:00:41.903753+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.903753+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.208912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219527+0000","last_change":"2024-03-30T14:59:09.624243+0000","last_active":"2024-03-30T15:00:48.219527+0000","last_peered":"2024-03-30T15:00:48.219527+0000","last_clean":"2024-03-30T15:00:48.219527+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:00:48.219527+0000","last_undegraded":"2024-03-30T15:00:48.219527+0000","last_fullsized":"2024-03-30T15:00:48.219527+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.624114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.624114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":451,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219934+0000","last_change":"2024-03-30T14:58:58.605917+0000","last_active":"2024-03-30T15:00:48.219934+0000","last_peered":"2024-03-30T15:00:48.219934+0000","last_clean":"2024-03-30T15:00:48.219934+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:00:48.219934+0000","last_undegraded":"2024-03-30T15:00:48.219934+0000","last_fullsized":"2024-03-30T15:00:48.219934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.605792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.605792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":454,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227919+0000","last_change":"2024-03-30T14:59:21.144421+0000","last_active":"2024-03-30T15:00:48.227919+0000","last_peered":"2024-03-30T15:00:48.227919+0000","last_clean":"2024-03-30T15:00:48.227919+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:00:48.227919+0000","last_undegraded":"2024-03-30T15:00:48.227919+0000","last_fullsized":"2024-03-30T15:00:48.227919+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.144335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.144335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.a","version":"115'10","reported_seq":373,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225388+0000","last_change":"2024-03-30T15:00:42.061005+0000","last_active":"2024-03-30T15:00:48.225388+0000","last_peered":"2024-03-30T15:00:48.225388+0000","last_clean":"2024-03-30T15:00:48.225388+0000","last_became_active":"2024-03-30T14:57:55.103400+0000","last_became_peered":"2024-03-30T14:57:55.103400+0000","last_unstale":"2024-03-30T15:00:48.225388+0000","last_undegraded":"2024-03-30T15:00:48.225388+0000","last_fullsized":"2024-03-30T15:00:48.225388+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:42.060906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:42.060906+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.650303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222797+0000","last_change":"2024-03-30T14:59:14.560489+0000","last_active":"2024-03-30T15:00:48.222797+0000","last_peered":"2024-03-30T15:00:48.222797+0000","last_clean":"2024-03-30T15:00:48.222797+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:00:48.222797+0000","last_undegraded":"2024-03-30T15:00:48.222797+0000","last_fullsized":"2024-03-30T15:00:48.222797+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.560377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.560377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":475,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219594+0000","last_change":"2024-03-30T14:59:10.626455+0000","last_active":"2024-03-30T15:00:48.219594+0000","last_peered":"2024-03-30T15:00:48.219594+0000","last_clean":"2024-03-30T15:00:48.219594+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:00:48.219594+0000","last_undegraded":"2024-03-30T15:00:48.219594+0000","last_fullsized":"2024-03-30T15:00:48.219594+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.626365+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":459,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217993+0000","last_change":"2024-03-30T14:58:32.272039+0000","last_active":"2024-03-30T15:00:48.217993+0000","last_peered":"2024-03-30T15:00:48.217993+0000","last_clean":"2024-03-30T15:00:48.217993+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:00:48.217993+0000","last_undegraded":"2024-03-30T15:00:48.217993+0000","last_fullsized":"2024-03-30T15:00:48.217993+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T14:58:32.271622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:32.271622+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":543,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219616+0000","last_change":"2024-03-30T14:58:59.472999+0000","last_active":"2024-03-30T15:00:48.219616+0000","last_peered":"2024-03-30T15:00:48.219616+0000","last_clean":"2024-03-30T15:00:48.219616+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:00:48.219616+0000","last_undegraded":"2024-03-30T15:00:48.219616+0000","last_fullsized":"2024-03-30T15:00:48.219616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.472884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.472884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":455,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227286+0000","last_change":"2024-03-30T15:00:36.069861+0000","last_active":"2024-03-30T15:00:48.227286+0000","last_peered":"2024-03-30T15:00:48.227286+0000","last_clean":"2024-03-30T15:00:48.227286+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:00:48.227286+0000","last_undegraded":"2024-03-30T15:00:48.227286+0000","last_fullsized":"2024-03-30T15:00:48.227286+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:00:36.069752+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:00:36.069752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:40.687480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"115'2","reported_seq":22,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224532+0000","last_change":"2024-03-30T15:00:45.260322+0000","last_active":"2024-03-30T15:00:48.224532+0000","last_peered":"2024-03-30T15:00:48.224532+0000","last_clean":"2024-03-30T15:00:48.224532+0000","last_became_active":"2024-03-30T15:00:45.244909+0000","last_became_peered":"2024-03-30T15:00:45.244909+0000","last_unstale":"2024-03-30T15:00:48.224532+0000","last_undegraded":"2024-03-30T15:00:48.224532+0000","last_fullsized":"2024-03-30T15:00:48.224532+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":363,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219759+0000","last_change":"2024-03-30T14:59:21.597544+0000","last_active":"2024-03-30T15:00:48.219759+0000","last_peered":"2024-03-30T15:00:48.219759+0000","last_clean":"2024-03-30T15:00:48.219759+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:00:48.219759+0000","last_undegraded":"2024-03-30T15:00:48.219759+0000","last_fullsized":"2024-03-30T15:00:48.219759+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":401,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.224071+0000","last_change":"2024-03-30T14:58:15.287344+0000","last_active":"2024-03-30T15:00:48.224071+0000","last_peered":"2024-03-30T15:00:48.224071+0000","last_clean":"2024-03-30T15:00:48.224071+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:00:48.224071+0000","last_undegraded":"2024-03-30T15:00:48.224071+0000","last_fullsized":"2024-03-30T15:00:48.224071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:15.286572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.286572+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":457,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223537+0000","last_change":"2024-03-30T14:58:31.196084+0000","last_active":"2024-03-30T15:00:48.223537+0000","last_peered":"2024-03-30T15:00:48.223537+0000","last_clean":"2024-03-30T15:00:48.223537+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:00:48.223537+0000","last_undegraded":"2024-03-30T15:00:48.223537+0000","last_fullsized":"2024-03-30T15:00:48.223537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:31.196010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.196010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.227824+0000","last_change":"2024-03-30T14:59:01.294973+0000","last_active":"2024-03-30T15:00:48.227824+0000","last_peered":"2024-03-30T15:00:48.227824+0000","last_clean":"2024-03-30T15:00:48.227824+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:00:48.227824+0000","last_undegraded":"2024-03-30T15:00:48.227824+0000","last_fullsized":"2024-03-30T15:00:48.227824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.294888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.294888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":359,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220035+0000","last_change":"2024-03-30T14:58:12.296903+0000","last_active":"2024-03-30T15:00:48.220035+0000","last_peered":"2024-03-30T15:00:48.220035+0000","last_clean":"2024-03-30T15:00:48.220035+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:00:48.220035+0000","last_undegraded":"2024-03-30T15:00:48.220035+0000","last_fullsized":"2024-03-30T15:00:48.220035+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:12.296793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.296793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"115'2","reported_seq":357,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220608+0000","last_change":"2024-03-30T14:57:55.109289+0000","last_active":"2024-03-30T15:00:48.220608+0000","last_peered":"2024-03-30T15:00:48.220608+0000","last_clean":"2024-03-30T15:00:48.220608+0000","last_became_active":"2024-03-30T14:57:55.108185+0000","last_became_peered":"2024-03-30T14:57:55.108185+0000","last_unstale":"2024-03-30T15:00:48.220608+0000","last_undegraded":"2024-03-30T15:00:48.220608+0000","last_fullsized":"2024-03-30T15:00:48.220608+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":362,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.229060+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:00:48.229060+0000","last_peered":"2024-03-30T15:00:48.229060+0000","last_clean":"2024-03-30T15:00:48.229060+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:00:48.229060+0000","last_undegraded":"2024-03-30T15:00:48.229060+0000","last_fullsized":"2024-03-30T15:00:48.229060+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.587803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.215780+0000","last_change":"2024-03-30T14:58:37.879183+0000","last_active":"2024-03-30T15:00:48.215780+0000","last_peered":"2024-03-30T15:00:48.215780+0000","last_clean":"2024-03-30T15:00:48.215780+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:00:48.215780+0000","last_undegraded":"2024-03-30T15:00:48.215780+0000","last_fullsized":"2024-03-30T15:00:48.215780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T14:58:37.879074+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":466,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221768+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:00:48.221768+0000","last_peered":"2024-03-30T15:00:48.221768+0000","last_clean":"2024-03-30T15:00:48.221768+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:00:48.221768+0000","last_undegraded":"2024-03-30T15:00:48.221768+0000","last_fullsized":"2024-03-30T15:00:48.221768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:21.606179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":450,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225359+0000","last_change":"2024-03-30T14:58:19.314405+0000","last_active":"2024-03-30T15:00:48.225359+0000","last_peered":"2024-03-30T15:00:48.225359+0000","last_clean":"2024-03-30T15:00:48.225359+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:00:48.225359+0000","last_undegraded":"2024-03-30T15:00:48.225359+0000","last_fullsized":"2024-03-30T15:00:48.225359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.314262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.314262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230880+0000","last_change":"2024-03-30T14:59:03.291569+0000","last_active":"2024-03-30T15:00:48.230880+0000","last_peered":"2024-03-30T15:00:48.230880+0000","last_clean":"2024-03-30T15:00:48.230880+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:00:48.230880+0000","last_undegraded":"2024-03-30T15:00:48.230880+0000","last_fullsized":"2024-03-30T15:00:48.230880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.291469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.291469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":352,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.216082+0000","last_change":"2024-03-30T14:57:56.258314+0000","last_active":"2024-03-30T15:00:48.216082+0000","last_peered":"2024-03-30T15:00:48.216082+0000","last_clean":"2024-03-30T15:00:48.216082+0000","last_became_active":"2024-03-30T14:57:56.257360+0000","last_became_peered":"2024-03-30T14:57:56.257360+0000","last_unstale":"2024-03-30T15:00:48.216082+0000","last_undegraded":"2024-03-30T15:00:48.216082+0000","last_fullsized":"2024-03-30T15:00:48.216082+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":364,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222634+0000","last_change":"2024-03-30T14:59:50.075598+0000","last_active":"2024-03-30T15:00:48.222634+0000","last_peered":"2024-03-30T15:00:48.222634+0000","last_clean":"2024-03-30T15:00:48.222634+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:00:48.222634+0000","last_undegraded":"2024-03-30T15:00:48.222634+0000","last_fullsized":"2024-03-30T15:00:48.222634+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.075512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.075512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.184144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":475,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.235664+0000","last_change":"2024-03-30T14:59:46.413217+0000","last_active":"2024-03-30T15:00:48.235664+0000","last_peered":"2024-03-30T15:00:48.235664+0000","last_clean":"2024-03-30T15:00:48.235664+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T15:00:48.235664+0000","last_undegraded":"2024-03-30T15:00:48.235664+0000","last_fullsized":"2024-03-30T15:00:48.235664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T14:59:46.413127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.413127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.902904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.228824+0000","last_change":"2024-03-30T14:58:15.115774+0000","last_active":"2024-03-30T15:00:48.228824+0000","last_peered":"2024-03-30T15:00:48.228824+0000","last_clean":"2024-03-30T15:00:48.228824+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:00:48.228824+0000","last_undegraded":"2024-03-30T15:00:48.228824+0000","last_fullsized":"2024-03-30T15:00:48.228824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:15.115694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.115694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.225853+0000","last_change":"2024-03-30T14:58:29.060905+0000","last_active":"2024-03-30T15:00:48.225853+0000","last_peered":"2024-03-30T15:00:48.225853+0000","last_clean":"2024-03-30T15:00:48.225853+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:00:48.225853+0000","last_undegraded":"2024-03-30T15:00:48.225853+0000","last_fullsized":"2024-03-30T15:00:48.225853+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:29.060822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:29.060822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":452,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220331+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:00:48.220331+0000","last_peered":"2024-03-30T15:00:48.220331+0000","last_clean":"2024-03-30T15:00:48.220331+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:00:48.220331+0000","last_undegraded":"2024-03-30T15:00:48.220331+0000","last_fullsized":"2024-03-30T15:00:48.220331+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.177682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":17,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222122+0000","last_change":"2024-03-30T15:00:45.246983+0000","last_active":"2024-03-30T15:00:48.222122+0000","last_peered":"2024-03-30T15:00:48.222122+0000","last_clean":"2024-03-30T15:00:48.222122+0000","last_became_active":"2024-03-30T15:00:45.246116+0000","last_became_peered":"2024-03-30T15:00:45.246116+0000","last_unstale":"2024-03-30T15:00:48.222122+0000","last_undegraded":"2024-03-30T15:00:48.222122+0000","last_fullsized":"2024-03-30T15:00:48.222122+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:16.765776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.765776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221711+0000","last_change":"2024-03-30T15:00:00.172131+0000","last_active":"2024-03-30T15:00:48.221711+0000","last_peered":"2024-03-30T15:00:48.221711+0000","last_clean":"2024-03-30T15:00:48.221711+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:00:48.221711+0000","last_undegraded":"2024-03-30T15:00:48.221711+0000","last_fullsized":"2024-03-30T15:00:48.221711+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.172053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.172053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:07.869357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223458+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:00:48.223458+0000","last_peered":"2024-03-30T15:00:48.223458+0000","last_clean":"2024-03-30T15:00:48.223458+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:00:48.223458+0000","last_undegraded":"2024-03-30T15:00:48.223458+0000","last_fullsized":"2024-03-30T15:00:48.223458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.313881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":456,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.230702+0000","last_change":"2024-03-30T14:58:25.078864+0000","last_active":"2024-03-30T15:00:48.230702+0000","last_peered":"2024-03-30T15:00:48.230702+0000","last_clean":"2024-03-30T15:00:48.230702+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:00:48.230702+0000","last_undegraded":"2024-03-30T15:00:48.230702+0000","last_fullsized":"2024-03-30T15:00:48.230702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.078771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.078771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223274+0000","last_change":"2024-03-30T14:58:19.191376+0000","last_active":"2024-03-30T15:00:48.223274+0000","last_peered":"2024-03-30T15:00:48.223274+0000","last_clean":"2024-03-30T15:00:48.223274+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:00:48.223274+0000","last_undegraded":"2024-03-30T15:00:48.223274+0000","last_fullsized":"2024-03-30T15:00:48.223274+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.191278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.191278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":454,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219625+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:00:48.219625+0000","last_peered":"2024-03-30T15:00:48.219625+0000","last_clean":"2024-03-30T15:00:48.219625+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:00:48.219625+0000","last_undegraded":"2024-03-30T15:00:48.219625+0000","last_fullsized":"2024-03-30T15:00:48.219625+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.919742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":20,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222896+0000","last_change":"2024-03-30T15:00:45.245276+0000","last_active":"2024-03-30T15:00:48.222896+0000","last_peered":"2024-03-30T15:00:48.222896+0000","last_clean":"2024-03-30T15:00:48.222896+0000","last_became_active":"2024-03-30T15:00:45.244380+0000","last_became_peered":"2024-03-30T15:00:45.244380+0000","last_unstale":"2024-03-30T15:00:48.222896+0000","last_undegraded":"2024-03-30T15:00:48.222896+0000","last_fullsized":"2024-03-30T15:00:48.222896+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.263392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.263392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:36.435258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":354,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219458+0000","last_change":"2024-03-30T14:57:55.106979+0000","last_active":"2024-03-30T15:00:48.219458+0000","last_peered":"2024-03-30T15:00:48.219458+0000","last_clean":"2024-03-30T15:00:48.219458+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:00:48.219458+0000","last_undegraded":"2024-03-30T15:00:48.219458+0000","last_fullsized":"2024-03-30T15:00:48.219458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":450,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.211794+0000","last_change":"2024-03-30T14:59:59.784365+0000","last_active":"2024-03-30T15:00:48.211794+0000","last_peered":"2024-03-30T15:00:48.211794+0000","last_clean":"2024-03-30T15:00:48.211794+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:00:48.211794+0000","last_undegraded":"2024-03-30T15:00:48.211794+0000","last_fullsized":"2024-03-30T15:00:48.211794+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:59:59.784262+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.784262+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.754986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":360,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.222500+0000","last_change":"2024-03-30T14:59:06.484972+0000","last_active":"2024-03-30T15:00:48.222500+0000","last_peered":"2024-03-30T15:00:48.222500+0000","last_clean":"2024-03-30T15:00:48.222500+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:00:48.222500+0000","last_undegraded":"2024-03-30T15:00:48.222500+0000","last_fullsized":"2024-03-30T15:00:48.222500+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:06.484877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T14:59:06.484877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217688+0000","last_change":"2024-03-30T14:59:12.396403+0000","last_active":"2024-03-30T15:00:48.217688+0000","last_peered":"2024-03-30T15:00:48.217688+0000","last_clean":"2024-03-30T15:00:48.217688+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:00:48.217688+0000","last_undegraded":"2024-03-30T15:00:48.217688+0000","last_fullsized":"2024-03-30T15:00:48.217688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.396321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":358,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.226445+0000","last_change":"2024-03-30T14:59:01.955656+0000","last_active":"2024-03-30T15:00:48.226445+0000","last_peered":"2024-03-30T15:00:48.226445+0000","last_clean":"2024-03-30T15:00:48.226445+0000","last_became_active":"2024-03-30T14:57:56.272206+0000","last_became_peered":"2024-03-30T14:57:56.272206+0000","last_unstale":"2024-03-30T15:00:48.226445+0000","last_undegraded":"2024-03-30T15:00:48.226445+0000","last_fullsized":"2024-03-30T15:00:48.226445+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.955549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.955549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":21,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.218238+0000","last_change":"2024-03-30T15:00:45.245980+0000","last_active":"2024-03-30T15:00:48.218238+0000","last_peered":"2024-03-30T15:00:48.218238+0000","last_clean":"2024-03-30T15:00:48.218238+0000","last_became_active":"2024-03-30T15:00:45.245596+0000","last_became_peered":"2024-03-30T15:00:45.245596+0000","last_unstale":"2024-03-30T15:00:48.218238+0000","last_undegraded":"2024-03-30T15:00:48.218238+0000","last_fullsized":"2024-03-30T15:00:48.218238+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.029978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.029978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:06.941515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":356,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.220006+0000","last_change":"2024-03-30T14:57:54.063914+0000","last_active":"2024-03-30T15:00:48.220006+0000","last_peered":"2024-03-30T15:00:48.220006+0000","last_clean":"2024-03-30T15:00:48.220006+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:00:48.220006+0000","last_undegraded":"2024-03-30T15:00:48.220006+0000","last_fullsized":"2024-03-30T15:00:48.220006+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217867+0000","last_change":"2024-03-30T14:59:57.415396+0000","last_active":"2024-03-30T15:00:48.217867+0000","last_peered":"2024-03-30T15:00:48.217867+0000","last_clean":"2024-03-30T15:00:48.217867+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T15:00:48.217867+0000","last_undegraded":"2024-03-30T15:00:48.217867+0000","last_fullsized":"2024-03-30T15:00:48.217867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.852042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":466,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.221450+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:00:48.221450+0000","last_peered":"2024-03-30T15:00:48.221450+0000","last_clean":"2024-03-30T15:00:48.221450+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:00:48.221450+0000","last_undegraded":"2024-03-30T15:00:48.221450+0000","last_fullsized":"2024-03-30T15:00:48.221450+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.262876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":450,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.217655+0000","last_change":"2024-03-30T14:59:04.268770+0000","last_active":"2024-03-30T15:00:48.217655+0000","last_peered":"2024-03-30T15:00:48.217655+0000","last_clean":"2024-03-30T15:00:48.217655+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:00:48.217655+0000","last_undegraded":"2024-03-30T15:00:48.217655+0000","last_fullsized":"2024-03-30T15:00:48.217655+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:04.268662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T14:59:04.268662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.223265+0000","last_change":"2024-03-30T14:58:27.174113+0000","last_active":"2024-03-30T15:00:48.223265+0000","last_peered":"2024-03-30T15:00:48.223265+0000","last_clean":"2024-03-30T15:00:48.223265+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:00:48.223265+0000","last_undegraded":"2024-03-30T15:00:48.223265+0000","last_fullsized":"2024-03-30T15:00:48.223265+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.174017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.174017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.d","version":"115'5","reported_seq":22,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-30T15:00:48.219612+0000","last_change":"2024-03-30T15:00:45.259199+0000","last_active":"2024-03-30T15:00:48.219612+0000","last_peered":"2024-03-30T15:00:48.219612+0000","last_clean":"2024-03-30T15:00:48.219612+0000","last_became_active":"2024-03-30T15:00:45.252183+0000","last_became_peered":"2024-03-30T15:00:45.252183+0000","last_unstale":"2024-03-30T15:00:48.219612+0000","last_undegraded":"2024-03-30T15:00:48.219612+0000","last_fullsized":"2024-03-30T15:00:48.219612+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.045519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.045519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.091151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":255,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":4195498,"num_objects":119,"num_object_clones":1,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":95,"num_read_kb":1,"num_write":539,"num_write_kb":8725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14020608,"data_stored":12586494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1787,"ondisk_log_size":1787,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":17,"num_pg":31,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005135,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72816,"kb_used_data":5528,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672336,"statfs":{"total":95995035648,"available":95920472064,"internally_reserved":0,"allocated":5660672,"data_stored":3402871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299476,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76940,"kb_used_data":5484,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668212,"statfs":{"total":95995035648,"available":95916249088,"internally_reserved":0,"allocated":5615616,"data_stored":3400836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72196,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672956,"statfs":{"total":95995035648,"available":95921106944,"internally_reserved":0,"allocated":5033984,"data_stored":2822221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54724,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93690428,"statfs":{"total":95995035648,"available":95938998272,"internally_reserved":0,"allocated":9867264,"data_stored":7607949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80492,"kb_used_data":9036,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664660,"statfs":{"total":95995035648,"available":95912611840,"internally_reserved":0,"allocated":9252864,"data_stored":7016145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80516,"kb_used_data":9076,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664636,"statfs":{"total":95995035648,"available":95912587264,"internally_reserved":0,"allocated":9293824,"data_stored":7017379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76396,"kb_used_data":4948,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668756,"statfs":{"total":95995035648,"available":95916806144,"internally_reserved":0,"allocated":5066752,"data_stored":2815724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659225,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76408,"kb_used_data":4960,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668744,"statfs":{"total":95995035648,"available":95916793856,"internally_reserved":0,"allocated":5079040,"data_stored":2825710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4370432,"data_stored":4194729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4382720,"data_stored":4194768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":4194795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:00:53.946 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-30T15:00:53.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:00:53.947 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T15:00:53.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:00:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:53.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:54.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:54.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:54.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:54.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:54.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:54.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:54.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.444+0000 7f6d83b6d700 1 -- 172.21.15.28:0/2176077963 >> v1:172.21.15.28:6790/0 conn(0x7f6d7c10d970 legacy=0x7f6d7c10fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.444+0000 7f6d83b6d700 1 -- 172.21.15.28:0/2176077963 shutdown_connections 2024-03-30T15:00:54.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.444+0000 7f6d83b6d700 1 -- 172.21.15.28:0/2176077963 >> 172.21.15.28:0/2176077963 conn(0x7f6d7c077370 msgr2=0x7f6d7c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:54.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.444+0000 7f6d83b6d700 1 -- 172.21.15.28:0/2176077963 shutdown_connections 2024-03-30T15:00:54.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.444+0000 7f6d83b6d700 1 -- 172.21.15.28:0/2176077963 wait complete. 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 librados: starting msgr at 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 librados: starting objecter 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 Processor -- start 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 -- start start 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 librados: setting wanted keys 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 librados: calling monclient init 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d7c106810 con 0x7f6d7c109e00 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d7c106a20 con 0x7f6d7c10d970 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.445+0000 7f6d83b6d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d7c1ab9b0 con 0x7f6d7c101810 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d8210a700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6d7c10d970 0x7f6d7c1a8230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:40616/0 (socket says 172.21.15.28:40616) 2024-03-30T15:00:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d81108700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6d7c109e00 0x7f6d7c103d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42486/0 (socket says 172.21.15.28:42486) 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d8210a700 1 -- 172.21.15.28:0/3861333177 learned_addr learned my addr 172.21.15.28:0/3861333177 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d81909700 1 --1- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6790/0 conn(0x7f6d7c101810 0x7f6d7c1035f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49548/0 (socket says 172.21.15.28:49548) 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter ms_handle_connect 0x7f6d7c109e00 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter ms_handle_connect 0x7f6d7c10d970 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter ms_handle_connect 0x7f6d7c101810 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797952787 0 0) 0x7f6d7c106a20 con 0x7f6d7c10d970 2024-03-30T15:00:54.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d680031f0 con 0x7f6d7c10d970 2024-03-30T15:00:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3649972101 0 0) 0x7f6d680031f0 con 0x7f6d7c10d970 2024-03-30T15:00:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d68004210 con 0x7f6d7c10d970 2024-03-30T15:00:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d740017b0 con 0x7f6d7c10d970 2024-03-30T15:00:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141046103 0 0) 0x7f6d7c1ab9b0 con 0x7f6d7c101810 2024-03-30T15:00:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d680031f0 con 0x7f6d7c101810 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1977782317 0 0) 0x7f6d68004210 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6790/0 conn(0x7f6d7c101810 legacy=0x7f6d7c1035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6789/0 conn(0x7f6d7c109e00 legacy=0x7f6d7c103d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c1acbd0 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.446+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7c1abbe0 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d83b6d700 10 client.49618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d7c1ac0a0 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d83b6d700 1 librados: init done 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6d740019d0 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d740050e0 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.447+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6d60005100 con 0x7f6d7c10d970 2024-03-30T15:00:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.448+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6d74005320 con 0x7f6d7c10d970 2024-03-30T15:00:54.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(290..290 src has 53..290) ==== 9323+0+0 (unknown 3971711090 0 0) 0x7f6d74003a80 con 0x7f6d7c10d970 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 osd_map(290..290 src has 53..290) 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 3 client.49618.objecter handle_osd_map got epochs [290,290] > 0 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 3 client.49618.objecter handle_osd_map decoding full epoch 290 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 20 client.49618.objecter dump_active .. 0 homeless 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.449+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v290) ==== 54+0+16 (unknown 4252067127 0 2420000192) 0x7f6d7405a450 con 0x7f6d7c10d970 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 1 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d68070850 2024-03-30T15:00:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 1 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 0 1 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 4 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 10 client.49618.objecter _send_command 1 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d60004b50 con 0x7f6d60004730 2024-03-30T15:00:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.450+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d60004730 2024-03-30T15:00:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.451+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d60004b50 con 0x7f6d60004730 2024-03-30T15:00:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.451+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 1: 0 ) 2024-03-30T15:00:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.451+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 1 = system:0 2024-03-30T15:00:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.451+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 4 2024-03-30T15:00:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.451+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 0 1 2024-03-30T15:00:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 2 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 2 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d600053e0 osd=0 3 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 0 2 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 4 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 10 client.49618.objecter _send_command 2 2024-03-30T15:00:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.470+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60007b00 con 0x7f6d60004730 2024-03-30T15:00:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.472+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60007b00 con 0x7f6d60004730 2024-03-30T15:00:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.472+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.472+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.472+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 4 2024-03-30T15:00:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.472+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 0 2 2024-03-30T15:00:54.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:00:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 1 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 4 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 10 client.49618.objecter _send_command 3 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6d600021c0 con 0x7f6d60004320 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.473+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d60004320 2024-03-30T15:00:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.474+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d600021c0 con 0x7f6d60004320 2024-03-30T15:00:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.474+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 3: 0 ) 2024-03-30T15:00:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.474+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 3 = system:0 2024-03-30T15:00:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.474+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 4 2024-03-30T15:00:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.474+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 1 3 2024-03-30T15:00:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 4 2024-03-30T15:00:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 4 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d600041b0 osd=1 3 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 1 4 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 4 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 10 client.49618.objecter _send_command 4 2024-03-30T15:00:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.493+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60001600 con 0x7f6d60004320 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60001600 con 0x7f6d60004320 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 4 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 1 4 2024-03-30T15:00:54.497 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 5 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 5 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 2 5 2024-03-30T15:00:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 4 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 10 client.49618.objecter _send_command 5 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6d60011070 con 0x7f6d600089d0 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.496+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d600089d0 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.498+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d60011070 con 0x7f6d600089d0 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.498+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 5: 0 ) 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.498+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 5 = system:0 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.498+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 4 2024-03-30T15:00:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.498+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 2 5 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 6 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 6 2024-03-30T15:00:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60001a00 osd=2 3 2024-03-30T15:00:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 2 6 2024-03-30T15:00:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 4 2024-03-30T15:00:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 10 client.49618.objecter _send_command 6 2024-03-30T15:00:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.516+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60007f60 con 0x7f6d600089d0 2024-03-30T15:00:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60007f60 con 0x7f6d600089d0 2024-03-30T15:00:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 4 2024-03-30T15:00:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 2 6 2024-03-30T15:00:54.520 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 7 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 7 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 3 7 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 4 2024-03-30T15:00:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 10 client.49618.objecter _send_command 7 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6d6000d4a0 con 0x7f6d60011250 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.518+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d60011250 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.519+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d6000d4a0 con 0x7f6d60011250 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.519+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 7: 0 ) 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.519+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 7 = system:0 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.519+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 4 2024-03-30T15:00:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.519+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 3 7 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 8 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 8 2024-03-30T15:00:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d6000cd30 osd=3 3 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 3 8 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 4 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 10 client.49618.objecter _send_command 8 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.536+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d600125c0 con 0x7f6d60011250 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d600125c0 con 0x7f6d60011250 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 4 2024-03-30T15:00:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 3 8 2024-03-30T15:00:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 9 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 9 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 4 9 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 4 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 10 client.49618.objecter _send_command 9 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.537+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6d6001a870 con 0x7f6d6000c870 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.538+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d6000c870 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.539+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d6001a870 con 0x7f6d6000c870 2024-03-30T15:00:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.539+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 9: 0 ) 2024-03-30T15:00:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.539+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 9 = system:0 2024-03-30T15:00:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.539+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 4 2024-03-30T15:00:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.539+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 4 9 2024-03-30T15:00:54.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 10 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 10 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60013740 osd=4 3 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 4 10 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 4 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 10 client.49618.objecter _send_command 10 2024-03-30T15:00:54.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.555+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60012810 con 0x7f6d6000c870 2024-03-30T15:00:54.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60012810 con 0x7f6d6000c870 2024-03-30T15:00:54.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 4 2024-03-30T15:00:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 4 10 2024-03-30T15:00:54.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 11 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 11 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 5 11 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 4 2024-03-30T15:00:54.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 10 client.49618.objecter _send_command 11 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.557+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6d60013340 con 0x7f6d60012e90 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d60012e90 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d60013340 con 0x7f6d60012e90 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 11: 0 ) 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 11 = system:0 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 4 2024-03-30T15:00:54.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.558+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 5 11 2024-03-30T15:00:54.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 12 2024-03-30T15:00:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 12 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60012810 osd=5 3 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 5 12 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 4 2024-03-30T15:00:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 10 client.49618.objecter _send_command 12 2024-03-30T15:00:54.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.575+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60013340 con 0x7f6d60012e90 2024-03-30T15:00:54.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60013340 con 0x7f6d60012e90 2024-03-30T15:00:54.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 4 2024-03-30T15:00:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 5 12 2024-03-30T15:00:54.580 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:00:54.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 13 2024-03-30T15:00:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 13 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 6 13 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 4 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 10 client.49618.objecter _send_command 13 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6d600232e0 con 0x7f6d6001f630 2024-03-30T15:00:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.577+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d6001f630 2024-03-30T15:00:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.578+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d600232e0 con 0x7f6d6001f630 2024-03-30T15:00:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.578+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 13: 0 ) 2024-03-30T15:00:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.578+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 13 = system:0 2024-03-30T15:00:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.578+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 4 2024-03-30T15:00:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.578+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 6 13 2024-03-30T15:00:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:54.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 14 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 14 2024-03-30T15:00:54.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60013340 osd=6 3 2024-03-30T15:00:54.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 6 14 2024-03-30T15:00:54.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 4 2024-03-30T15:00:54.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 10 client.49618.objecter _send_command 14 2024-03-30T15:00:54.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.595+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d60024c90 con 0x7f6d6001f630 2024-03-30T15:00:54.597 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:00:54.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.596+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d60024c90 con 0x7f6d6001f630 2024-03-30T15:00:54.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.596+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.596+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.596+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 4 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.596+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 6 14 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 15 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 15 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 7 15 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 4 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 10 client.49618.objecter _send_command 15 2024-03-30T15:00:54.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6d600258f0 con 0x7f6d600253b0 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.597+0000 7f6d72ffd700 10 client.49618.objecter ms_handle_connect 0x7f6d600253b0 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.598+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d600258f0 con 0x7f6d600253b0 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.598+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 15: 0 ) 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.598+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 15 = system:0 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.598+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 4 2024-03-30T15:00:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.598+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 7 15 2024-03-30T15:00:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 10 client.49618.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign -1 16 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 20 client.49618.objecter _get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_remove -1 16 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 20 client.49618.objecter get_session s=0x7f6d60024c90 osd=7 3 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 15 client.49618.objecter _session_command_op_assign 7 16 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 4 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 10 client.49618.objecter _send_command 16 2024-03-30T15:00:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.616+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6d600258f0 con 0x7f6d600253b0 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.618+0000 7f6d72ffd700 1 -- 172.21.15.28:0/3861333177 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d600258f0 con 0x7f6d600253b0 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.618+0000 7f6d72ffd700 10 client.49618.objecter ms_dispatch 0x7f6d7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.618+0000 7f6d72ffd700 10 client.49618.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.618+0000 7f6d72ffd700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 4 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.618+0000 7f6d72ffd700 15 client.49618.objecter _session_command_op_remove 7 16 2024-03-30T15:00:54.620 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 librados: watch_flush enter 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 librados: watch_flush exit 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.0 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6d60004730 legacy=0x7f6d60002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600053e0 osd=0 1 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.1 2024-03-30T15:00:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6d60004320 legacy=0x7f6d6000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d600041b0 osd=1 1 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.2 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6d600089d0 legacy=0x7f6d6000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60001a00 osd=2 1 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.3 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6d60011250 legacy=0x7f6d60011690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d6000cd30 osd=3 1 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.4 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d8210a700 1 -- 172.21.15.28:0/3861333177 reap_dead start 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6d6000c870 legacy=0x7f6d60019150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d8210a700 1 -- 172.21.15.28:0/3861333177 reap_dead start 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013740 osd=4 1 2024-03-30T15:00:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.5 2024-03-30T15:00:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.134:6804/3668151179 conn(0x7f6d60012e90 legacy=0x7f6d6001da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60012810 osd=5 1 2024-03-30T15:00:54.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.6 2024-03-30T15:00:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.134:6808/2135224194 conn(0x7f6d6001f630 legacy=0x7f6d60021a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60013340 osd=6 1 2024-03-30T15:00:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter close_session for osd.7 2024-03-30T15:00:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6d600253b0 legacy=0x7f6d600232e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter put_session s=0x7f6d60024c90 osd=7 1 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 20 client.49618.objecter shutdown clearing up homeless session... 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 10 client.49618.objecter successfully canceled tick 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6d68070850 legacy=0x7f6d68072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> v1:172.21.15.134:6789/0 conn(0x7f6d7c10d970 legacy=0x7f6d7c1a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d8210a700 1 -- 172.21.15.28:0/3861333177 reap_dead start 2024-03-30T15:00:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d8210a700 1 -- 172.21.15.28:0/3861333177 reap_dead start 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 shutdown_connections 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 >> 172.21.15.28:0/3861333177 conn(0x7f6d7c077370 msgr2=0x7f6d7c1105e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 shutdown_connections 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 -- 172.21.15.28:0/3861333177 wait complete. 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:54.619+0000 7f6d83b6d700 1 librados: shutdown 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:00:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:00:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:00:54.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:00:54.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:54.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:54.728 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: pgmap v374: 299 pgs: 6 creating+peering, 2 unknown, 291 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3861333177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:54.729 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: pgmap v374: 299 pgs: 6 creating+peering, 2 unknown, 291 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:54.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-30T15:00:54.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T15:00:54.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3861333177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:54.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:54.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:54.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:54.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: pgmap v374: 299 pgs: 6 creating+peering, 2 unknown, 291 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2485876767' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi028-92999-48"}]': finished 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: osdmap e290: 8 total, 8 up, 8 in 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3861333177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:54.919 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:54.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:54.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:54.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:55.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:55.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:55.165 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:00:55.165 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:00:55.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:55.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:55.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:55.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:55.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:55.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:55.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:55.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:55.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:55.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:55.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:00:55.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:55.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:55.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:55.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:56.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:56.017 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:56.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:56.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:56.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:56.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:56.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:56.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:56.419 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.420 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:56 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi028-92999-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: osdmap e291: 8 total, 8 up, 8 in 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:00:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:56.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:56 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:00:56.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:56.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:56.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:56.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:56.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:56.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:56.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:56.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:56.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.993+0000 7ff1e1837700 1 -- 172.21.15.28:0/1933772389 >> v1:172.21.15.28:6789/0 conn(0x7ff1dc10ea90 legacy=0x7ff1dc110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.993+0000 7ff1e1837700 1 -- 172.21.15.28:0/1933772389 shutdown_connections 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.993+0000 7ff1e1837700 1 -- 172.21.15.28:0/1933772389 >> 172.21.15.28:0/1933772389 conn(0x7ff1dc0fdb30 msgr2=0x7ff1dc0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.993+0000 7ff1e1837700 1 -- 172.21.15.28:0/1933772389 shutdown_connections 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.993+0000 7ff1e1837700 1 -- 172.21.15.28:0/1933772389 wait complete. 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 librados: starting msgr at 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 librados: starting objecter 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 Processor -- start 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 -- start start 2024-03-30T15:00:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 librados: setting wanted keys 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.994+0000 7ff1e1837700 1 librados: calling monclient init 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e1837700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1dc1a7170 con 0x7ff1dc10ea90 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e1837700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1dc1a8390 con 0x7ff1dc10aef0 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e1837700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1dc1a95d0 con 0x7ff1dc102260 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1db7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff1dc10ea90 0x7ff1dc1a5870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42534/0 (socket says 172.21.15.28:42534) 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1db7fe700 1 -- 172.21.15.28:0/3716318344 learned_addr learned my addr 172.21.15.28:0/3716318344 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1daffd700 1 --1- 172.21.15.28:0/3716318344 >> v1:172.21.15.28:6790/0 conn(0x7ff1dc102260 0x7ff1dc107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49600/0 (socket says 172.21.15.28:49600) 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter ms_handle_connect 0x7ff1dc10ea90 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter ms_handle_connect 0x7ff1dc102260 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter ms_handle_connect 0x7ff1dc10aef0 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:56.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968420315 0 0) 0x7ff1dc1a7170 con 0x7ff1dc10ea90 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1c40031f0 con 0x7ff1dc10ea90 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219150679 0 0) 0x7ff1dc1a8390 con 0x7ff1dc10aef0 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1dc1a7170 con 0x7ff1dc10aef0 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364199470 0 0) 0x7ff1dc1a95d0 con 0x7ff1dc102260 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.995+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1dc1a8390 con 0x7ff1dc102260 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3449249720 0 0) 0x7ff1dc1a7170 con 0x7ff1dc10aef0 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1dc1a95d0 con 0x7ff1dc10aef0 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 862277505 0 0) 0x7ff1dc1a8390 con 0x7ff1dc102260 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1dc1a7170 con 0x7ff1dc102260 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1d00017a0 con 0x7ff1dc10aef0 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3015947683 0 0) 0x7ff1c40031f0 con 0x7ff1dc10ea90 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1dc1a8390 con 0x7ff1dc10ea90 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1cc001830 con 0x7ff1dc102260 2024-03-30T15:00:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1d4004860 con 0x7ff1dc10ea90 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1780742810 0 0) 0x7ff1dc1a95d0 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 >> v1:172.21.15.28:6790/0 conn(0x7ff1dc102260 legacy=0x7ff1dc107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 >> v1:172.21.15.28:6789/0 conn(0x7ff1dc10ea90 legacy=0x7ff1dc1a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc1aa810 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc1a9800 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e1837700 10 client.49630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1dc1a9cc0 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e1837700 1 librados: init done 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff1d0003b20 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.996+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1d0004920 con 0x7ff1dc10aef0 2024-03-30T15:00:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.997+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c8005120 con 0x7ff1dc10aef0 2024-03-30T15:00:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.997+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff1d001b240 con 0x7ff1dc10aef0 2024-03-30T15:00:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.998+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(292..292 src has 53..292) ==== 9400+0+0 (unknown 3301655431 0 0) 0x7ff1d0088870 con 0x7ff1dc10aef0 2024-03-30T15:00:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.998+0000 7ff1e0835700 10 client.49630.objecter ms_dispatch 0x7ff1dc001550 osd_map(292..292 src has 53..292) 2024-03-30T15:00:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.998+0000 7ff1e0835700 3 client.49630.objecter handle_osd_map got epochs [292,292] > 0 2024-03-30T15:00:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.998+0000 7ff1e0835700 3 client.49630.objecter handle_osd_map decoding full epoch 292 2024-03-30T15:00:57.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.998+0000 7ff1e0835700 20 client.49630.objecter dump_active .. 0 homeless 2024-03-30T15:00:57.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:56.999+0000 7ff1e0835700 10 client.49630.objecter ms_handle_connect 0x7ff1c4074800 2024-03-30T15:00:57.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.003+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff1d0055590 con 0x7ff1dc10aef0 2024-03-30T15:00:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:57.031 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:57.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:57.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.216+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1c8000b90 con 0x7ff1c4074800 2024-03-30T15:00:57.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.249+0000 7ff1e0835700 1 -- 172.21.15.28:0/3716318344 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+738553 (unknown 2965378022 0 3477066501) 0x7ff1c8000b90 con 0x7ff1c4074800 2024-03-30T15:00:57.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:00:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 10 librados: watch_flush enter 2024-03-30T15:00:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 10 librados: watch_flush exit 2024-03-30T15:00:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 20 client.49630.objecter shutdown clearing up homeless session... 2024-03-30T15:00:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 10 client.49630.objecter successfully canceled tick 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff1c4074800 legacy=0x7ff1c4076cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 >> v1:172.21.15.134:6789/0 conn(0x7ff1dc10aef0 legacy=0x7ff1dc1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 shutdown_connections 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 >> 172.21.15.28:0/3716318344 conn(0x7ff1dc0fdb30 msgr2=0x7ff1dc111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 shutdown_connections 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 -- 172.21.15.28:0/3716318344 wait complete. 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:57.256+0000 7ff1e1837700 1 librados: shutdown 2024-03-30T15:00:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:00:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:57.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: pgmap v377: 291 pgs: 32 unknown, 259 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:57.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:57.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.376 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:57 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:57.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: pgmap v377: 291 pgs: 32 unknown, 259 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: pgmap v377: 291 pgs: 32 unknown, 259 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1782570749' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: osdmap e292: 8 total, 8 up, 8 in 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]: dispatch 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:00:57.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:57 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:00:57.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:57.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:57.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:00:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:00:57.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:57.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:57.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:57.931 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:00:57.931 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:00:58.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:58.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:58.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:58.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:00:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:58.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:00:58.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:58.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:00:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.49630 v1:172.21.15.28:0/3716318344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T15:00:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.49630 v1:172.21.15.28:0/3716318344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:58.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:58 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-03-30T15:00:56.220030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793877,"num_objects":335,"num_object_clones":1,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1399,"num_read_kb":1151,"num_write":1262,"num_write_kb":11252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1772962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2568,"ondisk_log_size":2568,"up":774,"acting":774,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":888,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":591140,"kb_used_data":54252,"kb_used_omap":12,"kb_used_meta":536691,"kb_avail":749370076,"statfs":{"total":767960285184,"available":767354957824,"internally_reserved":0,"allocated":55554048,"data_stored":37360089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":549572144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":136,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010047"},"pg_stats":[{"pgid":"258.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"258.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:00:55.212409+0000","last_change":"2024-03-30T15:00:55.212409+0000","last_active":"2024-03-30T15:00:55.212409+0000","last_peered":"2024-03-30T15:00:55.212409+0000","last_clean":"2024-03-30T15:00:55.212409+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:00:55.212409+0000","last_undegraded":"2024-03-30T15:00:55.212409+0000","last_fullsized":"2024-03-30T15:00:55.212409+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:55.212409+0000","last_clean_scrub_stamp":"2024-03-30T15:00:55.212409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224088+0000","last_change":"2024-03-30T15:00:43.233147+0000","last_active":"2024-03-30T15:00:53.224088+0000","last_peered":"2024-03-30T15:00:53.224088+0000","last_clean":"2024-03-30T15:00:53.224088+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:00:53.224088+0000","last_undegraded":"2024-03-30T15:00:53.224088+0000","last_fullsized":"2024-03-30T15:00:53.224088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01065052,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"229'85","reported_seq":508,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222154+0000","last_change":"2024-03-30T15:00:43.233337+0000","last_active":"2024-03-30T15:00:53.222154+0000","last_peered":"2024-03-30T15:00:53.222154+0000","last_clean":"2024-03-30T15:00:53.222154+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:00:53.222154+0000","last_undegraded":"2024-03-30T15:00:53.222154+0000","last_fullsized":"2024-03-30T15:00:53.222154+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025393439999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"230'87","reported_seq":512,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219783+0000","last_change":"2024-03-30T15:00:43.224129+0000","last_active":"2024-03-30T15:00:53.219783+0000","last_peered":"2024-03-30T15:00:53.219783+0000","last_clean":"2024-03-30T15:00:53.219783+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:00:53.219783+0000","last_undegraded":"2024-03-30T15:00:53.219783+0000","last_fullsized":"2024-03-30T15:00:53.219783+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001687252,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"204'102","reported_seq":529,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213014+0000","last_change":"2024-03-30T15:00:43.225813+0000","last_active":"2024-03-30T15:00:53.213014+0000","last_peered":"2024-03-30T15:00:53.213014+0000","last_clean":"2024-03-30T15:00:53.213014+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:00:53.213014+0000","last_undegraded":"2024-03-30T15:00:53.213014+0000","last_fullsized":"2024-03-30T15:00:53.213014+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001343522,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"286'36","reported_seq":480,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223531+0000","last_change":"2024-03-30T15:00:50.725913+0000","last_active":"2024-03-30T15:00:53.223531+0000","last_peered":"2024-03-30T15:00:53.223531+0000","last_clean":"2024-03-30T15:00:53.223531+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:00:53.223531+0000","last_undegraded":"2024-03-30T15:00:53.223531+0000","last_fullsized":"2024-03-30T15:00:53.223531+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.725810+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:07.548979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0087880479999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"283'65","reported_seq":558,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224078+0000","last_change":"2024-03-30T15:00:47.743389+0000","last_active":"2024-03-30T15:00:53.224078+0000","last_peered":"2024-03-30T15:00:53.224078+0000","last_clean":"2024-03-30T15:00:53.224078+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:00:53.224078+0000","last_undegraded":"2024-03-30T15:00:53.224078+0000","last_fullsized":"2024-03-30T15:00:53.224078+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'65","last_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_deep_scrub":"283'65","last_deep_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.743297+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.629294+0000","scrub_duration":10,"objects_trimmed":1,"snaptrim_duration":0.011100334,"stat_sum":{"num_bytes":4194341,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"222'34","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221528+0000","last_change":"2024-03-30T15:00:43.235577+0000","last_active":"2024-03-30T15:00:53.221528+0000","last_peered":"2024-03-30T15:00:53.221528+0000","last_clean":"2024-03-30T15:00:53.221528+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:00:53.221528+0000","last_undegraded":"2024-03-30T15:00:53.221528+0000","last_fullsized":"2024-03-30T15:00:53.221528+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021958989999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"204'51","reported_seq":481,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222388+0000","last_change":"2024-03-30T15:00:43.234880+0000","last_active":"2024-03-30T15:00:53.222388+0000","last_peered":"2024-03-30T15:00:53.222388+0000","last_clean":"2024-03-30T15:00:53.222388+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:00:53.222388+0000","last_undegraded":"2024-03-30T15:00:53.222388+0000","last_fullsized":"2024-03-30T15:00:53.222388+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095938800000000008,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"287'40","reported_seq":474,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218969+0000","last_change":"2024-03-30T15:00:43.212808+0000","last_active":"2024-03-30T15:00:53.218969+0000","last_peered":"2024-03-30T15:00:53.218969+0000","last_clean":"2024-03-30T15:00:53.218969+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:00:53.218969+0000","last_undegraded":"2024-03-30T15:00:53.218969+0000","last_fullsized":"2024-03-30T15:00:53.218969+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034693419999999998,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":470,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219908+0000","last_change":"2024-03-30T15:00:43.213650+0000","last_active":"2024-03-30T15:00:53.219908+0000","last_peered":"2024-03-30T15:00:53.219908+0000","last_clean":"2024-03-30T15:00:53.219908+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:00:53.219908+0000","last_undegraded":"2024-03-30T15:00:53.219908+0000","last_fullsized":"2024-03-30T15:00:53.219908+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041086289999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"0'0","reported_seq":439,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219972+0000","last_change":"2024-03-30T15:00:43.225727+0000","last_active":"2024-03-30T15:00:53.219972+0000","last_peered":"2024-03-30T15:00:53.219972+0000","last_clean":"2024-03-30T15:00:53.219972+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:00:53.219972+0000","last_undegraded":"2024-03-30T15:00:53.219972+0000","last_fullsized":"2024-03-30T15:00:53.219972+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030227269999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"204'17","reported_seq":456,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220577+0000","last_change":"2024-03-30T15:00:43.224756+0000","last_active":"2024-03-30T15:00:53.220577+0000","last_peered":"2024-03-30T15:00:53.220577+0000","last_clean":"2024-03-30T15:00:53.220577+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:00:53.220577+0000","last_undegraded":"2024-03-30T15:00:53.220577+0000","last_fullsized":"2024-03-30T15:00:53.220577+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020847700000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"217'17","reported_seq":452,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230769+0000","last_change":"2024-03-30T15:00:43.236048+0000","last_active":"2024-03-30T15:00:53.230769+0000","last_peered":"2024-03-30T15:00:53.230769+0000","last_clean":"2024-03-30T15:00:53.230769+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:00:53.230769+0000","last_undegraded":"2024-03-30T15:00:53.230769+0000","last_fullsized":"2024-03-30T15:00:53.230769+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013371539999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"241'73","reported_seq":508,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221435+0000","last_change":"2024-03-30T15:00:43.217166+0000","last_active":"2024-03-30T15:00:53.221435+0000","last_peered":"2024-03-30T15:00:53.221435+0000","last_clean":"2024-03-30T15:00:53.221435+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:00:53.221435+0000","last_undegraded":"2024-03-30T15:00:53.221435+0000","last_fullsized":"2024-03-30T15:00:53.221435+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001827506,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"233'51","reported_seq":484,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218050+0000","last_change":"2024-03-30T15:00:43.214272+0000","last_active":"2024-03-30T15:00:53.218050+0000","last_peered":"2024-03-30T15:00:53.218050+0000","last_clean":"2024-03-30T15:00:53.218050+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:00:53.218050+0000","last_undegraded":"2024-03-30T15:00:53.218050+0000","last_fullsized":"2024-03-30T15:00:53.218050+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00145184,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"267'90","reported_seq":520,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222857+0000","last_change":"2024-03-30T15:00:43.231273+0000","last_active":"2024-03-30T15:00:53.222857+0000","last_peered":"2024-03-30T15:00:53.222857+0000","last_clean":"2024-03-30T15:00:53.222857+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:00:53.222857+0000","last_undegraded":"2024-03-30T15:00:53.222857+0000","last_fullsized":"2024-03-30T15:00:53.222857+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'90","last_scrub_stamp":"2024-03-30T15:00:41.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.653662+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.010425+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0083710509999999991,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"204'34","reported_seq":469,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223432+0000","last_change":"2024-03-30T15:00:43.227678+0000","last_active":"2024-03-30T15:00:53.223432+0000","last_peered":"2024-03-30T15:00:53.223432+0000","last_clean":"2024-03-30T15:00:53.223432+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:00:53.223432+0000","last_undegraded":"2024-03-30T15:00:53.223432+0000","last_fullsized":"2024-03-30T15:00:53.223432+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047107750000000004,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":540,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220538+0000","last_change":"2024-03-30T15:00:43.219728+0000","last_active":"2024-03-30T15:00:53.220538+0000","last_peered":"2024-03-30T15:00:53.220538+0000","last_clean":"2024-03-30T15:00:53.220538+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:00:53.220538+0000","last_undegraded":"2024-03-30T15:00:53.220538+0000","last_fullsized":"2024-03-30T15:00:53.220538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:00:30.724310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.724310+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.577626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018219289999999999,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"204'34","reported_seq":478,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225318+0000","last_change":"2024-03-30T15:00:43.228511+0000","last_active":"2024-03-30T15:00:53.225318+0000","last_peered":"2024-03-30T15:00:53.225318+0000","last_clean":"2024-03-30T15:00:53.225318+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:00:53.225318+0000","last_undegraded":"2024-03-30T15:00:53.225318+0000","last_fullsized":"2024-03-30T15:00:53.225318+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T14:59:47.247116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.247116+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.949404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0019213380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":483,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222852+0000","last_change":"2024-03-30T15:00:43.218761+0000","last_active":"2024-03-30T15:00:53.222852+0000","last_peered":"2024-03-30T15:00:53.222852+0000","last_clean":"2024-03-30T15:00:53.222852+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:00:53.222852+0000","last_undegraded":"2024-03-30T15:00:53.222852+0000","last_fullsized":"2024-03-30T15:00:53.222852+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:00:37.797402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.797402+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.412666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024570769999999998,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":541,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225011+0000","last_change":"2024-03-30T15:00:43.219557+0000","last_active":"2024-03-30T15:00:53.225011+0000","last_peered":"2024-03-30T15:00:53.225011+0000","last_clean":"2024-03-30T15:00:53.225011+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:00:53.225011+0000","last_undegraded":"2024-03-30T15:00:53.225011+0000","last_fullsized":"2024-03-30T15:00:53.225011+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:00:31.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:31.737537+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:35.034652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017474070000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"248'56","reported_seq":494,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222662+0000","last_change":"2024-03-30T15:00:43.217112+0000","last_active":"2024-03-30T15:00:53.222662+0000","last_peered":"2024-03-30T15:00:53.222662+0000","last_clean":"2024-03-30T15:00:53.222662+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:00:53.222662+0000","last_undegraded":"2024-03-30T15:00:53.222662+0000","last_fullsized":"2024-03-30T15:00:53.222662+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00150314,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"204'51","reported_seq":489,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222679+0000","last_change":"2024-03-30T15:00:43.232136+0000","last_active":"2024-03-30T15:00:53.222679+0000","last_peered":"2024-03-30T15:00:53.222679+0000","last_clean":"2024-03-30T15:00:53.222679+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:00:53.222679+0000","last_undegraded":"2024-03-30T15:00:53.222679+0000","last_fullsized":"2024-03-30T15:00:53.222679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:00:29.555003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:29.555003+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.064502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0074406610000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"235'87","reported_seq":509,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227390+0000","last_change":"2024-03-30T15:00:43.234991+0000","last_active":"2024-03-30T15:00:53.227390+0000","last_peered":"2024-03-30T15:00:53.227390+0000","last_clean":"2024-03-30T15:00:53.227390+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:00:53.227390+0000","last_undegraded":"2024-03-30T15:00:53.227390+0000","last_fullsized":"2024-03-30T15:00:53.227390+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036646000000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"204'85","reported_seq":517,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214799+0000","last_change":"2024-03-30T15:00:43.230968+0000","last_active":"2024-03-30T15:00:53.214799+0000","last_peered":"2024-03-30T15:00:53.214799+0000","last_clean":"2024-03-30T15:00:53.214799+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:00:53.214799+0000","last_undegraded":"2024-03-30T15:00:53.214799+0000","last_fullsized":"2024-03-30T15:00:53.214799+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'85","last_scrub_stamp":"2024-03-30T15:00:35.252370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.252370+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:52.182528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053428640000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":482,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223184+0000","last_change":"2024-03-30T15:00:43.230248+0000","last_active":"2024-03-30T15:00:53.223184+0000","last_peered":"2024-03-30T15:00:53.223184+0000","last_clean":"2024-03-30T15:00:53.223184+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:00:53.223184+0000","last_undegraded":"2024-03-30T15:00:53.223184+0000","last_fullsized":"2024-03-30T15:00:53.223184+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:02.714228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.714228+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:26.058651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0084702249999999996,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"248'39","reported_seq":483,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222404+0000","last_change":"2024-03-30T15:00:43.227563+0000","last_active":"2024-03-30T15:00:53.222404+0000","last_peered":"2024-03-30T15:00:53.222404+0000","last_clean":"2024-03-30T15:00:53.222404+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:00:53.222404+0000","last_undegraded":"2024-03-30T15:00:53.222404+0000","last_fullsized":"2024-03-30T15:00:53.222404+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:04.710962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.710962+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:07.518453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0061256050000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"238'103","reported_seq":543,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223374+0000","last_change":"2024-03-30T15:00:43.219491+0000","last_active":"2024-03-30T15:00:53.223374+0000","last_peered":"2024-03-30T15:00:53.223374+0000","last_clean":"2024-03-30T15:00:53.223374+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:00:53.223374+0000","last_undegraded":"2024-03-30T15:00:53.223374+0000","last_fullsized":"2024-03-30T15:00:53.223374+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'103","last_scrub_stamp":"2024-03-30T15:00:35.786699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.786699+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.117141+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022054919999999999,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"204'22","reported_seq":463,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230324+0000","last_change":"2024-03-30T15:00:43.235889+0000","last_active":"2024-03-30T15:00:53.230324+0000","last_peered":"2024-03-30T15:00:53.230324+0000","last_clean":"2024-03-30T15:00:53.230324+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:00:53.230324+0000","last_undegraded":"2024-03-30T15:00:53.230324+0000","last_fullsized":"2024-03-30T15:00:53.230324+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091356760000000006,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"0'0","reported_seq":449,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224403+0000","last_change":"2024-03-30T15:00:43.234537+0000","last_active":"2024-03-30T15:00:53.224403+0000","last_peered":"2024-03-30T15:00:53.224403+0000","last_clean":"2024-03-30T15:00:53.224403+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:00:53.224403+0000","last_undegraded":"2024-03-30T15:00:53.224403+0000","last_fullsized":"2024-03-30T15:00:53.224403+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.644063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.644063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.771208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0083022960000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":495,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218189+0000","last_change":"2024-03-30T15:00:43.214361+0000","last_active":"2024-03-30T15:00:53.218189+0000","last_peered":"2024-03-30T15:00:53.218189+0000","last_clean":"2024-03-30T15:00:53.218189+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:00:53.218189+0000","last_undegraded":"2024-03-30T15:00:53.218189+0000","last_fullsized":"2024-03-30T15:00:53.218189+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001262364,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"204'102","reported_seq":533,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.229277+0000","last_change":"2024-03-30T15:00:43.233284+0000","last_active":"2024-03-30T15:00:53.229277+0000","last_peered":"2024-03-30T15:00:53.229277+0000","last_clean":"2024-03-30T15:00:53.229277+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:00:53.229277+0000","last_undegraded":"2024-03-30T15:00:53.229277+0000","last_fullsized":"2024-03-30T15:00:53.229277+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T14:59:48.278271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.278271+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:17.838490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078062200000000002,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228713+0000","last_change":"2024-03-30T14:57:56.252990+0000","last_active":"2024-03-30T15:00:53.228713+0000","last_peered":"2024-03-30T15:00:53.228713+0000","last_clean":"2024-03-30T15:00:53.228713+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:00:53.228713+0000","last_undegraded":"2024-03-30T15:00:53.228713+0000","last_fullsized":"2024-03-30T15:00:53.228713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222451+0000","last_change":"2024-03-30T14:59:10.495352+0000","last_active":"2024-03-30T15:00:53.222451+0000","last_peered":"2024-03-30T15:00:53.222451+0000","last_clean":"2024-03-30T15:00:53.222451+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:00:53.222451+0000","last_undegraded":"2024-03-30T15:00:53.222451+0000","last_fullsized":"2024-03-30T15:00:53.222451+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.495272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.495272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":367,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231129+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:00:53.231129+0000","last_peered":"2024-03-30T15:00:53.231129+0000","last_clean":"2024-03-30T15:00:53.231129+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:00:53.231129+0000","last_undegraded":"2024-03-30T15:00:53.231129+0000","last_fullsized":"2024-03-30T15:00:53.231129+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.937324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214942+0000","last_change":"2024-03-30T15:00:50.386813+0000","last_active":"2024-03-30T15:00:53.214942+0000","last_peered":"2024-03-30T15:00:53.214942+0000","last_clean":"2024-03-30T15:00:53.214942+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:00:53.214942+0000","last_undegraded":"2024-03-30T15:00:53.214942+0000","last_fullsized":"2024-03-30T15:00:53.214942+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:50.386674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.386674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:17.393728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":360,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219309+0000","last_change":"2024-03-30T14:57:56.261442+0000","last_active":"2024-03-30T15:00:53.219309+0000","last_peered":"2024-03-30T15:00:53.219309+0000","last_clean":"2024-03-30T15:00:53.219309+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:00:53.219309+0000","last_undegraded":"2024-03-30T15:00:53.219309+0000","last_fullsized":"2024-03-30T15:00:53.219309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222566+0000","last_change":"2024-03-30T14:59:56.788394+0000","last_active":"2024-03-30T15:00:53.222566+0000","last_peered":"2024-03-30T15:00:53.222566+0000","last_clean":"2024-03-30T15:00:53.222566+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:00:53.222566+0000","last_undegraded":"2024-03-30T15:00:53.222566+0000","last_fullsized":"2024-03-30T15:00:53.222566+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.788278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.788278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:25.851592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223112+0000","last_change":"2024-03-30T14:57:56.244353+0000","last_active":"2024-03-30T15:00:53.223112+0000","last_peered":"2024-03-30T15:00:53.223112+0000","last_clean":"2024-03-30T15:00:53.223112+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:00:53.223112+0000","last_undegraded":"2024-03-30T15:00:53.223112+0000","last_fullsized":"2024-03-30T15:00:53.223112+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227083+0000","last_change":"2024-03-30T14:57:56.239220+0000","last_active":"2024-03-30T15:00:53.227083+0000","last_peered":"2024-03-30T15:00:53.227083+0000","last_clean":"2024-03-30T15:00:53.227083+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:00:53.227083+0000","last_undegraded":"2024-03-30T15:00:53.227083+0000","last_fullsized":"2024-03-30T15:00:53.227083+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222553+0000","last_change":"2024-03-30T14:57:56.240474+0000","last_active":"2024-03-30T15:00:53.222553+0000","last_peered":"2024-03-30T15:00:53.222553+0000","last_clean":"2024-03-30T15:00:53.222553+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:00:53.222553+0000","last_undegraded":"2024-03-30T15:00:53.222553+0000","last_fullsized":"2024-03-30T15:00:53.222553+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214927+0000","last_change":"2024-03-30T15:00:43.330329+0000","last_active":"2024-03-30T15:00:53.214927+0000","last_peered":"2024-03-30T15:00:53.214927+0000","last_clean":"2024-03-30T15:00:53.214927+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:00:53.214927+0000","last_undegraded":"2024-03-30T15:00:53.214927+0000","last_fullsized":"2024-03-30T15:00:53.214927+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.330250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.998934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231084+0000","last_change":"2024-03-30T14:57:56.253494+0000","last_active":"2024-03-30T15:00:53.231084+0000","last_peered":"2024-03-30T15:00:53.231084+0000","last_clean":"2024-03-30T15:00:53.231084+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:00:53.231084+0000","last_undegraded":"2024-03-30T15:00:53.231084+0000","last_fullsized":"2024-03-30T15:00:53.231084+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223479+0000","last_change":"2024-03-30T14:59:52.803098+0000","last_active":"2024-03-30T15:00:53.223479+0000","last_peered":"2024-03-30T15:00:53.223479+0000","last_clean":"2024-03-30T15:00:53.223479+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:00:53.223479+0000","last_undegraded":"2024-03-30T15:00:53.223479+0000","last_fullsized":"2024-03-30T15:00:53.223479+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.803012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.803012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.827147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220221+0000","last_change":"2024-03-30T14:57:56.253984+0000","last_active":"2024-03-30T15:00:53.220221+0000","last_peered":"2024-03-30T15:00:53.220221+0000","last_clean":"2024-03-30T15:00:53.220221+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:00:53.220221+0000","last_undegraded":"2024-03-30T15:00:53.220221+0000","last_fullsized":"2024-03-30T15:00:53.220221+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.212833+0000","last_change":"2024-03-30T15:00:40.305048+0000","last_active":"2024-03-30T15:00:53.212833+0000","last_peered":"2024-03-30T15:00:53.212833+0000","last_clean":"2024-03-30T15:00:53.212833+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:00:53.212833+0000","last_undegraded":"2024-03-30T15:00:53.212833+0000","last_fullsized":"2024-03-30T15:00:53.212833+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:40.304903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:40.304903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.820755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223217+0000","last_change":"2024-03-30T14:59:17.774212+0000","last_active":"2024-03-30T15:00:53.223217+0000","last_peered":"2024-03-30T15:00:53.223217+0000","last_clean":"2024-03-30T15:00:53.223217+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:00:53.223217+0000","last_undegraded":"2024-03-30T15:00:53.223217+0000","last_fullsized":"2024-03-30T15:00:53.223217+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.774120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223974+0000","last_change":"2024-03-30T14:59:45.812010+0000","last_active":"2024-03-30T15:00:53.223974+0000","last_peered":"2024-03-30T15:00:53.223974+0000","last_clean":"2024-03-30T15:00:53.223974+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:00:53.223974+0000","last_undegraded":"2024-03-30T15:00:53.223974+0000","last_fullsized":"2024-03-30T15:00:53.223974+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:45.811929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:45.811929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.196406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227382+0000","last_change":"2024-03-30T14:57:56.248423+0000","last_active":"2024-03-30T15:00:53.227382+0000","last_peered":"2024-03-30T15:00:53.227382+0000","last_clean":"2024-03-30T15:00:53.227382+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:00:53.227382+0000","last_undegraded":"2024-03-30T15:00:53.227382+0000","last_fullsized":"2024-03-30T15:00:53.227382+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223180+0000","last_change":"2024-03-30T14:57:56.226093+0000","last_active":"2024-03-30T15:00:53.223180+0000","last_peered":"2024-03-30T15:00:53.223180+0000","last_clean":"2024-03-30T15:00:53.223180+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:00:53.223180+0000","last_undegraded":"2024-03-30T15:00:53.223180+0000","last_fullsized":"2024-03-30T15:00:53.223180+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230074+0000","last_change":"2024-03-30T14:57:56.276413+0000","last_active":"2024-03-30T15:00:53.230074+0000","last_peered":"2024-03-30T15:00:53.230074+0000","last_clean":"2024-03-30T15:00:53.230074+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:00:53.230074+0000","last_undegraded":"2024-03-30T15:00:53.230074+0000","last_fullsized":"2024-03-30T15:00:53.230074+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230999+0000","last_change":"2024-03-30T14:57:56.253085+0000","last_active":"2024-03-30T15:00:53.230999+0000","last_peered":"2024-03-30T15:00:53.230999+0000","last_clean":"2024-03-30T15:00:53.230999+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:00:53.230999+0000","last_undegraded":"2024-03-30T15:00:53.230999+0000","last_fullsized":"2024-03-30T15:00:53.230999+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213065+0000","last_change":"2024-03-30T14:59:55.177743+0000","last_active":"2024-03-30T15:00:53.213065+0000","last_peered":"2024-03-30T15:00:53.213065+0000","last_clean":"2024-03-30T15:00:53.213065+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:00:53.213065+0000","last_undegraded":"2024-03-30T15:00:53.213065+0000","last_fullsized":"2024-03-30T15:00:53.213065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.177623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.177623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.770057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":367,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219340+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:00:53.219340+0000","last_peered":"2024-03-30T15:00:53.219340+0000","last_clean":"2024-03-30T15:00:53.219340+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:00:53.219340+0000","last_undegraded":"2024-03-30T15:00:53.219340+0000","last_fullsized":"2024-03-30T15:00:53.219340+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:00.729852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220684+0000","last_change":"2024-03-30T15:00:32.754787+0000","last_active":"2024-03-30T15:00:53.220684+0000","last_peered":"2024-03-30T15:00:53.220684+0000","last_clean":"2024-03-30T15:00:53.220684+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:00:53.220684+0000","last_undegraded":"2024-03-30T15:00:53.220684+0000","last_fullsized":"2024-03-30T15:00:53.220684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.754592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.754592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.958130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223118+0000","last_change":"2024-03-30T14:59:13.804909+0000","last_active":"2024-03-30T15:00:53.223118+0000","last_peered":"2024-03-30T15:00:53.223118+0000","last_clean":"2024-03-30T15:00:53.223118+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:00:53.223118+0000","last_undegraded":"2024-03-30T15:00:53.223118+0000","last_fullsized":"2024-03-30T15:00:53.223118+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.804817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.804817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225717+0000","last_change":"2024-03-30T15:00:33.756912+0000","last_active":"2024-03-30T15:00:53.225717+0000","last_peered":"2024-03-30T15:00:53.225717+0000","last_clean":"2024-03-30T15:00:53.225717+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:00:53.225717+0000","last_undegraded":"2024-03-30T15:00:53.225717+0000","last_fullsized":"2024-03-30T15:00:53.225717+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:33.756810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.756810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:33.994340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214894+0000","last_change":"2024-03-30T15:00:03.114650+0000","last_active":"2024-03-30T15:00:53.214894+0000","last_peered":"2024-03-30T15:00:53.214894+0000","last_clean":"2024-03-30T15:00:53.214894+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:00:53.214894+0000","last_undegraded":"2024-03-30T15:00:53.214894+0000","last_fullsized":"2024-03-30T15:00:53.214894+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.114576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.114576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:10.524652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230581+0000","last_change":"2024-03-30T14:57:56.260622+0000","last_active":"2024-03-30T15:00:53.230581+0000","last_peered":"2024-03-30T15:00:53.230581+0000","last_clean":"2024-03-30T15:00:53.230581+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:00:53.230581+0000","last_undegraded":"2024-03-30T15:00:53.230581+0000","last_fullsized":"2024-03-30T15:00:53.230581+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228687+0000","last_change":"2024-03-30T14:57:56.249378+0000","last_active":"2024-03-30T15:00:53.228687+0000","last_peered":"2024-03-30T15:00:53.228687+0000","last_clean":"2024-03-30T15:00:53.228687+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:00:53.228687+0000","last_undegraded":"2024-03-30T15:00:53.228687+0000","last_fullsized":"2024-03-30T15:00:53.228687+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222979+0000","last_change":"2024-03-30T14:59:11.544255+0000","last_active":"2024-03-30T15:00:53.222979+0000","last_peered":"2024-03-30T15:00:53.222979+0000","last_clean":"2024-03-30T15:00:53.222979+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:00:53.222979+0000","last_undegraded":"2024-03-30T15:00:53.222979+0000","last_fullsized":"2024-03-30T15:00:53.222979+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:11.544174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:11.544174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225029+0000","last_change":"2024-03-30T14:57:56.271774+0000","last_active":"2024-03-30T15:00:53.225029+0000","last_peered":"2024-03-30T15:00:53.225029+0000","last_clean":"2024-03-30T15:00:53.225029+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:00:53.225029+0000","last_undegraded":"2024-03-30T15:00:53.225029+0000","last_fullsized":"2024-03-30T15:00:53.225029+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220213+0000","last_change":"2024-03-30T14:57:56.199752+0000","last_active":"2024-03-30T15:00:53.220213+0000","last_peered":"2024-03-30T15:00:53.220213+0000","last_clean":"2024-03-30T15:00:53.220213+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:00:53.220213+0000","last_undegraded":"2024-03-30T15:00:53.220213+0000","last_fullsized":"2024-03-30T15:00:53.220213+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":359,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222106+0000","last_change":"2024-03-30T14:57:56.228301+0000","last_active":"2024-03-30T15:00:53.222106+0000","last_peered":"2024-03-30T15:00:53.222106+0000","last_clean":"2024-03-30T15:00:53.222106+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:00:53.222106+0000","last_undegraded":"2024-03-30T15:00:53.222106+0000","last_fullsized":"2024-03-30T15:00:53.222106+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222537+0000","last_change":"2024-03-30T14:59:09.501246+0000","last_active":"2024-03-30T15:00:53.222537+0000","last_peered":"2024-03-30T15:00:53.222537+0000","last_clean":"2024-03-30T15:00:53.222537+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:00:53.222537+0000","last_undegraded":"2024-03-30T15:00:53.222537+0000","last_fullsized":"2024-03-30T15:00:53.222537+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.501169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.501169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":478,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223290+0000","last_change":"2024-03-30T14:58:15.280305+0000","last_active":"2024-03-30T15:00:53.223290+0000","last_peered":"2024-03-30T15:00:53.223290+0000","last_clean":"2024-03-30T15:00:53.223290+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:00:53.223290+0000","last_undegraded":"2024-03-30T15:00:53.223290+0000","last_fullsized":"2024-03-30T15:00:53.223290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:15.280217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.280217+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220333+0000","last_change":"2024-03-30T14:58:13.189214+0000","last_active":"2024-03-30T15:00:53.220333+0000","last_peered":"2024-03-30T15:00:53.220333+0000","last_clean":"2024-03-30T15:00:53.220333+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:00:53.220333+0000","last_undegraded":"2024-03-30T15:00:53.220333+0000","last_fullsized":"2024-03-30T15:00:53.220333+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.189081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.189081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":550,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224195+0000","last_change":"2024-03-30T14:58:31.806764+0000","last_active":"2024-03-30T15:00:53.224195+0000","last_peered":"2024-03-30T15:00:53.224195+0000","last_clean":"2024-03-30T15:00:53.224195+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:00:53.224195+0000","last_undegraded":"2024-03-30T15:00:53.224195+0000","last_fullsized":"2024-03-30T15:00:53.224195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:31.806606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.806606+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230648+0000","last_change":"2024-03-30T14:59:13.422572+0000","last_active":"2024-03-30T15:00:53.230648+0000","last_peered":"2024-03-30T15:00:53.230648+0000","last_clean":"2024-03-30T15:00:53.230648+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:53.230648+0000","last_undegraded":"2024-03-30T15:00:53.230648+0000","last_fullsized":"2024-03-30T15:00:53.230648+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.422453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.422453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":29,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218728+0000","last_change":"2024-03-30T15:00:45.239550+0000","last_active":"2024-03-30T15:00:53.218728+0000","last_peered":"2024-03-30T15:00:53.218728+0000","last_clean":"2024-03-30T15:00:53.218728+0000","last_became_active":"2024-03-30T15:00:45.238835+0000","last_became_peered":"2024-03-30T15:00:45.238835+0000","last_unstale":"2024-03-30T15:00:53.218728+0000","last_undegraded":"2024-03-30T15:00:53.218728+0000","last_fullsized":"2024-03-30T15:00:53.218728+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.045488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.045488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:09.864216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225892+0000","last_change":"2024-03-30T15:00:04.553363+0000","last_active":"2024-03-30T15:00:53.225892+0000","last_peered":"2024-03-30T15:00:53.225892+0000","last_clean":"2024-03-30T15:00:53.225892+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:00:53.225892+0000","last_undegraded":"2024-03-30T15:00:53.225892+0000","last_fullsized":"2024-03-30T15:00:53.225892+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.553281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.553281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:06.976142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":499,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223971+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:00:53.223971+0000","last_peered":"2024-03-30T15:00:53.223971+0000","last_clean":"2024-03-30T15:00:53.223971+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:00:53.223971+0000","last_undegraded":"2024-03-30T15:00:53.223971+0000","last_fullsized":"2024-03-30T15:00:53.223971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:08.635027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":475,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221601+0000","last_change":"2024-03-30T14:59:18.646158+0000","last_active":"2024-03-30T15:00:53.221601+0000","last_peered":"2024-03-30T15:00:53.221601+0000","last_clean":"2024-03-30T15:00:53.221601+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:00:53.221601+0000","last_undegraded":"2024-03-30T15:00:53.221601+0000","last_fullsized":"2024-03-30T15:00:53.221601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:18.646076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:18.646076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":468,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217317+0000","last_change":"2024-03-30T14:59:09.325167+0000","last_active":"2024-03-30T15:00:53.217317+0000","last_peered":"2024-03-30T15:00:53.217317+0000","last_clean":"2024-03-30T15:00:53.217317+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:00:53.217317+0000","last_undegraded":"2024-03-30T15:00:53.217317+0000","last_fullsized":"2024-03-30T15:00:53.217317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":462,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224403+0000","last_change":"2024-03-30T14:59:07.831752+0000","last_active":"2024-03-30T15:00:53.224403+0000","last_peered":"2024-03-30T15:00:53.224403+0000","last_clean":"2024-03-30T15:00:53.224403+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:00:53.224403+0000","last_undegraded":"2024-03-30T15:00:53.224403+0000","last_fullsized":"2024-03-30T15:00:53.224403+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.831675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.831675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.18","version":"115'2","reported_seq":51,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226209+0000","last_change":"2024-03-30T15:00:45.254965+0000","last_active":"2024-03-30T15:00:53.226209+0000","last_peered":"2024-03-30T15:00:53.226209+0000","last_clean":"2024-03-30T15:00:53.226209+0000","last_became_active":"2024-03-30T15:00:45.247625+0000","last_became_peered":"2024-03-30T15:00:45.247625+0000","last_unstale":"2024-03-30T15:00:53.226209+0000","last_undegraded":"2024-03-30T15:00:53.226209+0000","last_fullsized":"2024-03-30T15:00:53.226209+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217512+0000","last_change":"2024-03-30T14:59:54.144187+0000","last_active":"2024-03-30T15:00:53.217512+0000","last_peered":"2024-03-30T15:00:53.217512+0000","last_clean":"2024-03-30T15:00:53.217512+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:00:53.217512+0000","last_undegraded":"2024-03-30T15:00:53.217512+0000","last_fullsized":"2024-03-30T15:00:53.217512+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:54.144084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:54.144084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:04.029603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":487,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230557+0000","last_change":"2024-03-30T14:58:20.234367+0000","last_active":"2024-03-30T15:00:53.230557+0000","last_peered":"2024-03-30T15:00:53.230557+0000","last_clean":"2024-03-30T15:00:53.230557+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:00:53.230557+0000","last_undegraded":"2024-03-30T15:00:53.230557+0000","last_fullsized":"2024-03-30T15:00:53.230557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:58:20.234281+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.234281+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743514+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226526+0000","last_change":"2024-03-30T14:57:29.596356+0000","last_active":"2024-03-30T15:00:53.226526+0000","last_peered":"2024-03-30T15:00:53.226526+0000","last_clean":"2024-03-30T15:00:53.226526+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:00:53.226526+0000","last_undegraded":"2024-03-30T15:00:53.226526+0000","last_fullsized":"2024-03-30T15:00:53.226526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:29.596249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:29.596249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":458,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231477+0000","last_change":"2024-03-30T14:58:23.099186+0000","last_active":"2024-03-30T15:00:53.231477+0000","last_peered":"2024-03-30T15:00:53.231477+0000","last_clean":"2024-03-30T15:00:53.231477+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:00:53.231477+0000","last_undegraded":"2024-03-30T15:00:53.231477+0000","last_fullsized":"2024-03-30T15:00:53.231477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.099104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.099104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":377,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222682+0000","last_change":"2024-03-30T15:00:44.895646+0000","last_active":"2024-03-30T15:00:53.222682+0000","last_peered":"2024-03-30T15:00:53.222682+0000","last_clean":"2024-03-30T15:00:53.222682+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:00:53.222682+0000","last_undegraded":"2024-03-30T15:00:53.222682+0000","last_fullsized":"2024-03-30T15:00:53.222682+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.895504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.895504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.201212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1b","version":"115'10","reported_seq":398,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227045+0000","last_change":"2024-03-30T15:00:45.262813+0000","last_active":"2024-03-30T15:00:53.227045+0000","last_peered":"2024-03-30T15:00:53.227045+0000","last_clean":"2024-03-30T15:00:53.227045+0000","last_became_active":"2024-03-30T15:00:45.243932+0000","last_became_peered":"2024-03-30T15:00:45.243932+0000","last_unstale":"2024-03-30T15:00:53.227045+0000","last_undegraded":"2024-03-30T15:00:53.227045+0000","last_fullsized":"2024-03-30T15:00:53.227045+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:38.719347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:38.719347+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:08.349262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214109+0000","last_change":"2024-03-30T15:00:01.176797+0000","last_active":"2024-03-30T15:00:53.214109+0000","last_peered":"2024-03-30T15:00:53.214109+0000","last_clean":"2024-03-30T15:00:53.214109+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:00:53.214109+0000","last_undegraded":"2024-03-30T15:00:53.214109+0000","last_fullsized":"2024-03-30T15:00:53.214109+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.176673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.080692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":496,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214205+0000","last_change":"2024-03-30T14:58:31.256279+0000","last_active":"2024-03-30T15:00:53.214205+0000","last_peered":"2024-03-30T15:00:53.214205+0000","last_clean":"2024-03-30T15:00:53.214205+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:00:53.214205+0000","last_undegraded":"2024-03-30T15:00:53.214205+0000","last_fullsized":"2024-03-30T15:00:53.214205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T14:58:31.256139+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.256139+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213728+0000","last_change":"2024-03-30T14:59:07.286916+0000","last_active":"2024-03-30T15:00:53.213728+0000","last_peered":"2024-03-30T15:00:53.213728+0000","last_clean":"2024-03-30T15:00:53.213728+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:00:53.213728+0000","last_undegraded":"2024-03-30T15:00:53.213728+0000","last_fullsized":"2024-03-30T15:00:53.213728+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.286790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.286790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":543,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230577+0000","last_change":"2024-03-30T14:58:59.314094+0000","last_active":"2024-03-30T15:00:53.230577+0000","last_peered":"2024-03-30T15:00:53.230577+0000","last_clean":"2024-03-30T15:00:53.230577+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:00:53.230577+0000","last_undegraded":"2024-03-30T15:00:53.230577+0000","last_fullsized":"2024-03-30T15:00:53.230577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.314003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.314003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":454,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227568+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:00:53.227568+0000","last_peered":"2024-03-30T15:00:53.227568+0000","last_clean":"2024-03-30T15:00:53.227568+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:00:53.227568+0000","last_undegraded":"2024-03-30T15:00:53.227568+0000","last_fullsized":"2024-03-30T15:00:53.227568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1a","version":"115'5","reported_seq":42,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226708+0000","last_change":"2024-03-30T15:00:47.055900+0000","last_active":"2024-03-30T15:00:53.226708+0000","last_peered":"2024-03-30T15:00:53.226708+0000","last_clean":"2024-03-30T15:00:53.226708+0000","last_became_active":"2024-03-30T15:00:45.243189+0000","last_became_peered":"2024-03-30T15:00:45.243189+0000","last_unstale":"2024-03-30T15:00:53.226708+0000","last_undegraded":"2024-03-30T15:00:53.226708+0000","last_fullsized":"2024-03-30T15:00:53.226708+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:47.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.055803+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.569315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223916+0000","last_change":"2024-03-30T14:59:53.801443+0000","last_active":"2024-03-30T15:00:53.223916+0000","last_peered":"2024-03-30T15:00:53.223916+0000","last_clean":"2024-03-30T15:00:53.223916+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:00:53.223916+0000","last_undegraded":"2024-03-30T15:00:53.223916+0000","last_fullsized":"2024-03-30T15:00:53.223916+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.801332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.801332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.086851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":460,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226971+0000","last_change":"2024-03-30T14:58:21.250924+0000","last_active":"2024-03-30T15:00:53.226971+0000","last_peered":"2024-03-30T15:00:53.226971+0000","last_clean":"2024-03-30T15:00:53.226971+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:00:53.226971+0000","last_undegraded":"2024-03-30T15:00:53.226971+0000","last_fullsized":"2024-03-30T15:00:53.226971+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:21.250827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.250827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":458,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226111+0000","last_change":"2024-03-30T14:58:19.963766+0000","last_active":"2024-03-30T15:00:53.226111+0000","last_peered":"2024-03-30T15:00:53.226111+0000","last_clean":"2024-03-30T15:00:53.226111+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:00:53.226111+0000","last_undegraded":"2024-03-30T15:00:53.226111+0000","last_fullsized":"2024-03-30T15:00:53.226111+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:58:19.963665+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.963665+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":442,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230112+0000","last_change":"2024-03-30T14:59:10.014828+0000","last_active":"2024-03-30T15:00:53.230112+0000","last_peered":"2024-03-30T15:00:53.230112+0000","last_clean":"2024-03-30T15:00:53.230112+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:00:53.230112+0000","last_undegraded":"2024-03-30T15:00:53.230112+0000","last_fullsized":"2024-03-30T15:00:53.230112+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:10.014745+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.014745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224168+0000","last_change":"2024-03-30T14:59:02.261089+0000","last_active":"2024-03-30T15:00:53.224168+0000","last_peered":"2024-03-30T15:00:53.224168+0000","last_clean":"2024-03-30T15:00:53.224168+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:00:53.224168+0000","last_undegraded":"2024-03-30T15:00:53.224168+0000","last_fullsized":"2024-03-30T15:00:53.224168+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.260964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.260964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":29,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225428+0000","last_change":"2024-03-30T15:00:45.252550+0000","last_active":"2024-03-30T15:00:53.225428+0000","last_peered":"2024-03-30T15:00:53.225428+0000","last_clean":"2024-03-30T15:00:53.225428+0000","last_became_active":"2024-03-30T15:00:45.251660+0000","last_became_peered":"2024-03-30T15:00:45.251660+0000","last_unstale":"2024-03-30T15:00:53.225428+0000","last_undegraded":"2024-03-30T15:00:53.225428+0000","last_fullsized":"2024-03-30T15:00:53.225428+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.119871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.119871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:59.305863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":375,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222999+0000","last_change":"2024-03-30T14:59:48.794962+0000","last_active":"2024-03-30T15:00:53.222999+0000","last_peered":"2024-03-30T15:00:53.222999+0000","last_clean":"2024-03-30T15:00:53.222999+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:00:53.222999+0000","last_undegraded":"2024-03-30T15:00:53.222999+0000","last_fullsized":"2024-03-30T15:00:53.222999+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:48.794824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.794824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":430,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213090+0000","last_change":"2024-03-30T14:59:49.053482+0000","last_active":"2024-03-30T15:00:53.213090+0000","last_peered":"2024-03-30T15:00:53.213090+0000","last_clean":"2024-03-30T15:00:53.213090+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:00:53.213090+0000","last_undegraded":"2024-03-30T15:00:53.213090+0000","last_fullsized":"2024-03-30T15:00:53.213090+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:49.053394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:49.053394+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.339042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":493,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226728+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:00:53.226728+0000","last_peered":"2024-03-30T15:00:53.226728+0000","last_clean":"2024-03-30T15:00:53.226728+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:00:53.226728+0000","last_undegraded":"2024-03-30T15:00:53.226728+0000","last_fullsized":"2024-03-30T15:00:53.226728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.455990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":470,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221586+0000","last_change":"2024-03-30T14:58:25.170349+0000","last_active":"2024-03-30T15:00:53.221586+0000","last_peered":"2024-03-30T15:00:53.221586+0000","last_clean":"2024-03-30T15:00:53.221586+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:00:53.221586+0000","last_undegraded":"2024-03-30T15:00:53.221586+0000","last_fullsized":"2024-03-30T15:00:53.221586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.170239+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":463,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228240+0000","last_change":"2024-03-30T14:59:56.291563+0000","last_active":"2024-03-30T15:00:53.228240+0000","last_peered":"2024-03-30T15:00:53.228240+0000","last_clean":"2024-03-30T15:00:53.228240+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:00:53.228240+0000","last_undegraded":"2024-03-30T15:00:53.228240+0000","last_fullsized":"2024-03-30T15:00:53.228240+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.291453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.291453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.028131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"115'2","reported_seq":33,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217789+0000","last_change":"2024-03-30T15:00:45.251189+0000","last_active":"2024-03-30T15:00:53.217789+0000","last_peered":"2024-03-30T15:00:53.217789+0000","last_clean":"2024-03-30T15:00:53.217789+0000","last_became_active":"2024-03-30T15:00:45.241122+0000","last_became_peered":"2024-03-30T15:00:45.241122+0000","last_unstale":"2024-03-30T15:00:53.217789+0000","last_undegraded":"2024-03-30T15:00:53.217789+0000","last_fullsized":"2024-03-30T15:00:53.217789+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":375,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223165+0000","last_change":"2024-03-30T14:59:46.819021+0000","last_active":"2024-03-30T15:00:53.223165+0000","last_peered":"2024-03-30T15:00:53.223165+0000","last_clean":"2024-03-30T15:00:53.223165+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:00:53.223165+0000","last_undegraded":"2024-03-30T15:00:53.223165+0000","last_fullsized":"2024-03-30T15:00:53.223165+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:46.818941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.818941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.701804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":552,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220239+0000","last_change":"2024-03-30T15:00:30.490223+0000","last_active":"2024-03-30T15:00:53.220239+0000","last_peered":"2024-03-30T15:00:53.220239+0000","last_clean":"2024-03-30T15:00:53.220239+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:00:53.220239+0000","last_undegraded":"2024-03-30T15:00:53.220239+0000","last_fullsized":"2024-03-30T15:00:53.220239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:00:30.490104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.490104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.592628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":493,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218429+0000","last_change":"2024-03-30T14:59:10.283945+0000","last_active":"2024-03-30T15:00:53.218429+0000","last_peered":"2024-03-30T15:00:53.218429+0000","last_clean":"2024-03-30T15:00:53.218429+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:00:53.218429+0000","last_undegraded":"2024-03-30T15:00:53.218429+0000","last_fullsized":"2024-03-30T15:00:53.218429+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.283859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":474,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227668+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:00:53.227668+0000","last_peered":"2024-03-30T15:00:53.227668+0000","last_clean":"2024-03-30T15:00:53.227668+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:00:53.227668+0000","last_undegraded":"2024-03-30T15:00:53.227668+0000","last_fullsized":"2024-03-30T15:00:53.227668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251711+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":457,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218414+0000","last_change":"2024-03-30T14:58:23.157209+0000","last_active":"2024-03-30T15:00:53.218414+0000","last_peered":"2024-03-30T15:00:53.218414+0000","last_clean":"2024-03-30T15:00:53.218414+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:00:53.218414+0000","last_undegraded":"2024-03-30T15:00:53.218414+0000","last_fullsized":"2024-03-30T15:00:53.218414+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.157095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.157095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572753+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"115'5","reported_seq":52,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223491+0000","last_change":"2024-03-30T15:00:45.253584+0000","last_active":"2024-03-30T15:00:53.223491+0000","last_peered":"2024-03-30T15:00:53.223491+0000","last_clean":"2024-03-30T15:00:53.223491+0000","last_became_active":"2024-03-30T15:00:45.243133+0000","last_became_peered":"2024-03-30T15:00:45.243133+0000","last_unstale":"2024-03-30T15:00:53.223491+0000","last_undegraded":"2024-03-30T15:00:53.223491+0000","last_fullsized":"2024-03-30T15:00:53.223491+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T14:59:15.591612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.591612+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223926+0000","last_change":"2024-03-30T14:59:19.788455+0000","last_active":"2024-03-30T15:00:53.223926+0000","last_peered":"2024-03-30T15:00:53.223926+0000","last_clean":"2024-03-30T15:00:53.223926+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:00:53.223926+0000","last_undegraded":"2024-03-30T15:00:53.223926+0000","last_fullsized":"2024-03-30T15:00:53.223926+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.788329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.788329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221958+0000","last_change":"2024-03-30T14:59:01.518493+0000","last_active":"2024-03-30T15:00:53.221958+0000","last_peered":"2024-03-30T15:00:53.221958+0000","last_clean":"2024-03-30T15:00:53.221958+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:00:53.221958+0000","last_undegraded":"2024-03-30T15:00:53.221958+0000","last_fullsized":"2024-03-30T15:00:53.221958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.518414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.518414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":509,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220852+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:00:53.220852+0000","last_peered":"2024-03-30T15:00:53.220852+0000","last_clean":"2024-03-30T15:00:53.220852+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:00:53.220852+0000","last_undegraded":"2024-03-30T15:00:53.220852+0000","last_fullsized":"2024-03-30T15:00:53.220852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217157+0000","last_change":"2024-03-30T14:59:13.210437+0000","last_active":"2024-03-30T15:00:53.217157+0000","last_peered":"2024-03-30T15:00:53.217157+0000","last_clean":"2024-03-30T15:00:53.217157+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:00:53.217157+0000","last_undegraded":"2024-03-30T15:00:53.217157+0000","last_fullsized":"2024-03-30T15:00:53.217157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.210302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.210302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":464,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213449+0000","last_change":"2024-03-30T14:59:19.135083+0000","last_active":"2024-03-30T15:00:53.213449+0000","last_peered":"2024-03-30T15:00:53.213449+0000","last_clean":"2024-03-30T15:00:53.213449+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:00:53.213449+0000","last_undegraded":"2024-03-30T15:00:53.213449+0000","last_fullsized":"2024-03-30T15:00:53.213449+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.135006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.135006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"115'5","reported_seq":34,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224471+0000","last_change":"2024-03-30T15:00:45.258419+0000","last_active":"2024-03-30T15:00:53.224471+0000","last_peered":"2024-03-30T15:00:53.224471+0000","last_clean":"2024-03-30T15:00:53.224471+0000","last_became_active":"2024-03-30T15:00:45.244278+0000","last_became_peered":"2024-03-30T15:00:45.244278+0000","last_unstale":"2024-03-30T15:00:53.224471+0000","last_undegraded":"2024-03-30T15:00:53.224471+0000","last_fullsized":"2024-03-30T15:00:53.224471+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.109233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.109233+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:55.006862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":366,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221445+0000","last_change":"2024-03-30T14:57:54.026635+0000","last_active":"2024-03-30T15:00:53.221445+0000","last_peered":"2024-03-30T15:00:53.221445+0000","last_clean":"2024-03-30T15:00:53.221445+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:00:53.221445+0000","last_undegraded":"2024-03-30T15:00:53.221445+0000","last_fullsized":"2024-03-30T15:00:53.221445+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231482+0000","last_change":"2024-03-30T14:59:15.207502+0000","last_active":"2024-03-30T15:00:53.231482+0000","last_peered":"2024-03-30T15:00:53.231482+0000","last_clean":"2024-03-30T15:00:53.231482+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:00:53.231482+0000","last_undegraded":"2024-03-30T15:00:53.231482+0000","last_fullsized":"2024-03-30T15:00:53.231482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.207413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.207413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":490,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221350+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:00:53.221350+0000","last_peered":"2024-03-30T15:00:53.221350+0000","last_clean":"2024-03-30T15:00:53.221350+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:00:53.221350+0000","last_undegraded":"2024-03-30T15:00:53.221350+0000","last_fullsized":"2024-03-30T15:00:53.221350+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:59.702850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":849,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225729+0000","last_change":"2024-03-30T14:58:21.947917+0000","last_active":"2024-03-30T15:00:53.225729+0000","last_peered":"2024-03-30T15:00:53.225729+0000","last_clean":"2024-03-30T15:00:53.225729+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:00:53.225729+0000","last_undegraded":"2024-03-30T15:00:53.225729+0000","last_fullsized":"2024-03-30T15:00:53.225729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:58:21.947837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.947837+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":469,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219056+0000","last_change":"2024-03-30T14:59:05.612363+0000","last_active":"2024-03-30T15:00:53.219056+0000","last_peered":"2024-03-30T15:00:53.219056+0000","last_clean":"2024-03-30T15:00:53.219056+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:00:53.219056+0000","last_undegraded":"2024-03-30T15:00:53.219056+0000","last_fullsized":"2024-03-30T15:00:53.219056+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.612262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.612262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":456,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.215146+0000","last_change":"2024-03-30T14:58:20.292759+0000","last_active":"2024-03-30T15:00:53.215146+0000","last_peered":"2024-03-30T15:00:53.215146+0000","last_clean":"2024-03-30T15:00:53.215146+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:00:53.215146+0000","last_undegraded":"2024-03-30T15:00:53.215146+0000","last_fullsized":"2024-03-30T15:00:53.215146+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:20.292655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.292655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"115'9","reported_seq":44,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219997+0000","last_change":"2024-03-30T15:00:45.264420+0000","last_active":"2024-03-30T15:00:53.219997+0000","last_peered":"2024-03-30T15:00:53.219997+0000","last_clean":"2024-03-30T15:00:53.219997+0000","last_became_active":"2024-03-30T15:00:45.242541+0000","last_became_peered":"2024-03-30T15:00:45.242541+0000","last_unstale":"2024-03-30T15:00:53.219997+0000","last_undegraded":"2024-03-30T15:00:53.219997+0000","last_fullsized":"2024-03-30T15:00:53.219997+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":364,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.229000+0000","last_change":"2024-03-30T14:57:55.140517+0000","last_active":"2024-03-30T15:00:53.229000+0000","last_peered":"2024-03-30T15:00:53.229000+0000","last_clean":"2024-03-30T15:00:53.229000+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:00:53.229000+0000","last_undegraded":"2024-03-30T15:00:53.229000+0000","last_fullsized":"2024-03-30T15:00:53.229000+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":459,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228758+0000","last_change":"2024-03-30T14:58:35.074272+0000","last_active":"2024-03-30T15:00:53.228758+0000","last_peered":"2024-03-30T15:00:53.228758+0000","last_clean":"2024-03-30T15:00:53.228758+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:00:53.228758+0000","last_undegraded":"2024-03-30T15:00:53.228758+0000","last_fullsized":"2024-03-30T15:00:53.228758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:35.074185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.074185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":577,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226438+0000","last_change":"2024-03-30T15:00:39.833443+0000","last_active":"2024-03-30T15:00:53.226438+0000","last_peered":"2024-03-30T15:00:53.226438+0000","last_clean":"2024-03-30T15:00:53.226438+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:00:53.226438+0000","last_undegraded":"2024-03-30T15:00:53.226438+0000","last_fullsized":"2024-03-30T15:00:53.226438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:00:39.833329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:39.833329+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.420557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222271+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:00:53.222271+0000","last_peered":"2024-03-30T15:00:53.222271+0000","last_clean":"2024-03-30T15:00:53.222271+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:00:53.222271+0000","last_undegraded":"2024-03-30T15:00:53.222271+0000","last_fullsized":"2024-03-30T15:00:53.222271+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":726,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225891+0000","last_change":"2024-03-30T14:59:14.976524+0000","last_active":"2024-03-30T15:00:53.225891+0000","last_peered":"2024-03-30T15:00:53.225891+0000","last_clean":"2024-03-30T15:00:53.225891+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:00:53.225891+0000","last_undegraded":"2024-03-30T15:00:53.225891+0000","last_fullsized":"2024-03-30T15:00:53.225891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T14:59:14.976438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.976438+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":454,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231244+0000","last_change":"2024-03-30T14:59:05.166302+0000","last_active":"2024-03-30T15:00:53.231244+0000","last_peered":"2024-03-30T15:00:53.231244+0000","last_clean":"2024-03-30T15:00:53.231244+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:00:53.231244+0000","last_undegraded":"2024-03-30T15:00:53.231244+0000","last_fullsized":"2024-03-30T15:00:53.231244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.166200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.166200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225804+0000","last_change":"2024-03-30T15:00:45.242039+0000","last_active":"2024-03-30T15:00:53.225804+0000","last_peered":"2024-03-30T15:00:53.225804+0000","last_clean":"2024-03-30T15:00:53.225804+0000","last_became_active":"2024-03-30T15:00:45.240866+0000","last_became_peered":"2024-03-30T15:00:45.240866+0000","last_unstale":"2024-03-30T15:00:53.225804+0000","last_undegraded":"2024-03-30T15:00:53.225804+0000","last_fullsized":"2024-03-30T15:00:53.225804+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225335+0000","last_change":"2024-03-30T14:59:53.656204+0000","last_active":"2024-03-30T15:00:53.225335+0000","last_peered":"2024-03-30T15:00:53.225335+0000","last_clean":"2024-03-30T15:00:53.225335+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:00:53.225335+0000","last_undegraded":"2024-03-30T15:00:53.225335+0000","last_fullsized":"2024-03-30T15:00:53.225335+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.656111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.656111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:54.708968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224240+0000","last_change":"2024-03-30T14:59:03.841069+0000","last_active":"2024-03-30T15:00:53.224240+0000","last_peered":"2024-03-30T15:00:53.224240+0000","last_clean":"2024-03-30T15:00:53.224240+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:00:53.224240+0000","last_undegraded":"2024-03-30T15:00:53.224240+0000","last_fullsized":"2024-03-30T15:00:53.224240+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.840945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.840945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":504,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213753+0000","last_change":"2024-03-30T14:59:51.039814+0000","last_active":"2024-03-30T15:00:53.213753+0000","last_peered":"2024-03-30T15:00:53.213753+0000","last_clean":"2024-03-30T15:00:53.213753+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:00:53.213753+0000","last_undegraded":"2024-03-30T15:00:53.213753+0000","last_fullsized":"2024-03-30T15:00:53.213753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T14:59:51.039715+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.039715+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.286012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":484,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223274+0000","last_change":"2024-03-30T14:58:58.777774+0000","last_active":"2024-03-30T15:00:53.223274+0000","last_peered":"2024-03-30T15:00:53.223274+0000","last_clean":"2024-03-30T15:00:53.223274+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:00:53.223274+0000","last_undegraded":"2024-03-30T15:00:53.223274+0000","last_fullsized":"2024-03-30T15:00:53.223274+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.777653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213252+0000","last_change":"2024-03-30T14:59:22.144499+0000","last_active":"2024-03-30T15:00:53.213252+0000","last_peered":"2024-03-30T15:00:53.213252+0000","last_clean":"2024-03-30T15:00:53.213252+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:00:53.213252+0000","last_undegraded":"2024-03-30T15:00:53.213252+0000","last_fullsized":"2024-03-30T15:00:53.213252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:22.144389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:59:22.144389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":463,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.229366+0000","last_change":"2024-03-30T14:59:55.270674+0000","last_active":"2024-03-30T15:00:53.229366+0000","last_peered":"2024-03-30T15:00:53.229366+0000","last_clean":"2024-03-30T15:00:53.229366+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:00:53.229366+0000","last_undegraded":"2024-03-30T15:00:53.229366+0000","last_fullsized":"2024-03-30T15:00:53.229366+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.270595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.270595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.865117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":29,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223113+0000","last_change":"2024-03-30T15:00:45.248453+0000","last_active":"2024-03-30T15:00:53.223113+0000","last_peered":"2024-03-30T15:00:53.223113+0000","last_clean":"2024-03-30T15:00:53.223113+0000","last_became_active":"2024-03-30T15:00:45.248115+0000","last_became_peered":"2024-03-30T15:00:45.248115+0000","last_unstale":"2024-03-30T15:00:53.223113+0000","last_undegraded":"2024-03-30T15:00:53.223113+0000","last_fullsized":"2024-03-30T15:00:53.223113+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.031690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.031690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:22.634118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":366,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218523+0000","last_change":"2024-03-30T14:57:54.024875+0000","last_active":"2024-03-30T15:00:53.218523+0000","last_peered":"2024-03-30T15:00:53.218523+0000","last_clean":"2024-03-30T15:00:53.218523+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:00:53.218523+0000","last_undegraded":"2024-03-30T15:00:53.218523+0000","last_fullsized":"2024-03-30T15:00:53.218523+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":468,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220423+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:00:53.220423+0000","last_peered":"2024-03-30T15:00:53.220423+0000","last_clean":"2024-03-30T15:00:53.220423+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:00:53.220423+0000","last_undegraded":"2024-03-30T15:00:53.220423+0000","last_fullsized":"2024-03-30T15:00:53.220423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.197637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":541,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221624+0000","last_change":"2024-03-30T14:58:22.340066+0000","last_active":"2024-03-30T15:00:53.221624+0000","last_peered":"2024-03-30T15:00:53.221624+0000","last_clean":"2024-03-30T15:00:53.221624+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:00:53.221624+0000","last_undegraded":"2024-03-30T15:00:53.221624+0000","last_fullsized":"2024-03-30T15:00:53.221624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:58:22.339962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.339962+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251525+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":468,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220528+0000","last_change":"2024-03-30T14:58:19.237610+0000","last_active":"2024-03-30T15:00:53.220528+0000","last_peered":"2024-03-30T15:00:53.220528+0000","last_clean":"2024-03-30T15:00:53.220528+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:00:53.220528+0000","last_undegraded":"2024-03-30T15:00:53.220528+0000","last_fullsized":"2024-03-30T15:00:53.220528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.237530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.237530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":478,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222474+0000","last_change":"2024-03-30T14:58:36.096934+0000","last_active":"2024-03-30T15:00:53.222474+0000","last_peered":"2024-03-30T15:00:53.222474+0000","last_clean":"2024-03-30T15:00:53.222474+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:00:53.222474+0000","last_undegraded":"2024-03-30T15:00:53.222474+0000","last_fullsized":"2024-03-30T15:00:53.222474+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T14:58:36.096812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.096812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230360+0000","last_change":"2024-03-30T14:58:22.249707+0000","last_active":"2024-03-30T15:00:53.230360+0000","last_peered":"2024-03-30T15:00:53.230360+0000","last_clean":"2024-03-30T15:00:53.230360+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:00:53.230360+0000","last_undegraded":"2024-03-30T15:00:53.230360+0000","last_fullsized":"2024-03-30T15:00:53.230360+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.249628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.249628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743660+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.11","version":"115'7","reported_seq":35,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218724+0000","last_change":"2024-03-30T15:00:45.250504+0000","last_active":"2024-03-30T15:00:53.218724+0000","last_peered":"2024-03-30T15:00:53.218724+0000","last_clean":"2024-03-30T15:00:53.218724+0000","last_became_active":"2024-03-30T15:00:45.239363+0000","last_became_peered":"2024-03-30T15:00:45.239363+0000","last_unstale":"2024-03-30T15:00:53.218724+0000","last_undegraded":"2024-03-30T15:00:53.218724+0000","last_fullsized":"2024-03-30T15:00:53.218724+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":366,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220515+0000","last_change":"2024-03-30T14:57:54.046311+0000","last_active":"2024-03-30T15:00:53.220515+0000","last_peered":"2024-03-30T15:00:53.220515+0000","last_clean":"2024-03-30T15:00:53.220515+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:00:53.220515+0000","last_undegraded":"2024-03-30T15:00:53.220515+0000","last_fullsized":"2024-03-30T15:00:53.220515+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":488,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220425+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:00:53.220425+0000","last_peered":"2024-03-30T15:00:53.220425+0000","last_clean":"2024-03-30T15:00:53.220425+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:00:53.220425+0000","last_undegraded":"2024-03-30T15:00:53.220425+0000","last_fullsized":"2024-03-30T15:00:53.220425+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:39.060252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":474,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222500+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:00:53.222500+0000","last_peered":"2024-03-30T15:00:53.222500+0000","last_clean":"2024-03-30T15:00:53.222500+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:00:53.222500+0000","last_undegraded":"2024-03-30T15:00:53.222500+0000","last_fullsized":"2024-03-30T15:00:53.222500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:02.989386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223269+0000","last_change":"2024-03-30T14:58:30.843293+0000","last_active":"2024-03-30T15:00:53.223269+0000","last_peered":"2024-03-30T15:00:53.223269+0000","last_clean":"2024-03-30T15:00:53.223269+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:00:53.223269+0000","last_undegraded":"2024-03-30T15:00:53.223269+0000","last_fullsized":"2024-03-30T15:00:53.223269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:30.843163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:30.843163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704713+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":456,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219243+0000","last_change":"2024-03-30T14:58:36.234502+0000","last_active":"2024-03-30T15:00:53.219243+0000","last_peered":"2024-03-30T15:00:53.219243+0000","last_clean":"2024-03-30T15:00:53.219243+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:00:53.219243+0000","last_undegraded":"2024-03-30T15:00:53.219243+0000","last_fullsized":"2024-03-30T15:00:53.219243+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:36.234425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.234425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":364,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227464+0000","last_change":"2024-03-30T14:57:55.074016+0000","last_active":"2024-03-30T15:00:53.227464+0000","last_peered":"2024-03-30T15:00:53.227464+0000","last_clean":"2024-03-30T15:00:53.227464+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:00:53.227464+0000","last_undegraded":"2024-03-30T15:00:53.227464+0000","last_fullsized":"2024-03-30T15:00:53.227464+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":528,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221859+0000","last_change":"2024-03-30T14:59:00.517828+0000","last_active":"2024-03-30T15:00:53.221859+0000","last_peered":"2024-03-30T15:00:53.221859+0000","last_clean":"2024-03-30T15:00:53.221859+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:00:53.221859+0000","last_undegraded":"2024-03-30T15:00:53.221859+0000","last_fullsized":"2024-03-30T15:00:53.221859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:59:00.517731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:00.517731+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.229144+0000","last_change":"2024-03-30T14:58:23.951589+0000","last_active":"2024-03-30T15:00:53.229144+0000","last_peered":"2024-03-30T15:00:53.229144+0000","last_clean":"2024-03-30T15:00:53.229144+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:00:53.229144+0000","last_undegraded":"2024-03-30T15:00:53.229144+0000","last_fullsized":"2024-03-30T15:00:53.229144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.951507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223807+0000","last_change":"2024-03-30T14:58:33.889037+0000","last_active":"2024-03-30T15:00:53.223807+0000","last_peered":"2024-03-30T15:00:53.223807+0000","last_clean":"2024-03-30T15:00:53.223807+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:00:53.223807+0000","last_undegraded":"2024-03-30T15:00:53.223807+0000","last_fullsized":"2024-03-30T15:00:53.223807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:33.888931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:33.888931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225531+0000","last_change":"2024-03-30T14:59:12.985700+0000","last_active":"2024-03-30T15:00:53.225531+0000","last_peered":"2024-03-30T15:00:53.225531+0000","last_clean":"2024-03-30T15:00:53.225531+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:00:53.225531+0000","last_undegraded":"2024-03-30T15:00:53.225531+0000","last_fullsized":"2024-03-30T15:00:53.225531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.985620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.985620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":386,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223065+0000","last_change":"2024-03-30T14:59:51.656251+0000","last_active":"2024-03-30T15:00:53.223065+0000","last_peered":"2024-03-30T15:00:53.223065+0000","last_clean":"2024-03-30T15:00:53.223065+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:00:53.223065+0000","last_undegraded":"2024-03-30T15:00:53.223065+0000","last_fullsized":"2024-03-30T15:00:53.223065+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T14:59:51.656163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.656163+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.934763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223489+0000","last_change":"2024-03-30T14:59:08.794368+0000","last_active":"2024-03-30T15:00:53.223489+0000","last_peered":"2024-03-30T15:00:53.223489+0000","last_clean":"2024-03-30T15:00:53.223489+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:00:53.223489+0000","last_undegraded":"2024-03-30T15:00:53.223489+0000","last_fullsized":"2024-03-30T15:00:53.223489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.794288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.794288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":494,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226833+0000","last_change":"2024-03-30T14:58:25.223144+0000","last_active":"2024-03-30T15:00:53.226833+0000","last_peered":"2024-03-30T15:00:53.226833+0000","last_clean":"2024-03-30T15:00:53.226833+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:00:53.226833+0000","last_undegraded":"2024-03-30T15:00:53.226833+0000","last_fullsized":"2024-03-30T15:00:53.226833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:25.223044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.223044+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743710+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228849+0000","last_change":"2024-03-30T14:58:25.984629+0000","last_active":"2024-03-30T15:00:53.228849+0000","last_peered":"2024-03-30T15:00:53.228849+0000","last_clean":"2024-03-30T15:00:53.228849+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:00:53.228849+0000","last_undegraded":"2024-03-30T15:00:53.228849+0000","last_fullsized":"2024-03-30T15:00:53.228849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.984552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.984552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228306+0000","last_change":"2024-03-30T15:00:02.270908+0000","last_active":"2024-03-30T15:00:53.228306+0000","last_peered":"2024-03-30T15:00:53.228306+0000","last_clean":"2024-03-30T15:00:53.228306+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:00:53.228306+0000","last_undegraded":"2024-03-30T15:00:53.228306+0000","last_fullsized":"2024-03-30T15:00:53.228306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.270821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.039536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":29,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217898+0000","last_change":"2024-03-30T15:00:45.241964+0000","last_active":"2024-03-30T15:00:53.217898+0000","last_peered":"2024-03-30T15:00:53.217898+0000","last_clean":"2024-03-30T15:00:53.217898+0000","last_became_active":"2024-03-30T15:00:45.241487+0000","last_became_peered":"2024-03-30T15:00:45.241487+0000","last_unstale":"2024-03-30T15:00:53.217898+0000","last_undegraded":"2024-03-30T15:00:53.217898+0000","last_fullsized":"2024-03-30T15:00:53.217898+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.153490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.153490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:20.931394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":372,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220327+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:00:53.220327+0000","last_peered":"2024-03-30T15:00:53.220327+0000","last_clean":"2024-03-30T15:00:53.220327+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:00:53.220327+0000","last_undegraded":"2024-03-30T15:00:53.220327+0000","last_fullsized":"2024-03-30T15:00:53.220327+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.044750+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":543,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226383+0000","last_change":"2024-03-30T14:59:05.977968+0000","last_active":"2024-03-30T15:00:53.226383+0000","last_peered":"2024-03-30T15:00:53.226383+0000","last_clean":"2024-03-30T15:00:53.226383+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:00:53.226383+0000","last_undegraded":"2024-03-30T15:00:53.226383+0000","last_fullsized":"2024-03-30T15:00:53.226383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:05.977866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.977866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":487,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219475+0000","last_change":"2024-03-30T14:58:35.054537+0000","last_active":"2024-03-30T15:00:53.219475+0000","last_peered":"2024-03-30T15:00:53.219475+0000","last_clean":"2024-03-30T15:00:53.219475+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:00:53.219475+0000","last_undegraded":"2024-03-30T15:00:53.219475+0000","last_fullsized":"2024-03-30T15:00:53.219475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:58:35.054439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.054439+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":479,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230209+0000","last_change":"2024-03-30T14:58:24.985818+0000","last_active":"2024-03-30T15:00:53.230209+0000","last_peered":"2024-03-30T15:00:53.230209+0000","last_clean":"2024-03-30T15:00:53.230209+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:00:53.230209+0000","last_undegraded":"2024-03-30T15:00:53.230209+0000","last_fullsized":"2024-03-30T15:00:53.230209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T14:58:24.985660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.985660+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":454,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222687+0000","last_change":"2024-03-30T14:58:27.124070+0000","last_active":"2024-03-30T15:00:53.222687+0000","last_peered":"2024-03-30T15:00:53.222687+0000","last_clean":"2024-03-30T15:00:53.222687+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:00:53.222687+0000","last_undegraded":"2024-03-30T15:00:53.222687+0000","last_fullsized":"2024-03-30T15:00:53.222687+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.123941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.123941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"284'8","reported_seq":49,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225523+0000","last_change":"2024-03-30T15:00:48.895056+0000","last_active":"2024-03-30T15:00:53.225523+0000","last_peered":"2024-03-30T15:00:53.225523+0000","last_clean":"2024-03-30T15:00:53.225523+0000","last_became_active":"2024-03-30T15:00:45.253237+0000","last_became_peered":"2024-03-30T15:00:45.253237+0000","last_unstale":"2024-03-30T15:00:53.225523+0000","last_undegraded":"2024-03-30T15:00:53.225523+0000","last_fullsized":"2024-03-30T15:00:53.225523+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'8","last_scrub_stamp":"2024-03-30T15:00:48.894963+0000","last_deep_scrub":"284'8","last_deep_scrub_stamp":"2024-03-30T15:00:48.894963+0000","last_clean_scrub_stamp":"2024-03-30T15:00:48.894963+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:14.733838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225247+0000","last_change":"2024-03-30T14:59:56.678123+0000","last_active":"2024-03-30T15:00:53.225247+0000","last_peered":"2024-03-30T15:00:53.225247+0000","last_clean":"2024-03-30T15:00:53.225247+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:00:53.225247+0000","last_undegraded":"2024-03-30T15:00:53.225247+0000","last_fullsized":"2024-03-30T15:00:53.225247+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.678045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.678045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:12.776149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":388,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230234+0000","last_change":"2024-03-30T14:57:56.275636+0000","last_active":"2024-03-30T15:00:53.230234+0000","last_peered":"2024-03-30T15:00:53.230234+0000","last_clean":"2024-03-30T15:00:53.230234+0000","last_became_active":"2024-03-30T14:57:56.273316+0000","last_became_peered":"2024-03-30T14:57:56.273316+0000","last_unstale":"2024-03-30T15:00:53.230234+0000","last_undegraded":"2024-03-30T15:00:53.230234+0000","last_fullsized":"2024-03-30T15:00:53.230234+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:57:30.919064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:57:30.919064+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":475,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220185+0000","last_change":"2024-03-30T14:58:22.261893+0000","last_active":"2024-03-30T15:00:53.220185+0000","last_peered":"2024-03-30T15:00:53.220185+0000","last_clean":"2024-03-30T15:00:53.220185+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:00:53.220185+0000","last_undegraded":"2024-03-30T15:00:53.220185+0000","last_fullsized":"2024-03-30T15:00:53.220185+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.261803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.261803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934477+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":458,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231151+0000","last_change":"2024-03-30T14:58:26.076218+0000","last_active":"2024-03-30T15:00:53.231151+0000","last_peered":"2024-03-30T15:00:53.231151+0000","last_clean":"2024-03-30T15:00:53.231151+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:00:53.231151+0000","last_undegraded":"2024-03-30T15:00:53.231151+0000","last_fullsized":"2024-03-30T15:00:53.231151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:26.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:26.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228801+0000","last_change":"2024-03-30T14:59:51.262210+0000","last_active":"2024-03-30T15:00:53.228801+0000","last_peered":"2024-03-30T15:00:53.228801+0000","last_clean":"2024-03-30T15:00:53.228801+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:00:53.228801+0000","last_undegraded":"2024-03-30T15:00:53.228801+0000","last_fullsized":"2024-03-30T15:00:53.228801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:51.262127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.262127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.554991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":375,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227603+0000","last_change":"2024-03-30T15:00:37.677858+0000","last_active":"2024-03-30T15:00:53.227603+0000","last_peered":"2024-03-30T15:00:53.227603+0000","last_clean":"2024-03-30T15:00:53.227603+0000","last_became_active":"2024-03-30T14:57:55.139072+0000","last_became_peered":"2024-03-30T14:57:55.139072+0000","last_unstale":"2024-03-30T15:00:53.227603+0000","last_undegraded":"2024-03-30T15:00:53.227603+0000","last_fullsized":"2024-03-30T15:00:53.227603+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.677768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.677768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.348942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":364,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230250+0000","last_change":"2024-03-30T14:57:55.126527+0000","last_active":"2024-03-30T15:00:53.230250+0000","last_peered":"2024-03-30T15:00:53.230250+0000","last_clean":"2024-03-30T15:00:53.230250+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:00:53.230250+0000","last_undegraded":"2024-03-30T15:00:53.230250+0000","last_fullsized":"2024-03-30T15:00:53.230250+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":480,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222191+0000","last_change":"2024-03-30T14:59:15.487022+0000","last_active":"2024-03-30T15:00:53.222191+0000","last_peered":"2024-03-30T15:00:53.222191+0000","last_clean":"2024-03-30T15:00:53.222191+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:00:53.222191+0000","last_undegraded":"2024-03-30T15:00:53.222191+0000","last_fullsized":"2024-03-30T15:00:53.222191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:15.486892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.486892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":475,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225914+0000","last_change":"2024-03-30T15:00:37.049789+0000","last_active":"2024-03-30T15:00:53.225914+0000","last_peered":"2024-03-30T15:00:53.225914+0000","last_clean":"2024-03-30T15:00:53.225914+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:00:53.225914+0000","last_undegraded":"2024-03-30T15:00:53.225914+0000","last_fullsized":"2024-03-30T15:00:53.225914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.049685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.049685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.710602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214305+0000","last_change":"2024-03-30T14:58:24.291134+0000","last_active":"2024-03-30T15:00:53.214305+0000","last_peered":"2024-03-30T15:00:53.214305+0000","last_clean":"2024-03-30T15:00:53.214305+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:00:53.214305+0000","last_undegraded":"2024-03-30T15:00:53.214305+0000","last_fullsized":"2024-03-30T15:00:53.214305+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:24.291008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.291008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":462,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223367+0000","last_change":"2024-03-30T14:59:02.809926+0000","last_active":"2024-03-30T15:00:53.223367+0000","last_peered":"2024-03-30T15:00:53.223367+0000","last_clean":"2024-03-30T15:00:53.223367+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:00:53.223367+0000","last_undegraded":"2024-03-30T15:00:53.223367+0000","last_fullsized":"2024-03-30T15:00:53.223367+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.809814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.809814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3","version":"115'5","reported_seq":380,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220296+0000","last_change":"2024-03-30T15:00:49.710616+0000","last_active":"2024-03-30T15:00:53.220296+0000","last_peered":"2024-03-30T15:00:53.220296+0000","last_clean":"2024-03-30T15:00:53.220296+0000","last_became_active":"2024-03-30T14:57:55.089349+0000","last_became_peered":"2024-03-30T14:57:55.089349+0000","last_unstale":"2024-03-30T15:00:53.220296+0000","last_undegraded":"2024-03-30T15:00:53.220296+0000","last_fullsized":"2024-03-30T15:00:53.220296+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:49.710514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:49.710514+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:17.317289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220321+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:00:53.220321+0000","last_peered":"2024-03-30T15:00:53.220321+0000","last_clean":"2024-03-30T15:00:53.220321+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:00:53.220321+0000","last_undegraded":"2024-03-30T15:00:53.220321+0000","last_fullsized":"2024-03-30T15:00:53.220321+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.612522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":478,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227763+0000","last_change":"2024-03-30T14:58:31.097064+0000","last_active":"2024-03-30T15:00:53.227763+0000","last_peered":"2024-03-30T15:00:53.227763+0000","last_clean":"2024-03-30T15:00:53.227763+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:00:53.227763+0000","last_undegraded":"2024-03-30T15:00:53.227763+0000","last_fullsized":"2024-03-30T15:00:53.227763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:58:31.096968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.096968+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251559+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":476,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219146+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:00:53.219146+0000","last_peered":"2024-03-30T15:00:53.219146+0000","last_clean":"2024-03-30T15:00:53.219146+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:00:53.219146+0000","last_undegraded":"2024-03-30T15:00:53.219146+0000","last_fullsized":"2024-03-30T15:00:53.219146+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:05.736897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":429,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228529+0000","last_change":"2024-03-30T14:59:53.269594+0000","last_active":"2024-03-30T15:00:53.228529+0000","last_peered":"2024-03-30T15:00:53.228529+0000","last_clean":"2024-03-30T15:00:53.228529+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:00:53.228529+0000","last_undegraded":"2024-03-30T15:00:53.228529+0000","last_fullsized":"2024-03-30T15:00:53.228529+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:53.269452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.269452+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.687457+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230118+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:00:53.230118+0000","last_peered":"2024-03-30T15:00:53.230118+0000","last_clean":"2024-03-30T15:00:53.230118+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:00:53.230118+0000","last_undegraded":"2024-03-30T15:00:53.230118+0000","last_fullsized":"2024-03-30T15:00:53.230118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:11.363643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"115'5","reported_seq":372,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225254+0000","last_change":"2024-03-30T14:57:55.177168+0000","last_active":"2024-03-30T15:00:53.225254+0000","last_peered":"2024-03-30T15:00:53.225254+0000","last_clean":"2024-03-30T15:00:53.225254+0000","last_became_active":"2024-03-30T14:57:55.164327+0000","last_became_peered":"2024-03-30T14:57:55.164327+0000","last_unstale":"2024-03-30T15:00:53.225254+0000","last_undegraded":"2024-03-30T15:00:53.225254+0000","last_fullsized":"2024-03-30T15:00:53.225254+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":366,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218439+0000","last_change":"2024-03-30T14:57:54.020464+0000","last_active":"2024-03-30T15:00:53.218439+0000","last_peered":"2024-03-30T15:00:53.218439+0000","last_clean":"2024-03-30T15:00:53.218439+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:00:53.218439+0000","last_undegraded":"2024-03-30T15:00:53.218439+0000","last_fullsized":"2024-03-30T15:00:53.218439+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":403,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.217218+0000","last_change":"2024-03-30T14:58:13.210584+0000","last_active":"2024-03-30T15:00:53.217218+0000","last_peered":"2024-03-30T15:00:53.217218+0000","last_clean":"2024-03-30T15:00:53.217218+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:00:53.217218+0000","last_undegraded":"2024-03-30T15:00:53.217218+0000","last_fullsized":"2024-03-30T15:00:53.217218+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:13.210426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.210426+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228072+0000","last_change":"2024-03-30T14:59:09.029236+0000","last_active":"2024-03-30T15:00:53.228072+0000","last_peered":"2024-03-30T15:00:53.228072+0000","last_clean":"2024-03-30T15:00:53.228072+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:00:53.228072+0000","last_undegraded":"2024-03-30T15:00:53.228072+0000","last_fullsized":"2024-03-30T15:00:53.228072+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.029159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.029159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":545,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227636+0000","last_change":"2024-03-30T14:59:08.160662+0000","last_active":"2024-03-30T15:00:53.227636+0000","last_peered":"2024-03-30T15:00:53.227636+0000","last_clean":"2024-03-30T15:00:53.227636+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:00:53.227636+0000","last_undegraded":"2024-03-30T15:00:53.227636+0000","last_fullsized":"2024-03-30T15:00:53.227636+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:08.160528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.160528+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":458,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221241+0000","last_change":"2024-03-30T14:59:16.481218+0000","last_active":"2024-03-30T15:00:53.221241+0000","last_peered":"2024-03-30T15:00:53.221241+0000","last_clean":"2024-03-30T15:00:53.221241+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:00:53.221241+0000","last_undegraded":"2024-03-30T15:00:53.221241+0000","last_fullsized":"2024-03-30T15:00:53.221241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:16.481128+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.481128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"115'5","reported_seq":375,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218856+0000","last_change":"2024-03-30T14:57:54.041959+0000","last_active":"2024-03-30T15:00:53.218856+0000","last_peered":"2024-03-30T15:00:53.218856+0000","last_clean":"2024-03-30T15:00:53.218856+0000","last_became_active":"2024-03-30T14:57:54.041685+0000","last_became_peered":"2024-03-30T14:57:54.041685+0000","last_unstale":"2024-03-30T15:00:53.218856+0000","last_undegraded":"2024-03-30T15:00:53.218856+0000","last_fullsized":"2024-03-30T15:00:53.218856+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":372,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220770+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:00:53.220770+0000","last_peered":"2024-03-30T15:00:53.220770+0000","last_clean":"2024-03-30T15:00:53.220770+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:00:53.220770+0000","last_undegraded":"2024-03-30T15:00:53.220770+0000","last_fullsized":"2024-03-30T15:00:53.220770+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.403334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":487,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220859+0000","last_change":"2024-03-30T14:59:20.453682+0000","last_active":"2024-03-30T15:00:53.220859+0000","last_peered":"2024-03-30T15:00:53.220859+0000","last_clean":"2024-03-30T15:00:53.220859+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:00:53.220859+0000","last_undegraded":"2024-03-30T15:00:53.220859+0000","last_fullsized":"2024-03-30T15:00:53.220859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.453594+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223001+0000","last_change":"2024-03-30T14:57:31.554095+0000","last_active":"2024-03-30T15:00:53.223001+0000","last_peered":"2024-03-30T15:00:53.223001+0000","last_clean":"2024-03-30T15:00:53.223001+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:00:53.223001+0000","last_undegraded":"2024-03-30T15:00:53.223001+0000","last_fullsized":"2024-03-30T15:00:53.223001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:31.554003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:31.554003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":468,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218383+0000","last_change":"2024-03-30T14:59:15.144954+0000","last_active":"2024-03-30T15:00:53.218383+0000","last_peered":"2024-03-30T15:00:53.218383+0000","last_clean":"2024-03-30T15:00:53.218383+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:00:53.218383+0000","last_undegraded":"2024-03-30T15:00:53.218383+0000","last_fullsized":"2024-03-30T15:00:53.218383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.144864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219108+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:00:53.219108+0000","last_peered":"2024-03-30T15:00:53.219108+0000","last_clean":"2024-03-30T15:00:53.219108+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:00:53.219108+0000","last_undegraded":"2024-03-30T15:00:53.219108+0000","last_fullsized":"2024-03-30T15:00:53.219108+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:29.384685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"115'9","reported_seq":385,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224645+0000","last_change":"2024-03-30T14:59:47.655366+0000","last_active":"2024-03-30T15:00:53.224645+0000","last_peered":"2024-03-30T15:00:53.224645+0000","last_clean":"2024-03-30T15:00:53.224645+0000","last_became_active":"2024-03-30T14:57:55.181249+0000","last_became_peered":"2024-03-30T14:57:55.181249+0000","last_unstale":"2024-03-30T15:00:53.224645+0000","last_undegraded":"2024-03-30T15:00:53.224645+0000","last_fullsized":"2024-03-30T15:00:53.224645+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9","last_scrub_stamp":"2024-03-30T14:59:47.655290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.655290+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:10.972447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":372,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223904+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:00:53.223904+0000","last_peered":"2024-03-30T15:00:53.223904+0000","last_clean":"2024-03-30T15:00:53.223904+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:00:53.223904+0000","last_undegraded":"2024-03-30T15:00:53.223904+0000","last_fullsized":"2024-03-30T15:00:53.223904+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.153282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":480,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218493+0000","last_change":"2024-03-30T14:59:08.292999+0000","last_active":"2024-03-30T15:00:53.218493+0000","last_peered":"2024-03-30T15:00:53.218493+0000","last_clean":"2024-03-30T15:00:53.218493+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:00:53.218493+0000","last_undegraded":"2024-03-30T15:00:53.218493+0000","last_fullsized":"2024-03-30T15:00:53.218493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:08.292876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.292876+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":482,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228893+0000","last_change":"2024-03-30T14:58:12.927851+0000","last_active":"2024-03-30T15:00:53.228893+0000","last_peered":"2024-03-30T15:00:53.228893+0000","last_clean":"2024-03-30T15:00:53.228893+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:00:53.228893+0000","last_undegraded":"2024-03-30T15:00:53.228893+0000","last_fullsized":"2024-03-30T15:00:53.228893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:58:12.927762+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.927762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220606+0000","last_change":"2024-03-30T14:59:17.441377+0000","last_active":"2024-03-30T15:00:53.220606+0000","last_peered":"2024-03-30T15:00:53.220606+0000","last_clean":"2024-03-30T15:00:53.220606+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:00:53.220606+0000","last_undegraded":"2024-03-30T15:00:53.220606+0000","last_fullsized":"2024-03-30T15:00:53.220606+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.441289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.441289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":456,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218319+0000","last_change":"2024-03-30T14:58:18.177964+0000","last_active":"2024-03-30T15:00:53.218319+0000","last_peered":"2024-03-30T15:00:53.218319+0000","last_clean":"2024-03-30T15:00:53.218319+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:00:53.218319+0000","last_undegraded":"2024-03-30T15:00:53.218319+0000","last_fullsized":"2024-03-30T15:00:53.218319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:18.177806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:18.177806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572785+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222979+0000","last_change":"2024-03-30T14:57:54.013168+0000","last_active":"2024-03-30T15:00:53.222979+0000","last_peered":"2024-03-30T15:00:53.222979+0000","last_clean":"2024-03-30T15:00:53.222979+0000","last_became_active":"2024-03-30T14:57:54.012861+0000","last_became_peered":"2024-03-30T14:57:54.012861+0000","last_unstale":"2024-03-30T15:00:53.222979+0000","last_undegraded":"2024-03-30T15:00:53.222979+0000","last_fullsized":"2024-03-30T15:00:53.222979+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218929+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:00:53.218929+0000","last_peered":"2024-03-30T15:00:53.218929+0000","last_clean":"2024-03-30T15:00:53.218929+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:00:53.218929+0000","last_undegraded":"2024-03-30T15:00:53.218929+0000","last_fullsized":"2024-03-30T15:00:53.218929+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.187292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":406,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226252+0000","last_change":"2024-03-30T14:58:16.905699+0000","last_active":"2024-03-30T15:00:53.226252+0000","last_peered":"2024-03-30T15:00:53.226252+0000","last_clean":"2024-03-30T15:00:53.226252+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:00:53.226252+0000","last_undegraded":"2024-03-30T15:00:53.226252+0000","last_fullsized":"2024-03-30T15:00:53.226252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:58:16.905604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:16.905604+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.231242+0000","last_change":"2024-03-30T14:58:28.115261+0000","last_active":"2024-03-30T15:00:53.231242+0000","last_peered":"2024-03-30T15:00:53.231242+0000","last_clean":"2024-03-30T15:00:53.231242+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:00:53.231242+0000","last_undegraded":"2024-03-30T15:00:53.231242+0000","last_fullsized":"2024-03-30T15:00:53.231242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:28.115166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:28.115166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":459,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226001+0000","last_change":"2024-03-30T14:59:08.012564+0000","last_active":"2024-03-30T15:00:53.226001+0000","last_peered":"2024-03-30T15:00:53.226001+0000","last_clean":"2024-03-30T15:00:53.226001+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:00:53.226001+0000","last_undegraded":"2024-03-30T15:00:53.226001+0000","last_fullsized":"2024-03-30T15:00:53.226001+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.012470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.012470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220048+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:00:53.220048+0000","last_peered":"2024-03-30T15:00:53.220048+0000","last_clean":"2024-03-30T15:00:53.220048+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:00:53.220048+0000","last_undegraded":"2024-03-30T15:00:53.220048+0000","last_fullsized":"2024-03-30T15:00:53.220048+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.714400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218654+0000","last_change":"2024-03-30T14:57:54.019408+0000","last_active":"2024-03-30T15:00:53.218654+0000","last_peered":"2024-03-30T15:00:53.218654+0000","last_clean":"2024-03-30T15:00:53.218654+0000","last_became_active":"2024-03-30T14:57:54.014697+0000","last_became_peered":"2024-03-30T14:57:54.014697+0000","last_unstale":"2024-03-30T15:00:53.218654+0000","last_undegraded":"2024-03-30T15:00:53.218654+0000","last_fullsized":"2024-03-30T15:00:53.218654+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213467+0000","last_change":"2024-03-30T14:59:56.148641+0000","last_active":"2024-03-30T15:00:53.213467+0000","last_peered":"2024-03-30T15:00:53.213467+0000","last_clean":"2024-03-30T15:00:53.213467+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:00:53.213467+0000","last_undegraded":"2024-03-30T15:00:53.213467+0000","last_fullsized":"2024-03-30T15:00:53.213467+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.148529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.148529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.134818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":394,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213346+0000","last_change":"2024-03-30T14:58:17.304912+0000","last_active":"2024-03-30T15:00:53.213346+0000","last_peered":"2024-03-30T15:00:53.213346+0000","last_clean":"2024-03-30T15:00:53.213346+0000","last_became_active":"2024-03-30T14:57:56.224608+0000","last_became_peered":"2024-03-30T14:57:56.224608+0000","last_unstale":"2024-03-30T15:00:53.213346+0000","last_undegraded":"2024-03-30T15:00:53.213346+0000","last_fullsized":"2024-03-30T15:00:53.213346+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:58:17.304759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:17.304759+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":476,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222774+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:00:53.222774+0000","last_peered":"2024-03-30T15:00:53.222774+0000","last_clean":"2024-03-30T15:00:53.222774+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:00:53.222774+0000","last_undegraded":"2024-03-30T15:00:53.222774+0000","last_fullsized":"2024-03-30T15:00:53.222774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.891147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223482+0000","last_change":"2024-03-30T14:58:25.320605+0000","last_active":"2024-03-30T15:00:53.223482+0000","last_peered":"2024-03-30T15:00:53.223482+0000","last_clean":"2024-03-30T15:00:53.223482+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:00:53.223482+0000","last_undegraded":"2024-03-30T15:00:53.223482+0000","last_fullsized":"2024-03-30T15:00:53.223482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.320527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.320527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227052+0000","last_change":"2024-03-30T14:58:58.957529+0000","last_active":"2024-03-30T15:00:53.227052+0000","last_peered":"2024-03-30T15:00:53.227052+0000","last_clean":"2024-03-30T15:00:53.227052+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:00:53.227052+0000","last_undegraded":"2024-03-30T15:00:53.227052+0000","last_fullsized":"2024-03-30T15:00:53.227052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.957449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.957449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":376,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226690+0000","last_change":"2024-03-30T14:59:07.508558+0000","last_active":"2024-03-30T15:00:53.226690+0000","last_peered":"2024-03-30T15:00:53.226690+0000","last_clean":"2024-03-30T15:00:53.226690+0000","last_became_active":"2024-03-30T14:57:55.201639+0000","last_became_peered":"2024-03-30T14:57:55.201639+0000","last_unstale":"2024-03-30T15:00:53.226690+0000","last_undegraded":"2024-03-30T15:00:53.226690+0000","last_fullsized":"2024-03-30T15:00:53.226690+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.508480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.508480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225505+0000","last_change":"2024-03-30T15:00:03.553767+0000","last_active":"2024-03-30T15:00:53.225505+0000","last_peered":"2024-03-30T15:00:53.225505+0000","last_clean":"2024-03-30T15:00:53.225505+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:00:53.225505+0000","last_undegraded":"2024-03-30T15:00:53.225505+0000","last_fullsized":"2024-03-30T15:00:53.225505+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.553680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.553680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.545159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":495,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225507+0000","last_change":"2024-03-30T15:00:41.903846+0000","last_active":"2024-03-30T15:00:53.225507+0000","last_peered":"2024-03-30T15:00:53.225507+0000","last_clean":"2024-03-30T15:00:53.225507+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:00:53.225507+0000","last_undegraded":"2024-03-30T15:00:53.225507+0000","last_fullsized":"2024-03-30T15:00:53.225507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:00:41.903753+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.903753+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.208912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218521+0000","last_change":"2024-03-30T14:59:09.624243+0000","last_active":"2024-03-30T15:00:53.218521+0000","last_peered":"2024-03-30T15:00:53.218521+0000","last_clean":"2024-03-30T15:00:53.218521+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:00:53.218521+0000","last_undegraded":"2024-03-30T15:00:53.218521+0000","last_fullsized":"2024-03-30T15:00:53.218521+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.624114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.624114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":461,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222375+0000","last_change":"2024-03-30T14:58:58.605917+0000","last_active":"2024-03-30T15:00:53.222375+0000","last_peered":"2024-03-30T15:00:53.222375+0000","last_clean":"2024-03-30T15:00:53.222375+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:00:53.222375+0000","last_undegraded":"2024-03-30T15:00:53.222375+0000","last_fullsized":"2024-03-30T15:00:53.222375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.605792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.605792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":464,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.215176+0000","last_change":"2024-03-30T14:59:21.144421+0000","last_active":"2024-03-30T15:00:53.215176+0000","last_peered":"2024-03-30T15:00:53.215176+0000","last_clean":"2024-03-30T15:00:53.215176+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:00:53.215176+0000","last_undegraded":"2024-03-30T15:00:53.215176+0000","last_fullsized":"2024-03-30T15:00:53.215176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.144335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.144335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.a","version":"115'10","reported_seq":385,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226801+0000","last_change":"2024-03-30T15:00:42.061005+0000","last_active":"2024-03-30T15:00:53.226801+0000","last_peered":"2024-03-30T15:00:53.226801+0000","last_clean":"2024-03-30T15:00:53.226801+0000","last_became_active":"2024-03-30T14:57:55.103400+0000","last_became_peered":"2024-03-30T14:57:55.103400+0000","last_unstale":"2024-03-30T15:00:53.226801+0000","last_undegraded":"2024-03-30T15:00:53.226801+0000","last_fullsized":"2024-03-30T15:00:53.226801+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:42.060906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:42.060906+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:09.665039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225034+0000","last_change":"2024-03-30T14:59:14.560489+0000","last_active":"2024-03-30T15:00:53.225034+0000","last_peered":"2024-03-30T15:00:53.225034+0000","last_clean":"2024-03-30T15:00:53.225034+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:00:53.225034+0000","last_undegraded":"2024-03-30T15:00:53.225034+0000","last_fullsized":"2024-03-30T15:00:53.225034+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.560377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.560377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":485,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221692+0000","last_change":"2024-03-30T14:59:10.626455+0000","last_active":"2024-03-30T15:00:53.221692+0000","last_peered":"2024-03-30T15:00:53.221692+0000","last_clean":"2024-03-30T15:00:53.221692+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:00:53.221692+0000","last_undegraded":"2024-03-30T15:00:53.221692+0000","last_fullsized":"2024-03-30T15:00:53.221692+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.626365+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":469,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213619+0000","last_change":"2024-03-30T14:58:32.272039+0000","last_active":"2024-03-30T15:00:53.213619+0000","last_peered":"2024-03-30T15:00:53.213619+0000","last_clean":"2024-03-30T15:00:53.213619+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:00:53.213619+0000","last_undegraded":"2024-03-30T15:00:53.213619+0000","last_fullsized":"2024-03-30T15:00:53.213619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T14:58:32.271622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:32.271622+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":554,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225404+0000","last_change":"2024-03-30T14:58:59.472999+0000","last_active":"2024-03-30T15:00:53.225404+0000","last_peered":"2024-03-30T15:00:53.225404+0000","last_clean":"2024-03-30T15:00:53.225404+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:00:53.225404+0000","last_undegraded":"2024-03-30T15:00:53.225404+0000","last_fullsized":"2024-03-30T15:00:53.225404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.472884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.472884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":465,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228662+0000","last_change":"2024-03-30T15:00:36.069861+0000","last_active":"2024-03-30T15:00:53.228662+0000","last_peered":"2024-03-30T15:00:53.228662+0000","last_clean":"2024-03-30T15:00:53.228662+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:00:53.228662+0000","last_undegraded":"2024-03-30T15:00:53.228662+0000","last_fullsized":"2024-03-30T15:00:53.228662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:00:36.069752+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:00:36.069752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:40.687480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"115'2","reported_seq":34,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.219893+0000","last_change":"2024-03-30T15:00:45.260322+0000","last_active":"2024-03-30T15:00:53.219893+0000","last_peered":"2024-03-30T15:00:53.219893+0000","last_clean":"2024-03-30T15:00:53.219893+0000","last_became_active":"2024-03-30T15:00:45.244909+0000","last_became_peered":"2024-03-30T15:00:45.244909+0000","last_unstale":"2024-03-30T15:00:53.219893+0000","last_undegraded":"2024-03-30T15:00:53.219893+0000","last_fullsized":"2024-03-30T15:00:53.219893+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":373,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225805+0000","last_change":"2024-03-30T14:59:21.597544+0000","last_active":"2024-03-30T15:00:53.225805+0000","last_peered":"2024-03-30T15:00:53.225805+0000","last_clean":"2024-03-30T15:00:53.225805+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:00:53.225805+0000","last_undegraded":"2024-03-30T15:00:53.225805+0000","last_fullsized":"2024-03-30T15:00:53.225805+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":411,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214790+0000","last_change":"2024-03-30T14:58:15.287344+0000","last_active":"2024-03-30T15:00:53.214790+0000","last_peered":"2024-03-30T15:00:53.214790+0000","last_clean":"2024-03-30T15:00:53.214790+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:00:53.214790+0000","last_undegraded":"2024-03-30T15:00:53.214790+0000","last_fullsized":"2024-03-30T15:00:53.214790+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:15.286572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.286572+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":467,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227258+0000","last_change":"2024-03-30T14:58:31.196084+0000","last_active":"2024-03-30T15:00:53.227258+0000","last_peered":"2024-03-30T15:00:53.227258+0000","last_clean":"2024-03-30T15:00:53.227258+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:00:53.227258+0000","last_undegraded":"2024-03-30T15:00:53.227258+0000","last_fullsized":"2024-03-30T15:00:53.227258+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:31.196010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.196010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":459,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223409+0000","last_change":"2024-03-30T14:59:01.294973+0000","last_active":"2024-03-30T15:00:53.223409+0000","last_peered":"2024-03-30T15:00:53.223409+0000","last_clean":"2024-03-30T15:00:53.223409+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:00:53.223409+0000","last_undegraded":"2024-03-30T15:00:53.223409+0000","last_fullsized":"2024-03-30T15:00:53.223409+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.294888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.294888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":369,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225598+0000","last_change":"2024-03-30T14:58:12.296903+0000","last_active":"2024-03-30T15:00:53.225598+0000","last_peered":"2024-03-30T15:00:53.225598+0000","last_clean":"2024-03-30T15:00:53.225598+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:00:53.225598+0000","last_undegraded":"2024-03-30T15:00:53.225598+0000","last_fullsized":"2024-03-30T15:00:53.225598+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:12.296793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.296793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"115'2","reported_seq":377,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225161+0000","last_change":"2024-03-30T15:00:49.067522+0000","last_active":"2024-03-30T15:00:53.225161+0000","last_peered":"2024-03-30T15:00:53.225161+0000","last_clean":"2024-03-30T15:00:53.225161+0000","last_became_active":"2024-03-30T14:57:55.108185+0000","last_became_peered":"2024-03-30T14:57:55.108185+0000","last_unstale":"2024-03-30T15:00:53.225161+0000","last_undegraded":"2024-03-30T15:00:53.225161+0000","last_fullsized":"2024-03-30T15:00:53.225161+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-03-30T15:00:49.067432+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-03-30T15:00:49.067432+0000","last_clean_scrub_stamp":"2024-03-30T15:00:49.067432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:54.711501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":372,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222630+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:00:53.222630+0000","last_peered":"2024-03-30T15:00:53.222630+0000","last_clean":"2024-03-30T15:00:53.222630+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:00:53.222630+0000","last_undegraded":"2024-03-30T15:00:53.222630+0000","last_fullsized":"2024-03-30T15:00:53.222630+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.587803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":502,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223180+0000","last_change":"2024-03-30T14:58:37.879183+0000","last_active":"2024-03-30T15:00:53.223180+0000","last_peered":"2024-03-30T15:00:53.223180+0000","last_clean":"2024-03-30T15:00:53.223180+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:00:53.223180+0000","last_undegraded":"2024-03-30T15:00:53.223180+0000","last_fullsized":"2024-03-30T15:00:53.223180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T14:58:37.879074+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.704786+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":476,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220738+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:00:53.220738+0000","last_peered":"2024-03-30T15:00:53.220738+0000","last_clean":"2024-03-30T15:00:53.220738+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:00:53.220738+0000","last_undegraded":"2024-03-30T15:00:53.220738+0000","last_fullsized":"2024-03-30T15:00:53.220738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:21.606179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":460,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.215018+0000","last_change":"2024-03-30T14:58:19.314405+0000","last_active":"2024-03-30T15:00:53.215018+0000","last_peered":"2024-03-30T15:00:53.215018+0000","last_clean":"2024-03-30T15:00:53.215018+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:00:53.215018+0000","last_undegraded":"2024-03-30T15:00:53.215018+0000","last_fullsized":"2024-03-30T15:00:53.215018+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.314262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.314262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":456,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.230459+0000","last_change":"2024-03-30T14:59:03.291569+0000","last_active":"2024-03-30T15:00:53.230459+0000","last_peered":"2024-03-30T15:00:53.230459+0000","last_clean":"2024-03-30T15:00:53.230459+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:00:53.230459+0000","last_undegraded":"2024-03-30T15:00:53.230459+0000","last_fullsized":"2024-03-30T15:00:53.230459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.291469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.291469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":364,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218827+0000","last_change":"2024-03-30T14:57:56.258314+0000","last_active":"2024-03-30T15:00:53.218827+0000","last_peered":"2024-03-30T15:00:53.218827+0000","last_clean":"2024-03-30T15:00:53.218827+0000","last_became_active":"2024-03-30T14:57:56.257360+0000","last_became_peered":"2024-03-30T14:57:56.257360+0000","last_unstale":"2024-03-30T15:00:53.218827+0000","last_undegraded":"2024-03-30T15:00:53.218827+0000","last_fullsized":"2024-03-30T15:00:53.218827+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":374,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213360+0000","last_change":"2024-03-30T14:59:50.075598+0000","last_active":"2024-03-30T15:00:53.213360+0000","last_peered":"2024-03-30T15:00:53.213360+0000","last_clean":"2024-03-30T15:00:53.213360+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:00:53.213360+0000","last_undegraded":"2024-03-30T15:00:53.213360+0000","last_fullsized":"2024-03-30T15:00:53.213360+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.075512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.075512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:03.184144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":485,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220626+0000","last_change":"2024-03-30T14:59:46.413217+0000","last_active":"2024-03-30T15:00:53.220626+0000","last_peered":"2024-03-30T15:00:53.220626+0000","last_clean":"2024-03-30T15:00:53.220626+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T15:00:53.220626+0000","last_undegraded":"2024-03-30T15:00:53.220626+0000","last_fullsized":"2024-03-30T15:00:53.220626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T14:59:46.413127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.413127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.902904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222413+0000","last_change":"2024-03-30T14:58:15.115774+0000","last_active":"2024-03-30T15:00:53.222413+0000","last_peered":"2024-03-30T15:00:53.222413+0000","last_clean":"2024-03-30T15:00:53.222413+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:00:53.222413+0000","last_undegraded":"2024-03-30T15:00:53.222413+0000","last_fullsized":"2024-03-30T15:00:53.222413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:15.115694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.115694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":459,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.226997+0000","last_change":"2024-03-30T14:58:29.060905+0000","last_active":"2024-03-30T15:00:53.226997+0000","last_peered":"2024-03-30T15:00:53.226997+0000","last_clean":"2024-03-30T15:00:53.226997+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:00:53.226997+0000","last_undegraded":"2024-03-30T15:00:53.226997+0000","last_fullsized":"2024-03-30T15:00:53.226997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:29.060822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:29.060822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":462,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220680+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:00:53.220680+0000","last_peered":"2024-03-30T15:00:53.220680+0000","last_clean":"2024-03-30T15:00:53.220680+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:00:53.220680+0000","last_undegraded":"2024-03-30T15:00:53.220680+0000","last_fullsized":"2024-03-30T15:00:53.220680+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.177682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":29,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224381+0000","last_change":"2024-03-30T15:00:45.246983+0000","last_active":"2024-03-30T15:00:53.224381+0000","last_peered":"2024-03-30T15:00:53.224381+0000","last_clean":"2024-03-30T15:00:53.224381+0000","last_became_active":"2024-03-30T15:00:45.246116+0000","last_became_peered":"2024-03-30T15:00:45.246116+0000","last_unstale":"2024-03-30T15:00:53.224381+0000","last_undegraded":"2024-03-30T15:00:53.224381+0000","last_fullsized":"2024-03-30T15:00:53.224381+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:16.765776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.765776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.214443+0000","last_change":"2024-03-30T15:00:00.172131+0000","last_active":"2024-03-30T15:00:53.214443+0000","last_peered":"2024-03-30T15:00:53.214443+0000","last_clean":"2024-03-30T15:00:53.214443+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:00:53.214443+0000","last_undegraded":"2024-03-30T15:00:53.214443+0000","last_fullsized":"2024-03-30T15:00:53.214443+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.172053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.172053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:07.869357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":479,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227160+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:00:53.227160+0000","last_peered":"2024-03-30T15:00:53.227160+0000","last_clean":"2024-03-30T15:00:53.227160+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:00:53.227160+0000","last_undegraded":"2024-03-30T15:00:53.227160+0000","last_fullsized":"2024-03-30T15:00:53.227160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:53.313881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":466,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.227579+0000","last_change":"2024-03-30T14:58:25.078864+0000","last_active":"2024-03-30T15:00:53.227579+0000","last_peered":"2024-03-30T15:00:53.227579+0000","last_clean":"2024-03-30T15:00:53.227579+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:00:53.227579+0000","last_undegraded":"2024-03-30T15:00:53.227579+0000","last_fullsized":"2024-03-30T15:00:53.227579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.078771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.078771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":458,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.222780+0000","last_change":"2024-03-30T14:58:19.191376+0000","last_active":"2024-03-30T15:00:53.222780+0000","last_peered":"2024-03-30T15:00:53.222780+0000","last_clean":"2024-03-30T15:00:53.222780+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:00:53.222780+0000","last_undegraded":"2024-03-30T15:00:53.222780+0000","last_fullsized":"2024-03-30T15:00:53.222780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.191278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.191278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251490+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":464,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218589+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:00:53.218589+0000","last_peered":"2024-03-30T15:00:53.218589+0000","last_clean":"2024-03-30T15:00:53.218589+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:00:53.218589+0000","last_undegraded":"2024-03-30T15:00:53.218589+0000","last_fullsized":"2024-03-30T15:00:53.218589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.919742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"115'5","reported_seq":42,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.232293+0000","last_change":"2024-03-30T15:00:53.232293+0000","last_active":"2024-03-30T15:00:53.232293+0000","last_peered":"2024-03-30T15:00:53.232293+0000","last_clean":"2024-03-30T15:00:53.232293+0000","last_became_active":"2024-03-30T15:00:53.231995+0000","last_became_peered":"2024-03-30T15:00:53.231995+0000","last_unstale":"2024-03-30T15:00:53.232293+0000","last_undegraded":"2024-03-30T15:00:53.232293+0000","last_fullsized":"2024-03-30T15:00:53.232293+0000","mapping_epoch":280,"log_start":"115'5","ondisk_log_start":"115'5","created":111,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.263392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.263392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":5,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.300622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":364,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220429+0000","last_change":"2024-03-30T14:57:55.106979+0000","last_active":"2024-03-30T15:00:53.220429+0000","last_peered":"2024-03-30T15:00:53.220429+0000","last_clean":"2024-03-30T15:00:53.220429+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:00:53.220429+0000","last_undegraded":"2024-03-30T15:00:53.220429+0000","last_fullsized":"2024-03-30T15:00:53.220429+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.251286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":460,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.223365+0000","last_change":"2024-03-30T14:59:59.784365+0000","last_active":"2024-03-30T15:00:53.223365+0000","last_peered":"2024-03-30T15:00:53.223365+0000","last_clean":"2024-03-30T15:00:53.223365+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:00:53.223365+0000","last_undegraded":"2024-03-30T15:00:53.223365+0000","last_fullsized":"2024-03-30T15:00:53.223365+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:59:59.784262+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.784262+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.754986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":370,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224857+0000","last_change":"2024-03-30T14:59:06.484972+0000","last_active":"2024-03-30T15:00:53.224857+0000","last_peered":"2024-03-30T15:00:53.224857+0000","last_clean":"2024-03-30T15:00:53.224857+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:00:53.224857+0000","last_undegraded":"2024-03-30T15:00:53.224857+0000","last_fullsized":"2024-03-30T15:00:53.224857+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:06.484877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T14:59:06.484877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.934524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":459,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220249+0000","last_change":"2024-03-30T14:59:12.396403+0000","last_active":"2024-03-30T15:00:53.220249+0000","last_peered":"2024-03-30T15:00:53.220249+0000","last_clean":"2024-03-30T15:00:53.220249+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:00:53.220249+0000","last_undegraded":"2024-03-30T15:00:53.220249+0000","last_fullsized":"2024-03-30T15:00:53.220249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.396321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":368,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.228010+0000","last_change":"2024-03-30T14:59:01.955656+0000","last_active":"2024-03-30T15:00:53.228010+0000","last_peered":"2024-03-30T15:00:53.228010+0000","last_clean":"2024-03-30T15:00:53.228010+0000","last_became_active":"2024-03-30T14:57:56.272206+0000","last_became_peered":"2024-03-30T14:57:56.272206+0000","last_unstale":"2024-03-30T15:00:53.228010+0000","last_undegraded":"2024-03-30T15:00:53.228010+0000","last_fullsized":"2024-03-30T15:00:53.228010+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.955549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.955549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":33,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218356+0000","last_change":"2024-03-30T15:00:45.245980+0000","last_active":"2024-03-30T15:00:53.218356+0000","last_peered":"2024-03-30T15:00:53.218356+0000","last_clean":"2024-03-30T15:00:53.218356+0000","last_became_active":"2024-03-30T15:00:45.245596+0000","last_became_peered":"2024-03-30T15:00:45.245596+0000","last_unstale":"2024-03-30T15:00:53.218356+0000","last_undegraded":"2024-03-30T15:00:53.218356+0000","last_fullsized":"2024-03-30T15:00:53.218356+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.029978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.029978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:11.919390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":366,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.218925+0000","last_change":"2024-03-30T14:57:54.063914+0000","last_active":"2024-03-30T15:00:53.218925+0000","last_peered":"2024-03-30T15:00:53.218925+0000","last_clean":"2024-03-30T15:00:53.218925+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:00:53.218925+0000","last_undegraded":"2024-03-30T15:00:53.218925+0000","last_fullsized":"2024-03-30T15:00:53.218925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:57.572484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":487,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.220415+0000","last_change":"2024-03-30T14:59:57.415396+0000","last_active":"2024-03-30T15:00:53.220415+0000","last_peered":"2024-03-30T15:00:53.220415+0000","last_clean":"2024-03-30T15:00:53.220415+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T15:00:53.220415+0000","last_undegraded":"2024-03-30T15:00:53.220415+0000","last_fullsized":"2024-03-30T15:00:53.220415+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.852042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":477,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.221655+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:00:53.221655+0000","last_peered":"2024-03-30T15:00:53.221655+0000","last_clean":"2024-03-30T15:00:53.221655+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:00:53.221655+0000","last_undegraded":"2024-03-30T15:00:53.221655+0000","last_fullsized":"2024-03-30T15:00:53.221655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:58.262876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":460,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.213315+0000","last_change":"2024-03-30T14:59:04.268770+0000","last_active":"2024-03-30T15:00:53.213315+0000","last_peered":"2024-03-30T15:00:53.213315+0000","last_clean":"2024-03-30T15:00:53.213315+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:00:53.213315+0000","last_undegraded":"2024-03-30T15:00:53.213315+0000","last_fullsized":"2024-03-30T15:00:53.213315+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:04.268662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T14:59:04.268662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":455,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.224075+0000","last_change":"2024-03-30T14:58:27.174113+0000","last_active":"2024-03-30T15:00:53.224075+0000","last_peered":"2024-03-30T15:00:53.224075+0000","last_clean":"2024-03-30T15:00:53.224075+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:00:53.224075+0000","last_undegraded":"2024-03-30T15:00:53.224075+0000","last_fullsized":"2024-03-30T15:00:53.224075+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.174017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.174017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:00:56.743692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.d","version":"115'5","reported_seq":34,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-30T15:00:53.225622+0000","last_change":"2024-03-30T15:00:45.259199+0000","last_active":"2024-03-30T15:00:53.225622+0000","last_peered":"2024-03-30T15:00:53.225622+0000","last_clean":"2024-03-30T15:00:53.225622+0000","last_became_active":"2024-03-30T15:00:45.252183+0000","last_became_peered":"2024-03-30T15:00:45.252183+0000","last_unstale":"2024-03-30T15:00:53.225622+0000","last_undegraded":"2024-03-30T15:00:53.225622+0000","last_fullsized":"2024-03-30T15:00:53.225622+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.045519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.045519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:57.070397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":258,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":4195506,"num_objects":120,"num_object_clones":1,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":95,"num_read_kb":1,"num_write":540,"num_write_kb":8726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":12586518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1790,"ondisk_log_size":1790,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":17,"num_pg":30,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":90,"acting":90,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005136,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72896,"kb_used_data":5616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672256,"statfs":{"total":95995035648,"available":95920390144,"internally_reserved":0,"allocated":5750784,"data_stored":3461939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299477,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77000,"kb_used_data":5552,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668152,"statfs":{"total":95995035648,"available":95916187648,"internally_reserved":0,"allocated":5685248,"data_stored":3447037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72296,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672856,"statfs":{"total":95995035648,"available":95921004544,"internally_reserved":0,"allocated":5120000,"data_stored":2881238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54804,"kb_used_data":9724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93690348,"statfs":{"total":95995035648,"available":95938916352,"internally_reserved":0,"allocated":9957376,"data_stored":7666974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80568,"kb_used_data":9120,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664584,"statfs":{"total":95995035648,"available":95912534016,"internally_reserved":0,"allocated":9338880,"data_stored":7075162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80616,"kb_used_data":9160,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664536,"statfs":{"total":95995035648,"available":95912484864,"internally_reserved":0,"allocated":9379840,"data_stored":7076396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76476,"kb_used_data":5036,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668676,"statfs":{"total":95995035648,"available":95916724224,"internally_reserved":0,"allocated":5156864,"data_stored":2870704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659226,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76484,"kb_used_data":5044,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668668,"statfs":{"total":95995035648,"available":95916716032,"internally_reserved":0,"allocated":5165056,"data_stored":2880639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":4194737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4382720,"data_stored":4194768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":4194795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:00:58.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 1 2024-03-30T15:00:58.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:58.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:58.555 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.49630 v1:172.21.15.28:0/3716318344' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi028-92999-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.57578 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi028-92979-39"}]': finished 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: osdmap e293: 8 total, 8 up, 8 in 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:58 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-30T15:00:58.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:00:58.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:58.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:58.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:00:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:58.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:00:58.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:00:58.857 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:00:58.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:00:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:58.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:58.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:00:59.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:59.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.171 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:59.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:59.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:59.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:59.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:00:59.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: pgmap v380: 298 pgs: 40 unknown, 258 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: pgmap v380: 298 pgs: 40 unknown, 258 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:59.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-30T15:00:59.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T15:00:59.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:59.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:00:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:59.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:00:59.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:00:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:59.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:00:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 -- 172.21.15.28:0/702503307 >> v1:172.21.15.28:6789/0 conn(0x7fd40c101110 legacy=0x7fd40c1014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 -- 172.21.15.28:0/702503307 shutdown_connections 2024-03-30T15:00:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 -- 172.21.15.28:0/702503307 >> 172.21.15.28:0/702503307 conn(0x7fd40c0fc9c0 msgr2=0x7fd40c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:00:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 -- 172.21.15.28:0/702503307 shutdown_connections 2024-03-30T15:00:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 -- 172.21.15.28:0/702503307 wait complete. 2024-03-30T15:00:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 librados: starting msgr at 2024-03-30T15:00:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 librados: starting objecter 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.638+0000 7fd4117ba700 1 Processor -- start 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 -- start start 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 librados: setting wanted keys 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 librados: calling monclient init 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1a9300 con 0x7fd40c109da0 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1aa520 con 0x7fd40c101110 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd4117ba700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1ab760 con 0x7fd40c10d940 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd40a7fc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd40c109da0 0x7fd40c1a41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43372/0 (socket says 172.21.15.28:43372) 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd40a7fc700 1 -- 172.21.15.28:0/2780414744 learned_addr learned my addr 172.21.15.28:0/2780414744 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter ms_handle_connect 0x7fd40c109da0 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter ms_handle_connect 0x7fd40c10d940 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter ms_handle_connect 0x7fd40c101110 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175558211 0 0) 0x7fd40c1ab760 con 0x7fd40c10d940 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3f80039c0 con 0x7fd40c10d940 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 499632194 0 0) 0x7fd40c1a9300 con 0x7fd40c109da0 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c1ab760 con 0x7fd40c109da0 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249605209 0 0) 0x7fd40c1aa520 con 0x7fd40c101110 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.639+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c1a9300 con 0x7fd40c101110 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1377899737 0 0) 0x7fd3f80039c0 con 0x7fd40c10d940 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd40c1aa520 con 0x7fd40c10d940 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd3fc001020 con 0x7fd40c10d940 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2302883513 0 0) 0x7fd40c1ab760 con 0x7fd40c109da0 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3f80039c0 con 0x7fd40c109da0 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd3f40018f0 con 0x7fd40c109da0 2024-03-30T15:00:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2045246329 0 0) 0x7fd40c1a9300 con 0x7fd40c101110 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd40c1ab760 con 0x7fd40c101110 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd404001870 con 0x7fd40c101110 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1120828086 0 0) 0x7fd40c1aa520 con 0x7fd40c10d940 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 >> v1:172.21.15.134:6789/0 conn(0x7fd40c101110 legacy=0x7fd40c0fe890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 >> v1:172.21.15.28:6789/0 conn(0x7fd40c109da0 legacy=0x7fd40c1a41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c1ac9a0 con 0x7fd40c10d940 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c1ab990 con 0x7fd40c10d940 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd4117ba700 10 client.57620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd40c1abe50 con 0x7fd40c10d940 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd4117ba700 1 librados: init done 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd3fc001590 con 0x7fd40c10d940 2024-03-30T15:00:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.640+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd3fc004900 con 0x7fd40c10d940 2024-03-30T15:00:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.641+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ec005120 con 0x7fd40c10d940 2024-03-30T15:00:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.641+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd3fc003680 con 0x7fd40c10d940 2024-03-30T15:00:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(295..295 src has 53..295) ==== 9731+0+0 (unknown 3286005627 0 0) 0x7fd3fc088830 con 0x7fd40c10d940 2024-03-30T15:00:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 10 client.57620.objecter ms_dispatch 0x7fd40c001550 osd_map(295..295 src has 53..295) 2024-03-30T15:00:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 3 client.57620.objecter handle_osd_map got epochs [295,295] > 0 2024-03-30T15:00:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 3 client.57620.objecter handle_osd_map decoding full epoch 295 2024-03-30T15:00:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 20 client.57620.objecter dump_active .. 0 homeless 2024-03-30T15:00:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.642+0000 7fd403fff700 10 client.57620.objecter ms_handle_connect 0x7fd3f8070520 2024-03-30T15:00:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.646+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd3fc055420 con 0x7fd40c10d940 2024-03-30T15:00:59.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:00:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:59.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: pgmap v380: 298 pgs: 40 unknown, 258 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi028-92979-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: osdmap e294: 8 total, 8 up, 8 in 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:00:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:00:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]: dispatch 2024-03-30T15:00:59.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:00:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:00:59.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:00:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:00:59.865+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd3ec005410 con 0x7fd40c10d940 2024-03-30T15:00:59.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:00:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:00:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:00:59.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:00.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:00.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:00.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.246+0000 7fd403fff700 1 -- 172.21.15.28:0/2780414744 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v296) ==== 97+0+0 (unknown 2695371182 0 0) 0x7fd3fc058e70 con 0x7fd40c10d940 2024-03-30T15:01:00.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.250+0000 7fd4117ba700 10 librados: watch_flush enter 2024-03-30T15:01:00.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.250+0000 7fd4117ba700 10 librados: watch_flush exit 2024-03-30T15:01:00.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.250+0000 7fd4117ba700 20 client.57620.objecter shutdown clearing up homeless session... 2024-03-30T15:01:00.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.250+0000 7fd4117ba700 10 client.57620.objecter successfully canceled tick 2024-03-30T15:01:00.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.250+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd3f8070520 legacy=0x7fd3f80729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:00.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 >> v1:172.21.15.28:6790/0 conn(0x7fd40c10d940 legacy=0x7fd40c1a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:00.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 shutdown_connections 2024-03-30T15:01:00.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 >> 172.21.15.28:0/2780414744 conn(0x7fd40c0fc9c0 msgr2=0x7fd40c06ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:00.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 shutdown_connections 2024-03-30T15:01:00.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 -- 172.21.15.28:0/2780414744 wait complete. 2024-03-30T15:01:00.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:00.251+0000 7fd4117ba700 1 librados: shutdown 2024-03-30T15:01:00.254 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:01:00.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:00.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:00.327 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]': finished 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2780414744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.328 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:00 smithi134 ceph-mon[33714]: from='client.57620 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:00.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:00.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]': finished 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2780414744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[41614]: from='client.57620 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-33", "mode": "writeback"}]': finished 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: osdmap e295: 8 total, 8 up, 8 in 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2780414744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:00 smithi028 ceph-mon[29876]: from='client.57620 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:00.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:00.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:00.568 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:00.569 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:00.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:00.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:00.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:00.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:01.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:01.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:01.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:01.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:01:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:01.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:01.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:01.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:01.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: pgmap v383: 290 pgs: 2 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:01.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.57620 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:01.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T15:01:01.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:01.432 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:01.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.576 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:01.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:01.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:01.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: pgmap v383: 290 pgs: 2 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.57620 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T15:01:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: pgmap v383: 290 pgs: 2 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi028-92979-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.57620 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: osdmap e296: 8 total, 8 up, 8 in 2024-03-30T15:01:01.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]: dispatch 2024-03-30T15:01:01.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]: dispatch 2024-03-30T15:01:01.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:01.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:01.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:01.958 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:01.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:02.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:02.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:02.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:02.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:02.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:02.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:02.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:02.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:02.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:02.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:01:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T15:01:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:02.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-30T15:01:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-33"}]': finished 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3887675355' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi028-92999-49"}]': finished 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: osdmap e297: 8 total, 8 up, 8 in 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:02.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi028-92999-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: osdmap e298: 8 total, 8 up, 8 in 2024-03-30T15:01:02.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:02.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:02.840 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:02.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.840+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2556594118 >> v1:172.21.15.28:6790/0 conn(0x7f7f6410aed0 legacy=0x7f7f6410d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:02.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.840+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2556594118 shutdown_connections 2024-03-30T15:01:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.840+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2556594118 >> 172.21.15.28:0/2556594118 conn(0x7f7f640fdaf0 msgr2=0x7f7f640fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.840+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2556594118 shutdown_connections 2024-03-30T15:01:02.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.840+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2556594118 wait complete. 2024-03-30T15:01:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 librados: starting msgr at 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 librados: starting objecter 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 Processor -- start 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 -- start start 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 librados: setting wanted keys 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 librados: calling monclient init 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7f641a70e0 con 0x7f7f6410aed0 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7f641a8300 con 0x7f7f6410ea70 2024-03-30T15:01:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.841+0000 7f7f68ea3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7f641a9540 con 0x7f7f64102240 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f6259c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7f64102240 0x7f7f64107580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60932/0 (socket says 172.21.15.28:60932) 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f6259c700 1 -- 172.21.15.28:0/2369509838 learned_addr learned my addr 172.21.15.28:0/2369509838 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter ms_handle_connect 0x7f7f64102240 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter ms_handle_connect 0x7f7f6410aed0 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter ms_handle_connect 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51576269 0 0) 0x7f7f641a9540 con 0x7f7f64102240 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f4c0031f0 con 0x7f7f64102240 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798949609 0 0) 0x7f7f641a70e0 con 0x7f7f6410aed0 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f641a9540 con 0x7f7f6410aed0 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112961941 0 0) 0x7f7f641a8300 con 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f641a70e0 con 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2661792885 0 0) 0x7f7f4c0031f0 con 0x7f7f64102240 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f641a8300 con 0x7f7f64102240 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7f580012d0 con 0x7f7f64102240 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118371964 0 0) 0x7f7f641a70e0 con 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f4c0031f0 con 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7f5c001bb0 con 0x7f7f6410ea70 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3375101878 0 0) 0x7f7f641a9540 con 0x7f7f6410aed0 2024-03-30T15:01:02.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f641a70e0 con 0x7f7f6410aed0 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7f54001910 con 0x7f7f6410aed0 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 873924986 0 0) 0x7f7f641a8300 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.134:6789/0 conn(0x7f7f6410ea70 legacy=0x7f7f641a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6789/0 conn(0x7f7f6410aed0 legacy=0x7f7f641a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f641aa780 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.842+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f641a7310 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f68ea3700 10 client.57632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f641a77d0 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f68ea3700 1 librados: init done 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7f58001470 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7f58005c90 con 0x7f7f64102240 2024-03-30T15:01:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.843+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7f44005100 con 0x7f7f64102240 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.844+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7f58003930 con 0x7f7f64102240 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(298..298 src has 53..298) ==== 9233+0+0 (unknown 1469776745 0 0) 0x7f7f58089990 con 0x7f7f64102240 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 osd_map(298..298 src has 53..298) 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 3 client.57632.objecter handle_osd_map got epochs [298,298] > 0 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 3 client.57632.objecter handle_osd_map decoding full epoch 298 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 20 client.57632.objecter dump_active .. 0 homeless 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v298) ==== 54+0+16 (unknown 2854015103 0 2420000192) 0x7f7f58089da0 con 0x7f7f64102240 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f4c070370 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 1 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 1 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 0 1 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 4 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 10 client.57632.objecter _send_command 1 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f44004b50 con 0x7f7f44004730 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.845+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f44004730 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.846+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f44004b50 con 0x7f7f44004730 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.846+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.846+0000 7f7f537fe700 10 client.57632.objecter _finish_command 1 = system:0 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.846+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 4 2024-03-30T15:01:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.846+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 0 1 2024-03-30T15:01:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 2 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 2 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f440053e0 osd=0 3 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 0 2 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 4 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 10 client.57632.objecter _send_command 2 2024-03-30T15:01:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.863+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f44007b00 con 0x7f7f44004730 2024-03-30T15:01:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f44007b00 con 0x7f7f44004730 2024-03-30T15:01:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f537fe700 10 client.57632.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 4 2024-03-30T15:01:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 0 2 2024-03-30T15:01:02.867 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 1 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 4 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 10 client.57632.objecter _send_command 3 2024-03-30T15:01:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.865+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7f44002200 con 0x7f7f440042f0 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f440042f0 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f44002200 con 0x7f7f440042f0 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 10 client.57632.objecter _finish_command 3 = system:0 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 4 2024-03-30T15:01:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.866+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 1 3 2024-03-30T15:01:02.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 4 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 4 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44004180 osd=1 3 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 1 4 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 4 2024-03-30T15:01:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 10 client.57632.objecter _send_command 4 2024-03-30T15:01:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.884+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f44001600 con 0x7f7f440042f0 2024-03-30T15:01:02.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f44001600 con 0x7f7f440042f0 2024-03-30T15:01:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 10 client.57632.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 4 2024-03-30T15:01:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 1 4 2024-03-30T15:01:02.979 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:02.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 5 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 5 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 2 5 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 4 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 10 client.57632.objecter _send_command 5 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7f44015600 con 0x7f7f44011af0 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.886+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f44011af0 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.887+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f44015600 con 0x7f7f44011af0 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.887+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.887+0000 7f7f537fe700 10 client.57632.objecter _finish_command 5 = system:0 2024-03-30T15:01:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.887+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 4 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.887+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 2 5 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 6 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 6 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44008480 osd=2 3 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 2 6 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 4 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 10 client.57632.objecter _send_command 6 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.904+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f44007fc0 con 0x7f7f44011af0 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.905+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f44007fc0 con 0x7f7f44011af0 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.905+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.905+0000 7f7f537fe700 10 client.57632.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.905+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 4 2024-03-30T15:01:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.905+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 2 6 2024-03-30T15:01:02.982 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 7 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 7 2024-03-30T15:01:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 3 7 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 4 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 10 client.57632.objecter _send_command 7 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7f4401a440 con 0x7f7f44016930 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f44016930 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f4401a440 con 0x7f7f44016930 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 10 client.57632.objecter _finish_command 7 = system:0 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 4 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.906+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 3 7 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 8 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 8 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400cd30 osd=3 3 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 3 8 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 4 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 10 client.57632.objecter _send_command 8 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.923+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f4400c870 con 0x7f7f44016930 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f4400c870 con 0x7f7f44016930 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f537fe700 10 client.57632.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 4 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 3 8 2024-03-30T15:01:02.985 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 9 2024-03-30T15:01:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 9 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 4 9 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 4 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 10 client.57632.objecter _send_command 9 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.925+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7f4401efa0 con 0x7f7f44016490 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f44016490 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f4401efa0 con 0x7f7f44016490 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 10 client.57632.objecter _finish_command 9 = system:0 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 4 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.926+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 4 9 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 10 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 10 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400cae0 osd=4 3 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 4 10 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 4 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 10 client.57632.objecter _send_command 10 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.943+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f4400c8f0 con 0x7f7f44016490 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f4400c8f0 con 0x7f7f44016490 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f537fe700 10 client.57632.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 4 2024-03-30T15:01:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 4 10 2024-03-30T15:01:02.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 11 2024-03-30T15:01:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 11 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 5 11 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 4 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 10 client.57632.objecter _send_command 11 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.945+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7f440206f0 con 0x7f7f4401adc0 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.946+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f4401adc0 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.947+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f440206f0 con 0x7f7f4401adc0 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.947+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.947+0000 7f7f537fe700 10 client.57632.objecter _finish_command 11 = system:0 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.947+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 4 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.947+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 5 11 2024-03-30T15:01:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 12 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 12 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f4400c930 osd=5 3 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 5 12 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 4 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 10 client.57632.objecter _send_command 12 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.963+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f440206f0 con 0x7f7f4401adc0 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f440206f0 con 0x7f7f4401adc0 2024-03-30T15:01:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f537fe700 10 client.57632.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 4 2024-03-30T15:01:02.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 5 12 2024-03-30T15:01:02.994 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:02.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:01:02.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 13 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 13 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 6 13 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 4 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 10 client.57632.objecter _send_command 13 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.965+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7f440278e0 con 0x7f7f4401b330 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f4401b330 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f440278e0 con 0x7f7f4401b330 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 10 client.57632.objecter _finish_command 13 = system:0 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 4 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.966+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 6 13 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 14 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 14 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44021870 osd=6 3 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 6 14 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 4 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 10 client.57632.objecter _send_command 14 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.983+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f44027ac0 con 0x7f7f4401b330 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f44027ac0 con 0x7f7f4401b330 2024-03-30T15:01:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f537fe700 10 client.57632.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 4 2024-03-30T15:01:02.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 6 14 2024-03-30T15:01:02.998 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 15 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 15 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 7 15 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 4 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 10 client.57632.objecter _send_command 15 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.985+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7f44028e50 con 0x7f7f440281e0 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 10 client.57632.objecter ms_handle_connect 0x7f7f440281e0 2024-03-30T15:01:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7f44028e50 con 0x7f7f440281e0 2024-03-30T15:01:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 10 client.57632.objecter _finish_command 15 = system:0 2024-03-30T15:01:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 4 2024-03-30T15:01:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:02.986+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 7 15 2024-03-30T15:01:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:03.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 10 client.57632.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign -1 16 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 20 client.57632.objecter _get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_remove -1 16 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 20 client.57632.objecter get_session s=0x7f7f44027ac0 osd=7 3 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 15 client.57632.objecter _session_command_op_assign 7 16 2024-03-30T15:01:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 4 2024-03-30T15:01:03.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 10 client.57632.objecter _send_command 16 2024-03-30T15:01:03.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.003+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7f440290b0 con 0x7f7f440281e0 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f537fe700 1 -- 172.21.15.28:0/2369509838 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7f440290b0 con 0x7f7f440281e0 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f537fe700 10 client.57632.objecter ms_dispatch 0x7f7f64001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f537fe700 10 client.57632.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f537fe700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 4 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f537fe700 15 client.57632.objecter _session_command_op_remove 7 16 2024-03-30T15:01:03.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 librados: watch_flush enter 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 librados: watch_flush exit 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.0 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6801/3394323575 conn(0x7f7f44004730 legacy=0x7f7f44002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f440053e0 osd=0 1 2024-03-30T15:01:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.1 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6805/1850313547 conn(0x7f7f440042f0 legacy=0x7f7f4400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44004180 osd=1 1 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.2 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6809/3451325043 conn(0x7f7f44011af0 legacy=0x7f7f44013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f62d9d700 1 -- 172.21.15.28:0/2369509838 reap_dead start 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44008480 osd=2 1 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.3 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6813/1506157250 conn(0x7f7f44016930 legacy=0x7f7f44018d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cd30 osd=3 1 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.4 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.134:6800/3063775514 conn(0x7f7f44016490 legacy=0x7f7f4401b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400cae0 osd=4 1 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.5 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.134:6804/3668151179 conn(0x7f7f4401adc0 legacy=0x7f7f4401efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f4400c930 osd=5 1 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.6 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.134:6808/2135224194 conn(0x7f7f4401b330 legacy=0x7f7f44026070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44021870 osd=6 1 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter close_session for osd.7 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.134:6812/3758190298 conn(0x7f7f440281e0 legacy=0x7f7f44028670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter put_session s=0x7f7f44027ac0 osd=7 1 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 20 client.57632.objecter shutdown clearing up homeless session... 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 10 client.57632.objecter successfully canceled tick 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f62d9d700 1 -- 172.21.15.28:0/2369509838 reap_dead start 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7f4c070370 legacy=0x7f7f4c072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.005+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> v1:172.21.15.28:6790/0 conn(0x7f7f64102240 legacy=0x7f7f64107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.006+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 shutdown_connections 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.006+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 >> 172.21.15.28:0/2369509838 conn(0x7f7f640fdaf0 msgr2=0x7f7f64107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.006+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 shutdown_connections 2024-03-30T15:01:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.006+0000 7f7f68ea3700 1 -- 172.21.15.28:0/2369509838 wait complete. 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:03.006+0000 7f7f68ea3700 1 librados: shutdown 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:03.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:03.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:03.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:03.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:03.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:03.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:03.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:03.491 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:01:03.492 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:01:03.492 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:01:03.492 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:01:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:03.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:03.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:03.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: pgmap v386: 297 pgs: 8 unknown, 1 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2369509838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T15:01:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:03.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: pgmap v386: 297 pgs: 8 unknown, 1 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2369509838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:03.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: pgmap v386: 297 pgs: 8 unknown, 1 peering, 288 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2369509838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: osdmap e299: 8 total, 8 up, 8 in 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]: dispatch 2024-03-30T15:01:03.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:03.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:03.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:03.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:04.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:04.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:01:04.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:04.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:04.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:04.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:04.362 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:04.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:04.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:04.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:04.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:04.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:04.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:04.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:04.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.779+0000 7fa7603ac700 1 -- 172.21.15.28:0/776184417 >> v1:172.21.15.28:6790/0 conn(0x7fa75810eb50 legacy=0x7fa758111010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.779+0000 7fa7603ac700 1 -- 172.21.15.28:0/776184417 shutdown_connections 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.779+0000 7fa7603ac700 1 -- 172.21.15.28:0/776184417 >> 172.21.15.28:0/776184417 conn(0x7fa758077370 msgr2=0x7fa7580fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.779+0000 7fa7603ac700 1 -- 172.21.15.28:0/776184417 shutdown_connections 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.779+0000 7fa7603ac700 1 -- 172.21.15.28:0/776184417 wait complete. 2024-03-30T15:01:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 librados: starting msgr at 2024-03-30T15:01:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 librados: starting objecter 2024-03-30T15:01:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 Processor -- start 2024-03-30T15:01:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 -- start start 2024-03-30T15:01:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 librados: setting wanted keys 2024-03-30T15:01:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 librados: calling monclient init 2024-03-30T15:01:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa7581a7240 con 0x7fa75810eb50 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa7581a8460 con 0x7fa75810ae80 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.780+0000 7fa7603ac700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa7581a96a0 con 0x7fa758107000 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa75e148700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa758107000 0x7fa7580ff600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60960/0 (socket says 172.21.15.28:60960) 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa75e148700 1 -- 172.21.15.28:0/334941298 learned_addr learned my addr 172.21.15.28:0/334941298 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter ms_handle_connect 0x7fa75810eb50 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter ms_handle_connect 0x7fa758107000 2024-03-30T15:01:04.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter ms_handle_connect 0x7fa75810ae80 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205539629 0 0) 0x7fa7581a7240 con 0x7fa75810eb50 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7440039c0 con 0x7fa75810eb50 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937152010 0 0) 0x7fa7581a96a0 con 0x7fa758107000 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7581a7240 con 0x7fa758107000 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361856073 0 0) 0x7fa7581a8460 con 0x7fa75810ae80 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7581a96a0 con 0x7fa75810ae80 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3396178101 0 0) 0x7fa7440039c0 con 0x7fa75810eb50 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7581a8460 con 0x7fa75810eb50 2024-03-30T15:01:04.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa750001720 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2940649150 0 0) 0x7fa7581a7240 con 0x7fa758107000 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7440039c0 con 0x7fa758107000 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa754001890 con 0x7fa758107000 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1130670418 0 0) 0x7fa7581a8460 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 >> v1:172.21.15.28:6790/0 conn(0x7fa758107000 legacy=0x7fa7580ff600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 >> v1:172.21.15.134:6789/0 conn(0x7fa75810ae80 legacy=0x7fa7581a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.781+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7581aa8e0 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7581a98d0 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa7603ac700 10 client.24468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7581a9d90 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa7603ac700 1 librados: init done 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa750001f30 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.782+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa750005050 con 0x7fa75810eb50 2024-03-30T15:01:04.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.783+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73c005120 con 0x7fa75810eb50 2024-03-30T15:01:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.783+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa75001b910 con 0x7fa75810eb50 2024-03-30T15:01:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(300..300 src has 53..300) ==== 9708+0+0 (unknown 2283462404 0 0) 0x7fa750088f60 con 0x7fa75810eb50 2024-03-30T15:01:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 10 client.24468.objecter ms_dispatch 0x7fa758001550 osd_map(300..300 src has 53..300) 2024-03-30T15:01:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 3 client.24468.objecter handle_osd_map got epochs [300,300] > 0 2024-03-30T15:01:04.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 3 client.24468.objecter handle_osd_map decoding full epoch 300 2024-03-30T15:01:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 20 client.24468.objecter dump_active .. 0 homeless 2024-03-30T15:01:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.784+0000 7fa74f7fe700 10 client.24468.objecter ms_handle_connect 0x7fa744070410 2024-03-30T15:01:04.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:04.789+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa750055b60 con 0x7fa75810eb50 2024-03-30T15:01:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:04.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:04.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:04.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:05.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.008+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7fa73c005410 con 0x7fa75810eb50 2024-03-30T15:01:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:05.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.218+0000 7fa74f7fe700 1 -- 172.21.15.28:0/334941298 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v301) ==== 129+0+0 (unknown 1091170535 0 0) 0x7fa7500595b0 con 0x7fa75810eb50 2024-03-30T15:01:05.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.222+0000 7fa7603ac700 10 librados: watch_flush enter 2024-03-30T15:01:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.222+0000 7fa7603ac700 10 librados: watch_flush exit 2024-03-30T15:01:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.222+0000 7fa7603ac700 20 client.24468.objecter shutdown clearing up homeless session... 2024-03-30T15:01:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.222+0000 7fa7603ac700 10 client.24468.objecter successfully canceled tick 2024-03-30T15:01:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.223+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa744070410 legacy=0x7fa7440728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.223+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 >> v1:172.21.15.28:6789/0 conn(0x7fa75810eb50 legacy=0x7fa7581a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.223+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 shutdown_connections 2024-03-30T15:01:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.223+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 >> 172.21.15.28:0/334941298 conn(0x7fa758077370 msgr2=0x7fa758111e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.227+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 shutdown_connections 2024-03-30T15:01:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.227+0000 7fa7603ac700 1 -- 172.21.15.28:0/334941298 wait complete. 2024-03-30T15:01:05.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:05.227+0000 7fa7603ac700 1 librados: shutdown 2024-03-30T15:01:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 1 2024-03-30T15:01:05.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: pgmap v389: 289 pgs: 22 creating+peering, 10 unknown, 257 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:05.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: pgmap v389: 289 pgs: 22 creating+peering, 10 unknown, 257 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:05.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:05.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: pgmap v389: 289 pgs: 22 creating+peering, 10 unknown, 257 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:05.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi028-92999-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:05.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/720214906' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi028-92979-40"}]': finished 2024-03-30T15:01:05.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:05.382 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: osdmap e300: 8 total, 8 up, 8 in 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:05.383 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:05.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:05.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:05.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:05.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.780 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:05.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:05.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:05.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:05.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:05.911 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:01:05.911 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:01:05.911 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:01:05.911 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:01:05.911 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:01:06.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:06.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:06.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:06.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:06.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:06.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:06.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:06.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:01:06.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:06.530 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.531 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.532 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.532 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:06.532 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.532 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.532 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:06.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:06.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi028-92979-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/334941298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:06.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: osdmap e301: 8 total, 8 up, 8 in 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-30T15:01:06.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:06.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:01:06.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:06.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:06.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:06.798 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- 172.21.15.28:0/1387416212 >> v1:172.21.15.28:6789/0 conn(0x7f25ec102250 legacy=0x7f25ec102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- 172.21.15.28:0/1387416212 shutdown_connections 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- 172.21.15.28:0/1387416212 >> 172.21.15.28:0/1387416212 conn(0x7f25ec0fdb20 msgr2=0x7f25ec0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- 172.21.15.28:0/1387416212 shutdown_connections 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- 172.21.15.28:0/1387416212 wait complete. 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 librados: starting msgr at 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 librados: starting objecter 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 Processor -- start 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 -- start start 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 librados: setting wanted keys 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.824+0000 7f25f3a5c700 1 librados: calling monclient init 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25f3a5c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f25ec1a9370 con 0x7f25ec102250 2024-03-30T15:01:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25f3a5c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f25ec1aa590 con 0x7f25ec10ea80 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25f3a5c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f25ec1ab7d0 con 0x7f25ec10aee0 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25f0ff7700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f25ec10aee0 0x7f25ec1a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60984/0 (socket says 172.21.15.28:60984) 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25f0ff7700 1 -- 172.21.15.28:0/3334596921 learned_addr learned my addr 172.21.15.28:0/3334596921 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter ms_handle_connect 0x7f25ec10aee0 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter ms_handle_connect 0x7f25ec102250 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1241347189 0 0) 0x7f25ec1ab7d0 con 0x7f25ec10aee0 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25d80031f0 con 0x7f25ec10aee0 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921003258 0 0) 0x7f25ec1a9370 con 0x7f25ec102250 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25ec1ab7d0 con 0x7f25ec102250 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter ms_handle_connect 0x7f25ec10ea80 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224271599 0 0) 0x7f25ec1aa590 con 0x7f25ec10ea80 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25ec1a9370 con 0x7f25ec10ea80 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2424126349 0 0) 0x7f25d80031f0 con 0x7f25ec10aee0 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25ec1aa590 con 0x7f25ec10aee0 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25dc001870 con 0x7f25ec10aee0 2024-03-30T15:01:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1436753250 0 0) 0x7f25ec1ab7d0 con 0x7f25ec102250 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25d80031f0 con 0x7f25ec102250 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.825+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25e8004590 con 0x7f25ec102250 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4220186276 0 0) 0x7f25ec1a9370 con 0x7f25ec10ea80 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25ec1ab7d0 con 0x7f25ec10ea80 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25e40013f0 con 0x7f25ec10ea80 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 195576067 0 0) 0x7f25ec1aa590 con 0x7f25ec10aee0 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 >> v1:172.21.15.134:6789/0 conn(0x7f25ec10ea80 legacy=0x7f25ec1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 >> v1:172.21.15.28:6789/0 conn(0x7f25ec102250 legacy=0x7f25ec1075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25ec1aca10 con 0x7f25ec10aee0 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25ec1a95a0 con 0x7f25ec10aee0 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25f3a5c700 10 client.57650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f25ec1a9a60 con 0x7f25ec10aee0 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25f3a5c700 1 librados: init done 2024-03-30T15:01:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f25dc003b40 con 0x7f25ec10aee0 2024-03-30T15:01:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.826+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25dc005c20 con 0x7f25ec10aee0 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.827+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d0005150 con 0x7f25ec10aee0 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.827+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f25dc01c2c0 con 0x7f25ec10aee0 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(302..302 src has 53..302) ==== 9737+0+0 (unknown 2100960346 0 0) 0x7f25dc089c40 con 0x7f25ec10aee0 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 10 client.57650.objecter ms_dispatch 0x7f25ec001550 osd_map(302..302 src has 53..302) 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 3 client.57650.objecter handle_osd_map got epochs [302,302] > 0 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 3 client.57650.objecter handle_osd_map decoding full epoch 302 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 20 client.57650.objecter dump_active .. 0 homeless 2024-03-30T15:01:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.828+0000 7f25e27fc700 10 client.57650.objecter ms_handle_connect 0x7f25d8070530 2024-03-30T15:01:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:06.833+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f25dc056800 con 0x7f25ec10aee0 2024-03-30T15:01:06.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:06.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:06.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:07.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:07.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:07.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.048+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f25d0005440 con 0x7f25ec10aee0 2024-03-30T15:01:07.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:07.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:07.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.229+0000 7f25e27fc700 1 -- 172.21.15.28:0/3334596921 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v303) ==== 91+0+0 (unknown 874601106 0 0) 0x7f25dc05a250 con 0x7f25ec10aee0 2024-03-30T15:01:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.233+0000 7f25f3a5c700 10 librados: watch_flush enter 2024-03-30T15:01:07.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.233+0000 7f25f3a5c700 10 librados: watch_flush exit 2024-03-30T15:01:07.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.233+0000 7f25f3a5c700 20 client.57650.objecter shutdown clearing up homeless session... 2024-03-30T15:01:07.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.233+0000 7f25f3a5c700 10 client.57650.objecter successfully canceled tick 2024-03-30T15:01:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.235+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 >> v1:172.21.15.28:6800/3011431093 conn(0x7f25d8070530 legacy=0x7f25d80729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.235+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 >> v1:172.21.15.28:6790/0 conn(0x7f25ec10aee0 legacy=0x7f25ec1a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.235+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 shutdown_connections 2024-03-30T15:01:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.235+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 >> 172.21.15.28:0/3334596921 conn(0x7f25ec0fdb20 msgr2=0x7f25ec1116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.237+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 shutdown_connections 2024-03-30T15:01:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.237+0000 7f25f3a5c700 1 -- 172.21.15.28:0/3334596921 wait complete. 2024-03-30T15:01:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:07.238+0000 7f25f3a5c700 1 librados: shutdown 2024-03-30T15:01:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:01:07.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:07.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:07.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:07.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:07.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:07.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:07.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:07.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: pgmap v392: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:07.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3334596921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.57650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]': finished 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.57650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: pgmap v392: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3334596921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.57650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]': finished 2024-03-30T15:01:07.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:07.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.57650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:07.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T15:01:07.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.784 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:07.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:07.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:07.913 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: pgmap v392: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:07.913 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:07.913 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: osdmap e302: 8 total, 8 up, 8 in 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:07.914 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3334596921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.57650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi028-92979-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-35", "mode": "writeback"}]': finished 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.57650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: osdmap e303: 8 total, 8 up, 8 in 2024-03-30T15:01:07.915 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1398925' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.916 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]: dispatch 2024-03-30T15:01:07.916 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:07.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:08.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:08.060 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:08.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:08.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:08.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:08.400 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:08.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:08.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:08.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24838.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 202.21707387399999, 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8658181240000005, 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.558 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24838", 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3970114250", 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334486+0000", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.784e-06 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334493+0000", 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8900000000000001e-06 2024-03-30T15:01:08.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334501+0000", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5739999999999996e-06 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334512+0000", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0152e-05 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.342482+0000", 2024-03-30T15:01:08.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079708320000000006 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.342492+0000", 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3489999999999998e-06 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111380+0000", 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7688878199999998 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111508+0000", 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012799400000000001 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111645+0000", 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013785199999999999 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.119441+0000", 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0077957549999999997 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.184335+0000", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.064893748000000001 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200217+0000", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015881663000000001 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200263+0000", 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6193999999999999e-05 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200302+0000", 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9517000000000001e-05 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24877.0:2 18.16 18:6ceb5ccc:test-rados-api-smithi028-93253-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.456833467, 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.190799929, 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:01:08.565 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24877", 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2268236432", 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094727+0000", 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1579999999999999e-06 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094733+0000", 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0410000000000003e-06 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094744+0000", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1019000000000001e-05 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094755+0000", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0852e-05 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.095307+0000", 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055243399999999998 2024-03-30T15:01:08.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.095317+0000", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0013000000000001e-05 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.231293+0000", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1359758339999999 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.231407+0000", 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011398 2024-03-30T15:01:08.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285525+0000", 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.054117598000000003 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24964.0:2 30.1e 30:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.39620683300001, 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069668294000000006, 2024-03-30T15:01:08.570 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24964", 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2019932805", 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155353+0000", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.669e-06 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155363+0000", 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9850000000000001e-06 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158533+0000", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031701979999999999 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158546+0000", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3341e-05 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158606+0000", 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9194999999999998e-05 2024-03-30T15:01:08.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158724+0000", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011867 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160432+0000", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017071880000000001 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.171186+0000", 2024-03-30T15:01:08.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01075456 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.171405+0000", 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021920300000000001 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.224916+0000", 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053511156999999997 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.224980+0000", 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3842000000000003e-05 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.225020+0000", 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9286000000000003e-05 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24853.0:2 14.d 14:b0550ca8:test-rados-api-smithi028-93116-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:01:08.576 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.390737844, 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092514707000000002, 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24853", 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:01:08.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160822+0000", 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.857e-06 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160829+0000", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.781e-06 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160837+0000", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9640000000000003e-06 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160847+0000", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0117e-05 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160901+0000", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4135999999999998e-05 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160909+0000", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8890000000000002e-06 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175479+0000", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014569598 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175602+0000", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000122986 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175791+0000", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000189159 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.185216+0000", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0094253619999999996 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.223454+0000", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038237762000000002 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253285+0000", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.029830768000000001 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253311+0000", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6393e-05 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253335+0000", 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3935e-05 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24958.0:3 29.f 29:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.29616139199999, 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.040191584, 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.583 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24958", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/925060148", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255398+0000", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.251e-06 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255406+0000", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3119999999999994e-06 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255415+0000", 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3489999999999992e-06 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255422+0000", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1019999999999999e-06 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255533+0000", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011099600000000001 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255540+0000", 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4030000000000001e-06 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267675+0000", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0121351670000001 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267969+0000", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029406199999999999 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.268118+0000", 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014865700000000001 2024-03-30T15:01:08.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.270968+0000", 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028503249999999999 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274869+0000", 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0039004489999999998 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295496+0000", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020626938000000001 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295547+0000", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1697000000000003e-05 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295588+0000", 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0876e-05 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.589 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:14 11.a 11:56696d74:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.28386482900001, 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0178625910000001, 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:01:08.590 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267694+0000", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0100000000000004e-07 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267698+0000", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0899999999999998e-06 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267706+0000", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7330000000000003e-06 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267711+0000", 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0710000000000002e-06 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267749+0000", 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7834999999999999e-05 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267754+0000", 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.13e-06 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.226737+0000", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95898334299999999 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.226877+0000", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014019100000000001 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.227059+0000", 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018114199999999999 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.254281+0000", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027221973 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.254892+0000", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00061162599999999999 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285470+0000", 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030577887000000002 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285516+0000", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5532999999999999e-05 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285556+0000", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0235999999999999e-05 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24940.0:86 26.11 26:8f09172f:test-rados-api-smithi028-93750-14::testobj:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:01:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.21936912499999, 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94252751899999998, 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24940", 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 86 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:01:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332191+0000", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0600000000000002e-06 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332196+0000", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.977e-06 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332204+0000", 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3489999999999992e-06 2024-03-30T15:01:08.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332220+0000", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5622999999999999e-05 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332276+0000", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5619999999999999e-05 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332282+0000", 2024-03-30T15:01:08.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1589999999999998e-06 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274618+0000", 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94233667499999996 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274716+0000", 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8055999999999998e-05 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24856.0:7 15.1 15:8386110d:test-rados-api-smithi028-93086-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 192.385083831, 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.100957805, 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24856", 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/13297926", 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 7 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166476+0000", 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.967e-06 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166483+0000", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8109999999999997e-06 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166493+0000", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0135e-05 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166503+0000", 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0305999999999999e-05 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166635+0000", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013131800000000001 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166735+0000", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010037799999999999 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267432+0000", 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.10069689 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.605 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:1323 11.a 11:561c3777:test-rados-api-smithi028-93257-15::foo:head [setxattr a637 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 191.20870480400001, 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080122802000000007, 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1323 2024-03-30T15:01:08.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342854+0000", 2024-03-30T15:01:08.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1230000000000001e-06 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342860+0000", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8989999999999998e-06 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.344152+0000", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001291883 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.344661+0000", 2024-03-30T15:01:08.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00050922000000000001 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.354837+0000", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01017559 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.354946+0000", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010925900000000001 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.355269+0000", 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032268200000000003 2024-03-30T15:01:08.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.360362+0000", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050931029999999999 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.405152+0000", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044789735999999997 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422873+0000", 2024-03-30T15:01:08.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017721447000000001 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422931+0000", 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8053999999999997e-05 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422976+0000", 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4805999999999998e-05 2024-03-30T15:01:08.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:8 13.1 13:848d865e:test-rados-api-smithi028-93525-4::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.76223303800001, 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092471127, 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 8 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789327+0000", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7349999999999999e-06 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789333+0000", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5749999999999997e-06 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789343+0000", 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9259999999999995e-06 2024-03-30T15:01:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789353+0000", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4460000000000007e-06 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789406+0000", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3009e-05 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789412+0000", 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9329999999999998e-06 2024-03-30T15:01:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.827838+0000", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038426244999999998 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.827978+0000", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014053500000000001 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.828134+0000", 2024-03-30T15:01:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015600799999999999 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.832326+0000", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041916180000000003 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.867355+0000", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.035029326999999999 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881695+0000", 2024-03-30T15:01:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014340031 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881750+0000", 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4085999999999997e-05 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881796+0000", 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6653000000000002e-05 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:10 13.1 13:848d865e:test-rados-api-smithi028-93525-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.74758595200001, 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14530184500000001, 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 10 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803975+0000", 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.52e-06 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803985+0000", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.08e-05 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804006+0000", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0316e-05 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804019+0000", 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2860000000000001e-05 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804138+0000", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118913 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804151+0000", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3857e-05 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.843765+0000", 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.039613576999999997 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.843864+0000", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9179999999999996e-05 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.844054+0000", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019031300000000001 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.850640+0000", 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065853350000000003 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.896075+0000", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.045435206999999998 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949167+0000", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053091987 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949224+0000", 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6543999999999999e-05 2024-03-30T15:01:08.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949274+0000", 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0436000000000003e-05 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.32242.0:2 120.6 120:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 166.50159019700001, 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.23603611299999999, 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.627 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.32242", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/680296349", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049970+0000", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.154e-06 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049978+0000", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3329999999999997e-06 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.050672+0000", 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00069341499999999996 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.053009+0000", 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023367040000000002 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.059224+0000", 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062157419999999998 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.059235+0000", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0664e-05 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.078870+0000", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.019634743 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.078992+0000", 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012259900000000001 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.079132+0000", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013983100000000001 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.081146+0000", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020137979999999998 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.111454+0000", 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030308234999999999 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285896+0000", 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.17444207 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285955+0000", 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8797e-05 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286004+0000", 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9027999999999999e-05 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34738.0:2 124.0 124:0384ab8e:::oid:head [writefull 0~3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 165.25947281500001, 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1976773839999999, 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.634 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34738", 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3778949751", 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292088+0000", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3590000000000002e-06 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292095+0000", 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9450000000000004e-06 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292106+0000", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1911e-05 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292115+0000", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6910000000000003e-06 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292174+0000", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8742000000000001e-05 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292182+0000", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7950000000000008e-06 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.480809+0000", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1886268090000001 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.480965+0000", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000156176 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:01:08.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481066+0000", 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101111 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481431+0000", 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036508800000000001 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481637+0000", 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000206221 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489590+0000", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079524929999999997 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489668+0000", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8608000000000004e-05 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489763+0000", 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4435e-05 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.640 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.36958.0:2 131.0 131:0384ab8e:::oid:head [write 0~0,setxattr key1 (5) in=4b,omap-clear] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 161.702117413, 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.144649171, 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.36958", 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/962681329", 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.641 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849442+0000", 2024-03-30T15:01:08.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2240000000000001e-06 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849450+0000", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6520000000000006e-06 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849459+0000", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7499999999999998e-06 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849465+0000", 2024-03-30T15:01:08.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5010000000000004e-06 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849520+0000", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5136999999999997e-05 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849529+0000", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1339999999999993e-06 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967025+0000", 2024-03-30T15:01:08.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.117495906 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967148+0000", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012341300000000001 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967295+0000", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146317 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967911+0000", 2024-03-30T15:01:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00061612600000000004 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.968081+0000", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017055600000000001 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994027+0000", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025945373000000001 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994060+0000", 2024-03-30T15:01:08.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3176999999999998e-05 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994090+0000", 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9904999999999999e-05 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2079 11.a 11:5269a482:test-rados-api-smithi028-93257-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 159.64849693400001, 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13445506500000001, 2024-03-30T15:01:08.647 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2079 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903063+0000", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.747e-06 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903070+0000", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6710000000000002e-06 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903080+0000", 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0465999999999999e-05 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903087+0000", 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5150000000000003e-06 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903142+0000", 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5466000000000001e-05 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903275+0000", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000132578 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.972361+0000", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069086094000000001 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.983211+0000", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010850741000000001 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.984309+0000", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010978019999999999 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037413+0000", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053103996000000001 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037473+0000", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9632000000000001e-05 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037516+0000", 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3356999999999998e-05 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.42298.0:2 137.0 137:0384ab8e:::oid:head [append 0~64] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 157.24063234400001, 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.91052688900000001, 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.653 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.42298", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3149698539", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310928+0000", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8840000000000001e-06 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310934+0000", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2400000000000004e-06 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310962+0000", 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7712000000000002e-05 2024-03-30T15:01:08.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310971+0000", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3829999999999998e-06 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.338059+0000", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027088421000000001 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.338069+0000", 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8430000000000005e-06 2024-03-30T15:01:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213014+0000", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.87494432600000005 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213132+0000", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118268 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213274+0000", 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014212200000000001 2024-03-30T15:01:08.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.214845+0000", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015707869999999999 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.216399+0000", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001554366 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221386+0000", 2024-03-30T15:01:08.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049865179999999997 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221423+0000", 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7002000000000002e-05 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221453+0000", 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0017000000000001e-05 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35600.0:110 127.e 127:7056b563:test-rados-api-smithi028-93253-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 157.16924063900001, 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.082170402000000003, 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35600", 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 110 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382321+0000", 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5540000000000002e-06 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382332+0000", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0859999999999999e-05 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382359+0000", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6655000000000001e-05 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382380+0000", 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1384000000000001e-05 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382527+0000", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014746599999999999 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382828+0000", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030051500000000001 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.384285+0000", 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001456609 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.384354+0000", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9276999999999994e-05 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464212+0000", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.079858083999999996 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464285+0000", 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3312999999999994e-05 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464488+0000", 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000202685 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.53870.0:2 144.0 144:0384ab8e:::oid:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 154.208401528, 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.105832438, 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.665 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.53870", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2684867265", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:01:08.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343158+0000", 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.72e-06 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343165+0000", 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.72e-06 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343174+0000", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1290000000000002e-06 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343184+0000", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8700000000000004e-06 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343329+0000", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000145489 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343340+0000", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0159e-05 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427351+0000", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.084011002 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427475+0000", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012393300000000001 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:01:08.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427615+0000", 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014084500000000001 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428264+0000", 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064877700000000001 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.440421+0000", 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012156372 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448863+0000", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084419430000000004 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448933+0000", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0883999999999994e-05 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448989+0000", 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5594999999999999e-05 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.671 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.672 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.673 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49366.0:2 226.13 226:cbb9903e:::test_obj:head [writefull 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T15:01:08.673 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:01:08.673 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 76.356307113, 2024-03-30T15:01:08.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.161443155, 2024-03-30T15:01:08.674 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.674 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.674 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.674 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.674 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49366", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/796261573", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195254+0000", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8040000000000002e-06 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195266+0000", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2408999999999999e-05 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195279+0000", 2024-03-30T15:01:08.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2907e-05 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195292+0000", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3298999999999999e-05 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195630+0000", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033764299999999999 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195637+0000", 2024-03-30T15:01:08.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6730000000000003e-06 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.202689+0000", 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070526920000000002 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.202701+0000", 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1103e-05 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354226+0000", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.151525983 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354342+0000", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115528 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354464+0000", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012247000000000001 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:01:08.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354895+0000", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043055200000000001 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.355425+0000", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052977600000000003 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356598+0000", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011735140000000001 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356653+0000", 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4456000000000002e-05 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356694+0000", 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1346e-05 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49447.0:1 237.6s0 237:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.317208639999997, 2024-03-30T15:01:08.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012543746, 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49447", 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1989601426", 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234352+0000", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3429999999999998e-06 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234359+0000", 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5120000000000002e-06 2024-03-30T15:01:08.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234376+0000", 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6331999999999999e-05 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234387+0000", 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1476e-05 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.244898+0000", 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010510455 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.244901+0000", 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.382e-06 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.242735+0000", 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.997833796 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.242865+0000", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000130321 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.243228+0000", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036260699999999998 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.243478+0000", 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025050199999999999 2024-03-30T15:01:08.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.246817+0000", 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033390640000000001 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.246893+0000", 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5956000000000003e-05 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:01:08.687 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: 17.1 scrub starts 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: 17.1 scrub ok 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: 17.13 scrub starts 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: 17.13 scrub ok 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: pgmap v395: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:08.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[41614]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: 17.1 scrub starts 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: 17.1 scrub ok 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: 17.13 scrub starts 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: 17.13 scrub ok 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: pgmap v395: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:08.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:08 smithi028 ceph-mon[29876]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T15:01:08.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:08.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:08.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: 17.1 scrub starts 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: 17.1 scrub ok 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: 17.13 scrub starts 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: 17.13 scrub ok 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: pgmap v395: 296 pgs: 8 unknown, 1 clean+premerge+peered, 32 creating+peering, 255 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: from='client.57626 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi028-92999-50"}]': finished 2024-03-30T15:01:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:08 smithi134 ceph-mon[33714]: osdmap e304: 8 total, 8 up, 8 in 2024-03-30T15:01:08.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:08.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:08.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:09.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:09.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:09.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:09.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:09.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:09.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:09.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:09.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:09.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:01:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:09.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:09.632 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:01:09.632 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 17.9 scrub starts 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 17.9 scrub ok 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 101.1a scrub starts 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 101.1a scrub ok 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 17.0 scrub starts 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: 17.0 scrub ok 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1359925381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.633 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.57656 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.634 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.634 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.634 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 17.9 scrub starts 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 17.9 scrub ok 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 101.1a scrub starts 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 101.1a scrub ok 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 17.0 scrub starts 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: 17.0 scrub ok 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1359925381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.57656 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 17.9 scrub starts 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 17.9 scrub ok 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 101.1a scrub starts 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 101.1a scrub ok 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 17.0 scrub starts 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: 17.0 scrub ok 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: osdmap e305: 8 total, 8 up, 8 in 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:09.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1359925381' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.57656 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:09.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:09.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:09.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:09.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:09.858 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:09.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:09.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.969+0000 7f1b07701700 1 -- 172.21.15.28:0/3327876883 >> v1:172.21.15.28:6790/0 conn(0x7f1b00109dc0 legacy=0x7f1b0010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.969+0000 7f1b07701700 1 -- 172.21.15.28:0/3327876883 shutdown_connections 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.969+0000 7f1b07701700 1 -- 172.21.15.28:0/3327876883 >> 172.21.15.28:0/3327876883 conn(0x7f1b000fc9e0 msgr2=0x7f1b000fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.969+0000 7f1b07701700 1 -- 172.21.15.28:0/3327876883 shutdown_connections 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.969+0000 7f1b07701700 1 -- 172.21.15.28:0/3327876883 wait complete. 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 librados: starting msgr at 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 librados: starting objecter 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 Processor -- start 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 -- start start 2024-03-30T15:01:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 librados: setting wanted keys 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 librados: calling monclient init 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b001a7160 con 0x7f1b0010d960 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b001a8380 con 0x7f1b00101130 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b07701700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b001a95c0 con 0x7f1b00109dc0 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b04c9c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1b00109dc0 0x7f1b001a1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55868/0 (socket says 172.21.15.28:55868) 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b04c9c700 1 -- 172.21.15.28:0/137709776 learned_addr learned my addr 172.21.15.28:0/137709776 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1b05c9e700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b0010d960 0x7f1b001a5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33236/0 (socket says 172.21.15.28:33236) 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter ms_handle_connect 0x7f1b00109dc0 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter ms_handle_connect 0x7f1b00101130 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499144106 0 0) 0x7f1b001a95c0 con 0x7f1b00109dc0 2024-03-30T15:01:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1aec003aa0 con 0x7f1b00109dc0 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter ms_handle_connect 0x7f1b0010d960 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903414060 0 0) 0x7f1b001a8380 con 0x7f1b00101130 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b001a95c0 con 0x7f1b00101130 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.970+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982090060 0 0) 0x7f1b001a7160 con 0x7f1b0010d960 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b001a8380 con 0x7f1b0010d960 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 323833274 0 0) 0x7f1aec003aa0 con 0x7f1b00109dc0 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b001a7160 con 0x7f1b00109dc0 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1af00043f0 con 0x7f1b00109dc0 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3046194839 0 0) 0x7f1b001a95c0 con 0x7f1b00101130 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1aec003aa0 con 0x7f1b00101130 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1afc001840 con 0x7f1b00101130 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3403155372 0 0) 0x7f1b001a8380 con 0x7f1b0010d960 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b001a95c0 con 0x7f1b0010d960 2024-03-30T15:01:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1af8001b70 con 0x7f1b0010d960 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3421450310 0 0) 0x7f1b001a7160 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.134:6789/0 conn(0x7f1b00101130 legacy=0x7f1b00106480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6789/0 conn(0x7f1b0010d960 legacy=0x7f1b001a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b001aa800 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b001a85b0 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1b07701700 10 client.57662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b001a8a70 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1b07701700 1 librados: init done 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1af0004650 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.971+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1af0004e20 con 0x7f1b00109dc0 2024-03-30T15:01:09.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.972+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1ae4005100 con 0x7f1b00109dc0 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1af0001a30 con 0x7f1b00109dc0 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(305..305 src has 53..305) ==== 9673+0+0 (unknown 854921105 0 0) 0x7f1af0089d10 con 0x7f1b00109dc0 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 osd_map(305..305 src has 53..305) 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 3 client.57662.objecter handle_osd_map got epochs [305,305] > 0 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 3 client.57662.objecter handle_osd_map decoding full epoch 305 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 20 client.57662.objecter dump_active .. 0 homeless 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v305) ==== 54+0+16 (unknown 1285346396 0 2420000192) 0x7f1af008a0c0 con 0x7f1b00109dc0 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1aec070470 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 10 client.57662.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 1 2024-03-30T15:01:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 1 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 0 1 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 4 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 10 client.57662.objecter _send_command 1 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.973+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ae4004b50 con 0x7f1ae4004730 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae4004730 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae4004b50 con 0x7f1ae4004730 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 10 client.57662.objecter _finish_command 1 = system:0 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 4 2024-03-30T15:01:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.974+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 0 1 2024-03-30T15:01:09.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 10 client.57662.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 2 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 2 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae40053e0 osd=0 3 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 0 2 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 4 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 10 client.57662.objecter _send_command 2 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.992+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae40079f0 con 0x7f1ae4004730 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae40079f0 con 0x7f1ae4004730 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1af67fc700 10 client.57662.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 4 2024-03-30T15:01:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 0 2 2024-03-30T15:01:09.995 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 10 client.57662.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 3 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 3 2024-03-30T15:01:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 1 3 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 4 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 10 client.57662.objecter _send_command 3 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.993+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1ae4002180 con 0x7f1ae4004280 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae4004280 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae4002180 con 0x7f1ae4004280 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 10 client.57662.objecter _finish_command 3 = system:0 2024-03-30T15:01:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 4 2024-03-30T15:01:09.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:09.994+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 1 3 2024-03-30T15:01:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:10.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 10 client.57662.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 4 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 4 2024-03-30T15:01:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4004110 osd=1 3 2024-03-30T15:01:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 1 4 2024-03-30T15:01:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 4 2024-03-30T15:01:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 10 client.57662.objecter _send_command 4 2024-03-30T15:01:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.011+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae4001600 con 0x7f1ae4004280 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae4001600 con 0x7f1ae4004280 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 10 client.57662.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 4 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 1 4 2024-03-30T15:01:10.014 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 10 client.57662.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 5 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 5 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 2 5 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 4 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 10 client.57662.objecter _send_command 5 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1ae40110e0 con 0x7f1ae400d5d0 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.013+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae400d5d0 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.014+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae40110e0 con 0x7f1ae400d5d0 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.014+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.014+0000 7f1af67fc700 10 client.57662.objecter _finish_command 5 = system:0 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.014+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 4 2024-03-30T15:01:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.014+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 2 5 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 10 client.57662.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 6 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 6 2024-03-30T15:01:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4008370 osd=2 3 2024-03-30T15:01:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 2 6 2024-03-30T15:01:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 4 2024-03-30T15:01:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 10 client.57662.objecter _send_command 6 2024-03-30T15:01:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.031+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae4007eb0 con 0x7f1ae400d5d0 2024-03-30T15:01:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.032+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae4007eb0 con 0x7f1ae400d5d0 2024-03-30T15:01:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.032+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.032+0000 7f1af67fc700 10 client.57662.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.032+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 4 2024-03-30T15:01:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.032+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 2 6 2024-03-30T15:01:10.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 10 client.57662.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 7 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 7 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 3 7 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 4 2024-03-30T15:01:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 10 client.57662.objecter _send_command 7 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1ae4015f20 con 0x7f1ae4012410 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae4012410 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae4015f20 con 0x7f1ae4012410 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.033+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.034+0000 7f1af67fc700 10 client.57662.objecter _finish_command 7 = system:0 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.034+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 4 2024-03-30T15:01:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.034+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 3 7 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 10 client.57662.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 8 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 8 2024-03-30T15:01:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400cc20 osd=3 3 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 3 8 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 4 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 10 client.57662.objecter _send_command 8 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.051+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae400c700 con 0x7f1ae4012410 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.052+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae400c700 con 0x7f1ae4012410 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.052+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.052+0000 7f1af67fc700 10 client.57662.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.052+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 4 2024-03-30T15:01:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.052+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 3 8 2024-03-30T15:01:10.054 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 10 client.57662.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 9 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 9 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 4 9 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 4 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 10 client.57662.objecter _send_command 9 2024-03-30T15:01:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1ae400c6c0 con 0x7f1ae4011f70 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.053+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae4011f70 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.054+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae400c6c0 con 0x7f1ae4011f70 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.054+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.054+0000 7f1af67fc700 10 client.57662.objecter _finish_command 9 = system:0 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.054+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 4 2024-03-30T15:01:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.054+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 4 9 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 10 client.57662.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 10 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 10 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400c4b0 osd=4 3 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 4 10 2024-03-30T15:01:10.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 4 2024-03-30T15:01:10.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 10 client.57662.objecter _send_command 10 2024-03-30T15:01:10.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.071+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae401c150 con 0x7f1ae4011f70 2024-03-30T15:01:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.072+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae401c150 con 0x7f1ae4011f70 2024-03-30T15:01:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.072+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.072+0000 7f1af67fc700 10 client.57662.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.072+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 4 2024-03-30T15:01:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.072+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 4 10 2024-03-30T15:01:10.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 10 client.57662.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 11 2024-03-30T15:01:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 11 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 5 11 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 4 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 10 client.57662.objecter _send_command 11 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1ae401cd00 con 0x7f1ae401c870 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.073+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae401c870 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.074+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae401cd00 con 0x7f1ae401c870 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.074+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.074+0000 7f1af67fc700 10 client.57662.objecter _finish_command 11 = system:0 2024-03-30T15:01:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.074+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 4 2024-03-30T15:01:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.074+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 5 11 2024-03-30T15:01:10.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:01:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 10 client.57662.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 12 2024-03-30T15:01:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 12 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae400c720 osd=5 3 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 5 12 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 4 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 10 client.57662.objecter _send_command 12 2024-03-30T15:01:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.091+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae401cd00 con 0x7f1ae401c870 2024-03-30T15:01:10.094 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae401cd00 con 0x7f1ae401c870 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 10 client.57662.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 4 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 5 12 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 10 client.57662.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 13 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 13 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 6 13 2024-03-30T15:01:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 4 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 10 client.57662.objecter _send_command 13 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1ae4023140 con 0x7f1ae401ce70 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.093+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae401ce70 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.094+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae4023140 con 0x7f1ae401ce70 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.094+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.094+0000 7f1af67fc700 10 client.57662.objecter _finish_command 13 = system:0 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.094+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 4 2024-03-30T15:01:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.094+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 6 13 2024-03-30T15:01:10.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 10 client.57662.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 14 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 14 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae401cd00 osd=6 3 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 6 14 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 4 2024-03-30T15:01:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 10 client.57662.objecter _send_command 14 2024-03-30T15:01:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.113+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae4023320 con 0x7f1ae401ce70 2024-03-30T15:01:10.116 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.114+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae4023320 con 0x7f1ae401ce70 2024-03-30T15:01:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.114+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1af67fc700 10 client.57662.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 4 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 6 14 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 10 client.57662.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 15 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 15 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 7 15 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 4 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 10 client.57662.objecter _send_command 15 2024-03-30T15:01:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1ae4024620 con 0x7f1ae4023a40 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.115+0000 7f1af67fc700 10 client.57662.objecter ms_handle_connect 0x7f1ae4023a40 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.116+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1ae4024620 con 0x7f1ae4023a40 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.116+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.116+0000 7f1af67fc700 10 client.57662.objecter _finish_command 15 = system:0 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.116+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 4 2024-03-30T15:01:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.116+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 7 15 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 10 client.57662.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign -1 16 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 20 client.57662.objecter _get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 15 client.57662.objecter _session_command_op_remove -1 16 2024-03-30T15:01:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 20 client.57662.objecter get_session s=0x7f1ae4023320 osd=7 3 2024-03-30T15:01:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 15 client.57662.objecter _session_command_op_assign 7 16 2024-03-30T15:01:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 4 2024-03-30T15:01:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 10 client.57662.objecter _send_command 16 2024-03-30T15:01:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.133+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ae4024620 con 0x7f1ae4023a40 2024-03-30T15:01:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1af67fc700 1 -- 172.21.15.28:0/137709776 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1ae4024620 con 0x7f1ae4023a40 2024-03-30T15:01:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1af67fc700 10 client.57662.objecter ms_dispatch 0x7f1b00001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1af67fc700 10 client.57662.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1af67fc700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 4 2024-03-30T15:01:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1af67fc700 15 client.57662.objecter _session_command_op_remove 7 16 2024-03-30T15:01:10.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1b07701700 10 librados: watch_flush enter 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.135+0000 7f1b07701700 10 librados: watch_flush exit 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.0 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6801/3394323575 conn(0x7f1ae4004730 legacy=0x7f1ae4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae40053e0 osd=0 1 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.1 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6805/1850313547 conn(0x7f1ae4004280 legacy=0x7f1ae400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4004110 osd=1 1 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.2 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6809/3451325043 conn(0x7f1ae400d5d0 legacy=0x7f1ae400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4008370 osd=2 1 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.3 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6813/1506157250 conn(0x7f1ae4012410 legacy=0x7f1ae4014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400cc20 osd=3 1 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.4 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.134:6800/3063775514 conn(0x7f1ae4011f70 legacy=0x7f1ae4019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c4b0 osd=4 1 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.5 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.134:6804/3668151179 conn(0x7f1ae401c870 legacy=0x7f1ae401a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae400c720 osd=5 1 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b05c9e700 1 -- 172.21.15.28:0/137709776 reap_dead start 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.6 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.134:6808/2135224194 conn(0x7f1ae401ce70 legacy=0x7f1ae4021870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b05c9e700 1 -- 172.21.15.28:0/137709776 reap_dead start 2024-03-30T15:01:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b05c9e700 1 -- 172.21.15.28:0/137709776 reap_dead start 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b05c9e700 1 -- 172.21.15.28:0/137709776 reap_dead start 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae401cd00 osd=6 1 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter close_session for osd.7 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.134:6812/3758190298 conn(0x7f1ae4023a40 legacy=0x7f1ae4023ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter put_session s=0x7f1ae4023320 osd=7 1 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 20 client.57662.objecter shutdown clearing up homeless session... 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 10 client.57662.objecter successfully canceled tick 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1aec070470 legacy=0x7f1aec072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> v1:172.21.15.28:6790/0 conn(0x7f1b00109dc0 legacy=0x7f1b001a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 shutdown_connections 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 >> 172.21.15.28:0/137709776 conn(0x7f1b000fc9e0 msgr2=0x7f1b001105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 shutdown_connections 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 -- 172.21.15.28:0/137709776 wait complete. 2024-03-30T15:01:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:10.136+0000 7f1b07701700 1 librados: shutdown 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:10.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:10.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:10.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:10.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:10.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:10.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:10.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:10.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:10.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:10.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:10.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:10.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 101.19 scrub starts 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 101.19 scrub ok 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: pgmap v398: 319 pgs: 32 unknown, 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/137709776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 17.10 scrub starts 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 17.10 scrub ok 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.57656 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-30T15:01:10.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 42.5 scrub starts 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: 42.5 scrub ok 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 101.19 scrub starts 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 101.19 scrub ok 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: pgmap v398: 319 pgs: 32 unknown, 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:10.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/137709776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 17.10 scrub starts 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 17.10 scrub ok 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.57656 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.799 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 42.5 scrub starts 2024-03-30T15:01:10.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: 42.5 scrub ok 2024-03-30T15:01:10.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:10.878 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 101.19 scrub starts 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 101.19 scrub ok 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: pgmap v398: 319 pgs: 32 unknown, 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/137709776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 17.10 scrub starts 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 17.10 scrub ok 2024-03-30T15:01:10.879 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:10.880 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.57656 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi028-92999-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:10.880 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:10.880 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-30T15:01:10.880 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: osdmap e306: 8 total, 8 up, 8 in 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2923162696' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]: dispatch 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 42.5 scrub starts 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: 42.5 scrub ok 2024-03-30T15:01:10.881 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:10.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:10.885 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:10.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:11.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:11.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:11.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:11.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:11.400 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:11.400 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:11.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:11.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:11.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:11.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:11.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:11.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:11.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:11.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:11.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:01:12.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:12.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:12.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:12.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:12.193 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:12.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:12.369 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:01:12.369 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:01:12.369 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T15:01:12.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:01:12.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T15:01:12.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T15:01:12.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:12.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:12.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T15:01:12.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:12 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49657 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi028-92979-41"}]': finished 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: osdmap e307: 8 total, 8 up, 8 in 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:01:12.490 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi028-92979-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: osdmap e308: 8 total, 8 up, 8 in 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.491 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:12 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:12.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:12.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:12.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:12.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:01:12.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:12.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:13.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.058 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:13.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.099+0000 7fbc1bb81700 1 -- 172.21.15.28:0/1457909987 >> v1:172.21.15.28:6790/0 conn(0x7fbc1410aee0 legacy=0x7fbc1410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 -- 172.21.15.28:0/1457909987 shutdown_connections 2024-03-30T15:01:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 -- 172.21.15.28:0/1457909987 >> 172.21.15.28:0/1457909987 conn(0x7fbc140fdb20 msgr2=0x7fbc140fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 -- 172.21.15.28:0/1457909987 shutdown_connections 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 -- 172.21.15.28:0/1457909987 wait complete. 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 librados: starting msgr at 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 librados: starting objecter 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 Processor -- start 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 -- start start 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 librados: setting wanted keys 2024-03-30T15:01:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.100+0000 7fbc1bb81700 1 librados: calling monclient init 2024-03-30T15:01:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1bb81700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141a92d0 con 0x7fbc14102250 2024-03-30T15:01:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1bb81700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141aa510 con 0x7fbc1410ea80 2024-03-30T15:01:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1bb81700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbc141ab750 con 0x7fbc1410aee0 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1911c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbc1410aee0 0x7fbc141a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55916/0 (socket says 172.21.15.28:55916) 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1911c700 1 -- 172.21.15.28:0/2480465660 learned_addr learned my addr 172.21.15.28:0/2480465660 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc1991d700 1 --1- 172.21.15.28:0/2480465660 >> v1:172.21.15.28:6789/0 conn(0x7fbc14102250 0x7fbc14107320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33306/0 (socket says 172.21.15.28:33306) 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter ms_handle_connect 0x7fbc1410ea80 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter ms_handle_connect 0x7fbc1410aee0 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter ms_handle_connect 0x7fbc14102250 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 462007093 0 0) 0x7fbc141aa510 con 0x7fbc1410ea80 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc000031f0 con 0x7fbc1410ea80 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849250670 0 0) 0x7fbc141a92d0 con 0x7fbc14102250 2024-03-30T15:01:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc141aa510 con 0x7fbc14102250 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579458813 0 0) 0x7fbc141ab750 con 0x7fbc1410aee0 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc141a92d0 con 0x7fbc1410aee0 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3037414151 0 0) 0x7fbc000031f0 con 0x7fbc1410ea80 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc141ab750 con 0x7fbc1410ea80 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbc0c001cc0 con 0x7fbc1410ea80 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3158927254 0 0) 0x7fbc141aa510 con 0x7fbc14102250 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc000031f0 con 0x7fbc14102250 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbc100018b0 con 0x7fbc14102250 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2805506093 0 0) 0x7fbc141ab750 con 0x7fbc1410ea80 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 >> v1:172.21.15.28:6790/0 conn(0x7fbc1410aee0 legacy=0x7fbc141a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.101+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 >> v1:172.21.15.28:6789/0 conn(0x7fbc14102250 legacy=0x7fbc14107320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc141ac990 con 0x7fbc1410ea80 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc141ab980 con 0x7fbc1410ea80 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc1bb81700 10 client.49699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc141abe40 con 0x7fbc1410ea80 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc1bb81700 1 librados: init done 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbc0c003ea0 con 0x7fbc1410ea80 2024-03-30T15:01:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.102+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbc0c004db0 con 0x7fbc1410ea80 2024-03-30T15:01:13.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.103+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf8005120 con 0x7fbc1410ea80 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbc0c01b690 con 0x7fbc1410ea80 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(308..308 src has 53..308) ==== 9642+0+0 (unknown 61474468 0 0) 0x7fbc0c088e80 con 0x7fbc1410ea80 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 10 client.49699.objecter ms_dispatch 0x7fbc14001550 osd_map(308..308 src has 53..308) 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 3 client.49699.objecter handle_osd_map got epochs [308,308] > 0 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 3 client.49699.objecter handle_osd_map decoding full epoch 308 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.104+0000 7fbc0affd700 20 client.49699.objecter dump_active .. 0 homeless 2024-03-30T15:01:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.105+0000 7fbc0affd700 10 client.49699.objecter ms_handle_connect 0x7fbc00070330 2024-03-30T15:01:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.109+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbc0c055aa0 con 0x7fbc1410ea80 2024-03-30T15:01:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:13.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.183 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:13.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:13.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:13.362+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fbbf8005410 con 0x7fbc1410ea80 2024-03-30T15:01:13.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:13.421 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:13.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:13.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.561 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:13.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:13.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:13.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:13.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: pgmap v401: 287 pgs: 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 42.1f scrub starts 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 42.1f scrub ok 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:01:13.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: pgmap v401: 287 pgs: 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 42.1f scrub starts 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 42.1f scrub ok 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:01:13.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:01:13.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:13.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: pgmap v401: 287 pgs: 1 peering, 286 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 42.1f scrub starts 2024-03-30T15:01:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 42.1f scrub ok 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:01:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:13.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:13.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:13.945 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:13.945 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:14.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:14.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:14.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:14.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:14.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.222+0000 7fbc0affd700 1 -- 172.21.15.28:0/2480465660 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v310) ==== 101+0+0 (unknown 2491985441 0 0) 0x7fbc0c0594f0 con 0x7fbc1410ea80 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.225+0000 7fbc1bb81700 10 librados: watch_flush enter 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 10 librados: watch_flush exit 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 20 client.49699.objecter shutdown clearing up homeless session... 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 10 client.49699.objecter successfully canceled tick 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbc00070330 legacy=0x7fbc000727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 >> v1:172.21.15.134:6789/0 conn(0x7fbc1410ea80 legacy=0x7fbc141a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 shutdown_connections 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 >> 172.21.15.28:0/2480465660 conn(0x7fbc140fdb20 msgr2=0x7fbc14108eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.227+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 shutdown_connections 2024-03-30T15:01:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.230+0000 7fbc1bb81700 1 -- 172.21.15.28:0/2480465660 wait complete. 2024-03-30T15:01:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:14.230+0000 7fbc1bb81700 1 librados: shutdown 2024-03-30T15:01:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:01:14.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:14.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:14.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:14.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:01:14.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:14.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:14.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:14.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:14.786 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:01:14.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:14.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:14.907 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2480465660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: 17.14 scrub starts 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: 17.14 scrub ok 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: pgmap v404: 286 pgs: 286 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.49699 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:14.908 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:14.909 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.49699 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:14.909 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-30T15:01:14.909 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T15:01:14.909 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:14.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:14.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:14.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:15.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:15.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2480465660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: 17.14 scrub starts 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: 17.14 scrub ok 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: pgmap v404: 286 pgs: 286 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.49699 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:15.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.49699 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:15.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2480465660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: 17.14 scrub starts 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: 17.14 scrub ok 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: pgmap v404: 286 pgs: 286 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: osdmap e309: 8 total, 8 up, 8 in 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.49699 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi028-92979-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:15.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi028-92999-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:15.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-35"}]': finished 2024-03-30T15:01:15.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.49699 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:15.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-30T15:01:15.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: osdmap e310: 8 total, 8 up, 8 in 2024-03-30T15:01:15.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:15.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:15.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:15.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:15.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:15.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:15.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:15.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:15.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:15.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:15.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:15.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:15.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:15.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:15.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:16.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:16.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:16.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:16.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:01:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:16 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: osdmap e311: 8 total, 8 up, 8 in 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:16 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]: dispatch 2024-03-30T15:01:16.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:16.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:16.719 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:16.719 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:16.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:16.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 -- 172.21.15.28:0/1211216341 >> v1:172.21.15.28:6790/0 conn(0x7f6d581022a0 legacy=0x7f6d58102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 -- 172.21.15.28:0/1211216341 shutdown_connections 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 -- 172.21.15.28:0/1211216341 >> 172.21.15.28:0/1211216341 conn(0x7f6d580fdb50 msgr2=0x7f6d580fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 -- 172.21.15.28:0/1211216341 shutdown_connections 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 -- 172.21.15.28:0/1211216341 wait complete. 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 librados: starting msgr at 2024-03-30T15:01:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 librados: starting objecter 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.936+0000 7f6d60639700 1 Processor -- start 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 -- start start 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 librados: setting wanted keys 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 librados: calling monclient init 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d58107b10 con 0x7f6d581022a0 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d581a86a0 con 0x7f6d5810af30 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d60639700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d581a9880 con 0x7f6d5810ead0 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d5e3d5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6d581022a0 0x7f6d58104960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33326/0 (socket says 172.21.15.28:33326) 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d5e3d5700 1 -- 172.21.15.28:0/596003356 learned_addr learned my addr 172.21.15.28:0/596003356 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d5ebd6700 1 --1- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6790/0 conn(0x7f6d5810ead0 0x7f6d581a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55940/0 (socket says 172.21.15.28:55940) 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter ms_handle_connect 0x7f6d581022a0 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter ms_handle_connect 0x7f6d5810ead0 2024-03-30T15:01:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter ms_handle_connect 0x7f6d5810af30 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131728385 0 0) 0x7f6d58107b10 con 0x7f6d581022a0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d40003aa0 con 0x7f6d581022a0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835884634 0 0) 0x7f6d581a9880 con 0x7f6d5810ead0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d58107b10 con 0x7f6d5810ead0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494033930 0 0) 0x7f6d581a86a0 con 0x7f6d5810af30 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d581a9880 con 0x7f6d5810af30 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 154046607 0 0) 0x7f6d40003aa0 con 0x7f6d581022a0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d581a86a0 con 0x7f6d581022a0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d500015d0 con 0x7f6d581022a0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 31977791 0 0) 0x7f6d58107b10 con 0x7f6d5810ead0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d40003aa0 con 0x7f6d5810ead0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d48001bb0 con 0x7f6d5810ead0 2024-03-30T15:01:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4260022679 0 0) 0x7f6d581a9880 con 0x7f6d5810af30 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.937+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d58107b10 con 0x7f6d5810af30 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d4c001130 con 0x7f6d5810af30 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2883030023 0 0) 0x7f6d581a86a0 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6790/0 conn(0x7f6d5810ead0 legacy=0x7f6d581a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.134:6789/0 conn(0x7f6d5810af30 legacy=0x7f6d58105070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581aaac0 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d581a88d0 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d60639700 10 client.24510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d581a8dd0 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d60639700 1 librados: init done 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6d50001d20 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d50004e40 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.938+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6d44005100 con 0x7f6d581022a0 2024-03-30T15:01:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.939+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6d50001880 con 0x7f6d581022a0 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(312..312 src has 53..312) ==== 9786+0+0 (unknown 2971508555 0 0) 0x7f6d50089d80 con 0x7f6d581022a0 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 osd_map(312..312 src has 53..312) 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 3 client.24510.objecter handle_osd_map got epochs [312,312] > 0 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 3 client.24510.objecter handle_osd_map decoding full epoch 312 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 20 client.24510.objecter dump_active .. 0 homeless 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v312) ==== 54+0+16 (unknown 287102909 0 2420000192) 0x7f6d5008d020 con 0x7f6d581022a0 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d40070470 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 10 client.24510.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 1 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 1 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 0 1 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 4 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 10 client.24510.objecter _send_command 1 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d44004b50 con 0x7f6d44004730 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.940+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44004730 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.941+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d44004b50 con 0x7f6d44004730 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.941+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.941+0000 7f6d577fe700 10 client.24510.objecter _finish_command 1 = system:0 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.941+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 4 2024-03-30T15:01:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.941+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 0 1 2024-03-30T15:01:16.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:16.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.958+0000 7f6d60639700 10 client.24510.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 2 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 2 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d440053e0 osd=0 3 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 0 2 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 4 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 10 client.24510.objecter _send_command 2 2024-03-30T15:01:16.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.959+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d4400bf10 con 0x7f6d44004730 2024-03-30T15:01:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.960+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d4400bf10 con 0x7f6d44004730 2024-03-30T15:01:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.960+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.960+0000 7f6d577fe700 10 client.24510.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.960+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 4 2024-03-30T15:01:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.960+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 0 2 2024-03-30T15:01:16.963 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 10 client.24510.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 1 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 4 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 10 client.24510.objecter _send_command 3 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6d44002200 con 0x7f6d440042f0 2024-03-30T15:01:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d440042f0 2024-03-30T15:01:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d44002200 con 0x7f6d440042f0 2024-03-30T15:01:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 10 client.24510.objecter _finish_command 3 = system:0 2024-03-30T15:01:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 4 2024-03-30T15:01:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.961+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 1 3 2024-03-30T15:01:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:16.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:16.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 10 client.24510.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 4 2024-03-30T15:01:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 4 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44004180 osd=1 3 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 1 4 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 4 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 10 client.24510.objecter _send_command 4 2024-03-30T15:01:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.979+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d44001600 con 0x7f6d440042f0 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d44001600 con 0x7f6d440042f0 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 10 client.24510.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 4 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 1 4 2024-03-30T15:01:16.983 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 10 client.24510.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 5 2024-03-30T15:01:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 5 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 2 5 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 4 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 10 client.24510.objecter _send_command 5 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6d44015600 con 0x7f6d44011af0 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.981+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44011af0 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.982+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d44015600 con 0x7f6d44011af0 2024-03-30T15:01:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.982+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:16.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.982+0000 7f6d577fe700 10 client.24510.objecter _finish_command 5 = system:0 2024-03-30T15:01:16.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.982+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 4 2024-03-30T15:01:16.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.982+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 2 5 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 10 client.24510.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 6 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 6 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4400c890 osd=2 3 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 2 6 2024-03-30T15:01:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 4 2024-03-30T15:01:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 10 client.24510.objecter _send_command 6 2024-03-30T15:01:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:16.999+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d4400c3d0 con 0x7f6d44011af0 2024-03-30T15:01:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.000+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d4400c3d0 con 0x7f6d44011af0 2024-03-30T15:01:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.000+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.000+0000 7f6d577fe700 10 client.24510.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.000+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 4 2024-03-30T15:01:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.000+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 2 6 2024-03-30T15:01:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 10 client.24510.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 7 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 7 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 3 7 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 4 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 10 client.24510.objecter _send_command 7 2024-03-30T15:01:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6d4401a440 con 0x7f6d44016930 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.001+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44016930 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.002+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d4401a440 con 0x7f6d44016930 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.002+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.002+0000 7f6d577fe700 10 client.24510.objecter _finish_command 7 = system:0 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.002+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 4 2024-03-30T15:01:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.002+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 3 7 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 10 client.24510.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 8 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 8 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4400c660 osd=3 3 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 3 8 2024-03-30T15:01:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 4 2024-03-30T15:01:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 10 client.24510.objecter _send_command 8 2024-03-30T15:01:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.019+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d44010e70 con 0x7f6d44016930 2024-03-30T15:01:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.020+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d44010e70 con 0x7f6d44016930 2024-03-30T15:01:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.020+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.020+0000 7f6d577fe700 10 client.24510.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.020+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 4 2024-03-30T15:01:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.020+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 3 8 2024-03-30T15:01:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 10 client.24510.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 9 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 9 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 4 9 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 4 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 10 client.24510.objecter _send_command 9 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6d4401ecf0 con 0x7f6d44016490 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.021+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44016490 2024-03-30T15:01:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.022+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d4401ecf0 con 0x7f6d44016490 2024-03-30T15:01:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.022+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.022+0000 7f6d577fe700 10 client.24510.objecter _finish_command 9 = system:0 2024-03-30T15:01:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.022+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 4 2024-03-30T15:01:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.022+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 4 9 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 10 client.24510.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 10 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 10 2024-03-30T15:01:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44010c20 osd=4 3 2024-03-30T15:01:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 4 10 2024-03-30T15:01:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 4 2024-03-30T15:01:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 10 client.24510.objecter _send_command 10 2024-03-30T15:01:17.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.039+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d4401ecf0 con 0x7f6d44016490 2024-03-30T15:01:17.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d4401ecf0 con 0x7f6d44016490 2024-03-30T15:01:17.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d577fe700 10 client.24510.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 4 2024-03-30T15:01:17.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 4 10 2024-03-30T15:01:17.044 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 10 client.24510.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 11 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 11 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 5 11 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 4 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 10 client.24510.objecter _send_command 11 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.041+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6d44023030 con 0x7f6d4401f370 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d4401f370 2024-03-30T15:01:17.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d44023030 con 0x7f6d4401f370 2024-03-30T15:01:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 10 client.24510.objecter _finish_command 11 = system:0 2024-03-30T15:01:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 4 2024-03-30T15:01:17.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.042+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 5 11 2024-03-30T15:01:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 10 client.24510.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 12 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 12 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d4401ecf0 osd=5 3 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 5 12 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 4 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 10 client.24510.objecter _send_command 12 2024-03-30T15:01:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.063+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d44023210 con 0x7f6d4401f370 2024-03-30T15:01:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d44023210 con 0x7f6d4401f370 2024-03-30T15:01:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d577fe700 10 client.24510.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 4 2024-03-30T15:01:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 5 12 2024-03-30T15:01:17.068 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.065+0000 7f6d60639700 10 client.24510.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 13 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 13 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 6 13 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 4 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 10 client.24510.objecter _send_command 13 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6d44024530 con 0x7f6d44023930 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.066+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44023930 2024-03-30T15:01:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.067+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d44024530 con 0x7f6d44023930 2024-03-30T15:01:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.067+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.067+0000 7f6d577fe700 10 client.24510.objecter _finish_command 13 = system:0 2024-03-30T15:01:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.067+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 4 2024-03-30T15:01:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.067+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 6 13 2024-03-30T15:01:17.077 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 10 client.24510.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 14 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 14 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44023210 osd=6 3 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 6 14 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 4 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 10 client.24510.objecter _send_command 14 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.084+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d44024530 con 0x7f6d44023930 2024-03-30T15:01:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d44024530 con 0x7f6d44023930 2024-03-30T15:01:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 10 client.24510.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 4 2024-03-30T15:01:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 6 14 2024-03-30T15:01:17.088 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 10 client.24510.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 15 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 15 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 7 15 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 4 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 10 client.24510.objecter _send_command 15 2024-03-30T15:01:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6d4402ba60 con 0x7f6d44027f10 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.086+0000 7f6d577fe700 10 client.24510.objecter ms_handle_connect 0x7f6d44027f10 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.087+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6d4402ba60 con 0x7f6d44027f10 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.087+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.087+0000 7f6d577fe700 10 client.24510.objecter _finish_command 15 = system:0 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.087+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 4 2024-03-30T15:01:17.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.087+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 7 15 2024-03-30T15:01:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 10 client.24510.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign -1 16 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 20 client.24510.objecter _get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 15 client.24510.objecter _session_command_op_remove -1 16 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 20 client.24510.objecter get_session s=0x7f6d44024530 osd=7 3 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 15 client.24510.objecter _session_command_op_assign 7 16 2024-03-30T15:01:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 4 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 10 client.24510.objecter _send_command 16 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.105+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6d440110e0 con 0x7f6d44027f10 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d577fe700 1 -- 172.21.15.28:0/596003356 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6d440110e0 con 0x7f6d44027f10 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d577fe700 10 client.24510.objecter ms_dispatch 0x7f6d58001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d577fe700 10 client.24510.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d577fe700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 4 2024-03-30T15:01:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d577fe700 15 client.24510.objecter _session_command_op_remove 7 16 2024-03-30T15:01:17.110 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 librados: watch_flush enter 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 librados: watch_flush exit 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.0 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6d44004730 legacy=0x7f6d44002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d440053e0 osd=0 1 2024-03-30T15:01:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.1 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6d440042f0 legacy=0x7f6d4400f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44004180 osd=1 1 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.2 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6d44011af0 legacy=0x7f6d44013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c890 osd=2 1 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.3 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6d44016930 legacy=0x7f6d44018d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4400c660 osd=3 1 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.4 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6d44016490 legacy=0x7f6d4401d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44010c20 osd=4 1 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.5 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.134:6804/3668151179 conn(0x7f6d4401f370 legacy=0x7f6d44021760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d4401ecf0 osd=5 1 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.6 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.134:6808/2135224194 conn(0x7f6d44023930 legacy=0x7f6d44023d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44023210 osd=6 1 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter close_session for osd.7 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6d44027f10 legacy=0x7f6d4402a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter put_session s=0x7f6d44024530 osd=7 1 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 20 client.24510.objecter shutdown clearing up homeless session... 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 10 client.24510.objecter successfully canceled tick 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6d40070470 legacy=0x7f6d40072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> v1:172.21.15.28:6789/0 conn(0x7f6d581022a0 legacy=0x7f6d58104960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d5ebd6700 1 -- 172.21.15.28:0/596003356 reap_dead start 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.107+0000 7f6d5ebd6700 1 -- 172.21.15.28:0/596003356 reap_dead start 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d5ebd6700 1 -- 172.21.15.28:0/596003356 reap_dead start 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d5ebd6700 1 -- 172.21.15.28:0/596003356 reap_dead start 2024-03-30T15:01:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d5ebd6700 1 -- 172.21.15.28:0/596003356 reap_dead start 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 shutdown_connections 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 >> 172.21.15.28:0/596003356 conn(0x7f6d580fdb50 msgr2=0x7f6d58111fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 shutdown_connections 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d60639700 1 -- 172.21.15.28:0/596003356 wait complete. 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.108+0000 7f6d60639700 1 librados: shutdown 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.166+0000 7f7cdabc4700 1 -- 172.21.15.28:0/1986165997 >> v1:172.21.15.28:6789/0 conn(0x7f7cd4101140 legacy=0x7f7cd4101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.166+0000 7f7cdabc4700 1 -- 172.21.15.28:0/1986165997 shutdown_connections 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.166+0000 7f7cdabc4700 1 -- 172.21.15.28:0/1986165997 >> 172.21.15.28:0/1986165997 conn(0x7f7cd40fc9f0 msgr2=0x7f7cd40fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.166+0000 7f7cdabc4700 1 -- 172.21.15.28:0/1986165997 shutdown_connections 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.166+0000 7f7cdabc4700 1 -- 172.21.15.28:0/1986165997 wait complete. 2024-03-30T15:01:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 librados: starting msgr at 2024-03-30T15:01:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 librados: starting objecter 2024-03-30T15:01:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 Processor -- start 2024-03-30T15:01:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 -- start start 2024-03-30T15:01:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 librados: setting wanted keys 2024-03-30T15:01:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 librados: calling monclient init 2024-03-30T15:01:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7cd41a9100 con 0x7f7cd4109dd0 2024-03-30T15:01:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7cd41aa320 con 0x7f7cd4101140 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cdabc4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7cd41ab560 con 0x7f7cd410d970 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cd3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cd4109dd0 0x7f7cd41a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33332/0 (socket says 172.21.15.28:33332) 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.167+0000 7f7cd3fff700 1 -- 172.21.15.28:0/3307019983 learned_addr learned my addr 172.21.15.28:0/3307019983 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter ms_handle_connect 0x7f7cd410d970 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter ms_handle_connect 0x7f7cd4109dd0 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter ms_handle_connect 0x7f7cd4101140 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072578582 0 0) 0x7f7cd41aa320 con 0x7f7cd4101140 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc00032d0 con 0x7f7cd4101140 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375948177 0 0) 0x7f7cd41ab560 con 0x7f7cd410d970 2024-03-30T15:01:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cd41aa320 con 0x7f7cd410d970 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840851599 0 0) 0x7f7cd41a9100 con 0x7f7cd4109dd0 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cd41ab560 con 0x7f7cd4109dd0 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3211011021 0 0) 0x7f7cc00032d0 con 0x7f7cd4101140 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cd41a9100 con 0x7f7cd4101140 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cc4001430 con 0x7f7cd4101140 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1587958325 0 0) 0x7f7cd41ab560 con 0x7f7cd4109dd0 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cc00032d0 con 0x7f7cd4109dd0 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cbc001940 con 0x7f7cd4109dd0 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 795147542 0 0) 0x7f7cd41aa320 con 0x7f7cd410d970 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cd41ab560 con 0x7f7cd410d970 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.168+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3738878109 0 0) 0x7f7cd41a9100 con 0x7f7cd4101140 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 >> v1:172.21.15.28:6790/0 conn(0x7f7cd410d970 legacy=0x7f7cd41a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 >> v1:172.21.15.28:6789/0 conn(0x7f7cd4109dd0 legacy=0x7f7cd41a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd41ac7a0 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd41aa550 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cdabc4700 10 client.49711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7cd41aaa10 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cdabc4700 1 librados: init done 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7cc4001d20 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.169+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cc4004c80 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.170+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb4005120 con 0x7f7cd4101140 2024-03-30T15:01:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7cc4005ed0 con 0x7f7cd4101140 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(312..312 src has 53..312) ==== 9786+0+0 (unknown 2971508555 0 0) 0x7f7cc4088c60 con 0x7f7cd4101140 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 10 client.49711.objecter ms_dispatch 0x7f7cd4001550 osd_map(312..312 src has 53..312) 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 3 client.49711.objecter handle_osd_map got epochs [312,312] > 0 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 3 client.49711.objecter handle_osd_map decoding full epoch 312 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 20 client.49711.objecter dump_active .. 0 homeless 2024-03-30T15:01:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.171+0000 7f7cd1ffb700 10 client.49711.objecter ms_handle_connect 0x7f7cc0070570 2024-03-30T15:01:17.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.175+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7cc4059240 con 0x7f7cd4101140 2024-03-30T15:01:17.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: pgmap v407: 270 pgs: 16 unknown, 254 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: 17.18 deep-scrub starts 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: 17.18 deep-scrub ok 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.314 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/596003356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:17.315 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:17 smithi134 ceph-mon[33714]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T15:01:17.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:17.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:01:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:17.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: pgmap v407: 270 pgs: 16 unknown, 254 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:01:17.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:17.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:17.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: 17.18 deep-scrub starts 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: 17.18 deep-scrub ok 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:01:17.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/596003356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[41614]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: pgmap v407: 270 pgs: 16 unknown, 254 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: 17.18 deep-scrub starts 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: 17.18 deep-scrub ok 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pg_num","val":"11"}]': finished 2024-03-30T15:01:17.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: osdmap e312: 8 total, 8 up, 8 in 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/596003356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:17.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:17.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:17 smithi028 ceph-mon[29876]: osdmap e313: 8 total, 8 up, 8 in 2024-03-30T15:01:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.409+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7cb4000b90 con 0x7f7cc0070570 2024-03-30T15:01:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.428+0000 7f7cd1ffb700 1 -- 172.21.15.28:0/3307019983 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+722552 (unknown 2965378022 0 2346426299) 0x7f7cb4000b90 con 0x7f7cc0070570 2024-03-30T15:01:17.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:01:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 10 librados: watch_flush enter 2024-03-30T15:01:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 10 librados: watch_flush exit 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 20 client.49711.objecter shutdown clearing up homeless session... 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 10 client.49711.objecter successfully canceled tick 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7cc0070570 legacy=0x7f7cc0072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 >> v1:172.21.15.134:6789/0 conn(0x7f7cd4101140 legacy=0x7f7cd41063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 shutdown_connections 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 >> 172.21.15.28:0/3307019983 conn(0x7f7cd40fc9f0 msgr2=0x7f7cd4110e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 shutdown_connections 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 -- 172.21.15.28:0/3307019983 wait complete. 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:17.434+0000 7f7cdabc4700 1 librados: shutdown 2024-03-30T15:01:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:01:17.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:17.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:17.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:17.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:17.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:01:17.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:17.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:17.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:17.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:17.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:18.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:18.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:18.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAi 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6476 ms) 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7019 ms) 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6981 ms) 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7009 ms) 2024-03-30T15:01:18.231 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7014 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7002 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6999 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6992 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7015 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7002 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6987 ms) 2024-03-30T15:01:18.232 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7013 ms) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7005 ms) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6991 ms) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7011 ms) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6993 ms) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (111509 ms total) 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [----------] Global test environment tear-down 2024-03-30T15:01:18.233 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (213868 ms total) 2024-03-30T15:01:18.234 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-30T15:01:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:18.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:18.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:18.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 12.10 scrub starts 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: 12.10 scrub ok 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T15:01:18.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:01:18.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 12.10 scrub starts 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: 12.10 scrub ok 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:18.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:18.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:18.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:01:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3098895510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1709144287' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 12.10 scrub starts 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: 12.10 scrub ok 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-30T15:01:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.49690 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi028-92979-42"}]': finished 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.57671 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi028-92999-52"}]': finished 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: osdmap e314: 8 total, 8 up, 8 in 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:18.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:18.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:18.837 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:18.855 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-30T15:01:17.218885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793879,"num_objects":337,"num_object_clones":2,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1423,"num_read_kb":1174,"num_write":1299,"num_write_kb":11272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1772962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2560,"ondisk_log_size":2560,"up":759,"acting":759,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":879,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":602380,"kb_used_data":57136,"kb_used_omap":12,"kb_used_meta":545011,"kb_avail":749358836,"statfs":{"total":767960285184,"available":767343448064,"internally_reserved":0,"allocated":58507264,"data_stored":39209213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":558091824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001165"},"pg_stats":[{"pgid":"267.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"267.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:01:16.220718+0000","last_change":"2024-03-30T15:01:16.220718+0000","last_active":"2024-03-30T15:01:16.220718+0000","last_peered":"2024-03-30T15:01:16.220718+0000","last_clean":"2024-03-30T15:01:16.220718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:01:16.220718+0000","last_undegraded":"2024-03-30T15:01:16.220718+0000","last_fullsized":"2024-03-30T15:01:16.220718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:16.220718+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.220718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":541,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221046+0000","last_change":"2024-03-30T15:01:12.237476+0000","last_active":"2024-03-30T15:01:13.221046+0000","last_peered":"2024-03-30T15:01:13.221046+0000","last_clean":"2024-03-30T15:01:13.221046+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:01:13.221046+0000","last_undegraded":"2024-03-30T15:01:13.221046+0000","last_fullsized":"2024-03-30T15:01:13.221046+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:01:08.589132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:08.589132+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:20.282792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058544290000000004,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1b","version":"229'85","reported_seq":561,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253179+0000","last_change":"2024-03-30T15:01:12.253525+0000","last_active":"2024-03-30T15:01:12.253179+0000","last_peered":"2024-03-30T15:01:12.253179+0000","last_clean":"2024-03-30T15:01:12.253179+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:01:12.253179+0000","last_undegraded":"2024-03-30T15:01:12.253179+0000","last_fullsized":"2024-03-30T15:01:12.253179+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016816940000000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.18","version":"230'87","reported_seq":568,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223244+0000","last_change":"2024-03-30T15:01:12.219220+0000","last_active":"2024-03-30T15:01:13.223244+0000","last_peered":"2024-03-30T15:01:13.223244+0000","last_clean":"2024-03-30T15:01:13.223244+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:01:13.223244+0000","last_undegraded":"2024-03-30T15:01:13.223244+0000","last_fullsized":"2024-03-30T15:01:13.223244+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013249889999999999,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.19","version":"204'102","reported_seq":590,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.223140+0000","last_change":"2024-03-30T15:01:12.224172+0000","last_active":"2024-03-30T15:01:12.223140+0000","last_peered":"2024-03-30T15:01:12.223140+0000","last_clean":"2024-03-30T15:01:12.223140+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:01:12.223140+0000","last_undegraded":"2024-03-30T15:01:12.223140+0000","last_fullsized":"2024-03-30T15:01:12.223140+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T15:01:09.466765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:09.466765+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:27.839638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062489279999999999,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1e","version":"286'36","reported_seq":536,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220508+0000","last_change":"2024-03-30T15:01:12.235443+0000","last_active":"2024-03-30T15:01:13.220508+0000","last_peered":"2024-03-30T15:01:13.220508+0000","last_clean":"2024-03-30T15:01:13.220508+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:01:13.220508+0000","last_undegraded":"2024-03-30T15:01:13.220508+0000","last_fullsized":"2024-03-30T15:01:13.220508+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.725810+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:16.472004+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0065316139999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1f","version":"308'73","reported_seq":623,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220717+0000","last_change":"2024-03-30T15:01:12.247026+0000","last_active":"2024-03-30T15:01:13.220717+0000","last_peered":"2024-03-30T15:01:13.220717+0000","last_clean":"2024-03-30T15:01:13.220717+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:01:13.220717+0000","last_undegraded":"2024-03-30T15:01:13.220717+0000","last_fullsized":"2024-03-30T15:01:13.220717+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'65","last_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_deep_scrub":"283'65","last_deep_scrub_stamp":"2024-03-30T15:00:47.743297+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.743297+0000","objects_scrubbed":6,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.870507+0000","scrub_duration":10,"objects_trimmed":1,"snaptrim_duration":0.016684201999999999,"stat_sum":{"num_bytes":4194343,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":24,"num_write_kb":8213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1c","version":"222'34","reported_seq":519,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253300+0000","last_change":"2024-03-30T15:01:12.253733+0000","last_active":"2024-03-30T15:01:12.253300+0000","last_peered":"2024-03-30T15:01:12.253300+0000","last_clean":"2024-03-30T15:01:12.253300+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:01:12.253300+0000","last_undegraded":"2024-03-30T15:01:12.253300+0000","last_fullsized":"2024-03-30T15:01:12.253300+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698305,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1d","version":"204'51","reported_seq":537,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224588+0000","last_change":"2024-03-30T15:01:12.222217+0000","last_active":"2024-03-30T15:01:13.224588+0000","last_peered":"2024-03-30T15:01:13.224588+0000","last_clean":"2024-03-30T15:01:13.224588+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:01:13.224588+0000","last_undegraded":"2024-03-30T15:01:13.224588+0000","last_fullsized":"2024-03-30T15:01:13.224588+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036882529999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.12","version":"287'40","reported_seq":530,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589765+0000","last_change":"2024-03-30T15:01:12.234071+0000","last_active":"2024-03-30T15:01:13.589765+0000","last_peered":"2024-03-30T15:01:13.589765+0000","last_clean":"2024-03-30T15:01:13.589765+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:01:13.589765+0000","last_undegraded":"2024-03-30T15:01:13.589765+0000","last_fullsized":"2024-03-30T15:01:13.589765+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037348239999999999,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.13","version":"237'34","reported_seq":526,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589956+0000","last_change":"2024-03-30T15:01:12.238092+0000","last_active":"2024-03-30T15:01:13.589956+0000","last_peered":"2024-03-30T15:01:13.589956+0000","last_clean":"2024-03-30T15:01:13.589956+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:01:13.589956+0000","last_undegraded":"2024-03-30T15:01:13.589956+0000","last_fullsized":"2024-03-30T15:01:13.589956+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051724700000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.10","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223273+0000","last_change":"2024-03-30T15:01:12.219430+0000","last_active":"2024-03-30T15:01:13.223273+0000","last_peered":"2024-03-30T15:01:13.223273+0000","last_clean":"2024-03-30T15:01:13.223273+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:01:13.223273+0000","last_undegraded":"2024-03-30T15:01:13.223273+0000","last_fullsized":"2024-03-30T15:01:13.223273+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014867789999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.11","version":"204'17","reported_seq":512,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223743+0000","last_change":"2024-03-30T15:01:12.220746+0000","last_active":"2024-03-30T15:01:13.223743+0000","last_peered":"2024-03-30T15:01:13.223743+0000","last_clean":"2024-03-30T15:01:13.223743+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:01:13.223743+0000","last_undegraded":"2024-03-30T15:01:13.223743+0000","last_fullsized":"2024-03-30T15:01:13.223743+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026988469999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.16","version":"217'17","reported_seq":505,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253798+0000","last_change":"2024-03-30T15:01:12.254070+0000","last_active":"2024-03-30T15:01:12.253798+0000","last_peered":"2024-03-30T15:01:12.253798+0000","last_clean":"2024-03-30T15:01:12.253798+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:01:12.253798+0000","last_undegraded":"2024-03-30T15:01:12.253798+0000","last_fullsized":"2024-03-30T15:01:12.253798+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159227,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.17","version":"241'73","reported_seq":564,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591315+0000","last_change":"2024-03-30T15:01:12.248219+0000","last_active":"2024-03-30T15:01:13.591315+0000","last_peered":"2024-03-30T15:01:13.591315+0000","last_clean":"2024-03-30T15:01:13.591315+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:01:13.591315+0000","last_undegraded":"2024-03-30T15:01:13.591315+0000","last_fullsized":"2024-03-30T15:01:13.591315+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034647839999999998,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.14","version":"295'53","reported_seq":544,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590717+0000","last_change":"2024-03-30T15:01:12.244273+0000","last_active":"2024-03-30T15:01:13.590717+0000","last_peered":"2024-03-30T15:01:13.590717+0000","last_clean":"2024-03-30T15:01:13.590717+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:01:13.590717+0000","last_undegraded":"2024-03-30T15:01:13.590717+0000","last_fullsized":"2024-03-30T15:01:13.590717+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063330620000000004,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.15","version":"267'90","reported_seq":576,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219370+0000","last_change":"2024-03-30T15:01:12.236303+0000","last_active":"2024-03-30T15:01:13.219370+0000","last_peered":"2024-03-30T15:01:13.219370+0000","last_clean":"2024-03-30T15:01:13.219370+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:01:13.219370+0000","last_undegraded":"2024-03-30T15:01:13.219370+0000","last_fullsized":"2024-03-30T15:01:13.219370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'90","last_scrub_stamp":"2024-03-30T15:00:41.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.653662+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:44.300432+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0075118169999999996,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.a","version":"204'34","reported_seq":525,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224063+0000","last_change":"2024-03-30T15:01:12.224450+0000","last_active":"2024-03-30T15:01:13.224063+0000","last_peered":"2024-03-30T15:01:13.224063+0000","last_clean":"2024-03-30T15:01:13.224063+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:01:13.224063+0000","last_undegraded":"2024-03-30T15:01:13.224063+0000","last_fullsized":"2024-03-30T15:01:13.224063+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051419289999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.b","version":"238'104","reported_seq":593,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243840+0000","last_change":"2024-03-30T15:01:12.244118+0000","last_active":"2024-03-30T15:01:12.243840+0000","last_peered":"2024-03-30T15:01:12.243840+0000","last_clean":"2024-03-30T15:01:12.243840+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:01:12.243840+0000","last_undegraded":"2024-03-30T15:01:12.243840+0000","last_fullsized":"2024-03-30T15:01:12.243840+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:00:30.724310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.724310+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:54.547689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001401331,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.8","version":"204'34","reported_seq":534,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222000+0000","last_change":"2024-03-30T15:01:12.226433+0000","last_active":"2024-03-30T15:01:13.222000+0000","last_peered":"2024-03-30T15:01:13.222000+0000","last_clean":"2024-03-30T15:01:13.222000+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:01:13.222000+0000","last_undegraded":"2024-03-30T15:01:13.222000+0000","last_fullsized":"2024-03-30T15:01:13.222000+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T14:59:47.247116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.247116+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0036675259999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.9","version":"254'37","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243908+0000","last_change":"2024-03-30T15:01:12.244183+0000","last_active":"2024-03-30T15:01:12.243908+0000","last_peered":"2024-03-30T15:01:12.243908+0000","last_clean":"2024-03-30T15:01:12.243908+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:01:12.243908+0000","last_undegraded":"2024-03-30T15:01:12.243908+0000","last_fullsized":"2024-03-30T15:01:12.243908+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:00:37.797402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.797402+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.579376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014568300000000001,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.e","version":"204'108","reported_seq":594,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245962+0000","last_change":"2024-03-30T15:01:12.246123+0000","last_active":"2024-03-30T15:01:12.245962+0000","last_peered":"2024-03-30T15:01:12.245962+0000","last_clean":"2024-03-30T15:01:12.245962+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:01:12.245962+0000","last_undegraded":"2024-03-30T15:01:12.245962+0000","last_fullsized":"2024-03-30T15:01:12.245962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:00:31.737537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:31.737537+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.529701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00032509100000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.f","version":"248'56","reported_seq":550,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591341+0000","last_change":"2024-03-30T15:01:12.248926+0000","last_active":"2024-03-30T15:01:13.591341+0000","last_peered":"2024-03-30T15:01:13.591341+0000","last_clean":"2024-03-30T15:01:13.591341+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:01:13.591341+0000","last_undegraded":"2024-03-30T15:01:13.591341+0000","last_fullsized":"2024-03-30T15:01:13.591341+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039401879999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.c","version":"204'51","reported_seq":545,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224137+0000","last_change":"2024-03-30T15:01:12.224719+0000","last_active":"2024-03-30T15:01:13.224137+0000","last_peered":"2024-03-30T15:01:13.224137+0000","last_clean":"2024-03-30T15:01:13.224137+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:01:13.224137+0000","last_undegraded":"2024-03-30T15:01:13.224137+0000","last_fullsized":"2024-03-30T15:01:13.224137+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:00:29.555003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:29.555003+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:32.638304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0036771759999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.d","version":"235'87","reported_seq":562,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253611+0000","last_change":"2024-03-30T15:01:12.254140+0000","last_active":"2024-03-30T15:01:12.253611+0000","last_peered":"2024-03-30T15:01:12.253611+0000","last_clean":"2024-03-30T15:01:12.253611+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:01:12.253611+0000","last_undegraded":"2024-03-30T15:01:12.253611+0000","last_fullsized":"2024-03-30T15:01:12.253611+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017814040000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.2","version":"204'85","reported_seq":570,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.223706+0000","last_change":"2024-03-30T15:01:12.225036+0000","last_active":"2024-03-30T15:01:12.223706+0000","last_peered":"2024-03-30T15:01:12.223706+0000","last_clean":"2024-03-30T15:01:12.223706+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:01:12.223706+0000","last_undegraded":"2024-03-30T15:01:12.223706+0000","last_fullsized":"2024-03-30T15:01:12.223706+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'85","last_scrub_stamp":"2024-03-30T15:00:35.252370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.252370+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:01.777075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0071374050000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.3","version":"248'39","reported_seq":538,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219436+0000","last_change":"2024-03-30T15:01:12.233818+0000","last_active":"2024-03-30T15:01:13.219436+0000","last_peered":"2024-03-30T15:01:13.219436+0000","last_clean":"2024-03-30T15:01:13.219436+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:01:13.219436+0000","last_undegraded":"2024-03-30T15:01:13.219436+0000","last_fullsized":"2024-03-30T15:01:13.219436+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:02.714228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.714228+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055223779999999997,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.0","version":"248'39","reported_seq":539,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.218665+0000","last_change":"2024-03-30T15:01:12.231821+0000","last_active":"2024-03-30T15:01:13.218665+0000","last_peered":"2024-03-30T15:01:13.218665+0000","last_clean":"2024-03-30T15:01:13.218665+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:01:13.218665+0000","last_undegraded":"2024-03-30T15:01:13.218665+0000","last_fullsized":"2024-03-30T15:01:13.218665+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T15:00:04.710962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.710962+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.256385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.005125569,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.1","version":"238'103","reported_seq":596,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244955+0000","last_change":"2024-03-30T15:01:12.245189+0000","last_active":"2024-03-30T15:01:12.244955+0000","last_peered":"2024-03-30T15:01:12.244955+0000","last_clean":"2024-03-30T15:01:12.244955+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:01:12.244955+0000","last_undegraded":"2024-03-30T15:01:12.244955+0000","last_fullsized":"2024-03-30T15:01:12.244955+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'103","last_scrub_stamp":"2024-03-30T15:00:35.786699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.786699+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:54.343344+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001509961,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.6","version":"204'22","reported_seq":519,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226416+0000","last_change":"2024-03-30T15:01:12.223859+0000","last_active":"2024-03-30T15:01:13.226416+0000","last_peered":"2024-03-30T15:01:13.226416+0000","last_clean":"2024-03-30T15:01:13.226416+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:01:13.226416+0000","last_undegraded":"2024-03-30T15:01:13.226416+0000","last_fullsized":"2024-03-30T15:01:13.226416+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046413940000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.7","version":"0'0","reported_seq":505,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220990+0000","last_change":"2024-03-30T15:01:12.237147+0000","last_active":"2024-03-30T15:01:13.220990+0000","last_peered":"2024-03-30T15:01:13.220990+0000","last_clean":"2024-03-30T15:01:13.220990+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:01:13.220990+0000","last_undegraded":"2024-03-30T15:01:13.220990+0000","last_fullsized":"2024-03-30T15:01:13.220990+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.644063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.644063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:53.265770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063778710000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.4","version":"235'68","reported_seq":551,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590858+0000","last_change":"2024-03-30T15:01:12.244784+0000","last_active":"2024-03-30T15:01:13.590858+0000","last_peered":"2024-03-30T15:01:13.590858+0000","last_clean":"2024-03-30T15:01:13.590858+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:01:13.590858+0000","last_undegraded":"2024-03-30T15:01:13.590858+0000","last_fullsized":"2024-03-30T15:01:13.590858+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.034736+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065541890000000002,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"101.5","version":"204'102","reported_seq":589,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223240+0000","last_change":"2024-03-30T15:01:12.229254+0000","last_active":"2024-03-30T15:01:13.223240+0000","last_peered":"2024-03-30T15:01:13.223240+0000","last_clean":"2024-03-30T15:01:13.223240+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:01:13.223240+0000","last_undegraded":"2024-03-30T15:01:13.223240+0000","last_fullsized":"2024-03-30T15:01:13.223240+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T14:59:48.278271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.278271+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016655120000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"42.15","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222837+0000","last_change":"2024-03-30T14:57:56.252990+0000","last_active":"2024-03-30T15:01:13.222837+0000","last_peered":"2024-03-30T15:01:13.222837+0000","last_clean":"2024-03-30T15:01:13.222837+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:01:13.222837+0000","last_undegraded":"2024-03-30T15:01:13.222837+0000","last_fullsized":"2024-03-30T15:01:13.222837+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242755+0000","last_change":"2024-03-30T14:59:10.495352+0000","last_active":"2024-03-30T15:01:12.242755+0000","last_peered":"2024-03-30T15:01:12.242755+0000","last_clean":"2024-03-30T15:01:12.242755+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:01:12.242755+0000","last_undegraded":"2024-03-30T15:01:12.242755+0000","last_fullsized":"2024-03-30T15:01:12.242755+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.495272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.495272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":405,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.254027+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:01:12.254027+0000","last_peered":"2024-03-30T15:01:12.254027+0000","last_clean":"2024-03-30T15:01:12.254027+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:01:12.254027+0000","last_undegraded":"2024-03-30T15:01:12.254027+0000","last_fullsized":"2024-03-30T15:01:12.254027+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.937324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.219740+0000","last_change":"2024-03-30T15:00:50.386813+0000","last_active":"2024-03-30T15:01:12.219740+0000","last_peered":"2024-03-30T15:01:12.219740+0000","last_clean":"2024-03-30T15:01:12.219740+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:01:12.219740+0000","last_undegraded":"2024-03-30T15:01:12.219740+0000","last_fullsized":"2024-03-30T15:01:12.219740+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:50.386674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.386674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:17.393728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":400,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590181+0000","last_change":"2024-03-30T14:57:56.261442+0000","last_active":"2024-03-30T15:01:13.590181+0000","last_peered":"2024-03-30T15:01:13.590181+0000","last_clean":"2024-03-30T15:01:13.590181+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:01:13.590181+0000","last_undegraded":"2024-03-30T15:01:13.590181+0000","last_fullsized":"2024-03-30T15:01:13.590181+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.218611+0000","last_change":"2024-03-30T14:59:56.788394+0000","last_active":"2024-03-30T15:01:13.218611+0000","last_peered":"2024-03-30T15:01:13.218611+0000","last_clean":"2024-03-30T15:01:13.218611+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:01:13.218611+0000","last_undegraded":"2024-03-30T15:01:13.218611+0000","last_fullsized":"2024-03-30T15:01:13.218611+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.788278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.788278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:25.851592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":397,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.252275+0000","last_change":"2024-03-30T14:57:56.244353+0000","last_active":"2024-03-30T15:01:12.252275+0000","last_peered":"2024-03-30T15:01:12.252275+0000","last_clean":"2024-03-30T15:01:12.252275+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:01:12.252275+0000","last_undegraded":"2024-03-30T15:01:12.252275+0000","last_fullsized":"2024-03-30T15:01:12.252275+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221727+0000","last_change":"2024-03-30T14:57:56.239220+0000","last_active":"2024-03-30T15:01:13.221727+0000","last_peered":"2024-03-30T15:01:13.221727+0000","last_clean":"2024-03-30T15:01:13.221727+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:01:13.221727+0000","last_undegraded":"2024-03-30T15:01:13.221727+0000","last_fullsized":"2024-03-30T15:01:13.221727+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224787+0000","last_change":"2024-03-30T14:57:56.240474+0000","last_active":"2024-03-30T15:01:13.224787+0000","last_peered":"2024-03-30T15:01:13.224787+0000","last_clean":"2024-03-30T15:01:13.224787+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:01:13.224787+0000","last_undegraded":"2024-03-30T15:01:13.224787+0000","last_fullsized":"2024-03-30T15:01:13.224787+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.218424+0000","last_change":"2024-03-30T15:00:43.330329+0000","last_active":"2024-03-30T15:01:12.218424+0000","last_peered":"2024-03-30T15:01:12.218424+0000","last_clean":"2024-03-30T15:01:12.218424+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:01:12.218424+0000","last_undegraded":"2024-03-30T15:01:12.218424+0000","last_fullsized":"2024-03-30T15:01:12.218424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.330250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:50.998934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":405,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.319503+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:01:12.319503+0000","last_peered":"2024-03-30T15:01:12.319503+0000","last_clean":"2024-03-30T15:01:12.319503+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:01:12.319503+0000","last_undegraded":"2024-03-30T15:01:12.319503+0000","last_fullsized":"2024-03-30T15:01:12.319503+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:39.883961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220653+0000","last_change":"2024-03-30T14:59:52.803098+0000","last_active":"2024-03-30T15:01:13.220653+0000","last_peered":"2024-03-30T15:01:13.220653+0000","last_clean":"2024-03-30T15:01:13.220653+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:01:13.220653+0000","last_undegraded":"2024-03-30T15:01:13.220653+0000","last_fullsized":"2024-03-30T15:01:13.220653+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.803012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.803012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.827147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":397,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251556+0000","last_change":"2024-03-30T14:57:56.253984+0000","last_active":"2024-03-30T15:01:12.251556+0000","last_peered":"2024-03-30T15:01:12.251556+0000","last_clean":"2024-03-30T15:01:12.251556+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:01:12.251556+0000","last_undegraded":"2024-03-30T15:01:12.251556+0000","last_fullsized":"2024-03-30T15:01:12.251556+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221162+0000","last_change":"2024-03-30T15:00:40.305048+0000","last_active":"2024-03-30T15:01:12.221162+0000","last_peered":"2024-03-30T15:01:12.221162+0000","last_clean":"2024-03-30T15:01:12.221162+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:01:12.221162+0000","last_undegraded":"2024-03-30T15:01:12.221162+0000","last_fullsized":"2024-03-30T15:01:12.221162+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:40.304903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:40.304903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:04.820755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219618+0000","last_change":"2024-03-30T14:59:17.774212+0000","last_active":"2024-03-30T15:01:13.219618+0000","last_peered":"2024-03-30T15:01:13.219618+0000","last_clean":"2024-03-30T15:01:13.219618+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:01:13.219618+0000","last_undegraded":"2024-03-30T15:01:13.219618+0000","last_fullsized":"2024-03-30T15:01:13.219618+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:17.774120+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.774120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221152+0000","last_change":"2024-03-30T14:59:45.812010+0000","last_active":"2024-03-30T15:01:13.221152+0000","last_peered":"2024-03-30T15:01:13.221152+0000","last_clean":"2024-03-30T15:01:13.221152+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:01:13.221152+0000","last_undegraded":"2024-03-30T15:01:13.221152+0000","last_fullsized":"2024-03-30T15:01:13.221152+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:45.811929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:45.811929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":405,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253274+0000","last_change":"2024-03-30T15:01:10.292440+0000","last_active":"2024-03-30T15:01:12.253274+0000","last_peered":"2024-03-30T15:01:12.253274+0000","last_clean":"2024-03-30T15:01:12.253274+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:01:12.253274+0000","last_undegraded":"2024-03-30T15:01:12.253274+0000","last_fullsized":"2024-03-30T15:01:12.253274+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.292346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.292346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:31.381291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225242+0000","last_change":"2024-03-30T14:57:56.226093+0000","last_active":"2024-03-30T15:01:13.225242+0000","last_peered":"2024-03-30T15:01:13.225242+0000","last_clean":"2024-03-30T15:01:13.225242+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:01:13.225242+0000","last_undegraded":"2024-03-30T15:01:13.225242+0000","last_fullsized":"2024-03-30T15:01:13.225242+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224648+0000","last_change":"2024-03-30T14:57:56.276413+0000","last_active":"2024-03-30T15:01:13.224648+0000","last_peered":"2024-03-30T15:01:13.224648+0000","last_clean":"2024-03-30T15:01:13.224648+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:01:13.224648+0000","last_undegraded":"2024-03-30T15:01:13.224648+0000","last_fullsized":"2024-03-30T15:01:13.224648+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":397,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253416+0000","last_change":"2024-03-30T14:57:56.253085+0000","last_active":"2024-03-30T15:01:12.253416+0000","last_peered":"2024-03-30T15:01:12.253416+0000","last_clean":"2024-03-30T15:01:12.253416+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:01:12.253416+0000","last_undegraded":"2024-03-30T15:01:12.253416+0000","last_fullsized":"2024-03-30T15:01:12.253416+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.217826+0000","last_change":"2024-03-30T14:59:55.177743+0000","last_active":"2024-03-30T15:01:12.217826+0000","last_peered":"2024-03-30T15:01:12.217826+0000","last_clean":"2024-03-30T15:01:12.217826+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:01:12.217826+0000","last_undegraded":"2024-03-30T15:01:12.217826+0000","last_fullsized":"2024-03-30T15:01:12.217826+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.177623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.177623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":405,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251405+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:01:12.251405+0000","last_peered":"2024-03-30T15:01:12.251405+0000","last_clean":"2024-03-30T15:01:12.251405+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:01:12.251405+0000","last_undegraded":"2024-03-30T15:01:12.251405+0000","last_fullsized":"2024-03-30T15:01:12.251405+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242914+0000","last_change":"2024-03-30T15:00:32.754787+0000","last_active":"2024-03-30T15:01:12.242914+0000","last_peered":"2024-03-30T15:01:12.242914+0000","last_clean":"2024-03-30T15:01:12.242914+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:01:12.242914+0000","last_undegraded":"2024-03-30T15:01:12.242914+0000","last_fullsized":"2024-03-30T15:01:12.242914+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.754592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.754592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:49.958130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219121+0000","last_change":"2024-03-30T14:59:13.804909+0000","last_active":"2024-03-30T15:01:13.219121+0000","last_peered":"2024-03-30T15:01:13.219121+0000","last_clean":"2024-03-30T15:01:13.219121+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:01:13.219121+0000","last_undegraded":"2024-03-30T15:01:13.219121+0000","last_fullsized":"2024-03-30T15:01:13.219121+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.804817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.804817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244005+0000","last_change":"2024-03-30T15:00:33.756912+0000","last_active":"2024-03-30T15:01:12.244005+0000","last_peered":"2024-03-30T15:01:12.244005+0000","last_clean":"2024-03-30T15:01:12.244005+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:01:12.244005+0000","last_undegraded":"2024-03-30T15:01:12.244005+0000","last_fullsized":"2024-03-30T15:01:12.244005+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:33.756810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.756810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:33.994340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.218529+0000","last_change":"2024-03-30T15:00:03.114650+0000","last_active":"2024-03-30T15:01:12.218529+0000","last_peered":"2024-03-30T15:01:12.218529+0000","last_clean":"2024-03-30T15:01:12.218529+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:01:12.218529+0000","last_undegraded":"2024-03-30T15:01:12.218529+0000","last_fullsized":"2024-03-30T15:01:12.218529+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.114576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.114576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225472+0000","last_change":"2024-03-30T14:57:56.260622+0000","last_active":"2024-03-30T15:01:13.225472+0000","last_peered":"2024-03-30T15:01:13.225472+0000","last_clean":"2024-03-30T15:01:13.225472+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:01:13.225472+0000","last_undegraded":"2024-03-30T15:01:13.225472+0000","last_fullsized":"2024-03-30T15:01:13.225472+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223969+0000","last_change":"2024-03-30T14:57:56.249378+0000","last_active":"2024-03-30T15:01:13.223969+0000","last_peered":"2024-03-30T15:01:13.223969+0000","last_clean":"2024-03-30T15:01:13.223969+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:01:13.223969+0000","last_undegraded":"2024-03-30T15:01:13.223969+0000","last_fullsized":"2024-03-30T15:01:13.223969+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242879+0000","last_change":"2024-03-30T14:59:11.544255+0000","last_active":"2024-03-30T15:01:12.242879+0000","last_peered":"2024-03-30T15:01:12.242879+0000","last_clean":"2024-03-30T15:01:12.242879+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:01:12.242879+0000","last_undegraded":"2024-03-30T15:01:12.242879+0000","last_fullsized":"2024-03-30T15:01:12.242879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:11.544174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:11.544174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221383+0000","last_change":"2024-03-30T14:57:56.271774+0000","last_active":"2024-03-30T15:01:13.221383+0000","last_peered":"2024-03-30T15:01:13.221383+0000","last_clean":"2024-03-30T15:01:13.221383+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:01:13.221383+0000","last_undegraded":"2024-03-30T15:01:13.221383+0000","last_fullsized":"2024-03-30T15:01:13.221383+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":399,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224476+0000","last_change":"2024-03-30T14:57:56.199752+0000","last_active":"2024-03-30T15:01:13.224476+0000","last_peered":"2024-03-30T15:01:13.224476+0000","last_clean":"2024-03-30T15:01:13.224476+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:01:13.224476+0000","last_undegraded":"2024-03-30T15:01:13.224476+0000","last_fullsized":"2024-03-30T15:01:13.224476+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":397,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251647+0000","last_change":"2024-03-30T14:57:56.228301+0000","last_active":"2024-03-30T15:01:12.251647+0000","last_peered":"2024-03-30T15:01:12.251647+0000","last_clean":"2024-03-30T15:01:12.251647+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:01:12.251647+0000","last_undegraded":"2024-03-30T15:01:12.251647+0000","last_fullsized":"2024-03-30T15:01:12.251647+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:57:53.916755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242867+0000","last_change":"2024-03-30T14:59:09.501246+0000","last_active":"2024-03-30T15:01:12.242867+0000","last_peered":"2024-03-30T15:01:12.242867+0000","last_clean":"2024-03-30T15:01:12.242867+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:01:12.242867+0000","last_undegraded":"2024-03-30T15:01:12.242867+0000","last_fullsized":"2024-03-30T15:01:12.242867+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.501169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.501169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":518,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226303+0000","last_change":"2024-03-30T14:58:15.280305+0000","last_active":"2024-03-30T15:01:13.226303+0000","last_peered":"2024-03-30T15:01:13.226303+0000","last_clean":"2024-03-30T15:01:13.226303+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:01:13.226303+0000","last_undegraded":"2024-03-30T15:01:13.226303+0000","last_fullsized":"2024-03-30T15:01:13.226303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:15.280217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.280217+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":410,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590187+0000","last_change":"2024-03-30T14:58:13.189214+0000","last_active":"2024-03-30T15:01:13.590187+0000","last_peered":"2024-03-30T15:01:13.590187+0000","last_clean":"2024-03-30T15:01:13.590187+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:01:13.590187+0000","last_undegraded":"2024-03-30T15:01:13.590187+0000","last_fullsized":"2024-03-30T15:01:13.590187+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:13.189081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.189081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":594,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220220+0000","last_change":"2024-03-30T14:58:31.806764+0000","last_active":"2024-03-30T15:01:13.220220+0000","last_peered":"2024-03-30T15:01:13.220220+0000","last_clean":"2024-03-30T15:01:13.220220+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:01:13.220220+0000","last_undegraded":"2024-03-30T15:01:13.220220+0000","last_fullsized":"2024-03-30T15:01:13.220220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:31.806606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.806606+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226777+0000","last_change":"2024-03-30T14:59:13.422572+0000","last_active":"2024-03-30T15:01:13.226777+0000","last_peered":"2024-03-30T15:01:13.226777+0000","last_clean":"2024-03-30T15:01:13.226777+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:01:13.226777+0000","last_undegraded":"2024-03-30T15:01:13.226777+0000","last_fullsized":"2024-03-30T15:01:13.226777+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.422453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.422453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244922+0000","last_change":"2024-03-30T15:00:04.553363+0000","last_active":"2024-03-30T15:01:12.244922+0000","last_peered":"2024-03-30T15:01:12.244922+0000","last_clean":"2024-03-30T15:01:12.244922+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:01:12.244922+0000","last_undegraded":"2024-03-30T15:01:12.244922+0000","last_fullsized":"2024-03-30T15:01:12.244922+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.553281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.553281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":539,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225139+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:01:13.225139+0000","last_peered":"2024-03-30T15:01:13.225139+0000","last_clean":"2024-03-30T15:01:13.225139+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:01:13.225139+0000","last_undegraded":"2024-03-30T15:01:13.225139+0000","last_fullsized":"2024-03-30T15:01:13.225139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":514,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242717+0000","last_change":"2024-03-30T14:59:18.646158+0000","last_active":"2024-03-30T15:01:12.242717+0000","last_peered":"2024-03-30T15:01:12.242717+0000","last_clean":"2024-03-30T15:01:12.242717+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:01:12.242717+0000","last_undegraded":"2024-03-30T15:01:12.242717+0000","last_fullsized":"2024-03-30T15:01:12.242717+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:18.646076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:18.646076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":506,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221308+0000","last_change":"2024-03-30T14:59:09.325167+0000","last_active":"2024-03-30T15:01:12.221308+0000","last_peered":"2024-03-30T15:01:12.221308+0000","last_clean":"2024-03-30T15:01:12.221308+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:01:12.221308+0000","last_undegraded":"2024-03-30T15:01:12.221308+0000","last_fullsized":"2024-03-30T15:01:12.221308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":502,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221041+0000","last_change":"2024-03-30T14:59:07.831752+0000","last_active":"2024-03-30T15:01:13.221041+0000","last_peered":"2024-03-30T15:01:13.221041+0000","last_clean":"2024-03-30T15:01:13.221041+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:01:13.221041+0000","last_undegraded":"2024-03-30T15:01:13.221041+0000","last_fullsized":"2024-03-30T15:01:13.221041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.831675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.831675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.18","version":"115'2","reported_seq":101,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222403+0000","last_change":"2024-03-30T15:00:45.254965+0000","last_active":"2024-03-30T15:01:13.222403+0000","last_peered":"2024-03-30T15:01:13.222403+0000","last_clean":"2024-03-30T15:01:13.222403+0000","last_became_active":"2024-03-30T15:00:45.247625+0000","last_became_peered":"2024-03-30T15:00:45.247625+0000","last_unstale":"2024-03-30T15:01:13.222403+0000","last_undegraded":"2024-03-30T15:01:13.222403+0000","last_fullsized":"2024-03-30T15:01:13.222403+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":412,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220282+0000","last_change":"2024-03-30T14:59:54.144187+0000","last_active":"2024-03-30T15:01:12.220282+0000","last_peered":"2024-03-30T15:01:12.220282+0000","last_clean":"2024-03-30T15:01:12.220282+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:01:12.220282+0000","last_undegraded":"2024-03-30T15:01:12.220282+0000","last_fullsized":"2024-03-30T15:01:12.220282+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:54.144084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:54.144084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":527,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226654+0000","last_change":"2024-03-30T14:58:20.234367+0000","last_active":"2024-03-30T15:01:13.226654+0000","last_peered":"2024-03-30T15:01:13.226654+0000","last_clean":"2024-03-30T15:01:13.226654+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:01:13.226654+0000","last_undegraded":"2024-03-30T15:01:13.226654+0000","last_fullsized":"2024-03-30T15:01:13.226654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T14:58:20.234281+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T14:58:20.234281+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":508,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224084+0000","last_change":"2024-03-30T14:57:29.596356+0000","last_active":"2024-03-30T15:01:13.224084+0000","last_peered":"2024-03-30T15:01:13.224084+0000","last_clean":"2024-03-30T15:01:13.224084+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:01:13.224084+0000","last_undegraded":"2024-03-30T15:01:13.224084+0000","last_fullsized":"2024-03-30T15:01:13.224084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:29.596249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:29.596249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":496,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.254908+0000","last_change":"2024-03-30T14:58:23.099186+0000","last_active":"2024-03-30T15:01:12.254908+0000","last_peered":"2024-03-30T15:01:12.254908+0000","last_clean":"2024-03-30T15:01:12.254908+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:01:12.254908+0000","last_undegraded":"2024-03-30T15:01:12.254908+0000","last_fullsized":"2024-03-30T15:01:12.254908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.099104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.099104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":415,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243058+0000","last_change":"2024-03-30T15:00:44.895646+0000","last_active":"2024-03-30T15:01:12.243058+0000","last_peered":"2024-03-30T15:01:12.243058+0000","last_clean":"2024-03-30T15:01:12.243058+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:01:12.243058+0000","last_undegraded":"2024-03-30T15:01:12.243058+0000","last_fullsized":"2024-03-30T15:01:12.243058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.895504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.895504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:59.201212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":412,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220016+0000","last_change":"2024-03-30T15:00:01.176797+0000","last_active":"2024-03-30T15:01:12.220016+0000","last_peered":"2024-03-30T15:01:12.220016+0000","last_clean":"2024-03-30T15:01:12.220016+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:01:12.220016+0000","last_undegraded":"2024-03-30T15:01:12.220016+0000","last_fullsized":"2024-03-30T15:01:12.220016+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.176673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220086+0000","last_change":"2024-03-30T14:58:31.256279+0000","last_active":"2024-03-30T15:01:12.220086+0000","last_peered":"2024-03-30T15:01:12.220086+0000","last_clean":"2024-03-30T15:01:12.220086+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:01:12.220086+0000","last_undegraded":"2024-03-30T15:01:12.220086+0000","last_fullsized":"2024-03-30T15:01:12.220086+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T14:58:31.256139+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.256139+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221689+0000","last_change":"2024-03-30T14:59:07.286916+0000","last_active":"2024-03-30T15:01:12.221689+0000","last_peered":"2024-03-30T15:01:12.221689+0000","last_clean":"2024-03-30T15:01:12.221689+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:01:12.221689+0000","last_undegraded":"2024-03-30T15:01:12.221689+0000","last_fullsized":"2024-03-30T15:01:12.221689+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.286790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.286790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":587,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226720+0000","last_change":"2024-03-30T14:58:59.314094+0000","last_active":"2024-03-30T15:01:13.226720+0000","last_peered":"2024-03-30T15:01:13.226720+0000","last_clean":"2024-03-30T15:01:13.226720+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:01:13.226720+0000","last_undegraded":"2024-03-30T15:01:13.226720+0000","last_fullsized":"2024-03-30T15:01:13.226720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.314003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.314003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":492,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253472+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:01:12.253472+0000","last_peered":"2024-03-30T15:01:12.253472+0000","last_clean":"2024-03-30T15:01:12.253472+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:01:12.253472+0000","last_undegraded":"2024-03-30T15:01:12.253472+0000","last_fullsized":"2024-03-30T15:01:12.253472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219097+0000","last_change":"2024-03-30T14:59:53.801443+0000","last_active":"2024-03-30T15:01:13.219097+0000","last_peered":"2024-03-30T15:01:13.219097+0000","last_clean":"2024-03-30T15:01:13.219097+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:01:13.219097+0000","last_undegraded":"2024-03-30T15:01:13.219097+0000","last_fullsized":"2024-03-30T15:01:13.219097+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.801332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.801332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":500,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223875+0000","last_change":"2024-03-30T14:58:21.250924+0000","last_active":"2024-03-30T15:01:13.223875+0000","last_peered":"2024-03-30T15:01:13.223875+0000","last_clean":"2024-03-30T15:01:13.223875+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:01:13.223875+0000","last_undegraded":"2024-03-30T15:01:13.223875+0000","last_fullsized":"2024-03-30T15:01:13.223875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:21.250827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.250827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":498,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222363+0000","last_change":"2024-03-30T14:58:19.963766+0000","last_active":"2024-03-30T15:01:13.222363+0000","last_peered":"2024-03-30T15:01:13.222363+0000","last_clean":"2024-03-30T15:01:13.222363+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:01:13.222363+0000","last_undegraded":"2024-03-30T15:01:13.222363+0000","last_fullsized":"2024-03-30T15:01:13.222363+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T14:58:19.963665+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.963665+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":482,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224770+0000","last_change":"2024-03-30T14:59:10.014828+0000","last_active":"2024-03-30T15:01:13.224770+0000","last_peered":"2024-03-30T15:01:13.224770+0000","last_clean":"2024-03-30T15:01:13.224770+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:01:13.224770+0000","last_undegraded":"2024-03-30T15:01:13.224770+0000","last_fullsized":"2024-03-30T15:01:13.224770+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:10.014745+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.014745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":496,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224512+0000","last_change":"2024-03-30T14:59:02.261089+0000","last_active":"2024-03-30T15:01:13.224512+0000","last_peered":"2024-03-30T15:01:13.224512+0000","last_clean":"2024-03-30T15:01:13.224512+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:01:13.224512+0000","last_undegraded":"2024-03-30T15:01:13.224512+0000","last_fullsized":"2024-03-30T15:01:13.224512+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.260964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.260964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":77,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243860+0000","last_change":"2024-03-30T15:00:45.252550+0000","last_active":"2024-03-30T15:01:12.243860+0000","last_peered":"2024-03-30T15:01:12.243860+0000","last_clean":"2024-03-30T15:01:12.243860+0000","last_became_active":"2024-03-30T15:00:45.251660+0000","last_became_peered":"2024-03-30T15:00:45.251660+0000","last_unstale":"2024-03-30T15:01:12.243860+0000","last_undegraded":"2024-03-30T15:01:12.243860+0000","last_fullsized":"2024-03-30T15:01:12.243860+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.119871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.119871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.398361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":415,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.218987+0000","last_change":"2024-03-30T14:59:48.794962+0000","last_active":"2024-03-30T15:01:13.218987+0000","last_peered":"2024-03-30T15:01:13.218987+0000","last_clean":"2024-03-30T15:01:13.218987+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:01:13.218987+0000","last_undegraded":"2024-03-30T15:01:13.218987+0000","last_fullsized":"2024-03-30T15:01:13.218987+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:48.794824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:48.794824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":472,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221487+0000","last_change":"2024-03-30T14:59:49.053482+0000","last_active":"2024-03-30T15:01:12.221487+0000","last_peered":"2024-03-30T15:01:12.221487+0000","last_clean":"2024-03-30T15:01:12.221487+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:01:12.221487+0000","last_undegraded":"2024-03-30T15:01:12.221487+0000","last_fullsized":"2024-03-30T15:01:12.221487+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:49.053394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:49.053394+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":533,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224227+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:01:13.224227+0000","last_peered":"2024-03-30T15:01:13.224227+0000","last_clean":"2024-03-30T15:01:13.224227+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:01:13.224227+0000","last_undegraded":"2024-03-30T15:01:13.224227+0000","last_fullsized":"2024-03-30T15:01:13.224227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":511,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591169+0000","last_change":"2024-03-30T14:58:25.170349+0000","last_active":"2024-03-30T15:01:13.591169+0000","last_peered":"2024-03-30T15:01:13.591169+0000","last_clean":"2024-03-30T15:01:13.591169+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:01:13.591169+0000","last_undegraded":"2024-03-30T15:01:13.591169+0000","last_fullsized":"2024-03-30T15:01:13.591169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.170239+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":504,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221752+0000","last_change":"2024-03-30T14:59:56.291563+0000","last_active":"2024-03-30T15:01:13.221752+0000","last_peered":"2024-03-30T15:01:13.221752+0000","last_clean":"2024-03-30T15:01:13.221752+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:01:13.221752+0000","last_undegraded":"2024-03-30T15:01:13.221752+0000","last_fullsized":"2024-03-30T15:01:13.221752+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.291453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.291453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.14","version":"115'2","reported_seq":91,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590328+0000","last_change":"2024-03-30T15:01:13.588926+0000","last_active":"2024-03-30T15:01:13.590328+0000","last_peered":"2024-03-30T15:01:13.590328+0000","last_clean":"2024-03-30T15:01:13.590328+0000","last_became_active":"2024-03-30T15:00:45.241122+0000","last_became_peered":"2024-03-30T15:00:45.241122+0000","last_unstale":"2024-03-30T15:01:13.590328+0000","last_undegraded":"2024-03-30T15:01:13.590328+0000","last_fullsized":"2024-03-30T15:01:13.590328+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-03-30T15:01:13.588786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:13.588786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:14.792753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":415,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219410+0000","last_change":"2024-03-30T14:59:46.819021+0000","last_active":"2024-03-30T15:01:13.219410+0000","last_peered":"2024-03-30T15:01:13.219410+0000","last_clean":"2024-03-30T15:01:13.219410+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:01:13.219410+0000","last_undegraded":"2024-03-30T15:01:13.219410+0000","last_fullsized":"2024-03-30T15:01:13.219410+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:46.818941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.818941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":597,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590185+0000","last_change":"2024-03-30T15:00:30.490223+0000","last_active":"2024-03-30T15:01:13.590185+0000","last_peered":"2024-03-30T15:01:13.590185+0000","last_clean":"2024-03-30T15:01:13.590185+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:01:13.590185+0000","last_undegraded":"2024-03-30T15:01:13.590185+0000","last_fullsized":"2024-03-30T15:01:13.590185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:00:30.490104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:30.490104+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.592628+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":531,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221149+0000","last_change":"2024-03-30T14:59:10.283945+0000","last_active":"2024-03-30T15:01:12.221149+0000","last_peered":"2024-03-30T15:01:12.221149+0000","last_clean":"2024-03-30T15:01:12.221149+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:01:12.221149+0000","last_undegraded":"2024-03-30T15:01:12.221149+0000","last_fullsized":"2024-03-30T15:01:12.221149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.283859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":513,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253586+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:01:12.253586+0000","last_peered":"2024-03-30T15:01:12.253586+0000","last_clean":"2024-03-30T15:01:12.253586+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:01:12.253586+0000","last_undegraded":"2024-03-30T15:01:12.253586+0000","last_fullsized":"2024-03-30T15:01:12.253586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":497,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589064+0000","last_change":"2024-03-30T14:58:23.157209+0000","last_active":"2024-03-30T15:01:13.589064+0000","last_peered":"2024-03-30T15:01:13.589064+0000","last_clean":"2024-03-30T15:01:13.589064+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:01:13.589064+0000","last_undegraded":"2024-03-30T15:01:13.589064+0000","last_fullsized":"2024-03-30T15:01:13.589064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.157095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.157095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.17","version":"301'7","reported_seq":104,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.592276+0000","last_change":"2024-03-30T15:00:45.253584+0000","last_active":"2024-03-30T15:01:13.592276+0000","last_peered":"2024-03-30T15:01:13.592276+0000","last_clean":"2024-03-30T15:01:13.592276+0000","last_became_active":"2024-03-30T15:00:45.243133+0000","last_became_peered":"2024-03-30T15:00:45.243133+0000","last_unstale":"2024-03-30T15:01:13.592276+0000","last_undegraded":"2024-03-30T15:01:13.592276+0000","last_fullsized":"2024-03-30T15:01:13.592276+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T14:59:15.591612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.591612+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220429+0000","last_change":"2024-03-30T14:59:19.788455+0000","last_active":"2024-03-30T15:01:13.220429+0000","last_peered":"2024-03-30T15:01:13.220429+0000","last_clean":"2024-03-30T15:01:13.220429+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:01:13.220429+0000","last_undegraded":"2024-03-30T15:01:13.220429+0000","last_fullsized":"2024-03-30T15:01:13.220429+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.788329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.788329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":506,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243517+0000","last_change":"2024-03-30T14:59:01.518493+0000","last_active":"2024-03-30T15:01:12.243517+0000","last_peered":"2024-03-30T15:01:12.243517+0000","last_clean":"2024-03-30T15:01:12.243517+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:01:12.243517+0000","last_undegraded":"2024-03-30T15:01:12.243517+0000","last_fullsized":"2024-03-30T15:01:12.243517+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.518414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.518414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":549,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590857+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:01:13.590857+0000","last_peered":"2024-03-30T15:01:13.590857+0000","last_clean":"2024-03-30T15:01:13.590857+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:01:13.590857+0000","last_undegraded":"2024-03-30T15:01:13.590857+0000","last_fullsized":"2024-03-30T15:01:13.590857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":515,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221196+0000","last_change":"2024-03-30T14:59:13.210437+0000","last_active":"2024-03-30T15:01:12.221196+0000","last_peered":"2024-03-30T15:01:12.221196+0000","last_clean":"2024-03-30T15:01:12.221196+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:01:12.221196+0000","last_undegraded":"2024-03-30T15:01:12.221196+0000","last_fullsized":"2024-03-30T15:01:12.221196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.210302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.210302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":503,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.218464+0000","last_change":"2024-03-30T14:59:19.135083+0000","last_active":"2024-03-30T15:01:12.218464+0000","last_peered":"2024-03-30T15:01:12.218464+0000","last_clean":"2024-03-30T15:01:12.218464+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:01:12.218464+0000","last_undegraded":"2024-03-30T15:01:12.218464+0000","last_fullsized":"2024-03-30T15:01:12.218464+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:19.135006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:19.135006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.16","version":"301'7","reported_seq":84,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245490+0000","last_change":"2024-03-30T15:00:45.258419+0000","last_active":"2024-03-30T15:01:12.245490+0000","last_peered":"2024-03-30T15:01:12.245490+0000","last_clean":"2024-03-30T15:01:12.245490+0000","last_became_active":"2024-03-30T15:00:45.244278+0000","last_became_peered":"2024-03-30T15:00:45.244278+0000","last_unstale":"2024-03-30T15:01:12.245490+0000","last_undegraded":"2024-03-30T15:01:12.245490+0000","last_fullsized":"2024-03-30T15:01:12.245490+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.109233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.109233+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:36.629475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":406,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591136+0000","last_change":"2024-03-30T14:57:54.026635+0000","last_active":"2024-03-30T15:01:13.591136+0000","last_peered":"2024-03-30T15:01:13.591136+0000","last_clean":"2024-03-30T15:01:13.591136+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:01:13.591136+0000","last_undegraded":"2024-03-30T15:01:13.591136+0000","last_fullsized":"2024-03-30T15:01:13.591136+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":505,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.254924+0000","last_change":"2024-03-30T14:59:15.207502+0000","last_active":"2024-03-30T15:01:12.254924+0000","last_peered":"2024-03-30T15:01:12.254924+0000","last_clean":"2024-03-30T15:01:12.254924+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:01:12.254924+0000","last_undegraded":"2024-03-30T15:01:12.254924+0000","last_fullsized":"2024-03-30T15:01:12.254924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.207413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.207413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":531,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591082+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:01:13.591082+0000","last_peered":"2024-03-30T15:01:13.591082+0000","last_clean":"2024-03-30T15:01:13.591082+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:01:13.591082+0000","last_undegraded":"2024-03-30T15:01:13.591082+0000","last_fullsized":"2024-03-30T15:01:13.591082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":909,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222154+0000","last_change":"2024-03-30T14:58:21.947917+0000","last_active":"2024-03-30T15:01:13.222154+0000","last_peered":"2024-03-30T15:01:13.222154+0000","last_clean":"2024-03-30T15:01:13.222154+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:01:13.222154+0000","last_undegraded":"2024-03-30T15:01:13.222154+0000","last_fullsized":"2024-03-30T15:01:13.222154+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T14:58:21.947837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:21.947837+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":389,"num_read_kb":384,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":509,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589825+0000","last_change":"2024-03-30T14:59:05.612363+0000","last_active":"2024-03-30T15:01:13.589825+0000","last_peered":"2024-03-30T15:01:13.589825+0000","last_clean":"2024-03-30T15:01:13.589825+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:01:13.589825+0000","last_undegraded":"2024-03-30T15:01:13.589825+0000","last_fullsized":"2024-03-30T15:01:13.589825+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.612262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.612262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":503,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.423748+0000","last_change":"2024-03-30T15:01:12.423769+0000","last_active":"2024-03-30T15:01:12.423748+0000","last_peered":"2024-03-30T15:01:12.423748+0000","last_clean":"2024-03-30T15:01:12.423748+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:01:12.423748+0000","last_undegraded":"2024-03-30T15:01:12.423748+0000","last_fullsized":"2024-03-30T15:01:12.423748+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.423680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.423680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:37.670999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"301'11","reported_seq":104,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224151+0000","last_change":"2024-03-30T15:01:07.548867+0000","last_active":"2024-03-30T15:01:13.224151+0000","last_peered":"2024-03-30T15:01:13.224151+0000","last_clean":"2024-03-30T15:01:13.224151+0000","last_became_active":"2024-03-30T15:00:45.242541+0000","last_became_peered":"2024-03-30T15:00:45.242541+0000","last_unstale":"2024-03-30T15:01:13.224151+0000","last_undegraded":"2024-03-30T15:01:13.224151+0000","last_fullsized":"2024-03-30T15:01:13.224151+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'11","last_scrub_stamp":"2024-03-30T15:01:07.548781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:07.548781+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:34.037634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":404,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223053+0000","last_change":"2024-03-30T14:57:55.140517+0000","last_active":"2024-03-30T15:01:13.223053+0000","last_peered":"2024-03-30T15:01:13.223053+0000","last_clean":"2024-03-30T15:01:13.223053+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:01:13.223053+0000","last_undegraded":"2024-03-30T15:01:13.223053+0000","last_fullsized":"2024-03-30T15:01:13.223053+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":500,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223064+0000","last_change":"2024-03-30T14:58:35.074272+0000","last_active":"2024-03-30T15:01:13.223064+0000","last_peered":"2024-03-30T15:01:13.223064+0000","last_clean":"2024-03-30T15:01:13.223064+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:01:13.223064+0000","last_undegraded":"2024-03-30T15:01:13.223064+0000","last_fullsized":"2024-03-30T15:01:13.223064+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:35.074185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.074185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":620,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245221+0000","last_change":"2024-03-30T15:00:39.833443+0000","last_active":"2024-03-30T15:01:12.245221+0000","last_peered":"2024-03-30T15:01:12.245221+0000","last_clean":"2024-03-30T15:01:12.245221+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:01:12.245221+0000","last_undegraded":"2024-03-30T15:01:12.245221+0000","last_fullsized":"2024-03-30T15:01:12.245221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:00:39.833329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:39.833329+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:47.420557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":407,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251876+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:01:12.251876+0000","last_peered":"2024-03-30T15:01:12.251876+0000","last_clean":"2024-03-30T15:01:12.251876+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:01:12.251876+0000","last_undegraded":"2024-03-30T15:01:12.251876+0000","last_fullsized":"2024-03-30T15:01:12.251876+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":767,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222211+0000","last_change":"2024-03-30T14:59:14.976524+0000","last_active":"2024-03-30T15:01:13.222211+0000","last_peered":"2024-03-30T15:01:13.222211+0000","last_clean":"2024-03-30T15:01:13.222211+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:01:13.222211+0000","last_undegraded":"2024-03-30T15:01:13.222211+0000","last_fullsized":"2024-03-30T15:01:13.222211+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T14:59:14.976438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.976438+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":493,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.254226+0000","last_change":"2024-03-30T14:59:05.166302+0000","last_active":"2024-03-30T15:01:12.254226+0000","last_peered":"2024-03-30T15:01:12.254226+0000","last_clean":"2024-03-30T15:01:12.254226+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:01:12.254226+0000","last_undegraded":"2024-03-30T15:01:12.254226+0000","last_fullsized":"2024-03-30T15:01:12.254226+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.166200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.166200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":426,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222160+0000","last_change":"2024-03-30T15:01:10.022295+0000","last_active":"2024-03-30T15:01:13.222160+0000","last_peered":"2024-03-30T15:01:13.222160+0000","last_clean":"2024-03-30T15:01:13.222160+0000","last_became_active":"2024-03-30T15:00:45.240866+0000","last_became_peered":"2024-03-30T15:00:45.240866+0000","last_unstale":"2024-03-30T15:01:13.222160+0000","last_undegraded":"2024-03-30T15:01:13.222160+0000","last_fullsized":"2024-03-30T15:01:13.222160+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.022217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.022217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:16.314533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244273+0000","last_change":"2024-03-30T14:59:53.656204+0000","last_active":"2024-03-30T15:01:12.244273+0000","last_peered":"2024-03-30T15:01:12.244273+0000","last_clean":"2024-03-30T15:01:12.244273+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:01:12.244273+0000","last_undegraded":"2024-03-30T15:01:12.244273+0000","last_fullsized":"2024-03-30T15:01:12.244273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.656111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.656111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220840+0000","last_change":"2024-03-30T14:59:03.841069+0000","last_active":"2024-03-30T15:01:13.220840+0000","last_peered":"2024-03-30T15:01:13.220840+0000","last_clean":"2024-03-30T15:01:13.220840+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:01:13.220840+0000","last_undegraded":"2024-03-30T15:01:13.220840+0000","last_fullsized":"2024-03-30T15:01:13.220840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.840945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.840945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":551,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.219899+0000","last_change":"2024-03-30T15:01:11.446061+0000","last_active":"2024-03-30T15:01:12.219899+0000","last_peered":"2024-03-30T15:01:12.219899+0000","last_clean":"2024-03-30T15:01:12.219899+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:01:12.219899+0000","last_undegraded":"2024-03-30T15:01:12.219899+0000","last_fullsized":"2024-03-30T15:01:12.219899+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:01:11.445771+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.445771+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:11.603274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":524,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220463+0000","last_change":"2024-03-30T14:58:58.777774+0000","last_active":"2024-03-30T15:01:13.220463+0000","last_peered":"2024-03-30T15:01:13.220463+0000","last_clean":"2024-03-30T15:01:13.220463+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:01:13.220463+0000","last_undegraded":"2024-03-30T15:01:13.220463+0000","last_fullsized":"2024-03-30T15:01:13.220463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.777653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.777653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":515,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221552+0000","last_change":"2024-03-30T14:59:22.144499+0000","last_active":"2024-03-30T15:01:12.221552+0000","last_peered":"2024-03-30T15:01:12.221552+0000","last_clean":"2024-03-30T15:01:12.221552+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:01:12.221552+0000","last_undegraded":"2024-03-30T15:01:12.221552+0000","last_fullsized":"2024-03-30T15:01:12.221552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:22.144389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:04.301590+0000","last_clean_scrub_stamp":"2024-03-30T14:59:22.144389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":504,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223220+0000","last_change":"2024-03-30T14:59:55.270674+0000","last_active":"2024-03-30T15:01:13.223220+0000","last_peered":"2024-03-30T15:01:13.223220+0000","last_clean":"2024-03-30T15:01:13.223220+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:01:13.223220+0000","last_undegraded":"2024-03-30T15:01:13.223220+0000","last_fullsized":"2024-03-30T15:01:13.223220+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.270595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.270595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":79,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223664+0000","last_change":"2024-03-30T15:00:45.248453+0000","last_active":"2024-03-30T15:01:13.223664+0000","last_peered":"2024-03-30T15:01:13.223664+0000","last_clean":"2024-03-30T15:01:13.223664+0000","last_became_active":"2024-03-30T15:00:45.248115+0000","last_became_peered":"2024-03-30T15:00:45.248115+0000","last_unstale":"2024-03-30T15:01:13.223664+0000","last_undegraded":"2024-03-30T15:01:13.223664+0000","last_fullsized":"2024-03-30T15:01:13.223664+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:04.031690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:04.031690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:19.855770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":406,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591276+0000","last_change":"2024-03-30T14:57:54.024875+0000","last_active":"2024-03-30T15:01:13.591276+0000","last_peered":"2024-03-30T15:01:13.591276+0000","last_clean":"2024-03-30T15:01:13.591276+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:01:13.591276+0000","last_undegraded":"2024-03-30T15:01:13.591276+0000","last_fullsized":"2024-03-30T15:01:13.591276+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":509,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.588857+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:01:13.588857+0000","last_peered":"2024-03-30T15:01:13.588857+0000","last_clean":"2024-03-30T15:01:13.588857+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:01:13.588857+0000","last_undegraded":"2024-03-30T15:01:13.588857+0000","last_fullsized":"2024-03-30T15:01:13.588857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.197637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":583,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.252494+0000","last_change":"2024-03-30T14:58:22.340066+0000","last_active":"2024-03-30T15:01:12.252494+0000","last_peered":"2024-03-30T15:01:12.252494+0000","last_clean":"2024-03-30T15:01:12.252494+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:01:12.252494+0000","last_undegraded":"2024-03-30T15:01:12.252494+0000","last_fullsized":"2024-03-30T15:01:12.252494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T14:58:22.339962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.339962+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":508,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224648+0000","last_change":"2024-03-30T14:58:19.237610+0000","last_active":"2024-03-30T15:01:13.224648+0000","last_peered":"2024-03-30T15:01:13.224648+0000","last_clean":"2024-03-30T15:01:13.224648+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:01:13.224648+0000","last_undegraded":"2024-03-30T15:01:13.224648+0000","last_fullsized":"2024-03-30T15:01:13.224648+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.237530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.237530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":516,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251911+0000","last_change":"2024-03-30T14:58:36.096934+0000","last_active":"2024-03-30T15:01:12.251911+0000","last_peered":"2024-03-30T15:01:12.251911+0000","last_clean":"2024-03-30T15:01:12.251911+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:01:12.251911+0000","last_undegraded":"2024-03-30T15:01:12.251911+0000","last_fullsized":"2024-03-30T15:01:12.251911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T14:58:36.096812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.096812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226428+0000","last_change":"2024-03-30T14:58:22.249707+0000","last_active":"2024-03-30T15:01:13.226428+0000","last_peered":"2024-03-30T15:01:13.226428+0000","last_clean":"2024-03-30T15:01:13.226428+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:01:13.226428+0000","last_undegraded":"2024-03-30T15:01:13.226428+0000","last_fullsized":"2024-03-30T15:01:13.226428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:22.249628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:22.249628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.11","version":"301'9","reported_seq":87,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589448+0000","last_change":"2024-03-30T15:00:45.250504+0000","last_active":"2024-03-30T15:01:13.589448+0000","last_peered":"2024-03-30T15:01:13.589448+0000","last_clean":"2024-03-30T15:01:13.589448+0000","last_became_active":"2024-03-30T15:00:45.239363+0000","last_became_peered":"2024-03-30T15:00:45.239363+0000","last_unstale":"2024-03-30T15:01:13.589448+0000","last_undegraded":"2024-03-30T15:01:13.589448+0000","last_fullsized":"2024-03-30T15:01:13.589448+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":406,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590308+0000","last_change":"2024-03-30T14:57:54.046311+0000","last_active":"2024-03-30T15:01:13.590308+0000","last_peered":"2024-03-30T15:01:13.590308+0000","last_clean":"2024-03-30T15:01:13.590308+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:01:13.590308+0000","last_undegraded":"2024-03-30T15:01:13.590308+0000","last_fullsized":"2024-03-30T15:01:13.590308+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":528,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590310+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:01:13.590310+0000","last_peered":"2024-03-30T15:01:13.590310+0000","last_clean":"2024-03-30T15:01:13.590310+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:01:13.590310+0000","last_undegraded":"2024-03-30T15:01:13.590310+0000","last_fullsized":"2024-03-30T15:01:13.590310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:39.060252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":512,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253180+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:01:12.253180+0000","last_peered":"2024-03-30T15:01:12.253180+0000","last_clean":"2024-03-30T15:01:12.253180+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:01:12.253180+0000","last_undegraded":"2024-03-30T15:01:12.253180+0000","last_fullsized":"2024-03-30T15:01:12.253180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":515,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219609+0000","last_change":"2024-03-30T15:01:12.479941+0000","last_active":"2024-03-30T15:01:13.219609+0000","last_peered":"2024-03-30T15:01:13.219609+0000","last_clean":"2024-03-30T15:01:13.219609+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:01:13.219609+0000","last_undegraded":"2024-03-30T15:01:13.219609+0000","last_fullsized":"2024-03-30T15:01:13.219609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.479863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.479863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:22.895518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":497,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590249+0000","last_change":"2024-03-30T14:58:36.234502+0000","last_active":"2024-03-30T15:01:13.590249+0000","last_peered":"2024-03-30T15:01:13.590249+0000","last_clean":"2024-03-30T15:01:13.590249+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:01:13.590249+0000","last_undegraded":"2024-03-30T15:01:13.590249+0000","last_fullsized":"2024-03-30T15:01:13.590249+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:36.234425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.234425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":404,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225023+0000","last_change":"2024-03-30T14:57:55.074016+0000","last_active":"2024-03-30T15:01:13.225023+0000","last_peered":"2024-03-30T15:01:13.225023+0000","last_clean":"2024-03-30T15:01:13.225023+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:01:13.225023+0000","last_undegraded":"2024-03-30T15:01:13.225023+0000","last_fullsized":"2024-03-30T15:01:13.225023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":567,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243490+0000","last_change":"2024-03-30T14:59:00.517828+0000","last_active":"2024-03-30T15:01:12.243490+0000","last_peered":"2024-03-30T15:01:12.243490+0000","last_clean":"2024-03-30T15:01:12.243490+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:01:12.243490+0000","last_undegraded":"2024-03-30T15:01:12.243490+0000","last_fullsized":"2024-03-30T15:01:12.243490+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T14:59:00.517731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:00.517731+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223136+0000","last_change":"2024-03-30T14:58:23.951589+0000","last_active":"2024-03-30T15:01:13.223136+0000","last_peered":"2024-03-30T15:01:13.223136+0000","last_clean":"2024-03-30T15:01:13.223136+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:01:13.223136+0000","last_undegraded":"2024-03-30T15:01:13.223136+0000","last_fullsized":"2024-03-30T15:01:13.223136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T14:58:23.951507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":506,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220335+0000","last_change":"2024-03-30T14:58:33.889037+0000","last_active":"2024-03-30T15:01:13.220335+0000","last_peered":"2024-03-30T15:01:13.220335+0000","last_clean":"2024-03-30T15:01:13.220335+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:01:13.220335+0000","last_undegraded":"2024-03-30T15:01:13.220335+0000","last_fullsized":"2024-03-30T15:01:13.220335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:33.888931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:33.888931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":496,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222109+0000","last_change":"2024-03-30T14:59:12.985700+0000","last_active":"2024-03-30T15:01:13.222109+0000","last_peered":"2024-03-30T15:01:13.222109+0000","last_clean":"2024-03-30T15:01:13.222109+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:01:13.222109+0000","last_undegraded":"2024-03-30T15:01:13.222109+0000","last_fullsized":"2024-03-30T15:01:13.222109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.985620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.985620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":424,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.242999+0000","last_change":"2024-03-30T14:59:51.656251+0000","last_active":"2024-03-30T15:01:12.242999+0000","last_peered":"2024-03-30T15:01:12.242999+0000","last_clean":"2024-03-30T15:01:12.242999+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:01:12.242999+0000","last_undegraded":"2024-03-30T15:01:12.242999+0000","last_fullsized":"2024-03-30T15:01:12.242999+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T14:59:51.656163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.656163+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.218870+0000","last_change":"2024-03-30T14:59:08.794368+0000","last_active":"2024-03-30T15:01:13.218870+0000","last_peered":"2024-03-30T15:01:13.218870+0000","last_clean":"2024-03-30T15:01:13.218870+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:01:13.218870+0000","last_undegraded":"2024-03-30T15:01:13.218870+0000","last_fullsized":"2024-03-30T15:01:13.218870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.794288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.794288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":535,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224381+0000","last_change":"2024-03-30T14:58:25.223144+0000","last_active":"2024-03-30T15:01:13.224381+0000","last_peered":"2024-03-30T15:01:13.224381+0000","last_clean":"2024-03-30T15:01:13.224381+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:01:13.224381+0000","last_undegraded":"2024-03-30T15:01:13.224381+0000","last_fullsized":"2024-03-30T15:01:13.224381+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:25.223044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.223044+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224246+0000","last_change":"2024-03-30T14:58:25.984629+0000","last_active":"2024-03-30T15:01:13.224246+0000","last_peered":"2024-03-30T15:01:13.224246+0000","last_clean":"2024-03-30T15:01:13.224246+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:01:13.224246+0000","last_undegraded":"2024-03-30T15:01:13.224246+0000","last_fullsized":"2024-03-30T15:01:13.224246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.984552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.984552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222544+0000","last_change":"2024-03-30T15:00:02.270908+0000","last_active":"2024-03-30T15:01:13.222544+0000","last_peered":"2024-03-30T15:01:13.222544+0000","last_clean":"2024-03-30T15:01:13.222544+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:01:13.222544+0000","last_undegraded":"2024-03-30T15:01:13.222544+0000","last_fullsized":"2024-03-30T15:01:13.222544+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.270821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":412,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223615+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:01:13.223615+0000","last_peered":"2024-03-30T15:01:13.223615+0000","last_clean":"2024-03-30T15:01:13.223615+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:01:13.223615+0000","last_undegraded":"2024-03-30T15:01:13.223615+0000","last_fullsized":"2024-03-30T15:01:13.223615+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":588,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221612+0000","last_change":"2024-03-30T14:59:05.977968+0000","last_active":"2024-03-30T15:01:13.221612+0000","last_peered":"2024-03-30T15:01:13.221612+0000","last_clean":"2024-03-30T15:01:13.221612+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:01:13.221612+0000","last_undegraded":"2024-03-30T15:01:13.221612+0000","last_fullsized":"2024-03-30T15:01:13.221612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:05.977866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.977866+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":526,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251462+0000","last_change":"2024-03-30T14:58:35.054537+0000","last_active":"2024-03-30T15:01:12.251462+0000","last_peered":"2024-03-30T15:01:12.251462+0000","last_clean":"2024-03-30T15:01:12.251462+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:01:12.251462+0000","last_undegraded":"2024-03-30T15:01:12.251462+0000","last_fullsized":"2024-03-30T15:01:12.251462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:58:35.054439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.054439+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":519,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224982+0000","last_change":"2024-03-30T14:58:24.985818+0000","last_active":"2024-03-30T15:01:13.224982+0000","last_peered":"2024-03-30T15:01:13.224982+0000","last_clean":"2024-03-30T15:01:13.224982+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:01:13.224982+0000","last_undegraded":"2024-03-30T15:01:13.224982+0000","last_fullsized":"2024-03-30T15:01:13.224982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T14:58:24.985660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.985660+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":492,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.252251+0000","last_change":"2024-03-30T14:58:27.124070+0000","last_active":"2024-03-30T15:01:12.252251+0000","last_peered":"2024-03-30T15:01:12.252251+0000","last_clean":"2024-03-30T15:01:12.252251+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:01:12.252251+0000","last_undegraded":"2024-03-30T15:01:12.252251+0000","last_fullsized":"2024-03-30T15:01:12.252251+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.123941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.123941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244143+0000","last_change":"2024-03-30T14:59:56.678123+0000","last_active":"2024-03-30T15:01:12.244143+0000","last_peered":"2024-03-30T15:01:12.244143+0000","last_clean":"2024-03-30T15:01:12.244143+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:01:12.244143+0000","last_undegraded":"2024-03-30T15:01:12.244143+0000","last_fullsized":"2024-03-30T15:01:12.244143+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.678045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.678045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:12.776149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":428,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225104+0000","last_change":"2024-03-30T14:57:56.275636+0000","last_active":"2024-03-30T15:01:13.225104+0000","last_peered":"2024-03-30T15:01:13.225104+0000","last_clean":"2024-03-30T15:01:13.225104+0000","last_became_active":"2024-03-30T14:57:56.273316+0000","last_became_peered":"2024-03-30T14:57:56.273316+0000","last_unstale":"2024-03-30T15:01:13.225104+0000","last_undegraded":"2024-03-30T15:01:13.225104+0000","last_fullsized":"2024-03-30T15:01:13.225104+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T14:57:30.919064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:57:30.919064+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":521,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243048+0000","last_change":"2024-03-30T15:01:11.051090+0000","last_active":"2024-03-30T15:01:12.243048+0000","last_peered":"2024-03-30T15:01:12.243048+0000","last_clean":"2024-03-30T15:01:12.243048+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:01:12.243048+0000","last_undegraded":"2024-03-30T15:01:12.243048+0000","last_fullsized":"2024-03-30T15:01:12.243048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:11.051007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.051007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:12.715589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":496,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253815+0000","last_change":"2024-03-30T14:58:26.076218+0000","last_active":"2024-03-30T15:01:12.253815+0000","last_peered":"2024-03-30T15:01:12.253815+0000","last_clean":"2024-03-30T15:01:12.253815+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:01:12.253815+0000","last_undegraded":"2024-03-30T15:01:12.253815+0000","last_fullsized":"2024-03-30T15:01:12.253815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:26.076141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:26.076141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222920+0000","last_change":"2024-03-30T14:59:51.262210+0000","last_active":"2024-03-30T15:01:13.222920+0000","last_peered":"2024-03-30T15:01:13.222920+0000","last_clean":"2024-03-30T15:01:13.222920+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:01:13.222920+0000","last_undegraded":"2024-03-30T15:01:13.222920+0000","last_fullsized":"2024-03-30T15:01:13.222920+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:51.262127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.262127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.554991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":425,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225170+0000","last_change":"2024-03-30T15:00:37.677858+0000","last_active":"2024-03-30T15:01:13.225170+0000","last_peered":"2024-03-30T15:01:13.225170+0000","last_clean":"2024-03-30T15:01:13.225170+0000","last_became_active":"2024-03-30T14:57:55.139072+0000","last_became_peered":"2024-03-30T14:57:55.139072+0000","last_unstale":"2024-03-30T15:01:13.225170+0000","last_undegraded":"2024-03-30T15:01:13.225170+0000","last_fullsized":"2024-03-30T15:01:13.225170+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.677768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.677768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:06.999513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":412,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226301+0000","last_change":"2024-03-30T15:01:11.580352+0000","last_active":"2024-03-30T15:01:13.226301+0000","last_peered":"2024-03-30T15:01:13.226301+0000","last_clean":"2024-03-30T15:01:13.226301+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:01:13.226301+0000","last_undegraded":"2024-03-30T15:01:13.226301+0000","last_fullsized":"2024-03-30T15:01:13.226301+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:11.580276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.580276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:31.897339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":520,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591262+0000","last_change":"2024-03-30T14:59:15.487022+0000","last_active":"2024-03-30T15:01:13.591262+0000","last_peered":"2024-03-30T15:01:13.591262+0000","last_clean":"2024-03-30T15:01:13.591262+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:01:13.591262+0000","last_undegraded":"2024-03-30T15:01:13.591262+0000","last_fullsized":"2024-03-30T15:01:13.591262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:15.486892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.486892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":515,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221369+0000","last_change":"2024-03-30T15:00:37.049789+0000","last_active":"2024-03-30T15:01:13.221369+0000","last_peered":"2024-03-30T15:01:13.221369+0000","last_clean":"2024-03-30T15:01:13.221369+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:01:13.221369+0000","last_undegraded":"2024-03-30T15:01:13.221369+0000","last_fullsized":"2024-03-30T15:01:13.221369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.049685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.049685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:48.710602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":407,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220527+0000","last_change":"2024-03-30T14:58:24.291134+0000","last_active":"2024-03-30T15:01:12.220527+0000","last_peered":"2024-03-30T15:01:12.220527+0000","last_clean":"2024-03-30T15:01:12.220527+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:01:12.220527+0000","last_undegraded":"2024-03-30T15:01:12.220527+0000","last_fullsized":"2024-03-30T15:01:12.220527+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:24.291008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:24.291008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":502,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219611+0000","last_change":"2024-03-30T14:59:02.809926+0000","last_active":"2024-03-30T15:01:13.219611+0000","last_peered":"2024-03-30T15:01:13.219611+0000","last_clean":"2024-03-30T15:01:13.219611+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:01:13.219611+0000","last_undegraded":"2024-03-30T15:01:13.219611+0000","last_fullsized":"2024-03-30T15:01:13.219611+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.809814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.809814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3","version":"301'7","reported_seq":432,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224521+0000","last_change":"2024-03-30T15:00:49.710616+0000","last_active":"2024-03-30T15:01:13.224521+0000","last_peered":"2024-03-30T15:01:13.224521+0000","last_clean":"2024-03-30T15:01:13.224521+0000","last_became_active":"2024-03-30T14:57:55.089349+0000","last_became_peered":"2024-03-30T14:57:55.089349+0000","last_unstale":"2024-03-30T15:01:13.224521+0000","last_undegraded":"2024-03-30T15:01:13.224521+0000","last_fullsized":"2024-03-30T15:01:13.224521+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:49.710514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:49.710514+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.605558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":414,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591018+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:01:13.591018+0000","last_peered":"2024-03-30T15:01:13.591018+0000","last_clean":"2024-03-30T15:01:13.591018+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:01:13.591018+0000","last_undegraded":"2024-03-30T15:01:13.591018+0000","last_fullsized":"2024-03-30T15:01:13.591018+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.612522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":516,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253716+0000","last_change":"2024-03-30T14:58:31.097064+0000","last_active":"2024-03-30T15:01:12.253716+0000","last_peered":"2024-03-30T15:01:12.253716+0000","last_clean":"2024-03-30T15:01:12.253716+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:01:12.253716+0000","last_undegraded":"2024-03-30T15:01:12.253716+0000","last_fullsized":"2024-03-30T15:01:12.253716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:58:31.096968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.096968+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":516,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589877+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:01:13.589877+0000","last_peered":"2024-03-30T15:01:13.589877+0000","last_clean":"2024-03-30T15:01:13.589877+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:01:13.589877+0000","last_undegraded":"2024-03-30T15:01:13.589877+0000","last_fullsized":"2024-03-30T15:01:13.589877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":473,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222974+0000","last_change":"2024-03-30T14:59:53.269594+0000","last_active":"2024-03-30T15:01:13.222974+0000","last_peered":"2024-03-30T15:01:13.222974+0000","last_clean":"2024-03-30T15:01:13.222974+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:01:13.222974+0000","last_undegraded":"2024-03-30T15:01:13.222974+0000","last_fullsized":"2024-03-30T15:01:13.222974+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:53.269452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.269452+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226839+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:01:13.226839+0000","last_peered":"2024-03-30T15:01:13.226839+0000","last_clean":"2024-03-30T15:01:13.226839+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:01:13.226839+0000","last_undegraded":"2024-03-30T15:01:13.226839+0000","last_fullsized":"2024-03-30T15:01:13.226839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"301'7","reported_seq":432,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221865+0000","last_change":"2024-03-30T15:01:09.041433+0000","last_active":"2024-03-30T15:01:13.221865+0000","last_peered":"2024-03-30T15:01:13.221865+0000","last_clean":"2024-03-30T15:01:13.221865+0000","last_became_active":"2024-03-30T14:57:55.164327+0000","last_became_peered":"2024-03-30T14:57:55.164327+0000","last_unstale":"2024-03-30T15:01:13.221865+0000","last_undegraded":"2024-03-30T15:01:13.221865+0000","last_fullsized":"2024-03-30T15:01:13.221865+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'7","last_scrub_stamp":"2024-03-30T15:01:09.041344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:09.041344+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:16.138176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":406,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590936+0000","last_change":"2024-03-30T14:57:54.020464+0000","last_active":"2024-03-30T15:01:13.590936+0000","last_peered":"2024-03-30T15:01:13.590936+0000","last_clean":"2024-03-30T15:01:13.590936+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:01:13.590936+0000","last_undegraded":"2024-03-30T15:01:13.590936+0000","last_fullsized":"2024-03-30T15:01:13.590936+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":441,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221266+0000","last_change":"2024-03-30T14:58:13.210584+0000","last_active":"2024-03-30T15:01:12.221266+0000","last_peered":"2024-03-30T15:01:12.221266+0000","last_clean":"2024-03-30T15:01:12.221266+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:01:12.221266+0000","last_undegraded":"2024-03-30T15:01:12.221266+0000","last_fullsized":"2024-03-30T15:01:12.221266+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T14:58:13.210426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:13.210426+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221377+0000","last_change":"2024-03-30T14:59:09.029236+0000","last_active":"2024-03-30T15:01:13.221377+0000","last_peered":"2024-03-30T15:01:13.221377+0000","last_clean":"2024-03-30T15:01:13.221377+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:01:13.221377+0000","last_undegraded":"2024-03-30T15:01:13.221377+0000","last_fullsized":"2024-03-30T15:01:13.221377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.029159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.029159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":587,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253616+0000","last_change":"2024-03-30T14:59:08.160662+0000","last_active":"2024-03-30T15:01:12.253616+0000","last_peered":"2024-03-30T15:01:12.253616+0000","last_clean":"2024-03-30T15:01:12.253616+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:01:12.253616+0000","last_undegraded":"2024-03-30T15:01:12.253616+0000","last_fullsized":"2024-03-30T15:01:12.253616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:08.160528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.160528+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":498,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590979+0000","last_change":"2024-03-30T14:59:16.481218+0000","last_active":"2024-03-30T15:01:13.590979+0000","last_peered":"2024-03-30T15:01:13.590979+0000","last_clean":"2024-03-30T15:01:13.590979+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:01:13.590979+0000","last_undegraded":"2024-03-30T15:01:13.590979+0000","last_fullsized":"2024-03-30T15:01:13.590979+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:16.481128+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.481128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"301'7","reported_seq":435,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589712+0000","last_change":"2024-03-30T15:01:07.535937+0000","last_active":"2024-03-30T15:01:13.589712+0000","last_peered":"2024-03-30T15:01:13.589712+0000","last_clean":"2024-03-30T15:01:13.589712+0000","last_became_active":"2024-03-30T14:57:54.041685+0000","last_became_peered":"2024-03-30T14:57:54.041685+0000","last_unstale":"2024-03-30T15:01:13.589712+0000","last_undegraded":"2024-03-30T15:01:13.589712+0000","last_fullsized":"2024-03-30T15:01:13.589712+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'7","last_scrub_stamp":"2024-03-30T15:01:07.535830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:07.535830+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:18.946261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":412,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224806+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:01:13.224806+0000","last_peered":"2024-03-30T15:01:13.224806+0000","last_clean":"2024-03-30T15:01:13.224806+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:01:13.224806+0000","last_undegraded":"2024-03-30T15:01:13.224806+0000","last_fullsized":"2024-03-30T15:01:13.224806+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.403334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":527,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224841+0000","last_change":"2024-03-30T14:59:20.453682+0000","last_active":"2024-03-30T15:01:13.224841+0000","last_peered":"2024-03-30T15:01:13.224841+0000","last_clean":"2024-03-30T15:01:13.224841+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:01:13.224841+0000","last_undegraded":"2024-03-30T15:01:13.224841+0000","last_fullsized":"2024-03-30T15:01:13.224841+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.453594+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.225084+0000","last_change":"2024-03-30T14:57:31.554095+0000","last_active":"2024-03-30T15:01:13.225084+0000","last_peered":"2024-03-30T15:01:13.225084+0000","last_clean":"2024-03-30T15:01:13.225084+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:01:13.225084+0000","last_undegraded":"2024-03-30T15:01:13.225084+0000","last_fullsized":"2024-03-30T15:01:13.225084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:31.554003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:57:31.554003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":506,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220873+0000","last_change":"2024-03-30T14:59:15.144954+0000","last_active":"2024-03-30T15:01:12.220873+0000","last_peered":"2024-03-30T15:01:12.220873+0000","last_clean":"2024-03-30T15:01:12.220873+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:01:12.220873+0000","last_undegraded":"2024-03-30T15:01:12.220873+0000","last_fullsized":"2024-03-30T15:01:12.220873+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.144864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":409,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589945+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:01:13.589945+0000","last_peered":"2024-03-30T15:01:13.589945+0000","last_clean":"2024-03-30T15:01:13.589945+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:01:13.589945+0000","last_undegraded":"2024-03-30T15:01:13.589945+0000","last_fullsized":"2024-03-30T15:01:13.589945+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:29.384685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"301'11","reported_seq":435,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245614+0000","last_change":"2024-03-30T14:59:47.655366+0000","last_active":"2024-03-30T15:01:12.245614+0000","last_peered":"2024-03-30T15:01:12.245614+0000","last_clean":"2024-03-30T15:01:12.245614+0000","last_became_active":"2024-03-30T14:57:55.181249+0000","last_became_peered":"2024-03-30T14:57:55.181249+0000","last_unstale":"2024-03-30T15:01:12.245614+0000","last_undegraded":"2024-03-30T15:01:12.245614+0000","last_fullsized":"2024-03-30T15:01:12.245614+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9","last_scrub_stamp":"2024-03-30T14:59:47.655290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.655290+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":412,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224246+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:01:13.224246+0000","last_peered":"2024-03-30T15:01:13.224246+0000","last_clean":"2024-03-30T15:01:13.224246+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:01:13.224246+0000","last_undegraded":"2024-03-30T15:01:13.224246+0000","last_fullsized":"2024-03-30T15:01:13.224246+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:38.153282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":518,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221254+0000","last_change":"2024-03-30T14:59:08.292999+0000","last_active":"2024-03-30T15:01:12.221254+0000","last_peered":"2024-03-30T15:01:12.221254+0000","last_clean":"2024-03-30T15:01:12.221254+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:01:12.221254+0000","last_undegraded":"2024-03-30T15:01:12.221254+0000","last_fullsized":"2024-03-30T15:01:12.221254+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:08.292876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.292876+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":522,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222948+0000","last_change":"2024-03-30T14:58:12.927851+0000","last_active":"2024-03-30T15:01:13.222948+0000","last_peered":"2024-03-30T15:01:13.222948+0000","last_clean":"2024-03-30T15:01:13.222948+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:01:13.222948+0000","last_undegraded":"2024-03-30T15:01:13.222948+0000","last_fullsized":"2024-03-30T15:01:13.222948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T14:58:12.927762+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T14:58:12.927762+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":409,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590413+0000","last_change":"2024-03-30T14:59:17.441377+0000","last_active":"2024-03-30T15:01:13.590413+0000","last_peered":"2024-03-30T15:01:13.590413+0000","last_clean":"2024-03-30T15:01:13.590413+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:01:13.590413+0000","last_undegraded":"2024-03-30T15:01:13.590413+0000","last_fullsized":"2024-03-30T15:01:13.590413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.441289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.441289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":496,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589227+0000","last_change":"2024-03-30T14:58:18.177964+0000","last_active":"2024-03-30T15:01:13.589227+0000","last_peered":"2024-03-30T15:01:13.589227+0000","last_clean":"2024-03-30T15:01:13.589227+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:01:13.589227+0000","last_undegraded":"2024-03-30T15:01:13.589227+0000","last_fullsized":"2024-03-30T15:01:13.589227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:18.177806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:18.177806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":420,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591650+0000","last_change":"2024-03-30T14:57:54.013168+0000","last_active":"2024-03-30T15:01:13.591650+0000","last_peered":"2024-03-30T15:01:13.591650+0000","last_clean":"2024-03-30T15:01:13.591650+0000","last_became_active":"2024-03-30T14:57:54.012861+0000","last_became_peered":"2024-03-30T14:57:54.012861+0000","last_unstale":"2024-03-30T15:01:13.591650+0000","last_undegraded":"2024-03-30T15:01:13.591650+0000","last_fullsized":"2024-03-30T15:01:13.591650+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":414,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589732+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:01:13.589732+0000","last_peered":"2024-03-30T15:01:13.589732+0000","last_clean":"2024-03-30T15:01:13.589732+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:01:13.589732+0000","last_undegraded":"2024-03-30T15:01:13.589732+0000","last_fullsized":"2024-03-30T15:01:13.589732+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.187292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":446,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221477+0000","last_change":"2024-03-30T14:58:16.905699+0000","last_active":"2024-03-30T15:01:13.221477+0000","last_peered":"2024-03-30T15:01:13.221477+0000","last_clean":"2024-03-30T15:01:13.221477+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:01:13.221477+0000","last_undegraded":"2024-03-30T15:01:13.221477+0000","last_fullsized":"2024-03-30T15:01:13.221477+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T14:58:16.905604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:16.905604+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":504,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253929+0000","last_change":"2024-03-30T14:58:28.115261+0000","last_active":"2024-03-30T15:01:12.253929+0000","last_peered":"2024-03-30T15:01:12.253929+0000","last_clean":"2024-03-30T15:01:12.253929+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:01:12.253929+0000","last_undegraded":"2024-03-30T15:01:12.253929+0000","last_fullsized":"2024-03-30T15:01:12.253929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:28.115166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:28.115166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":499,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222313+0000","last_change":"2024-03-30T14:59:08.012564+0000","last_active":"2024-03-30T15:01:13.222313+0000","last_peered":"2024-03-30T15:01:13.222313+0000","last_clean":"2024-03-30T15:01:13.222313+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:01:13.222313+0000","last_undegraded":"2024-03-30T15:01:13.222313+0000","last_fullsized":"2024-03-30T15:01:13.222313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:08.012470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.012470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":409,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590000+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:01:13.590000+0000","last_peered":"2024-03-30T15:01:13.590000+0000","last_clean":"2024-03-30T15:01:13.590000+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:01:13.590000+0000","last_undegraded":"2024-03-30T15:01:13.590000+0000","last_fullsized":"2024-03-30T15:01:13.590000+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.714400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":420,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591394+0000","last_change":"2024-03-30T14:57:54.019408+0000","last_active":"2024-03-30T15:01:13.591394+0000","last_peered":"2024-03-30T15:01:13.591394+0000","last_clean":"2024-03-30T15:01:13.591394+0000","last_became_active":"2024-03-30T14:57:54.014697+0000","last_became_peered":"2024-03-30T14:57:54.014697+0000","last_unstale":"2024-03-30T15:01:13.591394+0000","last_undegraded":"2024-03-30T15:01:13.591394+0000","last_fullsized":"2024-03-30T15:01:13.591394+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":412,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221623+0000","last_change":"2024-03-30T14:59:56.148641+0000","last_active":"2024-03-30T15:01:12.221623+0000","last_peered":"2024-03-30T15:01:12.221623+0000","last_clean":"2024-03-30T15:01:12.221623+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:01:12.221623+0000","last_undegraded":"2024-03-30T15:01:12.221623+0000","last_fullsized":"2024-03-30T15:01:12.221623+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.148529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.148529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.134818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":432,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.221558+0000","last_change":"2024-03-30T14:58:17.304912+0000","last_active":"2024-03-30T15:01:12.221558+0000","last_peered":"2024-03-30T15:01:12.221558+0000","last_clean":"2024-03-30T15:01:12.221558+0000","last_became_active":"2024-03-30T14:57:56.224608+0000","last_became_peered":"2024-03-30T14:57:56.224608+0000","last_unstale":"2024-03-30T15:01:12.221558+0000","last_undegraded":"2024-03-30T15:01:12.221558+0000","last_fullsized":"2024-03-30T15:01:12.221558+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T14:58:17.304759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:17.304759+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":516,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591542+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:01:13.591542+0000","last_peered":"2024-03-30T15:01:13.591542+0000","last_clean":"2024-03-30T15:01:13.591542+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:01:13.591542+0000","last_undegraded":"2024-03-30T15:01:13.591542+0000","last_fullsized":"2024-03-30T15:01:13.591542+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.891147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":505,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243717+0000","last_change":"2024-03-30T14:58:25.320605+0000","last_active":"2024-03-30T15:01:12.243717+0000","last_peered":"2024-03-30T15:01:12.243717+0000","last_clean":"2024-03-30T15:01:12.243717+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:01:12.243717+0000","last_undegraded":"2024-03-30T15:01:12.243717+0000","last_fullsized":"2024-03-30T15:01:12.243717+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.320527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.320527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221663+0000","last_change":"2024-03-30T14:58:58.957529+0000","last_active":"2024-03-30T15:01:13.221663+0000","last_peered":"2024-03-30T15:01:13.221663+0000","last_clean":"2024-03-30T15:01:13.221663+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:01:13.221663+0000","last_undegraded":"2024-03-30T15:01:13.221663+0000","last_fullsized":"2024-03-30T15:01:13.221663+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.957449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.957449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":424,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245583+0000","last_change":"2024-03-30T14:59:07.508558+0000","last_active":"2024-03-30T15:01:12.245583+0000","last_peered":"2024-03-30T15:01:12.245583+0000","last_clean":"2024-03-30T15:01:12.245583+0000","last_became_active":"2024-03-30T14:57:55.201639+0000","last_became_peered":"2024-03-30T14:57:55.201639+0000","last_unstale":"2024-03-30T15:01:12.245583+0000","last_undegraded":"2024-03-30T15:01:12.245583+0000","last_fullsized":"2024-03-30T15:01:12.245583+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:07.508480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:07.508480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245018+0000","last_change":"2024-03-30T15:00:03.553767+0000","last_active":"2024-03-30T15:01:12.245018+0000","last_peered":"2024-03-30T15:01:12.245018+0000","last_clean":"2024-03-30T15:01:12.245018+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:01:12.245018+0000","last_undegraded":"2024-03-30T15:01:12.245018+0000","last_fullsized":"2024-03-30T15:01:12.245018+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.553680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.553680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:13.545159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245103+0000","last_change":"2024-03-30T15:00:41.903846+0000","last_active":"2024-03-30T15:01:12.245103+0000","last_peered":"2024-03-30T15:01:12.245103+0000","last_clean":"2024-03-30T15:01:12.245103+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:01:12.245103+0000","last_undegraded":"2024-03-30T15:01:12.245103+0000","last_fullsized":"2024-03-30T15:01:12.245103+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:00:41.903753+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:00:41.903753+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:05.208912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":410,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589249+0000","last_change":"2024-03-30T14:59:09.624243+0000","last_active":"2024-03-30T15:01:13.589249+0000","last_peered":"2024-03-30T15:01:13.589249+0000","last_clean":"2024-03-30T15:01:13.589249+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:01:13.589249+0000","last_undegraded":"2024-03-30T15:01:13.589249+0000","last_fullsized":"2024-03-30T15:01:13.589249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.624114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.624114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":501,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.588956+0000","last_change":"2024-03-30T14:58:58.605917+0000","last_active":"2024-03-30T15:01:13.588956+0000","last_peered":"2024-03-30T15:01:13.588956+0000","last_clean":"2024-03-30T15:01:13.588956+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:01:13.588956+0000","last_undegraded":"2024-03-30T15:01:13.588956+0000","last_fullsized":"2024-03-30T15:01:13.588956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:58.605792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:58.605792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":502,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220017+0000","last_change":"2024-03-30T14:59:21.144421+0000","last_active":"2024-03-30T15:01:12.220017+0000","last_peered":"2024-03-30T15:01:12.220017+0000","last_clean":"2024-03-30T15:01:12.220017+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:01:12.220017+0000","last_undegraded":"2024-03-30T15:01:12.220017+0000","last_fullsized":"2024-03-30T15:01:12.220017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.144335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.144335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.a","version":"301'14","reported_seq":449,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226032+0000","last_change":"2024-03-30T15:01:13.226032+0000","last_active":"2024-03-30T15:01:13.226032+0000","last_peered":"2024-03-30T15:01:13.226032+0000","last_clean":"2024-03-30T15:01:13.226032+0000","last_became_active":"2024-03-30T15:01:13.225726+0000","last_became_peered":"2024-03-30T15:01:13.225726+0000","last_unstale":"2024-03-30T15:01:13.226032+0000","last_undegraded":"2024-03-30T15:01:13.226032+0000","last_fullsized":"2024-03-30T15:01:13.226032+0000","mapping_epoch":111,"log_start":"301'14","ondisk_log_start":"301'14","created":111,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'10","last_scrub_stamp":"2024-03-30T15:00:42.060906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:42.060906+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":21,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:03.136605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245804+0000","last_change":"2024-03-30T14:59:14.560489+0000","last_active":"2024-03-30T15:01:12.245804+0000","last_peered":"2024-03-30T15:01:12.245804+0000","last_clean":"2024-03-30T15:01:12.245804+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:01:12.245804+0000","last_undegraded":"2024-03-30T15:01:12.245804+0000","last_fullsized":"2024-03-30T15:01:12.245804+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.560377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.560377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":525,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591230+0000","last_change":"2024-03-30T14:59:10.626455+0000","last_active":"2024-03-30T15:01:13.591230+0000","last_peered":"2024-03-30T15:01:13.591230+0000","last_clean":"2024-03-30T15:01:13.591230+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:01:13.591230+0000","last_undegraded":"2024-03-30T15:01:13.591230+0000","last_fullsized":"2024-03-30T15:01:13.591230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.626365+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":507,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220215+0000","last_change":"2024-03-30T14:58:32.272039+0000","last_active":"2024-03-30T15:01:12.220215+0000","last_peered":"2024-03-30T15:01:12.220215+0000","last_clean":"2024-03-30T15:01:12.220215+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:01:12.220215+0000","last_undegraded":"2024-03-30T15:01:12.220215+0000","last_fullsized":"2024-03-30T15:01:12.220215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T14:58:32.271622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:32.271622+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":597,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244418+0000","last_change":"2024-03-30T14:58:59.472999+0000","last_active":"2024-03-30T15:01:12.244418+0000","last_peered":"2024-03-30T15:01:12.244418+0000","last_clean":"2024-03-30T15:01:12.244418+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:01:12.244418+0000","last_undegraded":"2024-03-30T15:01:12.244418+0000","last_fullsized":"2024-03-30T15:01:12.244418+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.472884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.472884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":505,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.222983+0000","last_change":"2024-03-30T15:00:36.069861+0000","last_active":"2024-03-30T15:01:13.222983+0000","last_peered":"2024-03-30T15:01:13.222983+0000","last_clean":"2024-03-30T15:01:13.222983+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:01:13.222983+0000","last_undegraded":"2024-03-30T15:01:13.222983+0000","last_fullsized":"2024-03-30T15:01:13.222983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:00:36.069752+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:00:36.069752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:40.687480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.b","version":"301'14","reported_seq":94,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223852+0000","last_change":"2024-03-30T15:01:09.244795+0000","last_active":"2024-03-30T15:01:13.223852+0000","last_peered":"2024-03-30T15:01:13.223852+0000","last_clean":"2024-03-30T15:01:13.223852+0000","last_became_active":"2024-03-30T15:01:09.244422+0000","last_became_peered":"2024-03-30T15:01:09.244422+0000","last_unstale":"2024-03-30T15:01:13.223852+0000","last_undegraded":"2024-03-30T15:01:13.223852+0000","last_fullsized":"2024-03-30T15:01:13.223852+0000","mapping_epoch":280,"log_start":"301'14","ondisk_log_start":"301'14","created":111,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:57:48.654936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":411,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244309+0000","last_change":"2024-03-30T14:59:21.597544+0000","last_active":"2024-03-30T15:01:12.244309+0000","last_peered":"2024-03-30T15:01:12.244309+0000","last_clean":"2024-03-30T15:01:12.244309+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:01:12.244309+0000","last_undegraded":"2024-03-30T15:01:12.244309+0000","last_fullsized":"2024-03-30T15:01:12.244309+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:21.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:21.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":449,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.219502+0000","last_change":"2024-03-30T14:58:15.287344+0000","last_active":"2024-03-30T15:01:12.219502+0000","last_peered":"2024-03-30T15:01:12.219502+0000","last_clean":"2024-03-30T15:01:12.219502+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:01:12.219502+0000","last_undegraded":"2024-03-30T15:01:12.219502+0000","last_fullsized":"2024-03-30T15:01:12.219502+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:15.286572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.286572+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":507,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224767+0000","last_change":"2024-03-30T14:58:31.196084+0000","last_active":"2024-03-30T15:01:13.224767+0000","last_peered":"2024-03-30T15:01:13.224767+0000","last_clean":"2024-03-30T15:01:13.224767+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:01:13.224767+0000","last_undegraded":"2024-03-30T15:01:13.224767+0000","last_fullsized":"2024-03-30T15:01:13.224767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:31.196010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.196010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":499,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226306+0000","last_change":"2024-03-30T14:59:01.294973+0000","last_active":"2024-03-30T15:01:13.226306+0000","last_peered":"2024-03-30T15:01:13.226306+0000","last_clean":"2024-03-30T15:01:13.226306+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:01:13.226306+0000","last_undegraded":"2024-03-30T15:01:13.226306+0000","last_fullsized":"2024-03-30T15:01:13.226306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.294888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.294888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":415,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245160+0000","last_change":"2024-03-30T15:01:08.086168+0000","last_active":"2024-03-30T15:01:12.245160+0000","last_peered":"2024-03-30T15:01:12.245160+0000","last_clean":"2024-03-30T15:01:12.245160+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:01:12.245160+0000","last_undegraded":"2024-03-30T15:01:12.245160+0000","last_fullsized":"2024-03-30T15:01:12.245160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:08.086087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:08.086087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:15.793731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.8","version":"115'2","reported_seq":427,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221839+0000","last_change":"2024-03-30T15:00:49.067522+0000","last_active":"2024-03-30T15:01:13.221839+0000","last_peered":"2024-03-30T15:01:13.221839+0000","last_clean":"2024-03-30T15:01:13.221839+0000","last_became_active":"2024-03-30T14:57:55.108185+0000","last_became_peered":"2024-03-30T14:57:55.108185+0000","last_unstale":"2024-03-30T15:01:13.221839+0000","last_undegraded":"2024-03-30T15:01:13.221839+0000","last_fullsized":"2024-03-30T15:01:13.221839+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'2","last_scrub_stamp":"2024-03-30T15:00:49.067432+0000","last_deep_scrub":"115'2","last_deep_scrub_stamp":"2024-03-30T15:00:49.067432+0000","last_clean_scrub_stamp":"2024-03-30T15:00:49.067432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:00.819130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":410,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253316+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:01:12.253316+0000","last_peered":"2024-03-30T15:01:12.253316+0000","last_clean":"2024-03-30T15:01:12.253316+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:01:12.253316+0000","last_undegraded":"2024-03-30T15:01:12.253316+0000","last_fullsized":"2024-03-30T15:01:12.253316+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.587803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":542,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.219287+0000","last_change":"2024-03-30T14:58:37.879183+0000","last_active":"2024-03-30T15:01:13.219287+0000","last_peered":"2024-03-30T15:01:13.219287+0000","last_clean":"2024-03-30T15:01:13.219287+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:01:13.219287+0000","last_undegraded":"2024-03-30T15:01:13.219287+0000","last_fullsized":"2024-03-30T15:01:13.219287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T14:58:37.879074+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":516,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.590756+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:01:13.590756+0000","last_peered":"2024-03-30T15:01:13.590756+0000","last_clean":"2024-03-30T15:01:13.590756+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:01:13.590756+0000","last_undegraded":"2024-03-30T15:01:13.590756+0000","last_fullsized":"2024-03-30T15:01:13.590756+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:21.606179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":498,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.219833+0000","last_change":"2024-03-30T14:58:19.314405+0000","last_active":"2024-03-30T15:01:12.219833+0000","last_peered":"2024-03-30T15:01:12.219833+0000","last_clean":"2024-03-30T15:01:12.219833+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:01:12.219833+0000","last_undegraded":"2024-03-30T15:01:12.219833+0000","last_fullsized":"2024-03-30T15:01:12.219833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.314262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.314262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":496,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.226534+0000","last_change":"2024-03-30T14:59:03.291569+0000","last_active":"2024-03-30T15:01:13.226534+0000","last_peered":"2024-03-30T15:01:13.226534+0000","last_clean":"2024-03-30T15:01:13.226534+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:01:13.226534+0000","last_undegraded":"2024-03-30T15:01:13.226534+0000","last_fullsized":"2024-03-30T15:01:13.226534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.291469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.291469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":422,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589641+0000","last_change":"2024-03-30T15:01:08.487242+0000","last_active":"2024-03-30T15:01:13.589641+0000","last_peered":"2024-03-30T15:01:13.589641+0000","last_clean":"2024-03-30T15:01:13.589641+0000","last_became_active":"2024-03-30T14:57:56.257360+0000","last_became_peered":"2024-03-30T14:57:56.257360+0000","last_unstale":"2024-03-30T15:01:13.589641+0000","last_undegraded":"2024-03-30T15:01:13.589641+0000","last_fullsized":"2024-03-30T15:01:13.589641+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:08.487129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:01:08.487129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:02:36.310486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":412,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.218059+0000","last_change":"2024-03-30T14:59:50.075598+0000","last_active":"2024-03-30T15:01:12.218059+0000","last_peered":"2024-03-30T15:01:12.218059+0000","last_clean":"2024-03-30T15:01:12.218059+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:01:12.218059+0000","last_undegraded":"2024-03-30T15:01:12.218059+0000","last_fullsized":"2024-03-30T15:01:12.218059+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.075512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.075512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":525,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224734+0000","last_change":"2024-03-30T14:59:46.413217+0000","last_active":"2024-03-30T15:01:13.224734+0000","last_peered":"2024-03-30T15:01:13.224734+0000","last_clean":"2024-03-30T15:01:13.224734+0000","last_became_active":"2024-03-30T14:54:58.400636+0000","last_became_peered":"2024-03-30T14:54:58.400636+0000","last_unstale":"2024-03-30T15:01:13.224734+0000","last_undegraded":"2024-03-30T15:01:13.224734+0000","last_fullsized":"2024-03-30T15:01:13.224734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T14:59:46.413127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.413127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:15.902904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":504,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253177+0000","last_change":"2024-03-30T14:58:15.115774+0000","last_active":"2024-03-30T15:01:12.253177+0000","last_peered":"2024-03-30T15:01:12.253177+0000","last_clean":"2024-03-30T15:01:12.253177+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:01:12.253177+0000","last_undegraded":"2024-03-30T15:01:12.253177+0000","last_fullsized":"2024-03-30T15:01:12.253177+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:15.115694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:15.115694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":499,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221656+0000","last_change":"2024-03-30T14:58:29.060905+0000","last_active":"2024-03-30T15:01:13.221656+0000","last_peered":"2024-03-30T15:01:13.221656+0000","last_clean":"2024-03-30T15:01:13.221656+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:01:13.221656+0000","last_undegraded":"2024-03-30T15:01:13.221656+0000","last_fullsized":"2024-03-30T15:01:13.221656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:29.060822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:29.060822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":500,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251767+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:01:12.251767+0000","last_peered":"2024-03-30T15:01:12.251767+0000","last_clean":"2024-03-30T15:01:12.251767+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:01:12.251767+0000","last_undegraded":"2024-03-30T15:01:12.251767+0000","last_fullsized":"2024-03-30T15:01:12.251767+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:18.177682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":87,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.244950+0000","last_change":"2024-03-30T15:00:57.249517+0000","last_active":"2024-03-30T15:01:12.244950+0000","last_peered":"2024-03-30T15:01:12.244950+0000","last_clean":"2024-03-30T15:01:12.244950+0000","last_became_active":"2024-03-30T15:00:57.249210+0000","last_became_peered":"2024-03-30T15:00:57.249210+0000","last_unstale":"2024-03-30T15:01:12.244950+0000","last_undegraded":"2024-03-30T15:01:12.244950+0000","last_fullsized":"2024-03-30T15:01:12.244950+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:16.765776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.765776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":406,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.220206+0000","last_change":"2024-03-30T15:00:00.172131+0000","last_active":"2024-03-30T15:01:12.220206+0000","last_peered":"2024-03-30T15:01:12.220206+0000","last_clean":"2024-03-30T15:01:12.220206+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:01:12.220206+0000","last_undegraded":"2024-03-30T15:01:12.220206+0000","last_fullsized":"2024-03-30T15:01:12.220206+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.172053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.172053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":519,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224654+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:01:13.224654+0000","last_peered":"2024-03-30T15:01:13.224654+0000","last_clean":"2024-03-30T15:01:13.224654+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:01:13.224654+0000","last_undegraded":"2024-03-30T15:01:13.224654+0000","last_fullsized":"2024-03-30T15:01:13.224654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":504,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.253447+0000","last_change":"2024-03-30T14:58:25.078864+0000","last_active":"2024-03-30T15:01:12.253447+0000","last_peered":"2024-03-30T15:01:12.253447+0000","last_clean":"2024-03-30T15:01:12.253447+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:01:12.253447+0000","last_undegraded":"2024-03-30T15:01:12.253447+0000","last_fullsized":"2024-03-30T15:01:12.253447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.078771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.078771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":496,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.252055+0000","last_change":"2024-03-30T14:58:19.191376+0000","last_active":"2024-03-30T15:01:12.252055+0000","last_peered":"2024-03-30T15:01:12.252055+0000","last_clean":"2024-03-30T15:01:12.252055+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:01:12.252055+0000","last_undegraded":"2024-03-30T15:01:12.252055+0000","last_fullsized":"2024-03-30T15:01:12.252055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:19.191278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:19.191278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":504,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589323+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:01:13.589323+0000","last_peered":"2024-03-30T15:01:13.589323+0000","last_clean":"2024-03-30T15:01:13.589323+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:01:13.589323+0000","last_undegraded":"2024-03-30T15:01:13.589323+0000","last_fullsized":"2024-03-30T15:01:13.589323+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:43.919742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"301'7","reported_seq":94,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.592212+0000","last_change":"2024-03-30T15:00:53.232293+0000","last_active":"2024-03-30T15:01:13.592212+0000","last_peered":"2024-03-30T15:01:13.592212+0000","last_clean":"2024-03-30T15:01:13.592212+0000","last_became_active":"2024-03-30T15:00:53.231995+0000","last_became_peered":"2024-03-30T15:00:53.231995+0000","last_unstale":"2024-03-30T15:01:13.592212+0000","last_undegraded":"2024-03-30T15:01:13.592212+0000","last_fullsized":"2024-03-30T15:01:13.592212+0000","mapping_epoch":280,"log_start":"115'5","ondisk_log_start":"115'5","created":111,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.263392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.263392+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":5,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:46.109585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":402,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.251601+0000","last_change":"2024-03-30T14:57:55.106979+0000","last_active":"2024-03-30T15:01:12.251601+0000","last_peered":"2024-03-30T15:01:12.251601+0000","last_clean":"2024-03-30T15:01:12.251601+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:01:12.251601+0000","last_undegraded":"2024-03-30T15:01:12.251601+0000","last_fullsized":"2024-03-30T15:01:12.251601+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":500,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.220560+0000","last_change":"2024-03-30T14:59:59.784365+0000","last_active":"2024-03-30T15:01:13.220560+0000","last_peered":"2024-03-30T15:01:13.220560+0000","last_clean":"2024-03-30T15:01:13.220560+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:01:13.220560+0000","last_undegraded":"2024-03-30T15:01:13.220560+0000","last_fullsized":"2024-03-30T15:01:13.220560+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T14:59:59.784262+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.784262+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:16.754986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.245719+0000","last_change":"2024-03-30T14:59:06.484972+0000","last_active":"2024-03-30T15:01:12.245719+0000","last_peered":"2024-03-30T15:01:12.245719+0000","last_clean":"2024-03-30T15:01:12.245719+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:01:12.245719+0000","last_undegraded":"2024-03-30T15:01:12.245719+0000","last_fullsized":"2024-03-30T15:01:12.245719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:06.484877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T14:59:06.484877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":499,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223519+0000","last_change":"2024-03-30T14:59:12.396403+0000","last_active":"2024-03-30T15:01:13.223519+0000","last_peered":"2024-03-30T15:01:13.223519+0000","last_clean":"2024-03-30T15:01:13.223519+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:01:13.223519+0000","last_undegraded":"2024-03-30T15:01:13.223519+0000","last_fullsized":"2024-03-30T15:01:13.223519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.396321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":408,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.221320+0000","last_change":"2024-03-30T14:59:01.955656+0000","last_active":"2024-03-30T15:01:13.221320+0000","last_peered":"2024-03-30T15:01:13.221320+0000","last_clean":"2024-03-30T15:01:13.221320+0000","last_became_active":"2024-03-30T14:57:56.272206+0000","last_became_peered":"2024-03-30T14:57:56.272206+0000","last_unstale":"2024-03-30T15:01:13.221320+0000","last_undegraded":"2024-03-30T15:01:13.221320+0000","last_fullsized":"2024-03-30T15:01:13.221320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.955549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.955549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":93,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.588812+0000","last_change":"2024-03-30T15:01:05.421522+0000","last_active":"2024-03-30T15:01:13.588812+0000","last_peered":"2024-03-30T15:01:13.588812+0000","last_clean":"2024-03-30T15:01:13.588812+0000","last_became_active":"2024-03-30T15:01:05.420851+0000","last_became_peered":"2024-03-30T15:01:05.420851+0000","last_unstale":"2024-03-30T15:01:13.588812+0000","last_undegraded":"2024-03-30T15:01:13.588812+0000","last_fullsized":"2024-03-30T15:01:13.588812+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.029978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.029978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:58.437309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":406,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.589585+0000","last_change":"2024-03-30T14:57:54.063914+0000","last_active":"2024-03-30T15:01:13.589585+0000","last_peered":"2024-03-30T15:01:13.589585+0000","last_clean":"2024-03-30T15:01:13.589585+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:01:13.589585+0000","last_undegraded":"2024-03-30T15:01:13.589585+0000","last_fullsized":"2024-03-30T15:01:13.589585+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T14:57:46.318935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":527,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.223710+0000","last_change":"2024-03-30T14:59:57.415396+0000","last_active":"2024-03-30T15:01:13.223710+0000","last_peered":"2024-03-30T15:01:13.223710+0000","last_clean":"2024-03-30T15:01:13.223710+0000","last_became_active":"2024-03-30T14:54:39.335099+0000","last_became_peered":"2024-03-30T14:54:39.335099+0000","last_unstale":"2024-03-30T15:01:13.223710+0000","last_undegraded":"2024-03-30T15:01:13.223710+0000","last_fullsized":"2024-03-30T15:01:13.223710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:14.852042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":517,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.591431+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:01:13.591431+0000","last_peered":"2024-03-30T15:01:13.591431+0000","last_clean":"2024-03-30T15:01:13.591431+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:01:13.591431+0000","last_undegraded":"2024-03-30T15:01:13.591431+0000","last_fullsized":"2024-03-30T15:01:13.591431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":498,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.218168+0000","last_change":"2024-03-30T14:59:04.268770+0000","last_active":"2024-03-30T15:01:12.218168+0000","last_peered":"2024-03-30T15:01:12.218168+0000","last_clean":"2024-03-30T15:01:12.218168+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:01:12.218168+0000","last_undegraded":"2024-03-30T15:01:12.218168+0000","last_fullsized":"2024-03-30T15:01:12.218168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:04.268662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T14:59:04.268662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":495,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T15:01:13.224412+0000","last_change":"2024-03-30T14:58:27.174113+0000","last_active":"2024-03-30T15:01:13.224412+0000","last_peered":"2024-03-30T15:01:13.224412+0000","last_clean":"2024-03-30T15:01:13.224412+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:01:13.224412+0000","last_undegraded":"2024-03-30T15:01:13.224412+0000","last_fullsized":"2024-03-30T15:01:13.224412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:27.174017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:58:27.174017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.d","version":"301'12","reported_seq":96,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-30T15:01:12.243951+0000","last_change":"2024-03-30T15:01:01.258801+0000","last_active":"2024-03-30T15:01:12.243951+0000","last_peered":"2024-03-30T15:01:12.243951+0000","last_clean":"2024-03-30T15:01:12.243951+0000","last_became_active":"2024-03-30T15:01:01.258420+0000","last_became_peered":"2024-03-30T15:01:01.258420+0000","last_unstale":"2024-03-30T15:01:12.243951+0000","last_undegraded":"2024-03-30T15:01:12.243951+0000","last_fullsized":"2024-03-30T15:01:12.243951+0000","mapping_epoch":280,"log_start":"284'8","ondisk_log_start":"284'8","created":111,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'5","last_scrub_stamp":"2024-03-30T15:00:35.045519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:48.654936+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.045519+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:01:58.387708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":267,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":4195508,"num_objects":122,"num_object_clones":2,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":545,"num_write_kb":8730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14057472,"data_stored":12586524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1800,"ondisk_log_size":1800,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":389,"num_read_kb":384,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":17,"num_pg":25,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":75,"acting":75,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005140,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77448,"kb_used_data":5980,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93667704,"statfs":{"total":95995035648,"available":95915728896,"internally_reserved":0,"allocated":6123520,"data_stored":3693999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299481,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77372,"kb_used_data":5928,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93667780,"statfs":{"total":95995035648,"available":95915806720,"internally_reserved":0,"allocated":6070272,"data_stored":3691962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72632,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672520,"statfs":{"total":95995035648,"available":95920660480,"internally_reserved":0,"allocated":5476352,"data_stored":3102113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59340,"kb_used_data":10080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93685812,"statfs":{"total":95995035648,"available":95934271488,"internally_reserved":0,"allocated":10321920,"data_stored":7887851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80928,"kb_used_data":9484,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664224,"statfs":{"total":95995035648,"available":95912165376,"internally_reserved":0,"allocated":9711616,"data_stored":7307939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117097,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80968,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93664184,"statfs":{"total":95995035648,"available":95912124416,"internally_reserved":0,"allocated":9752576,"data_stored":7309173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76852,"kb_used_data":5408,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668300,"statfs":{"total":95995035648,"available":95916339200,"internally_reserved":0,"allocated":5537792,"data_stored":3114664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659230,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76840,"kb_used_data":5384,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668312,"statfs":{"total":95995035648,"available":95916351488,"internally_reserved":0,"allocated":5513216,"data_stored":3101512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4382720,"data_stored":4194739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4390912,"data_stored":4194770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4403200,"data_stored":4194797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:01:18.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 1 2024-03-30T15:01:18.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:18.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:18.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:19.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:19.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:19.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:19.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:19.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.49711 v1:172.21.15.28:0/3307019983' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: pgmap v410: 285 pgs: 32 unknown, 253 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.49711 v1:172.21.15.28:0/3307019983' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:01:19.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: pgmap v410: 285 pgs: 32 unknown, 253 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:01:19.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:19 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:19.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:19.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:19.596 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:19.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:19.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.49711 v1:172.21.15.28:0/3307019983' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:01:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: pgmap v410: 285 pgs: 32 unknown, 253 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-03-30T15:01:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:01:19.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: osdmap e315: 8 total, 8 up, 8 in 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]: dispatch 2024-03-30T15:01:19.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:19 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:19.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:19.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:19.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:19.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:19.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:20.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:20.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:01:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:20.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:20.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:20.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:20.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:20.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:20.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:20.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:20.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:20.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:20.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:20.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 17.4 scrub starts 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: 17.4 scrub ok 2024-03-30T15:01:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]': finished 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 17.4 scrub starts 2024-03-30T15:01:20.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: 17.4 scrub ok 2024-03-30T15:01:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]': finished 2024-03-30T15:01:20.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:20.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:20.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:20.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:20.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:20.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:20.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:01:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:01:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:01:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 17.4 scrub starts 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: 17.4 scrub ok 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-37", "mode": "writeback"}]': finished 2024-03-30T15:01:21.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:21.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:21.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:21.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:21.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:21.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:21.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.6 scrub starts 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.6 scrub ok 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: pgmap v413: 285 pgs: 2 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.8 scrub starts 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.8 scrub ok 2024-03-30T15:01:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.a scrub starts 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.a scrub ok 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.2 deep-scrub starts 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[41614]: 42.2 deep-scrub ok 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.6 scrub starts 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.6 scrub ok 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: pgmap v413: 285 pgs: 2 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.8 scrub starts 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.8 scrub ok 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:21.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.a scrub starts 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.a scrub ok 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.2 deep-scrub starts 2024-03-30T15:01:21.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:21 smithi028 ceph-mon[29876]: 42.2 deep-scrub ok 2024-03-30T15:01:21.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:21.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:21.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/1483978366 >> v1:172.21.15.28:6790/0 conn(0x7f5b94101130 legacy=0x7f5b94101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/1483978366 shutdown_connections 2024-03-30T15:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/1483978366 >> 172.21.15.28:0/1483978366 conn(0x7f5b940fc9e0 msgr2=0x7f5b940fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/1483978366 shutdown_connections 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/1483978366 wait complete. 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 librados: starting msgr at 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.897+0000 7f5b9c2a8700 1 librados: starting objecter 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 Processor -- start 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 -- start start 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 librados: setting wanted keys 2024-03-30T15:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 librados: calling monclient init 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5b940feb60 con 0x7f5b94109dc0 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5b940fed70 con 0x7f5b9410d960 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9c2a8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5b941ab940 con 0x7f5b94101130 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9a044700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5b94101130 0x7f5b940fce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52206/0 (socket says 172.21.15.28:52206) 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b9a044700 1 -- 172.21.15.28:0/81194448 learned_addr learned my addr 172.21.15.28:0/81194448 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f5b94101130 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f5b94109dc0 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.898+0000 7f5b8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 10 client.?.objecter ms_handle_connect 0x7f5b9410d960 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1907495370 0 0) 0x7f5b941ab940 con 0x7f5b94101130 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b800031f0 con 0x7f5b94101130 2024-03-30T15:01:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555576770 0 0) 0x7f5b940fed70 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b941ab940 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145765288 0 0) 0x7f5b940feb60 con 0x7f5b94109dc0 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b940fed70 con 0x7f5b94109dc0 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 617159695 0 0) 0x7f5b941ab940 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b940feb60 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5b8c001c30 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 424704318 0 0) 0x7f5b800031f0 con 0x7f5b94101130 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b941ab940 con 0x7f5b94101130 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5b900015d0 con 0x7f5b94101130 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3854443770 0 0) 0x7f5b940feb60 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.899+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 >> v1:172.21.15.28:6790/0 conn(0x7f5b94101130 legacy=0x7f5b940fce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 >> v1:172.21.15.28:6789/0 conn(0x7f5b94109dc0 legacy=0x7f5b940fd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b941acb60 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b941abb70 con 0x7f5b9410d960 2024-03-30T15:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b9c2a8700 10 client.49723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b941ac030 con 0x7f5b9410d960 2024-03-30T15:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b9c2a8700 1 librados: init done 2024-03-30T15:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5b8c003ec0 con 0x7f5b9410d960 2024-03-30T15:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.900+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5b8c004e90 con 0x7f5b9410d960 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.901+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b78005120 con 0x7f5b9410d960 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.901+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5b8c01b6d0 con 0x7f5b9410d960 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(317..317 src has 53..317) ==== 9764+0+0 (unknown 2792723897 0 0) 0x7f5b8c088f90 con 0x7f5b9410d960 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 10 client.49723.objecter ms_dispatch 0x7f5b94001550 osd_map(317..317 src has 53..317) 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 3 client.49723.objecter handle_osd_map got epochs [317,317] > 0 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 3 client.49723.objecter handle_osd_map decoding full epoch 317 2024-03-30T15:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 20 client.49723.objecter dump_active .. 0 homeless 2024-03-30T15:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.902+0000 7f5b8b7fe700 10 client.49723.objecter ms_handle_connect 0x7f5b80070330 2024-03-30T15:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:21.906+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5b8c055b40 con 0x7f5b9410d960 2024-03-30T15:01:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.6 scrub starts 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.6 scrub ok 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: pgmap v413: 285 pgs: 2 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.8 scrub starts 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.8 scrub ok 2024-03-30T15:01:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.a scrub starts 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.a scrub ok 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: osdmap e316: 8 total, 8 up, 8 in 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi028-92999-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: osdmap e317: 8 total, 8 up, 8 in 2024-03-30T15:01:21.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]: dispatch 2024-03-30T15:01:21.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:21.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.2 deep-scrub starts 2024-03-30T15:01:21.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:21 smithi134 ceph-mon[33714]: 42.2 deep-scrub ok 2024-03-30T15:01:22.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:22.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.126+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5b78005410 con 0x7f5b9410d960 2024-03-30T15:01:22.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:22.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.221+0000 7f5b8b7fe700 1 -- 172.21.15.28:0/81194448 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v318) ==== 87+0+0 (unknown 3899698685 0 0) 0x7f5b8c059590 con 0x7f5b9410d960 2024-03-30T15:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.225+0000 7f5b9c2a8700 10 librados: watch_flush enter 2024-03-30T15:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.225+0000 7f5b9c2a8700 10 librados: watch_flush exit 2024-03-30T15:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.225+0000 7f5b9c2a8700 20 client.49723.objecter shutdown clearing up homeless session... 2024-03-30T15:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.225+0000 7f5b9c2a8700 10 client.49723.objecter successfully canceled tick 2024-03-30T15:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.226+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5b80070330 legacy=0x7f5b800727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.226+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 >> v1:172.21.15.134:6789/0 conn(0x7f5b9410d960 legacy=0x7f5b941a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.227+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 shutdown_connections 2024-03-30T15:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.227+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 >> 172.21.15.28:0/81194448 conn(0x7f5b940fc9e0 msgr2=0x7f5b9406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.227+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 shutdown_connections 2024-03-30T15:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.228+0000 7f5b9c2a8700 1 -- 172.21.15.28:0/81194448 wait complete. 2024-03-30T15:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:22.228+0000 7f5b9c2a8700 1 librados: shutdown 2024-03-30T15:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:01:22.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.302 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:22.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:22.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:22.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:22.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:22.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:22.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:22.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:22.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:22.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:22.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: pgmap v416: 292 pgs: 8 unknown, 1 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/81194448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='client.49723 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: 42.7 scrub starts 2024-03-30T15:01:22.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: 42.7 scrub ok 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='client.49723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:22.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: pgmap v416: 292 pgs: 8 unknown, 1 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/81194448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='client.49723 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: 42.7 scrub starts 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: 42.7 scrub ok 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='client.49723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T15:01:22.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:22.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:22.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:22.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:23.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:23.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:23.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:01:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:01:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: pgmap v416: 292 pgs: 8 unknown, 1 peering, 283 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/81194448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='client.49723 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: 42.7 scrub starts 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: 42.7 scrub ok 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-37"}]': finished 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='client.49723 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: osdmap e318: 8 total, 8 up, 8 in 2024-03-30T15:01:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:23.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:23.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:23.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:23.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:23.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:23.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:23.595 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:23.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:23.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.852 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:23.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:23.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:23.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:23.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:24.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:24.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:24.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:24.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:24.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[29876]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[41614]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T15:01:24.377 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.378 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:24.378 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:24 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:24.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:24.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:24.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.650+0000 7f00c7170700 1 -- 172.21.15.28:0/2456494016 >> v1:172.21.15.28:6789/0 conn(0x7f00c0109dc0 legacy=0x7f00c010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.650+0000 7f00c7170700 1 -- 172.21.15.28:0/2456494016 shutdown_connections 2024-03-30T15:01:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.650+0000 7f00c7170700 1 -- 172.21.15.28:0/2456494016 >> 172.21.15.28:0/2456494016 conn(0x7f00c00fc9e0 msgr2=0x7f00c00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- 172.21.15.28:0/2456494016 shutdown_connections 2024-03-30T15:01:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- 172.21.15.28:0/2456494016 wait complete. 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 librados: starting msgr at 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 librados: starting objecter 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 Processor -- start 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- start start 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 librados: setting wanted keys 2024-03-30T15:01:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 librados: calling monclient init 2024-03-30T15:01:24.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f00c01a6070 con 0x7f00c0109dc0 2024-03-30T15:01:24.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f00c01a7290 con 0x7f00c010d960 2024-03-30T15:01:24.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00c7170700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f00c01a84d0 con 0x7f00c0101130 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00bffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f00c0109dc0 0x7f00c01a0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52768/0 (socket says 172.21.15.28:52768) 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.651+0000 7f00bffff700 1 -- 172.21.15.28:0/1828317325 learned_addr learned my addr 172.21.15.28:0/1828317325 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter ms_handle_connect 0x7f00c0109dc0 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter ms_handle_connect 0x7f00c010d960 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731332072 0 0) 0x7f00c01a7290 con 0x7f00c010d960 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00a80031f0 con 0x7f00c010d960 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4076671995 0 0) 0x7f00c01a6070 con 0x7f00c0109dc0 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00c01a7290 con 0x7f00c0109dc0 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter ms_handle_connect 0x7f00c0101130 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1199358786 0 0) 0x7f00a80031f0 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f00c01a6070 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f00b00017f0 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765373457 0 0) 0x7f00c01a84d0 con 0x7f00c0101130 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00a80031f0 con 0x7f00c0101130 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3484292853 0 0) 0x7f00c01a7290 con 0x7f00c0109dc0 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f00c01a84d0 con 0x7f00c0109dc0 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f00b4004610 con 0x7f00c0109dc0 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2376148026 0 0) 0x7f00c01a6070 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 >> v1:172.21.15.28:6790/0 conn(0x7f00c0101130 legacy=0x7f00c0106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 >> v1:172.21.15.28:6789/0 conn(0x7f00c0109dc0 legacy=0x7f00c01a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.652+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c01a9710 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f00c01a7460 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f00b0003b20 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f00b00049f0 con 0x7f00c010d960 2024-03-30T15:01:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00c7170700 10 client.49729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f00c01a7960 con 0x7f00c010d960 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.653+0000 7f00c7170700 1 librados: init done 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.654+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c0102610 con 0x7f00c010d960 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.654+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f00b0003680 con 0x7f00c010d960 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(320..320 src has 53..320) ==== 8988+0+0 (unknown 1247401394 0 0) 0x7f00b0088800 con 0x7f00c010d960 2024-03-30T15:01:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 10 client.49729.objecter ms_dispatch 0x7f00c0001550 osd_map(320..320 src has 53..320) 2024-03-30T15:01:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 3 client.49729.objecter handle_osd_map got epochs [320,320] > 0 2024-03-30T15:01:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 3 client.49729.objecter handle_osd_map decoding full epoch 320 2024-03-30T15:01:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 20 client.49729.objecter dump_active .. 0 homeless 2024-03-30T15:01:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.655+0000 7f00bdffb700 10 client.49729.objecter ms_handle_connect 0x7f00a80701d0 2024-03-30T15:01:24.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.659+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f00b00556b0 con 0x7f00c010d960 2024-03-30T15:01:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:24 smithi134 ceph-mon[33714]: osdmap e319: 8 total, 8 up, 8 in 2024-03-30T15:01:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:24 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:24.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:24.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:01:24.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:24.865+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f00c0104ae0 con 0x7f00c010d960 2024-03-30T15:01:24.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:24.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:01:24.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:24.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:24.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:25.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:25.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:25.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.244+0000 7f00bdffb700 1 -- 172.21.15.28:0/1828317325 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v321) ==== 129+0+0 (unknown 2403943880 0 0) 0x7f00b0059100 con 0x7f00c010d960 2024-03-30T15:01:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 10 librados: watch_flush enter 2024-03-30T15:01:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 10 librados: watch_flush exit 2024-03-30T15:01:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 20 client.49729.objecter shutdown clearing up homeless session... 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 10 client.49729.objecter successfully canceled tick 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 >> v1:172.21.15.28:6800/3011431093 conn(0x7f00a80701d0 legacy=0x7f00a8072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 >> v1:172.21.15.134:6789/0 conn(0x7f00c010d960 legacy=0x7f00c01a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 shutdown_connections 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 >> 172.21.15.28:0/1828317325 conn(0x7f00c00fc9e0 msgr2=0x7f00c01105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 shutdown_connections 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.249+0000 7f00c7170700 1 -- 172.21.15.28:0/1828317325 wait complete. 2024-03-30T15:01:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:25.250+0000 7f00c7170700 1 librados: shutdown 2024-03-30T15:01:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 1 2024-03-30T15:01:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:25.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:25.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:25.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:25.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: pgmap v419: 252 pgs: 2 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:01:25.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:25.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1828317325' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[41614]: from='client.49729 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: pgmap v419: 252 pgs: 2 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:25.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1828317325' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:25 smithi028 ceph-mon[29876]: from='client.49729 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:01:25.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: pgmap v419: 252 pgs: 2 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: osdmap e320: 8 total, 8 up, 8 in 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2330167966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]: dispatch 2024-03-30T15:01:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1828317325' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:25 smithi134 ceph-mon[33714]: from='client.49729 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:01:25.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:25.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:25.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:25.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:25.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:26.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:26.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:26.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:26.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:26.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:26.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:26.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:26.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.49729 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.49729 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:26.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:26.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:26.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.57695 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi028-92999-53"}]': finished 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.49729 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: osdmap e321: 8 total, 8 up, 8 in 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-30T15:01:26.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:26.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:26.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:26.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:27.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:27.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:27.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:27.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:27.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:27.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:27.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:27.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:27.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:27.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:27.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: pgmap v422: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:27.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: pgmap v422: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:27.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:27.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:27.798 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:27.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:27.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: pgmap v422: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi028-92999-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: osdmap e322: 8 total, 8 up, 8 in 2024-03-30T15:01:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:27.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:28.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:28.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:28.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:28.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:28.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:28.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:28.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:28.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:28.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:28.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:28.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:28.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:28.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:28.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.840 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:28.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:28.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:28.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: pgmap v425: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]': finished 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:28.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: pgmap v425: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]': finished 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: osdmap e323: 8 total, 8 up, 8 in 2024-03-30T15:01:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]: dispatch 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: pgmap v425: 283 pgs: 32 unknown, 1 clean+premerge+peered, 250 active+clean; 4.6 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi028-92999-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-39", "mode": "writeback"}]': finished 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: osdmap e324: 8 total, 8 up, 8 in 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:28.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:28.965 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:28.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:29.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.160+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/1918035749 >> v1:172.21.15.28:6790/0 conn(0x7f0a5810eb80 legacy=0x7f0a58110f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.160+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/1918035749 shutdown_connections 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/1918035749 >> 172.21.15.28:0/1918035749 conn(0x7f0a580fca30 msgr2=0x7f0a580fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/1918035749 shutdown_connections 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/1918035749 wait complete. 2024-03-30T15:01:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 librados: starting msgr at 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 librados: starting objecter 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 Processor -- start 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- start start 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 librados: setting wanted keys 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 librados: calling monclient init 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0a581a7160 con 0x7f0a58103260 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0a581a8380 con 0x7f0a5810eb80 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5f0cf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0a581a95c0 con 0x7f0a5810b070 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5ce6b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a58103260 0x7f0a5810a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37196/0 (socket says 172.21.15.28:37196) 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a5ce6b700 1 -- 172.21.15.28:0/3946905121 learned_addr learned my addr 172.21.15.28:0/3946905121 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a55ffb700 10 client.?.objecter ms_handle_connect 0x7f0a5810b070 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a55ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a55ffb700 10 client.?.objecter ms_handle_connect 0x7f0a58103260 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.161+0000 7f0a55ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 10 client.?.objecter ms_handle_connect 0x7f0a5810eb80 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990680395 0 0) 0x7f0a581a95c0 con 0x7f0a5810b070 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a480031f0 con 0x7f0a5810b070 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446812617 0 0) 0x7f0a581a7160 con 0x7f0a58103260 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a581a95c0 con 0x7f0a58103260 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1891739080 0 0) 0x7f0a581a8380 con 0x7f0a5810eb80 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a581a7160 con 0x7f0a5810eb80 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2322949498 0 0) 0x7f0a480031f0 con 0x7f0a5810b070 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a581a8380 con 0x7f0a5810b070 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0a44001a30 con 0x7f0a5810b070 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 462415087 0 0) 0x7f0a581a95c0 con 0x7f0a58103260 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a480031f0 con 0x7f0a58103260 2024-03-30T15:01:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0a4c001870 con 0x7f0a58103260 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3081960780 0 0) 0x7f0a581a7160 con 0x7f0a5810eb80 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a581a95c0 con 0x7f0a5810eb80 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0a50004880 con 0x7f0a5810eb80 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2893908118 0 0) 0x7f0a581a8380 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.134:6789/0 conn(0x7f0a5810eb80 legacy=0x7f0a581a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6789/0 conn(0x7f0a58103260 legacy=0x7f0a5810a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a581aa800 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a581a7390 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0a44001c90 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0a44004a00 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a5f0cf700 10 client.57719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a581a7850 con 0x7f0a5810b070 2024-03-30T15:01:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.162+0000 7f0a5f0cf700 1 librados: init done 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.163+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0a5806adb0 con 0x7f0a5810b070 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0a44004ba0 con 0x7f0a5810b070 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(324..324 src has 53..324) ==== 9744+0+0 (unknown 2028797272 0 0) 0x7f0a44088c00 con 0x7f0a5810b070 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 osd_map(324..324 src has 53..324) 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 3 client.57719.objecter handle_osd_map got epochs [324,324] > 0 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 3 client.57719.objecter handle_osd_map decoding full epoch 324 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 20 client.57719.objecter dump_active .. 0 homeless 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a48070550 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.164+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v324) ==== 54+0+16 (unknown 737359538 0 2420000192) 0x7f0a4408b060 con 0x7f0a5810b070 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 1 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 1 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 0 1 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 4 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 1 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a58108ed0 con 0x7f0a581a8550 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.165+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a581a8550 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.166+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a58108ed0 con 0x7f0a581a8550 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.166+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.166+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 1 = system:0 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.166+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 4 2024-03-30T15:01:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.166+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 0 1 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 2 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 2 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a58114ec0 osd=0 3 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 0 2 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 4 2024-03-30T15:01:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 2 2024-03-30T15:01:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.183+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a58068b30 con 0x7f0a581a8550 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a58068b30 con 0x7f0a581a8550 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 4 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 0 2 2024-03-30T15:01:29.187 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 1 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 4 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 3 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.186+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0a5802d9c0 con 0x7f0a5810b960 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a5810b960 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a5802d9c0 con 0x7f0a5810b960 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 3 = system:0 2024-03-30T15:01:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 4 2024-03-30T15:01:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.187+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 1 3 2024-03-30T15:01:29.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 4 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 4 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581a9290 osd=1 3 2024-03-30T15:01:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 1 4 2024-03-30T15:01:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 4 2024-03-30T15:01:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 4 2024-03-30T15:01:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.206+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a5802ddf0 con 0x7f0a5810b960 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a5802ddf0 con 0x7f0a5810b960 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 4 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 1 4 2024-03-30T15:01:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 5 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 5 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 2 5 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 4 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 5 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0a580632a0 con 0x7f0a58062e50 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.209+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a58062e50 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.210+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a580632a0 con 0x7f0a58062e50 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.210+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.210+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 5 = system:0 2024-03-30T15:01:29.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.210+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 4 2024-03-30T15:01:29.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.210+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 2 5 2024-03-30T15:01:29.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 6 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 6 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a5802dfa0 osd=2 3 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 2 6 2024-03-30T15:01:29.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 4 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 6 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.229+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581b3360 con 0x7f0a58062e50 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.230+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581b3360 con 0x7f0a58062e50 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.230+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.230+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.230+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 4 2024-03-30T15:01:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.230+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 2 6 2024-03-30T15:01:29.232 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 7 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 7 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 3 7 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 4 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 7 2024-03-30T15:01:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0a581af7c0 con 0x7f0a58062a40 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.231+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a58062a40 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.232+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a581af7c0 con 0x7f0a58062a40 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.232+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.232+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 7 = system:0 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.232+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 4 2024-03-30T15:01:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.232+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 3 7 2024-03-30T15:01:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 8 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 8 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a580628d0 osd=3 3 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 3 8 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 4 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 8 2024-03-30T15:01:29.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.249+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581afa40 con 0x7f0a58062a40 2024-03-30T15:01:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.252+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581afa40 con 0x7f0a58062a40 2024-03-30T15:01:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.252+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.252+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.252+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 4 2024-03-30T15:01:29.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.252+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 3 8 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 9 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 9 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 4 9 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 4 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 9 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.253+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0a581b4170 con 0x7f0a581b3ce0 2024-03-30T15:01:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.254+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a581b3ce0 2024-03-30T15:01:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.258+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a581b4170 con 0x7f0a581b3ce0 2024-03-30T15:01:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.258+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.258+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 9 = system:0 2024-03-30T15:01:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.258+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 4 2024-03-30T15:01:29.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.258+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 4 9 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 10 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 10 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581afa40 osd=4 3 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 4 10 2024-03-30T15:01:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 4 2024-03-30T15:01:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 10 2024-03-30T15:01:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.280+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581b43a0 con 0x7f0a581b3ce0 2024-03-30T15:01:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.281+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581b43a0 con 0x7f0a581b3ce0 2024-03-30T15:01:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 4 2024-03-30T15:01:29.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 4 10 2024-03-30T15:01:29.283 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 11 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 11 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 5 11 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 4 2024-03-30T15:01:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 11 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0a581bed90 con 0x7f0a58101300 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.282+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a58101300 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.283+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a581bed90 con 0x7f0a58101300 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.283+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.283+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 11 = system:0 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.283+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 4 2024-03-30T15:01:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.283+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 5 11 2024-03-30T15:01:29.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 12 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 12 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581b43a0 osd=5 3 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 5 12 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 4 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 12 2024-03-30T15:01:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.300+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581bed90 con 0x7f0a58101300 2024-03-30T15:01:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.302+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581bed90 con 0x7f0a58101300 2024-03-30T15:01:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.302+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.302+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.302+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 4 2024-03-30T15:01:29.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.302+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 5 12 2024-03-30T15:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 13 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 13 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 6 13 2024-03-30T15:01:29.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 4 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 13 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0a581c6eb0 con 0x7f0a581c33a0 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.303+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a581c33a0 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.304+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a581c6eb0 con 0x7f0a581c33a0 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.304+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.304+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 13 = system:0 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.304+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 4 2024-03-30T15:01:29.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.304+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 6 13 2024-03-30T15:01:29.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 14 2024-03-30T15:01:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 14 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581bed90 osd=6 3 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 6 14 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 4 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 14 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.320+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581c6eb0 con 0x7f0a581c33a0 2024-03-30T15:01:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581c6eb0 con 0x7f0a581c33a0 2024-03-30T15:01:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 4 2024-03-30T15:01:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 6 14 2024-03-30T15:01:29.324 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 15 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 15 2024-03-30T15:01:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 7 15 2024-03-30T15:01:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 4 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 15 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.322+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0a581cb1b0 con 0x7f0a581c75d0 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.323+0000 7f0a55ffb700 10 client.57719.objecter ms_handle_connect 0x7f0a581c75d0 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.324+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0a581cb1b0 con 0x7f0a581c75d0 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.324+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.324+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 15 = system:0 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.324+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 4 2024-03-30T15:01:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.324+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 7 15 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 10 client.57719.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign -1 16 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 20 client.57719.objecter _get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_remove -1 16 2024-03-30T15:01:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 20 client.57719.objecter get_session s=0x7f0a581c6eb0 osd=7 3 2024-03-30T15:01:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 15 client.57719.objecter _session_command_op_assign 7 16 2024-03-30T15:01:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 4 2024-03-30T15:01:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 10 client.57719.objecter _send_command 16 2024-03-30T15:01:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.343+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0a581cb1b0 con 0x7f0a581c75d0 2024-03-30T15:01:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:29.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a55ffb700 1 -- 172.21.15.28:0/3946905121 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0a581cb1b0 con 0x7f0a581c75d0 2024-03-30T15:01:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a55ffb700 10 client.57719.objecter ms_dispatch 0x7f0a58001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a55ffb700 10 client.57719.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a55ffb700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 4 2024-03-30T15:01:29.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a55ffb700 15 client.57719.objecter _session_command_op_remove 7 16 2024-03-30T15:01:29.347 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 librados: watch_flush enter 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 librados: watch_flush exit 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.0 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6801/3394323575 conn(0x7f0a581a8550 legacy=0x7f0a581a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a58114ec0 osd=0 1 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.1 2024-03-30T15:01:29.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6805/1850313547 conn(0x7f0a5810b960 legacy=0x7f0a5810bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581a9290 osd=1 1 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.2 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6809/3451325043 conn(0x7f0a58062e50 legacy=0x7f0a581b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a5802dfa0 osd=2 1 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.3 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6813/1506157250 conn(0x7f0a58062a40 legacy=0x7f0a581b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a580628d0 osd=3 1 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.4 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.134:6800/3063775514 conn(0x7f0a581b3ce0 legacy=0x7f0a581baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581afa40 osd=4 1 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.5 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.134:6804/3668151179 conn(0x7f0a58101300 legacy=0x7f0a581be680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581b43a0 osd=5 1 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.6 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.134:6808/2135224194 conn(0x7f0a581c33a0 legacy=0x7f0a581c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581bed90 osd=6 1 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter close_session for osd.7 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5d66c700 1 -- 172.21.15.28:0/3946905121 reap_dead start 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.134:6812/3758190298 conn(0x7f0a581c75d0 legacy=0x7f0a581c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5d66c700 1 -- 172.21.15.28:0/3946905121 reap_dead start 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5d66c700 1 -- 172.21.15.28:0/3946905121 reap_dead start 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5d66c700 1 -- 172.21.15.28:0/3946905121 reap_dead start 2024-03-30T15:01:29.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter put_session s=0x7f0a581c6eb0 osd=7 1 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 20 client.57719.objecter shutdown clearing up homeless session... 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 10 client.57719.objecter successfully canceled tick 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0a48070550 legacy=0x7f0a48072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.345+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> v1:172.21.15.28:6790/0 conn(0x7f0a5810b070 legacy=0x7f0a581a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.346+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 shutdown_connections 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.346+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 >> 172.21.15.28:0/3946905121 conn(0x7f0a580fca30 msgr2=0x7f0a580fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.346+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 shutdown_connections 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.346+0000 7f0a5f0cf700 1 -- 172.21.15.28:0/3946905121 wait complete. 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:29.346+0000 7f0a5f0cf700 1 librados: shutdown 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:29.351 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:29.352 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:29.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:29.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:29.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:29.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:29.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946905121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946905121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:29.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:29.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946905121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: osdmap e325: 8 total, 8 up, 8 in 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]: dispatch 2024-03-30T15:01:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:29.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:29.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:29.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:30.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:30.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:30.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:30.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:30.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:30.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:30.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:30.523 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:30.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: pgmap v428: 290 pgs: 8 creating+peering, 1 peering, 281 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 109 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:30.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: pgmap v428: 290 pgs: 8 creating+peering, 1 peering, 281 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 109 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:30.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:30.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T15:01:30.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T15:01:30.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:30.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:30.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:30.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:30.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:30.776 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:30.776 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:30.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:30.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:30.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: pgmap v428: 290 pgs: 8 creating+peering, 1 peering, 281 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 109 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-39"}]': finished 2024-03-30T15:01:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-30T15:01:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: osdmap e326: 8 total, 8 up, 8 in 2024-03-30T15:01:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:30.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:31.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:31.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:31.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:31.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: seed 93253 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (1358 ms) 2024-03-30T15:01:31.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (14065 ms) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (1 ms) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (15424 ms total) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (10335 ms) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (10915 ms) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9860 ms) 2024-03-30T15:01:31.258 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: my_snaps [3] 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: my_snaps [4,3] 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: promoting some heads 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: done waiting 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46038 ms) 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9982 ms) 2024-03-30T15:01:31.259 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-30T15:01:31.260 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8260 ms) 2024-03-30T15:01:31.260 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-30T15:01:31.260 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7732 ms) 2024-03-30T15:01:31.260 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-30T15:01:31.260 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8010 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10004 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9265 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9739 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12017 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7988 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8000 ms) 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-30T15:01:31.261 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13006 ms) 2024-03-30T15:01:31.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-30T15:01:31.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8007 ms) 2024-03-30T15:01:31.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-30T15:01:31.262 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8026 ms) 2024-03-30T15:01:31.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:01:31.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:01:31.326 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-30T15:01:31.326 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.131646 2024-03-30T15:01:31.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 3 0.13164586456244032 2024-03-30T15:01:31.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:31.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:31.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:31.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:31.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:31.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:31.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:31.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:31.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:31.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:31.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:31.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:31.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: osdmap e327: 8 total, 8 up, 8 in 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]: dispatch 2024-03-30T15:01:31.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:31.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:32.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:32.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:32.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:32.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:32.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:32.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:01:32.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:32.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:32.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:32.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:32.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:32.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:32.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:01:32.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:32.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:32.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:33.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:33.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:33.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:33.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.269+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1717206768 >> v1:172.21.15.28:6789/0 conn(0x7f1a34101130 legacy=0x7f1a34101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.269+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1717206768 shutdown_connections 2024-03-30T15:01:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.269+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1717206768 >> 172.21.15.28:0/1717206768 conn(0x7f1a340fc9e0 msgr2=0x7f1a340fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.269+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1717206768 shutdown_connections 2024-03-30T15:01:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.269+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1717206768 wait complete. 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: pgmap v431: 250 pgs: 1 peering, 249 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: pgmap v431: 250 pgs: 1 peering, 249 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T15:01:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 librados: starting msgr at 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 librados: starting objecter 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 Processor -- start 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 -- start start 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 librados: setting wanted keys 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.271+0000 7f1a3c16f700 1 librados: calling monclient init 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a3c16f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a341a71d0 con 0x7f1a3410d960 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a3c16f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a341a8410 con 0x7f1a34101130 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a39f0b700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1a34101130 0x7f1a341063d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:37328/0 (socket says 172.21.15.28:37328) 2024-03-30T15:01:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a39f0b700 1 -- 172.21.15.28:0/1500921484 learned_addr learned my addr 172.21.15.28:0/1500921484 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a3970a700 1 --1- 172.21.15.28:0/1500921484 >> v1:172.21.15.28:6790/0 conn(0x7f1a34109dc0 0x7f1a341a1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40836/0 (socket says 172.21.15.28:40836) 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1a341a9650 con 0x7f1a34109dc0 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter ms_handle_connect 0x7f1a34101130 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter ms_handle_connect 0x7f1a34109dc0 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627071940 0 0) 0x7f1a341a8410 con 0x7f1a34101130 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a200035b0 con 0x7f1a34101130 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter ms_handle_connect 0x7f1a3410d960 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525734201 0 0) 0x7f1a341a9650 con 0x7f1a34109dc0 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a341a8410 con 0x7f1a34109dc0 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.272+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4104497460 0 0) 0x7f1a200035b0 con 0x7f1a34101130 2024-03-30T15:01:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a341a9650 con 0x7f1a34101130 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1580220610 0 0) 0x7f1a341a71d0 con 0x7f1a3410d960 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a200035b0 con 0x7f1a3410d960 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1a2c001950 con 0x7f1a34101130 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2551815357 0 0) 0x7f1a341a8410 con 0x7f1a34109dc0 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a341a71d0 con 0x7f1a34109dc0 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2905909248 0 0) 0x7f1a200035b0 con 0x7f1a3410d960 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a341a8410 con 0x7f1a3410d960 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2546526426 0 0) 0x7f1a341a9650 con 0x7f1a34101130 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 >> v1:172.21.15.28:6790/0 conn(0x7f1a34109dc0 legacy=0x7f1a341a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 >> v1:172.21.15.28:6789/0 conn(0x7f1a3410d960 legacy=0x7f1a341a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a341aa890 con 0x7f1a34101130 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a341a9880 con 0x7f1a34101130 2024-03-30T15:01:33.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a3c16f700 10 client.49750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a341a9d40 con 0x7f1a34101130 2024-03-30T15:01:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.273+0000 7f1a3c16f700 1 librados: init done 2024-03-30T15:01:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.274+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1a2c0020a0 con 0x7f1a34101130 2024-03-30T15:01:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.275+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1a2c005100 con 0x7f1a34101130 2024-03-30T15:01:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.275+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a18005120 con 0x7f1a34101130 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.276+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1a2c001c00 con 0x7f1a34101130 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(329..329 src has 53..329) ==== 9212+0+0 (unknown 2641053404 0 0) 0x7f1a2c089ee0 con 0x7f1a34101130 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 10 client.49750.objecter ms_dispatch 0x7f1a34001550 osd_map(329..329 src has 53..329) 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 3 client.49750.objecter handle_osd_map got epochs [329,329] > 0 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 3 client.49750.objecter handle_osd_map decoding full epoch 329 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 20 client.49750.objecter dump_active .. 0 homeless 2024-03-30T15:01:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.277+0000 7f1a2affd700 10 client.49750.objecter ms_handle_connect 0x7f1a200749e0 2024-03-30T15:01:33.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.281+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1a2c056cb0 con 0x7f1a34101130 2024-03-30T15:01:33.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:33.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:33.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:33.504+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1a18005410 con 0x7f1a34101130 2024-03-30T15:01:33.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:33.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:33.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: pgmap v431: 250 pgs: 1 peering, 249 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2126246388' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi028-92999-54"}]': finished 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: osdmap e328: 8 total, 8 up, 8 in 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:33.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:33.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:33.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:33.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:33.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:33.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:33.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:34.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:34.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:34.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.242+0000 7f1a2affd700 1 -- 172.21.15.28:0/1500921484 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v330) ==== 97+0+0 (unknown 3353152056 0 0) 0x7f1a2c05a700 con 0x7f1a34101130 2024-03-30T15:01:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 10 librados: watch_flush enter 2024-03-30T15:01:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 10 librados: watch_flush exit 2024-03-30T15:01:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 20 client.49750.objecter shutdown clearing up homeless session... 2024-03-30T15:01:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 10 client.49750.objecter successfully canceled tick 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1a200749e0 legacy=0x7f1a20076ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.245+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 >> v1:172.21.15.134:6789/0 conn(0x7f1a34101130 legacy=0x7f1a341063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.246+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 shutdown_connections 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.246+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 >> 172.21.15.28:0/1500921484 conn(0x7f1a340fc9e0 msgr2=0x7f1a34110d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.246+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 shutdown_connections 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.246+0000 7f1a3c16f700 1 -- 172.21.15.28:0/1500921484 wait complete. 2024-03-30T15:01:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:34.246+0000 7f1a3c16f700 1 librados: shutdown 2024-03-30T15:01:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:01:34.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:34.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:34.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1500921484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='client.49750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1500921484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='client.49750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:34 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:34.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi028-92999-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: osdmap e329: 8 total, 8 up, 8 in 2024-03-30T15:01:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1500921484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='client.49750 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:34 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-30T15:01:34.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:34.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:34.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:34.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:35.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:35.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:35.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:35.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:35.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: pgmap v434: 281 pgs: 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:35.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.49750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: pgmap v434: 281 pgs: 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.49750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:35.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:35.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:35.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: pgmap v434: 281 pgs: 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.49750 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: osdmap e330: 8 total, 8 up, 8 in 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-30T15:01:35.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:01:35.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:35.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:35.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:35.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:35.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:35.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:35.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:36.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.123 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:36.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:36.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:36.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:36.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:36.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.508 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:36.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:36.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi028-92999-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.24865 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi028-93081-1","var":"pgp_num","val":"11"}]': finished 2024-03-30T15:01:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:01:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3133651075' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-30T15:01:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: osdmap e331: 8 total, 8 up, 8 in 2024-03-30T15:01:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:36.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:36.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:36.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:36.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:36.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:37.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:37.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:37.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:37.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:37.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:37.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:37.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:37.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:37.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: pgmap v437: 289 pgs: 8 unknown, 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:37.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:37.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: pgmap v437: 289 pgs: 8 unknown, 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T15:01:37.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:37.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:37.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:37.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:37.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: pgmap v437: 289 pgs: 8 unknown, 281 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: osdmap e332: 8 total, 8 up, 8 in 2024-03-30T15:01:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:01:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:37.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:37.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:37.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:37.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:37.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:38.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:38.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:38.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:38.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:38.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:38.265 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:38.265 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:38.275 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:38.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:38.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.459+0000 7fe628949700 1 -- 172.21.15.28:0/1621537749 >> v1:172.21.15.28:6789/0 conn(0x7fe624107000 legacy=0x7fe6241073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.459+0000 7fe628949700 1 -- 172.21.15.28:0/1621537749 shutdown_connections 2024-03-30T15:01:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.459+0000 7fe628949700 1 -- 172.21.15.28:0/1621537749 >> 172.21.15.28:0/1621537749 conn(0x7fe624077320 msgr2=0x7fe6240fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.459+0000 7fe628949700 1 -- 172.21.15.28:0/1621537749 shutdown_connections 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 -- 172.21.15.28:0/1621537749 wait complete. 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 librados: starting msgr at 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 librados: starting objecter 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 Processor -- start 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 -- start start 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 librados: setting wanted keys 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 librados: calling monclient init 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe62419ef80 con 0x7fe624107000 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe62419f190 con 0x7fe62410eb50 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.460+0000 7fe628949700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6241afc30 con 0x7fe62410ae80 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe62259c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe624107000 0x7fe624102f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37282/0 (socket says 172.21.15.28:37282) 2024-03-30T15:01:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe62259c700 1 -- 172.21.15.28:0/2089276909 learned_addr learned my addr 172.21.15.28:0/2089276909 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter ms_handle_connect 0x7fe624107000 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter ms_handle_connect 0x7fe62410ae80 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 982313404 0 0) 0x7fe62419ef80 con 0x7fe624107000 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6100031f0 con 0x7fe624107000 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter ms_handle_connect 0x7fe62410eb50 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783188470 0 0) 0x7fe6241afc30 con 0x7fe62410ae80 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62419ef80 con 0x7fe62410ae80 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322296903 0 0) 0x7fe62419f190 con 0x7fe62410eb50 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6241afc30 con 0x7fe62410eb50 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2080693926 0 0) 0x7fe6100031f0 con 0x7fe624107000 2024-03-30T15:01:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe62419f190 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3285092471 0 0) 0x7fe62419ef80 con 0x7fe62410ae80 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6100031f0 con 0x7fe62410ae80 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe60c0019b0 con 0x7fe62410ae80 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe614004990 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2671844379 0 0) 0x7fe6241afc30 con 0x7fe62410eb50 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe62419ef80 con 0x7fe62410eb50 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe61c0017b0 con 0x7fe62410eb50 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 639682000 0 0) 0x7fe62419f190 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 >> v1:172.21.15.28:6790/0 conn(0x7fe62410ae80 legacy=0x7fe62419dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 >> v1:172.21.15.134:6789/0 conn(0x7fe62410eb50 legacy=0x7fe62419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6241b0e70 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6241afe60 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe628949700 10 client.24567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6241b0320 con 0x7fe624107000 2024-03-30T15:01:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.461+0000 7fe628949700 1 librados: init done 2024-03-30T15:01:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.462+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe614001e90 con 0x7fe624107000 2024-03-30T15:01:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.462+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe614005220 con 0x7fe624107000 2024-03-30T15:01:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe604005120 con 0x7fe624107000 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe614005440 con 0x7fe624107000 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(334..334 src has 53..334) ==== 8618+0+0 (unknown 1033969699 0 0) 0x7fe614089be0 con 0x7fe624107000 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe61b7fe700 10 client.24567.objecter ms_dispatch 0x7fe624001550 osd_map(334..334 src has 53..334) 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe61b7fe700 3 client.24567.objecter handle_osd_map got epochs [334,334] > 0 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.463+0000 7fe61b7fe700 3 client.24567.objecter handle_osd_map decoding full epoch 334 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.464+0000 7fe61b7fe700 20 client.24567.objecter dump_active .. 0 homeless 2024-03-30T15:01:38.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.464+0000 7fe61b7fe700 10 client.24567.objecter ms_handle_connect 0x7fe6100704b0 2024-03-30T15:01:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.468+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe614056c20 con 0x7fe624107000 2024-03-30T15:01:38.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:38.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:01:38.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T15:01:38.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: pgmap v440: 280 pgs: 280 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:38.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:38 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:01:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:01:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T15:01:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: pgmap v440: 280 pgs: 280 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: osdmap e333: 8 total, 8 up, 8 in 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:38.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: pgmap v440: 280 pgs: 280 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:38.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.628 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:38 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-30T15:01:38.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:38.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:38.688+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032} v 0) -- 0x7fe604005410 con 0x7fe624107000 2024-03-30T15:01:38.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:38.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:38.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:38.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:39.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:39.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:39.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:39.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:01:39.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.288+0000 7fe61b7fe700 1 -- 172.21.15.28:0/2089276909 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]=0 set osd.3 primary-affinity to 0.131646 (886272) v335) ==== 155+0+0 (unknown 251181431 0 0) 0x7fe61405a670 con 0x7fe624107000 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 10 librados: watch_flush enter 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 10 librados: watch_flush exit 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 20 client.24567.objecter shutdown clearing up homeless session... 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 10 client.24567.objecter successfully canceled tick 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe6100704b0 legacy=0x7fe610072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 >> v1:172.21.15.28:6789/0 conn(0x7fe624107000 legacy=0x7fe624102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 shutdown_connections 2024-03-30T15:01:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.292+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 >> 172.21.15.28:0/2089276909 conn(0x7fe624077320 msgr2=0x7fe62410df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.293+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 shutdown_connections 2024-03-30T15:01:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.293+0000 7fe628949700 1 -- 172.21.15.28:0/2089276909 wait complete. 2024-03-30T15:01:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.293+0000 7fe628949700 1 librados: shutdown 2024-03-30T15:01:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 0.131646 (886272) 2024-03-30T15:01:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:39.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:39.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]: dispatch 2024-03-30T15:01:39.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[41614]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]: dispatch 2024-03-30T15:01:39.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:39 smithi028 ceph-mon[29876]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:39.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:39.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi028-93081-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-30T15:01:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: osdmap e334: 8 total, 8 up, 8 in 2024-03-30T15:01:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2113322517' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]: dispatch 2024-03-30T15:01:39.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:39 smithi134 ceph-mon[33714]: from='client.57725 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi028-92999-55"}]': finished 2024-03-30T15:01:39.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:39.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 -- 172.21.15.28:0/1655979171 >> v1:172.21.15.28:6789/0 conn(0x7f783010eaf0 legacy=0x7f7830110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 -- 172.21.15.28:0/1655979171 shutdown_connections 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 -- 172.21.15.28:0/1655979171 >> 172.21.15.28:0/1655979171 conn(0x7f78300fc9e0 msgr2=0x7f78300fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 -- 172.21.15.28:0/1655979171 shutdown_connections 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 -- 172.21.15.28:0/1655979171 wait complete. 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 librados: starting msgr at 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 librados: starting objecter 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.844+0000 7f7835ed0700 1 Processor -- start 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 -- start start 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 librados: setting wanted keys 2024-03-30T15:01:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 librados: calling monclient init 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f783010ab90 con 0x7f783010afe0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f783010ada0 con 0x7f783010eaf0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f7835ed0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78301a9640 con 0x7f78301031d0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782f7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f78301031d0 0x7f78301079e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44142/0 (socket says 172.21.15.28:44142) 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782f7fe700 1 -- 172.21.15.28:0/894198261 learned_addr learned my addr 172.21.15.28:0/894198261 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter ms_handle_connect 0x7f783010afe0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter ms_handle_connect 0x7f78301031d0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter ms_handle_connect 0x7f783010eaf0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030454077 0 0) 0x7f78301a9640 con 0x7f78301031d0 2024-03-30T15:01:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78180039c0 con 0x7f78301031d0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476609775 0 0) 0x7f783010ab90 con 0x7f783010afe0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78301a9640 con 0x7f783010afe0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 559029895 0 0) 0x7f783010ada0 con 0x7f783010eaf0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f783010ab90 con 0x7f783010eaf0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1187242141 0 0) 0x7f78180039c0 con 0x7f78301031d0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f783010ada0 con 0x7f78301031d0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.845+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78200013d0 con 0x7f78301031d0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 861257061 0 0) 0x7f78301a9640 con 0x7f783010afe0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78180039c0 con 0x7f783010afe0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7824001870 con 0x7f783010afe0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 839490505 0 0) 0x7f783010ab90 con 0x7f783010eaf0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78301a9640 con 0x7f783010eaf0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7828001d80 con 0x7f783010eaf0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4121642166 0 0) 0x7f783010ada0 con 0x7f78301031d0 2024-03-30T15:01:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.134:6789/0 conn(0x7f783010eaf0 legacy=0x7f78301a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6789/0 conn(0x7f783010afe0 legacy=0x7f78301080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78301aa880 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78301a9870 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f7835ed0700 10 client.57755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f78301a9d30 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f7835ed0700 1 librados: init done 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7820001680 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.846+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7820005cd0 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.847+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f781c005100 con 0x7f78301031d0 2024-03-30T15:01:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7820003930 con 0x7f78301031d0 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(335..335 src has 53..335) ==== 8835+0+0 (unknown 3785878979 0 0) 0x7f7820089860 con 0x7f78301031d0 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 osd_map(335..335 src has 53..335) 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 3 client.57755.objecter handle_osd_map got epochs [335,335] > 0 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 3 client.57755.objecter handle_osd_map decoding full epoch 335 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 20 client.57755.objecter dump_active .. 0 homeless 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v335) ==== 54+0+16 (unknown 1671976961 0 2420000192) 0x7f7820089ca0 con 0x7f78301031d0 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.848+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f78180704c0 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 10 client.57755.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 1 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 1 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 0 1 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 4 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 10 client.57755.objecter _send_command 1 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f781c004b50 con 0x7f781c004730 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.849+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c004730 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.850+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c004b50 con 0x7f781c004730 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.850+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.850+0000 7f782cff9700 10 client.57755.objecter _finish_command 1 = system:0 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.850+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 4 2024-03-30T15:01:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.850+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 0 1 2024-03-30T15:01:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 10 client.57755.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 2 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 2 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0053e0 osd=0 3 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 0 2 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 4 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 10 client.57755.objecter _send_command 2 2024-03-30T15:01:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.867+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c007b00 con 0x7f781c004730 2024-03-30T15:01:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.868+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c007b00 con 0x7f781c004730 2024-03-30T15:01:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.868+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.868+0000 7f782cff9700 10 client.57755.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.868+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 4 2024-03-30T15:01:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.868+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 0 2 2024-03-30T15:01:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 10 client.57755.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 3 2024-03-30T15:01:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 1 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 4 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 10 client.57755.objecter _send_command 3 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f781c002160 con 0x7f781c0042a0 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c0042a0 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c002160 con 0x7f781c0042a0 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 10 client.57755.objecter _finish_command 3 = system:0 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 4 2024-03-30T15:01:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.869+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 1 3 2024-03-30T15:01:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 10 client.57755.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 4 2024-03-30T15:01:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 4 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c004130 osd=1 3 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 1 4 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 4 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 10 client.57755.objecter _send_command 4 2024-03-30T15:01:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.890+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c001600 con 0x7f781c0042a0 2024-03-30T15:01:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c001600 con 0x7f781c0042a0 2024-03-30T15:01:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f782cff9700 10 client.57755.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 4 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 1 4 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 10 client.57755.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 5 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 5 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 2 5 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 4 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 10 client.57755.objecter _send_command 5 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.893+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f781c010fb0 con 0x7f781c008990 2024-03-30T15:01:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.894+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c008990 2024-03-30T15:01:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.895+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c010fb0 con 0x7f781c008990 2024-03-30T15:01:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.895+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.895+0000 7f782cff9700 10 client.57755.objecter _finish_command 5 = system:0 2024-03-30T15:01:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.895+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 4 2024-03-30T15:01:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.895+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 2 5 2024-03-30T15:01:39.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 10 client.57755.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 6 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 6 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c001a00 osd=2 3 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 2 6 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 4 2024-03-30T15:01:39.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 10 client.57755.objecter _send_command 6 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.913+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c008020 con 0x7f781c008990 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c008020 con 0x7f781c008990 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f782cff9700 10 client.57755.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 4 2024-03-30T15:01:39.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 2 6 2024-03-30T15:01:39.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:39.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 10 client.57755.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 7 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 7 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 3 7 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 4 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 10 client.57755.objecter _send_command 7 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.915+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f781c011ce0 con 0x7f781c011190 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c011190 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c011ce0 con 0x7f781c011190 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 10 client.57755.objecter _finish_command 7 = system:0 2024-03-30T15:01:39.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 4 2024-03-30T15:01:39.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.916+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 3 7 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 10 client.57755.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 8 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 8 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0082f0 osd=3 3 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 3 8 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 4 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 10 client.57755.objecter _send_command 8 2024-03-30T15:01:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.933+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c012690 con 0x7f781c011190 2024-03-30T15:01:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c012690 con 0x7f781c011190 2024-03-30T15:01:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f782cff9700 10 client.57755.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 4 2024-03-30T15:01:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 3 8 2024-03-30T15:01:39.937 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:39.938 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:39.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 10 client.57755.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 9 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 9 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 4 9 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 4 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 10 client.57755.objecter _send_command 9 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.935+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f781c01ea90 con 0x7f781c01af80 2024-03-30T15:01:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c01af80 2024-03-30T15:01:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c01ea90 con 0x7f781c01af80 2024-03-30T15:01:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 10 client.57755.objecter _finish_command 9 = system:0 2024-03-30T15:01:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 4 2024-03-30T15:01:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.936+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 4 9 2024-03-30T15:01:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 10 client.57755.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 10 2024-03-30T15:01:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 10 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c013810 osd=4 3 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 4 10 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 4 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 10 client.57755.objecter _send_command 10 2024-03-30T15:01:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.953+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c0128e0 con 0x7f781c01af80 2024-03-30T15:01:39.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c0128e0 con 0x7f781c01af80 2024-03-30T15:01:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 10 client.57755.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 4 2024-03-30T15:01:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 4 10 2024-03-30T15:01:39.957 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 10 client.57755.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 11 2024-03-30T15:01:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 11 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 5 11 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 4 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 10 client.57755.objecter _send_command 11 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f781c021330 con 0x7f781c013000 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.955+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c013000 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.956+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c021330 con 0x7f781c013000 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.956+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.956+0000 7f782cff9700 10 client.57755.objecter _finish_command 11 = system:0 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.956+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 4 2024-03-30T15:01:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.956+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 5 11 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 10 client.57755.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 12 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 12 2024-03-30T15:01:39.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c00c870 osd=5 3 2024-03-30T15:01:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 5 12 2024-03-30T15:01:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 4 2024-03-30T15:01:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 10 client.57755.objecter _send_command 12 2024-03-30T15:01:39.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.973+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c021330 con 0x7f781c013000 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.974+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c021330 con 0x7f781c013000 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.974+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.974+0000 7f782cff9700 10 client.57755.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.974+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 4 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.974+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 5 12 2024-03-30T15:01:39.976 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 10 client.57755.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 13 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 13 2024-03-30T15:01:39.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 6 13 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 4 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 10 client.57755.objecter _send_command 13 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f781c0275e0 con 0x7f781c023ad0 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.975+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c023ad0 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.976+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c0275e0 con 0x7f781c023ad0 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.976+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.976+0000 7f782cff9700 10 client.57755.objecter _finish_command 13 = system:0 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.976+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 4 2024-03-30T15:01:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.976+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 6 13 2024-03-30T15:01:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 10 client.57755.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 14 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 14 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c013690 osd=6 3 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 6 14 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 4 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 10 client.57755.objecter _send_command 14 2024-03-30T15:01:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.992+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c0275e0 con 0x7f781c023ad0 2024-03-30T15:01:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.994+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c0275e0 con 0x7f781c023ad0 2024-03-30T15:01:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.994+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.994+0000 7f782cff9700 10 client.57755.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.994+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 4 2024-03-30T15:01:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.994+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 6 14 2024-03-30T15:01:39.996 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 10 client.57755.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 15 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 15 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 7 15 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 4 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 10 client.57755.objecter _send_command 15 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f781c02b920 con 0x7f781c027c60 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.995+0000 7f782cff9700 10 client.57755.objecter ms_handle_connect 0x7f781c027c60 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.996+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f781c02b920 con 0x7f781c027c60 2024-03-30T15:01:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.996+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.996+0000 7f782cff9700 10 client.57755.objecter _finish_command 15 = system:0 2024-03-30T15:01:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.996+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 4 2024-03-30T15:01:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:39.996+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 7 15 2024-03-30T15:01:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 10 client.57755.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:01:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign -1 16 2024-03-30T15:01:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 20 client.57755.objecter _get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_remove -1 16 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 20 client.57755.objecter get_session s=0x7f781c0275e0 osd=7 3 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 15 client.57755.objecter _session_command_op_assign 7 16 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 4 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 10 client.57755.objecter _send_command 16 2024-03-30T15:01:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.014+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f781c012110 con 0x7f781c027c60 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782cff9700 1 -- 172.21.15.28:0/894198261 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f781c012110 con 0x7f781c027c60 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782cff9700 10 client.57755.objecter ms_dispatch 0x7f7830001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782cff9700 10 client.57755.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782cff9700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 4 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782cff9700 15 client.57755.objecter _session_command_op_remove 7 16 2024-03-30T15:01:40.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 librados: watch_flush enter 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 librados: watch_flush exit 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.0 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6801/3394323575 conn(0x7f781c004730 legacy=0x7f781c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0053e0 osd=0 1 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.1 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6805/1850313547 conn(0x7f781c0042a0 legacy=0x7f781c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c004130 osd=1 1 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.2 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6809/3451325043 conn(0x7f781c008990 legacy=0x7f781c00f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c001a00 osd=2 1 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.3 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6813/1506157250 conn(0x7f781c011190 legacy=0x7f781c0115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0082f0 osd=3 1 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.4 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.134:6800/3063775514 conn(0x7f781c01af80 legacy=0x7f781c01d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013810 osd=4 1 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.5 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.134:6804/3668151179 conn(0x7f781c013000 legacy=0x7f781c01fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c00c870 osd=5 1 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.6 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.134:6808/2135224194 conn(0x7f781c023ad0 legacy=0x7f781c025ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c013690 osd=6 1 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter close_session for osd.7 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.134:6812/3758190298 conn(0x7f781c027c60 legacy=0x7f781c02a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter put_session s=0x7f781c0275e0 osd=7 1 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 20 client.57755.objecter shutdown clearing up homeless session... 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 10 client.57755.objecter successfully canceled tick 2024-03-30T15:01:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782ffff700 1 -- 172.21.15.28:0/894198261 reap_dead start 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782ffff700 1 -- 172.21.15.28:0/894198261 reap_dead start 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782ffff700 1 -- 172.21.15.28:0/894198261 reap_dead start 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782ffff700 1 -- 172.21.15.28:0/894198261 reap_dead start 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f782ffff700 1 -- 172.21.15.28:0/894198261 reap_dead start 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6800/3011431093 conn(0x7f78180704c0 legacy=0x7f7818072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.016+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> v1:172.21.15.28:6790/0 conn(0x7f78301031d0 legacy=0x7f78301079e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.017+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 shutdown_connections 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.017+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 >> 172.21.15.28:0/894198261 conn(0x7f78300fc9e0 msgr2=0x7f7830110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.017+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 shutdown_connections 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.017+0000 7f7835ed0700 1 -- 172.21.15.28:0/894198261 wait complete. 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:40.017+0000 7f7835ed0700 1 librados: shutdown 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:40.021 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:40.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:40.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:40.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:40.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:40.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:40.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:40.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:40.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:40.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:40.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:40.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]': finished 2024-03-30T15:01:40.563 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]': finished 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/894198261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[41614]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]': finished 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]': finished 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:40.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/894198261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:40.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:40 smithi028 ceph-mon[29876]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:40.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:40.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi028-93081-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-41"}]': finished 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2089276909' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.13164586456244032}]': finished 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: osdmap e335: 8 total, 8 up, 8 in 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/894198261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-mon[33714]: pgmap v443: 260 pgs: 260 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:40.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:40.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:40.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:40.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:40.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:40.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:40.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:41.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:41.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:41.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:41.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:41.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:41.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:41.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:41.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:41.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:41.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T15:01:41.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T15:01:41.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:41 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi028-92999-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: osdmap e336: 8 total, 8 up, 8 in 2024-03-30T15:01:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:41 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:41.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:41.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:41.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:41.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:41.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:41.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:41.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:42.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:42.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:42.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:42.287 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSe expected=9 2024-03-30T15:01:42.287 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:de5d7c5f:::12:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:de5d7c5f:::12:head expected=17:de5d7c5f:::12:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:de5d7c5f:::12:head -> 12 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=12 expected=12 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:d83876eb:::4:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:d83876eb:::4:head expected=17:d83876eb:::4:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:d83876eb:::4:head -> 4 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=4 expected=4 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:cfc208b3:::3:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:cfc208b3:::3:head expected=17:cfc208b3:::3:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:cfc208b3:::3:head -> 3 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=3 expected=3 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:566253c9:::13:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:566253c9:::13:head expected=17:566253c9:::13:head 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:566253c9:::13:head -> 13 2024-03-30T15:01:42.288 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=13 expected=13 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:863748b0:::15:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:863748b0:::15:head expected=17:863748b0:::15:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:863748b0:::15:head -> 15 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=15 expected=15 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:6cac518f:::0:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:6cac518f:::0:head expected=17:6cac518f:::0:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:6cac518f:::0:head -> 0 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=0 expected=0 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:62a1935d:::14:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:62a1935d:::14:head expected=17:62a1935d:::14:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:62a1935d:::14:head -> 14 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=14 expected=14 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : seek to 17:5c6b0b28:::7:head 2024-03-30T15:01:42.289 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : cursor()=17:5c6b0b28:::7:head expected=17:5c6b0b28:::7:head 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: > 17:5c6b0b28:::7:head -> 7 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: : entry=7 expected=7 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (252 ms) 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (188657 ms) 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: split 0/5 -> MIN 17:33333333::::head 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: split 1/5 -> 17:33333333::::head 17:66666666::::head 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: split 2/5 -> 17:66666666::::head 17:99999999::::head 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: split 3/5 -> 17:99999999::::head 17:cccccccc::::head 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: split 4/5 -> 17:cccccccc::::head MAX 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (31873 ms) 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 7 tests from LibRadosList (222360 ms total) 2024-03-30T15:01:42.290 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (944 ms) 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo1 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo2 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo3 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo4 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo5 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo1 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset foo6,foo7 2024-03-30T15:01:42.291 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo7 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: foo6 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo4 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo5 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns2:foo7 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns2:foo6 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: ns1:foo1 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo1 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo2 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: :foo3 2024-03-30T15:01:42.292 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (31 ms) 2024-03-30T15:01:42.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:42.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:42.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:42.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:42.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: pgmap v446: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: pgmap v446: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:42.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:42.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:42.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T15:01:42.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi028-93081-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: osdmap e337: 8 total, 8 up, 8 in 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: pgmap v446: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi028-92999-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: osdmap e338: 8 total, 8 up, 8 in 2024-03-30T15:01:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:42.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:42.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:42.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:42.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:42.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:43.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:43.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:43.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:43.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:43.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:01:43.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:43.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:43.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:43.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:43.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:43.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:43.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:43.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:44.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:44.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:44.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:44.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.315+0000 7fdd28cc8700 1 -- 172.21.15.28:0/2436733456 >> v1:172.21.15.28:6790/0 conn(0x7fdd24102260 legacy=0x7fdd24102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.315+0000 7fdd28cc8700 1 -- 172.21.15.28:0/2436733456 shutdown_connections 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.315+0000 7fdd28cc8700 1 -- 172.21.15.28:0/2436733456 >> 172.21.15.28:0/2436733456 conn(0x7fdd240fdb10 msgr2=0x7fdd240fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.315+0000 7fdd28cc8700 1 -- 172.21.15.28:0/2436733456 shutdown_connections 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.315+0000 7fdd28cc8700 1 -- 172.21.15.28:0/2436733456 wait complete. 2024-03-30T15:01:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 librados: starting msgr at 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 librados: starting objecter 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 Processor -- start 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 -- start start 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 librados: setting wanted keys 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 librados: calling monclient init 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd241a7140 con 0x7fdd24102260 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd241a8360 con 0x7fdd2410ea90 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd28cc8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd241a95a0 con 0x7fdd2410aef0 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd21d9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdd2410aef0 0x7fdd241a1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44148/0 (socket says 172.21.15.28:44148) 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd21d9b700 1 -- 172.21.15.28:0/1841426120 learned_addr learned my addr 172.21.15.28:0/1841426120 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd2259c700 1 --1- 172.21.15.28:0/1841426120 >> v1:172.21.15.28:6789/0 conn(0x7fdd24102260 0x7fdd241075d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38950/0 (socket says 172.21.15.28:38950) 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd24102260 2024-03-30T15:01:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd2410ea90 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.316+0000 7fdd1b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3619082995 0 0) 0x7fdd241a95a0 con 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd10003aa0 con 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247813870 0 0) 0x7fdd241a8360 con 0x7fdd2410ea90 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd241a95a0 con 0x7fdd2410ea90 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411744198 0 0) 0x7fdd241a7140 con 0x7fdd24102260 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd241a8360 con 0x7fdd24102260 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4286898057 0 0) 0x7fdd10003aa0 con 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd241a7140 con 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2937557562 0 0) 0x7fdd241a95a0 con 0x7fdd2410ea90 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd10003aa0 con 0x7fdd2410ea90 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd0c0018f0 con 0x7fdd2410aef0 2024-03-30T15:01:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd140010f0 con 0x7fdd2410ea90 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 74789524 0 0) 0x7fdd241a8360 con 0x7fdd24102260 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd241a95a0 con 0x7fdd24102260 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd1c004980 con 0x7fdd24102260 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3182686285 0 0) 0x7fdd241a7140 con 0x7fdd2410aef0 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 >> v1:172.21.15.134:6789/0 conn(0x7fdd2410ea90 legacy=0x7fdd241a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 >> v1:172.21.15.28:6789/0 conn(0x7fdd24102260 legacy=0x7fdd241075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd241aa7e0 con 0x7fdd2410aef0 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd241a9770 con 0x7fdd2410aef0 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd28cc8700 10 client.57761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd241a9c30 con 0x7fdd2410aef0 2024-03-30T15:01:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd28cc8700 1 librados: init done 2024-03-30T15:01:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdd0c001cc0 con 0x7fdd2410aef0 2024-03-30T15:01:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.317+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd0c005d00 con 0x7fdd2410aef0 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.318+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd04005150 con 0x7fdd2410aef0 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdd0c003950 con 0x7fdd2410aef0 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(340..340 src has 53..340) ==== 9059+0+0 (unknown 2158629113 0 0) 0x7fdd0c089970 con 0x7fdd2410aef0 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 10 client.57761.objecter ms_dispatch 0x7fdd24001550 osd_map(340..340 src has 53..340) 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 3 client.57761.objecter handle_osd_map got epochs [340,340] > 0 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 3 client.57761.objecter handle_osd_map decoding full epoch 340 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 20 client.57761.objecter dump_active .. 0 homeless 2024-03-30T15:01:44.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.319+0000 7fdd1b7fe700 10 client.57761.objecter ms_handle_connect 0x7fdd100703b0 2024-03-30T15:01:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.324+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdd0c056800 con 0x7fdd2410aef0 2024-03-30T15:01:44.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:44.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:44.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:44.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:44.540+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fdd04005440 con 0x7fdd2410aef0 2024-03-30T15:01:44.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:44.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: osdmap e339: 8 total, 8 up, 8 in 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-6","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:44.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:44.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:44.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:44.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:44.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:44.970 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:01:44.970 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:01:44.970 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:01:44.971 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:01:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:44.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:45.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:45.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:45.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.236+0000 7fdd1b7fe700 1 -- 172.21.15.28:0/1841426120 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v341) ==== 91+0+0 (unknown 693560770 0 0) 0x7fdd0c05a250 con 0x7fdd2410aef0 2024-03-30T15:01:45.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 10 librados: watch_flush enter 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 10 librados: watch_flush exit 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 20 client.57761.objecter shutdown clearing up homeless session... 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 10 client.57761.objecter successfully canceled tick 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdd100703b0 legacy=0x7fdd10072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 >> v1:172.21.15.28:6790/0 conn(0x7fdd2410aef0 legacy=0x7fdd241a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 shutdown_connections 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.240+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 >> 172.21.15.28:0/1841426120 conn(0x7fdd240fdb10 msgr2=0x7fdd24111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.241+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 shutdown_connections 2024-03-30T15:01:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.241+0000 7fdd28cc8700 1 -- 172.21.15.28:0/1841426120 wait complete. 2024-03-30T15:01:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:45.241+0000 7fdd28cc8700 1 librados: shutdown 2024-03-30T15:01:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:01:45.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:45.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:45.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:45.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:45.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:01:45.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:01:45.594 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T15:01:45.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:01:45.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: pgmap v449: 268 pgs: 268 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1841426120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[29876]: from='client.57761 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: pgmap v449: 268 pgs: 268 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.623 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1841426120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:45 smithi028 ceph-mon[41614]: from='client.57761 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:45.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: pgmap v449: 268 pgs: 268 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: osdmap e340: 8 total, 8 up, 8 in 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1885953997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1841426120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:45 smithi134 ceph-mon[33714]: from='client.57761 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:01:45.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:45.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:01:45.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:45.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:45.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:46.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:46.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:46.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:46.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:46.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:46.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:46.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.57761 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: 101.0 scrub starts 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: 101.0 scrub ok 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:46.535 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.57761 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: 101.0 scrub starts 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: 101.0 scrub ok 2024-03-30T15:01:46.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:46 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:46.594 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:46.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:46.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:01:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_count","val": "8"}]': finished 2024-03-30T15:01:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.49756 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi028-93081-2"}]': finished 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.57761 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: osdmap e341: 8 total, 8 up, 8 in 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3544512235' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]: dispatch 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: 101.0 scrub starts 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: 101.0 scrub ok 2024-03-30T15:01:46.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:46 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:01:46.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:46.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:46.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:46.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:47.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:47.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:47.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:47.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:47.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:47.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:47.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:47.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:47.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T15:01:47.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.49786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: 101.3 scrub starts 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: 101.3 scrub ok 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.49786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T15:01:47.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.49786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: 101.3 scrub starts 2024-03-30T15:01:47.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: 101.3 scrub ok 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.49786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T15:01:47.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.492 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosListEC.LtRead 2024-03-30T15:01:47.492 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T15:01:47.492 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: ok, hit_set contains 271:602f83fe:::foo:head 2024-03-30T15:01:47.492 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10316 ms) 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg_num = 32 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 0 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 1 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 2 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 3 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 4 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 5 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 6 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 7 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 8 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 9 ls 1711810908,0 2024-03-30T15:01:47.493 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 10 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 11 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 12 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 13 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 14 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 15 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 16 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 17 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 18 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 19 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 20 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 21 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 22 ls 1711810908,0 2024-03-30T15:01:47.494 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 23 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 24 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 25 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 26 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 27 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 28 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 29 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 30 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg 31 ls 1711810908,0 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: pg_num = 32 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6cac518f:::0:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:02547ec2:::1:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f905c69b:::2:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:cfc208b3:::3:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d83876eb:::4:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b29083e3:::5:head 2024-03-30T15:01:47.495 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c4fdafeb:::6:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:5c6b0b28:::7:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:bd63b0f1:::8:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e960b815:::9:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:52ea6a34:::10:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:89d3ae78:::11:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:de5d7c5f:::12:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:566253c9:::13:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:62a1935d:::14:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:863748b0:::15:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3958e169:::16:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4d4dabf9:::17:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:8391935d:::18:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:28883081:::19:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:69259c59:::20:head 2024-03-30T15:01:47.496 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4bdb80b7:::21:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a11c5d71:::22:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:271af37b:::23:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:95b121be:::24:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:58d1031b:::25:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:0a050783:::26:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c709704c:::27:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:cbe56eaf:::28:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:86b4b162:::29:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:70d89383:::30:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:dd450c7c:::31:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6d5729b1:::32:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c388f3fb:::33:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:56cfea31:::34:head 2024-03-30T15:01:47.497 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9dbc1bf7:::35:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:40b74ccd:::36:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4d5aaf42:::37:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:920f362c:::38:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6cc53222:::39:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9cad833f:::40:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:1ea84d41:::41:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c4480ef6:::42:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a694361e:::43:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d1bd33e9:::44:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ddc2cd5d:::45:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:2b782207:::46:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7b187fca:::47:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:90ecdf6f:::48:head 2024-03-30T15:01:47.498 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a5ed95fe:::49:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ea0eaa55:::50:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f33ef17b:::51:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a0d1b2f6:::52:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:60c5229e:::53:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:edcbc575:::54:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:102cf253:::55:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:efb7fb0b:::56:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:50d0a326:::57:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d4dc5daf:::58:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3a130462:::59:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ec87ed71:::60:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d5bc9454:::61:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3ddfe313:::62:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7c2816b9:::63:head 2024-03-30T15:01:47.499 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:47e00e4d:::64:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c6410c18:::65:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b48ed237:::66:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:cd63ad31:::67:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b179e92b:::68:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:0d9f741a:::69:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6d3352ae:::70:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c6d5c19e:::71:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:bc4729c3:::72:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:77e930b9:::73:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:0abeecfd:::74:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b7c37e15:::75:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b6378398:::76:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:02bd68de:::77:head 2024-03-30T15:01:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:cc795d2d:::78:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:630d4fea:::79:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e0d29ef5:::80:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:fd6f13d2:::81:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:606461d5:::82:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:eadbdc43:::83:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:8761d0bb:::84:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9ef0186f:::85:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e0d41294:::86:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:961de695:::87:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:1423148f:::88:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:633a8fa2:::89:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a8653809:::90:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3dac8b33:::91:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:35aad435:::92:head 2024-03-30T15:01:47.501 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f6dcc343:::93:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:dbbdad87:::94:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:1cb48ce0:::95:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:03cd461c:::96:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:17a4ea99:::97:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9993c9a7:::98:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6394211c:::99:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:94c7ae57:::100:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6fdee5bb:::101:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9a477fd1:::102:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:eb850916:::103:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:affc56b9:::104:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b42dc814:::105:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f319f8f0:::106:head 2024-03-30T15:01:47.502 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9a40b9de:::107:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:8b524f28:::108:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e3de589f:::109:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:90f90a5b:::110:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a7b4f1d7:::111:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:af51766e:::112:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b6f90bd1:::113:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e0261208:::114:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c9569ef7:::115:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:61bebe50:::116:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:fe93412b:::117:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d3d38bee:::118:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3100ba0c:::119:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d0560ada:::120:head 2024-03-30T15:01:47.503 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f0ea8b35:::121:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:766f231a:::122:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a07a2582:::123:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:bd7c6b3a:::124:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:fb2ddaff:::125:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4408e1fe:::126:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ee1df7a7:::127:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c3002909:::128:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4f48ffa9:::129:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:edf38733:::130:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c08425c0:::131:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:5f902d98:::132:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:41ea2c93:::133:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:813cee13:::134:head 2024-03-30T15:01:47.504 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:0131818d:::135:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:26ba5a85:::136:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:381b8a5a:::137:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:28797e47:::138:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:bfca7f22:::139:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:36807075:::140:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:80b03975:::141:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:5c15709b:::142:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f39ea15e:::143:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ea992956:::144:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:48887b1c:::145:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:9f24a9dd:::146:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:987f100b:::147:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d2dd3581:::148:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7fed1808:::149:head 2024-03-30T15:01:47.505 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c80b70e9:::150:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:85ed90f9:::151:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:36428b24:::152:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d044c34a:::153:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7c18bf58:::154:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d1c21232:::155:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a7a3c575:::156:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:87da0633:::157:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d5ac3822:::158:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3f20522d:::159:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6ca26563:::160:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:532ce135:::161:head 2024-03-30T15:01:47.506 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:c78863e6:::162:head 2024-03-30T15:01:47.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:47.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:47.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:47.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: pgmap v452: 260 pgs: 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:01:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.49768 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi028-92999-56"}]': finished 2024-03-30T15:01:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: osdmap e342: 8 total, 8 up, 8 in 2024-03-30T15:01:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.49786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: 101.3 scrub starts 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: 101.3 scrub ok 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.49786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93081-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi028-92999-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: osdmap e343: 8 total, 8 up, 8 in 2024-03-30T15:01:47.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:47.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:47.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:47.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:47.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:48.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:48.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:48.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:48.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7462ddf6::istObjectsStart 2024-03-30T15:01:48.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 1 0 2024-03-30T15:01:48.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 10 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 13 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 7 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 14 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 0 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 15 0 2024-03-30T15:01:48.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 11 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 5 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 8 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 6 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 3 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 4 0 2024-03-30T15:01:48.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 12 0 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 9 0 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 2 0 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (45 ms) 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 3 tests from LibRadosListEC (1020 ms total) 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 2024-03-30T15:01:48.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2997 ms) 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] 1 test from LibRadosListNP (2997 ms total) 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [----------] Global test environment tear-down 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [==========] 11 tests from 3 test suites ran. (243821 ms total) 2024-03-30T15:01:48.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_list: [ PASSED ] 11 tests. 2024-03-30T15:01:48.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:48.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:48.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:48.457 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:48.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:48.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:01:48.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:48.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:48.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:01:48.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.49786 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[29876]: from='client.49786 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.49786 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:48 smithi028 ceph-mon[41614]: from='client.49786 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T15:01:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.754+0000 7fb4af2a7700 1 -- 172.21.15.28:0/2489912680 >> v1:172.21.15.134:6789/0 conn(0x7fb4a810eab0 legacy=0x7fb4a8110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.754+0000 7fb4af2a7700 1 -- 172.21.15.28:0/2489912680 shutdown_connections 2024-03-30T15:01:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.754+0000 7fb4af2a7700 1 -- 172.21.15.28:0/2489912680 >> 172.21.15.28:0/2489912680 conn(0x7fb4a80fc9e0 msgr2=0x7fb4a80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.754+0000 7fb4af2a7700 1 -- 172.21.15.28:0/2489912680 shutdown_connections 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.754+0000 7fb4af2a7700 1 -- 172.21.15.28:0/2489912680 wait complete. 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 librados: starting msgr at 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 librados: starting objecter 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 Processor -- start 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 -- start start 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 librados: setting wanted keys 2024-03-30T15:01:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 librados: calling monclient init 2024-03-30T15:01:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb4a81a91e0 con 0x7fb4a810afa0 2024-03-30T15:01:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb4a81aa400 con 0x7fb4a8103190 2024-03-30T15:01:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4af2a7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb4a81ab620 con 0x7fb4a810eab0 2024-03-30T15:01:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4ad844700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb4a810eab0 0x7fb4a81a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37754/0 (socket says 172.21.15.28:37754) 2024-03-30T15:01:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb4ad844700 1 -- 172.21.15.28:0/591130816 learned_addr learned my addr 172.21.15.28:0/591130816 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb49e7fc700 10 client.?.objecter ms_handle_connect 0x7fb4a810afa0 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb49e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb49e7fc700 10 client.?.objecter ms_handle_connect 0x7fb4a810eab0 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.755+0000 7fb49e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686635110 0 0) 0x7fb4a81ab620 con 0x7fb4a810eab0 2024-03-30T15:01:48.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4940039c0 con 0x7fb4a810eab0 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 10 client.?.objecter ms_handle_connect 0x7fb4a8103190 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252489669 0 0) 0x7fb4a81a91e0 con 0x7fb4a810afa0 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4a81ab620 con 0x7fb4a810afa0 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230986227 0 0) 0x7fb4a81aa400 con 0x7fb4a8103190 2024-03-30T15:01:48.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4a81a91e0 con 0x7fb4a8103190 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1893526752 0 0) 0x7fb4940039c0 con 0x7fb4a810eab0 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4a81aa400 con 0x7fb4a810eab0 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb4a0001b90 con 0x7fb4a810eab0 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 866072380 0 0) 0x7fb4a81a91e0 con 0x7fb4a8103190 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4940039c0 con 0x7fb4a8103190 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2040615430 0 0) 0x7fb4a81aa400 con 0x7fb4a810eab0 2024-03-30T15:01:48.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.134:6789/0 conn(0x7fb4a8103190 legacy=0x7fb4a810a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6789/0 conn(0x7fb4a810afa0 legacy=0x7fb4a81a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a81ac880 con 0x7fb4a810eab0 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.756+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a81ab850 con 0x7fb4a810eab0 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.757+0000 7fb4af2a7700 10 client.57779.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.757+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4a81abd10 con 0x7fb4a810eab0 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.757+0000 7fb4af2a7700 1 librados: init done 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.757+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb4a0001d90 con 0x7fb4a810eab0 2024-03-30T15:01:48.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.757+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb4a0005130 con 0x7fb4a810eab0 2024-03-30T15:01:48.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.758+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb48c005100 con 0x7fb4a810eab0 2024-03-30T15:01:48.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.759+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb4a0005110 con 0x7fb4a810eab0 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(344..344 src has 53..344) ==== 8931+0+0 (unknown 1213054829 0 0) 0x7fb4a0089c60 con 0x7fb4a810eab0 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 osd_map(344..344 src has 53..344) 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 3 client.57779.objecter handle_osd_map got epochs [344,344] > 0 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 3 client.57779.objecter handle_osd_map decoding full epoch 344 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 20 client.57779.objecter dump_active .. 0 homeless 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb4940700b0 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v344) ==== 54+0+16 (unknown 4193465294 0 2420000192) 0x7fb4a008c090 con 0x7fb4a810eab0 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 1 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 1 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 0 1 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 4 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 10 client.57779.objecter _send_command 1 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.760+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb48c004b70 con 0x7fb48c004730 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c004730 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c004b70 con 0x7fb48c004730 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 1: 0 ) 2024-03-30T15:01:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 1 = system:0 2024-03-30T15:01:48.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 4 2024-03-30T15:01:48.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.761+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 0 1 2024-03-30T15:01:48.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 2 2024-03-30T15:01:48.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 2 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c0053e0 osd=0 3 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 0 2 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 4 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 10 client.57779.objecter _send_command 2 2024-03-30T15:01:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.779+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c0079f0 con 0x7fb48c004730 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c0079f0 con 0x7fb48c004730 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 4 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 0 2 2024-03-30T15:01:48.782 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 1 3 2024-03-30T15:01:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 4 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 10 client.57779.objecter _send_command 3 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb48c0022a0 con 0x7fb48c004310 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.781+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c004310 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.783+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c0022a0 con 0x7fb48c004310 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.783+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 3: 0 ) 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.783+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 3 = system:0 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.783+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 4 2024-03-30T15:01:48.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.783+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 1 3 2024-03-30T15:01:48.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:48.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 4 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 4 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c004130 osd=1 3 2024-03-30T15:01:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 1 4 2024-03-30T15:01:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 4 2024-03-30T15:01:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 10 client.57779.objecter _send_command 4 2024-03-30T15:01:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.800+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c001720 con 0x7fb48c004310 2024-03-30T15:01:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.802+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c001720 con 0x7fb48c004310 2024-03-30T15:01:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.802+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.802+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.802+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 4 2024-03-30T15:01:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.802+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 1 4 2024-03-30T15:01:48.805 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:01:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 5 2024-03-30T15:01:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 5 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 2 5 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 4 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 10 client.57779.objecter _send_command 5 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb48c011170 con 0x7fb48c00d5d0 2024-03-30T15:01:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.803+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c00d5d0 2024-03-30T15:01:48.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.805+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c011170 con 0x7fb48c00d5d0 2024-03-30T15:01:48.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.805+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 5: 0 ) 2024-03-30T15:01:48.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.805+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 5 = system:0 2024-03-30T15:01:48.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.805+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 4 2024-03-30T15:01:48.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.805+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 2 5 2024-03-30T15:01:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:48.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 6 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 6 2024-03-30T15:01:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c001980 osd=2 3 2024-03-30T15:01:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 2 6 2024-03-30T15:01:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 4 2024-03-30T15:01:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 10 client.57779.objecter _send_command 6 2024-03-30T15:01:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.828+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c008100 con 0x7fb48c00d5d0 2024-03-30T15:01:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.829+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c008100 con 0x7fb48c00d5d0 2024-03-30T15:01:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.829+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.829+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.829+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 4 2024-03-30T15:01:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.829+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 2 6 2024-03-30T15:01:48.831 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 7 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 7 2024-03-30T15:01:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 3 7 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 4 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 10 client.57779.objecter _send_command 7 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb48c015fb0 con 0x7fb48c0124a0 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.830+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c0124a0 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.831+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c015fb0 con 0x7fb48c0124a0 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.831+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 7: 0 ) 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.831+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 7 = system:0 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.831+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 4 2024-03-30T15:01:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.831+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 3 7 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 8 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 8 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00cc20 osd=3 3 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 3 8 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 4 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 10 client.57779.objecter _send_command 8 2024-03-30T15:01:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.848+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c00c700 con 0x7fb48c0124a0 2024-03-30T15:01:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.849+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c00c700 con 0x7fb48c0124a0 2024-03-30T15:01:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.849+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.849+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.849+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 4 2024-03-30T15:01:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.849+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 3 8 2024-03-30T15:01:48.851 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:01:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 9 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 9 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 4 9 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 4 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 10 client.57779.objecter _send_command 9 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb48c00c6e0 con 0x7fb48c012000 2024-03-30T15:01:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.850+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c012000 2024-03-30T15:01:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.851+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c00c6e0 con 0x7fb48c012000 2024-03-30T15:01:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.851+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 9: 0 ) 2024-03-30T15:01:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.851+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 9 = system:0 2024-03-30T15:01:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.851+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 4 2024-03-30T15:01:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.851+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 4 9 2024-03-30T15:01:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 10 2024-03-30T15:01:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 10 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00c4b0 osd=4 3 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 4 10 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 4 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 10 client.57779.objecter _send_command 10 2024-03-30T15:01:48.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.868+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c01c210 con 0x7fb48c012000 2024-03-30T15:01:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c01c210 con 0x7fb48c012000 2024-03-30T15:01:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 4 2024-03-30T15:01:48.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 4 10 2024-03-30T15:01:48.872 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:01:48.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 11 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 11 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 5 11 2024-03-30T15:01:48.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 4 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 10 client.57779.objecter _send_command 11 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb48c01cdc0 con 0x7fb48c01c930 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.870+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c01c930 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.871+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c01cdc0 con 0x7fb48c01c930 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.871+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 11: 0 ) 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.871+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 11 = system:0 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.871+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 4 2024-03-30T15:01:48.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.871+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 5 11 2024-03-30T15:01:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 12 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 12 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c01c210 osd=5 3 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 5 12 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 4 2024-03-30T15:01:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 10 client.57779.objecter _send_command 12 2024-03-30T15:01:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.888+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c01cdc0 con 0x7fb48c01c930 2024-03-30T15:01:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c01cdc0 con 0x7fb48c01c930 2024-03-30T15:01:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 4 2024-03-30T15:01:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 5 12 2024-03-30T15:01:48.892 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:01:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 13 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 13 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 6 13 2024-03-30T15:01:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 4 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 10 client.57779.objecter _send_command 13 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb48c01d1d0 con 0x7fb48c01f900 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.890+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c01f900 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.891+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb4a0056b50 con 0x7fb48c01f900 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.891+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 13: 0 ) 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.891+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 13 = system:0 2024-03-30T15:01:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.891+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 4 2024-03-30T15:01:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.891+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 6 13 2024-03-30T15:01:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 14 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 14 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c01cdc0 osd=6 3 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 6 14 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 4 2024-03-30T15:01:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 10 client.57779.objecter _send_command 14 2024-03-30T15:01:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.909+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c011570 con 0x7fb48c01f900 2024-03-30T15:01:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c011570 con 0x7fb48c01f900 2024-03-30T15:01:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 4 2024-03-30T15:01:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 6 14 2024-03-30T15:01:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:01:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:01:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 15 2024-03-30T15:01:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 15 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 7 15 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 4 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 10 client.57779.objecter _send_command 15 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb48c017040 con 0x7fb48c011320 2024-03-30T15:01:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.911+0000 7fb49e7fc700 10 client.57779.objecter ms_handle_connect 0x7fb48c011320 2024-03-30T15:01:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.912+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb48c017040 con 0x7fb48c011320 2024-03-30T15:01:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.912+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 15: 0 ) 2024-03-30T15:01:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.912+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 15 = system:0 2024-03-30T15:01:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.912+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 4 2024-03-30T15:01:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.912+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 7 15 2024-03-30T15:01:48.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:01:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:01:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/957214817' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.49786 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi028-93253-43","var": "pg_num","format": "json"}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]: dispatch 2024-03-30T15:01:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:48 smithi134 ceph-mon[33714]: from='client.49786 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi028-93081-3","pool2":"test-rados-api-smithi028-93081-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 10 client.57779.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign -1 16 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 20 client.57779.objecter _get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_remove -1 16 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 20 client.57779.objecter get_session s=0x7fb48c00ca70 osd=7 3 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 15 client.57779.objecter _session_command_op_assign 7 16 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 4 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 10 client.57779.objecter _send_command 16 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.929+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb48c017040 con 0x7fb48c011320 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb49e7fc700 1 -- 172.21.15.28:0/591130816 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb48c017040 con 0x7fb48c011320 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb49e7fc700 10 client.57779.objecter ms_dispatch 0x7fb4a8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:01:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb49e7fc700 10 client.57779.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:01:48.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb49e7fc700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 4 2024-03-30T15:01:48.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb49e7fc700 15 client.57779.objecter _session_command_op_remove 7 16 2024-03-30T15:01:48.935 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:01:48.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 librados: watch_flush enter 2024-03-30T15:01:48.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 librados: watch_flush exit 2024-03-30T15:01:48.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.0 2024-03-30T15:01:48.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6801/3394323575 conn(0x7fb48c004730 legacy=0x7fb48c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c0053e0 osd=0 1 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.1 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6805/1850313547 conn(0x7fb48c004310 legacy=0x7fb48c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c004130 osd=1 1 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.2 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6809/3451325043 conn(0x7fb48c00d5d0 legacy=0x7fb48c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4ad844700 1 -- 172.21.15.28:0/591130816 reap_dead start 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c001980 osd=2 1 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.3 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6813/1506157250 conn(0x7fb48c0124a0 legacy=0x7fb48c014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00cc20 osd=3 1 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.4 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.134:6800/3063775514 conn(0x7fb48c012000 legacy=0x7fb48c0192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00c4b0 osd=4 1 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.5 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.134:6804/3668151179 conn(0x7fb48c01c930 legacy=0x7fb48c01aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01c210 osd=5 1 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.6 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.134:6808/2135224194 conn(0x7fb48c01f900 legacy=0x7fb48c021cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c01cdc0 osd=6 1 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter close_session for osd.7 2024-03-30T15:01:48.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.134:6812/3758190298 conn(0x7fb48c011320 legacy=0x7fb48c016930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4ad844700 1 -- 172.21.15.28:0/591130816 reap_dead start 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter put_session s=0x7fb48c00ca70 osd=7 1 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 20 client.57779.objecter shutdown clearing up homeless session... 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 10 client.57779.objecter successfully canceled tick 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb4940700b0 legacy=0x7fb494072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.931+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> v1:172.21.15.28:6790/0 conn(0x7fb4a810eab0 legacy=0x7fb4a81a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:48.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.932+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 shutdown_connections 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.932+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 >> 172.21.15.28:0/591130816 conn(0x7fb4a80fc9e0 msgr2=0x7fb4a8110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.932+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 shutdown_connections 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.932+0000 7fb4af2a7700 1 -- 172.21.15.28:0/591130816 wait complete. 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:48.932+0000 7fb4af2a7700 1 librados: shutdown 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:01:48.942 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:01:48.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:01:48.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:01:48.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:01:48.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:01:48.943 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:01:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:48.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:48.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:49.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:49.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:49.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:49.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:49.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:49.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:49.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:49.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:49.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:49.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]': finished 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T15:01:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/591130816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[41614]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]': finished 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:01:49.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T15:01:49.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T15:01:49.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T15:01:49.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/591130816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:49.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:49.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:49 smithi028 ceph-mon[29876]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T15:01:49.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:49.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-43"}]': finished 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T15:01:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: osdmap e344: 8 total, 8 up, 8 in 2024-03-30T15:01:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/591130816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:01:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi028-92999-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:49.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:49 smithi134 ceph-mon[33714]: osdmap e345: 8 total, 8 up, 8 in 2024-03-30T15:01:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:49.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:50.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:50.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:50.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:50.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:50.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:50.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:50.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:01:50.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:50.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:50.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:50.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:50.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:50.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:50.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: 42.15 scrub starts 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: 42.15 scrub ok 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: 42.15 scrub starts 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: 42.15 scrub ok 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:50.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: pgmap v458: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:01:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:01:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: 42.15 scrub starts 2024-03-30T15:01:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: 42.15 scrub ok 2024-03-30T15:01:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: osdmap e346: 8 total, 8 up, 8 in 2024-03-30T15:01:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:01:50.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:50.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:51.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:51.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:51.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.213 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:51.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:51.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:51.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:51.436 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:01:51.437 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:01:51.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:51.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:51.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:51.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:51.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:51.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:51.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:01:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:01:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:01:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:51.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: osdmap e347: 8 total, 8 up, 8 in 2024-03-30T15:01:51.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:01:51.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:51.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:51.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:51.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:52.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:52.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:52.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:52.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:52.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:52.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:52.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:52.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:52.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:52.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:52.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: 42.b scrub starts 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: 42.b scrub ok 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: 42.19 scrub starts 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: 42.19 scrub ok 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:52.814 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: 42.b scrub starts 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: 42.b scrub ok 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: 42.19 scrub starts 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: 42.19 scrub ok 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:52.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:52.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:52.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T15:01:52.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:52.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:52.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: 42.b scrub starts 2024-03-30T15:01:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: 42.b scrub ok 2024-03-30T15:01:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: 42.19 scrub starts 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: 42.19 scrub ok 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: pgmap v461: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: osdmap e348: 8 total, 8 up, 8 in 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]: dispatch 2024-03-30T15:01:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:01:52.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:01:52.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:53.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:53.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:53.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:53.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:53.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:53.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:01:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:53.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:53.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:53.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:53.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 12.1c scrub starts 2024-03-30T15:01:53.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:53.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:53.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:53.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:53.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 12.1c scrub starts 2024-03-30T15:01:53.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 12.1c scrub ok 2024-03-30T15:01:53.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:01:53.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 12.1c scrub ok 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T15:01:53.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:53.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:53 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:53.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:53.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:53.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 12.1c scrub starts 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 12.1c scrub ok 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T15:01:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/330470844' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi028-92999-57"}]': finished 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: osdmap e349: 8 total, 8 up, 8 in 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:53 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:01:54.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:54.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:54.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:54.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:54.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:54.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:54.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:54.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:54.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:54.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: pgmap v464: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:54.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3507727426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:01:54.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: pgmap v464: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3507727426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:01:54.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:54.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:54.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:01:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:01:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:01:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: pgmap v464: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi028-92999-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: osdmap e350: 8 total, 8 up, 8 in 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:01:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3507727426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:01:54.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:54.960 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:54.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:55.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:55.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:55.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:55.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:55.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:55.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 -- 172.21.15.28:0/1415515633 >> v1:172.21.15.28:6789/0 conn(0x7f6dd810d950 legacy=0x7f6dd810fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 -- 172.21.15.28:0/1415515633 shutdown_connections 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 -- 172.21.15.28:0/1415515633 >> 172.21.15.28:0/1415515633 conn(0x7f6dd8077370 msgr2=0x7f6dd80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 -- 172.21.15.28:0/1415515633 shutdown_connections 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 -- 172.21.15.28:0/1415515633 wait complete. 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 librados: starting msgr at 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.051+0000 7f6ddd10b700 1 librados: starting objecter 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 Processor -- start 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 -- start start 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 librados: setting wanted keys 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 librados: calling monclient init 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd81a91d0 con 0x7f6dd810d950 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd81aa3f0 con 0x7f6dd8105f00 2024-03-30T15:01:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6ddd10b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dd81ab630 con 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dd759e700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6dd810d950 0x7f6dd81a78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39890/0 (socket says 172.21.15.28:39890) 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dd759e700 1 -- 172.21.15.28:0/3507727426 learned_addr learned my addr 172.21.15.28:0/3507727426 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dbffff700 10 client.?.objecter ms_handle_connect 0x7f6dd810d950 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dbffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dbffff700 10 client.?.objecter ms_handle_connect 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.052+0000 7f6dbffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998952717 0 0) 0x7f6dd81ab630 con 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dc00032d0 con 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067449791 0 0) 0x7f6dd81a91d0 con 0x7f6dd810d950 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd81ab630 con 0x7f6dd810d950 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 10 client.?.objecter ms_handle_connect 0x7f6dd8105f00 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2795552534 0 0) 0x7f6dc00032d0 con 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dd81a91d0 con 0x7f6dd8109d80 2024-03-30T15:01:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6dcc0017b0 con 0x7f6dd8109d80 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1111817685 0 0) 0x7f6dd81ab630 con 0x7f6dd810d950 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dc00032d0 con 0x7f6dd810d950 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6dd0001250 con 0x7f6dd810d950 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 285856275 0 0) 0x7f6dd81aa3f0 con 0x7f6dd8105f00 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd81ab630 con 0x7f6dd8105f00 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1996894993 0 0) 0x7f6dd81a91d0 con 0x7f6dd8109d80 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 >> v1:172.21.15.134:6789/0 conn(0x7f6dd8105f00 legacy=0x7f6dd810cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 >> v1:172.21.15.28:6789/0 conn(0x7f6dd810d950 legacy=0x7f6dd81a78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd81ac870 con 0x7f6dd8109d80 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81ab860 con 0x7f6dd8109d80 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6ddd10b700 10 client.57791.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6dd81abda0 con 0x7f6dd8109d80 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6ddd10b700 1 librados: init done 2024-03-30T15:01:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6dcc003b20 con 0x7f6dd8109d80 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 350, 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:01:54.230428+0000", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:01:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 96, 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:01:55.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 278, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:01:55.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "321", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.334 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:01:55.335 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:01:55.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:01:55.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.338 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.339 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.340 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.342 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.343 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.346 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.348 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.349 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:01:55.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.353 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.354 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:01:55.355 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.356 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.053+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6dcc004a60 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.055+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc4005120 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.055+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6dcc01b2c0 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(349..349 src has 53..349) ==== 8515+0+0 (unknown 595511511 0 0) 0x7f6dcc088840 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 10 client.57791.objecter ms_dispatch 0x7f6dd8001550 osd_map(349..349 src has 53..349) 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 3 client.57791.objecter handle_osd_map got epochs [349,349] > 0 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 3 client.57791.objecter handle_osd_map decoding full epoch 349 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 20 client.57791.objecter dump_active .. 0 homeless 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.056+0000 7f6dbffff700 10 client.57791.objecter ms_handle_connect 0x7f6dc0078e60 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.061+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6dcc0558f0 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.277+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f6dc4005410 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.281+0000 7f6dbffff700 1 -- 172.21.15.28:0/3507727426 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v350) ==== 81+0+56398 (unknown 3170250307 0 2218504970) 0x7f6dcc059340 con 0x7f6dd8109d80 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 10 librados: watch_flush enter 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 10 librados: watch_flush exit 2024-03-30T15:01:55.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 20 client.57791.objecter shutdown clearing up homeless session... 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 10 client.57791.objecter successfully canceled tick 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6dc0078e60 legacy=0x7f6dc007b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 >> v1:172.21.15.28:6790/0 conn(0x7f6dd8109d80 legacy=0x7f6dd81a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 shutdown_connections 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.283+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 >> 172.21.15.28:0/3507727426 conn(0x7f6dd8077370 msgr2=0x7f6dd8101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.284+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 shutdown_connections 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.284+0000 7f6ddd10b700 1 -- 172.21.15.28:0/3507727426 wait complete. 2024-03-30T15:01:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:54.284+0000 7f6ddd10b700 1 librados: shutdown 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.362 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.363 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.365 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.367 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:01:55.368 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.369 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.371 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.372 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.373 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.375 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.377 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.378 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:01:55.379 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:01:55.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "348", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "325", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "325", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "325", 2024-03-30T15:01:55.382 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 14, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 348, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [ 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: 278 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.383 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:01:55.384 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.385 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 278, 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-45", 2024-03-30T15:01:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:01:49.503161+0000", 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:01:55.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "350", 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:01:55.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 14, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 348, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": 101, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:01:55.389 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 3, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 3, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:01:55.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [ 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: 1000000, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: 1000000, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: 1000000 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:01:55.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:01:55.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:01:55.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.397 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.400 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.401 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.404 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 342, 2024-03-30T15:01:55.405 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:01:55.408 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.409 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:01:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:01:55.412 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.413 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.414 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:01:55.415 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:01:55.416 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.417 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:01:55.421 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 346, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.422 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.423 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:01:55.424 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:01:55.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:39.892668+0000", 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:00:41.734628+0000", 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:15.875584+0000", 2024-03-30T15:01:55.427 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:39.649427+0000", 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.428 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:39.307801+0000", 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.429 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:05.898839+0000", 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:08.790186+0000", 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:01:55.430 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:01:47.572019+0000", 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:01:55.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1b", 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:01:55.433 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:01:55.434 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "testprofile-FlushAsyncPP_smithi028-92999-58": { 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-device-class": "", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:01:55.435 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-num-failure-domains": "0", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-osds-per-failure-domain": "0", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-root": "default", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van", 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "w": "8" 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.441 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:01:55.442 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.444 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:01:55.446 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.447 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.448 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:01:55.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:01:55.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:01:55.452 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:01:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:55.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:55.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:55.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:55.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:55.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:55.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:55.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: 42.4 scrub starts 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: 42.4 scrub ok 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T15:01:55.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: 42.4 scrub starts 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: 42.4 scrub ok 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T15:01:55.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:55 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:01:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:01:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: 42.4 scrub starts 2024-03-30T15:01:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: 42.4 scrub ok 2024-03-30T15:01:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:01:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: osdmap e351: 8 total, 8 up, 8 in 2024-03-30T15:01:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:55 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:01:55.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:55.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:55.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:56.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:56.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:56.174 INFO:tasks.thrashosds.thrasher:j is {'epoch': 350, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:01:54.230428+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 96, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 278, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '321', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '348', 'last_force_op_resend': '325', 'last_force_op_resend_prenautilus': '325', 'last_force_op_resend_preluminous': '325', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 348, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [278], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 278, 'pool_name': 'test-rados-api-smithi028-93253-45', 'create_time': '2024-03-30T15:01:49.503161+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '350', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 348, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 101, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 3, 'hit_set_count': 3, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 342, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 346, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:39.892668+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:00:41.734628+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:15.875584+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:39.649427+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:39.307801+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:05.898839+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:08.790186+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:01:47.572019+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-FlushAsyncPP_smithi028-92999-58': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:01:56.174 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1b 2024-03-30T15:01:56.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 4.1b 2024-03-30T15:01:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:56.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:56.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:56.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:56.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:56.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:56.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:56.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:56.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:56.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:56.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:56.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:56.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:56.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:57.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:57.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:57.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:57.252 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:57.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:57.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: 12.e scrub starts 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: 12.e scrub ok 2024-03-30T15:01:57.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: 12.e scrub starts 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: 12.e scrub ok 2024-03-30T15:01:57.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:57 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:57.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:57.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:57.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:01:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:01:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: pgmap v467: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:01:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi028-92999-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:01:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:01:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: osdmap e352: 8 total, 8 up, 8 in 2024-03-30T15:01:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: 12.e scrub starts 2024-03-30T15:01:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: 12.e scrub ok 2024-03-30T15:01:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:57 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:01:57.681 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 -- 172.21.15.28:0/2601855636 >> v1:172.21.15.28:6790/0 conn(0x7fda5c109dc0 legacy=0x7fda5c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 -- 172.21.15.28:0/2601855636 shutdown_connections 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 -- 172.21.15.28:0/2601855636 >> 172.21.15.28:0/2601855636 conn(0x7fda5c0fc9e0 msgr2=0x7fda5c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 -- 172.21.15.28:0/2601855636 shutdown_connections 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 -- 172.21.15.28:0/2601855636 wait complete. 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 librados: starting msgr at 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.679+0000 7fda6293e700 1 librados: starting objecter 2024-03-30T15:01:57.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 Processor -- start 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 -- start start 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 librados: setting wanted keys 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 librados: calling monclient init 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda5c1a7530 con 0x7fda5c10d960 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda5c1a8710 con 0x7fda5c101130 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda6293e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda5c1a9930 con 0x7fda5c109dc0 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda60edb700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fda5c10d960 0x7fda5c1a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39912/0 (socket says 172.21.15.28:39912) 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda60edb700 1 -- 172.21.15.28:0/3411685534 learned_addr learned my addr 172.21.15.28:0/3411685534 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda5b7fe700 1 --1- 172.21.15.28:0/3411685534 >> v1:172.21.15.28:6790/0 conn(0x7fda5c109dc0 0x7fda5c105090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37828/0 (socket says 172.21.15.28:37828) 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter ms_handle_connect 0x7fda5c109dc0 2024-03-30T15:01:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter ms_handle_connect 0x7fda5c10d960 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter ms_handle_connect 0x7fda5c101130 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.680+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387442653 0 0) 0x7fda5c1a7530 con 0x7fda5c10d960 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda440031f0 con 0x7fda5c10d960 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749355977 0 0) 0x7fda5c1a9930 con 0x7fda5c109dc0 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda5c1a7530 con 0x7fda5c109dc0 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101233219 0 0) 0x7fda5c1a8710 con 0x7fda5c101130 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda5c1a9930 con 0x7fda5c101130 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 5550327 0 0) 0x7fda5c1a7530 con 0x7fda5c109dc0 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda5c1a8710 con 0x7fda5c109dc0 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1172030148 0 0) 0x7fda440031f0 con 0x7fda5c10d960 2024-03-30T15:01:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda5c1a7530 con 0x7fda5c10d960 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fda50001860 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fda54001bf0 con 0x7fda5c10d960 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3326295022 0 0) 0x7fda5c1a8710 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 >> v1:172.21.15.134:6789/0 conn(0x7fda5c101130 legacy=0x7fda5c104980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 >> v1:172.21.15.28:6789/0 conn(0x7fda5c10d960 legacy=0x7fda5c1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c1aab90 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c1a7760 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda6293e700 10 client.57797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fda5c1a7c20 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda6293e700 1 librados: init done 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fda50004660 con 0x7fda5c109dc0 2024-03-30T15:01:57.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.681+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fda50005100 con 0x7fda5c109dc0 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda48005120 con 0x7fda5c109dc0 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fda50004960 con 0x7fda5c109dc0 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(353..353 src has 53..353) ==== 9388+0+0 (unknown 2995248709 0 0) 0x7fda50089d30 con 0x7fda5c109dc0 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda597fa700 10 client.57797.objecter ms_dispatch 0x7fda5c001550 osd_map(353..353 src has 53..353) 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda597fa700 3 client.57797.objecter handle_osd_map got epochs [353,353] > 0 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.683+0000 7fda597fa700 3 client.57797.objecter handle_osd_map decoding full epoch 353 2024-03-30T15:01:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.684+0000 7fda597fa700 20 client.57797.objecter dump_active .. 0 homeless 2024-03-30T15:01:57.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.684+0000 7fda597fa700 10 client.57797.objecter ms_handle_connect 0x7fda44070500 2024-03-30T15:01:57.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.690+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fda50056a50 con 0x7fda5c109dc0 2024-03-30T15:01:57.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:57.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:57.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:57.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:57.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:57.913+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fda48005410 con 0x7fda5c109dc0 2024-03-30T15:01:57.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:58.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:58.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:01:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:58.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:01:58.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.257+0000 7fda597fa700 1 -- 172.21.15.28:0/3411685534 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v354) ==== 101+0+0 (unknown 3433311493 0 0) 0x7fda5005a4a0 con 0x7fda5c109dc0 2024-03-30T15:01:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 10 librados: watch_flush enter 2024-03-30T15:01:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 10 librados: watch_flush exit 2024-03-30T15:01:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 20 client.57797.objecter shutdown clearing up homeless session... 2024-03-30T15:01:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 10 client.57797.objecter successfully canceled tick 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 >> v1:172.21.15.28:6800/3011431093 conn(0x7fda44070500 legacy=0x7fda440729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.259+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 >> v1:172.21.15.28:6790/0 conn(0x7fda5c109dc0 legacy=0x7fda5c105090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.260+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 shutdown_connections 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.260+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 >> 172.21.15.28:0/3411685534 conn(0x7fda5c0fc9e0 msgr2=0x7fda5c06a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.260+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 shutdown_connections 2024-03-30T15:01:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.260+0000 7fda6293e700 1 -- 172.21.15.28:0/3411685534 wait complete. 2024-03-30T15:01:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:01:58.260+0000 7fda6293e700 1 librados: shutdown 2024-03-30T15:01:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:01:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:58.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:01:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:58.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:01:58.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 42.e scrub starts 2024-03-30T15:01:58.463 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 42.e scrub ok 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 42.11 scrub starts 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: 42.11 scrub ok 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3411685534' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.464 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[41614]: from='client.57797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 42.e scrub starts 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 42.e scrub ok 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 42.11 scrub starts 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: 42.11 scrub ok 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3411685534' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-mon[29876]: from='client.57797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.465 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:01:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:01:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:01:58.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:58.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:58.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:58.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 42.e scrub starts 2024-03-30T15:01:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 42.e scrub ok 2024-03-30T15:01:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:01:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: osdmap e353: 8 total, 8 up, 8 in 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 42.11 scrub starts 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: 42.11 scrub ok 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3411685534' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:58 smithi134 ceph-mon[33714]: from='client.57797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:01:58.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:01:58.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:01:58.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:01:58.911 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:58.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:01:59.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:59.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:59.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:59.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:59.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:01:59.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.13 scrub starts 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.13 scrub ok 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: pgmap v470: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.f scrub starts 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.f scrub ok 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.9 scrub starts 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 42.9 scrub ok 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:59.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: from='client.57797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 12.3 scrub starts 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[29876]: 12.3 scrub ok 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.13 scrub starts 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.13 scrub ok 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: pgmap v470: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.f scrub starts 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.f scrub ok 2024-03-30T15:01:59.522 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.9 scrub starts 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 42.9 scrub ok 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: from='client.57797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 12.3 scrub starts 2024-03-30T15:01:59.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:01:59 smithi028 ceph-mon[41614]: 12.3 scrub ok 2024-03-30T15:01:59.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:01:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:01:59.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:01:59.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:01:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.13 scrub starts 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.13 scrub ok 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: pgmap v470: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.f scrub starts 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.f scrub ok 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.9 scrub starts 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 42.9 scrub ok 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: from='client.57797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:01:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: osdmap e354: 8 total, 8 up, 8 in 2024-03-30T15:01:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:01:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 12.3 scrub starts 2024-03-30T15:01:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:01:59 smithi134 ceph-mon[33714]: 12.3 scrub ok 2024-03-30T15:01:59.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:01:59.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:01:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:01:59.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:01:59.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:00.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:00.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:00.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:00.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:00.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:00.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:00.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:00.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 42.14 scrub starts 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 42.14 scrub ok 2024-03-30T15:02:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 42.12 scrub starts 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 42.12 scrub ok 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 3.0 deep-scrub starts 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 3.0 deep-scrub ok 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 101.1d scrub starts 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 101.1d scrub ok 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 12.c scrub starts 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: 12.c scrub ok 2024-03-30T15:02:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[41614]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 42.14 scrub starts 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 42.14 scrub ok 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 42.12 scrub starts 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 42.12 scrub ok 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 3.0 deep-scrub starts 2024-03-30T15:02:00.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 3.0 deep-scrub ok 2024-03-30T15:02:00.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 101.1d scrub starts 2024-03-30T15:02:00.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 101.1d scrub ok 2024-03-30T15:02:00.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 12.c scrub starts 2024-03-30T15:02:00.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: 12.c scrub ok 2024-03-30T15:02:00.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:00 smithi028 ceph-mon[29876]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:00.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:00.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:00.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:00.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:02:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:02:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 42.14 scrub starts 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 42.14 scrub ok 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 42.12 scrub starts 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 42.12 scrub ok 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: osdmap e355: 8 total, 8 up, 8 in 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4124774126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]: dispatch 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 3.0 deep-scrub starts 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 3.0 deep-scrub ok 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 101.1d scrub starts 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 101.1d scrub ok 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 12.c scrub starts 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: 12.c scrub ok 2024-03-30T15:02:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:00 smithi134 ceph-mon[33714]: pgmap v473: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:00.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:00.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:00.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2031944140 >> v1:172.21.15.28:6789/0 conn(0x7fcf1810ead0 legacy=0x7fcf18110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2031944140 shutdown_connections 2024-03-30T15:02:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2031944140 >> 172.21.15.28:0/2031944140 conn(0x7fcf180fdb50 msgr2=0x7fcf180fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2031944140 shutdown_connections 2024-03-30T15:02:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2031944140 wait complete. 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 librados: starting msgr at 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.902+0000 7fcf1d02b700 1 librados: starting objecter 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 Processor -- start 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 -- start start 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 librados: setting wanted keys 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 librados: calling monclient init 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcf18107950 con 0x7fcf1810af30 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcf18107b60 con 0x7fcf181022a0 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1d02b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcf181a96b0 con 0x7fcf1810ead0 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf1810af30 0x7fcf18104eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37714/0 (socket says 172.21.15.28:37714) 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1659c700 1 -- 172.21.15.28:0/2690646418 learned_addr learned my addr 172.21.15.28:0/2690646418 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf1759e700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fcf1810ead0 0x7fcf181a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52076/0 (socket says 172.21.15.28:52076) 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter ms_handle_connect 0x7fcf1810af30 2024-03-30T15:02:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter ms_handle_connect 0x7fcf1810ead0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter ms_handle_connect 0x7fcf181022a0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355610079 0 0) 0x7fcf18107950 con 0x7fcf1810af30 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf040031f0 con 0x7fcf1810af30 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121759713 0 0) 0x7fcf181a96b0 con 0x7fcf1810ead0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf18107950 con 0x7fcf1810ead0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3184530928 0 0) 0x7fcf18107b60 con 0x7fcf181022a0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.903+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf181a96b0 con 0x7fcf181022a0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3680165496 0 0) 0x7fcf18107950 con 0x7fcf1810ead0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf18107b60 con 0x7fcf1810ead0 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3698329905 0 0) 0x7fcf040031f0 con 0x7fcf1810af30 2024-03-30T15:02:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf18107950 con 0x7fcf1810af30 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcf10004880 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcf00001890 con 0x7fcf1810af30 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1426916719 0 0) 0x7fcf181a96b0 con 0x7fcf181022a0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf040031f0 con 0x7fcf181022a0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcf080017b0 con 0x7fcf181022a0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1856948585 0 0) 0x7fcf18107b60 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 >> v1:172.21.15.134:6789/0 conn(0x7fcf181022a0 legacy=0x7fcf181047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 >> v1:172.21.15.28:6789/0 conn(0x7fcf1810af30 legacy=0x7fcf18104eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf181aa8f0 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181a98e0 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf1d02b700 10 client.57809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf181a9da0 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf1d02b700 1 librados: init done 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fcf10001eb0 con 0x7fcf1810ead0 2024-03-30T15:02:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.904+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcf10005030 con 0x7fcf1810ead0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.905+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8005120 con 0x7fcf1810ead0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.905+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fcf10005270 con 0x7fcf1810ead0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(356..356 src has 53..356) ==== 8541+0+0 (unknown 3622634301 0 0) 0x7fcf10089ba0 con 0x7fcf1810ead0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 10 client.57809.objecter ms_dispatch 0x7fcf18001550 osd_map(356..356 src has 53..356) 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 3 client.57809.objecter handle_osd_map got epochs [356,356] > 0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 3 client.57809.objecter handle_osd_map decoding full epoch 356 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 20 client.57809.objecter dump_active .. 0 homeless 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.906+0000 7fcf0ffff700 10 client.57809.objecter ms_handle_connect 0x7fcf040748e0 2024-03-30T15:02:00.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:00.910+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fcf10056c30 con 0x7fcf1810ead0 2024-03-30T15:02:00.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:00.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:00.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:01.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:01.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.137+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} v 0) -- 0x7fcef8005410 con 0x7fcf1810ead0 2024-03-30T15:02:01.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:01.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:01.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.266+0000 7fcf0ffff700 1 -- 172.21.15.28:0/2690646418 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]=0 clear 4.1b pg_upmap_items mapping v357) ==== 118+0+0 (unknown 310682493 0 0) 0x7fcf1005a680 con 0x7fcf1810ead0 2024-03-30T15:02:01.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.268+0000 7fcf1d02b700 10 librados: watch_flush enter 2024-03-30T15:02:01.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 10 librados: watch_flush exit 2024-03-30T15:02:01.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 20 client.57809.objecter shutdown clearing up homeless session... 2024-03-30T15:02:01.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 10 client.57809.objecter successfully canceled tick 2024-03-30T15:02:01.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 >> v1:172.21.15.28:6800/3011431093 conn(0x7fcf040748e0 legacy=0x7fcf04076da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:01.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 >> v1:172.21.15.28:6790/0 conn(0x7fcf1810ead0 legacy=0x7fcf181a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:01.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 shutdown_connections 2024-03-30T15:02:01.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.269+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 >> 172.21.15.28:0/2690646418 conn(0x7fcf180fdb50 msgr2=0x7fcf180fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.270+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 shutdown_connections 2024-03-30T15:02:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.270+0000 7fcf1d02b700 1 -- 172.21.15.28:0/2690646418 wait complete. 2024-03-30T15:02:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:01.270+0000 7fcf1d02b700 1 librados: shutdown 2024-03-30T15:02:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:clear 4.1b pg_upmap_items mapping 2024-03-30T15:02:01.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:01.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:01.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:01.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 101.13 scrub starts 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 101.13 scrub ok 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2690646418' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[41614]: from='client.57809 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T15:02:01.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 101.13 scrub starts 2024-03-30T15:02:01.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 101.13 scrub ok 2024-03-30T15:02:01.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:02:01.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:02:01.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2690646418' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:01 smithi028 ceph-mon[29876]: from='client.57809 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.57785 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi028-92999-58"}]': finished 2024-03-30T15:02:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: osdmap e356: 8 total, 8 up, 8 in 2024-03-30T15:02:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 101.13 scrub starts 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 101.13 scrub ok 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:02:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2690646418' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:01 smithi134 ceph-mon[33714]: from='client.57809 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]: dispatch 2024-03-30T15:02:01.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:01.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:01.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:01.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:01.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:01.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.006+0000 7f81dc5b1700 1 -- 172.21.15.28:0/2822557517 >> v1:172.21.15.28:6790/0 conn(0x7f81d410d960 legacy=0x7f81d410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.006+0000 7f81dc5b1700 1 -- 172.21.15.28:0/2822557517 shutdown_connections 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.006+0000 7f81dc5b1700 1 -- 172.21.15.28:0/2822557517 >> 172.21.15.28:0/2822557517 conn(0x7f81d40fc9e0 msgr2=0x7f81d40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- 172.21.15.28:0/2822557517 shutdown_connections 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- 172.21.15.28:0/2822557517 wait complete. 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 librados: starting msgr at 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 librados: starting objecter 2024-03-30T15:02:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 Processor -- start 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- start start 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 librados: setting wanted keys 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 librados: calling monclient init 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81d406a950 con 0x7f81d4101130 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81d41a8540 con 0x7f81d4109dc0 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81dc5b1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f81d41a9720 con 0x7f81d410d960 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81da34d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81d4101130 0x7f81d41056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37746/0 (socket says 172.21.15.28:37746) 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.007+0000 7f81da34d700 1 -- 172.21.15.28:0/3774332920 learned_addr learned my addr 172.21.15.28:0/3774332920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter ms_handle_connect 0x7f81d4101130 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter ms_handle_connect 0x7f81d410d960 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:02.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter ms_handle_connect 0x7f81d4109dc0 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844851762 0 0) 0x7f81d406a950 con 0x7f81d4101130 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81c00031f0 con 0x7f81d4101130 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 690178862 0 0) 0x7f81d41a9720 con 0x7f81d410d960 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d406a950 con 0x7f81d410d960 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985716950 0 0) 0x7f81d41a8540 con 0x7f81d4109dc0 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d41a9720 con 0x7f81d4109dc0 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2723401351 0 0) 0x7f81c00031f0 con 0x7f81d4101130 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81d41a8540 con 0x7f81d4101130 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f81c4001930 con 0x7f81d4101130 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1491831931 0 0) 0x7f81d406a950 con 0x7f81d410d960 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81c00031f0 con 0x7f81d410d960 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f81cc004880 con 0x7f81d410d960 2024-03-30T15:02:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3425408896 0 0) 0x7f81d41a9720 con 0x7f81d4109dc0 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81d406a950 con 0x7f81d4109dc0 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f81d00010b0 con 0x7f81d4109dc0 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2676374696 0 0) 0x7f81d41a8540 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6790/0 conn(0x7f81d410d960 legacy=0x7f81d41a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.134:6789/0 conn(0x7f81d4109dc0 legacy=0x7f81d4105de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d41aa900 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f81d41a9950 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81dc5b1700 10 client.24639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f81d41a9e10 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.008+0000 7f81dc5b1700 1 librados: init done 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.009+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f81c4003b40 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.009+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f81c4005d30 con 0x7f81d4101130 2024-03-30T15:02:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.009+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f81b8005100 con 0x7f81d4101130 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.010+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f81c40036a0 con 0x7f81d4101130 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(357..357 src has 53..357) ==== 8847+0+0 (unknown 3676580251 0 0) 0x7f81c4089ac0 con 0x7f81d4101130 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 osd_map(357..357 src has 53..357) 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 3 client.24639.objecter handle_osd_map got epochs [357,357] > 0 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 3 client.24639.objecter handle_osd_map decoding full epoch 357 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 20 client.24639.objecter dump_active .. 0 homeless 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81c0070530 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v357) ==== 54+0+16 (unknown 2034508746 0 2420000192) 0x7f81c408a060 con 0x7f81d4101130 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 1 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 1 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 0 1 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 4 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 10 client.24639.objecter _send_command 1 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81b8004b50 con 0x7f81b8004730 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.011+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b8004730 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.012+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b8004b50 con 0x7f81b8004730 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.012+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.012+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 1 = system:0 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.012+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 4 2024-03-30T15:02:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.012+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 0 1 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 2 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 2 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b80053e0 osd=0 3 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 0 2 2024-03-30T15:02:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 4 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 10 client.24639.objecter _send_command 2 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.029+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b800be00 con 0x7f81b8004730 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b800be00 con 0x7f81b8004730 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 4 2024-03-30T15:02:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 0 2 2024-03-30T15:02:02.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:02.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 1 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 4 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 10 client.24639.objecter _send_command 3 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.031+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f81b8002160 con 0x7f81b8004280 2024-03-30T15:02:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b8004280 2024-03-30T15:02:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b8002160 con 0x7f81b8004280 2024-03-30T15:02:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 3 = system:0 2024-03-30T15:02:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 4 2024-03-30T15:02:02.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.032+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 1 3 2024-03-30T15:02:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 4 2024-03-30T15:02:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 4 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8004110 osd=1 3 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 1 4 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 4 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 10 client.24639.objecter _send_command 4 2024-03-30T15:02:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.050+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b8001600 con 0x7f81b8004280 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b8001600 con 0x7f81b8004280 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 4 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 1 4 2024-03-30T15:02:02.053 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 5 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 5 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 2 5 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 4 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 10 client.24639.objecter _send_command 5 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.051+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f81b800cf80 con 0x7f81b80119e0 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b80119e0 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b800cf80 con 0x7f81b80119e0 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 5 = system:0 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 4 2024-03-30T15:02:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.052+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 2 5 2024-03-30T15:02:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 6 2024-03-30T15:02:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 6 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b800c780 osd=2 3 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 2 6 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 4 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 10 client.24639.objecter _send_command 6 2024-03-30T15:02:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.069+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b800c2c0 con 0x7f81b80119e0 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b800c2c0 con 0x7f81b80119e0 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 4 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 2 6 2024-03-30T15:02:02.072 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 7 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 7 2024-03-30T15:02:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 3 7 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 4 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 10 client.24639.objecter _send_command 7 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f81b8016090 con 0x7f81b8015520 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.071+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b8015520 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.072+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b8016090 con 0x7f81b8015520 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.072+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.072+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 7 = system:0 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.072+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 4 2024-03-30T15:02:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.072+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 3 7 2024-03-30T15:02:02.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 8 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 8 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b800c550 osd=3 3 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 3 8 2024-03-30T15:02:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 4 2024-03-30T15:02:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 10 client.24639.objecter _send_command 8 2024-03-30T15:02:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.090+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b8016860 con 0x7f81b8015520 2024-03-30T15:02:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b8016860 con 0x7f81b8015520 2024-03-30T15:02:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 4 2024-03-30T15:02:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 3 8 2024-03-30T15:02:02.094 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:02.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 9 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 9 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 4 9 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 4 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 10 client.24639.objecter _send_command 9 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.092+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f81b801ea10 con 0x7f81b801af00 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.093+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b801af00 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.094+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b801ea10 con 0x7f81b801af00 2024-03-30T15:02:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.094+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.094+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 9 = system:0 2024-03-30T15:02:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.094+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 4 2024-03-30T15:02:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.094+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 4 9 2024-03-30T15:02:02.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 10 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 10 2024-03-30T15:02:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8017c70 osd=4 3 2024-03-30T15:02:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 4 10 2024-03-30T15:02:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 4 2024-03-30T15:02:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 10 client.24639.objecter _send_command 10 2024-03-30T15:02:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.111+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b8010b10 con 0x7f81b801af00 2024-03-30T15:02:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b8010b10 con 0x7f81b801af00 2024-03-30T15:02:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 4 2024-03-30T15:02:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 4 10 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.113+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 11 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 11 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 5 11 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 4 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 10 client.24639.objecter _send_command 11 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f81b8022dc0 con 0x7f81b801f090 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.114+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b801f090 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.115+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b8022dc0 con 0x7f81b801f090 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.115+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.115+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 11 = system:0 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.115+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 4 2024-03-30T15:02:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.115+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 5 11 2024-03-30T15:02:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 12 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 12 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b801ea10 osd=5 3 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 5 12 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 4 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 10 client.24639.objecter _send_command 12 2024-03-30T15:02:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.132+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b8024770 con 0x7f81b801f090 2024-03-30T15:02:02.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b8024770 con 0x7f81b801f090 2024-03-30T15:02:02.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 4 2024-03-30T15:02:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 5 12 2024-03-30T15:02:02.136 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 13 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 13 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 6 13 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 4 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 10 client.24639.objecter _send_command 13 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.134+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f81b80252f0 con 0x7f81b8024e90 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b8024e90 2024-03-30T15:02:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b80252f0 con 0x7f81b8024e90 2024-03-30T15:02:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 13 = system:0 2024-03-30T15:02:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 4 2024-03-30T15:02:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.135+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 6 13 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 14 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 14 2024-03-30T15:02:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b8024770 osd=6 3 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 6 14 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 4 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 10 client.24639.objecter _send_command 14 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.153+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b80252f0 con 0x7f81b8024e90 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.154+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b80252f0 con 0x7f81b8024e90 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.154+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.154+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.154+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 4 2024-03-30T15:02:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.154+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 6 14 2024-03-30T15:02:02.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 15 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 15 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 7 15 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 4 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 10 client.24639.objecter _send_command 15 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f81b802b7a0 con 0x7f81b8025520 2024-03-30T15:02:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.155+0000 7f81cb7fe700 10 client.24639.objecter ms_handle_connect 0x7f81b8025520 2024-03-30T15:02:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.156+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f81b802b7a0 con 0x7f81b8025520 2024-03-30T15:02:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.156+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.156+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 15 = system:0 2024-03-30T15:02:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.156+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 4 2024-03-30T15:02:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.156+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 7 15 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 10 client.24639.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign -1 16 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 20 client.24639.objecter _get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_remove -1 16 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 20 client.24639.objecter get_session s=0x7f81b80252f0 osd=7 3 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 15 client.24639.objecter _session_command_op_assign 7 16 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 4 2024-03-30T15:02:02.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 10 client.24639.objecter _send_command 16 2024-03-30T15:02:02.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.173+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f81b8010b10 con 0x7f81b8025520 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81cb7fe700 1 -- 172.21.15.28:0/3774332920 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f81b8010b10 con 0x7f81b8025520 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81cb7fe700 10 client.24639.objecter ms_dispatch 0x7f81d4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81cb7fe700 10 client.24639.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81cb7fe700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 4 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81cb7fe700 15 client.24639.objecter _session_command_op_remove 7 16 2024-03-30T15:02:02.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 librados: watch_flush enter 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 librados: watch_flush exit 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.0 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6801/3394323575 conn(0x7f81b8004730 legacy=0x7f81b8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80053e0 osd=0 1 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.1 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6805/1850313547 conn(0x7f81b8004280 legacy=0x7f81b800f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8004110 osd=1 1 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.2 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6809/3451325043 conn(0x7f81b80119e0 legacy=0x7f81b8013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c780 osd=2 1 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.3 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6813/1506157250 conn(0x7f81b8015520 legacy=0x7f81b8015980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b800c550 osd=3 1 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.4 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.134:6800/3063775514 conn(0x7f81b801af00 legacy=0x7f81b801d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8017c70 osd=4 1 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.5 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dab4e700 1 -- 172.21.15.28:0/3774332920 reap_dead start 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.134:6804/3668151179 conn(0x7f81b801f090 legacy=0x7f81b8021510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dab4e700 1 -- 172.21.15.28:0/3774332920 reap_dead start 2024-03-30T15:02:02.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b801ea10 osd=5 1 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.6 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.134:6808/2135224194 conn(0x7f81b8024e90 legacy=0x7f81b8022fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b8024770 osd=6 1 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter close_session for osd.7 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.134:6812/3758190298 conn(0x7f81b8025520 legacy=0x7f81b8029ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dab4e700 1 -- 172.21.15.28:0/3774332920 reap_dead start 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter put_session s=0x7f81b80252f0 osd=7 1 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 20 client.24639.objecter shutdown clearing up homeless session... 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 10 client.24639.objecter successfully canceled tick 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6800/3011431093 conn(0x7f81c0070530 legacy=0x7f81c00729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> v1:172.21.15.28:6789/0 conn(0x7f81d4101130 legacy=0x7f81d41056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 shutdown_connections 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 >> 172.21.15.28:0/3774332920 conn(0x7f81d40fc9e0 msgr2=0x7f81d41105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 shutdown_connections 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 -- 172.21.15.28:0/3774332920 wait complete. 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:02.175+0000 7f81dc5b1700 1 librados: shutdown 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:02.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:02.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:02.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:02.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:02.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:02.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:02.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: 42.1b deep-scrub starts 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: 42.1b deep-scrub ok 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: from='client.57809 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-03-30T15:02:02.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3774332920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: 42.1b deep-scrub starts 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: 42.1b deep-scrub ok 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: from='client.57809 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3774332920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:02.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:02.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:02.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:02.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: 42.1b deep-scrub starts 2024-03-30T15:02:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: 42.1b deep-scrub ok 2024-03-30T15:02:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: from='client.57809 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: osdmap e357: 8 total, 8 up, 8 in 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3774332920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:02.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:02.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:02.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:03.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:03.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:03.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:03.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:03.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:03.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:03.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:03.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:52:19.381206+0000", 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 584.14715051300004, 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93059181899999999, 2024-03-30T15:02:03.531 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14158", 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2199995094", 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381206+0000", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381208+0000", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8050000000000001e-06 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381216+0000", 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2020000000000004e-06 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381226+0000", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5489999999999995e-06 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381235+0000", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1109999999999997e-06 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381298+0000", 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2657999999999995e-05 2024-03-30T15:02:03.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:19.381306+0000", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3629999999999994e-06 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310542+0000", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92923640699999999 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310695+0000", 2024-03-30T15:02:03.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015222900000000001 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.310906+0000", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021148200000000001 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311551+0000", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064435399999999998 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311644+0000", 2024-03-30T15:02:03.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3702000000000005e-05 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311735+0000", 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0938000000000002e-05 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311769+0000", 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3439999999999998e-05 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:52:20.311798+0000", 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9578999999999999e-05 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34346.0:4 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.331748+0000", 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 446.196609056, 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14333030899999999, 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:02:03.538 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34346", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4086439972", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331748+0000", 2024-03-30T15:02:03.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331750+0000", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8610000000000001e-06 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331757+0000", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2699999999999999e-06 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331776+0000", 2024-03-30T15:02:03.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8924e-05 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331795+0000", 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8972999999999999e-05 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331857+0000", 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1649999999999994e-05 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.331866+0000", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5149999999999995e-06 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474614+0000", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14274793599999999 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.474650+0000", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5559999999999998e-05 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475078+0000", 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042862 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34352.0:2 3.0 3:00000000::::head [pgnls start_epoch 69] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e69)", 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:54:37.391127+0000", 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 446.13722952299997, 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084430388999999995, 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:02:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34352", 2024-03-30T15:02:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3269533966", 2024-03-30T15:02:03.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391127+0000", 2024-03-30T15:02:03.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391129+0000", 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.773e-06 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391135+0000", 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9989999999999997e-06 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391143+0000", 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1689999999999992e-06 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391151+0000", 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7300000000000005e-06 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391213+0000", 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1749999999999997e-05 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.547 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.391220+0000", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2359999999999998e-06 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475387+0000", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084166640000000001 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475417+0000", 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9899e-05 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.548 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:54:37.475558+0000", 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014119299999999999 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24907.0:2 24.b 24:d3fd6930:test-rados-api-smithi028-93417-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 248.56171189099999, 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27101285899999999, 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24907", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1522502118", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966647+0000", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.165e-06 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966656+0000", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4060000000000005e-06 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966667+0000", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.118e-05 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966679+0000", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.202e-05 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967482+0000", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080304200000000001 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967489+0000", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8129999999999998e-06 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093584+0000", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12609579200000001 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093596+0000", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1165000000000001e-05 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224181+0000", 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13058547400000001 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224307+0000", 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125637 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224451+0000", 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014380600000000001 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226643+0000", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021921750000000002 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227566+0000", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092316999999999996 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237564+0000", 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099978580000000001 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237612+0000", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8216000000000001e-05 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237658+0000", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5939999999999997e-05 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:2 23.10 23:0f4c2fcb:test-rados-api-smithi028-93408-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:02:03.556 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 248.55706608700001, 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26655647300000002, 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971294+0000", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.242e-06 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971305+0000", 2024-03-30T15:02:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0713999999999999e-05 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971327+0000", 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2089000000000001e-05 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971340+0000", 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2721999999999999e-05 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971419+0000", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9756e-05 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971426+0000", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5610000000000004e-06 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072900+0000", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.101473934 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072909+0000", 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5359999999999998e-06 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226905+0000", 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153995785 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227020+0000", 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011438300000000001 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227154+0000", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013489 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227892+0000", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073729500000000001 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.235325+0000", 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007433762 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237751+0000", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024258859999999999 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237804+0000", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2366e-05 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237847+0000", 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3551999999999999e-05 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.563 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24940.0:2 26.5 26:a21fad64:test-rados-api-smithi028-93750-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 248.54891636100001, 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25955751599999999, 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24940", 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T15:02:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979443+0000", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8150000000000002e-06 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979453+0000", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3689999999999996e-06 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979464+0000", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1061e-05 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979475+0000", 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1506e-05 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057216+0000", 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.077740857999999996 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057225+0000", 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210726+0000", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153500514 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210858+0000", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132724 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.211010+0000", 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015139900000000001 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.215902+0000", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048923489999999998 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.230960+0000", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015057615 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238899+0000", 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079398179999999995 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238954+0000", 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4842000000000001e-05 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238998+0000", 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3739000000000003e-05 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34880.0:2 13.16 13:6e692731:test-rados-api-smithi028-93525-2::foo:head [write 0~128] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 248.38073044199999, 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.148417239, 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34880", 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147629+0000", 2024-03-30T15:02:03.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147636+0000", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6860000000000006e-06 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147651+0000", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5071e-05 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147662+0000", 2024-03-30T15:02:03.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1103e-05 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147747+0000", 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4765000000000006e-05 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147757+0000", 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4099999999999997e-06 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280327+0000", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.132570166 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280452+0000", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125182 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280591+0000", 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013946 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282110+0000", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015181719999999999 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282802+0000", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069276799999999996 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295940+0000", 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013137729000000001 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295998+0000", 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.766e-05 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.296044+0000", 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5887000000000003e-05 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.577 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35207.0:2 45.6 45:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 247.39443985299999, 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151675741, 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35207", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/517891268", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133919+0000", 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.902e-06 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133926+0000", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4460000000000004e-06 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133942+0000", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5692000000000002e-05 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133954+0000", 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.169e-05 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134023+0000", 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8934000000000001e-05 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134029+0000", 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6760000000000001e-06 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223013+0000", 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088983584000000004 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223022+0000", 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5620000000000004e-06 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264315+0000", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041293273999999998 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264431+0000", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116506 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264556+0000", 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124237 2024-03-30T15:02:03.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.279474+0000", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014917977000000001 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280282+0000", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080855300000000005 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285498+0000", 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052162550000000004 2024-03-30T15:02:03.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285549+0000", 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0924999999999998e-05 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285593+0000", 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3528000000000001e-05 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2059 11.7 e116/110 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head v 116'1, mlcod=0'0)", 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.55824897400001, 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061806397999999999, 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970112+0000", 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0609999999999997e-06 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013285+0000", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.04317315 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013310+0000", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4615000000000001e-05 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013320+0000", 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0107e-05 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013400+0000", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0039999999999999e-05 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013425+0000", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5494e-05 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031865+0000", 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01844001 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031914+0000", 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8921000000000001e-05 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2069 11.7 e116/110 11:e17e9421:test-rados-api-smithi028-93257-17::big:head v 116'4, mlcod=116'2)", 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.298185+0000", 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.230171837, 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.042950631000000003, 2024-03-30T15:02:03.590 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298185+0000", 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298185+0000", 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.298186+0000", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2950000000000001e-06 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323173+0000", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024987056000000001 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323201+0000", 2024-03-30T15:02:03.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7954e-05 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323215+0000", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3456e-05 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323280+0000", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5036999999999994e-05 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.323303+0000", 2024-03-30T15:02:03.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.285e-05 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.341074+0000", 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017771022000000001 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.341136+0000", 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1960999999999998e-05 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2074 11.15 e117/110 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head v 117'1, mlcod=0'0)", 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 244.85091355700001, 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044884778, 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:02:03.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677447+0000", 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.811e-06 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689208+0000", 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011761303000000001 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689235+0000", 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6809e-05 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689248+0000", 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2918000000000001e-05 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689313+0000", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4999999999999994e-05 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689337+0000", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3526000000000001e-05 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722259+0000", 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032922117000000001 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722328+0000", 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9294000000000004e-05 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32236.0:2 121.0 121:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 222.51367578200001, 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2733807989999999, 2024-03-30T15:02:03.599 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.600 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.600 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32236", 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/520874703", 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:02:03.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014685+0000", 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4340000000000001e-06 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014695+0000", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0513000000000001e-05 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014709+0000", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3791000000000001e-05 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014721+0000", 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2165e-05 2024-03-30T15:02:03.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075744+0000", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061023012000000001 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075753+0000", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2e-06 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162599+0000", 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0868460719999999 2024-03-30T15:02:03.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162721+0000", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121662 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.171761+0000", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090404189999999992 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.176616+0000", 2024-03-30T15:02:03.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004854934 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.272952+0000", 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.096335658000000005 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.287962+0000", 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015010364 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288013+0000", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0917000000000001e-05 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288062+0000", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8658000000000001e-05 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1598 124.0 e147/140 124:0384ab8e:::oid:1 v 147'6, mlcod=144'3)", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:02:03.607 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 212.134147848, 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050952077999999998, 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394210+0000", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.102e-06 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394226+0000", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554e-05 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394235+0000", 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-03-30T15:02:03.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394244+0000", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2120000000000003e-06 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396824+0000", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025802059999999998 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396852+0000", 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8252999999999999e-05 2024-03-30T15:02:03.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445122+0000", 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048270362999999997 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445161+0000", 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8736999999999999e-05 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.611 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:35 111.1 e149/134 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head v 149'7, mlcod=148'4)", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 210.303406537, 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043878153000000003, 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224951+0000", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.91e-07 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224962+0000", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0858e-05 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224967+0000", 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9699999999999996e-06 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224977+0000", 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1079999999999999e-06 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235114+0000", 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010137607999999999 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235139+0000", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4437e-05 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268792+0000", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033653216 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268829+0000", 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6665000000000002e-05 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.48856.0:9 157.2s0 157:523311ec:test-rados-api-smithi028-93441-13::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 197.34266702900001, 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053529637999999997, 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.48856", 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1328251834", 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9 2024-03-30T15:02:03.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185693+0000", 2024-03-30T15:02:03.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1939999999999998e-06 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185705+0000", 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1446e-05 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185727+0000", 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2249e-05 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185739+0000", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185814+0000", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5013999999999997e-05 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185822+0000", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5329999999999997e-06 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198631+0000", 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012809375 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198727+0000", 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5884000000000006e-05 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.199305+0000", 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057763400000000005 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.211644+0000", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012338905000000001 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239140+0000", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027495908999999999 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239220+0000", 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9802999999999995e-05 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.56915.0:2 173.1b 173:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 194.34278505699999, 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.049005913, 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.56915", 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1224869641", 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185574+0000", 2024-03-30T15:02:03.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9039999999999999e-06 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185581+0000", 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7090000000000002e-06 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185595+0000", 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3257e-05 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185606+0000", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1419e-05 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193710+0000", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081039819999999992 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193719+0000", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1670000000000005e-06 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223661+0000", 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299414410000001 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223739+0000", 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8771999999999997e-05 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223867+0000", 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705999999999999 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.225806+0000", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019391670000000001 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.227421+0000", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00161532 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234490+0000", 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070693969999999998 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234537+0000", 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6709999999999998e-05 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234578+0000", 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0608000000000002e-05 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57314.0:2 223.9 223:94beac24:::oid_RoundTripCmpExtPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-30T15:02:03.628 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 134.32755854800001, 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16945650800000001, 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57314", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2222872349", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:02:03.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200800+0000", 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1320000000000001e-06 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200808+0000", 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2590000000000003e-06 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200817+0000", 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200827+0000", 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0472000000000001e-05 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200895+0000", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7424999999999999e-05 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200900+0000", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8420000000000001e-06 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362513+0000", 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16161309700000001 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362526+0000", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2933e-05 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.368887+0000", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063609690000000002 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369004+0000", 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116635 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369145+0000", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014146499999999999 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370068+0000", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092285399999999997 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370108+0000", 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9529e-05 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370170+0000", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1995999999999994e-05 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370216+0000", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5865000000000002e-05 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370255+0000", 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9128000000000002e-05 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.635 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.2.0:1538 101.3 e248/130 101:c3bfc9e7:test-rados-api-smithi028-93253-24::bar:8 v 248'39, mlcod=243'35)", 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 111.32306338799999, 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050164583999999998, 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:02:03.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6100000000000001e-07 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205300+0000", 2024-03-30T15:02:03.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2219999999999999e-06 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227707+0000", 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022406780000000001 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227716+0000", 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5339999999999999e-06 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231237+0000", 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035216029999999999 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231261+0000", 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3981999999999998e-05 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255425+0000", 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024164396000000001 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255458+0000", 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2605999999999999e-05 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24402.0:1 256.7s0 256:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 73.298804171, 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005296856, 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:03.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24402", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2485876767", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229555+0000", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229563+0000", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9759999999999995e-06 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229573+0000", 2024-03-30T15:02:03.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0592e-05 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229584+0000", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0533000000000001e-05 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229647+0000", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3118000000000001e-05 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229651+0000", 2024-03-30T15:02:03.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8680000000000001e-06 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232864+0000", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0032131120000001 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232980+0000", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011602100000000001 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.233406+0000", 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042549999999999999 2024-03-30T15:02:03.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234188+0000", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078191600000000001 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234777+0000", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00058978600000000002 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234850+0000", 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2254000000000003e-05 2024-03-30T15:02:03.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:4660 275.0 275:00000000::::head [pgnls start_epoch 344] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e344)", 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 15.283269178999999, 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.266339783, 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:02:03.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4660 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245090+0000", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.667e-06 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245101+0000", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0541e-05 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245116+0000", 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5091e-05 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245129+0000", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.278e-05 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245194+0000", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5180999999999998e-05 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245199+0000", 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.118e-06 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511197+0000", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26599746200000002 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511236+0000", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9444999999999999e-05 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511428+0000", 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019149799999999999 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.650 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:03.651 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:03.651 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:02:03.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:03.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:03.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:02:03.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 101.d scrub starts 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 101.d scrub ok 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: pgmap v476: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 3.a deep-scrub starts 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 3.a deep-scrub ok 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 101.12 deep-scrub starts 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 101.12 deep-scrub ok 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T15:02:03.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[29876]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 101.d scrub starts 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 101.d scrub ok 2024-03-30T15:02:03.741 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: pgmap v476: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 3.a deep-scrub starts 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 3.a deep-scrub ok 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 101.12 deep-scrub starts 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 101.12 deep-scrub ok 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:03.742 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:03 smithi028 ceph-mon[41614]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T15:02:03.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:03.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 101.d scrub starts 2024-03-30T15:02:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 101.d scrub ok 2024-03-30T15:02:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: pgmap v476: 260 pgs: 260 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 3.a deep-scrub starts 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 3.a deep-scrub ok 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: osdmap e358: 8 total, 8 up, 8 in 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 101.12 deep-scrub starts 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 101.12 deep-scrub ok 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi028-92999-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:03.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:03 smithi134 ceph-mon[33714]: osdmap e359: 8 total, 8 up, 8 in 2024-03-30T15:02:03.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:03.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:04.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:04.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:04.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:04.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:04.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:04.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:04.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:04.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:04.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:04.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.695 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:04.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:04.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 101.14 scrub starts 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: 101.14 scrub ok 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: pgmap v479: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:04.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[41614]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 101.14 scrub starts 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: 101.14 scrub ok 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: pgmap v479: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:04.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:04 smithi028 ceph-mon[29876]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T15:02:04.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:02:04.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 101.14 scrub starts 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: 101.14 scrub ok 2024-03-30T15:02:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: pgmap v479: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:04.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:04 smithi134 ceph-mon[33714]: osdmap e360: 8 total, 8 up, 8 in 2024-03-30T15:02:04.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:02:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:04.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:05.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:05.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:05.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.379 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:05.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:05.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:05.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:05 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:05.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:05.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:05.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:02:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: osdmap e361: 8 total, 8 up, 8 in 2024-03-30T15:02:05.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:05 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:05.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:06.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:06.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:06.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:06.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:06.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:06.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:06.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:06.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:06.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:06.575 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:02:06.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:02:06.576 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: pgmap v482: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:02:06.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: pgmap v482: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:06.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:06 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:06.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:06.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:06.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:06.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:06.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:06.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:06.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:02:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:02:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: pgmap v482: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:02:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:02:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: osdmap e362: 8 total, 8 up, 8 in 2024-03-30T15:02:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2168501096' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:06 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]: dispatch 2024-03-30T15:02:07.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:07.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:07.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:07.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout:o.RoundTripAppendPP (2997 ms) 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2992 ms) 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3016 ms) 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-30T15:02:07.247 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2989 ms) 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: setting pool EIO 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: max_success 99, min_failed 100 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4008 ms) 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3007 ms) 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (135848 ms total) 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-30T15:02:07.248 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3237 ms) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8755 ms) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15015 ms) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3015 ms) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30022 ms total) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2980 ms) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2980 ms total) 2024-03-30T15:02:07.249 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13998 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7279 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6759 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6982 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3001 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6992 ms) 2024-03-30T15:02:07.250 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7024 ms) 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6979 ms) 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7073 ms) 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-30T15:02:07.251 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6942 ms) 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6992 ms) 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7027 ms) 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-30T15:02:07.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6982 ms) 2024-03-30T15:02:07.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:07.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:07.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:07.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:07.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:07.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:07.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:07.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:07.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:07.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:02:07.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:02:07.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:07.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[41614]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:07 smithi028 ceph-mon[29876]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T15:02:07.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:07.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: from='client.57803 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi028-92999-59"}]': finished 2024-03-30T15:02:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:07 smithi134 ceph-mon[33714]: osdmap e363: 8 total, 8 up, 8 in 2024-03-30T15:02:08.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:08.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:08.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:08.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:08.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:08.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:08.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:08.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:08.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:08.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:08.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:08.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 101.f scrub starts 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 101.f scrub ok 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 42.1a deep-scrub starts 2024-03-30T15:02:08.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: 42.1a deep-scrub ok 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: pgmap v485: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1215912126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.57827 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.57827 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 101.f scrub starts 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 101.f scrub ok 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 42.1a deep-scrub starts 2024-03-30T15:02:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: 42.1a deep-scrub ok 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: pgmap v485: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1215912126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.57827 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.57827 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 101.f scrub starts 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 101.f scrub ok 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 42.1a deep-scrub starts 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: 42.1a deep-scrub ok 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: pgmap v485: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1215912126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.57827 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.57827 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: osdmap e364: 8 total, 8 up, 8 in 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]: dispatch 2024-03-30T15:02:08.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:08.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:08.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.734+0000 7f4353a50700 1 -- 172.21.15.28:0/4272851203 >> v1:172.21.15.28:6790/0 conn(0x7f434c10aed0 legacy=0x7f434c10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.734+0000 7f4353a50700 1 -- 172.21.15.28:0/4272851203 shutdown_connections 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.734+0000 7f4353a50700 1 -- 172.21.15.28:0/4272851203 >> 172.21.15.28:0/4272851203 conn(0x7f434c0fdaf0 msgr2=0x7f434c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- 172.21.15.28:0/4272851203 shutdown_connections 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- 172.21.15.28:0/4272851203 wait complete. 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 librados: starting msgr at 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 librados: starting objecter 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 Processor -- start 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- start start 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 librados: setting wanted keys 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 librados: calling monclient init 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f434c1a70f0 con 0x7f434c10ea70 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f434c1a8310 con 0x7f434c10aed0 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.735+0000 7f4353a50700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f434c1a9550 con 0x7f434c102240 2024-03-30T15:02:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43517ec700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f434c102240 0x7f434c107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52126/0 (socket says 172.21.15.28:52126) 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43517ec700 1 -- 172.21.15.28:0/1215912126 learned_addr learned my addr 172.21.15.28:0/1215912126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter ms_handle_connect 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter ms_handle_connect 0x7f434c10aed0 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764920907 0 0) 0x7f434c1a9550 con 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4338003aa0 con 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737895558 0 0) 0x7f434c1a8310 con 0x7f434c10aed0 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f434c1a9550 con 0x7f434c10aed0 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2733695216 0 0) 0x7f4338003aa0 con 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f434c1a8310 con 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4348001a30 con 0x7f434c102240 2024-03-30T15:02:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter ms_handle_connect 0x7f434c10ea70 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2216210520 0 0) 0x7f434c1a8310 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 >> v1:172.21.15.134:6789/0 conn(0x7f434c10aed0 legacy=0x7f434c1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 >> v1:172.21.15.28:6789/0 conn(0x7f434c10ea70 legacy=0x7f434c1a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c1aa790 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f434c1a8540 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f4353a50700 10 client.57827.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f434c1a8a00 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.736+0000 7f4353a50700 1 librados: init done 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.737+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4348001c90 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.737+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4348005d10 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.738+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4330005120 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.738+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f4348003930 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(363..363 src has 53..363) ==== 8512+0+0 (unknown 615726568 0 0) 0x7f4348089a80 con 0x7f434c102240 2024-03-30T15:02:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 10 client.57827.objecter ms_dispatch 0x7f434c001550 osd_map(363..363 src has 53..363) 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 3 client.57827.objecter handle_osd_map got epochs [363,363] > 0 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 3 client.57827.objecter handle_osd_map decoding full epoch 363 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 20 client.57827.objecter dump_active .. 0 homeless 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.739+0000 7f43427fc700 10 client.57827.objecter ms_handle_connect 0x7f433806ff40 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.743+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4348056b30 con 0x7f434c102240 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:07.958+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f4330005410 con 0x7f434c102240 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.244+0000 7f43427fc700 1 -- 172.21.15.28:0/1215912126 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v364) ==== 87+0+0 (unknown 682088944 0 0) 0x7f434805a580 con 0x7f434c102240 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.247+0000 7f4353a50700 10 librados: watch_flush enter 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.248+0000 7f4353a50700 10 librados: watch_flush exit 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.248+0000 7f4353a50700 20 client.57827.objecter shutdown clearing up homeless session... 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.248+0000 7f4353a50700 10 client.57827.objecter successfully canceled tick 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.248+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 >> v1:172.21.15.28:6800/3011431093 conn(0x7f433806ff40 legacy=0x7f4338072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.249+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 >> v1:172.21.15.28:6790/0 conn(0x7f434c102240 legacy=0x7f434c107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.249+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 shutdown_connections 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.249+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 >> 172.21.15.28:0/1215912126 conn(0x7f434c0fdaf0 msgr2=0x7f434c111c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.249+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 shutdown_connections 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.249+0000 7f4353a50700 1 -- 172.21.15.28:0/1215912126 wait complete. 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:08.250+0000 7f4353a50700 1 librados: shutdown 2024-03-30T15:02:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:02:09.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:09.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:02:09.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:02:09.075 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-30T15:02:09.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-30T15:02:09.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:09.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:09.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:09.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:09.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:09.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:09.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:09.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:09.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:09.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:09.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:09.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:09.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:10.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:10.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:10.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:10.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:10.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:10.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:10.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:10.509 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:02:10.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:02:10.509 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:02:10.509 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:02:10.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:10.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]': finished 2024-03-30T15:02:10.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[41614]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T15:02:10.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:10.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]': finished 2024-03-30T15:02:10.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:10 smithi028 ceph-mon[29876]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T15:02:10.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:10.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:10.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:10.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:10.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:10.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1001379214' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi028-92999-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-45"}]': finished 2024-03-30T15:02:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:10 smithi134 ceph-mon[33714]: osdmap e365: 8 total, 8 up, 8 in 2024-03-30T15:02:10.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:10.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:10.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:10.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:10.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:11.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:11.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.151+0000 7f1708269700 1 -- 172.21.15.28:0/1952564838 >> v1:172.21.15.28:6790/0 conn(0x7f17001022a0 legacy=0x7f1700102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.151+0000 7f1708269700 1 -- 172.21.15.28:0/1952564838 shutdown_connections 2024-03-30T15:02:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.151+0000 7f1708269700 1 -- 172.21.15.28:0/1952564838 >> 172.21.15.28:0/1952564838 conn(0x7f17000fdb50 msgr2=0x7f17000fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.151+0000 7f1708269700 1 -- 172.21.15.28:0/1952564838 shutdown_connections 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.151+0000 7f1708269700 1 -- 172.21.15.28:0/1952564838 wait complete. 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 librados: starting msgr at 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 librados: starting objecter 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 Processor -- start 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 -- start start 2024-03-30T15:02:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 librados: setting wanted keys 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 librados: calling monclient init 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17001a93e0 con 0x7f170010af30 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17001aa620 con 0x7f170010ead0 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1708269700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17001ab860 con 0x7f17001022a0 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1705804700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f170010af30 0x7f17001a4330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55618/0 (socket says 172.21.15.28:55618) 2024-03-30T15:02:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1706005700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f17001022a0 0x7f1700107440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39438/0 (socket says 172.21.15.28:39438) 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f1705804700 1 -- 172.21.15.28:0/4273601833 learned_addr learned my addr 172.21.15.28:0/4273601833 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f16f77fe700 10 client.?.objecter ms_handle_connect 0x7f170010af30 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f16f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f16f77fe700 10 client.?.objecter ms_handle_connect 0x7f17001022a0 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.152+0000 7f16f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672952483 0 0) 0x7f17001a93e0 con 0x7f170010af30 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16ec0035b0 con 0x7f170010af30 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791900873 0 0) 0x7f17001ab860 con 0x7f17001022a0 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17001a93e0 con 0x7f17001022a0 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 10 client.?.objecter ms_handle_connect 0x7f170010ead0 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2324850748 0 0) 0x7f16ec0035b0 con 0x7f170010af30 2024-03-30T15:02:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17001ab860 con 0x7f170010af30 2024-03-30T15:02:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f16fc001a70 con 0x7f170010af30 2024-03-30T15:02:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845229564 0 0) 0x7f17001aa620 con 0x7f170010ead0 2024-03-30T15:02:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16ec0035b0 con 0x7f170010ead0 2024-03-30T15:02:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 260971161 0 0) 0x7f17001a93e0 con 0x7f17001022a0 2024-03-30T15:02:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17001aa620 con 0x7f17001022a0 2024-03-30T15:02:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f16f0004540 con 0x7f17001022a0 2024-03-30T15:02:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1022605919 0 0) 0x7f17001ab860 con 0x7f170010af30 2024-03-30T15:02:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6790/0 conn(0x7f17001022a0 legacy=0x7f1700107440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.134:6789/0 conn(0x7f170010ead0 legacy=0x7f17001a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17001acaa0 con 0x7f170010af30 2024-03-30T15:02:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17001aba90 con 0x7f170010af30 2024-03-30T15:02:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f1708269700 10 client.24663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f17001abf50 con 0x7f170010af30 2024-03-30T15:02:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f1708269700 1 librados: init done 2024-03-30T15:02:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f16fc001d20 con 0x7f170010af30 2024-03-30T15:02:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.153+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f16fc005dc0 con 0x7f170010af30 2024-03-30T15:02:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.154+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f16e4005100 con 0x7f170010af30 2024-03-30T15:02:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.155+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f16fc003930 con 0x7f170010af30 2024-03-30T15:02:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.155+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(366..366 src has 53..366) ==== 8093+0+0 (unknown 1541674397 0 0) 0x7f16fc0897f0 con 0x7f170010af30 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.155+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 osd_map(366..366 src has 53..366) 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.155+0000 7f16f77fe700 3 client.24663.objecter handle_osd_map got epochs [366,366] > 0 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.155+0000 7f16f77fe700 3 client.24663.objecter handle_osd_map decoding full epoch 366 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f16f77fe700 20 client.24663.objecter dump_active .. 0 homeless 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v366) ==== 54+0+16 (unknown 824135033 0 2420000192) 0x7f16fc089bf0 con 0x7f170010af30 2024-03-30T15:02:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16ec0705b0 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 10 client.24663.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 1 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 1 2024-03-30T15:02:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 0 1 2024-03-30T15:02:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 4 2024-03-30T15:02:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 10 client.24663.objecter _send_command 1 2024-03-30T15:02:11.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16e4004b50 con 0x7f16e4004730 2024-03-30T15:02:11.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.156+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e4004730 2024-03-30T15:02:11.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.157+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e4004b50 con 0x7f16e4004730 2024-03-30T15:02:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.157+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.157+0000 7f16f77fe700 10 client.24663.objecter _finish_command 1 = system:0 2024-03-30T15:02:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.157+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 4 2024-03-30T15:02:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.157+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 0 1 2024-03-30T15:02:11.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 10 client.24663.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 2 2024-03-30T15:02:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 2 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e40053e0 osd=0 3 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 0 2 2024-03-30T15:02:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 4 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 10 client.24663.objecter _send_command 2 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.174+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e40079f0 con 0x7f16e4004730 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.175+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e40079f0 con 0x7f16e4004730 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.175+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.175+0000 7f16f77fe700 10 client.24663.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.175+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 4 2024-03-30T15:02:11.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.175+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 0 2 2024-03-30T15:02:11.180 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:11.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 10 client.24663.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 3 2024-03-30T15:02:11.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 3 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 1 3 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 4 2024-03-30T15:02:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 10 client.24663.objecter _send_command 3 2024-03-30T15:02:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f16e4002180 con 0x7f16e4004280 2024-03-30T15:02:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.176+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e4004280 2024-03-30T15:02:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.177+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e4002180 con 0x7f16e4004280 2024-03-30T15:02:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.177+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.177+0000 7f16f77fe700 10 client.24663.objecter _finish_command 3 = system:0 2024-03-30T15:02:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.177+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 4 2024-03-30T15:02:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.177+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 1 3 2024-03-30T15:02:11.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 10 client.24663.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 4 2024-03-30T15:02:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 4 2024-03-30T15:02:11.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4004110 osd=1 3 2024-03-30T15:02:11.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 1 4 2024-03-30T15:02:11.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 4 2024-03-30T15:02:11.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 10 client.24663.objecter _send_command 4 2024-03-30T15:02:11.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.194+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e4001600 con 0x7f16e4004280 2024-03-30T15:02:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e4001600 con 0x7f16e4004280 2024-03-30T15:02:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 10 client.24663.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 4 2024-03-30T15:02:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 1 4 2024-03-30T15:02:11.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 10 client.24663.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 5 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 5 2024-03-30T15:02:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 2 5 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 4 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 10 client.24663.objecter _send_command 5 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f16e40110e0 con 0x7f16e400d5d0 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.196+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e400d5d0 2024-03-30T15:02:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.197+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e40110e0 con 0x7f16e400d5d0 2024-03-30T15:02:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.197+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.197+0000 7f16f77fe700 10 client.24663.objecter _finish_command 5 = system:0 2024-03-30T15:02:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.197+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 4 2024-03-30T15:02:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.197+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 2 5 2024-03-30T15:02:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:11.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 10 client.24663.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 6 2024-03-30T15:02:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 6 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4008370 osd=2 3 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 2 6 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 4 2024-03-30T15:02:11.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 10 client.24663.objecter _send_command 6 2024-03-30T15:02:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.214+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e4007eb0 con 0x7f16e400d5d0 2024-03-30T15:02:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e4007eb0 con 0x7f16e400d5d0 2024-03-30T15:02:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f16f77fe700 10 client.24663.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 4 2024-03-30T15:02:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 2 6 2024-03-30T15:02:11.220 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 10 client.24663.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 7 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 7 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 3 7 2024-03-30T15:02:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 4 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 10 client.24663.objecter _send_command 7 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.215+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f16e4015f20 con 0x7f16e4012410 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e4012410 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e4015f20 con 0x7f16e4012410 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 10 client.24663.objecter _finish_command 7 = system:0 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 4 2024-03-30T15:02:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.216+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 3 7 2024-03-30T15:02:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 10 client.24663.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 8 2024-03-30T15:02:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 8 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e400cc20 osd=3 3 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 3 8 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 4 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 10 client.24663.objecter _send_command 8 2024-03-30T15:02:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.233+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e400c700 con 0x7f16e4012410 2024-03-30T15:02:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e400c700 con 0x7f16e4012410 2024-03-30T15:02:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 10 client.24663.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 4 2024-03-30T15:02:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 3 8 2024-03-30T15:02:11.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 10 client.24663.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 9 2024-03-30T15:02:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 9 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 4 9 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 4 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 10 client.24663.objecter _send_command 9 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f16e401a800 con 0x7f16e4011f70 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.235+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e4011f70 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.236+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e401a800 con 0x7f16e4011f70 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.236+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.236+0000 7f16f77fe700 10 client.24663.objecter _finish_command 9 = system:0 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.236+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 4 2024-03-30T15:02:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.236+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 4 9 2024-03-30T15:02:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 10 client.24663.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 10 2024-03-30T15:02:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 10 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e400c700 osd=4 3 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 4 10 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 4 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 10 client.24663.objecter _send_command 10 2024-03-30T15:02:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.259+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e401c1b0 con 0x7f16e4011f70 2024-03-30T15:02:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.264+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e401c1b0 con 0x7f16e4011f70 2024-03-30T15:02:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.264+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.264+0000 7f16f77fe700 10 client.24663.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.264+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 4 2024-03-30T15:02:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.264+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 4 10 2024-03-30T15:02:11.267 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 10 client.24663.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 11 2024-03-30T15:02:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 11 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 5 11 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 4 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 10 client.24663.objecter _send_command 11 2024-03-30T15:02:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.266+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f16e401cd60 con 0x7f16e401c8d0 2024-03-30T15:02:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.270+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e401c8d0 2024-03-30T15:02:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.272+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e401cd60 con 0x7f16e401c8d0 2024-03-30T15:02:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.272+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.272+0000 7f16f77fe700 10 client.24663.objecter _finish_command 11 = system:0 2024-03-30T15:02:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.272+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 4 2024-03-30T15:02:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.272+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 5 11 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 10 client.24663.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 12 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 12 2024-03-30T15:02:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e401c1b0 osd=5 3 2024-03-30T15:02:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 5 12 2024-03-30T15:02:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 4 2024-03-30T15:02:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 10 client.24663.objecter _send_command 12 2024-03-30T15:02:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.290+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e401cfe0 con 0x7f16e401c8d0 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e401cfe0 con 0x7f16e401c8d0 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f16f77fe700 10 client.24663.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 4 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 5 12 2024-03-30T15:02:11.293 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 10 client.24663.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 13 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 13 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 6 13 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 4 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 10 client.24663.objecter _send_command 13 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.292+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f16e40233b0 con 0x7f16e401f8a0 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e401f8a0 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e40233b0 con 0x7f16e401f8a0 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 10 client.24663.objecter _finish_command 13 = system:0 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 4 2024-03-30T15:02:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.293+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 6 13 2024-03-30T15:02:11.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 10 client.24663.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 14 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 14 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e401cdc0 osd=6 3 2024-03-30T15:02:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 6 14 2024-03-30T15:02:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 4 2024-03-30T15:02:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 10 client.24663.objecter _send_command 14 2024-03-30T15:02:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.310+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e400c9c0 con 0x7f16e401f8a0 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4273601833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.314 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4273601833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e400c9c0 con 0x7f16e401f8a0 2024-03-30T15:02:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f16f77fe700 10 client.24663.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 4 2024-03-30T15:02:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 6 14 2024-03-30T15:02:11.316 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 10 client.24663.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 15 2024-03-30T15:02:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 15 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 7 15 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 4 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 10 client.24663.objecter _send_command 15 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.312+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f16e40115c0 con 0x7f16e4016c00 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.313+0000 7f16f77fe700 10 client.24663.objecter ms_handle_connect 0x7f16e4016c00 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.314+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f16e40115c0 con 0x7f16e4016c00 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.314+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.314+0000 7f16f77fe700 10 client.24663.objecter _finish_command 15 = system:0 2024-03-30T15:02:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.314+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 4 2024-03-30T15:02:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.314+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 7 15 2024-03-30T15:02:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:11.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 10 client.24663.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign -1 16 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 20 client.24663.objecter _get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 15 client.24663.objecter _session_command_op_remove -1 16 2024-03-30T15:02:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 20 client.24663.objecter get_session s=0x7f16e4011290 osd=7 3 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 15 client.24663.objecter _session_command_op_assign 7 16 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 4 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 10 client.24663.objecter _send_command 16 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.333+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16e4011840 con 0x7f16e4016c00 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.335+0000 7f16f77fe700 1 -- 172.21.15.28:0/4273601833 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f16e4011840 con 0x7f16e4016c00 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.335+0000 7f16f77fe700 10 client.24663.objecter ms_dispatch 0x7f1700001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.335+0000 7f16f77fe700 10 client.24663.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.335+0000 7f16f77fe700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 4 2024-03-30T15:02:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.335+0000 7f16f77fe700 15 client.24663.objecter _session_command_op_remove 7 16 2024-03-30T15:02:11.337 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 librados: watch_flush enter 2024-03-30T15:02:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 librados: watch_flush exit 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.0 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6801/3394323575 conn(0x7f16e4004730 legacy=0x7f16e4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e40053e0 osd=0 1 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.1 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6805/1850313547 conn(0x7f16e4004280 legacy=0x7f16e400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4004110 osd=1 1 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.2 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6809/3451325043 conn(0x7f16e400d5d0 legacy=0x7f16e400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4008370 osd=2 1 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.3 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6813/1506157250 conn(0x7f16e4012410 legacy=0x7f16e4014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400cc20 osd=3 1 2024-03-30T15:02:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.4 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.134:6800/3063775514 conn(0x7f16e4011f70 legacy=0x7f16e40190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e400c700 osd=4 1 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.5 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.134:6804/3668151179 conn(0x7f16e401c8d0 legacy=0x7f16e401a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1706806700 1 -- 172.21.15.28:0/4273601833 reap_dead start 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401c1b0 osd=5 1 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.6 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.134:6808/2135224194 conn(0x7f16e401f8a0 legacy=0x7f16e4021c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1706806700 1 -- 172.21.15.28:0/4273601833 reap_dead start 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1706806700 1 -- 172.21.15.28:0/4273601833 reap_dead start 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1706806700 1 -- 172.21.15.28:0/4273601833 reap_dead start 2024-03-30T15:02:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e401cdc0 osd=6 1 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter close_session for osd.7 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.134:6812/3758190298 conn(0x7f16e4016c00 legacy=0x7f16e4026540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter put_session s=0x7f16e4011290 osd=7 1 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 20 client.24663.objecter shutdown clearing up homeless session... 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 10 client.24663.objecter successfully canceled tick 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6800/3011431093 conn(0x7f16ec0705b0 legacy=0x7f16ec072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> v1:172.21.15.28:6789/0 conn(0x7f170010af30 legacy=0x7f17001a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 shutdown_connections 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 >> 172.21.15.28:0/4273601833 conn(0x7f17000fdb50 msgr2=0x7f1700108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 shutdown_connections 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 -- 172.21.15.28:0/4273601833 wait complete. 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:11.336+0000 7f1708269700 1 librados: shutdown 2024-03-30T15:02:11.341 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:11.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:11.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:11.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:11.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: osdmap e366: 8 total, 8 up, 8 in 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4273601833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:11.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:11.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:02:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:11.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:11.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:11.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:11.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:02:11.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:11.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:11.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:12.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:12.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:12.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:12.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:12.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:12.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:12.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:12.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:12.556 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:12.577 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:12.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.617 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:12.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:12.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:12.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:12.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:12.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:12.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:12.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi028-92999-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: osdmap e367: 8 total, 8 up, 8 in 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:12.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:12.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:12.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:12.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:13.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:13.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:13.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:13.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:13.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:13.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:13.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:13.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:13.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:13.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:13.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:13.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: pgmap v491: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:13.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: pgmap v491: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:13.884 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:13.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: pgmap v491: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: osdmap e368: 8 total, 8 up, 8 in 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi028-92999-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: osdmap e369: 8 total, 8 up, 8 in 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:13.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:14.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:14.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:14.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:14.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:14.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:14.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:14.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:14.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:14.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:14.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:02:14.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:14.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:14.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:14.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:14.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:14.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:15.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:15.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:15.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:15.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:15.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:15.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:15.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:15.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:15.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[29876]: pgmap v494: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:15.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:15.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[29876]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T15:02:15.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.514 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[41614]: pgmap v494: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:15.514 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:15.514 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[41614]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T15:02:15.514 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.514 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:15.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:15.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:15 smithi134 ceph-mon[33714]: pgmap v494: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-30T15:02:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:15 smithi134 ceph-mon[33714]: osdmap e370: 8 total, 8 up, 8 in 2024-03-30T15:02:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]: dispatch 2024-03-30T15:02:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:15.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:15.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:15.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:15.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:15.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:16.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:16.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:16.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:16.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:16.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:16.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:16.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:16.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]': finished 2024-03-30T15:02:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T15:02:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]': finished 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:16 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:16.576 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:16.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-47", "mode": "writeback"}]': finished 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: osdmap e371: 8 total, 8 up, 8 in 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:16.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:16 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:16.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:16.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 -- 172.21.15.28:0/1266301854 >> v1:172.21.15.28:6789/0 conn(0x7fd6b010d9a0 legacy=0x7fd6b010fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 -- 172.21.15.28:0/1266301854 shutdown_connections 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 -- 172.21.15.28:0/1266301854 >> 172.21.15.28:0/1266301854 conn(0x7fd6b00fca20 msgr2=0x7fd6b00fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 -- 172.21.15.28:0/1266301854 shutdown_connections 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 -- 172.21.15.28:0/1266301854 wait complete. 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 librados: starting msgr at 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.677+0000 7fd6b87de700 1 librados: starting objecter 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 Processor -- start 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 -- start start 2024-03-30T15:02:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 librados: setting wanted keys 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 librados: calling monclient init 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6b01a5e30 con 0x7fd6b0109e00 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6b01a7050 con 0x7fd6b0101170 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b87de700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd6b01a8290 con 0x7fd6b010d9a0 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b5d79700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6b0109e00 0x7fd6b01a0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55630/0 (socket says 172.21.15.28:55630) 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b5d79700 1 -- 172.21.15.28:0/1834393401 learned_addr learned my addr 172.21.15.28:0/1834393401 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6b6d7b700 1 --1- 172.21.15.28:0/1834393401 >> v1:172.21.15.28:6790/0 conn(0x7fd6b010d9a0 0x7fd6b01a4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39456/0 (socket says 172.21.15.28:39456) 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter ms_handle_connect 0x7fd6b010d9a0 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter ms_handle_connect 0x7fd6b0109e00 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter ms_handle_connect 0x7fd6b0101170 2024-03-30T15:02:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3742595094 0 0) 0x7fd6b01a8290 con 0x7fd6b010d9a0 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd698003aa0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555631033 0 0) 0x7fd6b01a5e30 con 0x7fd6b0109e00 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b01a8290 con 0x7fd6b0109e00 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568455572 0 0) 0x7fd6b01a7050 con 0x7fd6b0101170 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b01a5e30 con 0x7fd6b0101170 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1586708455 0 0) 0x7fd698003aa0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.678+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6b01a7050 con 0x7fd6b010d9a0 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd6a80045e0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1748479204 0 0) 0x7fd6b01a8290 con 0x7fd6b0109e00 2024-03-30T15:02:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd698003aa0 con 0x7fd6b0109e00 2024-03-30T15:02:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd6a40017c0 con 0x7fd6b0109e00 2024-03-30T15:02:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4200990235 0 0) 0x7fd6b01a5e30 con 0x7fd6b0101170 2024-03-30T15:02:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6b01a8290 con 0x7fd6b0101170 2024-03-30T15:02:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3540184234 0 0) 0x7fd6b01a7050 con 0x7fd6b010d9a0 2024-03-30T15:02:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 >> v1:172.21.15.134:6789/0 conn(0x7fd6b0101170 legacy=0x7fd6b01062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 >> v1:172.21.15.28:6789/0 conn(0x7fd6b0109e00 legacy=0x7fd6b01a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b01a94d0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b01a6060 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6b87de700 10 client.57845.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6b01a6520 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6b87de700 1 librados: init done 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd6a8001070 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.679+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd6a8004cf0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.680+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69c005120 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.680+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd6a8004cf0 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(372..372 src has 53..372) ==== 8593+0+0 (unknown 4275546077 0 0) 0x7fd6a8089810 con 0x7fd6b010d9a0 2024-03-30T15:02:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 10 client.57845.objecter ms_dispatch 0x7fd6b0001550 osd_map(372..372 src has 53..372) 2024-03-30T15:02:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 3 client.57845.objecter handle_osd_map got epochs [372,372] > 0 2024-03-30T15:02:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 3 client.57845.objecter handle_osd_map decoding full epoch 372 2024-03-30T15:02:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 20 client.57845.objecter dump_active .. 0 homeless 2024-03-30T15:02:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.681+0000 7fd6af7fe700 10 client.57845.objecter ms_handle_connect 0x7fd6980726b0 2024-03-30T15:02:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.685+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd6a8056870 con 0x7fd6b010d9a0 2024-03-30T15:02:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:16.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:16.907+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd69c005410 con 0x7fd6b010d9a0 2024-03-30T15:02:16.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:16.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:17.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:17.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:17.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:17.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.255+0000 7fd6af7fe700 1 -- 172.21.15.28:0/1834393401 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v373) ==== 97+0+0 (unknown 4210852209 0 0) 0x7fd6a805a2c0 con 0x7fd6b010d9a0 2024-03-30T15:02:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 10 librados: watch_flush enter 2024-03-30T15:02:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 10 librados: watch_flush exit 2024-03-30T15:02:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 20 client.57845.objecter shutdown clearing up homeless session... 2024-03-30T15:02:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 10 client.57845.objecter successfully canceled tick 2024-03-30T15:02:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd6980726b0 legacy=0x7fd698074b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 >> v1:172.21.15.28:6790/0 conn(0x7fd6b010d9a0 legacy=0x7fd6b01a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 shutdown_connections 2024-03-30T15:02:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.260+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 >> 172.21.15.28:0/1834393401 conn(0x7fd6b00fca20 msgr2=0x7fd6b00fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.261+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 shutdown_connections 2024-03-30T15:02:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.261+0000 7fd6b87de700 1 -- 172.21.15.28:0/1834393401 wait complete. 2024-03-30T15:02:17.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:17.261+0000 7fd6b87de700 1 librados: shutdown 2024-03-30T15:02:17.263 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:02:17.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:17.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:02:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:02:17.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:17.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:17.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1834393401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[29876]: from='client.57845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1834393401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:17 smithi028 ceph-mon[41614]: from='client.57845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:17.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: pgmap v497: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-30T15:02:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: osdmap e372: 8 total, 8 up, 8 in 2024-03-30T15:02:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3946804484' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]: dispatch 2024-03-30T15:02:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1834393401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:17 smithi134 ceph-mon[33714]: from='client.57845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:17.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:17.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:17.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:17.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:17.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:17.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:18.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:18.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:18.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:18.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:18.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:18.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:18.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:18.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.57845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[41614]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.57845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T15:02:18.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-mon[29876]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:18.511 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:18.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:18.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.57833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi028-92999-61"}]': finished 2024-03-30T15:02:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.57845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: osdmap e373: 8 total, 8 up, 8 in 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-30T15:02:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:18 smithi134 ceph-mon[33714]: pgmap v500: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:18.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- 172.21.15.28:0/1972182387 >> v1:172.21.15.28:6789/0 conn(0x7fd030101130 legacy=0x7fd030101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- 172.21.15.28:0/1972182387 shutdown_connections 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- 172.21.15.28:0/1972182387 >> 172.21.15.28:0/1972182387 conn(0x7fd0300fc9e0 msgr2=0x7fd0300fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- 172.21.15.28:0/1972182387 shutdown_connections 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- 172.21.15.28:0/1972182387 wait complete. 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 librados: starting msgr at 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 librados: starting objecter 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 Processor -- start 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 -- start start 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 librados: setting wanted keys 2024-03-30T15:02:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.757+0000 7fd036818700 1 librados: calling monclient init 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd036818700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd0301068b0 con 0x7fd030109dc0 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd036818700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd0301a8540 con 0x7fd03010d960 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd034db5700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd03010d960 0x7fd0301a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:48148/0 (socket says 172.21.15.28:48148) 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd034db5700 1 -- 172.21.15.28:0/811813676 learned_addr learned my addr 172.21.15.28:0/811813676 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd0301a9720 con 0x7fd030101130 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter ms_handle_connect 0x7fd03010d960 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter ms_handle_connect 0x7fd030101130 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter ms_handle_connect 0x7fd030109dc0 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094497967 0 0) 0x7fd0301a8540 con 0x7fd03010d960 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0180031f0 con 0x7fd03010d960 2024-03-30T15:02:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.758+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256842629 0 0) 0x7fd0301a9720 con 0x7fd030101130 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0301a8540 con 0x7fd030101130 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429532173 0 0) 0x7fd0301068b0 con 0x7fd030109dc0 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0301a9720 con 0x7fd030109dc0 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4043920110 0 0) 0x7fd0180031f0 con 0x7fd03010d960 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0301068b0 con 0x7fd03010d960 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd028001d90 con 0x7fd03010d960 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1802867138 0 0) 0x7fd0301a8540 con 0x7fd030101130 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0180031f0 con 0x7fd030101130 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd020004490 con 0x7fd030101130 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3952035295 0 0) 0x7fd0301a9720 con 0x7fd030109dc0 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0301a8540 con 0x7fd030109dc0 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2729115758 0 0) 0x7fd0301068b0 con 0x7fd03010d960 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 >> v1:172.21.15.28:6790/0 conn(0x7fd030101130 legacy=0x7fd030104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 >> v1:172.21.15.28:6789/0 conn(0x7fd030109dc0 legacy=0x7fd030104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0301aa900 con 0x7fd03010d960 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd0301a9950 con 0x7fd03010d960 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd036818700 10 client.49876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd0301a9e10 con 0x7fd03010d960 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd036818700 1 librados: init done 2024-03-30T15:02:18.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.759+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd0280041d0 con 0x7fd03010d960 2024-03-30T15:02:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd028006180 con 0x7fd03010d960 2024-03-30T15:02:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd036818700 10 client.49876.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd036818700 15 client.49876.objecter _session_command_op_assign -1 1 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd036818700 20 client.49876.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd036818700 10 client.49876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.760+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd030102610 con 0x7fd03010d960 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.761+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd028003cb0 con 0x7fd03010d960 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(374..374 src has 53..374) ==== 8870+0+0 (unknown 212778753 0 0) 0x7fd028089d00 con 0x7fd03010d960 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 10 client.49876.objecter ms_dispatch 0x7fd030001550 osd_map(374..374 src has 53..374) 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 3 client.49876.objecter handle_osd_map got epochs [374,374] > 0 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 3 client.49876.objecter handle_osd_map decoding full epoch 374 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 10 client.49876.objecter checking command 1 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter _get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 15 client.49876.objecter _session_command_op_remove -1 1 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter _get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 15 client.49876.objecter _session_command_op_assign 0 1 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 4 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 10 client.49876.objecter _send_command 1 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0280041d0 con 0x7fd01807f160 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 20 client.49876.objecter dump_active .. 0 homeless 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=374) ==== 24+0+0 (unknown 2039789107 0 0) 0x7fd02808a0e0 con 0x7fd03010d960 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 10 client.49876.objecter ms_handle_connect 0x7fd0180702e0 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.762+0000 7fd02d7fa700 10 client.49876.objecter ms_handle_connect 0x7fd01807f160 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.763+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd0280041d0 con 0x7fd01807f160 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.763+0000 7fd02d7fa700 10 client.49876.objecter ms_dispatch 0x7fd030001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.763+0000 7fd02d7fa700 10 client.49876.objecter _finish_command 1 = system:0 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.763+0000 7fd02d7fa700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 4 2024-03-30T15:02:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.763+0000 7fd02d7fa700 15 client.49876.objecter _session_command_op_remove 0 1 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 10 client.49876.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 15 client.49876.objecter _session_command_op_assign -1 2 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 20 client.49876.objecter _get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 20 client.49876.objecter _get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 15 client.49876.objecter _session_command_op_remove -1 2 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 20 client.49876.objecter get_session s=0x7fd01807eab0 osd=0 3 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 15 client.49876.objecter _session_command_op_assign 0 2 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 4 2024-03-30T15:02:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 10 client.49876.objecter _send_command 2 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.783+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fd030068b30 con 0x7fd01807f160 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd02d7fa700 1 -- 172.21.15.28:0/811813676 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fd030068b30 con 0x7fd01807f160 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd02d7fa700 10 client.49876.objecter ms_dispatch 0x7fd030001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd02d7fa700 10 client.49876.objecter _finish_command 2 = system:0 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd02d7fa700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 4 2024-03-30T15:02:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd02d7fa700 15 client.49876.objecter _session_command_op_remove 0 2 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 10 librados: watch_flush enter 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 10 librados: watch_flush exit 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 10 client.49876.objecter close_session for osd.0 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 >> v1:172.21.15.28:6801/3394323575 conn(0x7fd01807f160 legacy=0x7fd018081600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 20 client.49876.objecter put_session s=0x7fd01807eab0 osd=0 1 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 20 client.49876.objecter shutdown clearing up homeless session... 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 10 client.49876.objecter successfully canceled tick 2024-03-30T15:02:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd0180702e0 legacy=0x7fd0180727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 >> v1:172.21.15.134:6789/0 conn(0x7fd03010d960 legacy=0x7fd0301a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd034db5700 1 -- 172.21.15.28:0/811813676 reap_dead start 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 shutdown_connections 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 >> 172.21.15.28:0/811813676 conn(0x7fd0300fc9e0 msgr2=0x7fd0301105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 shutdown_connections 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 -- 172.21.15.28:0/811813676 wait complete. 2024-03-30T15:02:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:18.784+0000 7fd036818700 1 librados: shutdown 2024-03-30T15:02:18.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:18.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:18.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:18.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:19.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:19.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:19.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:19.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:02:19.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:19.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:19.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:19.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:19.571 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:19.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:19.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:19.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:19.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:19.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-30T15:02:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T15:02:19.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:19 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: osdmap e374: 8 total, 8 up, 8 in 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:19.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:19 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:19.957 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:19.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:19.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-30T15:02:20.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:20.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:20.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:20.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:20.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:20.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:20.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:02:20.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:20.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:20.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: pgmap v503: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:20.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: pgmap v503: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:20.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: osdmap e375: 8 total, 8 up, 8 in 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: pgmap v503: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi028-92999-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:20.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:02:20.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: osdmap e376: 8 total, 8 up, 8 in 2024-03-30T15:02:20.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:02:20.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:20.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:21.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:21.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:21.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:21.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:21.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:21.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:21.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.538+0000 7f2fb263c700 1 -- 172.21.15.28:0/448206678 >> v1:172.21.15.28:6789/0 conn(0x7f2fac109dc0 legacy=0x7f2fac10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.538+0000 7f2fb263c700 1 -- 172.21.15.28:0/448206678 shutdown_connections 2024-03-30T15:02:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.538+0000 7f2fb263c700 1 -- 172.21.15.28:0/448206678 >> 172.21.15.28:0/448206678 conn(0x7f2fac0fc9e0 msgr2=0x7f2fac0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.538+0000 7f2fb263c700 1 -- 172.21.15.28:0/448206678 shutdown_connections 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.538+0000 7f2fb263c700 1 -- 172.21.15.28:0/448206678 wait complete. 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 librados: starting msgr at 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 librados: starting objecter 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 Processor -- start 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 -- start start 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 librados: setting wanted keys 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 librados: calling monclient init 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2fac1068f0 con 0x7f2fac109dc0 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2fac1a8540 con 0x7f2fac10d960 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fb263c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2fac1a9720 con 0x7f2fac101130 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fabfff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2fac101130 0x7f2fac1046c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53834/0 (socket says 172.21.15.28:53834) 2024-03-30T15:02:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fab7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2fac109dc0 0x7f2fac104dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52194/0 (socket says 172.21.15.28:52194) 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fabfff700 1 -- 172.21.15.28:0/3258347580 learned_addr learned my addr 172.21.15.28:0/3258347580 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fa97fa700 10 client.?.objecter ms_handle_connect 0x7f2fac101130 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fa97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fa97fa700 10 client.?.objecter ms_handle_connect 0x7f2fac109dc0 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.539+0000 7f2fa97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411784847 0 0) 0x7f2fac1068f0 con 0x7f2fac109dc0 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f940031f0 con 0x7f2fac109dc0 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920755558 0 0) 0x7f2fac1a9720 con 0x7f2fac101130 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fac1068f0 con 0x7f2fac101130 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 10 client.?.objecter ms_handle_connect 0x7f2fac10d960 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1958335076 0 0) 0x7f2fac1068f0 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fac1a9720 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 995453264 0 0) 0x7f2f940031f0 con 0x7f2fac109dc0 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fac1068f0 con 0x7f2fac109dc0 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2503282969 0 0) 0x7f2fac1a8540 con 0x7f2fac10d960 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f940031f0 con 0x7f2fac10d960 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2f9c001930 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2fa0001510 con 0x7f2fac109dc0 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3231788599 0 0) 0x7f2fac1a9720 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.134:6789/0 conn(0x7f2fac10d960 legacy=0x7f2fac1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6789/0 conn(0x7f2fac109dc0 legacy=0x7f2fac104dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac1aa900 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2fac1a7590 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fb263c700 10 client.57863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2fac1a7a50 con 0x7f2fac101130 2024-03-30T15:02:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fb263c700 1 librados: init done 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2f9c001cb0 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.540+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2f9c005d30 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.541+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2f98005100 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.542+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2f9c0040c0 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.542+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(377..377 src has 53..377) ==== 9396+0+0 (unknown 573172491 0 0) 0x7f2f9c089d70 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.542+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 osd_map(377..377 src has 53..377) 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.542+0000 7f2fa97fa700 3 client.57863.objecter handle_osd_map got epochs [377,377] > 0 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.542+0000 7f2fa97fa700 3 client.57863.objecter handle_osd_map decoding full epoch 377 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fa97fa700 20 client.57863.objecter dump_active .. 0 homeless 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v377) ==== 54+0+16 (unknown 2873370294 0 2420000192) 0x7f2f9c08c1a0 con 0x7f2fac101130 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f94070340 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 10 client.57863.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 1 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 1 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 0 1 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 4 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 10 client.57863.objecter _send_command 1 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f98004b50 con 0x7f2f98004730 2024-03-30T15:02:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.543+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f98004730 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.544+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f98004b50 con 0x7f2f98004730 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.544+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.544+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 1 = system:0 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.544+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 4 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.544+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 0 1 2024-03-30T15:02:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 10 client.57863.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 2 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 2 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f980053e0 osd=0 3 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 0 2 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 4 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 10 client.57863.objecter _send_command 2 2024-03-30T15:02:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.561+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f98007b00 con 0x7f2f98004730 2024-03-30T15:02:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f98007b00 con 0x7f2f98004730 2024-03-30T15:02:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 4 2024-03-30T15:02:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 0 2 2024-03-30T15:02:21.565 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 10 client.57863.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 3 2024-03-30T15:02:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 1 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 4 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 10 client.57863.objecter _send_command 3 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.563+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2f98002200 con 0x7f2f980042f0 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f980042f0 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f98002200 con 0x7f2f980042f0 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 3 = system:0 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 4 2024-03-30T15:02:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.564+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 1 3 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 10 client.57863.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 4 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 4 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98004180 osd=1 3 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 1 4 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 4 2024-03-30T15:02:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 10 client.57863.objecter _send_command 4 2024-03-30T15:02:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.586+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f98001600 con 0x7f2f980042f0 2024-03-30T15:02:21.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.587+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f98001600 con 0x7f2f980042f0 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.587+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.587+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.587+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 4 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.587+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 1 4 2024-03-30T15:02:21.589 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 10 client.57863.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 5 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 5 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 2 5 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 4 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 10 client.57863.objecter _send_command 5 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2f980111f0 con 0x7f2f9800d6e0 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.588+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f9800d6e0 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.589+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f980111f0 con 0x7f2f9800d6e0 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.589+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.589+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 5 = system:0 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.589+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 4 2024-03-30T15:02:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.589+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 2 5 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 10 client.57863.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 6 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 6 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98008480 osd=2 3 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 2 6 2024-03-30T15:02:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 4 2024-03-30T15:02:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 10 client.57863.objecter _send_command 6 2024-03-30T15:02:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.606+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f98007f60 con 0x7f2f9800d6e0 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f98007f60 con 0x7f2f9800d6e0 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 4 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 2 6 2024-03-30T15:02:21.609 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 10 client.57863.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 7 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 7 2024-03-30T15:02:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 3 7 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 4 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 10 client.57863.objecter _send_command 7 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2f98016030 con 0x7f2f98012520 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.608+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f98012520 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.609+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f98016030 con 0x7f2f98012520 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.609+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.609+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 7 = system:0 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.609+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 4 2024-03-30T15:02:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.609+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 3 7 2024-03-30T15:02:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 10 client.57863.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 8 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 8 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800cd30 osd=3 3 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 3 8 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 4 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 10 client.57863.objecter _send_command 8 2024-03-30T15:02:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.627+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f9800c7b0 con 0x7f2f98012520 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f9800c7b0 con 0x7f2f98012520 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 4 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 3 8 2024-03-30T15:02:21.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 10 client.57863.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 9 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 9 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 4 9 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 4 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 10 client.57863.objecter _send_command 9 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2f9800c740 con 0x7f2f98012080 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.629+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f98012080 2024-03-30T15:02:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.630+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f9800c740 con 0x7f2f98012080 2024-03-30T15:02:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.630+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.630+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 9 = system:0 2024-03-30T15:02:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.630+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 4 2024-03-30T15:02:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.630+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 4 9 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 10 client.57863.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 10 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 10 2024-03-30T15:02:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800c560 osd=4 3 2024-03-30T15:02:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 4 10 2024-03-30T15:02:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 4 2024-03-30T15:02:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 10 client.57863.objecter _send_command 10 2024-03-30T15:02:21.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.647+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f9800c7a0 con 0x7f2f98012080 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f9c05a4e0 con 0x7f2f98012080 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 4 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 4 10 2024-03-30T15:02:21.650 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 10 client.57863.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 11 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 11 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 5 11 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 4 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 10 client.57863.objecter _send_command 11 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.649+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2f98016eb0 con 0x7f2f980169b0 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f980169b0 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f98016eb0 con 0x7f2f980169b0 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 11 = system:0 2024-03-30T15:02:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 4 2024-03-30T15:02:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.650+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 5 11 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 10 client.57863.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 12 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 12 2024-03-30T15:02:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9800ca10 osd=5 3 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 5 12 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 4 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 10 client.57863.objecter _send_command 12 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.667+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f98023820 con 0x7f2f980169b0 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.669+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f98023820 con 0x7f2f980169b0 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.669+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.669+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.669+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 4 2024-03-30T15:02:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.669+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 5 12 2024-03-30T15:02:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 10 client.57863.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 13 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 13 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 6 13 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 4 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 10 client.57863.objecter _send_command 13 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2f98027af0 con 0x7f2f98023f40 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.670+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f98023f40 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.671+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f98027af0 con 0x7f2f98023f40 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.671+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.671+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 13 = system:0 2024-03-30T15:02:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.671+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 4 2024-03-30T15:02:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.671+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 6 13 2024-03-30T15:02:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 10 client.57863.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 14 2024-03-30T15:02:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 14 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f98016f10 osd=6 3 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 6 14 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 4 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 10 client.57863.objecter _send_command 14 2024-03-30T15:02:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.688+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f9801aa90 con 0x7f2f98023f40 2024-03-30T15:02:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f9801aa90 con 0x7f2f98023f40 2024-03-30T15:02:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 4 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 6 14 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 10 client.57863.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 15 2024-03-30T15:02:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 15 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 7 15 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 4 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 10 client.57863.objecter _send_command 15 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.690+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2f9801b610 con 0x7f2f9801b1b0 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.691+0000 7f2fa97fa700 10 client.57863.objecter ms_handle_connect 0x7f2f9801b1b0 2024-03-30T15:02:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.692+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f9801b610 con 0x7f2f9801b1b0 2024-03-30T15:02:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.692+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.692+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 15 = system:0 2024-03-30T15:02:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.692+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 4 2024-03-30T15:02:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.692+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 7 15 2024-03-30T15:02:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 10 client.57863.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign -1 16 2024-03-30T15:02:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 20 client.57863.objecter _get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_remove -1 16 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 20 client.57863.objecter get_session s=0x7f2f9801aa90 osd=7 3 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 15 client.57863.objecter _session_command_op_assign 7 16 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 4 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 10 client.57863.objecter _send_command 16 2024-03-30T15:02:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.710+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2f9801b610 con 0x7f2f9801b1b0 2024-03-30T15:02:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fa97fa700 1 -- 172.21.15.28:0/3258347580 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2f9801b610 con 0x7f2f9801b1b0 2024-03-30T15:02:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fa97fa700 10 client.57863.objecter ms_dispatch 0x7f2fac001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fa97fa700 10 client.57863.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fa97fa700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 4 2024-03-30T15:02:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fa97fa700 15 client.57863.objecter _session_command_op_remove 7 16 2024-03-30T15:02:21.714 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 librados: watch_flush enter 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 librados: watch_flush exit 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.0 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6801/3394323575 conn(0x7f2f98004730 legacy=0x7f2f98002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f980053e0 osd=0 1 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.1 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6805/1850313547 conn(0x7f2f980042f0 legacy=0x7f2f9800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98004180 osd=1 1 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.2 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6809/3451325043 conn(0x7f2f9800d6e0 legacy=0x7f2f9800fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98008480 osd=2 1 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.3 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6813/1506157250 conn(0x7f2f98012520 legacy=0x7f2f98014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800cd30 osd=3 1 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.4 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.134:6800/3063775514 conn(0x7f2f98012080 legacy=0x7f2f98019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb0bd9700 1 -- 172.21.15.28:0/3258347580 reap_dead start 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800c560 osd=4 1 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.5 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.134:6804/3668151179 conn(0x7f2f980169b0 legacy=0x7f2f9801bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb0bd9700 1 -- 172.21.15.28:0/3258347580 reap_dead start 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9800ca10 osd=5 1 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.6 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.134:6808/2135224194 conn(0x7f2f98023f40 legacy=0x7f2f98026330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f98016f10 osd=6 1 2024-03-30T15:02:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter close_session for osd.7 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.134:6812/3758190298 conn(0x7f2f9801b1b0 legacy=0x7f2f98027cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter put_session s=0x7f2f9801aa90 osd=7 1 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 20 client.57863.objecter shutdown clearing up homeless session... 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 10 client.57863.objecter successfully canceled tick 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb0bd9700 1 -- 172.21.15.28:0/3258347580 reap_dead start 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2f94070340 legacy=0x7f2f94072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.712+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> v1:172.21.15.28:6790/0 conn(0x7f2fac101130 legacy=0x7f2fac1046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.713+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 shutdown_connections 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.713+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 >> 172.21.15.28:0/3258347580 conn(0x7f2fac0fc9e0 msgr2=0x7f2fac110970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.713+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 shutdown_connections 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.713+0000 7f2fb263c700 1 -- 172.21.15.28:0/3258347580 wait complete. 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:21.713+0000 7f2fb263c700 1 librados: shutdown 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:21.717 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:21.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:21.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:21.803 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:21.803 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:02:21.803 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[41614]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T15:02:21.803 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[29876]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:21.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:02:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:21 smithi134 ceph-mon[33714]: osdmap e377: 8 total, 8 up, 8 in 2024-03-30T15:02:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:21.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:22.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:22.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:22.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:22.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:22.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:22.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:22.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:22.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:22.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:22.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:22.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:22.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:22.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:22.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:22.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:22.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3258347580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: pgmap v506: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T15:02:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:22 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3258347580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: pgmap v506: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3258347580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: pgmap v506: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: osdmap e378: 8 total, 8 up, 8 in 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:22.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:22 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:22.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:23.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:23.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:23.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:23.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:23.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:23.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:23.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:23.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:23.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:23.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:23.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:23.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:23.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:23.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T15:02:23.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]': finished 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: osdmap e379: 8 total, 8 up, 8 in 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/275545117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]: dispatch 2024-03-30T15:02:23.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:23.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:23.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-47"}]: dispatch 2024-03-30T15:02:24.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:24.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:24.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:24.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:24.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:24.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:24.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:24.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:24.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:24.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:24.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:24.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:24.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:24.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:24.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:24.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:25.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:25.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:25.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:02:25.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:25.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:25.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:25.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:25.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T15:02:25.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:25 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:25.576 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:25.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: pgmap v509: 260 pgs: 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.57851 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi028-92999-62"}]': finished 2024-03-30T15:02:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: osdmap e380: 8 total, 8 up, 8 in 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:25 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:25.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:25.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:25.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:25.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:25.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:26.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:26.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:26.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:26.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:26.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:26.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:26.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:26.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:26.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:26.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T15:02:26.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[29876]: pgmap v512: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:26 smithi028 ceph-mon[41614]: pgmap v512: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:26.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:26.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi028-92999-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: osdmap e381: 8 total, 8 up, 8 in 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:26 smithi134 ceph-mon[33714]: pgmap v512: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-30T15:02:26.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:26.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:26.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:26.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.897+0000 7f5cd0260700 1 -- 172.21.15.28:0/3642541546 >> v1:172.21.15.28:6789/0 conn(0x7f5cc8102250 legacy=0x7f5cc8102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.897+0000 7f5cd0260700 1 -- 172.21.15.28:0/3642541546 shutdown_connections 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.897+0000 7f5cd0260700 1 -- 172.21.15.28:0/3642541546 >> 172.21.15.28:0/3642541546 conn(0x7f5cc80fdb20 msgr2=0x7f5cc80fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.897+0000 7f5cd0260700 1 -- 172.21.15.28:0/3642541546 shutdown_connections 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.897+0000 7f5cd0260700 1 -- 172.21.15.28:0/3642541546 wait complete. 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 librados: starting msgr at 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 librados: starting objecter 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 Processor -- start 2024-03-30T15:02:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 -- start start 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 librados: setting wanted keys 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 librados: calling monclient init 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5cc81a9300 con 0x7f5cc810ea80 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5cc81aa520 con 0x7f5cc8102250 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cd0260700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5cc81ab760 con 0x7f5cc810aee0 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5ccd7fb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5cc810aee0 0x7f5cc81a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53860/0 (socket says 172.21.15.28:53860) 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cce7fd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5cc810ea80 0x7f5cc81a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52224/0 (socket says 172.21.15.28:52224) 2024-03-30T15:02:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5ccd7fb700 1 -- 172.21.15.28:0/3069359444 learned_addr learned my addr 172.21.15.28:0/3069359444 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter ms_handle_connect 0x7f5cc810ea80 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter ms_handle_connect 0x7f5cc810aee0 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter ms_handle_connect 0x7f5cc8102250 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.898+0000 7f5cc6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2950286019 0 0) 0x7f5cc81a9300 con 0x7f5cc810ea80 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb00031f0 con 0x7f5cc810ea80 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345796576 0 0) 0x7f5cc81ab760 con 0x7f5cc810aee0 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc81a9300 con 0x7f5cc810aee0 2024-03-30T15:02:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027397522 0 0) 0x7f5cc81aa520 con 0x7f5cc8102250 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc81ab760 con 0x7f5cc8102250 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3887133579 0 0) 0x7f5cc81a9300 con 0x7f5cc810aee0 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cc81aa520 con 0x7f5cc810aee0 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5cbc0018f0 con 0x7f5cc810aee0 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2513629275 0 0) 0x7f5cb00031f0 con 0x7f5cc810ea80 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cc81a9300 con 0x7f5cc810ea80 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5cc0001130 con 0x7f5cc810ea80 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 152498053 0 0) 0x7f5cc81ab760 con 0x7f5cc8102250 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cb00031f0 con 0x7f5cc8102250 2024-03-30T15:02:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5cb80048c0 con 0x7f5cc8102250 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2935365760 0 0) 0x7f5cc81aa520 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 >> v1:172.21.15.134:6789/0 conn(0x7f5cc8102250 legacy=0x7f5cc8107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 >> v1:172.21.15.28:6789/0 conn(0x7f5cc810ea80 legacy=0x7f5cc81a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc81ac9a0 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc81ab990 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cd0260700 10 client.57875.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5cc81abe50 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cd0260700 1 librados: init done 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5cbc003b40 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.899+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5cbc005d10 con 0x7f5cc810aee0 2024-03-30T15:02:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.900+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb4005120 con 0x7f5cc810aee0 2024-03-30T15:02:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5cbc0036a0 con 0x7f5cc810aee0 2024-03-30T15:02:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(382..382 src has 53..382) ==== 8941+0+0 (unknown 1591808632 0 0) 0x7f5cbc089b70 con 0x7f5cc810aee0 2024-03-30T15:02:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 10 client.57875.objecter ms_dispatch 0x7f5cc8001550 osd_map(382..382 src has 53..382) 2024-03-30T15:02:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 3 client.57875.objecter handle_osd_map got epochs [382,382] > 0 2024-03-30T15:02:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 3 client.57875.objecter handle_osd_map decoding full epoch 382 2024-03-30T15:02:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.901+0000 7f5cc6ffd700 20 client.57875.objecter dump_active .. 0 homeless 2024-03-30T15:02:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.902+0000 7f5cc6ffd700 10 client.57875.objecter ms_handle_connect 0x7f5cb0070520 2024-03-30T15:02:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:26.906+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5cbc056a50 con 0x7f5cc810aee0 2024-03-30T15:02:26.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:26.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:27.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.123+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5cb4005410 con 0x7f5cc810aee0 2024-03-30T15:02:27.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:27.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:27.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:27.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.271+0000 7f5cc6ffd700 1 -- 172.21.15.28:0/3069359444 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v383) ==== 91+0+0 (unknown 1003541544 0 0) 0x7f5cbc05a4a0 con 0x7f5cc810aee0 2024-03-30T15:02:27.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.275+0000 7f5cd0260700 10 librados: watch_flush enter 2024-03-30T15:02:27.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.275+0000 7f5cd0260700 10 librados: watch_flush exit 2024-03-30T15:02:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.275+0000 7f5cd0260700 20 client.57875.objecter shutdown clearing up homeless session... 2024-03-30T15:02:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.275+0000 7f5cd0260700 10 client.57875.objecter successfully canceled tick 2024-03-30T15:02:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.281+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5cb0070520 legacy=0x7f5cb00729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.281+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 >> v1:172.21.15.28:6790/0 conn(0x7f5cc810aee0 legacy=0x7f5cc81a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.281+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 shutdown_connections 2024-03-30T15:02:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.281+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 >> 172.21.15.28:0/3069359444 conn(0x7f5cc80fdb20 msgr2=0x7f5cc8107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.282+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 shutdown_connections 2024-03-30T15:02:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.282+0000 7f5cd0260700 1 -- 172.21.15.28:0/3069359444 wait complete. 2024-03-30T15:02:27.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:27.282+0000 7f5cd0260700 1 librados: shutdown 2024-03-30T15:02:27.284 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:02:27.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:27.377 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25081.0:2 37.6 37:602f83fe:::foo:head [create] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 272.177708564, 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092626624, 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25081", 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/317566051", 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197526+0000", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.86e-06 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197531+0000", 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2379999999999997e-06 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197542+0000", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0648e-05 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197552+0000", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0217000000000001e-05 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199081+0000", 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001528908 2024-03-30T15:02:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199091+0000", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6239999999999997e-06 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246480+0000", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.047389642 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246590+0000", 2024-03-30T15:02:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109821 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,3", 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246710+0000", 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120436 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256246+0000", 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095358879999999993 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267473+0000", 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011226707000000001 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290057+0000", 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022584090000000001 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290109+0000", 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2052000000000002e-05 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290151+0000", 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1492999999999997e-05 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24892.0:5 22.12 22:4a393722:test-rados-api-smithi028-93390-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 272.14194477900003, 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.057181524, 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24892", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1207320814", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233289+0000", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.091e-06 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233295+0000", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8749999999999997e-06 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233306+0000", 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0744000000000001e-05 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233313+0000", 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6140000000000003e-06 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233359+0000", 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6215999999999999e-05 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233366+0000", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0679999999999999e-06 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242316+0000", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0089497279999999 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242442+0000", 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012600999999999999 2024-03-30T15:02:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242608+0000", 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000165451 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259414+0000", 2024-03-30T15:02:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016806394999999998 2024-03-30T15:02:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259426+0000", 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2312e-05 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290387+0000", 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030960560000000002 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290434+0000", 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6702000000000001e-05 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290469+0000", 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5757999999999999e-05 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:4 23.2 23:4029b033:test-rados-api-smithi028-93408-2::nonexistent:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 272.134945587, 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99567974800000003, 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240288+0000", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.212e-06 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240293+0000", 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1719999999999998e-06 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240304+0000", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0149000000000001e-05 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240312+0000", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1480000000000006e-06 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241020+0000", 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070865900000000005 2024-03-30T15:02:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241028+0000", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9510000000000006e-06 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235868+0000", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99483949599999999 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235967+0000", 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8961000000000003e-05 2024-03-30T15:02:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25048.0:2 46.0s0 46:00000000::::head [pgnls start_epoch 116] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 269.94950726799999, 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2275935200000001, 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25048", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425727+0000", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8560000000000001e-06 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425733+0000", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6740000000000002e-06 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425742+0000", 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9740000000000008e-06 2024-03-30T15:02:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425750+0000", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6539999999999999e-06 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425815+0000", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5493999999999993e-05 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425818+0000", 2024-03-30T15:02:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.706e-06 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651104+0000", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2252865639999999 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651143+0000", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8489999999999999e-05 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.653319+0000", 2024-03-30T15:02:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021761079999999999 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2064 11.18 11:1888b8ad:test-rados-api-smithi028-93257-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 269.23469805600001, 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085813029999999998, 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2064 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140536+0000", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.023e-06 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140540+0000", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5650000000000003e-06 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140549+0000", 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.789e-06 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140555+0000", 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.947e-06 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140611+0000", 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5859999999999997e-05 2024-03-30T15:02:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:27.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140733+0000", 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121965 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.204443+0000", 2024-03-30T15:02:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063710599000000007 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.206091+0000", 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001648084 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226061+0000", 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019969810000000001 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226257+0000", 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019557099999999999 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226301+0000", 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4688000000000003e-05 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226348+0000", 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6128999999999998e-05 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2073 11.1e e116/110 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head v 116'8, mlcod=116'6)", 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 268.91212085199999, 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.121009771, 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:02:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463113+0000", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5859999999999999e-06 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471534+0000", 2024-03-30T15:02:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084207299999999995 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471550+0000", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5531000000000001e-05 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471557+0000", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5140000000000004e-06 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485095+0000", 2024-03-30T15:02:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013537693 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485121+0000", 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6554999999999999e-05 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584083+0000", 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.098961412999999998 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584122+0000", 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8748999999999998e-05 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25336.0:2 56.15 56:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 268.90421222800001, 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4325735549999998, 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25336", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/642826322", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471022+0000", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9400000000000001e-06 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471028+0000", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8690000000000001e-06 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471039+0000", 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0738e-05 2024-03-30T15:02:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471050+0000", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0791999999999999e-05 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.355117+0000", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.88406750999999995 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.355126+0000", 2024-03-30T15:02:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1200000000000008e-06 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875371+0000", 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5202449380000003 2024-03-30T15:02:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875483+0000", 2024-03-30T15:02:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011174499999999999 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875613+0000", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013038500000000001 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.880351+0000", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047371089999999998 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.887399+0000", 2024-03-30T15:02:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070482210000000003 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903507+0000", 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016108285999999999 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903554+0000", 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7107999999999997e-05 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903594+0000", 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9793999999999999e-05 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2074 11.15 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 268.74487629700002, 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.095916847999999999, 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2074 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630358+0000", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.623e-06 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666706+0000", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036348502999999997 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666726+0000", 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9771000000000001e-05 2024-03-30T15:02:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666738+0000", 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2181000000000001e-05 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669591+0000", 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028525410000000001 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669724+0000", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013316700000000001 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669879+0000", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015516999999999999 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.723327+0000", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053447407000000002 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.724496+0000", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001169123 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726138+0000", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016417020000000001 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726191+0000", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3597000000000002e-05 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726273+0000", 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2063000000000004e-05 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35393.0:4 55.1f 55:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:02:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 267.643099312, 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1634292589999999, 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35393", 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3403500224", 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732135+0000", 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1610000000000001e-06 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732140+0000", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9779999999999996e-06 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732152+0000", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2031000000000001e-05 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732160+0000", 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5549999999999997e-06 2024-03-30T15:02:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732204+0000", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3553999999999997e-05 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732209+0000", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7509999999999997e-06 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895502+0000", 2024-03-30T15:02:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1632922020000001 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895563+0000", 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1026999999999997e-05 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25342.0:3 57.f 57:f7483bc4:::src_object.2:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 267.64073364000001, 2024-03-30T15:02:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1724189570000001, 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25342", 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2592697076", 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734500+0000", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.381e-06 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734509+0000", 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0000000000000002e-06 2024-03-30T15:02:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734520+0000", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0373e-05 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734528+0000", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4200000000000007e-06 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734582+0000", 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3516000000000001e-05 2024-03-30T15:02:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734587+0000", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2009999999999998e-06 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.894986+0000", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1603986509999999 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895093+0000", 2024-03-30T15:02:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107148 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895229+0000", 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000136644 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.899876+0000", 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004646837 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.901346+0000", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001469459 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906825+0000", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054790519999999999 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906885+0000", 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0461000000000003e-05 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906918+0000", 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2814000000000001e-05 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35294.0:6 54.6 e118/117 54:602f83fe:::foo:head v 118'3, mlcod=118'1)", 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.730833+0000", 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 266.64439965600002, 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072366658, 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.730833+0000", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.730833+0000", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.730835+0000", 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.559e-06 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.730855+0000", 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0454000000000001e-05 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.787642+0000", 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056787048999999999 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.787653+0000", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0659000000000001e-05 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.797755+0000", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010102507 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.797778+0000", 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2898e-05 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.803166+0000", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053875019999999997 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.803200+0000", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4029999999999998e-05 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25501.0:2 65.c 65:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:02:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 266.385823985, 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9912078200000001, 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25501", 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3010623157", 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989412+0000", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2329999999999998e-06 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989425+0000", 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3040999999999999e-05 2024-03-30T15:02:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989439+0000", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4147e-05 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989452+0000", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2842e-05 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.516774+0000", 2024-03-30T15:02:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.52732199599999996 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.516784+0000", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0035e-05 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.933959+0000", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.417175176 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.934110+0000", 2024-03-30T15:02:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015060799999999999 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.934274+0000", 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000164468 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.942259+0000", 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079849589999999998 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.958103+0000", 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015843731 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980496+0000", 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022392689 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980572+0000", 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6538000000000005e-05 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980616+0000", 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4357000000000002e-05 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25450.0:2 60.6 60:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 265.56258022899999, 2024-03-30T15:02:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.093909628999999994, 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25450", 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4259934820", 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812654+0000", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9939999999999999e-06 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812662+0000", 2024-03-30T15:02:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1300000000000003e-06 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812671+0000", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6399999999999992e-06 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812682+0000", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.827604+0000", 2024-03-30T15:02:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014922308 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.827613+0000", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1759999999999999e-06 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.893746+0000", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.066132370999999995 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.893870+0000", 2024-03-30T15:02:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012424100000000001 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.894005+0000", 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013545199999999999 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895511+0000", 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015054510000000001 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895603+0000", 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1797000000000002e-05 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906478+0000", 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010874968 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906523+0000", 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5037000000000003e-05 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906562+0000", 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9456999999999998e-05 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:104 115.7s0 115:eb850916:::103:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 246.377767973, 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085521396999999999, 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997467+0000", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.65e-06 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997476+0000", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7739999999999993e-06 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997493+0000", 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6878999999999999e-05 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997503+0000", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0417e-05 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997580+0000", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6157999999999998e-05 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.998161+0000", 2024-03-30T15:02:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000581453 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.998600+0000", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043924900000000001 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.033062+0000", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034461749 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.082911+0000", 2024-03-30T15:02:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.049849088 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.082986+0000", 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4980000000000004e-05 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:256 115.6s0 115:7f07515b:::255:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.372576536, 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27174643399999998, 2024-03-30T15:02:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 256 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002658+0000", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2809999999999998e-06 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002767+0000", 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108225 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002785+0000", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7963999999999999e-05 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002796+0000", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1630999999999999e-05 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002903+0000", 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107241 2024-03-30T15:02:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002911+0000", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.605e-06 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074440+0000", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.071529408000000003 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074584+0000", 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014313800000000001 2024-03-30T15:02:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074986+0000", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000402488 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.137353+0000", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.062367140000000001 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.274320+0000", 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.136966596 2024-03-30T15:02:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.274403+0000", 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2717000000000001e-05 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:10 104.a e139/131 104:503bb488:test-rados-api-smithi028-93441-7::foo:head v 139'6, mlcod=136'3)", 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.190123903, 2024-03-30T15:02:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.100151402, 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0400000000000002e-07 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185114+0000", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0950000000000003e-06 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185121+0000", 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7469999999999999e-06 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185129+0000", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0949999999999996e-06 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.265981+0000", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080851965999999997 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.266007+0000", 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6166999999999999e-05 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285232+0000", 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019224465999999999 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285260+0000", 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8462e-05 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32242.0:2 120.6 e139/138 120:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-03-30T15:02:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.189870791, 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099712396999999994, 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5600000000000001e-07 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185367+0000", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4159999999999997e-06 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185373+0000", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1179999999999997e-06 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185379+0000", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0480000000000004e-06 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185623+0000", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024430699999999998 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185641+0000", 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7757999999999998e-05 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285042+0000", 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099401239000000002 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285074+0000", 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2155e-05 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32200.0:2 118.15 118:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.116253755, 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088700135999999999, 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32200", 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4179484271", 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258981+0000", 2024-03-30T15:02:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-06 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258991+0000", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0012e-05 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.259004+0000", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.242e-05 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.259014+0000", 2024-03-30T15:02:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6150000000000003e-06 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.268485+0000", 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0094716160000000004 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.268495+0000", 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.628e-06 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324078+0000", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055582950999999998 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324172+0000", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4473000000000002e-05 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324310+0000", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013811499999999999 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.331855+0000", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075448010000000003 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.331893+0000", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8108999999999997e-05 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347598+0000", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01570473 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347641+0000", 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3544000000000002e-05 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347679+0000", 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7533999999999997e-05 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2078 11.18 11:1888ddfb:test-rados-api-smithi028-93257-17::big.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 238.61817265400001, 2024-03-30T15:02:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14551070799999999, 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2078 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757062+0000", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6199999999999999e-06 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757071+0000", 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6010000000000001e-06 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757081+0000", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4860000000000003e-06 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757088+0000", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0899999999999999e-06 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757142+0000", 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3749000000000003e-05 2024-03-30T15:02:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757248+0000", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106406 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.884537+0000", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.127288966 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.887673+0000", 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003135826 2024-03-30T15:02:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902352+0000", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014678862000000001 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902473+0000", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121415 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902531+0000", 2024-03-30T15:02:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7527e-05 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902571+0000", 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0160000000000002e-05 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57050.0:2 190.6 190:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 200.16553398600001, 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075417256000000002, 2024-03-30T15:02:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57050", 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1148433196", 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209701+0000", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3800000000000001e-06 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209707+0000", 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9460000000000003e-06 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209718+0000", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0679000000000001e-05 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209729+0000", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0896999999999999e-05 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213086+0000", 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033575290000000002 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213092+0000", 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6489999999999996e-06 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220438+0000", 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073462149999999997 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220446+0000", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2770000000000005e-06 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272811+0000", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052364490999999999 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272945+0000", 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013451899999999999 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273074+0000", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012904 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273476+0000", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000401875 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.274847+0000", 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013706339999999999 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.284902+0000", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010055015 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285053+0000", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015113999999999999 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285116+0000", 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2970000000000002e-05 2024-03-30T15:02:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:27.503 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:02:27.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:27.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3069359444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[41614]: from='client.57875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:27.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T15:02:27.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3069359444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:27 smithi028 ceph-mon[29876]: from='client.57875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:27.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:27.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: osdmap e382: 8 total, 8 up, 8 in 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3069359444' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:27 smithi134 ceph-mon[33714]: from='client.57875 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:02:27.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:27.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:27.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:27.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:27.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:28.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:28.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:28.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:28.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: from='client.57875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: 101.2 scrub starts 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: 101.2 scrub ok 2024-03-30T15:02:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[29876]: pgmap v515: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: from='client.57875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: 101.2 scrub starts 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: 101.2 scrub ok 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-mon[41614]: pgmap v515: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:28.265 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:28.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:28.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:28.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:28.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:28.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:28.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:28.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi028-92999-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: from='client.57875 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: osdmap e383: 8 total, 8 up, 8 in 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:28.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: 101.2 scrub starts 2024-03-30T15:02:28.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: 101.2 scrub ok 2024-03-30T15:02:28.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:28 smithi134 ceph-mon[33714]: pgmap v515: 268 pgs: 40 unknown, 228 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:28.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:02:28.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:28.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:28.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:28.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:29.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:29.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:29.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:29.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:29.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:29.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:29.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:29.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: 101.5 scrub starts 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: 101.5 scrub ok 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: 101.1 scrub starts 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: 101.1 scrub ok 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T15:02:29.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: 101.5 scrub starts 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: 101.5 scrub ok 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: 101.1 scrub starts 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: 101.1 scrub ok 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 -- 172.21.15.28:0/3327715657 >> v1:172.21.15.28:6790/0 conn(0x7fdde010aef0 legacy=0x7fdde010d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 -- 172.21.15.28:0/3327715657 shutdown_connections 2024-03-30T15:02:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 -- 172.21.15.28:0/3327715657 >> 172.21.15.28:0/3327715657 conn(0x7fdde00fdb30 msgr2=0x7fdde00fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 -- 172.21.15.28:0/3327715657 shutdown_connections 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 -- 172.21.15.28:0/3327715657 wait complete. 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 librados: starting msgr at 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.575+0000 7fdde659d700 1 librados: starting objecter 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 Processor -- start 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 -- start start 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 librados: setting wanted keys 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 librados: calling monclient init 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdde01a9380 con 0x7fdde010aef0 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdde01aa5a0 con 0x7fdde010ea90 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdde659d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdde01ab7e0 con 0x7fdde0102260 2024-03-30T15:02:29.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdde0102260 0x7fdde01075a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57854/0 (socket says 172.21.15.28:57854) 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddffff700 1 -- 172.21.15.28:0/195971899 learned_addr learned my addr 172.21.15.28:0/195971899 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter ms_handle_connect 0x7fdde010aef0 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter ms_handle_connect 0x7fdde0102260 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:29.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter ms_handle_connect 0x7fdde010ea90 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.576+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423220898 0 0) 0x7fdde01a9380 con 0x7fdde010aef0 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddc80039c0 con 0x7fdde010aef0 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1350629689 0 0) 0x7fdde01ab7e0 con 0x7fdde0102260 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdde01a9380 con 0x7fdde0102260 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530492349 0 0) 0x7fdde01aa5a0 con 0x7fdde010ea90 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdde01ab7e0 con 0x7fdde010ea90 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2492958829 0 0) 0x7fddc80039c0 con 0x7fdde010aef0 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdde01aa5a0 con 0x7fdde010aef0 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fddd4001980 con 0x7fdde010aef0 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2452576107 0 0) 0x7fdde01a9380 con 0x7fdde0102260 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddc80039c0 con 0x7fdde0102260 2024-03-30T15:02:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2041982728 0 0) 0x7fdde01ab7e0 con 0x7fdde010ea90 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdde01a9380 con 0x7fdde010ea90 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fddd0001840 con 0x7fdde0102260 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fddd8001bb0 con 0x7fdde010ea90 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3241796301 0 0) 0x7fdde01aa5a0 con 0x7fdde010aef0 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 >> v1:172.21.15.28:6790/0 conn(0x7fdde0102260 legacy=0x7fdde01075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 >> v1:172.21.15.134:6789/0 conn(0x7fdde010ea90 legacy=0x7fdde01a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde01aca20 con 0x7fdde010aef0 2024-03-30T15:02:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdde01aba10 con 0x7fdde010aef0 2024-03-30T15:02:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdde659d700 10 client.24699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdde01abed0 con 0x7fdde010aef0 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdde659d700 1 librados: init done 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fddd4001c80 con 0x7fdde010aef0 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.577+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fddd4004cd0 con 0x7fdde010aef0 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.578+0000 7fdde659d700 10 client.24699.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.578+0000 7fdde659d700 15 client.24699.objecter _session_command_op_assign -1 1 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.578+0000 7fdde659d700 20 client.24699.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.578+0000 7fdde659d700 10 client.24699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.578+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fddcc002920 con 0x7fdde010aef0 2024-03-30T15:02:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:29.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.579+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fddd4003710 con 0x7fdde010aef0 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.579+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(385..385 src has 53..385) ==== 8949+0+0 (unknown 532681972 0 0) 0x7fddd40888e0 con 0x7fdde010aef0 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.579+0000 7fdddd7fa700 10 client.24699.objecter ms_dispatch 0x7fdde0001550 osd_map(385..385 src has 53..385) 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.579+0000 7fdddd7fa700 3 client.24699.objecter handle_osd_map got epochs [385,385] > 0 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.579+0000 7fdddd7fa700 3 client.24699.objecter handle_osd_map decoding full epoch 385 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter checking command 1 2024-03-30T15:02:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter _get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 15 client.24699.objecter _session_command_op_remove -1 1 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter _get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 15 client.24699.objecter _session_command_op_assign 1 1 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 4 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter _send_command 1 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6805/1850313547 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fddd4004050 con 0x7fddc807f3b0 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter dump_active .. 0 homeless 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=385) ==== 24+0+0 (unknown 2141971694 0 0) 0x7fddd4088cf0 con 0x7fdde010aef0 2024-03-30T15:02:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter ms_handle_connect 0x7fddc80703a0 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter ms_handle_connect 0x7fddc807f3b0 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fddd4004050 con 0x7fddc807f3b0 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter ms_dispatch 0x7fdde0001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 10 client.24699.objecter _finish_command 1 = system:0 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 4 2024-03-30T15:02:29.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.580+0000 7fdddd7fa700 15 client.24699.objecter _session_command_op_remove 1 1 2024-03-30T15:02:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 client.24699.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 15 client.24699.objecter _session_command_op_assign -1 2 2024-03-30T15:02:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter _get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter _get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 15 client.24699.objecter _session_command_op_remove -1 2 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter get_session s=0x7fddc807ed00 osd=1 3 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 15 client.24699.objecter _session_command_op_assign 1 2 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 4 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 client.24699.objecter _send_command 2 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fddcc0011f0 con 0x7fddc807f3b0 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdddd7fa700 1 -- 172.21.15.28:0/195971899 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fddcc0011f0 con 0x7fddc807f3b0 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdddd7fa700 10 client.24699.objecter ms_dispatch 0x7fdde0001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdddd7fa700 10 client.24699.objecter _finish_command 2 = system:0 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdddd7fa700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 4 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdddd7fa700 15 client.24699.objecter _session_command_op_remove 1 2 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 librados: watch_flush enter 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 librados: watch_flush exit 2024-03-30T15:02:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 client.24699.objecter close_session for osd.1 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 >> v1:172.21.15.28:6805/1850313547 conn(0x7fddc807f3b0 legacy=0x7fddc8081850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter put_session s=0x7fddc807ed00 osd=1 1 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 20 client.24699.objecter shutdown clearing up homeless session... 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 10 client.24699.objecter successfully canceled tick 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 >> v1:172.21.15.28:6800/3011431093 conn(0x7fddc80703a0 legacy=0x7fddc8072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 >> v1:172.21.15.28:6789/0 conn(0x7fdde010aef0 legacy=0x7fdde01a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.600+0000 7fdde4b3a700 1 -- 172.21.15.28:0/195971899 reap_dead start 2024-03-30T15:02:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.601+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 shutdown_connections 2024-03-30T15:02:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.601+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 >> 172.21.15.28:0/195971899 conn(0x7fdde00fdb30 msgr2=0x7fdde0111be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.601+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 shutdown_connections 2024-03-30T15:02:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.601+0000 7fdde659d700 1 -- 172.21.15.28:0/195971899 wait complete. 2024-03-30T15:02:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:29.601+0000 7fdde659d700 1 librados: shutdown 2024-03-30T15:02:29.618 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: 101.5 scrub starts 2024-03-30T15:02:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: 101.5 scrub ok 2024-03-30T15:02:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: 101.1 scrub starts 2024-03-30T15:02:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: 101.1 scrub ok 2024-03-30T15:02:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: osdmap e384: 8 total, 8 up, 8 in 2024-03-30T15:02:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]: dispatch 2024-03-30T15:02:29.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:29.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:29.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:29.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:29.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:29.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:30.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:30.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:30.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:30.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:30.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:30.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:30.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-30T15:02:30.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:30.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:30.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:30.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.16 scrub starts 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.16 scrub ok 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.7 scrub starts 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.7 scrub ok 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:30.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]': finished 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.10 scrub starts 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: 101.10 scrub ok 2024-03-30T15:02:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[41614]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.16 scrub starts 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.16 scrub ok 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.7 scrub starts 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.7 scrub ok 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]': finished 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.10 scrub starts 2024-03-30T15:02:30.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: 101.10 scrub ok 2024-03-30T15:02:30.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:30 smithi028 ceph-mon[29876]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:30.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.16 scrub starts 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.16 scrub ok 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.7 scrub starts 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.7 scrub ok 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-49", "mode": "readproxy"}]': finished 2024-03-30T15:02:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: osdmap e385: 8 total, 8 up, 8 in 2024-03-30T15:02:30.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:30.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.10 scrub starts 2024-03-30T15:02:30.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: 101.10 scrub ok 2024-03-30T15:02:30.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:30 smithi134 ceph-mon[33714]: pgmap v518: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:30.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:30.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:30.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:30.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:30.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:31.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:31.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:31.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:31.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:02:31.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:31.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:31.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:02:31.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:02:31.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:31 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:02:31.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:31.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: osdmap e386: 8 total, 8 up, 8 in 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/714138788' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]: dispatch 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:02:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:31 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:02:31.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:31.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.828 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:31.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:31.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:31.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:32.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:32.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:32.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:32.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:32.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:32.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:32.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:32.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 101.1c scrub starts 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 101.1c scrub ok 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:02:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 101.1c scrub starts 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 101.1c scrub ok 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T15:02:32.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:32.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T15:02:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T15:02:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 101.1c scrub starts 2024-03-30T15:02:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 101.1c scrub ok 2024-03-30T15:02:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.57869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi028-92999-63"}]': finished 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: osdmap e387: 8 total, 8 up, 8 in 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: pgmap v521: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi028-92999-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: osdmap e388: 8 total, 8 up, 8 in 2024-03-30T15:02:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:32.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:32.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:32.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:32.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:32.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:32.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:32.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:33.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:33.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:33.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:33.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:33.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:33.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:33.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:33.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:33.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:33.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:33.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:33.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:33.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 101.e scrub starts 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 101.e scrub ok 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 101.1b scrub starts 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 101.1b scrub ok 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 101.e scrub starts 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 101.e scrub ok 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 101.1b scrub starts 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 101.1b scrub ok 2024-03-30T15:02:33.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T15:02:33.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:33 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T15:02:33.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:33.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 101.e scrub starts 2024-03-30T15:02:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 101.e scrub ok 2024-03-30T15:02:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 101.1b scrub starts 2024-03-30T15:02:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 101.1b scrub ok 2024-03-30T15:02:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T15:02:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:33 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T15:02:33.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:33.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:33.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:34.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:34.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:34.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:34.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:34.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:34.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:34.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:34.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:34.588 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:34.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[41614]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:34.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:34 smithi028 ceph-mon[29876]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T15:02:34.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:34.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:34.733 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:02:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:02:34.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:34.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:34.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:34.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:34.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:02:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:02:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:02:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: osdmap e389: 8 total, 8 up, 8 in 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: pgmap v524: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi028-92999-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:34 smithi134 ceph-mon[33714]: osdmap e390: 8 total, 8 up, 8 in 2024-03-30T15:02:34.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:35.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:35.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:35.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:35.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:35.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:35.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:35.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.506+0000 7f403701e700 1 -- 172.21.15.28:0/1878683488 >> v1:172.21.15.28:6790/0 conn(0x7f403010aed0 legacy=0x7f403010d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.506+0000 7f403701e700 1 -- 172.21.15.28:0/1878683488 shutdown_connections 2024-03-30T15:02:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.506+0000 7f403701e700 1 -- 172.21.15.28:0/1878683488 >> 172.21.15.28:0/1878683488 conn(0x7f40300fdaf0 msgr2=0x7f40300fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.506+0000 7f403701e700 1 -- 172.21.15.28:0/1878683488 shutdown_connections 2024-03-30T15:02:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.506+0000 7f403701e700 1 -- 172.21.15.28:0/1878683488 wait complete. 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 librados: starting msgr at 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 librados: starting objecter 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 Processor -- start 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 -- start start 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 librados: setting wanted keys 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 librados: calling monclient init 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40301a7120 con 0x7f4030102240 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40301a8340 con 0x7f403010aed0 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.507+0000 7f403701e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40301a9580 con 0x7f403010ea70 2024-03-30T15:02:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f40355bb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f403010ea70 0x7f40301a57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57934/0 (socket says 172.21.15.28:57934) 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f40355bb700 1 -- 172.21.15.28:0/1544248853 learned_addr learned my addr 172.21.15.28:0/1544248853 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter ms_handle_connect 0x7f403010ea70 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter ms_handle_connect 0x7f403010aed0 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter ms_handle_connect 0x7f4030102240 2024-03-30T15:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.508+0000 7f402dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3457778935 0 0) 0x7f40301a7120 con 0x7f4030102240 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4020003aa0 con 0x7f4030102240 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183045451 0 0) 0x7f40301a8340 con 0x7f403010aed0 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40301a7120 con 0x7f403010aed0 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133811754 0 0) 0x7f40301a9580 con 0x7f403010ea70 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40301a8340 con 0x7f403010ea70 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 598486088 0 0) 0x7f40301a8340 con 0x7f403010ea70 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40301a9580 con 0x7f403010ea70 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4028001bb0 con 0x7f403010ea70 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1578378833 0 0) 0x7f4020003aa0 con 0x7f4030102240 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40301a8340 con 0x7f4030102240 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 931074880 0 0) 0x7f40301a7120 con 0x7f403010aed0 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4020003aa0 con 0x7f403010aed0 2024-03-30T15:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f40240010f0 con 0x7f4030102240 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f401c004600 con 0x7f403010aed0 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2477918749 0 0) 0x7f40301a9580 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 >> v1:172.21.15.134:6789/0 conn(0x7f403010aed0 legacy=0x7f40301a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 >> v1:172.21.15.28:6789/0 conn(0x7f4030102240 legacy=0x7f40301075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40301aa7c0 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40301a8510 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4028003ec0 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.511+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4028005f70 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.512+0000 7f403701e700 10 client.57899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.512+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40301a8970 con 0x7f403010ea70 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.512+0000 7f403701e700 1 librados: init done 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f403701e700 10 client.57899.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f403701e700 15 client.57899.objecter _session_command_op_assign -1 1 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f403701e700 20 client.57899.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:35.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f403701e700 10 client.57899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4030103720 con 0x7f403010ea70 2024-03-30T15:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.513+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f4028003a20 con 0x7f403010ea70 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(391..391 src has 53..391) ==== 9370+0+0 (unknown 739119937 0 0) 0x7f4028089ec0 con 0x7f403010ea70 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 10 client.57899.objecter ms_dispatch 0x7f4030001550 osd_map(391..391 src has 53..391) 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 3 client.57899.objecter handle_osd_map got epochs [391,391] > 0 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 3 client.57899.objecter handle_osd_map decoding full epoch 391 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 10 client.57899.objecter checking command 1 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter _get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 15 client.57899.objecter _session_command_op_remove -1 1 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter _get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 15 client.57899.objecter _session_command_op_assign 2 1 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 4 2024-03-30T15:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 10 client.57899.objecter _send_command 1 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6809/3451325043 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4020083aa0 con 0x7f402007fd90 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 20 client.57899.objecter dump_active .. 0 homeless 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=391) ==== 24+0+0 (unknown 1482368264 0 0) 0x7f402808a320 con 0x7f403010ea70 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 10 client.57899.objecter ms_handle_connect 0x7f4020070590 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.514+0000 7f402dffb700 10 client.57899.objecter ms_handle_connect 0x7f402007fd90 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.515+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4020083aa0 con 0x7f402007fd90 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.515+0000 7f402dffb700 10 client.57899.objecter ms_dispatch 0x7f4030001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.515+0000 7f402dffb700 10 client.57899.objecter _finish_command 1 = system:0 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.515+0000 7f402dffb700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 4 2024-03-30T15:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.515+0000 7f402dffb700 15 client.57899.objecter _session_command_op_remove 2 1 2024-03-30T15:02:35.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 10 client.57899.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 15 client.57899.objecter _session_command_op_assign -1 2 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 20 client.57899.objecter _get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 20 client.57899.objecter _get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 15 client.57899.objecter _session_command_op_remove -1 2 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 20 client.57899.objecter get_session s=0x7f402007f6e0 osd=2 3 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 15 client.57899.objecter _session_command_op_assign 2 2 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 4 2024-03-30T15:02:35.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 10 client.57899.objecter _send_command 2 2024-03-30T15:02:35.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 --> v1:172.21.15.28:6809/3451325043 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f4030068b30 con 0x7f402007fd90 2024-03-30T15:02:35.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f402dffb700 1 -- 172.21.15.28:0/1544248853 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f4030068b30 con 0x7f402007fd90 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f402dffb700 10 client.57899.objecter ms_dispatch 0x7f4030001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f402dffb700 10 client.57899.objecter _finish_command 2 = system:0 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f402dffb700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 4 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.536+0000 7f402dffb700 15 client.57899.objecter _session_command_op_remove 2 2 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 10 librados: watch_flush enter 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 10 librados: watch_flush exit 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 10 client.57899.objecter close_session for osd.2 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 >> v1:172.21.15.28:6809/3451325043 conn(0x7f402007fd90 legacy=0x7f4020082230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 20 client.57899.objecter put_session s=0x7f402007f6e0 osd=2 1 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 20 client.57899.objecter shutdown clearing up homeless session... 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 10 client.57899.objecter successfully canceled tick 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4020070590 legacy=0x7f4020072a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 >> v1:172.21.15.28:6790/0 conn(0x7f403010ea70 legacy=0x7f40301a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f40355bb700 1 -- 172.21.15.28:0/1544248853 reap_dead start 2024-03-30T15:02:35.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 shutdown_connections 2024-03-30T15:02:35.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 >> 172.21.15.28:0/1544248853 conn(0x7f40300fdaf0 msgr2=0x7f4030111bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:35.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 shutdown_connections 2024-03-30T15:02:35.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 -- 172.21.15.28:0/1544248853 wait complete. 2024-03-30T15:02:35.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:35.537+0000 7f403701e700 1 librados: shutdown 2024-03-30T15:02:35.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.561+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/4152619359 >> v1:172.21.15.28:6789/0 conn(0x7f2e540745e0 legacy=0x7f2e540749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.561+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/4152619359 shutdown_connections 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.561+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/4152619359 >> 172.21.15.28:0/4152619359 conn(0x7f2e540fdb00 msgr2=0x7f2e540fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/4152619359 shutdown_connections 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/4152619359 wait complete. 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 librados: starting msgr at 2024-03-30T15:02:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 librados: starting objecter 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 Processor -- start 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- start start 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 librados: setting wanted keys 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 librados: calling monclient init 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541ad4b0 con 0x7f2e540745e0 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541ae6d0 con 0x7f2e541151d0 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e5b4a7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2e541af910 con 0x7f2e540750d0 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e59243700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2e540745e0 0x7f2e54114760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38536/0 (socket says 172.21.15.28:38536) 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e58a42700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2e540750d0 0x7f2e541a84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57904/0 (socket says 172.21.15.28:57904) 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.562+0000 7f2e59243700 1 -- 172.21.15.28:0/952862960 learned_addr learned my addr 172.21.15.28:0/952862960 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter ms_handle_connect 0x7f2e540745e0 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter ms_handle_connect 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter ms_handle_connect 0x7f2e541151d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438374874 0 0) 0x7f2e541af910 con 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e3c003aa0 con 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125340918 0 0) 0x7f2e541ad4b0 con 0x7f2e540745e0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e541af910 con 0x7f2e540745e0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928308689 0 0) 0x7f2e541ae6d0 con 0x7f2e541151d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e541ad4b0 con 0x7f2e541151d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574031058 0 0) 0x7f2e3c003aa0 con 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e541ae6d0 con 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e48001830 con 0x7f2e540750d0 2024-03-30T15:02:35.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3953731829 0 0) 0x7f2e541af910 con 0x7f2e540745e0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e3c003aa0 con 0x7f2e540745e0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e44004860 con 0x7f2e540745e0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1172844139 0 0) 0x7f2e541ad4b0 con 0x7f2e541151d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e541af910 con 0x7f2e541151d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e4c001210 con 0x7f2e541151d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3863861679 0 0) 0x7f2e541ae6d0 con 0x7f2e540750d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.134:6789/0 conn(0x7f2e541151d0 legacy=0x7f2e541abc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6789/0 conn(0x7f2e540745e0 legacy=0x7f2e54114760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541b0b50 con 0x7f2e540750d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541afb40 con 0x7f2e540750d0 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e5b4a7700 10 client.57893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2e541b0000 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e5b4a7700 1 librados: init done 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2e48003b40 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.563+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2e48005c00 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.564+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2e40005100 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.565+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2e480036a0 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.565+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(390..390 src has 53..390) ==== 9370+0+0 (unknown 2639117728 0 0) 0x7f2e48089bc0 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.565+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 osd_map(390..390 src has 53..390) 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.565+0000 7f2e527fc700 3 client.57893.objecter handle_osd_map got epochs [390,390] > 0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.565+0000 7f2e527fc700 3 client.57893.objecter handle_osd_map decoding full epoch 390 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e527fc700 20 client.57893.objecter dump_active .. 0 homeless 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v390) ==== 54+0+16 (unknown 3927696887 0 2420000192) 0x7f2e4808a000 con 0x7f2e540750d0 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e3c070520 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 1 2024-03-30T15:02:35.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 1 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 0 1 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 1 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e40004b50 con 0x7f2e40004730 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.566+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e40004730 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.567+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40004b50 con 0x7f2e40004730 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.567+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:35.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.567+0000 7f2e527fc700 10 client.57893.objecter _finish_command 1 = system:0 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.567+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.567+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 0 1 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 2 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 2 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e400053e0 osd=0 3 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 0 2 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 2 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.584+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40007b00 con 0x7f2e40004730 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.585+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40007b00 con 0x7f2e40004730 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.585+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.585+0000 7f2e527fc700 10 client.57893.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.585+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 4 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.585+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 0 2 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 1 3 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 4 2024-03-30T15:02:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 3 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2e40002200 con 0x7f2e400042f0 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.586+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e400042f0 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.587+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40002200 con 0x7f2e400042f0 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.587+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.587+0000 7f2e527fc700 10 client.57893.objecter _finish_command 3 = system:0 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.587+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 4 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.587+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 1 3 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 4 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40004180 osd=1 3 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 1 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.604+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40001660 con 0x7f2e400042f0 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.605+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40001660 con 0x7f2e400042f0 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.605+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.605+0000 7f2e527fc700 10 client.57893.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.605+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.605+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 1 4 2024-03-30T15:02:35.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 5 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 5 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 2 5 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 4 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 5 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2e400111f0 con 0x7f2e4000d6e0 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.606+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e4000d6e0 2024-03-30T15:02:35.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.607+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e400111f0 con 0x7f2e4000d6e0 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.607+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.607+0000 7f2e527fc700 10 client.57893.objecter _finish_command 5 = system:0 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.607+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 4 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.607+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 2 5 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 6 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 6 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40008480 osd=2 3 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 2 6 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 4 2024-03-30T15:02:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 6 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.623+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40007fc0 con 0x7f2e4000d6e0 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40007fc0 con 0x7f2e4000d6e0 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e527fc700 10 client.57893.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 4 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 2 6 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 7 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 7 2024-03-30T15:02:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 3 7 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 4 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 7 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.625+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2e40016030 con 0x7f2e40012520 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:35.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e40012520 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40016030 con 0x7f2e40012520 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 10 client.57893.objecter _finish_command 7 = system:0 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 4 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.626+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 3 7 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 8 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 8 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000cd30 osd=3 3 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 3 8 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 4 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 8 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.644+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4000c810 con 0x7f2e40012520 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.645+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4000c810 con 0x7f2e40012520 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.645+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.645+0000 7f2e527fc700 10 client.57893.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.645+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 4 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.645+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 3 8 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 9 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 9 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 4 9 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 4 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 9 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2e4001a9b0 con 0x7f2e40012080 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.646+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e40012080 2024-03-30T15:02:35.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.647+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e4001a9b0 con 0x7f2e40012080 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.647+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.647+0000 7f2e527fc700 10 client.57893.objecter _finish_command 9 = system:0 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.647+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 4 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.647+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 4 9 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 10 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 10 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000c810 osd=4 3 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 4 10 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 4 2024-03-30T15:02:35.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 10 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.664+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4000cad0 con 0x7f2e40012080 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.665+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4000cad0 con 0x7f2e40012080 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.665+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.665+0000 7f2e527fc700 10 client.57893.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.665+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 4 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.665+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 4 10 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 11 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 11 2024-03-30T15:02:35.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 5 11 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 4 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 11 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2e4001c170 con 0x7f2e40016800 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.666+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e40016800 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.667+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e48056910 con 0x7f2e40016800 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.667+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.667+0000 7f2e527fc700 10 client.57893.objecter _finish_command 11 = system:0 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.667+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 4 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.667+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 5 11 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 12 2024-03-30T15:02:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 12 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e4000cad0 osd=5 3 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 5 12 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 4 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 12 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.684+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40016df0 con 0x7f2e40016800 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40016df0 con 0x7f2e40016800 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e527fc700 10 client.57893.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 4 2024-03-30T15:02:35.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 5 12 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 13 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 13 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 6 13 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 4 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 13 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.686+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2e40023560 con 0x7f2e4001fa50 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e4001fa50 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e40023560 con 0x7f2e4001fa50 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 10 client.57893.objecter _finish_command 13 = system:0 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 4 2024-03-30T15:02:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.687+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 6 13 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 14 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 14 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40016df0 osd=6 3 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 6 14 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 4 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 14 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.704+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e40016280 con 0x7f2e4001fa50 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e40016280 con 0x7f2e4001fa50 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e527fc700 10 client.57893.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 4 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 6 14 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 15 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 15 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 7 15 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 4 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 15 2024-03-30T15:02:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.706+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2e4001d080 con 0x7f2e4001cb00 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.707+0000 7f2e527fc700 10 client.57893.objecter ms_handle_connect 0x7f2e4001cb00 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.708+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2e4001d080 con 0x7f2e4001cb00 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.708+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.708+0000 7f2e527fc700 10 client.57893.objecter _finish_command 15 = system:0 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.708+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 4 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.708+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 7 15 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 10 client.57893.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign -1 16 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 20 client.57893.objecter _get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_remove -1 16 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 20 client.57893.objecter get_session s=0x7f2e40011400 osd=7 3 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 15 client.57893.objecter _session_command_op_assign 7 16 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 4 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 10 client.57893.objecter _send_command 16 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.725+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2e4001d300 con 0x7f2e4001cb00 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e527fc700 1 -- 172.21.15.28:0/952862960 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2e4001d300 con 0x7f2e4001cb00 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e527fc700 10 client.57893.objecter ms_dispatch 0x7f2e54001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e527fc700 10 client.57893.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e527fc700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 4 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e527fc700 15 client.57893.objecter _session_command_op_remove 7 16 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e5b4a7700 10 librados: watch_flush enter 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e5b4a7700 10 librados: watch_flush exit 2024-03-30T15:02:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.0 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.727+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6801/3394323575 conn(0x7f2e40004730 legacy=0x7f2e40002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e400053e0 osd=0 1 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.1 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6805/1850313547 conn(0x7f2e400042f0 legacy=0x7f2e4000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40004180 osd=1 1 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.2 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6809/3451325043 conn(0x7f2e4000d6e0 legacy=0x7f2e4000fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40008480 osd=2 1 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.3 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6813/1506157250 conn(0x7f2e40012520 legacy=0x7f2e40014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cd30 osd=3 1 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.4 2024-03-30T15:02:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.134:6800/3063775514 conn(0x7f2e40012080 legacy=0x7f2e400191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000c810 osd=4 1 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.5 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.134:6804/3668151179 conn(0x7f2e40016800 legacy=0x7f2e4001a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e4000cad0 osd=5 1 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.6 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.134:6808/2135224194 conn(0x7f2e4001fa50 legacy=0x7f2e40021e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40016df0 osd=6 1 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter close_session for osd.7 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.134:6812/3758190298 conn(0x7f2e4001cb00 legacy=0x7f2e400266f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter put_session s=0x7f2e40011400 osd=7 1 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 20 client.57893.objecter shutdown clearing up homeless session... 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 10 client.57893.objecter successfully canceled tick 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2e3c070520 legacy=0x7f2e3c0729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e59a44700 1 -- 172.21.15.28:0/952862960 reap_dead start 2024-03-30T15:02:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e59a44700 1 -- 172.21.15.28:0/952862960 reap_dead start 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> v1:172.21.15.28:6790/0 conn(0x7f2e540750d0 legacy=0x7f2e541a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e59a44700 1 -- 172.21.15.28:0/952862960 reap_dead start 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e59a44700 1 -- 172.21.15.28:0/952862960 reap_dead start 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e59a44700 1 -- 172.21.15.28:0/952862960 reap_dead start 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 shutdown_connections 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 >> 172.21.15.28:0/952862960 conn(0x7f2e540fdb00 msgr2=0x7f2e540ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 shutdown_connections 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 -- 172.21.15.28:0/952862960 wait complete. 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:34.728+0000 7f2e5b4a7700 1 librados: shutdown 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:35.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:35.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: 42.1d scrub starts 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: 42.1d scrub ok 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/952862960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:02:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[41614]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: 42.1d scrub starts 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: 42.1d scrub ok 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/952862960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:02:35.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:02:35.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:35.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:35.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:35 smithi028 ceph-mon[29876]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T15:02:35.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:35.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:35.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:35.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: 42.1d scrub starts 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: 42.1d scrub ok 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/952862960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:35 smithi134 ceph-mon[33714]: osdmap e391: 8 total, 8 up, 8 in 2024-03-30T15:02:36.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:36.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:36.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:36.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:36.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:36.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:36.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:36.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:36.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:36.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:36.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:36.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:02:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:02:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:02:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 101.11 scrub starts 2024-03-30T15:02:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 101.11 scrub ok 2024-03-30T15:02:36.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: pgmap v527: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 101.11 scrub starts 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 101.11 scrub ok 2024-03-30T15:02:36.782 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: pgmap v527: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T15:02:36.783 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:36.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:36.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:36.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 101.11 scrub starts 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 101.11 scrub ok 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: pgmap v527: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:02:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:02:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:02:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: osdmap e392: 8 total, 8 up, 8 in 2024-03-30T15:02:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:36.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:37.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:37.062 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:37.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:37.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-30T15:02:37.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:37.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:37.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:37.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:37.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:37.711 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:37.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:37.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: 101.6 scrub starts 2024-03-30T15:02:37.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: 101.6 scrub ok 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: 101.6 scrub starts 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: 101.6 scrub ok 2024-03-30T15:02:37.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T15:02:37.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: 101.6 scrub starts 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: 101.6 scrub ok 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:02:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:37.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: osdmap e393: 8 total, 8 up, 8 in 2024-03-30T15:02:37.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]: dispatch 2024-03-30T15:02:37.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:37.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:37.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:38.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:38.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:38.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:38.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:38.385 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:38.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:38.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:38.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:38.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 42.d scrub starts 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 42.d scrub ok 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: pgmap v530: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:38.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T15:02:38.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 42.d scrub starts 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 42.d scrub ok 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: pgmap v530: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:38 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:38.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:38.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 42.d scrub starts 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 42.d scrub ok 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: pgmap v530: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1623963110' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi028-92999-64"}]': finished 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: osdmap e394: 8 total, 8 up, 8 in 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:38.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.833 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:38 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:38.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:38.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:38.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:38.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:39.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:39.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:39.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:39.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:39.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:39.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:39.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:39.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:02:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:39.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:39.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:39.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:39.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:40.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.112 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:40.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:40.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:40.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:40.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:40.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 101.17 scrub starts 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 101.17 scrub ok 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:02:40.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[41614]: pgmap v533: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 101.17 scrub starts 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 101.17 scrub ok 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:02:40.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:02:40.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:40 smithi028 ceph-mon[29876]: pgmap v533: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:40.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:40.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:40.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:40.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:40.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:02:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 101.17 scrub starts 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 101.17 scrub ok 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi028-92999-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: osdmap e395: 8 total, 8 up, 8 in 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:40.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:40 smithi134 ceph-mon[33714]: pgmap v533: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:40.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:40.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:40.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:40.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:40.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:41.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:41.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:41.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:41.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:41.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:41.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:41.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 101.b scrub starts 2024-03-30T15:02:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 101.b scrub ok 2024-03-30T15:02:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:02:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:02:41.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 101.18 scrub starts 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 101.18 scrub ok 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[29876]: 3.b scrub ok 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 101.b scrub starts 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 101.b scrub ok 2024-03-30T15:02:41.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 101.18 scrub starts 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 101.18 scrub ok 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T15:02:41.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:41 smithi028 ceph-mon[41614]: 3.b scrub ok 2024-03-30T15:02:41.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:41.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 101.b scrub starts 2024-03-30T15:02:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 101.b scrub ok 2024-03-30T15:02:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:02:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:02:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 101.18 scrub starts 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 101.18 scrub ok 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: osdmap e396: 8 total, 8 up, 8 in 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T15:02:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:41 smithi134 ceph-mon[33714]: 3.b scrub ok 2024-03-30T15:02:41.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:41.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:41.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:41.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:41.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.967+0000 7fca5b91e700 1 -- 172.21.15.28:0/4094660295 >> v1:172.21.15.28:6790/0 conn(0x7fca54109dc0 legacy=0x7fca5410c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.967+0000 7fca5b91e700 1 -- 172.21.15.28:0/4094660295 shutdown_connections 2024-03-30T15:02:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.967+0000 7fca5b91e700 1 -- 172.21.15.28:0/4094660295 >> 172.21.15.28:0/4094660295 conn(0x7fca540fc9e0 msgr2=0x7fca540fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.967+0000 7fca5b91e700 1 -- 172.21.15.28:0/4094660295 shutdown_connections 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.967+0000 7fca5b91e700 1 -- 172.21.15.28:0/4094660295 wait complete. 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 librados: starting msgr at 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 librados: starting objecter 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 Processor -- start 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 -- start start 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 librados: setting wanted keys 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 librados: calling monclient init 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca540fea60 con 0x7fca54109dc0 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca540fec30 con 0x7fca54101130 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca5b91e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca541ab940 con 0x7fca5410d960 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca59ebb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fca5410d960 0x7fca541a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42026/0 (socket says 172.21.15.28:42026) 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca59ebb700 1 -- 172.21.15.28:0/1954523973 learned_addr learned my addr 172.21.15.28:0/1954523973 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:41.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca58eb9700 1 --1- 172.21.15.28:0/1954523973 >> v1:172.21.15.28:6789/0 conn(0x7fca54109dc0 0x7fca540fd5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38822/0 (socket says 172.21.15.28:38822) 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca5410d960 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca54109dc0 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter ms_handle_connect 0x7fca54101130 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142432854 0 0) 0x7fca541ab940 con 0x7fca5410d960 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca400031f0 con 0x7fca5410d960 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.968+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690756527 0 0) 0x7fca540fea60 con 0x7fca54109dc0 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca541ab940 con 0x7fca54109dc0 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555465977 0 0) 0x7fca540fec30 con 0x7fca54101130 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca540fea60 con 0x7fca54101130 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1266733542 0 0) 0x7fca400031f0 con 0x7fca5410d960 2024-03-30T15:02:41.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca540fec30 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fca4c001c50 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 355678409 0 0) 0x7fca541ab940 con 0x7fca54109dc0 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca400031f0 con 0x7fca54109dc0 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fca50001410 con 0x7fca54109dc0 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 654009275 0 0) 0x7fca540fea60 con 0x7fca54101130 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca541ab940 con 0x7fca54101130 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fca44001830 con 0x7fca54101130 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3595072127 0 0) 0x7fca540fec30 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 >> v1:172.21.15.134:6789/0 conn(0x7fca54101130 legacy=0x7fca540fce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 >> v1:172.21.15.28:6789/0 conn(0x7fca54109dc0 legacy=0x7fca540fd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca541acb20 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca541abb10 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca5b91e700 10 client.57911.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca541abfd0 con 0x7fca5410d960 2024-03-30T15:02:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca5b91e700 1 librados: init done 2024-03-30T15:02:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fca4c003ec0 con 0x7fca5410d960 2024-03-30T15:02:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.969+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fca4c006070 con 0x7fca5410d960 2024-03-30T15:02:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.970+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38005120 con 0x7fca5410d960 2024-03-30T15:02:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fca4c003a20 con 0x7fca5410d960 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(397..397 src has 53..397) ==== 9330+0+0 (unknown 2126965920 0 0) 0x7fca4c089df0 con 0x7fca5410d960 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 10 client.57911.objecter ms_dispatch 0x7fca54001550 osd_map(397..397 src has 53..397) 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 3 client.57911.objecter handle_osd_map got epochs [397,397] > 0 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 3 client.57911.objecter handle_osd_map decoding full epoch 397 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 20 client.57911.objecter dump_active .. 0 homeless 2024-03-30T15:02:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.971+0000 7fca4a7fc700 10 client.57911.objecter ms_handle_connect 0x7fca400705f0 2024-03-30T15:02:41.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:41.976+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fca4c056b50 con 0x7fca5410d960 2024-03-30T15:02:41.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:42.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:42.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:02:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:02:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.205+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fca38005410 con 0x7fca5410d960 2024-03-30T15:02:42.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.247+0000 7fca4a7fc700 1 -- 172.21.15.28:0/1954523973 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v398) ==== 101+0+0 (unknown 4220563546 0 0) 0x7fca4c05a5a0 con 0x7fca5410d960 2024-03-30T15:02:42.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:42.252 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2:163:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:5d165639:::164:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f43765fc:::165:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b4c720e9:::166:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e694b040:::167:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:afa38db2:::168:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:77ba9f53:::169:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:87495034:::170:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7c96bf0e:::171:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:dbe346cc:::172:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:e943ec24:::173:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f97a9c0c:::174:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:6f26e74d:::175:head 2024-03-30T15:02:42.253 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4f95e106:::176:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:0e6f2f8f:::177:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:05db05f1:::178:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:38a78d66:::179:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:d095610b:::180:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:a1a9d709:::181:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:1e5d39db:::182:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:f7df4fb9:::183:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:03a7f161:::184:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ba70721e:::185:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:28e5662d:::186:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:973d52de:::187:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:4303eb1c:::188:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:b990b48e:::189:head 2024-03-30T15:02:42.254 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:29b8165b:::190:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:3547f197:::191:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:7e260936:::192:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:1abec7b1:::193:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:10fdda93:::194:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:15817eea:::195:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:770bab57:::196:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:ed9e13e7:::197:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:71471a8f:::198:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: checking for 275:10fb1d02:::199:head 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7672 ms) 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,0 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: first is 1711810917 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,0 2024-03-30T15:02:42.255 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,1711810925,1711810926,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,1711810925,1711810926,0 2024-03-30T15:02:42.256 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810917,1711810919,1711810920,1711810922,1711810923,1711810925,1711810926,0 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711810920,1711810922,1711810923,1711810925,1711810926,1711810928,1711810929,0 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: first now 1711810920, trimmed 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21008 ms) 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: foo0 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14001 ms) 2024-03-30T15:02:42.257 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.255+0000 7fca5b91e700 10 librados: watch_flush enter 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.255+0000 7fca5b91e700 10 librados: watch_flush exit 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.255+0000 7fca5b91e700 20 client.57911.objecter shutdown clearing up homeless session... 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.255+0000 7fca5b91e700 10 client.57911.objecter successfully canceled tick 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.256+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 >> v1:172.21.15.28:6800/3011431093 conn(0x7fca400705f0 legacy=0x7fca40072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.256+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 >> v1:172.21.15.28:6790/0 conn(0x7fca5410d960 legacy=0x7fca541a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.257+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 shutdown_connections 2024-03-30T15:02:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.257+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 >> 172.21.15.28:0/1954523973 conn(0x7fca540fc9e0 msgr2=0x7fca5406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.258+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 shutdown_connections 2024-03-30T15:02:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.258+0000 7fca5b91e700 1 -- 172.21.15.28:0/1954523973 wait complete. 2024-03-30T15:02:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:42.259+0000 7fca5b91e700 1 librados: shutdown 2024-03-30T15:02:42.262 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:02:42.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:42.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:42.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:42.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:42.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:42.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:42.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:42.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 42.3 scrub starts 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 42.3 scrub ok 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 42.1 scrub starts 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 42.1 scrub ok 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1954523973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.57911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: from='client.57911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[41614]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 42.3 scrub starts 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 42.3 scrub ok 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 42.1 scrub starts 2024-03-30T15:02:42.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 42.1 scrub ok 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1954523973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.57911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:42.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:02:42.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: from='client.57911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:42.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:42 smithi028 ceph-mon[29876]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T15:02:42.862 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:42.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:42.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 42.3 scrub starts 2024-03-30T15:02:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 42.3 scrub ok 2024-03-30T15:02:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 42.1 scrub starts 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 42.1 scrub ok 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi028-92999-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]': finished 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: osdmap e397: 8 total, 8 up, 8 in 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-49"}]: dispatch 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1954523973' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.57911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: from='client.57911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:02:42.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:42 smithi134 ceph-mon[33714]: osdmap e398: 8 total, 8 up, 8 in 2024-03-30T15:02:43.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:43.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:43.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:43.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:43.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:43.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:43.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:43.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:43.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:02:43.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:43.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:43.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:43.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:43.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:43.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:43.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.9 scrub starts 2024-03-30T15:02:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.9 scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 42.10 scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 42.10 scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: pgmap v536: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.a scrub starts 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.a scrub ok 2024-03-30T15:02:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.4 scrub starts 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: 101.4 scrub ok 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.9 scrub starts 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.9 scrub ok 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 42.10 scrub starts 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 42.10 scrub ok 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: pgmap v536: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:43.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.a scrub starts 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.a scrub ok 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.4 scrub starts 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: 101.4 scrub ok 2024-03-30T15:02:43.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.9 scrub starts 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.9 scrub ok 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 42.10 scrub starts 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 42.10 scrub ok 2024-03-30T15:02:43.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: pgmap v536: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.a scrub starts 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.a scrub ok 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.4 scrub starts 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: 101.4 scrub ok 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: osdmap e399: 8 total, 8 up, 8 in 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:43.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:02:43.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:43.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:44.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:44.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:44.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:44.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.214+0000 7f5017290700 1 -- 172.21.15.28:0/2862005482 >> v1:172.21.15.28:6790/0 conn(0x7f5010109dd0 legacy=0x7f501010c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.214+0000 7f5017290700 1 -- 172.21.15.28:0/2862005482 shutdown_connections 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.214+0000 7f5017290700 1 -- 172.21.15.28:0/2862005482 >> 172.21.15.28:0/2862005482 conn(0x7f50100fc9f0 msgr2=0x7f50100fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.214+0000 7f5017290700 1 -- 172.21.15.28:0/2862005482 shutdown_connections 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.214+0000 7f5017290700 1 -- 172.21.15.28:0/2862005482 wait complete. 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 librados: starting msgr at 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 librados: starting objecter 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 Processor -- start 2024-03-30T15:02:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 -- start start 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 librados: setting wanted keys 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 librados: calling monclient init 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50101a93c0 con 0x7f501010d970 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50101aa5e0 con 0x7f5010101140 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f5017290700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50101ab820 con 0x7f5010109dd0 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f501482b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5010109dd0 0x7f50101a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42040/0 (socket says 172.21.15.28:42040) 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f501582d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f501010d970 0x7f50101a7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38842/0 (socket says 172.21.15.28:38842) 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f501482b700 1 -- 172.21.15.28:0/2182877483 learned_addr learned my addr 172.21.15.28:0/2182877483 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f500e7fc700 10 client.?.objecter ms_handle_connect 0x7f5010109dd0 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f500e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f500e7fc700 10 client.?.objecter ms_handle_connect 0x7f501010d970 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.215+0000 7f500e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 10 client.?.objecter ms_handle_connect 0x7f5010101140 2024-03-30T15:02:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1658442789 0 0) 0x7f50101ab820 con 0x7f5010109dd0 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ff8003aa0 con 0x7f5010109dd0 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570449661 0 0) 0x7f50101a93c0 con 0x7f501010d970 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50101ab820 con 0x7f501010d970 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445171857 0 0) 0x7f50101aa5e0 con 0x7f5010101140 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50101a93c0 con 0x7f5010101140 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2406923379 0 0) 0x7f4ff8003aa0 con 0x7f5010109dd0 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50101aa5e0 con 0x7f5010109dd0 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5004004530 con 0x7f5010109dd0 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3873275177 0 0) 0x7f50101ab820 con 0x7f501010d970 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ff8003aa0 con 0x7f501010d970 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f50080013d0 con 0x7f501010d970 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2658755432 0 0) 0x7f50101a93c0 con 0x7f5010101140 2024-03-30T15:02:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50101ab820 con 0x7f5010101140 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5000001bb0 con 0x7f5010101140 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 471378438 0 0) 0x7f50101aa5e0 con 0x7f5010109dd0 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 >> v1:172.21.15.134:6789/0 conn(0x7f5010101140 legacy=0x7f50100fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 >> v1:172.21.15.28:6789/0 conn(0x7f501010d970 legacy=0x7f50101a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50101aca60 con 0x7f5010109dd0 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50101a95f0 con 0x7f5010109dd0 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f5017290700 10 client.57917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f50101a9ab0 con 0x7f5010109dd0 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f5017290700 1 librados: init done 2024-03-30T15:02:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f50040019e0 con 0x7f5010109dd0 2024-03-30T15:02:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.216+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5004004dc0 con 0x7f5010109dd0 2024-03-30T15:02:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.217+0000 7f5017290700 10 client.57917.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.217+0000 7f5017290700 15 client.57917.objecter _session_command_op_assign -1 1 2024-03-30T15:02:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.217+0000 7f5017290700 20 client.57917.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.217+0000 7f5017290700 10 client.57917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.217+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4ffc002920 con 0x7f5010109dd0 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5004004da0 con 0x7f5010109dd0 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(399..399 src has 53..399) ==== 8916+0+0 (unknown 566506163 0 0) 0x7f50040899d0 con 0x7f5010109dd0 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 10 client.57917.objecter ms_dispatch 0x7f5010001550 osd_map(399..399 src has 53..399) 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 3 client.57917.objecter handle_osd_map got epochs [399,399] > 0 2024-03-30T15:02:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 3 client.57917.objecter handle_osd_map decoding full epoch 399 2024-03-30T15:02:44.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 10 client.57917.objecter checking command 1 2024-03-30T15:02:44.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter _get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 15 client.57917.objecter _session_command_op_remove -1 1 2024-03-30T15:02:44.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter _get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 15 client.57917.objecter _session_command_op_assign 3 1 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 4 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 10 client.57917.objecter _send_command 1 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6813/1506157250 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ff8082fd0 con 0x7f4ff807f2c0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 20 client.57917.objecter dump_active .. 0 homeless 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=399) ==== 24+0+0 (unknown 3518892591 0 0) 0x7f5004089d80 con 0x7f5010109dd0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.218+0000 7f500e7fc700 10 client.57917.objecter ms_handle_connect 0x7f4ff80703c0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 10 client.57917.objecter ms_handle_connect 0x7f4ff807f2c0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4ff8082fd0 con 0x7f4ff807f2c0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 10 client.57917.objecter ms_dispatch 0x7f5010001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 10 client.57917.objecter _finish_command 1 = system:0 2024-03-30T15:02:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 4 2024-03-30T15:02:44.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.219+0000 7f500e7fc700 15 client.57917.objecter _session_command_op_remove 3 1 2024-03-30T15:02:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 client.57917.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 15 client.57917.objecter _session_command_op_assign -1 2 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter _get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter _get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 15 client.57917.objecter _session_command_op_remove -1 2 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter get_session s=0x7f4ff807ec10 osd=3 3 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 15 client.57917.objecter _session_command_op_assign 3 2 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 4 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 client.57917.objecter _send_command 2 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 --> v1:172.21.15.28:6813/1506157250 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f4ffc0011f0 con 0x7f4ff807f2c0 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f500e7fc700 1 -- 172.21.15.28:0/2182877483 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f4ffc0011f0 con 0x7f4ff807f2c0 2024-03-30T15:02:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f500e7fc700 10 client.57917.objecter ms_dispatch 0x7f5010001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f500e7fc700 10 client.57917.objecter _finish_command 2 = system:0 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f500e7fc700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 4 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f500e7fc700 15 client.57917.objecter _session_command_op_remove 3 2 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 librados: watch_flush enter 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 librados: watch_flush exit 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 client.57917.objecter close_session for osd.3 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 >> v1:172.21.15.28:6813/1506157250 conn(0x7f4ff807f2c0 legacy=0x7f4ff8081760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter put_session s=0x7f4ff807ec10 osd=3 1 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 20 client.57917.objecter shutdown clearing up homeless session... 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 10 client.57917.objecter successfully canceled tick 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4ff80703c0 legacy=0x7f4ff8072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 >> v1:172.21.15.28:6790/0 conn(0x7f5010109dd0 legacy=0x7f50101a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f501582d700 1 -- 172.21.15.28:0/2182877483 reap_dead start 2024-03-30T15:02:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 shutdown_connections 2024-03-30T15:02:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 >> 172.21.15.28:0/2182877483 conn(0x7f50100fc9f0 msgr2=0x7f501006a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 shutdown_connections 2024-03-30T15:02:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.239+0000 7f5017290700 1 -- 172.21.15.28:0/2182877483 wait complete. 2024-03-30T15:02:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:44.240+0000 7f5017290700 1 librados: shutdown 2024-03-30T15:02:44.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:44.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:44.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:44.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:44.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:44.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:44.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:44.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 42.c deep-scrub starts 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: 42.c deep-scrub ok 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: pgmap v539: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 42.c deep-scrub starts 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: 42.c deep-scrub ok 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: pgmap v539: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T15:02:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:44.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:02:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:02:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:02:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 42.c deep-scrub starts 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: 42.c deep-scrub ok 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: pgmap v539: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: osdmap e400: 8 total, 8 up, 8 in 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3790227579' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]: dispatch 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:02:44.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:44.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:44.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:45.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:45.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:45.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:45.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:45.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:45.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:45.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:45.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:45.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 5.1b deep-scrub starts 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: 5.1b deep-scrub ok 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T15:02:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:02:45.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 5.1b deep-scrub starts 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: 5.1b deep-scrub ok 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:45 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:45.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-30T15:02:45.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T15:02:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T15:02:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:02:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 5.1b deep-scrub starts 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: 5.1b deep-scrub ok 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.57905 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi028-92999-65"}]': finished 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: osdmap e401: 8 total, 8 up, 8 in 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:45 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:02:45.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:02:45.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:45.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:45.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:46.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:46.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:46.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:46.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:46.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:46.372 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:46.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:46.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:46.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:46.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:46.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:46.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:46.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:46.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:46.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:46.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:47.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:47.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:47.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:47.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:47.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.411 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:47.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.483+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2248989088 >> v1:172.21.15.28:6790/0 conn(0x7fdcb41031f0 legacy=0x7fdcb41035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.483+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2248989088 shutdown_connections 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.483+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2248989088 >> 172.21.15.28:0/2248989088 conn(0x7fdcb40fc9e0 msgr2=0x7fdcb40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.483+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2248989088 shutdown_connections 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.483+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2248989088 wait complete. 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 librados: starting msgr at 2024-03-30T15:02:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 librados: starting objecter 2024-03-30T15:02:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 Processor -- start 2024-03-30T15:02:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 -- start start 2024-03-30T15:02:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 librados: setting wanted keys 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 librados: calling monclient init 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcb410ace0 con 0x7fdcb41031f0 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcb41a8650 con 0x7fdcb410b000 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb8d16700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcb41a9830 con 0x7fdcb410eb10 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb2d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdcb410eb10 0x7fdcb41a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42060/0 (socket says 172.21.15.28:42060) 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb2d9d700 1 -- 172.21.15.28:0/2447477218 learned_addr learned my addr 172.21.15.28:0/2447477218 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.484+0000 7fdcb1d9b700 1 --1- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6789/0 conn(0x7fdcb410b000 0x7fdcb4108240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:59854/0 (socket says 172.21.15.28:59854) 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter ms_handle_connect 0x7fdcb410b000 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter ms_handle_connect 0x7fdcb410eb10 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter ms_handle_connect 0x7fdcb41031f0 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3196333882 0 0) 0x7fdcb41a8650 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc9c0035b0 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110105784 0 0) 0x7fdcb41a9830 con 0x7fdcb410eb10 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcb41a8650 con 0x7fdcb410eb10 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1268232139 0 0) 0x7fdc9c0035b0 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcb41a9830 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdca8001820 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4115313448 0 0) 0x7fdcb41a8650 con 0x7fdcb410eb10 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc9c0035b0 con 0x7fdcb410eb10 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdcac001bb0 con 0x7fdcb410eb10 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2081543713 0 0) 0x7fdcb41a9830 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6790/0 conn(0x7fdcb410eb10 legacy=0x7fdcb41a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6789/0 conn(0x7fdcb41031f0 legacy=0x7fdcb4107b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb41aaa10 con 0x7fdcb410b000 2024-03-30T15:02:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41a9a60 con 0x7fdcb410b000 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdcb8d16700 10 client.49948.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdcb41a9f20 con 0x7fdcb410b000 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdcb8d16700 1 librados: init done 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdca8001c50 con 0x7fdcb410b000 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.485+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdca8005d10 con 0x7fdcb410b000 2024-03-30T15:02:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.486+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdc94005100 con 0x7fdcb410b000 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.487+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdca8003950 con 0x7fdcb410b000 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.487+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(403..403 src has 53..403) ==== 8949+0+0 (unknown 2552654901 0 0) 0x7fdca8089a80 con 0x7fdcb410b000 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.487+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 osd_map(403..403 src has 53..403) 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.487+0000 7fdca37fe700 3 client.49948.objecter handle_osd_map got epochs [403,403] > 0 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.487+0000 7fdca37fe700 3 client.49948.objecter handle_osd_map decoding full epoch 403 2024-03-30T15:02:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdca37fe700 20 client.49948.objecter dump_active .. 0 homeless 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v403) ==== 54+0+16 (unknown 1704715114 0 2420000192) 0x7fdca8089e70 con 0x7fdcb410b000 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 1 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 1 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 0 1 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 4 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 10 client.49948.objecter _send_command 1 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc9c070330 2024-03-30T15:02:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc94004b50 con 0x7fdc94004730 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.488+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc94004730 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.489+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94004b50 con 0x7fdc94004730 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.489+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.489+0000 7fdca37fe700 10 client.49948.objecter _finish_command 1 = system:0 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.489+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 4 2024-03-30T15:02:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.489+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 0 1 2024-03-30T15:02:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 2 2024-03-30T15:02:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 2 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc940053e0 osd=0 3 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 0 2 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 4 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 10 client.49948.objecter _send_command 2 2024-03-30T15:02:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.506+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc9400bf10 con 0x7fdc94004730 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc9400bf10 con 0x7fdc94004730 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 10 client.49948.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 4 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 0 2 2024-03-30T15:02:47.509 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 3 2024-03-30T15:02:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 3 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 1 3 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 4 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 10 client.49948.objecter _send_command 3 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdc94002160 con 0x7fdc940042a0 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.508+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc940042a0 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.509+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94002160 con 0x7fdc940042a0 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.509+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.509+0000 7fdca37fe700 10 client.49948.objecter _finish_command 3 = system:0 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.509+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 4 2024-03-30T15:02:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.509+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 1 3 2024-03-30T15:02:47.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 4 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 4 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94004130 osd=1 3 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 1 4 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 4 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 10 client.49948.objecter _send_command 4 2024-03-30T15:02:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.527+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc94001600 con 0x7fdc940042a0 2024-03-30T15:02:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc94001600 con 0x7fdc940042a0 2024-03-30T15:02:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdca37fe700 10 client.49948.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 4 2024-03-30T15:02:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 1 4 2024-03-30T15:02:47.531 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 5 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 5 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 2 5 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 4 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 10 client.49948.objecter _send_command 5 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.529+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdc94015420 con 0x7fdc9400cda0 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc9400cda0 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94015420 con 0x7fdc9400cda0 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 10 client.49948.objecter _finish_command 5 = system:0 2024-03-30T15:02:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 4 2024-03-30T15:02:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.530+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 2 5 2024-03-30T15:02:47.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 6 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 6 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94001a00 osd=2 3 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 2 6 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 4 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 10 client.49948.objecter _send_command 6 2024-03-30T15:02:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.547+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc9400c430 con 0x7fdc9400cda0 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc9400c430 con 0x7fdc9400cda0 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 10 client.49948.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 4 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 2 6 2024-03-30T15:02:47.550 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 7 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 7 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 3 7 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 4 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 10 client.49948.objecter _send_command 7 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdc94016120 con 0x7fdc940155d0 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.549+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc940155d0 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.550+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94016120 con 0x7fdc940155d0 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.550+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.550+0000 7fdca37fe700 10 client.49948.objecter _finish_command 7 = system:0 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.550+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 4 2024-03-30T15:02:47.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.550+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 3 7 2024-03-30T15:02:47.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 8 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 8 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc9400c680 osd=3 3 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 3 8 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 4 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 10 client.49948.objecter _send_command 8 2024-03-30T15:02:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.567+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc94016ad0 con 0x7fdc940155d0 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc94016ad0 con 0x7fdc940155d0 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdca37fe700 10 client.49948.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 4 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 3 8 2024-03-30T15:02:47.571 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 9 2024-03-30T15:02:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 9 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 4 9 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 4 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 10 client.49948.objecter _send_command 9 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.569+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdc9401ec40 con 0x7fdc9401afe0 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc9401afe0 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc9401ec40 con 0x7fdc9401afe0 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 10 client.49948.objecter _finish_command 9 = system:0 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 4 2024-03-30T15:02:47.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.570+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 4 9 2024-03-30T15:02:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 10 2024-03-30T15:02:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 10 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94016ad0 osd=4 3 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 4 10 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 4 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 10 client.49948.objecter _send_command 10 2024-03-30T15:02:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.587+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc9401ee20 con 0x7fdc9401afe0 2024-03-30T15:02:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc9401ee20 con 0x7fdc9401afe0 2024-03-30T15:02:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdca37fe700 10 client.49948.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 4 2024-03-30T15:02:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 4 10 2024-03-30T15:02:47.591 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 11 2024-03-30T15:02:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 11 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 5 11 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 4 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 10 client.49948.objecter _send_command 11 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.589+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdc94020210 con 0x7fdc9401f540 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc9401f540 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94020210 con 0x7fdc9401f540 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 10 client.49948.objecter _finish_command 11 = system:0 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 4 2024-03-30T15:02:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.590+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 5 11 2024-03-30T15:02:47.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: 3.10 deep-scrub starts 2024-03-30T15:02:47.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: 3.10 deep-scrub ok 2024-03-30T15:02:47.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: pgmap v542: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:47.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: 3.10 deep-scrub starts 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: 3.10 deep-scrub ok 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: pgmap v542: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:47.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T15:02:47.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:47.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:47 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 12 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 12 2024-03-30T15:02:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc9401ee20 osd=5 3 2024-03-30T15:02:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 5 12 2024-03-30T15:02:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 4 2024-03-30T15:02:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 10 client.49948.objecter _send_command 12 2024-03-30T15:02:47.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.610+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc94020210 con 0x7fdc9401f540 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc94020210 con 0x7fdc9401f540 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdca37fe700 10 client.49948.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 4 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 5 12 2024-03-30T15:02:47.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:47.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 13 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 13 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 6 13 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 4 2024-03-30T15:02:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 10 client.49948.objecter _send_command 13 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.612+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdc940276a0 con 0x7fdc94023b20 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc94023b20 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc940276a0 con 0x7fdc94023b20 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 10 client.49948.objecter _finish_command 13 = system:0 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 4 2024-03-30T15:02:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.613+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 6 13 2024-03-30T15:02:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 14 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 14 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc94020470 osd=6 3 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 6 14 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 4 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 10 client.49948.objecter _send_command 14 2024-03-30T15:02:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.630+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc940276a0 con 0x7fdc94023b20 2024-03-30T15:02:47.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc940276a0 con 0x7fdc94023b20 2024-03-30T15:02:47.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdca37fe700 10 client.49948.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 4 2024-03-30T15:02:47.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 6 14 2024-03-30T15:02:47.634 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 15 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 15 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 7 15 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 4 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 10 client.49948.objecter _send_command 15 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.632+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdc94028980 con 0x7fdc94029610 2024-03-30T15:02:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.633+0000 7fdca37fe700 10 client.49948.objecter ms_handle_connect 0x7fdc94029610 2024-03-30T15:02:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.634+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdc94028980 con 0x7fdc94029610 2024-03-30T15:02:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.634+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.634+0000 7fdca37fe700 10 client.49948.objecter _finish_command 15 = system:0 2024-03-30T15:02:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.634+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 4 2024-03-30T15:02:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.634+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 7 15 2024-03-30T15:02:47.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 10 client.49948.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:02:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign -1 16 2024-03-30T15:02:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 20 client.49948.objecter _get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_remove -1 16 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 20 client.49948.objecter get_session s=0x7fdc940276a0 osd=7 3 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 15 client.49948.objecter _session_command_op_assign 7 16 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 4 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 10 client.49948.objecter _send_command 16 2024-03-30T15:02:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.650+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc94028980 con 0x7fdc94029610 2024-03-30T15:02:47.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdca37fe700 1 -- 172.21.15.28:0/2447477218 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdc94028980 con 0x7fdc94029610 2024-03-30T15:02:47.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdca37fe700 10 client.49948.objecter ms_dispatch 0x7fdcb4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:47.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdca37fe700 10 client.49948.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdca37fe700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 4 2024-03-30T15:02:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdca37fe700 15 client.49948.objecter _session_command_op_remove 7 16 2024-03-30T15:02:47.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:47.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 10 librados: watch_flush enter 2024-03-30T15:02:47.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 10 librados: watch_flush exit 2024-03-30T15:02:47.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.0 2024-03-30T15:02:47.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6801/3394323575 conn(0x7fdc94004730 legacy=0x7fdc94002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940053e0 osd=0 1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6805/1850313547 conn(0x7fdc940042a0 legacy=0x7fdc9400f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.652+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94004130 osd=1 1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.2 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6809/3451325043 conn(0x7fdc9400cda0 legacy=0x7fdc94013b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94001a00 osd=2 1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.3 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6813/1506157250 conn(0x7fdc940155d0 legacy=0x7fdc94015a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9400c680 osd=3 1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.4 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6800/3063775514 conn(0x7fdc9401afe0 legacy=0x7fdc9401d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94016ad0 osd=4 1 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.5 2024-03-30T15:02:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6804/3668151179 conn(0x7fdc9401f540 legacy=0x7fdc9401f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc9401ee20 osd=5 1 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.6 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6808/2135224194 conn(0x7fdc94023b20 legacy=0x7fdc94025f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb2d9d700 1 -- 172.21.15.28:0/2447477218 reap_dead start 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb2d9d700 1 -- 172.21.15.28:0/2447477218 reap_dead start 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc94020470 osd=6 1 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter close_session for osd.7 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6812/3758190298 conn(0x7fdc94029610 legacy=0x7fdc94029a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter put_session s=0x7fdc940276a0 osd=7 1 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 20 client.49948.objecter shutdown clearing up homeless session... 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 10 client.49948.objecter successfully canceled tick 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdc9c070330 legacy=0x7fdc9c0727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> v1:172.21.15.134:6789/0 conn(0x7fdcb410b000 legacy=0x7fdcb4108240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb2d9d700 1 -- 172.21.15.28:0/2447477218 reap_dead start 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 shutdown_connections 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 >> 172.21.15.28:0/2447477218 conn(0x7fdcb40fc9e0 msgr2=0x7fdcb40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 shutdown_connections 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 -- 172.21.15.28:0/2447477218 wait complete. 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:47.653+0000 7fdcb8d16700 1 librados: shutdown 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:47.659 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:47.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:47.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:47.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:47.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:47.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: 3.10 deep-scrub starts 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: 3.10 deep-scrub ok 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: pgmap v542: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi028-92999-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: osdmap e402: 8 total, 8 up, 8 in 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]: dispatch 2024-03-30T15:02:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:47 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:48.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:48.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:48.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:48.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:48.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:48.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:48.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:48.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:48.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:48.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:48.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:48.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:48.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 6.a deep-scrub starts 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 6.a deep-scrub ok 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]': finished 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2447477218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 101.1f deep-scrub starts 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: 101.1f deep-scrub ok 2024-03-30T15:02:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: pgmap v545: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T15:02:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:48.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:48.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:48.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 6.a deep-scrub starts 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 6.a deep-scrub ok 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]': finished 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2447477218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 101.1f deep-scrub starts 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: 101.1f deep-scrub ok 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: pgmap v545: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 6.a deep-scrub starts 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 6.a deep-scrub ok 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-51", "mode": "writeback"}]': finished 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2447477218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: osdmap e403: 8 total, 8 up, 8 in 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 101.1f deep-scrub starts 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: 101.1f deep-scrub ok 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: pgmap v545: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi028-92999-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: osdmap e404: 8 total, 8 up, 8 in 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:02:49.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:49.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:49.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:49.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:49.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:49.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:49.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.440 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:49.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:49.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:49.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:49.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:49.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:49.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:49.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:49.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:49.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:49.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:50.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:50.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.197 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:50.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:50.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:50.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:50.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:50.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:50.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:02:50.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:02:50.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:02:50.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:50 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:02:50.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:50.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.635+0000 7f9626418700 1 -- 172.21.15.28:0/2096872884 >> v1:172.21.15.28:6790/0 conn(0x7f962010afa0 legacy=0x7f962010d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.635+0000 7f9626418700 1 -- 172.21.15.28:0/2096872884 shutdown_connections 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.635+0000 7f9626418700 1 -- 172.21.15.28:0/2096872884 >> 172.21.15.28:0/2096872884 conn(0x7f96200fc9e0 msgr2=0x7f96200fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.635+0000 7f9626418700 1 -- 172.21.15.28:0/2096872884 shutdown_connections 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.635+0000 7f9626418700 1 -- 172.21.15.28:0/2096872884 wait complete. 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 librados: starting msgr at 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 librados: starting objecter 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 Processor -- start 2024-03-30T15:02:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 -- start start 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 librados: setting wanted keys 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.636+0000 7f9626418700 1 librados: calling monclient init 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f9626418700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96201a9270 con 0x7f962010eab0 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f9626418700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96201aa490 con 0x7f962010afa0 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f9626418700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96201ab6d0 con 0x7f9620103190 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f96249b5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f962010eab0 0x7f96201a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41272/0 (socket says 172.21.15.28:41272) 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f96249b5700 1 -- 172.21.15.28:0/3876615985 learned_addr learned my addr 172.21.15.28:0/3876615985 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter ms_handle_connect 0x7f962010eab0 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter ms_handle_connect 0x7f9620103190 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter ms_handle_connect 0x7f962010afa0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290043810 0 0) 0x7f96201a9270 con 0x7f962010eab0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96080031f0 con 0x7f962010eab0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050461328 0 0) 0x7f96201ab6d0 con 0x7f9620103190 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96201a9270 con 0x7f9620103190 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788454006 0 0) 0x7f96201aa490 con 0x7f962010afa0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96201ab6d0 con 0x7f962010afa0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.637+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1727105155 0 0) 0x7f96201a9270 con 0x7f9620103190 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96201aa490 con 0x7f9620103190 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 75191108 0 0) 0x7f96080031f0 con 0x7f962010eab0 2024-03-30T15:02:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96201a9270 con 0x7f962010eab0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9610001250 con 0x7f9620103190 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9618001c10 con 0x7f962010eab0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3132591226 0 0) 0x7f96201ab6d0 con 0x7f962010afa0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96080031f0 con 0x7f962010afa0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9614003910 con 0x7f962010afa0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4015949950 0 0) 0x7f96201a9270 con 0x7f962010eab0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 >> v1:172.21.15.28:6790/0 conn(0x7f9620103190 legacy=0x7f962010a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 >> v1:172.21.15.134:6789/0 conn(0x7f962010afa0 legacy=0x7f96201a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96201ac910 con 0x7f962010eab0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f96201ab900 con 0x7f962010eab0 2024-03-30T15:02:50.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9618003ea0 con 0x7f962010eab0 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9618006020 con 0x7f962010eab0 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f9626418700 10 client.24747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f96201abdc0 con 0x7f962010eab0 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.638+0000 7f9626418700 1 librados: init done 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.639+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962006ad30 con 0x7f962010eab0 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.640+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9618003a00 con 0x7f962010eab0 2024-03-30T15:02:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.640+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(406..406 src has 53..406) ==== 8975+0+0 (unknown 2551758342 0 0) 0x7f9618089d00 con 0x7f962010eab0 2024-03-30T15:02:50.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.640+0000 7f961d7fa700 10 client.24747.objecter ms_dispatch 0x7f9620001550 osd_map(406..406 src has 53..406) 2024-03-30T15:02:50.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.640+0000 7f961d7fa700 3 client.24747.objecter handle_osd_map got epochs [406,406] > 0 2024-03-30T15:02:50.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.640+0000 7f961d7fa700 3 client.24747.objecter handle_osd_map decoding full epoch 406 2024-03-30T15:02:50.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.641+0000 7f961d7fa700 20 client.24747.objecter dump_active .. 0 homeless 2024-03-30T15:02:50.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.641+0000 7f961d7fa700 10 client.24747.objecter ms_handle_connect 0x7f9608078c70 2024-03-30T15:02:50.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.650+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9618056be0 con 0x7f962010eab0 2024-03-30T15:02:50.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:50.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:02:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T15:02:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T15:02:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:02:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: osdmap e405: 8 total, 8 up, 8 in 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:02:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:50 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:02:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:50.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:50.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.828 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:50.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:50.878+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f9620102320 con 0x7f962010eab0 2024-03-30T15:02:50.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:50.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:50.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:51.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:51.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:51.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:51.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.269+0000 7f961d7fa700 1 -- 172.21.15.28:0/3876615985 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v407) ==== 87+0+0 (unknown 856551717 0 0) 0x7f961805a630 con 0x7f962010eab0 2024-03-30T15:02:51.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.272+0000 7f9626418700 10 librados: watch_flush enter 2024-03-30T15:02:51.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.273+0000 7f9626418700 10 librados: watch_flush exit 2024-03-30T15:02:51.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.273+0000 7f9626418700 20 client.24747.objecter shutdown clearing up homeless session... 2024-03-30T15:02:51.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.273+0000 7f9626418700 10 client.24747.objecter successfully canceled tick 2024-03-30T15:02:51.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.273+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9608078c70 legacy=0x7f960807b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:51.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.273+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 >> v1:172.21.15.28:6789/0 conn(0x7f962010eab0 legacy=0x7f96201a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.274+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 shutdown_connections 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.274+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 >> 172.21.15.28:0/3876615985 conn(0x7f96200fc9e0 msgr2=0x7f96200fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.274+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 shutdown_connections 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.274+0000 7f9626418700 1 -- 172.21.15.28:0/3876615985 wait complete. 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:51.274+0000 7f9626418700 1 librados: shutdown 2024-03-30T15:02:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:02:51.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:51.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:02:51.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: pgmap v548: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T15:02:51.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: pgmap v548: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:51.318 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:51.319 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:51.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:51.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:51.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:51.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:02:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:02:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: pgmap v548: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: osdmap e406: 8 total, 8 up, 8 in 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:02:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:51.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:02:51.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:51.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(14.1d epoch 114/111 rep_tid 87 entries 114'2 (0'0) error 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head by client.24853.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T15:02:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 295.12800245900002, 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023963393999999999, 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173902+0000", 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7300000000000002e-07 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173911+0000", 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8720000000000008e-06 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173929+0000", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8666000000000001e-05 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173946+0000", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6724000000000001e-05 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.193026+0000", 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019079608000000001 2024-03-30T15:02:51.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.197865+0000", 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048391509999999999 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(15.1 epoch 114/111 rep_tid 80 entries 114'2 (0'0) error 15:8386110d:test-rados-api-smithi028-93086-4::foo:head by client.24856.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 295.09453359499997, 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047719461999999997, 2024-03-30T15:02:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207371+0000", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4999999999999998e-07 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207383+0000", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2333e-05 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207399+0000", 2024-03-30T15:02:51.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6235999999999998e-05 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207417+0000", 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7303999999999999e-05 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.247826+0000", 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040409395000000001 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.255090+0000", 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072638440000000002 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34820.0:20 9.8 e115/114 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head v 115'1, mlcod=0'0)", 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 295.020754522, 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021284207999999999, 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281150+0000", 2024-03-30T15:02:51.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0800000000000001e-07 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281157+0000", 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5959999999999998e-06 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281163+0000", 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4249999999999997e-06 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.842 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281170+0000", 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2110000000000001e-06 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282755+0000", 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001584577 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282773+0000", 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8474e-05 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302395+0000", 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019622245999999999 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302434+0000", 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8371000000000002e-05 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24853.0:11 14.15 e115/111 14:ada94957:test-rados-api-smithi028-93116-5::foo:head v 115'3, mlcod=115'1)", 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 293.92913554, 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038572226000000001, 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372769+0000", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0950000000000001e-06 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372783+0000", 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3463e-05 2024-03-30T15:02:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372792+0000", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.448e-06 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372802+0000", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7890000000000006e-06 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387040+0000", 2024-03-30T15:02:51.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014237757 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387062+0000", 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1857000000000001e-05 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411293+0000", 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024231447 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.848 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411341+0000", 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7370000000000002e-05 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32242.0:2 120.6 e139/138 120:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 269.22101782999999, 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029704504999999999, 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8999999999999998e-07 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080897+0000", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0572e-05 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080903+0000", 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4659999999999996e-06 2024-03-30T15:02:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080913+0000", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.431e-06 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097177+0000", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016263732999999999 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097197+0000", 2024-03-30T15:02:51.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0520999999999999e-05 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110549+0000", 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013351633 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110591+0000", 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1859e-05 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(115.6s1 139/136 ECSubWrite(tid=562, reqid=client.27400.0:256, at_version=139'28, trim_to=0'0, roll_forward_to=138'27))", 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 269.21829010900001, 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026921001, 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:02:51.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7399999999999999e-07 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083633+0000", 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8479999999999999e-05 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083641+0000", 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7190000000000007e-06 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083652+0000", 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0902e-05 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097409+0000", 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013756496 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097418+0000", 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110508+0000", 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013089735999999999 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110535+0000", 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7146000000000001e-05 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.858 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32239.0:2 119.1b e139/137 119:d96a01e0:::testobj:head v 139'1, mlcod=0'0)", 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 269.19134645299999, 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024229506000000001, 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110558+0000", 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3299999999999998e-07 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110568+0000", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0029e-05 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110575+0000", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3660000000000003e-06 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110585+0000", 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3400000000000004e-06 2024-03-30T15:02:51.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111135+0000", 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055055900000000005 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111154+0000", 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9269000000000001e-05 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134746+0000", 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023591918999999999 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134787+0000", 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0691000000000001e-05 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:12 104.1c e141/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:head v 141'1, mlcod=0'0)", 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:02:51.863 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 266.89765906100001, 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041302041999999997, 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404246+0000", 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5799999999999999e-06 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404257+0000", 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193999999999999e-05 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404267+0000", 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0215e-05 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404276+0000", 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4520000000000003e-06 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.866 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433295+0000", 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029018773000000001 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433316+0000", 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0636999999999999e-05 2024-03-30T15:02:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445508+0000", 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012192305000000001 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445547+0000", 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8886e-05 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.8 epoch 142/140 rep_tid 1005 entries 142'13 (0'0) error 123:1179ae53:::test_441:head by client.44099.0:448 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 142'11)", 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 264.483293108, 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038477480000000001, 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818612+0000", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.37e-07 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818622+0000", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0477e-05 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851833+0000", 2024-03-30T15:02:51.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033211076999999999 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851867+0000", 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3874999999999997e-05 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851967+0000", 2024-03-30T15:02:51.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100182 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.857088+0000", 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051211319999999996 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.1f epoch 143/140 rep_tid 611 entries 143'25 (0'0) error 123:fbee941d:::test_708:head by client.44099.0:715 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 143'23)", 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 263.466138672, 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.114938996, 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835768+0000", 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.892e-06 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835782+0000", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3456e-05 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835803+0000", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.101e-05 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835820+0000", 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-05 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.935436+0000", 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099616275000000004 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.950704+0000", 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015268273000000001 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1491 104.1c e144/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:5 v 144'6, mlcod=143'3)", 2024-03-30T15:02:51.877 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 263.399668534, 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022363271000000001, 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:02:51.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902236+0000", 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5999999999999999e-07 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902245+0000", 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9339999999999995e-06 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902253+0000", 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6200000000000005e-06 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902263+0000", 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1749999999999994e-06 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903266+0000", 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010038880000000001 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903291+0000", 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4907999999999999e-05 2024-03-30T15:02:51.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924561+0000", 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021269507 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924599+0000", 2024-03-30T15:02:51.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7778999999999999e-05 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1533 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 262.38303997399998, 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046703298999999997, 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:02:51.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918865+0000", 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3909999999999999e-06 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955732+0000", 2024-03-30T15:02:51.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036866290000000003 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955761+0000", 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9051e-05 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955774+0000", 2024-03-30T15:02:51.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3448999999999999e-05 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955856+0000", 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1716000000000002e-05 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955881+0000", 2024-03-30T15:02:51.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5361e-05 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965519+0000", 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096378399999999999 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965567+0000", 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8201000000000001e-05 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.888 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1535 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 262.34343042299997, 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055252190999999999, 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958476+0000", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2699999999999999e-06 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004606+0000", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046129950000000003 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004628+0000", 2024-03-30T15:02:51.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2775000000000001e-05 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004638+0000", 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3659999999999998e-06 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004730+0000", 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2026000000000003e-05 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004753+0000", 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2898e-05 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013677+0000", 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089245069999999999 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013726+0000", 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8399000000000002e-05 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1536 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:02:51.894 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 262.29420891900003, 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029420031999999999, 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007696+0000", 2024-03-30T15:02:51.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4410000000000001e-06 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024530+0000", 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016833292 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024541+0000", 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1619e-05 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024550+0000", 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2840000000000006e-06 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024603+0000", 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3566999999999997e-05 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024628+0000", 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4326999999999999e-05 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037063+0000", 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012435493000000001 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037115+0000", 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2009000000000003e-05 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.3 epoch 144/143 rep_tid 1544 entries 144'5 (0'0) error 130:c0c9adc9:::test_234:head by client.46235.0:241 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 144'3)", 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:02:51.903 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 262.164905272, 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031151346999999999, 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137001+0000", 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1840000000000002e-06 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137017+0000", 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-03-30T15:02:51.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150695+0000", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013677962 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150725+0000", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.042e-05 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150878+0000", 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015338299999999999 2024-03-30T15:02:51.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.168150+0000", 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017271531 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.908 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1599 127.e e147/142 127:7056b563:test-rados-api-smithi028-93253-10::foo:head v 147'1, mlcod=0'0)", 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 259.91428012, 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075902677000000002, 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7899999999999999e-07 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387631+0000", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3709999999999996e-06 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387636+0000", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2689999999999999e-06 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387641+0000", 2024-03-30T15:02:51.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2129999999999999e-06 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.388146+0000", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050455300000000003 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.388164+0000", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8002000000000001e-05 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463488+0000", 2024-03-30T15:02:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075323946000000003 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463526+0000", 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8943999999999998e-05 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:23 104.7 e147/131 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head v 147'3, mlcod=0'0)", 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:02:51.913 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 259.91135696600003, 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072716648999999994, 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390548+0000", 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1399999999999997e-07 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390561+0000", 2024-03-30T15:02:51.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3682e-05 2024-03-30T15:02:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390568+0000", 2024-03-30T15:02:51.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7220000000000002e-06 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390577+0000", 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7150000000000004e-06 2024-03-30T15:02:51.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390748+0000", 2024-03-30T15:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000170817 2024-03-30T15:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390769+0000", 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1851000000000001e-05 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463222+0000", 2024-03-30T15:02:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072453039999999996 2024-03-30T15:02:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463264+0000", 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1208000000000003e-05 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:25 104.7 e149/131 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head v 149'4, mlcod=146'1)", 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 258.04116948500001, 2024-03-30T15:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027373235999999999, 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:02:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260735+0000", 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3999999999999997e-07 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260741+0000", 2024-03-30T15:02:51.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4799999999999998e-06 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270493+0000", 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097522140000000004 2024-03-30T15:02:51.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270511+0000", 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7096e-05 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270773+0000", 2024-03-30T15:02:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000262188 2024-03-30T15:02:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270797+0000", 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4264999999999999e-05 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.288065+0000", 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017268426 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.288108+0000", 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2227000000000001e-05 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:63 111.18 e162/134 111:1e07d18e:test-rados-api-smithi028-93525-14::foo:head v 162'1, mlcod=0'0)", 2024-03-30T15:02:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:02:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.106876352, 2024-03-30T15:02:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020132151000000001, 2024-03-30T15:02:51.933 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6100000000000002e-07 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195035+0000", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7700000000000004e-06 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195041+0000", 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.964e-06 2024-03-30T15:02:51.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195049+0000", 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9390000000000003e-06 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195528+0000", 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047816500000000001 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195550+0000", 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1985999999999999e-05 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.215114+0000", 2024-03-30T15:02:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019564406999999999 2024-03-30T15:02:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.215160+0000", 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5658999999999998e-05 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.0.0:954 101.4 e201/130 101:24de1dd9:test-rados-api-smithi028-93253-14::foo38:6 v 201'40, mlcod=167'34)", 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 206.083288275, 2024-03-30T15:02:51.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025813787000000001, 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:02:51.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:02:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:02:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:02:51.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218617+0000", 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.336e-06 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218625+0000", 2024-03-30T15:02:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7419999999999997e-06 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218639+0000", 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3995999999999999e-05 2024-03-30T15:02:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218646+0000", 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9009999999999997e-06 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.228251+0000", 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096057060000000003 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.228270+0000", 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8728000000000001e-05 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:02:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.244397+0000", 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016126699000000001 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.244429+0000", 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2678999999999997e-05 2024-03-30T15:02:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:02:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:02:51.950 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:02:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:51.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:52.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:52.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:52.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:52.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:52.264 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ 2024-03-30T15:02:52.264 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3277 ms) 2024-03-30T15:02:52.264 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6744 ms) 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6989 ms) 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7029 ms) 2024-03-30T15:02:52.265 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-30T15:02:52.266 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6973 ms) 2024-03-30T15:02:52.266 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-30T15:02:52.266 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6997 ms) 2024-03-30T15:02:52.266 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-30T15:02:52.266 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7004 ms) 2024-03-30T15:02:52.267 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139044 ms total) 2024-03-30T15:02:52.267 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: 2024-03-30T15:02:52.267 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-30T15:02:52.267 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (307894 ms total) 2024-03-30T15:02:52.267 INFO:tasks.workunit.client.0.smithi028.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-30T15:02:52.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:52.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:52.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:52.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: 101.15 scrub starts 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: 101.15 scrub ok 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: pgmap v551: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[29876]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: 101.15 scrub starts 2024-03-30T15:02:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: 101.15 scrub ok 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: pgmap v551: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:02:52.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:52 smithi028 ceph-mon[41614]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T15:02:52.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:52.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:02:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: 101.15 scrub starts 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: 101.15 scrub ok 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:02:52.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3876615985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: osdmap e407: 8 total, 8 up, 8 in 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3909045240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]: dispatch 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: pgmap v551: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-51","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: from='client.57923 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi028-92999-66"}]': finished 2024-03-30T15:02:52.608 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:52 smithi134 ceph-mon[33714]: osdmap e408: 8 total, 8 up, 8 in 2024-03-30T15:02:52.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:52.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:52.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:52.820 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:02:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:52.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:52.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:52.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:53.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:53.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:53.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:53.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:53.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:53.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:53.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:53.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:53 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:02:53.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:53 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:02:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:53 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:02:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:53 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:02:53.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:53.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:53.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:53 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:02:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:53 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:02:53.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:53.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:53.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:53.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:53.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:53.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:53.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:54.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:54.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:54.115 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.116+0000 7fa2abdd7700 1 -- 172.21.15.28:0/3991108282 >> v1:172.21.15.28:6789/0 conn(0x7fa2a410aee0 legacy=0x7fa2a410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.116+0000 7fa2abdd7700 1 -- 172.21.15.28:0/3991108282 shutdown_connections 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.116+0000 7fa2abdd7700 1 -- 172.21.15.28:0/3991108282 >> 172.21.15.28:0/3991108282 conn(0x7fa2a40fdae0 msgr2=0x7fa2a40fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.116+0000 7fa2abdd7700 1 -- 172.21.15.28:0/3991108282 shutdown_connections 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.116+0000 7fa2abdd7700 1 -- 172.21.15.28:0/3991108282 wait complete. 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 librados: starting msgr at 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 librados: starting objecter 2024-03-30T15:02:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 Processor -- start 2024-03-30T15:02:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 -- start start 2024-03-30T15:02:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 librados: setting wanted keys 2024-03-30T15:02:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 librados: calling monclient init 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2a41a6f40 con 0x7fa2a410ea80 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2a41a8160 con 0x7fa2a4102250 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2abdd7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa2a41a93a0 con 0x7fa2a410aee0 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2a9372700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa2a410aee0 0x7fa2a41a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37054/0 (socket says 172.21.15.28:37054) 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa2a9372700 1 -- 172.21.15.28:0/1788019955 learned_addr learned my addr 172.21.15.28:0/1788019955 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa29affd700 10 client.?.objecter ms_handle_connect 0x7fa2a410aee0 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa29affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa29affd700 10 client.?.objecter ms_handle_connect 0x7fa2a410ea80 2024-03-30T15:02:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.117+0000 7fa29affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 10 client.?.objecter ms_handle_connect 0x7fa2a4102250 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786796576 0 0) 0x7fa2a41a6f40 con 0x7fa2a410ea80 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2900031f0 con 0x7fa2a410ea80 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33155750 0 0) 0x7fa2a41a93a0 con 0x7fa2a410aee0 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2a41a6f40 con 0x7fa2a410aee0 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978082889 0 0) 0x7fa2a41a8160 con 0x7fa2a4102250 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2a41a93a0 con 0x7fa2a4102250 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3374571228 0 0) 0x7fa2900031f0 con 0x7fa2a410ea80 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2a41a8160 con 0x7fa2a410ea80 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa29c001cd0 con 0x7fa2a410ea80 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1241838381 0 0) 0x7fa2a41a93a0 con 0x7fa2a4102250 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2900031f0 con 0x7fa2a4102250 2024-03-30T15:02:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2914594939 0 0) 0x7fa2a41a6f40 con 0x7fa2a410aee0 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2a41a93a0 con 0x7fa2a410aee0 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa2a0001870 con 0x7fa2a4102250 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa294001370 con 0x7fa2a410aee0 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.118+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2890004046 0 0) 0x7fa2a41a8160 con 0x7fa2a410ea80 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 >> v1:172.21.15.28:6790/0 conn(0x7fa2a410aee0 legacy=0x7fa2a41a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 >> v1:172.21.15.134:6789/0 conn(0x7fa2a4102250 legacy=0x7fa2a41073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a41aa5e0 con 0x7fa2a410ea80 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a41a7170 con 0x7fa2a410ea80 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa2abdd7700 10 client.24753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa2a41a7630 con 0x7fa2a410ea80 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa2abdd7700 1 librados: init done 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa29c003ea0 con 0x7fa2a410ea80 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.119+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa29c004d20 con 0x7fa2a410ea80 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.120+0000 7fa2abdd7700 10 client.24753.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.120+0000 7fa2abdd7700 15 client.24753.objecter _session_command_op_assign -1 1 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.120+0000 7fa2abdd7700 20 client.24753.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.120+0000 7fa2abdd7700 10 client.24753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.120+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa288002920 con 0x7fa2a410ea80 2024-03-30T15:02:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.121+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa29c003a00 con 0x7fa2a410ea80 2024-03-30T15:02:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.121+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(408..408 src has 53..408) ==== 8541+0+0 (unknown 1244626324 0 0) 0x7fa29c088740 con 0x7fa2a410ea80 2024-03-30T15:02:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 10 client.24753.objecter ms_dispatch 0x7fa2a4001550 osd_map(408..408 src has 53..408) 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 3 client.24753.objecter handle_osd_map got epochs [408,408] > 0 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 3 client.24753.objecter handle_osd_map decoding full epoch 408 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 10 client.24753.objecter checking command 1 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter _get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 15 client.24753.objecter _session_command_op_remove -1 1 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter _get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 15 client.24753.objecter _session_command_op_assign 4 1 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 4 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 10 client.24753.objecter _send_command 1 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.134:6800/3063775514 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa290082b40 con 0x7fa29007ee30 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 20 client.24753.objecter dump_active .. 0 homeless 2024-03-30T15:02:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=408) ==== 24+0+0 (unknown 308604552 0 0) 0x7fa29c088b50 con 0x7fa2a410ea80 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 10 client.24753.objecter ms_handle_connect 0x7fa290070410 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.122+0000 7fa29affd700 10 client.24753.objecter ms_handle_connect 0x7fa29007ee30 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.123+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa290082b40 con 0x7fa29007ee30 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.123+0000 7fa29affd700 10 client.24753.objecter ms_dispatch 0x7fa2a4001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.123+0000 7fa29affd700 10 client.24753.objecter _finish_command 1 = system:0 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.123+0000 7fa29affd700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 4 2024-03-30T15:02:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.123+0000 7fa29affd700 15 client.24753.objecter _session_command_op_remove 4 1 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 client.24753.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 15 client.24753.objecter _session_command_op_assign -1 2 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter _get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter _get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 15 client.24753.objecter _session_command_op_remove -1 2 2024-03-30T15:02:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter get_session s=0x7fa29007e780 osd=4 3 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 15 client.24753.objecter _session_command_op_assign 4 2 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 4 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 client.24753.objecter _send_command 2 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 --> v1:172.21.15.134:6800/3063775514 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fa2880011f0 con 0x7fa29007ee30 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa29affd700 1 -- 172.21.15.28:0/1788019955 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fa2880011f0 con 0x7fa29007ee30 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa29affd700 10 client.24753.objecter ms_dispatch 0x7fa2a4001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa29affd700 10 client.24753.objecter _finish_command 2 = system:0 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa29affd700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 4 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa29affd700 15 client.24753.objecter _session_command_op_remove 4 2 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 librados: watch_flush enter 2024-03-30T15:02:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 librados: watch_flush exit 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 client.24753.objecter close_session for osd.4 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 >> v1:172.21.15.134:6800/3063775514 conn(0x7fa29007ee30 legacy=0x7fa2900812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter put_session s=0x7fa29007e780 osd=4 1 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 20 client.24753.objecter shutdown clearing up homeless session... 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.162+0000 7fa2abdd7700 10 client.24753.objecter successfully canceled tick 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa290070410 legacy=0x7fa2900728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 >> v1:172.21.15.28:6789/0 conn(0x7fa2a410ea80 legacy=0x7fa2a41a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2aa374700 1 -- 172.21.15.28:0/1788019955 reap_dead start 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 shutdown_connections 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 >> 172.21.15.28:0/1788019955 conn(0x7fa2a40fdae0 msgr2=0x7fa2a40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 shutdown_connections 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 -- 172.21.15.28:0/1788019955 wait complete. 2024-03-30T15:02:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:54.163+0000 7fa2abdd7700 1 librados: shutdown 2024-03-30T15:02:54.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:54.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:54.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:54.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:54.421 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:54 smithi134 ceph-mon[33714]: pgmap v553: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:54.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:54.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:54.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:54.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:54.672 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:02:54.672 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:02:54.672 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:02:54.672 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:02:54.704 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:54 smithi028 ceph-mon[41614]: pgmap v553: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:54.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:54 smithi028 ceph-mon[29876]: pgmap v553: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:54.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:02:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:54.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:02:54.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:54.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:54.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:54.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:55.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:55.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:55.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-30T15:02:55.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:55.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:02:55.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:55.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:55.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:55 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:02:55.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:55 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:02:55.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:55.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:55.575 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:02:55.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:55.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:55.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:02:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:55 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:02:55.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:55.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:55.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:55.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:55.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:56.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:56.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:56.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:56.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:56.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:56 smithi028 ceph-mon[41614]: pgmap v554: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:56.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:56 smithi028 ceph-mon[29876]: pgmap v554: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:56.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:56.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:56.557 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:56 smithi134 ceph-mon[33714]: pgmap v554: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:56.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.626+0000 7f22f626d700 1 -- 172.21.15.28:0/3472904817 >> v1:172.21.15.28:6790/0 conn(0x7f22f0109d80 legacy=0x7f22f010c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.626+0000 7f22f626d700 1 -- 172.21.15.28:0/3472904817 shutdown_connections 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.626+0000 7f22f626d700 1 -- 172.21.15.28:0/3472904817 >> 172.21.15.28:0/3472904817 conn(0x7f22f0077370 msgr2=0x7f22f00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.626+0000 7f22f626d700 1 -- 172.21.15.28:0/3472904817 shutdown_connections 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.626+0000 7f22f626d700 1 -- 172.21.15.28:0/3472904817 wait complete. 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 librados: starting msgr at 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 librados: starting objecter 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 Processor -- start 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 -- start start 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 librados: setting wanted keys 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 librados: calling monclient init 2024-03-30T15:02:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22f01a7080 con 0x7f22f0109d80 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22f01a82a0 con 0x7f22f010d950 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22f626d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22f01a94e0 con 0x7f22f0105f00 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22effff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f22f0105f00 0x7f22f010d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37084/0 (socket says 172.21.15.28:37084) 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22effff700 1 -- 172.21.15.28:0/4011259104 learned_addr learned my addr 172.21.15.28:0/4011259104 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22ef7fe700 1 --1- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6789/0 conn(0x7f22f0109d80 0x7f22f01a1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41324/0 (socket says 172.21.15.28:41324) 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22ed7fa700 10 client.?.objecter ms_handle_connect 0x7f22f0105f00 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22ed7fa700 10 client.?.objecter ms_handle_connect 0x7f22f0109d80 2024-03-30T15:02:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.627+0000 7f22ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 10 client.?.objecter ms_handle_connect 0x7f22f010d950 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516699861 0 0) 0x7f22f01a94e0 con 0x7f22f0105f00 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d80039c0 con 0x7f22f0105f00 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3628768799 0 0) 0x7f22f01a7080 con 0x7f22f0109d80 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22f01a94e0 con 0x7f22f0109d80 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508266784 0 0) 0x7f22f01a82a0 con 0x7f22f010d950 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22f01a7080 con 0x7f22f010d950 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2163103397 0 0) 0x7f22d80039c0 con 0x7f22f0105f00 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22f01a82a0 con 0x7f22f0105f00 2024-03-30T15:02:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22e0001a30 con 0x7f22f0105f00 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115558823 0 0) 0x7f22f01a94e0 con 0x7f22f0109d80 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d80039c0 con 0x7f22f0109d80 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1222598575 0 0) 0x7f22f01a7080 con 0x7f22f010d950 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22f01a94e0 con 0x7f22f010d950 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22e40043e0 con 0x7f22f0109d80 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22e8001b30 con 0x7f22f010d950 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 476645617 0 0) 0x7f22f01a82a0 con 0x7f22f0105f00 2024-03-30T15:02:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.134:6789/0 conn(0x7f22f010d950 legacy=0x7f22f01a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6789/0 conn(0x7f22f0109d80 legacy=0x7f22f01a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f01aa720 con 0x7f22f0105f00 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22f01a72b0 con 0x7f22f0105f00 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22f626d700 10 client.57947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f22e0001ce0 con 0x7f22f0105f00 2024-03-30T15:02:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f22f01a7770 con 0x7f22f0105f00 2024-03-30T15:02:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22f626d700 1 librados: init done 2024-03-30T15:02:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.628+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f22e0005d80 con 0x7f22f0105f00 2024-03-30T15:02:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.629+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f22dc005100 con 0x7f22f0105f00 2024-03-30T15:02:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.630+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f22e0003930 con 0x7f22f0105f00 2024-03-30T15:02:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.630+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(408..408 src has 53..408) ==== 8541+0+0 (unknown 1244626324 0 0) 0x7f22e0089910 con 0x7f22f0105f00 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.630+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 osd_map(408..408 src has 53..408) 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.630+0000 7f22ed7fa700 3 client.57947.objecter handle_osd_map got epochs [408,408] > 0 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.630+0000 7f22ed7fa700 3 client.57947.objecter handle_osd_map decoding full epoch 408 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22ed7fa700 20 client.57947.objecter dump_active .. 0 homeless 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v408) ==== 54+0+16 (unknown 767593945 0 2420000192) 0x7f22e0089d50 con 0x7f22f0105f00 2024-03-30T15:02:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22d80704f0 2024-03-30T15:02:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 10 client.57947.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 1 2024-03-30T15:02:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 1 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 0 1 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 4 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 10 client.57947.objecter _send_command 1 2024-03-30T15:02:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22dc004b50 con 0x7f22dc004730 2024-03-30T15:02:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.631+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc004730 2024-03-30T15:02:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.632+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc004b50 con 0x7f22dc004730 2024-03-30T15:02:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.632+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.632+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 1 = system:0 2024-03-30T15:02:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.632+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 4 2024-03-30T15:02:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.632+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 0 1 2024-03-30T15:02:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 10 client.57947.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 2 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 2 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc0053e0 osd=0 3 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 0 2 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 4 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 10 client.57947.objecter _send_command 2 2024-03-30T15:02:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.650+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc007b00 con 0x7f22dc004730 2024-03-30T15:02:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc007b00 con 0x7f22dc004730 2024-03-30T15:02:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 4 2024-03-30T15:02:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 0 2 2024-03-30T15:02:56.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 10 client.57947.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 1 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 4 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 10 client.57947.objecter _send_command 3 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f22dc002160 con 0x7f22dc0042a0 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.653+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc0042a0 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.654+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc002160 con 0x7f22dc0042a0 2024-03-30T15:02:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.654+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 3: 0 ) 2024-03-30T15:02:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.654+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 3 = system:0 2024-03-30T15:02:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.654+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 4 2024-03-30T15:02:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.654+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 1 3 2024-03-30T15:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 10 client.57947.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 4 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 4 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc004130 osd=1 3 2024-03-30T15:02:56.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 1 4 2024-03-30T15:02:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 4 2024-03-30T15:02:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 10 client.57947.objecter _send_command 4 2024-03-30T15:02:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.680+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc001600 con 0x7f22dc0042a0 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.681+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc001600 con 0x7f22dc0042a0 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.681+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.681+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.681+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 4 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.681+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 1 4 2024-03-30T15:02:56.683 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 10 client.57947.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 5 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 5 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 2 5 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 4 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 10 client.57947.objecter _send_command 5 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f22dc008bd0 con 0x7f22dc00d6e0 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.682+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc00d6e0 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.683+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc008bd0 con 0x7f22dc00d6e0 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.683+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 5: 0 ) 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.683+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 5 = system:0 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.683+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 4 2024-03-30T15:02:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.683+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 2 5 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.702+0000 7f22f626d700 10 client.57947.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 6 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 6 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc008480 osd=2 3 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 2 6 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 4 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 10 client.57947.objecter _send_command 6 2024-03-30T15:02:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.703+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc00ea50 con 0x7f22dc00d6e0 2024-03-30T15:02:56.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.705+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc00ea50 con 0x7f22dc00d6e0 2024-03-30T15:02:56.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.705+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.705+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.705+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 4 2024-03-30T15:02:56.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.705+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 2 6 2024-03-30T15:02:56.707 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 10 client.57947.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 7 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 7 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 3 7 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 4 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 10 client.57947.objecter _send_command 7 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f22dc01a310 con 0x7f22dc0167c0 2024-03-30T15:02:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.706+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc0167c0 2024-03-30T15:02:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.707+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc01a310 con 0x7f22dc0167c0 2024-03-30T15:02:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.707+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 7: 0 ) 2024-03-30T15:02:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.707+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 7 = system:0 2024-03-30T15:02:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.707+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 4 2024-03-30T15:02:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.707+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 3 7 2024-03-30T15:02:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 10 client.57947.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 8 2024-03-30T15:02:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 8 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00ecb0 osd=3 3 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 3 8 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 4 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 10 client.57947.objecter _send_command 8 2024-03-30T15:02:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.735+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc00e4d0 con 0x7f22dc0167c0 2024-03-30T15:02:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc00e4d0 con 0x7f22dc0167c0 2024-03-30T15:02:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 4 2024-03-30T15:02:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 3 8 2024-03-30T15:02:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:02:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 10 client.57947.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 9 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 9 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 4 9 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 4 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 10 client.57947.objecter _send_command 9 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.738+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f22dc00d470 con 0x7f22dc00e280 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.739+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc00e280 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.740+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc00d470 con 0x7f22dc00e280 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.740+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 9: 0 ) 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.740+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 9 = system:0 2024-03-30T15:02:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.740+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 4 2024-03-30T15:02:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.740+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 4 9 2024-03-30T15:02:56.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 10 client.57947.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 10 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 10 2024-03-30T15:02:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc01a330 osd=4 3 2024-03-30T15:02:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 4 10 2024-03-30T15:02:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 4 2024-03-30T15:02:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 10 client.57947.objecter _send_command 10 2024-03-30T15:02:56.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.757+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc00d470 con 0x7f22dc00e280 2024-03-30T15:02:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc00d470 con 0x7f22dc00e280 2024-03-30T15:02:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 4 2024-03-30T15:02:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 4 10 2024-03-30T15:02:56.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:02:56.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 10 client.57947.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 11 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 11 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 5 11 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 4 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 10 client.57947.objecter _send_command 11 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.759+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f22dc020f00 con 0x7f22dc01f250 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc01f250 2024-03-30T15:02:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc020f00 con 0x7f22dc01f250 2024-03-30T15:02:56.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 11: 0 ) 2024-03-30T15:02:56.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 11 = system:0 2024-03-30T15:02:56.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 4 2024-03-30T15:02:56.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.760+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 5 11 2024-03-30T15:02:56.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 10 client.57947.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 12 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 12 2024-03-30T15:02:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00d470 osd=5 3 2024-03-30T15:02:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 5 12 2024-03-30T15:02:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 4 2024-03-30T15:02:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 10 client.57947.objecter _send_command 12 2024-03-30T15:02:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.778+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc021180 con 0x7f22dc01f250 2024-03-30T15:02:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc021180 con 0x7f22dc01f250 2024-03-30T15:02:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 4 2024-03-30T15:02:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 5 12 2024-03-30T15:02:56.782 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:02:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 10 client.57947.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 13 2024-03-30T15:02:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 13 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 6 13 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 4 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 10 client.57947.objecter _send_command 13 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f22dc0273e0 con 0x7f22dc021180 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.780+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc021180 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.781+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc0273e0 con 0x7f22dc021180 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.781+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 13: 0 ) 2024-03-30T15:02:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.781+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 13 = system:0 2024-03-30T15:02:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.781+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 4 2024-03-30T15:02:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.781+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 6 13 2024-03-30T15:02:56.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 10 client.57947.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 14 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 14 2024-03-30T15:02:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc020f00 osd=6 3 2024-03-30T15:02:56.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 6 14 2024-03-30T15:02:56.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 4 2024-03-30T15:02:56.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 10 client.57947.objecter _send_command 14 2024-03-30T15:02:56.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.798+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc00c7b0 con 0x7f22dc021180 2024-03-30T15:02:56.802 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.800+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc00c7b0 con 0x7f22dc021180 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.800+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.800+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.800+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 4 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.800+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 6 14 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 10 client.57947.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 15 2024-03-30T15:02:56.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 15 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 7 15 2024-03-30T15:02:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 4 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 10 client.57947.objecter _send_command 15 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f22dc00c6d0 con 0x7f22dc01acc0 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.801+0000 7f22ed7fa700 10 client.57947.objecter ms_handle_connect 0x7f22dc01acc0 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.802+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f22dc00c6d0 con 0x7f22dc01acc0 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.802+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 15: 0 ) 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.802+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 15 = system:0 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.802+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 4 2024-03-30T15:02:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.802+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 7 15 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 10 client.57947.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign -1 16 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 20 client.57947.objecter _get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 15 client.57947.objecter _session_command_op_remove -1 16 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 20 client.57947.objecter get_session s=0x7f22dc00c560 osd=7 3 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 15 client.57947.objecter _session_command_op_assign 7 16 2024-03-30T15:02:56.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 4 2024-03-30T15:02:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 10 client.57947.objecter _send_command 16 2024-03-30T15:02:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.819+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f22dc0008d0 con 0x7f22dc01acc0 2024-03-30T15:02:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22ed7fa700 1 -- 172.21.15.28:0/4011259104 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f22dc0008d0 con 0x7f22dc01acc0 2024-03-30T15:02:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22ed7fa700 10 client.57947.objecter ms_dispatch 0x7f22f0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:02:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22ed7fa700 10 client.57947.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:02:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22ed7fa700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 4 2024-03-30T15:02:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22ed7fa700 15 client.57947.objecter _session_command_op_remove 7 16 2024-03-30T15:02:56.823 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 librados: watch_flush enter 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 librados: watch_flush exit 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.0 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6801/3394323575 conn(0x7f22dc004730 legacy=0x7f22dc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc0053e0 osd=0 1 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.1 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6805/1850313547 conn(0x7f22dc0042a0 legacy=0x7f22dc00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc004130 osd=1 1 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.2 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6809/3451325043 conn(0x7f22dc00d6e0 legacy=0x7f22dc00db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc008480 osd=2 1 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.3 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6813/1506157250 conn(0x7f22dc0167c0 legacy=0x7f22dc018bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00ecb0 osd=3 1 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.4 2024-03-30T15:02:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.134:6800/3063775514 conn(0x7f22dc00e280 legacy=0x7f22dc00cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc01a330 osd=4 1 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.5 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.134:6804/3668151179 conn(0x7f22dc01f250 legacy=0x7f22dc01f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f480a700 1 -- 172.21.15.28:0/4011259104 reap_dead start 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f480a700 1 -- 172.21.15.28:0/4011259104 reap_dead start 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f480a700 1 -- 172.21.15.28:0/4011259104 reap_dead start 2024-03-30T15:02:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00d470 osd=5 1 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 10 client.57947.objecter close_session for osd.6 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.821+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.134:6808/2135224194 conn(0x7f22dc021180 legacy=0x7f22dc025be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc020f00 osd=6 1 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 10 client.57947.objecter close_session for osd.7 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.134:6812/3758190298 conn(0x7f22dc01acc0 legacy=0x7f22dc02ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 20 client.57947.objecter put_session s=0x7f22dc00c560 osd=7 1 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 20 client.57947.objecter shutdown clearing up homeless session... 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 10 client.57947.objecter successfully canceled tick 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6800/3011431093 conn(0x7f22d80704f0 legacy=0x7f22d80729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> v1:172.21.15.28:6790/0 conn(0x7f22f0105f00 legacy=0x7f22f010d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f480a700 1 -- 172.21.15.28:0/4011259104 reap_dead start 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 shutdown_connections 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 >> 172.21.15.28:0/4011259104 conn(0x7f22f0077370 msgr2=0x7f22f0105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 shutdown_connections 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 -- 172.21.15.28:0/4011259104 wait complete. 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:56.822+0000 7f22f626d700 1 librados: shutdown 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:02:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:02:56.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:56.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:56.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:56.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:57.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:57.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:57.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:57.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:57.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:57.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:57.407 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:02:57.407 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:02:57.416 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:57 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:57.416 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:57 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:57.416 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4011259104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:57.416 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:57 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:57.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:57.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:57.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:57.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4011259104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4011259104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:02:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:57 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:02:57.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:57.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:57.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:57.907 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:57.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:57.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:02:58.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:02:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:58.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:02:58.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:58.215 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:02:58.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:58.281 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:02:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:58.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:58.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:58.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:58.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:58.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:58 smithi134 ceph-mon[33714]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T15:02:58.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:02:58.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:02:58.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:58 smithi134 ceph-mon[33714]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:58.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:58.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:58.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[29876]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T15:02:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:02:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:02:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[29876]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:58.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:02:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:02:58.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[41614]: osdmap e409: 8 total, 8 up, 8 in 2024-03-30T15:02:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:02:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:02:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:58 smithi028 ceph-mon[41614]: pgmap v556: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:02:58.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:58.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:02:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:58.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:02:59.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:59.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:59.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:59.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:02:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.306 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:59.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:02:59.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:59.408 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:02:59 smithi134 ceph-mon[33714]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T15:02:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:02:59.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:02:59 smithi028 ceph-mon[29876]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T15:02:59.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:02:59 smithi028 ceph-mon[41614]: osdmap e410: 8 total, 8 up, 8 in 2024-03-30T15:02:59.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:02:59.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:02:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.648+0000 7f824a7a6700 1 -- 172.21.15.28:0/2731514659 >> v1:172.21.15.28:6789/0 conn(0x7f8244101140 legacy=0x7f8244101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.648+0000 7f824a7a6700 1 -- 172.21.15.28:0/2731514659 shutdown_connections 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.648+0000 7f824a7a6700 1 -- 172.21.15.28:0/2731514659 >> 172.21.15.28:0/2731514659 conn(0x7f82440fc9f0 msgr2=0x7f82440fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.648+0000 7f824a7a6700 1 -- 172.21.15.28:0/2731514659 shutdown_connections 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.648+0000 7f824a7a6700 1 -- 172.21.15.28:0/2731514659 wait complete. 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 librados: starting msgr at 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 librados: starting objecter 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 Processor -- start 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 -- start start 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 librados: setting wanted keys 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 librados: calling monclient init 2024-03-30T15:02:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8244106860 con 0x7f8244101140 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f82441a8550 con 0x7f8244109dd0 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f824a7a6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f82441a9730 con 0x7f824410d970 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f8243fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8244101140 0x7f824406a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43806/0 (socket says 172.21.15.28:43806) 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f8243fff700 1 -- 172.21.15.28:0/3040142807 learned_addr learned my addr 172.21.15.28:0/3040142807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f82417fa700 10 client.?.objecter ms_handle_connect 0x7f824410d970 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f82417fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f82417fa700 10 client.?.objecter ms_handle_connect 0x7f8244101140 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.649+0000 7f82417fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 10 client.?.objecter ms_handle_connect 0x7f8244109dd0 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247124994 0 0) 0x7f82441a9730 con 0x7f824410d970 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f822c0031f0 con 0x7f824410d970 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610034427 0 0) 0x7f8244106860 con 0x7f8244101140 2024-03-30T15:02:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82441a9730 con 0x7f8244101140 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20562151 0 0) 0x7f82441a8550 con 0x7f8244109dd0 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8244106860 con 0x7f8244109dd0 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3426012564 0 0) 0x7f822c0031f0 con 0x7f824410d970 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82441a8550 con 0x7f824410d970 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2569383811 0 0) 0x7f82441a9730 con 0x7f8244101140 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f822c0031f0 con 0x7f8244101140 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f823c001c30 con 0x7f824410d970 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8234001370 con 0x7f8244101140 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 61562773 0 0) 0x7f8244106860 con 0x7f8244109dd0 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82441a9730 con 0x7f8244109dd0 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8238001870 con 0x7f8244109dd0 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1748905049 0 0) 0x7f82441a8550 con 0x7f824410d970 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 >> v1:172.21.15.134:6789/0 conn(0x7f8244109dd0 legacy=0x7f824406ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 >> v1:172.21.15.28:6789/0 conn(0x7f8244101140 legacy=0x7f824406a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82441aa910 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f82441a7540 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f824a7a6700 10 client.57953.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f82441a7a00 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f824a7a6700 1 librados: init done 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f823c003ec0 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.650+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f823c004dc0 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.651+0000 7f824a7a6700 10 client.57953.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.651+0000 7f824a7a6700 15 client.57953.objecter _session_command_op_assign -1 1 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.651+0000 7f824a7a6700 20 client.57953.objecter _get_session osd=-1 returning homeless 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.651+0000 7f824a7a6700 10 client.57953.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.651+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8230002920 con 0x7f824410d970 2024-03-30T15:02:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f823c003a20 con 0x7f824410d970 2024-03-30T15:02:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(410..410 src has 53..410) ==== 8541+0+0 (unknown 2815930682 0 0) 0x7f823c088860 con 0x7f824410d970 2024-03-30T15:02:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 10 client.57953.objecter ms_dispatch 0x7f8244001550 osd_map(410..410 src has 53..410) 2024-03-30T15:02:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 3 client.57953.objecter handle_osd_map got epochs [410,410] > 0 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 3 client.57953.objecter handle_osd_map decoding full epoch 410 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 10 client.57953.objecter checking command 1 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter _get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 15 client.57953.objecter _session_command_op_remove -1 1 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter _get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 15 client.57953.objecter _session_command_op_assign 5 1 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 4 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 10 client.57953.objecter _send_command 1 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.134:6804/3668151179 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f822c082da0 con 0x7f822c07f090 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 20 client.57953.objecter dump_active .. 0 homeless 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.653+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=410) ==== 24+0+0 (unknown 4086334341 0 0) 0x7f823c088c60 con 0x7f824410d970 2024-03-30T15:02:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.654+0000 7f82417fa700 10 client.57953.objecter ms_handle_connect 0x7f822c0704c0 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.654+0000 7f82417fa700 10 client.57953.objecter ms_handle_connect 0x7f822c07f090 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.655+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f822c082da0 con 0x7f822c07f090 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.655+0000 7f82417fa700 10 client.57953.objecter ms_dispatch 0x7f8244001550 command_reply(tid 1: 0 ) 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.655+0000 7f82417fa700 10 client.57953.objecter _finish_command 1 = system:0 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.655+0000 7f82417fa700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 4 2024-03-30T15:02:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.655+0000 7f82417fa700 15 client.57953.objecter _session_command_op_remove 5 1 2024-03-30T15:02:59.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 10 client.57953.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 15 client.57953.objecter _session_command_op_assign -1 2 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 20 client.57953.objecter _get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 20 client.57953.objecter _get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 15 client.57953.objecter _session_command_op_remove -1 2 2024-03-30T15:02:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 20 client.57953.objecter get_session s=0x7f822c07e9e0 osd=5 3 2024-03-30T15:02:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 15 client.57953.objecter _session_command_op_assign 5 2 2024-03-30T15:02:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 4 2024-03-30T15:02:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 10 client.57953.objecter _send_command 2 2024-03-30T15:02:59.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 --> v1:172.21.15.134:6804/3668151179 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f82300011f0 con 0x7f822c07f090 2024-03-30T15:02:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f82417fa700 1 -- 172.21.15.28:0/3040142807 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f82300011f0 con 0x7f822c07f090 2024-03-30T15:02:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f82417fa700 10 client.57953.objecter ms_dispatch 0x7f8244001550 command_reply(tid 2: 0 ) 2024-03-30T15:02:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f82417fa700 10 client.57953.objecter _finish_command 2 = system:0 2024-03-30T15:02:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f82417fa700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 4 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.674+0000 7f82417fa700 15 client.57953.objecter _session_command_op_remove 5 2 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 10 librados: watch_flush enter 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 10 librados: watch_flush exit 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 10 client.57953.objecter close_session for osd.5 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 >> v1:172.21.15.134:6804/3668151179 conn(0x7f822c07f090 legacy=0x7f822c081530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 20 client.57953.objecter put_session s=0x7f822c07e9e0 osd=5 1 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 20 client.57953.objecter shutdown clearing up homeless session... 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 10 client.57953.objecter successfully canceled tick 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 >> v1:172.21.15.28:6800/3011431093 conn(0x7f822c0704c0 legacy=0x7f822c072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 >> v1:172.21.15.28:6790/0 conn(0x7f824410d970 legacy=0x7f82441a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f8248d43700 1 -- 172.21.15.28:0/3040142807 reap_dead start 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 shutdown_connections 2024-03-30T15:02:59.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 >> 172.21.15.28:0/3040142807 conn(0x7f82440fc9f0 msgr2=0x7f8244110b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:02:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 shutdown_connections 2024-03-30T15:02:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 -- 172.21.15.28:0/3040142807 wait complete. 2024-03-30T15:02:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:02:59.675+0000 7f824a7a6700 1 librados: shutdown 2024-03-30T15:02:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:02:59.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:02:59.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:02:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:59.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:02:59.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:02:59.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:02:59.918 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:02:59.918 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:02:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:02:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:02:59.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:00.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:00.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:00.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:00.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:00.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:00.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:00.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:00.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:00.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:00 smithi028 ceph-mon[29876]: pgmap v558: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:00.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:00 smithi028 ceph-mon[41614]: pgmap v558: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:00.522 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:00.522 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:00 smithi134 ceph-mon[33714]: pgmap v558: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:00.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:00.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:03:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:00.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:00.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:00.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:00.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:00.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:00.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:00.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:00.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:01.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:01.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:01.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:01.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:01.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:01.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:01.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:01.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:01.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-30T15:03:01.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:01.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 -- 172.21.15.28:0/853243857 >> v1:172.21.15.28:6790/0 conn(0x7f753c10ea80 legacy=0x7f753c110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 -- 172.21.15.28:0/853243857 shutdown_connections 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 -- 172.21.15.28:0/853243857 >> 172.21.15.28:0/853243857 conn(0x7f753c0fdb20 msgr2=0x7f753c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 -- 172.21.15.28:0/853243857 shutdown_connections 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 -- 172.21.15.28:0/853243857 wait complete. 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 librados: starting msgr at 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.708+0000 7f7542377700 1 librados: starting objecter 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 Processor -- start 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 -- start start 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 librados: setting wanted keys 2024-03-30T15:03:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 librados: calling monclient init 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f753c076210 con 0x7f753c102250 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f753c076420 con 0x7f753c10aee0 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f7542377700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f753c1aba50 con 0x7f753c10ea80 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f753bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f753c102250 0x7f753c074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43814/0 (socket says 172.21.15.28:43814) 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f753bfff700 1 -- 172.21.15.28:0/2031059807 learned_addr learned my addr 172.21.15.28:0/2031059807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f75397fa700 10 client.?.objecter ms_handle_connect 0x7f753c10ea80 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f75397fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f75397fa700 10 client.?.objecter ms_handle_connect 0x7f753c102250 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.709+0000 7f75397fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 10 client.?.objecter ms_handle_connect 0x7f753c10aee0 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232533106 0 0) 0x7f753c1aba50 con 0x7f753c10ea80 2024-03-30T15:03:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75280031f0 con 0x7f753c10ea80 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141609516 0 0) 0x7f753c076210 con 0x7f753c102250 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753c1aba50 con 0x7f753c102250 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805479487 0 0) 0x7f753c076420 con 0x7f753c10aee0 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753c076210 con 0x7f753c10aee0 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1221997848 0 0) 0x7f75280031f0 con 0x7f753c10ea80 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f753c076420 con 0x7f753c10ea80 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7534004af0 con 0x7f753c10ea80 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3125158687 0 0) 0x7f753c1aba50 con 0x7f753c102250 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75280031f0 con 0x7f753c102250 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f752c001930 con 0x7f753c102250 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2440766595 0 0) 0x7f753c076210 con 0x7f753c10aee0 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f753c1aba50 con 0x7f753c10aee0 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f75240018d0 con 0x7f753c10aee0 2024-03-30T15:03:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 426262741 0 0) 0x7f753c076420 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 >> v1:172.21.15.134:6789/0 conn(0x7f753c10aee0 legacy=0x7f753c074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 >> v1:172.21.15.28:6789/0 conn(0x7f753c102250 legacy=0x7f753c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c1acc70 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f753c1abc80 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f7542377700 10 client.57959.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f753c1ac140 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f7542377700 1 librados: init done 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7534001de0 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.710+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7534005040 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.711+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f751c005150 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7534003ba0 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(410..410 src has 53..410) ==== 8541+0+0 (unknown 2815930682 0 0) 0x7f7534088a30 con 0x7f753c10ea80 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 10 client.57959.objecter ms_dispatch 0x7f753c001550 osd_map(410..410 src has 53..410) 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 3 client.57959.objecter handle_osd_map got epochs [410,410] > 0 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 3 client.57959.objecter handle_osd_map decoding full epoch 410 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 20 client.57959.objecter dump_active .. 0 homeless 2024-03-30T15:03:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.712+0000 7f75397fa700 10 client.57959.objecter ms_handle_connect 0x7f75280703e0 2024-03-30T15:03:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.717+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7534055ac0 con 0x7f753c10ea80 2024-03-30T15:03:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:01.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:01.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:01.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:01.911 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:01.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:01.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:01.942+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f751c005440 con 0x7f753c10ea80 2024-03-30T15:03:02.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:02.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:02.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:02.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:02.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.312+0000 7f75397fa700 1 -- 172.21.15.28:0/2031059807 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v411) ==== 97+0+0 (unknown 500673491 0 0) 0x7f7534059510 con 0x7f753c10ea80 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 10 librados: watch_flush enter 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 10 librados: watch_flush exit 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 20 client.57959.objecter shutdown clearing up homeless session... 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 10 client.57959.objecter successfully canceled tick 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 >> v1:172.21.15.28:6800/3011431093 conn(0x7f75280703e0 legacy=0x7f75280728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.314+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 >> v1:172.21.15.28:6790/0 conn(0x7f753c10ea80 legacy=0x7f753c1a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.315+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 shutdown_connections 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.315+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 >> 172.21.15.28:0/2031059807 conn(0x7f753c0fdb20 msgr2=0x7f753c1116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.315+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 shutdown_connections 2024-03-30T15:03:02.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.315+0000 7f7542377700 1 -- 172.21.15.28:0/2031059807 wait complete. 2024-03-30T15:03:02.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:02.315+0000 7f7542377700 1 librados: shutdown 2024-03-30T15:03:02.317 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:03:02.380 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:02.380 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:02.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:02.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:02.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:02.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:02.617 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:02.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:02.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2031059807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:02 smithi134 ceph-mon[33714]: from='client.57959 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:02 smithi134 ceph-mon[33714]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:03:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2031059807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[29876]: from='client.57959 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[29876]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2031059807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[41614]: from='client.57959 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[41614]: pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:02.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:02.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:02.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:02.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:03.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:03.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:03:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:03.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:03.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:03.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:03.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:03.278 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:03.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:03.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:03.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:03.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: from='client.57959 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: from='client.57959 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T15:03:03.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:03.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:03.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:03.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: from='client.57959 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:03.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:03:03.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: osdmap e411: 8 total, 8 up, 8 in 2024-03-30T15:03:03.565 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.565 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:03.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:03.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:03.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:03.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:03.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:03.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:03.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:04.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:04.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:04.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:04.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:04.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:04.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:04.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:03:04.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T15:03:04.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[29876]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.358 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.359 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:04 smithi028 ceph-mon[41614]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:04.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]': finished 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: osdmap e412: 8 total, 8 up, 8 in 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-51"}]: dispatch 2024-03-30T15:03:04.412 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:04 smithi134 ceph-mon[33714]: pgmap v562: 260 pgs: 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:04.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:04.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:04.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:04.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:04.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:04.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:04.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:04.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:04.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:04.964 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:04.965 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:05.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:05.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:05.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:03:05.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:05.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:05.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:05.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:05.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[29876]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:03:05.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:03:05.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:05 smithi028 ceph-mon[41614]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T15:03:05.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:05.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.325+0000 7fd4470ec700 1 -- 172.21.15.28:0/3217434170 >> v1:172.21.15.28:6789/0 conn(0x7fd440101140 legacy=0x7fd440101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.326+0000 7fd4470ec700 1 -- 172.21.15.28:0/3217434170 shutdown_connections 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.326+0000 7fd4470ec700 1 -- 172.21.15.28:0/3217434170 >> 172.21.15.28:0/3217434170 conn(0x7fd4400fc9d0 msgr2=0x7fd4400fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.327+0000 7fd4470ec700 1 -- 172.21.15.28:0/3217434170 shutdown_connections 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.327+0000 7fd4470ec700 1 -- 172.21.15.28:0/3217434170 wait complete. 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 librados: starting msgr at 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 librados: starting objecter 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 Processor -- start 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 -- start start 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 librados: setting wanted keys 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.328+0000 7fd4470ec700 1 librados: calling monclient init 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.329+0000 7fd4470ec700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4401a4f50 con 0x7fd44010d970 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.329+0000 7fd4470ec700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4401a6170 con 0x7fd440101140 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.329+0000 7fd4470ec700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4401a73b0 con 0x7fd440109dd0 2024-03-30T15:03:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.329+0000 7fd444e88700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd440101140 0x7fd440106370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:41532/0 (socket says 172.21.15.28:41532) 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.329+0000 7fd444e88700 1 -- 172.21.15.28:0/2904473291 learned_addr learned my addr 172.21.15.28:0/2904473291 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter ms_handle_connect 0x7fd440101140 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter ms_handle_connect 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2888559161 0 0) 0x7fd4401a73b0 con 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4300035b0 con 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter ms_handle_connect 0x7fd44010d970 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3452291496 0 0) 0x7fd4300035b0 con 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4401a73b0 con 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd42c0019d0 con 0x7fd440109dd0 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229269625 0 0) 0x7fd4401a4f50 con 0x7fd44010d970 2024-03-30T15:03:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.330+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4300035b0 con 0x7fd44010d970 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2033294939 0 0) 0x7fd4401a73b0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.134:6789/0 conn(0x7fd440101140 legacy=0x7fd440106370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6789/0 conn(0x7fd44010d970 legacy=0x7fd4401a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4401a85f0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4401a75e0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd4470ec700 10 client.57965.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4401a7aa0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd4470ec700 1 librados: init done 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd42c003b20 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.331+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd42c004ad0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.332+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd424005100 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.333+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd42c0043a0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(414..414 src has 53..414) ==== 8494+0+0 (unknown 2193375239 0 0) 0x7fd42c0886e0 con 0x7fd440109dd0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 osd_map(414..414 src has 53..414) 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 3 client.57965.objecter handle_osd_map got epochs [414,414] > 0 2024-03-30T15:03:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 3 client.57965.objecter handle_osd_map decoding full epoch 414 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 20 client.57965.objecter dump_active .. 0 homeless 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v414) ==== 54+0+16 (unknown 330119727 0 2420000192) 0x7fd42c08ab00 con 0x7fd440109dd0 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.334+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd430074810 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:05.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 10 client.57965.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 1 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 1 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 0 1 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 4 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 10 client.57965.objecter _send_command 1 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.335+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd424004b50 con 0x7fd424004730 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.336+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd424004730 2024-03-30T15:03:05.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.337+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd424004b50 con 0x7fd424004730 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.337+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.337+0000 7fd43dffb700 10 client.57965.objecter _finish_command 1 = system:0 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.337+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 4 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.337+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 0 1 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 10 client.57965.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 2 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 2 2024-03-30T15:03:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd4240053e0 osd=0 3 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 0 2 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 4 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 10 client.57965.objecter _send_command 2 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.355+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd424007b00 con 0x7fd424004730 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.357+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd424007b00 con 0x7fd424004730 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.357+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.357+0000 7fd43dffb700 10 client.57965.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.357+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 4 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.357+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 0 2 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 10 client.57965.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 3 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 4 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 10 client.57965.objecter _send_command 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.358+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd424002160 con 0x7fd4240042a0 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd4240042a0 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd424002160 con 0x7fd4240042a0 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 10 client.57965.objecter _finish_command 3 = system:0 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 4 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.359+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 1 3 2024-03-30T15:03:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 10 client.57965.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 4 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 4 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424004130 osd=1 3 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 1 4 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 4 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 10 client.57965.objecter _send_command 4 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.381+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd424001600 con 0x7fd4240042a0 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd424001600 con 0x7fd4240042a0 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd43dffb700 10 client.57965.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 4 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 1 4 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 10 client.57965.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 5 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 5 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 2 5 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 4 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 10 client.57965.objecter _send_command 5 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.383+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd424011010 con 0x7fd424008990 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.384+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd424008990 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.385+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd424011010 con 0x7fd424008990 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.385+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.385+0000 7fd43dffb700 10 client.57965.objecter _finish_command 5 = system:0 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.385+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 4 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.385+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 2 5 2024-03-30T15:03:05.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 10 client.57965.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 6 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 6 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424001a00 osd=2 3 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 2 6 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 4 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 10 client.57965.objecter _send_command 6 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.403+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd424008020 con 0x7fd424008990 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.404+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd424008020 con 0x7fd424008990 2024-03-30T15:03:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.404+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.404+0000 7fd43dffb700 10 client.57965.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.404+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 4 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.404+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 2 6 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 10 client.57965.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 7 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 7 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 3 7 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 4 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 10 client.57965.objecter _send_command 7 2024-03-30T15:03:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd424011d10 con 0x7fd4240111c0 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.405+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd4240111c0 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.406+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd424011d10 con 0x7fd4240111c0 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.406+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.406+0000 7fd43dffb700 10 client.57965.objecter _finish_command 7 = system:0 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.406+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 4 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.406+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 3 7 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 10 client.57965.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 8 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 8 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd424008270 osd=3 3 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 3 8 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 4 2024-03-30T15:03:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 10 client.57965.objecter _send_command 8 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.423+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd4240126c0 con 0x7fd4240111c0 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd4240126c0 con 0x7fd4240111c0 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 10 client.57965.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 4 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 3 8 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 10 client.57965.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 9 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 9 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 4 9 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 4 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 10 client.57965.objecter _send_command 9 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd42401a890 con 0x7fd424016bd0 2024-03-30T15:03:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.425+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd424016bd0 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.426+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd42401a890 con 0x7fd424016bd0 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.426+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.426+0000 7fd43dffb700 10 client.57965.objecter _finish_command 9 = system:0 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.426+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 4 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.426+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 4 9 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 10 client.57965.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 10 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 10 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd4240126c0 osd=4 3 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 4 10 2024-03-30T15:03:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 4 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 10 client.57965.objecter _send_command 10 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.443+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd42401aa70 con 0x7fd424016bd0 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd42401aa70 con 0x7fd424016bd0 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 10 client.57965.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 4 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 4 10 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:05.503 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 10 client.57965.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 11 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 11 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 5 11 2024-03-30T15:03:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 4 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 10 client.57965.objecter _send_command 11 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd42401bd10 con 0x7fd42401b190 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.445+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd42401b190 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.446+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd42401bd10 con 0x7fd42401b190 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.446+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.446+0000 7fd43dffb700 10 client.57965.objecter _finish_command 11 = system:0 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.446+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 4 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.446+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 5 11 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 10 client.57965.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 12 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 12 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42401aa70 osd=5 3 2024-03-30T15:03:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 5 12 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 4 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 10 client.57965.objecter _send_command 12 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.463+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd42401bf90 con 0x7fd42401b190 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd42401bf90 con 0x7fd42401b190 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 10 client.57965.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 4 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 5 12 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 10 client.57965.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 13 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 13 2024-03-30T15:03:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 6 13 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 4 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 10 client.57965.objecter _send_command 13 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd424023280 con 0x7fd42401f770 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.465+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd42401f770 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.466+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd424023280 con 0x7fd42401f770 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.466+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.466+0000 7fd43dffb700 10 client.57965.objecter _finish_command 13 = system:0 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.466+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 4 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.466+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 6 13 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 10 client.57965.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 14 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 14 2024-03-30T15:03:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42401bd70 osd=6 3 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 6 14 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 4 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 10 client.57965.objecter _send_command 14 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.483+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd42400c870 con 0x7fd42401f770 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd42400c870 con 0x7fd42401f770 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 10 client.57965.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 4 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 6 14 2024-03-30T15:03:05.508 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 10 client.57965.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 15 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 15 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 7 15 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 4 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 10 client.57965.objecter _send_command 15 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd4240135f0 con 0x7fd424013190 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.485+0000 7fd43dffb700 10 client.57965.objecter ms_handle_connect 0x7fd424013190 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.486+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd4240135f0 con 0x7fd424013190 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.486+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.486+0000 7fd43dffb700 10 client.57965.objecter _finish_command 15 = system:0 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.486+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 4 2024-03-30T15:03:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.486+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 7 15 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 10 client.57965.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign -1 16 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 20 client.57965.objecter _get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_remove -1 16 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 20 client.57965.objecter get_session s=0x7fd42400c870 osd=7 3 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 15 client.57965.objecter _session_command_op_assign 7 16 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 4 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 10 client.57965.objecter _send_command 16 2024-03-30T15:03:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.509+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd4240135f0 con 0x7fd424013190 2024-03-30T15:03:05.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.511+0000 7fd43dffb700 1 -- 172.21.15.28:0/2904473291 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd4240135f0 con 0x7fd424013190 2024-03-30T15:03:05.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.511+0000 7fd43dffb700 10 client.57965.objecter ms_dispatch 0x7fd440001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:05.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.511+0000 7fd43dffb700 10 client.57965.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:05.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.511+0000 7fd43dffb700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 4 2024-03-30T15:03:05.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.511+0000 7fd43dffb700 15 client.57965.objecter _session_command_op_remove 7 16 2024-03-30T15:03:05.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 librados: watch_flush enter 2024-03-30T15:03:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 librados: watch_flush exit 2024-03-30T15:03:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.0 2024-03-30T15:03:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6801/3394323575 conn(0x7fd424004730 legacy=0x7fd424002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240053e0 osd=0 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6805/1850313547 conn(0x7fd4240042a0 legacy=0x7fd42400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424004130 osd=1 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.2 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6809/3451325043 conn(0x7fd424008990 legacy=0x7fd42400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424001a00 osd=2 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.3 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6813/1506157250 conn(0x7fd4240111c0 legacy=0x7fd424011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd424008270 osd=3 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.4 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.134:6800/3063775514 conn(0x7fd424016bd0 legacy=0x7fd424018fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd4240126c0 osd=4 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.5 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.134:6804/3668151179 conn(0x7fd42401b190 legacy=0x7fd42401b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401aa70 osd=5 1 2024-03-30T15:03:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.6 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd445689700 1 -- 172.21.15.28:0/2904473291 reap_dead start 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.134:6808/2135224194 conn(0x7fd42401f770 legacy=0x7fd424021b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd445689700 1 -- 172.21.15.28:0/2904473291 reap_dead start 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd445689700 1 -- 172.21.15.28:0/2904473291 reap_dead start 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42401bd70 osd=6 1 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter close_session for osd.7 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.134:6812/3758190298 conn(0x7fd424013190 legacy=0x7fd424024400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter put_session s=0x7fd42400c870 osd=7 1 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 20 client.57965.objecter shutdown clearing up homeless session... 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 10 client.57965.objecter successfully canceled tick 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd430074810 legacy=0x7fd430076cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> v1:172.21.15.28:6790/0 conn(0x7fd440109dd0 legacy=0x7fd44019fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 shutdown_connections 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 >> 172.21.15.28:0/2904473291 conn(0x7fd4400fc9d0 msgr2=0x7fd4400fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 shutdown_connections 2024-03-30T15:03:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 -- 172.21.15.28:0/2904473291 wait complete. 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:05.512+0000 7fd4470ec700 1 librados: shutdown 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:05.539 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.563 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:05.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:05.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:05 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:03:05.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:05 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:03:05.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:05 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:03:05.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:05 smithi134 ceph-mon[33714]: osdmap e413: 8 total, 8 up, 8 in 2024-03-30T15:03:05.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:03:05.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:05.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:05.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:05.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:05.883 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:05.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:05.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:05.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:06.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.088 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:06.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:06.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:06.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- 172.21.15.28:0/2469600101 >> v1:172.21.15.28:6790/0 conn(0x7f68a8109dc0 legacy=0x7f68a810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- 172.21.15.28:0/2469600101 shutdown_connections 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- 172.21.15.28:0/2469600101 >> 172.21.15.28:0/2469600101 conn(0x7f68a80fc9e0 msgr2=0x7f68a80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- 172.21.15.28:0/2469600101 shutdown_connections 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- 172.21.15.28:0/2469600101 wait complete. 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 librados: starting msgr at 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 librados: starting objecter 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 Processor -- start 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 -- start start 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 librados: setting wanted keys 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.263+0000 7f68b05f6700 1 librados: calling monclient init 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68b05f6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f68a81a9220 con 0x7f68a8101130 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68b05f6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f68a81aa440 con 0x7f68a8109dc0 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68b05f6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f68a81ab680 con 0x7f68a810d960 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68aeb93700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f68a810d960 0x7f68a81a7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60590/0 (socket says 172.21.15.28:60590) 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68ae392700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f68a8101130 0x7f68a8106320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43850/0 (socket says 172.21.15.28:43850) 2024-03-30T15:03:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68aeb93700 1 -- 172.21.15.28:0/1449361983 learned_addr learned my addr 172.21.15.28:0/1449361983 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter ms_handle_connect 0x7f68a8101130 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter ms_handle_connect 0x7f68a810d960 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter ms_handle_connect 0x7f68a8109dc0 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143761418 0 0) 0x7f68a81ab680 con 0x7f68a810d960 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68900031f0 con 0x7f68a810d960 2024-03-30T15:03:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244541338 0 0) 0x7f68a81a9220 con 0x7f68a8101130 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a81ab680 con 0x7f68a8101130 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221622673 0 0) 0x7f68a81aa440 con 0x7f68a8109dc0 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a81a9220 con 0x7f68a8109dc0 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1408464737 0 0) 0x7f68a81a9220 con 0x7f68a8109dc0 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68a81aa440 con 0x7f68a8109dc0 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f689c0019e0 con 0x7f68a8109dc0 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1147561708 0 0) 0x7f68900031f0 con 0x7f68a810d960 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68a81a9220 con 0x7f68a810d960 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4184578434 0 0) 0x7f68a81ab680 con 0x7f68a8101130 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68900031f0 con 0x7f68a8101130 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6898004310 con 0x7f68a810d960 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.264+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f68a0001890 con 0x7f68a8101130 2024-03-30T15:03:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3421788518 0 0) 0x7f68a81aa440 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 >> v1:172.21.15.28:6790/0 conn(0x7f68a810d960 legacy=0x7f68a81a7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 >> v1:172.21.15.28:6789/0 conn(0x7f68a8101130 legacy=0x7f68a8106320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a81ac8c0 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f68a81ab8b0 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 10 client.49987.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f68a81abd70 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 1 librados: init done 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f689c004760 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f689c0050b0 con 0x7f68a8109dc0 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 10 client.49987.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 15 client.49987.objecter _session_command_op_assign -1 1 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 20 client.49987.objecter _get_session osd=-1 returning homeless 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 10 client.49987.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.265+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6894002920 con 0x7f68a8109dc0 2024-03-30T15:03:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.266+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f689c003860 con 0x7f68a8109dc0 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(414..414 src has 53..414) ==== 8494+0+0 (unknown 2193375239 0 0) 0x7f689c088640 con 0x7f68a8109dc0 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 10 client.49987.objecter ms_dispatch 0x7f68a8001550 osd_map(414..414 src has 53..414) 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 3 client.49987.objecter handle_osd_map got epochs [414,414] > 0 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 3 client.49987.objecter handle_osd_map decoding full epoch 414 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 10 client.49987.objecter checking command 1 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter _get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 15 client.49987.objecter _session_command_op_remove -1 1 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter _get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 15 client.49987.objecter _session_command_op_assign 6 1 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 4 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 10 client.49987.objecter _send_command 1 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6808/2135224194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f689c001ce0 con 0x7f689007edc0 2024-03-30T15:03:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 20 client.49987.objecter dump_active .. 0 homeless 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=414) ==== 24+0+0 (unknown 898903918 0 0) 0x7f689c088a20 con 0x7f68a8109dc0 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 10 client.49987.objecter ms_handle_connect 0x7f6890070500 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.267+0000 7f68a77fe700 10 client.49987.objecter ms_handle_connect 0x7f689007edc0 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.268+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f689c001ce0 con 0x7f689007edc0 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.268+0000 7f68a77fe700 10 client.49987.objecter ms_dispatch 0x7f68a8001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.268+0000 7f68a77fe700 10 client.49987.objecter _finish_command 1 = system:0 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.268+0000 7f68a77fe700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 4 2024-03-30T15:03:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.268+0000 7f68a77fe700 15 client.49987.objecter _session_command_op_remove 6 1 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 10 client.49987.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 15 client.49987.objecter _session_command_op_assign -1 2 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 20 client.49987.objecter _get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 20 client.49987.objecter _get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 15 client.49987.objecter _session_command_op_remove -1 2 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 20 client.49987.objecter get_session s=0x7f689007e710 osd=6 3 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 15 client.49987.objecter _session_command_op_assign 6 2 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 4 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 10 client.49987.objecter _send_command 2 2024-03-30T15:03:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.289+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 --> v1:172.21.15.134:6808/2135224194 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f68940011f0 con 0x7f689007edc0 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68a77fe700 1 -- 172.21.15.28:0/1449361983 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f68940011f0 con 0x7f689007edc0 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68a77fe700 10 client.49987.objecter ms_dispatch 0x7f68a8001550 command_reply(tid 2: 0 ) 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68a77fe700 10 client.49987.objecter _finish_command 2 = system:0 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68a77fe700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 4 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68a77fe700 15 client.49987.objecter _session_command_op_remove 6 2 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 10 librados: watch_flush enter 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 10 librados: watch_flush exit 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 10 client.49987.objecter close_session for osd.6 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 >> v1:172.21.15.134:6808/2135224194 conn(0x7f689007edc0 legacy=0x7f6890081260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 20 client.49987.objecter put_session s=0x7f689007e710 osd=6 1 2024-03-30T15:03:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 20 client.49987.objecter shutdown clearing up homeless session... 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 10 client.49987.objecter successfully canceled tick 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6890070500 legacy=0x7f68900729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 >> v1:172.21.15.134:6789/0 conn(0x7f68a8109dc0 legacy=0x7f68a81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.290+0000 7f68aeb93700 1 -- 172.21.15.28:0/1449361983 reap_dead start 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.291+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 shutdown_connections 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.291+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 >> 172.21.15.28:0/1449361983 conn(0x7f68a80fc9e0 msgr2=0x7f68a8106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.291+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 shutdown_connections 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.291+0000 7f68b05f6700 1 -- 172.21.15.28:0/1449361983 wait complete. 2024-03-30T15:03:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:06.291+0000 7f68b05f6700 1 librados: shutdown 2024-03-30T15:03:06.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:06.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:06.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:06.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[41614]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T15:03:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2904473291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[41614]: pgmap v565: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[29876]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2904473291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:06.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:06 smithi028 ceph-mon[29876]: pgmap v565: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T15:03:06.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:06 smithi134 ceph-mon[33714]: osdmap e414: 8 total, 8 up, 8 in 2024-03-30T15:03:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2904473291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:06 smithi134 ceph-mon[33714]: pgmap v565: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-30T15:03:06.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:06.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:06.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:06.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:06.909 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:06.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:06.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:06.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:07.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:07.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:07.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:07.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:07.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:07.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:07.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:07.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:07.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:07.631 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:07 smithi134 ceph-mon[33714]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:917 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94326471858816 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:54:57.389505+0000", 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 490.239019923, 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0143281449999999, 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.632 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 917 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389505+0000", 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389505+0000", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389508+0000", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9900000000000002e-06 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389519+0000", 2024-03-30T15:03:07.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0617e-05 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389536+0000", 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7238e-05 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.389550+0000", 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3613e-05 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.396980+0000", 2024-03-30T15:03:07.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074302750000000001 2024-03-30T15:03:07.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.396990+0000", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6569999999999992e-06 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.400791+0000", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0038010900000001 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.400953+0000", 2024-03-30T15:03:07.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016236500000000001 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402085+0000", 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011316469999999999 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402190+0000", 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010480999999999999 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402593+0000", 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040285000000000001 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402647+0000", 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4255000000000001e-05 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403834+0000", 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011867379999999999 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:924 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94326471858816 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:54:57.517418+0000", 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 490.11110687399997, 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.88591363700000003, 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 924 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.640 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517418+0000", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517418+0000", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517420+0000", 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0920000000000001e-06 2024-03-30T15:03:07.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517425+0000", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.835e-06 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517437+0000", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1479e-05 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517448+0000", 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0969e-05 2024-03-30T15:03:07.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517523+0000", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4740000000000006e-05 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:57.517534+0000", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1192999999999999e-05 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.401224+0000", 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.88369055500000004 2024-03-30T15:03:07.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.401274+0000", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9820000000000001e-05 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402290+0000", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010158739999999999 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.402387+0000", 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7085999999999998e-05 2024-03-30T15:03:07.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403039+0000", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00065200800000000004 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403122+0000", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2751999999999996e-05 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:54:58.403332+0000", 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021023400000000001 2024-03-30T15:03:07.645 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25069.0:3 35.f 35:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 312.42814840099999, 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0835292080000001, 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25069", 2024-03-30T15:03:07.646 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3135164836", 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200379+0000", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6729999999999999e-06 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200387+0000", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1799999999999996e-06 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200395+0000", 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3850000000000002e-06 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200404+0000", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6219999999999998e-06 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200468+0000", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4142999999999999e-05 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200475+0000", 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4580000000000004e-06 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259289+0000", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0588138250000001 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259412+0000", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012236699999999999 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,6", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.260458+0000", 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010460790000000001 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.261342+0000", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00088464100000000003 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.268237+0000", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068943329999999999 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283840+0000", 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015603258 2024-03-30T15:03:07.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283874+0000", 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3736999999999999e-05 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283906+0000", 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2506999999999999e-05 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25048.0:1 46.6s0 46:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:03:07.652 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 311.49571708100001, 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2874541669999999, 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25048", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:03:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132810+0000", 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0250000000000001e-06 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132815+0000", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9869999999999999e-06 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132823+0000", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0560000000000005e-06 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132833+0000", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5200000000000003e-06 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145608+0000", 2024-03-30T15:03:07.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012775376999999999 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145613+0000", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5079999999999999e-06 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397079+0000", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2514661540000001 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397194+0000", 2024-03-30T15:03:07.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115407 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397531+0000", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000336745 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.402316+0000", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047853660000000001 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420194+0000", 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017877613000000001 2024-03-30T15:03:07.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420262+0000", 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8409000000000002e-05 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34886.0:1 50.0s0 50:0c828f53:test-rados-api-smithi028-93540-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 310.291284873, 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3929710050000002, 2024-03-30T15:03:07.658 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.659 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.659 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.659 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.659 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34886", 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4069139007", 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337242+0000", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.792e-06 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337249+0000", 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9430000000000003e-06 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337260+0000", 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0535e-05 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337269+0000", 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8009999999999993e-06 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342443+0000", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051742710000000003 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342445+0000", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.446e-06 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718228+0000", 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3757831299999999 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718356+0000", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012730400000000001 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719072+0000", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071628200000000005 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.720051+0000", 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00097890299999999993 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730124+0000", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010073343 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730212+0000", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7255000000000004e-05 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2059 11.7 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head [writefull 0~25165836] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:03:07.665 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 309.68274377500001, 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.086346471999999994, 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2059 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945784+0000", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0310000000000001e-06 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969470+0000", 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023686577 2024-03-30T15:03:07.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969496+0000", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6210000000000001e-05 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969507+0000", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0723000000000001e-05 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969596+0000", 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8653999999999995e-05 2024-03-30T15:03:07.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969758+0000", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016222 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969914+0000", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015554699999999999 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030515+0000", 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060601742 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030889+0000", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037352300000000001 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032049+0000", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011604639999999999 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032088+0000", 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8621000000000002e-05 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032128+0000", 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0160000000000002e-05 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:5 13.b 13:d4d78c7e:test-rados-api-smithi028-93525-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 309.15614914299999, 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15821444200000001, 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.671 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 5 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472378+0000", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2000000000000001e-06 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472386+0000", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8760000000000005e-06 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472397+0000", 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0900000000000001e-05 2024-03-30T15:03:07.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472409+0000", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1379999999999999e-05 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472488+0000", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9796999999999995e-05 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472495+0000", 2024-03-30T15:03:07.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5749999999999997e-06 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517331+0000", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044836187 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517472+0000", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014135100000000001 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.521687+0000", 2024-03-30T15:03:07.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042141230000000002 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.534881+0000", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01319443 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.553213+0000", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018331719 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630475+0000", 2024-03-30T15:03:07.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.077262443 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630530+0000", 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4910000000000001e-05 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630591+0000", 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0550999999999997e-05 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.677 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25483.0:2 63.1b 63:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 306.63903683799998, 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9747056280000002, 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25483", 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1101472932", 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989490+0000", 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5880000000000001e-06 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989498+0000", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.396e-06 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989506+0000", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8459999999999997e-06 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989518+0000", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1309e-05 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989578+0000", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0393000000000003e-05 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989589+0000", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0829000000000001e-05 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914500+0000", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9249114629999999 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914631+0000", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013069899999999999 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914772+0000", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014078499999999999 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.917517+0000", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027451939999999998 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962191+0000", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044673933999999998 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964110+0000", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019193019999999999 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964150+0000", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0212000000000001e-05 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964194+0000", 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3677999999999998e-05 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35456.0:2 69.0 69:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 305.83925130799997, 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729640450000001, 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.684 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35456", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3389323125", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789276+0000", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.317e-06 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789282+0000", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0510000000000002e-06 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789297+0000", 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4766999999999999e-05 2024-03-30T15:03:07.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789307+0000", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8810000000000008e-06 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806392+0000", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017084881 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806402+0000", 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9089999999999995e-06 2024-03-30T15:03:07.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885233+0000", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0788315669999999 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885345+0000", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111907 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885469+0000", 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012389899999999999 2024-03-30T15:03:07.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.901898+0000", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016428991 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.902088+0000", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000189459 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962138+0000", 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060050234000000001 2024-03-30T15:03:07.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962190+0000", 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2432000000000002e-05 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962238+0000", 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7750000000000002e-05 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25489.0:3 64.f 64:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 304.71432359200003, 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.084413231000000005, 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25489", 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2934336176", 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914204+0000", 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.905e-06 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914214+0000", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9399999999999997e-06 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914224+0000", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0190999999999999e-05 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914235+0000", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0686999999999999e-05 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914298+0000", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3063000000000003e-05 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914308+0000", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0642000000000001e-05 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995802+0000", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.081493828000000004 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995889+0000", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7015000000000006e-05 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995993+0000", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104177 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996545+0000", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055223100000000003 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996864+0000", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031808999999999999 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998524+0000", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016600040000000001 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998579+0000", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5368000000000003e-05 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998615+0000", 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6090000000000002e-05 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49076.0:2 135.1b 135:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 278.45494176800003, 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0617338199999999, 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49076", 2024-03-30T15:03:07.697 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2505490190", 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173586+0000", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9530000000000002e-06 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173594+0000", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4069999999999993e-06 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173615+0000", 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0788000000000002e-05 2024-03-30T15:03:07.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173632+0000", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7589999999999999e-05 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176962+0000", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033293039999999999 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176970+0000", 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9889999999999992e-06 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226049+0000", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.049079015 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226159+0000", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110021 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226288+0000", 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012942800000000001 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.229542+0000", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032538939999999998 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.231623+0000", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020808440000000001 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235241+0000", 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036180539999999999 2024-03-30T15:03:07.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235288+0000", 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7382000000000001e-05 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235317+0000", 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9150999999999999e-05 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:23 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [write 0~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:03:07.706 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 276.28210295899999, 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12244129300000001, 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 23 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346424+0000", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9690000000000001e-06 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346432+0000", 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5979999999999999e-06 2024-03-30T15:03:07.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346442+0000", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0431000000000001e-05 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346453+0000", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0896e-05 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356731+0000", 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010277319 2024-03-30T15:03:07.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356836+0000", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105664 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.357052+0000", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021593299999999999 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.366074+0000", 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090220760000000004 2024-03-30T15:03:07.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.388374+0000", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022299229 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468758+0000", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080384786999999999 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468818+0000", 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9493000000000001e-05 2024-03-30T15:03:07.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468864+0000", 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5898e-05 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:25 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [rollback 8] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 274.416631532, 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12770631599999999, 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.712 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 25 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:03:07.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211896+0000", 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0789999999999999e-06 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211903+0000", 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2359999999999998e-06 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211912+0000", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4109999999999996e-06 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211923+0000", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1038e-05 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212228+0000", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030524600000000001 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212330+0000", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102433 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212488+0000", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015767099999999999 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.213555+0000", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001067045 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.292128+0000", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.078572806999999995 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339484+0000", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.047356374999999999 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339548+0000", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4126000000000002e-05 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.717 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339600+0000", 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1848999999999998e-05 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57281.0:1 221.6s0 221:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 204.420284106, 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1519453040000001, 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.718 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57281", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4286010033", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208243+0000", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1550000000000001e-06 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208253+0000", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3139999999999993e-06 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208264+0000", 2024-03-30T15:03:07.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1592e-05 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208277+0000", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2703e-05 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209635+0000", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001357638 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209639+0000", 2024-03-30T15:03:07.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6949999999999999e-06 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358223+0000", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1485833839999999 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358319+0000", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6644999999999994e-05 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358620+0000", 2024-03-30T15:03:07.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030071200000000002 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.359145+0000", 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052535199999999998 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360113+0000", 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096798800000000003 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360187+0000", 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3126000000000004e-05 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49354.0:1 224.6s1 224:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 197.424270302, 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025966369, 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.724 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49354", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1904812330", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204256+0000", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.39e-06 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204263+0000", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0669999999999997e-06 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204271+0000", 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1880000000000002e-06 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204279+0000", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5639999999999997e-06 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228587+0000", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.024308065 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228592+0000", 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-03-30T15:03:07.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222766+0000", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99417422799999999 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222903+0000", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013736600000000001 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.223366+0000", 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000463187 2024-03-30T15:03:07.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.226776+0000", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003409287 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230143+0000", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033669149999999998 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230221+0000", 2024-03-30T15:03:07.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8844000000000006e-05 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57626.0:1 262.5s0 262:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 123.392955151, 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0934634089999999, 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.730 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57626", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1398925", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235572+0000", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9889999999999999e-06 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235581+0000", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8370000000000003e-06 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235591+0000", 2024-03-30T15:03:07.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0292000000000001e-05 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235602+0000", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0976e-05 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237121+0000", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015190379999999999 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237126+0000", 2024-03-30T15:03:07.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1679999999999997e-06 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326125+0000", 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0889992150000001 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326273+0000", 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014821499999999999 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326919+0000", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064574200000000004 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.327411+0000", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00049205899999999999 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.328990+0000", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001578861 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.329034+0000", 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4017000000000002e-05 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49657.0:1 263.6s0 263:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 120.400076656, 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.017319423, 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.736 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49657", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2923162696", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228451+0000", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4700000000000001e-06 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228459+0000", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8490000000000006e-06 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228469+0000", 2024-03-30T15:03:07.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5219999999999996e-06 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228478+0000", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2150000000000007e-06 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238328+0000", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098497410000000004 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238331+0000", 2024-03-30T15:03:07.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9009999999999996e-06 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242049+0000", 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0037178710000001 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242184+0000", 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0001351 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242588+0000", 2024-03-30T15:03:07.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:07.742 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[29876]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: osdmap e415: 8 total, 8 up, 8 in 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-53"}]: dispatch 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:07.743 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:07 smithi028 ceph-mon[41614]: osdmap e416: 8 total, 8 up, 8 in 2024-03-30T15:03:07.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040368400000000002 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.244103+0000", 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015145740000000001 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245701+0000", 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015980180000000001 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245768+0000", 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7477999999999993e-05 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24537.0:1 270.6s0 270:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 99.371670791, 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0134851680000001, 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24537", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2126246388", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256858+0000", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6389999999999998e-06 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256868+0000", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0231000000000001e-05 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256886+0000", 2024-03-30T15:03:07.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7988000000000001e-05 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256901+0000", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4817999999999999e-05 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261660+0000", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004759012 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261665+0000", 2024-03-30T15:03:07.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0649999999999998e-06 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267449+0000", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0057839420000001 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267596+0000", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014701699999999999 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.268682+0000", 2024-03-30T15:03:07.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001085325 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.269079+0000", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039733100000000001 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270298+0000", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001219354 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270340+0000", 2024-03-30T15:03:07.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1446000000000003e-05 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57869.0:1 286.5s0 286:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 40.354030647000002, 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0109918019999999, 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57869", 2024-03-30T15:03:07.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/714138788", 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274498+0000", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9220000000000001e-06 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274509+0000", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1678999999999999e-05 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274525+0000", 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5781e-05 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274540+0000", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4808e-05 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277431+0000", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028910849999999998 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277436+0000", 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4589999999999997e-06 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.282966+0000", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0055296760000001 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283097+0000", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013078699999999999 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283531+0000", 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043440599999999997 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283992+0000", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046019300000000002 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285413+0000", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014217050000000001 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285487+0000", 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3300999999999995e-05 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57923.0:1 290.7s0 290:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.368130260000001, 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.011345699, 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57923", 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3909045240", 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.757 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260397+0000", 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.733e-06 2024-03-30T15:03:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260404+0000", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.221e-06 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260415+0000", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0659000000000001e-05 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260425+0000", 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0703999999999999e-05 2024-03-30T15:03:07.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260972+0000", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054698500000000005 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260976+0000", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1020000000000001e-06 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269376+0000", 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.008400816 2024-03-30T15:03:07.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269503+0000", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126236 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269916+0000", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000413743 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271218+0000", 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013020810000000001 2024-03-30T15:03:07.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271674+0000", 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00045557200000000001 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271741+0000", 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6847000000000006e-05 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:07.762 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:03:07.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-30T15:03:07.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:07.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:07.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:08.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:08.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:08.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:08.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:08.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:08.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:08.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:08.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:08.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:08.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:03:08.600 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:08.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:08.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:08.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:08.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:03:08.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:08.835 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:08.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:08.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:08.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:09.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:09.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:09.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:09.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:09.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:09.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:09.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[29876]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[29876]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[41614]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[41614]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.516 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:09 smithi134 ceph-mon[33714]: pgmap v568: 228 pgs: 228 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:09.516 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:09 smithi134 ceph-mon[33714]: osdmap e417: 8 total, 8 up, 8 in 2024-03-30T15:03:09.516 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.516 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.529+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/558497303 >> v1:172.21.15.28:6789/0 conn(0x7f0ac010ea80 legacy=0x7f0ac0110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.529+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/558497303 shutdown_connections 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.529+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/558497303 >> 172.21.15.28:0/558497303 conn(0x7f0ac00fdb20 msgr2=0x7f0ac00fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.529+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/558497303 shutdown_connections 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.529+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/558497303 wait complete. 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 librados: starting msgr at 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 librados: starting objecter 2024-03-30T15:03:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 Processor -- start 2024-03-30T15:03:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 -- start start 2024-03-30T15:03:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 librados: setting wanted keys 2024-03-30T15:03:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 librados: calling monclient init 2024-03-30T15:03:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ac01a7170 con 0x7f0ac0102250 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ac01a8390 con 0x7f0ac010ea80 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ac4c1a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0ac01a95d0 con 0x7f0ac010aee0 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0abe59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ac0102250 0x7f0ac0107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51656/0 (socket says 172.21.15.28:51656) 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0abe59c700 1 -- 172.21.15.28:0/3063497253 learned_addr learned my addr 172.21.15.28:0/3063497253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ab77fe700 10 client.?.objecter ms_handle_connect 0x7f0ac010aee0 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ab77fe700 10 client.?.objecter ms_handle_connect 0x7f0ac0102250 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.530+0000 7f0ab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 10 client.?.objecter ms_handle_connect 0x7f0ac010ea80 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3962564807 0 0) 0x7f0ac01a7170 con 0x7f0ac0102250 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0aac0031f0 con 0x7f0ac0102250 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794990978 0 0) 0x7f0ac01a95d0 con 0x7f0ac010aee0 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ac01a7170 con 0x7f0ac010aee0 2024-03-30T15:03:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 139296998 0 0) 0x7f0ac01a8390 con 0x7f0ac010ea80 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ac01a95d0 con 0x7f0ac010ea80 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2570948249 0 0) 0x7f0aac0031f0 con 0x7f0ac0102250 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ac01a8390 con 0x7f0ac0102250 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0aa8001930 con 0x7f0ac0102250 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3055279789 0 0) 0x7f0ac01a7170 con 0x7f0ac010aee0 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0aac0031f0 con 0x7f0ac010aee0 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0ab0001890 con 0x7f0ac010aee0 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 454392691 0 0) 0x7f0ac01a95d0 con 0x7f0ac010ea80 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ac01a7170 con 0x7f0ac010ea80 2024-03-30T15:03:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0ab8004730 con 0x7f0ac010ea80 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1176391594 0 0) 0x7f0ac01a8390 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 >> v1:172.21.15.28:6790/0 conn(0x7f0ac010aee0 legacy=0x7f0ac01a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 >> v1:172.21.15.134:6789/0 conn(0x7f0ac010ea80 legacy=0x7f0ac01a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac01aa810 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac01a73a0 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ac4c1a700 10 client.24789.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ac01a7860 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ac4c1a700 1 librados: init done 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0aa8001d00 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.531+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0aa8005d70 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.532+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa0005120 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0aa801c670 con 0x7f0ac0102250 2024-03-30T15:03:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(418..418 src has 53..418) ==== 8507+0+0 (unknown 467308374 0 0) 0x7f0aa80899d0 con 0x7f0ac0102250 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 10 client.24789.objecter ms_dispatch 0x7f0ac0001550 osd_map(418..418 src has 53..418) 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 3 client.24789.objecter handle_osd_map got epochs [418,418] > 0 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 3 client.24789.objecter handle_osd_map decoding full epoch 418 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.533+0000 7f0ab77fe700 20 client.24789.objecter dump_active .. 0 homeless 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.534+0000 7f0ab77fe700 10 client.24789.objecter ms_handle_connect 0x7f0aac070390 2024-03-30T15:03:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.538+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0aa8056a80 con 0x7f0ac0102250 2024-03-30T15:03:09.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:09.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:09.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:09.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:09.757+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f0aa0005410 con 0x7f0ac0102250 2024-03-30T15:03:09.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:09.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:09.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:09.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:09.959 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:10.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:10.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:10.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:10.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:10.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.261+0000 7f0ab77fe700 1 -- 172.21.15.28:0/3063497253 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v419) ==== 91+0+0 (unknown 113733077 0 0) 0x7f0aa805a4d0 con 0x7f0ac0102250 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.264+0000 7f0ac4c1a700 10 librados: watch_flush enter 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.264+0000 7f0ac4c1a700 10 librados: watch_flush exit 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.264+0000 7f0ac4c1a700 20 client.24789.objecter shutdown clearing up homeless session... 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.264+0000 7f0ac4c1a700 10 client.24789.objecter successfully canceled tick 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.265+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0aac070390 legacy=0x7f0aac072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.265+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 >> v1:172.21.15.28:6789/0 conn(0x7f0ac0102250 legacy=0x7f0ac0107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.266+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 shutdown_connections 2024-03-30T15:03:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.266+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 >> 172.21.15.28:0/3063497253 conn(0x7f0ac00fdb20 msgr2=0x7f0ac01116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.266+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 shutdown_connections 2024-03-30T15:03:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.266+0000 7f0ac4c1a700 1 -- 172.21.15.28:0/3063497253 wait complete. 2024-03-30T15:03:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:10.266+0000 7f0ac4c1a700 1 librados: shutdown 2024-03-30T15:03:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:03:10.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:10.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:10.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T15:03:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:10.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:10 smithi134 ceph-mon[33714]: pgmap v571: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:10.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:10.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:10.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[41614]: pgmap v571: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: osdmap e418: 8 total, 8 up, 8 in 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:10.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-55"}]: dispatch 2024-03-30T15:03:10.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:10.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:10 smithi028 ceph-mon[29876]: pgmap v571: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:10.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:10.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:10.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:10.699 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:03:10.699 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:03:10.699 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:03:10.699 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:03:10.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:10.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:10.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:10.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:10.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:11.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:11.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:11.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:11.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:03:11.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:11.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:11.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[29876]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[41614]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T15:03:11.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:11.545 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:11.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:11.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:11.653 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:11 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:11.653 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3063497253' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:11.653 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:11 smithi134 ceph-mon[33714]: osdmap e419: 8 total, 8 up, 8 in 2024-03-30T15:03:11.653 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:11.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:11.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:11.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:11.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:11.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:11.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:12.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:12.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.254+0000 7ff57d496700 1 -- 172.21.15.28:0/727935506 >> v1:172.21.15.28:6790/0 conn(0x7ff57810b050 legacy=0x7ff57810d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.254+0000 7ff57d496700 1 -- 172.21.15.28:0/727935506 shutdown_connections 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.254+0000 7ff57d496700 1 -- 172.21.15.28:0/727935506 >> 172.21.15.28:0/727935506 conn(0x7ff5780fca30 msgr2=0x7ff5780fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.254+0000 7ff57d496700 1 -- 172.21.15.28:0/727935506 shutdown_connections 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.254+0000 7ff57d496700 1 -- 172.21.15.28:0/727935506 wait complete. 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 librados: starting msgr at 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 librados: starting objecter 2024-03-30T15:03:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 Processor -- start 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 -- start start 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 librados: setting wanted keys 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 librados: calling monclient init 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5781a94b0 con 0x7ff57810eb60 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5781aa690 con 0x7ff57810b050 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff57d496700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff5781ab870 con 0x7ff578103240 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff576ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff578103240 0x7ff57819f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60362/0 (socket says 172.21.15.28:60362) 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.255+0000 7ff576ffd700 1 -- 172.21.15.28:0/2082758574 learned_addr learned my addr 172.21.15.28:0/2082758574 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff57810eb60 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff578103240 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 759931314 0 0) 0x7ff5781a94b0 con 0x7ff57810eb60 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff560003aa0 con 0x7ff57810eb60 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter ms_handle_connect 0x7ff57810b050 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684883281 0 0) 0x7ff5781ab870 con 0x7ff578103240 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5781a94b0 con 0x7ff578103240 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919231172 0 0) 0x7ff5781aa690 con 0x7ff57810b050 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5781ab870 con 0x7ff57810b050 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 673492320 0 0) 0x7ff560003aa0 con 0x7ff57810eb60 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5781aa690 con 0x7ff57810eb60 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff570001b70 con 0x7ff57810eb60 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 982954774 0 0) 0x7ff5781a94b0 con 0x7ff578103240 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff560003aa0 con 0x7ff578103240 2024-03-30T15:03:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff568001230 con 0x7ff578103240 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 165496403 0 0) 0x7ff5781ab870 con 0x7ff57810b050 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5781a94b0 con 0x7ff57810b050 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff56c003860 con 0x7ff57810b050 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3748896418 0 0) 0x7ff5781aa690 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 >> v1:172.21.15.28:6790/0 conn(0x7ff578103240 legacy=0x7ff57819f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 >> v1:172.21.15.134:6789/0 conn(0x7ff57810b050 legacy=0x7ff5781a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5781aca50 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5781abaa0 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff57d496700 10 client.24795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5781abf60 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.256+0000 7ff57d496700 1 librados: init done 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff570003ea0 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff570004de0 con 0x7ff57810eb60 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff57d496700 10 client.24795.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff57d496700 15 client.24795.objecter _session_command_op_assign -1 1 2024-03-30T15:03:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff57d496700 20 client.24795.objecter _get_session osd=-1 returning homeless 2024-03-30T15:03:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff57d496700 10 client.24795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.257+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff564002920 con 0x7ff57810eb60 2024-03-30T15:03:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.258+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff570003a00 con 0x7ff57810eb60 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.258+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(420..420 src has 53..420) ==== 8494+0+0 (unknown 685224004 0 0) 0x7ff5700888b0 con 0x7ff57810eb60 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.258+0000 7ff55ffff700 10 client.24795.objecter ms_dispatch 0x7ff578001550 osd_map(420..420 src has 53..420) 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.258+0000 7ff55ffff700 3 client.24795.objecter handle_osd_map got epochs [420,420] > 0 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.258+0000 7ff55ffff700 3 client.24795.objecter handle_osd_map decoding full epoch 420 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 10 client.24795.objecter checking command 1 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter _get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 15 client.24795.objecter _session_command_op_remove -1 1 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter _get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 15 client.24795.objecter _session_command_op_assign 7 1 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 4 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 10 client.24795.objecter _send_command 1 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.134:6812/3758190298 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff560082a20 con 0x7ff56007ed10 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 20 client.24795.objecter dump_active .. 0 homeless 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=420) ==== 24+0+0 (unknown 1743721605 0 0) 0x7ff570088cb0 con 0x7ff57810eb60 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 10 client.24795.objecter ms_handle_connect 0x7ff5600704d0 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.259+0000 7ff55ffff700 10 client.24795.objecter ms_handle_connect 0x7ff56007ed10 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.260+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff560082a20 con 0x7ff56007ed10 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.260+0000 7ff55ffff700 10 client.24795.objecter ms_dispatch 0x7ff578001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.260+0000 7ff55ffff700 10 client.24795.objecter _finish_command 1 = system:0 2024-03-30T15:03:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.260+0000 7ff55ffff700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 4 2024-03-30T15:03:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.260+0000 7ff55ffff700 15 client.24795.objecter _session_command_op_remove 7 1 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 10 client.24795.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 15 client.24795.objecter _session_command_op_assign -1 2 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 20 client.24795.objecter _get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 20 client.24795.objecter _get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 15 client.24795.objecter _session_command_op_remove -1 2 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 20 client.24795.objecter get_session s=0x7ff56007e660 osd=7 3 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 15 client.24795.objecter _session_command_op_assign 7 2 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 4 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 10 client.24795.objecter _send_command 2 2024-03-30T15:03:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.284+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 --> v1:172.21.15.134:6812/3758190298 -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7ff5640011f0 con 0x7ff56007ed10 2024-03-30T15:03:12.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.285+0000 7ff55ffff700 1 -- 172.21.15.28:0/2082758574 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7ff5640011f0 con 0x7ff56007ed10 2024-03-30T15:03:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.285+0000 7ff55ffff700 10 client.24795.objecter ms_dispatch 0x7ff578001550 command_reply(tid 2: 0 ) 2024-03-30T15:03:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.285+0000 7ff55ffff700 10 client.24795.objecter _finish_command 2 = system:0 2024-03-30T15:03:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.285+0000 7ff55ffff700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 4 2024-03-30T15:03:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.285+0000 7ff55ffff700 15 client.24795.objecter _session_command_op_remove 7 2 2024-03-30T15:03:12.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.286+0000 7ff57d496700 10 librados: watch_flush enter 2024-03-30T15:03:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 10 librados: watch_flush exit 2024-03-30T15:03:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 10 client.24795.objecter close_session for osd.7 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 >> v1:172.21.15.134:6812/3758190298 conn(0x7ff56007ed10 legacy=0x7ff5600811b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 20 client.24795.objecter put_session s=0x7ff56007e660 osd=7 1 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 20 client.24795.objecter shutdown clearing up homeless session... 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 10 client.24795.objecter successfully canceled tick 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff5600704d0 legacy=0x7ff560072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.287+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 >> v1:172.21.15.28:6789/0 conn(0x7ff57810eb60 legacy=0x7ff5781a7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.289+0000 7ff5777fe700 1 -- 172.21.15.28:0/2082758574 reap_dead start 2024-03-30T15:03:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.289+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 shutdown_connections 2024-03-30T15:03:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.289+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 >> 172.21.15.28:0/2082758574 conn(0x7ff5780fca30 msgr2=0x7ff578110fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.292+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 shutdown_connections 2024-03-30T15:03:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.292+0000 7ff57d496700 1 -- 172.21.15.28:0/2082758574 wait complete. 2024-03-30T15:03:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:12.293+0000 7ff57d496700 1 librados: shutdown 2024-03-30T15:03:12.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: pgmap v574: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:12.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: pgmap v574: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:12.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:12 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:12.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:12.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:12.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: osdmap e420: 8 total, 8 up, 8 in 2024-03-30T15:03:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:03:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:03:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: pgmap v574: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:03:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:12 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:12.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:12.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:12.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:12.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:12.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:12.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:13.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:13.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:13.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:03:13.209 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:03:13.209 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:03:13.209 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:03:13.209 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:03:13.209 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:03:13.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:13.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:13.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:13.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:13.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:13.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:13.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:13.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:03:13.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:03:13.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:13.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:13.874 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: osdmap e421: 8 total, 8 up, 8 in 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.875 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-57"}]: dispatch 2024-03-30T15:03:13.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:13.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:13.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:03:13.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:14.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:14.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 -- 172.21.15.28:0/2767839902 >> v1:172.21.15.28:6790/0 conn(0x7f39b810eb50 legacy=0x7f39b8111010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 -- 172.21.15.28:0/2767839902 shutdown_connections 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 -- 172.21.15.28:0/2767839902 >> 172.21.15.28:0/2767839902 conn(0x7f39b8077320 msgr2=0x7f39b80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 -- 172.21.15.28:0/2767839902 shutdown_connections 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 -- 172.21.15.28:0/2767839902 wait complete. 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 librados: starting msgr at 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.125+0000 7f39bf14c700 1 librados: starting objecter 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 Processor -- start 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 -- start start 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 librados: setting wanted keys 2024-03-30T15:03:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 librados: calling monclient init 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f39b81a7170 con 0x7f39b8107000 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f39b81a83b0 con 0x7f39b810ae80 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bf14c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f39b81a95f0 con 0x7f39b810eb50 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bcee8700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39b8107000 0x7f39b80ff580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51702/0 (socket says 172.21.15.28:51702) 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39bcee8700 1 -- 172.21.15.28:0/3009370329 learned_addr learned my addr 172.21.15.28:0/3009370329 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter ms_handle_connect 0x7f39b810eb50 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter ms_handle_connect 0x7f39b8107000 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter ms_handle_connect 0x7f39b810ae80 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365699882 0 0) 0x7f39b81a7170 con 0x7f39b8107000 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39a80031f0 con 0x7f39b8107000 2024-03-30T15:03:14.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1409992211 0 0) 0x7f39b81a95f0 con 0x7f39b810eb50 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39b81a7170 con 0x7f39b810eb50 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940590566 0 0) 0x7f39b81a83b0 con 0x7f39b810ae80 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.126+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39b81a95f0 con 0x7f39b810ae80 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 248986952 0 0) 0x7f39a80031f0 con 0x7f39b8107000 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39b81a83b0 con 0x7f39b8107000 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f39ac0018b0 con 0x7f39b8107000 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4205249809 0 0) 0x7f39b81a7170 con 0x7f39b810eb50 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39a80031f0 con 0x7f39b810eb50 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f39b0001d60 con 0x7f39b810eb50 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3941528374 0 0) 0x7f39b81a95f0 con 0x7f39b810ae80 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39b81a7170 con 0x7f39b810ae80 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f39a40012d0 con 0x7f39b810ae80 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2159547481 0 0) 0x7f39b81a83b0 con 0x7f39b8107000 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6790/0 conn(0x7f39b810eb50 legacy=0x7f39b81a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.134:6789/0 conn(0x7f39b810ae80 legacy=0x7f39b81a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b81aa830 con 0x7f39b8107000 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f39b81a73a0 con 0x7f39b8107000 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f39ac003b40 con 0x7f39b8107000 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f39ac005d30 con 0x7f39b8107000 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39bf14c700 10 client.24801.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f39b81a7860 con 0x7f39b8107000 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.127+0000 7f39bf14c700 1 librados: init done 2024-03-30T15:03:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.128+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f39b8103740 con 0x7f39b8107000 2024-03-30T15:03:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.129+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f39ac0036a0 con 0x7f39b8107000 2024-03-30T15:03:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(422..422 src has 53..422) ==== 8122+0+0 (unknown 983260520 0 0) 0x7f39ac0895e0 con 0x7f39b8107000 2024-03-30T15:03:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 osd_map(422..422 src has 53..422) 2024-03-30T15:03:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 3 client.24801.objecter handle_osd_map got epochs [422,422] > 0 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 3 client.24801.objecter handle_osd_map decoding full epoch 422 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 20 client.24801.objecter dump_active .. 0 homeless 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v422) ==== 54+0+16 (unknown 2990622246 0 2420000192) 0x7f39ac08bad0 con 0x7f39b8107000 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39a80702f0 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 10 client.24801.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 1 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 1 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 0 1 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 4 2024-03-30T15:03:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 10 client.24801.objecter _send_command 1 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.130+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39b81a8060 con 0x7f39b81a8580 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.131+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b81a8580 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.131+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81a8060 con 0x7f39b81a8580 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.131+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.132+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 1 = system:0 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.132+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 4 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.132+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 0 1 2024-03-30T15:03:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:14.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 10 client.24801.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 2 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 2 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b80fdbc0 osd=0 3 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 0 2 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 4 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 10 client.24801.objecter _send_command 2 2024-03-30T15:03:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.149+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81a92c0 con 0x7f39b81a8580 2024-03-30T15:03:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81a92c0 con 0x7f39b81a8580 2024-03-30T15:03:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 4 2024-03-30T15:03:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 0 2 2024-03-30T15:03:14.154 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 10 client.24801.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 1 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 4 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 10 client.24801.objecter _send_command 3 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.152+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f39b8103540 con 0x7f39b810e4f0 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b810e4f0 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b8103540 con 0x7f39b810e4f0 2024-03-30T15:03:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 3 = system:0 2024-03-30T15:03:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 4 2024-03-30T15:03:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.153+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 1 3 2024-03-30T15:03:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 10 client.24801.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 4 2024-03-30T15:03:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 4 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b810e380 osd=1 3 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 1 4 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 4 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 10 client.24801.objecter _send_command 4 2024-03-30T15:03:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.173+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b80635b0 con 0x7f39b810e4f0 2024-03-30T15:03:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b80635b0 con 0x7f39b810e4f0 2024-03-30T15:03:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 4 2024-03-30T15:03:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 1 4 2024-03-30T15:03:14.177 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 10 client.24801.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 5 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 5 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 2 5 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 4 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 10 client.24801.objecter _send_command 5 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.175+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f39b810cf50 con 0x7f39b8062a40 2024-03-30T15:03:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b8062a40 2024-03-30T15:03:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b810cf50 con 0x7f39b8062a40 2024-03-30T15:03:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 5 = system:0 2024-03-30T15:03:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 4 2024-03-30T15:03:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.176+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 2 5 2024-03-30T15:03:14.183 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 10 client.24801.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 6 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 6 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b80628d0 osd=2 3 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 2 6 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 4 2024-03-30T15:03:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 10 client.24801.objecter _send_command 6 2024-03-30T15:03:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.193+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b5790 con 0x7f39b8062a40 2024-03-30T15:03:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.194+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81b5790 con 0x7f39b8062a40 2024-03-30T15:03:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.194+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.194+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.194+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 4 2024-03-30T15:03:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.194+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 2 6 2024-03-30T15:03:14.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 10 client.24801.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 7 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 7 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 3 7 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 4 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 10 client.24801.objecter _send_command 7 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f39b81b1bc0 con 0x7f39b810c630 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.195+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b810c630 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.196+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81b1bc0 con 0x7f39b810c630 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.196+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.196+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 7 = system:0 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.196+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 4 2024-03-30T15:03:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.196+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 3 7 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 10 client.24801.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 8 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 8 2024-03-30T15:03:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b810c4c0 osd=3 3 2024-03-30T15:03:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 3 8 2024-03-30T15:03:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 4 2024-03-30T15:03:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 10 client.24801.objecter _send_command 8 2024-03-30T15:03:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.213+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b1e90 con 0x7f39b810c630 2024-03-30T15:03:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81b1e90 con 0x7f39b810c630 2024-03-30T15:03:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 4 2024-03-30T15:03:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 3 8 2024-03-30T15:03:14.216 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 10 client.24801.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 9 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 9 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 4 9 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 4 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 10 client.24801.objecter _send_command 9 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f39b81b12a0 con 0x7f39b81b6620 2024-03-30T15:03:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.215+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b81b6620 2024-03-30T15:03:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.216+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81b12a0 con 0x7f39b81b6620 2024-03-30T15:03:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.216+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.216+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 9 = system:0 2024-03-30T15:03:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.216+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 4 2024-03-30T15:03:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.216+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 4 9 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.232+0000 7f39bf14c700 10 client.24801.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 10 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 10 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b1e90 osd=4 3 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 4 10 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 4 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 10 client.24801.objecter _send_command 10 2024-03-30T15:03:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.233+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b12a0 con 0x7f39b81b6620 2024-03-30T15:03:14.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.234+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81b12a0 con 0x7f39b81b6620 2024-03-30T15:03:14.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.234+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.234+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.234+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 4 2024-03-30T15:03:14.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.234+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 4 10 2024-03-30T15:03:14.236 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 10 client.24801.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 11 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 11 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 5 11 2024-03-30T15:03:14.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 4 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 10 client.24801.objecter _send_command 11 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f39b81c2b30 con 0x7f39b81befb0 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.235+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b81befb0 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.236+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81c2b30 con 0x7f39b81befb0 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.236+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.236+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 11 = system:0 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.236+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 4 2024-03-30T15:03:14.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.236+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 5 11 2024-03-30T15:03:14.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 10 client.24801.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 12 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 12 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b12a0 osd=5 3 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 5 12 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 4 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 10 client.24801.objecter _send_command 12 2024-03-30T15:03:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.253+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b0fa0 con 0x7f39b81befb0 2024-03-30T15:03:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39a400c9e0 con 0x7f39b81befb0 2024-03-30T15:03:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 4 2024-03-30T15:03:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 5 12 2024-03-30T15:03:14.256 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 10 client.24801.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 13 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 13 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 6 13 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 4 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 10 client.24801.objecter _send_command 13 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.254+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f39b81c42c0 con 0x7f39b81baed0 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.255+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b81baed0 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.256+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81c42c0 con 0x7f39b81baed0 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.256+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.256+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 13 = system:0 2024-03-30T15:03:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.256+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 4 2024-03-30T15:03:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.256+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 6 13 2024-03-30T15:03:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 10 client.24801.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 14 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 14 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b5bf0 osd=6 3 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 6 14 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 4 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 10 client.24801.objecter _send_command 14 2024-03-30T15:03:14.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.273+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b5f00 con 0x7f39b81baed0 2024-03-30T15:03:14.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:14.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81b5f00 con 0x7f39b81baed0 2024-03-30T15:03:14.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 4 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 6 14 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 10 client.24801.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 15 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 15 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 7 15 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 4 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 10 client.24801.objecter _send_command 15 2024-03-30T15:03:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.275+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f39b81cb6e0 con 0x7f39b81c7bd0 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 10 client.24801.objecter ms_handle_connect 0x7f39b81c7bd0 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f39b81cb6e0 con 0x7f39b81c7bd0 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 15 = system:0 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 4 2024-03-30T15:03:14.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.276+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 7 15 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.301+0000 7f39bf14c700 10 client.24801.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign -1 16 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 20 client.24801.objecter _get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_remove -1 16 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 20 client.24801.objecter get_session s=0x7f39b81b5f00 osd=7 3 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 15 client.24801.objecter _session_command_op_assign 7 16 2024-03-30T15:03:14.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 4 2024-03-30T15:03:14.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 10 client.24801.objecter _send_command 16 2024-03-30T15:03:14.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.302+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f39b81b59a0 con 0x7f39b81c7bd0 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.303+0000 7f39b5ffb700 1 -- 172.21.15.28:0/3009370329 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f39b81b59a0 con 0x7f39b81c7bd0 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.303+0000 7f39b5ffb700 10 client.24801.objecter ms_dispatch 0x7f39b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.303+0000 7f39b5ffb700 10 client.24801.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.303+0000 7f39b5ffb700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 4 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.303+0000 7f39b5ffb700 15 client.24801.objecter _session_command_op_remove 7 16 2024-03-30T15:03:14.305 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 librados: watch_flush enter 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 librados: watch_flush exit 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.0 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6801/3394323575 conn(0x7f39b81a8580 legacy=0x7f39b81a8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80fdbc0 osd=0 1 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.1 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6805/1850313547 conn(0x7f39b810e4f0 legacy=0x7f39b802c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810e380 osd=1 1 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.2 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6809/3451325043 conn(0x7f39b8062a40 legacy=0x7f39b81b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b80628d0 osd=2 1 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.3 2024-03-30T15:03:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6813/1506157250 conn(0x7f39b810c630 legacy=0x7f39b81b8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b810c4c0 osd=3 1 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.4 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.134:6800/3063775514 conn(0x7f39b81b6620 legacy=0x7f39b81bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b1e90 osd=4 1 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.5 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bd6e9700 1 -- 172.21.15.28:0/3009370329 reap_dead start 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.134:6804/3668151179 conn(0x7f39b81befb0 legacy=0x7f39b81c13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bd6e9700 1 -- 172.21.15.28:0/3009370329 reap_dead start 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b12a0 osd=5 1 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.6 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.134:6808/2135224194 conn(0x7f39b81baed0 legacy=0x7f39b81c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5bf0 osd=6 1 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter close_session for osd.7 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.134:6812/3758190298 conn(0x7f39b81c7bd0 legacy=0x7f39b81c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter put_session s=0x7f39b81b5f00 osd=7 1 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bd6e9700 1 -- 172.21.15.28:0/3009370329 reap_dead start 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 20 client.24801.objecter shutdown clearing up homeless session... 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 10 client.24801.objecter successfully canceled tick 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6800/3011431093 conn(0x7f39a80702f0 legacy=0x7f39a80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> v1:172.21.15.28:6789/0 conn(0x7f39b8107000 legacy=0x7f39b80ff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 shutdown_connections 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 >> 172.21.15.28:0/3009370329 conn(0x7f39b8077320 msgr2=0x7f39b810da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 shutdown_connections 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 -- 172.21.15.28:0/3009370329 wait complete. 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:14.304+0000 7f39bf14c700 1 librados: shutdown 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:14.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:14.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:14.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:14.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:14.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:14.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3009370329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T15:03:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3009370329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T15:03:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:14.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:14.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:14.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:14.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: osdmap e422: 8 total, 8 up, 8 in 2024-03-30T15:03:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:03:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:03:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3009370329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: osdmap e423: 8 total, 8 up, 8 in 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:15.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:15.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:15.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:15.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:15.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:15.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:15.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:15.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.534 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:15.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:15.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:15.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:15.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:15.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:15.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: 3.6 scrub starts 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: 3.6 scrub ok 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: 3.6 scrub starts 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: 3.6 scrub ok 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.847 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.847 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: 3.6 scrub starts 2024-03-30T15:03:15.861 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: 3.6 scrub ok 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: osdmap e424: 8 total, 8 up, 8 in 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:15.863 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.863 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-59"}]: dispatch 2024-03-30T15:03:15.871 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:15.871 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:15.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:15.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:15.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:16.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:16.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:16.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:16.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:16.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:16.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:16.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:16.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:16.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:16.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:16.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:03:16.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:16.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:16.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: 42.1c scrub starts 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: 42.1c scrub ok 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: pgmap v580: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[29876]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: 42.1c scrub starts 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: 42.1c scrub ok 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: pgmap v580: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:16.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:16 smithi028 ceph-mon[41614]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T15:03:16.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.830+0000 7fc714468700 1 -- 172.21.15.28:0/1623589585 >> v1:172.21.15.28:6790/0 conn(0x7fc70c10ea90 legacy=0x7fc70c110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.830+0000 7fc714468700 1 -- 172.21.15.28:0/1623589585 shutdown_connections 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.830+0000 7fc714468700 1 -- 172.21.15.28:0/1623589585 >> 172.21.15.28:0/1623589585 conn(0x7fc70c0fdb30 msgr2=0x7fc70c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.830+0000 7fc714468700 1 -- 172.21.15.28:0/1623589585 shutdown_connections 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.830+0000 7fc714468700 1 -- 172.21.15.28:0/1623589585 wait complete. 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 librados: starting msgr at 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 librados: starting objecter 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 Processor -- start 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 -- start start 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 librados: setting wanted keys 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 librados: calling monclient init 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc70c1a7100 con 0x7fc70c10aef0 2024-03-30T15:03:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc70c1a8320 con 0x7fc70c10ea90 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc714468700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc70c1a9560 con 0x7fc70c102260 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc712204700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc70c102260 0x7fc70c107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:60390/0 (socket says 172.21.15.28:60390) 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc712204700 1 -- 172.21.15.28:0/3358552597 learned_addr learned my addr 172.21.15.28:0/3358552597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter ms_handle_connect 0x7fc70c10aef0 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter ms_handle_connect 0x7fc70c102260 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter ms_handle_connect 0x7fc70c10ea90 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531431093 0 0) 0x7fc70c1a9560 con 0x7fc70c102260 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.831+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6f40035b0 con 0x7fc70c102260 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681291367 0 0) 0x7fc70c1a7100 con 0x7fc70c10aef0 2024-03-30T15:03:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc70c1a9560 con 0x7fc70c10aef0 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 288932660 0 0) 0x7fc70c1a8320 con 0x7fc70c10ea90 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc70c1a7100 con 0x7fc70c10ea90 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2302021496 0 0) 0x7fc6f40035b0 con 0x7fc70c102260 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc70c1a8320 con 0x7fc70c102260 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc704001650 con 0x7fc70c102260 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3259068575 0 0) 0x7fc70c1a9560 con 0x7fc70c10aef0 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6f40035b0 con 0x7fc70c10aef0 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc700001870 con 0x7fc70c10aef0 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3236906459 0 0) 0x7fc70c1a7100 con 0x7fc70c10ea90 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc70c1a9560 con 0x7fc70c10ea90 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc6fc004540 con 0x7fc70c10ea90 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1845057973 0 0) 0x7fc70c1a8320 con 0x7fc70c102260 2024-03-30T15:03:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 >> v1:172.21.15.134:6789/0 conn(0x7fc70c10ea90 legacy=0x7fc70c1a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 >> v1:172.21.15.28:6789/0 conn(0x7fc70c10aef0 legacy=0x7fc70c1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c1aa7a0 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c1a9790 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc714468700 10 client.57995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc70c1a9c50 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc714468700 1 librados: init done 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc7040018d0 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.832+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc704006050 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.833+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f8005120 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc70401c950 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(425..425 src has 53..425) ==== 8122+0+0 (unknown 1133766625 0 0) 0x7fc704089a90 con 0x7fc70c102260 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 10 client.57995.objecter ms_dispatch 0x7fc70c001550 osd_map(425..425 src has 53..425) 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 3 client.57995.objecter handle_osd_map got epochs [425,425] > 0 2024-03-30T15:03:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 3 client.57995.objecter handle_osd_map decoding full epoch 425 2024-03-30T15:03:16.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 20 client.57995.objecter dump_active .. 0 homeless 2024-03-30T15:03:16.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.834+0000 7fc70b7fe700 10 client.57995.objecter ms_handle_connect 0x7fc6f4074810 2024-03-30T15:03:16.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:16.839+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc704056e10 con 0x7fc70c102260 2024-03-30T15:03:16.854 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:16.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:16.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: 42.1c scrub starts 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: 42.1c scrub ok 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: pgmap v580: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:03:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:16 smithi134 ceph-mon[33714]: osdmap e425: 8 total, 8 up, 8 in 2024-03-30T15:03:17.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:17.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.054+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fc6f8005410 con 0x7fc70c102260 2024-03-30T15:03:17.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:17.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:03:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.257+0000 7fc70b7fe700 1 -- 172.21.15.28:0/3358552597 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v426) ==== 101+0+0 (unknown 3030619653 0 0) 0x7fc70405a860 con 0x7fc70c102260 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.261+0000 7fc714468700 10 librados: watch_flush enter 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.261+0000 7fc714468700 10 librados: watch_flush exit 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.261+0000 7fc714468700 20 client.57995.objecter shutdown clearing up homeless session... 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.261+0000 7fc714468700 10 client.57995.objecter successfully canceled tick 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.262+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc6f4074810 legacy=0x7fc6f4076cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.262+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 >> v1:172.21.15.28:6790/0 conn(0x7fc70c102260 legacy=0x7fc70c107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:17.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.262+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 shutdown_connections 2024-03-30T15:03:17.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.262+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 >> 172.21.15.28:0/3358552597 conn(0x7fc70c0fdb30 msgr2=0x7fc70c107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:17.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.264+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 shutdown_connections 2024-03-30T15:03:17.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.264+0000 7fc714468700 1 -- 172.21.15.28:0/3358552597 wait complete. 2024-03-30T15:03:17.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:17.265+0000 7fc714468700 1 librados: shutdown 2024-03-30T15:03:17.267 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:03:17.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:17.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.427 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:17.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:17.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:17.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:17.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:17.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:17.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:17.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:03:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 5.1e scrub starts 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: 5.1e scrub ok 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3358552597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: from='client.57995 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: from='client.57995 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T15:03:17.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:17.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:17.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:17.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 5.1e scrub starts 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: 5.1e scrub ok 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3358552597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: from='client.57995 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: from='client.57995 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:03:17.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 5.1e scrub starts 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: 5.1e scrub ok 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3358552597' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: from='client.57995 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: from='client.57995 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: osdmap e426: 8 total, 8 up, 8 in 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:17.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:18.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:03:18.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:03:18.026 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T15:03:18.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:03:18.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:18.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:18.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:18.266 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:18.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:18.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:18.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:18.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:18.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:18.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:18.787 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:18.787 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:18.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:18.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:18.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:18.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:18.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:19.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:19.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:19.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:19.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:19.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:19.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:19.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: pgmap v583: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T15:03:19.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T15:03:19.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:19 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:19.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:19.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: pgmap v583: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T15:03:19.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: pgmap v583: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: osdmap e427: 8 total, 8 up, 8 in 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T15:03:19.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:19 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:19.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:03:19.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.571 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:19.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:19.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:19.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:19.718 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:19.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:19.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:19.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:19.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:20.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:03:20.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:20.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:20.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:20.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:20.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:20.355 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 101.1e scrub starts 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 101.1e scrub ok 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:03:20.356 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-mon[33714]: pgmap v586: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 101.1e scrub starts 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 101.1e scrub ok 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[29876]: pgmap v586: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 101.1e scrub starts 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 101.1e scrub ok 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: osdmap e428: 8 total, 8 up, 8 in 2024-03-30T15:03:20.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-61"}]: dispatch 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:03:20.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:20 smithi028 ceph-mon[41614]: pgmap v586: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:20.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:20.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:20.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:20.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:20.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:20.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:20.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:20.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:20.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:20.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:20.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.354+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/1283878883 >> v1:172.21.15.134:6789/0 conn(0x7fc6e010eb50 legacy=0x7fc6e0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.354+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/1283878883 shutdown_connections 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.354+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/1283878883 >> 172.21.15.28:0/1283878883 conn(0x7fc6e00fc9e0 msgr2=0x7fc6e00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.354+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/1283878883 shutdown_connections 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.354+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/1283878883 wait complete. 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 librados: starting msgr at 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 librados: starting objecter 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 Processor -- start 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 -- start start 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 librados: setting wanted keys 2024-03-30T15:03:20.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.355+0000 7fc6e6ecf700 1 librados: calling monclient init 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6e6ecf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6e010ade0 con 0x7fc6e010b040 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6e6ecf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6e01a8630 con 0x7fc6e010eb50 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6e6ecf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc6e01a9850 con 0x7fc6e0103230 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6e4c6b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc6e0103230 0x7fc6e0107c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54532/0 (socket says 172.21.15.28:54532) 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6e4c6b700 1 -- 172.21.15.28:0/3681934887 learned_addr learned my addr 172.21.15.28:0/3681934887 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter ms_handle_connect 0x7fc6e0103230 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter ms_handle_connect 0x7fc6e010eb50 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter ms_handle_connect 0x7fc6e010b040 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459230453 0 0) 0x7fc6e01a8630 con 0x7fc6e010eb50 2024-03-30T15:03:20.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d00031f0 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645089247 0 0) 0x7fc6e01a9850 con 0x7fc6e0103230 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6e01a8630 con 0x7fc6e0103230 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3386781407 0 0) 0x7fc6d00031f0 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6e01a9850 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc6d8001d50 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1665951127 0 0) 0x7fc6e01a8630 con 0x7fc6e0103230 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6d00031f0 con 0x7fc6e0103230 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.356+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc6d40018f0 con 0x7fc6e0103230 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3651534972 0 0) 0x7fc6e01a9850 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6790/0 conn(0x7fc6e0103230 legacy=0x7fc6e0107c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6789/0 conn(0x7fc6e010b040 legacy=0x7fc6e0108340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e01aaa70 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e01a9a80 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc6d8004a00 con 0x7fc6e010eb50 2024-03-30T15:03:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc6d8005390 con 0x7fc6e010eb50 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6e6ecf700 10 client.50017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6e01a9f40 con 0x7fc6e010eb50 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.357+0000 7fc6e6ecf700 1 librados: init done 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.358+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc6c4005100 con 0x7fc6e010eb50 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.358+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc6d801bbd0 con 0x7fc6e010eb50 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(429..429 src has 53..429) ==== 8138+0+0 (unknown 3728928599 0 0) 0x7fc6d8088960 con 0x7fc6e010eb50 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 osd_map(429..429 src has 53..429) 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 3 client.50017.objecter handle_osd_map got epochs [429,429] > 0 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 3 client.50017.objecter handle_osd_map decoding full epoch 429 2024-03-30T15:03:20.960 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 20 client.50017.objecter dump_active .. 0 homeless 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v429) ==== 54+0+16 (unknown 4196018325 0 2420000192) 0x7fc6d8088d50 con 0x7fc6e010eb50 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6d00702f0 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 1 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 1 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 0 1 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 4 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 1 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.359+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6c4004b50 con 0x7fc6c4004730 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c4004730 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c4004b50 con 0x7fc6c4004730 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 1 = system:0 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 4 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.360+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 0 1 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 2 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 2 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c40053e0 osd=0 3 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 0 2 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 4 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 2 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.378+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c4007b00 con 0x7fc6c4004730 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c4007b00 con 0x7fc6c4004730 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 4 2024-03-30T15:03:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 0 2 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 1 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 4 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 3 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.380+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc6c4002160 con 0x7fc6c40042a0 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c40042a0 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c4002160 con 0x7fc6c40042a0 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:20.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 3 = system:0 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.381+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 1 3 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4004130 osd=1 3 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 1 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 4 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.398+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c4001600 con 0x7fc6c40042a0 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c4001600 con 0x7fc6c40042a0 2024-03-30T15:03:20.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 4 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 1 4 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 5 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 5 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 2 5 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 4 2024-03-30T15:03:20.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 5 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc6c4015600 con 0x7fc6c4011af0 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.400+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c4011af0 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.401+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c4015600 con 0x7fc6c4011af0 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.401+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.401+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 5 = system:0 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.401+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 4 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.401+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 2 5 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:20.967 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 6 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 6 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4008480 osd=2 3 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 2 6 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 4 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 6 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.418+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c4007f00 con 0x7fc6c4011af0 2024-03-30T15:03:20.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.419+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c4007f00 con 0x7fc6c4011af0 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 4 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 2 6 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 7 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 7 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 3 7 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 4 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 7 2024-03-30T15:03:20.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc6c401a1d0 con 0x7fc6c400d240 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.420+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c400d240 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.421+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6d803ca50 con 0x7fc6c400d240 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.421+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.421+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 7 = system:0 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.421+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 4 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.421+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 3 7 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 8 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 8 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4008300 osd=3 3 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 3 8 2024-03-30T15:03:20.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 4 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 8 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.438+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c400c8d0 con 0x7fc6c400d240 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c400c8d0 con 0x7fc6c400d240 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 4 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 3 8 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 9 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 9 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 4 9 2024-03-30T15:03:20.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 4 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 9 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.440+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc6c401ed80 con 0x7fc6c4016490 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c4016490 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c401ed80 con 0x7fc6c4016490 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 9 = system:0 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 4 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.441+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 4 9 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 10 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 10 2024-03-30T15:03:20.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c400c8d0 osd=4 3 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 4 10 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 4 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 10 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.458+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c401ed80 con 0x7fc6c4016490 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c401ed80 con 0x7fc6c4016490 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 4 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 4 10 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 11 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 11 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 5 11 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 4 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 11 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.460+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc6c40201b0 con 0x7fc6c401f4e0 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:20.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c401f4e0 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c40201b0 con 0x7fc6c401f4e0 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 11 = system:0 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 4 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.461+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 5 11 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 12 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 12 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c401edc0 osd=5 3 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 5 12 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 4 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 12 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.479+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c4020430 con 0x7fc6c401f4e0 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.480+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c4020430 con 0x7fc6c401f4e0 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.480+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 4 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 5 12 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 13 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 13 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 6 13 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 4 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 13 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc6c40277a0 con 0x7fc6c4023c90 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.481+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c4023c90 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.482+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c40277a0 con 0x7fc6c4023c90 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.482+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.482+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 13 = system:0 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.482+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 4 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.482+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 6 13 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 14 2024-03-30T15:03:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 14 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c4020430 osd=6 3 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 6 14 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 4 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 14 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.499+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c400c6e0 con 0x7fc6c4023c90 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c400c6e0 con 0x7fc6c4023c90 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 4 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 6 14 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 15 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 15 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 7 15 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 4 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 15 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.500+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc6c4028d60 con 0x7fc6c4028920 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.501+0000 7fc6ddffb700 10 client.50017.objecter ms_handle_connect 0x7fc6c4028920 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.502+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6c4028d60 con 0x7fc6c4028920 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.502+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.502+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 15 = system:0 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.502+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 4 2024-03-30T15:03:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.502+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 7 15 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 10 client.50017.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign -1 16 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 20 client.50017.objecter _get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_remove -1 16 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 20 client.50017.objecter get_session s=0x7fc6c400c6e0 osd=7 3 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 15 client.50017.objecter _session_command_op_assign 7 16 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 4 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 10 client.50017.objecter _send_command 16 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.518+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc6c4028d60 con 0x7fc6c4028920 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6ddffb700 1 -- 172.21.15.28:0/3681934887 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc6c4028d60 con 0x7fc6c4028920 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6ddffb700 10 client.50017.objecter ms_dispatch 0x7fc6e0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6ddffb700 10 client.50017.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6ddffb700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 4 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6ddffb700 15 client.50017.objecter _session_command_op_remove 7 16 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 librados: watch_flush enter 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 librados: watch_flush exit 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.0 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6801/3394323575 conn(0x7fc6c4004730 legacy=0x7fc6c4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c40053e0 osd=0 1 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.1 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6805/1850313547 conn(0x7fc6c40042a0 legacy=0x7fc6c400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4004130 osd=1 1 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.2 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6809/3451325043 conn(0x7fc6c4011af0 legacy=0x7fc6c4013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008480 osd=2 1 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.3 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6813/1506157250 conn(0x7fc6c400d240 legacy=0x7fc6c4018940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4008300 osd=3 1 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.4 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.134:6800/3063775514 conn(0x7fc6c4016490 legacy=0x7fc6c401a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e546c700 1 -- 172.21.15.28:0/3681934887 reap_dead start 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c8d0 osd=4 1 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.5 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.134:6804/3668151179 conn(0x7fc6c401f4e0 legacy=0x7fc6c401f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e546c700 1 -- 172.21.15.28:0/3681934887 reap_dead start 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c401edc0 osd=5 1 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.6 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e546c700 1 -- 172.21.15.28:0/3681934887 reap_dead start 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.134:6808/2135224194 conn(0x7fc6c4023c90 legacy=0x7fc6c4026080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c4020430 osd=6 1 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter close_session for osd.7 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.134:6812/3758190298 conn(0x7fc6c4028920 legacy=0x7fc6c401c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter put_session s=0x7fc6c400c6e0 osd=7 1 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 20 client.50017.objecter shutdown clearing up homeless session... 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.520+0000 7fc6e6ecf700 10 client.50017.objecter successfully canceled tick 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc6d00702f0 legacy=0x7fc6d00727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> v1:172.21.15.134:6789/0 conn(0x7fc6e010eb50 legacy=0x7fc6e01a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 shutdown_connections 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 >> 172.21.15.28:0/3681934887 conn(0x7fc6e00fc9e0 msgr2=0x7fc6e00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 shutdown_connections 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 -- 172.21.15.28:0/3681934887 wait complete. 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:20.521+0000 7fc6e6ecf700 1 librados: shutdown 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:21.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:21.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:21.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:21.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:21.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:21.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:21.342 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:03:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:03:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:03:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:03:21.343 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:21.352 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3681934887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:21.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:21.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:21.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:21.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:21.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:21.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:03:21.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:03:21.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T15:03:21.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T15:03:21.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T15:03:21.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:21.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3681934887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: osdmap e429: 8 total, 8 up, 8 in 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:21.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3681934887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:21.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:21.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:21.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:21.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:21.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:22.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:03:22.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:22.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:22.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:22.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:22.273 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:03:22.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T15:03:22.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: pgmap v589: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T15:03:22.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T15:03:22.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: pgmap v589: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:22.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:22.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:22.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:22.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:22.548 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: osdmap e430: 8 total, 8 up, 8 in 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: pgmap v589: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 668 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:22.549 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:22.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:22.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:22.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:22.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:22.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:22.847 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:22.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:22.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:22.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:23.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:23.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 101.1a scrub starts 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 101.1a scrub ok 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 3.1d deep-scrub starts 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 3.1d deep-scrub ok 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:03:23.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:03:23.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:23.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 -- 172.21.15.28:0/4144823466 >> v1:172.21.15.28:6790/0 conn(0x7fbb081031a0 legacy=0x7fbb08103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 -- 172.21.15.28:0/4144823466 shutdown_connections 2024-03-30T15:03:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 -- 172.21.15.28:0/4144823466 >> 172.21.15.28:0/4144823466 conn(0x7fbb080fc9f0 msgr2=0x7fbb080fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 -- 172.21.15.28:0/4144823466 shutdown_connections 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 -- 172.21.15.28:0/4144823466 wait complete. 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 librados: starting msgr at 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.340+0000 7fbb0dd27700 1 librados: starting objecter 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 Processor -- start 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 -- start start 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 librados: setting wanted keys 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 librados: calling monclient init 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb081a92f0 con 0x7fbb081031a0 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb081aa510 con 0x7fbb0810eac0 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb0dd27700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbb081ab750 con 0x7fbb0810afb0 2024-03-30T15:03:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb06ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbb0810afb0 0x7fbb081a42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54540/0 (socket says 172.21.15.28:54540) 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb06ffd700 1 -- 172.21.15.28:0/2960905778 learned_addr learned my addr 172.21.15.28:0/2960905778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter ms_handle_connect 0x7fbb081031a0 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter ms_handle_connect 0x7fbb0810afb0 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter ms_handle_connect 0x7fbb0810eac0 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682383155 0 0) 0x7fbb081a92f0 con 0x7fbb081031a0 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaf00031f0 con 0x7fbb081031a0 2024-03-30T15:03:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802715830 0 0) 0x7fbb081ab750 con 0x7fbb0810afb0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb081a92f0 con 0x7fbb0810afb0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.341+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272127290 0 0) 0x7fbb081aa510 con 0x7fbb0810eac0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb081ab750 con 0x7fbb0810eac0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 429517263 0 0) 0x7fbaf00031f0 con 0x7fbb081031a0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb081aa510 con 0x7fbb081031a0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2086653513 0 0) 0x7fbb081a92f0 con 0x7fbb0810afb0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbaf00031f0 con 0x7fbb0810afb0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbaf8004540 con 0x7fbb081031a0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbafc001870 con 0x7fbb0810afb0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1353972882 0 0) 0x7fbb081ab750 con 0x7fbb0810eac0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb081a92f0 con 0x7fbb0810eac0 2024-03-30T15:03:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbb00001470 con 0x7fbb0810eac0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3946538399 0 0) 0x7fbb081aa510 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 >> v1:172.21.15.28:6790/0 conn(0x7fbb0810afb0 legacy=0x7fbb081a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 >> v1:172.21.15.134:6789/0 conn(0x7fbb0810eac0 legacy=0x7fbb081a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb081ac990 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb081ab980 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb0dd27700 10 client.24813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb081abe40 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb0dd27700 1 librados: init done 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbaf8003760 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.342+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbaf8004d50 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.343+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf4005120 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbaf8003910 con 0x7fbb081031a0 2024-03-30T15:03:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(432..432 src has 53..432) ==== 8523+0+0 (unknown 3637106646 0 0) 0x7fbaf8088810 con 0x7fbb081031a0 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 10 client.24813.objecter ms_dispatch 0x7fbb08001550 osd_map(432..432 src has 53..432) 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 3 client.24813.objecter handle_osd_map got epochs [432,432] > 0 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 3 client.24813.objecter handle_osd_map decoding full epoch 432 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 20 client.24813.objecter dump_active .. 0 homeless 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.344+0000 7fbb04ff9700 10 client.24813.objecter ms_handle_connect 0x7fbaf0070450 2024-03-30T15:03:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.349+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbaf80558b0 con 0x7fbb081031a0 2024-03-30T15:03:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:03:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:03:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 101.1a scrub starts 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 101.1a scrub ok 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 3.1d deep-scrub starts 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 3.1d deep-scrub ok 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:03:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:23 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:03:23.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:23.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 101.1a scrub starts 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 101.1a scrub ok 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 3.1d deep-scrub starts 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 3.1d deep-scrub ok 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: osdmap e431: 8 total, 8 up, 8 in 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:03:23.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:23 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:03:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.564+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbaf4000b90 con 0x7fbaf0070450 2024-03-30T15:03:23.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.581+0000 7fbb04ff9700 1 -- 172.21.15.28:0/2960905778 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+660075 (unknown 2965378022 0 2471523963) 0x7fbaf4000b90 con 0x7fbaf0070450 2024-03-30T15:03:23.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 10 librados: watch_flush enter 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 10 librados: watch_flush exit 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 20 client.24813.objecter shutdown clearing up homeless session... 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 10 client.24813.objecter successfully canceled tick 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbaf0070450 legacy=0x7fbaf0072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 >> v1:172.21.15.28:6789/0 conn(0x7fbb081031a0 legacy=0x7fbb0810a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 shutdown_connections 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 >> 172.21.15.28:0/2960905778 conn(0x7fbb080fc9f0 msgr2=0x7fbb08110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 shutdown_connections 2024-03-30T15:03:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 -- 172.21.15.28:0/2960905778 wait complete. 2024-03-30T15:03:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:23.585+0000 7fbb0dd27700 1 librados: shutdown 2024-03-30T15:03:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:03:23.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:23.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:23.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:23.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:23.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:23.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24865.0:4 17.10 17:08000000::::head [pgnls start_epoch 114] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:24.079 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 328.81693077199998, 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.016273526, 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24865", 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3133651075", 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 4 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259411+0000", 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1000000000000001e-06 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259418+0000", 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6809999999999998e-06 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259427+0000", 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8659999999999995e-06 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259432+0000", 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2320000000000001e-06 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259486+0000", 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3547000000000001e-05 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259495+0000", 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0040000000000005e-06 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.273975+0000", 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.014480228 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274008+0000", 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2824000000000003e-05 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.275683+0000", 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016750440000000001 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24853.0:11 14.15 14:ada94957:test-rados-api-smithi028-93116-5::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 326.705044402, 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.040512728999999997, 2024-03-30T15:03:24.086 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24853", 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 11 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371298+0000", 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.973e-06 2024-03-30T15:03:24.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371305+0000", 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.956e-06 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371315+0000", 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0227999999999999e-05 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371325+0000", 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0009e-05 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371393+0000", 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7558000000000006e-05 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371494+0000", 2024-03-30T15:03:24.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010136000000000001 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,4", 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371687+0000", 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019319699999999999 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.375970+0000", 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042825850000000002 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.401997+0000", 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.026027127000000001 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411745+0000", 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0097477950000000001 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411781+0000", 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6205000000000003e-05 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411809+0000", 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7736e-05 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2070 11.1e 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 325.72076575099999, 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.090460287, 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2070 2024-03-30T15:03:24.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355577+0000", 2024-03-30T15:03:24.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5239999999999999e-06 2024-03-30T15:03:24.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381402+0000", 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025824751 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381428+0000", 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6114000000000001e-05 2024-03-30T15:03:24.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381442+0000", 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3645000000000001e-05 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381556+0000", 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114195 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381660+0000", 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104416 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381768+0000", 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108322 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.423084+0000", 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041315927000000002 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445887+0000", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022802991000000002 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445906+0000", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8759000000000001e-05 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445976+0000", 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9926999999999996e-05 2024-03-30T15:03:24.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.446035+0000", 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8715999999999998e-05 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.102 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2073 11.1e 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 325.62529764999999, 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16928738600000001, 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2073 2024-03-30T15:03:24.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451044+0000", 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111e-06 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454143+0000", 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030991600000000001 2024-03-30T15:03:24.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454169+0000", 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6114000000000001e-05 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454182+0000", 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3062e-05 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454263+0000", 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0641000000000003e-05 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454373+0000", 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109875 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.458049+0000", 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036762639999999998 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.474113+0000", 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016064330000000002 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.533037+0000", 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058924057000000002 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620236+0000", 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.087198494000000001 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620291+0000", 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4829e-05 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620330+0000", 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9449000000000001e-05 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.110 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25342.0:2 57.1e 57:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 325.60907629399998, 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2667642749999999, 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25342", 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2592697076", 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467266+0000", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9099999999999999e-06 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467277+0000", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0706999999999999e-05 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467287+0000", 2024-03-30T15:03:24.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0508e-05 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467298+0000", 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0519999999999999e-05 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467376+0000", 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8356000000000007e-05 2024-03-30T15:03:24.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467386+0000", 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9620000000000005e-06 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.727985+0000", 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.260599096 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.728092+0000", 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107036 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.728213+0000", 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012053900000000001 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729270+0000", 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001057721 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729293+0000", 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2796999999999999e-05 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733920+0000", 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046273019999999998 2024-03-30T15:03:24.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733982+0000", 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1978999999999997e-05 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.734028+0000", 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5841999999999999e-05 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.118 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35294.0:2 54.6 54:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 325.603002624, 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.251836993, 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.119 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35294", 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/806098981", 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473339+0000", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8220000000000001e-06 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473345+0000", 2024-03-30T15:03:24.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3179999999999997e-06 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473356+0000", 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0954e-05 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473365+0000", 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7769999999999997e-06 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473429+0000", 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3115999999999997e-05 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473438+0000", 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7580000000000004e-06 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719452+0000", 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.246013944 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719691+0000", 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000238197 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:03:24.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719925+0000", 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023494400000000001 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.721930+0000", 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002004329 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.722343+0000", 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00041327999999999997 2024-03-30T15:03:24.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725066+0000", 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027224760000000001 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725131+0000", 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5407999999999994e-05 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725175+0000", 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.367e-05 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35294.0:6 54.6 54:602f83fe:::foo:head [watch unwatch cookie 93835005080800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 323.34620428699998, 2024-03-30T15:03:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.084326659999999998, 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35294", 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/806098981", 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 6 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730139+0000", 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.179e-06 2024-03-30T15:03:24.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730148+0000", 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9379999999999998e-06 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730168+0000", 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9602e-05 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730181+0000", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3796e-05 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730418+0000", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023605800000000001 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730534+0000", 2024-03-30T15:03:24.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116564 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730722+0000", 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000187894 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.731099+0000", 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000376993 2024-03-30T15:03:24.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.731523+0000", 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042433299999999999 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814309+0000", 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.082786025999999999 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814379+0000", 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0009e-05 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814463+0000", 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3268000000000003e-05 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25489.0:2 64.1a 64:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 323.08680096699999, 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.91822508, 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25489", 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2934336176", 2024-03-30T15:03:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989541+0000", 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8899999999999999e-06 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989549+0000", 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6000000000000001e-06 2024-03-30T15:03:24.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989557+0000", 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6819999999999992e-06 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989564+0000", 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7649999999999996e-06 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989627+0000", 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2780000000000005e-05 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989635+0000", 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6029999999999999e-06 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894091+0000", 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.904456081 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894206+0000", 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115474 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894346+0000", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139486 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.896778+0000", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002432156 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907464+0000", 2024-03-30T15:03:24.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010685741 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907629+0000", 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000165055 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907721+0000", 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2670000000000006e-05 2024-03-30T15:03:24.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907764+0000", 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3096999999999998e-05 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35396.0:1 66.0s0 66:074d3272:test-rados-api-smithi028-93069-23::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:03:24.143 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 322.22329465899998, 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.114964936, 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35396", 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/658745307", 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853048+0000", 2024-03-30T15:03:24.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9520000000000001e-06 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853055+0000", 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.576e-06 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853065+0000", 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9790000000000005e-06 2024-03-30T15:03:24.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853076+0000", 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0764999999999999e-05 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.854153+0000", 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010775750000000001 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.854157+0000", 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.331e-06 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939401+0000", 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.085244367 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939520+0000", 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119022 2024-03-30T15:03:24.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939930+0000", 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040942499999999999 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.940245+0000", 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031576299999999999 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.967952+0000", 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027706232000000001 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.968011+0000", 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8949000000000001e-05 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.150 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35956.0:2 125.1b 125:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 299.67928596500002, 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.270631477, 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35956", 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4007703721", 2024-03-30T15:03:24.151 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397056+0000", 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6249999999999999e-06 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397063+0000", 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8850000000000001e-06 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397072+0000", 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3989999999999993e-06 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397082+0000", 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0210000000000001e-05 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.404263+0000", 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007180484 2024-03-30T15:03:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.404283+0000", 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9973e-05 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.662959+0000", 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.258676382 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663072+0000", 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113008 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,6", 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663217+0000", 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014457599999999999 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663678+0000", 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046156600000000001 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663884+0000", 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020523500000000001 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667630+0000", 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003746096 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667660+0000", 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0332999999999999e-05 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667686+0000", 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5704999999999999e-05 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35977.0:2 126.16 126:68f45781:::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 298.47961303900001, 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.079154859999999994, 2024-03-30T15:03:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35977", 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2482478745", 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596729+0000", 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8959999999999999e-06 2024-03-30T15:03:24.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596736+0000", 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.489e-06 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596744+0000", 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7849999999999997e-06 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596754+0000", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1789999999999997e-06 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596838+0000", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4194000000000001e-05 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596846+0000", 2024-03-30T15:03:24.163 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6330000000000002e-06 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.662948+0000", 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.066101602999999995 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663060+0000", 2024-03-30T15:03:24.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112466 2024-03-30T15:03:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663207+0000", 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014695100000000001 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663682+0000", 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00047461399999999999 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667728+0000", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040456939999999999 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675811+0000", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080834860000000008 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675843+0000", 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1751000000000001e-05 2024-03-30T15:03:24.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675882+0000", 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9119000000000003e-05 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46235.0:515 130.4 130:24299189:::test_508:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e145)", 2024-03-30T15:03:24.169 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 293.93175846399998, 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062198679999999999, 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46235", 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 515 2024-03-30T15:03:24.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144583+0000", 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2890000000000001e-06 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144591+0000", 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0630000000000006e-06 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.145783+0000", 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011915199999999999 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.145797+0000", 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4464000000000001e-05 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.168993+0000", 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023195436 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.169128+0000", 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013532799999999999 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.206780+0000", 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.037652579999999998 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.56882.0:2 166.18 166:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 277.872849804, 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030150073999999999, 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.175 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.56882", 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1234052616", 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:03:24.176 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203492+0000", 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0470000000000001e-06 2024-03-30T15:03:24.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203504+0000", 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1810999999999999e-05 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203514+0000", 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0009e-05 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203535+0000", 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0738e-05 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.213352+0000", 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098172729999999996 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.213361+0000", 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1069999999999994e-06 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226469+0000", 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013108066 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226597+0000", 2024-03-30T15:03:24.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012716999999999999 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226734+0000", 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013731299999999999 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.227951+0000", 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012165450000000001 2024-03-30T15:03:24.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.232907+0000", 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049565479999999999 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233549+0000", 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064232700000000005 2024-03-30T15:03:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233599+0000", 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9830000000000002e-05 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233641+0000", 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1289999999999999e-05 2024-03-30T15:03:24.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57104.0:13 195.6s0 195:71611c3d:test-rados-api-smithi028-93525-20::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 244.86700384900001, 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036499746999999999, 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.184 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57104", 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2817065657", 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 13 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209339+0000", 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.322e-06 2024-03-30T15:03:24.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209348+0000", 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7499999999999998e-06 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209362+0000", 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3577000000000001e-05 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209374+0000", 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2077e-05 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.210150+0000", 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00077631599999999999 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.188 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.210159+0000", 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0359999999999995e-06 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219259+0000", 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090994409999999998 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219399+0000", 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013971400000000001 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219883+0000", 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00048410599999999999 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.222618+0000", 2024-03-30T15:03:24.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027352790000000002 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.245760+0000", 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023141623 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.245836+0000", 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6506000000000002e-05 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.191 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57182.0:2 206.11 206:8e43fa19:test-rados-api-smithi028-93257-34::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 240.88499010000001, 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.032742122999999998, 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.192 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57182", 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/354569125", 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.193 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191353+0000", 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3800000000000001e-06 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.194 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191368+0000", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5608e-05 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191379+0000", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1093e-05 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191391+0000", 2024-03-30T15:03:24.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1999e-05 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.193786+0000", 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023949930000000002 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.193797+0000", 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0971999999999999e-05 2024-03-30T15:03:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214441+0000", 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020643470000000001 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214580+0000", 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013977799999999999 2024-03-30T15:03:24.197 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214734+0000", 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000153351 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.218361+0000", 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036272399999999999 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.221480+0000", 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003119442 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.223991+0000", 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025103450000000002 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.224045+0000", 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4627999999999999e-05 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.224092+0000", 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6823999999999998e-05 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.200 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57227.0:2 211.d 211:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 235.87327381099999, 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.040305306999999999, 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57227", 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1866544138", 2024-03-30T15:03:24.201 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:03:24.202 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203068+0000", 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9690000000000001e-06 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203077+0000", 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9230000000000008e-06 2024-03-30T15:03:24.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203088+0000", 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0525e-05 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203097+0000", 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6260000000000007e-06 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203155+0000", 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7924999999999999e-05 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203160+0000", 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9439999999999996e-06 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.210784+0000", 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076239929999999999 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.210794+0000", 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7180000000000008e-06 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.206 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.227995+0000", 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017200653999999999 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.228136+0000", 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141679 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:03:24.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.228273+0000", 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013709700000000001 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.232978+0000", 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047047629999999998 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.241577+0000", 2024-03-30T15:03:24.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0085986069999999994 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243117+0000", 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015401970000000001 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243325+0000", 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020754599999999999 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243372+0000", 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7141000000000001e-05 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57224.0:1 212.6s0 212:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-03-30T15:03:24.210 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 234.85851140899999, 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0139440129999999, 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57224", 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2535484916", 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217831+0000", 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9479999999999998e-06 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217842+0000", 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0769e-05 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217855+0000", 2024-03-30T15:03:24.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3015e-05 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217863+0000", 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3860000000000007e-06 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217925+0000", 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2392999999999997e-05 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217928+0000", 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9780000000000001e-06 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230208+0000", 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0122793919999999 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.215 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230351+0000", 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014353200000000001 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230724+0000", 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037236099999999998 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231099+0000", 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037526500000000001 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.216 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231702+0000", 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00060294299999999999 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231773+0000", 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1031000000000005e-05 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57476.0:1 245.6s0 245:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 178.84433282099999, 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0238222990000001, 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57476", 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4051625719", 2024-03-30T15:03:24.218 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232010+0000", 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.604e-06 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232018+0000", 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3520000000000007e-06 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232031+0000", 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2765999999999999e-05 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232043+0000", 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1732e-05 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232111+0000", 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8403000000000002e-05 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232115+0000", 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7220000000000001e-06 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252496+0000", 2024-03-30T15:03:24.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0203810069999999 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252648+0000", 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000152017 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.254117+0000", 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014689320000000001 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255025+0000", 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090758899999999999 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255742+0000", 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071763700000000003 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255830+0000", 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7538000000000001e-05 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49756.0:1 273.6s0 273:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 102.762442702, 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93441624899999998, 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.225 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49756", 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1885953997", 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:03:24.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313900+0000", 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.785e-06 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313911+0000", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0925e-05 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313947+0000", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6205000000000003e-05 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313960+0000", 2024-03-30T15:03:24.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2768999999999999e-05 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525411+0000", 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.21145103200000001 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525415+0000", 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.014e-06 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.246985+0000", 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.72157016200000001 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247116+0000", 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013093499999999999 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247514+0000", 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039808699999999999 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247951+0000", 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043615700000000001 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:03:24.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248270+0000", 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031954200000000002 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248314+0000", 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3636e-05 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.232 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49768.0:1 274.5s0 274:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e338)", 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 101.844121093, 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0270852340000001, 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49768", 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3544512235", 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.233 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232221+0000", 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3120000000000001e-06 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232228+0000", 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1680000000000001e-06 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:03:24.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232240+0000", 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2248e-05 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232251+0000", 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1559000000000001e-05 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234151+0000", 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001899277 2024-03-30T15:03:24.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234167+0000", 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5872e-05 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259198+0000", 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025031091 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:03:24.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259304+0000", 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010670699999999999 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:03:24.238 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:03:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:24.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:24.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:03:24.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:24.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 42.18 scrub starts 2024-03-30T15:03:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 42.18 scrub ok 2024-03-30T15:03:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:03:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: from='client.24813 v1:172.21.15.28:0/2960905778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: pgmap v592: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 101.8 scrub starts 2024-03-30T15:03:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:24 smithi134 ceph-mon[33714]: 101.8 scrub ok 2024-03-30T15:03:24.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 42.18 scrub starts 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 42.18 scrub ok 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:03:24.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: from='client.24813 v1:172.21.15.28:0/2960905778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: pgmap v592: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 101.8 scrub starts 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[41614]: 101.8 scrub ok 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 42.18 scrub starts 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 42.18 scrub ok 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: osdmap e432: 8 total, 8 up, 8 in 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:03:24.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:03:24.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: from='client.24813 v1:172.21.15.28:0/2960905778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:03:24.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: pgmap v592: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:24.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 101.8 scrub starts 2024-03-30T15:03:24.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:24 smithi028 ceph-mon[29876]: 101.8 scrub ok 2024-03-30T15:03:24.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:24.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:24.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:24.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:24.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:24.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:24.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:24.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2024-03-30T15:03:23.277498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792082,"num_objects":355,"num_object_clones":2,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":1559,"num_read_kb":1305,"num_write":1212,"num_write_kb":15147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2543,"ondisk_log_size":2543,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":715,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":684244,"kb_used_data":84908,"kb_used_omap":12,"kb_used_meta":599091,"kb_avail":749276972,"statfs":{"total":767960285184,"available":767259619328,"internally_reserved":0,"allocated":86945792,"data_stored":60708291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":613469744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015688"},"pg_stats":[{"pgid":"296.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"296.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:03:21.266079+0000","last_change":"2024-03-30T15:03:21.266079+0000","last_active":"2024-03-30T15:03:21.266079+0000","last_peered":"2024-03-30T15:03:21.266079+0000","last_clean":"2024-03-30T15:03:21.266079+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:03:21.266079+0000","last_undegraded":"2024-03-30T15:03:21.266079+0000","last_fullsized":"2024-03-30T15:03:21.266079+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:21.266079+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.266079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":805,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266231+0000","last_change":"2024-03-30T15:01:12.237476+0000","last_active":"2024-03-30T15:03:18.266231+0000","last_peered":"2024-03-30T15:03:18.266231+0000","last_clean":"2024-03-30T15:03:18.266231+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:03:18.266231+0000","last_undegraded":"2024-03-30T15:03:18.266231+0000","last_fullsized":"2024-03-30T15:03:18.266231+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:01:08.589132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:08.589132+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058544290000000004,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":834,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269123+0000","last_change":"2024-03-30T15:02:31.961593+0000","last_active":"2024-03-30T15:03:18.269123+0000","last_peered":"2024-03-30T15:03:18.269123+0000","last_clean":"2024-03-30T15:03:18.269123+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:03:18.269123+0000","last_undegraded":"2024-03-30T15:03:18.269123+0000","last_fullsized":"2024-03-30T15:03:18.269123+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:31.961438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.961438+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:40.263914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016816940000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"400'89","reported_seq":843,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.257908+0000","last_change":"2024-03-30T15:02:40.447187+0000","last_active":"2024-03-30T15:03:18.257908+0000","last_peered":"2024-03-30T15:03:18.257908+0000","last_clean":"2024-03-30T15:03:18.257908+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:03:18.257908+0000","last_undegraded":"2024-03-30T15:03:18.257908+0000","last_fullsized":"2024-03-30T15:03:18.257908+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'87","last_scrub_stamp":"2024-03-30T15:02:40.447037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.447037+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:46.192952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0013249889999999999,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":863,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264419+0000","last_change":"2024-03-30T15:01:12.224172+0000","last_active":"2024-03-30T15:03:18.264419+0000","last_peered":"2024-03-30T15:03:18.264419+0000","last_clean":"2024-03-30T15:03:18.264419+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:03:18.264419+0000","last_undegraded":"2024-03-30T15:03:18.264419+0000","last_fullsized":"2024-03-30T15:03:18.264419+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'102","last_scrub_stamp":"2024-03-30T15:01:09.466765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:09.466765+0000","objects_scrubbed":6,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062489279999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"286'36","reported_seq":800,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271222+0000","last_change":"2024-03-30T15:01:12.235443+0000","last_active":"2024-03-30T15:03:18.271222+0000","last_peered":"2024-03-30T15:03:18.271222+0000","last_clean":"2024-03-30T15:03:18.271222+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:03:18.271222+0000","last_undegraded":"2024-03-30T15:03:18.271222+0000","last_fullsized":"2024-03-30T15:03:18.271222+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.725810+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0065316139999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":904,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273251+0000","last_change":"2024-03-30T15:02:47.930062+0000","last_active":"2024-03-30T15:03:18.273251+0000","last_peered":"2024-03-30T15:03:18.273251+0000","last_clean":"2024-03-30T15:03:18.273251+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:03:18.273251+0000","last_undegraded":"2024-03-30T15:03:18.273251+0000","last_fullsized":"2024-03-30T15:03:18.273251+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:55.011112+0000","scrub_duration":12,"objects_trimmed":1,"snaptrim_duration":0.016684201999999999,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":796,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268620+0000","last_change":"2024-03-30T15:02:30.913136+0000","last_active":"2024-03-30T15:03:18.268620+0000","last_peered":"2024-03-30T15:03:18.268620+0000","last_clean":"2024-03-30T15:03:18.268620+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:03:18.268620+0000","last_undegraded":"2024-03-30T15:03:18.268620+0000","last_fullsized":"2024-03-30T15:03:18.268620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:46.599450+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001698305,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":808,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258649+0000","last_change":"2024-03-30T15:01:59.441437+0000","last_active":"2024-03-30T15:03:18.258649+0000","last_peered":"2024-03-30T15:03:18.258649+0000","last_clean":"2024-03-30T15:03:18.258649+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:03:18.258649+0000","last_undegraded":"2024-03-30T15:03:18.258649+0000","last_fullsized":"2024-03-30T15:03:18.258649+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:01:59.441350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.441350+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036882529999999998,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":806,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258598+0000","last_change":"2024-03-30T15:02:02.557714+0000","last_active":"2024-03-30T15:03:18.258598+0000","last_peered":"2024-03-30T15:03:18.258598+0000","last_clean":"2024-03-30T15:03:18.258598+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:03:18.258598+0000","last_undegraded":"2024-03-30T15:03:18.258598+0000","last_fullsized":"2024-03-30T15:03:18.258598+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.557627+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037348239999999999,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":798,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258564+0000","last_change":"2024-03-30T15:02:00.557168+0000","last_active":"2024-03-30T15:03:18.258564+0000","last_peered":"2024-03-30T15:03:18.258564+0000","last_clean":"2024-03-30T15:03:18.258564+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:03:18.258564+0000","last_undegraded":"2024-03-30T15:03:18.258564+0000","last_fullsized":"2024-03-30T15:03:18.258564+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:02:00.557079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.557079+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051724700000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"421'7","reported_seq":771,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.257953+0000","last_change":"2024-03-30T15:02:29.567221+0000","last_active":"2024-03-30T15:03:18.257953+0000","last_peered":"2024-03-30T15:03:18.257953+0000","last_clean":"2024-03-30T15:03:18.257953+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:03:18.257953+0000","last_undegraded":"2024-03-30T15:03:18.257953+0000","last_fullsized":"2024-03-30T15:03:18.257953+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'5","last_scrub_stamp":"2024-03-30T15:02:29.567113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:29.567113+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:33.919594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0014867789999999999,"stat_sum":{"num_bytes":18,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"421'19","reported_seq":783,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258557+0000","last_change":"2024-03-30T15:02:35.464869+0000","last_active":"2024-03-30T15:03:18.258557+0000","last_peered":"2024-03-30T15:03:18.258557+0000","last_clean":"2024-03-30T15:03:18.258557+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:03:18.258557+0000","last_undegraded":"2024-03-30T15:03:18.258557+0000","last_fullsized":"2024-03-30T15:03:18.258557+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'17","last_scrub_stamp":"2024-03-30T15:02:35.464703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.464703+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:41.881914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0026988469999999999,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":779,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269462+0000","last_change":"2024-03-30T15:02:28.859585+0000","last_active":"2024-03-30T15:03:18.269462+0000","last_peered":"2024-03-30T15:03:18.269462+0000","last_clean":"2024-03-30T15:03:18.269462+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:03:18.269462+0000","last_undegraded":"2024-03-30T15:03:18.269462+0000","last_fullsized":"2024-03-30T15:03:18.269462+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:54.947610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00159227,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":842,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260817+0000","last_change":"2024-03-30T15:02:38.712411+0000","last_active":"2024-03-30T15:03:18.260817+0000","last_peered":"2024-03-30T15:03:18.260817+0000","last_clean":"2024-03-30T15:03:18.260817+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:03:18.260817+0000","last_undegraded":"2024-03-30T15:03:18.260817+0000","last_fullsized":"2024-03-30T15:03:18.260817+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:39.003325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0034647839999999998,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"368'54","reported_seq":817,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258494+0000","last_change":"2024-03-30T15:02:03.544671+0000","last_active":"2024-03-30T15:03:18.258494+0000","last_peered":"2024-03-30T15:03:18.258494+0000","last_clean":"2024-03-30T15:03:18.258494+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:03:18.258494+0000","last_undegraded":"2024-03-30T15:03:18.258494+0000","last_fullsized":"2024-03-30T15:03:18.258494+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:33.407632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063330620000000004,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":849,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271990+0000","last_change":"2024-03-30T15:02:50.933106+0000","last_active":"2024-03-30T15:03:18.271990+0000","last_peered":"2024-03-30T15:03:18.271990+0000","last_clean":"2024-03-30T15:03:18.271990+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:03:18.271990+0000","last_undegraded":"2024-03-30T15:03:18.271990+0000","last_fullsized":"2024-03-30T15:03:18.271990+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.089487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0075118169999999996,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"368'35","reported_seq":795,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259107+0000","last_change":"2024-03-30T15:02:42.437538+0000","last_active":"2024-03-30T15:03:18.259107+0000","last_peered":"2024-03-30T15:03:18.259107+0000","last_clean":"2024-03-30T15:03:18.259107+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:03:18.259107+0000","last_undegraded":"2024-03-30T15:03:18.259107+0000","last_fullsized":"2024-03-30T15:03:18.259107+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'35","last_scrub_stamp":"2024-03-30T15:02:42.436904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.436904+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:52.929370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0051419289999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":864,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264634+0000","last_change":"2024-03-30T15:02:39.411420+0000","last_active":"2024-03-30T15:03:18.264634+0000","last_peered":"2024-03-30T15:03:18.264634+0000","last_clean":"2024-03-30T15:03:18.264634+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:03:18.264634+0000","last_undegraded":"2024-03-30T15:03:18.264634+0000","last_fullsized":"2024-03-30T15:03:18.264634+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:02:39.411331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.411331+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:01.941801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001401331,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"204'34","reported_seq":795,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265863+0000","last_change":"2024-03-30T15:01:12.226433+0000","last_active":"2024-03-30T15:03:18.265863+0000","last_peered":"2024-03-30T15:03:18.265863+0000","last_clean":"2024-03-30T15:03:18.265863+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:03:18.265863+0000","last_undegraded":"2024-03-30T15:03:18.265863+0000","last_fullsized":"2024-03-30T15:03:18.265863+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'34","last_scrub_stamp":"2024-03-30T14:59:47.247116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T14:59:47.247116+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0036675259999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":807,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269041+0000","last_change":"2024-03-30T15:02:41.402516+0000","last_active":"2024-03-30T15:03:18.269041+0000","last_peered":"2024-03-30T15:03:18.269041+0000","last_clean":"2024-03-30T15:03:18.269041+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:03:18.269041+0000","last_undegraded":"2024-03-30T15:03:18.269041+0000","last_fullsized":"2024-03-30T15:03:18.269041+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:02:41.402421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.402421+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:48.957834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014568300000000001,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":865,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263166+0000","last_change":"2024-03-30T15:02:31.357257+0000","last_active":"2024-03-30T15:03:18.263166+0000","last_peered":"2024-03-30T15:03:18.263166+0000","last_clean":"2024-03-30T15:03:18.263166+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:03:18.263166+0000","last_undegraded":"2024-03-30T15:03:18.263166+0000","last_fullsized":"2024-03-30T15:03:18.263166+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:02:31.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.356987+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:31.807490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00032509100000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":826,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260928+0000","last_change":"2024-03-30T15:02:07.572745+0000","last_active":"2024-03-30T15:03:18.260928+0000","last_peered":"2024-03-30T15:03:18.260928+0000","last_clean":"2024-03-30T15:03:18.260928+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:03:18.260928+0000","last_undegraded":"2024-03-30T15:03:18.260928+0000","last_fullsized":"2024-03-30T15:03:18.260928+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'56","last_scrub_stamp":"2024-03-30T15:02:07.572656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.572656+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0039401879999999999,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"424'56","reported_seq":812,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259864+0000","last_change":"2024-03-30T15:01:12.224719+0000","last_active":"2024-03-30T15:03:18.259864+0000","last_peered":"2024-03-30T15:03:18.259864+0000","last_clean":"2024-03-30T15:03:18.259864+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:03:18.259864+0000","last_undegraded":"2024-03-30T15:03:18.259864+0000","last_fullsized":"2024-03-30T15:03:18.259864+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-03-30T15:00:29.555003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:00:29.555003+0000","objects_scrubbed":3,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0036771759999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"400'89","reported_seq":838,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270395+0000","last_change":"2024-03-30T15:02:01.657048+0000","last_active":"2024-03-30T15:03:18.270395+0000","last_peered":"2024-03-30T15:03:18.270395+0000","last_clean":"2024-03-30T15:03:18.270395+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:03:18.270395+0000","last_undegraded":"2024-03-30T15:03:18.270395+0000","last_fullsized":"2024-03-30T15:03:18.270395+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'87","last_scrub_stamp":"2024-03-30T15:02:01.656924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.656924+0000","objects_scrubbed":6,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0017814040000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"368'86","reported_seq":843,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262744+0000","last_change":"2024-03-30T15:02:27.539764+0000","last_active":"2024-03-30T15:03:18.262744+0000","last_peered":"2024-03-30T15:03:18.262744+0000","last_clean":"2024-03-30T15:03:18.262744+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:03:18.262744+0000","last_undegraded":"2024-03-30T15:03:18.262744+0000","last_fullsized":"2024-03-30T15:03:18.262744+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:39.676822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0071374050000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":810,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266442+0000","last_change":"2024-03-30T15:01:46.416810+0000","last_active":"2024-03-30T15:03:18.266442+0000","last_peered":"2024-03-30T15:03:18.266442+0000","last_clean":"2024-03-30T15:03:18.266442+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:03:18.266442+0000","last_undegraded":"2024-03-30T15:03:18.266442+0000","last_fullsized":"2024-03-30T15:03:18.266442+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:01:46.416710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.416710+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0055223779999999997,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"248'39","reported_seq":811,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265563+0000","last_change":"2024-03-30T15:01:45.468045+0000","last_active":"2024-03-30T15:03:18.265563+0000","last_peered":"2024-03-30T15:03:18.265563+0000","last_clean":"2024-03-30T15:03:18.265563+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:03:18.265563+0000","last_undegraded":"2024-03-30T15:03:18.265563+0000","last_fullsized":"2024-03-30T15:03:18.265563+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:01:45.467945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:45.467945+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.005125569,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"324'104","reported_seq":868,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269721+0000","last_change":"2024-03-30T15:02:28.234858+0000","last_active":"2024-03-30T15:03:18.269721+0000","last_peered":"2024-03-30T15:03:18.269721+0000","last_clean":"2024-03-30T15:03:18.269721+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:03:18.269721+0000","last_undegraded":"2024-03-30T15:03:18.269721+0000","last_fullsized":"2024-03-30T15:03:18.269721+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:02:28.234739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.234739+0000","objects_scrubbed":8,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:41.073727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001509961,"stat_sum":{"num_bytes":4000074,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":3937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":789,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260977+0000","last_change":"2024-03-30T15:02:36.478453+0000","last_active":"2024-03-30T15:03:18.260977+0000","last_peered":"2024-03-30T15:03:18.260977+0000","last_clean":"2024-03-30T15:03:18.260977+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:03:18.260977+0000","last_undegraded":"2024-03-30T15:03:18.260977+0000","last_fullsized":"2024-03-30T15:03:18.260977+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:02:36.478360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.478360+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.409942+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0046413940000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":779,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273519+0000","last_change":"2024-03-30T15:02:28.967570+0000","last_active":"2024-03-30T15:03:18.273519+0000","last_peered":"2024-03-30T15:03:18.273519+0000","last_clean":"2024-03-30T15:03:18.273519+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:03:18.273519+0000","last_undegraded":"2024-03-30T15:03:18.273519+0000","last_fullsized":"2024-03-30T15:03:18.273519+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:02:28.967486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.967486+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:43.672530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0063778710000000002,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":823,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258603+0000","last_change":"2024-03-30T15:02:42.619819+0000","last_active":"2024-03-30T15:03:18.258603+0000","last_peered":"2024-03-30T15:03:18.258603+0000","last_clean":"2024-03-30T15:03:18.258603+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:03:18.258603+0000","last_undegraded":"2024-03-30T15:03:18.258603+0000","last_fullsized":"2024-03-30T15:03:18.258603+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:44.428562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0065541890000000002,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":859,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267848+0000","last_change":"2024-03-30T15:02:27.989384+0000","last_active":"2024-03-30T15:03:18.267848+0000","last_peered":"2024-03-30T15:03:18.267848+0000","last_clean":"2024-03-30T15:03:18.267848+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:03:18.267848+0000","last_undegraded":"2024-03-30T15:03:18.267848+0000","last_fullsized":"2024-03-30T15:03:18.267848+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:02:27.989261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.989261+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:29.258872+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016655120000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267638+0000","last_change":"2024-03-30T15:01:50.157960+0000","last_active":"2024-03-30T15:03:18.267638+0000","last_peered":"2024-03-30T15:03:18.267638+0000","last_clean":"2024-03-30T15:03:18.267638+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:03:18.267638+0000","last_undegraded":"2024-03-30T15:03:18.267638+0000","last_fullsized":"2024-03-30T15:03:18.267638+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.157883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.157883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":650,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262215+0000","last_change":"2024-03-30T15:01:59.125848+0000","last_active":"2024-03-30T15:03:18.262215+0000","last_peered":"2024-03-30T15:03:18.262215+0000","last_clean":"2024-03-30T15:03:18.262215+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:03:18.262215+0000","last_undegraded":"2024-03-30T15:03:18.262215+0000","last_fullsized":"2024-03-30T15:03:18.262215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.125727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.125727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:24.130224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270388+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:03:18.270388+0000","last_peered":"2024-03-30T15:03:18.270388+0000","last_clean":"2024-03-30T15:03:18.270388+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:03:18.270388+0000","last_undegraded":"2024-03-30T15:03:18.270388+0000","last_fullsized":"2024-03-30T15:03:18.270388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.272436+0000","last_change":"2024-03-30T15:00:50.386813+0000","last_active":"2024-03-30T15:03:18.272436+0000","last_peered":"2024-03-30T15:03:18.272436+0000","last_clean":"2024-03-30T15:03:18.272436+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:03:18.272436+0000","last_undegraded":"2024-03-30T15:03:18.272436+0000","last_fullsized":"2024-03-30T15:03:18.272436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:50.386674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:50.386674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":645,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259616+0000","last_change":"2024-03-30T15:01:57.530926+0000","last_active":"2024-03-30T15:03:18.259616+0000","last_peered":"2024-03-30T15:03:18.259616+0000","last_clean":"2024-03-30T15:03:18.259616+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:03:18.259616+0000","last_undegraded":"2024-03-30T15:03:18.259616+0000","last_fullsized":"2024-03-30T15:03:18.259616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.530827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.530827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:27.144111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265663+0000","last_change":"2024-03-30T15:02:41.928550+0000","last_active":"2024-03-30T15:03:18.265663+0000","last_peered":"2024-03-30T15:03:18.265663+0000","last_clean":"2024-03-30T15:03:18.265663+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:03:18.265663+0000","last_undegraded":"2024-03-30T15:03:18.265663+0000","last_fullsized":"2024-03-30T15:03:18.265663+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.928469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.928469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.706155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270483+0000","last_change":"2024-03-30T15:01:57.663288+0000","last_active":"2024-03-30T15:03:18.270483+0000","last_peered":"2024-03-30T15:03:18.270483+0000","last_clean":"2024-03-30T15:03:18.270483+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:03:18.270483+0000","last_undegraded":"2024-03-30T15:03:18.270483+0000","last_fullsized":"2024-03-30T15:03:18.270483+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.663204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.663204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:22.694299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264008+0000","last_change":"2024-03-30T15:01:59.181493+0000","last_active":"2024-03-30T15:03:18.264008+0000","last_peered":"2024-03-30T15:03:18.264008+0000","last_clean":"2024-03-30T15:03:18.264008+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:03:18.264008+0000","last_undegraded":"2024-03-30T15:03:18.264008+0000","last_fullsized":"2024-03-30T15:03:18.264008+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.181401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.181401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259721+0000","last_change":"2024-03-30T15:02:34.464153+0000","last_active":"2024-03-30T15:03:18.259721+0000","last_peered":"2024-03-30T15:03:18.259721+0000","last_clean":"2024-03-30T15:03:18.259721+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:03:18.259721+0000","last_undegraded":"2024-03-30T15:03:18.259721+0000","last_fullsized":"2024-03-30T15:03:18.259721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:34.464067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.464067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:52.932209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270544+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:03:18.270544+0000","last_peered":"2024-03-30T15:03:18.270544+0000","last_clean":"2024-03-30T15:03:18.270544+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:03:18.270544+0000","last_undegraded":"2024-03-30T15:03:18.270544+0000","last_fullsized":"2024-03-30T15:03:18.270544+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.860968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270371+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:03:18.270371+0000","last_peered":"2024-03-30T15:03:18.270371+0000","last_clean":"2024-03-30T15:03:18.270371+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:03:18.270371+0000","last_undegraded":"2024-03-30T15:03:18.270371+0000","last_fullsized":"2024-03-30T15:03:18.270371+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.272416+0000","last_change":"2024-03-30T15:02:38.936916+0000","last_active":"2024-03-30T15:03:18.272416+0000","last_peered":"2024-03-30T15:03:18.272416+0000","last_clean":"2024-03-30T15:03:18.272416+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:03:18.272416+0000","last_undegraded":"2024-03-30T15:03:18.272416+0000","last_fullsized":"2024-03-30T15:03:18.272416+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:38.936820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.936820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:03.472567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267539+0000","last_change":"2024-03-30T15:01:51.636327+0000","last_active":"2024-03-30T15:03:18.267539+0000","last_peered":"2024-03-30T15:03:18.267539+0000","last_clean":"2024-03-30T15:03:18.267539+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:03:18.267539+0000","last_undegraded":"2024-03-30T15:03:18.267539+0000","last_fullsized":"2024-03-30T15:03:18.267539+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.636231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.636231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262889+0000","last_change":"2024-03-30T15:00:40.305048+0000","last_active":"2024-03-30T15:03:18.262889+0000","last_peered":"2024-03-30T15:03:18.262889+0000","last_clean":"2024-03-30T15:03:18.262889+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:03:18.262889+0000","last_undegraded":"2024-03-30T15:03:18.262889+0000","last_fullsized":"2024-03-30T15:03:18.262889+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:40.304903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:40.304903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266572+0000","last_change":"2024-03-30T15:02:00.582792+0000","last_active":"2024-03-30T15:03:18.266572+0000","last_peered":"2024-03-30T15:03:18.266572+0000","last_clean":"2024-03-30T15:03:18.266572+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:03:18.266572+0000","last_undegraded":"2024-03-30T15:03:18.266572+0000","last_fullsized":"2024-03-30T15:03:18.266572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.582709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266333+0000","last_change":"2024-03-30T15:02:07.597777+0000","last_active":"2024-03-30T15:03:18.266333+0000","last_peered":"2024-03-30T15:03:18.266333+0000","last_clean":"2024-03-30T15:03:18.266333+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:03:18.266333+0000","last_undegraded":"2024-03-30T15:03:18.266333+0000","last_fullsized":"2024-03-30T15:03:18.266333+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.597700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:19.362527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270612+0000","last_change":"2024-03-30T15:01:10.292440+0000","last_active":"2024-03-30T15:03:18.270612+0000","last_peered":"2024-03-30T15:03:18.270612+0000","last_clean":"2024-03-30T15:03:18.270612+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:03:18.270612+0000","last_undegraded":"2024-03-30T15:03:18.270612+0000","last_fullsized":"2024-03-30T15:03:18.270612+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.292346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.292346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260438+0000","last_change":"2024-03-30T15:01:54.513739+0000","last_active":"2024-03-30T15:03:18.260438+0000","last_peered":"2024-03-30T15:03:18.260438+0000","last_clean":"2024-03-30T15:03:18.260438+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:03:18.260438+0000","last_undegraded":"2024-03-30T15:03:18.260438+0000","last_fullsized":"2024-03-30T15:03:18.260438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.513601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.513601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267670+0000","last_change":"2024-03-30T15:01:22.172967+0000","last_active":"2024-03-30T15:03:18.267670+0000","last_peered":"2024-03-30T15:03:18.267670+0000","last_clean":"2024-03-30T15:03:18.267670+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:03:18.267670+0000","last_undegraded":"2024-03-30T15:03:18.267670+0000","last_fullsized":"2024-03-30T15:03:18.267670+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:22.172854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:22.172854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269640+0000","last_change":"2024-03-30T15:01:19.395733+0000","last_active":"2024-03-30T15:03:18.269640+0000","last_peered":"2024-03-30T15:03:18.269640+0000","last_clean":"2024-03-30T15:03:18.269640+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:03:18.269640+0000","last_undegraded":"2024-03-30T15:03:18.269640+0000","last_fullsized":"2024-03-30T15:03:18.269640+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265487+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:03:18.265487+0000","last_peered":"2024-03-30T15:03:18.265487+0000","last_clean":"2024-03-30T15:03:18.265487+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:03:18.265487+0000","last_undegraded":"2024-03-30T15:03:18.265487+0000","last_fullsized":"2024-03-30T15:03:18.265487+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.690679+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266978+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:03:18.266978+0000","last_peered":"2024-03-30T15:03:18.266978+0000","last_clean":"2024-03-30T15:03:18.266978+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:03:18.266978+0000","last_undegraded":"2024-03-30T15:03:18.266978+0000","last_fullsized":"2024-03-30T15:03:18.266978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":650,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264594+0000","last_change":"2024-03-30T15:02:40.426121+0000","last_active":"2024-03-30T15:03:18.264594+0000","last_peered":"2024-03-30T15:03:18.264594+0000","last_clean":"2024-03-30T15:03:18.264594+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:03:18.264594+0000","last_undegraded":"2024-03-30T15:03:18.264594+0000","last_fullsized":"2024-03-30T15:03:18.264594+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.426031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.426031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:58.345766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265484+0000","last_change":"2024-03-30T15:01:21.414752+0000","last_active":"2024-03-30T15:03:18.265484+0000","last_peered":"2024-03-30T15:03:18.265484+0000","last_clean":"2024-03-30T15:03:18.265484+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:03:18.265484+0000","last_undegraded":"2024-03-30T15:03:18.265484+0000","last_fullsized":"2024-03-30T15:03:18.265484+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:01:21.414672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":650,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265493+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:03:18.265493+0000","last_peered":"2024-03-30T15:03:18.265493+0000","last_clean":"2024-03-30T15:03:18.265493+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:03:18.265493+0000","last_undegraded":"2024-03-30T15:03:18.265493+0000","last_fullsized":"2024-03-30T15:03:18.265493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:04.572831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270650+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:03:18.270650+0000","last_peered":"2024-03-30T15:03:18.270650+0000","last_clean":"2024-03-30T15:03:18.270650+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:03:18.270650+0000","last_undegraded":"2024-03-30T15:03:18.270650+0000","last_fullsized":"2024-03-30T15:03:18.270650+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.321710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268452+0000","last_change":"2024-03-30T15:01:58.138003+0000","last_active":"2024-03-30T15:03:18.268452+0000","last_peered":"2024-03-30T15:03:18.268452+0000","last_clean":"2024-03-30T15:03:18.268452+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:03:18.268452+0000","last_undegraded":"2024-03-30T15:03:18.268452+0000","last_fullsized":"2024-03-30T15:03:18.268452+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.137887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.137887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:26.595005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266261+0000","last_change":"2024-03-30T15:01:57.095860+0000","last_active":"2024-03-30T15:03:18.266261+0000","last_peered":"2024-03-30T15:03:18.266261+0000","last_clean":"2024-03-30T15:03:18.266261+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:03:18.266261+0000","last_undegraded":"2024-03-30T15:03:18.266261+0000","last_fullsized":"2024-03-30T15:03:18.266261+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.095752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.095752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:20.098612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":650,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269150+0000","last_change":"2024-03-30T15:01:58.162003+0000","last_active":"2024-03-30T15:03:18.269150+0000","last_peered":"2024-03-30T15:03:18.269150+0000","last_clean":"2024-03-30T15:03:18.269150+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:03:18.269150+0000","last_undegraded":"2024-03-30T15:03:18.269150+0000","last_fullsized":"2024-03-30T15:03:18.269150+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.161871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.161871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:26.818497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265699+0000","last_change":"2024-03-30T15:01:20.107429+0000","last_active":"2024-03-30T15:03:18.265699+0000","last_peered":"2024-03-30T15:03:18.265699+0000","last_clean":"2024-03-30T15:03:18.265699+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:03:18.265699+0000","last_undegraded":"2024-03-30T15:03:18.265699+0000","last_fullsized":"2024-03-30T15:03:18.265699+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:20.107350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.107350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259751+0000","last_change":"2024-03-30T15:01:51.481224+0000","last_active":"2024-03-30T15:03:18.259751+0000","last_peered":"2024-03-30T15:03:18.259751+0000","last_clean":"2024-03-30T15:03:18.259751+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:03:18.259751+0000","last_undegraded":"2024-03-30T15:03:18.259751+0000","last_fullsized":"2024-03-30T15:03:18.259751+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.481143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.481143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":642,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268062+0000","last_change":"2024-03-30T15:01:20.445532+0000","last_active":"2024-03-30T15:03:18.268062+0000","last_peered":"2024-03-30T15:03:18.268062+0000","last_clean":"2024-03-30T15:03:18.268062+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:03:18.268062+0000","last_undegraded":"2024-03-30T15:03:18.268062+0000","last_fullsized":"2024-03-30T15:03:18.268062+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:20.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262337+0000","last_change":"2024-03-30T15:01:51.100729+0000","last_active":"2024-03-30T15:03:18.262337+0000","last_peered":"2024-03-30T15:03:18.262337+0000","last_clean":"2024-03-30T15:03:18.262337+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:03:18.262337+0000","last_undegraded":"2024-03-30T15:03:18.262337+0000","last_fullsized":"2024-03-30T15:03:18.262337+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.100634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.100634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":761,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260532+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:03:18.260532+0000","last_peered":"2024-03-30T15:03:18.260532+0000","last_clean":"2024-03-30T15:03:18.260532+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:03:18.260532+0000","last_undegraded":"2024-03-30T15:03:18.260532+0000","last_fullsized":"2024-03-30T15:03:18.260532+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.497042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259060+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:03:18.259060+0000","last_peered":"2024-03-30T15:03:18.259060+0000","last_clean":"2024-03-30T15:03:18.259060+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:03:18.259060+0000","last_undegraded":"2024-03-30T15:03:18.259060+0000","last_fullsized":"2024-03-30T15:03:18.259060+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:47.630837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":864,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.272491+0000","last_change":"2024-03-30T15:01:16.395457+0000","last_active":"2024-03-30T15:03:18.272491+0000","last_peered":"2024-03-30T15:03:18.272491+0000","last_clean":"2024-03-30T15:03:18.272491+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:03:18.272491+0000","last_undegraded":"2024-03-30T15:03:18.272491+0000","last_fullsized":"2024-03-30T15:03:18.272491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:16.395364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.395364+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":730,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261321+0000","last_change":"2024-03-30T14:59:13.422572+0000","last_active":"2024-03-30T15:03:18.261321+0000","last_peered":"2024-03-30T15:03:18.261321+0000","last_clean":"2024-03-30T15:03:18.261321+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:03:18.261321+0000","last_undegraded":"2024-03-30T15:03:18.261321+0000","last_fullsized":"2024-03-30T15:03:18.261321+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:13.422453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:13.422453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265676+0000","last_change":"2024-03-30T15:02:07.241570+0000","last_active":"2024-03-30T15:03:18.265676+0000","last_peered":"2024-03-30T15:03:18.265676+0000","last_clean":"2024-03-30T15:03:18.265676+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:03:18.265676+0000","last_undegraded":"2024-03-30T15:03:18.265676+0000","last_fullsized":"2024-03-30T15:03:18.265676+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.241467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.241467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:22.628639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":774,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260062+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:03:18.260062+0000","last_peered":"2024-03-30T15:03:18.260062+0000","last_clean":"2024-03-30T15:03:18.260062+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:03:18.260062+0000","last_undegraded":"2024-03-30T15:03:18.260062+0000","last_fullsized":"2024-03-30T15:03:18.260062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":758,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263446+0000","last_change":"2024-03-30T15:02:02.104334+0000","last_active":"2024-03-30T15:03:18.263446+0000","last_peered":"2024-03-30T15:03:18.263446+0000","last_clean":"2024-03-30T15:03:18.263446+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:03:18.263446+0000","last_undegraded":"2024-03-30T15:03:18.263446+0000","last_fullsized":"2024-03-30T15:03:18.263446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.104245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:27.404456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273514+0000","last_change":"2024-03-30T15:01:48.369544+0000","last_active":"2024-03-30T15:03:18.273514+0000","last_peered":"2024-03-30T15:03:18.273514+0000","last_clean":"2024-03-30T15:03:18.273514+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:03:18.273514+0000","last_undegraded":"2024-03-30T15:03:18.273514+0000","last_fullsized":"2024-03-30T15:03:18.273514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:48.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":747,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273672+0000","last_change":"2024-03-30T15:02:04.594739+0000","last_active":"2024-03-30T15:03:18.273672+0000","last_peered":"2024-03-30T15:03:18.273672+0000","last_clean":"2024-03-30T15:03:18.273672+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:03:18.273672+0000","last_undegraded":"2024-03-30T15:03:18.273672+0000","last_fullsized":"2024-03-30T15:03:18.273672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:04.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:04.594651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.272800+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:03:18.272800+0000","last_peered":"2024-03-30T15:03:18.272800+0000","last_clean":"2024-03-30T15:03:18.272800+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:03:18.272800+0000","last_undegraded":"2024-03-30T15:03:18.272800+0000","last_fullsized":"2024-03-30T15:03:18.272800+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:45.801330+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":770,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261230+0000","last_change":"2024-03-30T15:02:03.449650+0000","last_active":"2024-03-30T15:03:18.261230+0000","last_peered":"2024-03-30T15:03:18.261230+0000","last_clean":"2024-03-30T15:03:18.261230+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:03:18.261230+0000","last_undegraded":"2024-03-30T15:03:18.261230+0000","last_fullsized":"2024-03-30T15:03:18.261230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:03.449549+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.449549+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:23.456461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259001+0000","last_change":"2024-03-30T15:01:50.503242+0000","last_active":"2024-03-30T15:03:18.259001+0000","last_peered":"2024-03-30T15:03:18.259001+0000","last_clean":"2024-03-30T15:03:18.259001+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:03:18.259001+0000","last_undegraded":"2024-03-30T15:03:18.259001+0000","last_fullsized":"2024-03-30T15:03:18.259001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.503156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.503156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:18.294000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270914+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:03:18.270914+0000","last_peered":"2024-03-30T15:03:18.270914+0000","last_clean":"2024-03-30T15:03:18.270914+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:03:18.270914+0000","last_undegraded":"2024-03-30T15:03:18.270914+0000","last_fullsized":"2024-03-30T15:03:18.270914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:48.181332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262566+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:03:18.262566+0000","last_peered":"2024-03-30T15:03:18.262566+0000","last_clean":"2024-03-30T15:03:18.262566+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:03:18.262566+0000","last_undegraded":"2024-03-30T15:03:18.262566+0000","last_fullsized":"2024-03-30T15:03:18.262566+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.693552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270865+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:03:18.270865+0000","last_peered":"2024-03-30T15:03:18.270865+0000","last_clean":"2024-03-30T15:03:18.270865+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:03:18.270865+0000","last_undegraded":"2024-03-30T15:03:18.270865+0000","last_fullsized":"2024-03-30T15:03:18.270865+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:57.924644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":779,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271024+0000","last_change":"2024-03-30T15:01:46.424704+0000","last_active":"2024-03-30T15:03:18.271024+0000","last_peered":"2024-03-30T15:03:18.271024+0000","last_clean":"2024-03-30T15:03:18.271024+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:03:18.271024+0000","last_undegraded":"2024-03-30T15:03:18.271024+0000","last_fullsized":"2024-03-30T15:03:18.271024+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T15:01:46.424546+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.424546+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269964+0000","last_change":"2024-03-30T15:01:50.443926+0000","last_active":"2024-03-30T15:03:18.269964+0000","last_peered":"2024-03-30T15:03:18.269964+0000","last_clean":"2024-03-30T15:03:18.269964+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:03:18.269964+0000","last_undegraded":"2024-03-30T15:03:18.269964+0000","last_fullsized":"2024-03-30T15:03:18.269964+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.443839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.443839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":847,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261266+0000","last_change":"2024-03-30T14:58:59.314094+0000","last_active":"2024-03-30T15:03:18.261266+0000","last_peered":"2024-03-30T15:03:18.261266+0000","last_clean":"2024-03-30T15:03:18.261266+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:03:18.261266+0000","last_undegraded":"2024-03-30T15:03:18.261266+0000","last_fullsized":"2024-03-30T15:03:18.261266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:58:59.314003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:58:59.314003+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":729,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270795+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:03:18.270795+0000","last_peered":"2024-03-30T15:03:18.270795+0000","last_clean":"2024-03-30T15:03:18.270795+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:03:18.270795+0000","last_undegraded":"2024-03-30T15:03:18.270795+0000","last_fullsized":"2024-03-30T15:03:18.270795+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266079+0000","last_change":"2024-03-30T15:02:30.902281+0000","last_active":"2024-03-30T15:03:18.266079+0000","last_peered":"2024-03-30T15:03:18.266079+0000","last_clean":"2024-03-30T15:03:18.266079+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:03:18.266079+0000","last_undegraded":"2024-03-30T15:03:18.266079+0000","last_fullsized":"2024-03-30T15:03:18.266079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.902199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.902199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:44.826826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":743,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258873+0000","last_change":"2024-03-30T15:02:39.473396+0000","last_active":"2024-03-30T15:03:18.258873+0000","last_peered":"2024-03-30T15:03:18.258873+0000","last_clean":"2024-03-30T15:03:18.258873+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:03:18.258873+0000","last_undegraded":"2024-03-30T15:03:18.258873+0000","last_fullsized":"2024-03-30T15:03:18.258873+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:39.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.473279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:45.969335+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266197+0000","last_change":"2024-03-30T15:02:34.958123+0000","last_active":"2024-03-30T15:03:18.266197+0000","last_peered":"2024-03-30T15:03:18.266197+0000","last_clean":"2024-03-30T15:03:18.266197+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:03:18.266197+0000","last_undegraded":"2024-03-30T15:03:18.266197+0000","last_fullsized":"2024-03-30T15:03:18.266197+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T15:02:34.958001+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.958001+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:46.407868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":725,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268153+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:03:18.268153+0000","last_peered":"2024-03-30T15:03:18.268153+0000","last_clean":"2024-03-30T15:03:18.268153+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:03:18.268153+0000","last_undegraded":"2024-03-30T15:03:18.268153+0000","last_fullsized":"2024-03-30T15:03:18.268153+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.286838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":731,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259720+0000","last_change":"2024-03-30T14:59:02.261089+0000","last_active":"2024-03-30T15:03:18.259720+0000","last_peered":"2024-03-30T15:03:18.259720+0000","last_clean":"2024-03-30T15:03:18.259720+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:03:18.259720+0000","last_undegraded":"2024-03-30T15:03:18.259720+0000","last_fullsized":"2024-03-30T15:03:18.259720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:02.260964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:02.260964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":660,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264811+0000","last_change":"2024-03-30T15:02:06.597375+0000","last_active":"2024-03-30T15:03:18.264811+0000","last_peered":"2024-03-30T15:03:18.264811+0000","last_clean":"2024-03-30T15:03:18.264811+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:03:18.264811+0000","last_undegraded":"2024-03-30T15:03:18.264811+0000","last_fullsized":"2024-03-30T15:03:18.264811+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:06.597250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:06.597250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269467+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:03:18.269467+0000","last_peered":"2024-03-30T15:03:18.269467+0000","last_clean":"2024-03-30T15:03:18.269467+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:03:18.269467+0000","last_undegraded":"2024-03-30T15:03:18.269467+0000","last_fullsized":"2024-03-30T15:03:18.269467+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.527010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":768,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259087+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:03:18.259087+0000","last_peered":"2024-03-30T15:03:18.259087+0000","last_clean":"2024-03-30T15:03:18.259087+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:03:18.259087+0000","last_undegraded":"2024-03-30T15:03:18.259087+0000","last_fullsized":"2024-03-30T15:03:18.259087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":756,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261528+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:03:18.261528+0000","last_peered":"2024-03-30T15:03:18.261528+0000","last_clean":"2024-03-30T15:03:18.261528+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:03:18.261528+0000","last_undegraded":"2024-03-30T15:03:18.261528+0000","last_fullsized":"2024-03-30T15:03:18.261528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.403595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":739,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267111+0000","last_change":"2024-03-30T14:59:56.291563+0000","last_active":"2024-03-30T15:03:18.267111+0000","last_peered":"2024-03-30T15:03:18.267111+0000","last_clean":"2024-03-30T15:03:18.267111+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:03:18.267111+0000","last_undegraded":"2024-03-30T15:03:18.267111+0000","last_fullsized":"2024-03-30T15:03:18.267111+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:56.291453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:56.291453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":660,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265620+0000","last_change":"2024-03-30T15:02:05.626016+0000","last_active":"2024-03-30T15:03:18.265620+0000","last_peered":"2024-03-30T15:03:18.265620+0000","last_clean":"2024-03-30T15:03:18.265620+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:03:18.265620+0000","last_undegraded":"2024-03-30T15:03:18.265620+0000","last_fullsized":"2024-03-30T15:03:18.265620+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:05.625940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.625940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":867,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258962+0000","last_change":"2024-03-30T15:01:48.471009+0000","last_active":"2024-03-30T15:03:18.258962+0000","last_peered":"2024-03-30T15:03:18.258962+0000","last_clean":"2024-03-30T15:03:18.258962+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:03:18.258962+0000","last_undegraded":"2024-03-30T15:03:18.258962+0000","last_fullsized":"2024-03-30T15:03:18.258962+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:48.470884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.470884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":776,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269192+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:03:18.269192+0000","last_peered":"2024-03-30T15:03:18.269192+0000","last_clean":"2024-03-30T15:03:18.269192+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:03:18.269192+0000","last_undegraded":"2024-03-30T15:03:18.269192+0000","last_fullsized":"2024-03-30T15:03:18.269192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271125+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:03:18.271125+0000","last_peered":"2024-03-30T15:03:18.271125+0000","last_clean":"2024-03-30T15:03:18.271125+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:03:18.271125+0000","last_undegraded":"2024-03-30T15:03:18.271125+0000","last_fullsized":"2024-03-30T15:03:18.271125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.257998+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:03:18.257998+0000","last_peered":"2024-03-30T15:03:18.257998+0000","last_clean":"2024-03-30T15:03:18.257998+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:03:18.257998+0000","last_undegraded":"2024-03-30T15:03:18.257998+0000","last_fullsized":"2024-03-30T15:03:18.257998+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:42.688293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267450+0000","last_change":"2024-03-30T15:01:52.535873+0000","last_active":"2024-03-30T15:03:18.267450+0000","last_peered":"2024-03-30T15:03:18.267450+0000","last_clean":"2024-03-30T15:03:18.267450+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:03:18.267450+0000","last_undegraded":"2024-03-30T15:03:18.267450+0000","last_fullsized":"2024-03-30T15:03:18.267450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.535788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.535788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263814+0000","last_change":"2024-03-30T15:01:53.060917+0000","last_active":"2024-03-30T15:03:18.263814+0000","last_peered":"2024-03-30T15:03:18.263814+0000","last_clean":"2024-03-30T15:03:18.263814+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:03:18.263814+0000","last_undegraded":"2024-03-30T15:03:18.263814+0000","last_fullsized":"2024-03-30T15:03:18.263814+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.060820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.060820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:19.872352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":786,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259752+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:03:18.259752+0000","last_peered":"2024-03-30T15:03:18.259752+0000","last_clean":"2024-03-30T15:03:18.259752+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:03:18.259752+0000","last_undegraded":"2024-03-30T15:03:18.259752+0000","last_fullsized":"2024-03-30T15:03:18.259752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273229+0000","last_change":"2024-03-30T15:01:57.409471+0000","last_active":"2024-03-30T15:03:18.273229+0000","last_peered":"2024-03-30T15:03:18.273229+0000","last_clean":"2024-03-30T15:03:18.273229+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:03:18.273229+0000","last_undegraded":"2024-03-30T15:03:18.273229+0000","last_fullsized":"2024-03-30T15:03:18.273229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.409374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.409374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":748,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264720+0000","last_change":"2024-03-30T15:01:54.450014+0000","last_active":"2024-03-30T15:03:18.264720+0000","last_peered":"2024-03-30T15:03:18.264720+0000","last_clean":"2024-03-30T15:03:18.264720+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:03:18.264720+0000","last_undegraded":"2024-03-30T15:03:18.264720+0000","last_fullsized":"2024-03-30T15:03:18.264720+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.449931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.449931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261380+0000","last_change":"2024-03-30T15:01:56.570174+0000","last_active":"2024-03-30T15:03:18.261380+0000","last_peered":"2024-03-30T15:03:18.261380+0000","last_clean":"2024-03-30T15:03:18.261380+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:03:18.261380+0000","last_undegraded":"2024-03-30T15:03:18.261380+0000","last_fullsized":"2024-03-30T15:03:18.261380+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:56.570063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:56.570063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271108+0000","last_change":"2024-03-30T14:59:15.207502+0000","last_active":"2024-03-30T15:03:18.271108+0000","last_peered":"2024-03-30T15:03:18.271108+0000","last_clean":"2024-03-30T15:03:18.271108+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:03:18.271108+0000","last_undegraded":"2024-03-30T15:03:18.271108+0000","last_fullsized":"2024-03-30T15:03:18.271108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:15.207413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.207413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":768,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261330+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:03:18.261330+0000","last_peered":"2024-03-30T15:03:18.261330+0000","last_clean":"2024-03-30T15:03:18.261330+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:03:18.261330+0000","last_undegraded":"2024-03-30T15:03:18.261330+0000","last_fullsized":"2024-03-30T15:03:18.261330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":1276,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265979+0000","last_change":"2024-03-30T15:02:03.014253+0000","last_active":"2024-03-30T15:03:18.265979+0000","last_peered":"2024-03-30T15:03:18.265979+0000","last_clean":"2024-03-30T15:03:18.265979+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:03:18.265979+0000","last_undegraded":"2024-03-30T15:03:18.265979+0000","last_fullsized":"2024-03-30T15:03:18.265979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T15:02:03.014132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.014132+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:22.813895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":513,"num_read_kb":508,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":746,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258851+0000","last_change":"2024-03-30T14:59:05.612363+0000","last_active":"2024-03-30T15:03:18.258851+0000","last_peered":"2024-03-30T15:03:18.258851+0000","last_clean":"2024-03-30T15:03:18.258851+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:03:18.258851+0000","last_undegraded":"2024-03-30T15:03:18.258851+0000","last_fullsized":"2024-03-30T15:03:18.258851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.612262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.612262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":740,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273004+0000","last_change":"2024-03-30T15:01:12.423769+0000","last_active":"2024-03-30T15:03:18.273004+0000","last_peered":"2024-03-30T15:03:18.273004+0000","last_clean":"2024-03-30T15:03:18.273004+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:03:18.273004+0000","last_undegraded":"2024-03-30T15:03:18.273004+0000","last_fullsized":"2024-03-30T15:03:18.273004+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.423680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.423680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267728+0000","last_change":"2024-03-30T15:01:18.064257+0000","last_active":"2024-03-30T15:03:18.267728+0000","last_peered":"2024-03-30T15:03:18.267728+0000","last_clean":"2024-03-30T15:03:18.267728+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:03:18.267728+0000","last_undegraded":"2024-03-30T15:03:18.267728+0000","last_fullsized":"2024-03-30T15:03:18.267728+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:18.064173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.064173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":743,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266865+0000","last_change":"2024-03-30T15:02:33.948293+0000","last_active":"2024-03-30T15:03:18.266865+0000","last_peered":"2024-03-30T15:03:18.266865+0000","last_clean":"2024-03-30T15:03:18.266865+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:03:18.266865+0000","last_undegraded":"2024-03-30T15:03:18.266865+0000","last_fullsized":"2024-03-30T15:03:18.266865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:33.948156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.948156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:40.346164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":889,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265869+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:03:18.265869+0000","last_peered":"2024-03-30T15:03:18.265869+0000","last_clean":"2024-03-30T15:03:18.265869+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:03:18.265869+0000","last_undegraded":"2024-03-30T15:03:18.265869+0000","last_fullsized":"2024-03-30T15:03:18.265869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.146207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":644,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270330+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:03:18.270330+0000","last_peered":"2024-03-30T15:03:18.270330+0000","last_clean":"2024-03-30T15:03:18.270330+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:03:18.270330+0000","last_undegraded":"2024-03-30T15:03:18.270330+0000","last_fullsized":"2024-03-30T15:03:18.270330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1011,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266043+0000","last_change":"2024-03-30T15:02:39.890565+0000","last_active":"2024-03-30T15:03:18.266043+0000","last_peered":"2024-03-30T15:03:18.266043+0000","last_clean":"2024-03-30T15:03:18.266043+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:03:18.266043+0000","last_undegraded":"2024-03-30T15:03:18.266043+0000","last_fullsized":"2024-03-30T15:03:18.266043+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:02:39.890462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.890462+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:51.359472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":729,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270593+0000","last_change":"2024-03-30T14:59:05.166302+0000","last_active":"2024-03-30T15:03:18.270593+0000","last_peered":"2024-03-30T15:03:18.270593+0000","last_clean":"2024-03-30T15:03:18.270593+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:03:18.270593+0000","last_undegraded":"2024-03-30T15:03:18.270593+0000","last_fullsized":"2024-03-30T15:03:18.270593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:05.166200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:05.166200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264576+0000","last_change":"2024-03-30T15:02:06.204746+0000","last_active":"2024-03-30T15:03:18.264576+0000","last_peered":"2024-03-30T15:03:18.264576+0000","last_clean":"2024-03-30T15:03:18.264576+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:03:18.264576+0000","last_undegraded":"2024-03-30T15:03:18.264576+0000","last_fullsized":"2024-03-30T15:03:18.264576+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:06.204628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:06.204628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":752,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.272912+0000","last_change":"2024-03-30T15:01:58.576718+0000","last_active":"2024-03-30T15:03:18.272912+0000","last_peered":"2024-03-30T15:03:18.272912+0000","last_clean":"2024-03-30T15:03:18.272912+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:03:18.272912+0000","last_undegraded":"2024-03-30T15:03:18.272912+0000","last_fullsized":"2024-03-30T15:03:18.272912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.576585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.576585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":788,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270796+0000","last_change":"2024-03-30T15:01:11.446061+0000","last_active":"2024-03-30T15:03:18.270796+0000","last_peered":"2024-03-30T15:03:18.270796+0000","last_clean":"2024-03-30T15:03:18.270796+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:03:18.270796+0000","last_undegraded":"2024-03-30T15:03:18.270796+0000","last_fullsized":"2024-03-30T15:03:18.270796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:01:11.445771+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.445771+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264979+0000","last_change":"2024-03-30T15:01:53.529768+0000","last_active":"2024-03-30T15:03:18.264979+0000","last_peered":"2024-03-30T15:03:18.264979+0000","last_clean":"2024-03-30T15:03:18.264979+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:03:18.264979+0000","last_undegraded":"2024-03-30T15:03:18.264979+0000","last_fullsized":"2024-03-30T15:03:18.264979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.529037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.529037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:21.071241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269629+0000","last_change":"2024-03-30T15:01:59.392426+0000","last_active":"2024-03-30T15:03:18.269629+0000","last_peered":"2024-03-30T15:03:18.269629+0000","last_clean":"2024-03-30T15:03:18.269629+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:03:18.269629+0000","last_undegraded":"2024-03-30T15:03:18.269629+0000","last_fullsized":"2024-03-30T15:03:18.269629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.392341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":739,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268014+0000","last_change":"2024-03-30T14:59:55.270674+0000","last_active":"2024-03-30T15:03:18.268014+0000","last_peered":"2024-03-30T15:03:18.268014+0000","last_clean":"2024-03-30T15:03:18.268014+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:03:18.268014+0000","last_undegraded":"2024-03-30T15:03:18.268014+0000","last_fullsized":"2024-03-30T15:03:18.268014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.270595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.270595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259542+0000","last_change":"2024-03-30T15:01:59.537303+0000","last_active":"2024-03-30T15:03:18.259542+0000","last_peered":"2024-03-30T15:03:18.259542+0000","last_clean":"2024-03-30T15:03:18.259542+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:03:18.259542+0000","last_undegraded":"2024-03-30T15:03:18.259542+0000","last_fullsized":"2024-03-30T15:03:18.259542+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.537225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.537225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:22.819340+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":746,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259839+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:03:18.259839+0000","last_peered":"2024-03-30T15:03:18.259839+0000","last_clean":"2024-03-30T15:03:18.259839+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:03:18.259839+0000","last_undegraded":"2024-03-30T15:03:18.259839+0000","last_fullsized":"2024-03-30T15:03:18.259839+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":853,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268730+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:03:18.268730+0000","last_peered":"2024-03-30T15:03:18.268730+0000","last_clean":"2024-03-30T15:03:18.268730+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:03:18.268730+0000","last_undegraded":"2024-03-30T15:03:18.268730+0000","last_fullsized":"2024-03-30T15:03:18.268730+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.234178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259419+0000","last_change":"2024-03-30T15:02:00.474885+0000","last_active":"2024-03-30T15:03:18.259419+0000","last_peered":"2024-03-30T15:03:18.259419+0000","last_clean":"2024-03-30T15:03:18.259419+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:03:18.259419+0000","last_undegraded":"2024-03-30T15:03:18.259419+0000","last_fullsized":"2024-03-30T15:03:18.259419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.474742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.474742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270377+0000","last_change":"2024-03-30T14:58:36.096934+0000","last_active":"2024-03-30T15:03:18.270377+0000","last_peered":"2024-03-30T15:03:18.270377+0000","last_clean":"2024-03-30T15:03:18.270377+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:03:18.270377+0000","last_undegraded":"2024-03-30T15:03:18.270377+0000","last_fullsized":"2024-03-30T15:03:18.270377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T14:58:36.096812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T14:58:36.096812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261080+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:03:18.261080+0000","last_peered":"2024-03-30T15:03:18.261080+0000","last_clean":"2024-03-30T15:03:18.261080+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:03:18.261080+0000","last_undegraded":"2024-03-30T15:03:18.261080+0000","last_fullsized":"2024-03-30T15:03:18.261080+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:19.574807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259259+0000","last_change":"2024-03-30T15:01:58.563335+0000","last_active":"2024-03-30T15:03:18.259259+0000","last_peered":"2024-03-30T15:03:18.259259+0000","last_clean":"2024-03-30T15:03:18.259259+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:03:18.259259+0000","last_undegraded":"2024-03-30T15:03:18.259259+0000","last_fullsized":"2024-03-30T15:03:18.259259+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":765,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259155+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:03:18.259155+0000","last_peered":"2024-03-30T15:03:18.259155+0000","last_clean":"2024-03-30T15:03:18.259155+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:03:18.259155+0000","last_undegraded":"2024-03-30T15:03:18.259155+0000","last_fullsized":"2024-03-30T15:03:18.259155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269203+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:03:18.269203+0000","last_peered":"2024-03-30T15:03:18.269203+0000","last_clean":"2024-03-30T15:03:18.269203+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:03:18.269203+0000","last_undegraded":"2024-03-30T15:03:18.269203+0000","last_fullsized":"2024-03-30T15:03:18.269203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265898+0000","last_change":"2024-03-30T15:01:12.479941+0000","last_active":"2024-03-30T15:03:18.265898+0000","last_peered":"2024-03-30T15:03:18.265898+0000","last_clean":"2024-03-30T15:03:18.265898+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:03:18.265898+0000","last_undegraded":"2024-03-30T15:03:18.265898+0000","last_fullsized":"2024-03-30T15:03:18.265898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.479863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.479863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259583+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:03:18.259583+0000","last_peered":"2024-03-30T15:03:18.259583+0000","last_clean":"2024-03-30T15:03:18.259583+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:03:18.259583+0000","last_undegraded":"2024-03-30T15:03:18.259583+0000","last_fullsized":"2024-03-30T15:03:18.259583+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:14.649484+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260521+0000","last_change":"2024-03-30T15:01:53.490679+0000","last_active":"2024-03-30T15:03:18.260521+0000","last_peered":"2024-03-30T15:03:18.260521+0000","last_clean":"2024-03-30T15:03:18.260521+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:03:18.260521+0000","last_undegraded":"2024-03-30T15:03:18.260521+0000","last_fullsized":"2024-03-30T15:03:18.260521+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.490591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.490591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":818,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263718+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:03:18.263718+0000","last_peered":"2024-03-30T15:03:18.263718+0000","last_clean":"2024-03-30T15:03:18.263718+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:03:18.263718+0000","last_undegraded":"2024-03-30T15:03:18.263718+0000","last_fullsized":"2024-03-30T15:03:18.263718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:41.052823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268477+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:03:18.268477+0000","last_peered":"2024-03-30T15:03:18.268477+0000","last_clean":"2024-03-30T15:03:18.268477+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:03:18.268477+0000","last_undegraded":"2024-03-30T15:03:18.268477+0000","last_fullsized":"2024-03-30T15:03:18.268477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.609405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267323+0000","last_change":"2024-03-30T15:01:15.350072+0000","last_active":"2024-03-30T15:03:18.267323+0000","last_peered":"2024-03-30T15:03:18.267323+0000","last_clean":"2024-03-30T15:03:18.267323+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:03:18.267323+0000","last_undegraded":"2024-03-30T15:03:18.267323+0000","last_fullsized":"2024-03-30T15:03:18.267323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:15.349949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:15.349949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265930+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:03:18.265930+0000","last_peered":"2024-03-30T15:03:18.265930+0000","last_clean":"2024-03-30T15:03:18.265930+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:03:18.265930+0000","last_undegraded":"2024-03-30T15:03:18.265930+0000","last_fullsized":"2024-03-30T15:03:18.265930+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:55.110433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":676,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270092+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:03:18.270092+0000","last_peered":"2024-03-30T15:03:18.270092+0000","last_clean":"2024-03-30T15:03:18.270092+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:03:18.270092+0000","last_undegraded":"2024-03-30T15:03:18.270092+0000","last_fullsized":"2024-03-30T15:03:18.270092+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.286336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265803+0000","last_change":"2024-03-30T15:01:55.535170+0000","last_active":"2024-03-30T15:03:18.265803+0000","last_peered":"2024-03-30T15:03:18.265803+0000","last_clean":"2024-03-30T15:03:18.265803+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:03:18.265803+0000","last_undegraded":"2024-03-30T15:03:18.265803+0000","last_fullsized":"2024-03-30T15:03:18.265803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:55.535057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:01:55.535057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":769,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259147+0000","last_change":"2024-03-30T14:58:25.223144+0000","last_active":"2024-03-30T15:03:18.259147+0000","last_peered":"2024-03-30T15:03:18.259147+0000","last_clean":"2024-03-30T15:03:18.259147+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:03:18.259147+0000","last_undegraded":"2024-03-30T15:03:18.259147+0000","last_fullsized":"2024-03-30T15:03:18.259147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:58:25.223044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.223044+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267162+0000","last_change":"2024-03-30T15:02:01.090736+0000","last_active":"2024-03-30T15:03:18.267162+0000","last_peered":"2024-03-30T15:03:18.267162+0000","last_clean":"2024-03-30T15:03:18.267162+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:03:18.267162+0000","last_undegraded":"2024-03-30T15:03:18.267162+0000","last_fullsized":"2024-03-30T15:03:18.267162+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:01.090640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.090640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:21.533214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267257+0000","last_change":"2024-03-30T15:00:02.270908+0000","last_active":"2024-03-30T15:03:18.267257+0000","last_peered":"2024-03-30T15:03:18.267257+0000","last_clean":"2024-03-30T15:03:18.267257+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:03:18.267257+0000","last_undegraded":"2024-03-30T15:03:18.267257+0000","last_fullsized":"2024-03-30T15:03:18.267257+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.270821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258089+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:03:18.258089+0000","last_peered":"2024-03-30T15:03:18.258089+0000","last_clean":"2024-03-30T15:03:18.258089+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:03:18.258089+0000","last_undegraded":"2024-03-30T15:03:18.258089+0000","last_fullsized":"2024-03-30T15:03:18.258089+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":855,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265082+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:03:18.265082+0000","last_peered":"2024-03-30T15:03:18.265082+0000","last_clean":"2024-03-30T15:03:18.265082+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:03:18.265082+0000","last_undegraded":"2024-03-30T15:03:18.265082+0000","last_fullsized":"2024-03-30T15:03:18.265082+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.736984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":763,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267027+0000","last_change":"2024-03-30T14:58:35.054537+0000","last_active":"2024-03-30T15:03:18.267027+0000","last_peered":"2024-03-30T15:03:18.267027+0000","last_clean":"2024-03-30T15:03:18.267027+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:03:18.267027+0000","last_undegraded":"2024-03-30T15:03:18.267027+0000","last_fullsized":"2024-03-30T15:03:18.267027+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T14:58:35.054439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:35.054439+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268547+0000","last_change":"2024-03-30T15:02:05.004181+0000","last_active":"2024-03-30T15:03:18.268547+0000","last_peered":"2024-03-30T15:03:18.268547+0000","last_clean":"2024-03-30T15:03:18.268547+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:03:18.268547+0000","last_undegraded":"2024-03-30T15:03:18.268547+0000","last_fullsized":"2024-03-30T15:03:18.268547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:02:05.004073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.004073+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":737,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270626+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:03:18.270626+0000","last_peered":"2024-03-30T15:03:18.270626+0000","last_clean":"2024-03-30T15:03:18.270626+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:03:18.270626+0000","last_undegraded":"2024-03-30T15:03:18.270626+0000","last_fullsized":"2024-03-30T15:03:18.270626+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:55.717440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264460+0000","last_change":"2024-03-30T15:02:32.380969+0000","last_active":"2024-03-30T15:03:18.264460+0000","last_peered":"2024-03-30T15:03:18.264460+0000","last_clean":"2024-03-30T15:03:18.264460+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:03:18.264460+0000","last_undegraded":"2024-03-30T15:03:18.264460+0000","last_fullsized":"2024-03-30T15:03:18.264460+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.380822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.380822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:39.592472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":319,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-03-30T15:03:17.264185+0000","last_change":"2024-03-30T15:02:49.221941+0000","last_active":"2024-03-30T15:03:17.264185+0000","last_peered":"2024-03-30T15:03:17.264185+0000","last_clean":"2024-03-30T15:03:17.264185+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:03:17.264185+0000","last_undegraded":"2024-03-30T15:03:17.264185+0000","last_fullsized":"2024-03-30T15:03:17.264185+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T15:02:49.221822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.221822+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:52.007992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":765,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262075+0000","last_change":"2024-03-30T15:02:45.427403+0000","last_active":"2024-03-30T15:03:18.262075+0000","last_peered":"2024-03-30T15:03:18.262075+0000","last_clean":"2024-03-30T15:03:18.262075+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:03:18.262075+0000","last_undegraded":"2024-03-30T15:03:18.262075+0000","last_fullsized":"2024-03-30T15:03:18.262075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:02:45.427234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:02.575772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269792+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:03:18.269792+0000","last_peered":"2024-03-30T15:03:18.269792+0000","last_clean":"2024-03-30T15:03:18.269792+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:03:18.269792+0000","last_undegraded":"2024-03-30T15:03:18.269792+0000","last_fullsized":"2024-03-30T15:03:18.269792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.325342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":730,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267459+0000","last_change":"2024-03-30T14:59:51.262210+0000","last_active":"2024-03-30T15:03:18.267459+0000","last_peered":"2024-03-30T15:03:18.267459+0000","last_clean":"2024-03-30T15:03:18.267459+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:03:18.267459+0000","last_undegraded":"2024-03-30T15:03:18.267459+0000","last_fullsized":"2024-03-30T15:03:18.267459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:51.262127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.262127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261026+0000","last_change":"2024-03-30T15:01:11.580352+0000","last_active":"2024-03-30T15:03:18.261026+0000","last_peered":"2024-03-30T15:03:18.261026+0000","last_clean":"2024-03-30T15:03:18.261026+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:03:18.261026+0000","last_undegraded":"2024-03-30T15:03:18.261026+0000","last_fullsized":"2024-03-30T15:03:18.261026+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:11.580276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.580276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":757,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261687+0000","last_change":"2024-03-30T14:59:15.487022+0000","last_active":"2024-03-30T15:03:18.261687+0000","last_peered":"2024-03-30T15:03:18.261687+0000","last_clean":"2024-03-30T15:03:18.261687+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:03:18.261687+0000","last_undegraded":"2024-03-30T15:03:18.261687+0000","last_fullsized":"2024-03-30T15:03:18.261687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:15.486892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:15.486892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264745+0000","last_change":"2024-03-30T15:00:37.049789+0000","last_active":"2024-03-30T15:03:18.264745+0000","last_peered":"2024-03-30T15:03:18.264745+0000","last_clean":"2024-03-30T15:03:18.264745+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:03:18.264745+0000","last_undegraded":"2024-03-30T15:03:18.264745+0000","last_fullsized":"2024-03-30T15:03:18.264745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:37.049685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:00:37.049685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":652,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271617+0000","last_change":"2024-03-30T15:01:19.390146+0000","last_active":"2024-03-30T15:03:18.271617+0000","last_peered":"2024-03-30T15:03:18.271617+0000","last_clean":"2024-03-30T15:03:18.271617+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:03:18.271617+0000","last_undegraded":"2024-03-30T15:03:18.271617+0000","last_fullsized":"2024-03-30T15:03:18.271617+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.390029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.390029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":747,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271813+0000","last_change":"2024-03-30T15:01:19.394940+0000","last_active":"2024-03-30T15:03:18.271813+0000","last_peered":"2024-03-30T15:03:18.271813+0000","last_clean":"2024-03-30T15:03:18.271813+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:03:18.271813+0000","last_undegraded":"2024-03-30T15:03:18.271813+0000","last_fullsized":"2024-03-30T15:03:18.271813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.394860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.394860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259745+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:03:18.259745+0000","last_peered":"2024-03-30T15:03:18.259745+0000","last_clean":"2024-03-30T15:03:18.259745+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:03:18.259745+0000","last_undegraded":"2024-03-30T15:03:18.259745+0000","last_fullsized":"2024-03-30T15:03:18.259745+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271341+0000","last_change":"2024-03-30T14:58:31.097064+0000","last_active":"2024-03-30T15:03:18.271341+0000","last_peered":"2024-03-30T15:03:18.271341+0000","last_clean":"2024-03-30T15:03:18.271341+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:03:18.271341+0000","last_undegraded":"2024-03-30T15:03:18.271341+0000","last_fullsized":"2024-03-30T15:03:18.271341+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:58:31.096968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.096968+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258999+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:03:18.258999+0000","last_peered":"2024-03-30T15:03:18.258999+0000","last_clean":"2024-03-30T15:03:18.258999+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:03:18.258999+0000","last_undegraded":"2024-03-30T15:03:18.258999+0000","last_fullsized":"2024-03-30T15:03:18.258999+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":733,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266765+0000","last_change":"2024-03-30T14:59:53.269594+0000","last_active":"2024-03-30T15:03:18.266765+0000","last_peered":"2024-03-30T15:03:18.266765+0000","last_clean":"2024-03-30T15:03:18.266765+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:03:18.266765+0000","last_undegraded":"2024-03-30T15:03:18.266765+0000","last_fullsized":"2024-03-30T15:03:18.266765+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T14:59:53.269452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.269452+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":730,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264281+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:03:18.264281+0000","last_peered":"2024-03-30T15:03:18.264281+0000","last_clean":"2024-03-30T15:03:18.264281+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:03:18.264281+0000","last_undegraded":"2024-03-30T15:03:18.264281+0000","last_fullsized":"2024-03-30T15:03:18.264281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258776+0000","last_change":"2024-03-30T15:01:52.471709+0000","last_active":"2024-03-30T15:03:18.258776+0000","last_peered":"2024-03-30T15:03:18.258776+0000","last_clean":"2024-03-30T15:03:18.258776+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:03:18.258776+0000","last_undegraded":"2024-03-30T15:03:18.258776+0000","last_fullsized":"2024-03-30T15:03:18.258776+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.471603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.471603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":686,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.273325+0000","last_change":"2024-03-30T15:01:18.430986+0000","last_active":"2024-03-30T15:03:18.273325+0000","last_peered":"2024-03-30T15:03:18.273325+0000","last_clean":"2024-03-30T15:03:18.273325+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:03:18.273325+0000","last_undegraded":"2024-03-30T15:03:18.273325+0000","last_fullsized":"2024-03-30T15:03:18.273325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:01:18.430887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.430887+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266883+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:03:18.266883+0000","last_peered":"2024-03-30T15:03:18.266883+0000","last_clean":"2024-03-30T15:03:18.266883+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:03:18.266883+0000","last_undegraded":"2024-03-30T15:03:18.266883+0000","last_fullsized":"2024-03-30T15:03:18.266883+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:37.842104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":849,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271023+0000","last_change":"2024-03-30T14:59:08.160662+0000","last_active":"2024-03-30T15:03:18.271023+0000","last_peered":"2024-03-30T15:03:18.271023+0000","last_clean":"2024-03-30T15:03:18.271023+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:03:18.271023+0000","last_undegraded":"2024-03-30T15:03:18.271023+0000","last_fullsized":"2024-03-30T15:03:18.271023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T14:59:08.160528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:08.160528+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":735,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261173+0000","last_change":"2024-03-30T14:59:16.481218+0000","last_active":"2024-03-30T15:03:18.261173+0000","last_peered":"2024-03-30T15:03:18.261173+0000","last_clean":"2024-03-30T15:03:18.261173+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:03:18.261173+0000","last_undegraded":"2024-03-30T15:03:18.261173+0000","last_fullsized":"2024-03-30T15:03:18.261173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T14:59:16.481128+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T14:59:16.481128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259891+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:03:18.259891+0000","last_peered":"2024-03-30T15:03:18.259891+0000","last_clean":"2024-03-30T15:03:18.259891+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:03:18.259891+0000","last_undegraded":"2024-03-30T15:03:18.259891+0000","last_fullsized":"2024-03-30T15:03:18.259891+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259922+0000","last_change":"2024-03-30T14:59:20.453682+0000","last_active":"2024-03-30T15:03:18.259922+0000","last_peered":"2024-03-30T15:03:18.259922+0000","last_clean":"2024-03-30T15:03:18.259922+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:03:18.259922+0000","last_undegraded":"2024-03-30T15:03:18.259922+0000","last_fullsized":"2024-03-30T15:03:18.259922+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.453594+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260211+0000","last_change":"2024-03-30T15:01:21.453162+0000","last_active":"2024-03-30T15:03:18.260211+0000","last_peered":"2024-03-30T15:03:18.260211+0000","last_clean":"2024-03-30T15:03:18.260211+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:03:18.260211+0000","last_undegraded":"2024-03-30T15:03:18.260211+0000","last_fullsized":"2024-03-30T15:03:18.260211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:21.453084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:21.453084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":751,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269098+0000","last_change":"2024-03-30T15:02:03.477727+0000","last_active":"2024-03-30T15:03:18.269098+0000","last_peered":"2024-03-30T15:03:18.269098+0000","last_clean":"2024-03-30T15:03:18.269098+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:03:18.269098+0000","last_undegraded":"2024-03-30T15:03:18.269098+0000","last_fullsized":"2024-03-30T15:03:18.269098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.477653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.477653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":646,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259322+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:03:18.259322+0000","last_peered":"2024-03-30T15:03:18.259322+0000","last_clean":"2024-03-30T15:03:18.259322+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:03:18.259322+0000","last_undegraded":"2024-03-30T15:03:18.259322+0000","last_fullsized":"2024-03-30T15:03:18.259322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259316+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:03:18.259316+0000","last_peered":"2024-03-30T15:03:18.259316+0000","last_clean":"2024-03-30T15:03:18.259316+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:03:18.259316+0000","last_undegraded":"2024-03-30T15:03:18.259316+0000","last_fullsized":"2024-03-30T15:03:18.259316+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":763,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269290+0000","last_change":"2024-03-30T15:02:00.404994+0000","last_active":"2024-03-30T15:03:18.269290+0000","last_peered":"2024-03-30T15:03:18.269290+0000","last_clean":"2024-03-30T15:03:18.269290+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:03:18.269290+0000","last_undegraded":"2024-03-30T15:03:18.269290+0000","last_fullsized":"2024-03-30T15:03:18.269290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:02:00.404907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.404907+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":765,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267559+0000","last_change":"2024-03-30T15:02:02.044673+0000","last_active":"2024-03-30T15:03:18.267559+0000","last_peered":"2024-03-30T15:03:18.267559+0000","last_clean":"2024-03-30T15:03:18.267559+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:03:18.267559+0000","last_undegraded":"2024-03-30T15:03:18.267559+0000","last_fullsized":"2024-03-30T15:03:18.267559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:02:02.044556+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.044556+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:26.077889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":646,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259346+0000","last_change":"2024-03-30T14:59:17.441377+0000","last_active":"2024-03-30T15:03:18.259346+0000","last_peered":"2024-03-30T15:03:18.259346+0000","last_clean":"2024-03-30T15:03:18.259346+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:03:18.259346+0000","last_undegraded":"2024-03-30T15:03:18.259346+0000","last_fullsized":"2024-03-30T15:03:18.259346+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:17.441289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:17.441289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.257998+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:03:18.257998+0000","last_peered":"2024-03-30T15:03:18.257998+0000","last_clean":"2024-03-30T15:03:18.257998+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:03:18.257998+0000","last_undegraded":"2024-03-30T15:03:18.257998+0000","last_fullsized":"2024-03-30T15:03:18.257998+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.024180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.257987+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:03:18.257987+0000","last_peered":"2024-03-30T15:03:18.257987+0000","last_clean":"2024-03-30T15:03:18.257987+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:03:18.257987+0000","last_undegraded":"2024-03-30T15:03:18.257987+0000","last_fullsized":"2024-03-30T15:03:18.257987+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":689,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264971+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:03:18.264971+0000","last_peered":"2024-03-30T15:03:18.264971+0000","last_clean":"2024-03-30T15:03:18.264971+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:03:18.264971+0000","last_undegraded":"2024-03-30T15:03:18.264971+0000","last_fullsized":"2024-03-30T15:03:18.264971+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:02.391503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269886+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:03:18.269886+0000","last_peered":"2024-03-30T15:03:18.269886+0000","last_clean":"2024-03-30T15:03:18.269886+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:03:18.269886+0000","last_undegraded":"2024-03-30T15:03:18.269886+0000","last_fullsized":"2024-03-30T15:03:18.269886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:52.403587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266178+0000","last_change":"2024-03-30T15:02:41.903716+0000","last_active":"2024-03-30T15:03:18.266178+0000","last_peered":"2024-03-30T15:03:18.266178+0000","last_clean":"2024-03-30T15:03:18.266178+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:03:18.266178+0000","last_undegraded":"2024-03-30T15:03:18.266178+0000","last_fullsized":"2024-03-30T15:03:18.266178+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.903609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.903609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:44.902005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":646,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258745+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:03:18.258745+0000","last_peered":"2024-03-30T15:03:18.258745+0000","last_clean":"2024-03-30T15:03:18.258745+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:03:18.258745+0000","last_undegraded":"2024-03-30T15:03:18.258745+0000","last_fullsized":"2024-03-30T15:03:18.258745+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269780+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:03:18.269780+0000","last_peered":"2024-03-30T15:03:18.269780+0000","last_clean":"2024-03-30T15:03:18.269780+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:03:18.269780+0000","last_undegraded":"2024-03-30T15:03:18.269780+0000","last_fullsized":"2024-03-30T15:03:18.269780+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.008953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":310,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.661735+0000","last_change":"2024-03-30T15:02:42.208942+0000","last_active":"2024-03-30T15:03:18.661735+0000","last_peered":"2024-03-30T15:03:18.661735+0000","last_clean":"2024-03-30T15:03:18.661735+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:03:18.661735+0000","last_undegraded":"2024-03-30T15:03:18.661735+0000","last_fullsized":"2024-03-30T15:03:18.661735+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:02:42.208837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.208837+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:59.566569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261121+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:03:18.261121+0000","last_peered":"2024-03-30T15:03:18.261121+0000","last_clean":"2024-03-30T15:03:18.261121+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:03:18.261121+0000","last_undegraded":"2024-03-30T15:03:18.261121+0000","last_fullsized":"2024-03-30T15:03:18.261121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":757,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270026+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:03:18.270026+0000","last_peered":"2024-03-30T15:03:18.270026+0000","last_clean":"2024-03-30T15:03:18.270026+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:03:18.270026+0000","last_undegraded":"2024-03-30T15:03:18.270026+0000","last_fullsized":"2024-03-30T15:03:18.270026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.713330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263886+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:03:18.263886+0000","last_peered":"2024-03-30T15:03:18.263886+0000","last_clean":"2024-03-30T15:03:18.263886+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:03:18.263886+0000","last_undegraded":"2024-03-30T15:03:18.263886+0000","last_fullsized":"2024-03-30T15:03:18.263886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.141724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264825+0000","last_change":"2024-03-30T15:02:35.380537+0000","last_active":"2024-03-30T15:03:18.264825+0000","last_peered":"2024-03-30T15:03:18.264825+0000","last_clean":"2024-03-30T15:03:18.264825+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:03:18.264825+0000","last_undegraded":"2024-03-30T15:03:18.264825+0000","last_fullsized":"2024-03-30T15:03:18.264825+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.380454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.380454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:40.477448+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265229+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:03:18.265229+0000","last_peered":"2024-03-30T15:03:18.265229+0000","last_clean":"2024-03-30T15:03:18.265229+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:03:18.265229+0000","last_undegraded":"2024-03-30T15:03:18.265229+0000","last_fullsized":"2024-03-30T15:03:18.265229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:04.864954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258192+0000","last_change":"2024-03-30T14:59:09.624243+0000","last_active":"2024-03-30T15:03:18.258192+0000","last_peered":"2024-03-30T15:03:18.258192+0000","last_clean":"2024-03-30T15:03:18.258192+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:03:18.258192+0000","last_undegraded":"2024-03-30T15:03:18.258192+0000","last_fullsized":"2024-03-30T15:03:18.258192+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:09.624114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:09.624114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":746,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262408+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:03:18.262408+0000","last_peered":"2024-03-30T15:03:18.262408+0000","last_clean":"2024-03-30T15:03:18.262408+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:03:18.262408+0000","last_undegraded":"2024-03-30T15:03:18.262408+0000","last_fullsized":"2024-03-30T15:03:18.262408+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:43.075639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":747,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270960+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:03:18.270960+0000","last_peered":"2024-03-30T15:03:18.270960+0000","last_clean":"2024-03-30T15:03:18.270960+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:03:18.270960+0000","last_undegraded":"2024-03-30T15:03:18.270960+0000","last_fullsized":"2024-03-30T15:03:18.270960+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:37.960602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":663,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263298+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:03:18.263298+0000","last_peered":"2024-03-30T15:03:18.263298+0000","last_clean":"2024-03-30T15:03:18.263298+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:03:18.263298+0000","last_undegraded":"2024-03-30T15:03:18.263298+0000","last_fullsized":"2024-03-30T15:03:18.263298+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:29.813607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":762,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261590+0000","last_change":"2024-03-30T14:59:10.626455+0000","last_active":"2024-03-30T15:03:18.261590+0000","last_peered":"2024-03-30T15:03:18.261590+0000","last_clean":"2024-03-30T15:03:18.261590+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:03:18.261590+0000","last_undegraded":"2024-03-30T15:03:18.261590+0000","last_fullsized":"2024-03-30T15:03:18.261590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.626365+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":752,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264885+0000","last_change":"2024-03-30T15:01:47.383230+0000","last_active":"2024-03-30T15:03:18.264885+0000","last_peered":"2024-03-30T15:03:18.264885+0000","last_clean":"2024-03-30T15:03:18.264885+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:03:18.264885+0000","last_undegraded":"2024-03-30T15:03:18.264885+0000","last_fullsized":"2024-03-30T15:03:18.264885+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:01:47.383148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.383148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":874,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264662+0000","last_change":"2024-03-30T15:02:51.451850+0000","last_active":"2024-03-30T15:03:18.264662+0000","last_peered":"2024-03-30T15:03:18.264662+0000","last_clean":"2024-03-30T15:03:18.264662+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:03:18.264662+0000","last_undegraded":"2024-03-30T15:03:18.264662+0000","last_fullsized":"2024-03-30T15:03:18.264662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:51.451690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:51.451690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:00.171722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":740,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.266817+0000","last_change":"2024-03-30T15:00:36.069861+0000","last_active":"2024-03-30T15:03:18.266817+0000","last_peered":"2024-03-30T15:03:18.266817+0000","last_clean":"2024-03-30T15:03:18.266817+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:03:18.266817+0000","last_undegraded":"2024-03-30T15:03:18.266817+0000","last_fullsized":"2024-03-30T15:03:18.266817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:00:36.069752+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:00:36.069752+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":655,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265587+0000","last_change":"2024-03-30T15:02:03.146988+0000","last_active":"2024-03-30T15:03:18.265587+0000","last_peered":"2024-03-30T15:03:18.265587+0000","last_clean":"2024-03-30T15:03:18.265587+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:03:18.265587+0000","last_undegraded":"2024-03-30T15:03:18.265587+0000","last_fullsized":"2024-03-30T15:03:18.265587+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.146842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.146842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:24.075939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":694,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270525+0000","last_change":"2024-03-30T15:01:20.341161+0000","last_active":"2024-03-30T15:03:18.270525+0000","last_peered":"2024-03-30T15:03:18.270525+0000","last_clean":"2024-03-30T15:03:18.270525+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:03:18.270525+0000","last_undegraded":"2024-03-30T15:03:18.270525+0000","last_fullsized":"2024-03-30T15:03:18.270525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:01:20.341036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.341036+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260367+0000","last_change":"2024-03-30T14:58:31.196084+0000","last_active":"2024-03-30T15:03:18.260367+0000","last_peered":"2024-03-30T15:03:18.260367+0000","last_clean":"2024-03-30T15:03:18.260367+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:03:18.260367+0000","last_undegraded":"2024-03-30T15:03:18.260367+0000","last_fullsized":"2024-03-30T15:03:18.260367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:31.196010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:31.196010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":734,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260676+0000","last_change":"2024-03-30T14:59:01.294973+0000","last_active":"2024-03-30T15:03:18.260676+0000","last_peered":"2024-03-30T15:03:18.260676+0000","last_clean":"2024-03-30T15:03:18.260676+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:03:18.260676+0000","last_undegraded":"2024-03-30T15:03:18.260676+0000","last_fullsized":"2024-03-30T15:03:18.260676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:01.294888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:01.294888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265021+0000","last_change":"2024-03-30T15:02:49.425842+0000","last_active":"2024-03-30T15:03:18.265021+0000","last_peered":"2024-03-30T15:03:18.265021+0000","last_clean":"2024-03-30T15:03:18.265021+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:03:18.265021+0000","last_undegraded":"2024-03-30T15:03:18.265021+0000","last_fullsized":"2024-03-30T15:03:18.265021+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:49.425762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.425762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:51.518922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269353+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:03:18.269353+0000","last_peered":"2024-03-30T15:03:18.269353+0000","last_clean":"2024-03-30T15:03:18.269353+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:03:18.269353+0000","last_undegraded":"2024-03-30T15:03:18.269353+0000","last_fullsized":"2024-03-30T15:03:18.269353+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":787,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.271471+0000","last_change":"2024-03-30T15:01:18.365147+0000","last_active":"2024-03-30T15:03:18.271471+0000","last_peered":"2024-03-30T15:03:18.271471+0000","last_clean":"2024-03-30T15:03:18.271471+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:03:18.271471+0000","last_undegraded":"2024-03-30T15:03:18.271471+0000","last_fullsized":"2024-03-30T15:03:18.271471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:01:18.365052+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.365052+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":753,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259987+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:03:18.259987+0000","last_peered":"2024-03-30T15:03:18.259987+0000","last_clean":"2024-03-30T15:03:18.259987+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:03:18.259987+0000","last_undegraded":"2024-03-30T15:03:18.259987+0000","last_fullsized":"2024-03-30T15:03:18.259987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":743,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270865+0000","last_change":"2024-03-30T15:01:15.391982+0000","last_active":"2024-03-30T15:03:18.270865+0000","last_peered":"2024-03-30T15:03:18.270865+0000","last_clean":"2024-03-30T15:03:18.270865+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:03:18.270865+0000","last_undegraded":"2024-03-30T15:03:18.270865+0000","last_fullsized":"2024-03-30T15:03:18.270865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:15.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:01:15.391906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":731,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261114+0000","last_change":"2024-03-30T14:59:03.291569+0000","last_active":"2024-03-30T15:03:18.261114+0000","last_peered":"2024-03-30T15:03:18.261114+0000","last_clean":"2024-03-30T15:03:18.261114+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:03:18.261114+0000","last_undegraded":"2024-03-30T15:03:18.261114+0000","last_fullsized":"2024-03-30T15:03:18.261114+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:03.291469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.291469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":657,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263370+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:03:18.263370+0000","last_peered":"2024-03-30T15:03:18.263370+0000","last_clean":"2024-03-30T15:03:18.263370+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:03:18.263370+0000","last_undegraded":"2024-03-30T15:03:18.263370+0000","last_fullsized":"2024-03-30T15:03:18.263370+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:42.642572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":772,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259796+0000","last_change":"2024-03-30T15:02:02.493916+0000","last_active":"2024-03-30T15:03:18.259796+0000","last_peered":"2024-03-30T15:03:18.259796+0000","last_clean":"2024-03-30T15:03:18.259796+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:03:18.259796+0000","last_undegraded":"2024-03-30T15:03:18.259796+0000","last_fullsized":"2024-03-30T15:03:18.259796+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T14:59:46.413127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:46.413127+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":749,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269103+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:03:18.269103+0000","last_peered":"2024-03-30T15:03:18.269103+0000","last_clean":"2024-03-30T15:03:18.269103+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:03:18.269103+0000","last_undegraded":"2024-03-30T15:03:18.269103+0000","last_fullsized":"2024-03-30T15:03:18.269103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.425048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.262355+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:03:18.262355+0000","last_peered":"2024-03-30T15:03:18.262355+0000","last_clean":"2024-03-30T15:03:18.262355+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:03:18.262355+0000","last_undegraded":"2024-03-30T15:03:18.262355+0000","last_fullsized":"2024-03-30T15:03:18.262355+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:55.847447+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":737,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.267997+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:03:18.267997+0000","last_peered":"2024-03-30T15:03:18.267997+0000","last_clean":"2024-03-30T15:03:18.267997+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:03:18.267997+0000","last_undegraded":"2024-03-30T15:03:18.267997+0000","last_fullsized":"2024-03-30T15:03:18.267997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268661+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:03:18.268661+0000","last_peered":"2024-03-30T15:03:18.268661+0000","last_clean":"2024-03-30T15:03:18.268661+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:03:18.268661+0000","last_undegraded":"2024-03-30T15:03:18.268661+0000","last_fullsized":"2024-03-30T15:03:18.268661+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.603935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.260342+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:03:18.260342+0000","last_peered":"2024-03-30T15:03:18.260342+0000","last_clean":"2024-03-30T15:03:18.260342+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:03:18.260342+0000","last_undegraded":"2024-03-30T15:03:18.260342+0000","last_fullsized":"2024-03-30T15:03:18.260342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.270894+0000","last_change":"2024-03-30T14:58:25.078864+0000","last_active":"2024-03-30T15:03:18.270894+0000","last_peered":"2024-03-30T15:03:18.270894+0000","last_clean":"2024-03-30T15:03:18.270894+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:03:18.270894+0000","last_undegraded":"2024-03-30T15:03:18.270894+0000","last_fullsized":"2024-03-30T15:03:18.270894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:58:25.078771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:58:25.078771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.269350+0000","last_change":"2024-03-30T15:02:44.998020+0000","last_active":"2024-03-30T15:03:18.269350+0000","last_peered":"2024-03-30T15:03:18.269350+0000","last_clean":"2024-03-30T15:03:18.269350+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:03:18.269350+0000","last_undegraded":"2024-03-30T15:03:18.269350+0000","last_fullsized":"2024-03-30T15:03:18.269350+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.997898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:45.179955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258582+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:03:18.258582+0000","last_peered":"2024-03-30T15:03:18.258582+0000","last_clean":"2024-03-30T15:03:18.258582+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:03:18.258582+0000","last_undegraded":"2024-03-30T15:03:18.258582+0000","last_fullsized":"2024-03-30T15:03:18.258582+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":647,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.268456+0000","last_change":"2024-03-30T15:01:16.348182+0000","last_active":"2024-03-30T15:03:18.268456+0000","last_peered":"2024-03-30T15:03:18.268456+0000","last_clean":"2024-03-30T15:03:18.268456+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:03:18.268456+0000","last_undegraded":"2024-03-30T15:03:18.268456+0000","last_fullsized":"2024-03-30T15:03:18.268456+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.348072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.348072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":745,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.265076+0000","last_change":"2024-03-30T15:02:33.875911+0000","last_active":"2024-03-30T15:03:18.265076+0000","last_peered":"2024-03-30T15:03:18.265076+0000","last_clean":"2024-03-30T15:03:18.265076+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:03:18.265076+0000","last_undegraded":"2024-03-30T15:03:18.265076+0000","last_fullsized":"2024-03-30T15:03:18.265076+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T15:02:33.875778+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.875778+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:03:45.369826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":652,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.263036+0000","last_change":"2024-03-30T15:01:47.131788+0000","last_active":"2024-03-30T15:03:18.263036+0000","last_peered":"2024-03-30T15:03:18.263036+0000","last_clean":"2024-03-30T15:03:18.263036+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:03:18.263036+0000","last_undegraded":"2024-03-30T15:03:18.263036+0000","last_fullsized":"2024-03-30T15:03:18.263036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:47.131690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.131690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":734,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258082+0000","last_change":"2024-03-30T14:59:12.396403+0000","last_active":"2024-03-30T15:03:18.258082+0000","last_peered":"2024-03-30T15:03:18.258082+0000","last_clean":"2024-03-30T15:03:18.258082+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:03:18.258082+0000","last_undegraded":"2024-03-30T15:03:18.258082+0000","last_fullsized":"2024-03-30T15:03:18.258082+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:12.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T14:59:12.396321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":304,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.663878+0000","last_change":"2024-03-30T15:03:16.163490+0000","last_active":"2024-03-30T15:03:18.663878+0000","last_peered":"2024-03-30T15:03:18.663878+0000","last_clean":"2024-03-30T15:03:18.663878+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:03:18.663878+0000","last_undegraded":"2024-03-30T15:03:18.663878+0000","last_fullsized":"2024-03-30T15:03:18.663878+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.163376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.163376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:36.277238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":651,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.259062+0000","last_change":"2024-03-30T15:01:17.584059+0000","last_active":"2024-03-30T15:03:18.259062+0000","last_peered":"2024-03-30T15:03:18.259062+0000","last_clean":"2024-03-30T15:03:18.259062+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:03:18.259062+0000","last_undegraded":"2024-03-30T15:03:18.259062+0000","last_fullsized":"2024-03-30T15:03:18.259062+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:17.583984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:17.583984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":778,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.258357+0000","last_change":"2024-03-30T15:02:19.657930+0000","last_active":"2024-03-30T15:03:18.258357+0000","last_peered":"2024-03-30T15:03:18.258357+0000","last_clean":"2024-03-30T15:03:18.258357+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:03:18.258357+0000","last_undegraded":"2024-03-30T15:03:18.258357+0000","last_fullsized":"2024-03-30T15:03:18.258357+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":754,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.261025+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:03:18.261025+0000","last_peered":"2024-03-30T15:03:18.261025+0000","last_clean":"2024-03-30T15:03:18.261025+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:03:18.261025+0000","last_undegraded":"2024-03-30T15:03:18.261025+0000","last_fullsized":"2024-03-30T15:03:18.261025+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":743,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.264617+0000","last_change":"2024-03-30T15:01:53.452032+0000","last_active":"2024-03-30T15:03:18.264617+0000","last_peered":"2024-03-30T15:03:18.264617+0000","last_clean":"2024-03-30T15:03:18.264617+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:03:18.264617+0000","last_undegraded":"2024-03-30T15:03:18.264617+0000","last_fullsized":"2024-03-30T15:03:18.264617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.451927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.451927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-03-30T15:03:18.547539+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:03:18.547539+0000","last_peered":"2024-03-30T15:03:18.547539+0000","last_clean":"2024-03-30T15:03:18.547539+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:03:18.547539+0000","last_undegraded":"2024-03-30T15:03:18.547539+0000","last_fullsized":"2024-03-30T15:03:18.547539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:40.511390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":296,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195759,"num_objects":156,"num_object_clones":2,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":590,"num_write_kb":12679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26431488,"data_stored":24587277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1861,"ondisk_log_size":1861,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":513,"num_read_kb":508,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005165,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87480,"kb_used_data":11876,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657672,"statfs":{"total":95995035648,"available":95905456128,"internally_reserved":0,"allocated":12161024,"data_stored":8882412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299506,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91596,"kb_used_data":11832,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93653556,"statfs":{"total":95995035648,"available":95901241344,"internally_reserved":0,"allocated":12115968,"data_stored":8889088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81656262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82972,"kb_used_data":7356,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662180,"statfs":{"total":95995035648,"available":95910072320,"internally_reserved":0,"allocated":7532544,"data_stored":4290795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65484,"kb_used_data":12076,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93679668,"statfs":{"total":95995035648,"available":95927980032,"internally_reserved":0,"allocated":12365824,"data_stored":9085105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95160,"kb_used_data":15384,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93649992,"statfs":{"total":95995035648,"available":95897591808,"internally_reserved":0,"allocated":15753216,"data_stored":12484353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117122,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91320,"kb_used_data":11544,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93653832,"statfs":{"total":95995035648,"available":95901523968,"internally_reserved":0,"allocated":11821056,"data_stored":8486240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953596,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87212,"kb_used_data":7424,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93657940,"statfs":{"total":95995035648,"available":95905730560,"internally_reserved":0,"allocated":7602176,"data_stored":4299832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659255,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83020,"kb_used_data":7416,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93662132,"statfs":{"total":95995035648,"available":95910023168,"internally_reserved":0,"allocated":7593984,"data_stored":4290466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4194807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8441856,"data_stored":8194873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4460544,"data_stored":4194913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4000467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4000556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:03:24.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 1 2024-03-30T15:03:24.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:03:25.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:25.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:25.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:25.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:25.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:25.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:25.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:25.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:25.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[41614]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:03:25.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:25.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:25 smithi028 ceph-mon[29876]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T15:03:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.513+0000 7f99635ea700 1 -- 172.21.15.28:0/699438376 >> v1:172.21.15.28:6789/0 conn(0x7f995c1094a0 legacy=0x7f995c109880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.514+0000 7f99635ea700 1 -- 172.21.15.28:0/699438376 shutdown_connections 2024-03-30T15:03:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.514+0000 7f99635ea700 1 -- 172.21.15.28:0/699438376 >> 172.21.15.28:0/699438376 conn(0x7f995c06d550 msgr2=0x7f995c06d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.514+0000 7f99635ea700 1 -- 172.21.15.28:0/699438376 shutdown_connections 2024-03-30T15:03:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.514+0000 7f99635ea700 1 -- 172.21.15.28:0/699438376 wait complete. 2024-03-30T15:03:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 librados: starting msgr at 2024-03-30T15:03:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 librados: starting objecter 2024-03-30T15:03:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 Processor -- start 2024-03-30T15:03:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 -- start start 2024-03-30T15:03:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 librados: setting wanted keys 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 librados: calling monclient init 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f995c10e870 con 0x7f995c1094a0 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f995c1b7680 con 0x7f995c1158d0 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99635ea700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f995c1b8860 con 0x7f995c111d30 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f9960b85700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f995c111d30 0x7f995c1a4a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54560/0 (socket says 172.21.15.28:54560) 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f9960b85700 1 -- 172.21.15.28:0/1878138280 learned_addr learned my addr 172.21.15.28:0/1878138280 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f9961386700 1 --1- 172.21.15.28:0/1878138280 >> v1:172.21.15.28:6789/0 conn(0x7f995c1094a0 0x7f995c10e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47428/0 (socket says 172.21.15.28:47428) 2024-03-30T15:03:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99527fc700 10 client.?.objecter ms_handle_connect 0x7f995c1094a0 2024-03-30T15:03:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99527fc700 10 client.?.objecter ms_handle_connect 0x7f995c111d30 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.515+0000 7f99527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 10 client.?.objecter ms_handle_connect 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768314135 0 0) 0x7f995c1b7680 con 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99480031f0 con 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136708107 0 0) 0x7f995c10e870 con 0x7f995c1094a0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f995c1b7680 con 0x7f995c1094a0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950978146 0 0) 0x7f995c1b8860 con 0x7f995c111d30 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f995c10e870 con 0x7f995c111d30 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3190584484 0 0) 0x7f99480031f0 con 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f995c1b8860 con 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9954001870 con 0x7f995c1158d0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3677034736 0 0) 0x7f995c1b7680 con 0x7f995c1094a0 2024-03-30T15:03:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99480031f0 con 0x7f995c1094a0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9958001950 con 0x7f995c1094a0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1019936864 0 0) 0x7f995c1b8860 con 0x7f995c1158d0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 >> v1:172.21.15.28:6790/0 conn(0x7f995c111d30 legacy=0x7f995c1a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 >> v1:172.21.15.28:6789/0 conn(0x7f995c1094a0 legacy=0x7f995c10e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.516+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c1b9a40 con 0x7f995c1158d0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f995c1b8a90 con 0x7f995c1158d0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99635ea700 10 client.50029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f995c1b8f50 con 0x7f995c1158d0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99635ea700 1 librados: init done 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9954003b20 con 0x7f995c1158d0 2024-03-30T15:03:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.517+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9954004bb0 con 0x7f995c1158d0 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.518+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9940005120 con 0x7f995c1158d0 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.519+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9954004e80 con 0x7f995c1158d0 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.519+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(434..434 src has 53..434) ==== 8523+0+0 (unknown 3659546658 0 0) 0x7f9954088890 con 0x7f995c1158d0 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.519+0000 7f99527fc700 10 client.50029.objecter ms_dispatch 0x7f995c001550 osd_map(434..434 src has 53..434) 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.519+0000 7f99527fc700 3 client.50029.objecter handle_osd_map got epochs [434,434] > 0 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.519+0000 7f99527fc700 3 client.50029.objecter handle_osd_map decoding full epoch 434 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.520+0000 7f99527fc700 20 client.50029.objecter dump_active .. 0 homeless 2024-03-30T15:03:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.520+0000 7f99527fc700 10 client.50029.objecter ms_handle_connect 0x7f99480703d0 2024-03-30T15:03:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.524+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9954055910 con 0x7f995c1158d0 2024-03-30T15:03:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:25.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:25.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:25 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:03:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:25 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:03:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:25 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:03:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:25 smithi134 ceph-mon[33714]: osdmap e433: 8 total, 8 up, 8 in 2024-03-30T15:03:25.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:25.751+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f9940005410 con 0x7f995c1158d0 2024-03-30T15:03:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:25.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:25.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:25.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:26.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:26.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:26.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:26.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:26.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.301+0000 7f99527fc700 1 -- 172.21.15.28:0/1878138280 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v435) ==== 87+0+0 (unknown 2393879144 0 0) 0x7f9954059360 con 0x7f995c1158d0 2024-03-30T15:03:26.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 10 librados: watch_flush enter 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 10 librados: watch_flush exit 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 20 client.50029.objecter shutdown clearing up homeless session... 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 10 client.50029.objecter successfully canceled tick 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 >> v1:172.21.15.28:6800/3011431093 conn(0x7f99480703d0 legacy=0x7f9948072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 >> v1:172.21.15.134:6789/0 conn(0x7f995c1158d0 legacy=0x7f995c1a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 shutdown_connections 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 >> 172.21.15.28:0/1878138280 conn(0x7f995c06d550 msgr2=0x7f995c118520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 shutdown_connections 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 -- 172.21.15.28:0/1878138280 wait complete. 2024-03-30T15:03:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:26.304+0000 7f99635ea700 1 librados: shutdown 2024-03-30T15:03:26.307 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:03:26.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T15:03:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1878138280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: from='client.50029 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[29876]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T15:03:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1878138280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: from='client.50029 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:26 smithi028 ceph-mon[41614]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:26.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:26.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:26.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:26.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: osdmap e434: 8 total, 8 up, 8 in 2024-03-30T15:03:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1878138280' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: from='client.50029 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:03:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:26 smithi134 ceph-mon[33714]: pgmap v595: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-30T15:03:26.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:26.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:26.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:27.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:27.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:27.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:27.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:27.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:03:27.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:03:27.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: from='client.50029 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[29876]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:03:27.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:27.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: from='client.50029 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:03:27.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T15:03:27.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:27 smithi028 ceph-mon[41614]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T15:03:27.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:27.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: from='client.50029 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: osdmap e435: 8 total, 8 up, 8 in 2024-03-30T15:03:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:27 smithi134 ceph-mon[33714]: osdmap e436: 8 total, 8 up, 8 in 2024-03-30T15:03:27.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:27.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:27.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:27.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:27.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:27.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:28.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:28.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:28.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:28.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:28.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:28.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:28.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:28.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:28.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:28 smithi028 ceph-mon[41614]: pgmap v598: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:28.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:28 smithi028 ceph-mon[29876]: pgmap v598: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:28.459 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:28.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:28.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:28 smithi134 ceph-mon[33714]: pgmap v598: 260 pgs: 260 active+clean; 8.4 MiB data, 673 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:03:28.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:28.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:28.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:28.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:28.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:28.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:28.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:29.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:29.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:29.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:29.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:03:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:29.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:29.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:29.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:29.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:29.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:29.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:29.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:29.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:29.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:29.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:29.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:29.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:30.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:30.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:30.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:30.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:30.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:30.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:30.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.669+0000 7f3fa084c700 1 -- 172.21.15.28:0/765902057 >> v1:172.21.15.28:6790/0 conn(0x7f3f9c109dc0 legacy=0x7f3f9c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 -- 172.21.15.28:0/765902057 shutdown_connections 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 -- 172.21.15.28:0/765902057 >> 172.21.15.28:0/765902057 conn(0x7f3f9c0fc9e0 msgr2=0x7f3f9c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 -- 172.21.15.28:0/765902057 shutdown_connections 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 -- 172.21.15.28:0/765902057 wait complete. 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 librados: starting msgr at 2024-03-30T15:03:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 librados: starting objecter 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 Processor -- start 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 -- start start 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 librados: setting wanted keys 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.670+0000 7f3fa084c700 1 librados: calling monclient init 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3fa084c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f9c1a9350 con 0x7f3f9c109dc0 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3fa084c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f9c1aa570 con 0x7f3f9c10d960 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3fa084c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f9c1ab7b0 con 0x7f3f9c101130 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f9a59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3f9c101130 0x7f3f9c0fe8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40516/0 (socket says 172.21.15.28:40516) 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f99d9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f9c109dc0 0x7f3f9c1a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35476/0 (socket says 172.21.15.28:35476) 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f9a59c700 1 -- 172.21.15.28:0/3588071198 learned_addr learned my addr 172.21.15.28:0/3588071198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter ms_handle_connect 0x7f3f9c101130 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter ms_handle_connect 0x7f3f9c109dc0 2024-03-30T15:03:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter ms_handle_connect 0x7f3f9c10d960 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478232023 0 0) 0x7f3f9c1ab7b0 con 0x7f3f9c101130 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f840031f0 con 0x7f3f9c101130 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864803953 0 0) 0x7f3f9c1a9350 con 0x7f3f9c109dc0 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f9c1ab7b0 con 0x7f3f9c109dc0 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280459081 0 0) 0x7f3f9c1aa570 con 0x7f3f9c10d960 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f9c1a9350 con 0x7f3f9c10d960 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3236492664 0 0) 0x7f3f840031f0 con 0x7f3f9c101130 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f9c1aa570 con 0x7f3f9c101130 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f90001930 con 0x7f3f9c101130 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 814587414 0 0) 0x7f3f9c1ab7b0 con 0x7f3f9c109dc0 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f840031f0 con 0x7f3f9c109dc0 2024-03-30T15:03:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f8c003820 con 0x7f3f9c109dc0 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2945496367 0 0) 0x7f3f9c1a9350 con 0x7f3f9c10d960 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f9c1ab7b0 con 0x7f3f9c10d960 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.671+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f94001bf0 con 0x7f3f9c10d960 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2978613163 0 0) 0x7f3f9c1aa570 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 >> v1:172.21.15.134:6789/0 conn(0x7f3f9c10d960 legacy=0x7f3f9c1a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 >> v1:172.21.15.28:6789/0 conn(0x7f3f9c109dc0 legacy=0x7f3f9c1a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c1ac9f0 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c1a9580 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3fa084c700 10 client.58019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f9c1a9a40 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3fa084c700 1 librados: init done 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3f90003b40 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.672+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f90005d30 con 0x7f3f9c101130 2024-03-30T15:03:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.673+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7c005120 con 0x7f3f9c101130 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.673+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3f900036a0 con 0x7f3f9c101130 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(436..436 src has 53..436) ==== 8579+0+0 (unknown 2545333297 0 0) 0x7f3f90089930 con 0x7f3f9c101130 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 10 client.58019.objecter ms_dispatch 0x7f3f9c001550 osd_map(436..436 src has 53..436) 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 3 client.58019.objecter handle_osd_map got epochs [436,436] > 0 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 3 client.58019.objecter handle_osd_map decoding full epoch 436 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 20 client.58019.objecter dump_active .. 0 homeless 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.674+0000 7f3f8b7fe700 10 client.58019.objecter ms_handle_connect 0x7f3f84070530 2024-03-30T15:03:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.678+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3f900569a0 con 0x7f3f9c101130 2024-03-30T15:03:30.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:30.725 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:30.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.757+0000 7faf45f3e700 1 -- 172.21.15.28:0/239516540 >> v1:172.21.15.28:6789/0 conn(0x7faf40101140 legacy=0x7faf40101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.757+0000 7faf45f3e700 1 -- 172.21.15.28:0/239516540 shutdown_connections 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 -- 172.21.15.28:0/239516540 >> 172.21.15.28:0/239516540 conn(0x7faf400fc9f0 msgr2=0x7faf400fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 -- 172.21.15.28:0/239516540 shutdown_connections 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 -- 172.21.15.28:0/239516540 wait complete. 2024-03-30T15:03:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 librados: starting msgr at 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 librados: starting objecter 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 Processor -- start 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 -- start start 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 librados: setting wanted keys 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.758+0000 7faf45f3e700 1 librados: calling monclient init 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf45f3e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf401a5e30 con 0x7faf40101140 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf45f3e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf401a7050 con 0x7faf4010d970 2024-03-30T15:03:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf45f3e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf401a8290 con 0x7faf40109dd0 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3f7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faf40101140 0x7faf401062d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35488/0 (socket says 172.21.15.28:35488) 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3f7fe700 1 -- 172.21.15.28:0/1149797515 learned_addr learned my addr 172.21.15.28:0/1149797515 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter ms_handle_connect 0x7faf40101140 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter ms_handle_connect 0x7faf40109dd0 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter ms_handle_connect 0x7faf4010d970 2024-03-30T15:03:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156042505 0 0) 0x7faf401a5e30 con 0x7faf40101140 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf280031f0 con 0x7faf40101140 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1919737267 0 0) 0x7faf401a8290 con 0x7faf40109dd0 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf401a5e30 con 0x7faf40109dd0 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301342793 0 0) 0x7faf401a7050 con 0x7faf4010d970 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf401a8290 con 0x7faf4010d970 2024-03-30T15:03:30.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2143074072 0 0) 0x7faf280031f0 con 0x7faf40101140 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf401a7050 con 0x7faf40101140 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf300045c0 con 0x7faf40101140 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2430339634 0 0) 0x7faf401a5e30 con 0x7faf40109dd0 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf280031f0 con 0x7faf40109dd0 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf34001870 con 0x7faf40109dd0 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 781988026 0 0) 0x7faf401a8290 con 0x7faf4010d970 2024-03-30T15:03:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf401a5e30 con 0x7faf4010d970 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf380013f0 con 0x7faf4010d970 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3297576423 0 0) 0x7faf401a7050 con 0x7faf40101140 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6790/0 conn(0x7faf40109dd0 legacy=0x7faf401a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.134:6789/0 conn(0x7faf4010d970 legacy=0x7faf401a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.759+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf401a94d0 con 0x7faf40101140 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faf401a84c0 con 0x7faf40101140 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf45f3e700 10 client.24831.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faf401a8980 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf45f3e700 1 librados: init done 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7faf300010b0 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf30004e30 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.760+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7faf2c005100 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.761+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7faf300037f0 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(436..436 src has 53..436) ==== 8579+0+0 (unknown 2545333297 0 0) 0x7faf30089900 con 0x7faf40101140 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 osd_map(436..436 src has 53..436) 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 3 client.24831.objecter handle_osd_map got epochs [436,436] > 0 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 3 client.24831.objecter handle_osd_map decoding full epoch 436 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 20 client.24831.objecter dump_active .. 0 homeless 2024-03-30T15:03:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v436) ==== 54+0+16 (unknown 155271652 0 2420000192) 0x7faf30089d40 con 0x7faf40101140 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf280704e0 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 10 client.24831.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 1 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 1 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 0 1 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 4 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 10 client.24831.objecter _send_command 1 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf2c004b50 con 0x7faf2c004730 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.762+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c004730 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.763+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c004b50 con 0x7faf2c004730 2024-03-30T15:03:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.763+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.763+0000 7faf3cff9700 10 client.24831.objecter _finish_command 1 = system:0 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.763+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 4 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.763+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 0 1 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 10 client.24831.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 2 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 2 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0053e0 osd=0 3 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 0 2 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 4 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 10 client.24831.objecter _send_command 2 2024-03-30T15:03:30.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.780+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c007b00 con 0x7faf2c004730 2024-03-30T15:03:30.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c007b00 con 0x7faf2c004730 2024-03-30T15:03:30.870 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:30.870 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:30.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 10 client.24831.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 4 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 0 2 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 10 client.24831.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 1 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 4 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 10 client.24831.objecter _send_command 3 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7faf2c002160 con 0x7faf2c0042a0 2024-03-30T15:03:30.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.782+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c0042a0 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.783+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c002160 con 0x7faf2c0042a0 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.783+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.783+0000 7faf3cff9700 10 client.24831.objecter _finish_command 3 = system:0 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.783+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 4 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.783+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 1 3 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 10 client.24831.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 4 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 4 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c004130 osd=1 3 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 1 4 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 4 2024-03-30T15:03:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 10 client.24831.objecter _send_command 4 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.799+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c001600 con 0x7faf2c0042a0 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c001600 con 0x7faf2c0042a0 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf3cff9700 10 client.24831.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 4 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 1 4 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 10 client.24831.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 5 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 5 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 2 5 2024-03-30T15:03:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 4 2024-03-30T15:03:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 10 client.24831.objecter _send_command 5 2024-03-30T15:03:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.801+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7faf2c015400 con 0x7faf2c0117c0 2024-03-30T15:03:30.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:30.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.802+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c0117c0 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.803+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c015400 con 0x7faf2c0117c0 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.803+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.803+0000 7faf3cff9700 10 client.24831.objecter _finish_command 5 = system:0 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.803+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 4 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.803+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 2 5 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 10 client.24831.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 6 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 6 2024-03-30T15:03:30.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c011140 osd=2 3 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 2 6 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 4 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 10 client.24831.objecter _send_command 6 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.819+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c009850 con 0x7faf2c0117c0 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c009850 con 0x7faf2c0117c0 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf3cff9700 10 client.24831.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 4 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 2 6 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 10 client.24831.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 7 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 7 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 3 7 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 4 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 10 client.24831.objecter _send_command 7 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.821+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7faf2c016150 con 0x7faf2c0155e0 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c0155e0 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c016150 con 0x7faf2c0155e0 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 10 client.24831.objecter _finish_command 7 = system:0 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 4 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.822+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 3 7 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 10 client.24831.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 8 2024-03-30T15:03:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 8 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c009ae0 osd=3 3 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 3 8 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 4 2024-03-30T15:03:30.878 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 10 client.24831.objecter _send_command 8 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.839+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c017c80 con 0x7faf2c0155e0 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c017c80 con 0x7faf2c0155e0 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 10 client.24831.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 4 2024-03-30T15:03:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 3 8 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 10 client.24831.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 9 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 9 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 4 9 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 4 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 10 client.24831.objecter _send_command 9 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7faf2c01ebc0 con 0x7faf2c01afc0 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.841+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c01afc0 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.842+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c01ebc0 con 0x7faf2c01afc0 2024-03-30T15:03:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.842+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.842+0000 7faf3cff9700 10 client.24831.objecter _finish_command 9 = system:0 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.842+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 4 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.842+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 4 9 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 10 client.24831.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 10 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 10 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c017c80 osd=4 3 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 4 10 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 4 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 10 client.24831.objecter _send_command 10 2024-03-30T15:03:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.861+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c020f90 con 0x7faf2c01afc0 2024-03-30T15:03:30.882 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c020f90 con 0x7faf2c01afc0 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf3cff9700 10 client.24831.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 4 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 4 10 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 10 client.24831.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 11 2024-03-30T15:03:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 11 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 5 11 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 4 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 10 client.24831.objecter _send_command 11 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.863+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7faf2c0202e0 con 0x7faf2c020d40 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.864+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c020d40 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.865+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c0202e0 con 0x7faf2c020d40 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.865+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.865+0000 7faf3cff9700 10 client.24831.objecter _finish_command 11 = system:0 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.865+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 4 2024-03-30T15:03:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.865+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 5 11 2024-03-30T15:03:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:30.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 10 client.24831.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 12 2024-03-30T15:03:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 12 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0213f0 osd=5 3 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 5 12 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 4 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 10 client.24831.objecter _send_command 12 2024-03-30T15:03:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.889+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c0202e0 con 0x7faf2c020d40 2024-03-30T15:03:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c0202e0 con 0x7faf2c020d40 2024-03-30T15:03:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf3cff9700 10 client.24831.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 4 2024-03-30T15:03:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 5 12 2024-03-30T15:03:30.893 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 10 client.24831.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 13 2024-03-30T15:03:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 13 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 6 13 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 4 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 10 client.24831.objecter _send_command 13 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.891+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7faf2c017a90 con 0x7faf2c023c80 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.892+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c023c80 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.893+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c017a90 con 0x7faf2c023c80 2024-03-30T15:03:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.893+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.893+0000 7faf3cff9700 10 client.24831.objecter _finish_command 13 = system:0 2024-03-30T15:03:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.893+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 4 2024-03-30T15:03:30.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.893+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 6 13 2024-03-30T15:03:30.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.902+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f3f7c005410 con 0x7f3f9c101130 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 10 client.24831.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 14 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 14 2024-03-30T15:03:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c0202e0 osd=6 3 2024-03-30T15:03:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 6 14 2024-03-30T15:03:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 4 2024-03-30T15:03:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 10 client.24831.objecter _send_command 14 2024-03-30T15:03:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.910+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c028f60 con 0x7faf2c023c80 2024-03-30T15:03:30.913 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c028f60 con 0x7faf2c023c80 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf3cff9700 10 client.24831.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 4 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 6 14 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 10 client.24831.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 15 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 15 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 7 15 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 4 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 10 client.24831.objecter _send_command 15 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.912+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7faf2c029ae0 con 0x7faf2c029680 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.913+0000 7faf3cff9700 10 client.24831.objecter ms_handle_connect 0x7faf2c029680 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.914+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faf2c029ae0 con 0x7faf2c029680 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.914+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.914+0000 7faf3cff9700 10 client.24831.objecter _finish_command 15 = system:0 2024-03-30T15:03:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.914+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 4 2024-03-30T15:03:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.914+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 7 15 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 10 client.24831.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign -1 16 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 20 client.24831.objecter _get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_remove -1 16 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 20 client.24831.objecter get_session s=0x7faf2c028f60 osd=7 3 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 15 client.24831.objecter _session_command_op_assign 7 16 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 4 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 10 client.24831.objecter _send_command 16 2024-03-30T15:03:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.931+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf2c029ae0 con 0x7faf2c029680 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3cff9700 1 -- 172.21.15.28:0/1149797515 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faf2c029ae0 con 0x7faf2c029680 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3cff9700 10 client.24831.objecter ms_dispatch 0x7faf40001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3cff9700 10 client.24831.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3cff9700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 4 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3cff9700 15 client.24831.objecter _session_command_op_remove 7 16 2024-03-30T15:03:30.935 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:30.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 librados: watch_flush enter 2024-03-30T15:03:30.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 librados: watch_flush exit 2024-03-30T15:03:30.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.0 2024-03-30T15:03:30.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6801/3394323575 conn(0x7faf2c004730 legacy=0x7faf2c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0053e0 osd=0 1 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.1 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6805/1850313547 conn(0x7faf2c0042a0 legacy=0x7faf2c008e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c004130 osd=1 1 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.2 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6809/3451325043 conn(0x7faf2c0117c0 legacy=0x7faf2c013bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c011140 osd=2 1 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.3 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6813/1506157250 conn(0x7faf2c0155e0 legacy=0x7faf2c015a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c009ae0 osd=3 1 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.4 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.134:6800/3063775514 conn(0x7faf2c01afc0 legacy=0x7faf2c01d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c017c80 osd=4 1 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.5 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.134:6804/3668151179 conn(0x7faf2c020d40 legacy=0x7faf2c01ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3ffff700 1 -- 172.21.15.28:0/1149797515 reap_dead start 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0213f0 osd=5 1 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.6 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3ffff700 1 -- 172.21.15.28:0/1149797515 reap_dead start 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.134:6808/2135224194 conn(0x7faf2c023c80 legacy=0x7faf2c026070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c0202e0 osd=6 1 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter close_session for osd.7 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.134:6812/3758190298 conn(0x7faf2c029680 legacy=0x7faf2c027790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter put_session s=0x7faf2c028f60 osd=7 1 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 20 client.24831.objecter shutdown clearing up homeless session... 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 10 client.24831.objecter successfully canceled tick 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3ffff700 1 -- 172.21.15.28:0/1149797515 reap_dead start 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf3ffff700 1 -- 172.21.15.28:0/1149797515 reap_dead start 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6800/3011431093 conn(0x7faf280704e0 legacy=0x7faf280729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> v1:172.21.15.28:6789/0 conn(0x7faf40101140 legacy=0x7faf401062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 shutdown_connections 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 >> 172.21.15.28:0/1149797515 conn(0x7faf400fc9f0 msgr2=0x7faf400fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 shutdown_connections 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.934+0000 7faf45f3e700 1 -- 172.21.15.28:0/1149797515 wait complete. 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:30.935+0000 7faf45f3e700 1 librados: shutdown 2024-03-30T15:03:30.939 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:30.940 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:30.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:31.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.043+0000 7f3f8b7fe700 1 -- 172.21.15.28:0/3588071198 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v437) ==== 129+0+0 (unknown 1089889432 0 0) 0x7f3f9005a3f0 con 0x7f3f9c101130 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 10 librados: watch_flush enter 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 10 librados: watch_flush exit 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 20 client.58019.objecter shutdown clearing up homeless session... 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 10 client.58019.objecter successfully canceled tick 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3f84070530 legacy=0x7f3f840729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 >> v1:172.21.15.28:6790/0 conn(0x7f3f9c101130 legacy=0x7f3f9c0fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 shutdown_connections 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 >> 172.21.15.28:0/3588071198 conn(0x7f3f9c0fc9e0 msgr2=0x7f3f9c106cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 shutdown_connections 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 -- 172.21.15.28:0/3588071198 wait complete. 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:31.046+0000 7f3fa084c700 1 librados: shutdown 2024-03-30T15:03:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 1 2024-03-30T15:03:31.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:31.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[41614]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T15:03:31.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1149797515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:31.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3588071198' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[41614]: from='client.58019 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[29876]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T15:03:31.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1149797515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:31.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3588071198' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:31 smithi028 ceph-mon[29876]: from='client.58019 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:31.266 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:31.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:31.495 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:31 smithi134 ceph-mon[33714]: pgmap v599: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T15:03:31.495 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1149797515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:31.496 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3588071198' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.496 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:31 smithi134 ceph-mon[33714]: from='client.58019 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-30T15:03:31.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:31.591 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:31.591 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:31.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:31.649 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:31.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:31.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:31.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:31.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:32.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:32.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:32.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:32.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:32.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[41614]: from='client.58019 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:03:32.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[41614]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T15:03:32.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:32.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[29876]: from='client.58019 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:03:32.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:32 smithi028 ceph-mon[29876]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T15:03:32.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:32.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:32.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:32 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:32.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:32 smithi134 ceph-mon[33714]: from='client.58019 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-30T15:03:32.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:32 smithi134 ceph-mon[33714]: osdmap e437: 8 total, 8 up, 8 in 2024-03-30T15:03:32.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:32.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:03:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:32.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:32.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:32.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:32.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:32.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:32.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:33.036 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:33.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:33.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:33.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:33.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:33.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:33.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:33.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:33.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:33 smithi028 ceph-mon[41614]: pgmap v601: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T15:03:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:33 smithi028 ceph-mon[41614]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T15:03:33.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:33 smithi028 ceph-mon[29876]: pgmap v601: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T15:03:33.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:33 smithi028 ceph-mon[29876]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T15:03:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:33 smithi134 ceph-mon[33714]: pgmap v601: 260 pgs: 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-30T15:03:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:33 smithi134 ceph-mon[33714]: osdmap e438: 8 total, 8 up, 8 in 2024-03-30T15:03:33.682 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:33.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:33.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:33.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:33.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:33.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:33.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:34.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:34.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:34.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:34.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:34.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:34.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:34.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:34.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:34.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:34.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:34.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:34.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:34.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:34.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:03:34.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:34.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:34.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:35.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:35.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:35.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.239 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:35.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.305+0000 7f5d8f366700 1 -- 172.21.15.28:0/190133150 >> v1:172.21.15.28:6789/0 conn(0x7f5d8810ea70 legacy=0x7f5d88110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.305+0000 7f5d8f366700 1 -- 172.21.15.28:0/190133150 shutdown_connections 2024-03-30T15:03:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.305+0000 7f5d8f366700 1 -- 172.21.15.28:0/190133150 >> 172.21.15.28:0/190133150 conn(0x7f5d880fdaf0 msgr2=0x7f5d880fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 -- 172.21.15.28:0/190133150 shutdown_connections 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 -- 172.21.15.28:0/190133150 wait complete. 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 librados: starting msgr at 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 librados: starting objecter 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 Processor -- start 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 -- start start 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 librados: setting wanted keys 2024-03-30T15:03:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.306+0000 7f5d8f366700 1 librados: calling monclient init 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8f366700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d881a6f30 con 0x7f5d88102240 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8f366700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d881a8150 con 0x7f5d8810ea70 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8f366700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d881a9390 con 0x7f5d8810aed0 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8d102700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d88102240 0x7f5d881073d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35502/0 (socket says 172.21.15.28:35502) 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8c901700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5d8810aed0 0x7f5d881a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40566/0 (socket says 172.21.15.28:40566) 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d8d102700 1 -- 172.21.15.28:0/1142621614 learned_addr learned my addr 172.21.15.28:0/1142621614 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8810aed0 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter ms_handle_connect 0x7f5d88102240 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8810ea70 2024-03-30T15:03:35.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104707331 0 0) 0x7f5d881a9390 con 0x7f5d8810aed0 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d740031f0 con 0x7f5d8810aed0 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373913090 0 0) 0x7f5d881a8150 con 0x7f5d8810ea70 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d881a9390 con 0x7f5d8810ea70 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3920661975 0 0) 0x7f5d881a6f30 con 0x7f5d88102240 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d881a8150 con 0x7f5d88102240 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 659392578 0 0) 0x7f5d740031f0 con 0x7f5d8810aed0 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d881a6f30 con 0x7f5d8810aed0 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d78001910 con 0x7f5d8810aed0 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3132474460 0 0) 0x7f5d881a9390 con 0x7f5d8810ea70 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d740031f0 con 0x7f5d8810ea70 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2735944691 0 0) 0x7f5d881a8150 con 0x7f5d88102240 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d881a9390 con 0x7f5d88102240 2024-03-30T15:03:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d80004730 con 0x7f5d8810ea70 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d84001230 con 0x7f5d88102240 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.307+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1390864923 0 0) 0x7f5d881a6f30 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 >> v1:172.21.15.134:6789/0 conn(0x7f5d8810ea70 legacy=0x7f5d881a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 >> v1:172.21.15.28:6789/0 conn(0x7f5d88102240 legacy=0x7f5d881073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d881aa5d0 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d881a8380 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5d78003df0 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d8f366700 10 client.58031.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d78005dd0 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d881a8840 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.308+0000 7f5d8f366700 1 librados: init done 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.309+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d88103720 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.309+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5d78001bc0 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(438..438 src has 53..438) ==== 8551+0+0 (unknown 159875066 0 0) 0x7f5d78089980 con 0x7f5d8810aed0 2024-03-30T15:03:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 10 client.58031.objecter ms_dispatch 0x7f5d88001550 osd_map(438..438 src has 53..438) 2024-03-30T15:03:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 3 client.58031.objecter handle_osd_map got epochs [438,438] > 0 2024-03-30T15:03:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 3 client.58031.objecter handle_osd_map decoding full epoch 438 2024-03-30T15:03:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 20 client.58031.objecter dump_active .. 0 homeless 2024-03-30T15:03:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.310+0000 7f5d7e7fc700 10 client.58031.objecter ms_handle_connect 0x7f5d74070320 2024-03-30T15:03:35.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.314+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5d78056a10 con 0x7f5d8810aed0 2024-03-30T15:03:35.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:35.357 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:35 smithi028 ceph-mon[41614]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 913 B/s wr, 4 op/s 2024-03-30T15:03:35.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:35 smithi028 ceph-mon[29876]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 913 B/s wr, 4 op/s 2024-03-30T15:03:35.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:35.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:35.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:35.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:35.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:35.533+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f5d88068b30 con 0x7f5d8810aed0 2024-03-30T15:03:35.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:35 smithi134 ceph-mon[33714]: pgmap v603: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 913 B/s wr, 4 op/s 2024-03-30T15:03:35.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:35.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:35.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:35.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:35.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:35.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:35.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:36.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:36.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:36.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:36.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.270+0000 7f5d7e7fc700 1 -- 172.21.15.28:0/1142621614 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v439) ==== 97+0+0 (unknown 2257801178 0 0) 0x7f5d7805a460 con 0x7f5d8810aed0 2024-03-30T15:03:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 10 librados: watch_flush enter 2024-03-30T15:03:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 10 librados: watch_flush exit 2024-03-30T15:03:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 20 client.58031.objecter shutdown clearing up homeless session... 2024-03-30T15:03:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 10 client.58031.objecter successfully canceled tick 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5d74070320 legacy=0x7f5d740727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 >> v1:172.21.15.28:6790/0 conn(0x7f5d8810aed0 legacy=0x7f5d881a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 shutdown_connections 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 >> 172.21.15.28:0/1142621614 conn(0x7f5d880fdaf0 msgr2=0x7f5d880fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 shutdown_connections 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 -- 172.21.15.28:0/1142621614 wait complete. 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:36.274+0000 7f5d8f366700 1 librados: shutdown 2024-03-30T15:03:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:03:36.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:36.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:36.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:36.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:36.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1142621614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:36 smithi028 ceph-mon[41614]: from='client.58031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1142621614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:36 smithi028 ceph-mon[29876]: from='client.58031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:36.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1142621614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:36 smithi134 ceph-mon[33714]: from='client.58031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:03:36.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:36.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:36.847 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:36.847 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:36.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:36.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:36.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:37.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:37.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:37.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:37.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:37.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:37.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:37.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: pgmap v604: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-30T15:03:37.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:37.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: from='client.58031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:37.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T15:03:37.457 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[41614]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: pgmap v604: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: from='client.58031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:37.458 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:37 smithi028 ceph-mon[29876]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T15:03:37.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:37.534 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:37.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:37.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: pgmap v604: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: from='client.58031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: osdmap e439: 8 total, 8 up, 8 in 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:37 smithi134 ceph-mon[33714]: osdmap e440: 8 total, 8 up, 8 in 2024-03-30T15:03:37.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:37.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:37.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:37.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:38.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:38.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:38.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:38.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:38.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:38.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:38.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:38.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:38.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:03:38.429 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:03:38.429 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-30T15:03:38.430 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-30T15:03:38.430 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-30T15:03:38.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-30T15:03:38.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:38.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:38.581 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:38 smithi134 ceph-mon[33714]: pgmap v607: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:03:38.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:38.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:38 smithi028 ceph-mon[29876]: pgmap v607: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:03:38.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:38 smithi028 ceph-mon[41614]: pgmap v607: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:03:38.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:38.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:38.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:38.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:38.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:38.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:39.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:39.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:39.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:39.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:39.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:03:39.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:39.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:39.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:39.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:39 smithi028 ceph-mon[41614]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T15:03:39.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:39 smithi028 ceph-mon[29876]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T15:03:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.544+0000 7f056da07700 1 -- 172.21.15.28:0/754710550 >> v1:172.21.15.28:6789/0 conn(0x7f056810ea70 legacy=0x7f0568110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.544+0000 7f056da07700 1 -- 172.21.15.28:0/754710550 shutdown_connections 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.544+0000 7f056da07700 1 -- 172.21.15.28:0/754710550 >> 172.21.15.28:0/754710550 conn(0x7f05680fdaf0 msgr2=0x7f05680fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.544+0000 7f056da07700 1 -- 172.21.15.28:0/754710550 shutdown_connections 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.544+0000 7f056da07700 1 -- 172.21.15.28:0/754710550 wait complete. 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 librados: starting msgr at 2024-03-30T15:03:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 librados: starting objecter 2024-03-30T15:03:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 Processor -- start 2024-03-30T15:03:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 -- start start 2024-03-30T15:03:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 librados: setting wanted keys 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 librados: calling monclient init 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05681a4dd0 con 0x7f0568102240 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05681a6010 con 0x7f056810aed0 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f056da07700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f05681a7250 con 0x7f056810ea70 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f05677fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f056810ea70 0x7f05681a35f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40176/0 (socket says 172.21.15.28:40176) 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.545+0000 7f05677fe700 1 -- 172.21.15.28:0/1065780755 learned_addr learned my addr 172.21.15.28:0/1065780755 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter ms_handle_connect 0x7f0568102240 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter ms_handle_connect 0x7f056810aed0 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter ms_handle_connect 0x7f056810ea70 2024-03-30T15:03:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187507542 0 0) 0x7f05681a6010 con 0x7f056810aed0 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05540031f0 con 0x7f056810aed0 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596564210 0 0) 0x7f05681a4dd0 con 0x7f0568102240 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05681a6010 con 0x7f0568102240 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1866026306 0 0) 0x7f05681a7250 con 0x7f056810ea70 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05681a4dd0 con 0x7f056810ea70 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3998041683 0 0) 0x7f05540031f0 con 0x7f056810aed0 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05681a7250 con 0x7f056810aed0 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4095994527 0 0) 0x7f05681a4dd0 con 0x7f056810ea70 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05540031f0 con 0x7f056810ea70 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2042074508 0 0) 0x7f05681a6010 con 0x7f0568102240 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05681a4dd0 con 0x7f0568102240 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0550001830 con 0x7f056810aed0 2024-03-30T15:03:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f05600043c0 con 0x7f056810ea70 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.546+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0558001b70 con 0x7f0568102240 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2365120404 0 0) 0x7f05681a7250 con 0x7f056810aed0 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6790/0 conn(0x7f056810ea70 legacy=0x7f05681a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6789/0 conn(0x7f0568102240 legacy=0x7f05681072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05681a7480 con 0x7f056810aed0 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05681a7770 con 0x7f056810aed0 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056da07700 10 client.50053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f05681a63b0 con 0x7f056810aed0 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056da07700 1 librados: init done 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0550003b20 con 0x7f056810aed0 2024-03-30T15:03:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.547+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0550004a60 con 0x7f056810aed0 2024-03-30T15:03:39.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.548+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0548005100 con 0x7f056810aed0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.548+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f055001b0e0 con 0x7f056810aed0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(441..441 src has 53..441) ==== 8579+0+0 (unknown 1878224483 0 0) 0x7f0550088610 con 0x7f056810aed0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 osd_map(441..441 src has 53..441) 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 3 client.50053.objecter handle_osd_map got epochs [441,441] > 0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 3 client.50053.objecter handle_osd_map decoding full epoch 441 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 20 client.50053.objecter dump_active .. 0 homeless 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v441) ==== 54+0+16 (unknown 2138544289 0 2420000192) 0x7f055008aa40 con 0x7f056810aed0 2024-03-30T15:03:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f0554070510 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056da07700 10 client.50053.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 1 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.549+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 1 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 0 1 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 4 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 10 client.50053.objecter _send_command 1 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0548004b50 con 0x7f0548004730 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.550+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f0548004730 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.551+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548004b50 con 0x7f0548004730 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.551+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.551+0000 7f056ca05700 10 client.50053.objecter _finish_command 1 = system:0 2024-03-30T15:03:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.551+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 4 2024-03-30T15:03:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.551+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 0 1 2024-03-30T15:03:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 10 client.50053.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 2 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 2 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f05480053e0 osd=0 3 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 0 2 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 4 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 10 client.50053.objecter _send_command 2 2024-03-30T15:03:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.573+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0548007b00 con 0x7f0548004730 2024-03-30T15:03:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0548007b00 con 0x7f0548004730 2024-03-30T15:03:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056ca05700 10 client.50053.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 4 2024-03-30T15:03:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 0 2 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 10 client.50053.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 1 3 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 4 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 10 client.50053.objecter _send_command 3 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.576+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0548002160 con 0x7f05480042a0 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.577+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f05480042a0 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.578+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548002160 con 0x7f05480042a0 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.578+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.578+0000 7f056ca05700 10 client.50053.objecter _finish_command 3 = system:0 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.578+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 4 2024-03-30T15:03:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.578+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 1 3 2024-03-30T15:03:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 10 client.50053.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 4 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 4 2024-03-30T15:03:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548004130 osd=1 3 2024-03-30T15:03:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 1 4 2024-03-30T15:03:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 4 2024-03-30T15:03:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 10 client.50053.objecter _send_command 4 2024-03-30T15:03:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.600+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0548001600 con 0x7f05480042a0 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0548001600 con 0x7f05480042a0 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 10 client.50053.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 4 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 1 4 2024-03-30T15:03:39.604 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 10 client.50053.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 5 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 5 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 2 5 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 4 2024-03-30T15:03:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 10 client.50053.objecter _send_command 5 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0548011010 con 0x7f0548008990 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.602+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f0548008990 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.603+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548011010 con 0x7f0548008990 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.603+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.603+0000 7f056ca05700 10 client.50053.objecter _finish_command 5 = system:0 2024-03-30T15:03:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.603+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 4 2024-03-30T15:03:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.603+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 2 5 2024-03-30T15:03:39.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 10 client.50053.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 6 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 6 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548001a00 osd=2 3 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 2 6 2024-03-30T15:03:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 4 2024-03-30T15:03:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 10 client.50053.objecter _send_command 6 2024-03-30T15:03:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.628+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0548008020 con 0x7f0548008990 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0548008020 con 0x7f0548008990 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 10 client.50053.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 4 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 2 6 2024-03-30T15:03:39.632 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 10 client.50053.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 7 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 7 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 3 7 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 4 2024-03-30T15:03:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 10 client.50053.objecter _send_command 7 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0548011d10 con 0x7f05480111c0 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.630+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f05480111c0 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.631+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548011d10 con 0x7f05480111c0 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.631+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.631+0000 7f056ca05700 10 client.50053.objecter _finish_command 7 = system:0 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.631+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 4 2024-03-30T15:03:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.631+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 3 7 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 10 client.50053.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 8 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 8 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548008270 osd=3 3 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 3 8 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 4 2024-03-30T15:03:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 10 client.50053.objecter _send_command 8 2024-03-30T15:03:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.660+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f05480126c0 con 0x7f05480111c0 2024-03-30T15:03:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.661+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f05480126c0 con 0x7f05480111c0 2024-03-30T15:03:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.661+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.661+0000 7f056ca05700 10 client.50053.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.661+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 4 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.661+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 3 8 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 10 client.50053.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 9 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 9 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 4 9 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 4 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 10 client.50053.objecter _send_command 9 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f054801a810 con 0x7f0548016bd0 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.662+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f0548016bd0 2024-03-30T15:03:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.663+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f054801a810 con 0x7f0548016bd0 2024-03-30T15:03:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.663+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.663+0000 7f056ca05700 10 client.50053.objecter _finish_command 9 = system:0 2024-03-30T15:03:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.663+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 4 2024-03-30T15:03:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.663+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 4 9 2024-03-30T15:03:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:39 smithi134 ceph-mon[33714]: osdmap e441: 8 total, 8 up, 8 in 2024-03-30T15:03:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 10 client.50053.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 10 2024-03-30T15:03:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 10 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548013840 osd=4 3 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 4 10 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 4 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 10 client.50053.objecter _send_command 10 2024-03-30T15:03:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.679+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0548013070 con 0x7f0548016bd0 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0548013070 con 0x7f0548016bd0 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 10 client.50053.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 4 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 4 10 2024-03-30T15:03:39.683 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 10 client.50053.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 11 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 11 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 5 11 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 4 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 10 client.50053.objecter _send_command 11 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0548012e70 con 0x7f05480133f0 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.681+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f05480133f0 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.682+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548012e70 con 0x7f05480133f0 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.682+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.682+0000 7f056ca05700 10 client.50053.objecter _finish_command 11 = system:0 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.682+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 4 2024-03-30T15:03:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.682+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 5 11 2024-03-30T15:03:39.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 10 client.50053.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 12 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 12 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548013280 osd=5 3 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 5 12 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 4 2024-03-30T15:03:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 10 client.50053.objecter _send_command 12 2024-03-30T15:03:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.699+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0548012e70 con 0x7f05480133f0 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.700+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0548012e70 con 0x7f05480133f0 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.700+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.700+0000 7f056ca05700 10 client.50053.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.700+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 4 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.700+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 5 12 2024-03-30T15:03:39.703 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 10 client.50053.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 13 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 13 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 6 13 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 4 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 10 client.50053.objecter _send_command 13 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0548023310 con 0x7f054801f6f0 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.701+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f054801f6f0 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.702+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0548023310 con 0x7f054801f6f0 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.702+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.702+0000 7f056ca05700 10 client.50053.objecter _finish_command 13 = system:0 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.702+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 4 2024-03-30T15:03:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.702+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 6 13 2024-03-30T15:03:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 10 client.50053.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 14 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 14 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548012e70 osd=6 3 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 6 14 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 4 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 10 client.50053.objecter _send_command 14 2024-03-30T15:03:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.719+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f05480232d0 con 0x7f054801f6f0 2024-03-30T15:03:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.720+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f05480232d0 con 0x7f054801f6f0 2024-03-30T15:03:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.720+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.720+0000 7f056ca05700 10 client.50053.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.720+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 4 2024-03-30T15:03:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.720+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 6 14 2024-03-30T15:03:39.723 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 10 client.50053.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 15 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 15 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 7 15 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 4 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 10 client.50053.objecter _send_command 15 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f05480246d0 con 0x7f05480239d0 2024-03-30T15:03:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.721+0000 7f056ca05700 10 client.50053.objecter ms_handle_connect 0x7f05480239d0 2024-03-30T15:03:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.722+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f05480246d0 con 0x7f05480239d0 2024-03-30T15:03:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.722+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.722+0000 7f056ca05700 10 client.50053.objecter _finish_command 15 = system:0 2024-03-30T15:03:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.722+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 4 2024-03-30T15:03:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.722+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 7 15 2024-03-30T15:03:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 10 client.50053.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign -1 16 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 20 client.50053.objecter _get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 15 client.50053.objecter _session_command_op_remove -1 16 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 20 client.50053.objecter get_session s=0x7f0548023310 osd=7 3 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 15 client.50053.objecter _session_command_op_assign 7 16 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 4 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 10 client.50053.objecter _send_command 16 2024-03-30T15:03:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.738+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f05480246d0 con 0x7f05480239d0 2024-03-30T15:03:39.742 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.740+0000 7f056ca05700 1 -- 172.21.15.28:0/1065780755 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f05480246d0 con 0x7f05480239d0 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.740+0000 7f056ca05700 10 client.50053.objecter ms_dispatch 0x7f0568001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.740+0000 7f056ca05700 10 client.50053.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.740+0000 7f056ca05700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 4 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.740+0000 7f056ca05700 15 client.50053.objecter _session_command_op_remove 7 16 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 librados: watch_flush enter 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 librados: watch_flush exit 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.0 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6801/3394323575 conn(0x7f0548004730 legacy=0x7f0548002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f05480053e0 osd=0 1 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.1 2024-03-30T15:03:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6805/1850313547 conn(0x7f05480042a0 legacy=0x7f054800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548004130 osd=1 1 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.2 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6809/3451325043 conn(0x7f0548008990 legacy=0x7f054800f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548001a00 osd=2 1 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.3 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6813/1506157250 conn(0x7f05480111c0 legacy=0x7f0548011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548008270 osd=3 1 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.4 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.134:6800/3063775514 conn(0x7f0548016bd0 legacy=0x7f0548018fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013840 osd=4 1 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.5 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.134:6804/3668151179 conn(0x7f05480133f0 legacy=0x7f05480126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548013280 osd=5 1 2024-03-30T15:03:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.6 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f05677fe700 1 -- 172.21.15.28:0/1065780755 reap_dead start 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.134:6808/2135224194 conn(0x7f054801f6f0 legacy=0x7f0548021ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f05677fe700 1 -- 172.21.15.28:0/1065780755 reap_dead start 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f05677fe700 1 -- 172.21.15.28:0/1065780755 reap_dead start 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548012e70 osd=6 1 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter close_session for osd.7 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.134:6812/3758190298 conn(0x7f05480239d0 legacy=0x7f0548023e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter put_session s=0x7f0548023310 osd=7 1 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 20 client.50053.objecter shutdown clearing up homeless session... 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 10 client.50053.objecter successfully canceled tick 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0554070510 legacy=0x7f05540729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> v1:172.21.15.134:6789/0 conn(0x7f056810aed0 legacy=0x7f056819fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 shutdown_connections 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 >> 172.21.15.28:0/1065780755 conn(0x7f05680fdaf0 msgr2=0x7f0568074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 shutdown_connections 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 -- 172.21.15.28:0/1065780755 wait complete. 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:39.741+0000 7f056da07700 1 librados: shutdown 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:39.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:39.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:39.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:39.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:40.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:40.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:40.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:40.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:40.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:40.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:40.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1065780755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:40 smithi134 ceph-mon[33714]: pgmap v609: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:40.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:40.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:40.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:40.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1065780755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:40.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:40 smithi028 ceph-mon[41614]: pgmap v609: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1065780755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:40 smithi028 ceph-mon[29876]: pgmap v609: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:40.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:40.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:40.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:03:40.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:40.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:41.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:41.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:41.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:41.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:41.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:41.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:41.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:41.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:41.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:41.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:41.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:41.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:41.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:41.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:41.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:41.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:42.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:42.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:42.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:42.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:42.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:42.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:42.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:42.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:42.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:42.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:42.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:42.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:42.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:03:42.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:42.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:03:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-30T15:03:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:03:42.958 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:42.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:43.016 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:43.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:43.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:43.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:43.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:43.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:43.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:43 smithi028 ceph-mon[29876]: pgmap v610: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:43.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:43 smithi028 ceph-mon[29876]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T15:03:43.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:43 smithi028 ceph-mon[41614]: pgmap v610: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:43.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:43 smithi028 ceph-mon[41614]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T15:03:43.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:43.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:43.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:43.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:43 smithi134 ceph-mon[33714]: pgmap v610: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:03:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:43 smithi134 ceph-mon[33714]: osdmap e442: 8 total, 8 up, 8 in 2024-03-30T15:03:43.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:43.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:43.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:43.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:44.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:44.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:44.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:44.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:44.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:44.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:44.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:44.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:44.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:44.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:44.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:44.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:44.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:44.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:44.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:44.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:44.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:45.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:45.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:45.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:45.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:45.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:45.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:45.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:45.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:45.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:45.629 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:03:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24910.0:2 25.17 25:ecb6a661:test-rados-api-smithi028-93441-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 349.655680652, 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3400814589999999, 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.630 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24910", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1422378701", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966913+0000", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8819999999999999e-06 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966920+0000", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0570000000000004e-06 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966930+0000", 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0156000000000001e-05 2024-03-30T15:03:45.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966940+0000", 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0147e-05 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083450+0000", 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.116509813 2024-03-30T15:03:45.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083460+0000", 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.292378+0000", 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2089187990000001 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294004+0000", 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00162598 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294807+0000", 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080235899999999999 2024-03-30T15:03:45.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295802+0000", 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000994834 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.299106+0000", 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003304202 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306871+0000", 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077654550000000001 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306919+0000", 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7447000000000002e-05 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306992+0000", 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3418999999999996e-05 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24961.0:2 27.15 27:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 349.65393363800001, 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23402463300000001, 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:03:45.638 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24961", 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1241392178", 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:03:45.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968659+0000", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.73e-06 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968665+0000", 2024-03-30T15:03:45.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8200000000000002e-06 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968676+0000", 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0528e-05 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968685+0000", 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.329e-06 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969119+0000", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043351199999999999 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969125+0000", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2280000000000003e-06 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049864+0000", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080739092999999998 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049873+0000", 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7539999999999995e-06 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202601+0000", 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.152728379 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202682+0000", 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1260000000000005e-05 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35042.0:4 21.1f 21:ff7b1f36:::obj1:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 349.44766016099999, 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0929989090000001, 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:03:45.644 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35042", 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/917910517", 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174933+0000", 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.421e-06 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174937+0000", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4139999999999996e-06 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174945+0000", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2090000000000005e-06 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174955+0000", 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5300000000000002e-06 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175014+0000", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9110000000000002e-05 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175023+0000", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2920000000000001e-06 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267861+0000", 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0928374670000001 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267930+0000", 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9466000000000002e-05 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34820.0:20 9.8 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 349.40824253300002, 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.089392117, 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34820", 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4253613591", 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:03:45.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214351+0000", 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.103e-06 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214358+0000", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5320000000000001e-06 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214370+0000", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1377e-05 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214379+0000", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9090000000000006e-06 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214437+0000", 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8692e-05 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214446+0000", 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8680000000000005e-06 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253200+0000", 2024-03-30T15:03:45.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.038753864 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253307+0000", 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010676299999999999 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253432+0000", 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124957 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.266525+0000", 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013092763 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.268732+0000", 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022073129999999998 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303646+0000", 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034913911999999998 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303698+0000", 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1986e-05 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303741+0000", 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3078000000000003e-05 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25075.0:2 39.1b 39:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 349.37931645700002, 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0594717039999999, 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25075", 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3988199589", 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243277+0000", 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.796e-06 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243283+0000", 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2740000000000004e-06 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243299+0000", 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6532e-05 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243310+0000", 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0463e-05 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243383+0000", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3207000000000006e-05 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243392+0000", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8559999999999996e-06 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267447+0000", 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.024054757 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267568+0000", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012110199999999999 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267732+0000", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016446500000000001 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280028+0000", 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012295451000000001 2024-03-30T15:03:45.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.284857+0000", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048288619999999997 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302648+0000", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017791609999999999 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302697+0000", 2024-03-30T15:03:45.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9116999999999997e-05 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302747+0000", 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9212000000000002e-05 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24853.0:7 14.1d 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:03:45.664 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 348.45806561299997, 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.092228541999999997, 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24853", 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164528+0000", 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.328e-06 2024-03-30T15:03:45.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164538+0000", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0189e-05 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164550+0000", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1353e-05 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164560+0000", 2024-03-30T15:03:45.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0828e-05 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167851+0000", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032908939999999999 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167956+0000", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010505699999999999 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256754+0000", 2024-03-30T15:03:45.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088797893000000003 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35405.0:2 62.13 62:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 343.63282726599999, 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.977776094, 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35405", 2024-03-30T15:03:45.670 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1251877021", 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989767+0000", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.615e-06 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989773+0000", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2430000000000002e-06 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989784+0000", 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1043000000000001e-05 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989793+0000", 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5279999999999992e-06 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511344+0000", 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.52155030499999999 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511352+0000", 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4479999999999994e-06 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949501+0000", 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.43814895 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949615+0000", 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114282 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949747+0000", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000131564 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956142+0000", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063947639999999998 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956280+0000", 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000138496 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967445+0000", 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011165029999999999 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967488+0000", 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2397000000000001e-05 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967540+0000", 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2429000000000002e-05 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:483 115.4s0 115:2235e3a1:::482:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:03:45.677 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 321.36361411199999, 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15220373500000001, 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 483 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:03:45.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258980+0000", 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5359999999999999e-06 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258989+0000", 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281859+0000", 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022870352 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281876+0000", 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6492000000000001e-05 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338663+0000", 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056787694 2024-03-30T15:03:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338791+0000", 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012725400000000001 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.339139+0000", 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034834299999999998 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:03:45.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.368274+0000", 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029135306 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411112+0000", 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.042837274000000002 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411181+0000", 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9336000000000002e-05 2024-03-30T15:03:45.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:2 123.15 123:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 321.258267289, 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2153437069999999, 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T15:03:45.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364326+0000", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.435e-06 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364336+0000", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6139999999999998e-06 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364348+0000", 2024-03-30T15:03:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2194e-05 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364360+0000", 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1511e-05 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364419+0000", 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8956999999999998e-05 2024-03-30T15:03:45.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364429+0000", 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0026e-05 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538529+0000", 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.174100744 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538649+0000", 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011977900000000001 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538787+0000", 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000137458 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.541812+0000", 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030249740000000002 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.545225+0000", 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034129009999999999 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571213+0000", 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025988165000000001 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571265+0000", 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2385000000000001e-05 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.579668+0000", 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084025639999999995 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:45 smithi028 ceph-mon[41614]: pgmap v612: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:45.692 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:715 123.1f 123:fbee941d:::test_708:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e143)", 2024-03-30T15:03:45.692 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 316.78782085400002, 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146558462, 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 715 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:03:45.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834773+0000", 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.57e-06 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834784+0000", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1277999999999999e-05 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834803+0000", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8281999999999999e-05 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834813+0000", 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.078e-05 2024-03-30T15:03:45.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834910+0000", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6772000000000001e-05 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835058+0000", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148066 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.981329+0000", 2024-03-30T15:03:45.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146270714 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.2 epoch 146/143 rep_tid 752 entries 146'28 (0'0) error 130:40466eb8:::test_509:head by client.46235.0:516 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 145'26)", 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 314.38826143799997, 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024038321000000001, 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:03:45.697 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:03:45.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.72e-07 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234339+0000", 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9590000000000001e-06 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234353+0000", 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3560000000000001e-05 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234368+0000", 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5308e-05 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.241947+0000", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007578904 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.258368+0000", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016421318000000001 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.46235.0:517 130.5 130:a3243b6a:::test_510:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:03:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 314.352456061, 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023663910999999999, 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.46235", 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 517 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270137+0000", 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.356e-06 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270143+0000", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5889999999999999e-06 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270151+0000", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5000000000000002e-06 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270160+0000", 2024-03-30T15:03:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6399999999999992e-06 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270522+0000", 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036188600000000001 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270653+0000", 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000130791 2024-03-30T15:03:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.293799+0000", 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023146149000000001 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42667.0:2 138.0 138:0384ab8e:::oid:head [write 0~7,setxattr key1 (5) in=4b,truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:03:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 313.29298638099999, 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93406830600000001, 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42667", 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2528295921", 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329607+0000", 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.874e-06 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329614+0000", 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1169999999999998e-06 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329624+0000", 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0457e-05 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329636+0000", 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1382999999999999e-05 2024-03-30T15:03:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350953+0000", 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021317545 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350963+0000", 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193e-05 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.249891+0000", 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89892724199999996 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250074+0000", 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018359100000000001 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250291+0000", 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021668499999999999 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252632+0000", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023406579999999998 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257757+0000", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051254320000000001 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263567+0000", 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058095780000000001 2024-03-30T15:03:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263628+0000", 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1237999999999999e-05 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263673+0000", 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5312999999999998e-05 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42271.0:2 136.6 136:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 312.445632977, 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.090549882999999998, 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42271", 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/553852123", 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176961+0000", 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-06 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176971+0000", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9720000000000004e-06 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176993+0000", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2331e-05 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.177006+0000", 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3244999999999999e-05 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198821+0000", 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021814229000000001 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198830+0000", 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8740000000000001e-06 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250126+0000", 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.051296025000000002 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250242+0000", 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116638 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250389+0000", 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014667799999999999 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252485+0000", 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020958959999999999 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257681+0000", 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005195987 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267377+0000", 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096964649999999996 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267466+0000", 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8781000000000003e-05 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267508+0000", 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2174e-05 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.43429.0:2 140.7 140:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 311.31164060899999, 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.059809913999999999, 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.43429", 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1777674724", 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:45 smithi028 ceph-mon[29876]: pgmap v612: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310953+0000", 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2840000000000001e-06 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310961+0000", 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7340000000000008e-06 2024-03-30T15:03:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313256+0000", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022953600000000002 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313269+0000", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2472e-05 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313347+0000", 2024-03-30T15:03:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8175000000000003e-05 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313356+0000", 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9029999999999993e-06 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316618+0000", 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003262587 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316732+0000", 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011378 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316884+0000", 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015224100000000001 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.332636+0000", 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015752226000000001 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.342429+0000", 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097926090000000007 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370655+0000", 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.028226161999999999 2024-03-30T15:03:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370712+0000", 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6952e-05 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370761+0000", 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8429e-05 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24384.0:1 254.6s0 254:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 178.389296863, 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.003921587, 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24384", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3107101299", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233296+0000", 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8679999999999999e-06 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233304+0000", 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5299999999999999e-06 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233315+0000", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1473e-05 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233327+0000", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1598e-05 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233890+0000", 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056354599999999995 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233894+0000", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3400000000000002e-06 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.232901+0000", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99900681199999997 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233056+0000", 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000155811 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233360+0000", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000303982 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233629+0000", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000268712 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.237139+0000", 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035102390000000001 2024-03-30T15:03:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.237216+0000", 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6675999999999996e-05 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24865.0:1050 17.c 17:30000000::::head [pgnls start_epoch 300] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e300)", 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 160.116373318, 2024-03-30T15:03:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91512182399999997, 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24865", 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3133651075", 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1050 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506219+0000", 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5000000000000001e-07 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506223+0000", 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4689999999999999e-06 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506232+0000", 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6540000000000005e-06 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506239+0000", 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.148e-06 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506281+0000", 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1637000000000001e-05 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506287+0000", 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4649999999999999e-06 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421133+0000", 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91484526300000002 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421177+0000", 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4280000000000003e-05 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421340+0000", 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016295799999999999 2024-03-30T15:03:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:3042 261.1f 261:f8f7c133:test-rados-api-smithi028-93253-36::foo:head [writefull 0~1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 157.39141506199999, 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027248054000000001, 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:03:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3042 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231181+0000", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5569999999999997e-06 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231199+0000", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8062000000000001e-05 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231227+0000", 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8402999999999999e-05 2024-03-30T15:03:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231242+0000", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4956999999999999e-05 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231319+0000", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6625999999999994e-05 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231326+0000", 2024-03-30T15:03:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.832e-06 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.242303+0000", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010977551 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for blocked object", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.242479+0000", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000175878 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243461+0000", 2024-03-30T15:03:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098175100000000011 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243576+0000", 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115201 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243753+0000", 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017675199999999999 2024-03-30T15:03:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.244990+0000", 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001236975 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.249284+0000", 2024-03-30T15:03:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042942049999999997 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258343+0000", 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090588839999999997 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258384+0000", 2024-03-30T15:03:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1437999999999999e-05 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258424+0000", 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9981999999999998e-05 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:3240 267.4 267:20000000:%01:::head [pgnls start_epoch 316] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e316)", 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 144.371465406, 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21607105099999999, 2024-03-30T15:03:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3240 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251129+0000", 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7829999999999999e-06 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251138+0000", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4029999999999996e-06 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251152+0000", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4414e-05 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251162+0000", 2024-03-30T15:03:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3789999999999995e-06 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251207+0000", 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5121999999999999e-05 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:03:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251211+0000", 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0770000000000001e-06 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467004+0000", 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21579247800000001 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467041+0000", 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7539000000000001e-05 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467197+0000", 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015585600000000001 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57695.0:1 268.2s1 268:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 143.390051554, 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.026729614, 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:03:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57695", 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2330167966", 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232541+0000", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5269999999999999e-06 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232555+0000", 2024-03-30T15:03:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3358e-05 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232566+0000", 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0951000000000001e-05 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232576+0000", 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0631e-05 2024-03-30T15:03:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232958+0000", 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038161199999999998 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232962+0000", 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8369999999999999e-06 2024-03-30T15:03:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255130+0000", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0221682700000001 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255262+0000", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132003 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255735+0000", 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047280999999999999 2024-03-30T15:03:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.257894+0000", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021590899999999998 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.258958+0000", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010644039999999999 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.259269+0000", 2024-03-30T15:03:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031112099999999998 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:45.766 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:03:45.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:45.770 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:45 smithi134 ceph-mon[33714]: pgmap v612: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:03:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:45.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:45.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:45.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:46.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:46.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:46.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:46.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:46.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:46.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:46.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:03:46.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:46.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:46.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:46.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:46 smithi028 ceph-mon[29876]: pgmap v613: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:46.625 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:46 smithi028 ceph-mon[41614]: pgmap v613: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:46.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:46 smithi134 ceph-mon[33714]: pgmap v613: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:46.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:46.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:46.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:46.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:46.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:47.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:47.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:47.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:47.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:47.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:47.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:47.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:47.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:47.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:47.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:47.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:47.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:47.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:47.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:47.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:48.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:48.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:48.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:48.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:48.347 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:48 smithi028 ceph-mon[41614]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T15:03:48.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:48 smithi028 ceph-mon[29876]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T15:03:48.348 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:48.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:48.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:48.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:48.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:48 smithi134 ceph-mon[33714]: osdmap e443: 8 total, 8 up, 8 in 2024-03-30T15:03:48.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:48.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:48.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- 172.21.15.28:0/527706104 >> v1:172.21.15.28:6790/0 conn(0x7fec98115290 legacy=0x7fec98117680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- 172.21.15.28:0/527706104 shutdown_connections 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- 172.21.15.28:0/527706104 >> 172.21.15.28:0/527706104 conn(0x7fec980fc9a0 msgr2=0x7fec980fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- 172.21.15.28:0/527706104 shutdown_connections 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- 172.21.15.28:0/527706104 wait complete. 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 librados: starting msgr at 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 librados: starting objecter 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 Processor -- start 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 -- start start 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 librados: setting wanted keys 2024-03-30T15:03:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.847+0000 7fec9d678700 1 librados: calling monclient init 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec9d678700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec98114ef0 con 0x7fec98115290 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec9d678700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec981aec50 con 0x7fec98106d80 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec9d678700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec981afe30 con 0x7fec98100f80 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec96ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fec98100f80 0x7fec98111d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34432/0 (socket says 172.21.15.28:34432) 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec96ffd700 1 -- 172.21.15.28:0/3013189885 learned_addr learned my addr 172.21.15.28:0/3013189885 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter ms_handle_connect 0x7fec98100f80 2024-03-30T15:03:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter ms_handle_connect 0x7fec98115290 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter ms_handle_connect 0x7fec98106d80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571010328 0 0) 0x7fec981afe30 con 0x7fec98100f80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec800031f0 con 0x7fec98100f80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036450011 0 0) 0x7fec98114ef0 con 0x7fec98115290 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec981afe30 con 0x7fec98115290 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1771656278 0 0) 0x7fec981aec50 con 0x7fec98106d80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec98114ef0 con 0x7fec98106d80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3382507349 0 0) 0x7fec800031f0 con 0x7fec98100f80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec981aec50 con 0x7fec98100f80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec88001930 con 0x7fec98100f80 2024-03-30T15:03:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3314110450 0 0) 0x7fec981afe30 con 0x7fec98115290 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec800031f0 con 0x7fec98115290 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec900017b0 con 0x7fec98115290 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1366467904 0 0) 0x7fec98114ef0 con 0x7fec98106d80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec981afe30 con 0x7fec98106d80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec8c0018b0 con 0x7fec98106d80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.848+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3143200600 0 0) 0x7fec981aec50 con 0x7fec98100f80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 >> v1:172.21.15.134:6789/0 conn(0x7fec98106d80 legacy=0x7fec98112450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 >> v1:172.21.15.28:6789/0 conn(0x7fec98115290 legacy=0x7fec981ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec981b1010 con 0x7fec98100f80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec981adca0 con 0x7fec98100f80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec9d678700 10 client.58043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fec981ae160 con 0x7fec98100f80 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec9d678700 1 librados: init done 2024-03-30T15:03:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fec88003b40 con 0x7fec98100f80 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.849+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec88005ca0 con 0x7fec98100f80 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9806ae90 con 0x7fec98100f80 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fec8801c340 con 0x7fec98100f80 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(443..443 src has 53..443) ==== 8551+0+0 (unknown 900227648 0 0) 0x7fec880898d0 con 0x7fec98100f80 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec7ffff700 10 client.58043.objecter ms_dispatch 0x7fec98001550 osd_map(443..443 src has 53..443) 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec7ffff700 3 client.58043.objecter handle_osd_map got epochs [443,443] > 0 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.850+0000 7fec7ffff700 3 client.58043.objecter handle_osd_map decoding full epoch 443 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.851+0000 7fec7ffff700 20 client.58043.objecter dump_active .. 0 homeless 2024-03-30T15:03:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.851+0000 7fec7ffff700 10 client.58043.objecter ms_handle_connect 0x7fec80070480 2024-03-30T15:03:48.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:48.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:48.855+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fec88056960 con 0x7fec98100f80 2024-03-30T15:03:48.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:48.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:48.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:49.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.067+0000 7fc9eb155700 1 -- 172.21.15.28:0/287337680 >> v1:172.21.15.28:6790/0 conn(0x7fc9e410ea70 legacy=0x7fc9e4110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.067+0000 7fc9eb155700 1 -- 172.21.15.28:0/287337680 shutdown_connections 2024-03-30T15:03:49.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.067+0000 7fc9eb155700 1 -- 172.21.15.28:0/287337680 >> 172.21.15.28:0/287337680 conn(0x7fc9e40fdaf0 msgr2=0x7fc9e40fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.067+0000 7fc9eb155700 1 -- 172.21.15.28:0/287337680 shutdown_connections 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.067+0000 7fc9eb155700 1 -- 172.21.15.28:0/287337680 wait complete. 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 librados: starting msgr at 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 librados: starting objecter 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 Processor -- start 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 -- start start 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 librados: setting wanted keys 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.068+0000 7fc9eb155700 1 librados: calling monclient init 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9eb155700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9e41a7160 con 0x7fc9e410aed0 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9eb155700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9e41a8380 con 0x7fc9e410ea70 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e8ef1700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc9e4102240 0x7fc9e41075f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34456/0 (socket says 172.21.15.28:34456) 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e8ef1700 1 -- 172.21.15.28:0/3215098166 learned_addr learned my addr 172.21.15.28:0/3215098166 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9e41a95c0 con 0x7fc9e4102240 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter ms_handle_connect 0x7fc9e4102240 2024-03-30T15:03:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter ms_handle_connect 0x7fc9e410aed0 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter ms_handle_connect 0x7fc9e410ea70 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458063560 0 0) 0x7fc9e41a95c0 con 0x7fc9e4102240 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9d40031f0 con 0x7fc9e4102240 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2888035799 0 0) 0x7fc9e41a7160 con 0x7fc9e410aed0 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e41a95c0 con 0x7fc9e410aed0 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872792142 0 0) 0x7fc9e41a8380 con 0x7fc9e410ea70 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e41a7160 con 0x7fc9e410ea70 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2417746352 0 0) 0x7fc9d40031f0 con 0x7fc9e4102240 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e41a8380 con 0x7fc9e4102240 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9d8001930 con 0x7fc9e4102240 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2537460551 0 0) 0x7fc9e41a7160 con 0x7fc9e410ea70 2024-03-30T15:03:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9d40031f0 con 0x7fc9e410ea70 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9dc001bc0 con 0x7fc9e410ea70 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 175331397 0 0) 0x7fc9e41a95c0 con 0x7fc9e410aed0 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e41a7160 con 0x7fc9e410aed0 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.069+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9d0001130 con 0x7fc9e410aed0 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2224782588 0 0) 0x7fc9e41a8380 con 0x7fc9e4102240 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.134:6789/0 conn(0x7fc9e410ea70 legacy=0x7fc9e41a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6789/0 conn(0x7fc9e410aed0 legacy=0x7fc9e41a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e41aa800 con 0x7fc9e4102240 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e41a7390 con 0x7fc9e4102240 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc9d8003b40 con 0x7fc9e4102240 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9d8005d90 con 0x7fc9e4102240 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9eb155700 10 client.58049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:49.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e41a7850 con 0x7fc9e4102240 2024-03-30T15:03:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.070+0000 7fc9eb155700 1 librados: init done 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.071+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc9e4103720 con 0x7fc9e4102240 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.071+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc9d80036a0 con 0x7fc9e4102240 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(443..443 src has 53..443) ==== 8551+0+0 (unknown 900227648 0 0) 0x7fc9d8089a10 con 0x7fc9e4102240 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 osd_map(443..443 src has 53..443) 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 3 client.58049.objecter handle_osd_map got epochs [443,443] > 0 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 3 client.58049.objecter handle_osd_map decoding full epoch 443 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 20 client.58049.objecter dump_active .. 0 homeless 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v443) ==== 54+0+16 (unknown 1789720051 0 2420000192) 0x7fc9d8089e50 con 0x7fc9e4102240 2024-03-30T15:03:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9d40703f0 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 10 client.58049.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 1 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 1 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 0 1 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 4 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 10 client.58049.objecter _send_command 1 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.072+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4105bd0 con 0x7fc9e41a8550 2024-03-30T15:03:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.073+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e41a8550 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.073+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fec98068b30 con 0x7fec98100f80 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.074+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4105bd0 con 0x7fc9e41a8550 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.074+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.074+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 1 = system:0 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.074+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 4 2024-03-30T15:03:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.074+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 0 1 2024-03-30T15:03:49.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 10 client.58049.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 2 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 2 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e4112a20 osd=0 3 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 0 2 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 4 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 10 client.58049.objecter _send_command 2 2024-03-30T15:03:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.097+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e4068b30 con 0x7fc9e41a8550 2024-03-30T15:03:49.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.098+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4068b30 con 0x7fc9e41a8550 2024-03-30T15:03:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.098+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.098+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.098+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 4 2024-03-30T15:03:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.098+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 0 2 2024-03-30T15:03:49.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 10 client.58049.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 3 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 3 2024-03-30T15:03:49.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 1 3 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 4 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 10 client.58049.objecter _send_command 3 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc9e402d9c0 con 0x7fc9e410f3d0 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.099+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e410f3d0 2024-03-30T15:03:49.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.100+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e402d9c0 con 0x7fc9e410f3d0 2024-03-30T15:03:49.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.100+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:49.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.100+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 3 = system:0 2024-03-30T15:03:49.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.100+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 4 2024-03-30T15:03:49.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.100+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 1 3 2024-03-30T15:03:49.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 10 client.58049.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 4 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 4 2024-03-30T15:03:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41a9290 osd=1 3 2024-03-30T15:03:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 1 4 2024-03-30T15:03:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 4 2024-03-30T15:03:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 10 client.58049.objecter _send_command 4 2024-03-30T15:03:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.117+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e402ddc0 con 0x7fc9e410f3d0 2024-03-30T15:03:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e402ddc0 con 0x7fc9e410f3d0 2024-03-30T15:03:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 4 2024-03-30T15:03:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 1 4 2024-03-30T15:03:49.122 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:49.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 10 client.58049.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 5 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 5 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 2 5 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 4 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 10 client.58049.objecter _send_command 5 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.119+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4063290 con 0x7fc9e4062e50 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e4062e50 2024-03-30T15:03:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4063290 con 0x7fc9e4062e50 2024-03-30T15:03:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 5 = system:0 2024-03-30T15:03:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 4 2024-03-30T15:03:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.120+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 2 5 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 10 client.58049.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 6 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 6 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e402df70 osd=2 3 2024-03-30T15:03:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 2 6 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 4 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 10 client.58049.objecter _send_command 6 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.138+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e41b3360 con 0x7fc9e4062e50 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e41b3360 con 0x7fc9e4062e50 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 4 2024-03-30T15:03:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 2 6 2024-03-30T15:03:49.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 10 client.58049.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 7 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 7 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 3 7 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 4 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 10 client.58049.objecter _send_command 7 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.139+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc9e41af830 con 0x7fc9e4062a40 2024-03-30T15:03:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e4062a40 2024-03-30T15:03:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e41af830 con 0x7fc9e4062a40 2024-03-30T15:03:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 7 = system:0 2024-03-30T15:03:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 4 2024-03-30T15:03:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.140+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 3 7 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 10 client.58049.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 8 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 8 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e40628d0 osd=3 3 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 3 8 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 4 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 10 client.58049.objecter _send_command 8 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.158+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e41aed00 con 0x7fc9e4062a40 2024-03-30T15:03:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e41aed00 con 0x7fc9e4062a40 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 4 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 3 8 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 10 client.58049.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 9 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 9 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 4 9 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 4 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 10 client.58049.objecter _send_command 9 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc9e41aec20 con 0x7fc9e41b41f0 2024-03-30T15:03:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.160+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e41b41f0 2024-03-30T15:03:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.161+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e41aec20 con 0x7fc9e41b41f0 2024-03-30T15:03:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.161+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.161+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 9 = system:0 2024-03-30T15:03:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.161+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 4 2024-03-30T15:03:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.161+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 4 9 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 10 client.58049.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 10 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 10 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41aeab0 osd=4 3 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 4 10 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 4 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 10 client.58049.objecter _send_command 10 2024-03-30T15:03:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.178+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e41aec20 con 0x7fc9e41b41f0 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e41aec20 con 0x7fc9e41b41f0 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 4 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 4 10 2024-03-30T15:03:49.181 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 10 client.58049.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 11 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 11 2024-03-30T15:03:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 5 11 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 4 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 10 client.58049.objecter _send_command 11 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc9e41c0840 con 0x7fc9e41bcb80 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.180+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e41bcb80 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.181+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e41c0840 con 0x7fc9e41bcb80 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.181+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.181+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 11 = system:0 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.181+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 4 2024-03-30T15:03:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.181+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 5 11 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 10 client.58049.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 12 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 12 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41aec20 osd=5 3 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 5 12 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 4 2024-03-30T15:03:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 10 client.58049.objecter _send_command 12 2024-03-30T15:03:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.198+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e41c2220 con 0x7fc9e41bcb80 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e41c2220 con 0x7fc9e41bcb80 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 4 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 5 12 2024-03-30T15:03:49.201 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 10 client.58049.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 13 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 13 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 6 13 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 4 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 10 client.58049.objecter _send_command 13 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4076480 con 0x7fc9e41c2940 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.200+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e41c2940 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.201+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4076480 con 0x7fc9e41c2940 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.201+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.201+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 13 = system:0 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.201+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 4 2024-03-30T15:03:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.201+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 6 13 2024-03-30T15:03:49.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 10 client.58049.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 14 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 14 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e41c08a0 osd=6 3 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 6 14 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 4 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 10 client.58049.objecter _send_command 14 2024-03-30T15:03:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.218+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e4076480 con 0x7fc9e41c2940 2024-03-30T15:03:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4076480 con 0x7fc9e41c2940 2024-03-30T15:03:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 4 2024-03-30T15:03:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 6 14 2024-03-30T15:03:49.222 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 10 client.58049.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 15 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 15 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 7 15 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 4 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 10 client.58049.objecter _send_command 15 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.220+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc9e41cb360 con 0x7fc9e41c7820 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.221+0000 7fc9e1ffb700 10 client.58049.objecter ms_handle_connect 0x7fc9e41c7820 2024-03-30T15:03:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.222+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e41cb360 con 0x7fc9e41c7820 2024-03-30T15:03:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.222+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.222+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 15 = system:0 2024-03-30T15:03:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.222+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 4 2024-03-30T15:03:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.222+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 7 15 2024-03-30T15:03:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 10 client.58049.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign -1 16 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 20 client.58049.objecter _get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_remove -1 16 2024-03-30T15:03:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 20 client.58049.objecter get_session s=0x7fc9e4076480 osd=7 3 2024-03-30T15:03:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 15 client.58049.objecter _session_command_op_assign 7 16 2024-03-30T15:03:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 4 2024-03-30T15:03:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 10 client.58049.objecter _send_command 16 2024-03-30T15:03:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.239+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9e41b3760 con 0x7fc9e41c7820 2024-03-30T15:03:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.241+0000 7fc9e1ffb700 1 -- 172.21.15.28:0/3215098166 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e41b3760 con 0x7fc9e41c7820 2024-03-30T15:03:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.241+0000 7fc9e1ffb700 10 client.58049.objecter ms_dispatch 0x7fc9e4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.241+0000 7fc9e1ffb700 10 client.58049.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.241+0000 7fc9e1ffb700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 4 2024-03-30T15:03:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.241+0000 7fc9e1ffb700 15 client.58049.objecter _session_command_op_remove 7 16 2024-03-30T15:03:49.243 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 librados: watch_flush enter 2024-03-30T15:03:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 librados: watch_flush exit 2024-03-30T15:03:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.0 2024-03-30T15:03:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6801/3394323575 conn(0x7fc9e41a8550 legacy=0x7fc9e41a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4112a20 osd=0 1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6805/1850313547 conn(0x7fc9e410f3d0 legacy=0x7fc9e410f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41a9290 osd=1 1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.2 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6809/3451325043 conn(0x7fc9e4062e50 legacy=0x7fc9e41b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e402df70 osd=2 1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.3 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6813/1506157250 conn(0x7fc9e4062a40 legacy=0x7fc9e41b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e40628d0 osd=3 1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.4 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.134:6800/3063775514 conn(0x7fc9e41b41f0 legacy=0x7fc9e41baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aeab0 osd=4 1 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.5 2024-03-30T15:03:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9e96f2700 1 -- 172.21.15.28:0/3215098166 reap_dead start 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.134:6804/3668151179 conn(0x7fc9e41bcb80 legacy=0x7fc9e41bef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9e96f2700 1 -- 172.21.15.28:0/3215098166 reap_dead start 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9e96f2700 1 -- 172.21.15.28:0/3215098166 reap_dead start 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41aec20 osd=5 1 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.6 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.134:6808/2135224194 conn(0x7fc9e41c2940 legacy=0x7fc9e41c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e41c08a0 osd=6 1 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter close_session for osd.7 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.134:6812/3758190298 conn(0x7fc9e41c7820 legacy=0x7fc9e41c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter put_session s=0x7fc9e4076480 osd=7 1 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 20 client.58049.objecter shutdown clearing up homeless session... 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 10 client.58049.objecter successfully canceled tick 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc9d40703f0 legacy=0x7fc9d40728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> v1:172.21.15.28:6790/0 conn(0x7fc9e4102240 legacy=0x7fc9e41075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9e96f2700 1 -- 172.21.15.28:0/3215098166 reap_dead start 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 shutdown_connections 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 >> 172.21.15.28:0/3215098166 conn(0x7fc9e40fdaf0 msgr2=0x7fc9e4111f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 shutdown_connections 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 -- 172.21.15.28:0/3215098166 wait complete. 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.242+0000 7fc9eb155700 1 librados: shutdown 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:49.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.275+0000 7fec7ffff700 1 -- 172.21.15.28:0/3013189885 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v444) ==== 91+0+0 (unknown 1325488459 0 0) 0x7fec8805a3b0 con 0x7fec98100f80 2024-03-30T15:03:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 10 librados: watch_flush enter 2024-03-30T15:03:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 10 librados: watch_flush exit 2024-03-30T15:03:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 20 client.58043.objecter shutdown clearing up homeless session... 2024-03-30T15:03:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 10 client.58043.objecter successfully canceled tick 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 >> v1:172.21.15.28:6800/3011431093 conn(0x7fec80070480 legacy=0x7fec80072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 >> v1:172.21.15.28:6790/0 conn(0x7fec98100f80 legacy=0x7fec98111d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 shutdown_connections 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 >> 172.21.15.28:0/3013189885 conn(0x7fec980fc9a0 msgr2=0x7fec980fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 shutdown_connections 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 -- 172.21.15.28:0/3013189885 wait complete. 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:49.278+0000 7fec9d678700 1 librados: shutdown 2024-03-30T15:03:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:03:49.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:49.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[29876]: pgmap v615: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3215098166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3013189885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[29876]: from='client.58043 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[41614]: pgmap v615: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3215098166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3013189885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:49 smithi028 ceph-mon[41614]: from='client.58043 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:03:49.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:03:49.388 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T15:03:49.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:03:49.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:49.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:49.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:49.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:49 smithi134 ceph-mon[33714]: pgmap v615: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3215098166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3013189885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:49 smithi134 ceph-mon[33714]: from='client.58043 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:03:49.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:49.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:49.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:49.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:49.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:49.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:49.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:50.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:50.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:50.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:50.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:50.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:50.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: from='client.58043 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:50.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: 101.c scrub starts 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: 101.c scrub ok 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[41614]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: from='client.58043 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T15:03:50.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: 101.c scrub starts 2024-03-30T15:03:50.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: 101.c scrub ok 2024-03-30T15:03:50.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:50 smithi028 ceph-mon[29876]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:50.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:50.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:50.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:50.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:50.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:03:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:03:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: from='client.58043 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:03:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: osdmap e444: 8 total, 8 up, 8 in 2024-03-30T15:03:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: 101.c scrub starts 2024-03-30T15:03:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: 101.c scrub ok 2024-03-30T15:03:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:50 smithi134 ceph-mon[33714]: pgmap v617: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:03:50.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:50.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:50.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:50.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:50.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:50.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:50.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:51.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:51.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:51.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:51.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 101.1 scrub starts 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 101.1 scrub ok 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 101.1d scrub starts 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 101.1d scrub ok 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 101.1 scrub starts 2024-03-30T15:03:51.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 101.1 scrub ok 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 101.1d scrub starts 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 101.1d scrub ok 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T15:03:51.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:03:51.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:51.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 101.1 scrub starts 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 101.1 scrub ok 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 101.1d scrub starts 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 101.1d scrub ok 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T15:03:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T15:03:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:03:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:51.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:51.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:51.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:51.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:52.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:52.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:52.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:52.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:52.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:52.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:52.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:52.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:52.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: 101.5 scrub starts 2024-03-30T15:03:52.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: 101.5 scrub ok 2024-03-30T15:03:52.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:03:52.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:03:52.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: 101.5 scrub starts 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: 101.5 scrub ok 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:03:52.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:52.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:52.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: 101.5 scrub starts 2024-03-30T15:03:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: 101.5 scrub ok 2024-03-30T15:03:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:03:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:03:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:03:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: pgmap v618: 260 pgs: 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:03:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:03:52.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:52.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:03:52.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:52.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:52.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:52.904 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:03:52.904 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:03:52.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:53.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:53.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:53.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:53.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:03:53.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:53.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:53.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:53.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T15:03:53.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T15:03:53.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:53 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T15:03:53.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:53.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:53.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:03:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T15:03:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:53 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T15:03:53.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:53.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:53.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:53.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:53.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:54.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:54.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:54.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:54.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:54.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:54.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:54.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:03:54.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:54.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:54.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:54.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[29876]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:03:54.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:54 smithi028 ceph-mon[41614]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:03:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:54.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- 172.21.15.28:0/3391712526 >> v1:172.21.15.28:6789/0 conn(0x7f629810eac0 legacy=0x7f6298110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- 172.21.15.28:0/3391712526 shutdown_connections 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- 172.21.15.28:0/3391712526 >> 172.21.15.28:0/3391712526 conn(0x7f62980761e0 msgr2=0x7f62980765f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- 172.21.15.28:0/3391712526 shutdown_connections 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- 172.21.15.28:0/3391712526 wait complete. 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 librados: starting msgr at 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 librados: starting objecter 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 Processor -- start 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- start start 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 librados: setting wanted keys 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 librados: calling monclient init 2024-03-30T15:03:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f629819eee0 con 0x7f6298100ac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f629819f0f0 con 0x7f629810af50 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.630+0000 7f629f9ad700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62981afba0 con 0x7f629810eac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629d749700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6298100ac0 0x7f6298107530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53600/0 (socket says 172.21.15.28:53600) 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629d749700 1 -- 172.21.15.28:0/1004008904 learned_addr learned my addr 172.21.15.28:0/1004008904 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter ms_handle_connect 0x7f6298100ac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter ms_handle_connect 0x7f629810eac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter ms_handle_connect 0x7f629810af50 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246685015 0 0) 0x7f629819eee0 con 0x7f6298100ac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62840039c0 con 0x7f6298100ac0 2024-03-30T15:03:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980094335 0 0) 0x7f62981afba0 con 0x7f629810eac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f629819eee0 con 0x7f629810eac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922871270 0 0) 0x7f629819f0f0 con 0x7f629810af50 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62981afba0 con 0x7f629810af50 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1873320713 0 0) 0x7f62840039c0 con 0x7f6298100ac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f629819f0f0 con 0x7f6298100ac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6290001d30 con 0x7f6298100ac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1820284575 0 0) 0x7f629819eee0 con 0x7f629810eac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62840039c0 con 0x7f629810eac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62940043b0 con 0x7f629810eac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 801585545 0 0) 0x7f62981afba0 con 0x7f629810af50 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f629819eee0 con 0x7f629810af50 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2061532045 0 0) 0x7f629819f0f0 con 0x7f6298100ac0 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 >> v1:172.21.15.28:6790/0 conn(0x7f629810eac0 legacy=0x7f629819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 >> v1:172.21.15.134:6789/0 conn(0x7f629810af50 legacy=0x7f629819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62981b0e00 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62981afdd0 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629f9ad700 10 client.24861.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62981b0290 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f629f9ad700 1 librados: init done 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.631+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6290002030 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.632+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6290006140 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.632+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f627c005120 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.633+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6290003d50 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.633+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(444..444 src has 53..444) ==== 8579+0+0 (unknown 4082456312 0 0) 0x7f6290089ff0 con 0x7f6298100ac0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.633+0000 7f628e7fc700 10 client.24861.objecter ms_dispatch 0x7f6298001550 osd_map(444..444 src has 53..444) 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.633+0000 7f628e7fc700 3 client.24861.objecter handle_osd_map got epochs [444,444] > 0 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.633+0000 7f628e7fc700 3 client.24861.objecter handle_osd_map decoding full epoch 444 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.634+0000 7f628e7fc700 20 client.24861.objecter dump_active .. 0 homeless 2024-03-30T15:03:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.634+0000 7f628e7fc700 10 client.24861.objecter ms_handle_connect 0x7f6284070530 2024-03-30T15:03:54.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.638+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6290057060 con 0x7f6298100ac0 2024-03-30T15:03:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:54 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:03:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:54 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:03:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:54 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:03:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:54 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:03:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:54 smithi134 ceph-mon[33714]: pgmap v619: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:54.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:54.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:54.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.865+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f627c005410 con 0x7f6298100ac0 2024-03-30T15:03:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.868+0000 7f628e7fc700 1 -- 172.21.15.28:0/1004008904 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v444) ==== 81+0+56224 (unknown 1611544978 0 171834375) 0x7f629005aab0 con 0x7f6298100ac0 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 444, 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:03:49.270854+0000", 2024-03-30T15:03:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 112, 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:03:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 296, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.873 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.874 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "437", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.875 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.876 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.877 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.879 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.881 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.884 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.885 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.886 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.887 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.888 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.890 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.892 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.895 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.896 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:03:54.897 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.898 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.899 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.900 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.901 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-03-30T15:03:54.902 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.904 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.905 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.906 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.907 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.908 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.910 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.911 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.912 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.913 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:03:54.914 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.915 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.916 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.919 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.920 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.921 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.922 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "444", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "411", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "411", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "411", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 17, 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 444, 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.923 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.924 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.925 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:03:54.926 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 296, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-63", 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:03:20.531707+0000", 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:03:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:03:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "432", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:03:54.929 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:03:54.930 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:03:54.931 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.933 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 10 librados: watch_flush enter 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 10 librados: watch_flush exit 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 20 client.24861.objecter shutdown clearing up homeless session... 2024-03-30T15:03:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 10 client.24861.objecter successfully canceled tick 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6284070530 legacy=0x7f62840729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 >> v1:172.21.15.28:6789/0 conn(0x7f6298100ac0 legacy=0x7f6298107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 shutdown_connections 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 >> 172.21.15.28:0/1004008904 conn(0x7f62980761e0 msgr2=0x7f6298111ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 shutdown_connections 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 -- 172.21.15.28:0/1004008904 wait complete. 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:54.870+0000 7f629f9ad700 1 librados: shutdown 2024-03-30T15:03:54.936 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:03:54.938 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.939 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.940 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:03:54.941 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:03:54.942 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.943 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.944 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.945 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.946 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.947 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 417, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:03:54.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.951 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:03:54.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:03:54.955 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:03:54.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.959 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:03:54.960 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:03:54.962 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:03:54.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:03:54.964 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 430, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.967 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.968 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:04.098499+0000", 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:13.415223+0000", 2024-03-30T15:03:54.969 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:41.445505+0000", 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:03:54.970 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:56.548276+0000", 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.971 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:47.348318+0000", 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:29.093089+0000", 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.972 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:35.216771+0000", 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:03:54.973 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:57.539763+0000", 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:03:54.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:03:54.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:03:54.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:03:54.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:03:54.979 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 16, 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.981 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:03:54.983 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:03:54.986 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:03:54.987 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.988 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:03:54.989 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:03:54.991 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:03:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [ 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 16, 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:03:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:03:54.994 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:03:54.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:55.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:55.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:55.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:55.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:55.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:55.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:55.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:55.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 101.e scrub starts 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 101.e scrub ok 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1004008904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T15:03:55.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 101.e scrub starts 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 101.e scrub ok 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1004008904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T15:03:55.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:55 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T15:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 101.e scrub starts 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 101.e scrub ok 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:03:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:03:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1004008904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:03:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T15:03:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:55 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T15:03:55.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:55.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:55.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:55.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:56.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:56.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:56.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:03:56.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:03:56.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:56.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[41614]: 42.16 scrub starts 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[41614]: 42.16 scrub ok 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[41614]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[29876]: 42.16 scrub starts 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[29876]: 42.16 scrub ok 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T15:03:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:56 smithi028 ceph-mon[29876]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:03:56.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:56.574 INFO:tasks.thrashosds.thrasher:j is {'epoch': 444, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:03:49.270854+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 112, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 296, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '437', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '444', 'last_force_op_resend': '411', 'last_force_op_resend_prenautilus': '411', 'last_force_op_resend_preluminous': '411', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 444, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 296, 'pool_name': 'test-rados-api-smithi028-93253-63', 'create_time': '2024-03-30T15:03:20.531707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '432', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:04.098499+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:13.415223+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:41.445505+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:56.548276+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:47.348318+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:29.093089+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:35.216771+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:57.539763+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 101, 'snaps': [{'begin': 16, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [{'pool': 101, 'snaps': [{'begin': 16, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:03:56.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:03:56.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:56.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:56 smithi134 ceph-mon[33714]: 42.16 scrub starts 2024-03-30T15:03:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:56 smithi134 ceph-mon[33714]: 42.16 scrub ok 2024-03-30T15:03:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:56 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T15:03:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:56 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T15:03:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:56 smithi134 ceph-mon[33714]: pgmap v620: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:03:56.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:56.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:03:56.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:56.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:57.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:57.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:57.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:57.229 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:57.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:57.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:57.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:57.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:57.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:57.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:57.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:57 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T15:03:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:57 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T15:03:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:57 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T15:03:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:57 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T15:03:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:57 smithi134 ceph-mon[33714]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[29876]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T15:03:57.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:57 smithi028 ceph-mon[41614]: osdmap e445: 8 total, 8 up, 8 in 2024-03-30T15:03:57.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:57.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:57.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:57.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:57.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:58.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:58.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:58.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:03:58.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:03:58.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:03:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:03:58.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:58.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:03:58.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:58.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: 42.2 scrub starts 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: 42.2 scrub ok 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-30T15:03:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:58 smithi134 ceph-mon[33714]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: 42.2 scrub starts 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: 42.2 scrub ok 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:03:58.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[41614]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: 42.2 scrub starts 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: 42.2 scrub ok 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:03:58.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:03:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:03:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:03:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: pgmap v622: 260 pgs: 260 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-30T15:03:58.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-mon[29876]: osdmap e446: 8 total, 8 up, 8 in 2024-03-30T15:03:58.684 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:03:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:03:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:03:58.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:58.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:58.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:58.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:58.965 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:58.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:59.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:59.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:59.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 -- 172.21.15.28:0/1947936923 >> v1:172.21.15.28:6789/0 conn(0x7fa1a410eb10 legacy=0x7fa1a4110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 -- 172.21.15.28:0/1947936923 shutdown_connections 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 -- 172.21.15.28:0/1947936923 >> 172.21.15.28:0/1947936923 conn(0x7fa1a40fc9e0 msgr2=0x7fa1a40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 -- 172.21.15.28:0/1947936923 shutdown_connections 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 -- 172.21.15.28:0/1947936923 wait complete. 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 librados: starting msgr at 2024-03-30T15:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.286+0000 7fa1a8890700 1 librados: starting objecter 2024-03-30T15:03:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 Processor -- start 2024-03-30T15:03:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 -- start start 2024-03-30T15:03:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 librados: setting wanted keys 2024-03-30T15:03:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 librados: calling monclient init 2024-03-30T15:03:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1a410aa00 con 0x7fa1a410b000 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1a410ac10 con 0x7fa1a41031f0 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a8890700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1a410ae20 con 0x7fa1a410eb10 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1a410b000 0x7fa1a4101560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51434/0 (socket says 172.21.15.28:51434) 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa1a2d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa1a410eb10 0x7fa1a41a8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56460/0 (socket says 172.21.15.28:56460) 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19bfff700 1 -- 172.21.15.28:0/1686600809 learned_addr learned my addr 172.21.15.28:0/1686600809 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter ms_handle_connect 0x7fa1a410b000 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter ms_handle_connect 0x7fa1a410eb10 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter ms_handle_connect 0x7fa1a41031f0 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772793865 0 0) 0x7fa1a410aa00 con 0x7fa1a410b000 2024-03-30T15:03:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1900031f0 con 0x7fa1a410b000 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7420788 0 0) 0x7fa1a4115f00 con 0x7fa1a410eb10 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.287+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a410aa00 con 0x7fa1a410eb10 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686560342 0 0) 0x7fa1a410ac10 con 0x7fa1a41031f0 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a4115f00 con 0x7fa1a41031f0 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4035883805 0 0) 0x7fa1900031f0 con 0x7fa1a410b000 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1a410ac10 con 0x7fa1a410b000 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa18c001a10 con 0x7fa1a410b000 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1766314012 0 0) 0x7fa1a410aa00 con 0x7fa1a410eb10 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1900031f0 con 0x7fa1a410eb10 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa19c004810 con 0x7fa1a410eb10 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1391798903 0 0) 0x7fa1a410ac10 con 0x7fa1a410b000 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6790/0 conn(0x7fa1a410eb10 legacy=0x7fa1a41a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.134:6789/0 conn(0x7fa1a41031f0 legacy=0x7fa1a4100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a41acac0 con 0x7fa1a410b000 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a41a9a90 con 0x7fa1a410b000 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa1a8890700 10 client.24867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1a41a9f50 con 0x7fa1a410b000 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa1a8890700 1 librados: init done 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.288+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa18c001cd0 con 0x7fa1a410b000 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.289+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa18c005d40 con 0x7fa1a410b000 2024-03-30T15:03:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.289+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa184005100 con 0x7fa1a410b000 2024-03-30T15:03:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.290+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa18c003930 con 0x7fa1a410b000 2024-03-30T15:03:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.290+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(446..446 src has 53..446) ==== 8523+0+0 (unknown 3562352666 0 0) 0x7fa18c0897a0 con 0x7fa1a410b000 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.290+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 osd_map(446..446 src has 53..446) 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.290+0000 7fa19b7fe700 3 client.24867.objecter handle_osd_map got epochs [446,446] > 0 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.290+0000 7fa19b7fe700 3 client.24867.objecter handle_osd_map decoding full epoch 446 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa19b7fe700 20 client.24867.objecter dump_active .. 0 homeless 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v446) ==== 54+0+16 (unknown 1274518014 0 2420000192) 0x7fa18c089b50 con 0x7fa1a410b000 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa190070160 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 10 client.24867.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 1 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 1 2024-03-30T15:03:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 0 1 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 4 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 10 client.24867.objecter _send_command 1 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa184004b50 con 0x7fa184004730 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.291+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa184004730 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.292+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa184004b50 con 0x7fa184004730 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.292+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 1: 0 ) 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.292+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 1 = system:0 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.292+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 4 2024-03-30T15:03:59.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.292+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 0 1 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 10 client.24867.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 2 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 2 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa1840053e0 osd=0 3 2024-03-30T15:03:59.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 0 2 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 4 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 10 client.24867.objecter _send_command 2 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.310+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa1840079f0 con 0x7fa184004730 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.311+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa1840079f0 con 0x7fa184004730 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.311+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.311+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.311+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 4 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.311+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 0 2 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 10 client.24867.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 3 2024-03-30T15:03:59.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 1 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 4 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 10 client.24867.objecter _send_command 3 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa184002180 con 0x7fa184004280 2024-03-30T15:03:59.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.312+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa184004280 2024-03-30T15:03:59.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.313+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa184002180 con 0x7fa184004280 2024-03-30T15:03:59.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.313+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 3: 0 ) 2024-03-30T15:03:59.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.313+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 3 = system:0 2024-03-30T15:03:59.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.313+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 4 2024-03-30T15:03:59.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.313+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 1 3 2024-03-30T15:03:59.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:03:59.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 10 client.24867.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 4 2024-03-30T15:03:59.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 4 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184004110 osd=1 3 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 1 4 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 4 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 10 client.24867.objecter _send_command 4 2024-03-30T15:03:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.330+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa184001600 con 0x7fa184004280 2024-03-30T15:03:59.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T15:03:59.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T15:03:59.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1686600809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:59.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T15:03:59.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T15:03:59.333 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:03:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1686600809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:59.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa184001600 con 0x7fa184004280 2024-03-30T15:03:59.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 4 2024-03-30T15:03:59.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 1 4 2024-03-30T15:03:59.334 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 10 client.24867.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 5 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 5 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 2 5 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 4 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 10 client.24867.objecter _send_command 5 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa1840110e0 con 0x7fa18400d5d0 2024-03-30T15:03:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.332+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa18400d5d0 2024-03-30T15:03:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.333+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa1840110e0 con 0x7fa18400d5d0 2024-03-30T15:03:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.333+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 5: 0 ) 2024-03-30T15:03:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.333+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 5 = system:0 2024-03-30T15:03:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.333+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 4 2024-03-30T15:03:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.333+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 2 5 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 10 client.24867.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 6 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 6 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184008370 osd=2 3 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 2 6 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 4 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 10 client.24867.objecter _send_command 6 2024-03-30T15:03:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.350+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa184007eb0 con 0x7fa18400d5d0 2024-03-30T15:03:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:59.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:03:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa184007eb0 con 0x7fa18400d5d0 2024-03-30T15:03:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 4 2024-03-30T15:03:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 2 6 2024-03-30T15:03:59.354 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 10 client.24867.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 7 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 7 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 3 7 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 4 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 10 client.24867.objecter _send_command 7 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.351+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa184015f20 con 0x7fa184012410 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa184012410 2024-03-30T15:03:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa184015f20 con 0x7fa184012410 2024-03-30T15:03:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 7: 0 ) 2024-03-30T15:03:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 7 = system:0 2024-03-30T15:03:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 4 2024-03-30T15:03:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.352+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 3 7 2024-03-30T15:03:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 10 client.24867.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 8 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 8 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400cc20 osd=3 3 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 3 8 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 4 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 10 client.24867.objecter _send_command 8 2024-03-30T15:03:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.369+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa18400c700 con 0x7fa184012410 2024-03-30T15:03:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa18400c700 con 0x7fa184012410 2024-03-30T15:03:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 4 2024-03-30T15:03:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 3 8 2024-03-30T15:03:59.373 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:03:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 10 client.24867.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 9 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 9 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 4 9 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 4 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 10 client.24867.objecter _send_command 9 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.371+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa18401aa20 con 0x7fa184011f70 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.372+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa184011f70 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.372+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa18401aa20 con 0x7fa184011f70 2024-03-30T15:03:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.372+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 9: 0 ) 2024-03-30T15:03:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.373+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 9 = system:0 2024-03-30T15:03:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.373+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 4 2024-03-30T15:03:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.373+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 4 9 2024-03-30T15:03:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 10 client.24867.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 10 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 10 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400c700 osd=4 3 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 4 10 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 4 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 10 client.24867.objecter _send_command 10 2024-03-30T15:03:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.389+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa18400c9c0 con 0x7fa184011f70 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.391+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa18400c9c0 con 0x7fa184011f70 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.391+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.391+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.391+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 4 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.391+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 4 10 2024-03-30T15:03:59.393 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 10 client.24867.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 11 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 11 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 5 11 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 4 2024-03-30T15:03:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 10 client.24867.objecter _send_command 11 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa18401c1b0 con 0x7fa1840168a0 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.392+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa1840168a0 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.393+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa18c056840 con 0x7fa1840168a0 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.393+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 11: 0 ) 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.393+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 11 = system:0 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.393+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 4 2024-03-30T15:03:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.393+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 5 11 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 10 client.24867.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 12 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 12 2024-03-30T15:03:59.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa18400c9c0 osd=5 3 2024-03-30T15:03:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 5 12 2024-03-30T15:03:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 4 2024-03-30T15:03:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 10 client.24867.objecter _send_command 12 2024-03-30T15:03:59.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.410+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa1840112f0 con 0x7fa1840168a0 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa1840112f0 con 0x7fa1840168a0 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 4 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 5 12 2024-03-30T15:03:59.413 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 10 client.24867.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 13 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 13 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 6 13 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 4 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 10 client.24867.objecter _send_command 13 2024-03-30T15:03:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa1840235e0 con 0x7fa18401fa90 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.412+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa18401fa90 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.413+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa1840235e0 con 0x7fa18401fa90 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.413+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 13: 0 ) 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.413+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 13 = system:0 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.413+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 4 2024-03-30T15:03:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.413+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 6 13 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 10 client.24867.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 14 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 14 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa184016ea0 osd=6 3 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 6 14 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 4 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 10 client.24867.objecter _send_command 14 2024-03-30T15:03:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.431+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa184016320 con 0x7fa18401fa90 2024-03-30T15:03:59.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.433+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa184016320 con 0x7fa18401fa90 2024-03-30T15:03:59.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.433+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.433+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.433+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 4 2024-03-30T15:03:59.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.433+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 6 14 2024-03-30T15:03:59.435 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:03:59.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 10 client.24867.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 15 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 15 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 7 15 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 4 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 10 client.24867.objecter _send_command 15 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa184016240 con 0x7fa18401cb40 2024-03-30T15:03:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.434+0000 7fa19b7fe700 10 client.24867.objecter ms_handle_connect 0x7fa18401cb40 2024-03-30T15:03:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.435+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa184016240 con 0x7fa18401cb40 2024-03-30T15:03:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.435+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 15: 0 ) 2024-03-30T15:03:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.435+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 15 = system:0 2024-03-30T15:03:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.435+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 4 2024-03-30T15:03:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.435+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 7 15 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 10 client.24867.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign -1 16 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 20 client.24867.objecter _get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_remove -1 16 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 20 client.24867.objecter get_session s=0x7fa1840160d0 osd=7 3 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 15 client.24867.objecter _session_command_op_assign 7 16 2024-03-30T15:03:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 4 2024-03-30T15:03:59.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 10 client.24867.objecter _send_command 16 2024-03-30T15:03:59.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.452+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa184016240 con 0x7fa18401cb40 2024-03-30T15:03:59.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa19b7fe700 1 -- 172.21.15.28:0/1686600809 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa184016240 con 0x7fa18401cb40 2024-03-30T15:03:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa19b7fe700 10 client.24867.objecter ms_dispatch 0x7fa1a4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:03:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa19b7fe700 10 client.24867.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:03:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa19b7fe700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 4 2024-03-30T15:03:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa19b7fe700 15 client.24867.objecter _session_command_op_remove 7 16 2024-03-30T15:03:59.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 librados: watch_flush enter 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 librados: watch_flush exit 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.0 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6801/3394323575 conn(0x7fa184004730 legacy=0x7fa184002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840053e0 osd=0 1 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.1 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6805/1850313547 conn(0x7fa184004280 legacy=0x7fa18400ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184004110 osd=1 1 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.2 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6809/3451325043 conn(0x7fa18400d5d0 legacy=0x7fa18400f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184008370 osd=2 1 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.3 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6813/1506157250 conn(0x7fa184012410 legacy=0x7fa184014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400cc20 osd=3 1 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.4 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.134:6800/3063775514 conn(0x7fa184011f70 legacy=0x7fa184019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c700 osd=4 1 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.5 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.134:6804/3668151179 conn(0x7fa1840168a0 legacy=0x7fa18401aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa18400c9c0 osd=5 1 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.6 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.134:6808/2135224194 conn(0x7fa18401fa90 legacy=0x7fa184021e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa184016ea0 osd=6 1 2024-03-30T15:03:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter close_session for osd.7 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.134:6812/3758190298 conn(0x7fa18401cb40 legacy=0x7fa184026f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter put_session s=0x7fa1840160d0 osd=7 1 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 20 client.24867.objecter shutdown clearing up homeless session... 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 10 client.24867.objecter successfully canceled tick 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a2d9d700 1 -- 172.21.15.28:0/1686600809 reap_dead start 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa190070160 legacy=0x7fa190072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a2d9d700 1 -- 172.21.15.28:0/1686600809 reap_dead start 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> v1:172.21.15.28:6789/0 conn(0x7fa1a410b000 legacy=0x7fa1a4101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a2d9d700 1 -- 172.21.15.28:0/1686600809 reap_dead start 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a2d9d700 1 -- 172.21.15.28:0/1686600809 reap_dead start 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a2d9d700 1 -- 172.21.15.28:0/1686600809 reap_dead start 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 shutdown_connections 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.454+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 >> 172.21.15.28:0/1686600809 conn(0x7fa1a40fc9e0 msgr2=0x7fa1a40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:03:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.455+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 shutdown_connections 2024-03-30T15:03:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.455+0000 7fa1a8890700 1 -- 172.21.15.28:0/1686600809 wait complete. 2024-03-30T15:03:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:03:59.455+0000 7fa1a8890700 1 librados: shutdown 2024-03-30T15:03:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:03:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:03:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:03:59.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:59.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:03:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:59.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:03:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:59 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T15:03:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:59 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T15:03:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:03:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1686600809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:03:59.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:03:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:59.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:03:59.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:03:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:03:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:03:59.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:03:59.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:03:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:03:59.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:00.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:00.122 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:00.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:00.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:00.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:04:00.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[41614]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:04:00.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:00 smithi028 ceph-mon[29876]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:04:00.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:00.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:00.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:00.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:04:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-63"}]: dispatch 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:04:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:00 smithi134 ceph-mon[33714]: pgmap v624: 260 pgs: 260 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:04:00.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:00.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:00.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:00.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:00.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:01.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:01.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.113+0000 7f934d46f700 1 -- 172.21.15.28:0/4223795688 >> v1:172.21.15.28:6789/0 conn(0x7f934810da20 legacy=0x7f934810fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 -- 172.21.15.28:0/4223795688 shutdown_connections 2024-03-30T15:04:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 -- 172.21.15.28:0/4223795688 >> 172.21.15.28:0/4223795688 conn(0x7f93480fcaa0 msgr2=0x7f93480feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 -- 172.21.15.28:0/4223795688 shutdown_connections 2024-03-30T15:04:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 -- 172.21.15.28:0/4223795688 wait complete. 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 librados: starting msgr at 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 librados: starting objecter 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.114+0000 7f934d46f700 1 Processor -- start 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 -- start start 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 librados: setting wanted keys 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 librados: calling monclient init 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93481a7030 con 0x7f9348109e80 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93481a8270 con 0x7f934810da20 2024-03-30T15:04:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f934d46f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93481a94b0 con 0x7f93481011f0 2024-03-30T15:04:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f9346ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f93481011f0 0x7f934806ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56484/0 (socket says 172.21.15.28:56484) 2024-03-30T15:04:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f9346ffd700 1 -- 172.21.15.28:0/1395796025 learned_addr learned my addr 172.21.15.28:0/1395796025 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter ms_handle_connect 0x7f93481011f0 2024-03-30T15:04:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter ms_handle_connect 0x7f9348109e80 2024-03-30T15:04:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter ms_handle_connect 0x7f934810da20 2024-03-30T15:04:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897787806 0 0) 0x7f93481a94b0 con 0x7f93481011f0 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9330003aa0 con 0x7f93481011f0 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773799956 0 0) 0x7f93481a7030 con 0x7f9348109e80 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93481a94b0 con 0x7f9348109e80 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292731072 0 0) 0x7f93481a8270 con 0x7f934810da20 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93481a7030 con 0x7f934810da20 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.115+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3392216926 0 0) 0x7f9330003aa0 con 0x7f93481011f0 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93481a8270 con 0x7f93481011f0 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f93380019c0 con 0x7f93481011f0 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 278332538 0 0) 0x7f93481a94b0 con 0x7f9348109e80 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9330003aa0 con 0x7f9348109e80 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3682749432 0 0) 0x7f93481a7030 con 0x7f934810da20 2024-03-30T15:04:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93481a94b0 con 0x7f934810da20 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f933c001a30 con 0x7f9348109e80 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9340001d80 con 0x7f934810da20 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3783447613 0 0) 0x7f93481a8270 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 >> v1:172.21.15.134:6789/0 conn(0x7f934810da20 legacy=0x7f93481a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 >> v1:172.21.15.28:6789/0 conn(0x7f9348109e80 legacy=0x7f93481a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93481aa6f0 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93481a7260 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f934d46f700 10 client.58067.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f93481a7720 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f934d46f700 1 librados: init done 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9338003df0 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.116+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9338005eb0 con 0x7f93481011f0 2024-03-30T15:04:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.117+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9334005120 con 0x7f93481011f0 2024-03-30T15:04:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.117+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9338003950 con 0x7f93481011f0 2024-03-30T15:04:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(447..447 src has 53..447) ==== 8138+0+0 (unknown 2922271259 0 0) 0x7f9338089650 con 0x7f93481011f0 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 10 client.58067.objecter ms_dispatch 0x7f9348001550 osd_map(447..447 src has 53..447) 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 3 client.58067.objecter handle_osd_map got epochs [447,447] > 0 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 3 client.58067.objecter handle_osd_map decoding full epoch 447 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 20 client.58067.objecter dump_active .. 0 homeless 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.118+0000 7f932ffff700 10 client.58067.objecter ms_handle_connect 0x7f9330070400 2024-03-30T15:04:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.122+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f93380569a0 con 0x7f93481011f0 2024-03-30T15:04:01.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:01.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:01.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:01.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:01.364+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f9334005410 con 0x7f93481011f0 2024-03-30T15:04:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:01.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:01.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:01.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:01.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:01.645 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T15:04:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:01 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T15:04:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: osdmap e447: 8 total, 8 up, 8 in 2024-03-30T15:04:01.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:01.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T15:04:01.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:01 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T15:04:01.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:01.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:01.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:01.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:01.982 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:04:01.982 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:04:01.982 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:04:01.982 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:04:01.982 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:04:01.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:02.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:02.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:02.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.312+0000 7f932ffff700 1 -- 172.21.15.28:0/1395796025 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v449) ==== 101+0+0 (unknown 3599922298 0 0) 0x7f933805a3f0 con 0x7f93481011f0 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.315+0000 7f934d46f700 10 librados: watch_flush enter 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.315+0000 7f934d46f700 10 librados: watch_flush exit 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.315+0000 7f934d46f700 20 client.58067.objecter shutdown clearing up homeless session... 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.315+0000 7f934d46f700 10 client.58067.objecter successfully canceled tick 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.316+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9330070400 legacy=0x7f93300728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.316+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 >> v1:172.21.15.28:6790/0 conn(0x7f93481011f0 legacy=0x7f934806ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.316+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 shutdown_connections 2024-03-30T15:04:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.316+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 >> 172.21.15.28:0/1395796025 conn(0x7f93480fcaa0 msgr2=0x7f93480fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.317+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 shutdown_connections 2024-03-30T15:04:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.317+0000 7f934d46f700 1 -- 172.21.15.28:0/1395796025 wait complete. 2024-03-30T15:04:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:02.318+0000 7f934d46f700 1 librados: shutdown 2024-03-30T15:04:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:04:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:02.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:02.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:02.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:02.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:02.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:04:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:04:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T15:04:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1395796025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: from='client.58067 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:04:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:02 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:04:02.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1395796025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: from='client.58067 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:04:02.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: osdmap e448: 8 total, 8 up, 8 in 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1395796025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: from='client.58067 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: pgmap v627: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:04:02.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:02 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:04:02.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:02.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:02.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:02.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:02.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:03.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:03.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:03.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:03.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:03.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:03.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:03.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:03.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:03.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: 101.19 scrub starts 2024-03-30T15:04:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: 101.19 scrub ok 2024-03-30T15:04:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: from='client.58067 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: 42.8 scrub starts 2024-03-30T15:04:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:03 smithi134 ceph-mon[33714]: 42.8 scrub ok 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: 101.19 scrub starts 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: 101.19 scrub ok 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: from='client.58067 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: 42.8 scrub starts 2024-03-30T15:04:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[41614]: 42.8 scrub ok 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: 101.19 scrub starts 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: 101.19 scrub ok 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: from='client.58067 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: osdmap e449: 8 total, 8 up, 8 in 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: 42.8 scrub starts 2024-03-30T15:04:03.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:03 smithi028 ceph-mon[29876]: 42.8 scrub ok 2024-03-30T15:04:03.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:03.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:03.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:03.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:03.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:04.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:04.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:04.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:04.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:04:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:04.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:04.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:04.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:04.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:04.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[41614]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:04.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:04.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:04:04.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:04:04.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:04 smithi028 ceph-mon[29876]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 -- 172.21.15.28:0/3940655022 >> v1:172.21.15.28:6789/0 conn(0x7f1250102250 legacy=0x7f1250102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 -- 172.21.15.28:0/3940655022 shutdown_connections 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 -- 172.21.15.28:0/3940655022 >> 172.21.15.28:0/3940655022 conn(0x7f12500fdb20 msgr2=0x7f12500fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 -- 172.21.15.28:0/3940655022 shutdown_connections 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 -- 172.21.15.28:0/3940655022 wait complete. 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 librados: starting msgr at 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 librados: starting objecter 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.391+0000 7f12554c3700 1 Processor -- start 2024-03-30T15:04:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 -- start start 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 librados: setting wanted keys 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 librados: calling monclient init 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f12501a9300 con 0x7f125010ea80 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f12501aa520 con 0x7f125010aee0 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f12554c3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f12501ab760 con 0x7f1250102250 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f124effd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1250102250 0x7f1250107530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56490/0 (socket says 172.21.15.28:56490) 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f124effd700 1 -- 172.21.15.28:0/1101537077 learned_addr learned my addr 172.21.15.28:0/1101537077 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter ms_handle_connect 0x7f125010ea80 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter ms_handle_connect 0x7f1250102250 2024-03-30T15:04:04.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter ms_handle_connect 0x7f125010aee0 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705566734 0 0) 0x7f12501a9300 con 0x7f125010ea80 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12380031f0 con 0x7f125010ea80 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250165411 0 0) 0x7f12501ab760 con 0x7f1250102250 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12501a9300 con 0x7f1250102250 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611671007 0 0) 0x7f12501aa520 con 0x7f125010aee0 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.392+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12501ab760 con 0x7f125010aee0 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3780049071 0 0) 0x7f12380031f0 con 0x7f125010ea80 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12501aa520 con 0x7f125010ea80 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1190767932 0 0) 0x7f12501ab760 con 0x7f125010aee0 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12380031f0 con 0x7f125010aee0 2024-03-30T15:04:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1248001c70 con 0x7f125010ea80 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1244001870 con 0x7f125010aee0 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505994200 0 0) 0x7f12501a9300 con 0x7f1250102250 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12501ab760 con 0x7f1250102250 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12400046e0 con 0x7f1250102250 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3787376159 0 0) 0x7f12380031f0 con 0x7f125010aee0 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 >> v1:172.21.15.28:6790/0 conn(0x7f1250102250 legacy=0x7f1250107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 >> v1:172.21.15.28:6789/0 conn(0x7f125010ea80 legacy=0x7f12501a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12501ac9a0 con 0x7f125010aee0 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12501ab990 con 0x7f125010aee0 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f12554c3700 10 client.50089.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f12501abe50 con 0x7f125010aee0 2024-03-30T15:04:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f12554c3700 1 librados: init done 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1244003b40 con 0x7f125010aee0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.393+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12440049b0 con 0x7f125010aee0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.394+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f123c005120 con 0x7f125010aee0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.394+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1244004990 con 0x7f125010aee0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(451..451 src has 53..451) ==== 8523+0+0 (unknown 3216225720 0 0) 0x7f12440883e0 con 0x7f125010aee0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 10 client.50089.objecter ms_dispatch 0x7f1250001550 osd_map(451..451 src has 53..451) 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 3 client.50089.objecter handle_osd_map got epochs [451,451] > 0 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 3 client.50089.objecter handle_osd_map decoding full epoch 451 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 20 client.50089.objecter dump_active .. 0 homeless 2024-03-30T15:04:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.395+0000 7f1237fff700 10 client.50089.objecter ms_handle_connect 0x7f1238070440 2024-03-30T15:04:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.399+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1244055460 con 0x7f125010aee0 2024-03-30T15:04:04.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:04.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:04.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:04.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.616+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f123c000b90 con 0x7f1238070440 2024-03-30T15:04:04.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:04.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.634+0000 7f1237fff700 1 -- 172.21.15.28:0/1101537077 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+665326 (unknown 2965378022 0 3672449167) 0x7f123c000b90 con 0x7f1238070440 2024-03-30T15:04:04.645 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 10 librados: watch_flush enter 2024-03-30T15:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 10 librados: watch_flush exit 2024-03-30T15:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 20 client.50089.objecter shutdown clearing up homeless session... 2024-03-30T15:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 10 client.50089.objecter successfully canceled tick 2024-03-30T15:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1238070440 legacy=0x7f1238072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 >> v1:172.21.15.134:6789/0 conn(0x7f125010aee0 legacy=0x7f12501a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 shutdown_connections 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.639+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 >> 172.21.15.28:0/1101537077 conn(0x7f12500fdb20 msgr2=0x7f1250111c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.640+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 shutdown_connections 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.640+0000 7f12554c3700 1 -- 172.21.15.28:0/1101537077 wait complete. 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:04.640+0000 7f12554c3700 1 librados: shutdown 2024-03-30T15:04:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:04:04.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: osdmap e450: 8 total, 8 up, 8 in 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:04:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:04:04.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:04 smithi134 ceph-mon[33714]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:04.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:04.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:04.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:04.910 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:04.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:05.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:05.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:05.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:05.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:05.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:05.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:05.510 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-03-30T15:04:04.330190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792134,"num_objects":364,"num_object_clones":2,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":0,"num_read":1607,"num_read_kb":1350,"num_write":1230,"num_write_kb":15162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2580,"ondisk_log_size":2580,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":743,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":608224,"kb_used_data":87904,"kb_used_omap":12,"kb_used_meta":520051,"kb_avail":749352992,"statfs":{"total":767960285184,"available":767337463808,"internally_reserved":0,"allocated":90013696,"data_stored":62356302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":532532784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029207"},"pg_stats":[{"pgid":"297.1f","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339882+0000","last_change":"2024-03-30T15:04:02.327312+0000","last_active":"2024-03-30T15:04:03.339882+0000","last_peered":"2024-03-30T15:04:03.339882+0000","last_clean":"2024-03-30T15:04:03.339882+0000","last_became_active":"2024-03-30T15:04:02.327008+0000","last_became_peered":"2024-03-30T15:04:02.327008+0000","last_unstale":"2024-03-30T15:04:03.339882+0000","last_undegraded":"2024-03-30T15:04:03.339882+0000","last_fullsized":"2024-03-30T15:04:03.339882+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.680394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1e","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328983+0000","last_change":"2024-03-30T15:04:02.575225+0000","last_active":"2024-03-30T15:04:03.328983+0000","last_peered":"2024-03-30T15:04:03.328983+0000","last_clean":"2024-03-30T15:04:03.328983+0000","last_became_active":"2024-03-30T15:04:02.574926+0000","last_became_peered":"2024-03-30T15:04:02.574926+0000","last_unstale":"2024-03-30T15:04:03.328983+0000","last_undegraded":"2024-03-30T15:04:03.328983+0000","last_fullsized":"2024-03-30T15:04:03.328983+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:10.338692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1d","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338853+0000","last_change":"2024-03-30T15:04:02.575194+0000","last_active":"2024-03-30T15:04:03.338853+0000","last_peered":"2024-03-30T15:04:03.338853+0000","last_clean":"2024-03-30T15:04:03.338853+0000","last_became_active":"2024-03-30T15:04:02.574887+0000","last_became_peered":"2024-03-30T15:04:02.574887+0000","last_unstale":"2024-03-30T15:04:03.338853+0000","last_undegraded":"2024-03-30T15:04:03.338853+0000","last_fullsized":"2024-03-30T15:04:03.338853+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:10.270398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.1c","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344031+0000","last_change":"2024-03-30T15:04:02.575185+0000","last_active":"2024-03-30T15:04:03.344031+0000","last_peered":"2024-03-30T15:04:03.344031+0000","last_clean":"2024-03-30T15:04:03.344031+0000","last_became_active":"2024-03-30T15:04:02.574860+0000","last_became_peered":"2024-03-30T15:04:02.574860+0000","last_unstale":"2024-03-30T15:04:03.344031+0000","last_undegraded":"2024-03-30T15:04:03.344031+0000","last_fullsized":"2024-03-30T15:04:03.344031+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.405093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.1b","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331182+0000","last_change":"2024-03-30T15:04:02.327712+0000","last_active":"2024-03-30T15:04:03.331182+0000","last_peered":"2024-03-30T15:04:03.331182+0000","last_clean":"2024-03-30T15:04:03.331182+0000","last_became_active":"2024-03-30T15:04:02.327364+0000","last_became_peered":"2024-03-30T15:04:02.327364+0000","last_unstale":"2024-03-30T15:04:03.331182+0000","last_undegraded":"2024-03-30T15:04:03.331182+0000","last_fullsized":"2024-03-30T15:04:03.331182+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.316889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.1a","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343483+0000","last_change":"2024-03-30T15:04:02.576586+0000","last_active":"2024-03-30T15:04:03.343483+0000","last_peered":"2024-03-30T15:04:03.343483+0000","last_clean":"2024-03-30T15:04:03.343483+0000","last_became_active":"2024-03-30T15:04:02.576114+0000","last_became_peered":"2024-03-30T15:04:02.576114+0000","last_unstale":"2024-03-30T15:04:03.343483+0000","last_undegraded":"2024-03-30T15:04:03.343483+0000","last_fullsized":"2024-03-30T15:04:03.343483+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.599932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.19","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327759+0000","last_change":"2024-03-30T15:04:02.575983+0000","last_active":"2024-03-30T15:04:03.327759+0000","last_peered":"2024-03-30T15:04:03.327759+0000","last_clean":"2024-03-30T15:04:03.327759+0000","last_became_active":"2024-03-30T15:04:02.575651+0000","last_became_peered":"2024-03-30T15:04:02.575651+0000","last_unstale":"2024-03-30T15:04:03.327759+0000","last_undegraded":"2024-03-30T15:04:03.327759+0000","last_fullsized":"2024-03-30T15:04:03.327759+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:25.940747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.18","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327570+0000","last_change":"2024-03-30T15:04:02.576433+0000","last_active":"2024-03-30T15:04:03.327570+0000","last_peered":"2024-03-30T15:04:03.327570+0000","last_clean":"2024-03-30T15:04:03.327570+0000","last_became_active":"2024-03-30T15:04:02.576098+0000","last_became_peered":"2024-03-30T15:04:02.576098+0000","last_unstale":"2024-03-30T15:04:03.327570+0000","last_undegraded":"2024-03-30T15:04:03.327570+0000","last_fullsized":"2024-03-30T15:04:03.327570+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:03.438286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.17","version":"450'1","reported_seq":14,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.337732+0000","last_change":"2024-03-30T15:04:02.331072+0000","last_active":"2024-03-30T15:04:03.337732+0000","last_peered":"2024-03-30T15:04:03.337732+0000","last_clean":"2024-03-30T15:04:03.337732+0000","last_became_active":"2024-03-30T15:04:02.330610+0000","last_became_peered":"2024-03-30T15:04:02.330610+0000","last_unstale":"2024-03-30T15:04:03.337732+0000","last_undegraded":"2024-03-30T15:04:03.337732+0000","last_fullsized":"2024-03-30T15:04:03.337732+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.779919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"297.16","version":"450'2","reported_seq":15,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.353421+0000","last_change":"2024-03-30T15:04:02.323696+0000","last_active":"2024-03-30T15:04:03.353421+0000","last_peered":"2024-03-30T15:04:03.353421+0000","last_clean":"2024-03-30T15:04:03.353421+0000","last_became_active":"2024-03-30T15:04:02.323410+0000","last_became_peered":"2024-03-30T15:04:02.323410+0000","last_unstale":"2024-03-30T15:04:03.353421+0000","last_undegraded":"2024-03-30T15:04:03.353421+0000","last_fullsized":"2024-03-30T15:04:03.353421+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.066108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.15","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349214+0000","last_change":"2024-03-30T15:04:02.331266+0000","last_active":"2024-03-30T15:04:03.349214+0000","last_peered":"2024-03-30T15:04:03.349214+0000","last_clean":"2024-03-30T15:04:03.349214+0000","last_became_active":"2024-03-30T15:04:02.330908+0000","last_became_peered":"2024-03-30T15:04:02.330908+0000","last_unstale":"2024-03-30T15:04:03.349214+0000","last_undegraded":"2024-03-30T15:04:03.349214+0000","last_fullsized":"2024-03-30T15:04:03.349214+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.863110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.14","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344159+0000","last_change":"2024-03-30T15:04:02.327694+0000","last_active":"2024-03-30T15:04:03.344159+0000","last_peered":"2024-03-30T15:04:03.344159+0000","last_clean":"2024-03-30T15:04:03.344159+0000","last_became_active":"2024-03-30T15:04:02.327208+0000","last_became_peered":"2024-03-30T15:04:02.327208+0000","last_unstale":"2024-03-30T15:04:03.344159+0000","last_undegraded":"2024-03-30T15:04:03.344159+0000","last_fullsized":"2024-03-30T15:04:03.344159+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:17.171469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.13","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327816+0000","last_change":"2024-03-30T15:04:02.574744+0000","last_active":"2024-03-30T15:04:03.327816+0000","last_peered":"2024-03-30T15:04:03.327816+0000","last_clean":"2024-03-30T15:04:03.327816+0000","last_became_active":"2024-03-30T15:04:02.574381+0000","last_became_peered":"2024-03-30T15:04:02.574381+0000","last_unstale":"2024-03-30T15:04:03.327816+0000","last_undegraded":"2024-03-30T15:04:03.327816+0000","last_fullsized":"2024-03-30T15:04:03.327816+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:23.744104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.12","version":"450'2","reported_seq":15,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.357777+0000","last_change":"2024-03-30T15:04:02.575194+0000","last_active":"2024-03-30T15:04:03.357777+0000","last_peered":"2024-03-30T15:04:03.357777+0000","last_clean":"2024-03-30T15:04:03.357777+0000","last_became_active":"2024-03-30T15:04:02.574848+0000","last_became_peered":"2024-03-30T15:04:02.574848+0000","last_unstale":"2024-03-30T15:04:03.357777+0000","last_undegraded":"2024-03-30T15:04:03.357777+0000","last_fullsized":"2024-03-30T15:04:03.357777+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.479757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.11","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338301+0000","last_change":"2024-03-30T15:04:02.577381+0000","last_active":"2024-03-30T15:04:03.338301+0000","last_peered":"2024-03-30T15:04:03.338301+0000","last_clean":"2024-03-30T15:04:03.338301+0000","last_became_active":"2024-03-30T15:04:02.577074+0000","last_became_peered":"2024-03-30T15:04:02.577074+0000","last_unstale":"2024-03-30T15:04:03.338301+0000","last_undegraded":"2024-03-30T15:04:03.338301+0000","last_fullsized":"2024-03-30T15:04:03.338301+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.927816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.10","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330802+0000","last_change":"2024-03-30T15:04:02.329337+0000","last_active":"2024-03-30T15:04:03.330802+0000","last_peered":"2024-03-30T15:04:03.330802+0000","last_clean":"2024-03-30T15:04:03.330802+0000","last_became_active":"2024-03-30T15:04:02.328930+0000","last_became_peered":"2024-03-30T15:04:02.328930+0000","last_unstale":"2024-03-30T15:04:03.330802+0000","last_undegraded":"2024-03-30T15:04:03.330802+0000","last_fullsized":"2024-03-30T15:04:03.330802+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.221654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.f","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332182+0000","last_change":"2024-03-30T15:04:02.332952+0000","last_active":"2024-03-30T15:04:03.332182+0000","last_peered":"2024-03-30T15:04:03.332182+0000","last_clean":"2024-03-30T15:04:03.332182+0000","last_became_active":"2024-03-30T15:04:02.332571+0000","last_became_peered":"2024-03-30T15:04:02.332571+0000","last_unstale":"2024-03-30T15:04:03.332182+0000","last_undegraded":"2024-03-30T15:04:03.332182+0000","last_fullsized":"2024-03-30T15:04:03.332182+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:26.429736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.e","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349503+0000","last_change":"2024-03-30T15:04:02.576026+0000","last_active":"2024-03-30T15:04:03.349503+0000","last_peered":"2024-03-30T15:04:03.349503+0000","last_clean":"2024-03-30T15:04:03.349503+0000","last_became_active":"2024-03-30T15:04:02.575725+0000","last_became_peered":"2024-03-30T15:04:02.575725+0000","last_unstale":"2024-03-30T15:04:03.349503+0000","last_undegraded":"2024-03-30T15:04:03.349503+0000","last_fullsized":"2024-03-30T15:04:03.349503+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.660766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.d","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328667+0000","last_change":"2024-03-30T15:04:02.329312+0000","last_active":"2024-03-30T15:04:03.328667+0000","last_peered":"2024-03-30T15:04:03.328667+0000","last_clean":"2024-03-30T15:04:03.328667+0000","last_became_active":"2024-03-30T15:04:02.327473+0000","last_became_peered":"2024-03-30T15:04:02.327473+0000","last_unstale":"2024-03-30T15:04:03.328667+0000","last_undegraded":"2024-03-30T15:04:03.328667+0000","last_fullsized":"2024-03-30T15:04:03.328667+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.001117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.c","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349104+0000","last_change":"2024-03-30T15:04:02.326424+0000","last_active":"2024-03-30T15:04:03.349104+0000","last_peered":"2024-03-30T15:04:03.349104+0000","last_clean":"2024-03-30T15:04:03.349104+0000","last_became_active":"2024-03-30T15:04:02.326093+0000","last_became_peered":"2024-03-30T15:04:02.326093+0000","last_unstale":"2024-03-30T15:04:03.349104+0000","last_undegraded":"2024-03-30T15:04:03.349104+0000","last_fullsized":"2024-03-30T15:04:03.349104+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.507562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"297.b","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327957+0000","last_change":"2024-03-30T15:04:02.329421+0000","last_active":"2024-03-30T15:04:03.327957+0000","last_peered":"2024-03-30T15:04:03.327957+0000","last_clean":"2024-03-30T15:04:03.327957+0000","last_became_active":"2024-03-30T15:04:02.329139+0000","last_became_peered":"2024-03-30T15:04:02.329139+0000","last_unstale":"2024-03-30T15:04:03.327957+0000","last_undegraded":"2024-03-30T15:04:03.327957+0000","last_fullsized":"2024-03-30T15:04:03.327957+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.272465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.a","version":"450'2","reported_seq":15,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.355675+0000","last_change":"2024-03-30T15:04:02.576089+0000","last_active":"2024-03-30T15:04:03.355675+0000","last_peered":"2024-03-30T15:04:03.355675+0000","last_clean":"2024-03-30T15:04:03.355675+0000","last_became_active":"2024-03-30T15:04:02.575447+0000","last_became_peered":"2024-03-30T15:04:02.575447+0000","last_unstale":"2024-03-30T15:04:03.355675+0000","last_undegraded":"2024-03-30T15:04:03.355675+0000","last_fullsized":"2024-03-30T15:04:03.355675+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:10.460645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.9","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334330+0000","last_change":"2024-03-30T15:04:02.325161+0000","last_active":"2024-03-30T15:04:03.334330+0000","last_peered":"2024-03-30T15:04:03.334330+0000","last_clean":"2024-03-30T15:04:03.334330+0000","last_became_active":"2024-03-30T15:04:02.324825+0000","last_became_peered":"2024-03-30T15:04:02.324825+0000","last_unstale":"2024-03-30T15:04:03.334330+0000","last_undegraded":"2024-03-30T15:04:03.334330+0000","last_fullsized":"2024-03-30T15:04:03.334330+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.885546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"297.8","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330810+0000","last_change":"2024-03-30T15:04:02.575237+0000","last_active":"2024-03-30T15:04:03.330810+0000","last_peered":"2024-03-30T15:04:03.330810+0000","last_clean":"2024-03-30T15:04:03.330810+0000","last_became_active":"2024-03-30T15:04:02.574918+0000","last_became_peered":"2024-03-30T15:04:02.574918+0000","last_unstale":"2024-03-30T15:04:03.330810+0000","last_undegraded":"2024-03-30T15:04:03.330810+0000","last_fullsized":"2024-03-30T15:04:03.330810+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.709966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"297.7","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328925+0000","last_change":"2024-03-30T15:04:02.576291+0000","last_active":"2024-03-30T15:04:03.328925+0000","last_peered":"2024-03-30T15:04:03.328925+0000","last_clean":"2024-03-30T15:04:03.328925+0000","last_became_active":"2024-03-30T15:04:02.575970+0000","last_became_peered":"2024-03-30T15:04:02.575970+0000","last_unstale":"2024-03-30T15:04:03.328925+0000","last_undegraded":"2024-03-30T15:04:03.328925+0000","last_fullsized":"2024-03-30T15:04:03.328925+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.579825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"297.6","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335364+0000","last_change":"2024-03-30T15:04:02.327460+0000","last_active":"2024-03-30T15:04:03.335364+0000","last_peered":"2024-03-30T15:04:03.335364+0000","last_clean":"2024-03-30T15:04:03.335364+0000","last_became_active":"2024-03-30T15:04:02.326600+0000","last_became_peered":"2024-03-30T15:04:02.326600+0000","last_unstale":"2024-03-30T15:04:03.335364+0000","last_undegraded":"2024-03-30T15:04:03.335364+0000","last_fullsized":"2024-03-30T15:04:03.335364+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:04.731236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"297.5","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329032+0000","last_change":"2024-03-30T15:04:02.574793+0000","last_active":"2024-03-30T15:04:03.329032+0000","last_peered":"2024-03-30T15:04:03.329032+0000","last_clean":"2024-03-30T15:04:03.329032+0000","last_became_active":"2024-03-30T15:04:02.574273+0000","last_became_peered":"2024-03-30T15:04:02.574273+0000","last_unstale":"2024-03-30T15:04:03.329032+0000","last_undegraded":"2024-03-30T15:04:03.329032+0000","last_fullsized":"2024-03-30T15:04:03.329032+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.568451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.4","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344263+0000","last_change":"2024-03-30T15:04:02.574913+0000","last_active":"2024-03-30T15:04:03.344263+0000","last_peered":"2024-03-30T15:04:03.344263+0000","last_clean":"2024-03-30T15:04:03.344263+0000","last_became_active":"2024-03-30T15:04:02.574420+0000","last_became_peered":"2024-03-30T15:04:02.574420+0000","last_unstale":"2024-03-30T15:04:03.344263+0000","last_undegraded":"2024-03-30T15:04:03.344263+0000","last_fullsized":"2024-03-30T15:04:03.344263+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:27.877130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"297.3","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338682+0000","last_change":"2024-03-30T15:04:02.322970+0000","last_active":"2024-03-30T15:04:03.338682+0000","last_peered":"2024-03-30T15:04:03.338682+0000","last_clean":"2024-03-30T15:04:03.338682+0000","last_became_active":"2024-03-30T15:04:02.322310+0000","last_became_peered":"2024-03-30T15:04:02.322310+0000","last_unstale":"2024-03-30T15:04:03.338682+0000","last_undegraded":"2024-03-30T15:04:03.338682+0000","last_fullsized":"2024-03-30T15:04:03.338682+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:17.635772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"297.2","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328091+0000","last_change":"2024-03-30T15:04:02.329133+0000","last_active":"2024-03-30T15:04:03.328091+0000","last_peered":"2024-03-30T15:04:03.328091+0000","last_clean":"2024-03-30T15:04:03.328091+0000","last_became_active":"2024-03-30T15:04:02.327480+0000","last_became_peered":"2024-03-30T15:04:02.327480+0000","last_unstale":"2024-03-30T15:04:03.328091+0000","last_undegraded":"2024-03-30T15:04:03.328091+0000","last_fullsized":"2024-03-30T15:04:03.328091+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.583275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"297.1","version":"0'0","reported_seq":13,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334347+0000","last_change":"2024-03-30T15:04:02.575697+0000","last_active":"2024-03-30T15:04:03.334347+0000","last_peered":"2024-03-30T15:04:03.334347+0000","last_clean":"2024-03-30T15:04:03.334347+0000","last_became_active":"2024-03-30T15:04:02.575369+0000","last_became_peered":"2024-03-30T15:04:02.575369+0000","last_unstale":"2024-03-30T15:04:03.334347+0000","last_undegraded":"2024-03-30T15:04:03.334347+0000","last_fullsized":"2024-03-30T15:04:03.334347+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:21.335606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"297.0","version":"450'1","reported_seq":14,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.352027+0000","last_change":"2024-03-30T15:04:02.329668+0000","last_active":"2024-03-30T15:04:03.352027+0000","last_peered":"2024-03-30T15:04:03.352027+0000","last_clean":"2024-03-30T15:04:03.352027+0000","last_became_active":"2024-03-30T15:04:02.329404+0000","last_became_peered":"2024-03-30T15:04:02.329404+0000","last_unstale":"2024-03-30T15:04:03.352027+0000","last_undegraded":"2024-03-30T15:04:03.352027+0000","last_fullsized":"2024-03-30T15:04:03.352027+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:01.300718+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.300718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.187965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":880,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327758+0000","last_change":"2024-03-30T15:03:49.868242+0000","last_active":"2024-03-30T15:04:03.327758+0000","last_peered":"2024-03-30T15:04:03.327758+0000","last_clean":"2024-03-30T15:04:03.327758+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:04:03.327758+0000","last_undegraded":"2024-03-30T15:04:03.327758+0000","last_fullsized":"2024-03-30T15:04:03.327758+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:32.552784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013223219999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":900,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334758+0000","last_change":"2024-03-30T15:03:49.312726+0000","last_active":"2024-03-30T15:04:03.334758+0000","last_peered":"2024-03-30T15:04:03.334758+0000","last_clean":"2024-03-30T15:04:03.334758+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:04:03.334758+0000","last_undegraded":"2024-03-30T15:04:03.334758+0000","last_fullsized":"2024-03-30T15:04:03.334758+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:31.961438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.961438+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00068545100000000001,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"400'89","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343155+0000","last_change":"2024-03-30T15:03:49.280034+0000","last_active":"2024-03-30T15:04:03.343155+0000","last_peered":"2024-03-30T15:04:03.343155+0000","last_clean":"2024-03-30T15:04:03.343155+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:04:03.343155+0000","last_undegraded":"2024-03-30T15:04:03.343155+0000","last_fullsized":"2024-03-30T15:04:03.343155+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'87","last_scrub_stamp":"2024-03-30T15:02:40.447037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.447037+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012654579999999999,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":937,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330960+0000","last_change":"2024-03-30T15:04:01.987571+0000","last_active":"2024-03-30T15:04:03.330960+0000","last_peered":"2024-03-30T15:04:03.330960+0000","last_clean":"2024-03-30T15:04:03.330960+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:04:03.330960+0000","last_undegraded":"2024-03-30T15:04:03.330960+0000","last_fullsized":"2024-03-30T15:04:03.330960+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:23.144383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000669957,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"286'36","reported_seq":875,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328334+0000","last_change":"2024-03-30T15:03:49.869695+0000","last_active":"2024-03-30T15:04:03.328334+0000","last_peered":"2024-03-30T15:04:03.328334+0000","last_clean":"2024-03-30T15:04:03.328334+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:04:03.328334+0000","last_undegraded":"2024-03-30T15:04:03.328334+0000","last_fullsized":"2024-03-30T15:04:03.328334+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:03:19.020165+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.020165+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.915160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017317389999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":971,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329038+0000","last_change":"2024-03-30T15:03:49.870157+0000","last_active":"2024-03-30T15:04:03.329038+0000","last_peered":"2024-03-30T15:04:03.329038+0000","last_clean":"2024-03-30T15:04:03.329038+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:04:03.329038+0000","last_undegraded":"2024-03-30T15:04:03.329038+0000","last_fullsized":"2024-03-30T15:04:03.329038+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0012316670000000001,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":862,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334830+0000","last_change":"2024-03-30T15:03:49.312946+0000","last_active":"2024-03-30T15:04:03.334830+0000","last_peered":"2024-03-30T15:04:03.334830+0000","last_clean":"2024-03-30T15:04:03.334830+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:04:03.334830+0000","last_undegraded":"2024-03-30T15:04:03.334830+0000","last_fullsized":"2024-03-30T15:04:03.334830+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00085074499999999997,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":882,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344250+0000","last_change":"2024-03-30T15:03:50.417752+0000","last_active":"2024-03-30T15:04:03.344250+0000","last_peered":"2024-03-30T15:04:03.344250+0000","last_clean":"2024-03-30T15:04:03.344250+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:04:03.344250+0000","last_undegraded":"2024-03-30T15:04:03.344250+0000","last_fullsized":"2024-03-30T15:04:03.344250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:51.976435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010235820000000001,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":873,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338401+0000","last_change":"2024-03-30T15:03:49.424988+0000","last_active":"2024-03-30T15:04:03.338401+0000","last_peered":"2024-03-30T15:04:03.338401+0000","last_clean":"2024-03-30T15:04:03.338401+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:04:03.338401+0000","last_undegraded":"2024-03-30T15:04:03.338401+0000","last_fullsized":"2024-03-30T15:04:03.338401+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.557627+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00052679599999999995,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338896+0000","last_change":"2024-03-30T15:03:49.426566+0000","last_active":"2024-03-30T15:04:03.338896+0000","last_peered":"2024-03-30T15:04:03.338896+0000","last_clean":"2024-03-30T15:04:03.338896+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:04:03.338896+0000","last_undegraded":"2024-03-30T15:04:03.338896+0000","last_fullsized":"2024-03-30T15:04:03.338896+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:02:00.557079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.557079+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001666521,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"428'9","reported_seq":839,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343319+0000","last_change":"2024-03-30T15:03:49.280037+0000","last_active":"2024-03-30T15:04:03.343319+0000","last_peered":"2024-03-30T15:04:03.343319+0000","last_clean":"2024-03-30T15:04:03.343319+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:04:03.343319+0000","last_undegraded":"2024-03-30T15:04:03.343319+0000","last_fullsized":"2024-03-30T15:04:03.343319+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'5","last_scrub_stamp":"2024-03-30T15:02:29.567113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:29.567113+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001191459,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"428'21","reported_seq":851,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343288+0000","last_change":"2024-03-30T15:03:49.280569+0000","last_active":"2024-03-30T15:04:03.343288+0000","last_peered":"2024-03-30T15:04:03.343288+0000","last_clean":"2024-03-30T15:04:03.343288+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:04:03.343288+0000","last_undegraded":"2024-03-30T15:04:03.343288+0000","last_fullsized":"2024-03-30T15:04:03.343288+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'17","last_scrub_stamp":"2024-03-30T15:02:35.464703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.464703+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017288749999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":845,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335443+0000","last_change":"2024-03-30T15:03:49.313293+0000","last_active":"2024-03-30T15:04:03.335443+0000","last_peered":"2024-03-30T15:04:03.335443+0000","last_clean":"2024-03-30T15:04:03.335443+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:04:03.335443+0000","last_undegraded":"2024-03-30T15:04:03.335443+0000","last_fullsized":"2024-03-30T15:04:03.335443+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064837,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339903+0000","last_change":"2024-03-30T15:03:49.426629+0000","last_active":"2024-03-30T15:04:03.339903+0000","last_peered":"2024-03-30T15:04:03.339903+0000","last_clean":"2024-03-30T15:04:03.339903+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:04:03.339903+0000","last_undegraded":"2024-03-30T15:04:03.339903+0000","last_fullsized":"2024-03-30T15:04:03.339903+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00071320900000000002,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"368'54","reported_seq":884,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338692+0000","last_change":"2024-03-30T15:03:49.425793+0000","last_active":"2024-03-30T15:04:03.338692+0000","last_peered":"2024-03-30T15:04:03.338692+0000","last_clean":"2024-03-30T15:04:03.338692+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:04:03.338692+0000","last_undegraded":"2024-03-30T15:04:03.338692+0000","last_fullsized":"2024-03-30T15:04:03.338692+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00070943999999999998,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":916,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328511+0000","last_change":"2024-03-30T15:03:49.869564+0000","last_active":"2024-03-30T15:04:03.328511+0000","last_peered":"2024-03-30T15:04:03.328511+0000","last_clean":"2024-03-30T15:04:03.328511+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:04:03.328511+0000","last_undegraded":"2024-03-30T15:04:03.328511+0000","last_fullsized":"2024-03-30T15:04:03.328511+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:20.520994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021704710000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"368'35","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343831+0000","last_change":"2024-03-30T15:03:49.280713+0000","last_active":"2024-03-30T15:04:03.343831+0000","last_peered":"2024-03-30T15:04:03.343831+0000","last_clean":"2024-03-30T15:04:03.343831+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:04:03.343831+0000","last_undegraded":"2024-03-30T15:04:03.343831+0000","last_fullsized":"2024-03-30T15:04:03.343831+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'35","last_scrub_stamp":"2024-03-30T15:02:42.436904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.436904+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.431614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001387175,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":931,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348927+0000","last_change":"2024-03-30T15:03:49.336064+0000","last_active":"2024-03-30T15:04:03.348927+0000","last_peered":"2024-03-30T15:04:03.348927+0000","last_clean":"2024-03-30T15:04:03.348927+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:04:03.348927+0000","last_undegraded":"2024-03-30T15:04:03.348927+0000","last_fullsized":"2024-03-30T15:04:03.348927+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:02:39.411331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.411331+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081013099999999996,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"444'55","reported_seq":893,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328016+0000","last_change":"2024-03-30T15:03:49.869477+0000","last_active":"2024-03-30T15:04:03.328016+0000","last_peered":"2024-03-30T15:04:03.328016+0000","last_clean":"2024-03-30T15:04:03.328016+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:04:03.328016+0000","last_undegraded":"2024-03-30T15:04:03.328016+0000","last_fullsized":"2024-03-30T15:04:03.328016+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:37.520784+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.002834689,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":874,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348489+0000","last_change":"2024-03-30T15:03:49.336069+0000","last_active":"2024-03-30T15:04:03.348489+0000","last_peered":"2024-03-30T15:04:03.348489+0000","last_clean":"2024-03-30T15:04:03.348489+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:04:03.348489+0000","last_undegraded":"2024-03-30T15:04:03.348489+0000","last_fullsized":"2024-03-30T15:04:03.348489+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:02:41.402421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.402421+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00095881999999999998,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":940,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350181+0000","last_change":"2024-03-30T15:03:53.395482+0000","last_active":"2024-03-30T15:04:03.350181+0000","last_peered":"2024-03-30T15:04:03.350181+0000","last_clean":"2024-03-30T15:04:03.350181+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:04:03.350181+0000","last_undegraded":"2024-03-30T15:04:03.350181+0000","last_fullsized":"2024-03-30T15:04:03.350181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.903600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035289799999999998,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":893,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340015+0000","last_change":"2024-03-30T15:03:49.426814+0000","last_active":"2024-03-30T15:04:03.340015+0000","last_peered":"2024-03-30T15:04:03.340015+0000","last_clean":"2024-03-30T15:04:03.340015+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:04:03.340015+0000","last_undegraded":"2024-03-30T15:04:03.340015+0000","last_fullsized":"2024-03-30T15:04:03.340015+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'56","last_scrub_stamp":"2024-03-30T15:02:07.572656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.572656+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00084214999999999995,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"424'56","reported_seq":886,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345079+0000","last_change":"2024-03-30T15:03:49.463622+0000","last_active":"2024-03-30T15:04:03.345079+0000","last_peered":"2024-03-30T15:04:03.345079+0000","last_clean":"2024-03-30T15:04:03.345079+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:04:03.345079+0000","last_undegraded":"2024-03-30T15:04:03.345079+0000","last_fullsized":"2024-03-30T15:04:03.345079+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:59.225185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001170854,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"400'89","reported_seq":904,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335194+0000","last_change":"2024-03-30T15:03:49.313260+0000","last_active":"2024-03-30T15:04:03.335194+0000","last_peered":"2024-03-30T15:04:03.335194+0000","last_clean":"2024-03-30T15:04:03.335194+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:04:03.335194+0000","last_undegraded":"2024-03-30T15:04:03.335194+0000","last_fullsized":"2024-03-30T15:04:03.335194+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'87","last_scrub_stamp":"2024-03-30T15:02:01.656924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.656924+0000","objects_scrubbed":6,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00085783500000000004,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"450'90","reported_seq":913,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.356550+0000","last_change":"2024-03-30T15:03:49.276744+0000","last_active":"2024-03-30T15:04:03.356550+0000","last_peered":"2024-03-30T15:04:03.356550+0000","last_clean":"2024-03-30T15:04:03.356550+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:04:03.356550+0000","last_undegraded":"2024-03-30T15:04:03.356550+0000","last_fullsized":"2024-03-30T15:04:03.356550+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001085892,"stat_sum":{"num_bytes":73,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":877,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327789+0000","last_change":"2024-03-30T15:03:49.868008+0000","last_active":"2024-03-30T15:04:03.327789+0000","last_peered":"2024-03-30T15:04:03.327789+0000","last_clean":"2024-03-30T15:04:03.327789+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:04:03.327789+0000","last_undegraded":"2024-03-30T15:04:03.327789+0000","last_fullsized":"2024-03-30T15:04:03.327789+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:01:46.416710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.416710+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001026051,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"248'39","reported_seq":878,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327034+0000","last_change":"2024-03-30T15:03:49.867995+0000","last_active":"2024-03-30T15:04:03.327034+0000","last_peered":"2024-03-30T15:04:03.327034+0000","last_clean":"2024-03-30T15:04:03.327034+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:04:03.327034+0000","last_undegraded":"2024-03-30T15:04:03.327034+0000","last_fullsized":"2024-03-30T15:04:03.327034+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:01:45.467945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:45.467945+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0012596459999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"324'104","reported_seq":943,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348957+0000","last_change":"2024-03-30T15:03:50.351456+0000","last_active":"2024-03-30T15:04:03.348957+0000","last_peered":"2024-03-30T15:04:03.348957+0000","last_clean":"2024-03-30T15:04:03.348957+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:04:03.348957+0000","last_undegraded":"2024-03-30T15:04:03.348957+0000","last_fullsized":"2024-03-30T15:04:03.348957+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:03.884461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010326560000000001,"stat_sum":{"num_bytes":4000074,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":3937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":855,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345161+0000","last_change":"2024-03-30T15:03:49.282699+0000","last_active":"2024-03-30T15:04:03.345161+0000","last_peered":"2024-03-30T15:04:03.345161+0000","last_clean":"2024-03-30T15:04:03.345161+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:04:03.345161+0000","last_undegraded":"2024-03-30T15:04:03.345161+0000","last_fullsized":"2024-03-30T15:04:03.345161+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:02:36.478360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.478360+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0011790839999999999,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329244+0000","last_change":"2024-03-30T15:03:49.870156+0000","last_active":"2024-03-30T15:04:03.329244+0000","last_peered":"2024-03-30T15:04:03.329244+0000","last_clean":"2024-03-30T15:04:03.329244+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:04:03.329244+0000","last_undegraded":"2024-03-30T15:04:03.329244+0000","last_fullsized":"2024-03-30T15:04:03.329244+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:02:28.967486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.967486+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00048192900000000003,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":890,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338726+0000","last_change":"2024-03-30T15:03:49.426323+0000","last_active":"2024-03-30T15:04:03.338726+0000","last_peered":"2024-03-30T15:04:03.338726+0000","last_clean":"2024-03-30T15:04:03.338726+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:04:03.338726+0000","last_undegraded":"2024-03-30T15:04:03.338726+0000","last_fullsized":"2024-03-30T15:04:03.338726+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00099438400000000007,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":934,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330264+0000","last_change":"2024-03-30T15:03:51.152395+0000","last_active":"2024-03-30T15:04:03.330264+0000","last_peered":"2024-03-30T15:04:03.330264+0000","last_clean":"2024-03-30T15:04:03.330264+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:04:03.330264+0000","last_undegraded":"2024-03-30T15:04:03.330264+0000","last_fullsized":"2024-03-30T15:04:03.330264+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.082026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082763,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329024+0000","last_change":"2024-03-30T15:01:50.157960+0000","last_active":"2024-03-30T15:04:03.329024+0000","last_peered":"2024-03-30T15:04:03.329024+0000","last_clean":"2024-03-30T15:04:03.329024+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:04:03.329024+0000","last_undegraded":"2024-03-30T15:04:03.329024+0000","last_fullsized":"2024-03-30T15:04:03.329024+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.157883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.157883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349410+0000","last_change":"2024-03-30T15:01:59.125848+0000","last_active":"2024-03-30T15:04:03.349410+0000","last_peered":"2024-03-30T15:04:03.349410+0000","last_clean":"2024-03-30T15:04:03.349410+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:04:03.349410+0000","last_undegraded":"2024-03-30T15:04:03.349410+0000","last_fullsized":"2024-03-30T15:04:03.349410+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.125727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.125727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335750+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:04:03.335750+0000","last_peered":"2024-03-30T15:04:03.335750+0000","last_clean":"2024-03-30T15:04:03.335750+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:04:03.335750+0000","last_undegraded":"2024-03-30T15:04:03.335750+0000","last_fullsized":"2024-03-30T15:04:03.335750+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332230+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:04:03.332230+0000","last_peered":"2024-03-30T15:04:03.332230+0000","last_clean":"2024-03-30T15:04:03.332230+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:04:03.332230+0000","last_undegraded":"2024-03-30T15:04:03.332230+0000","last_fullsized":"2024-03-30T15:04:03.332230+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.852443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":691,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338648+0000","last_change":"2024-03-30T15:01:57.530926+0000","last_active":"2024-03-30T15:04:03.338648+0000","last_peered":"2024-03-30T15:04:03.338648+0000","last_clean":"2024-03-30T15:04:03.338648+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:04:03.338648+0000","last_undegraded":"2024-03-30T15:04:03.338648+0000","last_fullsized":"2024-03-30T15:04:03.338648+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.530827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.530827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":699,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327062+0000","last_change":"2024-03-30T15:02:41.928550+0000","last_active":"2024-03-30T15:04:03.327062+0000","last_peered":"2024-03-30T15:04:03.327062+0000","last_clean":"2024-03-30T15:04:03.327062+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:04:03.327062+0000","last_undegraded":"2024-03-30T15:04:03.327062+0000","last_fullsized":"2024-03-30T15:04:03.327062+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.928469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.928469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.706155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335144+0000","last_change":"2024-03-30T15:01:57.663288+0000","last_active":"2024-03-30T15:04:03.335144+0000","last_peered":"2024-03-30T15:04:03.335144+0000","last_clean":"2024-03-30T15:04:03.335144+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:04:03.335144+0000","last_undegraded":"2024-03-30T15:04:03.335144+0000","last_fullsized":"2024-03-30T15:04:03.335144+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.663204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.663204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328427+0000","last_change":"2024-03-30T15:01:59.181493+0000","last_active":"2024-03-30T15:04:03.328427+0000","last_peered":"2024-03-30T15:04:03.328427+0000","last_clean":"2024-03-30T15:04:03.328427+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:04:03.328427+0000","last_undegraded":"2024-03-30T15:04:03.328427+0000","last_fullsized":"2024-03-30T15:04:03.328427+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.181401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.181401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344462+0000","last_change":"2024-03-30T15:02:34.464153+0000","last_active":"2024-03-30T15:04:03.344462+0000","last_peered":"2024-03-30T15:04:03.344462+0000","last_clean":"2024-03-30T15:04:03.344462+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:04:03.344462+0000","last_undegraded":"2024-03-30T15:04:03.344462+0000","last_fullsized":"2024-03-30T15:04:03.344462+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:34.464067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.464067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331390+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:04:03.331390+0000","last_peered":"2024-03-30T15:04:03.331390+0000","last_clean":"2024-03-30T15:04:03.331390+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:04:03.331390+0000","last_undegraded":"2024-03-30T15:04:03.331390+0000","last_fullsized":"2024-03-30T15:04:03.331390+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.860968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335758+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:04:03.335758+0000","last_peered":"2024-03-30T15:04:03.335758+0000","last_clean":"2024-03-30T15:04:03.335758+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:04:03.335758+0000","last_undegraded":"2024-03-30T15:04:03.335758+0000","last_fullsized":"2024-03-30T15:04:03.335758+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":700,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328230+0000","last_change":"2024-03-30T15:02:38.936916+0000","last_active":"2024-03-30T15:04:03.328230+0000","last_peered":"2024-03-30T15:04:03.328230+0000","last_clean":"2024-03-30T15:04:03.328230+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:04:03.328230+0000","last_undegraded":"2024-03-30T15:04:03.328230+0000","last_fullsized":"2024-03-30T15:04:03.328230+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:38.936820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.936820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:03.472567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334407+0000","last_change":"2024-03-30T15:01:51.636327+0000","last_active":"2024-03-30T15:04:03.334407+0000","last_peered":"2024-03-30T15:04:03.334407+0000","last_clean":"2024-03-30T15:04:03.334407+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:04:03.334407+0000","last_undegraded":"2024-03-30T15:04:03.334407+0000","last_fullsized":"2024-03-30T15:04:03.334407+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.636231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.636231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330882+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:04:03.330882+0000","last_peered":"2024-03-30T15:04:03.330882+0000","last_clean":"2024-03-30T15:04:03.330882+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:04:03.330882+0000","last_undegraded":"2024-03-30T15:04:03.330882+0000","last_fullsized":"2024-03-30T15:04:03.330882+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.681651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":699,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327883+0000","last_change":"2024-03-30T15:02:00.582792+0000","last_active":"2024-03-30T15:04:03.327883+0000","last_peered":"2024-03-30T15:04:03.327883+0000","last_clean":"2024-03-30T15:04:03.327883+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:04:03.327883+0000","last_undegraded":"2024-03-30T15:04:03.327883+0000","last_fullsized":"2024-03-30T15:04:03.327883+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.582709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":699,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327738+0000","last_change":"2024-03-30T15:02:07.597777+0000","last_active":"2024-03-30T15:04:03.327738+0000","last_peered":"2024-03-30T15:04:03.327738+0000","last_clean":"2024-03-30T15:04:03.327738+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:04:03.327738+0000","last_undegraded":"2024-03-30T15:04:03.327738+0000","last_fullsized":"2024-03-30T15:04:03.327738+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.597700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335365+0000","last_change":"2024-03-30T15:01:10.292440+0000","last_active":"2024-03-30T15:04:03.335365+0000","last_peered":"2024-03-30T15:04:03.335365+0000","last_clean":"2024-03-30T15:04:03.335365+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:04:03.335365+0000","last_undegraded":"2024-03-30T15:04:03.335365+0000","last_fullsized":"2024-03-30T15:04:03.335365+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.292346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.292346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345581+0000","last_change":"2024-03-30T15:01:54.513739+0000","last_active":"2024-03-30T15:04:03.345581+0000","last_peered":"2024-03-30T15:04:03.345581+0000","last_clean":"2024-03-30T15:04:03.345581+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:04:03.345581+0000","last_undegraded":"2024-03-30T15:04:03.345581+0000","last_fullsized":"2024-03-30T15:04:03.345581+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.513601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.513601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329331+0000","last_change":"2024-03-30T15:01:22.172967+0000","last_active":"2024-03-30T15:04:03.329331+0000","last_peered":"2024-03-30T15:04:03.329331+0000","last_clean":"2024-03-30T15:04:03.329331+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:04:03.329331+0000","last_undegraded":"2024-03-30T15:04:03.329331+0000","last_fullsized":"2024-03-30T15:04:03.329331+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:22.172854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:22.172854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335607+0000","last_change":"2024-03-30T15:01:19.395733+0000","last_active":"2024-03-30T15:04:03.335607+0000","last_peered":"2024-03-30T15:04:03.335607+0000","last_clean":"2024-03-30T15:04:03.335607+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:04:03.335607+0000","last_undegraded":"2024-03-30T15:04:03.335607+0000","last_fullsized":"2024-03-30T15:04:03.335607+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330940+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:04:03.330940+0000","last_peered":"2024-03-30T15:04:03.330940+0000","last_clean":"2024-03-30T15:04:03.330940+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:04:03.330940+0000","last_undegraded":"2024-03-30T15:04:03.330940+0000","last_fullsized":"2024-03-30T15:04:03.330940+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.690679+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334250+0000","last_change":"2024-03-30T14:59:58.070758+0000","last_active":"2024-03-30T15:04:03.334250+0000","last_peered":"2024-03-30T15:04:03.334250+0000","last_clean":"2024-03-30T15:04:03.334250+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:04:03.334250+0000","last_undegraded":"2024-03-30T15:04:03.334250+0000","last_fullsized":"2024-03-30T15:04:03.334250+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:58.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T14:59:58.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349011+0000","last_change":"2024-03-30T15:02:40.426121+0000","last_active":"2024-03-30T15:04:03.349011+0000","last_peered":"2024-03-30T15:04:03.349011+0000","last_clean":"2024-03-30T15:04:03.349011+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:04:03.349011+0000","last_undegraded":"2024-03-30T15:04:03.349011+0000","last_fullsized":"2024-03-30T15:04:03.349011+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.426031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.426031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":707,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327218+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:04:03.327218+0000","last_peered":"2024-03-30T15:04:03.327218+0000","last_clean":"2024-03-30T15:04:03.327218+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:04:03.327218+0000","last_undegraded":"2024-03-30T15:04:03.327218+0000","last_fullsized":"2024-03-30T15:04:03.327218+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.425095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349108+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:04:03.349108+0000","last_peered":"2024-03-30T15:04:03.349108+0000","last_clean":"2024-03-30T15:04:03.349108+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:04:03.349108+0000","last_undegraded":"2024-03-30T15:04:03.349108+0000","last_fullsized":"2024-03-30T15:04:03.349108+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:04.572831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331501+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:04:03.331501+0000","last_peered":"2024-03-30T15:04:03.331501+0000","last_clean":"2024-03-30T15:04:03.331501+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:04:03.331501+0000","last_undegraded":"2024-03-30T15:04:03.331501+0000","last_fullsized":"2024-03-30T15:04:03.331501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330154+0000","last_change":"2024-03-30T15:01:58.138003+0000","last_active":"2024-03-30T15:04:03.330154+0000","last_peered":"2024-03-30T15:04:03.330154+0000","last_clean":"2024-03-30T15:04:03.330154+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:04:03.330154+0000","last_undegraded":"2024-03-30T15:04:03.330154+0000","last_fullsized":"2024-03-30T15:04:03.330154+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.137887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.137887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":688,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329055+0000","last_change":"2024-03-30T15:01:57.095860+0000","last_active":"2024-03-30T15:04:03.329055+0000","last_peered":"2024-03-30T15:04:03.329055+0000","last_clean":"2024-03-30T15:04:03.329055+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:04:03.329055+0000","last_undegraded":"2024-03-30T15:04:03.329055+0000","last_fullsized":"2024-03-30T15:04:03.329055+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.095752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.095752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348455+0000","last_change":"2024-03-30T15:01:58.162003+0000","last_active":"2024-03-30T15:04:03.348455+0000","last_peered":"2024-03-30T15:04:03.348455+0000","last_clean":"2024-03-30T15:04:03.348455+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:04:03.348455+0000","last_undegraded":"2024-03-30T15:04:03.348455+0000","last_fullsized":"2024-03-30T15:04:03.348455+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.161871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.161871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327641+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:04:03.327641+0000","last_peered":"2024-03-30T15:04:03.327641+0000","last_clean":"2024-03-30T15:04:03.327641+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:04:03.327641+0000","last_undegraded":"2024-03-30T15:04:03.327641+0000","last_fullsized":"2024-03-30T15:04:03.327641+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.836391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343621+0000","last_change":"2024-03-30T15:01:51.481224+0000","last_active":"2024-03-30T15:04:03.343621+0000","last_peered":"2024-03-30T15:04:03.343621+0000","last_clean":"2024-03-30T15:04:03.343621+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:04:03.343621+0000","last_undegraded":"2024-03-30T15:04:03.343621+0000","last_fullsized":"2024-03-30T15:04:03.343621+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.481143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.481143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":687,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334409+0000","last_change":"2024-03-30T15:01:20.445532+0000","last_active":"2024-03-30T15:04:03.334409+0000","last_peered":"2024-03-30T15:04:03.334409+0000","last_clean":"2024-03-30T15:04:03.334409+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:04:03.334409+0000","last_undegraded":"2024-03-30T15:04:03.334409+0000","last_fullsized":"2024-03-30T15:04:03.334409+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:20.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":710,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349687+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:04:03.349687+0000","last_peered":"2024-03-30T15:04:03.349687+0000","last_clean":"2024-03-30T15:04:03.349687+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:04:03.349687+0000","last_undegraded":"2024-03-30T15:04:03.349687+0000","last_fullsized":"2024-03-30T15:04:03.349687+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:38.423498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":807,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345551+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:04:03.345551+0000","last_peered":"2024-03-30T15:04:03.345551+0000","last_clean":"2024-03-30T15:04:03.345551+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:04:03.345551+0000","last_undegraded":"2024-03-30T15:04:03.345551+0000","last_fullsized":"2024-03-30T15:04:03.345551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.497042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":701,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339838+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:04:03.339838+0000","last_peered":"2024-03-30T15:04:03.339838+0000","last_clean":"2024-03-30T15:04:03.339838+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:04:03.339838+0000","last_undegraded":"2024-03-30T15:04:03.339838+0000","last_fullsized":"2024-03-30T15:04:03.339838+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":928,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328990+0000","last_change":"2024-03-30T15:04:03.157433+0000","last_active":"2024-03-30T15:04:03.328990+0000","last_peered":"2024-03-30T15:04:03.328990+0000","last_clean":"2024-03-30T15:04:03.328990+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:04:03.328990+0000","last_undegraded":"2024-03-30T15:04:03.328990+0000","last_fullsized":"2024-03-30T15:04:03.328990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.245942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":784,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345729+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:04:03.345729+0000","last_peered":"2024-03-30T15:04:03.345729+0000","last_clean":"2024-03-30T15:04:03.345729+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:03.345729+0000","last_undegraded":"2024-03-30T15:04:03.345729+0000","last_fullsized":"2024-03-30T15:04:03.345729+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.569612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349374+0000","last_change":"2024-03-30T15:02:07.241570+0000","last_active":"2024-03-30T15:04:03.349374+0000","last_peered":"2024-03-30T15:04:03.349374+0000","last_clean":"2024-03-30T15:04:03.349374+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:04:03.349374+0000","last_undegraded":"2024-03-30T15:04:03.349374+0000","last_fullsized":"2024-03-30T15:04:03.349374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.241467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.241467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":820,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345243+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:04:03.345243+0000","last_peered":"2024-03-30T15:04:03.345243+0000","last_clean":"2024-03-30T15:04:03.345243+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:04:03.345243+0000","last_undegraded":"2024-03-30T15:04:03.345243+0000","last_fullsized":"2024-03-30T15:04:03.345243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":804,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348799+0000","last_change":"2024-03-30T15:02:02.104334+0000","last_active":"2024-03-30T15:04:03.348799+0000","last_peered":"2024-03-30T15:04:03.348799+0000","last_clean":"2024-03-30T15:04:03.348799+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:04:03.348799+0000","last_undegraded":"2024-03-30T15:04:03.348799+0000","last_fullsized":"2024-03-30T15:04:03.348799+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.104245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331916+0000","last_change":"2024-03-30T15:01:48.369544+0000","last_active":"2024-03-30T15:04:03.331916+0000","last_peered":"2024-03-30T15:04:03.331916+0000","last_clean":"2024-03-30T15:04:03.331916+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:04:03.331916+0000","last_undegraded":"2024-03-30T15:04:03.331916+0000","last_fullsized":"2024-03-30T15:04:03.331916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:48.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":794,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329205+0000","last_change":"2024-03-30T15:02:04.594739+0000","last_active":"2024-03-30T15:04:03.329205+0000","last_peered":"2024-03-30T15:04:03.329205+0000","last_clean":"2024-03-30T15:04:03.329205+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:04:03.329205+0000","last_undegraded":"2024-03-30T15:04:03.329205+0000","last_fullsized":"2024-03-30T15:04:03.329205+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:04.594651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:04.594651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332366+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:04:03.332366+0000","last_peered":"2024-03-30T15:04:03.332366+0000","last_clean":"2024-03-30T15:04:03.332366+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:04:03.332366+0000","last_undegraded":"2024-03-30T15:04:03.332366+0000","last_fullsized":"2024-03-30T15:04:03.332366+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":816,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345506+0000","last_change":"2024-03-30T15:02:03.449650+0000","last_active":"2024-03-30T15:04:03.345506+0000","last_peered":"2024-03-30T15:04:03.345506+0000","last_clean":"2024-03-30T15:04:03.345506+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:04:03.345506+0000","last_undegraded":"2024-03-30T15:04:03.345506+0000","last_fullsized":"2024-03-30T15:04:03.345506+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:03.449549+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.449549+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":796,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343948+0000","last_change":"2024-03-30T15:01:50.503242+0000","last_active":"2024-03-30T15:04:03.343948+0000","last_peered":"2024-03-30T15:04:03.343948+0000","last_clean":"2024-03-30T15:04:03.343948+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:04:03.343948+0000","last_undegraded":"2024-03-30T15:04:03.343948+0000","last_fullsized":"2024-03-30T15:04:03.343948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.503156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.503156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":787,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.336544+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:04:03.336544+0000","last_peered":"2024-03-30T15:04:03.336544+0000","last_clean":"2024-03-30T15:04:03.336544+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:04:03.336544+0000","last_undegraded":"2024-03-30T15:04:03.336544+0000","last_fullsized":"2024-03-30T15:04:03.336544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":705,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349883+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:04:03.349883+0000","last_peered":"2024-03-30T15:04:03.349883+0000","last_clean":"2024-03-30T15:04:03.349883+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:04:03.349883+0000","last_undegraded":"2024-03-30T15:04:03.349883+0000","last_fullsized":"2024-03-30T15:04:03.349883+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.693552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332226+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:04:03.332226+0000","last_peered":"2024-03-30T15:04:03.332226+0000","last_clean":"2024-03-30T15:04:03.332226+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:04:03.332226+0000","last_undegraded":"2024-03-30T15:04:03.332226+0000","last_fullsized":"2024-03-30T15:04:03.332226+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":825,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332265+0000","last_change":"2024-03-30T15:01:46.424704+0000","last_active":"2024-03-30T15:04:03.332265+0000","last_peered":"2024-03-30T15:04:03.332265+0000","last_clean":"2024-03-30T15:04:03.332265+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:04:03.332265+0000","last_undegraded":"2024-03-30T15:04:03.332265+0000","last_fullsized":"2024-03-30T15:04:03.332265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T15:01:46.424546+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.424546+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":698,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331490+0000","last_change":"2024-03-30T15:01:50.443926+0000","last_active":"2024-03-30T15:04:03.331490+0000","last_peered":"2024-03-30T15:04:03.331490+0000","last_clean":"2024-03-30T15:04:03.331490+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:04:03.331490+0000","last_undegraded":"2024-03-30T15:04:03.331490+0000","last_fullsized":"2024-03-30T15:04:03.331490+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.443839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.443839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":910,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345700+0000","last_change":"2024-03-30T15:03:22.567558+0000","last_active":"2024-03-30T15:04:03.345700+0000","last_peered":"2024-03-30T15:04:03.345700+0000","last_clean":"2024-03-30T15:04:03.345700+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:04:03.345700+0000","last_undegraded":"2024-03-30T15:04:03.345700+0000","last_fullsized":"2024-03-30T15:04:03.345700+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:22.567465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.567465+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.595071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":775,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335467+0000","last_change":"2024-03-30T14:59:14.201212+0000","last_active":"2024-03-30T15:04:03.335467+0000","last_peered":"2024-03-30T15:04:03.335467+0000","last_clean":"2024-03-30T15:04:03.335467+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:04:03.335467+0000","last_undegraded":"2024-03-30T15:04:03.335467+0000","last_fullsized":"2024-03-30T15:04:03.335467+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T14:59:14.201056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":699,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327419+0000","last_change":"2024-03-30T15:02:30.902281+0000","last_active":"2024-03-30T15:04:03.327419+0000","last_peered":"2024-03-30T15:04:03.327419+0000","last_clean":"2024-03-30T15:04:03.327419+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:04:03.327419+0000","last_undegraded":"2024-03-30T15:04:03.327419+0000","last_fullsized":"2024-03-30T15:04:03.327419+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.902199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.902199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343555+0000","last_change":"2024-03-30T15:02:39.473396+0000","last_active":"2024-03-30T15:04:03.343555+0000","last_peered":"2024-03-30T15:04:03.343555+0000","last_clean":"2024-03-30T15:04:03.343555+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:04:03.343555+0000","last_undegraded":"2024-03-30T15:04:03.343555+0000","last_fullsized":"2024-03-30T15:04:03.343555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:39.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.473279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328738+0000","last_change":"2024-03-30T15:02:34.958123+0000","last_active":"2024-03-30T15:04:03.328738+0000","last_peered":"2024-03-30T15:04:03.328738+0000","last_clean":"2024-03-30T15:04:03.328738+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:04:03.328738+0000","last_undegraded":"2024-03-30T15:04:03.328738+0000","last_fullsized":"2024-03-30T15:04:03.328738+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T15:02:34.958001+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.958001+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":771,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329395+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:04:03.329395+0000","last_peered":"2024-03-30T15:04:03.329395+0000","last_clean":"2024-03-30T15:04:03.329395+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:04:03.329395+0000","last_undegraded":"2024-03-30T15:04:03.329395+0000","last_fullsized":"2024-03-30T15:04:03.329395+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":784,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344194+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:04:03.344194+0000","last_peered":"2024-03-30T15:04:03.344194+0000","last_clean":"2024-03-30T15:04:03.344194+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:04:03.344194+0000","last_undegraded":"2024-03-30T15:04:03.344194+0000","last_fullsized":"2024-03-30T15:04:03.344194+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.431637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":706,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327188+0000","last_change":"2024-03-30T15:02:06.597375+0000","last_active":"2024-03-30T15:04:03.327188+0000","last_peered":"2024-03-30T15:04:03.327188+0000","last_clean":"2024-03-30T15:04:03.327188+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:04:03.327188+0000","last_undegraded":"2024-03-30T15:04:03.327188+0000","last_fullsized":"2024-03-30T15:04:03.327188+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:06.597250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:06.597250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331125+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:04:03.331125+0000","last_peered":"2024-03-30T15:04:03.331125+0000","last_clean":"2024-03-30T15:04:03.331125+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:04:03.331125+0000","last_undegraded":"2024-03-30T15:04:03.331125+0000","last_fullsized":"2024-03-30T15:04:03.331125+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.527010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":813,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343780+0000","last_change":"2024-03-30T14:59:51.377378+0000","last_active":"2024-03-30T15:04:03.343780+0000","last_peered":"2024-03-30T15:04:03.343780+0000","last_clean":"2024-03-30T15:04:03.343780+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:04:03.343780+0000","last_undegraded":"2024-03-30T15:04:03.343780+0000","last_fullsized":"2024-03-30T15:04:03.343780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T14:59:51.377224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:51.377224+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":802,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339823+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:04:03.339823+0000","last_peered":"2024-03-30T15:04:03.339823+0000","last_clean":"2024-03-30T15:04:03.339823+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:04:03.339823+0000","last_undegraded":"2024-03-30T15:04:03.339823+0000","last_fullsized":"2024-03-30T15:04:03.339823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.403595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":793,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328611+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:04:03.328611+0000","last_peered":"2024-03-30T15:04:03.328611+0000","last_clean":"2024-03-30T15:04:03.328611+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:04:03.328611+0000","last_undegraded":"2024-03-30T15:04:03.328611+0000","last_fullsized":"2024-03-30T15:04:03.328611+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.356946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":706,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327304+0000","last_change":"2024-03-30T15:02:05.626016+0000","last_active":"2024-03-30T15:04:03.327304+0000","last_peered":"2024-03-30T15:04:03.327304+0000","last_clean":"2024-03-30T15:04:03.327304+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:04:03.327304+0000","last_undegraded":"2024-03-30T15:04:03.327304+0000","last_fullsized":"2024-03-30T15:04:03.327304+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:05.625940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.625940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":922,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339798+0000","last_change":"2024-03-30T15:01:48.471009+0000","last_active":"2024-03-30T15:04:03.339798+0000","last_peered":"2024-03-30T15:04:03.339798+0000","last_clean":"2024-03-30T15:04:03.339798+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:04:03.339798+0000","last_undegraded":"2024-03-30T15:04:03.339798+0000","last_fullsized":"2024-03-30T15:04:03.339798+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:48.470884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.470884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":821,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331894+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:04:03.331894+0000","last_peered":"2024-03-30T15:04:03.331894+0000","last_clean":"2024-03-30T15:04:03.331894+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:04:03.331894+0000","last_undegraded":"2024-03-30T15:04:03.331894+0000","last_fullsized":"2024-03-30T15:04:03.331894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":795,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335785+0000","last_change":"2024-03-30T14:59:20.244715+0000","last_active":"2024-03-30T15:04:03.335785+0000","last_peered":"2024-03-30T15:04:03.335785+0000","last_clean":"2024-03-30T15:04:03.335785+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:04:03.335785+0000","last_undegraded":"2024-03-30T15:04:03.335785+0000","last_fullsized":"2024-03-30T15:04:03.335785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:20.244586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T14:59:20.244586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338084+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:04:03.338084+0000","last_peered":"2024-03-30T15:04:03.338084+0000","last_clean":"2024-03-30T15:04:03.338084+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:04:03.338084+0000","last_undegraded":"2024-03-30T15:04:03.338084+0000","last_fullsized":"2024-03-30T15:04:03.338084+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":699,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328134+0000","last_change":"2024-03-30T15:01:52.535873+0000","last_active":"2024-03-30T15:04:03.328134+0000","last_peered":"2024-03-30T15:04:03.328134+0000","last_clean":"2024-03-30T15:04:03.328134+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:04:03.328134+0000","last_undegraded":"2024-03-30T15:04:03.328134+0000","last_fullsized":"2024-03-30T15:04:03.328134+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.535788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.535788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":804,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349097+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:04:03.349097+0000","last_peered":"2024-03-30T15:04:03.349097+0000","last_clean":"2024-03-30T15:04:03.349097+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:04:03.349097+0000","last_undegraded":"2024-03-30T15:04:03.349097+0000","last_fullsized":"2024-03-30T15:04:03.349097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.848800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":832,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340137+0000","last_change":"2024-03-30T14:59:03.618708+0000","last_active":"2024-03-30T15:04:03.340137+0000","last_peered":"2024-03-30T15:04:03.340137+0000","last_clean":"2024-03-30T15:04:03.340137+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:04:03.340137+0000","last_undegraded":"2024-03-30T15:04:03.340137+0000","last_fullsized":"2024-03-30T15:04:03.340137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T14:59:03.618586+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T14:59:03.618586+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":805,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331810+0000","last_change":"2024-03-30T15:01:57.409471+0000","last_active":"2024-03-30T15:04:03.331810+0000","last_peered":"2024-03-30T15:04:03.331810+0000","last_clean":"2024-03-30T15:04:03.331810+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:04:03.331810+0000","last_undegraded":"2024-03-30T15:04:03.331810+0000","last_fullsized":"2024-03-30T15:04:03.331810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.409374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.409374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":793,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331174+0000","last_change":"2024-03-30T15:01:54.450014+0000","last_active":"2024-03-30T15:04:03.331174+0000","last_peered":"2024-03-30T15:04:03.331174+0000","last_clean":"2024-03-30T15:04:03.331174+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:04:03.331174+0000","last_undegraded":"2024-03-30T15:04:03.331174+0000","last_fullsized":"2024-03-30T15:04:03.331174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.449931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.449931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339777+0000","last_change":"2024-03-30T15:01:56.570174+0000","last_active":"2024-03-30T15:04:03.339777+0000","last_peered":"2024-03-30T15:04:03.339777+0000","last_clean":"2024-03-30T15:04:03.339777+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:04:03.339777+0000","last_undegraded":"2024-03-30T15:04:03.339777+0000","last_fullsized":"2024-03-30T15:04:03.339777+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:56.570063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:56.570063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":795,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.336628+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:04:03.336628+0000","last_peered":"2024-03-30T15:04:03.336628+0000","last_clean":"2024-03-30T15:04:03.336628+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:04:03.336628+0000","last_undegraded":"2024-03-30T15:04:03.336628+0000","last_fullsized":"2024-03-30T15:04:03.336628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.978863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":814,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339712+0000","last_change":"2024-03-30T14:59:59.388663+0000","last_active":"2024-03-30T15:04:03.339712+0000","last_peered":"2024-03-30T15:04:03.339712+0000","last_clean":"2024-03-30T15:04:03.339712+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:04:03.339712+0000","last_undegraded":"2024-03-30T15:04:03.339712+0000","last_fullsized":"2024-03-30T15:04:03.339712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T14:59:59.388548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:59.388548+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":1367,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328226+0000","last_change":"2024-03-30T15:02:03.014253+0000","last_active":"2024-03-30T15:04:03.328226+0000","last_peered":"2024-03-30T15:04:03.328226+0000","last_clean":"2024-03-30T15:04:03.328226+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:04:03.328226+0000","last_undegraded":"2024-03-30T15:04:03.328226+0000","last_fullsized":"2024-03-30T15:04:03.328226+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T15:02:03.014132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.014132+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":558,"num_read_kb":553,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":800,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338418+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:04:03.338418+0000","last_peered":"2024-03-30T15:04:03.338418+0000","last_clean":"2024-03-30T15:04:03.338418+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:04:03.338418+0000","last_undegraded":"2024-03-30T15:04:03.338418+0000","last_fullsized":"2024-03-30T15:04:03.338418+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:44.443232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":785,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331575+0000","last_change":"2024-03-30T15:01:12.423769+0000","last_active":"2024-03-30T15:04:03.331575+0000","last_peered":"2024-03-30T15:04:03.331575+0000","last_clean":"2024-03-30T15:04:03.331575+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:04:03.331575+0000","last_undegraded":"2024-03-30T15:04:03.331575+0000","last_fullsized":"2024-03-30T15:04:03.331575+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.423680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.423680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":693,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330147+0000","last_change":"2024-03-30T15:01:18.064257+0000","last_active":"2024-03-30T15:04:03.330147+0000","last_peered":"2024-03-30T15:04:03.330147+0000","last_clean":"2024-03-30T15:04:03.330147+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:04:03.330147+0000","last_undegraded":"2024-03-30T15:04:03.330147+0000","last_fullsized":"2024-03-30T15:04:03.330147+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:18.064173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.064173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":789,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329116+0000","last_change":"2024-03-30T15:02:33.948293+0000","last_active":"2024-03-30T15:04:03.329116+0000","last_peered":"2024-03-30T15:04:03.329116+0000","last_clean":"2024-03-30T15:04:03.329116+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:04:03.329116+0000","last_undegraded":"2024-03-30T15:04:03.329116+0000","last_fullsized":"2024-03-30T15:04:03.329116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:33.948156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.948156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":944,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349536+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:04:03.349536+0000","last_peered":"2024-03-30T15:04:03.349536+0000","last_clean":"2024-03-30T15:04:03.349536+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:04:03.349536+0000","last_undegraded":"2024-03-30T15:04:03.349536+0000","last_fullsized":"2024-03-30T15:04:03.349536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.146207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":689,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334570+0000","last_change":"2024-03-30T14:59:10.163771+0000","last_active":"2024-03-30T15:04:03.334570+0000","last_peered":"2024-03-30T15:04:03.334570+0000","last_clean":"2024-03-30T15:04:03.334570+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:04:03.334570+0000","last_undegraded":"2024-03-30T15:04:03.334570+0000","last_fullsized":"2024-03-30T15:04:03.334570+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:10.163688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:10.163688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1058,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328392+0000","last_change":"2024-03-30T15:02:39.890565+0000","last_active":"2024-03-30T15:04:03.328392+0000","last_peered":"2024-03-30T15:04:03.328392+0000","last_clean":"2024-03-30T15:04:03.328392+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:04:03.328392+0000","last_undegraded":"2024-03-30T15:04:03.328392+0000","last_fullsized":"2024-03-30T15:04:03.328392+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:02:39.890462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.890462+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":783,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.336362+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:04:03.336362+0000","last_peered":"2024-03-30T15:04:03.336362+0000","last_clean":"2024-03-30T15:04:03.336362+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:04:03.336362+0000","last_undegraded":"2024-03-30T15:04:03.336362+0000","last_fullsized":"2024-03-30T15:04:03.336362+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.463993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":709,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349974+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:04:03.349974+0000","last_peered":"2024-03-30T15:04:03.349974+0000","last_clean":"2024-03-30T15:04:03.349974+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:04:03.349974+0000","last_undegraded":"2024-03-30T15:04:03.349974+0000","last_fullsized":"2024-03-30T15:04:03.349974+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.021923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":798,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328541+0000","last_change":"2024-03-30T15:01:58.576718+0000","last_active":"2024-03-30T15:04:03.328541+0000","last_peered":"2024-03-30T15:04:03.328541+0000","last_clean":"2024-03-30T15:04:03.328541+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:04:03.328541+0000","last_undegraded":"2024-03-30T15:04:03.328541+0000","last_fullsized":"2024-03-30T15:04:03.328541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.576585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.576585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":841,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332149+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:04:03.332149+0000","last_peered":"2024-03-30T15:04:03.332149+0000","last_clean":"2024-03-30T15:04:03.332149+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:04:03.332149+0000","last_undegraded":"2024-03-30T15:04:03.332149+0000","last_fullsized":"2024-03-30T15:04:03.332149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:54.532654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":815,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327291+0000","last_change":"2024-03-30T15:01:53.529768+0000","last_active":"2024-03-30T15:04:03.327291+0000","last_peered":"2024-03-30T15:04:03.327291+0000","last_clean":"2024-03-30T15:04:03.327291+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:04:03.327291+0000","last_undegraded":"2024-03-30T15:04:03.327291+0000","last_fullsized":"2024-03-30T15:04:03.327291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.529037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.529037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":805,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331244+0000","last_change":"2024-03-30T15:01:59.392426+0000","last_active":"2024-03-30T15:04:03.331244+0000","last_peered":"2024-03-30T15:04:03.331244+0000","last_clean":"2024-03-30T15:04:03.331244+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:04:03.331244+0000","last_undegraded":"2024-03-30T15:04:03.331244+0000","last_fullsized":"2024-03-30T15:04:03.331244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.392341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":793,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330299+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:04:03.330299+0000","last_peered":"2024-03-30T15:04:03.330299+0000","last_clean":"2024-03-30T15:04:03.330299+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:04:03.330299+0000","last_undegraded":"2024-03-30T15:04:03.330299+0000","last_fullsized":"2024-03-30T15:04:03.330299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.756028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338919+0000","last_change":"2024-03-30T15:01:59.537303+0000","last_active":"2024-03-30T15:04:03.338919+0000","last_peered":"2024-03-30T15:04:03.338919+0000","last_clean":"2024-03-30T15:04:03.338919+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:04:03.338919+0000","last_undegraded":"2024-03-30T15:04:03.338919+0000","last_fullsized":"2024-03-30T15:04:03.338919+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.537225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.537225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":792,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338822+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:04:03.338822+0000","last_peered":"2024-03-30T15:04:03.338822+0000","last_clean":"2024-03-30T15:04:03.338822+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:04:03.338822+0000","last_undegraded":"2024-03-30T15:04:03.338822+0000","last_fullsized":"2024-03-30T15:04:03.338822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":908,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334831+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:04:03.334831+0000","last_peered":"2024-03-30T15:04:03.334831+0000","last_clean":"2024-03-30T15:04:03.334831+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:04:03.334831+0000","last_undegraded":"2024-03-30T15:04:03.334831+0000","last_fullsized":"2024-03-30T15:04:03.334831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.234178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":796,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343845+0000","last_change":"2024-03-30T15:02:00.474885+0000","last_active":"2024-03-30T15:04:03.343845+0000","last_peered":"2024-03-30T15:04:03.343845+0000","last_clean":"2024-03-30T15:04:03.343845+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:04:03.343845+0000","last_undegraded":"2024-03-30T15:04:03.343845+0000","last_fullsized":"2024-03-30T15:04:03.343845+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.474742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.474742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":806,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334610+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:04:03.334610+0000","last_peered":"2024-03-30T15:04:03.334610+0000","last_clean":"2024-03-30T15:04:03.334610+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:04:03.334610+0000","last_undegraded":"2024-03-30T15:04:03.334610+0000","last_fullsized":"2024-03-30T15:04:03.334610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.645581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345092+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:04:03.345092+0000","last_peered":"2024-03-30T15:04:03.345092+0000","last_clean":"2024-03-30T15:04:03.345092+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:04:03.345092+0000","last_undegraded":"2024-03-30T15:04:03.345092+0000","last_fullsized":"2024-03-30T15:04:03.345092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:19.574807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339968+0000","last_change":"2024-03-30T15:01:58.563335+0000","last_active":"2024-03-30T15:04:03.339968+0000","last_peered":"2024-03-30T15:04:03.339968+0000","last_clean":"2024-03-30T15:04:03.339968+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:04:03.339968+0000","last_undegraded":"2024-03-30T15:04:03.339968+0000","last_fullsized":"2024-03-30T15:04:03.339968+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":812,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339899+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:04:03.339899+0000","last_peered":"2024-03-30T15:04:03.339899+0000","last_clean":"2024-03-30T15:04:03.339899+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:04:03.339899+0000","last_undegraded":"2024-03-30T15:04:03.339899+0000","last_fullsized":"2024-03-30T15:04:03.339899+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":795,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335380+0000","last_change":"2024-03-30T14:59:50.165454+0000","last_active":"2024-03-30T15:04:03.335380+0000","last_peered":"2024-03-30T15:04:03.335380+0000","last_clean":"2024-03-30T15:04:03.335380+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:04:03.335380+0000","last_undegraded":"2024-03-30T15:04:03.335380+0000","last_fullsized":"2024-03-30T15:04:03.335380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.165351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.165351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":806,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327544+0000","last_change":"2024-03-30T15:03:26.133453+0000","last_active":"2024-03-30T15:04:03.327544+0000","last_peered":"2024-03-30T15:04:03.327544+0000","last_clean":"2024-03-30T15:04:03.327544+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:04:03.327544+0000","last_undegraded":"2024-03-30T15:04:03.327544+0000","last_fullsized":"2024-03-30T15:04:03.327544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:26.133342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:26.133342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.286995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338675+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:04:03.338675+0000","last_peered":"2024-03-30T15:04:03.338675+0000","last_clean":"2024-03-30T15:04:03.338675+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:04:03.338675+0000","last_undegraded":"2024-03-30T15:04:03.338675+0000","last_fullsized":"2024-03-30T15:04:03.338675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:14.649484+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":693,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345031+0000","last_change":"2024-03-30T15:01:53.490679+0000","last_active":"2024-03-30T15:04:03.345031+0000","last_peered":"2024-03-30T15:04:03.345031+0000","last_clean":"2024-03-30T15:04:03.345031+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:04:03.345031+0000","last_undegraded":"2024-03-30T15:04:03.345031+0000","last_fullsized":"2024-03-30T15:04:03.345031+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.490591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.490591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349097+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:04:03.349097+0000","last_peered":"2024-03-30T15:04:03.349097+0000","last_clean":"2024-03-30T15:04:03.349097+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:04:03.349097+0000","last_undegraded":"2024-03-30T15:04:03.349097+0000","last_fullsized":"2024-03-30T15:04:03.349097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:41.052823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330201+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:04:03.330201+0000","last_peered":"2024-03-30T15:04:03.330201+0000","last_clean":"2024-03-30T15:04:03.330201+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:04:03.330201+0000","last_undegraded":"2024-03-30T15:04:03.330201+0000","last_fullsized":"2024-03-30T15:04:03.330201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":806,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328004+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:04:03.328004+0000","last_peered":"2024-03-30T15:04:03.328004+0000","last_clean":"2024-03-30T15:04:03.328004+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:04:03.328004+0000","last_undegraded":"2024-03-30T15:04:03.328004+0000","last_fullsized":"2024-03-30T15:04:03.328004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.803620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":785,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327981+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:04:03.327981+0000","last_peered":"2024-03-30T15:04:03.327981+0000","last_clean":"2024-03-30T15:04:03.327981+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:04:03.327981+0000","last_undegraded":"2024-03-30T15:04:03.327981+0000","last_fullsized":"2024-03-30T15:04:03.327981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":723,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348623+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:04:03.348623+0000","last_peered":"2024-03-30T15:04:03.348623+0000","last_clean":"2024-03-30T15:04:03.348623+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:04:03.348623+0000","last_undegraded":"2024-03-30T15:04:03.348623+0000","last_fullsized":"2024-03-30T15:04:03.348623+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.286336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":798,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327220+0000","last_change":"2024-03-30T15:01:55.535170+0000","last_active":"2024-03-30T15:04:03.327220+0000","last_peered":"2024-03-30T15:04:03.327220+0000","last_clean":"2024-03-30T15:04:03.327220+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:04:03.327220+0000","last_undegraded":"2024-03-30T15:04:03.327220+0000","last_fullsized":"2024-03-30T15:04:03.327220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:55.535057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:01:55.535057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":823,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343958+0000","last_change":"2024-03-30T15:03:19.519526+0000","last_active":"2024-03-30T15:04:03.343958+0000","last_peered":"2024-03-30T15:04:03.343958+0000","last_clean":"2024-03-30T15:04:03.343958+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:04:03.343958+0000","last_undegraded":"2024-03-30T15:04:03.343958+0000","last_fullsized":"2024-03-30T15:04:03.343958+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:03:19.519440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.519440+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.770107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329260+0000","last_change":"2024-03-30T15:02:01.090736+0000","last_active":"2024-03-30T15:04:03.329260+0000","last_peered":"2024-03-30T15:04:03.329260+0000","last_clean":"2024-03-30T15:04:03.329260+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:04:03.329260+0000","last_undegraded":"2024-03-30T15:04:03.329260+0000","last_fullsized":"2024-03-30T15:04:03.329260+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:01.090640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.090640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328668+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:04:03.328668+0000","last_peered":"2024-03-30T15:04:03.328668+0000","last_clean":"2024-03-30T15:04:03.328668+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:04:03.328668+0000","last_undegraded":"2024-03-30T15:04:03.328668+0000","last_fullsized":"2024-03-30T15:04:03.328668+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.943192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":693,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343472+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:04:03.343472+0000","last_peered":"2024-03-30T15:04:03.343472+0000","last_clean":"2024-03-30T15:04:03.343472+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:04:03.343472+0000","last_undegraded":"2024-03-30T15:04:03.343472+0000","last_fullsized":"2024-03-30T15:04:03.343472+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":911,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328746+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:04:03.328746+0000","last_peered":"2024-03-30T15:04:03.328746+0000","last_clean":"2024-03-30T15:04:03.328746+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:04:03.328746+0000","last_undegraded":"2024-03-30T15:04:03.328746+0000","last_fullsized":"2024-03-30T15:04:03.328746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.736984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":816,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334285+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:04:03.334285+0000","last_peered":"2024-03-30T15:04:03.334285+0000","last_clean":"2024-03-30T15:04:03.334285+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:04:03.334285+0000","last_undegraded":"2024-03-30T15:04:03.334285+0000","last_fullsized":"2024-03-30T15:04:03.334285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:30.964164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":809,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329511+0000","last_change":"2024-03-30T15:02:05.004181+0000","last_active":"2024-03-30T15:04:03.329511+0000","last_peered":"2024-03-30T15:04:03.329511+0000","last_clean":"2024-03-30T15:04:03.329511+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:04:03.329511+0000","last_undegraded":"2024-03-30T15:04:03.329511+0000","last_fullsized":"2024-03-30T15:04:03.329511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:02:05.004073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.004073+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":783,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334859+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:04:03.334859+0000","last_peered":"2024-03-30T15:04:03.334859+0000","last_clean":"2024-03-30T15:04:03.334859+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:03.334859+0000","last_undegraded":"2024-03-30T15:04:03.334859+0000","last_fullsized":"2024-03-30T15:04:03.334859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349967+0000","last_change":"2024-03-30T15:02:32.380969+0000","last_active":"2024-03-30T15:04:03.349967+0000","last_peered":"2024-03-30T15:04:03.349967+0000","last_clean":"2024-03-30T15:04:03.349967+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:04:03.349967+0000","last_undegraded":"2024-03-30T15:04:03.349967+0000","last_fullsized":"2024-03-30T15:04:03.349967+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.380822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.380822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":375,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348777+0000","last_change":"2024-03-30T15:03:52.115510+0000","last_active":"2024-03-30T15:04:03.348777+0000","last_peered":"2024-03-30T15:04:03.348777+0000","last_clean":"2024-03-30T15:04:03.348777+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:04:03.348777+0000","last_undegraded":"2024-03-30T15:04:03.348777+0000","last_fullsized":"2024-03-30T15:04:03.348777+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T15:03:52.115417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.115417+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.716598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":812,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348460+0000","last_change":"2024-03-30T15:02:45.427403+0000","last_active":"2024-03-30T15:04:03.348460+0000","last_peered":"2024-03-30T15:04:03.348460+0000","last_clean":"2024-03-30T15:04:03.348460+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:04:03.348460+0000","last_undegraded":"2024-03-30T15:04:03.348460+0000","last_fullsized":"2024-03-30T15:04:03.348460+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:02:45.427234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":786,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335730+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:04:03.335730+0000","last_peered":"2024-03-30T15:04:03.335730+0000","last_clean":"2024-03-30T15:04:03.335730+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:04:03.335730+0000","last_undegraded":"2024-03-30T15:04:03.335730+0000","last_fullsized":"2024-03-30T15:04:03.335730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.325342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":784,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.329175+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:04:03.329175+0000","last_peered":"2024-03-30T15:04:03.329175+0000","last_clean":"2024-03-30T15:04:03.329175+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:04:03.329175+0000","last_undegraded":"2024-03-30T15:04:03.329175+0000","last_fullsized":"2024-03-30T15:04:03.329175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.682878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":693,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344661+0000","last_change":"2024-03-30T15:01:11.580352+0000","last_active":"2024-03-30T15:04:03.344661+0000","last_peered":"2024-03-30T15:04:03.344661+0000","last_clean":"2024-03-30T15:04:03.344661+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:04:03.344661+0000","last_undegraded":"2024-03-30T15:04:03.344661+0000","last_fullsized":"2024-03-30T15:04:03.344661+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:11.580276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.580276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":811,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339962+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:04:03.339962+0000","last_peered":"2024-03-30T15:04:03.339962+0000","last_clean":"2024-03-30T15:04:03.339962+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:04:03.339962+0000","last_undegraded":"2024-03-30T15:04:03.339962+0000","last_fullsized":"2024-03-30T15:04:03.339962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.283455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":805,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328281+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:04:03.328281+0000","last_peered":"2024-03-30T15:04:03.328281+0000","last_clean":"2024-03-30T15:04:03.328281+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:04:03.328281+0000","last_undegraded":"2024-03-30T15:04:03.328281+0000","last_fullsized":"2024-03-30T15:04:03.328281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:30.940258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332284+0000","last_change":"2024-03-30T15:01:19.390146+0000","last_active":"2024-03-30T15:04:03.332284+0000","last_peered":"2024-03-30T15:04:03.332284+0000","last_clean":"2024-03-30T15:04:03.332284+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:04:03.332284+0000","last_undegraded":"2024-03-30T15:04:03.332284+0000","last_fullsized":"2024-03-30T15:04:03.332284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.390029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.390029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":801,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328210+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:04:03.328210+0000","last_peered":"2024-03-30T15:04:03.328210+0000","last_clean":"2024-03-30T15:04:03.328210+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:04:03.328210+0000","last_undegraded":"2024-03-30T15:04:03.328210+0000","last_fullsized":"2024-03-30T15:04:03.328210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.180120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338751+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:04:03.338751+0000","last_peered":"2024-03-30T15:04:03.338751+0000","last_clean":"2024-03-30T15:04:03.338751+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:04:03.338751+0000","last_undegraded":"2024-03-30T15:04:03.338751+0000","last_fullsized":"2024-03-30T15:04:03.338751+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":806,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.336296+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:04:03.336296+0000","last_peered":"2024-03-30T15:04:03.336296+0000","last_clean":"2024-03-30T15:04:03.336296+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:04:03.336296+0000","last_undegraded":"2024-03-30T15:04:03.336296+0000","last_fullsized":"2024-03-30T15:04:03.336296+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.204528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":800,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338547+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:04:03.338547+0000","last_peered":"2024-03-30T15:04:03.338547+0000","last_clean":"2024-03-30T15:04:03.338547+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:04:03.338547+0000","last_undegraded":"2024-03-30T15:04:03.338547+0000","last_fullsized":"2024-03-30T15:04:03.338547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":796,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328632+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:04:03.328632+0000","last_peered":"2024-03-30T15:04:03.328632+0000","last_clean":"2024-03-30T15:04:03.328632+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:04:03.328632+0000","last_undegraded":"2024-03-30T15:04:03.328632+0000","last_fullsized":"2024-03-30T15:04:03.328632+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.585322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":775,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345662+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:04:03.345662+0000","last_peered":"2024-03-30T15:04:03.345662+0000","last_clean":"2024-03-30T15:04:03.345662+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:04:03.345662+0000","last_undegraded":"2024-03-30T15:04:03.345662+0000","last_fullsized":"2024-03-30T15:04:03.345662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338829+0000","last_change":"2024-03-30T15:01:52.471709+0000","last_active":"2024-03-30T15:04:03.338829+0000","last_peered":"2024-03-30T15:04:03.338829+0000","last_clean":"2024-03-30T15:04:03.338829+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:04:03.338829+0000","last_undegraded":"2024-03-30T15:04:03.338829+0000","last_fullsized":"2024-03-30T15:04:03.338829+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.471603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.471603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":731,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331864+0000","last_change":"2024-03-30T15:01:18.430986+0000","last_active":"2024-03-30T15:04:03.331864+0000","last_peered":"2024-03-30T15:04:03.331864+0000","last_clean":"2024-03-30T15:04:03.331864+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:04:03.331864+0000","last_undegraded":"2024-03-30T15:04:03.331864+0000","last_fullsized":"2024-03-30T15:04:03.331864+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:01:18.430887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.430887+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328181+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:04:03.328181+0000","last_peered":"2024-03-30T15:04:03.328181+0000","last_clean":"2024-03-30T15:04:03.328181+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:04:03.328181+0000","last_undegraded":"2024-03-30T15:04:03.328181+0000","last_fullsized":"2024-03-30T15:04:03.328181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:37.842104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":911,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335609+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:04:03.335609+0000","last_peered":"2024-03-30T15:04:03.335609+0000","last_clean":"2024-03-30T15:04:03.335609+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:04:03.335609+0000","last_undegraded":"2024-03-30T15:04:03.335609+0000","last_fullsized":"2024-03-30T15:04:03.335609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.083796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":789,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339654+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:04:03.339654+0000","last_peered":"2024-03-30T15:04:03.339654+0000","last_clean":"2024-03-30T15:04:03.339654+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:04:03.339654+0000","last_undegraded":"2024-03-30T15:04:03.339654+0000","last_fullsized":"2024-03-30T15:04:03.339654+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.685331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344097+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:04:03.344097+0000","last_peered":"2024-03-30T15:04:03.344097+0000","last_clean":"2024-03-30T15:04:03.344097+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:04:03.344097+0000","last_undegraded":"2024-03-30T15:04:03.344097+0000","last_fullsized":"2024-03-30T15:04:03.344097+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":816,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344175+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:04:03.344175+0000","last_peered":"2024-03-30T15:04:03.344175+0000","last_clean":"2024-03-30T15:04:03.344175+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:04:03.344175+0000","last_undegraded":"2024-03-30T15:04:03.344175+0000","last_fullsized":"2024-03-30T15:04:03.344175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.342982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":796,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345332+0000","last_change":"2024-03-30T15:01:21.453162+0000","last_active":"2024-03-30T15:04:03.345332+0000","last_peered":"2024-03-30T15:04:03.345332+0000","last_clean":"2024-03-30T15:04:03.345332+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:04:03.345332+0000","last_undegraded":"2024-03-30T15:04:03.345332+0000","last_fullsized":"2024-03-30T15:04:03.345332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:21.453084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:21.453084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331818+0000","last_change":"2024-03-30T15:02:03.477727+0000","last_active":"2024-03-30T15:04:03.331818+0000","last_peered":"2024-03-30T15:04:03.331818+0000","last_clean":"2024-03-30T15:04:03.331818+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:04:03.331818+0000","last_undegraded":"2024-03-30T15:04:03.331818+0000","last_fullsized":"2024-03-30T15:04:03.331818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.477653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.477653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338442+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:04:03.338442+0000","last_peered":"2024-03-30T15:04:03.338442+0000","last_clean":"2024-03-30T15:04:03.338442+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:04:03.338442+0000","last_undegraded":"2024-03-30T15:04:03.338442+0000","last_fullsized":"2024-03-30T15:04:03.338442+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343984+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:04:03.343984+0000","last_peered":"2024-03-30T15:04:03.343984+0000","last_clean":"2024-03-30T15:04:03.343984+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:04:03.343984+0000","last_undegraded":"2024-03-30T15:04:03.343984+0000","last_fullsized":"2024-03-30T15:04:03.343984+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":809,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331997+0000","last_change":"2024-03-30T15:02:00.404994+0000","last_active":"2024-03-30T15:04:03.331997+0000","last_peered":"2024-03-30T15:04:03.331997+0000","last_clean":"2024-03-30T15:04:03.331997+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:04:03.331997+0000","last_undegraded":"2024-03-30T15:04:03.331997+0000","last_fullsized":"2024-03-30T15:04:03.331997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:02:00.404907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.404907+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":811,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.330066+0000","last_change":"2024-03-30T15:02:02.044673+0000","last_active":"2024-03-30T15:04:03.330066+0000","last_peered":"2024-03-30T15:04:03.330066+0000","last_clean":"2024-03-30T15:04:03.330066+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:04:03.330066+0000","last_undegraded":"2024-03-30T15:04:03.330066+0000","last_fullsized":"2024-03-30T15:04:03.330066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:02:02.044556+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.044556+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":700,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340021+0000","last_change":"2024-03-30T15:03:59.603193+0000","last_active":"2024-03-30T15:04:03.340021+0000","last_peered":"2024-03-30T15:04:03.340021+0000","last_clean":"2024-03-30T15:04:03.340021+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:04:03.340021+0000","last_undegraded":"2024-03-30T15:04:03.340021+0000","last_fullsized":"2024-03-30T15:04:03.340021+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.946216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338138+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:04:03.338138+0000","last_peered":"2024-03-30T15:04:03.338138+0000","last_clean":"2024-03-30T15:04:03.338138+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:04:03.338138+0000","last_undegraded":"2024-03-30T15:04:03.338138+0000","last_fullsized":"2024-03-30T15:04:03.338138+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:07.024180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338305+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:04:03.338305+0000","last_peered":"2024-03-30T15:04:03.338305+0000","last_clean":"2024-03-30T15:04:03.338305+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:04:03.338305+0000","last_undegraded":"2024-03-30T15:04:03.338305+0000","last_fullsized":"2024-03-30T15:04:03.338305+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":735,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328622+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:04:03.328622+0000","last_peered":"2024-03-30T15:04:03.328622+0000","last_clean":"2024-03-30T15:04:03.328622+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:04:03.328622+0000","last_undegraded":"2024-03-30T15:04:03.328622+0000","last_fullsized":"2024-03-30T15:04:03.328622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":794,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335766+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:04:03.335766+0000","last_peered":"2024-03-30T15:04:03.335766+0000","last_clean":"2024-03-30T15:04:03.335766+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:04:03.335766+0000","last_undegraded":"2024-03-30T15:04:03.335766+0000","last_fullsized":"2024-03-30T15:04:03.335766+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":789,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328743+0000","last_change":"2024-03-30T15:02:41.903716+0000","last_active":"2024-03-30T15:04:03.328743+0000","last_peered":"2024-03-30T15:04:03.328743+0000","last_clean":"2024-03-30T15:04:03.328743+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:04:03.328743+0000","last_undegraded":"2024-03-30T15:04:03.328743+0000","last_fullsized":"2024-03-30T15:04:03.328743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.903609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.903609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339677+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:04:03.339677+0000","last_peered":"2024-03-30T15:04:03.339677+0000","last_clean":"2024-03-30T15:04:03.339677+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:04:03.339677+0000","last_undegraded":"2024-03-30T15:04:03.339677+0000","last_fullsized":"2024-03-30T15:04:03.339677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331358+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:04:03.331358+0000","last_peered":"2024-03-30T15:04:03.331358+0000","last_clean":"2024-03-30T15:04:03.331358+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:04:03.331358+0000","last_undegraded":"2024-03-30T15:04:03.331358+0000","last_fullsized":"2024-03-30T15:04:03.331358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.008953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":365,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.347458+0000","last_change":"2024-03-30T15:04:01.043987+0000","last_active":"2024-03-30T15:04:03.347458+0000","last_peered":"2024-03-30T15:04:03.347458+0000","last_clean":"2024-03-30T15:04:03.347458+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:04:03.347458+0000","last_undegraded":"2024-03-30T15:04:03.347458+0000","last_fullsized":"2024-03-30T15:04:03.347458+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:04:01.043885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.043885+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.133940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":799,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340154+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:04:03.340154+0000","last_peered":"2024-03-30T15:04:03.340154+0000","last_clean":"2024-03-30T15:04:03.340154+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:04:03.340154+0000","last_undegraded":"2024-03-30T15:04:03.340154+0000","last_fullsized":"2024-03-30T15:04:03.340154+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":804,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349347+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:04:03.349347+0000","last_peered":"2024-03-30T15:04:03.349347+0000","last_clean":"2024-03-30T15:04:03.349347+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:04:03.349347+0000","last_undegraded":"2024-03-30T15:04:03.349347+0000","last_fullsized":"2024-03-30T15:04:03.349347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:15.713330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":785,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328517+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:04:03.328517+0000","last_peered":"2024-03-30T15:04:03.328517+0000","last_clean":"2024-03-30T15:04:03.328517+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:04:03.328517+0000","last_undegraded":"2024-03-30T15:04:03.328517+0000","last_fullsized":"2024-03-30T15:04:03.328517+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:06.141724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":701,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350185+0000","last_change":"2024-03-30T15:02:35.380537+0000","last_active":"2024-03-30T15:04:03.350185+0000","last_peered":"2024-03-30T15:04:03.350185+0000","last_clean":"2024-03-30T15:04:03.350185+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:04:03.350185+0000","last_undegraded":"2024-03-30T15:04:03.350185+0000","last_fullsized":"2024-03-30T15:04:03.350185+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.380454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.380454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":824,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350193+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:04:03.350193+0000","last_peered":"2024-03-30T15:04:03.350193+0000","last_clean":"2024-03-30T15:04:03.350193+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:04:03.350193+0000","last_undegraded":"2024-03-30T15:04:03.350193+0000","last_fullsized":"2024-03-30T15:04:03.350193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:04.864954+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":701,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338210+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:04:03.338210+0000","last_peered":"2024-03-30T15:04:03.338210+0000","last_clean":"2024-03-30T15:04:03.338210+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:04:03.338210+0000","last_undegraded":"2024-03-30T15:04:03.338210+0000","last_fullsized":"2024-03-30T15:04:03.338210+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.782926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":792,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340098+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:04:03.340098+0000","last_peered":"2024-03-30T15:04:03.340098+0000","last_clean":"2024-03-30T15:04:03.340098+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:04:03.340098+0000","last_undegraded":"2024-03-30T15:04:03.340098+0000","last_fullsized":"2024-03-30T15:04:03.340098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:43.075639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":792,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.332040+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:04:03.332040+0000","last_peered":"2024-03-30T15:04:03.332040+0000","last_clean":"2024-03-30T15:04:03.332040+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:04:03.332040+0000","last_undegraded":"2024-03-30T15:04:03.332040+0000","last_fullsized":"2024-03-30T15:04:03.332040+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":709,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350306+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:04:03.350306+0000","last_peered":"2024-03-30T15:04:03.350306+0000","last_clean":"2024-03-30T15:04:03.350306+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:04:03.350306+0000","last_undegraded":"2024-03-30T15:04:03.350306+0000","last_fullsized":"2024-03-30T15:04:03.350306+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:29.813607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":816,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339912+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:04:03.339912+0000","last_peered":"2024-03-30T15:04:03.339912+0000","last_clean":"2024-03-30T15:04:03.339912+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:04:03.339912+0000","last_undegraded":"2024-03-30T15:04:03.339912+0000","last_fullsized":"2024-03-30T15:04:03.339912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:38.106835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":797,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331378+0000","last_change":"2024-03-30T15:01:47.383230+0000","last_active":"2024-03-30T15:04:03.331378+0000","last_peered":"2024-03-30T15:04:03.331378+0000","last_clean":"2024-03-30T15:04:03.331378+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:04:03.331378+0000","last_undegraded":"2024-03-30T15:04:03.331378+0000","last_fullsized":"2024-03-30T15:04:03.331378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:01:47.383148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.383148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":929,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350044+0000","last_change":"2024-03-30T15:02:51.451850+0000","last_active":"2024-03-30T15:04:03.350044+0000","last_peered":"2024-03-30T15:04:03.350044+0000","last_clean":"2024-03-30T15:04:03.350044+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:04:03.350044+0000","last_undegraded":"2024-03-30T15:04:03.350044+0000","last_fullsized":"2024-03-30T15:04:03.350044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:51.451690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:51.451690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":794,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328690+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:04:03.328690+0000","last_peered":"2024-03-30T15:04:03.328690+0000","last_clean":"2024-03-30T15:04:03.328690+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:04:03.328690+0000","last_undegraded":"2024-03-30T15:04:03.328690+0000","last_fullsized":"2024-03-30T15:04:03.328690+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:07.241016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":701,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349245+0000","last_change":"2024-03-30T15:02:03.146988+0000","last_active":"2024-03-30T15:04:03.349245+0000","last_peered":"2024-03-30T15:04:03.349245+0000","last_clean":"2024-03-30T15:04:03.349245+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:04:03.349245+0000","last_undegraded":"2024-03-30T15:04:03.349245+0000","last_fullsized":"2024-03-30T15:04:03.349245+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.146842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.146842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":739,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331832+0000","last_change":"2024-03-30T15:01:20.341161+0000","last_active":"2024-03-30T15:04:03.331832+0000","last_peered":"2024-03-30T15:04:03.331832+0000","last_clean":"2024-03-30T15:04:03.331832+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:04:03.331832+0000","last_undegraded":"2024-03-30T15:04:03.331832+0000","last_fullsized":"2024-03-30T15:04:03.331832+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:01:20.341036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.341036+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":795,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344414+0000","last_change":"2024-03-30T15:03:21.522112+0000","last_active":"2024-03-30T15:04:03.344414+0000","last_peered":"2024-03-30T15:04:03.344414+0000","last_clean":"2024-03-30T15:04:03.344414+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:04:03.344414+0000","last_undegraded":"2024-03-30T15:04:03.344414+0000","last_fullsized":"2024-03-30T15:04:03.344414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.522038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.522038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.587391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":787,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345685+0000","last_change":"2024-03-30T15:03:23.551230+0000","last_active":"2024-03-30T15:04:03.345685+0000","last_peered":"2024-03-30T15:04:03.345685+0000","last_clean":"2024-03-30T15:04:03.345685+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:04:03.345685+0000","last_undegraded":"2024-03-30T15:04:03.345685+0000","last_fullsized":"2024-03-30T15:04:03.345685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:23.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.577963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":705,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.350288+0000","last_change":"2024-03-30T15:02:49.425842+0000","last_active":"2024-03-30T15:04:03.350288+0000","last_peered":"2024-03-30T15:04:03.350288+0000","last_clean":"2024-03-30T15:04:03.350288+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:04:03.350288+0000","last_undegraded":"2024-03-30T15:04:03.350288+0000","last_fullsized":"2024-03-30T15:04:03.350288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:49.425762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.425762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335581+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:04:03.335581+0000","last_peered":"2024-03-30T15:04:03.335581+0000","last_clean":"2024-03-30T15:04:03.335581+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:04:03.335581+0000","last_undegraded":"2024-03-30T15:04:03.335581+0000","last_fullsized":"2024-03-30T15:04:03.335581+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":841,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328024+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:04:03.328024+0000","last_peered":"2024-03-30T15:04:03.328024+0000","last_clean":"2024-03-30T15:04:03.328024+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:04:03.328024+0000","last_undegraded":"2024-03-30T15:04:03.328024+0000","last_fullsized":"2024-03-30T15:04:03.328024+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.076544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":799,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.340130+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:04:03.340130+0000","last_peered":"2024-03-30T15:04:03.340130+0000","last_clean":"2024-03-30T15:04:03.340130+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:04:03.340130+0000","last_undegraded":"2024-03-30T15:04:03.340130+0000","last_fullsized":"2024-03-30T15:04:03.340130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331972+0000","last_change":"2024-03-30T15:01:15.391982+0000","last_active":"2024-03-30T15:04:03.331972+0000","last_peered":"2024-03-30T15:04:03.331972+0000","last_clean":"2024-03-30T15:04:03.331972+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:04:03.331972+0000","last_undegraded":"2024-03-30T15:04:03.331972+0000","last_fullsized":"2024-03-30T15:04:03.331972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:15.391906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:01:15.391906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":784,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.345240+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:04:03.345240+0000","last_peered":"2024-03-30T15:04:03.345240+0000","last_clean":"2024-03-30T15:04:03.345240+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:04:03.345240+0000","last_undegraded":"2024-03-30T15:04:03.345240+0000","last_fullsized":"2024-03-30T15:04:03.345240+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:57.177473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":702,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331261+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:04:03.331261+0000","last_peered":"2024-03-30T15:04:03.331261+0000","last_clean":"2024-03-30T15:04:03.331261+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:04:03.331261+0000","last_undegraded":"2024-03-30T15:04:03.331261+0000","last_fullsized":"2024-03-30T15:04:03.331261+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":825,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.516432+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:04:03.516432+0000","last_peered":"2024-03-30T15:04:03.516432+0000","last_clean":"2024-03-30T15:04:03.516432+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:04:03.516432+0000","last_undegraded":"2024-03-30T15:04:03.516432+0000","last_fullsized":"2024-03-30T15:04:03.516432+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.236100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":794,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335364+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:04:03.335364+0000","last_peered":"2024-03-30T15:04:03.335364+0000","last_clean":"2024-03-30T15:04:03.335364+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:04:03.335364+0000","last_undegraded":"2024-03-30T15:04:03.335364+0000","last_fullsized":"2024-03-30T15:04:03.335364+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.425048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.328157+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:04:03.328157+0000","last_peered":"2024-03-30T15:04:03.328157+0000","last_clean":"2024-03-30T15:04:03.328157+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:04:03.328157+0000","last_undegraded":"2024-03-30T15:04:03.328157+0000","last_fullsized":"2024-03-30T15:04:03.328157+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":782,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334742+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:04:03.334742+0000","last_peered":"2024-03-30T15:04:03.334742+0000","last_clean":"2024-03-30T15:04:03.334742+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:04:03.334742+0000","last_undegraded":"2024-03-30T15:04:03.334742+0000","last_fullsized":"2024-03-30T15:04:03.334742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":696,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331247+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:04:03.331247+0000","last_peered":"2024-03-30T15:04:03.331247+0000","last_clean":"2024-03-30T15:04:03.331247+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:04:03.331247+0000","last_undegraded":"2024-03-30T15:04:03.331247+0000","last_fullsized":"2024-03-30T15:04:03.331247+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:08.603935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":799,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344406+0000","last_change":"2024-03-30T14:59:52.399600+0000","last_active":"2024-03-30T15:04:03.344406+0000","last_peered":"2024-03-30T15:04:03.344406+0000","last_clean":"2024-03-30T15:04:03.344406+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:04:03.344406+0000","last_undegraded":"2024-03-30T15:04:03.344406+0000","last_fullsized":"2024-03-30T15:04:03.344406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:52.399475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.399475+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":794,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.335502+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:04:03.335502+0000","last_peered":"2024-03-30T15:04:03.335502+0000","last_clean":"2024-03-30T15:04:03.335502+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:04:03.335502+0000","last_undegraded":"2024-03-30T15:04:03.335502+0000","last_fullsized":"2024-03-30T15:04:03.335502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:51.403630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":786,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334927+0000","last_change":"2024-03-30T15:02:44.998020+0000","last_active":"2024-03-30T15:04:03.334927+0000","last_peered":"2024-03-30T15:04:03.334927+0000","last_clean":"2024-03-30T15:04:03.334927+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:04:03.334927+0000","last_undegraded":"2024-03-30T15:04:03.334927+0000","last_fullsized":"2024-03-30T15:04:03.334927+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.997898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":787,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338279+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:04:03.338279+0000","last_peered":"2024-03-30T15:04:03.338279+0000","last_clean":"2024-03-30T15:04:03.338279+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:04:03.338279+0000","last_undegraded":"2024-03-30T15:04:03.338279+0000","last_fullsized":"2024-03-30T15:04:03.338279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":692,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.334530+0000","last_change":"2024-03-30T15:01:16.348182+0000","last_active":"2024-03-30T15:04:03.334530+0000","last_peered":"2024-03-30T15:04:03.334530+0000","last_clean":"2024-03-30T15:04:03.334530+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:04:03.334530+0000","last_undegraded":"2024-03-30T15:04:03.334530+0000","last_fullsized":"2024-03-30T15:04:03.334530+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.348072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.348072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":791,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.327344+0000","last_change":"2024-03-30T15:02:33.875911+0000","last_active":"2024-03-30T15:04:03.327344+0000","last_peered":"2024-03-30T15:04:03.327344+0000","last_clean":"2024-03-30T15:04:03.327344+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:04:03.327344+0000","last_undegraded":"2024-03-30T15:04:03.327344+0000","last_fullsized":"2024-03-30T15:04:03.327344+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T15:02:33.875778+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.875778+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":707,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.349995+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:04:03.349995+0000","last_peered":"2024-03-30T15:04:03.349995+0000","last_clean":"2024-03-30T15:04:03.349995+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:04:03.349995+0000","last_undegraded":"2024-03-30T15:04:03.349995+0000","last_fullsized":"2024-03-30T15:04:03.349995+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:32.273285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":787,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343388+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:04:03.343388+0000","last_peered":"2024-03-30T15:04:03.343388+0000","last_clean":"2024-03-30T15:04:03.343388+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:04:03.343388+0000","last_undegraded":"2024-03-30T15:04:03.343388+0000","last_fullsized":"2024-03-30T15:04:03.343388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.751185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":350,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.348312+0000","last_change":"2024-03-30T15:03:16.163490+0000","last_active":"2024-03-30T15:04:03.348312+0000","last_peered":"2024-03-30T15:04:03.348312+0000","last_clean":"2024-03-30T15:04:03.348312+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:04:03.348312+0000","last_undegraded":"2024-03-30T15:04:03.348312+0000","last_fullsized":"2024-03-30T15:04:03.348312+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.163376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.163376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:36.277238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":697,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.338487+0000","last_change":"2024-03-30T15:01:17.584059+0000","last_active":"2024-03-30T15:04:03.338487+0000","last_peered":"2024-03-30T15:04:03.338487+0000","last_clean":"2024-03-30T15:04:03.338487+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:04:03.338487+0000","last_undegraded":"2024-03-30T15:04:03.338487+0000","last_fullsized":"2024-03-30T15:04:03.338487+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:17.583984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:17.583984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":823,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.343516+0000","last_change":"2024-03-30T15:02:19.657930+0000","last_active":"2024-03-30T15:04:03.343516+0000","last_peered":"2024-03-30T15:04:03.343516+0000","last_clean":"2024-03-30T15:04:03.343516+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:04:03.343516+0000","last_undegraded":"2024-03-30T15:04:03.343516+0000","last_fullsized":"2024-03-30T15:04:03.343516+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":800,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.339997+0000","last_change":"2024-03-30T14:59:53.411916+0000","last_active":"2024-03-30T15:04:03.339997+0000","last_peered":"2024-03-30T15:04:03.339997+0000","last_clean":"2024-03-30T15:04:03.339997+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:04:03.339997+0000","last_undegraded":"2024-03-30T15:04:03.339997+0000","last_fullsized":"2024-03-30T15:04:03.339997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:53.411813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:53.411813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":788,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.331164+0000","last_change":"2024-03-30T15:01:53.452032+0000","last_active":"2024-03-30T15:04:03.331164+0000","last_peered":"2024-03-30T15:04:03.331164+0000","last_clean":"2024-03-30T15:04:03.331164+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:04:03.331164+0000","last_undegraded":"2024-03-30T15:04:03.331164+0000","last_fullsized":"2024-03-30T15:04:03.331164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.451927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.451927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":783,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-30T15:04:03.344083+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:04:03.344083+0000","last_peered":"2024-03-30T15:04:03.344083+0000","last_clean":"2024-03-30T15:04:03.344083+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:04:03.344083+0000","last_undegraded":"2024-03-30T15:04:03.344083+0000","last_fullsized":"2024-03-30T15:04:03.344083+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:40.511390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":297,"num_pg":32,"stat_sum":{"num_bytes":18,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195793,"num_objects":160,"num_object_clones":2,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":600,"num_write_kb":12689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26492928,"data_stored":24587379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1890,"ondisk_log_size":1890,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":558,"num_read_kb":553,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005174,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87856,"kb_used_data":12252,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657296,"statfs":{"total":95995035648,"available":95905071104,"internally_reserved":0,"allocated":12546048,"data_stored":9089469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299515,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75340,"kb_used_data":12192,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669812,"statfs":{"total":95995035648,"available":95917887488,"internally_reserved":0,"allocated":12484608,"data_stored":9087042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83356,"kb_used_data":7728,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93661796,"statfs":{"total":95995035648,"available":95909679104,"internally_reserved":0,"allocated":7913472,"data_stored":4497852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65864,"kb_used_data":12456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93679288,"statfs":{"total":95995035648,"available":95927590912,"internally_reserved":0,"allocated":12754944,"data_stored":9292484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411471,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78888,"kb_used_data":15764,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93666264,"statfs":{"total":95995035648,"available":95914254336,"internally_reserved":0,"allocated":16142336,"data_stored":12691418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117131,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75068,"kb_used_data":11920,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670084,"statfs":{"total":95995035648,"available":95918166016,"internally_reserved":0,"allocated":12206080,"data_stored":8693305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953605,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70944,"kb_used_data":7808,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674208,"statfs":{"total":95995035648,"available":95922388992,"internally_reserved":0,"allocated":7995392,"data_stored":4507217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659264,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70908,"kb_used_data":7784,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674244,"statfs":{"total":95995035648,"available":95922425856,"internally_reserved":0,"allocated":7970816,"data_stored":4497515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4194807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":8194889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4194929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4000475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":4000574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":14,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":297,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:04:05.520 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-03-30T15:04:05.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:05.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:05.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:04:05.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[29876]: from='client.50089 v1:172.21.15.28:0/1101537077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T15:04:05.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T15:04:05.615 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T15:04:05.615 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:05 smithi028 ceph-mon[41614]: from='client.50089 v1:172.21.15.28:0/1101537077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:05.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T15:04:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T15:04:05.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: osdmap e451: 8 total, 8 up, 8 in 2024-03-30T15:04:05.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:05 smithi134 ceph-mon[33714]: from='client.50089 v1:172.21.15.28:0/1101537077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:05.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:04:05.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:05.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:05.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:05.905 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:05.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:05.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:06.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:06.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:06.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:06.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:06.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:06.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:06.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:06.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:06.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[41614]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[41614]: 4.3 scrub starts 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[41614]: 4.3 scrub ok 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[41614]: pgmap v633: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[29876]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[29876]: 4.3 scrub starts 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[29876]: 4.3 scrub ok 2024-03-30T15:04:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:06 smithi028 ceph-mon[29876]: pgmap v633: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T15:04:06.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:06.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:06 smithi134 ceph-mon[33714]: osdmap e452: 8 total, 8 up, 8 in 2024-03-30T15:04:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:06 smithi134 ceph-mon[33714]: 4.3 scrub starts 2024-03-30T15:04:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:06 smithi134 ceph-mon[33714]: 4.3 scrub ok 2024-03-30T15:04:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:06 smithi134 ceph-mon[33714]: pgmap v633: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-30T15:04:06.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:06.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:06.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:07.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:07.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:07.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:07.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:07.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:07.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 101.0 deep-scrub starts 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 101.0 deep-scrub ok 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T15:04:07.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[41614]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 101.0 deep-scrub starts 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 101.0 deep-scrub ok 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T15:04:07.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T15:04:07.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T15:04:07.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T15:04:07.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:07.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:07.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:07 smithi028 ceph-mon[29876]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T15:04:07.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:07.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 101.0 deep-scrub starts 2024-03-30T15:04:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 101.0 deep-scrub ok 2024-03-30T15:04:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: osdmap e453: 8 total, 8 up, 8 in 2024-03-30T15:04:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:07 smithi134 ceph-mon[33714]: osdmap e454: 8 total, 8 up, 8 in 2024-03-30T15:04:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:07.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:07.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:07.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:07.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:07.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:08.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:08.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.108+0000 7fd414210700 1 -- 172.21.15.28:0/1864846700 >> v1:172.21.15.28:6790/0 conn(0x7fd40c109dd0 legacy=0x7fd40c10c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.108+0000 7fd414210700 1 -- 172.21.15.28:0/1864846700 shutdown_connections 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.108+0000 7fd414210700 1 -- 172.21.15.28:0/1864846700 >> 172.21.15.28:0/1864846700 conn(0x7fd40c0fc9d0 msgr2=0x7fd40c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.108+0000 7fd414210700 1 -- 172.21.15.28:0/1864846700 shutdown_connections 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.108+0000 7fd414210700 1 -- 172.21.15.28:0/1864846700 wait complete. 2024-03-30T15:04:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 librados: starting msgr at 2024-03-30T15:04:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 librados: starting objecter 2024-03-30T15:04:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 Processor -- start 2024-03-30T15:04:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 -- start start 2024-03-30T15:04:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 librados: setting wanted keys 2024-03-30T15:04:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 librados: calling monclient init 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1a9360 con 0x7fd40c101140 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1aa580 con 0x7fd40c109dd0 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd414210700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd40c1ab7c0 con 0x7fd40c10d970 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd411fac700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd40c101140 0x7fd40c0fe8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51490/0 (socket says 172.21.15.28:51490) 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd411fac700 1 -- 172.21.15.28:0/89746027 learned_addr learned my addr 172.21.15.28:0/89746027 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd402ffd700 10 client.?.objecter ms_handle_connect 0x7fd40c101140 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd402ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.109+0000 7fd402ffd700 10 client.?.objecter ms_handle_connect 0x7fd40c10d970 2024-03-30T15:04:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 10 client.?.objecter ms_handle_connect 0x7fd40c109dd0 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392083021 0 0) 0x7fd40c1ab7c0 con 0x7fd40c10d970 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3f80031f0 con 0x7fd40c10d970 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619561009 0 0) 0x7fd40c1a9360 con 0x7fd40c101140 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c1ab7c0 con 0x7fd40c101140 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 629011411 0 0) 0x7fd40c1aa580 con 0x7fd40c109dd0 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c1a9360 con 0x7fd40c109dd0 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 440519538 0 0) 0x7fd3f80031f0 con 0x7fd40c10d970 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd40c1aa580 con 0x7fd40c10d970 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd404001c10 con 0x7fd40c10d970 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 73793507 0 0) 0x7fd40c1ab7c0 con 0x7fd40c101140 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3f80031f0 con 0x7fd40c101140 2024-03-30T15:04:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd408001870 con 0x7fd40c101140 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 643795034 0 0) 0x7fd40c1a9360 con 0x7fd40c109dd0 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd40c1ab7c0 con 0x7fd40c109dd0 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 349709320 0 0) 0x7fd40c1aa580 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.134:6789/0 conn(0x7fd40c109dd0 legacy=0x7fd40c1a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6789/0 conn(0x7fd40c101140 legacy=0x7fd40c0fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c1aca00 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c1a9590 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd414210700 10 client.58079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd40c1a9a50 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd414210700 1 librados: init done 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd404003ea0 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.110+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd404004c60 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.111+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd3f0005100 con 0x7fd40c10d970 2024-03-30T15:04:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd404003a00 con 0x7fd40c10d970 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(454..454 src has 53..454) ==== 8551+0+0 (unknown 1318987890 0 0) 0x7fd4040886d0 con 0x7fd40c10d970 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 osd_map(454..454 src has 53..454) 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 3 client.58079.objecter handle_osd_map got epochs [454,454] > 0 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 3 client.58079.objecter handle_osd_map decoding full epoch 454 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 20 client.58079.objecter dump_active .. 0 homeless 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) ==== 54+0+16 (unknown 1521394261 0 2420000192) 0x7fd40408ab30 con 0x7fd40c10d970 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.112+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f80704a0 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 10 client.58079.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 1 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 1 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 0 1 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 4 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 10 client.58079.objecter _send_command 1 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3f0004b50 con 0x7fd3f0004730 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.113+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f0004730 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.114+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f0004b50 con 0x7fd3f0004730 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.114+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 1: 0 ) 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.114+0000 7fd402ffd700 10 client.58079.objecter _finish_command 1 = system:0 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.114+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 4 2024-03-30T15:04:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.114+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 0 1 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 10 client.58079.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 2 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 2 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f00053e0 osd=0 3 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 0 2 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 4 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 10 client.58079.objecter _send_command 2 2024-03-30T15:04:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.135+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f0007b00 con 0x7fd3f0004730 2024-03-30T15:04:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f0007b00 con 0x7fd3f0004730 2024-03-30T15:04:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd402ffd700 10 client.58079.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 4 2024-03-30T15:04:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 0 2 2024-03-30T15:04:08.139 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 10 client.58079.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 1 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 4 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 10 client.58079.objecter _send_command 3 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.137+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd3f0002100 con 0x7fd3f00042a0 2024-03-30T15:04:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f00042a0 2024-03-30T15:04:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f0002100 con 0x7fd3f00042a0 2024-03-30T15:04:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 3: 0 ) 2024-03-30T15:04:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 10 client.58079.objecter _finish_command 3 = system:0 2024-03-30T15:04:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 4 2024-03-30T15:04:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.138+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 1 3 2024-03-30T15:04:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 10 client.58079.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 4 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 4 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0004130 osd=1 3 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 1 4 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 4 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 10 client.58079.objecter _send_command 4 2024-03-30T15:04:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.155+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f0001600 con 0x7fd3f00042a0 2024-03-30T15:04:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f0001600 con 0x7fd3f00042a0 2024-03-30T15:04:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 10 client.58079.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 4 2024-03-30T15:04:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 1 4 2024-03-30T15:04:08.159 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:04:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 10 client.58079.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 5 2024-03-30T15:04:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 5 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 2 5 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 4 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 10 client.58079.objecter _send_command 5 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd3f0010fb0 con 0x7fd3f0008990 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.157+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f0008990 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.158+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f0010fb0 con 0x7fd3f0008990 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.158+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 5: 0 ) 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.158+0000 7fd402ffd700 10 client.58079.objecter _finish_command 5 = system:0 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.158+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 4 2024-03-30T15:04:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.158+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 2 5 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 10 client.58079.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 6 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 6 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0001a00 osd=2 3 2024-03-30T15:04:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 2 6 2024-03-30T15:04:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 4 2024-03-30T15:04:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 10 client.58079.objecter _send_command 6 2024-03-30T15:04:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.175+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f0007fc0 con 0x7fd3f0008990 2024-03-30T15:04:08.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.176+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f0007fc0 con 0x7fd3f0008990 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.176+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.176+0000 7fd402ffd700 10 client.58079.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.176+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 4 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.176+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 2 6 2024-03-30T15:04:08.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 10 client.58079.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 7 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 7 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 3 7 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 4 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 10 client.58079.objecter _send_command 7 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd3f000d3e0 con 0x7fd3f000cf80 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.177+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f000cf80 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.178+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f000d3e0 con 0x7fd3f000cf80 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.178+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 7: 0 ) 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.178+0000 7fd402ffd700 10 client.58079.objecter _finish_command 7 = system:0 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.178+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 4 2024-03-30T15:04:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.178+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 3 7 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 10 client.58079.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 8 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 8 2024-03-30T15:04:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f0008220 osd=3 3 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 3 8 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 4 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.194+0000 7fd414210700 10 client.58079.objecter _send_command 8 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.195+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f000ca00 con 0x7fd3f000cf80 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f000ca00 con 0x7fd3f000cf80 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd402ffd700 10 client.58079.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 4 2024-03-30T15:04:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 3 8 2024-03-30T15:04:08.198 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:04:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 10 client.58079.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 9 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 9 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 4 9 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 4 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 10 client.58079.objecter _send_command 9 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.196+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd3f000c9e0 con 0x7fd3f0011e40 2024-03-30T15:04:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.197+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f0011e40 2024-03-30T15:04:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.198+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f000c9e0 con 0x7fd3f0011e40 2024-03-30T15:04:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.198+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 9: 0 ) 2024-03-30T15:04:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.198+0000 7fd402ffd700 10 client.58079.objecter _finish_command 9 = system:0 2024-03-30T15:04:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.198+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 4 2024-03-30T15:04:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.198+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 4 9 2024-03-30T15:04:08.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:08.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 10 client.58079.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 10 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 10 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f000c7b0 osd=4 3 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 4 10 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 4 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 10 client.58079.objecter _send_command 10 2024-03-30T15:04:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.214+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f001a940 con 0x7fd3f0011e40 2024-03-30T15:04:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.216+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f001a940 con 0x7fd3f0011e40 2024-03-30T15:04:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.216+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.216+0000 7fd402ffd700 10 client.58079.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.216+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 4 2024-03-30T15:04:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.216+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 4 10 2024-03-30T15:04:08.218 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 10 client.58079.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 11 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 11 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 5 11 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 4 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 10 client.58079.objecter _send_command 11 2024-03-30T15:04:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd3f001cc90 con 0x7fd3f001b060 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.217+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f001b060 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.218+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f001cc90 con 0x7fd3f001b060 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.218+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 11: 0 ) 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.218+0000 7fd402ffd700 10 client.58079.objecter _finish_command 11 = system:0 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.218+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 4 2024-03-30T15:04:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.218+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 5 11 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 10 client.58079.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 12 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 12 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f001a940 osd=5 3 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 5 12 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 4 2024-03-30T15:04:08.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 10 client.58079.objecter _send_command 12 2024-03-30T15:04:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.234+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f001cef0 con 0x7fd3f001b060 2024-03-30T15:04:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.236+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f001cef0 con 0x7fd3f001b060 2024-03-30T15:04:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd402ffd700 10 client.58079.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 4 2024-03-30T15:04:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 5 12 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 10 client.58079.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 13 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 13 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 6 13 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 4 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 10 client.58079.objecter _send_command 13 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd3f00232e0 con 0x7fd3f001f7d0 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.237+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f001f7d0 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.238+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f00232e0 con 0x7fd3f001f7d0 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.238+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 13: 0 ) 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.238+0000 7fd402ffd700 10 client.58079.objecter _finish_command 13 = system:0 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.238+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 4 2024-03-30T15:04:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.238+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 6 13 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 10 client.58079.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 14 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 14 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f001cef0 osd=6 3 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 6 14 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 4 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 10 client.58079.objecter _send_command 14 2024-03-30T15:04:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.256+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f00232e0 con 0x7fd3f001f7d0 2024-03-30T15:04:08.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f00232e0 con 0x7fd3f001f7d0 2024-03-30T15:04:08.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd402ffd700 10 client.58079.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 4 2024-03-30T15:04:08.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 6 14 2024-03-30T15:04:08.260 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:04:08.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 10 client.58079.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 15 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 15 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 7 15 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 4 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 10 client.58079.objecter _send_command 15 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.258+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd3f00275e0 con 0x7fd3f0023960 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.259+0000 7fd402ffd700 10 client.58079.objecter ms_handle_connect 0x7fd3f0023960 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.260+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3f00275e0 con 0x7fd3f0023960 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.260+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 15: 0 ) 2024-03-30T15:04:08.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.260+0000 7fd402ffd700 10 client.58079.objecter _finish_command 15 = system:0 2024-03-30T15:04:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.260+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 4 2024-03-30T15:04:08.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.260+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 7 15 2024-03-30T15:04:08.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 10 client.58079.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:08.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign -1 16 2024-03-30T15:04:08.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 20 client.58079.objecter _get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 15 client.58079.objecter _session_command_op_remove -1 16 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 20 client.58079.objecter get_session s=0x7fd3f00232e0 osd=7 3 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 15 client.58079.objecter _session_command_op_assign 7 16 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 4 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 10 client.58079.objecter _send_command 16 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.277+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd3f0028fc0 con 0x7fd3f0023960 2024-03-30T15:04:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.278+0000 7fd402ffd700 1 -- 172.21.15.28:0/89746027 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd3f0028fc0 con 0x7fd3f0023960 2024-03-30T15:04:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.278+0000 7fd402ffd700 10 client.58079.objecter ms_dispatch 0x7fd40c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.278+0000 7fd402ffd700 10 client.58079.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.278+0000 7fd402ffd700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 4 2024-03-30T15:04:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.278+0000 7fd402ffd700 15 client.58079.objecter _session_command_op_remove 7 16 2024-03-30T15:04:08.280 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:04:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 librados: watch_flush enter 2024-03-30T15:04:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 librados: watch_flush exit 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.0 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6801/3394323575 conn(0x7fd3f0004730 legacy=0x7fd3f0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00053e0 osd=0 1 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.1 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6805/1850313547 conn(0x7fd3f00042a0 legacy=0x7fd3f000ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0004130 osd=1 1 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.2 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6809/3451325043 conn(0x7fd3f0008990 legacy=0x7fd3f000f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0001a00 osd=2 1 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.3 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6813/1506157250 conn(0x7fd3f000cf80 legacy=0x7fd3f0014970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd4127ad700 1 -- 172.21.15.28:0/89746027 reap_dead start 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f0008220 osd=3 1 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.4 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.134:6800/3063775514 conn(0x7fd3f0011e40 legacy=0x7fd3f0019220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f000c7b0 osd=4 1 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.5 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd4127ad700 1 -- 172.21.15.28:0/89746027 reap_dead start 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.134:6804/3668151179 conn(0x7fd3f001b060 legacy=0x7fd3f001b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001a940 osd=5 1 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.6 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.134:6808/2135224194 conn(0x7fd3f001f7d0 legacy=0x7fd3f0021bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f001cef0 osd=6 1 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter close_session for osd.7 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.134:6812/3758190298 conn(0x7fd3f0023960 legacy=0x7fd3f0025d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter put_session s=0x7fd3f00232e0 osd=7 1 2024-03-30T15:04:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 20 client.58079.objecter shutdown clearing up homeless session... 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 10 client.58079.objecter successfully canceled tick 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd3f80704a0 legacy=0x7fd3f8072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> v1:172.21.15.28:6790/0 conn(0x7fd40c10d970 legacy=0x7fd40c1a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd4127ad700 1 -- 172.21.15.28:0/89746027 reap_dead start 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 shutdown_connections 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 >> 172.21.15.28:0/89746027 conn(0x7fd40c0fc9d0 msgr2=0x7fd40c06a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 shutdown_connections 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 -- 172.21.15.28:0/89746027 wait complete. 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:08.279+0000 7fd414210700 1 librados: shutdown 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:04:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:04:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:04:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:04:08.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:08.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:08.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:08.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:08.470 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:08.471 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/89746027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:08.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/89746027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:08.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:08.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T15:04:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T15:04:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: pgmap v636: 260 pgs: 260 active+clean; 8.4 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/89746027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:08.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:08.711 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:08.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:08.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:08.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:08.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:08.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:09.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:09.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:09.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:09.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:09.358 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:04:09.358 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:04:09.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:09.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:09.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:09 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T15:04:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:09 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T15:04:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:09 smithi134 ceph-mon[33714]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T15:04:09.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T15:04:09.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T15:04:09.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[41614]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T15:04:09.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T15:04:09.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T15:04:09.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:09 smithi028 ceph-mon[29876]: osdmap e455: 8 total, 8 up, 8 in 2024-03-30T15:04:09.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:09.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:09.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:09.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:09.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:09.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:10.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:10.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:10.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:10.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:10.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:10.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:10.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:04:10.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:04:10.521 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T15:04:10.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:04:10.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:10.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 42.0 scrub starts 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 42.0 scrub ok 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[29876]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 879 B/s wr, 6 op/s 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:04:10.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 42.0 scrub starts 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 42.0 scrub ok 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T15:04:10.643 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:10 smithi028 ceph-mon[41614]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 879 B/s wr, 6 op/s 2024-03-30T15:04:10.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:04:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:04:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 42.0 scrub starts 2024-03-30T15:04:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 42.0 scrub ok 2024-03-30T15:04:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:04:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:04:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T15:04:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T15:04:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:10 smithi134 ceph-mon[33714]: pgmap v638: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 879 B/s wr, 6 op/s 2024-03-30T15:04:10.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:10.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:10.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:10.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:10.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:10.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:11.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:11.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.125+0000 7faced18f700 1 -- 172.21.15.28:0/3978156402 >> v1:172.21.15.28:6789/0 conn(0x7face810d960 legacy=0x7face810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.125+0000 7faced18f700 1 -- 172.21.15.28:0/3978156402 shutdown_connections 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.125+0000 7faced18f700 1 -- 172.21.15.28:0/3978156402 >> 172.21.15.28:0/3978156402 conn(0x7face80fc9e0 msgr2=0x7face80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.125+0000 7faced18f700 1 -- 172.21.15.28:0/3978156402 shutdown_connections 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.125+0000 7faced18f700 1 -- 172.21.15.28:0/3978156402 wait complete. 2024-03-30T15:04:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 librados: starting msgr at 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 librados: starting objecter 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 Processor -- start 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 -- start start 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 librados: setting wanted keys 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.126+0000 7faced18f700 1 librados: calling monclient init 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faced18f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7face81a93d0 con 0x7face8109dc0 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faced18f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7face81aa610 con 0x7face810d960 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faced18f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7face81ab850 con 0x7face8101130 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7face6d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7face8101130 0x7face81063d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:33640/0 (socket says 172.21.15.28:33640) 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7face6d9d700 1 -- 172.21.15.28:0/335020227 learned_addr learned my addr 172.21.15.28:0/335020227 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter ms_handle_connect 0x7face8101130 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter ms_handle_connect 0x7face8109dc0 2024-03-30T15:04:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608174446 0 0) 0x7face81ab850 con 0x7face8101130 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facd00039c0 con 0x7face8101130 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490910123 0 0) 0x7face81a93d0 con 0x7face8109dc0 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7face81ab850 con 0x7face8109dc0 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter ms_handle_connect 0x7face810d960 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1457427187 0 0) 0x7face81ab850 con 0x7face8109dc0 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7face81a93d0 con 0x7face8109dc0 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 513711713 0 0) 0x7facd00039c0 con 0x7face8101130 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7face81ab850 con 0x7face8101130 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7facd80018b0 con 0x7face8109dc0 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.127+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207348798 0 0) 0x7face81aa610 con 0x7face810d960 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facd00039c0 con 0x7face810d960 2024-03-30T15:04:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7facdc0013d0 con 0x7face8101130 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1000067591 0 0) 0x7face81a93d0 con 0x7face8109dc0 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 >> v1:172.21.15.28:6790/0 conn(0x7face8101130 legacy=0x7face81063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 >> v1:172.21.15.134:6789/0 conn(0x7face810d960 legacy=0x7face81a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face81aca90 con 0x7face8109dc0 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7face81a9600 con 0x7face8109dc0 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faced18f700 10 client.24891.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7face81a9ac0 con 0x7face8109dc0 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faced18f700 1 librados: init done 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7facd8003b40 con 0x7face8109dc0 2024-03-30T15:04:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.128+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7facd8004a40 con 0x7face8109dc0 2024-03-30T15:04:11.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.129+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd4005120 con 0x7face8109dc0 2024-03-30T15:04:11.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.129+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7facd801b2a0 con 0x7face8109dc0 2024-03-30T15:04:11.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(455..455 src has 53..455) ==== 8551+0+0 (unknown 2709645540 0 0) 0x7facd8088670 con 0x7face8109dc0 2024-03-30T15:04:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 10 client.24891.objecter ms_dispatch 0x7face8001550 osd_map(455..455 src has 53..455) 2024-03-30T15:04:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 3 client.24891.objecter handle_osd_map got epochs [455,455] > 0 2024-03-30T15:04:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 3 client.24891.objecter handle_osd_map decoding full epoch 455 2024-03-30T15:04:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 20 client.24891.objecter dump_active .. 0 homeless 2024-03-30T15:04:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.130+0000 7faccffff700 10 client.24891.objecter ms_handle_connect 0x7facd0070420 2024-03-30T15:04:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.135+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7facd8055700 con 0x7face8109dc0 2024-03-30T15:04:11.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:11.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:11.355+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7facd4005410 con 0x7face8109dc0 2024-03-30T15:04:11.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:11.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:11.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:11.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:11 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:04:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:11 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:04:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:11 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:04:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:11 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:04:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:04:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:11.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:11.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:04:11.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:11.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:12.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:12.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:12.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:12.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.270+0000 7faccffff700 1 -- 172.21.15.28:0/335020227 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v456) ==== 87+0+0 (unknown 2900707099 0 0) 0x7facd8059150 con 0x7face8109dc0 2024-03-30T15:04:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.273+0000 7faced18f700 10 librados: watch_flush enter 2024-03-30T15:04:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.274+0000 7faced18f700 10 librados: watch_flush exit 2024-03-30T15:04:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.274+0000 7faced18f700 20 client.24891.objecter shutdown clearing up homeless session... 2024-03-30T15:04:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.274+0000 7faced18f700 10 client.24891.objecter successfully canceled tick 2024-03-30T15:04:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.274+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 >> v1:172.21.15.28:6800/3011431093 conn(0x7facd0070420 legacy=0x7facd00728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.274+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 >> v1:172.21.15.28:6789/0 conn(0x7face8109dc0 legacy=0x7face81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.275+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 shutdown_connections 2024-03-30T15:04:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.275+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 >> 172.21.15.28:0/335020227 conn(0x7face80fc9e0 msgr2=0x7face8106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.275+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 shutdown_connections 2024-03-30T15:04:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.275+0000 7faced18f700 1 -- 172.21.15.28:0/335020227 wait complete. 2024-03-30T15:04:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:12.275+0000 7faced18f700 1 librados: shutdown 2024-03-30T15:04:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:04:12.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:12.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:12.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[41614]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:12.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T15:04:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:12.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:12 smithi028 ceph-mon[29876]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T15:04:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:12.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:12.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:12.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:12.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: pgmap v639: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/335020227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:12 smithi134 ceph-mon[33714]: osdmap e456: 8 total, 8 up, 8 in 2024-03-30T15:04:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:12.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:12.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:12.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:12.822 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:12.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:12.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:13.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:13.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:13.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:13.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:13.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:13.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:13.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:13.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:13.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:13 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T15:04:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:13 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T15:04:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:13.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:13 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T15:04:13.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:13 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T15:04:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:13 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T15:04:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:13 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T15:04:13.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:13.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:13.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:13.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:13.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:14.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:14.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:14.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:14.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:14.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:14.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:14.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:14.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:14 smithi028 ceph-mon[29876]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 609 B/s wr, 4 op/s 2024-03-30T15:04:14.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:14 smithi028 ceph-mon[41614]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 609 B/s wr, 4 op/s 2024-03-30T15:04:14.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:14.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:14 smithi134 ceph-mon[33714]: pgmap v641: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 609 B/s wr, 4 op/s 2024-03-30T15:04:14.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:14.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:14.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:14.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:14.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:15.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:15.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:15.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:15.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.381+0000 7fc4e6721700 1 -- 172.21.15.28:0/769220582 >> v1:172.21.15.28:6789/0 conn(0x7fc4e0109dc0 legacy=0x7fc4e010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.381+0000 7fc4e6721700 1 -- 172.21.15.28:0/769220582 shutdown_connections 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.381+0000 7fc4e6721700 1 -- 172.21.15.28:0/769220582 >> 172.21.15.28:0/769220582 conn(0x7fc4e00fc9e0 msgr2=0x7fc4e00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.381+0000 7fc4e6721700 1 -- 172.21.15.28:0/769220582 shutdown_connections 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.381+0000 7fc4e6721700 1 -- 172.21.15.28:0/769220582 wait complete. 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 librados: starting msgr at 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 librados: starting objecter 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 Processor -- start 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 -- start start 2024-03-30T15:04:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 librados: setting wanted keys 2024-03-30T15:04:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 librados: calling monclient init 2024-03-30T15:04:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc4e01a5da0 con 0x7fc4e0101130 2024-03-30T15:04:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc4e01a6fc0 con 0x7fc4e010d960 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4e6721700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc4e01a8200 con 0x7fc4e0109dc0 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4dffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc4e0101130 0x7fc4e0106230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41266/0 (socket says 172.21.15.28:41266) 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.382+0000 7fc4dffff700 1 -- 172.21.15.28:0/2555398882 learned_addr learned my addr 172.21.15.28:0/2555398882 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fc4e0101130 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fc4e0109dc0 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883509711 0 0) 0x7fc4e01a5da0 con 0x7fc4e0101130 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4c8003aa0 con 0x7fc4e0101130 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter ms_handle_connect 0x7fc4e010d960 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102661892 0 0) 0x7fc4e01a8200 con 0x7fc4e0109dc0 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4e01a5da0 con 0x7fc4e0109dc0 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 618192775 0 0) 0x7fc4c8003aa0 con 0x7fc4e0101130 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4e01a8200 con 0x7fc4e0101130 2024-03-30T15:04:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc4d0001910 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 110381573 0 0) 0x7fc4e01a5da0 con 0x7fc4e0109dc0 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4c8003aa0 con 0x7fc4e0109dc0 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2681701558 0 0) 0x7fc4e01a6fc0 con 0x7fc4e010d960 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4e01a5da0 con 0x7fc4e010d960 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc4d40015b0 con 0x7fc4e0109dc0 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2829258331 0 0) 0x7fc4e01a8200 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 >> v1:172.21.15.28:6790/0 conn(0x7fc4e0109dc0 legacy=0x7fc4e01a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 >> v1:172.21.15.134:6789/0 conn(0x7fc4e010d960 legacy=0x7fc4e01a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e01a9440 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e01a8430 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4e6721700 10 client.24897.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc4e01a88f0 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4e6721700 1 librados: init done 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc4d0001c60 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.383+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc4d0005da0 con 0x7fc4e0101130 2024-03-30T15:04:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.384+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc005120 con 0x7fc4e0101130 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.385+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc4d001c6a0 con 0x7fc4e0101130 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.385+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(456..456 src has 53..456) ==== 8551+0+0 (unknown 1182597763 0 0) 0x7fc4d0089ac0 con 0x7fc4e0101130 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.385+0000 7fc4dd7fa700 10 client.24897.objecter ms_dispatch 0x7fc4e0001550 osd_map(456..456 src has 53..456) 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.385+0000 7fc4dd7fa700 3 client.24897.objecter handle_osd_map got epochs [456,456] > 0 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.385+0000 7fc4dd7fa700 3 client.24897.objecter handle_osd_map decoding full epoch 456 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.386+0000 7fc4dd7fa700 20 client.24897.objecter dump_active .. 0 homeless 2024-03-30T15:04:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.386+0000 7fc4dd7fa700 10 client.24897.objecter ms_handle_connect 0x7fc4c80725d0 2024-03-30T15:04:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.390+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc4d0056b50 con 0x7fc4e0101130 2024-03-30T15:04:15.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:15.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:04:15.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.608+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fc4cc005410 con 0x7fc4e0101130 2024-03-30T15:04:15.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.611+0000 7fc4dd7fa700 1 -- 172.21.15.28:0/2555398882 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v456) ==== 81+0+56030 (unknown 2429210580 0 1766944265) 0x7fc4d005a5a0 con 0x7fc4e0101130 2024-03-30T15:04:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 456, 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:04:12.269433+0000", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:04:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 112, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 297, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:04:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "437", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:04:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:04:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:04:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:04:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:04:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:04:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:04:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:04:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "453", 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "411", 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "411", 2024-03-30T15:04:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "411", 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 19, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 453, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:04:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 297, 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-65", 2024-03-30T15:04:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:04:00.565991+0000", 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "450", 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:04:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 10 librados: watch_flush enter 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 10 librados: watch_flush exit 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 20 client.24897.objecter shutdown clearing up homeless session... 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 10 client.24897.objecter successfully canceled tick 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc4c80725d0 legacy=0x7fc4c8074a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 >> v1:172.21.15.28:6789/0 conn(0x7fc4e0101130 legacy=0x7fc4e0106230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 shutdown_connections 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 >> 172.21.15.28:0/2555398882 conn(0x7fc4e00fc9e0 msgr2=0x7fc4e00fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 shutdown_connections 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 -- 172.21.15.28:0/2555398882 wait complete. 2024-03-30T15:04:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:15.613+0000 7fc4e6721700 1 librados: shutdown 2024-03-30T15:04:15.682 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:15.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:04:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:04:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:04:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:04:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 417, 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:04:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:04:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:04:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:04:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 448, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:04:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:04:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:04.098499+0000", 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:13.415223+0000", 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:57.434430+0000", 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:06.444221+0000", 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:07.577028+0000", 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:57.286337+0000", 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:04:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:03.344622+0000", 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:02:57.539763+0000", 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:04:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:04:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:04:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:04:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [ 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 19, 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:04:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:15.739 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:04:15.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:15.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2555398882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:15.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:15.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:15.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2555398882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2555398882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:16.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:16.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:16.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:16.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:16.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:16.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:16.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:16.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:16.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:16.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:16.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:04:16.718 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:04:16.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:16.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:16 smithi134 ceph-mon[33714]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T15:04:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:16 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:16.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:16.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:16.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[29876]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T15:04:16.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[41614]: pgmap v642: 260 pgs: 260 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:16 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-65"}]: dispatch 2024-03-30T15:04:17.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:17.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:17.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:17.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:17.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:17.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:17.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:17.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:17.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:17.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:17.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[41614]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[41614]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[29876]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[29876]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:17.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:17.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:04:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:17 smithi134 ceph-mon[33714]: osdmap e457: 8 total, 8 up, 8 in 2024-03-30T15:04:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:17 smithi134 ceph-mon[33714]: osdmap e458: 8 total, 8 up, 8 in 2024-03-30T15:04:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:17.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:17.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:17.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:18.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:18.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:18.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:18.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:18.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:18.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:18.565 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:18.609 INFO:tasks.thrashosds.thrasher:j is {'epoch': 456, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:04:12.269433+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 112, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 297, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '437', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '453', 'last_force_op_resend': '411', 'last_force_op_resend_prenautilus': '411', 'last_force_op_resend_preluminous': '411', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 453, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 297, 'pool_name': 'test-rados-api-smithi028-93253-65', 'create_time': '2024-03-30T15:04:00.565991+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '450', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:04.098499+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:13.415223+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:57.434430+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:06.444221+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:07.577028+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:57.286337+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:03.344622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:02:57.539763+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 101, 'snaps': [{'begin': 19, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:04:18.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:04:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:18.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:18.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:18.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:18.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:18.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[29876]: pgmap v645: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:18.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[29876]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[41614]: pgmap v645: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[41614]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T15:04:18.898 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:18 smithi134 ceph-mon[33714]: pgmap v645: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:18 smithi134 ceph-mon[33714]: osdmap e459: 8 total, 8 up, 8 in 2024-03-30T15:04:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:18.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:18.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:19.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:19.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:19.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:19.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:19.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:19.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:19.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:19.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:19.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:19.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:19.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:19.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: osdmap e460: 8 total, 8 up, 8 in 2024-03-30T15:04:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-67"}]: dispatch 2024-03-30T15:04:19.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:19.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:20.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:20.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:20.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:20.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:20.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:20.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:20.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:20.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:20.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:20 smithi028 ceph-mon[41614]: pgmap v648: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:20.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:20 smithi028 ceph-mon[41614]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T15:04:20.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:20 smithi028 ceph-mon[29876]: pgmap v648: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:20.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:20 smithi028 ceph-mon[29876]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T15:04:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:20.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:20.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:20.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:20.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:21.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.720+0000 7fd52c03c700 1 -- 172.21.15.28:0/423915454 >> v1:172.21.15.28:6790/0 conn(0x7fd52410afb0 legacy=0x7fd52410d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.720+0000 7fd52c03c700 1 -- 172.21.15.28:0/423915454 shutdown_connections 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.720+0000 7fd52c03c700 1 -- 172.21.15.28:0/423915454 >> 172.21.15.28:0/423915454 conn(0x7fd5240fc9d0 msgr2=0x7fd5240fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- 172.21.15.28:0/423915454 shutdown_connections 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- 172.21.15.28:0/423915454 wait complete. 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 librados: starting msgr at 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 librados: starting objecter 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 Processor -- start 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- start start 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 librados: setting wanted keys 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 librados: calling monclient init 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd52410ac20 con 0x7fd52410eac0 2024-03-30T15:04:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd5241a8640 con 0x7fd52410afb0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.721+0000 7fd52c03c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd5241a9820 con 0x7fd5241031a0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd529dd8700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd5241031a0 0x7fd524107a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44286/0 (socket says 172.21.15.28:44286) 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd529dd8700 1 -- 172.21.15.28:0/1136019644 learned_addr learned my addr 172.21.15.28:0/1136019644 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter ms_handle_connect 0x7fd52410eac0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter ms_handle_connect 0x7fd5241031a0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter ms_handle_connect 0x7fd52410afb0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475213258 0 0) 0x7fd52410ac20 con 0x7fd52410eac0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5100031f0 con 0x7fd52410eac0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2120620125 0 0) 0x7fd5241a9820 con 0x7fd5241031a0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd52410ac20 con 0x7fd5241031a0 2024-03-30T15:04:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634486955 0 0) 0x7fd5241a8640 con 0x7fd52410afb0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5241a9820 con 0x7fd52410afb0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 642177229 0 0) 0x7fd5100031f0 con 0x7fd52410eac0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5241a8640 con 0x7fd52410eac0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd51c001c50 con 0x7fd52410eac0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3209527301 0 0) 0x7fd52410ac20 con 0x7fd5241031a0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5100031f0 con 0x7fd5241031a0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3327466670 0 0) 0x7fd5241a9820 con 0x7fd52410afb0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd52410ac20 con 0x7fd52410afb0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd5200018f0 con 0x7fd5241031a0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd514001430 con 0x7fd52410afb0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3951358040 0 0) 0x7fd5241a8640 con 0x7fd52410eac0 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6790/0 conn(0x7fd5241031a0 legacy=0x7fd524107a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.134:6789/0 conn(0x7fd52410afb0 legacy=0x7fd524108180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5241aaa00 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.722+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5241a9a50 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd52c03c700 10 client.24903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5241a9f10 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd52c03c700 1 librados: init done 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd51c003ec0 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd51c004d10 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.723+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd508005100 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd51c003a20 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(461..461 src has 53..461) ==== 8138+0+0 (unknown 1849486651 0 0) 0x7fd51c0884e0 con 0x7fd52410eac0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 osd_map(461..461 src has 53..461) 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 3 client.24903.objecter handle_osd_map got epochs [461,461] > 0 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 3 client.24903.objecter handle_osd_map decoding full epoch 461 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 20 client.24903.objecter dump_active .. 0 homeless 2024-03-30T15:04:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.725+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v461) ==== 54+0+16 (unknown 318098662 0 2420000192) 0x7fd51c0889a0 con 0x7fd52410eac0 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd5100704a0 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 10 client.24903.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 1 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 1 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 0 1 2024-03-30T15:04:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 4 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 10 client.24903.objecter _send_command 1 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd508004b50 con 0x7fd508004730 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.726+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508004730 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.727+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd508004b50 con 0x7fd508004730 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.727+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 1: 0 ) 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.727+0000 7fd51affd700 10 client.24903.objecter _finish_command 1 = system:0 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.727+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 4 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.727+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 0 1 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 10 client.24903.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 2 2024-03-30T15:04:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 2 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd5080053e0 osd=0 3 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 0 2 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 4 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 10 client.24903.objecter _send_command 2 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.746+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd5080079f0 con 0x7fd508004730 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd5080079f0 con 0x7fd508004730 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 10 client.24903.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 4 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 0 2 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 10 client.24903.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 3 2024-03-30T15:04:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 1 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 4 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 10 client.24903.objecter _send_command 3 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd508002130 con 0x7fd508004280 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.748+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508004280 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.749+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd508002130 con 0x7fd508004280 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.749+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 3: 0 ) 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.749+0000 7fd51affd700 10 client.24903.objecter _finish_command 3 = system:0 2024-03-30T15:04:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.749+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 4 2024-03-30T15:04:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.749+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 1 3 2024-03-30T15:04:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 10 client.24903.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 4 2024-03-30T15:04:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508004110 osd=1 3 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 1 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 10 client.24903.objecter _send_command 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd508001600 con 0x7fd508004280 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd508001600 con 0x7fd508004280 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd51affd700 10 client.24903.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.768+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 1 4 2024-03-30T15:04:21.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 10 client.24903.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 5 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 5 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 2 5 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 4 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 10 client.24903.objecter _send_command 5 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd508010f60 con 0x7fd5080088c0 2024-03-30T15:04:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd5080088c0 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd508010f60 con 0x7fd5080088c0 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 5: 0 ) 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 10 client.24903.objecter _finish_command 5 = system:0 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 4 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.769+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 2 5 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 10 client.24903.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 6 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 6 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508001a00 osd=2 3 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 2 6 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 4 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 10 client.24903.objecter _send_command 6 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.786+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd508007e50 con 0x7fd5080088c0 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd508007e50 con 0x7fd5080088c0 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 10 client.24903.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 4 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 2 6 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 10 client.24903.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 7 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 7 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 3 7 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 4 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 10 client.24903.objecter _send_command 7 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd508011c40 con 0x7fd508011110 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.789+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508011110 2024-03-30T15:04:21.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.790+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd508011c40 con 0x7fd508011110 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.790+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 7: 0 ) 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.790+0000 7fd51affd700 10 client.24903.objecter _finish_command 7 = system:0 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.790+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 4 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.790+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 3 7 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 10 client.24903.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 8 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 8 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd50800cc20 osd=3 3 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 3 8 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 4 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 10 client.24903.objecter _send_command 8 2024-03-30T15:04:21.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.807+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd508012410 con 0x7fd508011110 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd508012410 con 0x7fd508011110 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd51affd700 10 client.24903.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 4 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 3 8 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:04:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 10 client.24903.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 9 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 9 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 4 9 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 4 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 10 client.24903.objecter _send_command 9 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.808+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd50801ea40 con 0x7fd50801af30 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.809+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd50801af30 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.810+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd50801ea40 con 0x7fd50801af30 2024-03-30T15:04:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.810+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 9: 0 ) 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.810+0000 7fd51affd700 10 client.24903.objecter _finish_command 9 = system:0 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.810+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 4 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.810+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 4 9 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 10 client.24903.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 10 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 10 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508012410 osd=4 3 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 4 10 2024-03-30T15:04:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 4 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 10 client.24903.objecter _send_command 10 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.826+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd50800c4b0 con 0x7fd50801af30 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd50800c4b0 con 0x7fd50801af30 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 10 client.24903.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 4 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 4 10 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 10 client.24903.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 11 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 11 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 5 11 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 4 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 10 client.24903.objecter _send_command 11 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd5080133f0 con 0x7fd508012e70 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.828+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508012e70 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.829+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd5080133f0 con 0x7fd508012e70 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.829+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 11: 0 ) 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.829+0000 7fd51affd700 10 client.24903.objecter _finish_command 11 = system:0 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.829+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 4 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.829+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 5 11 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 10 client.24903.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 12 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 12 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd50800c4b0 osd=5 3 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 5 12 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 4 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 10 client.24903.objecter _send_command 12 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.846+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd5080133f0 con 0x7fd508012e70 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:04:21.163 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.847+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd5080133f0 con 0x7fd508012e70 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.847+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.847+0000 7fd51affd700 10 client.24903.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.847+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 4 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.847+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 5 12 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 10 client.24903.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 13 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 13 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 6 13 2024-03-30T15:04:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 4 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 10 client.24903.objecter _send_command 13 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd508027350 con 0x7fd508023800 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.848+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508023800 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.849+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd508027350 con 0x7fd508023800 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.849+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 13: 0 ) 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.849+0000 7fd51affd700 10 client.24903.objecter _finish_command 13 = system:0 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.849+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 4 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.849+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 6 13 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 10 client.24903.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 14 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 14 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd5080133f0 osd=6 3 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 6 14 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 4 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 10 client.24903.objecter _send_command 14 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.865+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd5080128a0 con 0x7fd508023800 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd5080128a0 con 0x7fd508023800 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd51affd700 10 client.24903.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 4 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 6 14 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 10 client.24903.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 15 2024-03-30T15:04:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 15 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 7 15 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 4 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 10 client.24903.objecter _send_command 15 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.867+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd508028b10 con 0x7fd508012650 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.868+0000 7fd51affd700 10 client.24903.objecter ms_handle_connect 0x7fd508012650 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.869+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd52000e900 con 0x7fd508012650 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.869+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 15: 0 ) 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.869+0000 7fd51affd700 10 client.24903.objecter _finish_command 15 = system:0 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.869+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 4 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.869+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 7 15 2024-03-30T15:04:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 10 client.24903.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign -1 16 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 20 client.24903.objecter _get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_remove -1 16 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 20 client.24903.objecter get_session s=0x7fd508012d00 osd=7 3 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 15 client.24903.objecter _session_command_op_assign 7 16 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 4 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 10 client.24903.objecter _send_command 16 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.885+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd50801f7f0 con 0x7fd508012650 2024-03-30T15:04:21.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd51affd700 1 -- 172.21.15.28:0/1136019644 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd50801f7f0 con 0x7fd508012650 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd51affd700 10 client.24903.objecter ms_dispatch 0x7fd524001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd51affd700 10 client.24903.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd51affd700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 4 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd51affd700 15 client.24903.objecter _session_command_op_remove 7 16 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.887+0000 7fd52c03c700 10 librados: watch_flush enter 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 librados: watch_flush exit 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.0 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6801/3394323575 conn(0x7fd508004730 legacy=0x7fd508002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080053e0 osd=0 1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6805/1850313547 conn(0x7fd508004280 legacy=0x7fd50800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508004110 osd=1 1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.2 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6809/3451325043 conn(0x7fd5080088c0 legacy=0x7fd50800f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508001a00 osd=2 1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52a5d9700 1 -- 172.21.15.28:0/1136019644 reap_dead start 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.3 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6813/1506157250 conn(0x7fd508011110 legacy=0x7fd508011530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800cc20 osd=3 1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.4 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.134:6800/3063775514 conn(0x7fd50801af30 legacy=0x7fd50801d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012410 osd=4 1 2024-03-30T15:04:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.5 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.134:6804/3668151179 conn(0x7fd508012e70 legacy=0x7fd508021bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd50800c4b0 osd=5 1 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.6 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.134:6808/2135224194 conn(0x7fd508023800 legacy=0x7fd508025bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd5080133f0 osd=6 1 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter close_session for osd.7 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.134:6812/3758190298 conn(0x7fd508012650 legacy=0x7fd508027350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter put_session s=0x7fd508012d00 osd=7 1 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 20 client.24903.objecter shutdown clearing up homeless session... 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 10 client.24903.objecter successfully canceled tick 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52a5d9700 1 -- 172.21.15.28:0/1136019644 reap_dead start 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd5100704a0 legacy=0x7fd510072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> v1:172.21.15.28:6789/0 conn(0x7fd52410eac0 legacy=0x7fd5241a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 shutdown_connections 2024-03-30T15:04:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 >> 172.21.15.28:0/1136019644 conn(0x7fd5240fc9d0 msgr2=0x7fd524110ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 shutdown_connections 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 -- 172.21.15.28:0/1136019644 wait complete. 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:20.888+0000 7fd52c03c700 1 librados: shutdown 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:04:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:04:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:20 smithi134 ceph-mon[33714]: pgmap v648: 260 pgs: 7 creating+activating, 253 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:20 smithi134 ceph-mon[33714]: osdmap e461: 8 total, 8 up, 8 in 2024-03-30T15:04:21.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:21.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:21.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:21.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:21.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:21.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:21.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1136019644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[29876]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T15:04:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:21.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1136019644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[41614]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T15:04:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:21.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:21.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:21.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:21.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:21.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:21.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3018240937 >> v1:172.21.15.28:6790/0 conn(0x7f8e48102240 legacy=0x7f8e48102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3018240937 shutdown_connections 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3018240937 >> 172.21.15.28:0/3018240937 conn(0x7f8e480fdaf0 msgr2=0x7f8e480fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3018240937 shutdown_connections 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3018240937 wait complete. 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 librados: starting msgr at 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.096+0000 7f8e4ceba700 1 librados: starting objecter 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 Processor -- start 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 -- start start 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 librados: setting wanted keys 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 librados: calling monclient init 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e48076130 con 0x7f8e48102240 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e48076340 con 0x7f8e4810ea70 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4ceba700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8e481aba40 con 0x7f8e4810aed0 2024-03-30T15:04:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e45d9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8e4810aed0 0x7f8e48074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44308/0 (socket says 172.21.15.28:44308) 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e45d9b700 1 -- 172.21.15.28:0/3477380348 learned_addr learned my addr 172.21.15.28:0/3477380348 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e4659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e48102240 0x7f8e48074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53380/0 (socket says 172.21.15.28:53380) 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e4810aed0 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e48102240 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f8e4810ea70 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161676792 0 0) 0x7f8e48076130 con 0x7f8e48102240 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e340031f0 con 0x7f8e48102240 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801033857 0 0) 0x7f8e481aba40 con 0x7f8e4810aed0 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e48076130 con 0x7f8e4810aed0 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4055813403 0 0) 0x7f8e48076340 con 0x7f8e4810ea70 2024-03-30T15:04:22.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e481aba40 con 0x7f8e4810ea70 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2008430089 0 0) 0x7f8e340031f0 con 0x7f8e48102240 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e48076340 con 0x7f8e48102240 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8e400017b0 con 0x7f8e48102240 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.097+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 305233359 0 0) 0x7f8e48076130 con 0x7f8e4810aed0 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e340031f0 con 0x7f8e4810aed0 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8e300018b0 con 0x7f8e4810aed0 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 973477494 0 0) 0x7f8e481aba40 con 0x7f8e4810ea70 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e48076130 con 0x7f8e4810ea70 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8e38001850 con 0x7f8e4810ea70 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2616218593 0 0) 0x7f8e48076340 con 0x7f8e48102240 2024-03-30T15:04:22.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 >> v1:172.21.15.28:6790/0 conn(0x7f8e4810aed0 legacy=0x7f8e48074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 >> v1:172.21.15.134:6789/0 conn(0x7f8e4810ea70 legacy=0x7f8e481a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e481acc20 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e481abc70 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e4ceba700 10 client.24909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e481ac130 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e4ceba700 1 librados: init done 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8e40001eb0 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.098+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8e400051b0 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.099+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e28005120 con 0x7f8e48102240 2024-03-30T15:04:22.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.099+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8e40003be0 con 0x7f8e48102240 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(462..462 src has 53..462) ==== 8510+0+0 (unknown 4015628941 0 0) 0x7f8e40089b30 con 0x7f8e48102240 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 10 client.24909.objecter ms_dispatch 0x7f8e48001550 osd_map(462..462 src has 53..462) 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 3 client.24909.objecter handle_osd_map got epochs [462,462] > 0 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 3 client.24909.objecter handle_osd_map decoding full epoch 462 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 20 client.24909.objecter dump_active .. 0 homeless 2024-03-30T15:04:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.100+0000 7f8e3f7fe700 10 client.24909.objecter ms_handle_connect 0x7f8e340704e0 2024-03-30T15:04:22.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:22.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.105+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8e40056be0 con 0x7f8e48102240 2024-03-30T15:04:22.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1136019644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:21 smithi134 ceph-mon[33714]: osdmap e462: 8 total, 8 up, 8 in 2024-03-30T15:04:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:22.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:22.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:22.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:22.366+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f8e28005410 con 0x7f8e48102240 2024-03-30T15:04:22.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:22.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:22.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:22.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:22.670 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:22.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:22.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:22.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:22.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:22.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:23.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:23.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:23.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: pgmap v651: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:04:23.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:23.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: pgmap v651: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:04:23.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:23.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:23.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:23.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.287+0000 7f8e3f7fe700 1 -- 172.21.15.28:0/3477380348 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v464) ==== 97+0+0 (unknown 2295822960 0 0) 0x7f8e4005a630 con 0x7f8e48102240 2024-03-30T15:04:23.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.289+0000 7f8e4ceba700 10 librados: watch_flush enter 2024-03-30T15:04:23.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.290+0000 7f8e4ceba700 10 librados: watch_flush exit 2024-03-30T15:04:23.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.290+0000 7f8e4ceba700 20 client.24909.objecter shutdown clearing up homeless session... 2024-03-30T15:04:23.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.290+0000 7f8e4ceba700 10 client.24909.objecter successfully canceled tick 2024-03-30T15:04:23.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8e340704e0 legacy=0x7f8e340729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 >> v1:172.21.15.28:6789/0 conn(0x7f8e48102240 legacy=0x7f8e48074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 shutdown_connections 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 >> 172.21.15.28:0/3477380348 conn(0x7f8e480fdaf0 msgr2=0x7f8e4810a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 shutdown_connections 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 -- 172.21.15.28:0/3477380348 wait complete. 2024-03-30T15:04:23.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:23.291+0000 7f8e4ceba700 1 librados: shutdown 2024-03-30T15:04:23.294 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:04:23.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: pgmap v651: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: osdmap e463: 8 total, 8 up, 8 in 2024-03-30T15:04:23.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-69"}]: dispatch 2024-03-30T15:04:23.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:23.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:23.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:23.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:04:23.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:23.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:23.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:23.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:23.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:24.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:24.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:24.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:24.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:24.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:24.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:24.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:24.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[41614]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T15:04:24.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:24.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:24.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:24 smithi028 ceph-mon[29876]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T15:04:24.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:24.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:24 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3477380348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:24 smithi134 ceph-mon[33714]: osdmap e464: 8 total, 8 up, 8 in 2024-03-30T15:04:24.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:24.542 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:04:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:04:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:04:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:04:24.543 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:04:24.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:24.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:24.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:24.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:24.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:25.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:25.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:25.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[41614]: pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:04:25.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:25.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[41614]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T15:04:25.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[29876]: pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:04:25.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:25.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[29876]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T15:04:25.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:25 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:25.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:25.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:25.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:25 smithi134 ceph-mon[33714]: pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:04:25.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:25 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:25.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:25 smithi134 ceph-mon[33714]: osdmap e465: 8 total, 8 up, 8 in 2024-03-30T15:04:25.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:25 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:25.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:25.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:25.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:25.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:25.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:04:25.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.711 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:25.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:25.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:25.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:25.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:25.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:25.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:26.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:26.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:26.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:04:26.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- 172.21.15.28:0/4276104076 >> v1:172.21.15.28:6790/0 conn(0x7ffbb410ae80 legacy=0x7ffbb410d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- 172.21.15.28:0/4276104076 shutdown_connections 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- 172.21.15.28:0/4276104076 >> 172.21.15.28:0/4276104076 conn(0x7ffbb40761d0 msgr2=0x7ffbb40765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- 172.21.15.28:0/4276104076 shutdown_connections 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- 172.21.15.28:0/4276104076 wait complete. 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 librados: starting msgr at 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 librados: starting objecter 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 Processor -- start 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 -- start start 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 librados: setting wanted keys 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.131+0000 7ffbba8a4700 1 librados: calling monclient init 2024-03-30T15:04:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbba8a4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a70b0 con 0x7ffbb410ae80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbba8a4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a82d0 con 0x7ffbb410ea80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbba8a4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a9510 con 0x7ffbb4108110 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb3fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ffbb4108110 0x7ffbb41073c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44320/0 (socket says 172.21.15.28:44320) 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb3fff700 1 -- 172.21.15.28:0/1000572631 learned_addr learned my addr 172.21.15.28:0/1000572631 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter ms_handle_connect 0x7ffbb4108110 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter ms_handle_connect 0x7ffbb410ae80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter ms_handle_connect 0x7ffbb410ea80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515854428 0 0) 0x7ffbb41a9510 con 0x7ffbb4108110 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb9c003aa0 con 0x7ffbb4108110 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904205880 0 0) 0x7ffbb41a70b0 con 0x7ffbb410ae80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb41a9510 con 0x7ffbb410ae80 2024-03-30T15:04:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247834351 0 0) 0x7ffbb41a82d0 con 0x7ffbb410ea80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.132+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb41a70b0 con 0x7ffbb410ea80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4294072177 0 0) 0x7ffb9c003aa0 con 0x7ffbb4108110 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbb41a82d0 con 0x7ffbb4108110 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffba4001930 con 0x7ffbb4108110 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4285470524 0 0) 0x7ffbb41a9510 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb9c003aa0 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3391127027 0 0) 0x7ffbb41a70b0 con 0x7ffbb410ea80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbb41a9510 con 0x7ffbb410ea80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffba8001490 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffbac001c10 con 0x7ffbb410ea80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1287986226 0 0) 0x7ffb9c003aa0 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 >> v1:172.21.15.28:6790/0 conn(0x7ffbb4108110 legacy=0x7ffbb41073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 >> v1:172.21.15.134:6789/0 conn(0x7ffbb410ea80 legacy=0x7ffbb41a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb41aa750 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb41a72e0 con 0x7ffbb410ae80 2024-03-30T15:04:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbba8a4700 10 client.24915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbb41a77a0 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.133+0000 7ffbba8a4700 1 librados: init done 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.134+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ffba8003ad0 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.134+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffba8004eb0 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.135+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb406a1f0 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.135+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ffba801b550 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(466..466 src has 53..466) ==== 8523+0+0 (unknown 2844508642 0 0) 0x7ffba80887d0 con 0x7ffbb410ae80 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 10 client.24915.objecter ms_dispatch 0x7ffbb4001550 osd_map(466..466 src has 53..466) 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 3 client.24915.objecter handle_osd_map got epochs [466,466] > 0 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 3 client.24915.objecter handle_osd_map decoding full epoch 466 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 20 client.24915.objecter dump_active .. 0 homeless 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.136+0000 7ffbb17fa700 10 client.24915.objecter ms_handle_connect 0x7ffb9c070470 2024-03-30T15:04:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.141+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ffba8055870 con 0x7ffbb410ae80 2024-03-30T15:04:26.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:26.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:26.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:26.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.363+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffbb4068b30 con 0x7ffb9c070470 2024-03-30T15:04:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.382+0000 7ffbb17fa700 1 -- 172.21.15.28:0/1000572631 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+664637 (unknown 2965378022 0 1669296229) 0x7ffbb4068b30 con 0x7ffb9c070470 2024-03-30T15:04:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 10 librados: watch_flush enter 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 10 librados: watch_flush exit 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 20 client.24915.objecter shutdown clearing up homeless session... 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 10 client.24915.objecter successfully canceled tick 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 >> v1:172.21.15.28:6800/3011431093 conn(0x7ffb9c070470 legacy=0x7ffb9c072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 >> v1:172.21.15.28:6789/0 conn(0x7ffbb410ae80 legacy=0x7ffbb41a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 shutdown_connections 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 >> 172.21.15.28:0/1000572631 conn(0x7ffbb40761d0 msgr2=0x7ffbb4109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 shutdown_connections 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 -- 172.21.15.28:0/1000572631 wait complete. 2024-03-30T15:04:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:26.387+0000 7ffbba8a4700 1 librados: shutdown 2024-03-30T15:04:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:04:26.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:26.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T15:04:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[41614]: pgmap v657: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:26 smithi028 ceph-mon[29876]: pgmap v657: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:04:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:26.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:26.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: osdmap e466: 8 total, 8 up, 8 in 2024-03-30T15:04:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:04:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:26 smithi134 ceph-mon[33714]: pgmap v657: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-30T15:04:26.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:26.728 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:26.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:26.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:26.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:26.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:27.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:27.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:27.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:27.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:27.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:27.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:27.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:27.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:27.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:27.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[29876]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T15:04:27.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[29876]: from='client.24915 v1:172.21.15.28:0/1000572631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:27.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[29876]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T15:04:27.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:27.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[41614]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T15:04:27.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[41614]: from='client.24915 v1:172.21.15.28:0/1000572631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:27.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:27 smithi028 ceph-mon[41614]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T15:04:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:27.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:27.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:27.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:27.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:27.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:27 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:04:28.045 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:27 smithi134 ceph-mon[33714]: osdmap e467: 8 total, 8 up, 8 in 2024-03-30T15:04:28.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:27 smithi134 ceph-mon[33714]: from='client.24915 v1:172.21.15.28:0/1000572631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:04:28.046 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:27 smithi134 ceph-mon[33714]: osdmap e468: 8 total, 8 up, 8 in 2024-03-30T15:04:28.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:28.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:28.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:28.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:28.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:28.306 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:28.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:28.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:28.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:28.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:28.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:28.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:28.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[41614]: pgmap v660: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:28.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:28.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:28.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:28 smithi028 ceph-mon[29876]: pgmap v660: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:28.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2024-03-30T15:04:26.300679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792145,"num_objects":363,"num_object_clones":4,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":1626,"num_read_kb":1371,"num_write":1231,"num_write_kb":15166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2598,"ondisk_log_size":2598,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":748,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":610052,"kb_used_data":89720,"kb_used_omap":12,"kb_used_meta":520051,"kb_avail":749351164,"statfs":{"total":767960285184,"available":767335591936,"internally_reserved":0,"allocated":91873280,"data_stored":63358279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":532532784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013314"},"pg_stats":[{"pgid":"300.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"300.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-30T15:04:24.288323+0000","last_change":"2024-03-30T15:04:24.288323+0000","last_active":"2024-03-30T15:04:24.288323+0000","last_peered":"2024-03-30T15:04:24.288323+0000","last_clean":"2024-03-30T15:04:24.288323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:04:24.288323+0000","last_undegraded":"2024-03-30T15:04:24.288323+0000","last_fullsized":"2024-03-30T15:04:24.288323+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:24.288323+0000","last_clean_scrub_stamp":"2024-03-30T15:04:24.288323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":916,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291812+0000","last_change":"2024-03-30T15:04:06.344710+0000","last_active":"2024-03-30T15:04:23.291812+0000","last_peered":"2024-03-30T15:04:23.291812+0000","last_clean":"2024-03-30T15:04:23.291812+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:04:23.291812+0000","last_undegraded":"2024-03-30T15:04:23.291812+0000","last_fullsized":"2024-03-30T15:04:23.291812+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:40.995948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":936,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380963+0000","last_change":"2024-03-30T15:04:06.379921+0000","last_active":"2024-03-30T15:04:23.380963+0000","last_peered":"2024-03-30T15:04:23.380963+0000","last_clean":"2024-03-30T15:04:23.380963+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:04:23.380963+0000","last_undegraded":"2024-03-30T15:04:23.380963+0000","last_fullsized":"2024-03-30T15:04:23.380963+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:31.961438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.961438+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"400'89","reported_seq":945,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290985+0000","last_change":"2024-03-30T15:04:06.471415+0000","last_active":"2024-03-30T15:04:23.290985+0000","last_peered":"2024-03-30T15:04:23.290985+0000","last_clean":"2024-03-30T15:04:23.290985+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:04:23.290985+0000","last_undegraded":"2024-03-30T15:04:23.290985+0000","last_fullsized":"2024-03-30T15:04:23.290985+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'87","last_scrub_stamp":"2024-03-30T15:02:40.447037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.447037+0000","objects_scrubbed":7,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617106+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":971,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.290867+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:04:22.290867+0000","last_peered":"2024-03-30T15:04:22.290867+0000","last_clean":"2024-03-30T15:04:22.290867+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:04:22.290867+0000","last_undegraded":"2024-03-30T15:04:22.290867+0000","last_fullsized":"2024-03-30T15:04:22.290867+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.569931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"286'36","reported_seq":912,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292048+0000","last_change":"2024-03-30T15:04:06.346222+0000","last_active":"2024-03-30T15:04:23.292048+0000","last_peered":"2024-03-30T15:04:23.292048+0000","last_clean":"2024-03-30T15:04:23.292048+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:04:23.292048+0000","last_undegraded":"2024-03-30T15:04:23.292048+0000","last_fullsized":"2024-03-30T15:04:23.292048+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:03:19.020165+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.020165+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:31.903181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":1007,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293116+0000","last_change":"2024-03-30T15:04:06.347583+0000","last_active":"2024-03-30T15:04:23.293116+0000","last_peered":"2024-03-30T15:04:23.293116+0000","last_clean":"2024-03-30T15:04:23.293116+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:04:23.293116+0000","last_undegraded":"2024-03-30T15:04:23.293116+0000","last_fullsized":"2024-03-30T15:04:23.293116+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053208+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":898,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381858+0000","last_change":"2024-03-30T15:04:06.381417+0000","last_active":"2024-03-30T15:04:23.381858+0000","last_peered":"2024-03-30T15:04:23.381858+0000","last_clean":"2024-03-30T15:04:23.381858+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:04:23.381858+0000","last_undegraded":"2024-03-30T15:04:23.381858+0000","last_fullsized":"2024-03-30T15:04:23.381858+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":918,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292820+0000","last_change":"2024-03-30T15:04:06.473326+0000","last_active":"2024-03-30T15:04:23.292820+0000","last_peered":"2024-03-30T15:04:23.292820+0000","last_clean":"2024-03-30T15:04:23.292820+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:04:23.292820+0000","last_undegraded":"2024-03-30T15:04:23.292820+0000","last_fullsized":"2024-03-30T15:04:23.292820+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.880512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":909,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603509+0000","last_change":"2024-03-30T15:04:06.473729+0000","last_active":"2024-03-30T15:04:23.603509+0000","last_peered":"2024-03-30T15:04:23.603509+0000","last_clean":"2024-03-30T15:04:23.603509+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:04:23.603509+0000","last_undegraded":"2024-03-30T15:04:23.603509+0000","last_fullsized":"2024-03-30T15:04:23.603509+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.557627+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":901,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604578+0000","last_change":"2024-03-30T15:04:06.474203+0000","last_active":"2024-03-30T15:04:23.604578+0000","last_peered":"2024-03-30T15:04:23.604578+0000","last_clean":"2024-03-30T15:04:23.604578+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:04:23.604578+0000","last_undegraded":"2024-03-30T15:04:23.604578+0000","last_fullsized":"2024-03-30T15:04:23.604578+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:02:00.557079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.557079+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"428'9","reported_seq":875,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291113+0000","last_change":"2024-03-30T15:04:06.471477+0000","last_active":"2024-03-30T15:04:23.291113+0000","last_peered":"2024-03-30T15:04:23.291113+0000","last_clean":"2024-03-30T15:04:23.291113+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:04:23.291113+0000","last_undegraded":"2024-03-30T15:04:23.291113+0000","last_fullsized":"2024-03-30T15:04:23.291113+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'5","last_scrub_stamp":"2024-03-30T15:02:29.567113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:29.567113+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":26,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"428'21","reported_seq":887,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291063+0000","last_change":"2024-03-30T15:04:06.472183+0000","last_active":"2024-03-30T15:04:23.291063+0000","last_peered":"2024-03-30T15:04:23.291063+0000","last_clean":"2024-03-30T15:04:23.291063+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:04:23.291063+0000","last_undegraded":"2024-03-30T15:04:23.291063+0000","last_fullsized":"2024-03-30T15:04:23.291063+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'17","last_scrub_stamp":"2024-03-30T15:02:35.464703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.464703+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":881,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382303+0000","last_change":"2024-03-30T15:04:06.381760+0000","last_active":"2024-03-30T15:04:23.382303+0000","last_peered":"2024-03-30T15:04:23.382303+0000","last_clean":"2024-03-30T15:04:23.382303+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:04:23.382303+0000","last_undegraded":"2024-03-30T15:04:23.382303+0000","last_fullsized":"2024-03-30T15:04:23.382303+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":945,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605621+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:04:23.605621+0000","last_peered":"2024-03-30T15:04:23.605621+0000","last_clean":"2024-03-30T15:04:23.605621+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:04:23.605621+0000","last_undegraded":"2024-03-30T15:04:23.605621+0000","last_fullsized":"2024-03-30T15:04:23.605621+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"460'60","reported_seq":926,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603690+0000","last_change":"2024-03-30T15:04:06.474098+0000","last_active":"2024-03-30T15:04:23.603690+0000","last_peered":"2024-03-30T15:04:23.603690+0000","last_clean":"2024-03-30T15:04:23.603690+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:04:23.603690+0000","last_undegraded":"2024-03-30T15:04:23.603690+0000","last_fullsized":"2024-03-30T15:04:23.603690+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":952,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291375+0000","last_change":"2024-03-30T15:04:06.346253+0000","last_active":"2024-03-30T15:04:23.291375+0000","last_peered":"2024-03-30T15:04:23.291375+0000","last_clean":"2024-03-30T15:04:23.291375+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:04:23.291375+0000","last_undegraded":"2024-03-30T15:04:23.291375+0000","last_fullsized":"2024-03-30T15:04:23.291375+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"368'35","reported_seq":897,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292797+0000","last_change":"2024-03-30T15:04:06.473836+0000","last_active":"2024-03-30T15:04:23.292797+0000","last_peered":"2024-03-30T15:04:23.292797+0000","last_clean":"2024-03-30T15:04:23.292797+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:04:23.292797+0000","last_undegraded":"2024-03-30T15:04:23.292797+0000","last_fullsized":"2024-03-30T15:04:23.292797+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'35","last_scrub_stamp":"2024-03-30T15:02:42.436904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.436904+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":965,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294930+0000","last_change":"2024-03-30T15:04:06.383533+0000","last_active":"2024-03-30T15:04:22.294930+0000","last_peered":"2024-03-30T15:04:22.294930+0000","last_clean":"2024-03-30T15:04:22.294930+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:04:22.294930+0000","last_undegraded":"2024-03-30T15:04:22.294930+0000","last_fullsized":"2024-03-30T15:04:22.294930+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:02:39.411331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.411331+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"444'55","reported_seq":927,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302875+0000","last_change":"2024-03-30T15:04:07.016310+0000","last_active":"2024-03-30T15:04:22.302875+0000","last_peered":"2024-03-30T15:04:22.302875+0000","last_clean":"2024-03-30T15:04:22.302875+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:04:22.302875+0000","last_undegraded":"2024-03-30T15:04:22.302875+0000","last_fullsized":"2024-03-30T15:04:22.302875+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:30.361533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":908,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294483+0000","last_change":"2024-03-30T15:04:06.383117+0000","last_active":"2024-03-30T15:04:22.294483+0000","last_peered":"2024-03-30T15:04:22.294483+0000","last_clean":"2024-03-30T15:04:22.294483+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:04:22.294483+0000","last_undegraded":"2024-03-30T15:04:22.294483+0000","last_fullsized":"2024-03-30T15:04:22.294483+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:02:41.402421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.402421+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":974,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.296520+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:04:22.296520+0000","last_peered":"2024-03-30T15:04:22.296520+0000","last_clean":"2024-03-30T15:04:22.296520+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:04:22.296520+0000","last_undegraded":"2024-03-30T15:04:22.296520+0000","last_fullsized":"2024-03-30T15:04:22.296520+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:00.435336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":929,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605722+0000","last_change":"2024-03-30T15:04:06.474819+0000","last_active":"2024-03-30T15:04:23.605722+0000","last_peered":"2024-03-30T15:04:23.605722+0000","last_clean":"2024-03-30T15:04:23.605722+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:04:23.605722+0000","last_undegraded":"2024-03-30T15:04:23.605722+0000","last_fullsized":"2024-03-30T15:04:23.605722+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'56","last_scrub_stamp":"2024-03-30T15:02:07.572656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.572656+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":928,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292915+0000","last_change":"2024-03-30T15:04:06.473834+0000","last_active":"2024-03-30T15:04:23.292915+0000","last_peered":"2024-03-30T15:04:23.292915+0000","last_clean":"2024-03-30T15:04:23.292915+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:04:23.292915+0000","last_undegraded":"2024-03-30T15:04:23.292915+0000","last_fullsized":"2024-03-30T15:04:23.292915+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.510044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"400'89","reported_seq":940,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381508+0000","last_change":"2024-03-30T15:04:06.381269+0000","last_active":"2024-03-30T15:04:23.381508+0000","last_peered":"2024-03-30T15:04:23.381508+0000","last_clean":"2024-03-30T15:04:23.381508+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:04:23.381508+0000","last_undegraded":"2024-03-30T15:04:23.381508+0000","last_fullsized":"2024-03-30T15:04:23.381508+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'87","last_scrub_stamp":"2024-03-30T15:02:01.656924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.656924+0000","objects_scrubbed":6,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":957,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301551+0000","last_change":"2024-03-30T15:04:07.014089+0000","last_active":"2024-03-30T15:04:22.301551+0000","last_peered":"2024-03-30T15:04:22.301551+0000","last_clean":"2024-03-30T15:04:22.301551+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:04:22.301551+0000","last_undegraded":"2024-03-30T15:04:22.301551+0000","last_fullsized":"2024-03-30T15:04:22.301551+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.190261+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":913,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292655+0000","last_change":"2024-03-30T15:04:06.343935+0000","last_active":"2024-03-30T15:04:23.292655+0000","last_peered":"2024-03-30T15:04:23.292655+0000","last_clean":"2024-03-30T15:04:23.292655+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:04:23.292655+0000","last_undegraded":"2024-03-30T15:04:23.292655+0000","last_fullsized":"2024-03-30T15:04:23.292655+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:01:46.416710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.416710+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"452'42","reported_seq":926,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.289854+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:04:23.289854+0000","last_peered":"2024-03-30T15:04:23.289854+0000","last_clean":"2024-03-30T15:04:23.289854+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:04:23.289854+0000","last_undegraded":"2024-03-30T15:04:23.289854+0000","last_fullsized":"2024-03-30T15:04:23.289854+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:21.788734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"324'104","reported_seq":977,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.295095+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:04:22.295095+0000","last_peered":"2024-03-30T15:04:22.295095+0000","last_clean":"2024-03-30T15:04:22.295095+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:04:22.295095+0000","last_undegraded":"2024-03-30T15:04:22.295095+0000","last_fullsized":"2024-03-30T15:04:22.295095+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.242607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000074,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":3937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":891,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293206+0000","last_change":"2024-03-30T15:04:06.474417+0000","last_active":"2024-03-30T15:04:23.293206+0000","last_peered":"2024-03-30T15:04:23.293206+0000","last_clean":"2024-03-30T15:04:23.293206+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:04:23.293206+0000","last_undegraded":"2024-03-30T15:04:23.293206+0000","last_fullsized":"2024-03-30T15:04:23.293206+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:02:36.478360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.478360+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":882,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.294283+0000","last_change":"2024-03-30T15:04:06.347655+0000","last_active":"2024-03-30T15:04:23.294283+0000","last_peered":"2024-03-30T15:04:23.294283+0000","last_clean":"2024-03-30T15:04:23.294283+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:04:23.294283+0000","last_undegraded":"2024-03-30T15:04:23.294283+0000","last_fullsized":"2024-03-30T15:04:23.294283+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:02:28.967486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.967486+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":926,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603838+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:04:23.603838+0000","last_peered":"2024-03-30T15:04:23.603838+0000","last_clean":"2024-03-30T15:04:23.603838+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:04:23.603838+0000","last_undegraded":"2024-03-30T15:04:23.603838+0000","last_fullsized":"2024-03-30T15:04:23.603838+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":968,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.306256+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:04:22.306256+0000","last_peered":"2024-03-30T15:04:22.306256+0000","last_clean":"2024-03-30T15:04:22.306256+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:04:22.306256+0000","last_undegraded":"2024-03-30T15:04:22.306256+0000","last_fullsized":"2024-03-30T15:04:22.306256+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:58.687173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":714,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.305164+0000","last_change":"2024-03-30T15:01:50.157960+0000","last_active":"2024-03-30T15:04:22.305164+0000","last_peered":"2024-03-30T15:04:22.305164+0000","last_clean":"2024-03-30T15:04:22.305164+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:04:22.305164+0000","last_undegraded":"2024-03-30T15:04:22.305164+0000","last_fullsized":"2024-03-30T15:04:22.305164+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.157883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.157883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.297963+0000","last_change":"2024-03-30T15:01:59.125848+0000","last_active":"2024-03-30T15:04:22.297963+0000","last_peered":"2024-03-30T15:04:22.297963+0000","last_clean":"2024-03-30T15:04:22.297963+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:04:22.297963+0000","last_undegraded":"2024-03-30T15:04:22.297963+0000","last_fullsized":"2024-03-30T15:04:22.297963+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.125727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.125727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382927+0000","last_change":"2024-03-30T15:00:43.156604+0000","last_active":"2024-03-30T15:04:23.382927+0000","last_peered":"2024-03-30T15:04:23.382927+0000","last_clean":"2024-03-30T15:04:23.382927+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:04:23.382927+0000","last_undegraded":"2024-03-30T15:04:23.382927+0000","last_fullsized":"2024-03-30T15:04:23.382927+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:43.156516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:00:43.156516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304860+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:04:22.304860+0000","last_peered":"2024-03-30T15:04:22.304860+0000","last_clean":"2024-03-30T15:04:22.304860+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:04:22.304860+0000","last_undegraded":"2024-03-30T15:04:22.304860+0000","last_fullsized":"2024-03-30T15:04:22.304860+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.852443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":719,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604294+0000","last_change":"2024-03-30T15:01:57.530926+0000","last_active":"2024-03-30T15:04:23.604294+0000","last_peered":"2024-03-30T15:04:23.604294+0000","last_clean":"2024-03-30T15:04:23.604294+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:04:23.604294+0000","last_undegraded":"2024-03-30T15:04:23.604294+0000","last_fullsized":"2024-03-30T15:04:23.604294+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.530827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.530827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":727,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.289962+0000","last_change":"2024-03-30T15:02:41.928550+0000","last_active":"2024-03-30T15:04:23.289962+0000","last_peered":"2024-03-30T15:04:23.289962+0000","last_clean":"2024-03-30T15:04:23.289962+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:04:23.289962+0000","last_undegraded":"2024-03-30T15:04:23.289962+0000","last_fullsized":"2024-03-30T15:04:23.289962+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.928469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.928469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381493+0000","last_change":"2024-03-30T15:01:57.663288+0000","last_active":"2024-03-30T15:04:23.381493+0000","last_peered":"2024-03-30T15:04:23.381493+0000","last_clean":"2024-03-30T15:04:23.381493+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:04:23.381493+0000","last_undegraded":"2024-03-30T15:04:23.381493+0000","last_fullsized":"2024-03-30T15:04:23.381493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.663204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.663204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":714,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.300830+0000","last_change":"2024-03-30T15:01:59.181493+0000","last_active":"2024-03-30T15:04:22.300830+0000","last_peered":"2024-03-30T15:04:22.300830+0000","last_clean":"2024-03-30T15:04:22.300830+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:04:22.300830+0000","last_undegraded":"2024-03-30T15:04:22.300830+0000","last_fullsized":"2024-03-30T15:04:22.300830+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.181401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.181401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293065+0000","last_change":"2024-03-30T15:02:34.464153+0000","last_active":"2024-03-30T15:04:23.293065+0000","last_peered":"2024-03-30T15:04:23.293065+0000","last_clean":"2024-03-30T15:04:23.293065+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:04:23.293065+0000","last_undegraded":"2024-03-30T15:04:23.293065+0000","last_fullsized":"2024-03-30T15:04:23.293065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:34.464067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.464067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302972+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:04:22.302972+0000","last_peered":"2024-03-30T15:04:22.302972+0000","last_clean":"2024-03-30T15:04:22.302972+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:04:22.302972+0000","last_undegraded":"2024-03-30T15:04:22.302972+0000","last_fullsized":"2024-03-30T15:04:22.302972+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.860968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":716,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382934+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:04:23.382934+0000","last_peered":"2024-03-30T15:04:23.382934+0000","last_clean":"2024-03-30T15:04:23.382934+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:04:23.382934+0000","last_undegraded":"2024-03-30T15:04:23.382934+0000","last_fullsized":"2024-03-30T15:04:23.382934+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":728,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292161+0000","last_change":"2024-03-30T15:02:38.936916+0000","last_active":"2024-03-30T15:04:23.292161+0000","last_peered":"2024-03-30T15:04:23.292161+0000","last_clean":"2024-03-30T15:04:23.292161+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:04:23.292161+0000","last_undegraded":"2024-03-30T15:04:23.292161+0000","last_fullsized":"2024-03-30T15:04:23.292161+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:38.936820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.936820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380585+0000","last_change":"2024-03-30T15:01:51.636327+0000","last_active":"2024-03-30T15:04:23.380585+0000","last_peered":"2024-03-30T15:04:23.380585+0000","last_clean":"2024-03-30T15:04:23.380585+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:04:23.380585+0000","last_undegraded":"2024-03-30T15:04:23.380585+0000","last_fullsized":"2024-03-30T15:04:23.380585+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.636231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.636231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.288858+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:04:22.288858+0000","last_peered":"2024-03-30T15:04:22.288858+0000","last_clean":"2024-03-30T15:04:22.288858+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:04:22.288858+0000","last_undegraded":"2024-03-30T15:04:22.288858+0000","last_fullsized":"2024-03-30T15:04:22.288858+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.681651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":727,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292763+0000","last_change":"2024-03-30T15:02:00.582792+0000","last_active":"2024-03-30T15:04:23.292763+0000","last_peered":"2024-03-30T15:04:23.292763+0000","last_clean":"2024-03-30T15:04:23.292763+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:04:23.292763+0000","last_undegraded":"2024-03-30T15:04:23.292763+0000","last_fullsized":"2024-03-30T15:04:23.292763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.582709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":727,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291928+0000","last_change":"2024-03-30T15:02:07.597777+0000","last_active":"2024-03-30T15:04:23.291928+0000","last_peered":"2024-03-30T15:04:23.291928+0000","last_clean":"2024-03-30T15:04:23.291928+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:04:23.291928+0000","last_undegraded":"2024-03-30T15:04:23.291928+0000","last_fullsized":"2024-03-30T15:04:23.291928+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.597700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":716,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381766+0000","last_change":"2024-03-30T15:01:10.292440+0000","last_active":"2024-03-30T15:04:23.381766+0000","last_peered":"2024-03-30T15:04:23.381766+0000","last_clean":"2024-03-30T15:04:23.381766+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:04:23.381766+0000","last_undegraded":"2024-03-30T15:04:23.381766+0000","last_fullsized":"2024-03-30T15:04:23.381766+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.292346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.292346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":716,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293102+0000","last_change":"2024-03-30T15:01:54.513739+0000","last_active":"2024-03-30T15:04:23.293102+0000","last_peered":"2024-03-30T15:04:23.293102+0000","last_clean":"2024-03-30T15:04:23.293102+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:04:23.293102+0000","last_undegraded":"2024-03-30T15:04:23.293102+0000","last_fullsized":"2024-03-30T15:04:23.293102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.513601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.513601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599289+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":714,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303119+0000","last_change":"2024-03-30T15:01:22.172967+0000","last_active":"2024-03-30T15:04:22.303119+0000","last_peered":"2024-03-30T15:04:22.303119+0000","last_clean":"2024-03-30T15:04:22.303119+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:04:22.303119+0000","last_undegraded":"2024-03-30T15:04:22.303119+0000","last_fullsized":"2024-03-30T15:04:22.303119+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:22.172854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:22.172854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382800+0000","last_change":"2024-03-30T15:01:19.395733+0000","last_active":"2024-03-30T15:04:23.382800+0000","last_peered":"2024-03-30T15:04:23.382800+0000","last_clean":"2024-03-30T15:04:23.382800+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:04:23.382800+0000","last_undegraded":"2024-03-30T15:04:23.382800+0000","last_fullsized":"2024-03-30T15:04:23.382800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:19.395643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:19.395643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":723,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.293637+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:04:22.293637+0000","last_peered":"2024-03-30T15:04:22.293637+0000","last_clean":"2024-03-30T15:04:22.293637+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:04:22.293637+0000","last_undegraded":"2024-03-30T15:04:22.293637+0000","last_fullsized":"2024-03-30T15:04:22.293637+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171594+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":724,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380587+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:04:23.380587+0000","last_peered":"2024-03-30T15:04:23.380587+0000","last_clean":"2024-03-30T15:04:23.380587+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:04:23.380587+0000","last_undegraded":"2024-03-30T15:04:23.380587+0000","last_fullsized":"2024-03-30T15:04:23.380587+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.116140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":723,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.295100+0000","last_change":"2024-03-30T15:02:40.426121+0000","last_active":"2024-03-30T15:04:22.295100+0000","last_peered":"2024-03-30T15:04:22.295100+0000","last_clean":"2024-03-30T15:04:22.295100+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:04:22.295100+0000","last_undegraded":"2024-03-30T15:04:22.295100+0000","last_fullsized":"2024-03-30T15:04:22.295100+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.426031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.426031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":736,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291262+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:04:23.291262+0000","last_peered":"2024-03-30T15:04:23.291262+0000","last_clean":"2024-03-30T15:04:23.291262+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:04:23.291262+0000","last_undegraded":"2024-03-30T15:04:23.291262+0000","last_fullsized":"2024-03-30T15:04:23.291262+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.425095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294208+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:04:22.294208+0000","last_peered":"2024-03-30T15:04:22.294208+0000","last_clean":"2024-03-30T15:04:22.294208+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:04:22.294208+0000","last_undegraded":"2024-03-30T15:04:22.294208+0000","last_fullsized":"2024-03-30T15:04:22.294208+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303077+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:04:22.303077+0000","last_peered":"2024-03-30T15:04:22.303077+0000","last_clean":"2024-03-30T15:04:22.303077+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:04:22.303077+0000","last_undegraded":"2024-03-30T15:04:22.303077+0000","last_fullsized":"2024-03-30T15:04:22.303077+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":714,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303634+0000","last_change":"2024-03-30T15:01:58.138003+0000","last_active":"2024-03-30T15:04:22.303634+0000","last_peered":"2024-03-30T15:04:22.303634+0000","last_clean":"2024-03-30T15:04:22.303634+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:04:22.303634+0000","last_undegraded":"2024-03-30T15:04:22.303634+0000","last_fullsized":"2024-03-30T15:04:22.303634+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.137887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.137887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":714,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304756+0000","last_change":"2024-03-30T15:01:57.095860+0000","last_active":"2024-03-30T15:04:22.304756+0000","last_peered":"2024-03-30T15:04:22.304756+0000","last_clean":"2024-03-30T15:04:22.304756+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:04:22.304756+0000","last_undegraded":"2024-03-30T15:04:22.304756+0000","last_fullsized":"2024-03-30T15:04:22.304756+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.095752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.095752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294606+0000","last_change":"2024-03-30T15:01:58.162003+0000","last_active":"2024-03-30T15:04:22.294606+0000","last_peered":"2024-03-30T15:04:22.294606+0000","last_clean":"2024-03-30T15:04:22.294606+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:04:22.294606+0000","last_undegraded":"2024-03-30T15:04:22.294606+0000","last_fullsized":"2024-03-30T15:04:22.294606+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.161871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.161871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302704+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:04:22.302704+0000","last_peered":"2024-03-30T15:04:22.302704+0000","last_clean":"2024-03-30T15:04:22.302704+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:04:22.302704+0000","last_undegraded":"2024-03-30T15:04:22.302704+0000","last_fullsized":"2024-03-30T15:04:22.302704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.836391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291534+0000","last_change":"2024-03-30T15:01:51.481224+0000","last_active":"2024-03-30T15:04:23.291534+0000","last_peered":"2024-03-30T15:04:23.291534+0000","last_clean":"2024-03-30T15:04:23.291534+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:04:23.291534+0000","last_undegraded":"2024-03-30T15:04:23.291534+0000","last_fullsized":"2024-03-30T15:04:23.291534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.481143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.481143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":715,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380955+0000","last_change":"2024-03-30T15:01:20.445532+0000","last_active":"2024-03-30T15:04:23.380955+0000","last_peered":"2024-03-30T15:04:23.380955+0000","last_clean":"2024-03-30T15:04:23.380955+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:04:23.380955+0000","last_undegraded":"2024-03-30T15:04:23.380955+0000","last_fullsized":"2024-03-30T15:04:23.380955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:20.445448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.445448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":736,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298057+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:04:22.298057+0000","last_peered":"2024-03-30T15:04:22.298057+0000","last_clean":"2024-03-30T15:04:22.298057+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:04:22.298057+0000","last_undegraded":"2024-03-30T15:04:22.298057+0000","last_fullsized":"2024-03-30T15:04:22.298057+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:38.423498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"76'10","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293217+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:04:23.293217+0000","last_peered":"2024-03-30T15:04:23.293217+0000","last_clean":"2024-03-30T15:04:23.293217+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:04:23.293217+0000","last_undegraded":"2024-03-30T15:04:23.293217+0000","last_fullsized":"2024-03-30T15:04:23.293217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":729,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605177+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:04:23.605177+0000","last_peered":"2024-03-30T15:04:23.605177+0000","last_clean":"2024-03-30T15:04:23.605177+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:04:23.605177+0000","last_undegraded":"2024-03-30T15:04:23.605177+0000","last_fullsized":"2024-03-30T15:04:23.605177+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":960,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291894+0000","last_change":"2024-03-30T15:04:03.157433+0000","last_active":"2024-03-30T15:04:23.291894+0000","last_peered":"2024-03-30T15:04:23.291894+0000","last_clean":"2024-03-30T15:04:23.291894+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:04:23.291894+0000","last_undegraded":"2024-03-30T15:04:23.291894+0000","last_fullsized":"2024-03-30T15:04:23.291894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.245942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293783+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:04:23.293783+0000","last_peered":"2024-03-30T15:04:23.293783+0000","last_clean":"2024-03-30T15:04:23.293783+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:23.293783+0000","last_undegraded":"2024-03-30T15:04:23.293783+0000","last_fullsized":"2024-03-30T15:04:23.293783+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.569612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":736,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294405+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:04:22.294405+0000","last_peered":"2024-03-30T15:04:22.294405+0000","last_clean":"2024-03-30T15:04:22.294405+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:04:22.294405+0000","last_undegraded":"2024-03-30T15:04:22.294405+0000","last_fullsized":"2024-03-30T15:04:22.294405+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:26.821252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'17","reported_seq":848,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293393+0000","last_change":"2024-03-30T15:00:02.464828+0000","last_active":"2024-03-30T15:04:23.293393+0000","last_peered":"2024-03-30T15:04:23.293393+0000","last_clean":"2024-03-30T15:04:23.293393+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:04:23.293393+0000","last_undegraded":"2024-03-30T15:04:23.293393+0000","last_fullsized":"2024-03-30T15:04:23.293393+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:00:02.464690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.464690+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":830,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.297077+0000","last_change":"2024-03-30T15:02:02.104334+0000","last_active":"2024-03-30T15:04:22.297077+0000","last_peered":"2024-03-30T15:04:22.297077+0000","last_clean":"2024-03-30T15:04:22.297077+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:04:22.297077+0000","last_undegraded":"2024-03-30T15:04:22.297077+0000","last_fullsized":"2024-03-30T15:04:22.297077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.104245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":823,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303634+0000","last_change":"2024-03-30T15:01:48.369544+0000","last_active":"2024-03-30T15:04:22.303634+0000","last_peered":"2024-03-30T15:04:22.303634+0000","last_clean":"2024-03-30T15:04:22.303634+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:04:22.303634+0000","last_undegraded":"2024-03-30T15:04:22.303634+0000","last_fullsized":"2024-03-30T15:04:22.303634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:48.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.294348+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:04:23.294348+0000","last_peered":"2024-03-30T15:04:23.294348+0000","last_clean":"2024-03-30T15:04:23.294348+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:04:23.294348+0000","last_undegraded":"2024-03-30T15:04:23.294348+0000","last_fullsized":"2024-03-30T15:04:23.294348+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:33.116494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":729,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.309419+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:04:22.309419+0000","last_peered":"2024-03-30T15:04:22.309419+0000","last_clean":"2024-03-30T15:04:22.309419+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:04:22.309419+0000","last_undegraded":"2024-03-30T15:04:22.309419+0000","last_fullsized":"2024-03-30T15:04:22.309419+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171527+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"93'16","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293659+0000","last_change":"2024-03-30T15:02:03.449650+0000","last_active":"2024-03-30T15:04:23.293659+0000","last_peered":"2024-03-30T15:04:23.293659+0000","last_clean":"2024-03-30T15:04:23.293659+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:04:23.293659+0000","last_undegraded":"2024-03-30T15:04:23.293659+0000","last_fullsized":"2024-03-30T15:04:23.293659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:03.449549+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.449549+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291537+0000","last_change":"2024-03-30T15:01:50.503242+0000","last_active":"2024-03-30T15:04:23.291537+0000","last_peered":"2024-03-30T15:04:23.291537+0000","last_clean":"2024-03-30T15:04:23.291537+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:04:23.291537+0000","last_undegraded":"2024-03-30T15:04:23.291537+0000","last_fullsized":"2024-03-30T15:04:23.291537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.503156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.503156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":815,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.383185+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:04:23.383185+0000","last_peered":"2024-03-30T15:04:23.383185+0000","last_clean":"2024-03-30T15:04:23.383185+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:04:23.383185+0000","last_undegraded":"2024-03-30T15:04:23.383185+0000","last_fullsized":"2024-03-30T15:04:23.383185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":731,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298212+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:04:22.298212+0000","last_peered":"2024-03-30T15:04:22.298212+0000","last_clean":"2024-03-30T15:04:22.298212+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:04:22.298212+0000","last_undegraded":"2024-03-30T15:04:22.298212+0000","last_fullsized":"2024-03-30T15:04:22.298212+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":729,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.309264+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:04:22.309264+0000","last_peered":"2024-03-30T15:04:22.309264+0000","last_clean":"2024-03-30T15:04:22.309264+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:04:22.309264+0000","last_undegraded":"2024-03-30T15:04:22.309264+0000","last_fullsized":"2024-03-30T15:04:22.309264+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"94'22","reported_seq":851,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.309352+0000","last_change":"2024-03-30T15:01:46.424704+0000","last_active":"2024-03-30T15:04:22.309352+0000","last_peered":"2024-03-30T15:04:22.309352+0000","last_clean":"2024-03-30T15:04:22.309352+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:04:22.309352+0000","last_undegraded":"2024-03-30T15:04:22.309352+0000","last_fullsized":"2024-03-30T15:04:22.309352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T15:01:46.424546+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.424546+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":724,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.289757+0000","last_change":"2024-03-30T15:01:50.443926+0000","last_active":"2024-03-30T15:04:22.289757+0000","last_peered":"2024-03-30T15:04:22.289757+0000","last_clean":"2024-03-30T15:04:22.289757+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:04:22.289757+0000","last_undegraded":"2024-03-30T15:04:22.289757+0000","last_fullsized":"2024-03-30T15:04:22.289757+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.443839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.443839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":942,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293770+0000","last_change":"2024-03-30T15:03:22.567558+0000","last_active":"2024-03-30T15:04:23.293770+0000","last_peered":"2024-03-30T15:04:23.293770+0000","last_clean":"2024-03-30T15:04:23.293770+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:04:23.293770+0000","last_undegraded":"2024-03-30T15:04:23.293770+0000","last_fullsized":"2024-03-30T15:04:23.293770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:22.567465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.567465+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.595071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":811,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381911+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:04:23.381911+0000","last_peered":"2024-03-30T15:04:23.381911+0000","last_clean":"2024-03-30T15:04:23.381911+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:04:23.381911+0000","last_undegraded":"2024-03-30T15:04:23.381911+0000","last_fullsized":"2024-03-30T15:04:23.381911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:11.636902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":727,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290343+0000","last_change":"2024-03-30T15:02:30.902281+0000","last_active":"2024-03-30T15:04:23.290343+0000","last_peered":"2024-03-30T15:04:23.290343+0000","last_clean":"2024-03-30T15:04:23.290343+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:04:23.290343+0000","last_undegraded":"2024-03-30T15:04:23.290343+0000","last_fullsized":"2024-03-30T15:04:23.290343+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.902199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.902199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291364+0000","last_change":"2024-03-30T15:02:39.473396+0000","last_active":"2024-03-30T15:04:23.291364+0000","last_peered":"2024-03-30T15:04:23.291364+0000","last_clean":"2024-03-30T15:04:23.291364+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:04:23.291364+0000","last_undegraded":"2024-03-30T15:04:23.291364+0000","last_fullsized":"2024-03-30T15:04:23.291364+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:39.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.473279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":814,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303595+0000","last_change":"2024-03-30T15:02:34.958123+0000","last_active":"2024-03-30T15:04:22.303595+0000","last_peered":"2024-03-30T15:04:22.303595+0000","last_clean":"2024-03-30T15:04:22.303595+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:04:22.303595+0000","last_undegraded":"2024-03-30T15:04:22.303595+0000","last_fullsized":"2024-03-30T15:04:22.303595+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T15:02:34.958001+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.958001+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":797,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303216+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:04:22.303216+0000","last_peered":"2024-03-30T15:04:22.303216+0000","last_clean":"2024-03-30T15:04:22.303216+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:04:22.303216+0000","last_undegraded":"2024-03-30T15:04:22.303216+0000","last_fullsized":"2024-03-30T15:04:22.303216+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293373+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:04:23.293373+0000","last_peered":"2024-03-30T15:04:23.293373+0000","last_clean":"2024-03-30T15:04:23.293373+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:04:23.293373+0000","last_undegraded":"2024-03-30T15:04:23.293373+0000","last_fullsized":"2024-03-30T15:04:23.293373+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.431637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":735,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290624+0000","last_change":"2024-03-30T15:02:06.597375+0000","last_active":"2024-03-30T15:04:23.290624+0000","last_peered":"2024-03-30T15:04:23.290624+0000","last_clean":"2024-03-30T15:04:23.290624+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:04:23.290624+0000","last_undegraded":"2024-03-30T15:04:23.290624+0000","last_fullsized":"2024-03-30T15:04:23.290624+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:06.597250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:06.597250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":827,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.289227+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:04:22.289227+0000","last_peered":"2024-03-30T15:04:22.289227+0000","last_clean":"2024-03-30T15:04:22.289227+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:04:22.289227+0000","last_undegraded":"2024-03-30T15:04:22.289227+0000","last_fullsized":"2024-03-30T15:04:22.289227+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"76'14","reported_seq":849,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291739+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:04:23.291739+0000","last_peered":"2024-03-30T15:04:23.291739+0000","last_clean":"2024-03-30T15:04:23.291739+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:04:23.291739+0000","last_undegraded":"2024-03-30T15:04:23.291739+0000","last_fullsized":"2024-03-30T15:04:23.291739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.978250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605837+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:04:23.605837+0000","last_peered":"2024-03-30T15:04:23.605837+0000","last_clean":"2024-03-30T15:04:23.605837+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:04:23.605837+0000","last_undegraded":"2024-03-30T15:04:23.605837+0000","last_fullsized":"2024-03-30T15:04:23.605837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.403595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":819,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301319+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:04:22.301319+0000","last_peered":"2024-03-30T15:04:22.301319+0000","last_clean":"2024-03-30T15:04:22.301319+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:04:22.301319+0000","last_undegraded":"2024-03-30T15:04:22.301319+0000","last_fullsized":"2024-03-30T15:04:22.301319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:45.356946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":735,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292005+0000","last_change":"2024-03-30T15:02:05.626016+0000","last_active":"2024-03-30T15:04:23.292005+0000","last_peered":"2024-03-30T15:04:23.292005+0000","last_clean":"2024-03-30T15:04:23.292005+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:04:23.292005+0000","last_undegraded":"2024-03-30T15:04:23.292005+0000","last_fullsized":"2024-03-30T15:04:23.292005+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:05.625940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.625940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":954,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605079+0000","last_change":"2024-03-30T15:01:48.471009+0000","last_active":"2024-03-30T15:04:23.605079+0000","last_peered":"2024-03-30T15:04:23.605079+0000","last_clean":"2024-03-30T15:04:23.605079+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:04:23.605079+0000","last_undegraded":"2024-03-30T15:04:23.605079+0000","last_fullsized":"2024-03-30T15:04:23.605079+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:48.470884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.470884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"186'15","reported_seq":847,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304158+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:04:22.304158+0000","last_peered":"2024-03-30T15:04:22.304158+0000","last_clean":"2024-03-30T15:04:22.304158+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:04:22.304158+0000","last_undegraded":"2024-03-30T15:04:22.304158+0000","last_fullsized":"2024-03-30T15:04:22.304158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382160+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:04:23.382160+0000","last_peered":"2024-03-30T15:04:23.382160+0000","last_clean":"2024-03-30T15:04:23.382160+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:04:23.382160+0000","last_undegraded":"2024-03-30T15:04:23.382160+0000","last_fullsized":"2024-03-30T15:04:23.382160+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.920854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603884+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:04:23.603884+0000","last_peered":"2024-03-30T15:04:23.603884+0000","last_clean":"2024-03-30T15:04:23.603884+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:04:23.603884+0000","last_undegraded":"2024-03-30T15:04:23.603884+0000","last_fullsized":"2024-03-30T15:04:23.603884+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":727,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293067+0000","last_change":"2024-03-30T15:01:52.535873+0000","last_active":"2024-03-30T15:04:23.293067+0000","last_peered":"2024-03-30T15:04:23.293067+0000","last_clean":"2024-03-30T15:04:23.293067+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:04:23.293067+0000","last_undegraded":"2024-03-30T15:04:23.293067+0000","last_fullsized":"2024-03-30T15:04:23.293067+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.535788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.535788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":830,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.297446+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:04:22.297446+0000","last_peered":"2024-03-30T15:04:22.297446+0000","last_clean":"2024-03-30T15:04:22.297446+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:04:22.297446+0000","last_undegraded":"2024-03-30T15:04:22.297446+0000","last_fullsized":"2024-03-30T15:04:22.297446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.848800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"95'27","reported_seq":868,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605508+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:04:23.605508+0000","last_peered":"2024-03-30T15:04:23.605508+0000","last_clean":"2024-03-30T15:04:23.605508+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:04:23.605508+0000","last_undegraded":"2024-03-30T15:04:23.605508+0000","last_fullsized":"2024-03-30T15:04:23.605508+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.338727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":831,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303439+0000","last_change":"2024-03-30T15:01:57.409471+0000","last_active":"2024-03-30T15:04:22.303439+0000","last_peered":"2024-03-30T15:04:22.303439+0000","last_clean":"2024-03-30T15:04:22.303439+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:04:22.303439+0000","last_undegraded":"2024-03-30T15:04:22.303439+0000","last_fullsized":"2024-03-30T15:04:22.303439+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.409374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.409374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":819,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294012+0000","last_change":"2024-03-30T15:01:54.450014+0000","last_active":"2024-03-30T15:04:22.294012+0000","last_peered":"2024-03-30T15:04:22.294012+0000","last_clean":"2024-03-30T15:04:22.294012+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:04:22.294012+0000","last_undegraded":"2024-03-30T15:04:22.294012+0000","last_fullsized":"2024-03-30T15:04:22.294012+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.449931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.449931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171375+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605711+0000","last_change":"2024-03-30T15:01:56.570174+0000","last_active":"2024-03-30T15:04:23.605711+0000","last_peered":"2024-03-30T15:04:23.605711+0000","last_clean":"2024-03-30T15:04:23.605711+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:04:23.605711+0000","last_undegraded":"2024-03-30T15:04:23.605711+0000","last_fullsized":"2024-03-30T15:04:23.605711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:56.570063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:56.570063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.383190+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:04:23.383190+0000","last_peered":"2024-03-30T15:04:23.383190+0000","last_clean":"2024-03-30T15:04:23.383190+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:04:23.383190+0000","last_undegraded":"2024-03-30T15:04:23.383190+0000","last_fullsized":"2024-03-30T15:04:23.383190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.978863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"76'12","reported_seq":850,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603216+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:04:23.603216+0000","last_peered":"2024-03-30T15:04:23.603216+0000","last_clean":"2024-03-30T15:04:23.603216+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:04:23.603216+0000","last_undegraded":"2024-03-30T15:04:23.603216+0000","last_fullsized":"2024-03-30T15:04:23.603216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.711973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"77'5","reported_seq":1413,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303219+0000","last_change":"2024-03-30T15:02:03.014253+0000","last_active":"2024-03-30T15:04:22.303219+0000","last_peered":"2024-03-30T15:04:22.303219+0000","last_clean":"2024-03-30T15:04:22.303219+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:04:22.303219+0000","last_undegraded":"2024-03-30T15:04:22.303219+0000","last_fullsized":"2024-03-30T15:04:22.303219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-03-30T15:02:03.014132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.014132+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":578,"num_read_kb":573,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603937+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:04:23.603937+0000","last_peered":"2024-03-30T15:04:23.603937+0000","last_clean":"2024-03-30T15:04:23.603937+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:04:23.603937+0000","last_undegraded":"2024-03-30T15:04:23.603937+0000","last_fullsized":"2024-03-30T15:04:23.603937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:44.443232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":819,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303224+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:04:22.303224+0000","last_peered":"2024-03-30T15:04:22.303224+0000","last_clean":"2024-03-30T15:04:22.303224+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:04:22.303224+0000","last_undegraded":"2024-03-30T15:04:22.303224+0000","last_fullsized":"2024-03-30T15:04:22.303224+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.928596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":719,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.305928+0000","last_change":"2024-03-30T15:01:18.064257+0000","last_active":"2024-03-30T15:04:22.305928+0000","last_peered":"2024-03-30T15:04:22.305928+0000","last_clean":"2024-03-30T15:04:22.305928+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:04:22.305928+0000","last_undegraded":"2024-03-30T15:04:22.305928+0000","last_fullsized":"2024-03-30T15:04:22.305928+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:18.064173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.064173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":815,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302816+0000","last_change":"2024-03-30T15:02:33.948293+0000","last_active":"2024-03-30T15:04:22.302816+0000","last_peered":"2024-03-30T15:04:22.302816+0000","last_clean":"2024-03-30T15:04:22.302816+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:04:22.302816+0000","last_undegraded":"2024-03-30T15:04:22.302816+0000","last_fullsized":"2024-03-30T15:04:22.302816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:33.948156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.948156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"76'16","reported_seq":974,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294668+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:04:22.294668+0000","last_peered":"2024-03-30T15:04:22.294668+0000","last_clean":"2024-03-30T15:04:22.294668+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:04:22.294668+0000","last_undegraded":"2024-03-30T15:04:22.294668+0000","last_fullsized":"2024-03-30T15:04:22.294668+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:27.345625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":726,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381147+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:04:23.381147+0000","last_peered":"2024-03-30T15:04:23.381147+0000","last_clean":"2024-03-30T15:04:23.381147+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:04:23.381147+0000","last_undegraded":"2024-03-30T15:04:23.381147+0000","last_fullsized":"2024-03-30T15:04:23.381147+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.548386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1084,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303278+0000","last_change":"2024-03-30T15:02:39.890565+0000","last_active":"2024-03-30T15:04:22.303278+0000","last_peered":"2024-03-30T15:04:22.303278+0000","last_clean":"2024-03-30T15:04:22.303278+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:04:22.303278+0000","last_undegraded":"2024-03-30T15:04:22.303278+0000","last_fullsized":"2024-03-30T15:04:22.303278+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:02:39.890462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.890462+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":811,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.383059+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:04:23.383059+0000","last_peered":"2024-03-30T15:04:23.383059+0000","last_clean":"2024-03-30T15:04:23.383059+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:04:23.383059+0000","last_undegraded":"2024-03-30T15:04:23.383059+0000","last_fullsized":"2024-03-30T15:04:23.383059+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.463993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":735,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.296260+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:04:22.296260+0000","last_peered":"2024-03-30T15:04:22.296260+0000","last_clean":"2024-03-30T15:04:22.296260+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:04:22.296260+0000","last_undegraded":"2024-03-30T15:04:22.296260+0000","last_fullsized":"2024-03-30T15:04:22.296260+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.021923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292433+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:04:23.292433+0000","last_peered":"2024-03-30T15:04:23.292433+0000","last_clean":"2024-03-30T15:04:23.292433+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:04:23.292433+0000","last_undegraded":"2024-03-30T15:04:23.292433+0000","last_fullsized":"2024-03-30T15:04:23.292433+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.873816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"106'18","reported_seq":867,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.309708+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:04:22.309708+0000","last_peered":"2024-03-30T15:04:22.309708+0000","last_clean":"2024-03-30T15:04:22.309708+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:04:22.309708+0000","last_undegraded":"2024-03-30T15:04:22.309708+0000","last_fullsized":"2024-03-30T15:04:22.309708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:54.532654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":843,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290734+0000","last_change":"2024-03-30T15:01:53.529768+0000","last_active":"2024-03-30T15:04:23.290734+0000","last_peered":"2024-03-30T15:04:23.290734+0000","last_clean":"2024-03-30T15:04:23.290734+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:04:23.290734+0000","last_undegraded":"2024-03-30T15:04:23.290734+0000","last_fullsized":"2024-03-30T15:04:23.290734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.529037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.529037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":831,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.289334+0000","last_change":"2024-03-30T15:01:59.392426+0000","last_active":"2024-03-30T15:04:22.289334+0000","last_peered":"2024-03-30T15:04:22.289334+0000","last_clean":"2024-03-30T15:04:22.289334+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:04:22.289334+0000","last_undegraded":"2024-03-30T15:04:22.289334+0000","last_fullsized":"2024-03-30T15:04:22.289334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.392341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":819,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.306444+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:04:22.306444+0000","last_peered":"2024-03-30T15:04:22.306444+0000","last_clean":"2024-03-30T15:04:22.306444+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:04:22.306444+0000","last_undegraded":"2024-03-30T15:04:22.306444+0000","last_fullsized":"2024-03-30T15:04:22.306444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.756028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604739+0000","last_change":"2024-03-30T15:01:59.537303+0000","last_active":"2024-03-30T15:04:23.604739+0000","last_peered":"2024-03-30T15:04:23.604739+0000","last_clean":"2024-03-30T15:04:23.604739+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:04:23.604739+0000","last_undegraded":"2024-03-30T15:04:23.604739+0000","last_fullsized":"2024-03-30T15:04:23.604739+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.537225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.537225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602766+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":820,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605098+0000","last_change":"2024-03-30T15:00:44.580234+0000","last_active":"2024-03-30T15:04:23.605098+0000","last_peered":"2024-03-30T15:04:23.605098+0000","last_clean":"2024-03-30T15:04:23.605098+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:04:23.605098+0000","last_undegraded":"2024-03-30T15:04:23.605098+0000","last_fullsized":"2024-03-30T15:04:23.605098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:44.580140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:00:44.580140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"79'29","reported_seq":940,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381990+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:04:23.381990+0000","last_peered":"2024-03-30T15:04:23.381990+0000","last_clean":"2024-03-30T15:04:23.381990+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:04:23.381990+0000","last_undegraded":"2024-03-30T15:04:23.381990+0000","last_fullsized":"2024-03-30T15:04:23.381990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291842+0000","last_change":"2024-03-30T15:02:00.474885+0000","last_active":"2024-03-30T15:04:23.291842+0000","last_peered":"2024-03-30T15:04:23.291842+0000","last_clean":"2024-03-30T15:04:23.291842+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:04:23.291842+0000","last_undegraded":"2024-03-30T15:04:23.291842+0000","last_fullsized":"2024-03-30T15:04:23.291842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:00.474742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.474742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599344+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381292+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:04:23.381292+0000","last_peered":"2024-03-30T15:04:23.381292+0000","last_clean":"2024-03-30T15:04:23.381292+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:04:23.381292+0000","last_undegraded":"2024-03-30T15:04:23.381292+0000","last_fullsized":"2024-03-30T15:04:23.381292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.645581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":724,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293323+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:04:23.293323+0000","last_peered":"2024-03-30T15:04:23.293323+0000","last_clean":"2024-03-30T15:04:23.293323+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:04:23.293323+0000","last_undegraded":"2024-03-30T15:04:23.293323+0000","last_fullsized":"2024-03-30T15:04:23.293323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.617161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":726,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605421+0000","last_change":"2024-03-30T15:01:58.563335+0000","last_active":"2024-03-30T15:04:23.605421+0000","last_peered":"2024-03-30T15:04:23.605421+0000","last_clean":"2024-03-30T15:04:23.605421+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:04:23.605421+0000","last_undegraded":"2024-03-30T15:04:23.605421+0000","last_fullsized":"2024-03-30T15:04:23.605421+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602673+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"76'9","reported_seq":840,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603199+0000","last_change":"2024-03-30T15:00:33.505380+0000","last_active":"2024-03-30T15:04:23.603199+0000","last_peered":"2024-03-30T15:04:23.603199+0000","last_clean":"2024-03-30T15:04:23.603199+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:04:23.603199+0000","last_undegraded":"2024-03-30T15:04:23.603199+0000","last_fullsized":"2024-03-30T15:04:23.603199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:00:33.505287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:00:33.505287+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":831,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382241+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:04:23.382241+0000","last_peered":"2024-03-30T15:04:23.382241+0000","last_clean":"2024-03-30T15:04:23.382241+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:04:23.382241+0000","last_undegraded":"2024-03-30T15:04:23.382241+0000","last_fullsized":"2024-03-30T15:04:23.382241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.634280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291460+0000","last_change":"2024-03-30T15:03:26.133453+0000","last_active":"2024-03-30T15:04:23.291460+0000","last_peered":"2024-03-30T15:04:23.291460+0000","last_clean":"2024-03-30T15:04:23.291460+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:04:23.291460+0000","last_undegraded":"2024-03-30T15:04:23.291460+0000","last_fullsized":"2024-03-30T15:04:23.291460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:26.133342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:26.133342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.286995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604196+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:04:23.604196+0000","last_peered":"2024-03-30T15:04:23.604196+0000","last_clean":"2024-03-30T15:04:23.604196+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:04:23.604196+0000","last_undegraded":"2024-03-30T15:04:23.604196+0000","last_fullsized":"2024-03-30T15:04:23.604196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":721,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.294586+0000","last_change":"2024-03-30T15:01:53.490679+0000","last_active":"2024-03-30T15:04:23.294586+0000","last_peered":"2024-03-30T15:04:23.294586+0000","last_clean":"2024-03-30T15:04:23.294586+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:04:23.294586+0000","last_undegraded":"2024-03-30T15:04:23.294586+0000","last_fullsized":"2024-03-30T15:04:23.294586+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.490591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.490591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'19","reported_seq":891,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.297360+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:04:22.297360+0000","last_peered":"2024-03-30T15:04:22.297360+0000","last_clean":"2024-03-30T15:04:22.297360+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:04:22.297360+0000","last_undegraded":"2024-03-30T15:04:22.297360+0000","last_fullsized":"2024-03-30T15:04:22.297360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:41.052823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":823,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.306132+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:04:22.306132+0000","last_peered":"2024-03-30T15:04:22.306132+0000","last_clean":"2024-03-30T15:04:22.306132+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:04:22.306132+0000","last_undegraded":"2024-03-30T15:04:22.306132+0000","last_fullsized":"2024-03-30T15:04:22.306132+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":834,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293020+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:04:23.293020+0000","last_peered":"2024-03-30T15:04:23.293020+0000","last_clean":"2024-03-30T15:04:23.293020+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:04:23.293020+0000","last_undegraded":"2024-03-30T15:04:23.293020+0000","last_fullsized":"2024-03-30T15:04:23.293020+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.803620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":811,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303069+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:04:22.303069+0000","last_peered":"2024-03-30T15:04:22.303069+0000","last_clean":"2024-03-30T15:04:22.303069+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:04:22.303069+0000","last_undegraded":"2024-03-30T15:04:22.303069+0000","last_fullsized":"2024-03-30T15:04:22.303069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":749,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294715+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:04:22.294715+0000","last_peered":"2024-03-30T15:04:22.294715+0000","last_clean":"2024-03-30T15:04:22.294715+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:04:22.294715+0000","last_undegraded":"2024-03-30T15:04:22.294715+0000","last_fullsized":"2024-03-30T15:04:22.294715+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.286336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":826,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290105+0000","last_change":"2024-03-30T15:01:55.535170+0000","last_active":"2024-03-30T15:04:23.290105+0000","last_peered":"2024-03-30T15:04:23.290105+0000","last_clean":"2024-03-30T15:04:23.290105+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:04:23.290105+0000","last_undegraded":"2024-03-30T15:04:23.290105+0000","last_fullsized":"2024-03-30T15:04:23.290105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:55.535057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:01:55.535057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"76'12","reported_seq":851,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291888+0000","last_change":"2024-03-30T15:03:19.519526+0000","last_active":"2024-03-30T15:04:23.291888+0000","last_peered":"2024-03-30T15:04:23.291888+0000","last_clean":"2024-03-30T15:04:23.291888+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:04:23.291888+0000","last_undegraded":"2024-03-30T15:04:23.291888+0000","last_fullsized":"2024-03-30T15:04:23.291888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:03:19.519440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.519440+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.770107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":823,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.305052+0000","last_change":"2024-03-30T15:02:01.090736+0000","last_active":"2024-03-30T15:04:22.305052+0000","last_peered":"2024-03-30T15:04:22.305052+0000","last_clean":"2024-03-30T15:04:22.305052+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:04:22.305052+0000","last_undegraded":"2024-03-30T15:04:22.305052+0000","last_fullsized":"2024-03-30T15:04:22.305052+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:01.090640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.090640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":723,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301482+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:04:22.301482+0000","last_peered":"2024-03-30T15:04:22.301482+0000","last_clean":"2024-03-30T15:04:22.301482+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:04:22.301482+0000","last_undegraded":"2024-03-30T15:04:22.301482+0000","last_fullsized":"2024-03-30T15:04:22.301482+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.943192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":721,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291385+0000","last_change":"2024-03-30T15:00:03.515720+0000","last_active":"2024-03-30T15:04:23.291385+0000","last_peered":"2024-03-30T15:04:23.291385+0000","last_clean":"2024-03-30T15:04:23.291385+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:04:23.291385+0000","last_undegraded":"2024-03-30T15:04:23.291385+0000","last_fullsized":"2024-03-30T15:04:23.291385+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:03.515635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:03.515635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599092+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304125+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:04:22.304125+0000","last_peered":"2024-03-30T15:04:22.304125+0000","last_clean":"2024-03-30T15:04:22.304125+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:04:22.304125+0000","last_undegraded":"2024-03-30T15:04:22.304125+0000","last_fullsized":"2024-03-30T15:04:22.304125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"76'15","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380597+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:04:23.380597+0000","last_peered":"2024-03-30T15:04:23.380597+0000","last_clean":"2024-03-30T15:04:23.380597+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:04:23.380597+0000","last_undegraded":"2024-03-30T15:04:23.380597+0000","last_fullsized":"2024-03-30T15:04:23.380597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:30.964164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":835,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303370+0000","last_change":"2024-03-30T15:02:05.004181+0000","last_active":"2024-03-30T15:04:22.303370+0000","last_peered":"2024-03-30T15:04:22.303370+0000","last_clean":"2024-03-30T15:04:22.303370+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:04:22.303370+0000","last_undegraded":"2024-03-30T15:04:22.303370+0000","last_fullsized":"2024-03-30T15:04:22.303370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:02:05.004073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:05.004073+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":811,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381507+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:04:23.381507+0000","last_peered":"2024-03-30T15:04:23.381507+0000","last_clean":"2024-03-30T15:04:23.381507+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:23.381507+0000","last_undegraded":"2024-03-30T15:04:23.381507+0000","last_fullsized":"2024-03-30T15:04:23.381507+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":728,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.296162+0000","last_change":"2024-03-30T15:02:32.380969+0000","last_active":"2024-03-30T15:04:22.296162+0000","last_peered":"2024-03-30T15:04:22.296162+0000","last_clean":"2024-03-30T15:04:22.296162+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:04:22.296162+0000","last_undegraded":"2024-03-30T15:04:22.296162+0000","last_fullsized":"2024-03-30T15:04:22.296162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.380822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.380822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307359+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"76'6","reported_seq":404,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380369+0000","last_change":"2024-03-30T15:03:52.115510+0000","last_active":"2024-03-30T15:04:23.380369+0000","last_peered":"2024-03-30T15:04:23.380369+0000","last_clean":"2024-03-30T15:04:23.380369+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:04:23.380369+0000","last_undegraded":"2024-03-30T15:04:23.380369+0000","last_fullsized":"2024-03-30T15:04:23.380369+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T15:03:52.115417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.115417+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.716598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":838,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.293777+0000","last_change":"2024-03-30T15:02:45.427403+0000","last_active":"2024-03-30T15:04:22.293777+0000","last_peered":"2024-03-30T15:04:22.293777+0000","last_clean":"2024-03-30T15:04:22.293777+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:04:22.293777+0000","last_undegraded":"2024-03-30T15:04:22.293777+0000","last_fullsized":"2024-03-30T15:04:22.293777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:02:45.427234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":815,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.383155+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:04:23.383155+0000","last_peered":"2024-03-30T15:04:23.383155+0000","last_clean":"2024-03-30T15:04:23.383155+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:04:23.383155+0000","last_undegraded":"2024-03-30T15:04:23.383155+0000","last_fullsized":"2024-03-30T15:04:23.383155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:33.325342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":811,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.305289+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:04:22.305289+0000","last_peered":"2024-03-30T15:04:22.305289+0000","last_clean":"2024-03-30T15:04:22.305289+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:04:22.305289+0000","last_undegraded":"2024-03-30T15:04:22.305289+0000","last_fullsized":"2024-03-30T15:04:22.305289+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.682878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":721,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293200+0000","last_change":"2024-03-30T15:01:11.580352+0000","last_active":"2024-03-30T15:04:23.293200+0000","last_peered":"2024-03-30T15:04:23.293200+0000","last_clean":"2024-03-30T15:04:23.293200+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:04:23.293200+0000","last_undegraded":"2024-03-30T15:04:23.293200+0000","last_fullsized":"2024-03-30T15:04:23.293200+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:11.580276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:11.580276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"76'9","reported_seq":840,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.606435+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:04:23.606435+0000","last_peered":"2024-03-30T15:04:23.606435+0000","last_clean":"2024-03-30T15:04:23.606435+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:04:23.606435+0000","last_undegraded":"2024-03-30T15:04:23.606435+0000","last_fullsized":"2024-03-30T15:04:23.606435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.283455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":831,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303718+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:04:22.303718+0000","last_peered":"2024-03-30T15:04:22.303718+0000","last_clean":"2024-03-30T15:04:22.303718+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:04:22.303718+0000","last_undegraded":"2024-03-30T15:04:22.303718+0000","last_fullsized":"2024-03-30T15:04:22.303718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:30.940258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":731,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.309478+0000","last_change":"2024-03-30T15:04:03.988041+0000","last_active":"2024-03-30T15:04:22.309478+0000","last_peered":"2024-03-30T15:04:22.309478+0000","last_clean":"2024-03-30T15:04:22.309478+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:04:22.309478+0000","last_undegraded":"2024-03-30T15:04:22.309478+0000","last_fullsized":"2024-03-30T15:04:22.309478+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.333733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":830,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.294566+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:04:23.294566+0000","last_peered":"2024-03-30T15:04:23.294566+0000","last_clean":"2024-03-30T15:04:23.294566+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:04:23.294566+0000","last_undegraded":"2024-03-30T15:04:23.294566+0000","last_fullsized":"2024-03-30T15:04:23.294566+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.180120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605007+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:04:23.605007+0000","last_peered":"2024-03-30T15:04:23.605007+0000","last_clean":"2024-03-30T15:04:23.605007+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:04:23.605007+0000","last_undegraded":"2024-03-30T15:04:23.605007+0000","last_fullsized":"2024-03-30T15:04:23.605007+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"76'9","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382402+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:04:23.382402+0000","last_peered":"2024-03-30T15:04:23.382402+0000","last_clean":"2024-03-30T15:04:23.382402+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:04:23.382402+0000","last_undegraded":"2024-03-30T15:04:23.382402+0000","last_fullsized":"2024-03-30T15:04:23.382402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.204528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603625+0000","last_change":"2024-03-30T14:59:50.390565+0000","last_active":"2024-03-30T15:04:23.603625+0000","last_peered":"2024-03-30T15:04:23.603625+0000","last_clean":"2024-03-30T15:04:23.603625+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:04:23.603625+0000","last_undegraded":"2024-03-30T15:04:23.603625+0000","last_fullsized":"2024-03-30T15:04:23.603625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:50.390475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:50.390475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":826,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302760+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:04:22.302760+0000","last_peered":"2024-03-30T15:04:22.302760+0000","last_clean":"2024-03-30T15:04:22.302760+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:04:22.302760+0000","last_undegraded":"2024-03-30T15:04:22.302760+0000","last_fullsized":"2024-03-30T15:04:22.302760+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:35.585322+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":804,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293815+0000","last_change":"2024-03-30T14:59:55.336534+0000","last_active":"2024-03-30T15:04:23.293815+0000","last_peered":"2024-03-30T15:04:23.293815+0000","last_clean":"2024-03-30T15:04:23.293815+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:04:23.293815+0000","last_undegraded":"2024-03-30T15:04:23.293815+0000","last_fullsized":"2024-03-30T15:04:23.293815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:55.336452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T14:59:55.336452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604104+0000","last_change":"2024-03-30T15:01:52.471709+0000","last_active":"2024-03-30T15:04:23.604104+0000","last_peered":"2024-03-30T15:04:23.604104+0000","last_clean":"2024-03-30T15:04:23.604104+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:04:23.604104+0000","last_undegraded":"2024-03-30T15:04:23.604104+0000","last_fullsized":"2024-03-30T15:04:23.604104+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.471603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.471603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602640+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'10","reported_seq":757,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303533+0000","last_change":"2024-03-30T15:01:18.430986+0000","last_active":"2024-03-30T15:04:22.303533+0000","last_peered":"2024-03-30T15:04:22.303533+0000","last_clean":"2024-03-30T15:04:22.303533+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:04:22.303533+0000","last_undegraded":"2024-03-30T15:04:22.303533+0000","last_fullsized":"2024-03-30T15:04:22.303533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:01:18.430887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:18.430887+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":823,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301078+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:04:22.301078+0000","last_peered":"2024-03-30T15:04:22.301078+0000","last_clean":"2024-03-30T15:04:22.301078+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:04:22.301078+0000","last_undegraded":"2024-03-30T15:04:22.301078+0000","last_fullsized":"2024-03-30T15:04:22.301078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:37.842104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":944,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382050+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:04:23.382050+0000","last_peered":"2024-03-30T15:04:23.382050+0000","last_clean":"2024-03-30T15:04:23.382050+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:04:23.382050+0000","last_undegraded":"2024-03-30T15:04:23.382050+0000","last_fullsized":"2024-03-30T15:04:23.382050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.083796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":818,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605633+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:04:23.605633+0000","last_peered":"2024-03-30T15:04:23.605633+0000","last_clean":"2024-03-30T15:04:23.605633+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:04:23.605633+0000","last_undegraded":"2024-03-30T15:04:23.605633+0000","last_fullsized":"2024-03-30T15:04:23.605633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.685331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292038+0000","last_change":"2024-03-30T15:00:32.662003+0000","last_active":"2024-03-30T15:04:23.292038+0000","last_peered":"2024-03-30T15:04:23.292038+0000","last_clean":"2024-03-30T15:04:23.292038+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:04:23.292038+0000","last_undegraded":"2024-03-30T15:04:23.292038+0000","last_fullsized":"2024-03-30T15:04:23.292038+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.661889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.661889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"196'16","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292073+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:04:23.292073+0000","last_peered":"2024-03-30T15:04:23.292073+0000","last_clean":"2024-03-30T15:04:23.292073+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:04:23.292073+0000","last_undegraded":"2024-03-30T15:04:23.292073+0000","last_fullsized":"2024-03-30T15:04:23.292073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.342982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":824,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292811+0000","last_change":"2024-03-30T15:01:21.453162+0000","last_active":"2024-03-30T15:04:23.292811+0000","last_peered":"2024-03-30T15:04:23.292811+0000","last_clean":"2024-03-30T15:04:23.292811+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:04:23.292811+0000","last_undegraded":"2024-03-30T15:04:23.292811+0000","last_fullsized":"2024-03-30T15:04:23.292811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:21.453084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:21.453084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":823,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304057+0000","last_change":"2024-03-30T15:02:03.477727+0000","last_active":"2024-03-30T15:04:22.304057+0000","last_peered":"2024-03-30T15:04:22.304057+0000","last_clean":"2024-03-30T15:04:22.304057+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:04:22.304057+0000","last_undegraded":"2024-03-30T15:04:22.304057+0000","last_fullsized":"2024-03-30T15:04:22.304057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.477653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.477653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603950+0000","last_change":"2024-03-30T15:00:02.518872+0000","last_active":"2024-03-30T15:04:23.603950+0000","last_peered":"2024-03-30T15:04:23.603950+0000","last_clean":"2024-03-30T15:04:23.603950+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:04:23.603950+0000","last_undegraded":"2024-03-30T15:04:23.603950+0000","last_fullsized":"2024-03-30T15:04:23.603950+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:00:02.518782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.292790+0000","last_change":"2024-03-30T15:00:35.626574+0000","last_active":"2024-03-30T15:04:23.292790+0000","last_peered":"2024-03-30T15:04:23.292790+0000","last_clean":"2024-03-30T15:04:23.292790+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:04:23.292790+0000","last_undegraded":"2024-03-30T15:04:23.292790+0000","last_fullsized":"2024-03-30T15:04:23.292790+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:35.626482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:35.626482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"76'9","reported_seq":835,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304166+0000","last_change":"2024-03-30T15:02:00.404994+0000","last_active":"2024-03-30T15:04:22.304166+0000","last_peered":"2024-03-30T15:04:22.304166+0000","last_clean":"2024-03-30T15:04:22.304166+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:04:22.304166+0000","last_undegraded":"2024-03-30T15:04:22.304166+0000","last_fullsized":"2024-03-30T15:04:22.304166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:02:00.404907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.404907+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":838,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.305393+0000","last_change":"2024-03-30T15:02:02.044673+0000","last_active":"2024-03-30T15:04:22.305393+0000","last_peered":"2024-03-30T15:04:22.305393+0000","last_clean":"2024-03-30T15:04:22.305393+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:04:22.305393+0000","last_undegraded":"2024-03-30T15:04:22.305393+0000","last_fullsized":"2024-03-30T15:04:22.305393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:02:02.044556+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:02:02.044556+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":728,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605313+0000","last_change":"2024-03-30T15:03:59.603193+0000","last_active":"2024-03-30T15:04:23.605313+0000","last_peered":"2024-03-30T15:04:23.605313+0000","last_clean":"2024-03-30T15:04:23.605313+0000","last_became_active":"2024-03-30T14:57:56.241939+0000","last_became_peered":"2024-03-30T14:57:56.241939+0000","last_unstale":"2024-03-30T15:04:23.605313+0000","last_undegraded":"2024-03-30T15:04:23.605313+0000","last_fullsized":"2024-03-30T15:04:23.605313+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.946216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603877+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:04:23.603877+0000","last_peered":"2024-03-30T15:04:23.603877+0000","last_clean":"2024-03-30T15:04:23.603877+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:04:23.603877+0000","last_undegraded":"2024-03-30T15:04:23.603877+0000","last_fullsized":"2024-03-30T15:04:23.603877+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603826+0000","last_change":"2024-03-30T15:00:46.540190+0000","last_active":"2024-03-30T15:04:23.603826+0000","last_peered":"2024-03-30T15:04:23.603826+0000","last_clean":"2024-03-30T15:04:23.603826+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:04:23.603826+0000","last_undegraded":"2024-03-30T15:04:23.603826+0000","last_fullsized":"2024-03-30T15:04:23.603826+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:00:46.540109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"76'11","reported_seq":761,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303960+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:04:22.303960+0000","last_peered":"2024-03-30T15:04:22.303960+0000","last_clean":"2024-03-30T15:04:22.303960+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:04:22.303960+0000","last_undegraded":"2024-03-30T15:04:22.303960+0000","last_fullsized":"2024-03-30T15:04:22.303960+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.383256+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:04:23.383256+0000","last_peered":"2024-03-30T15:04:23.383256+0000","last_clean":"2024-03-30T15:04:23.383256+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:04:23.383256+0000","last_undegraded":"2024-03-30T15:04:23.383256+0000","last_fullsized":"2024-03-30T15:04:23.383256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":815,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.303502+0000","last_change":"2024-03-30T15:02:41.903716+0000","last_active":"2024-03-30T15:04:22.303502+0000","last_peered":"2024-03-30T15:04:22.303502+0000","last_clean":"2024-03-30T15:04:22.303502+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:04:22.303502+0000","last_undegraded":"2024-03-30T15:04:22.303502+0000","last_fullsized":"2024-03-30T15:04:22.303502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.903609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.903609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604865+0000","last_change":"2024-03-30T15:00:00.432700+0000","last_active":"2024-03-30T15:04:23.604865+0000","last_peered":"2024-03-30T15:04:23.604865+0000","last_clean":"2024-03-30T15:04:23.604865+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:04:23.604865+0000","last_undegraded":"2024-03-30T15:04:23.604865+0000","last_fullsized":"2024-03-30T15:04:23.604865+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:00.432607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:00.432607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602266+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":728,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.289481+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:04:22.289481+0000","last_peered":"2024-03-30T15:04:22.289481+0000","last_clean":"2024-03-30T15:04:22.289481+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:04:22.289481+0000","last_undegraded":"2024-03-30T15:04:22.289481+0000","last_fullsized":"2024-03-30T15:04:22.289481+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.190282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"76'4","reported_seq":393,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.377201+0000","last_change":"2024-03-30T15:04:01.043987+0000","last_active":"2024-03-30T15:04:23.377201+0000","last_peered":"2024-03-30T15:04:23.377201+0000","last_clean":"2024-03-30T15:04:23.377201+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:04:23.377201+0000","last_undegraded":"2024-03-30T15:04:23.377201+0000","last_fullsized":"2024-03-30T15:04:23.377201+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:04:01.043885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.043885+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.133940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":828,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.606041+0000","last_change":"2024-03-30T15:00:01.470154+0000","last_active":"2024-03-30T15:04:23.606041+0000","last_peered":"2024-03-30T15:04:23.606041+0000","last_clean":"2024-03-30T15:04:23.606041+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:04:23.606041+0000","last_undegraded":"2024-03-30T15:04:23.606041+0000","last_fullsized":"2024-03-30T15:04:23.606041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:00:01.470075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":830,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.295424+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:04:22.295424+0000","last_peered":"2024-03-30T15:04:22.295424+0000","last_clean":"2024-03-30T15:04:22.295424+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:04:22.295424+0000","last_undegraded":"2024-03-30T15:04:22.295424+0000","last_fullsized":"2024-03-30T15:04:22.295424+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":811,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.300730+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:04:22.300730+0000","last_peered":"2024-03-30T15:04:22.300730+0000","last_clean":"2024-03-30T15:04:22.300730+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:04:22.300730+0000","last_undegraded":"2024-03-30T15:04:22.300730+0000","last_fullsized":"2024-03-30T15:04:22.300730+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":727,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298159+0000","last_change":"2024-03-30T15:02:35.380537+0000","last_active":"2024-03-30T15:04:22.298159+0000","last_peered":"2024-03-30T15:04:22.298159+0000","last_clean":"2024-03-30T15:04:22.298159+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:04:22.298159+0000","last_undegraded":"2024-03-30T15:04:22.298159+0000","last_fullsized":"2024-03-30T15:04:22.298159+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.380454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.380454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"93'16","reported_seq":850,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298297+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:04:22.298297+0000","last_peered":"2024-03-30T15:04:22.298297+0000","last_clean":"2024-03-30T15:04:22.298297+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:04:22.298297+0000","last_undegraded":"2024-03-30T15:04:22.298297+0000","last_fullsized":"2024-03-30T15:04:22.298297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":729,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.603993+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:04:23.603993+0000","last_peered":"2024-03-30T15:04:23.603993+0000","last_clean":"2024-03-30T15:04:23.603993+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:04:23.603993+0000","last_undegraded":"2024-03-30T15:04:23.603993+0000","last_fullsized":"2024-03-30T15:04:23.603993+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.782926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":820,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.606535+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:04:23.606535+0000","last_peered":"2024-03-30T15:04:23.606535+0000","last_clean":"2024-03-30T15:04:23.606535+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:04:23.606535+0000","last_undegraded":"2024-03-30T15:04:23.606535+0000","last_fullsized":"2024-03-30T15:04:23.606535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:43.075639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":818,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304713+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:04:22.304713+0000","last_peered":"2024-03-30T15:04:22.304713+0000","last_clean":"2024-03-30T15:04:22.304713+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:04:22.304713+0000","last_undegraded":"2024-03-30T15:04:22.304713+0000","last_fullsized":"2024-03-30T15:04:22.304713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":735,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.296616+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:04:22.296616+0000","last_peered":"2024-03-30T15:04:22.296616+0000","last_clean":"2024-03-30T15:04:22.296616+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:04:22.296616+0000","last_undegraded":"2024-03-30T15:04:22.296616+0000","last_fullsized":"2024-03-30T15:04:22.296616+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:29.813607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"186'15","reported_seq":844,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.606001+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:04:23.606001+0000","last_peered":"2024-03-30T15:04:23.606001+0000","last_clean":"2024-03-30T15:04:23.606001+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:04:23.606001+0000","last_undegraded":"2024-03-30T15:04:23.606001+0000","last_fullsized":"2024-03-30T15:04:23.606001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:38.106835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":824,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301124+0000","last_change":"2024-03-30T15:01:47.383230+0000","last_active":"2024-03-30T15:04:22.301124+0000","last_peered":"2024-03-30T15:04:22.301124+0000","last_clean":"2024-03-30T15:04:22.301124+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:04:22.301124+0000","last_undegraded":"2024-03-30T15:04:22.301124+0000","last_fullsized":"2024-03-30T15:04:22.301124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:01:47.383148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.383148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":959,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298965+0000","last_change":"2024-03-30T15:02:51.451850+0000","last_active":"2024-03-30T15:04:22.298965+0000","last_peered":"2024-03-30T15:04:22.298965+0000","last_clean":"2024-03-30T15:04:22.298965+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:04:22.298965+0000","last_undegraded":"2024-03-30T15:04:22.298965+0000","last_fullsized":"2024-03-30T15:04:22.298965+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:51.451690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:51.451690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:27.345505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":820,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.302737+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:04:22.302737+0000","last_peered":"2024-03-30T15:04:22.302737+0000","last_clean":"2024-03-30T15:04:22.302737+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:04:22.302737+0000","last_undegraded":"2024-03-30T15:04:22.302737+0000","last_fullsized":"2024-03-30T15:04:22.302737+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:07.241016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":735,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.294313+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:04:22.294313+0000","last_peered":"2024-03-30T15:04:22.294313+0000","last_clean":"2024-03-30T15:04:22.294313+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:04:22.294313+0000","last_undegraded":"2024-03-30T15:04:22.294313+0000","last_fullsized":"2024-03-30T15:04:22.294313+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:17.458537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"76'12","reported_seq":765,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304386+0000","last_change":"2024-03-30T15:01:20.341161+0000","last_active":"2024-03-30T15:04:22.304386+0000","last_peered":"2024-03-30T15:04:22.304386+0000","last_clean":"2024-03-30T15:04:22.304386+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:04:22.304386+0000","last_undegraded":"2024-03-30T15:04:22.304386+0000","last_fullsized":"2024-03-30T15:04:22.304386+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:01:20.341036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:01:20.341036+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":823,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293531+0000","last_change":"2024-03-30T15:03:21.522112+0000","last_active":"2024-03-30T15:04:23.293531+0000","last_peered":"2024-03-30T15:04:23.293531+0000","last_clean":"2024-03-30T15:04:23.293531+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:04:23.293531+0000","last_undegraded":"2024-03-30T15:04:23.293531+0000","last_fullsized":"2024-03-30T15:04:23.293531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.522038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.522038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:42.587391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":815,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293561+0000","last_change":"2024-03-30T15:03:23.551230+0000","last_active":"2024-03-30T15:04:23.293561+0000","last_peered":"2024-03-30T15:04:23.293561+0000","last_clean":"2024-03-30T15:04:23.293561+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:04:23.293561+0000","last_undegraded":"2024-03-30T15:04:23.293561+0000","last_fullsized":"2024-03-30T15:04:23.293561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:23.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.577963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":732,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.298639+0000","last_change":"2024-03-30T15:02:49.425842+0000","last_active":"2024-03-30T15:04:22.298639+0000","last_peered":"2024-03-30T15:04:22.298639+0000","last_clean":"2024-03-30T15:04:22.298639+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:04:22.298639+0000","last_undegraded":"2024-03-30T15:04:22.298639+0000","last_fullsized":"2024-03-30T15:04:22.298639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:49.425762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.425762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:22.307411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382591+0000","last_change":"2024-03-30T15:00:34.064166+0000","last_active":"2024-03-30T15:04:23.382591+0000","last_peered":"2024-03-30T15:04:23.382591+0000","last_clean":"2024-03-30T15:04:23.382591+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:04:23.382591+0000","last_undegraded":"2024-03-30T15:04:23.382591+0000","last_fullsized":"2024-03-30T15:04:23.382591+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:34.064067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:34.064067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"153'22","reported_seq":869,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291913+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:04:23.291913+0000","last_peered":"2024-03-30T15:04:23.291913+0000","last_clean":"2024-03-30T15:04:23.291913+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:04:23.291913+0000","last_undegraded":"2024-03-30T15:04:23.291913+0000","last_fullsized":"2024-03-30T15:04:23.291913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:50.076544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":827,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605475+0000","last_change":"2024-03-30T14:59:52.405122+0000","last_active":"2024-03-30T15:04:23.605475+0000","last_peered":"2024-03-30T15:04:23.605475+0000","last_clean":"2024-03-30T15:04:23.605475+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:04:23.605475+0000","last_undegraded":"2024-03-30T15:04:23.605475+0000","last_fullsized":"2024-03-30T15:04:23.605475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:52.405049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T14:59:52.405049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":822,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.304561+0000","last_change":"2024-03-30T15:04:07.011782+0000","last_active":"2024-03-30T15:04:22.304561+0000","last_peered":"2024-03-30T15:04:22.304561+0000","last_clean":"2024-03-30T15:04:22.304561+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:04:22.304561+0000","last_undegraded":"2024-03-30T15:04:22.304561+0000","last_fullsized":"2024-03-30T15:04:22.304561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.011661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.011661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.278098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":812,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293537+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:04:23.293537+0000","last_peered":"2024-03-30T15:04:23.293537+0000","last_clean":"2024-03-30T15:04:23.293537+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:04:23.293537+0000","last_undegraded":"2024-03-30T15:04:23.293537+0000","last_fullsized":"2024-03-30T15:04:23.293537+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:57.177473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":728,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301814+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:04:22.301814+0000","last_peered":"2024-03-30T15:04:22.301814+0000","last_clean":"2024-03-30T15:04:22.301814+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:04:22.301814+0000","last_undegraded":"2024-03-30T15:04:22.301814+0000","last_fullsized":"2024-03-30T15:04:22.301814+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"76'5","reported_seq":853,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291964+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:04:23.291964+0000","last_peered":"2024-03-30T15:04:23.291964+0000","last_clean":"2024-03-30T15:04:23.291964+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:04:23.291964+0000","last_undegraded":"2024-03-30T15:04:23.291964+0000","last_fullsized":"2024-03-30T15:04:23.291964+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.236100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":822,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382222+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:04:23.382222+0000","last_peered":"2024-03-30T15:04:23.382222+0000","last_clean":"2024-03-30T15:04:23.382222+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:04:23.382222+0000","last_undegraded":"2024-03-30T15:04:23.382222+0000","last_fullsized":"2024-03-30T15:04:23.382222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":814,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.300631+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:04:22.300631+0000","last_peered":"2024-03-30T15:04:22.300631+0000","last_clean":"2024-03-30T15:04:22.300631+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:04:22.300631+0000","last_undegraded":"2024-03-30T15:04:22.300631+0000","last_fullsized":"2024-03-30T15:04:22.300631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.130630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":810,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380792+0000","last_change":"2024-03-30T14:59:57.093620+0000","last_active":"2024-03-30T15:04:23.380792+0000","last_peered":"2024-03-30T15:04:23.380792+0000","last_clean":"2024-03-30T15:04:23.380792+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:04:23.380792+0000","last_undegraded":"2024-03-30T15:04:23.380792+0000","last_fullsized":"2024-03-30T15:04:23.380792+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.093513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":722,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.301893+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:04:22.301893+0000","last_peered":"2024-03-30T15:04:22.301893+0000","last_clean":"2024-03-30T15:04:22.301893+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:04:22.301893+0000","last_undegraded":"2024-03-30T15:04:22.301893+0000","last_fullsized":"2024-03-30T15:04:22.301893+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"76'9","reported_seq":835,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293565+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:04:23.293565+0000","last_peered":"2024-03-30T15:04:23.293565+0000","last_clean":"2024-03-30T15:04:23.293565+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:04:23.293565+0000","last_undegraded":"2024-03-30T15:04:23.293565+0000","last_fullsized":"2024-03-30T15:04:23.293565+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.139552+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":822,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.382029+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:04:23.382029+0000","last_peered":"2024-03-30T15:04:23.382029+0000","last_clean":"2024-03-30T15:04:23.382029+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:04:23.382029+0000","last_undegraded":"2024-03-30T15:04:23.382029+0000","last_fullsized":"2024-03-30T15:04:23.382029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:51.403630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":814,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.381235+0000","last_change":"2024-03-30T15:02:44.998020+0000","last_active":"2024-03-30T15:04:23.381235+0000","last_peered":"2024-03-30T15:04:23.381235+0000","last_clean":"2024-03-30T15:04:23.381235+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:04:23.381235+0000","last_undegraded":"2024-03-30T15:04:23.381235+0000","last_fullsized":"2024-03-30T15:04:23.381235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.997898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604119+0000","last_change":"2024-03-30T15:00:32.473129+0000","last_active":"2024-03-30T15:04:23.604119+0000","last_peered":"2024-03-30T15:04:23.604119+0000","last_clean":"2024-03-30T15:04:23.604119+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:04:23.604119+0000","last_undegraded":"2024-03-30T15:04:23.604119+0000","last_fullsized":"2024-03-30T15:04:23.604119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:32.472977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:00:32.472977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":720,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.380678+0000","last_change":"2024-03-30T15:01:16.348182+0000","last_active":"2024-03-30T15:04:23.380678+0000","last_peered":"2024-03-30T15:04:23.380678+0000","last_clean":"2024-03-30T15:04:23.380678+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:04:23.380678+0000","last_undegraded":"2024-03-30T15:04:23.380678+0000","last_fullsized":"2024-03-30T15:04:23.380678+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:16.348072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:16.348072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"93'22","reported_seq":819,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.290831+0000","last_change":"2024-03-30T15:02:33.875911+0000","last_active":"2024-03-30T15:04:23.290831+0000","last_peered":"2024-03-30T15:04:23.290831+0000","last_clean":"2024-03-30T15:04:23.290831+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:04:23.290831+0000","last_undegraded":"2024-03-30T15:04:23.290831+0000","last_fullsized":"2024-03-30T15:04:23.290831+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T15:02:33.875778+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.875778+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:26.053172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":733,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.296426+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:04:22.296426+0000","last_peered":"2024-03-30T15:04:22.296426+0000","last_clean":"2024-03-30T15:04:22.296426+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:04:22.296426+0000","last_undegraded":"2024-03-30T15:04:22.296426+0000","last_fullsized":"2024-03-30T15:04:22.296426+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:32.273285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":816,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291335+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:04:23.291335+0000","last_peered":"2024-03-30T15:04:23.291335+0000","last_clean":"2024-03-30T15:04:23.291335+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:04:23.291335+0000","last_undegraded":"2024-03-30T15:04:23.291335+0000","last_fullsized":"2024-03-30T15:04:23.291335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.751185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":379,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.378148+0000","last_change":"2024-03-30T15:03:16.163490+0000","last_active":"2024-03-30T15:04:23.378148+0000","last_peered":"2024-03-30T15:04:23.378148+0000","last_clean":"2024-03-30T15:04:23.378148+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:04:23.378148+0000","last_undegraded":"2024-03-30T15:04:23.378148+0000","last_fullsized":"2024-03-30T15:04:23.378148+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.163376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.163376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:36.277238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":725,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.604266+0000","last_change":"2024-03-30T15:01:17.584059+0000","last_active":"2024-03-30T15:04:23.604266+0000","last_peered":"2024-03-30T15:04:23.604266+0000","last_clean":"2024-03-30T15:04:23.604266+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:04:23.604266+0000","last_undegraded":"2024-03-30T15:04:23.604266+0000","last_fullsized":"2024-03-30T15:04:23.604266+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:17.583984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:17.583984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:28.602541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"76'9","reported_seq":852,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.291558+0000","last_change":"2024-03-30T15:02:19.657930+0000","last_active":"2024-03-30T15:04:23.291558+0000","last_peered":"2024-03-30T15:04:23.291558+0000","last_clean":"2024-03-30T15:04:23.291558+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:04:23.291558+0000","last_undegraded":"2024-03-30T15:04:23.291558+0000","last_fullsized":"2024-03-30T15:04:23.291558+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T14:59:57.415259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T14:59:57.415259+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:25.599125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":836,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.605922+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:04:23.605922+0000","last_peered":"2024-03-30T15:04:23.605922+0000","last_clean":"2024-03-30T15:04:23.605922+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:04:23.605922+0000","last_undegraded":"2024-03-30T15:04:23.605922+0000","last_fullsized":"2024-03-30T15:04:23.605922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.406410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":814,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-03-30T15:04:22.293880+0000","last_change":"2024-03-30T15:01:53.452032+0000","last_active":"2024-03-30T15:04:22.293880+0000","last_peered":"2024-03-30T15:04:22.293880+0000","last_clean":"2024-03-30T15:04:22.293880+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:04:22.293880+0000","last_undegraded":"2024-03-30T15:04:22.293880+0000","last_fullsized":"2024-03-30T15:04:22.293880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.451927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.451927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:24.171430+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":811,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-30T15:04:23.293340+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:04:23.293340+0000","last_peered":"2024-03-30T15:04:23.293340+0000","last_clean":"2024-03-30T15:04:23.293340+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:04:23.293340+0000","last_undegraded":"2024-03-30T15:04:23.293340+0000","last_fullsized":"2024-03-30T15:04:23.293340+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:40.511390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":300,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195822,"num_objects":164,"num_object_clones":4,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":111,"num_read_kb":10,"num_write":609,"num_write_kb":12698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26542080,"data_stored":24587499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1916,"ondisk_log_size":1916,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":578,"num_read_kb":573,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":461,"ondisk_log_size":461,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005178,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88120,"kb_used_data":12492,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93657032,"statfs":{"total":95995035648,"available":95904800768,"internally_reserved":0,"allocated":12791808,"data_stored":9216003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299519,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75552,"kb_used_data":12416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669600,"statfs":{"total":95995035648,"available":95917670400,"internally_reserved":0,"allocated":12713984,"data_stored":9213536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83568,"kb_used_data":7964,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93661584,"statfs":{"total":95995035648,"available":95909462016,"internally_reserved":0,"allocated":8155136,"data_stored":4624376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738525,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66080,"kb_used_data":12660,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93679072,"statfs":{"total":95995035648,"available":95927369728,"internally_reserved":0,"allocated":12963840,"data_stored":9409565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411475,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79136,"kb_used_data":15988,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93666016,"statfs":{"total":95995035648,"available":95914000384,"internally_reserved":0,"allocated":16371712,"data_stored":12817912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117135,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75268,"kb_used_data":12144,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669884,"statfs":{"total":95995035648,"available":95917961216,"internally_reserved":0,"allocated":12435456,"data_stored":8819799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953609,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71160,"kb_used_data":8036,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673992,"statfs":{"total":95995035648,"available":95922167808,"internally_reserved":0,"allocated":8228864,"data_stored":4633049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659268,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71168,"kb_used_data":8020,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673984,"statfs":{"total":95995035648,"available":95922159616,"internally_reserved":0,"allocated":8212480,"data_stored":4624039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":4194827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":8194889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4194929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4000475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4247552,"data_stored":4000604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:04:28.590 INFO:tasks.thrashosds.thrasher:Setting 5.13 to [2, 5, 7, 1, 3, 4] 2024-03-30T15:04:28.590 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.13', '2', '5', '7', '1', '3', '4'] 2024-03-30T15:04:28.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap-items 5.13 2 5 7 1 3 4 2024-03-30T15:04:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:28.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:28 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:28 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:28 smithi134 ceph-mon[33714]: pgmap v660: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:04:28.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:28.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:28.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:28.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:28.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:29.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:29.102 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:29.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:29.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:29.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:29.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:29.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:29.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:29.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:29 smithi028 ceph-mon[29876]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T15:04:29.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:29 smithi028 ceph-mon[41614]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T15:04:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:29 smithi134 ceph-mon[33714]: osdmap e469: 8 total, 8 up, 8 in 2024-03-30T15:04:29.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:29.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:29.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:29.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:29.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:29.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:29.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:30.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:30.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:30.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:30.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:30.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:30.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:30 smithi028 ceph-mon[29876]: pgmap v662: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-30T15:04:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:30 smithi028 ceph-mon[41614]: pgmap v662: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-30T15:04:30.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:30.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:30.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:30.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:30 smithi134 ceph-mon[33714]: pgmap v662: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-03-30T15:04:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:30.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:30.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:30.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:30.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:30.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.016+0000 7f29c3031700 1 -- 172.21.15.28:0/612227111 >> v1:172.21.15.28:6789/0 conn(0x7f29bc101ba0 legacy=0x7f29bc101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- 172.21.15.28:0/612227111 shutdown_connections 2024-03-30T15:04:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- 172.21.15.28:0/612227111 >> 172.21.15.28:0/612227111 conn(0x7f29bc0761c0 msgr2=0x7f29bc0765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- 172.21.15.28:0/612227111 shutdown_connections 2024-03-30T15:04:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- 172.21.15.28:0/612227111 wait complete. 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 librados: starting msgr at 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 librados: starting objecter 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 Processor -- start 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- start start 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 librados: setting wanted keys 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 librados: calling monclient init 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29bc1a9370 con 0x7f29bc101ba0 2024-03-30T15:04:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29bc1aa590 con 0x7f29bc109e20 2024-03-30T15:04:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c3031700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29bc1ab7d0 con 0x7f29bc10d9c0 2024-03-30T15:04:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c0dcd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29bc101ba0 0x7f29bc101230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42820/0 (socket says 172.21.15.28:42820) 2024-03-30T15:04:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.017+0000 7f29c0dcd700 1 -- 172.21.15.28:0/3571783868 learned_addr learned my addr 172.21.15.28:0/3571783868 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter ms_handle_connect 0x7f29bc101ba0 2024-03-30T15:04:31.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:31.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter ms_handle_connect 0x7f29bc10d9c0 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121356137 0 0) 0x7f29bc1a9370 con 0x7f29bc101ba0 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a80039c0 con 0x7f29bc101ba0 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter ms_handle_connect 0x7f29bc109e20 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561666545 0 0) 0x7f29bc1ab7d0 con 0x7f29bc10d9c0 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29bc1a9370 con 0x7f29bc10d9c0 2024-03-30T15:04:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214645434 0 0) 0x7f29bc1aa590 con 0x7f29bc109e20 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29bc1ab7d0 con 0x7f29bc109e20 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3691080213 0 0) 0x7f29a80039c0 con 0x7f29bc101ba0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29bc1aa590 con 0x7f29bc101ba0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29a4004260 con 0x7f29bc101ba0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3645464372 0 0) 0x7f29bc1a9370 con 0x7f29bc10d9c0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a80039c0 con 0x7f29bc10d9c0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29b4001c30 con 0x7f29bc10d9c0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1916265822 0 0) 0x7f29bc1ab7d0 con 0x7f29bc109e20 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29bc1a9370 con 0x7f29bc109e20 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29ac001a30 con 0x7f29bc109e20 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2586675091 0 0) 0x7f29bc1aa590 con 0x7f29bc101ba0 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6790/0 conn(0x7f29bc10d9c0 legacy=0x7f29bc1a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.134:6789/0 conn(0x7f29bc109e20 legacy=0x7f29bc1a4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc1aca10 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc1aba00 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.018+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f29a4001970 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.019+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f29a4004fe0 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.019+0000 7f29c3031700 10 client.24921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.019+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29bc1abec0 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.019+0000 7f29c3031700 1 librados: init done 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.019+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f29bc06a2e0 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f29a4005200 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(469..469 src has 53..469) ==== 8523+0+0 (unknown 2948968964 0 0) 0x7f29a4089790 con 0x7f29bc101ba0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 osd_map(469..469 src has 53..469) 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 3 client.24921.objecter handle_osd_map got epochs [469,469] > 0 2024-03-30T15:04:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 3 client.24921.objecter handle_osd_map decoding full epoch 469 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 20 client.24921.objecter dump_active .. 0 homeless 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.020+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v469) ==== 54+0+16 (unknown 3947041598 0 2420000192) 0x7f29a408bb70 con 0x7f29bc101ba0 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29a8070550 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 10 client.24921.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 1 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 1 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 0 1 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 4 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 10 client.24921.objecter _send_command 1 2024-03-30T15:04:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29bc10f450 con 0x7f29bc1aac70 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.021+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1aac70 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.022+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc10f450 con 0x7f29bc1aac70 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.022+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 1: 0 ) 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.022+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 1 = system:0 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.022+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 4 2024-03-30T15:04:31.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.022+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 0 1 2024-03-30T15:04:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 10 client.24921.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 2 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 2 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc113df0 osd=0 3 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 0 2 2024-03-30T15:04:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 4 2024-03-30T15:04:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 10 client.24921.objecter _send_command 2 2024-03-30T15:04:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.039+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc068b30 con 0x7f29bc1aac70 2024-03-30T15:04:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.040+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc068b30 con 0x7f29bc1aac70 2024-03-30T15:04:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.040+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.040+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.040+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 4 2024-03-30T15:04:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.040+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 0 2 2024-03-30T15:04:31.042 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 10 client.24921.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 1 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 4 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 10 client.24921.objecter _send_command 3 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1ac7c0 con 0x7f29bc062e50 2024-03-30T15:04:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc062e50 2024-03-30T15:04:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29a405a280 con 0x7f29bc062e50 2024-03-30T15:04:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 3: 0 ) 2024-03-30T15:04:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 3 = system:0 2024-03-30T15:04:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 4 2024-03-30T15:04:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.041+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 1 3 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 10 client.24921.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 4 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 4 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1ac650 osd=1 3 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 1 4 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 4 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 10 client.24921.objecter _send_command 4 2024-03-30T15:04:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.058+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc1095e0 con 0x7f29bc062e50 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc1095e0 con 0x7f29bc062e50 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 4 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 1 4 2024-03-30T15:04:31.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:04:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 10 client.24921.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 5 2024-03-30T15:04:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 5 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 2 5 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 4 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 10 client.24921.objecter _send_command 5 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1b4e50 con 0x7f29bc1b1340 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.060+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1b1340 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.061+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc1b4e50 con 0x7f29bc1b1340 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.061+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 5: 0 ) 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.061+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 5 = system:0 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.061+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 4 2024-03-30T15:04:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.061+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 2 5 2024-03-30T15:04:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 10 client.24921.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 6 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 6 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1b0cc0 osd=2 3 2024-03-30T15:04:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 2 6 2024-03-30T15:04:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 4 2024-03-30T15:04:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 10 client.24921.objecter _send_command 6 2024-03-30T15:04:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.077+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc062930 con 0x7f29bc1b1340 2024-03-30T15:04:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.079+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc062930 con 0x7f29bc1b1340 2024-03-30T15:04:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.079+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.079+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.079+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 4 2024-03-30T15:04:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.079+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 2 6 2024-03-30T15:04:31.083 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:04:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 10 client.24921.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 7 2024-03-30T15:04:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 7 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 3 7 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 4 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 10 client.24921.objecter _send_command 7 2024-03-30T15:04:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1b9c90 con 0x7f29bc1b6180 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1b6180 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc1b9c90 con 0x7f29bc1b6180 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 7: 0 ) 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 7 = system:0 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 4 2024-03-30T15:04:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.080+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 3 7 2024-03-30T15:04:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 10 client.24921.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 8 2024-03-30T15:04:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 8 2024-03-30T15:04:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02d180 osd=3 3 2024-03-30T15:04:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 3 8 2024-03-30T15:04:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 4 2024-03-30T15:04:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 10 client.24921.objecter _send_command 8 2024-03-30T15:04:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.097+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc02cc00 con 0x7f29bc1b6180 2024-03-30T15:04:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc02cc00 con 0x7f29bc1b6180 2024-03-30T15:04:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 4 2024-03-30T15:04:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 3 8 2024-03-30T15:04:31.102 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:04:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 10 client.24921.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 9 2024-03-30T15:04:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 9 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 4 9 2024-03-30T15:04:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 4 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 10 client.24921.objecter _send_command 9 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.099+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f29bc02cb90 con 0x7f29bc1b5ce0 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.100+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1b5ce0 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.100+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc02cb90 con 0x7f29bc1b5ce0 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.100+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 9: 0 ) 2024-03-30T15:04:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.101+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 9 = system:0 2024-03-30T15:04:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.101+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 4 2024-03-30T15:04:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.101+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 4 9 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 10 client.24921.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 10 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 10 2024-03-30T15:04:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02c9b0 osd=4 3 2024-03-30T15:04:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 4 10 2024-03-30T15:04:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 4 2024-03-30T15:04:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 10 client.24921.objecter _send_command 10 2024-03-30T15:04:31.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.117+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc1be540 con 0x7f29bc1b5ce0 2024-03-30T15:04:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc1be540 con 0x7f29bc1b5ce0 2024-03-30T15:04:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 4 2024-03-30T15:04:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 4 10 2024-03-30T15:04:31.121 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 10 client.24921.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 11 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 11 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 5 11 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 4 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 10 client.24921.objecter _send_command 11 2024-03-30T15:04:31.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.119+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1c27c0 con 0x7f29bc1bea50 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.120+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1bea50 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.121+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc1c27c0 con 0x7f29bc1bea50 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.121+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 11: 0 ) 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.121+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 11 = system:0 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.121+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 4 2024-03-30T15:04:31.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.121+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 5 11 2024-03-30T15:04:31.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 10 client.24921.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 12 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 12 2024-03-30T15:04:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc02cc50 osd=5 3 2024-03-30T15:04:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 5 12 2024-03-30T15:04:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 4 2024-03-30T15:04:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 10 client.24921.objecter _send_command 12 2024-03-30T15:04:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.137+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc1c4170 con 0x7f29bc1bea50 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.139+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc1c4170 con 0x7f29bc1bea50 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.139+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.139+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.139+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 4 2024-03-30T15:04:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.139+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 5 12 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 10 client.24921.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 13 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 13 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 6 13 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 4 2024-03-30T15:04:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 10 client.24921.objecter _send_command 13 2024-03-30T15:04:31.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.140+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1c4d80 con 0x7f29bc1c4890 2024-03-30T15:04:31.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.141+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1c4890 2024-03-30T15:04:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.142+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc1c4d80 con 0x7f29bc1c4890 2024-03-30T15:04:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.142+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 13: 0 ) 2024-03-30T15:04:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.142+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 13 = system:0 2024-03-30T15:04:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.142+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 4 2024-03-30T15:04:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.142+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 6 13 2024-03-30T15:04:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.161+0000 7f29c3031700 10 client.24921.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 14 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 14 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1c4170 osd=6 3 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 6 14 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 4 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 10 client.24921.objecter _send_command 14 2024-03-30T15:04:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.162+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc1c4d80 con 0x7f29bc1c4890 2024-03-30T15:04:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc1c4d80 con 0x7f29bc1c4890 2024-03-30T15:04:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 4 2024-03-30T15:04:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 6 14 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 10 client.24921.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 15 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 15 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 7 15 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 4 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 10 client.24921.objecter _send_command 15 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f29bc1cb390 con 0x7f29bc1c7880 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.164+0000 7f29b9ffb700 10 client.24921.objecter ms_handle_connect 0x7f29bc1c7880 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.165+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f29bc1cb390 con 0x7f29bc1c7880 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.165+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 15: 0 ) 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.165+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 15 = system:0 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.165+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 4 2024-03-30T15:04:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.165+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 7 15 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 10 client.24921.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign -1 16 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 20 client.24921.objecter _get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 15 client.24921.objecter _session_command_op_remove -1 16 2024-03-30T15:04:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 20 client.24921.objecter get_session s=0x7f29bc1c4d80 osd=7 3 2024-03-30T15:04:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 15 client.24921.objecter _session_command_op_assign 7 16 2024-03-30T15:04:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 4 2024-03-30T15:04:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 10 client.24921.objecter _send_command 16 2024-03-30T15:04:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.182+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f29bc1cb390 con 0x7f29bc1c7880 2024-03-30T15:04:31.185 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:04:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29b9ffb700 1 -- 172.21.15.28:0/3571783868 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f29bc1cb390 con 0x7f29bc1c7880 2024-03-30T15:04:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29b9ffb700 10 client.24921.objecter ms_dispatch 0x7f29bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29b9ffb700 10 client.24921.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29b9ffb700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 4 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29b9ffb700 15 client.24921.objecter _session_command_op_remove 7 16 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 librados: watch_flush enter 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 librados: watch_flush exit 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.0 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6801/3394323575 conn(0x7f29bc1aac70 legacy=0x7f29bc1ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc113df0 osd=0 1 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.1 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6805/1850313547 conn(0x7f29bc062e50 legacy=0x7f29bc107ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1ac650 osd=1 1 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.2 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6809/3451325043 conn(0x7f29bc1b1340 legacy=0x7f29bc1b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1b0cc0 osd=2 1 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.3 2024-03-30T15:04:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6813/1506157250 conn(0x7f29bc1b6180 legacy=0x7f29bc1b8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02d180 osd=3 1 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.4 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.134:6800/3063775514 conn(0x7f29bc1b5ce0 legacy=0x7f29bc1bce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c15ce700 1 -- 172.21.15.28:0/3571783868 reap_dead start 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02c9b0 osd=4 1 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.5 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.134:6804/3668151179 conn(0x7f29bc1bea50 legacy=0x7f29bc1c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c15ce700 1 -- 172.21.15.28:0/3571783868 reap_dead start 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc02cc50 osd=5 1 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.6 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.134:6808/2135224194 conn(0x7f29bc1c4890 legacy=0x7f29bc1c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4170 osd=6 1 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter close_session for osd.7 2024-03-30T15:04:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.134:6812/3758190298 conn(0x7f29bc1c7880 legacy=0x7f29bc1c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter put_session s=0x7f29bc1c4d80 osd=7 1 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 20 client.24921.objecter shutdown clearing up homeless session... 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 10 client.24921.objecter successfully canceled tick 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c15ce700 1 -- 172.21.15.28:0/3571783868 reap_dead start 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6800/3011431093 conn(0x7f29a8070550 legacy=0x7f29a8072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> v1:172.21.15.28:6789/0 conn(0x7f29bc101ba0 legacy=0x7f29bc101230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 shutdown_connections 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.184+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 >> 172.21.15.28:0/3571783868 conn(0x7f29bc0761c0 msgr2=0x7f29bc110ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.185+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 shutdown_connections 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.185+0000 7f29c3031700 1 -- 172.21.15.28:0/3571783868 wait complete. 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:31.185+0000 7f29c3031700 1 librados: shutdown 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:04:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:04:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:04:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:04:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:04:31.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:31.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:31.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:31.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:31.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:31.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3571783868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:31.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:31.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3571783868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:31.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:31.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3571783868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:31.762 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:04:31.762 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:04:31.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:31.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:31.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:31.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:31.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:32.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:32.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:32.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:32.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:32.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:32.309 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:32.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[29876]: pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[29876]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:32.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[41614]: pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T15:04:32.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:32 smithi028 ceph-mon[41614]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T15:04:32.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:32.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:32.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:32 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:32 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:32 smithi134 ceph-mon[33714]: pgmap v663: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-30T15:04:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:32 smithi134 ceph-mon[33714]: osdmap e470: 8 total, 8 up, 8 in 2024-03-30T15:04:32.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:32.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:32.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:32.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:32.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:32.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:33.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.033+0000 7f1687e06700 1 -- 172.21.15.28:0/2998182841 >> v1:172.21.15.28:6789/0 conn(0x7f1680108130 legacy=0x7f1680108510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.033+0000 7f1687e06700 1 -- 172.21.15.28:0/2998182841 shutdown_connections 2024-03-30T15:04:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.033+0000 7f1687e06700 1 -- 172.21.15.28:0/2998182841 >> 172.21.15.28:0/2998182841 conn(0x7f1680076230 msgr2=0x7f1680076640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.033+0000 7f1687e06700 1 -- 172.21.15.28:0/2998182841 shutdown_connections 2024-03-30T15:04:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.033+0000 7f1687e06700 1 -- 172.21.15.28:0/2998182841 wait complete. 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 librados: starting msgr at 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 librados: starting objecter 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 Processor -- start 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 -- start start 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 librados: setting wanted keys 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 librados: calling monclient init 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16801a93b0 con 0x7f1680108130 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16801aa5d0 con 0x7f168010aea0 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1687e06700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16801ab810 con 0x7f168010eaa0 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1685ba2700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1680108130 0x7f1680107520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42826/0 (socket says 172.21.15.28:42826) 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1685ba2700 1 -- 172.21.15.28:0/3234456577 learned_addr learned my addr 172.21.15.28:0/3234456577 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f16863a3700 1 --1- 172.21.15.28:0/3234456577 >> v1:172.21.15.28:6790/0 conn(0x7f168010eaa0 0x7f16801a7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48094/0 (socket says 172.21.15.28:48094) 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter ms_handle_connect 0x7f1680108130 2024-03-30T15:04:33.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter ms_handle_connect 0x7f168010eaa0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2086112162 0 0) 0x7f16801a93b0 con 0x7f1680108130 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f166c0039c0 con 0x7f1680108130 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter ms_handle_connect 0x7f168010aea0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.034+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1554439368 0 0) 0x7f16801ab810 con 0x7f168010eaa0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16801a93b0 con 0x7f168010eaa0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607849331 0 0) 0x7f16801aa5d0 con 0x7f168010aea0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16801ab810 con 0x7f168010aea0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1193026922 0 0) 0x7f166c0039c0 con 0x7f1680108130 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16801aa5d0 con 0x7f1680108130 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f16700044c0 con 0x7f1680108130 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3801536445 0 0) 0x7f16801a93b0 con 0x7f168010eaa0 2024-03-30T15:04:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f166c0039c0 con 0x7f168010eaa0 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1678001bc0 con 0x7f168010eaa0 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 786775393 0 0) 0x7f16801ab810 con 0x7f168010aea0 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16801a93b0 con 0x7f168010aea0 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f167c001870 con 0x7f168010aea0 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 423481270 0 0) 0x7f16801aa5d0 con 0x7f1680108130 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 >> v1:172.21.15.28:6790/0 conn(0x7f168010eaa0 legacy=0x7f16801a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 >> v1:172.21.15.134:6789/0 conn(0x7f168010aea0 legacy=0x7f16801a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16801aca50 con 0x7f1680108130 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16801aba40 con 0x7f1680108130 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1687e06700 10 client.24927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16801abf00 con 0x7f1680108130 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1687e06700 1 librados: init done 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1670001b10 con 0x7f1680108130 2024-03-30T15:04:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.035+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1670004d70 con 0x7f1680108130 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.036+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1664005120 con 0x7f1680108130 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1670003860 con 0x7f1680108130 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(470..470 src has 53..470) ==== 8523+0+0 (unknown 2891509258 0 0) 0x7f1670089a00 con 0x7f1680108130 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 10 client.24927.objecter ms_dispatch 0x7f1680001550 osd_map(470..470 src has 53..470) 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 3 client.24927.objecter handle_osd_map got epochs [470,470] > 0 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 3 client.24927.objecter handle_osd_map decoding full epoch 470 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 20 client.24927.objecter dump_active .. 0 homeless 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.037+0000 7f1676ffd700 10 client.24927.objecter ms_handle_connect 0x7f166c070400 2024-03-30T15:04:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.042+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1670056aa0 con 0x7f1680108130 2024-03-30T15:04:33.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.270+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f1664005410 con 0x7f1680108130 2024-03-30T15:04:33.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:33.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.321+0000 7f1676ffd700 1 -- 172.21.15.28:0/3234456577 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v471) ==== 91+0+0 (unknown 1908937600 0 0) 0x7f167005a4f0 con 0x7f1680108130 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 10 librados: watch_flush enter 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 10 librados: watch_flush exit 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 20 client.24927.objecter shutdown clearing up homeless session... 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 10 client.24927.objecter successfully canceled tick 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 >> v1:172.21.15.28:6800/3011431093 conn(0x7f166c070400 legacy=0x7f166c0728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 >> v1:172.21.15.28:6789/0 conn(0x7f1680108130 legacy=0x7f1680107520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 shutdown_connections 2024-03-30T15:04:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.325+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 >> 172.21.15.28:0/3234456577 conn(0x7f1680076230 msgr2=0x7f1680109660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.327+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 shutdown_connections 2024-03-30T15:04:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:33.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.328+0000 7f1687e06700 1 -- 172.21.15.28:0/3234456577 wait complete. 2024-03-30T15:04:33.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:33.328+0000 7f1687e06700 1 librados: shutdown 2024-03-30T15:04:33.330 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:04:33.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:33.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:33.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:04:33.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:04:33.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:33.584 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:33.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:04:33.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:33.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:33.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:33.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:33.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:33.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:33.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:34.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.098 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:34.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:34.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:34.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:34.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[29876]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[29876]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[41614]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T15:04:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:34 smithi028 ceph-mon[41614]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-30T15:04:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:34.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:34.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:04:34.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:34 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3234456577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:04:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:34 smithi134 ceph-mon[33714]: osdmap e471: 8 total, 8 up, 8 in 2024-03-30T15:04:34.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:34 smithi134 ceph-mon[33714]: pgmap v666: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-30T15:04:34.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:34.757 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:34.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:04:34.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:34.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:34.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:35.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:35.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:35.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:35.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:35.272 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:35.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:35.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:35.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:35.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 42.14 scrub starts 2024-03-30T15:04:35.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 42.14 scrub ok 2024-03-30T15:04:35.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 101.12 scrub starts 2024-03-30T15:04:35.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 101.12 scrub ok 2024-03-30T15:04:35.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 42.14 scrub starts 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 42.14 scrub ok 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 101.12 scrub starts 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 101.12 scrub ok 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:04:35.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:35 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:04:35.552 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:35.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:35.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 42.14 scrub starts 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 42.14 scrub ok 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 101.12 scrub starts 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 101.12 scrub ok 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:04:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:35 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:04:35.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.808+0000 7f42b9b38700 1 -- 172.21.15.28:0/1366507608 >> v1:172.21.15.28:6789/0 conn(0x7f42b4100780 legacy=0x7f42b4100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.808+0000 7f42b9b38700 1 -- 172.21.15.28:0/1366507608 shutdown_connections 2024-03-30T15:04:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.808+0000 7f42b9b38700 1 -- 172.21.15.28:0/1366507608 >> 172.21.15.28:0/1366507608 conn(0x7f42b4077370 msgr2=0x7f42b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.808+0000 7f42b9b38700 1 -- 172.21.15.28:0/1366507608 shutdown_connections 2024-03-30T15:04:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.808+0000 7f42b9b38700 1 -- 172.21.15.28:0/1366507608 wait complete. 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 librados: starting msgr at 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 librados: starting objecter 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 Processor -- start 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 -- start start 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 librados: setting wanted keys 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 librados: calling monclient init 2024-03-30T15:04:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b4103200 con 0x7f42b410af00 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b4103410 con 0x7f42b4100780 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b9b38700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f42b4103640 con 0x7f42b410ea70 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b3fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f42b410ea70 0x7f42b41a8300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48118/0 (socket says 172.21.15.28:48118) 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.809+0000 7f42b3fff700 1 -- 172.21.15.28:0/1865628450 learned_addr learned my addr 172.21.15.28:0/1865628450 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter ms_handle_connect 0x7f42b410ea70 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter ms_handle_connect 0x7f42b4100780 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716779020 0 0) 0x7f42b4103410 con 0x7f42b4100780 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f429c0031f0 con 0x7f42b4100780 2024-03-30T15:04:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112245373 0 0) 0x7f42b4103640 con 0x7f42b410ea70 2024-03-30T15:04:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b4103410 con 0x7f42b410ea70 2024-03-30T15:04:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter ms_handle_connect 0x7f42b410af00 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2547954089 0 0) 0x7f429c0031f0 con 0x7f42b4100780 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b4103640 con 0x7f42b4100780 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f42a8004470 con 0x7f42b4100780 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 327951716 0 0) 0x7f42b4103410 con 0x7f42b410ea70 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f429c0031f0 con 0x7f42b410ea70 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f42ac0015b0 con 0x7f42b410ea70 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273566801 0 0) 0x7f42b4103200 con 0x7f42b410af00 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b4103410 con 0x7f42b410af00 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1274114305 0 0) 0x7f42b4103640 con 0x7f42b4100780 2024-03-30T15:04:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 >> v1:172.21.15.28:6790/0 conn(0x7f42b410ea70 legacy=0x7f42b41a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 >> v1:172.21.15.28:6789/0 conn(0x7f42b410af00 legacy=0x7f42b4101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.810+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b41aca90 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42b41a9a80 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b9b38700 10 client.50143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42b41a9f40 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b9b38700 1 librados: init done 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f42a8001af0 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.811+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f42a8004fd0 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.812+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b4103840 con 0x7f42b4100780 2024-03-30T15:04:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f42a801b920 con 0x7f42b4100780 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(472..472 src has 53..472) ==== 8523+0+0 (unknown 2697202118 0 0) 0x7f42a8088af0 con 0x7f42b4100780 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 10 client.50143.objecter ms_dispatch 0x7f42b4001550 osd_map(472..472 src has 53..472) 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 3 client.50143.objecter handle_osd_map got epochs [472,472] > 0 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 3 client.50143.objecter handle_osd_map decoding full epoch 472 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.813+0000 7f42b0ff9700 20 client.50143.objecter dump_active .. 0 homeless 2024-03-30T15:04:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.814+0000 7f42b0ff9700 10 client.50143.objecter ms_handle_connect 0x7f429c0705b0 2024-03-30T15:04:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:35.818+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f42a8055b70 con 0x7f42b4100780 2024-03-30T15:04:35.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:35.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:35.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:36.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.043+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]} v 0) -- 0x7f42b4068b30 con 0x7f42b4100780 2024-03-30T15:04:36.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:36.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:36.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:04:36.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.344+0000 7f42b0ff9700 1 -- 172.21.15.28:0/1865628450 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]=0 set 5.13 pg_upmap_items mapping to [2->5,7->1,3->4] v473) ==== 159+0+0 (unknown 4129283898 0 0) 0x7f42a80595c0 con 0x7f42b4100780 2024-03-30T15:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.346+0000 7f42b9b38700 10 librados: watch_flush enter 2024-03-30T15:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.346+0000 7f42b9b38700 10 librados: watch_flush exit 2024-03-30T15:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.346+0000 7f42b9b38700 20 client.50143.objecter shutdown clearing up homeless session... 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.346+0000 7f42b9b38700 10 client.50143.objecter successfully canceled tick 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 >> v1:172.21.15.28:6800/3011431093 conn(0x7f429c0705b0 legacy=0x7f429c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 >> v1:172.21.15.134:6789/0 conn(0x7f42b4100780 legacy=0x7f42b4101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 shutdown_connections 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 >> 172.21.15.28:0/1865628450 conn(0x7f42b4077370 msgr2=0x7f42b40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 shutdown_connections 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 -- 172.21.15.28:0/1865628450 wait complete. 2024-03-30T15:04:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:36.347+0000 7f42b9b38700 1 librados: shutdown 2024-03-30T15:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:set 5.13 pg_upmap_items mapping to [2->5,7->1,3->4] 2024-03-30T15:04:36.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:36.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 101.f scrub starts 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 101.f scrub ok 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:04:36.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1865628450' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: from='client.50143 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 42.7 scrub starts 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[29876]: 42.7 scrub ok 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 101.f scrub starts 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 101.f scrub ok 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:04:36.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:04:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:04:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1865628450' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: from='client.50143 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 42.7 scrub starts 2024-03-30T15:04:36.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:36 smithi028 ceph-mon[41614]: 42.7 scrub ok 2024-03-30T15:04:36.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:36.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: osdmap e472: 8 total, 8 up, 8 in 2024-03-30T15:04:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 101.f scrub starts 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 101.f scrub ok 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: pgmap v668: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1865628450' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: from='client.50143 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]: dispatch 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 42.7 scrub starts 2024-03-30T15:04:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:36 smithi134 ceph-mon[33714]: 42.7 scrub ok 2024-03-30T15:04:36.744 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:36.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:36.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:37.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:37.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:37.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:37.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 101.3 scrub starts 2024-03-30T15:04:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 101.3 scrub ok 2024-03-30T15:04:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:04:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:04:37.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: from='client.50143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]': finished 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:04:37.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[29876]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 101.3 scrub starts 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 101.3 scrub ok 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: from='client.50143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]': finished 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:04:37.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:37 smithi028 ceph-mon[41614]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T15:04:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:37.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:37.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 101.3 scrub starts 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 101.3 scrub ok 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: from='client.50143 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.13", "id": [2, 5, 7, 1, 3, 4]}]': finished 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: osdmap e473: 8 total, 8 up, 8 in 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T15:04:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T15:04:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:04:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:04:37.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:37 smithi134 ceph-mon[33714]: osdmap e474: 8 total, 8 up, 8 in 2024-03-30T15:04:37.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:37.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:37.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:37.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:37.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:38.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.079 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:38.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:38.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:38.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:38.341 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:38.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:38.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:38.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 42.1b scrub starts 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 42.1b scrub ok 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:04:38.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 42.12 scrub starts 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[41614]: 42.12 scrub ok 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 42.1b scrub starts 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 42.1b scrub ok 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 42.12 scrub starts 2024-03-30T15:04:38.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:38 smithi028 ceph-mon[29876]: 42.12 scrub ok 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 42.1b scrub starts 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 42.1b scrub ok 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: pgmap v671: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 42.12 scrub starts 2024-03-30T15:04:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:38 smithi134 ceph-mon[33714]: 42.12 scrub ok 2024-03-30T15:04:38.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:38.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:38.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.882+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1446498673 >> v1:172.21.15.28:6789/0 conn(0x7f6a7c10d960 legacy=0x7f6a7c10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.882+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1446498673 shutdown_connections 2024-03-30T15:04:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.882+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1446498673 >> 172.21.15.28:0/1446498673 conn(0x7f6a7c0fc9e0 msgr2=0x7f6a7c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1446498673 shutdown_connections 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1446498673 wait complete. 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 librados: starting msgr at 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 librados: starting objecter 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 Processor -- start 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- start start 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 librados: setting wanted keys 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 librados: calling monclient init 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a7c1a60a0 con 0x7f6a7c101130 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a7c1a72c0 con 0x7f6a7c109dc0 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.883+0000 7f6a83dc6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6a7c1a8500 con 0x7f6a7c10d960 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a82363700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6a7c10d960 0x7f6a7c1a4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53872/0 (socket says 172.21.15.28:53872) 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a82363700 1 -- 172.21.15.28:0/1192271039 learned_addr learned my addr 172.21.15.28:0/1192271039 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter ms_handle_connect 0x7f6a7c10d960 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter ms_handle_connect 0x7f6a7c101130 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter ms_handle_connect 0x7f6a7c109dc0 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370559635 0 0) 0x7f6a7c1a8500 con 0x7f6a7c10d960 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a68003aa0 con 0x7f6a7c10d960 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894948498 0 0) 0x7f6a7c1a60a0 con 0x7f6a7c101130 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a7c1a8500 con 0x7f6a7c101130 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756333297 0 0) 0x7f6a7c1a72c0 con 0x7f6a7c109dc0 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a7c1a60a0 con 0x7f6a7c109dc0 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 265973092 0 0) 0x7f6a68003aa0 con 0x7f6a7c10d960 2024-03-30T15:04:38.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a7c1a72c0 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2416937557 0 0) 0x7f6a7c1a8500 con 0x7f6a7c101130 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a68003aa0 con 0x7f6a7c101130 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a780013d0 con 0x7f6a7c101130 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a74004860 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3251185445 0 0) 0x7f6a7c1a60a0 con 0x7f6a7c109dc0 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a7c1a8500 con 0x7f6a7c109dc0 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a6c001830 con 0x7f6a7c109dc0 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 726381496 0 0) 0x7f6a7c1a72c0 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.134:6789/0 conn(0x7f6a7c109dc0 legacy=0x7f6a7c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6789/0 conn(0x7f6a7c101130 legacy=0x7f6a7c1064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7c1a9740 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7c1a62d0 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a83dc6700 10 client.58133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a7c1a6790 con 0x7f6a7c10d960 2024-03-30T15:04:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.884+0000 7f6a83dc6700 1 librados: init done 2024-03-30T15:04:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.885+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6a74001e90 con 0x7f6a7c10d960 2024-03-30T15:04:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.885+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6a74005100 con 0x7f6a7c10d960 2024-03-30T15:04:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.885+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6a60005100 con 0x7f6a7c10d960 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.886+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6a740050e0 con 0x7f6a7c10d960 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(474..474 src has 53..474) ==== 8552+0+0 (unknown 3956060529 0 0) 0x7f6a74089a00 con 0x7f6a7c10d960 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 osd_map(474..474 src has 53..474) 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 3 client.58133.objecter handle_osd_map got epochs [474,474] > 0 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 3 client.58133.objecter handle_osd_map decoding full epoch 474 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 20 client.58133.objecter dump_active .. 0 homeless 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a68074730 2024-03-30T15:04:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v474) ==== 54+0+16 (unknown 2293147433 0 2420000192) 0x7f6a7408a060 con 0x7f6a7c10d960 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 1 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 1 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 0 1 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 4 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 10 client.58133.objecter _send_command 1 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a60004b50 con 0x7f6a60004730 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.887+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a60004730 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.888+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a60004b50 con 0x7f6a60004730 2024-03-30T15:04:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.888+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 1: 0 ) 2024-03-30T15:04:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.888+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 1 = system:0 2024-03-30T15:04:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.888+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 4 2024-03-30T15:04:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.888+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 0 1 2024-03-30T15:04:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 2 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 2 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a600053e0 osd=0 3 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 0 2 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 4 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 10 client.58133.objecter _send_command 2 2024-03-30T15:04:38.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.905+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a600079f0 con 0x7f6a60004730 2024-03-30T15:04:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a600079f0 con 0x7f6a60004730 2024-03-30T15:04:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 4 2024-03-30T15:04:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 0 2 2024-03-30T15:04:38.909 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:04:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 1 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 4 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 10 client.58133.objecter _send_command 3 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6a60002180 con 0x7f6a60004280 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.907+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a60004280 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.908+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a60002180 con 0x7f6a60004280 2024-03-30T15:04:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.908+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 3: 0 ) 2024-03-30T15:04:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.908+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 3 = system:0 2024-03-30T15:04:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.908+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 4 2024-03-30T15:04:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.908+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 1 3 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 4 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 4 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60004110 osd=1 3 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 1 4 2024-03-30T15:04:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 4 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 10 client.58133.objecter _send_command 4 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.925+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a60001600 con 0x7f6a60004280 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.926+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a60001600 con 0x7f6a60004280 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.926+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.926+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.926+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 4 2024-03-30T15:04:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.926+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 1 4 2024-03-30T15:04:38.928 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:04:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 5 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 5 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 2 5 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 4 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 10 client.58133.objecter _send_command 5 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6a600110e0 con 0x7f6a6000d5d0 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a6000d5d0 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a600110e0 con 0x7f6a6000d5d0 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 5: 0 ) 2024-03-30T15:04:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 5 = system:0 2024-03-30T15:04:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 4 2024-03-30T15:04:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.927+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 2 5 2024-03-30T15:04:38.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 6 2024-03-30T15:04:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 6 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60008370 osd=2 3 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 2 6 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 4 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 10 client.58133.objecter _send_command 6 2024-03-30T15:04:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.944+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a60007e50 con 0x7f6a6000d5d0 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.945+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a60007e50 con 0x7f6a6000d5d0 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 4 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 2 6 2024-03-30T15:04:38.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:04:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 7 2024-03-30T15:04:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 7 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 3 7 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 4 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 10 client.58133.objecter _send_command 7 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6a6000d400 con 0x7f6a60012410 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.946+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a60012410 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.947+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a6000d400 con 0x7f6a60012410 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.947+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 7: 0 ) 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.947+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 7 = system:0 2024-03-30T15:04:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.947+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 4 2024-03-30T15:04:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.947+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 3 7 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 8 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 8 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6000cc20 osd=3 3 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 3 8 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 4 2024-03-30T15:04:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 10 client.58133.objecter _send_command 8 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.963+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a6000c760 con 0x7f6a60012410 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a6000c760 con 0x7f6a60012410 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 4 2024-03-30T15:04:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 3 8 2024-03-30T15:04:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:04:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 9 2024-03-30T15:04:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 9 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 4 9 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 4 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 10 client.58133.objecter _send_command 9 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.965+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6a6001a900 con 0x7f6a60011f70 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a60011f70 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a6001a900 con 0x7f6a60011f70 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 9: 0 ) 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 9 = system:0 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 4 2024-03-30T15:04:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.966+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 4 9 2024-03-30T15:04:38.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:38.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:38.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 10 2024-03-30T15:04:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 10 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6000c760 osd=4 3 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 4 10 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 4 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 10 client.58133.objecter _send_command 10 2024-03-30T15:04:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.983+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a6001a900 con 0x7f6a60011f70 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.984+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a6001a900 con 0x7f6a60011f70 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.984+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.984+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 4 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 4 10 2024-03-30T15:04:38.986 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:04:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 11 2024-03-30T15:04:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 11 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 5 11 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 4 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 10 client.58133.objecter _send_command 11 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6a6001ec60 con 0x7f6a6001af80 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.985+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a6001af80 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.986+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a6001ec60 con 0x7f6a6001af80 2024-03-30T15:04:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.986+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 11: 0 ) 2024-03-30T15:04:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.986+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 11 = system:0 2024-03-30T15:04:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.986+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 4 2024-03-30T15:04:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:38.986+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 5 11 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 12 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 12 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a6001a900 osd=5 3 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 5 12 2024-03-30T15:04:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 4 2024-03-30T15:04:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 10 client.58133.objecter _send_command 12 2024-03-30T15:04:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.003+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a60020610 con 0x7f6a6001af80 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.004+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a60020610 con 0x7f6a6001af80 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.004+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.004+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.004+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 4 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.004+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 5 12 2024-03-30T15:04:39.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 13 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 13 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 6 13 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 4 2024-03-30T15:04:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 10 client.58133.objecter _send_command 13 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6a60021190 con 0x7f6a60020d30 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.005+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a60020d30 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.006+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a60021190 con 0x7f6a60020d30 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.006+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 13: 0 ) 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.006+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 13 = system:0 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.006+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 4 2024-03-30T15:04:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.006+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 6 13 2024-03-30T15:04:39.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:39.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 14 2024-03-30T15:04:39.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 14 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60020610 osd=6 3 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 6 14 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 4 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 10 client.58133.objecter _send_command 14 2024-03-30T15:04:39.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.023+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a60021190 con 0x7f6a60020d30 2024-03-30T15:04:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a60021190 con 0x7f6a60020d30 2024-03-30T15:04:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 4 2024-03-30T15:04:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 6 14 2024-03-30T15:04:39.027 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:04:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 15 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 15 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 7 15 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 4 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 10 client.58133.objecter _send_command 15 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6a60027600 con 0x7f6a600213c0 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.025+0000 7f6a72ffd700 10 client.58133.objecter ms_handle_connect 0x7f6a600213c0 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.026+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6a60027600 con 0x7f6a600213c0 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.026+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 15: 0 ) 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.026+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 15 = system:0 2024-03-30T15:04:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.026+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 4 2024-03-30T15:04:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.026+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 7 15 2024-03-30T15:04:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 10 client.58133.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:04:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign -1 16 2024-03-30T15:04:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 20 client.58133.objecter _get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_remove -1 16 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 20 client.58133.objecter get_session s=0x7f6a60021190 osd=7 3 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 15 client.58133.objecter _session_command_op_assign 7 16 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 4 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 10 client.58133.objecter _send_command 16 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.044+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a600277e0 con 0x7f6a600213c0 2024-03-30T15:04:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a72ffd700 1 -- 172.21.15.28:0/1192271039 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6a600277e0 con 0x7f6a600213c0 2024-03-30T15:04:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a72ffd700 10 client.58133.objecter ms_dispatch 0x7f6a7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a72ffd700 10 client.58133.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a72ffd700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 4 2024-03-30T15:04:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a72ffd700 15 client.58133.objecter _session_command_op_remove 7 16 2024-03-30T15:04:39.047 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:04:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 librados: watch_flush enter 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 librados: watch_flush exit 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.0 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6a60004730 legacy=0x7f6a60002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a600053e0 osd=0 1 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.1 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6a60004280 legacy=0x7f6a6000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60004110 osd=1 1 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.2 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6a6000d5d0 legacy=0x7f6a6000f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60008370 osd=2 1 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.3 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6a60012410 legacy=0x7f6a60014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000cc20 osd=3 1 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.4 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6a60011f70 legacy=0x7f6a60015f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6000c760 osd=4 1 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.5 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.134:6804/3668151179 conn(0x7f6a6001af80 legacy=0x7f6a6001d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a6001a900 osd=5 1 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.6 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.134:6808/2135224194 conn(0x7f6a60020d30 legacy=0x7f6a6001ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60020610 osd=6 1 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a82363700 1 -- 172.21.15.28:0/1192271039 reap_dead start 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter close_session for osd.7 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6a600213c0 legacy=0x7f6a60025d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a82363700 1 -- 172.21.15.28:0/1192271039 reap_dead start 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter put_session s=0x7f6a60021190 osd=7 1 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 20 client.58133.objecter shutdown clearing up homeless session... 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 10 client.58133.objecter successfully canceled tick 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a82363700 1 -- 172.21.15.28:0/1192271039 reap_dead start 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a82363700 1 -- 172.21.15.28:0/1192271039 reap_dead start 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6a68074730 legacy=0x7f6a68076bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> v1:172.21.15.28:6790/0 conn(0x7f6a7c10d960 legacy=0x7f6a7c1a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 shutdown_connections 2024-03-30T15:04:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 >> 172.21.15.28:0/1192271039 conn(0x7f6a7c0fc9e0 msgr2=0x7f6a7c1105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 shutdown_connections 2024-03-30T15:04:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 -- 172.21.15.28:0/1192271039 wait complete. 2024-03-30T15:04:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:39.046+0000 7f6a83dc6700 1 librados: shutdown 2024-03-30T15:04:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:04:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:04:39.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:39.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:39.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:39.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:39.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:39.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:39.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:39.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:04:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:04:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1192271039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:39.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:04:39.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:04:39.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1192271039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:39.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:39.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:39.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:39 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:04:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:39 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:04:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1192271039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:39.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:39.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:39.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:39.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:39.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:39.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:40.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:40.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:40.244 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:40.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:40.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:40.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:40.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:40.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:40.626 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:40.635 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:04:40.636 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:04:40.636 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:04:40.636 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[41614]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:04:40.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:04:40.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:04:40.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T15:04:40.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T15:04:40.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:40 smithi028 ceph-mon[29876]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:04:40.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:04:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:04:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T15:04:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T15:04:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:40 smithi134 ceph-mon[33714]: pgmap v672: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:04:40.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:40.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:40.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:40.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:40.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:40.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:41.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:41.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:41.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:41.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:41.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:41.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:41.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:41.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:04:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:04:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:04:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:04:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:04:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:41 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:04:41.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:04:41.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:04:41.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:41 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:04:41.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:41.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:41.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:41.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:41.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:04:42.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:42.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:42.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:04:42.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:42.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:42.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:42.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:42.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:42.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:42.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:42.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:04:42.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[41614]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 460 B/s wr, 3 op/s 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[41614]: 42.15 scrub starts 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[41614]: 42.15 scrub ok 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[29876]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 460 B/s wr, 3 op/s 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[29876]: 42.15 scrub starts 2024-03-30T15:04:42.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:42 smithi028 ceph-mon[29876]: 42.15 scrub ok 2024-03-30T15:04:42.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:42 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:04:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:42 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:04:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:42 smithi134 ceph-mon[33714]: pgmap v673: 260 pgs: 260 active+clean; 8.4 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 460 B/s wr, 3 op/s 2024-03-30T15:04:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:42 smithi134 ceph-mon[33714]: 42.15 scrub starts 2024-03-30T15:04:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:42 smithi134 ceph-mon[33714]: 42.15 scrub ok 2024-03-30T15:04:42.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:42.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:42.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:42.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:42.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:42.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:43.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 -- 172.21.15.28:0/3821813346 >> v1:172.21.15.28:6789/0 conn(0x7f639010ea80 legacy=0x7f6390110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 -- 172.21.15.28:0/3821813346 shutdown_connections 2024-03-30T15:04:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 -- 172.21.15.28:0/3821813346 >> 172.21.15.28:0/3821813346 conn(0x7f63900761c0 msgr2=0x7f63900765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 -- 172.21.15.28:0/3821813346 shutdown_connections 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 -- 172.21.15.28:0/3821813346 wait complete. 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 librados: starting msgr at 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 librados: starting objecter 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.206+0000 7f6397144700 1 Processor -- start 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 -- start start 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 librados: setting wanted keys 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 librados: calling monclient init 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63901a93b0 con 0x7f639010ea80 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63901aa5d0 con 0x7f6390100a90 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f6397144700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63901ab810 con 0x7f639010af10 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f639010af10 0x7f63901a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53890/0 (socket says 172.21.15.28:53890) 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638ffff700 1 -- 172.21.15.28:0/3554251057 learned_addr learned my addr 172.21.15.28:0/3554251057 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter ms_handle_connect 0x7f639010ea80 2024-03-30T15:04:43.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter ms_handle_connect 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter ms_handle_connect 0x7f639010af10 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597092312 0 0) 0x7f63901a93b0 con 0x7f639010ea80 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6380003aa0 con 0x7f639010ea80 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351021324 0 0) 0x7f63901aa5d0 con 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63901a93b0 con 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220622266 0 0) 0x7f63901ab810 con 0x7f639010af10 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63901aa5d0 con 0x7f639010af10 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2967846116 0 0) 0x7f63901a93b0 con 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63901ab810 con 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.207+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6384001a30 con 0x7f6390100a90 2024-03-30T15:04:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3715447051 0 0) 0x7f63901aa5d0 con 0x7f639010af10 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63901a93b0 con 0x7f639010af10 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1895283328 0 0) 0x7f6380003aa0 con 0x7f639010ea80 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63901aa5d0 con 0x7f639010ea80 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f637c0017b0 con 0x7f639010af10 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f63880047c0 con 0x7f639010ea80 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2341512719 0 0) 0x7f63901ab810 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 >> v1:172.21.15.28:6790/0 conn(0x7f639010af10 legacy=0x7f63901a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 >> v1:172.21.15.28:6789/0 conn(0x7f639010ea80 legacy=0x7f63901a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63901aca50 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63901aa800 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6384001c90 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6384004af0 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f6397144700 10 client.50155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63901aacc0 con 0x7f6390100a90 2024-03-30T15:04:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.208+0000 7f6397144700 1 librados: init done 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.209+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639006a200 con 0x7f6390100a90 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.209+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6384003930 con 0x7f6390100a90 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(474..474 src has 53..474) ==== 8552+0+0 (unknown 3956060529 0 0) 0x7f63840887c0 con 0x7f6390100a90 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 10 client.50155.objecter ms_dispatch 0x7f6390001550 osd_map(474..474 src has 53..474) 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 3 client.50155.objecter handle_osd_map got epochs [474,474] > 0 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 3 client.50155.objecter handle_osd_map decoding full epoch 474 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 20 client.50155.objecter dump_active .. 0 homeless 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.210+0000 7f638dffb700 10 client.50155.objecter ms_handle_connect 0x7f6380070510 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.215+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6384055830 con 0x7f6390100a90 2024-03-30T15:04:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:43.400 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:43.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:43.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:43.450+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f6390068b60 con 0x7f6390100a90 2024-03-30T15:04:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[29876]: 42.17 scrub starts 2024-03-30T15:04:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[29876]: 42.17 scrub ok 2024-03-30T15:04:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T15:04:43.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T15:04:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[41614]: 42.17 scrub starts 2024-03-30T15:04:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[41614]: 42.17 scrub ok 2024-03-30T15:04:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T15:04:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:43 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T15:04:43.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:43.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:43.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:43.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:43.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:43.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:43 smithi134 ceph-mon[33714]: 42.17 scrub starts 2024-03-30T15:04:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:43 smithi134 ceph-mon[33714]: 42.17 scrub ok 2024-03-30T15:04:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:43 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T15:04:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:43 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T15:04:43.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:43.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:43.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:43.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:43.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:44.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:44.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:44.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:44.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:44.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:44.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.361+0000 7f638dffb700 1 -- 172.21.15.28:0/3554251057 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v475) ==== 101+0+0 (unknown 2662034252 0 0) 0x7f6390068b60 con 0x7f6390100a90 2024-03-30T15:04:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 10 librados: watch_flush enter 2024-03-30T15:04:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 10 librados: watch_flush exit 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 20 client.50155.objecter shutdown clearing up homeless session... 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 10 client.50155.objecter successfully canceled tick 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6380070510 legacy=0x7f63800729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 >> v1:172.21.15.134:6789/0 conn(0x7f6390100a90 legacy=0x7f6390107500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 shutdown_connections 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 >> 172.21.15.28:0/3554251057 conn(0x7f63900761c0 msgr2=0x7f6390109770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 shutdown_connections 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.363+0000 7f6397144700 1 -- 172.21.15.28:0/3554251057 wait complete. 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:44.364+0000 7f6397144700 1 librados: shutdown 2024-03-30T15:04:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:04:44.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:44.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:44.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:44.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:44.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3554251057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: from='client.50155 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 6.11 scrub starts 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 6.11 scrub ok 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:04:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:44 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:04:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:44.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3554251057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: from='client.50155 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 6.11 scrub starts 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 6.11 scrub ok 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-30T15:04:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:04:44.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:04:44.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3554251057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: from='client.50155 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 6.11 scrub starts 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 6.11 scrub ok 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: pgmap v674: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:04:44.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:44 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:04:44.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:44.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:44.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:44.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:45.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:45.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:45.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:45.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:45.328 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:45.401 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:04:45.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:04:45.402 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T15:04:45.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:04:45.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:45.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 42.9 scrub starts 2024-03-30T15:04:45.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 42.9 scrub ok 2024-03-30T15:04:45.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: from='client.50155 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 42.e scrub starts 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[41614]: 42.e scrub ok 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 42.9 scrub starts 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 42.9 scrub ok 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: from='client.50155 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T15:04:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:04:45.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:04:45.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 42.e scrub starts 2024-03-30T15:04:45.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:45 smithi028 ceph-mon[29876]: 42.e scrub ok 2024-03-30T15:04:45.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:45.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 42.9 scrub starts 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 42.9 scrub ok 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: from='client.50155 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: osdmap e475: 8 total, 8 up, 8 in 2024-03-30T15:04:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:04:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:04:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 42.e scrub starts 2024-03-30T15:04:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:45 smithi134 ceph-mon[33714]: 42.e scrub ok 2024-03-30T15:04:45.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:45.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:45.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:45.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:45.973 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:45.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:46.069 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:46.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:46.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:46.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:46.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:46.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:46.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:46.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[29876]: 42.4 scrub starts 2024-03-30T15:04:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[29876]: 42.4 scrub ok 2024-03-30T15:04:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[29876]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 352 B/s wr, 3 op/s 2024-03-30T15:04:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[41614]: 42.4 scrub starts 2024-03-30T15:04:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[41614]: 42.4 scrub ok 2024-03-30T15:04:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:46 smithi028 ceph-mon[41614]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 352 B/s wr, 3 op/s 2024-03-30T15:04:46.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:46.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:46.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:46 smithi134 ceph-mon[33714]: 42.4 scrub starts 2024-03-30T15:04:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:46 smithi134 ceph-mon[33714]: 42.4 scrub ok 2024-03-30T15:04:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:46 smithi134 ceph-mon[33714]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 352 B/s wr, 3 op/s 2024-03-30T15:04:46.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:46.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:46.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:46.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:46.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:47.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:47.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:47.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:47.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:47.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:47.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:47.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:47.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:04:47.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T15:04:47.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T15:04:47.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: osdmap e476: 8 total, 8 up, 8 in 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-71"}]: dispatch 2024-03-30T15:04:47.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:47.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.774+0000 7f90f43da700 1 -- 172.21.15.28:0/2101234447 >> v1:172.21.15.28:6790/0 conn(0x7f90ec10eab0 legacy=0x7f90ec110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.774+0000 7f90f43da700 1 -- 172.21.15.28:0/2101234447 shutdown_connections 2024-03-30T15:04:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.774+0000 7f90f43da700 1 -- 172.21.15.28:0/2101234447 >> 172.21.15.28:0/2101234447 conn(0x7f90ec0fc9e0 msgr2=0x7f90ec0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- 172.21.15.28:0/2101234447 shutdown_connections 2024-03-30T15:04:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- 172.21.15.28:0/2101234447 wait complete. 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 librados: starting msgr at 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 librados: starting objecter 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 Processor -- start 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- start start 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 librados: setting wanted keys 2024-03-30T15:04:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 librados: calling monclient init 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90ec1a9330 con 0x7f90ec10afa0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90ec1aa550 con 0x7f90ec103190 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.775+0000 7f90f43da700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90ec1ab790 con 0x7f90ec10eab0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90f2977700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f90ec10eab0 0x7f90ec1a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53908/0 (socket says 172.21.15.28:53908) 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 learned_addr learned my addr 172.21.15.28:0/45266660 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter ms_handle_connect 0x7f90ec10eab0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter ms_handle_connect 0x7f90ec10afa0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter ms_handle_connect 0x7f90ec103190 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488414431 0 0) 0x7f90ec1a9330 con 0x7f90ec10afa0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90d80031f0 con 0x7f90ec10afa0 2024-03-30T15:04:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1081938551 0 0) 0x7f90ec1aa550 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90ec1a9330 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197491122 0 0) 0x7f90ec1ab790 con 0x7f90ec10eab0 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90ec1aa550 con 0x7f90ec10eab0 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2404222413 0 0) 0x7f90ec1a9330 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90ec1ab790 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f90e8001870 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3492880450 0 0) 0x7f90d80031f0 con 0x7f90ec10afa0 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90ec1a9330 con 0x7f90ec10afa0 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f90e0001130 con 0x7f90ec10afa0 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2557304912 0 0) 0x7f90ec1ab790 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6790/0 conn(0x7f90ec10eab0 legacy=0x7f90ec1a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6789/0 conn(0x7f90ec10afa0 legacy=0x7f90ec1a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ec1ac9d0 con 0x7f90ec103190 2024-03-30T15:04:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.776+0000 7f90f43da700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f90ec1a9560 con 0x7f90ec103190 2024-03-30T15:04:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.777+0000 7f90f43da700 10 client.50161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.777+0000 7f90f43da700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90ec1a9a20 con 0x7f90ec103190 2024-03-30T15:04:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.777+0000 7f90f43da700 1 librados: init done 2024-03-30T15:04:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.777+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f90e8003b40 con 0x7f90ec103190 2024-03-30T15:04:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.777+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f90e8004ad0 con 0x7f90ec103190 2024-03-30T15:04:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.778+0000 7f90f43da700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f90d0005100 con 0x7f90ec103190 2024-03-30T15:04:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.778+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f90e80036a0 con 0x7f90ec103190 2024-03-30T15:04:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(476..476 src has 53..476) ==== 8552+0+0 (unknown 3751236241 0 0) 0x7f90e80887b0 con 0x7f90ec103190 2024-03-30T15:04:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 osd_map(476..476 src has 53..476) 2024-03-30T15:04:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 3 client.50161.objecter handle_osd_map got epochs [476,476] > 0 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 3 client.50161.objecter handle_osd_map decoding full epoch 476 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 20 client.50161.objecter dump_active .. 0 homeless 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v476) ==== 54+0+16 (unknown 3066237151 0 2420000192) 0x7f90e808abe0 con 0x7f90ec103190 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d8070150 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 1 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 1 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 0 1 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 4 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 10 client.50161.objecter _send_command 1 2024-03-30T15:04:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90d4006d40 con 0x7f90d4002fd0 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.779+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d4002fd0 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.780+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4006d40 con 0x7f90d4002fd0 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.780+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 1: 0 ) 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.780+0000 7f90df7fe700 10 client.50161.objecter _finish_command 1 = system:0 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.780+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 4 2024-03-30T15:04:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.780+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 0 1 2024-03-30T15:04:47.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 2 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 2 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40028e0 osd=0 3 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 0 2 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 4 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 10 client.50161.objecter _send_command 2 2024-03-30T15:04:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.797+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4001240 con 0x7f90d4002fd0 2024-03-30T15:04:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.799+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4001240 con 0x7f90d4002fd0 2024-03-30T15:04:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.799+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.799+0000 7f90df7fe700 10 client.50161.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.799+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 4 2024-03-30T15:04:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.799+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 0 2 2024-03-30T15:04:47.801 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:04:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 1 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 4 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 10 client.50161.objecter _send_command 3 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f90d4008cc0 con 0x7f90d4008100 2024-03-30T15:04:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.800+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d4008100 2024-03-30T15:04:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.801+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4008cc0 con 0x7f90d4008100 2024-03-30T15:04:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.801+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 3: 0 ) 2024-03-30T15:04:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.801+0000 7f90df7fe700 10 client.50161.objecter _finish_command 3 = system:0 2024-03-30T15:04:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.801+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 4 2024-03-30T15:04:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.801+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 1 3 2024-03-30T15:04:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 4 2024-03-30T15:04:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 4 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4001f50 osd=1 3 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 1 4 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 4 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 10 client.50161.objecter _send_command 4 2024-03-30T15:04:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.818+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4009680 con 0x7f90d4008100 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4009680 con 0x7f90d4008100 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 10 client.50161.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 4 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 1 4 2024-03-30T15:04:47.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:04:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 5 2024-03-30T15:04:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 5 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 2 5 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 4 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 10 client.50161.objecter _send_command 5 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f90d4010ae0 con 0x7f90d400ce70 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.820+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d400ce70 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.821+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4010ae0 con 0x7f90d400ce70 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.821+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 5: 0 ) 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.821+0000 7f90df7fe700 10 client.50161.objecter _finish_command 5 = system:0 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.821+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 4 2024-03-30T15:04:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.821+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 2 5 2024-03-30T15:04:47.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 6 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 6 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4001c90 osd=2 3 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 2 6 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 4 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 10 client.50161.objecter _send_command 6 2024-03-30T15:04:47.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.838+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4009220 con 0x7f90d400ce70 2024-03-30T15:04:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4009220 con 0x7f90d400ce70 2024-03-30T15:04:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90df7fe700 10 client.50161.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 4 2024-03-30T15:04:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 2 6 2024-03-30T15:04:47.841 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:04:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 7 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 7 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 3 7 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 4 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 10 client.50161.objecter _send_command 7 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.839+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f90d4007ed0 con 0x7f90d40079a0 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d40079a0 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4007ed0 con 0x7f90d40079a0 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 7: 0 ) 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 10 client.50161.objecter _finish_command 7 = system:0 2024-03-30T15:04:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 4 2024-03-30T15:04:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.840+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 3 7 2024-03-30T15:04:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:47.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 8 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 8 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40094e0 osd=3 3 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 3 8 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 4 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 10 client.50161.objecter _send_command 8 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.857+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4007420 con 0x7f90d40079a0 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.858+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4007420 con 0x7f90d40079a0 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.858+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.858+0000 7f90df7fe700 10 client.50161.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.858+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 4 2024-03-30T15:04:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.858+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 3 8 2024-03-30T15:04:47.861 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:04:47.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 9 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 9 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 4 9 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 4 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 10 client.50161.objecter _send_command 9 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f90d4007340 con 0x7f90d40119a0 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.859+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d40119a0 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.860+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4007340 con 0x7f90d40119a0 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.860+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 9: 0 ) 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.860+0000 7f90df7fe700 10 client.50161.objecter _finish_command 9 = system:0 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.860+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 4 2024-03-30T15:04:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.860+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 4 9 2024-03-30T15:04:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 10 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 10 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d40071d0 osd=4 3 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 4 10 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 4 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 10 client.50161.objecter _send_command 10 2024-03-30T15:04:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.876+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4007340 con 0x7f90d40119a0 2024-03-30T15:04:47.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.878+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4007340 con 0x7f90d40119a0 2024-03-30T15:04:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.878+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.878+0000 7f90df7fe700 10 client.50161.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.878+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 4 2024-03-30T15:04:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.878+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 4 10 2024-03-30T15:04:47.880 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 11 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 11 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 5 11 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 4 2024-03-30T15:04:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 10 client.50161.objecter _send_command 11 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f90d401e640 con 0x7f90d401aa50 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.879+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d401aa50 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.880+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d401e640 con 0x7f90d401aa50 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.880+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 11: 0 ) 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.880+0000 7f90df7fe700 10 client.50161.objecter _finish_command 11 = system:0 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.880+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 4 2024-03-30T15:04:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.880+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 5 11 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 12 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 12 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d4007340 osd=5 3 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 5 12 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 4 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 10 client.50161.objecter _send_command 12 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.898+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d401e820 con 0x7f90d401aa50 2024-03-30T15:04:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.899+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d401e820 con 0x7f90d401aa50 2024-03-30T15:04:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.899+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90df7fe700 10 client.50161.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 4 2024-03-30T15:04:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 5 12 2024-03-30T15:04:47.901 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 13 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 13 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 6 13 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 4 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 10 client.50161.objecter _send_command 13 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f90d401fb30 con 0x7f90d401ef40 2024-03-30T15:04:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.900+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d401ef40 2024-03-30T15:04:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.901+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d401fb30 con 0x7f90d401ef40 2024-03-30T15:04:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.901+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 13: 0 ) 2024-03-30T15:04:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.901+0000 7f90df7fe700 10 client.50161.objecter _finish_command 13 = system:0 2024-03-30T15:04:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.901+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 4 2024-03-30T15:04:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.901+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 6 13 2024-03-30T15:04:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 14 2024-03-30T15:04:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 14 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d401e820 osd=6 3 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 6 14 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 4 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 10 client.50161.objecter _send_command 14 2024-03-30T15:04:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.920+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d401fb30 con 0x7f90d401ef40 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.922+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d401fb30 con 0x7f90d401ef40 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.922+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.922+0000 7f90df7fe700 10 client.50161.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.922+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 4 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.922+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 6 14 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:04:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 15 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 15 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 7 15 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 4 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 10 client.50161.objecter _send_command 15 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f90d4027030 con 0x7f90d4023520 2024-03-30T15:04:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.923+0000 7f90df7fe700 10 client.50161.objecter ms_handle_connect 0x7f90d4023520 2024-03-30T15:04:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.924+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90d4027030 con 0x7f90d4023520 2024-03-30T15:04:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.924+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 15: 0 ) 2024-03-30T15:04:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.924+0000 7f90df7fe700 10 client.50161.objecter _finish_command 15 = system:0 2024-03-30T15:04:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.924+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 4 2024-03-30T15:04:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.924+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 7 15 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 10 client.50161.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign -1 16 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 20 client.50161.objecter _get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_remove -1 16 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 20 client.50161.objecter get_session s=0x7f90d401fe00 osd=7 3 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 15 client.50161.objecter _session_command_op_assign 7 16 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 4 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 10 client.50161.objecter _send_command 16 2024-03-30T15:04:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.943+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f90d4000930 con 0x7f90d4023520 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90df7fe700 1 -- 172.21.15.28:0/45266660 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f90d4000930 con 0x7f90d4023520 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90df7fe700 10 client.50161.objecter ms_dispatch 0x7f90ec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90df7fe700 10 client.50161.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90df7fe700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 4 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90df7fe700 15 client.50161.objecter _session_command_op_remove 7 16 2024-03-30T15:04:47.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:04:47.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 librados: watch_flush enter 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 librados: watch_flush exit 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.0 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6801/3394323575 conn(0x7f90d4002fd0 legacy=0x7f90d4005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40028e0 osd=0 1 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.1 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6805/1850313547 conn(0x7f90d4008100 legacy=0x7f90d40085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001f50 osd=1 1 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.2 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6809/3451325043 conn(0x7f90d400ce70 legacy=0x7f90d400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4001c90 osd=2 1 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.3 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6813/1506157250 conn(0x7f90d40079a0 legacy=0x7f90d40144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40094e0 osd=3 1 2024-03-30T15:04:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.4 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.134:6800/3063775514 conn(0x7f90d40119a0 legacy=0x7f90d4016d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d40071d0 osd=4 1 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.5 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.134:6804/3668151179 conn(0x7f90d401aa50 legacy=0x7f90d401ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d4007340 osd=5 1 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.6 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.134:6808/2135224194 conn(0x7f90d401ef40 legacy=0x7f90d401f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401e820 osd=6 1 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter close_session for osd.7 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.134:6812/3758190298 conn(0x7f90d4023520 legacy=0x7f90d4025910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter put_session s=0x7f90d401fe00 osd=7 1 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 20 client.50161.objecter shutdown clearing up homeless session... 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 10 client.50161.objecter successfully canceled tick 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 reap_dead start 2024-03-30T15:04:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 reap_dead start 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 reap_dead start 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 reap_dead start 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90f2977700 1 -- 172.21.15.28:0/45266660 reap_dead start 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.28:6800/3011431093 conn(0x7f90d8070150 legacy=0x7f90d8072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> v1:172.21.15.134:6789/0 conn(0x7f90ec103190 legacy=0x7f90ec10a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 shutdown_connections 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 >> 172.21.15.28:0/45266660 conn(0x7f90ec0fc9e0 msgr2=0x7f90ec110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.946+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 shutdown_connections 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.947+0000 7f90dd7fa700 1 -- 172.21.15.28:0/45266660 wait complete. 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:47.947+0000 7f90dd7fa700 1 librados: shutdown 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:04:47.951 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:04:47.952 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:04:47.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:47.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:47.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:48.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:48.201 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:04:48.201 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:04:48.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:48.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:48.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:48.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:48.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:48.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 12.a deep-scrub starts 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 12.a deep-scrub ok 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/45266660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 12.a deep-scrub starts 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 12.a deep-scrub ok 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T15:04:48.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T15:04:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/45266660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:04:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:04:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:48 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:04:48.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:48.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:48.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 12.a deep-scrub starts 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 12.a deep-scrub ok 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/45266660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: pgmap v678: 260 pgs: 260 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:04:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:04:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:48 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:04:48.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:48.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:48.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:48.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:48.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:48.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:49.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:49.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:49.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:49.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:49.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[29876]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[41614]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:04:49.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:49 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:04:49.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:49.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:49.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:49 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:04:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:49 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:04:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:49 smithi134 ceph-mon[33714]: osdmap e477: 8 total, 8 up, 8 in 2024-03-30T15:04:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:49 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:04:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:49 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:04:49.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:49.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:04:49.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:49.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:49.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:49.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:50.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:50.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:50.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:50.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:50.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:50.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.398 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:50.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 12.9 deep-scrub starts 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 12.9 deep-scrub ok 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 12.19 scrub starts 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 12.19 scrub ok 2024-03-30T15:04:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: pgmap v681: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 2.0 deep-scrub starts 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 2.0 deep-scrub ok 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 42.6 deep-scrub starts 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[41614]: 42.6 deep-scrub ok 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 12.9 deep-scrub starts 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 12.9 deep-scrub ok 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 12.19 scrub starts 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 12.19 scrub ok 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: pgmap v681: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 2.0 deep-scrub starts 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 2.0 deep-scrub ok 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 42.6 deep-scrub starts 2024-03-30T15:04:50.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:50 smithi028 ceph-mon[29876]: 42.6 deep-scrub ok 2024-03-30T15:04:50.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:50.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:50.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:50.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:04:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 12.9 deep-scrub starts 2024-03-30T15:04:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 12.9 deep-scrub ok 2024-03-30T15:04:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: osdmap e478: 8 total, 8 up, 8 in 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 12.19 scrub starts 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 12.19 scrub ok 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: pgmap v681: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 2.0 deep-scrub starts 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 2.0 deep-scrub ok 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 42.6 deep-scrub starts 2024-03-30T15:04:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:50 smithi134 ceph-mon[33714]: 42.6 deep-scrub ok 2024-03-30T15:04:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:50.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.766+0000 7f117ef47700 1 -- 172.21.15.28:0/1431538365 >> v1:172.21.15.28:6790/0 conn(0x7f1178102240 legacy=0x7f1178102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.766+0000 7f117ef47700 1 -- 172.21.15.28:0/1431538365 shutdown_connections 2024-03-30T15:04:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.766+0000 7f117ef47700 1 -- 172.21.15.28:0/1431538365 >> 172.21.15.28:0/1431538365 conn(0x7f11780fdaf0 msgr2=0x7f11780fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.766+0000 7f117ef47700 1 -- 172.21.15.28:0/1431538365 shutdown_connections 2024-03-30T15:04:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.766+0000 7f117ef47700 1 -- 172.21.15.28:0/1431538365 wait complete. 2024-03-30T15:04:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 librados: starting msgr at 2024-03-30T15:04:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 librados: starting objecter 2024-03-30T15:04:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 Processor -- start 2024-03-30T15:04:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 -- start start 2024-03-30T15:04:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 librados: setting wanted keys 2024-03-30T15:04:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117ef47700 1 librados: calling monclient init 2024-03-30T15:04:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117cce3700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1178102240 0x7f11781048e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45814/0 (socket says 172.21.15.28:45814) 2024-03-30T15:04:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.767+0000 7f117cce3700 1 -- 172.21.15.28:0/4266500567 learned_addr learned my addr 172.21.15.28:0/4266500567 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1178107a90 con 0x7f117810aed0 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11781a8640 con 0x7f117810ea70 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11781a9820 con 0x7f1178102240 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter ms_handle_connect 0x7f1178102240 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter ms_handle_connect 0x7f117810ea70 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter ms_handle_connect 0x7f117810aed0 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264925952 0 0) 0x7f11781a9820 con 0x7f1178102240 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11680035b0 con 0x7f1178102240 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371791606 0 0) 0x7f11781a8640 con 0x7f117810ea70 2024-03-30T15:04:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11781a9820 con 0x7f117810ea70 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186426106 0 0) 0x7f1178107a90 con 0x7f117810aed0 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11781a8640 con 0x7f117810aed0 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3495505656 0 0) 0x7f11680035b0 con 0x7f1178102240 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1178107a90 con 0x7f1178102240 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.768+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f116c001970 con 0x7f1178102240 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2067949396 0 0) 0x7f11781a9820 con 0x7f117810ea70 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11680035b0 con 0x7f117810ea70 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1170001c50 con 0x7f117810ea70 2024-03-30T15:04:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 889837155 0 0) 0x7f1178107a90 con 0x7f1178102240 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 >> v1:172.21.15.134:6789/0 conn(0x7f117810ea70 legacy=0x7f11781a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 >> v1:172.21.15.28:6789/0 conn(0x7f117810aed0 legacy=0x7f1178104ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11781a99f0 con 0x7f1178102240 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11781a9ce0 con 0x7f1178102240 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f117ef47700 10 client.58151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f11781aa1a0 con 0x7f1178102240 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f116c004770 con 0x7f1178102240 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f117ef47700 1 librados: init done 2024-03-30T15:04:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.769+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f116c0051a0 con 0x7f1178102240 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.770+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c005120 con 0x7f1178102240 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.771+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f116c005340 con 0x7f1178102240 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(479..479 src has 53..479) ==== 8552+0+0 (unknown 3317343073 0 0) 0x7f116c089b40 con 0x7f1178102240 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 10 client.58151.objecter ms_dispatch 0x7f1178001550 osd_map(479..479 src has 53..479) 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 3 client.58151.objecter handle_osd_map got epochs [479,479] > 0 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 3 client.58151.objecter handle_osd_map decoding full epoch 479 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 20 client.58151.objecter dump_active .. 0 homeless 2024-03-30T15:04:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.772+0000 7f1175ffb700 10 client.58151.objecter ms_handle_connect 0x7f1168078c30 2024-03-30T15:04:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.778+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f116c056bb0 con 0x7f1178102240 2024-03-30T15:04:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:50.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:50.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.985+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f115c005410 con 0x7f1178102240 2024-03-30T15:04:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.988+0000 7f1175ffb700 1 -- 172.21.15.28:0/4266500567 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v479) ==== 81+0+56012 (unknown 378553489 0 458483029) 0x7f116c05a600 con 0x7f1178102240 2024-03-30T15:04:50.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:04:50.991 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:04:50.991 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 479, 2024-03-30T15:04:50.991 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:04:50.991 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:04:50.374043+0000", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 112, 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:04:50.992 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 301, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:04:50.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:04:50.994 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "437", 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:50.995 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:50.996 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:50.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:50.998 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:04:50.999 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:04:51.000 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.001 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.002 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:04:51.004 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.005 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.006 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.007 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.007 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.007 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.007 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.009 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.010 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.011 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.012 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.013 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.014 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.016 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.017 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.018 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.019 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.020 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-03-30T15:04:51.022 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.025 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.026 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.027 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:51.028 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.029 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.030 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.031 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.032 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.033 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:04:51.034 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.035 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.036 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.037 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.039 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.041 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "468", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "411", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "411", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "411", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 21, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 468, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.045 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 301, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-73", 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:04:48.624272+0000", 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 32769, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,creating", 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:04:51.047 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:04:51.048 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "479", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:04:51.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:04:51.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:04:51.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:51.052 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.053 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.054 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 10 librados: watch_flush enter 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 10 librados: watch_flush exit 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 20 client.58151.objecter shutdown clearing up homeless session... 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 10 client.58151.objecter successfully canceled tick 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1168078c30 legacy=0x7f116807b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 >> v1:172.21.15.28:6790/0 conn(0x7f1178102240 legacy=0x7f11781048e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 shutdown_connections 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.991+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 >> 172.21.15.28:0/4266500567 conn(0x7f11780fdaf0 msgr2=0x7f117810a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.992+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 shutdown_connections 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.992+0000 7f117ef47700 1 -- 172.21.15.28:0/4266500567 wait complete. 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:50.992+0000 7f117ef47700 1 librados: shutdown 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.056 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.059 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:04:51.060 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.061 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.062 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.063 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:51.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 417, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.068 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:04:51.069 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.071 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:04:51.074 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:04:51.075 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:04:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 478, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:34.941959+0000", 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:31.463076+0000", 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:57.434430+0000", 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.090 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:06.444221+0000", 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.091 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:07.577028+0000", 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:03:57.286337+0000", 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:03.344622+0000", 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:04:51.093 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:27.766874+0000", 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.095 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:04:51.096 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:04:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:04:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.101 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.104 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:04:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.107 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.108 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:04:51.109 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.110 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:04:51.112 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:04:51.113 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:04:51.113 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:04:51.113 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:04:51.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:04:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:04:51.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:51.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:51.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:04:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:51.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:51.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:51.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:51.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4266500567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:04:51.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4266500567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:04:51.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:51 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:04:51.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:51.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:04:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:04:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: osdmap e479: 8 total, 8 up, 8 in 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4266500567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:04:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:51 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:04:51.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:51.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:51.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:51.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:52.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:52.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:52.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:52.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:52.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:52.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:52.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: pgmap v684: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 171 op/s 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:04:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: pgmap v684: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 171 op/s 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:04:52.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:52 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:04:52.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:52.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:52.663 INFO:tasks.thrashosds.thrasher:j is {'epoch': 479, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:04:50.374043+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 112, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 301, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '437', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '468', 'last_force_op_resend': '411', 'last_force_op_resend_prenautilus': '411', 'last_force_op_resend_preluminous': '411', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 468, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 301, 'pool_name': 'test-rados-api-smithi028-93253-73', 'create_time': '2024-03-30T15:04:48.624272+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '479', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:34.941959+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:31.463076+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:57.434430+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:06.444221+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:07.577028+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:03:57.286337+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:03.344622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:27.766874+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.13', 'mappings': [{'from': 3, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:04:52.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:04:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: osdmap e480: 8 total, 8 up, 8 in 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: pgmap v684: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 0 B/s wr, 171 op/s 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:04:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:04:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:04:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:52 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:04:52.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:52.787 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:52.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:52.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:53.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:53.038 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:53.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:53.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:53.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:53.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:53.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:53.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:04:53.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:04:53.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T15:04:53.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: 42.a scrub starts 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[41614]: 42.a scrub ok 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: 42.a scrub starts 2024-03-30T15:04:53.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:53 smithi028 ceph-mon[29876]: 42.a scrub ok 2024-03-30T15:04:53.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:53.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:53.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: osdmap e481: 8 total, 8 up, 8 in 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: 42.a scrub starts 2024-03-30T15:04:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:53 smithi134 ceph-mon[33714]: 42.a scrub ok 2024-03-30T15:04:53.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:53.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:53.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:53.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:54.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:54.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:54.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:54.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:04:54.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:04:54.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:54.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:54.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:54.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:54.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 12.b deep-scrub starts 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 12.b deep-scrub ok 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[41614]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 12.b deep-scrub starts 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 12.b deep-scrub ok 2024-03-30T15:04:54.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T15:04:54.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T15:04:54.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:04:54.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:04:54.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:54 smithi028 ceph-mon[29876]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-30T15:04:54.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 12.b deep-scrub starts 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 12.b deep-scrub ok 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:04:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:54 smithi134 ceph-mon[33714]: pgmap v686: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-30T15:04:54.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:54.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:54.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.914+0000 7fd44c21a700 1 -- 172.21.15.28:0/4207877081 >> v1:172.21.15.28:6790/0 conn(0x7fd444106650 legacy=0x7fd444106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:54.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.914+0000 7fd44c21a700 1 -- 172.21.15.28:0/4207877081 shutdown_connections 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.914+0000 7fd44c21a700 1 -- 172.21.15.28:0/4207877081 >> 172.21.15.28:0/4207877081 conn(0x7fd444077370 msgr2=0x7fd4440fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.914+0000 7fd44c21a700 1 -- 172.21.15.28:0/4207877081 shutdown_connections 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.914+0000 7fd44c21a700 1 -- 172.21.15.28:0/4207877081 wait complete. 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 librados: starting msgr at 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 librados: starting objecter 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 Processor -- start 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 -- start start 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 librados: setting wanted keys 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 librados: calling monclient init 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4441a9350 con 0x7fd444109500 2024-03-30T15:04:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4441aa550 con 0x7fd444106650 2024-03-30T15:04:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd44c21a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4441ab790 con 0x7fd44410d0d0 2024-03-30T15:04:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd4497b5700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd444109500 0x7fd444109ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39936/0 (socket says 172.21.15.28:39936) 2024-03-30T15:04:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.915+0000 7fd4497b5700 1 -- 172.21.15.28:0/2770418878 learned_addr learned my addr 172.21.15.28:0/2770418878 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter ms_handle_connect 0x7fd44410d0d0 2024-03-30T15:04:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter ms_handle_connect 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008645635 0 0) 0x7fd4441ab790 con 0x7fd44410d0d0 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd430003c50 con 0x7fd44410d0d0 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737585265 0 0) 0x7fd4441a9350 con 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4441ab790 con 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2815353394 0 0) 0x7fd4441ab790 con 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4441a9350 con 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter ms_handle_connect 0x7fd444106650 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1804266513 0 0) 0x7fd430003c50 con 0x7fd44410d0d0 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4441ab790 con 0x7fd44410d0d0 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd4400018b0 con 0x7fd444109500 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd43c001590 con 0x7fd44410d0d0 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585037011 0 0) 0x7fd4441aa550 con 0x7fd444106650 2024-03-30T15:04:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd430003c50 con 0x7fd444106650 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1263148505 0 0) 0x7fd4441a9350 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 >> v1:172.21.15.28:6790/0 conn(0x7fd44410d0d0 legacy=0x7fd4441a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 >> v1:172.21.15.134:6789/0 conn(0x7fd444106650 legacy=0x7fd4441098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4441ac9d0 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4441a9580 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd44c21a700 10 client.24963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4441a9a40 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.916+0000 7fd44c21a700 1 librados: init done 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.917+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd440003b40 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.917+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd440005d10 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.918+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd428005120 con 0x7fd444109500 2024-03-30T15:04:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.918+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd4400036a0 con 0x7fd444109500 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(481..481 src has 53..481) ==== 8552+0+0 (unknown 435639280 0 0) 0x7fd4400898e0 con 0x7fd444109500 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 10 client.24963.objecter ms_dispatch 0x7fd444001550 osd_map(481..481 src has 53..481) 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 3 client.24963.objecter handle_osd_map got epochs [481,481] > 0 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 3 client.24963.objecter handle_osd_map decoding full epoch 481 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 20 client.24963.objecter dump_active .. 0 homeless 2024-03-30T15:04:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.919+0000 7fd43affd700 10 client.24963.objecter ms_handle_connect 0x7fd430070470 2024-03-30T15:04:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:54.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:54.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:54.924+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd440056970 con 0x7fd444109500 2024-03-30T15:04:55.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:55.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:55.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.156+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd428005410 con 0x7fd444109500 2024-03-30T15:04:55.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:04:55.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:55.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:55.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.394+0000 7fd43affd700 1 -- 172.21.15.28:0/2770418878 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v482) ==== 87+0+0 (unknown 1520824860 0 0) 0x7fd44005a3c0 con 0x7fd444109500 2024-03-30T15:04:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.397+0000 7fd44c21a700 10 librados: watch_flush enter 2024-03-30T15:04:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 10 librados: watch_flush exit 2024-03-30T15:04:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 20 client.24963.objecter shutdown clearing up homeless session... 2024-03-30T15:04:55.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 10 client.24963.objecter successfully canceled tick 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd430070470 legacy=0x7fd430072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 >> v1:172.21.15.28:6789/0 conn(0x7fd444109500 legacy=0x7fd444109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 shutdown_connections 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.398+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 >> 172.21.15.28:0/2770418878 conn(0x7fd444077370 msgr2=0x7fd44410fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.399+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 shutdown_connections 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.399+0000 7fd44c21a700 1 -- 172.21.15.28:0/2770418878 wait complete. 2024-03-30T15:04:55.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:55.399+0000 7fd44c21a700 1 librados: shutdown 2024-03-30T15:04:55.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:55.403 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:04:55.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 3.0 deep-scrub starts 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 3.0 deep-scrub ok 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.b scrub starts 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.b scrub ok 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.f scrub starts 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.f scrub ok 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 101.9 scrub starts 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 101.9 scrub ok 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.5 scrub starts 2024-03-30T15:04:55.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[29876]: 42.5 scrub ok 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 3.0 deep-scrub starts 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 3.0 deep-scrub ok 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.b scrub starts 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.b scrub ok 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.f scrub starts 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.f scrub ok 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:55.435 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 101.9 scrub starts 2024-03-30T15:04:55.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 101.9 scrub ok 2024-03-30T15:04:55.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.5 scrub starts 2024-03-30T15:04:55.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:55 smithi028 ceph-mon[41614]: 42.5 scrub ok 2024-03-30T15:04:55.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:55.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:55.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:55.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 3.0 deep-scrub starts 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 3.0 deep-scrub ok 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.b scrub starts 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.b scrub ok 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.f scrub starts 2024-03-30T15:04:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.f scrub ok 2024-03-30T15:04:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:04:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 101.9 scrub starts 2024-03-30T15:04:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 101.9 scrub ok 2024-03-30T15:04:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.5 scrub starts 2024-03-30T15:04:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:55 smithi134 ceph-mon[33714]: 42.5 scrub ok 2024-03-30T15:04:55.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:55.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:55.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:55.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:55.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:56.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:56.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:56.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:56.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:56.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:56.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:56.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:56.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:56.478 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:04:56.478 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:04:56.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:56.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:56.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:04:56.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:04:56.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: 42.1a scrub starts 2024-03-30T15:04:56.570 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: 42.1a scrub ok 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[41614]: pgmap v688: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: 42.1a scrub starts 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: 42.1a scrub ok 2024-03-30T15:04:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:56.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:56.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T15:04:56.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:56 smithi028 ceph-mon[29876]: pgmap v688: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-30T15:04:56.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: 42.1a scrub starts 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: 42.1a scrub ok 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2770418878' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: osdmap e482: 8 total, 8 up, 8 in 2024-03-30T15:04:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:56 smithi134 ceph-mon[33714]: pgmap v688: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-30T15:04:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:56.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:56.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:56.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:56.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:57.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:57.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:57.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:57.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:57.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:57.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:57.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:57.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:57.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:57.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:57 smithi134 ceph-mon[33714]: 2.1 deep-scrub starts 2024-03-30T15:04:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:57 smithi134 ceph-mon[33714]: 2.1 deep-scrub ok 2024-03-30T15:04:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:57 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:57.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[29876]: 2.1 deep-scrub starts 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[29876]: 2.1 deep-scrub ok 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[41614]: 2.1 deep-scrub starts 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[41614]: 2.1 deep-scrub ok 2024-03-30T15:04:57.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:57 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:04:57.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:04:57.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:57.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:57.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:57.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:57.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:58.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:04:58.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:04:58.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:58.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:04:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:58.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:58.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:58.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:58.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:58.467 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:04:58.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:04:58 smithi028 ceph-mon[41614]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T15:04:58.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:04:58 smithi028 ceph-mon[29876]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T15:04:58.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:04:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:04:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:04:58.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:58.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:58.585 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:04:58 smithi134 ceph-mon[33714]: pgmap v689: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-30T15:04:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:58.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:04:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:58.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:04:58.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:58.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:58.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:04:58.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:58.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:04:59.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:59.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:04:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:04:59.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:59.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.576+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2897876065 >> v1:172.21.15.28:6790/0 conn(0x7fe0541031a0 legacy=0x7fe054103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.576+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2897876065 shutdown_connections 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.576+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2897876065 >> 172.21.15.28:0/2897876065 conn(0x7fe0540fc9f0 msgr2=0x7fe0540fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.576+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2897876065 shutdown_connections 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.576+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2897876065 wait complete. 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 librados: starting msgr at 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 librados: starting objecter 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 Processor -- start 2024-03-30T15:04:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 -- start start 2024-03-30T15:04:59.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 librados: setting wanted keys 2024-03-30T15:04:59.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 librados: calling monclient init 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0541a6f30 con 0x7fe05410eac0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0541a8150 con 0x7fe0541031a0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05bfc8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe0541a9390 con 0x7fe05410afb0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe059563700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe05410afb0 0x7fe0541a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39644/0 (socket says 172.21.15.28:39644) 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe059563700 1 -- 172.21.15.28:0/2826720494 learned_addr learned my addr 172.21.15.28:0/2826720494 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe05a565700 1 --1- 172.21.15.28:0/2826720494 >> v1:172.21.15.28:6789/0 conn(0x7fe05410eac0 0x7fe0541a5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37240/0 (socket says 172.21.15.28:37240) 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe046ffd700 10 client.?.objecter ms_handle_connect 0x7fe05410afb0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe046ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe046ffd700 10 client.?.objecter ms_handle_connect 0x7fe05410eac0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.577+0000 7fe046ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 10 client.?.objecter ms_handle_connect 0x7fe0541031a0 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:04:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3742943746 0 0) 0x7fe0541a9390 con 0x7fe05410afb0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0400031f0 con 0x7fe05410afb0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274137116 0 0) 0x7fe0541a6f30 con 0x7fe05410eac0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0541a9390 con 0x7fe05410eac0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996232115 0 0) 0x7fe0541a8150 con 0x7fe0541031a0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0541a6f30 con 0x7fe0541031a0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1918694977 0 0) 0x7fe0400031f0 con 0x7fe05410afb0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0541a8150 con 0x7fe05410afb0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe0480013d0 con 0x7fe05410afb0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2189720471 0 0) 0x7fe0541a9390 con 0x7fe05410eac0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0400031f0 con 0x7fe05410eac0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2502257455 0 0) 0x7fe0541a6f30 con 0x7fe0541031a0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0541a9390 con 0x7fe0541031a0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe04c001b50 con 0x7fe05410eac0 2024-03-30T15:04:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe050004470 con 0x7fe0541031a0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1125241928 0 0) 0x7fe0541a8150 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 >> v1:172.21.15.134:6789/0 conn(0x7fe0541031a0 legacy=0x7fe05410a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 >> v1:172.21.15.28:6789/0 conn(0x7fe05410eac0 legacy=0x7fe0541a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0541aa5d0 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0541a7160 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe05bfc8700 10 client.58163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0541a7620 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe05bfc8700 1 librados: init done 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.578+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe048001570 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.579+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe048005c70 con 0x7fe05410afb0 2024-03-30T15:04:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.579+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05406aed0 con 0x7fe05410afb0 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.580+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe048003fc0 con 0x7fe05410afb0 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(482..482 src has 53..482) ==== 8552+0+0 (unknown 64659968 0 0) 0x7fe048089ae0 con 0x7fe05410afb0 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 10 client.58163.objecter ms_dispatch 0x7fe054001550 osd_map(482..482 src has 53..482) 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 3 client.58163.objecter handle_osd_map got epochs [482,482] > 0 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 3 client.58163.objecter handle_osd_map decoding full epoch 482 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 20 client.58163.objecter dump_active .. 0 homeless 2024-03-30T15:04:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.581+0000 7fe046ffd700 10 client.58163.objecter ms_handle_connect 0x7fe040070470 2024-03-30T15:04:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.585+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe048056b50 con 0x7fe05410afb0 2024-03-30T15:04:59.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:59.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:59.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:04:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.786+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe054108c30 con 0x7fe040070470 2024-03-30T15:04:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.804+0000 7fe046ffd700 1 -- 172.21.15.28:0/2826720494 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+666180 (unknown 2965378022 0 3812890986) 0x7fe054108c30 con 0x7fe040070470 2024-03-30T15:04:59.813 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 10 librados: watch_flush enter 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 10 librados: watch_flush exit 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 20 client.58163.objecter shutdown clearing up homeless session... 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 10 client.58163.objecter successfully canceled tick 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe040070470 legacy=0x7fe040072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 >> v1:172.21.15.28:6790/0 conn(0x7fe05410afb0 legacy=0x7fe0541a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:04:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 shutdown_connections 2024-03-30T15:04:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 >> 172.21.15.28:0/2826720494 conn(0x7fe0540fc9f0 msgr2=0x7fe0540fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:04:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 shutdown_connections 2024-03-30T15:04:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.810+0000 7fe05bfc8700 1 -- 172.21.15.28:0/2826720494 wait complete. 2024-03-30T15:04:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:04:59.811+0000 7fe05bfc8700 1 librados: shutdown 2024-03-30T15:04:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:04:59.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:04:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:04:59.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:04:59.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:04:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:04:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:04:59.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:00.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:00.165 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:00.165 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:00.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:00.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:00.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:00.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.475+0000 7f249f22f700 1 -- 172.21.15.28:0/3352980640 >> v1:172.21.15.28:6790/0 conn(0x7f249810d9b0 legacy=0x7f249810fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- 172.21.15.28:0/3352980640 shutdown_connections 2024-03-30T15:05:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- 172.21.15.28:0/3352980640 >> 172.21.15.28:0/3352980640 conn(0x7f24980fca30 msgr2=0x7f24980fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- 172.21.15.28:0/3352980640 shutdown_connections 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- 172.21.15.28:0/3352980640 wait complete. 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 librados: starting msgr at 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 librados: starting objecter 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 Processor -- start 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- start start 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 librados: setting wanted keys 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 librados: calling monclient init 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f24980fea20 con 0x7f249810d9b0 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f24980fec30 con 0x7f2498101180 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.476+0000 7f249f22f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f24980fee40 con 0x7f2498109e10 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2497fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2498109e10 0x7f24980fd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39664/0 (socket says 172.21.15.28:39664) 2024-03-30T15:05:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2497fff700 1 -- 172.21.15.28:0/2458893042 learned_addr learned my addr 172.21.15.28:0/2458893042 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter ms_handle_connect 0x7f2498109e10 2024-03-30T15:05:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter ms_handle_connect 0x7f2498101180 2024-03-30T15:05:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter ms_handle_connect 0x7f249810d9b0 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186308399 0 0) 0x7f24980fec30 con 0x7f2498101180 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24840031f0 con 0x7f2498101180 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532587774 0 0) 0x7f24840035b0 con 0x7f2498109e10 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24980fec30 con 0x7f2498109e10 2024-03-30T15:05:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867985387 0 0) 0x7f24980fea20 con 0x7f249810d9b0 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24840035b0 con 0x7f249810d9b0 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2626449090 0 0) 0x7f24840031f0 con 0x7f2498101180 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24980fea20 con 0x7f2498101180 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f24880018b0 con 0x7f2498101180 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3890019280 0 0) 0x7f24840035b0 con 0x7f249810d9b0 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24840031f0 con 0x7f249810d9b0 2024-03-30T15:05:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990917229 0 0) 0x7f24980fec30 con 0x7f2498109e10 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24840035b0 con 0x7f2498109e10 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2490001790 con 0x7f249810d9b0 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2480001810 con 0x7f2498109e10 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 524963261 0 0) 0x7f24980fea20 con 0x7f2498101180 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6790/0 conn(0x7f2498109e10 legacy=0x7f24980fd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6789/0 conn(0x7f249810d9b0 legacy=0x7f24981a8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24981aca00 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.477+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f24981a99d0 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.478+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2488003b40 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.478+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2488004b20 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.478+0000 7f249f22f700 10 client.50182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.478+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f24981a9e90 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.478+0000 7f249f22f700 1 librados: init done 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2498102660 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f248801b400 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(482..482 src has 53..482) ==== 8552+0+0 (unknown 64659968 0 0) 0x7f24880887a0 con 0x7f2498101180 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 osd_map(482..482 src has 53..482) 2024-03-30T15:05:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f2495ffb700 3 client.50182.objecter handle_osd_map got epochs [482,482] > 0 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.479+0000 7f2495ffb700 3 client.50182.objecter handle_osd_map decoding full epoch 482 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f2495ffb700 20 client.50182.objecter dump_active .. 0 homeless 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v482) ==== 54+0+16 (unknown 692247328 0 2420000192) 0x7f248808abd0 con 0x7f2498101180 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f2484070520 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 10 client.50182.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 1 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 1 2024-03-30T15:05:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 0 1 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 4 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 10 client.50182.objecter _send_command 1 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24980fe410 con 0x7f24981082f0 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.480+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981082f0 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.481+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f24980fe410 con 0x7f24981082f0 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.481+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.481+0000 7f2495ffb700 10 client.50182.objecter _finish_command 1 = system:0 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.481+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 4 2024-03-30T15:05:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.481+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 0 1 2024-03-30T15:05:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 10 client.50182.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 2 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 2 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498111960 osd=0 3 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 0 2 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 4 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 10 client.50182.objecter _send_command 2 2024-03-30T15:05:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.498+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2498068b30 con 0x7f24981082f0 2024-03-30T15:05:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2498068b30 con 0x7f24981082f0 2024-03-30T15:05:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 10 client.50182.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 4 2024-03-30T15:05:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 0 2 2024-03-30T15:05:00.502 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 10 client.50182.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 1 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 4 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 10 client.50182.objecter _send_command 3 2024-03-30T15:05:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2498108be0 con 0x7f24981acbd0 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.500+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981acbd0 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.501+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2498108be0 con 0x7f24981acbd0 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.501+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.501+0000 7f2495ffb700 10 client.50182.objecter _finish_command 3 = system:0 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.501+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 4 2024-03-30T15:05:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.501+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 1 3 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 10 client.50182.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 4 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 4 2024-03-30T15:05:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f249802de60 osd=1 3 2024-03-30T15:05:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 1 4 2024-03-30T15:05:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 4 2024-03-30T15:05:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 10 client.50182.objecter _send_command 4 2024-03-30T15:05:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.519+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f2498108d90 con 0x7f24981acbd0 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2498108d90 con 0x7f24981acbd0 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 10 client.50182.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 4 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 1 4 2024-03-30T15:05:00.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 10 client.50182.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 5 2024-03-30T15:05:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 5 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 2 5 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 4 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 10 client.50182.objecter _send_command 5 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f24981113b0 con 0x7f2498063360 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.521+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f2498063360 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.522+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f248000a9e0 con 0x7f2498063360 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.522+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.522+0000 7f2495ffb700 10 client.50182.objecter _finish_command 5 = system:0 2024-03-30T15:05:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.522+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 4 2024-03-30T15:05:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.522+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 2 5 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 10 client.50182.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 6 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 6 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498111240 osd=2 3 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 2 6 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 4 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 10 client.50182.objecter _send_command 6 2024-03-30T15:05:00.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.538+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24980629f0 con 0x7f2498063360 2024-03-30T15:05:00.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24980629f0 con 0x7f2498063360 2024-03-30T15:05:00.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 10 client.50182.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 4 2024-03-30T15:05:00.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 2 6 2024-03-30T15:05:00.542 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 10 client.50182.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 7 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 7 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 3 7 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 4 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 10 client.50182.objecter _send_command 7 2024-03-30T15:05:00.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f24981b5ac0 con 0x7f24981b4f90 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.540+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981b4f90 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.541+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f24981b5ac0 con 0x7f24981b4f90 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.541+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.541+0000 7f2495ffb700 10 client.50182.objecter _finish_command 7 = system:0 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.541+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 4 2024-03-30T15:05:00.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.541+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 3 7 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 10 client.50182.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 8 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 8 2024-03-30T15:05:00.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f2498062c50 osd=3 3 2024-03-30T15:05:00.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 3 8 2024-03-30T15:05:00.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 4 2024-03-30T15:05:00.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 10 client.50182.objecter _send_command 8 2024-03-30T15:05:00.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.558+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24981b7410 con 0x7f24981b4f90 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24981b7410 con 0x7f24981b4f90 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 10 client.50182.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 4 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 3 8 2024-03-30T15:05:00.561 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 10 client.50182.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 9 2024-03-30T15:05:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 9 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 4 9 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 4 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 10 client.50182.objecter _send_command 9 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f24981be480 con 0x7f24981ba970 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.560+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981ba970 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.561+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f24981be480 con 0x7f24981ba970 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.561+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.561+0000 7f2495ffb700 10 client.50182.objecter _finish_command 9 = system:0 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.561+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 4 2024-03-30T15:05:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.561+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 4 9 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 10 client.50182.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 10 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 10 2024-03-30T15:05:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981b7410 osd=4 3 2024-03-30T15:05:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 4 10 2024-03-30T15:05:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 4 2024-03-30T15:05:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 10 client.50182.objecter _send_command 10 2024-03-30T15:05:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.578+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24981b7680 con 0x7f24981ba970 2024-03-30T15:05:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24981b7680 con 0x7f24981ba970 2024-03-30T15:05:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f2495ffb700 10 client.50182.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 4 2024-03-30T15:05:00.582 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 4 10 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 10 client.50182.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 11 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 11 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 5 11 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 4 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 10 client.50182.objecter _send_command 11 2024-03-30T15:05:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.580+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f24981c2870 con 0x7f24981beb00 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981beb00 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f24981c2870 con 0x7f24981beb00 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 10 client.50182.objecter _finish_command 11 = system:0 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 4 2024-03-30T15:05:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.581+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 5 11 2024-03-30T15:05:00.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 10 client.50182.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 12 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 12 2024-03-30T15:05:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981be480 osd=5 3 2024-03-30T15:05:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 5 12 2024-03-30T15:05:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 4 2024-03-30T15:05:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 10 client.50182.objecter _send_command 12 2024-03-30T15:05:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.598+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24981c3bd0 con 0x7f24981beb00 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24981c3bd0 con 0x7f24981beb00 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 10 client.50182.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 4 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 5 12 2024-03-30T15:05:00.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 10 client.50182.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 13 2024-03-30T15:05:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 13 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 6 13 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 4 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 10 client.50182.objecter _send_command 13 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f24981c4f20 con 0x7f24981c3de0 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.600+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981c3de0 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.601+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f24981c4f20 con 0x7f24981c3de0 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.601+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.601+0000 7f2495ffb700 10 client.50182.objecter _finish_command 13 = system:0 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.601+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 4 2024-03-30T15:05:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.601+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 6 13 2024-03-30T15:05:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 10 client.50182.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 14 2024-03-30T15:05:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 14 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981c3bd0 osd=6 3 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 6 14 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 4 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 10 client.50182.objecter _send_command 14 2024-03-30T15:05:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.619+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24981c4f20 con 0x7f24981c3de0 2024-03-30T15:05:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24981c4f20 con 0x7f24981c3de0 2024-03-30T15:05:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f2495ffb700 10 client.50182.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 4 2024-03-30T15:05:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 6 14 2024-03-30T15:05:00.623 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:00.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 10 client.50182.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:00.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 15 2024-03-30T15:05:00.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 15 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 7 15 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 4 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 10 client.50182.objecter _send_command 15 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.621+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f24981c5200 con 0x7f24981c7930 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 10 client.50182.objecter ms_handle_connect 0x7f24981c7930 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2488086260 con 0x7f24981c7930 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 10 client.50182.objecter _finish_command 15 = system:0 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 4 2024-03-30T15:05:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.622+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 7 15 2024-03-30T15:05:00.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 10 client.50182.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign -1 16 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 20 client.50182.objecter _get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 15 client.50182.objecter _session_command_op_remove -1 16 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 20 client.50182.objecter get_session s=0x7f24981c4f20 osd=7 3 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 15 client.50182.objecter _session_command_op_assign 7 16 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 4 2024-03-30T15:05:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 10 client.50182.objecter _send_command 16 2024-03-30T15:05:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.641+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f24981b7680 con 0x7f24981c7930 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.642+0000 7f2495ffb700 1 -- 172.21.15.28:0/2458893042 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f24981b7680 con 0x7f24981c7930 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f2495ffb700 10 client.50182.objecter ms_dispatch 0x7f2498001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f2495ffb700 10 client.50182.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f2495ffb700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 4 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f2495ffb700 15 client.50182.objecter _session_command_op_remove 7 16 2024-03-30T15:05:00.644 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 librados: watch_flush enter 2024-03-30T15:05:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 librados: watch_flush exit 2024-03-30T15:05:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.0 2024-03-30T15:05:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6801/3394323575 conn(0x7f24981082f0 legacy=0x7f24981ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111960 osd=0 1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6805/1850313547 conn(0x7f24981acbd0 legacy=0x7f24981aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f249802de60 osd=1 1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.2 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6809/3451325043 conn(0x7f2498063360 legacy=0x7f24981b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498111240 osd=2 1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.3 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6813/1506157250 conn(0x7f24981b4f90 legacy=0x7f24981b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f2498062c50 osd=3 1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.4 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.134:6800/3063775514 conn(0x7f24981ba970 legacy=0x7f24981bcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981b7410 osd=4 1 2024-03-30T15:05:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.5 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.134:6804/3668151179 conn(0x7f24981beb00 legacy=0x7f24981c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981be480 osd=5 1 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.6 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.134:6808/2135224194 conn(0x7f24981c3de0 legacy=0x7f24981b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249d7cc700 1 -- 172.21.15.28:0/2458893042 reap_dead start 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c3bd0 osd=6 1 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter close_session for osd.7 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.134:6812/3758190298 conn(0x7f24981c7930 legacy=0x7f24981c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249d7cc700 1 -- 172.21.15.28:0/2458893042 reap_dead start 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter put_session s=0x7f24981c4f20 osd=7 1 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 20 client.50182.objecter shutdown clearing up homeless session... 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 10 client.50182.objecter successfully canceled tick 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249d7cc700 1 -- 172.21.15.28:0/2458893042 reap_dead start 2024-03-30T15:05:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249d7cc700 1 -- 172.21.15.28:0/2458893042 reap_dead start 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2484070520 legacy=0x7f24840729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.643+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> v1:172.21.15.134:6789/0 conn(0x7f2498101180 legacy=0x7f24980fcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.644+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 shutdown_connections 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.644+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 >> 172.21.15.28:0/2458893042 conn(0x7f24980fca30 msgr2=0x7f2498110690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.644+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 shutdown_connections 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.644+0000 7f249f22f700 1 -- 172.21.15.28:0/2458893042 wait complete. 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:00.644+0000 7f249f22f700 1 librados: shutdown 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:00.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:00.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:00.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:00.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:05:00.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:00.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:00.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:00.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:01.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:01.047 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:01.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:01.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:01.174 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2024-03-30T15:04:58.008823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792235,"num_objects":372,"num_object_clones":7,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":2094,"num_read_kb":1835,"num_write":1538,"num_write_kb":15181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2924,"ondisk_log_size":2924,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":763,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":604128,"kb_used_data":92116,"kb_used_omap":12,"kb_used_meta":511731,"kb_avail":749357088,"statfs":{"total":767960285184,"available":767341658112,"internally_reserved":0,"allocated":94326784,"data_stored":64638874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":524013104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.624809"},"pg_stats":[{"pgid":"301.1f","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995084+0000","last_change":"2024-03-30T15:04:50.399250+0000","last_active":"2024-03-30T15:04:52.995084+0000","last_peered":"2024-03-30T15:04:52.995084+0000","last_clean":"2024-03-30T15:04:52.995084+0000","last_became_active":"2024-03-30T15:04:50.398265+0000","last_became_peered":"2024-03-30T15:04:50.398265+0000","last_unstale":"2024-03-30T15:04:52.995084+0000","last_undegraded":"2024-03-30T15:04:52.995084+0000","last_fullsized":"2024-03-30T15:04:52.995084+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:56.780465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.1e","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392512+0000","last_change":"2024-03-30T15:04:50.398972+0000","last_active":"2024-03-30T15:04:52.392512+0000","last_peered":"2024-03-30T15:04:52.392512+0000","last_clean":"2024-03-30T15:04:52.392512+0000","last_became_active":"2024-03-30T15:04:50.398441+0000","last_became_peered":"2024-03-30T15:04:50.398441+0000","last_unstale":"2024-03-30T15:04:52.392512+0000","last_undegraded":"2024-03-30T15:04:52.392512+0000","last_fullsized":"2024-03-30T15:04:52.392512+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:12.449484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.1d","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404189+0000","last_change":"2024-03-30T15:04:50.403883+0000","last_active":"2024-03-30T15:04:52.404189+0000","last_peered":"2024-03-30T15:04:52.404189+0000","last_clean":"2024-03-30T15:04:52.404189+0000","last_became_active":"2024-03-30T15:04:50.401989+0000","last_became_peered":"2024-03-30T15:04:50.401989+0000","last_unstale":"2024-03-30T15:04:52.404189+0000","last_undegraded":"2024-03-30T15:04:52.404189+0000","last_fullsized":"2024-03-30T15:04:52.404189+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.652592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1c","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403689+0000","last_change":"2024-03-30T15:04:50.408548+0000","last_active":"2024-03-30T15:04:52.403689+0000","last_peered":"2024-03-30T15:04:52.403689+0000","last_clean":"2024-03-30T15:04:52.403689+0000","last_became_active":"2024-03-30T15:04:50.408288+0000","last_became_peered":"2024-03-30T15:04:50.408288+0000","last_unstale":"2024-03-30T15:04:52.403689+0000","last_undegraded":"2024-03-30T15:04:52.403689+0000","last_fullsized":"2024-03-30T15:04:52.403689+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:18.206460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1b","version":"481'4","reported_seq":19,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.428361+0000","last_change":"2024-03-30T15:04:50.402372+0000","last_active":"2024-03-30T15:04:52.428361+0000","last_peered":"2024-03-30T15:04:52.428361+0000","last_clean":"2024-03-30T15:04:52.428361+0000","last_became_active":"2024-03-30T15:04:50.401588+0000","last_became_peered":"2024-03-30T15:04:50.401588+0000","last_unstale":"2024-03-30T15:04:52.428361+0000","last_undegraded":"2024-03-30T15:04:52.428361+0000","last_fullsized":"2024-03-30T15:04:52.428361+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:05.880602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.1a","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391016+0000","last_change":"2024-03-30T15:04:50.396307+0000","last_active":"2024-03-30T15:04:52.391016+0000","last_peered":"2024-03-30T15:04:52.391016+0000","last_clean":"2024-03-30T15:04:52.391016+0000","last_became_active":"2024-03-30T15:04:50.395754+0000","last_became_peered":"2024-03-30T15:04:50.395754+0000","last_unstale":"2024-03-30T15:04:52.391016+0000","last_undegraded":"2024-03-30T15:04:52.391016+0000","last_fullsized":"2024-03-30T15:04:52.391016+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:09.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.19","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412097+0000","last_change":"2024-03-30T15:04:50.403723+0000","last_active":"2024-03-30T15:04:52.412097+0000","last_peered":"2024-03-30T15:04:52.412097+0000","last_clean":"2024-03-30T15:04:52.412097+0000","last_became_active":"2024-03-30T15:04:50.403129+0000","last_became_peered":"2024-03-30T15:04:50.403129+0000","last_unstale":"2024-03-30T15:04:52.412097+0000","last_undegraded":"2024-03-30T15:04:52.412097+0000","last_fullsized":"2024-03-30T15:04:52.412097+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:12.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.18","version":"481'2","reported_seq":18,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.428265+0000","last_change":"2024-03-30T15:04:50.403462+0000","last_active":"2024-03-30T15:04:52.428265+0000","last_peered":"2024-03-30T15:04:52.428265+0000","last_clean":"2024-03-30T15:04:52.428265+0000","last_became_active":"2024-03-30T15:04:50.403167+0000","last_became_peered":"2024-03-30T15:04:50.403167+0000","last_unstale":"2024-03-30T15:04:52.428265+0000","last_undegraded":"2024-03-30T15:04:52.428265+0000","last_fullsized":"2024-03-30T15:04:52.428265+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:55.336850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.17","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995106+0000","last_change":"2024-03-30T15:04:50.400522+0000","last_active":"2024-03-30T15:04:52.995106+0000","last_peered":"2024-03-30T15:04:52.995106+0000","last_clean":"2024-03-30T15:04:52.995106+0000","last_became_active":"2024-03-30T15:04:50.400160+0000","last_became_peered":"2024-03-30T15:04:50.400160+0000","last_unstale":"2024-03-30T15:04:52.995106+0000","last_undegraded":"2024-03-30T15:04:52.995106+0000","last_fullsized":"2024-03-30T15:04:52.995106+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.699152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.16","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995047+0000","last_change":"2024-03-30T15:04:50.399878+0000","last_active":"2024-03-30T15:04:52.995047+0000","last_peered":"2024-03-30T15:04:52.995047+0000","last_clean":"2024-03-30T15:04:52.995047+0000","last_became_active":"2024-03-30T15:04:50.399407+0000","last_became_peered":"2024-03-30T15:04:50.399407+0000","last_unstale":"2024-03-30T15:04:52.995047+0000","last_undegraded":"2024-03-30T15:04:52.995047+0000","last_fullsized":"2024-03-30T15:04:52.995047+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:52.906849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.15","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424342+0000","last_change":"2024-03-30T15:04:50.398417+0000","last_active":"2024-03-30T15:04:52.424342+0000","last_peered":"2024-03-30T15:04:52.424342+0000","last_clean":"2024-03-30T15:04:52.424342+0000","last_became_active":"2024-03-30T15:04:50.398090+0000","last_became_peered":"2024-03-30T15:04:50.398090+0000","last_unstale":"2024-03-30T15:04:52.424342+0000","last_undegraded":"2024-03-30T15:04:52.424342+0000","last_fullsized":"2024-03-30T15:04:52.424342+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.891807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"301.14","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994479+0000","last_change":"2024-03-30T15:04:50.393970+0000","last_active":"2024-03-30T15:04:52.994479+0000","last_peered":"2024-03-30T15:04:52.994479+0000","last_clean":"2024-03-30T15:04:52.994479+0000","last_became_active":"2024-03-30T15:04:50.393638+0000","last_became_peered":"2024-03-30T15:04:50.393638+0000","last_unstale":"2024-03-30T15:04:52.994479+0000","last_undegraded":"2024-03-30T15:04:52.994479+0000","last_fullsized":"2024-03-30T15:04:52.994479+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.102527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.13","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.390933+0000","last_change":"2024-03-30T15:04:50.407329+0000","last_active":"2024-03-30T15:04:52.390933+0000","last_peered":"2024-03-30T15:04:52.390933+0000","last_clean":"2024-03-30T15:04:52.390933+0000","last_became_active":"2024-03-30T15:04:50.406996+0000","last_became_peered":"2024-03-30T15:04:50.406996+0000","last_unstale":"2024-03-30T15:04:52.390933+0000","last_undegraded":"2024-03-30T15:04:52.390933+0000","last_fullsized":"2024-03-30T15:04:52.390933+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.159780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.12","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408948+0000","last_change":"2024-03-30T15:04:50.395888+0000","last_active":"2024-03-30T15:04:52.408948+0000","last_peered":"2024-03-30T15:04:52.408948+0000","last_clean":"2024-03-30T15:04:52.408948+0000","last_became_active":"2024-03-30T15:04:50.394567+0000","last_became_peered":"2024-03-30T15:04:50.394567+0000","last_unstale":"2024-03-30T15:04:52.408948+0000","last_undegraded":"2024-03-30T15:04:52.408948+0000","last_fullsized":"2024-03-30T15:04:52.408948+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:55.067944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.11","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412124+0000","last_change":"2024-03-30T15:04:50.400541+0000","last_active":"2024-03-30T15:04:52.412124+0000","last_peered":"2024-03-30T15:04:52.412124+0000","last_clean":"2024-03-30T15:04:52.412124+0000","last_became_active":"2024-03-30T15:04:50.400206+0000","last_became_peered":"2024-03-30T15:04:50.400206+0000","last_unstale":"2024-03-30T15:04:52.412124+0000","last_undegraded":"2024-03-30T15:04:52.412124+0000","last_fullsized":"2024-03-30T15:04:52.412124+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.845482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.10","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405290+0000","last_change":"2024-03-30T15:04:50.393172+0000","last_active":"2024-03-30T15:04:52.405290+0000","last_peered":"2024-03-30T15:04:52.405290+0000","last_clean":"2024-03-30T15:04:52.405290+0000","last_became_active":"2024-03-30T15:04:50.392869+0000","last_became_peered":"2024-03-30T15:04:50.392869+0000","last_unstale":"2024-03-30T15:04:52.405290+0000","last_undegraded":"2024-03-30T15:04:52.405290+0000","last_fullsized":"2024-03-30T15:04:52.405290+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.050894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"301.f","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413866+0000","last_change":"2024-03-30T15:04:50.402397+0000","last_active":"2024-03-30T15:04:52.413866+0000","last_peered":"2024-03-30T15:04:52.413866+0000","last_clean":"2024-03-30T15:04:52.413866+0000","last_became_active":"2024-03-30T15:04:50.402001+0000","last_became_peered":"2024-03-30T15:04:50.402001+0000","last_unstale":"2024-03-30T15:04:52.413866+0000","last_undegraded":"2024-03-30T15:04:52.413866+0000","last_fullsized":"2024-03-30T15:04:52.413866+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:18.195966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.e","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392609+0000","last_change":"2024-03-30T15:04:50.407540+0000","last_active":"2024-03-30T15:04:52.392609+0000","last_peered":"2024-03-30T15:04:52.392609+0000","last_clean":"2024-03-30T15:04:52.392609+0000","last_became_active":"2024-03-30T15:04:50.406992+0000","last_became_peered":"2024-03-30T15:04:50.406992+0000","last_unstale":"2024-03-30T15:04:52.392609+0000","last_undegraded":"2024-03-30T15:04:52.392609+0000","last_fullsized":"2024-03-30T15:04:52.392609+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.911027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.d","version":"481'2","reported_seq":18,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.427854+0000","last_change":"2024-03-30T15:04:50.397687+0000","last_active":"2024-03-30T15:04:52.427854+0000","last_peered":"2024-03-30T15:04:52.427854+0000","last_clean":"2024-03-30T15:04:52.427854+0000","last_became_active":"2024-03-30T15:04:50.397392+0000","last_became_peered":"2024-03-30T15:04:50.397392+0000","last_unstale":"2024-03-30T15:04:52.427854+0000","last_undegraded":"2024-03-30T15:04:52.427854+0000","last_fullsized":"2024-03-30T15:04:52.427854+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:56.527662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.c","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412607+0000","last_change":"2024-03-30T15:04:50.398714+0000","last_active":"2024-03-30T15:04:52.412607+0000","last_peered":"2024-03-30T15:04:52.412607+0000","last_clean":"2024-03-30T15:04:52.412607+0000","last_became_active":"2024-03-30T15:04:50.398385+0000","last_became_peered":"2024-03-30T15:04:50.398385+0000","last_unstale":"2024-03-30T15:04:52.412607+0000","last_undegraded":"2024-03-30T15:04:52.412607+0000","last_fullsized":"2024-03-30T15:04:52.412607+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:55.575098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.b","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.394980+0000","last_change":"2024-03-30T15:04:50.395801+0000","last_active":"2024-03-30T15:04:52.394980+0000","last_peered":"2024-03-30T15:04:52.394980+0000","last_clean":"2024-03-30T15:04:52.394980+0000","last_became_active":"2024-03-30T15:04:50.395043+0000","last_became_peered":"2024-03-30T15:04:50.395043+0000","last_unstale":"2024-03-30T15:04:52.394980+0000","last_undegraded":"2024-03-30T15:04:52.394980+0000","last_fullsized":"2024-03-30T15:04:52.394980+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:52.500590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.a","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409034+0000","last_change":"2024-03-30T15:04:50.395020+0000","last_active":"2024-03-30T15:04:52.409034+0000","last_peered":"2024-03-30T15:04:52.409034+0000","last_clean":"2024-03-30T15:04:52.409034+0000","last_became_active":"2024-03-30T15:04:50.394734+0000","last_became_peered":"2024-03-30T15:04:50.394734+0000","last_unstale":"2024-03-30T15:04:52.409034+0000","last_undegraded":"2024-03-30T15:04:52.409034+0000","last_fullsized":"2024-03-30T15:04:52.409034+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.971936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.9","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423527+0000","last_change":"2024-03-30T15:04:50.402115+0000","last_active":"2024-03-30T15:04:52.423527+0000","last_peered":"2024-03-30T15:04:52.423527+0000","last_clean":"2024-03-30T15:04:52.423527+0000","last_became_active":"2024-03-30T15:04:50.401805+0000","last_became_peered":"2024-03-30T15:04:50.401805+0000","last_unstale":"2024-03-30T15:04:52.423527+0000","last_undegraded":"2024-03-30T15:04:52.423527+0000","last_fullsized":"2024-03-30T15:04:52.423527+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.480838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"301.8","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408418+0000","last_change":"2024-03-30T15:04:50.398527+0000","last_active":"2024-03-30T15:04:52.408418+0000","last_peered":"2024-03-30T15:04:52.408418+0000","last_clean":"2024-03-30T15:04:52.408418+0000","last_became_active":"2024-03-30T15:04:50.398113+0000","last_became_peered":"2024-03-30T15:04:50.398113+0000","last_unstale":"2024-03-30T15:04:52.408418+0000","last_undegraded":"2024-03-30T15:04:52.408418+0000","last_fullsized":"2024-03-30T15:04:52.408418+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.538558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.7","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410281+0000","last_change":"2024-03-30T15:04:50.394651+0000","last_active":"2024-03-30T15:04:52.410281+0000","last_peered":"2024-03-30T15:04:52.410281+0000","last_clean":"2024-03-30T15:04:52.410281+0000","last_became_active":"2024-03-30T15:04:50.394374+0000","last_became_peered":"2024-03-30T15:04:50.394374+0000","last_unstale":"2024-03-30T15:04:52.410281+0000","last_undegraded":"2024-03-30T15:04:52.410281+0000","last_fullsized":"2024-03-30T15:04:52.410281+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:02.387068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"301.6","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413437+0000","last_change":"2024-03-30T15:04:50.403706+0000","last_active":"2024-03-30T15:04:52.413437+0000","last_peered":"2024-03-30T15:04:52.413437+0000","last_clean":"2024-03-30T15:04:52.413437+0000","last_became_active":"2024-03-30T15:04:50.403320+0000","last_became_peered":"2024-03-30T15:04:50.403320+0000","last_unstale":"2024-03-30T15:04:52.413437+0000","last_undegraded":"2024-03-30T15:04:52.413437+0000","last_fullsized":"2024-03-30T15:04:52.413437+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:51.888433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"301.5","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405955+0000","last_change":"2024-03-30T15:04:50.394439+0000","last_active":"2024-03-30T15:04:52.405955+0000","last_peered":"2024-03-30T15:04:52.405955+0000","last_clean":"2024-03-30T15:04:52.405955+0000","last_became_active":"2024-03-30T15:04:50.394104+0000","last_became_peered":"2024-03-30T15:04:50.394104+0000","last_unstale":"2024-03-30T15:04:52.405955+0000","last_undegraded":"2024-03-30T15:04:52.405955+0000","last_fullsized":"2024-03-30T15:04:52.405955+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.281470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"301.4","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403644+0000","last_change":"2024-03-30T15:04:50.407575+0000","last_active":"2024-03-30T15:04:52.403644+0000","last_peered":"2024-03-30T15:04:52.403644+0000","last_clean":"2024-03-30T15:04:52.403644+0000","last_became_active":"2024-03-30T15:04:50.407206+0000","last_became_peered":"2024-03-30T15:04:50.407206+0000","last_unstale":"2024-03-30T15:04:52.403644+0000","last_undegraded":"2024-03-30T15:04:52.403644+0000","last_fullsized":"2024-03-30T15:04:52.403644+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:55.704050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"301.3","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.395000+0000","last_change":"2024-03-30T15:04:50.396155+0000","last_active":"2024-03-30T15:04:52.395000+0000","last_peered":"2024-03-30T15:04:52.395000+0000","last_clean":"2024-03-30T15:04:52.395000+0000","last_became_active":"2024-03-30T15:04:50.395839+0000","last_became_peered":"2024-03-30T15:04:50.395839+0000","last_unstale":"2024-03-30T15:04:52.395000+0000","last_undegraded":"2024-03-30T15:04:52.395000+0000","last_fullsized":"2024-03-30T15:04:52.395000+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:50.723704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"301.2","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994252+0000","last_change":"2024-03-30T15:04:50.405495+0000","last_active":"2024-03-30T15:04:52.994252+0000","last_peered":"2024-03-30T15:04:52.994252+0000","last_clean":"2024-03-30T15:04:52.994252+0000","last_became_active":"2024-03-30T15:04:50.405141+0000","last_became_peered":"2024-03-30T15:04:50.405141+0000","last_unstale":"2024-03-30T15:04:52.994252+0000","last_undegraded":"2024-03-30T15:04:52.994252+0000","last_fullsized":"2024-03-30T15:04:52.994252+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:12.181843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"301.1","version":"481'2","reported_seq":18,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.428631+0000","last_change":"2024-03-30T15:04:50.395109+0000","last_active":"2024-03-30T15:04:52.428631+0000","last_peered":"2024-03-30T15:04:52.428631+0000","last_clean":"2024-03-30T15:04:52.428631+0000","last_became_active":"2024-03-30T15:04:50.394161+0000","last_became_peered":"2024-03-30T15:04:50.394161+0000","last_unstale":"2024-03-30T15:04:52.428631+0000","last_undegraded":"2024-03-30T15:04:52.428631+0000","last_fullsized":"2024-03-30T15:04:52.428631+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.869128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"301.0","version":"0'0","reported_seq":15,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.390650+0000","last_change":"2024-03-30T15:04:50.398153+0000","last_active":"2024-03-30T15:04:52.390650+0000","last_peered":"2024-03-30T15:04:52.390650+0000","last_clean":"2024-03-30T15:04:52.390650+0000","last_became_active":"2024-03-30T15:04:50.397826+0000","last_became_peered":"2024-03-30T15:04:50.397826+0000","last_unstale":"2024-03-30T15:04:52.390650+0000","last_undegraded":"2024-03-30T15:04:52.390650+0000","last_fullsized":"2024-03-30T15:04:52.390650+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":478,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:49.368220+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.368220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:49.700632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":954,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.394233+0000","last_change":"2024-03-30T15:04:06.344710+0000","last_active":"2024-03-30T15:04:52.394233+0000","last_peered":"2024-03-30T15:04:52.394233+0000","last_clean":"2024-03-30T15:04:52.394233+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:04:52.394233+0000","last_undegraded":"2024-03-30T15:04:52.394233+0000","last_fullsized":"2024-03-30T15:04:52.394233+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":974,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423986+0000","last_change":"2024-03-30T15:04:06.379921+0000","last_active":"2024-03-30T15:04:52.423986+0000","last_peered":"2024-03-30T15:04:52.423986+0000","last_clean":"2024-03-30T15:04:52.423986+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:04:52.423986+0000","last_undegraded":"2024-03-30T15:04:52.423986+0000","last_fullsized":"2024-03-30T15:04:52.423986+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:31.961438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.961438+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"474'106","reported_seq":999,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.402685+0000","last_change":"2024-03-30T15:04:06.471415+0000","last_active":"2024-03-30T15:04:52.402685+0000","last_peered":"2024-03-30T15:04:52.402685+0000","last_clean":"2024-03-30T15:04:52.402685+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:04:52.402685+0000","last_undegraded":"2024-03-30T15:04:52.402685+0000","last_fullsized":"2024-03-30T15:04:52.402685+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'87","last_scrub_stamp":"2024-03-30T15:02:40.447037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.447037+0000","objects_scrubbed":7,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":98,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":1011,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405471+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:04:52.405471+0000","last_peered":"2024-03-30T15:04:52.405471+0000","last_clean":"2024-03-30T15:04:52.405471+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:04:52.405471+0000","last_undegraded":"2024-03-30T15:04:52.405471+0000","last_fullsized":"2024-03-30T15:04:52.405471+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:23.354917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"286'36","reported_seq":950,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392702+0000","last_change":"2024-03-30T15:04:06.346222+0000","last_active":"2024-03-30T15:04:52.392702+0000","last_peered":"2024-03-30T15:04:52.392702+0000","last_clean":"2024-03-30T15:04:52.392702+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:04:52.392702+0000","last_undegraded":"2024-03-30T15:04:52.392702+0000","last_fullsized":"2024-03-30T15:04:52.392702+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:03:19.020165+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.020165+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":1046,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391854+0000","last_change":"2024-03-30T15:04:06.347583+0000","last_active":"2024-03-30T15:04:52.391854+0000","last_peered":"2024-03-30T15:04:52.391854+0000","last_clean":"2024-03-30T15:04:52.391854+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:04:52.391854+0000","last_undegraded":"2024-03-30T15:04:52.391854+0000","last_fullsized":"2024-03-30T15:04:52.391854+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":936,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424441+0000","last_change":"2024-03-30T15:04:06.381417+0000","last_active":"2024-03-30T15:04:52.424441+0000","last_peered":"2024-03-30T15:04:52.424441+0000","last_clean":"2024-03-30T15:04:52.424441+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:04:52.424441+0000","last_undegraded":"2024-03-30T15:04:52.424441+0000","last_fullsized":"2024-03-30T15:04:52.424441+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":956,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404256+0000","last_change":"2024-03-30T15:04:06.473326+0000","last_active":"2024-03-30T15:04:52.404256+0000","last_peered":"2024-03-30T15:04:52.404256+0000","last_clean":"2024-03-30T15:04:52.404256+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:04:52.404256+0000","last_undegraded":"2024-03-30T15:04:52.404256+0000","last_fullsized":"2024-03-30T15:04:52.404256+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.219102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":955,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409404+0000","last_change":"2024-03-30T15:04:34.489872+0000","last_active":"2024-03-30T15:04:52.409404+0000","last_peered":"2024-03-30T15:04:52.409404+0000","last_clean":"2024-03-30T15:04:52.409404+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:04:52.409404+0000","last_undegraded":"2024-03-30T15:04:52.409404+0000","last_fullsized":"2024-03-30T15:04:52.409404+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:04:34.489778+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.489778+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.808911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":939,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409355+0000","last_change":"2024-03-30T15:04:06.474203+0000","last_active":"2024-03-30T15:04:52.409355+0000","last_peered":"2024-03-30T15:04:52.409355+0000","last_clean":"2024-03-30T15:04:52.409355+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:04:52.409355+0000","last_undegraded":"2024-03-30T15:04:52.409355+0000","last_fullsized":"2024-03-30T15:04:52.409355+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:02:00.557079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.557079+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"481'21","reported_seq":923,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.426236+0000","last_change":"2024-03-30T15:04:06.471477+0000","last_active":"2024-03-30T15:04:52.426236+0000","last_peered":"2024-03-30T15:04:52.426236+0000","last_clean":"2024-03-30T15:04:52.426236+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:04:52.426236+0000","last_undegraded":"2024-03-30T15:04:52.426236+0000","last_fullsized":"2024-03-30T15:04:52.426236+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'5","last_scrub_stamp":"2024-03-30T15:02:29.567113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:29.567113+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"428'21","reported_seq":925,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.402794+0000","last_change":"2024-03-30T15:04:06.472183+0000","last_active":"2024-03-30T15:04:52.402794+0000","last_peered":"2024-03-30T15:04:52.402794+0000","last_clean":"2024-03-30T15:04:52.402794+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:04:52.402794+0000","last_undegraded":"2024-03-30T15:04:52.402794+0000","last_fullsized":"2024-03-30T15:04:52.402794+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'17","last_scrub_stamp":"2024-03-30T15:02:35.464703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.464703+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":919,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424794+0000","last_change":"2024-03-30T15:04:06.381760+0000","last_active":"2024-03-30T15:04:52.424794+0000","last_peered":"2024-03-30T15:04:52.424794+0000","last_clean":"2024-03-30T15:04:52.424794+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:04:52.424794+0000","last_undegraded":"2024-03-30T15:04:52.424794+0000","last_fullsized":"2024-03-30T15:04:52.424794+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":983,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410484+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:04:52.410484+0000","last_peered":"2024-03-30T15:04:52.410484+0000","last_clean":"2024-03-30T15:04:52.410484+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:04:52.410484+0000","last_undegraded":"2024-03-30T15:04:52.410484+0000","last_fullsized":"2024-03-30T15:04:52.410484+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"460'60","reported_seq":964,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409656+0000","last_change":"2024-03-30T15:04:06.474098+0000","last_active":"2024-03-30T15:04:52.409656+0000","last_peered":"2024-03-30T15:04:52.409656+0000","last_clean":"2024-03-30T15:04:52.409656+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:04:52.409656+0000","last_undegraded":"2024-03-30T15:04:52.409656+0000","last_fullsized":"2024-03-30T15:04:52.409656+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":990,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391367+0000","last_change":"2024-03-30T15:04:06.346253+0000","last_active":"2024-03-30T15:04:52.391367+0000","last_peered":"2024-03-30T15:04:52.391367+0000","last_clean":"2024-03-30T15:04:52.391367+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:04:52.391367+0000","last_undegraded":"2024-03-30T15:04:52.391367+0000","last_fullsized":"2024-03-30T15:04:52.391367+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"368'35","reported_seq":936,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403043+0000","last_change":"2024-03-30T15:04:06.473836+0000","last_active":"2024-03-30T15:04:52.403043+0000","last_peered":"2024-03-30T15:04:52.403043+0000","last_clean":"2024-03-30T15:04:52.403043+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:04:52.403043+0000","last_undegraded":"2024-03-30T15:04:52.403043+0000","last_fullsized":"2024-03-30T15:04:52.403043+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'35","last_scrub_stamp":"2024-03-30T15:02:42.436904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.436904+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":1005,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412454+0000","last_change":"2024-03-30T15:04:06.383533+0000","last_active":"2024-03-30T15:04:52.412454+0000","last_peered":"2024-03-30T15:04:52.412454+0000","last_clean":"2024-03-30T15:04:52.412454+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:04:52.412454+0000","last_undegraded":"2024-03-30T15:04:52.412454+0000","last_fullsized":"2024-03-30T15:04:52.412454+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:02:39.411331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.411331+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"444'55","reported_seq":968,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994233+0000","last_change":"2024-03-30T15:04:07.016310+0000","last_active":"2024-03-30T15:04:52.994233+0000","last_peered":"2024-03-30T15:04:52.994233+0000","last_clean":"2024-03-30T15:04:52.994233+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:04:52.994233+0000","last_undegraded":"2024-03-30T15:04:52.994233+0000","last_fullsized":"2024-03-30T15:04:52.994233+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":949,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412217+0000","last_change":"2024-03-30T15:04:06.383117+0000","last_active":"2024-03-30T15:04:52.412217+0000","last_peered":"2024-03-30T15:04:52.412217+0000","last_clean":"2024-03-30T15:04:52.412217+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:04:52.412217+0000","last_undegraded":"2024-03-30T15:04:52.412217+0000","last_fullsized":"2024-03-30T15:04:52.412217+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:02:41.402421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.402421+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":1014,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413665+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:04:52.413665+0000","last_peered":"2024-03-30T15:04:52.413665+0000","last_clean":"2024-03-30T15:04:52.413665+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:04:52.413665+0000","last_undegraded":"2024-03-30T15:04:52.413665+0000","last_fullsized":"2024-03-30T15:04:52.413665+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:06.371753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":975,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410635+0000","last_change":"2024-03-30T15:04:35.492607+0000","last_active":"2024-03-30T15:04:52.410635+0000","last_peered":"2024-03-30T15:04:52.410635+0000","last_clean":"2024-03-30T15:04:52.410635+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:04:52.410635+0000","last_undegraded":"2024-03-30T15:04:52.410635+0000","last_fullsized":"2024-03-30T15:04:52.410635+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'60","last_scrub_stamp":"2024-03-30T15:04:35.492526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.492526+0000","objects_scrubbed":6,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.066090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":966,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403794+0000","last_change":"2024-03-30T15:04:06.473834+0000","last_active":"2024-03-30T15:04:52.403794+0000","last_peered":"2024-03-30T15:04:52.403794+0000","last_clean":"2024-03-30T15:04:52.403794+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:04:52.403794+0000","last_undegraded":"2024-03-30T15:04:52.403794+0000","last_fullsized":"2024-03-30T15:04:52.403794+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:04.989653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"400'89","reported_seq":978,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424643+0000","last_change":"2024-03-30T15:04:06.381269+0000","last_active":"2024-03-30T15:04:52.424643+0000","last_peered":"2024-03-30T15:04:52.424643+0000","last_clean":"2024-03-30T15:04:52.424643+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:04:52.424643+0000","last_undegraded":"2024-03-30T15:04:52.424643+0000","last_fullsized":"2024-03-30T15:04:52.424643+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'87","last_scrub_stamp":"2024-03-30T15:02:01.656924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.656924+0000","objects_scrubbed":6,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":998,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405639+0000","last_change":"2024-03-30T15:04:07.014089+0000","last_active":"2024-03-30T15:04:52.405639+0000","last_peered":"2024-03-30T15:04:52.405639+0000","last_clean":"2024-03-30T15:04:52.405639+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:04:52.405639+0000","last_undegraded":"2024-03-30T15:04:52.405639+0000","last_fullsized":"2024-03-30T15:04:52.405639+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":960,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.395060+0000","last_change":"2024-03-30T15:04:35.992306+0000","last_active":"2024-03-30T15:04:52.395060+0000","last_peered":"2024-03-30T15:04:52.395060+0000","last_clean":"2024-03-30T15:04:52.395060+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:04:52.395060+0000","last_undegraded":"2024-03-30T15:04:52.395060+0000","last_fullsized":"2024-03-30T15:04:52.395060+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:04:35.992167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.992167+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.042534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"452'42","reported_seq":964,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391043+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:04:52.391043+0000","last_peered":"2024-03-30T15:04:52.391043+0000","last_clean":"2024-03-30T15:04:52.391043+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:04:52.391043+0000","last_undegraded":"2024-03-30T15:04:52.391043+0000","last_fullsized":"2024-03-30T15:04:52.391043+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.602954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"324'104","reported_seq":1018,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412804+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:04:52.412804+0000","last_peered":"2024-03-30T15:04:52.412804+0000","last_clean":"2024-03-30T15:04:52.412804+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:04:52.412804+0000","last_undegraded":"2024-03-30T15:04:52.412804+0000","last_fullsized":"2024-03-30T15:04:52.412804+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:57.323659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000074,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":3937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":930,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404778+0000","last_change":"2024-03-30T15:04:06.474417+0000","last_active":"2024-03-30T15:04:52.404778+0000","last_peered":"2024-03-30T15:04:52.404778+0000","last_clean":"2024-03-30T15:04:52.404778+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:04:52.404778+0000","last_undegraded":"2024-03-30T15:04:52.404778+0000","last_fullsized":"2024-03-30T15:04:52.404778+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:02:36.478360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.478360+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":921,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392809+0000","last_change":"2024-03-30T15:04:06.347655+0000","last_active":"2024-03-30T15:04:52.392809+0000","last_peered":"2024-03-30T15:04:52.392809+0000","last_clean":"2024-03-30T15:04:52.392809+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:04:52.392809+0000","last_undegraded":"2024-03-30T15:04:52.392809+0000","last_fullsized":"2024-03-30T15:04:52.392809+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:02:28.967486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.967486+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":965,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409670+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:04:52.409670+0000","last_peered":"2024-03-30T15:04:52.409670+0000","last_clean":"2024-03-30T15:04:52.409670+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:04:52.409670+0000","last_undegraded":"2024-03-30T15:04:52.409670+0000","last_fullsized":"2024-03-30T15:04:52.409670+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":1009,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995642+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:04:52.995642+0000","last_peered":"2024-03-30T15:04:52.995642+0000","last_clean":"2024-03-30T15:04:52.995642+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:04:52.995642+0000","last_undegraded":"2024-03-30T15:04:52.995642+0000","last_fullsized":"2024-03-30T15:04:52.995642+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:57.696240+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995112+0000","last_change":"2024-03-30T15:04:42.056061+0000","last_active":"2024-03-30T15:04:52.995112+0000","last_peered":"2024-03-30T15:04:52.995112+0000","last_clean":"2024-03-30T15:04:52.995112+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:04:52.995112+0000","last_undegraded":"2024-03-30T15:04:52.995112+0000","last_fullsized":"2024-03-30T15:04:52.995112+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.055955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.055955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.618053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":766,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412997+0000","last_change":"2024-03-30T15:04:34.266981+0000","last_active":"2024-03-30T15:04:52.412997+0000","last_peered":"2024-03-30T15:04:52.412997+0000","last_clean":"2024-03-30T15:04:52.412997+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:04:52.412997+0000","last_undegraded":"2024-03-30T15:04:52.412997+0000","last_fullsized":"2024-03-30T15:04:52.412997+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:34.266865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.266865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:49.944697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":757,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425406+0000","last_change":"2024-03-30T15:04:42.277788+0000","last_active":"2024-03-30T15:04:52.425406+0000","last_peered":"2024-03-30T15:04:52.425406+0000","last_clean":"2024-03-30T15:04:52.425406+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:04:52.425406+0000","last_undegraded":"2024-03-30T15:04:52.425406+0000","last_fullsized":"2024-03-30T15:04:52.425406+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.712874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406009+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:04:52.406009+0000","last_peered":"2024-03-30T15:04:52.406009+0000","last_clean":"2024-03-30T15:04:52.406009+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:04:52.406009+0000","last_undegraded":"2024-03-30T15:04:52.406009+0000","last_fullsized":"2024-03-30T15:04:52.406009+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.852443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":753,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409210+0000","last_change":"2024-03-30T15:01:57.530926+0000","last_active":"2024-03-30T15:04:52.409210+0000","last_peered":"2024-03-30T15:04:52.409210+0000","last_clean":"2024-03-30T15:04:52.409210+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:04:52.409210+0000","last_undegraded":"2024-03-30T15:04:52.409210+0000","last_fullsized":"2024-03-30T15:04:52.409210+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.530827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.530827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":761,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.390912+0000","last_change":"2024-03-30T15:02:41.928550+0000","last_active":"2024-03-30T15:04:52.390912+0000","last_peered":"2024-03-30T15:04:52.390912+0000","last_clean":"2024-03-30T15:04:52.390912+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:04:52.390912+0000","last_undegraded":"2024-03-30T15:04:52.390912+0000","last_fullsized":"2024-03-30T15:04:52.390912+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.928469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.928469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":749,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424349+0000","last_change":"2024-03-30T15:01:57.663288+0000","last_active":"2024-03-30T15:04:52.424349+0000","last_peered":"2024-03-30T15:04:52.424349+0000","last_clean":"2024-03-30T15:04:52.424349+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:04:52.424349+0000","last_undegraded":"2024-03-30T15:04:52.424349+0000","last_fullsized":"2024-03-30T15:04:52.424349+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.663204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.663204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994666+0000","last_change":"2024-03-30T15:04:38.084291+0000","last_active":"2024-03-30T15:04:52.994666+0000","last_peered":"2024-03-30T15:04:52.994666+0000","last_clean":"2024-03-30T15:04:52.994666+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:04:52.994666+0000","last_undegraded":"2024-03-30T15:04:52.994666+0000","last_fullsized":"2024-03-30T15:04:52.994666+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:38.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:40.487265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":749,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404305+0000","last_change":"2024-03-30T15:02:34.464153+0000","last_active":"2024-03-30T15:04:52.404305+0000","last_peered":"2024-03-30T15:04:52.404305+0000","last_clean":"2024-03-30T15:04:52.404305+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:04:52.404305+0000","last_undegraded":"2024-03-30T15:04:52.404305+0000","last_fullsized":"2024-03-30T15:04:52.404305+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:34.464067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.464067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406171+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:04:52.406171+0000","last_peered":"2024-03-30T15:04:52.406171+0000","last_clean":"2024-03-30T15:04:52.406171+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:04:52.406171+0000","last_undegraded":"2024-03-30T15:04:52.406171+0000","last_fullsized":"2024-03-30T15:04:52.406171+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":750,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425269+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:04:52.425269+0000","last_peered":"2024-03-30T15:04:52.425269+0000","last_clean":"2024-03-30T15:04:52.425269+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:04:52.425269+0000","last_undegraded":"2024-03-30T15:04:52.425269+0000","last_fullsized":"2024-03-30T15:04:52.425269+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392787+0000","last_change":"2024-03-30T15:02:38.936916+0000","last_active":"2024-03-30T15:04:52.392787+0000","last_peered":"2024-03-30T15:04:52.392787+0000","last_clean":"2024-03-30T15:04:52.392787+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:04:52.392787+0000","last_undegraded":"2024-03-30T15:04:52.392787+0000","last_fullsized":"2024-03-30T15:04:52.392787+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:38.936820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.936820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":749,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423578+0000","last_change":"2024-03-30T15:01:51.636327+0000","last_active":"2024-03-30T15:04:52.423578+0000","last_peered":"2024-03-30T15:04:52.423578+0000","last_clean":"2024-03-30T15:04:52.423578+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:04:52.423578+0000","last_undegraded":"2024-03-30T15:04:52.423578+0000","last_fullsized":"2024-03-30T15:04:52.423578+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.636231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.636231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405351+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:04:52.405351+0000","last_peered":"2024-03-30T15:04:52.405351+0000","last_clean":"2024-03-30T15:04:52.405351+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:04:52.405351+0000","last_undegraded":"2024-03-30T15:04:52.405351+0000","last_fullsized":"2024-03-30T15:04:52.405351+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":769,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.395061+0000","last_change":"2024-03-30T15:04:37.008749+0000","last_active":"2024-03-30T15:04:52.395061+0000","last_peered":"2024-03-30T15:04:52.395061+0000","last_clean":"2024-03-30T15:04:52.395061+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:04:52.395061+0000","last_undegraded":"2024-03-30T15:04:52.395061+0000","last_fullsized":"2024-03-30T15:04:52.395061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.008666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.008666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.100396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":761,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.394523+0000","last_change":"2024-03-30T15:02:07.597777+0000","last_active":"2024-03-30T15:04:52.394523+0000","last_peered":"2024-03-30T15:04:52.394523+0000","last_clean":"2024-03-30T15:04:52.394523+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:04:52.394523+0000","last_undegraded":"2024-03-30T15:04:52.394523+0000","last_fullsized":"2024-03-30T15:04:52.394523+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.597700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":750,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424862+0000","last_change":"2024-03-30T15:01:10.292440+0000","last_active":"2024-03-30T15:04:52.424862+0000","last_peered":"2024-03-30T15:04:52.424862+0000","last_clean":"2024-03-30T15:04:52.424862+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:04:52.424862+0000","last_undegraded":"2024-03-30T15:04:52.424862+0000","last_fullsized":"2024-03-30T15:04:52.424862+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:10.292346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:10.292346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404384+0000","last_change":"2024-03-30T15:04:45.608747+0000","last_active":"2024-03-30T15:04:52.404384+0000","last_peered":"2024-03-30T15:04:52.404384+0000","last_clean":"2024-03-30T15:04:52.404384+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:04:52.404384+0000","last_undegraded":"2024-03-30T15:04:52.404384+0000","last_fullsized":"2024-03-30T15:04:52.404384+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.608652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.608652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:05.611823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995516+0000","last_change":"2024-03-30T15:04:36.136344+0000","last_active":"2024-03-30T15:04:52.995516+0000","last_peered":"2024-03-30T15:04:52.995516+0000","last_clean":"2024-03-30T15:04:52.995516+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:04:52.995516+0000","last_undegraded":"2024-03-30T15:04:52.995516+0000","last_fullsized":"2024-03-30T15:04:52.995516+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.136228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.136228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:36.176305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425094+0000","last_change":"2024-03-30T15:04:50.298381+0000","last_active":"2024-03-30T15:04:52.425094+0000","last_peered":"2024-03-30T15:04:52.425094+0000","last_clean":"2024-03-30T15:04:52.425094+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:04:52.425094+0000","last_undegraded":"2024-03-30T15:04:52.425094+0000","last_fullsized":"2024-03-30T15:04:52.425094+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.298282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.756120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405515+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:04:52.405515+0000","last_peered":"2024-03-30T15:04:52.405515+0000","last_clean":"2024-03-30T15:04:52.405515+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:04:52.405515+0000","last_undegraded":"2024-03-30T15:04:52.405515+0000","last_fullsized":"2024-03-30T15:04:52.405515+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423505+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:04:52.423505+0000","last_peered":"2024-03-30T15:04:52.423505+0000","last_clean":"2024-03-30T15:04:52.423505+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:04:52.423505+0000","last_undegraded":"2024-03-30T15:04:52.423505+0000","last_fullsized":"2024-03-30T15:04:52.423505+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.116140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412500+0000","last_change":"2024-03-30T15:02:40.426121+0000","last_active":"2024-03-30T15:04:52.412500+0000","last_peered":"2024-03-30T15:04:52.412500+0000","last_clean":"2024-03-30T15:04:52.412500+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:04:52.412500+0000","last_undegraded":"2024-03-30T15:04:52.412500+0000","last_fullsized":"2024-03-30T15:04:52.412500+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.426031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.426031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":770,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393609+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:04:52.393609+0000","last_peered":"2024-03-30T15:04:52.393609+0000","last_clean":"2024-03-30T15:04:52.393609+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:04:52.393609+0000","last_undegraded":"2024-03-30T15:04:52.393609+0000","last_fullsized":"2024-03-30T15:04:52.393609+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.425095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412884+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:04:52.412884+0000","last_peered":"2024-03-30T15:04:52.412884+0000","last_clean":"2024-03-30T15:04:52.412884+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:04:52.412884+0000","last_undegraded":"2024-03-30T15:04:52.412884+0000","last_fullsized":"2024-03-30T15:04:52.412884+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406281+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:04:52.406281+0000","last_peered":"2024-03-30T15:04:52.406281+0000","last_clean":"2024-03-30T15:04:52.406281+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:04:52.406281+0000","last_undegraded":"2024-03-30T15:04:52.406281+0000","last_fullsized":"2024-03-30T15:04:52.406281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":751,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995773+0000","last_change":"2024-03-30T15:01:58.138003+0000","last_active":"2024-03-30T15:04:52.995773+0000","last_peered":"2024-03-30T15:04:52.995773+0000","last_clean":"2024-03-30T15:04:52.995773+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:04:52.995773+0000","last_undegraded":"2024-03-30T15:04:52.995773+0000","last_fullsized":"2024-03-30T15:04:52.995773+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.137887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.137887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995119+0000","last_change":"2024-03-30T15:04:45.130399+0000","last_active":"2024-03-30T15:04:52.995119+0000","last_peered":"2024-03-30T15:04:52.995119+0000","last_clean":"2024-03-30T15:04:52.995119+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:04:52.995119+0000","last_undegraded":"2024-03-30T15:04:52.995119+0000","last_fullsized":"2024-03-30T15:04:52.995119+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.130322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.130322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.109084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":767,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412188+0000","last_change":"2024-03-30T15:04:44.264561+0000","last_active":"2024-03-30T15:04:52.412188+0000","last_peered":"2024-03-30T15:04:52.412188+0000","last_clean":"2024-03-30T15:04:52.412188+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:04:52.412188+0000","last_undegraded":"2024-03-30T15:04:52.412188+0000","last_fullsized":"2024-03-30T15:04:52.412188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.264430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.264430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.388512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994063+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:04:52.994063+0000","last_peered":"2024-03-30T15:04:52.994063+0000","last_clean":"2024-03-30T15:04:52.994063+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:04:52.994063+0000","last_undegraded":"2024-03-30T15:04:52.994063+0000","last_fullsized":"2024-03-30T15:04:52.994063+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.836391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":750,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403346+0000","last_change":"2024-03-30T15:01:51.481224+0000","last_active":"2024-03-30T15:04:52.403346+0000","last_peered":"2024-03-30T15:04:52.403346+0000","last_clean":"2024-03-30T15:04:52.403346+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:04:52.403346+0000","last_undegraded":"2024-03-30T15:04:52.403346+0000","last_fullsized":"2024-03-30T15:04:52.403346+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.481143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.481143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:53.311855+0000","last_change":"2024-03-30T15:04:53.311876+0000","last_active":"2024-03-30T15:04:53.311855+0000","last_peered":"2024-03-30T15:04:53.311855+0000","last_clean":"2024-03-30T15:04:53.311855+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:04:53.311855+0000","last_undegraded":"2024-03-30T15:04:53.311855+0000","last_fullsized":"2024-03-30T15:04:53.311855+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.311797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.311797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.989454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":772,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413102+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:04:52.413102+0000","last_peered":"2024-03-30T15:04:52.413102+0000","last_clean":"2024-03-30T15:04:52.413102+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:04:52.413102+0000","last_undegraded":"2024-03-30T15:04:52.413102+0000","last_fullsized":"2024-03-30T15:04:52.413102+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":884,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404384+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:04:52.404384+0000","last_peered":"2024-03-30T15:04:52.404384+0000","last_clean":"2024-03-30T15:04:52.404384+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:04:52.404384+0000","last_undegraded":"2024-03-30T15:04:52.404384+0000","last_fullsized":"2024-03-30T15:04:52.404384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":764,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409695+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:04:52.409695+0000","last_peered":"2024-03-30T15:04:52.409695+0000","last_clean":"2024-03-30T15:04:52.409695+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:04:52.409695+0000","last_undegraded":"2024-03-30T15:04:52.409695+0000","last_fullsized":"2024-03-30T15:04:52.409695+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"76'8","reported_seq":1000,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393479+0000","last_change":"2024-03-30T15:04:03.157433+0000","last_active":"2024-03-30T15:04:52.393479+0000","last_peered":"2024-03-30T15:04:52.393479+0000","last_clean":"2024-03-30T15:04:52.393479+0000","last_became_active":"2024-03-30T14:54:41.343232+0000","last_became_peered":"2024-03-30T14:54:41.343232+0000","last_unstale":"2024-03-30T15:04:52.393479+0000","last_undegraded":"2024-03-30T15:04:52.393479+0000","last_fullsized":"2024-03-30T15:04:52.393479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.245942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":846,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406411+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:04:52.406411+0000","last_peered":"2024-03-30T15:04:52.406411+0000","last_clean":"2024-03-30T15:04:52.406411+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:52.406411+0000","last_undegraded":"2024-03-30T15:04:52.406411+0000","last_fullsized":"2024-03-30T15:04:52.406411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.569612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":772,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412999+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:04:52.412999+0000","last_peered":"2024-03-30T15:04:52.412999+0000","last_clean":"2024-03-30T15:04:52.412999+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:04:52.412999+0000","last_undegraded":"2024-03-30T15:04:52.412999+0000","last_fullsized":"2024-03-30T15:04:52.412999+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:26.821252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":911,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404566+0000","last_change":"2024-03-30T15:04:34.512958+0000","last_active":"2024-03-30T15:04:52.404566+0000","last_peered":"2024-03-30T15:04:52.404566+0000","last_clean":"2024-03-30T15:04:52.404566+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:04:52.404566+0000","last_undegraded":"2024-03-30T15:04:52.404566+0000","last_fullsized":"2024-03-30T15:04:52.404566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:04:34.512838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.512838+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.123121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":874,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412301+0000","last_change":"2024-03-30T15:04:48.217127+0000","last_active":"2024-03-30T15:04:52.412301+0000","last_peered":"2024-03-30T15:04:52.412301+0000","last_clean":"2024-03-30T15:04:52.412301+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:04:52.412301+0000","last_undegraded":"2024-03-30T15:04:52.412301+0000","last_fullsized":"2024-03-30T15:04:52.412301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.217046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.217046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.053751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407468+0000","last_change":"2024-03-30T15:04:48.997109+0000","last_active":"2024-03-30T15:04:52.407468+0000","last_peered":"2024-03-30T15:04:52.407468+0000","last_clean":"2024-03-30T15:04:52.407468+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:04:52.407468+0000","last_undegraded":"2024-03-30T15:04:52.407468+0000","last_fullsized":"2024-03-30T15:04:52.407468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.997021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.997021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.181602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":864,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392980+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:04:52.392980+0000","last_peered":"2024-03-30T15:04:52.392980+0000","last_clean":"2024-03-30T15:04:52.392980+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:04:52.392980+0000","last_undegraded":"2024-03-30T15:04:52.392980+0000","last_fullsized":"2024-03-30T15:04:52.392980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:33.116494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":765,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406380+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:04:52.406380+0000","last_peered":"2024-03-30T15:04:52.406380+0000","last_clean":"2024-03-30T15:04:52.406380+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:04:52.406380+0000","last_undegraded":"2024-03-30T15:04:52.406380+0000","last_fullsized":"2024-03-30T15:04:52.406380+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":890,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404965+0000","last_change":"2024-03-30T15:02:03.449650+0000","last_active":"2024-03-30T15:04:52.404965+0000","last_peered":"2024-03-30T15:04:52.404965+0000","last_clean":"2024-03-30T15:04:52.404965+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:04:52.404965+0000","last_undegraded":"2024-03-30T15:04:52.404965+0000","last_fullsized":"2024-03-30T15:04:52.404965+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:03.449549+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.449549+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":858,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403140+0000","last_change":"2024-03-30T15:01:50.503242+0000","last_active":"2024-03-30T15:04:52.403140+0000","last_peered":"2024-03-30T15:04:52.403140+0000","last_clean":"2024-03-30T15:04:52.403140+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:04:52.403140+0000","last_undegraded":"2024-03-30T15:04:52.403140+0000","last_fullsized":"2024-03-30T15:04:52.403140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.503156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.503156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":849,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425702+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:04:52.425702+0000","last_peered":"2024-03-30T15:04:52.425702+0000","last_clean":"2024-03-30T15:04:52.425702+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:04:52.425702+0000","last_undegraded":"2024-03-30T15:04:52.425702+0000","last_fullsized":"2024-03-30T15:04:52.425702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":768,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413236+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:04:52.413236+0000","last_peered":"2024-03-30T15:04:52.413236+0000","last_clean":"2024-03-30T15:04:52.413236+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:04:52.413236+0000","last_undegraded":"2024-03-30T15:04:52.413236+0000","last_fullsized":"2024-03-30T15:04:52.413236+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":765,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406107+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:04:52.406107+0000","last_peered":"2024-03-30T15:04:52.406107+0000","last_clean":"2024-03-30T15:04:52.406107+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:04:52.406107+0000","last_undegraded":"2024-03-30T15:04:52.406107+0000","last_fullsized":"2024-03-30T15:04:52.406107+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":905,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406312+0000","last_change":"2024-03-30T15:01:46.424704+0000","last_active":"2024-03-30T15:04:52.406312+0000","last_peered":"2024-03-30T15:04:52.406312+0000","last_clean":"2024-03-30T15:04:52.406312+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:04:52.406312+0000","last_undegraded":"2024-03-30T15:04:52.406312+0000","last_fullsized":"2024-03-30T15:04:52.406312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T15:01:46.424546+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.424546+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":769,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406626+0000","last_change":"2024-03-30T15:04:50.960049+0000","last_active":"2024-03-30T15:04:52.406626+0000","last_peered":"2024-03-30T15:04:52.406626+0000","last_clean":"2024-03-30T15:04:52.406626+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:04:52.406626+0000","last_undegraded":"2024-03-30T15:04:52.406626+0000","last_fullsized":"2024-03-30T15:04:52.406626+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.959968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.959968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.339138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":982,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406402+0000","last_change":"2024-03-30T15:03:22.567558+0000","last_active":"2024-03-30T15:04:52.406402+0000","last_peered":"2024-03-30T15:04:52.406402+0000","last_clean":"2024-03-30T15:04:52.406402+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:04:52.406402+0000","last_undegraded":"2024-03-30T15:04:52.406402+0000","last_fullsized":"2024-03-30T15:04:52.406402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:22.567465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.567465+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":845,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424957+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:04:52.424957+0000","last_peered":"2024-03-30T15:04:52.424957+0000","last_clean":"2024-03-30T15:04:52.424957+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:04:52.424957+0000","last_undegraded":"2024-03-30T15:04:52.424957+0000","last_fullsized":"2024-03-30T15:04:52.424957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:11.636902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391501+0000","last_change":"2024-03-30T15:02:30.902281+0000","last_active":"2024-03-30T15:04:52.391501+0000","last_peered":"2024-03-30T15:04:52.391501+0000","last_clean":"2024-03-30T15:04:52.391501+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:04:52.391501+0000","last_undegraded":"2024-03-30T15:04:52.391501+0000","last_fullsized":"2024-03-30T15:04:52.391501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.902199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.902199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":850,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403045+0000","last_change":"2024-03-30T15:02:39.473396+0000","last_active":"2024-03-30T15:04:52.403045+0000","last_peered":"2024-03-30T15:04:52.403045+0000","last_clean":"2024-03-30T15:04:52.403045+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:04:52.403045+0000","last_undegraded":"2024-03-30T15:04:52.403045+0000","last_fullsized":"2024-03-30T15:04:52.403045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:39.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.473279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994705+0000","last_change":"2024-03-30T15:02:34.958123+0000","last_active":"2024-03-30T15:04:52.994705+0000","last_peered":"2024-03-30T15:04:52.994705+0000","last_clean":"2024-03-30T15:04:52.994705+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:04:52.994705+0000","last_undegraded":"2024-03-30T15:04:52.994705+0000","last_fullsized":"2024-03-30T15:04:52.994705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T15:02:34.958001+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.958001+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":834,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995527+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:04:52.995527+0000","last_peered":"2024-03-30T15:04:52.995527+0000","last_clean":"2024-03-30T15:04:52.995527+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:04:52.995527+0000","last_undegraded":"2024-03-30T15:04:52.995527+0000","last_fullsized":"2024-03-30T15:04:52.995527+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":846,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403412+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:04:52.403412+0000","last_peered":"2024-03-30T15:04:52.403412+0000","last_clean":"2024-03-30T15:04:52.403412+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:04:52.403412+0000","last_undegraded":"2024-03-30T15:04:52.403412+0000","last_fullsized":"2024-03-30T15:04:52.403412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":777,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391402+0000","last_change":"2024-03-30T15:04:48.979522+0000","last_active":"2024-03-30T15:04:52.391402+0000","last_peered":"2024-03-30T15:04:52.391402+0000","last_clean":"2024-03-30T15:04:52.391402+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:04:52.391402+0000","last_undegraded":"2024-03-30T15:04:52.391402+0000","last_fullsized":"2024-03-30T15:04:52.391402+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.979438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.432525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":869,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405577+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:04:52.405577+0000","last_peered":"2024-03-30T15:04:52.405577+0000","last_clean":"2024-03-30T15:04:52.405577+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:04:52.405577+0000","last_undegraded":"2024-03-30T15:04:52.405577+0000","last_fullsized":"2024-03-30T15:04:52.405577+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":904,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403257+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:04:52.403257+0000","last_peered":"2024-03-30T15:04:52.403257+0000","last_clean":"2024-03-30T15:04:52.403257+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:04:52.403257+0000","last_undegraded":"2024-03-30T15:04:52.403257+0000","last_fullsized":"2024-03-30T15:04:52.403257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.978250+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":864,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411142+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:04:52.411142+0000","last_peered":"2024-03-30T15:04:52.411142+0000","last_clean":"2024-03-30T15:04:52.411142+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:04:52.411142+0000","last_undegraded":"2024-03-30T15:04:52.411142+0000","last_fullsized":"2024-03-30T15:04:52.411142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994625+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:04:52.994625+0000","last_peered":"2024-03-30T15:04:52.994625+0000","last_clean":"2024-03-30T15:04:52.994625+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:04:52.994625+0000","last_undegraded":"2024-03-30T15:04:52.994625+0000","last_fullsized":"2024-03-30T15:04:52.994625+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":777,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393700+0000","last_change":"2024-03-30T15:04:47.018205+0000","last_active":"2024-03-30T15:04:52.393700+0000","last_peered":"2024-03-30T15:04:52.393700+0000","last_clean":"2024-03-30T15:04:52.393700+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:04:52.393700+0000","last_undegraded":"2024-03-30T15:04:52.393700+0000","last_fullsized":"2024-03-30T15:04:52.393700+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.018083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.520606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":994,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409695+0000","last_change":"2024-03-30T15:01:48.471009+0000","last_active":"2024-03-30T15:04:52.409695+0000","last_peered":"2024-03-30T15:04:52.409695+0000","last_clean":"2024-03-30T15:04:52.409695+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:04:52.409695+0000","last_undegraded":"2024-03-30T15:04:52.409695+0000","last_fullsized":"2024-03-30T15:04:52.409695+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:48.470884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.470884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":898,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407334+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:04:52.407334+0000","last_peered":"2024-03-30T15:04:52.407334+0000","last_clean":"2024-03-30T15:04:52.407334+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:04:52.407334+0000","last_undegraded":"2024-03-30T15:04:52.407334+0000","last_fullsized":"2024-03-30T15:04:52.407334+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":865,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425095+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:04:52.425095+0000","last_peered":"2024-03-30T15:04:52.425095+0000","last_clean":"2024-03-30T15:04:52.425095+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:04:52.425095+0000","last_undegraded":"2024-03-30T15:04:52.425095+0000","last_fullsized":"2024-03-30T15:04:52.425095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:09.920854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":851,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408503+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:04:52.408503+0000","last_peered":"2024-03-30T15:04:52.408503+0000","last_clean":"2024-03-30T15:04:52.408503+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:04:52.408503+0000","last_undegraded":"2024-03-30T15:04:52.408503+0000","last_fullsized":"2024-03-30T15:04:52.408503+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":770,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.952237+0000","last_change":"2024-03-30T15:04:52.952256+0000","last_active":"2024-03-30T15:04:52.952237+0000","last_peered":"2024-03-30T15:04:52.952237+0000","last_clean":"2024-03-30T15:04:52.952237+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:04:52.952237+0000","last_undegraded":"2024-03-30T15:04:52.952237+0000","last_fullsized":"2024-03-30T15:04:52.952237+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.952177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.313787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412592+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:04:52.412592+0000","last_peered":"2024-03-30T15:04:52.412592+0000","last_clean":"2024-03-30T15:04:52.412592+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:04:52.412592+0000","last_undegraded":"2024-03-30T15:04:52.412592+0000","last_fullsized":"2024-03-30T15:04:52.412592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.848800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":917,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410075+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:04:52.410075+0000","last_peered":"2024-03-30T15:04:52.410075+0000","last_clean":"2024-03-30T15:04:52.410075+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:04:52.410075+0000","last_undegraded":"2024-03-30T15:04:52.410075+0000","last_fullsized":"2024-03-30T15:04:52.410075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.338727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":875,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407266+0000","last_change":"2024-03-30T15:04:51.927488+0000","last_active":"2024-03-30T15:04:52.407266+0000","last_peered":"2024-03-30T15:04:52.407266+0000","last_clean":"2024-03-30T15:04:52.407266+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:04:52.407266+0000","last_undegraded":"2024-03-30T15:04:52.407266+0000","last_fullsized":"2024-03-30T15:04:52.407266+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.927412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.927412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.838858+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405733+0000","last_change":"2024-03-30T15:01:54.450014+0000","last_active":"2024-03-30T15:04:52.405733+0000","last_peered":"2024-03-30T15:04:52.405733+0000","last_clean":"2024-03-30T15:04:52.405733+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:04:52.405733+0000","last_undegraded":"2024-03-30T15:04:52.405733+0000","last_fullsized":"2024-03-30T15:04:52.405733+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.449931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.449931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411136+0000","last_change":"2024-03-30T15:01:56.570174+0000","last_active":"2024-03-30T15:04:52.411136+0000","last_peered":"2024-03-30T15:04:52.411136+0000","last_clean":"2024-03-30T15:04:52.411136+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:04:52.411136+0000","last_undegraded":"2024-03-30T15:04:52.411136+0000","last_fullsized":"2024-03-30T15:04:52.411136+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:56.570063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:56.570063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":857,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425760+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:04:52.425760+0000","last_peered":"2024-03-30T15:04:52.425760+0000","last_clean":"2024-03-30T15:04:52.425760+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:04:52.425760+0000","last_undegraded":"2024-03-30T15:04:52.425760+0000","last_fullsized":"2024-03-30T15:04:52.425760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.978863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":902,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411030+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:04:52.411030+0000","last_peered":"2024-03-30T15:04:52.411030+0000","last_clean":"2024-03-30T15:04:52.411030+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:04:52.411030+0000","last_undegraded":"2024-03-30T15:04:52.411030+0000","last_fullsized":"2024-03-30T15:04:52.411030+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.711973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1489,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994368+0000","last_change":"2024-03-30T15:04:50.115333+0000","last_active":"2024-03-30T15:04:52.994368+0000","last_peered":"2024-03-30T15:04:52.994368+0000","last_clean":"2024-03-30T15:04:52.994368+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:04:52.994368+0000","last_undegraded":"2024-03-30T15:04:52.994368+0000","last_fullsized":"2024-03-30T15:04:52.994368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.115166+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.947111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":608,"num_read_kb":603,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":862,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408767+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:04:52.408767+0000","last_peered":"2024-03-30T15:04:52.408767+0000","last_clean":"2024-03-30T15:04:52.408767+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:04:52.408767+0000","last_undegraded":"2024-03-30T15:04:52.408767+0000","last_fullsized":"2024-03-30T15:04:52.408767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406526+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:04:52.406526+0000","last_peered":"2024-03-30T15:04:52.406526+0000","last_clean":"2024-03-30T15:04:52.406526+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:04:52.406526+0000","last_undegraded":"2024-03-30T15:04:52.406526+0000","last_fullsized":"2024-03-30T15:04:52.406526+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.928596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":764,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995477+0000","last_change":"2024-03-30T15:04:37.123232+0000","last_active":"2024-03-30T15:04:52.995477+0000","last_peered":"2024-03-30T15:04:52.995477+0000","last_clean":"2024-03-30T15:04:52.995477+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:04:52.995477+0000","last_undegraded":"2024-03-30T15:04:52.995477+0000","last_fullsized":"2024-03-30T15:04:52.995477+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.957564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":851,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994887+0000","last_change":"2024-03-30T15:02:33.948293+0000","last_active":"2024-03-30T15:04:52.994887+0000","last_peered":"2024-03-30T15:04:52.994887+0000","last_clean":"2024-03-30T15:04:52.994887+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:04:52.994887+0000","last_undegraded":"2024-03-30T15:04:52.994887+0000","last_fullsized":"2024-03-30T15:04:52.994887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:33.948156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.948156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1034,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413418+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:04:52.413418+0000","last_peered":"2024-03-30T15:04:52.413418+0000","last_clean":"2024-03-30T15:04:52.413418+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:04:52.413418+0000","last_undegraded":"2024-03-30T15:04:52.413418+0000","last_fullsized":"2024-03-30T15:04:52.413418+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423951+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:04:52.423951+0000","last_peered":"2024-03-30T15:04:52.423951+0000","last_clean":"2024-03-30T15:04:52.423951+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:04:52.423951+0000","last_undegraded":"2024-03-30T15:04:52.423951+0000","last_fullsized":"2024-03-30T15:04:52.423951+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.548386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1121,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994408+0000","last_change":"2024-03-30T15:02:39.890565+0000","last_active":"2024-03-30T15:04:52.994408+0000","last_peered":"2024-03-30T15:04:52.994408+0000","last_clean":"2024-03-30T15:04:52.994408+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:04:52.994408+0000","last_undegraded":"2024-03-30T15:04:52.994408+0000","last_fullsized":"2024-03-30T15:04:52.994408+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:02:39.890462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.890462+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":845,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425513+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:04:52.425513+0000","last_peered":"2024-03-30T15:04:52.425513+0000","last_clean":"2024-03-30T15:04:52.425513+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:04:52.425513+0000","last_undegraded":"2024-03-30T15:04:52.425513+0000","last_fullsized":"2024-03-30T15:04:52.425513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.463993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":772,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413929+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:04:52.413929+0000","last_peered":"2024-03-30T15:04:52.413929+0000","last_clean":"2024-03-30T15:04:52.413929+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:04:52.413929+0000","last_undegraded":"2024-03-30T15:04:52.413929+0000","last_fullsized":"2024-03-30T15:04:52.413929+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393134+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:04:52.393134+0000","last_peered":"2024-03-30T15:04:52.393134+0000","last_clean":"2024-03-30T15:04:52.393134+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:04:52.393134+0000","last_undegraded":"2024-03-30T15:04:52.393134+0000","last_fullsized":"2024-03-30T15:04:52.393134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.873816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":909,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406015+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:04:52.406015+0000","last_peered":"2024-03-30T15:04:52.406015+0000","last_clean":"2024-03-30T15:04:52.406015+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:04:52.406015+0000","last_undegraded":"2024-03-30T15:04:52.406015+0000","last_fullsized":"2024-03-30T15:04:52.406015+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:54.532654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":877,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391540+0000","last_change":"2024-03-30T15:01:53.529768+0000","last_active":"2024-03-30T15:04:52.391540+0000","last_peered":"2024-03-30T15:04:52.391540+0000","last_clean":"2024-03-30T15:04:52.391540+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:04:52.391540+0000","last_undegraded":"2024-03-30T15:04:52.391540+0000","last_fullsized":"2024-03-30T15:04:52.391540+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.529037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:49.700022+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.529037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405607+0000","last_change":"2024-03-30T15:01:59.392426+0000","last_active":"2024-03-30T15:04:52.405607+0000","last_peered":"2024-03-30T15:04:52.405607+0000","last_clean":"2024-03-30T15:04:52.405607+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:04:52.405607+0000","last_undegraded":"2024-03-30T15:04:52.405607+0000","last_fullsized":"2024-03-30T15:04:52.405607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:59.392341+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.392341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995630+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:04:52.995630+0000","last_peered":"2024-03-30T15:04:52.995630+0000","last_clean":"2024-03-30T15:04:52.995630+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:04:52.995630+0000","last_undegraded":"2024-03-30T15:04:52.995630+0000","last_fullsized":"2024-03-30T15:04:52.995630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.756028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409962+0000","last_change":"2024-03-30T15:01:59.537303+0000","last_active":"2024-03-30T15:04:52.409962+0000","last_peered":"2024-03-30T15:04:52.409962+0000","last_clean":"2024-03-30T15:04:52.409962+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:04:52.409962+0000","last_undegraded":"2024-03-30T15:04:52.409962+0000","last_fullsized":"2024-03-30T15:04:52.409962+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.537225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.537225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":862,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:53.572029+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:04:53.572029+0000","last_peered":"2024-03-30T15:04:53.572029+0000","last_clean":"2024-03-30T15:04:53.572029+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:04:53.572029+0000","last_undegraded":"2024-03-30T15:04:53.572029+0000","last_fullsized":"2024-03-30T15:04:53.572029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.100823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":998,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424182+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:04:52.424182+0000","last_peered":"2024-03-30T15:04:52.424182+0000","last_clean":"2024-03-30T15:04:52.424182+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:04:52.424182+0000","last_undegraded":"2024-03-30T15:04:52.424182+0000","last_fullsized":"2024-03-30T15:04:52.424182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:53.505173+0000","last_change":"2024-03-30T15:04:53.505197+0000","last_active":"2024-03-30T15:04:53.505173+0000","last_peered":"2024-03-30T15:04:53.505173+0000","last_clean":"2024-03-30T15:04:53.505173+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:04:53.505173+0000","last_undegraded":"2024-03-30T15:04:53.505173+0000","last_fullsized":"2024-03-30T15:04:53.505173+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.505107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.505107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.825117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423963+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:04:52.423963+0000","last_peered":"2024-03-30T15:04:52.423963+0000","last_clean":"2024-03-30T15:04:52.423963+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:04:52.423963+0000","last_undegraded":"2024-03-30T15:04:52.423963+0000","last_fullsized":"2024-03-30T15:04:52.423963+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404807+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:04:52.404807+0000","last_peered":"2024-03-30T15:04:52.404807+0000","last_clean":"2024-03-30T15:04:52.404807+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:04:52.404807+0000","last_undegraded":"2024-03-30T15:04:52.404807+0000","last_fullsized":"2024-03-30T15:04:52.404807+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409919+0000","last_change":"2024-03-30T15:01:58.563335+0000","last_active":"2024-03-30T15:04:52.409919+0000","last_peered":"2024-03-30T15:04:52.409919+0000","last_clean":"2024-03-30T15:04:52.409919+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:04:52.409919+0000","last_undegraded":"2024-03-30T15:04:52.409919+0000","last_fullsized":"2024-03-30T15:04:52.409919+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":891,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409769+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:04:52.409769+0000","last_peered":"2024-03-30T15:04:52.409769+0000","last_clean":"2024-03-30T15:04:52.409769+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:04:52.409769+0000","last_undegraded":"2024-03-30T15:04:52.409769+0000","last_fullsized":"2024-03-30T15:04:52.409769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:00.062419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":865,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424574+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:04:52.424574+0000","last_peered":"2024-03-30T15:04:52.424574+0000","last_clean":"2024-03-30T15:04:52.424574+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:04:52.424574+0000","last_undegraded":"2024-03-30T15:04:52.424574+0000","last_fullsized":"2024-03-30T15:04:52.424574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.634280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393865+0000","last_change":"2024-03-30T15:03:26.133453+0000","last_active":"2024-03-30T15:04:52.393865+0000","last_peered":"2024-03-30T15:04:52.393865+0000","last_clean":"2024-03-30T15:04:52.393865+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:04:52.393865+0000","last_undegraded":"2024-03-30T15:04:52.393865+0000","last_fullsized":"2024-03-30T15:04:52.393865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:26.133342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:26.133342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":850,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409090+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:04:52.409090+0000","last_peered":"2024-03-30T15:04:52.409090+0000","last_clean":"2024-03-30T15:04:52.409090+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:04:52.409090+0000","last_undegraded":"2024-03-30T15:04:52.409090+0000","last_fullsized":"2024-03-30T15:04:52.409090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404228+0000","last_change":"2024-03-30T15:04:46.590243+0000","last_active":"2024-03-30T15:04:52.404228+0000","last_peered":"2024-03-30T15:04:52.404228+0000","last_clean":"2024-03-30T15:04:52.404228+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:04:52.404228+0000","last_undegraded":"2024-03-30T15:04:52.404228+0000","last_fullsized":"2024-03-30T15:04:52.404228+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:46.590171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.590171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.642800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":951,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412534+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:04:52.412534+0000","last_peered":"2024-03-30T15:04:52.412534+0000","last_clean":"2024-03-30T15:04:52.412534+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:04:52.412534+0000","last_undegraded":"2024-03-30T15:04:52.412534+0000","last_fullsized":"2024-03-30T15:04:52.412534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995532+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:04:52.995532+0000","last_peered":"2024-03-30T15:04:52.995532+0000","last_clean":"2024-03-30T15:04:52.995532+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:04:52.995532+0000","last_undegraded":"2024-03-30T15:04:52.995532+0000","last_fullsized":"2024-03-30T15:04:52.995532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.395772+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:04:52.395772+0000","last_peered":"2024-03-30T15:04:52.395772+0000","last_clean":"2024-03-30T15:04:52.395772+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:04:52.395772+0000","last_undegraded":"2024-03-30T15:04:52.395772+0000","last_fullsized":"2024-03-30T15:04:52.395772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.803620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":847,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994253+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:04:52.994253+0000","last_peered":"2024-03-30T15:04:52.994253+0000","last_clean":"2024-03-30T15:04:52.994253+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:04:52.994253+0000","last_undegraded":"2024-03-30T15:04:52.994253+0000","last_fullsized":"2024-03-30T15:04:52.994253+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":785,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412346+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:04:52.412346+0000","last_peered":"2024-03-30T15:04:52.412346+0000","last_clean":"2024-03-30T15:04:52.412346+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:04:52.412346+0000","last_undegraded":"2024-03-30T15:04:52.412346+0000","last_fullsized":"2024-03-30T15:04:52.412346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391224+0000","last_change":"2024-03-30T15:04:49.959125+0000","last_active":"2024-03-30T15:04:52.391224+0000","last_peered":"2024-03-30T15:04:52.391224+0000","last_clean":"2024-03-30T15:04:52.391224+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:04:52.391224+0000","last_undegraded":"2024-03-30T15:04:52.391224+0000","last_fullsized":"2024-03-30T15:04:52.391224+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.959020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.959020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.221503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":894,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403307+0000","last_change":"2024-03-30T15:03:19.519526+0000","last_active":"2024-03-30T15:04:52.403307+0000","last_peered":"2024-03-30T15:04:52.403307+0000","last_clean":"2024-03-30T15:04:52.403307+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:04:52.403307+0000","last_undegraded":"2024-03-30T15:04:52.403307+0000","last_fullsized":"2024-03-30T15:04:52.403307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:03:19.519440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.519440+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":867,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995507+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:04:52.995507+0000","last_peered":"2024-03-30T15:04:52.995507+0000","last_clean":"2024-03-30T15:04:52.995507+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:04:52.995507+0000","last_undegraded":"2024-03-30T15:04:52.995507+0000","last_fullsized":"2024-03-30T15:04:52.995507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.756007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994820+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:04:52.994820+0000","last_peered":"2024-03-30T15:04:52.994820+0000","last_clean":"2024-03-30T15:04:52.994820+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:04:52.994820+0000","last_undegraded":"2024-03-30T15:04:52.994820+0000","last_fullsized":"2024-03-30T15:04:52.994820+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.943192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.402919+0000","last_change":"2024-03-30T15:04:37.589562+0000","last_active":"2024-03-30T15:04:52.402919+0000","last_peered":"2024-03-30T15:04:52.402919+0000","last_clean":"2024-03-30T15:04:52.402919+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:04:52.402919+0000","last_undegraded":"2024-03-30T15:04:52.402919+0000","last_fullsized":"2024-03-30T15:04:52.402919+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:50.917173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":983,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994900+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:04:52.994900+0000","last_peered":"2024-03-30T15:04:52.994900+0000","last_clean":"2024-03-30T15:04:52.994900+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:04:52.994900+0000","last_undegraded":"2024-03-30T15:04:52.994900+0000","last_fullsized":"2024-03-30T15:04:52.994900+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":896,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423512+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:04:52.423512+0000","last_peered":"2024-03-30T15:04:52.423512+0000","last_clean":"2024-03-30T15:04:52.423512+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:04:52.423512+0000","last_undegraded":"2024-03-30T15:04:52.423512+0000","last_fullsized":"2024-03-30T15:04:52.423512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":879,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995650+0000","last_change":"2024-03-30T15:04:44.117459+0000","last_active":"2024-03-30T15:04:52.995650+0000","last_peered":"2024-03-30T15:04:52.995650+0000","last_clean":"2024-03-30T15:04:52.995650+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:04:52.995650+0000","last_undegraded":"2024-03-30T15:04:52.995650+0000","last_fullsized":"2024-03-30T15:04:52.995650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:04:44.117358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.117358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.327955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":845,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424139+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:04:52.424139+0000","last_peered":"2024-03-30T15:04:52.424139+0000","last_clean":"2024-03-30T15:04:52.424139+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:04:52.424139+0000","last_undegraded":"2024-03-30T15:04:52.424139+0000","last_fullsized":"2024-03-30T15:04:52.424139+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":772,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413865+0000","last_change":"2024-03-30T15:04:49.244698+0000","last_active":"2024-03-30T15:04:52.413865+0000","last_peered":"2024-03-30T15:04:52.413865+0000","last_clean":"2024-03-30T15:04:52.413865+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:04:52.413865+0000","last_undegraded":"2024-03-30T15:04:52.413865+0000","last_fullsized":"2024-03-30T15:04:52.413865+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.244616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.244616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:56.099737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":447,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.396180+0000","last_change":"2024-03-30T15:03:52.115510+0000","last_active":"2024-03-30T15:04:52.396180+0000","last_peered":"2024-03-30T15:04:52.396180+0000","last_clean":"2024-03-30T15:04:52.396180+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:04:52.396180+0000","last_undegraded":"2024-03-30T15:04:52.396180+0000","last_fullsized":"2024-03-30T15:04:52.396180+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'6","last_scrub_stamp":"2024-03-30T15:03:52.115417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.115417+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.716598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":874,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412189+0000","last_change":"2024-03-30T15:02:45.427403+0000","last_active":"2024-03-30T15:04:52.412189+0000","last_peered":"2024-03-30T15:04:52.412189+0000","last_clean":"2024-03-30T15:04:52.412189+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:04:52.412189+0000","last_undegraded":"2024-03-30T15:04:52.412189+0000","last_fullsized":"2024-03-30T15:04:52.412189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:02:45.427234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":849,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425193+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:04:52.425193+0000","last_peered":"2024-03-30T15:04:52.425193+0000","last_clean":"2024-03-30T15:04:52.425193+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:04:52.425193+0000","last_undegraded":"2024-03-30T15:04:52.425193+0000","last_fullsized":"2024-03-30T15:04:52.425193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":847,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995118+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:04:52.995118+0000","last_peered":"2024-03-30T15:04:52.995118+0000","last_clean":"2024-03-30T15:04:52.995118+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:04:52.995118+0000","last_undegraded":"2024-03-30T15:04:52.995118+0000","last_fullsized":"2024-03-30T15:04:52.995118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:02.682878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404740+0000","last_change":"2024-03-30T15:04:44.618218+0000","last_active":"2024-03-30T15:04:52.404740+0000","last_peered":"2024-03-30T15:04:52.404740+0000","last_clean":"2024-03-30T15:04:52.404740+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:04:52.404740+0000","last_undegraded":"2024-03-30T15:04:52.404740+0000","last_fullsized":"2024-03-30T15:04:52.404740+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.618135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.618135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.110883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":883,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411251+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:04:52.411251+0000","last_peered":"2024-03-30T15:04:52.411251+0000","last_clean":"2024-03-30T15:04:52.411251+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:04:52.411251+0000","last_undegraded":"2024-03-30T15:04:52.411251+0000","last_fullsized":"2024-03-30T15:04:52.411251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.283455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":867,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994703+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:04:52.994703+0000","last_peered":"2024-03-30T15:04:52.994703+0000","last_clean":"2024-03-30T15:04:52.994703+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:04:52.994703+0000","last_undegraded":"2024-03-30T15:04:52.994703+0000","last_fullsized":"2024-03-30T15:04:52.994703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:30.940258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":767,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406693+0000","last_change":"2024-03-30T15:04:03.988041+0000","last_active":"2024-03-30T15:04:52.406693+0000","last_peered":"2024-03-30T15:04:52.406693+0000","last_clean":"2024-03-30T15:04:52.406693+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:04:52.406693+0000","last_undegraded":"2024-03-30T15:04:52.406693+0000","last_fullsized":"2024-03-30T15:04:52.406693+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.333733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":864,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.395192+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:04:52.395192+0000","last_peered":"2024-03-30T15:04:52.395192+0000","last_clean":"2024-03-30T15:04:52.395192+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:04:52.395192+0000","last_undegraded":"2024-03-30T15:04:52.395192+0000","last_fullsized":"2024-03-30T15:04:52.395192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.180120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409892+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:04:52.409892+0000","last_peered":"2024-03-30T15:04:52.409892+0000","last_clean":"2024-03-30T15:04:52.409892+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:04:52.409892+0000","last_undegraded":"2024-03-30T15:04:52.409892+0000","last_fullsized":"2024-03-30T15:04:52.409892+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":878,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425320+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:04:52.425320+0000","last_peered":"2024-03-30T15:04:52.425320+0000","last_clean":"2024-03-30T15:04:52.425320+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:04:52.425320+0000","last_undegraded":"2024-03-30T15:04:52.425320+0000","last_fullsized":"2024-03-30T15:04:52.425320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":870,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409268+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:04:52.409268+0000","last_peered":"2024-03-30T15:04:52.409268+0000","last_clean":"2024-03-30T15:04:52.409268+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:04:52.409268+0000","last_undegraded":"2024-03-30T15:04:52.409268+0000","last_fullsized":"2024-03-30T15:04:52.409268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.659468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":868,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994765+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:04:52.994765+0000","last_peered":"2024-03-30T15:04:52.994765+0000","last_clean":"2024-03-30T15:04:52.994765+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:04:52.994765+0000","last_undegraded":"2024-03-30T15:04:52.994765+0000","last_fullsized":"2024-03-30T15:04:52.994765+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":846,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406440+0000","last_change":"2024-03-30T15:04:35.553690+0000","last_active":"2024-03-30T15:04:52.406440+0000","last_peered":"2024-03-30T15:04:52.406440+0000","last_clean":"2024-03-30T15:04:52.406440+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:04:52.406440+0000","last_undegraded":"2024-03-30T15:04:52.406440+0000","last_fullsized":"2024-03-30T15:04:52.406440+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.553605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.553605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.879462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":759,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409869+0000","last_change":"2024-03-30T15:01:52.471709+0000","last_active":"2024-03-30T15:04:52.409869+0000","last_peered":"2024-03-30T15:04:52.409869+0000","last_clean":"2024-03-30T15:04:52.409869+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:04:52.409869+0000","last_undegraded":"2024-03-30T15:04:52.409869+0000","last_fullsized":"2024-03-30T15:04:52.409869+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.471603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.471603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":816,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407370+0000","last_change":"2024-03-30T15:04:38.199333+0000","last_active":"2024-03-30T15:04:52.407370+0000","last_peered":"2024-03-30T15:04:52.407370+0000","last_clean":"2024-03-30T15:04:52.407370+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:04:52.407370+0000","last_undegraded":"2024-03-30T15:04:52.407370+0000","last_fullsized":"2024-03-30T15:04:52.407370+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:04:38.199239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.199239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.783575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994483+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:04:52.994483+0000","last_peered":"2024-03-30T15:04:52.994483+0000","last_clean":"2024-03-30T15:04:52.994483+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:04:52.994483+0000","last_undegraded":"2024-03-30T15:04:52.994483+0000","last_fullsized":"2024-03-30T15:04:52.994483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":984,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425134+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:04:52.425134+0000","last_peered":"2024-03-30T15:04:52.425134+0000","last_clean":"2024-03-30T15:04:52.425134+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:04:52.425134+0000","last_undegraded":"2024-03-30T15:04:52.425134+0000","last_fullsized":"2024-03-30T15:04:52.425134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:56.083796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":852,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411000+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:04:52.411000+0000","last_peered":"2024-03-30T15:04:52.411000+0000","last_clean":"2024-03-30T15:04:52.411000+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:04:52.411000+0000","last_undegraded":"2024-03-30T15:04:52.411000+0000","last_fullsized":"2024-03-30T15:04:52.411000+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:01.685331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403700+0000","last_change":"2024-03-30T15:04:41.620132+0000","last_active":"2024-03-30T15:04:52.403700+0000","last_peered":"2024-03-30T15:04:52.403700+0000","last_clean":"2024-03-30T15:04:52.403700+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:04:52.403700+0000","last_undegraded":"2024-03-30T15:04:52.403700+0000","last_fullsized":"2024-03-30T15:04:52.403700+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:41.620027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:41.620027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.996471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":890,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403711+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:04:52.403711+0000","last_peered":"2024-03-30T15:04:52.403711+0000","last_clean":"2024-03-30T15:04:52.403711+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:04:52.403711+0000","last_undegraded":"2024-03-30T15:04:52.403711+0000","last_fullsized":"2024-03-30T15:04:52.403711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.342982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404234+0000","last_change":"2024-03-30T15:04:43.577081+0000","last_active":"2024-03-30T15:04:52.404234+0000","last_peered":"2024-03-30T15:04:52.404234+0000","last_clean":"2024-03-30T15:04:52.404234+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:04:52.404234+0000","last_undegraded":"2024-03-30T15:04:52.404234+0000","last_fullsized":"2024-03-30T15:04:52.404234+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.576993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.576993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.972808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407323+0000","last_change":"2024-03-30T15:02:03.477727+0000","last_active":"2024-03-30T15:04:52.407323+0000","last_peered":"2024-03-30T15:04:52.407323+0000","last_clean":"2024-03-30T15:04:52.407323+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:04:52.407323+0000","last_undegraded":"2024-03-30T15:04:52.407323+0000","last_fullsized":"2024-03-30T15:04:52.407323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.477653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.477653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409282+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:04:52.409282+0000","last_peered":"2024-03-30T15:04:52.409282+0000","last_clean":"2024-03-30T15:04:52.409282+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:04:52.409282+0000","last_undegraded":"2024-03-30T15:04:52.409282+0000","last_fullsized":"2024-03-30T15:04:52.409282+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.074178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403078+0000","last_change":"2024-03-30T15:04:40.601679+0000","last_active":"2024-03-30T15:04:52.403078+0000","last_peered":"2024-03-30T15:04:52.403078+0000","last_clean":"2024-03-30T15:04:52.403078+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:04:52.403078+0000","last_undegraded":"2024-03-30T15:04:52.403078+0000","last_fullsized":"2024-03-30T15:04:52.403078+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.601541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.601541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":880,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.407408+0000","last_change":"2024-03-30T15:02:00.404994+0000","last_active":"2024-03-30T15:04:52.407408+0000","last_peered":"2024-03-30T15:04:52.407408+0000","last_clean":"2024-03-30T15:04:52.407408+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:04:52.407408+0000","last_undegraded":"2024-03-30T15:04:52.407408+0000","last_fullsized":"2024-03-30T15:04:52.407408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:02:00.404907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.404907+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":882,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.995419+0000","last_change":"2024-03-30T15:04:52.183278+0000","last_active":"2024-03-30T15:04:52.995419+0000","last_peered":"2024-03-30T15:04:52.995419+0000","last_clean":"2024-03-30T15:04:52.995419+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:04:52.995419+0000","last_undegraded":"2024-03-30T15:04:52.995419+0000","last_fullsized":"2024-03-30T15:04:52.995419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:04:52.183194+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.183194+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.754133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403245+0000","last_change":"2024-03-30T15:04:37.298232+0000","last_active":"2024-03-30T15:04:52.403245+0000","last_peered":"2024-03-30T15:04:52.403245+0000","last_clean":"2024-03-30T15:04:52.403245+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:04:52.403245+0000","last_undegraded":"2024-03-30T15:04:52.403245+0000","last_fullsized":"2024-03-30T15:04:52.403245+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:05.561872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":850,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408475+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:04:52.408475+0000","last_peered":"2024-03-30T15:04:52.408475+0000","last_clean":"2024-03-30T15:04:52.408475+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:04:52.408475+0000","last_undegraded":"2024-03-30T15:04:52.408475+0000","last_fullsized":"2024-03-30T15:04:52.408475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":767,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408558+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:04:52.408558+0000","last_peered":"2024-03-30T15:04:52.408558+0000","last_clean":"2024-03-30T15:04:52.408558+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:04:52.408558+0000","last_undegraded":"2024-03-30T15:04:52.408558+0000","last_fullsized":"2024-03-30T15:04:52.408558+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.076034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":809,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994789+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:04:52.994789+0000","last_peered":"2024-03-30T15:04:52.994789+0000","last_clean":"2024-03-30T15:04:52.994789+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:04:52.994789+0000","last_undegraded":"2024-03-30T15:04:52.994789+0000","last_fullsized":"2024-03-30T15:04:52.994789+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":857,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.425308+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:04:52.425308+0000","last_peered":"2024-03-30T15:04:52.425308+0000","last_clean":"2024-03-30T15:04:52.425308+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:04:52.425308+0000","last_undegraded":"2024-03-30T15:04:52.425308+0000","last_fullsized":"2024-03-30T15:04:52.425308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":851,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994602+0000","last_change":"2024-03-30T15:02:41.903716+0000","last_active":"2024-03-30T15:04:52.994602+0000","last_peered":"2024-03-30T15:04:52.994602+0000","last_clean":"2024-03-30T15:04:52.994602+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:04:52.994602+0000","last_undegraded":"2024-03-30T15:04:52.994602+0000","last_fullsized":"2024-03-30T15:04:52.994602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.903609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.903609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":762,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409531+0000","last_change":"2024-03-30T15:04:40.631484+0000","last_active":"2024-03-30T15:04:52.409531+0000","last_peered":"2024-03-30T15:04:52.409531+0000","last_clean":"2024-03-30T15:04:52.409531+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:04:52.409531+0000","last_undegraded":"2024-03-30T15:04:52.409531+0000","last_fullsized":"2024-03-30T15:04:52.409531+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.631340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.631340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.585112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":764,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405720+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:04:52.405720+0000","last_peered":"2024-03-30T15:04:52.405720+0000","last_clean":"2024-03-30T15:04:52.405720+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:04:52.405720+0000","last_undegraded":"2024-03-30T15:04:52.405720+0000","last_fullsized":"2024-03-30T15:04:52.405720+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":433,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.392410+0000","last_change":"2024-03-30T15:04:01.043987+0000","last_active":"2024-03-30T15:04:52.392410+0000","last_peered":"2024-03-30T15:04:52.392410+0000","last_clean":"2024-03-30T15:04:52.392410+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:04:52.392410+0000","last_undegraded":"2024-03-30T15:04:52.392410+0000","last_fullsized":"2024-03-30T15:04:52.392410+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:04:01.043885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.043885+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.133940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":870,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410782+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:04:52.410782+0000","last_peered":"2024-03-30T15:04:52.410782+0000","last_clean":"2024-03-30T15:04:52.410782+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:04:52.410782+0000","last_undegraded":"2024-03-30T15:04:52.410782+0000","last_fullsized":"2024-03-30T15:04:52.410782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.517062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":866,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413050+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:04:52.413050+0000","last_peered":"2024-03-30T15:04:52.413050+0000","last_clean":"2024-03-30T15:04:52.413050+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:04:52.413050+0000","last_undegraded":"2024-03-30T15:04:52.413050+0000","last_fullsized":"2024-03-30T15:04:52.413050+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":847,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994500+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:04:52.994500+0000","last_peered":"2024-03-30T15:04:52.994500+0000","last_clean":"2024-03-30T15:04:52.994500+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:04:52.994500+0000","last_undegraded":"2024-03-30T15:04:52.994500+0000","last_fullsized":"2024-03-30T15:04:52.994500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":771,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.414058+0000","last_change":"2024-03-30T15:04:51.176640+0000","last_active":"2024-03-30T15:04:52.414058+0000","last_peered":"2024-03-30T15:04:52.414058+0000","last_clean":"2024-03-30T15:04:52.414058+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:04:52.414058+0000","last_undegraded":"2024-03-30T15:04:52.414058+0000","last_fullsized":"2024-03-30T15:04:52.414058+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.176549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.176549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.079483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":898,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.414112+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:04:52.414112+0000","last_peered":"2024-03-30T15:04:52.414112+0000","last_clean":"2024-03-30T15:04:52.414112+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:04:52.414112+0000","last_undegraded":"2024-03-30T15:04:52.414112+0000","last_fullsized":"2024-03-30T15:04:52.414112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408633+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:04:52.408633+0000","last_peered":"2024-03-30T15:04:52.408633+0000","last_clean":"2024-03-30T15:04:52.408633+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:04:52.408633+0000","last_undegraded":"2024-03-30T15:04:52.408633+0000","last_fullsized":"2024-03-30T15:04:52.408633+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.782926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411352+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:04:52.411352+0000","last_peered":"2024-03-30T15:04:52.411352+0000","last_clean":"2024-03-30T15:04:52.411352+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:04:52.411352+0000","last_undegraded":"2024-03-30T15:04:52.411352+0000","last_fullsized":"2024-03-30T15:04:52.411352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":855,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406535+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:04:52.406535+0000","last_peered":"2024-03-30T15:04:52.406535+0000","last_clean":"2024-03-30T15:04:52.406535+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:04:52.406535+0000","last_undegraded":"2024-03-30T15:04:52.406535+0000","last_fullsized":"2024-03-30T15:04:52.406535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":771,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413721+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:04:52.413721+0000","last_peered":"2024-03-30T15:04:52.413721+0000","last_clean":"2024-03-30T15:04:52.413721+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:04:52.413721+0000","last_undegraded":"2024-03-30T15:04:52.413721+0000","last_fullsized":"2024-03-30T15:04:52.413721+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":894,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.411245+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:04:52.411245+0000","last_peered":"2024-03-30T15:04:52.411245+0000","last_clean":"2024-03-30T15:04:52.411245+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:04:52.411245+0000","last_undegraded":"2024-03-30T15:04:52.411245+0000","last_fullsized":"2024-03-30T15:04:52.411245+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":860,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405930+0000","last_change":"2024-03-30T15:01:47.383230+0000","last_active":"2024-03-30T15:04:52.405930+0000","last_peered":"2024-03-30T15:04:52.405930+0000","last_clean":"2024-03-30T15:04:52.405930+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:04:52.405930+0000","last_undegraded":"2024-03-30T15:04:52.405930+0000","last_fullsized":"2024-03-30T15:04:52.405930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:01:47.383148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.383148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1001,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.414009+0000","last_change":"2024-03-30T15:02:51.451850+0000","last_active":"2024-03-30T15:04:52.414009+0000","last_peered":"2024-03-30T15:04:52.414009+0000","last_clean":"2024-03-30T15:04:52.414009+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:04:52.414009+0000","last_undegraded":"2024-03-30T15:04:52.414009+0000","last_fullsized":"2024-03-30T15:04:52.414009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:51.451690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:51.451690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":857,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994740+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:04:52.994740+0000","last_peered":"2024-03-30T15:04:52.994740+0000","last_clean":"2024-03-30T15:04:52.994740+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:04:52.994740+0000","last_undegraded":"2024-03-30T15:04:52.994740+0000","last_fullsized":"2024-03-30T15:04:52.994740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:07.241016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":771,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.412989+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:04:52.412989+0000","last_peered":"2024-03-30T15:04:52.412989+0000","last_clean":"2024-03-30T15:04:52.412989+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:04:52.412989+0000","last_undegraded":"2024-03-30T15:04:52.412989+0000","last_fullsized":"2024-03-30T15:04:52.412989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:17.458537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":818,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406193+0000","last_change":"2024-03-30T15:04:40.214721+0000","last_active":"2024-03-30T15:04:52.406193+0000","last_peered":"2024-03-30T15:04:52.406193+0000","last_clean":"2024-03-30T15:04:52.406193+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:04:52.406193+0000","last_undegraded":"2024-03-30T15:04:52.406193+0000","last_fullsized":"2024-03-30T15:04:52.406193+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:40.214633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.214633+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:09.122719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":858,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403562+0000","last_change":"2024-03-30T15:03:21.522112+0000","last_active":"2024-03-30T15:04:52.403562+0000","last_peered":"2024-03-30T15:04:52.403562+0000","last_clean":"2024-03-30T15:04:52.403562+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:04:52.403562+0000","last_undegraded":"2024-03-30T15:04:52.403562+0000","last_fullsized":"2024-03-30T15:04:52.403562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.522038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.522038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":850,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404495+0000","last_change":"2024-03-30T15:03:23.551230+0000","last_active":"2024-03-30T15:04:52.404495+0000","last_peered":"2024-03-30T15:04:52.404495+0000","last_clean":"2024-03-30T15:04:52.404495+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:04:52.404495+0000","last_undegraded":"2024-03-30T15:04:52.404495+0000","last_fullsized":"2024-03-30T15:04:52.404495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:23.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":768,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.414194+0000","last_change":"2024-03-30T15:02:49.425842+0000","last_active":"2024-03-30T15:04:52.414194+0000","last_peered":"2024-03-30T15:04:52.414194+0000","last_clean":"2024-03-30T15:04:52.414194+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:04:52.414194+0000","last_undegraded":"2024-03-30T15:04:52.414194+0000","last_fullsized":"2024-03-30T15:04:52.414194+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:49.425762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.425762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424690+0000","last_change":"2024-03-30T15:04:36.274480+0000","last_active":"2024-03-30T15:04:52.424690+0000","last_peered":"2024-03-30T15:04:52.424690+0000","last_clean":"2024-03-30T15:04:52.424690+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:04:52.424690+0000","last_undegraded":"2024-03-30T15:04:52.424690+0000","last_fullsized":"2024-03-30T15:04:52.424690+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.274391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.274391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:57.565878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":922,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.394789+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:04:52.394789+0000","last_peered":"2024-03-30T15:04:52.394789+0000","last_clean":"2024-03-30T15:04:52.394789+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:04:52.394789+0000","last_undegraded":"2024-03-30T15:04:52.394789+0000","last_fullsized":"2024-03-30T15:04:52.394789+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":870,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410170+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:04:52.410170+0000","last_peered":"2024-03-30T15:04:52.410170+0000","last_clean":"2024-03-30T15:04:52.410170+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:04:52.410170+0000","last_undegraded":"2024-03-30T15:04:52.410170+0000","last_fullsized":"2024-03-30T15:04:52.410170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.088353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":859,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406380+0000","last_change":"2024-03-30T15:04:07.011782+0000","last_active":"2024-03-30T15:04:52.406380+0000","last_peered":"2024-03-30T15:04:52.406380+0000","last_clean":"2024-03-30T15:04:52.406380+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:04:52.406380+0000","last_undegraded":"2024-03-30T15:04:52.406380+0000","last_fullsized":"2024-03-30T15:04:52.406380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.011661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.011661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:12.278098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":847,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.404880+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:04:52.404880+0000","last_peered":"2024-03-30T15:04:52.404880+0000","last_clean":"2024-03-30T15:04:52.404880+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:04:52.404880+0000","last_undegraded":"2024-03-30T15:04:52.404880+0000","last_fullsized":"2024-03-30T15:04:52.404880+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:04:57.177473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":765,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406117+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:04:52.406117+0000","last_peered":"2024-03-30T15:04:52.406117+0000","last_clean":"2024-03-30T15:04:52.406117+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:04:52.406117+0000","last_undegraded":"2024-03-30T15:04:52.406117+0000","last_fullsized":"2024-03-30T15:04:52.406117+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":891,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403597+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:04:52.403597+0000","last_peered":"2024-03-30T15:04:52.403597+0000","last_clean":"2024-03-30T15:04:52.403597+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:04:52.403597+0000","last_undegraded":"2024-03-30T15:04:52.403597+0000","last_fullsized":"2024-03-30T15:04:52.403597+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.236100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":857,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424571+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:04:52.424571+0000","last_peered":"2024-03-30T15:04:52.424571+0000","last_clean":"2024-03-30T15:04:52.424571+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:04:52.424571+0000","last_undegraded":"2024-03-30T15:04:52.424571+0000","last_fullsized":"2024-03-30T15:04:52.424571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":851,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.994469+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:04:52.994469+0000","last_peered":"2024-03-30T15:04:52.994469+0000","last_clean":"2024-03-30T15:04:52.994469+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:04:52.994469+0000","last_undegraded":"2024-03-30T15:04:52.994469+0000","last_fullsized":"2024-03-30T15:04:52.994469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":853,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423734+0000","last_change":"2024-03-30T15:04:35.253973+0000","last_active":"2024-03-30T15:04:52.423734+0000","last_peered":"2024-03-30T15:04:52.423734+0000","last_clean":"2024-03-30T15:04:52.423734+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:04:52.423734+0000","last_undegraded":"2024-03-30T15:04:52.423734+0000","last_fullsized":"2024-03-30T15:04:52.423734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.253856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.253856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.131487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":758,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.406578+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:04:52.406578+0000","last_peered":"2024-03-30T15:04:52.406578+0000","last_clean":"2024-03-30T15:04:52.406578+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:04:52.406578+0000","last_undegraded":"2024-03-30T15:04:52.406578+0000","last_fullsized":"2024-03-30T15:04:52.406578+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":879,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403572+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:04:52.403572+0000","last_peered":"2024-03-30T15:04:52.403572+0000","last_clean":"2024-03-30T15:04:52.403572+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:04:52.403572+0000","last_undegraded":"2024-03-30T15:04:52.403572+0000","last_fullsized":"2024-03-30T15:04:52.403572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.139552+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":857,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424985+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:04:52.424985+0000","last_peered":"2024-03-30T15:04:52.424985+0000","last_clean":"2024-03-30T15:04:52.424985+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:04:52.424985+0000","last_undegraded":"2024-03-30T15:04:52.424985+0000","last_fullsized":"2024-03-30T15:04:52.424985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":849,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.424363+0000","last_change":"2024-03-30T15:02:44.998020+0000","last_active":"2024-03-30T15:04:52.424363+0000","last_peered":"2024-03-30T15:04:52.424363+0000","last_clean":"2024-03-30T15:04:52.424363+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:04:52.424363+0000","last_undegraded":"2024-03-30T15:04:52.424363+0000","last_fullsized":"2024-03-30T15:04:52.424363+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.997898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":858,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.408673+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:04:52.408673+0000","last_peered":"2024-03-30T15:04:52.408673+0000","last_clean":"2024-03-30T15:04:52.408673+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:04:52.408673+0000","last_undegraded":"2024-03-30T15:04:52.408673+0000","last_fullsized":"2024-03-30T15:04:52.408673+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.231315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":763,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.423666+0000","last_change":"2024-03-30T15:04:52.290105+0000","last_active":"2024-03-30T15:04:52.423666+0000","last_peered":"2024-03-30T15:04:52.423666+0000","last_clean":"2024-03-30T15:04:52.423666+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:04:52.423666+0000","last_undegraded":"2024-03-30T15:04:52.423666+0000","last_fullsized":"2024-03-30T15:04:52.423666+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.289998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.289998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.214437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":871,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.391628+0000","last_change":"2024-03-30T15:02:33.875911+0000","last_active":"2024-03-30T15:04:52.391628+0000","last_peered":"2024-03-30T15:04:52.391628+0000","last_clean":"2024-03-30T15:04:52.391628+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:04:52.391628+0000","last_undegraded":"2024-03-30T15:04:52.391628+0000","last_fullsized":"2024-03-30T15:04:52.391628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T15:02:33.875778+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.875778+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":769,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.413635+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:04:52.413635+0000","last_peered":"2024-03-30T15:04:52.413635+0000","last_clean":"2024-03-30T15:04:52.413635+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:04:52.413635+0000","last_undegraded":"2024-03-30T15:04:52.413635+0000","last_fullsized":"2024-03-30T15:04:52.413635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":850,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.402857+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:04:52.402857+0000","last_peered":"2024-03-30T15:04:52.402857+0000","last_clean":"2024-03-30T15:04:52.402857+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:04:52.402857+0000","last_undegraded":"2024-03-30T15:04:52.402857+0000","last_fullsized":"2024-03-30T15:04:52.402857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:08.751185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":421,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.393405+0000","last_change":"2024-03-30T15:04:39.209738+0000","last_active":"2024-03-30T15:04:52.393405+0000","last_peered":"2024-03-30T15:04:52.393405+0000","last_clean":"2024-03-30T15:04:52.393405+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:04:52.393405+0000","last_undegraded":"2024-03-30T15:04:52.393405+0000","last_fullsized":"2024-03-30T15:04:52.393405+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:39.209655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.209655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.749896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":760,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.409009+0000","last_change":"2024-03-30T15:01:17.584059+0000","last_active":"2024-03-30T15:04:52.409009+0000","last_peered":"2024-03-30T15:04:52.409009+0000","last_clean":"2024-03-30T15:04:52.409009+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:04:52.409009+0000","last_undegraded":"2024-03-30T15:04:52.409009+0000","last_fullsized":"2024-03-30T15:04:52.409009+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:17.583984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:17.583984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":903,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403021+0000","last_change":"2024-03-30T15:04:39.592435+0000","last_active":"2024-03-30T15:04:52.403021+0000","last_peered":"2024-03-30T15:04:52.403021+0000","last_clean":"2024-03-30T15:04:52.403021+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:04:52.403021+0000","last_undegraded":"2024-03-30T15:04:52.403021+0000","last_fullsized":"2024-03-30T15:04:52.403021+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:39.592324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.592324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.690325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":870,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.410628+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:04:52.410628+0000","last_peered":"2024-03-30T15:04:52.410628+0000","last_clean":"2024-03-30T15:04:52.410628+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:04:52.410628+0000","last_undegraded":"2024-03-30T15:04:52.410628+0000","last_fullsized":"2024-03-30T15:04:52.410628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.406410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":851,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.405630+0000","last_change":"2024-03-30T15:01:53.452032+0000","last_active":"2024-03-30T15:04:52.405630+0000","last_peered":"2024-03-30T15:04:52.405630+0000","last_clean":"2024-03-30T15:04:52.405630+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:04:52.405630+0000","last_undegraded":"2024-03-30T15:04:52.405630+0000","last_fullsized":"2024-03-30T15:04:52.405630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.451927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.451927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":846,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-03-30T15:04:52.403386+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:04:52.403386+0000","last_peered":"2024-03-30T15:04:52.403386+0000","last_clean":"2024-03-30T15:04:52.403386+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:04:52.403386+0000","last_undegraded":"2024-03-30T15:04:52.403386+0000","last_fullsized":"2024-03-30T15:04:52.403386+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":301,"num_pg":32,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195872,"num_objects":169,"num_object_clones":7,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":111,"num_read_kb":10,"num_write":620,"num_write_kb":12709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":24587649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1945,"ondisk_log_size":1945,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":608,"num_read_kb":603,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005184,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75904,"kb_used_data":12780,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669248,"statfs":{"total":95995035648,"available":95917309952,"internally_reserved":0,"allocated":13086720,"data_stored":9377135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299525,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75852,"kb_used_data":12716,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669300,"statfs":{"total":95995035648,"available":95917363200,"internally_reserved":0,"allocated":13021184,"data_stored":9374698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83868,"kb_used_data":8264,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93661284,"statfs":{"total":95995035648,"available":95909154816,"internally_reserved":0,"allocated":8462336,"data_stored":4785538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70536,"kb_used_data":12956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93674616,"statfs":{"total":95995035648,"available":95922806784,"internally_reserved":0,"allocated":13266944,"data_stored":9570717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79436,"kb_used_data":16288,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93665716,"statfs":{"total":95995035648,"available":95913693184,"internally_reserved":0,"allocated":16678912,"data_stored":12979074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117141,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75580,"kb_used_data":12456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669572,"statfs":{"total":95995035648,"available":95917641728,"internally_reserved":0,"allocated":12754944,"data_stored":8980991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953615,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71484,"kb_used_data":8336,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673668,"statfs":{"total":95995035648,"available":95921836032,"internally_reserved":0,"allocated":8536064,"data_stored":4785520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659274,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71468,"kb_used_data":8320,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673684,"statfs":{"total":95995035648,"available":95921852416,"internally_reserved":0,"allocated":8519680,"data_stored":4785201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":4194827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":8194889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":4194979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4000475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4247552,"data_stored":4000604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":301,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:05:01.185 INFO:tasks.thrashosds.thrasher:Setting 5.d to [3, 7, 2, 6, 5, 4] 2024-03-30T15:05:01.186 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.d', '3', '7', '2', '6', '5', '4'] 2024-03-30T15:05:01.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap-items 5.d 3 7 2 6 5 4 2024-03-30T15:05:01.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:01.323 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:01 smithi134 ceph-mon[33714]: from='client.58163 v1:172.21.15.28:0/2826720494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:01.323 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:01 smithi134 ceph-mon[33714]: pgmap v690: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-30T15:05:01.323 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2458893042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:01.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:01.379 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[41614]: from='client.58163 v1:172.21.15.28:0/2826720494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[41614]: pgmap v690: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2458893042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[29876]: from='client.58163 v1:172.21.15.28:0/2826720494' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[29876]: pgmap v690: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-30T15:05:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2458893042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:01.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:01.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:01.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:01.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:01.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:01.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:01.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:01.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:01.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:01.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:02.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:02.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:02.131 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:02 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:02.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:02.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:02.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:02.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:02.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:02 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:02.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:02 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:02.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:02.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:02.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:02.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:02.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:02.679 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:05:02.679 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:05:02.679 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:05:02.679 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:05:02.679 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:05:02.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:02.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:02.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:02.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:02.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:02.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:03.016 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:03.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:03.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:03.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:05:03.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:03.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:03.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:03.555 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:03.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:03.677 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:03 smithi134 ceph-mon[33714]: pgmap v691: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 745 B/s wr, 3 op/s 2024-03-30T15:05:03.677 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.677 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.677 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.677 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[29876]: pgmap v691: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 745 B/s wr, 3 op/s 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[41614]: pgmap v691: 260 pgs: 260 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 745 B/s wr, 3 op/s 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-73"}]: dispatch 2024-03-30T15:05:03.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:03.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:05:03.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:03.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:03.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:04.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:04.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:04.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:04.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:04.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:04.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.398 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:04.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:04.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:04.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:04.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[41614]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T15:05:04.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[41614]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:05:04.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[41614]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T15:05:04.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[29876]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T15:05:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[29876]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:05:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[29876]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T15:05:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:04.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:04 smithi134 ceph-mon[33714]: osdmap e483: 8 total, 8 up, 8 in 2024-03-30T15:05:04.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:04 smithi134 ceph-mon[33714]: pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:05:04.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:04 smithi134 ceph-mon[33714]: osdmap e484: 8 total, 8 up, 8 in 2024-03-30T15:05:04.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.630 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.725+0000 7f73a7b03700 1 -- 172.21.15.28:0/3535268120 >> v1:172.21.15.28:6790/0 conn(0x7f73a010ea80 legacy=0x7f73a0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.725+0000 7f73a7b03700 1 -- 172.21.15.28:0/3535268120 shutdown_connections 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.725+0000 7f73a7b03700 1 -- 172.21.15.28:0/3535268120 >> 172.21.15.28:0/3535268120 conn(0x7f73a00761d0 msgr2=0x7f73a00765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.725+0000 7f73a7b03700 1 -- 172.21.15.28:0/3535268120 shutdown_connections 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.725+0000 7f73a7b03700 1 -- 172.21.15.28:0/3535268120 wait complete. 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 librados: starting msgr at 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 librados: starting objecter 2024-03-30T15:05:04.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 Processor -- start 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 -- start start 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 librados: setting wanted keys 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 librados: calling monclient init 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73a01a9200 con 0x7f73a010ea80 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73a01aa420 con 0x7f73a0108110 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a7b03700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73a01ab640 con 0x7f73a010ae80 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a509e700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f73a010ae80 0x7f73a01a4320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39674/0 (socket says 172.21.15.28:39674) 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a509e700 1 -- 172.21.15.28:0/1699015035 learned_addr learned my addr 172.21.15.28:0/1699015035 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f73a60a0700 1 --1- 172.21.15.28:0/1699015035 >> v1:172.21.15.28:6789/0 conn(0x7f73a010ea80 0x7f73a01a7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37276/0 (socket says 172.21.15.28:37276) 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f7396ffd700 10 client.?.objecter ms_handle_connect 0x7f73a010ae80 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f7396ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f7396ffd700 10 client.?.objecter ms_handle_connect 0x7f73a010ea80 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.726+0000 7f7396ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 10 client.?.objecter ms_handle_connect 0x7f73a0108110 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094621589 0 0) 0x7f73a01ab640 con 0x7f73a010ae80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f738c0031f0 con 0x7f73a010ae80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3914005010 0 0) 0x7f73a01a9200 con 0x7f73a010ea80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a01ab640 con 0x7f73a010ea80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685397460 0 0) 0x7f73a01aa420 con 0x7f73a0108110 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a01a9200 con 0x7f73a0108110 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2983734547 0 0) 0x7f738c0031f0 con 0x7f73a010ae80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73a01aa420 con 0x7f73a010ae80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7390004060 con 0x7f73a010ae80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3172837926 0 0) 0x7f73a01ab640 con 0x7f73a010ea80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f738c0031f0 con 0x7f73a010ea80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7398001890 con 0x7f73a010ea80 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4022665590 0 0) 0x7f73a01a9200 con 0x7f73a0108110 2024-03-30T15:05:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73a01ab640 con 0x7f73a0108110 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f739c001a30 con 0x7f73a0108110 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 94295968 0 0) 0x7f73a01aa420 con 0x7f73a010ae80 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 >> v1:172.21.15.134:6789/0 conn(0x7f73a0108110 legacy=0x7f73a01073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 >> v1:172.21.15.28:6789/0 conn(0x7f73a010ea80 legacy=0x7f73a01a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a01ac880 con 0x7f73a010ae80 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73a01a9430 con 0x7f73a010ae80 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f73a7b03700 10 client.58175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f73a01a98f0 con 0x7f73a010ae80 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f73a7b03700 1 librados: init done 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7390004450 con 0x7f73a010ae80 2024-03-30T15:05:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.727+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7390005ce0 con 0x7f73a010ae80 2024-03-30T15:05:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.728+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7384005120 con 0x7f73a010ae80 2024-03-30T15:05:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7390001880 con 0x7f73a010ae80 2024-03-30T15:05:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(484..484 src has 53..484) ==== 8539+0+0 (unknown 267227205 0 0) 0x7f73900897c0 con 0x7f73a010ae80 2024-03-30T15:05:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 10 client.58175.objecter ms_dispatch 0x7f73a0001550 osd_map(484..484 src has 53..484) 2024-03-30T15:05:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 3 client.58175.objecter handle_osd_map got epochs [484,484] > 0 2024-03-30T15:05:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 3 client.58175.objecter handle_osd_map decoding full epoch 484 2024-03-30T15:05:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 20 client.58175.objecter dump_active .. 0 homeless 2024-03-30T15:05:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.729+0000 7f7396ffd700 10 client.58175.objecter ms_handle_connect 0x7f738c0703e0 2024-03-30T15:05:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.735+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7390056850 con 0x7f73a010ae80 2024-03-30T15:05:04.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:04.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:04.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:04.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:04.963+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]} v 0) -- 0x7f7384005410 con 0x7f73a010ae80 2024-03-30T15:05:05.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:05.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.078+0000 7f7396ffd700 1 -- 172.21.15.28:0/1699015035 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]=0 set 5.d pg_upmap_items mapping to [3->7,2->6,5->4] v485) ==== 157+0+0 (unknown 1996518130 0 0) 0x7f739005a2a0 con 0x7f73a010ae80 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 10 librados: watch_flush enter 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 10 librados: watch_flush exit 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 20 client.58175.objecter shutdown clearing up homeless session... 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 10 client.58175.objecter successfully canceled tick 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 >> v1:172.21.15.28:6800/3011431093 conn(0x7f738c0703e0 legacy=0x7f738c0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 >> v1:172.21.15.28:6790/0 conn(0x7f73a010ae80 legacy=0x7f73a01a4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 shutdown_connections 2024-03-30T15:05:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.082+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 >> 172.21.15.28:0/1699015035 conn(0x7f73a00761d0 msgr2=0x7f73a0108db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.086+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 shutdown_connections 2024-03-30T15:05:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.088+0000 7f73a7b03700 1 -- 172.21.15.28:0/1699015035 wait complete. 2024-03-30T15:05:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.088+0000 7f73a7b03700 1 librados: shutdown 2024-03-30T15:05:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:set 5.d pg_upmap_items mapping to [3->7,2->6,5->4] 2024-03-30T15:05:05.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:05.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:05.161 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:05:05.161 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:05:05.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:05:05.161 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24892.0:2 22.10 22:0c63179d:test-rados-api-smithi028-93390-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 430.102843146, 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16686387499999999, 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24892", 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1207320814", 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.055998+0000", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056000+0000", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.897e-06 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056007+0000", 2024-03-30T15:05:05.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8290000000000002e-06 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056018+0000", 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1647999999999999e-05 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056029+0000", 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0702e-05 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056154+0000", 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012552099999999999 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.056165+0000", 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0147e-05 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097196+0000", 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041031265999999997 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097299+0000", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010320200000000001 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.097434+0000", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013449600000000001 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.104018+0000", 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.006584047 2024-03-30T15:05:05.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.184696+0000", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080678643999999994 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222775+0000", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038078911999999999 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222823+0000", 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7997000000000001e-05 2024-03-30T15:05:05.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.222862+0000", 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8566999999999999e-05 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24832.0:14 10.1b 10:db53fca8:test-rados-api-smithi028-93032-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 429.85977688700001, 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99005185600000001, 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.169 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24832", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2605619531", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.170 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299064+0000", 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299065+0000", 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4160000000000001e-06 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299071+0000", 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8510000000000004e-06 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299081+0000", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8670000000000006e-06 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299087+0000", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7559999999999996e-06 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299131+0000", 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4267999999999997e-05 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.299138+0000", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2779999999999999e-06 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.199889+0000", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.90075121199999997 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.200039+0000", 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015004800000000001 2024-03-30T15:05:05.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.200192+0000", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015276299999999999 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.209842+0000", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0096505029999999995 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.220748+0000", 2024-03-30T15:05:05.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010905475 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289041+0000", 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.068293870000000007 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289083+0000", 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1668000000000002e-05 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.289116+0000", 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2880999999999999e-05 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.176 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35288.0:2 53.0 53:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 427.74669173400002, 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3211012919999998, 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35288", 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3734394375", 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.177 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412149+0000", 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412151+0000", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5829999999999999e-06 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412159+0000", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7840000000000004e-06 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412168+0000", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6229999999999992e-06 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412175+0000", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7240000000000003e-06 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412220+0000", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5077999999999998e-05 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.412226+0000", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8359999999999997e-06 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.180 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729625+0000", 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3173991090000001 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729721+0000", 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6324999999999997e-05 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729844+0000", 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012299199999999999 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733011+0000", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031669369999999999 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733063+0000", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1668000000000001e-05 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733112+0000", 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9196999999999997e-05 2024-03-30T15:05:05.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733163+0000", 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1159000000000003e-05 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733251+0000", 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7276999999999998e-05 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.34853.0:2059 11.7 e116/110 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head v 116'1, mlcod=0'0)", 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 427.18874794099997, 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060642963000000001, 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970093+0000", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.970096+0000", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3900000000000002e-06 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983905+0000", 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013808289 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983934+0000", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8943999999999999e-05 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.983958+0000", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4706e-05 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.984028+0000", 2024-03-30T15:05:05.186 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9355000000000003e-05 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.984046+0000", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8637000000000001e-05 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030684+0000", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.046637985999999999 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030736+0000", 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1656000000000001e-05 2024-03-30T15:05:05.187 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.34853.0:2069 11.7 e116/110 11:e17e9421:test-rados-api-smithi028-93257-17::big:head v 116'4, mlcod=116'2)", 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 426.86063341699997, 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.056908669000000002, 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.188 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298208+0000", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.298210+0000", 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.508e-06 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.189 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306447+0000", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0082367710000000004 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306462+0000", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.506e-05 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306473+0000", 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0987e-05 2024-03-30T15:05:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306529+0000", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6245000000000001e-05 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.306548+0000", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8896e-05 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355055+0000", 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.048506794999999998 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.191 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355116+0000", 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1407000000000003e-05 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35393.0:3 55.14 55:2a647cd4:::obj0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 426.60449013200002, 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.176375602, 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.192 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35393", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3403500224", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.193 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554351+0000", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554353+0000", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.99e-06 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554359+0000", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2380000000000002e-06 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.194 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554645+0000", 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028636500000000001 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.554656+0000", 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0383e-05 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.555818+0000", 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001162649 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.195 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.555833+0000", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5004999999999999e-05 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730652+0000", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.174818852 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730726+0000", 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4120000000000002e-05 2024-03-30T15:05:05.196 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25357.0:2 59.15 59:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 423.36988993799997, 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.083096852999999998, 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25357", 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4132935590", 2024-03-30T15:05:05.197 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788951+0000", 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.198 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788953+0000", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.125e-06 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788959+0000", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.285e-06 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788974+0000", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4980999999999999e-05 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.788986+0000", 2024-03-30T15:05:05.199 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1239e-05 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789049+0000", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3813999999999996e-05 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789060+0000", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0365e-05 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.809307+0000", 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020247586000000001 2024-03-30T15:05:05.200 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.809316+0000", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.867e-06 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860035+0000", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.050718621999999998 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860156+0000", 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120742 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.201 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.860291+0000", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013580600000000001 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.862487+0000", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002195865 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.865606+0000", 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031187010000000002 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.202 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.871952+0000", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0063455330000000004 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.872002+0000", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0482000000000002e-05 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.872048+0000", 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5840000000000002e-05 2024-03-30T15:05:05.203 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:10 104.a 104:503bb488:test-rados-api-smithi028-93441-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 403.10335349299999, 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.23059840000000001, 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:05:05.204 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 10 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055487+0000", 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.205 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055490+0000", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.289e-06 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055499+0000", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8130000000000002e-06 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055511+0000", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1936000000000001e-05 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.055521+0000", 2024-03-30T15:05:05.206 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0862e-05 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076707+0000", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.021186060999999999 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076816+0000", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108422 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.076977+0000", 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160798 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.207 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.081552+0000", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045756740000000001 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.118486+0000", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036933360999999998 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285988+0000", 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16750200700000001 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.208 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286037+0000", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9163999999999999e-05 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286086+0000", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9013e-05 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46169.0:2 128.0 128:0384ab8e:::oid:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:05:05.209 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 399.572898049, 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.399107962, 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46169", 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2731295739", 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.210 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585943+0000", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585945+0000", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7409999999999999e-06 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585952+0000", 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1110000000000002e-06 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.211 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585961+0000", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6439999999999995e-06 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.585972+0000", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0444e-05 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.586033+0000", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0677999999999997e-05 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.212 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.586041+0000", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7020000000000007e-06 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.960969+0000", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3749282319999998 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.961084+0000", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114992 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,6", 2024-03-30T15:05:05.213 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.961221+0000", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013679 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.962152+0000", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00093109599999999996 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.962664+0000", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00051144500000000002 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.214 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.984986+0000", 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022322437000000001 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.985022+0000", 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5281000000000003e-05 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.985051+0000", 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9369e-05 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.215 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2080 11.1b 11:df4c7475:test-rados-api-smithi028-93257-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 396.121034157, 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.156258491, 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2080 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.216 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037807+0000", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037808+0000", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.54e-07 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037811+0000", 2024-03-30T15:05:05.217 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.737e-06 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037816+0000", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5689999999999998e-06 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037820+0000", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0799999999999999e-06 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037856+0000", 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6032999999999999e-05 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037932+0000", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5718999999999998e-05 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.112269+0000", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.074337635999999999 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.123357+0000", 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.011087211 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.123994+0000", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00063726099999999997 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.193965+0000", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069971071999999995 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.194023+0000", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7599999999999997e-05 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.194065+0000", 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2719000000000002e-05 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46235.0:777 130.f 130:f18d14f3:::test_770:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 393.85064253600001, 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058944257, 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46235", 2024-03-30T15:05:05.221 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 777 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308198+0000", 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308201+0000", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3910000000000001e-06 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308208+0000", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9299999999999997e-06 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308223+0000", 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4952e-05 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308231+0000", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1880000000000002e-06 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308292+0000", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1274999999999999e-05 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.308402+0000", 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109546 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.367143+0000", 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058740975000000001 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:28 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 0~128] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 392.98411459800002, 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.070825352999999994, 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.225 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 28 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174726+0000", 2024-03-30T15:05:05.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174728+0000", 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.761e-06 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174743+0000", 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.485e-05 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174754+0000", 2024-03-30T15:05:05.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1413999999999999e-05 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174765+0000", 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0715e-05 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174834+0000", 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8806999999999994e-05 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.174967+0000", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013262199999999999 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.175110+0000", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014391499999999999 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.176336+0000", 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001225603 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.183259+0000", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0069225720000000001 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245465+0000", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062206836000000001 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245511+0000", 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5689999999999998e-05 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.245552+0000", 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0568000000000002e-05 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:35 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 128~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 391.97565415100001, 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14093118900000001, 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.232 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 35 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183187+0000", 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183189+0000", 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7710000000000001e-06 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183197+0000", 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.977e-06 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183208+0000", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1868999999999999e-05 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.183219+0000", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1049e-05 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.191898+0000", 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0086782060000000008 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.192015+0000", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011683 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.192223+0000", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000208383 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.199864+0000", 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076411250000000003 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.199969+0000", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104924 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324017+0000", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.124047831 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324072+0000", 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5185000000000001e-05 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.324118+0000", 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6038999999999997e-05 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.43456.0:2 141.1b 141:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 391.956916132, 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.141261546, 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.238 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.43456", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3977280399", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201925+0000", 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201927+0000", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201940+0000", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.344e-05 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201952+0000", 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.189e-05 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.240 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.201962+0000", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.628e-06 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.208727+0000", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067653949999999996 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.208737+0000", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0134e-05 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337524+0000", 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1287864860000001 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337650+0000", 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012597699999999999 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.337783+0000", 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013337200000000001 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.242 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.340765+0000", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029821169999999998 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.342269+0000", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015038790000000001 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343072+0000", 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00080310199999999998 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343137+0000", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4850000000000004e-05 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343186+0000", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9258000000000002e-05 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.27442.0:42 111.1 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head [write 128~128] snapc 8=[8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:05:05.244 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 389.85217856100002, 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12159848600000001, 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.27442", 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3216563079", 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 42 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306662+0000", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306664+0000", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.877e-06 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306674+0000", 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3179999999999996e-06 2024-03-30T15:05:05.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306685+0000", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1630999999999999e-05 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.306696+0000", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0427e-05 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.331819+0000", 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025123125 2024-03-30T15:05:05.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.331826+0000", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1829999999999997e-06 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337586+0000", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057599330000000001 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337695+0000", 2024-03-30T15:05:05.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109527 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.337897+0000", 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020184399999999999 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.340104+0000", 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022068370000000001 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.365863+0000", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025758547 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427962+0000", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062099013000000002 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428068+0000", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106545 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428261+0000", 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019267899999999999 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.48940.0:2 169.1b 169:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 378.95429102600002, 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0270853879999999, 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.251 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.48940", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/728758827", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204550+0000", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204552+0000", 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.204560+0000", 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4739999999999999e-06 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216581+0000", 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012021041 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216592+0000", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1868000000000001e-05 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216715+0000", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012253599999999999 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.216724+0000", 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5960000000000004e-06 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224347+0000", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0076235769999999 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224466+0000", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118509 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.224603+0000", 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013684600000000001 2024-03-30T15:05:05.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.225507+0000", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090434700000000001 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.226930+0000", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00142267 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231488+0000", 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004558472 2024-03-30T15:05:05.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231539+0000", 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0834999999999997e-05 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.231635+0000", 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6477999999999993e-05 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.48946.0:2 170.0 170:0384ab8e:::oid:head [append 0~128,setxattr attr1 (11) in=5b,setxattr attr2 (256) in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 377.95265011599997, 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012141285, 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.48946", 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/417824189", 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206191+0000", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206193+0000", 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1280000000000002e-06 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206205+0000", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1595e-05 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206215+0000", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0397000000000001e-05 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206223+0000", 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7859999999999997e-06 2024-03-30T15:05:05.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206739+0000", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00051627499999999996 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:47.206756+0000", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7142999999999998e-05 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212703+0000", 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0059471760000001 2024-03-30T15:05:05.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212819+0000", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115915 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.212988+0000", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016922899999999999 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.216595+0000", 2024-03-30T15:05:05.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036063169999999999 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.217141+0000", 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000545901 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218232+0000", 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010908669999999999 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218287+0000", 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5744e-05 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:48.218332+0000", 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4811999999999997e-05 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24069.0:1 209.1s0 209:8f59135b:test-rados-api-smithi028-93257-37::foo:head [write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 337.95319433600002, 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0606089889999999, 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.264 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24069", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/556303511", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205647+0000", 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205648+0000", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5939999999999999e-06 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205655+0000", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3509999999999998e-06 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205664+0000", 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0880000000000001e-06 2024-03-30T15:05:05.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.205671+0000", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2819999999999999e-06 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.209177+0000", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035059549999999998 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:27.209181+0000", 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7529999999999999e-06 2024-03-30T15:05:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.255633+0000", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.046452484 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.255983+0000", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000349656 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.256346+0000", 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036334200000000001 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.268 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.256897+0000", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055132599999999999 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.266188+0000", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0092902729999999999 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.266256+0000", 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7885000000000005e-05 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.269 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57506.0:1 249.6s0 249:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 272.94395397, 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0321323259999999, 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57506", 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/174647268", 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.270 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214887+0000", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214890+0000", 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5160000000000001e-06 2024-03-30T15:05:05.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214899+0000", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1560000000000001e-06 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214912+0000", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3456e-05 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.214925+0000", 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3271000000000001e-05 2024-03-30T15:05:05.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.215120+0000", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019473900000000001 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:32.215125+0000", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3480000000000003e-06 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242234+0000", 2024-03-30T15:05:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.027108057 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242343+0000", 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109604 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.242674+0000", 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033041199999999998 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:05:05.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.243135+0000", 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000461108 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.246953+0000", 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038186940000000001 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:33.247019+0000", 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5964999999999996e-05 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57578.0:1 257.6s1 257:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:53.216777+0000", 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 251.94206415100001, 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0552720630000001, 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57578", 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1782570749", 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.276 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216777+0000", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216777+0000", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216780+0000", 2024-03-30T15:05:05.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9699999999999999e-06 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216788+0000", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8630000000000008e-06 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216797+0000", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7310000000000005e-06 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.216808+0000", 2024-03-30T15:05:05.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0127e-05 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.227199+0000", 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010391136 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:53.227203+0000", 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.065e-06 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:05.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.267266+0000", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.040063239 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.267398+0000", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013180399999999999 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.267868+0000", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00047055399999999998 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:05:05.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.268465+0000", 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00059647599999999997 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.271996+0000", 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035308800000000001 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:54.272049+0000", 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3217999999999998e-05 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:05.282 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:05:05.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.278+0000 7ff00264f700 1 -- 172.21.15.28:0/622691155 >> v1:172.21.15.28:6790/0 conn(0x7feffc101130 legacy=0x7feffc101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.278+0000 7ff00264f700 1 -- 172.21.15.28:0/622691155 shutdown_connections 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.278+0000 7ff00264f700 1 -- 172.21.15.28:0/622691155 >> 172.21.15.28:0/622691155 conn(0x7feffc0fc9e0 msgr2=0x7feffc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.278+0000 7ff00264f700 1 -- 172.21.15.28:0/622691155 shutdown_connections 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.278+0000 7ff00264f700 1 -- 172.21.15.28:0/622691155 wait complete. 2024-03-30T15:05:05.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 librados: starting msgr at 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 librados: starting objecter 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 Processor -- start 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 -- start start 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 librados: setting wanted keys 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 librados: calling monclient init 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feffc106900 con 0x7feffc10d960 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feffc1a8540 con 0x7feffc109dc0 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7ff00264f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feffc1a9720 con 0x7feffc101130 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.279+0000 7feffbfff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7feffc101130 0x7feffc104720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39704/0 (socket says 172.21.15.28:39704) 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feffbfff700 1 -- 172.21.15.28:0/2102178589 learned_addr learned my addr 172.21.15.28:0/2102178589 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7ff000bec700 1 --1- 172.21.15.28:0/2102178589 >> v1:172.21.15.28:6789/0 conn(0x7feffc10d960 0x7feffc1a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37302/0 (socket says 172.21.15.28:37302) 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter ms_handle_connect 0x7feffc101130 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:05.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter ms_handle_connect 0x7feffc109dc0 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter ms_handle_connect 0x7feffc10d960 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3496606958 0 0) 0x7feffc1a8540 con 0x7feffc109dc0 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe40039c0 con 0x7feffc109dc0 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13630435 0 0) 0x7feffc1a9720 con 0x7feffc101130 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feffc1a8540 con 0x7feffc101130 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857678469 0 0) 0x7feffc106900 con 0x7feffc10d960 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feffc1a9720 con 0x7feffc10d960 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2700551357 0 0) 0x7fefe40039c0 con 0x7feffc109dc0 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feffc106900 con 0x7feffc109dc0 2024-03-30T15:05:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feff00018b0 con 0x7feffc109dc0 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3164381473 0 0) 0x7feffc1a8540 con 0x7feffc101130 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefe40039c0 con 0x7feffc101130 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384429970 0 0) 0x7feffc1a9720 con 0x7feffc10d960 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feffc1a8540 con 0x7feffc10d960 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fefec001470 con 0x7feffc101130 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feff4001c10 con 0x7feffc10d960 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3728490380 0 0) 0x7feffc106900 con 0x7feffc109dc0 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 >> v1:172.21.15.28:6790/0 conn(0x7feffc101130 legacy=0x7feffc104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 >> v1:172.21.15.28:6789/0 conn(0x7feffc10d960 legacy=0x7feffc1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc1aa900 con 0x7feffc109dc0 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feffc1a8770 con 0x7feffc109dc0 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7ff00264f700 10 client.50191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feffc1a8c30 con 0x7feffc109dc0 2024-03-30T15:05:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7ff00264f700 1 librados: init done 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7feff0001c60 con 0x7feffc109dc0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.280+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7feff0004b60 con 0x7feffc109dc0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe8005120 con 0x7feffc109dc0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7feff0003950 con 0x7feffc109dc0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(485..485 src has 53..485) ==== 8589+0+0 (unknown 4142684862 0 0) 0x7feff00885c0 con 0x7feffc109dc0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7feff97fa700 10 client.50191.objecter ms_dispatch 0x7feffc001550 osd_map(485..485 src has 53..485) 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7feff97fa700 3 client.50191.objecter handle_osd_map got epochs [485,485] > 0 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.282+0000 7feff97fa700 3 client.50191.objecter handle_osd_map decoding full epoch 485 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.283+0000 7feff97fa700 20 client.50191.objecter dump_active .. 0 homeless 2024-03-30T15:05:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.283+0000 7feff97fa700 10 client.50191.objecter ms_handle_connect 0x7fefe40704f0 2024-03-30T15:05:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.287+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7feff0055600 con 0x7feffc109dc0 2024-03-30T15:05:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:05.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:05.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1699015035' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: from='client.58175 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: from='client.58175 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]': finished 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T15:05:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:05.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1699015035' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: from='client.58175 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: from='client.58175 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]': finished 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1699015035' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: from='client.58175 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]: dispatch 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: from='client.58175 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.d", "id": [3, 7, 2, 6, 5, 4]}]': finished 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: osdmap e485: 8 total, 8 up, 8 in 2024-03-30T15:05:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:05.508+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fefe8005410 con 0x7feffc109dc0 2024-03-30T15:05:05.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:05.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:05.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:05.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:05.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:05.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:05:05.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:05.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:06.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:06.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.077+0000 7feff97fa700 1 -- 172.21.15.28:0/2102178589 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v486) ==== 97+0+0 (unknown 3156243194 0 0) 0x7feff0059050 con 0x7feffc109dc0 2024-03-30T15:05:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 10 librados: watch_flush enter 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 10 librados: watch_flush exit 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 20 client.50191.objecter shutdown clearing up homeless session... 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 10 client.50191.objecter successfully canceled tick 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 >> v1:172.21.15.28:6800/3011431093 conn(0x7fefe40704f0 legacy=0x7fefe40729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 >> v1:172.21.15.134:6789/0 conn(0x7feffc109dc0 legacy=0x7feffc104e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 shutdown_connections 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 >> 172.21.15.28:0/2102178589 conn(0x7feffc0fc9e0 msgr2=0x7feffc069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:06.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 shutdown_connections 2024-03-30T15:05:06.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 -- 172.21.15.28:0/2102178589 wait complete. 2024-03-30T15:05:06.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:06.079+0000 7ff00264f700 1 librados: shutdown 2024-03-30T15:05:06.082 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:05:06.158 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:06.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:06.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:06.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:06.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:06.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:06.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:06.533 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2102178589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.533 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: from='client.50191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.534 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:06.534 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:06.534 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: from='client.50191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:06.534 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:06 smithi134 ceph-mon[33714]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T15:05:06.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2102178589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: from='client.50191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: from='client.50191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[29876]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2102178589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: from='client.50191 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: pgmap v696: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: from='client.50191 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:06.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:06 smithi028 ceph-mon[41614]: osdmap e486: 8 total, 8 up, 8 in 2024-03-30T15:05:06.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:06.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:06.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:06.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:06.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:06.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:06.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:07.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:07.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:07.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:07.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:07.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:07.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:07.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:07.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:07.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:07.884 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:05:07.884 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:05:07.884 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:05:07.885 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:05:07.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:07.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:07.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:08.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:08.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:08 smithi134 ceph-mon[33714]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T15:05:08.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:08.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T15:05:08.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[29876]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: osdmap e487: 8 total, 8 up, 8 in 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-75"}]: dispatch 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:08.247 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:08.248 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-mon[41614]: osdmap e488: 8 total, 8 up, 8 in 2024-03-30T15:05:08.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:08.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:08.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:08.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:08.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:05:08.536 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.545+0000 7faf0f5cc700 1 -- 172.21.15.28:0/2368097156 >> v1:172.21.15.28:6789/0 conn(0x7faf0810ea70 legacy=0x7faf08110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.545+0000 7faf0f5cc700 1 -- 172.21.15.28:0/2368097156 shutdown_connections 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.545+0000 7faf0f5cc700 1 -- 172.21.15.28:0/2368097156 >> 172.21.15.28:0/2368097156 conn(0x7faf080fdaf0 msgr2=0x7faf080fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.545+0000 7faf0f5cc700 1 -- 172.21.15.28:0/2368097156 shutdown_connections 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.545+0000 7faf0f5cc700 1 -- 172.21.15.28:0/2368097156 wait complete. 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 librados: starting msgr at 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 librados: starting objecter 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 Processor -- start 2024-03-30T15:05:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 -- start start 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 librados: setting wanted keys 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 librados: calling monclient init 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf081a93c0 con 0x7faf08102240 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf081aa5e0 con 0x7faf0810ea70 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.546+0000 7faf0f5cc700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faf081ab820 con 0x7faf0810aed0 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faf0d368700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faf08102240 0x7faf081075f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37320/0 (socket says 172.21.15.28:37320) 2024-03-30T15:05:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faf0cb67700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7faf0810aed0 0x7faf081a42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39724/0 (socket says 172.21.15.28:39724) 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faf0d368700 1 -- 172.21.15.28:0/4230644476 learned_addr learned my addr 172.21.15.28:0/4230644476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter ms_handle_connect 0x7faf08102240 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter ms_handle_connect 0x7faf0810aed0 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter ms_handle_connect 0x7faf0810ea70 2024-03-30T15:05:08.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956241183 0 0) 0x7faf081a93c0 con 0x7faf08102240 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faef4003aa0 con 0x7faf08102240 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940270723 0 0) 0x7faf081aa5e0 con 0x7faf0810ea70 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf081a93c0 con 0x7faf0810ea70 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236968368 0 0) 0x7faf081ab820 con 0x7faf0810aed0 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf081aa5e0 con 0x7faf0810aed0 2024-03-30T15:05:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2172104412 0 0) 0x7faf081a93c0 con 0x7faf0810ea70 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf081ab820 con 0x7faf0810ea70 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf00004730 con 0x7faf0810ea70 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 631326645 0 0) 0x7faef4003aa0 con 0x7faf08102240 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf081a93c0 con 0x7faf08102240 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3151297739 0 0) 0x7faf081aa5e0 con 0x7faf0810aed0 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faef4003aa0 con 0x7faf0810aed0 2024-03-30T15:05:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf040018b0 con 0x7faf08102240 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faef8001230 con 0x7faf0810aed0 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1472828457 0 0) 0x7faf081ab820 con 0x7faf0810ea70 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6790/0 conn(0x7faf0810aed0 legacy=0x7faf081a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6789/0 conn(0x7faf08102240 legacy=0x7faf081075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.547+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf081aca60 con 0x7faf0810ea70 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faf081aba50 con 0x7faf0810ea70 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faf0f5cc700 10 client.50197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faf081abf10 con 0x7faf0810ea70 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faf0f5cc700 1 librados: init done 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7faf00001e30 con 0x7faf0810ea70 2024-03-30T15:05:08.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7faf00004ff0 con 0x7faf0810ea70 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.548+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7faeec005100 con 0x7faf0810ea70 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.549+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7faf00003a90 con 0x7faf0810ea70 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(489..489 src has 53..489) ==== 8576+0+0 (unknown 2254911158 0 0) 0x7faf00088b10 con 0x7faf0810ea70 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 osd_map(489..489 src has 53..489) 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 3 client.50197.objecter handle_osd_map got epochs [489,489] > 0 2024-03-30T15:05:08.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 3 client.50197.objecter handle_osd_map decoding full epoch 489 2024-03-30T15:05:08.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 20 client.50197.objecter dump_active .. 0 homeless 2024-03-30T15:05:08.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v489) ==== 54+0+16 (unknown 1629074835 0 2420000192) 0x7faf00089060 con 0x7faf0810ea70 2024-03-30T15:05:08.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faef40704d0 2024-03-30T15:05:08.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 1 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 1 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 0 1 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 4 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 10 client.50197.objecter _send_command 1 2024-03-30T15:05:08.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faeec004b50 con 0x7faeec004730 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.550+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec004730 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.551+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec004b50 con 0x7faeec004730 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.551+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.551+0000 7faefe7fc700 10 client.50197.objecter _finish_command 1 = system:0 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.551+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 4 2024-03-30T15:05:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.551+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 0 1 2024-03-30T15:05:08.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 2 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 2 2024-03-30T15:05:08.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec0053e0 osd=0 3 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 0 2 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 4 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 10 client.50197.objecter _send_command 2 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.568+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec0079f0 con 0x7faeec004730 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec0079f0 con 0x7faeec004730 2024-03-30T15:05:08.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faefe7fc700 10 client.50197.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 4 2024-03-30T15:05:08.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 0 2 2024-03-30T15:05:08.572 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 1 3 2024-03-30T15:05:08.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 4 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 10 client.50197.objecter _send_command 3 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.570+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7faeec002180 con 0x7faeec004280 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec004280 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec002180 con 0x7faeec004280 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 10 client.50197.objecter _finish_command 3 = system:0 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 4 2024-03-30T15:05:08.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.571+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 1 3 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 4 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 4 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec004110 osd=1 3 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 1 4 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 4 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 10 client.50197.objecter _send_command 4 2024-03-30T15:05:08.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.588+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec001600 con 0x7faeec004280 2024-03-30T15:05:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec001600 con 0x7faeec004280 2024-03-30T15:05:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faefe7fc700 10 client.50197.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 4 2024-03-30T15:05:08.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 1 4 2024-03-30T15:05:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 5 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 5 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 2 5 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 4 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 10 client.50197.objecter _send_command 5 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.590+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7faeec0110e0 con 0x7faeec00d5d0 2024-03-30T15:05:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec00d5d0 2024-03-30T15:05:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec0110e0 con 0x7faeec00d5d0 2024-03-30T15:05:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 10 client.50197.objecter _finish_command 5 = system:0 2024-03-30T15:05:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 4 2024-03-30T15:05:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.591+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 2 5 2024-03-30T15:05:08.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 6 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 6 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec008370 osd=2 3 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 2 6 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 4 2024-03-30T15:05:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 10 client.50197.objecter _send_command 6 2024-03-30T15:05:08.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.608+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec007e50 con 0x7faeec00d5d0 2024-03-30T15:05:08.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec007e50 con 0x7faeec00d5d0 2024-03-30T15:05:08.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faefe7fc700 10 client.50197.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 4 2024-03-30T15:05:08.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 2 6 2024-03-30T15:05:08.612 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 7 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 7 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 3 7 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 4 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 10 client.50197.objecter _send_command 7 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.610+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7faeec00d400 con 0x7faeec012410 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec012410 2024-03-30T15:05:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec00d400 con 0x7faeec012410 2024-03-30T15:05:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 10 client.50197.objecter _finish_command 7 = system:0 2024-03-30T15:05:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 4 2024-03-30T15:05:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.611+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 3 7 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 8 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 8 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec00cc20 osd=3 3 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 3 8 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 4 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 10 client.50197.objecter _send_command 8 2024-03-30T15:05:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.629+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec00c6a0 con 0x7faeec012410 2024-03-30T15:05:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec00c6a0 con 0x7faeec012410 2024-03-30T15:05:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faefe7fc700 10 client.50197.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 4 2024-03-30T15:05:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 3 8 2024-03-30T15:05:08.632 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 9 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 9 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 4 9 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 4 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 10 client.50197.objecter _send_command 9 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.630+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7faeec00c5c0 con 0x7faeec011f70 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.631+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec011f70 2024-03-30T15:05:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.632+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec00c5c0 con 0x7faeec011f70 2024-03-30T15:05:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.632+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.632+0000 7faefe7fc700 10 client.50197.objecter _finish_command 9 = system:0 2024-03-30T15:05:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.632+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 4 2024-03-30T15:05:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.632+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 4 9 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 10 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 10 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec00c450 osd=4 3 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 4 10 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 4 2024-03-30T15:05:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 10 client.50197.objecter _send_command 10 2024-03-30T15:05:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.649+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec00c5c0 con 0x7faeec011f70 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.650+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec00c5c0 con 0x7faeec011f70 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.650+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.650+0000 7faefe7fc700 10 client.50197.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.650+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 4 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.650+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 4 10 2024-03-30T15:05:08.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 11 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 11 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 5 11 2024-03-30T15:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 4 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 10 client.50197.objecter _send_command 11 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7faeec01eac0 con 0x7faeec01af80 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.651+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec01af80 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.652+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec01eac0 con 0x7faeec01af80 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.652+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.652+0000 7faefe7fc700 10 client.50197.objecter _finish_command 11 = system:0 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.652+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 4 2024-03-30T15:05:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.652+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 5 11 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 12 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 12 2024-03-30T15:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec01a900 osd=5 3 2024-03-30T15:05:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 5 12 2024-03-30T15:05:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 4 2024-03-30T15:05:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 10 client.50197.objecter _send_command 12 2024-03-30T15:05:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.669+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec020470 con 0x7faeec01af80 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec020470 con 0x7faeec01af80 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 10 client.50197.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 4 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 5 12 2024-03-30T15:05:08.672 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 13 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 13 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 6 13 2024-03-30T15:05:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 4 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 10 client.50197.objecter _send_command 13 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7faeec020fb0 con 0x7faeec020b90 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.671+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec020b90 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.672+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec020fb0 con 0x7faeec020b90 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.672+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.672+0000 7faefe7fc700 10 client.50197.objecter _finish_command 13 = system:0 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.672+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 4 2024-03-30T15:05:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.672+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 6 13 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 14 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 14 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec020470 osd=6 3 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 6 14 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 4 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 10 client.50197.objecter _send_command 14 2024-03-30T15:05:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.690+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec020fb0 con 0x7faeec020b90 2024-03-30T15:05:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec020fb0 con 0x7faeec020b90 2024-03-30T15:05:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faefe7fc700 10 client.50197.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 4 2024-03-30T15:05:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 6 14 2024-03-30T15:05:08.693 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.691+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 15 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 15 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 7 15 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 4 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 10 client.50197.objecter _send_command 15 2024-03-30T15:05:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7faeec027460 con 0x7faeec0211e0 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.692+0000 7faefe7fc700 10 client.50197.objecter ms_handle_connect 0x7faeec0211e0 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.693+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faeec027460 con 0x7faeec0211e0 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.693+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.693+0000 7faefe7fc700 10 client.50197.objecter _finish_command 15 = system:0 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.693+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 4 2024-03-30T15:05:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.693+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 7 15 2024-03-30T15:05:08.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:08.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 10 client.50197.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign -1 16 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 20 client.50197.objecter _get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_remove -1 16 2024-03-30T15:05:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 20 client.50197.objecter get_session s=0x7faeec020fb0 osd=7 3 2024-03-30T15:05:08.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 15 client.50197.objecter _session_command_op_assign 7 16 2024-03-30T15:05:08.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 4 2024-03-30T15:05:08.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 10 client.50197.objecter _send_command 16 2024-03-30T15:05:08.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.709+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeec027460 con 0x7faeec0211e0 2024-03-30T15:05:08.713 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faefe7fc700 1 -- 172.21.15.28:0/4230644476 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7faeec027460 con 0x7faeec0211e0 2024-03-30T15:05:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faefe7fc700 10 client.50197.objecter ms_dispatch 0x7faf08001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faefe7fc700 10 client.50197.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faefe7fc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 4 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faefe7fc700 15 client.50197.objecter _session_command_op_remove 7 16 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 10 librados: watch_flush enter 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 10 librados: watch_flush exit 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.0 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6801/3394323575 conn(0x7faeec004730 legacy=0x7faeec002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec0053e0 osd=0 1 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.1 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6805/1850313547 conn(0x7faeec004280 legacy=0x7faeec00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec004110 osd=1 1 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.2 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.711+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6809/3451325043 conn(0x7faeec00d5d0 legacy=0x7faeec00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0db69700 1 -- 172.21.15.28:0/4230644476 reap_dead start 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec008370 osd=2 1 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.3 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6813/1506157250 conn(0x7faeec012410 legacy=0x7faeec014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00cc20 osd=3 1 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.4 2024-03-30T15:05:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.134:6800/3063775514 conn(0x7faeec011f70 legacy=0x7faeec015f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec00c450 osd=4 1 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.5 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.134:6804/3668151179 conn(0x7faeec01af80 legacy=0x7faeec01d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec01a900 osd=5 1 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.6 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.134:6808/2135224194 conn(0x7faeec020b90 legacy=0x7faeec01eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020470 osd=6 1 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter close_session for osd.7 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.134:6812/3758190298 conn(0x7faeec0211e0 legacy=0x7faeec025bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter put_session s=0x7faeec020fb0 osd=7 1 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 20 client.50197.objecter shutdown clearing up homeless session... 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 10 client.50197.objecter successfully canceled tick 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0db69700 1 -- 172.21.15.28:0/4230644476 reap_dead start 2024-03-30T15:05:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.28:6800/3011431093 conn(0x7faef40704d0 legacy=0x7faef4072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> v1:172.21.15.134:6789/0 conn(0x7faf0810ea70 legacy=0x7faf081a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 shutdown_connections 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 >> 172.21.15.28:0/4230644476 conn(0x7faf080fdaf0 msgr2=0x7faf08111c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 shutdown_connections 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 -- 172.21.15.28:0/4230644476 wait complete. 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:08.712+0000 7faf0f5cc700 1 librados: shutdown 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:08.719 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:08.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:08.781 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:08.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:08.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:08.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:05:08.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:08.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:08.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:09.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:09.155 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:09 smithi134 ceph-mon[33714]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:09.155 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:09 smithi134 ceph-mon[33714]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T15:05:09.155 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.155 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.155 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4230644476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:09.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:09.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[29876]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:09.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[29876]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T15:05:09.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4230644476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[41614]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[41614]: osdmap e489: 8 total, 8 up, 8 in 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:09.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4230644476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:09.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:09.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:09.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:09.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:09.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:09.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:09.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:09.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:09.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:09.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:09.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:10.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:10.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:10.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:10.351 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:10.351 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:10 smithi134 ceph-mon[33714]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T15:05:10.351 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:10.351 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:10 smithi134 ceph-mon[33714]: pgmap v703: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 780 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:10.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:10.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:10.634 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:10.634 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:10.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:10.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:10.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:10.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[29876]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[29876]: pgmap v703: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 780 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[41614]: osdmap e490: 8 total, 8 up, 8 in 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:10.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:10 smithi028 ceph-mon[41614]: pgmap v703: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 780 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:10.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:10.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:10.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:10.997 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:05:10.998 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:05:10.998 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-30T15:05:10.998 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-30T15:05:10.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:05:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:11.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:11.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:11.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:11.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:11.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:05:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:11.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:11.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.394+0000 7f8aac5b1700 1 -- 172.21.15.28:0/2500319248 >> v1:172.21.15.28:6790/0 conn(0x7f8aa4101130 legacy=0x7f8aa4101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.394+0000 7f8aac5b1700 1 -- 172.21.15.28:0/2500319248 shutdown_connections 2024-03-30T15:05:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.395+0000 7f8aac5b1700 1 -- 172.21.15.28:0/2500319248 >> 172.21.15.28:0/2500319248 conn(0x7f8aa40fc9e0 msgr2=0x7f8aa40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.395+0000 7f8aac5b1700 1 -- 172.21.15.28:0/2500319248 shutdown_connections 2024-03-30T15:05:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.395+0000 7f8aac5b1700 1 -- 172.21.15.28:0/2500319248 wait complete. 2024-03-30T15:05:11.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T15:05:11.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.397 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.395+0000 7f8aac5b1700 1 librados: starting msgr at 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.395+0000 7f8aac5b1700 1 librados: starting objecter 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 Processor -- start 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 -- start start 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 librados: setting wanted keys 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 librados: calling monclient init 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8aa4106900 con 0x7f8aa4109dc0 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8aa41a7530 con 0x7f8aa410d960 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aaa34d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8aa4101130 0x7f8aa4103750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56306/0 (socket says 172.21.15.28:56306) 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aac5b1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8aa41a8710 con 0x7f8aa4101130 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8aaa34d700 1 -- 172.21.15.28:0/4097869800 learned_addr learned my addr 172.21.15.28:0/4097869800 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f8aa4101130 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f8aa4109dc0 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.396+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142316316 0 0) 0x7f8aa41a7530 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a900035b0 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4079633305 0 0) 0x7f8aa41a8710 con 0x7f8aa4101130 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa41a7530 con 0x7f8aa4101130 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2399651921 0 0) 0x7f8a900035b0 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8aa41a8710 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8a9c001ae0 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 85851993 0 0) 0x7f8aa41a8710 con 0x7f8aa410d960 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 >> v1:172.21.15.28:6790/0 conn(0x7f8aa4101130 legacy=0x7f8aa4103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 >> v1:172.21.15.28:6789/0 conn(0x7f8aa4109dc0 legacy=0x7f8aa4103e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa41a98f0 con 0x7f8aa410d960 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa41a7760 con 0x7f8aa410d960 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8aac5b1700 10 client.50203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8aa41a7bc0 con 0x7f8aa410d960 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8aac5b1700 1 librados: init done 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.397+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8a9c003ec0 con 0x7f8aa410d960 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.398+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8a9c004df0 con 0x7f8aa410d960 2024-03-30T15:05:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.398+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a88005120 con 0x7f8aa410d960 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.399+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8a9c006110 con 0x7f8aa410d960 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.399+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(492..492 src has 53..492) ==== 8204+0+0 (unknown 3722751153 0 0) 0x7f8a9c088990 con 0x7f8aa410d960 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.399+0000 7f8a9b7fe700 10 client.50203.objecter ms_dispatch 0x7f8aa4001550 osd_map(492..492 src has 53..492) 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.399+0000 7f8a9b7fe700 3 client.50203.objecter handle_osd_map got epochs [492,492] > 0 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.399+0000 7f8a9b7fe700 3 client.50203.objecter handle_osd_map decoding full epoch 492 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.400+0000 7f8a9b7fe700 20 client.50203.objecter dump_active .. 0 homeless 2024-03-30T15:05:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.400+0000 7f8a9b7fe700 10 client.50203.objecter ms_handle_connect 0x7f8a90070090 2024-03-30T15:05:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.404+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8a9c055b50 con 0x7f8aa410d960 2024-03-30T15:05:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:11.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:11.491 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:11.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:11.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:05:11.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: osdmap e491: 8 total, 8 up, 8 in 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-77"}]: dispatch 2024-03-30T15:05:11.613 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:11.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:11.641+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8a88005410 con 0x7f8aa410d960 2024-03-30T15:05:11.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:11.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:11.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:11.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:11.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:12.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:12.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:12.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:12.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:12.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:12.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[29876]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T15:05:12.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4097869800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[29876]: from='client.50203 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[29876]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.313+0000 7f8a9b7fe700 1 -- 172.21.15.28:0/4097869800 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v493) ==== 91+0+0 (unknown 3367606226 0 0) 0x7f8a9c0595a0 con 0x7f8aa410d960 2024-03-30T15:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.317+0000 7f8aac5b1700 10 librados: watch_flush enter 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.317+0000 7f8aac5b1700 10 librados: watch_flush exit 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.317+0000 7f8aac5b1700 20 client.50203.objecter shutdown clearing up homeless session... 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.317+0000 7f8aac5b1700 10 client.50203.objecter successfully canceled tick 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8a90070090 legacy=0x7f8a90072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 >> v1:172.21.15.134:6789/0 conn(0x7f8aa410d960 legacy=0x7f8aa41a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 shutdown_connections 2024-03-30T15:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 >> 172.21.15.28:0/4097869800 conn(0x7f8aa40fc9e0 msgr2=0x7f8aa41105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 shutdown_connections 2024-03-30T15:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 -- 172.21.15.28:0/4097869800 wait complete. 2024-03-30T15:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:12.318+0000 7f8aac5b1700 1 librados: shutdown 2024-03-30T15:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:05:12.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:12.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:12.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:12.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:12 smithi134 ceph-mon[33714]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T15:05:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4097869800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:12 smithi134 ceph-mon[33714]: from='client.50203 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:12 smithi134 ceph-mon[33714]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:12.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[41614]: osdmap e492: 8 total, 8 up, 8 in 2024-03-30T15:05:12.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4097869800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[41614]: from='client.50203 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:12.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:12 smithi028 ceph-mon[41614]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T15:05:12.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:12.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:12.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:12.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:12.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:12.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:13.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:13.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:13.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:13.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:13.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:13.291 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:13.291 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:13.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:13.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:13.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:13.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:13.591 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:13 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:13.591 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:13 smithi134 ceph-mon[33714]: from='client.50203 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:13.591 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:13 smithi134 ceph-mon[33714]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T15:05:13.591 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.591 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:13.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[29876]: from='client.50203 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[29876]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[41614]: from='client.50203 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[41614]: osdmap e493: 8 total, 8 up, 8 in 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:13.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:13.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:13.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:05:13.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:13.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:13.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:14.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:14.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:14.131 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:14.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:14.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:14.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:14.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:14.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:14.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:14.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:14.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: 101.7 scrub starts 2024-03-30T15:05:14.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: 101.7 scrub ok 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[29876]: pgmap v709: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: 101.7 scrub starts 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: 101.7 scrub ok 2024-03-30T15:05:14.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:14.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T15:05:14.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:14.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:05:14.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:05:14.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:14 smithi028 ceph-mon[41614]: pgmap v709: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.576+0000 7f37f07ec700 1 -- 172.21.15.28:0/385249215 >> v1:172.21.15.28:6789/0 conn(0x7f37e8101180 legacy=0x7f37e8101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.576+0000 7f37f07ec700 1 -- 172.21.15.28:0/385249215 shutdown_connections 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.576+0000 7f37f07ec700 1 -- 172.21.15.28:0/385249215 >> 172.21.15.28:0/385249215 conn(0x7f37e80fca30 msgr2=0x7f37e80fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.576+0000 7f37f07ec700 1 -- 172.21.15.28:0/385249215 shutdown_connections 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.576+0000 7f37f07ec700 1 -- 172.21.15.28:0/385249215 wait complete. 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 librados: starting msgr at 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 librados: starting objecter 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 Processor -- start 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 -- start start 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 librados: setting wanted keys 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 librados: calling monclient init 2024-03-30T15:05:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37e81a92f0 con 0x7f37e810d9b0 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37e81aa510 con 0x7f37e8109e10 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37f07ec700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f37e81ab750 con 0x7f37e8101180 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37ee588700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f37e8101180 0x7f37e8106520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56328/0 (socket says 172.21.15.28:56328) 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37ee588700 1 -- 172.21.15.28:0/912703590 learned_addr learned my addr 172.21.15.28:0/912703590 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter ms_handle_connect 0x7f37e8101180 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter ms_handle_connect 0x7f37e810d9b0 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter ms_handle_connect 0x7f37e8109e10 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778977490 0 0) 0x7f37e81ab750 con 0x7f37e8101180 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d40039c0 con 0x7f37e8101180 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533577850 0 0) 0x7f37e81a92f0 con 0x7f37e810d9b0 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e81ab750 con 0x7f37e810d9b0 2024-03-30T15:05:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2180044255 0 0) 0x7f37e81aa510 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.577+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37e81a92f0 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 844042758 0 0) 0x7f37d40039c0 con 0x7f37e8101180 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37e81aa510 con 0x7f37e8101180 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3758649034 0 0) 0x7f37e81a92f0 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d40039c0 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 207901803 0 0) 0x7f37e81ab750 con 0x7f37e810d9b0 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37e81a92f0 con 0x7f37e810d9b0 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37e4001350 con 0x7f37e8101180 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37d8001890 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37e0001bb0 con 0x7f37e810d9b0 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2716210273 0 0) 0x7f37d40039c0 con 0x7f37e8109e10 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 >> v1:172.21.15.28:6790/0 conn(0x7f37e8101180 legacy=0x7f37e8106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 >> v1:172.21.15.28:6789/0 conn(0x7f37e810d9b0 legacy=0x7f37e81a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e81ac990 con 0x7f37e8109e10 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37e81a9520 con 0x7f37e8109e10 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37f07ec700 10 client.50209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f37e81a99e0 con 0x7f37e8109e10 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37f07ec700 1 librados: init done 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f37d8003b20 con 0x7f37e8109e10 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.578+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f37d8004880 con 0x7f37e8109e10 2024-03-30T15:05:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.579+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc005120 con 0x7f37e8109e10 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.579+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f37d8003680 con 0x7f37e8109e10 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(495..495 src has 442..495) ==== 8589+0+0 (unknown 2381992141 0 0) 0x7f37d80884a0 con 0x7f37e8109e10 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 10 client.50209.objecter ms_dispatch 0x7f37e8001550 osd_map(495..495 src has 442..495) 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 3 client.50209.objecter handle_osd_map got epochs [495,495] > 0 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 3 client.50209.objecter handle_osd_map decoding full epoch 495 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 20 client.50209.objecter dump_active .. 0 homeless 2024-03-30T15:05:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.580+0000 7f37df7fe700 10 client.50209.objecter ms_handle_connect 0x7f37d40703d0 2024-03-30T15:05:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.585+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f37d80554e0 con 0x7f37e8109e10 2024-03-30T15:05:14.586 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: 101.7 scrub starts 2024-03-30T15:05:14.586 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: 101.7 scrub ok 2024-03-30T15:05:14.586 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:14.586 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: osdmap e494: 8 total, 8 up, 8 in 2024-03-30T15:05:14.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:14.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:05:14.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:05:14.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:14 smithi134 ceph-mon[33714]: pgmap v709: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:14.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:14.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:14.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:14.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.788+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f37cc000b90 con 0x7f37d40703d0 2024-03-30T15:05:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.805+0000 7f37df7fe700 1 -- 172.21.15.28:0/912703590 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+667650 (unknown 2965378022 0 86688410) 0x7f37cc000b90 con 0x7f37d40703d0 2024-03-30T15:05:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:14.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:14.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:05:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.809+0000 7f37f07ec700 10 librados: watch_flush enter 2024-03-30T15:05:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.809+0000 7f37f07ec700 10 librados: watch_flush exit 2024-03-30T15:05:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 20 client.50209.objecter shutdown clearing up homeless session... 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 10 client.50209.objecter successfully canceled tick 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 >> v1:172.21.15.28:6800/3011431093 conn(0x7f37d40703d0 legacy=0x7f37d4072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 >> v1:172.21.15.134:6789/0 conn(0x7f37e8109e10 legacy=0x7f37e81a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 shutdown_connections 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 >> 172.21.15.28:0/912703590 conn(0x7f37e80fca30 msgr2=0x7f37e806ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 shutdown_connections 2024-03-30T15:05:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 -- 172.21.15.28:0/912703590 wait complete. 2024-03-30T15:05:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:14.810+0000 7f37f07ec700 1 librados: shutdown 2024-03-30T15:05:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:05:14.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:14.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:14.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:15.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:15.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:15.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:15.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:15.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:15.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:15.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 12.1d scrub starts 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 12.1d scrub ok 2024-03-30T15:05:15.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: from='client.50209 v1:172.21.15.28:0/912703590' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:15.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:15.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 12.1d scrub starts 2024-03-30T15:05:15.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 12.1d scrub ok 2024-03-30T15:05:15.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: from='client.50209 v1:172.21.15.28:0/912703590' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:15.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:05:15.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:15 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:05:15.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:15.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:15.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:05:15.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:05:15.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: osdmap e495: 8 total, 8 up, 8 in 2024-03-30T15:05:15.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 12.1d scrub starts 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 12.1d scrub ok 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: from='client.50209 v1:172.21.15.28:0/912703590' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:05:15.668 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:15 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:05:15.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:15.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:15.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:15.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:15.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:15.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:15.957 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2024-03-30T15:05:14.322595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792270,"num_objects":376,"num_object_clones":10,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":2109,"num_read_kb":1851,"num_write":1539,"num_write_kb":15185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2940,"ondisk_log_size":2940,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":753,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":618064,"kb_used_data":92344,"kb_used_omap":12,"kb_used_meta":524211,"kb_avail":749343152,"statfs":{"total":767960285184,"available":767327387648,"internally_reserved":0,"allocated":94560256,"data_stored":64815872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":536792624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025351"},"pg_stats":[{"pgid":"304.1f","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344354+0000","last_change":"2024-03-30T15:05:13.343866+0000","last_active":"2024-03-30T15:05:13.344354+0000","last_peered":"2024-03-30T15:05:13.344354+0000","last_clean":"2024-03-30T15:05:13.344354+0000","last_became_active":"2024-03-30T15:05:13.343591+0000","last_became_peered":"2024-03-30T15:05:13.343591+0000","last_unstale":"2024-03-30T15:05:13.344354+0000","last_undegraded":"2024-03-30T15:05:13.344354+0000","last_fullsized":"2024-03-30T15:05:13.344354+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:26.119005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1e","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344486+0000","last_change":"2024-03-30T15:05:13.344170+0000","last_active":"2024-03-30T15:05:13.344486+0000","last_peered":"2024-03-30T15:05:13.344486+0000","last_clean":"2024-03-30T15:05:13.344486+0000","last_became_active":"2024-03-30T15:05:13.343841+0000","last_became_peered":"2024-03-30T15:05:13.343841+0000","last_unstale":"2024-03-30T15:05:13.344486+0000","last_undegraded":"2024-03-30T15:05:13.344486+0000","last_fullsized":"2024-03-30T15:05:13.344486+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.509434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1d","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338819+0000","last_change":"2024-03-30T15:05:13.337980+0000","last_active":"2024-03-30T15:05:13.338819+0000","last_peered":"2024-03-30T15:05:13.338819+0000","last_clean":"2024-03-30T15:05:13.338819+0000","last_became_active":"2024-03-30T15:05:13.337694+0000","last_became_peered":"2024-03-30T15:05:13.337694+0000","last_unstale":"2024-03-30T15:05:13.338819+0000","last_undegraded":"2024-03-30T15:05:13.338819+0000","last_fullsized":"2024-03-30T15:05:13.338819+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.205653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"304.1c","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332390+0000","last_change":"2024-03-30T15:05:13.332109+0000","last_active":"2024-03-30T15:05:13.332390+0000","last_peered":"2024-03-30T15:05:13.332390+0000","last_clean":"2024-03-30T15:05:13.332390+0000","last_became_active":"2024-03-30T15:05:13.331788+0000","last_became_peered":"2024-03-30T15:05:13.331788+0000","last_unstale":"2024-03-30T15:05:13.332390+0000","last_undegraded":"2024-03-30T15:05:13.332390+0000","last_fullsized":"2024-03-30T15:05:13.332390+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:41.628138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.1b","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344348+0000","last_change":"2024-03-30T15:05:13.344037+0000","last_active":"2024-03-30T15:05:13.344348+0000","last_peered":"2024-03-30T15:05:13.344348+0000","last_clean":"2024-03-30T15:05:13.344348+0000","last_became_active":"2024-03-30T15:05:13.343558+0000","last_became_peered":"2024-03-30T15:05:13.343558+0000","last_unstale":"2024-03-30T15:05:13.344348+0000","last_undegraded":"2024-03-30T15:05:13.344348+0000","last_fullsized":"2024-03-30T15:05:13.344348+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.022174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1a","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.337173+0000","last_change":"2024-03-30T15:05:13.336866+0000","last_active":"2024-03-30T15:05:13.337173+0000","last_peered":"2024-03-30T15:05:13.337173+0000","last_clean":"2024-03-30T15:05:13.337173+0000","last_became_active":"2024-03-30T15:05:13.336536+0000","last_became_peered":"2024-03-30T15:05:13.336536+0000","last_unstale":"2024-03-30T15:05:13.337173+0000","last_undegraded":"2024-03-30T15:05:13.337173+0000","last_fullsized":"2024-03-30T15:05:13.337173+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.874284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.19","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.347525+0000","last_change":"2024-03-30T15:05:13.347183+0000","last_active":"2024-03-30T15:05:13.347525+0000","last_peered":"2024-03-30T15:05:13.347525+0000","last_clean":"2024-03-30T15:05:13.347525+0000","last_became_active":"2024-03-30T15:05:13.346878+0000","last_became_peered":"2024-03-30T15:05:13.346878+0000","last_unstale":"2024-03-30T15:05:13.347525+0000","last_undegraded":"2024-03-30T15:05:13.347525+0000","last_fullsized":"2024-03-30T15:05:13.347525+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:26.319998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.18","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338794+0000","last_change":"2024-03-30T15:05:13.336581+0000","last_active":"2024-03-30T15:05:13.338794+0000","last_peered":"2024-03-30T15:05:13.338794+0000","last_clean":"2024-03-30T15:05:13.338794+0000","last_became_active":"2024-03-30T15:05:13.336207+0000","last_became_peered":"2024-03-30T15:05:13.336207+0000","last_unstale":"2024-03-30T15:05:13.338794+0000","last_undegraded":"2024-03-30T15:05:13.338794+0000","last_fullsized":"2024-03-30T15:05:13.338794+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.17","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.340303+0000","last_change":"2024-03-30T15:05:13.339981+0000","last_active":"2024-03-30T15:05:13.340303+0000","last_peered":"2024-03-30T15:05:13.340303+0000","last_clean":"2024-03-30T15:05:13.340303+0000","last_became_active":"2024-03-30T15:05:13.339642+0000","last_became_peered":"2024-03-30T15:05:13.339642+0000","last_unstale":"2024-03-30T15:05:13.340303+0000","last_undegraded":"2024-03-30T15:05:13.340303+0000","last_fullsized":"2024-03-30T15:05:13.340303+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:12.774909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"304.16","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344714+0000","last_change":"2024-03-30T15:05:13.343874+0000","last_active":"2024-03-30T15:05:13.344714+0000","last_peered":"2024-03-30T15:05:13.344714+0000","last_clean":"2024-03-30T15:05:13.344714+0000","last_became_active":"2024-03-30T15:05:13.343568+0000","last_became_peered":"2024-03-30T15:05:13.343568+0000","last_unstale":"2024-03-30T15:05:13.344714+0000","last_undegraded":"2024-03-30T15:05:13.344714+0000","last_fullsized":"2024-03-30T15:05:13.344714+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.792022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.15","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338539+0000","last_change":"2024-03-30T15:05:13.334764+0000","last_active":"2024-03-30T15:05:13.338539+0000","last_peered":"2024-03-30T15:05:13.338539+0000","last_clean":"2024-03-30T15:05:13.338539+0000","last_became_active":"2024-03-30T15:05:13.334353+0000","last_became_peered":"2024-03-30T15:05:13.334353+0000","last_unstale":"2024-03-30T15:05:13.338539+0000","last_undegraded":"2024-03-30T15:05:13.338539+0000","last_fullsized":"2024-03-30T15:05:13.338539+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.706627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.14","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329881+0000","last_change":"2024-03-30T15:05:13.329452+0000","last_active":"2024-03-30T15:05:13.329881+0000","last_peered":"2024-03-30T15:05:13.329881+0000","last_clean":"2024-03-30T15:05:13.329881+0000","last_became_active":"2024-03-30T15:05:13.329130+0000","last_became_peered":"2024-03-30T15:05:13.329130+0000","last_unstale":"2024-03-30T15:05:13.329881+0000","last_undegraded":"2024-03-30T15:05:13.329881+0000","last_fullsized":"2024-03-30T15:05:13.329881+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.13","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.345256+0000","last_change":"2024-03-30T15:05:13.344097+0000","last_active":"2024-03-30T15:05:13.345256+0000","last_peered":"2024-03-30T15:05:13.345256+0000","last_clean":"2024-03-30T15:05:13.345256+0000","last_became_active":"2024-03-30T15:05:13.343627+0000","last_became_peered":"2024-03-30T15:05:13.343627+0000","last_unstale":"2024-03-30T15:05:13.345256+0000","last_undegraded":"2024-03-30T15:05:13.345256+0000","last_fullsized":"2024-03-30T15:05:13.345256+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.426729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.12","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338198+0000","last_change":"2024-03-30T15:05:13.336998+0000","last_active":"2024-03-30T15:05:13.338198+0000","last_peered":"2024-03-30T15:05:13.338198+0000","last_clean":"2024-03-30T15:05:13.338198+0000","last_became_active":"2024-03-30T15:05:13.336699+0000","last_became_peered":"2024-03-30T15:05:13.336699+0000","last_unstale":"2024-03-30T15:05:13.338198+0000","last_undegraded":"2024-03-30T15:05:13.338198+0000","last_fullsized":"2024-03-30T15:05:13.338198+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.404059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.11","version":"494'1","reported_seq":11,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.339002+0000","last_change":"2024-03-30T15:05:13.329376+0000","last_active":"2024-03-30T15:05:13.339002+0000","last_peered":"2024-03-30T15:05:13.339002+0000","last_clean":"2024-03-30T15:05:13.339002+0000","last_became_active":"2024-03-30T15:05:13.329076+0000","last_became_peered":"2024-03-30T15:05:13.329076+0000","last_unstale":"2024-03-30T15:05:13.339002+0000","last_undegraded":"2024-03-30T15:05:13.339002+0000","last_fullsized":"2024-03-30T15:05:13.339002+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:18.397376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.10","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.345867+0000","last_change":"2024-03-30T15:05:13.345632+0000","last_active":"2024-03-30T15:05:13.345867+0000","last_peered":"2024-03-30T15:05:13.345867+0000","last_clean":"2024-03-30T15:05:13.345867+0000","last_became_active":"2024-03-30T15:05:13.345171+0000","last_became_peered":"2024-03-30T15:05:13.345171+0000","last_unstale":"2024-03-30T15:05:13.345867+0000","last_undegraded":"2024-03-30T15:05:13.345867+0000","last_fullsized":"2024-03-30T15:05:13.345867+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:18.260298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.f","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338435+0000","last_change":"2024-03-30T15:05:13.336249+0000","last_active":"2024-03-30T15:05:13.338435+0000","last_peered":"2024-03-30T15:05:13.338435+0000","last_clean":"2024-03-30T15:05:13.338435+0000","last_became_active":"2024-03-30T15:05:13.335669+0000","last_became_peered":"2024-03-30T15:05:13.335669+0000","last_unstale":"2024-03-30T15:05:13.338435+0000","last_undegraded":"2024-03-30T15:05:13.338435+0000","last_fullsized":"2024-03-30T15:05:13.338435+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.907258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.e","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338269+0000","last_change":"2024-03-30T15:05:13.338019+0000","last_active":"2024-03-30T15:05:13.338269+0000","last_peered":"2024-03-30T15:05:13.338269+0000","last_clean":"2024-03-30T15:05:13.338269+0000","last_became_active":"2024-03-30T15:05:13.337705+0000","last_became_peered":"2024-03-30T15:05:13.337705+0000","last_unstale":"2024-03-30T15:05:13.338269+0000","last_undegraded":"2024-03-30T15:05:13.338269+0000","last_fullsized":"2024-03-30T15:05:13.338269+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:36.592287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.d","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338518+0000","last_change":"2024-03-30T15:05:13.333258+0000","last_active":"2024-03-30T15:05:13.338518+0000","last_peered":"2024-03-30T15:05:13.338518+0000","last_clean":"2024-03-30T15:05:13.338518+0000","last_became_active":"2024-03-30T15:05:13.332983+0000","last_became_peered":"2024-03-30T15:05:13.332983+0000","last_unstale":"2024-03-30T15:05:13.338518+0000","last_undegraded":"2024-03-30T15:05:13.338518+0000","last_fullsized":"2024-03-30T15:05:13.338518+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:24.150335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.c","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.346255+0000","last_change":"2024-03-30T15:05:13.343804+0000","last_active":"2024-03-30T15:05:13.346255+0000","last_peered":"2024-03-30T15:05:13.346255+0000","last_clean":"2024-03-30T15:05:13.346255+0000","last_became_active":"2024-03-30T15:05:13.343461+0000","last_became_peered":"2024-03-30T15:05:13.343461+0000","last_unstale":"2024-03-30T15:05:13.346255+0000","last_undegraded":"2024-03-30T15:05:13.346255+0000","last_fullsized":"2024-03-30T15:05:13.346255+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:20.599182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.b","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344515+0000","last_change":"2024-03-30T15:05:13.343804+0000","last_active":"2024-03-30T15:05:13.344515+0000","last_peered":"2024-03-30T15:05:13.344515+0000","last_clean":"2024-03-30T15:05:13.344515+0000","last_became_active":"2024-03-30T15:05:13.343512+0000","last_became_peered":"2024-03-30T15:05:13.343512+0000","last_unstale":"2024-03-30T15:05:13.344515+0000","last_undegraded":"2024-03-30T15:05:13.344515+0000","last_fullsized":"2024-03-30T15:05:13.344515+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.192366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.a","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332528+0000","last_change":"2024-03-30T15:05:13.332286+0000","last_active":"2024-03-30T15:05:13.332528+0000","last_peered":"2024-03-30T15:05:13.332528+0000","last_clean":"2024-03-30T15:05:13.332528+0000","last_became_active":"2024-03-30T15:05:13.331872+0000","last_became_peered":"2024-03-30T15:05:13.331872+0000","last_unstale":"2024-03-30T15:05:13.332528+0000","last_undegraded":"2024-03-30T15:05:13.332528+0000","last_fullsized":"2024-03-30T15:05:13.332528+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:26.528562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.9","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.346006+0000","last_change":"2024-03-30T15:05:13.344837+0000","last_active":"2024-03-30T15:05:13.346006+0000","last_peered":"2024-03-30T15:05:13.346006+0000","last_clean":"2024-03-30T15:05:13.346006+0000","last_became_active":"2024-03-30T15:05:13.343863+0000","last_became_peered":"2024-03-30T15:05:13.343863+0000","last_unstale":"2024-03-30T15:05:13.346006+0000","last_undegraded":"2024-03-30T15:05:13.346006+0000","last_fullsized":"2024-03-30T15:05:13.346006+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.763726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.8","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.346373+0000","last_change":"2024-03-30T15:05:13.345256+0000","last_active":"2024-03-30T15:05:13.346373+0000","last_peered":"2024-03-30T15:05:13.346373+0000","last_clean":"2024-03-30T15:05:13.346373+0000","last_became_active":"2024-03-30T15:05:13.344921+0000","last_became_peered":"2024-03-30T15:05:13.344921+0000","last_unstale":"2024-03-30T15:05:13.346373+0000","last_undegraded":"2024-03-30T15:05:13.346373+0000","last_fullsized":"2024-03-30T15:05:13.346373+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:36.516647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.7","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327435+0000","last_change":"2024-03-30T15:05:13.326046+0000","last_active":"2024-03-30T15:05:13.327435+0000","last_peered":"2024-03-30T15:05:13.327435+0000","last_clean":"2024-03-30T15:05:13.327435+0000","last_became_active":"2024-03-30T15:05:13.325739+0000","last_became_peered":"2024-03-30T15:05:13.325739+0000","last_unstale":"2024-03-30T15:05:13.327435+0000","last_undegraded":"2024-03-30T15:05:13.327435+0000","last_fullsized":"2024-03-30T15:05:13.327435+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.274010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.6","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.340072+0000","last_change":"2024-03-30T15:05:13.339237+0000","last_active":"2024-03-30T15:05:13.340072+0000","last_peered":"2024-03-30T15:05:13.340072+0000","last_clean":"2024-03-30T15:05:13.340072+0000","last_became_active":"2024-03-30T15:05:13.338947+0000","last_became_peered":"2024-03-30T15:05:13.338947+0000","last_unstale":"2024-03-30T15:05:13.340072+0000","last_undegraded":"2024-03-30T15:05:13.340072+0000","last_fullsized":"2024-03-30T15:05:13.340072+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.939848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.5","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.339912+0000","last_change":"2024-03-30T15:05:13.338870+0000","last_active":"2024-03-30T15:05:13.339912+0000","last_peered":"2024-03-30T15:05:13.339912+0000","last_clean":"2024-03-30T15:05:13.339912+0000","last_became_active":"2024-03-30T15:05:13.338587+0000","last_became_peered":"2024-03-30T15:05:13.338587+0000","last_unstale":"2024-03-30T15:05:13.339912+0000","last_undegraded":"2024-03-30T15:05:13.339912+0000","last_fullsized":"2024-03-30T15:05:13.339912+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.493534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.4","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.346108+0000","last_change":"2024-03-30T15:05:13.344393+0000","last_active":"2024-03-30T15:05:13.346108+0000","last_peered":"2024-03-30T15:05:13.346108+0000","last_clean":"2024-03-30T15:05:13.346108+0000","last_became_active":"2024-03-30T15:05:13.343541+0000","last_became_peered":"2024-03-30T15:05:13.343541+0000","last_unstale":"2024-03-30T15:05:13.346108+0000","last_undegraded":"2024-03-30T15:05:13.346108+0000","last_fullsized":"2024-03-30T15:05:13.346108+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.446279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.3","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338393+0000","last_change":"2024-03-30T15:05:13.338045+0000","last_active":"2024-03-30T15:05:13.338393+0000","last_peered":"2024-03-30T15:05:13.338393+0000","last_clean":"2024-03-30T15:05:13.338393+0000","last_became_active":"2024-03-30T15:05:13.337722+0000","last_became_peered":"2024-03-30T15:05:13.337722+0000","last_unstale":"2024-03-30T15:05:13.338393+0000","last_undegraded":"2024-03-30T15:05:13.338393+0000","last_fullsized":"2024-03-30T15:05:13.338393+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.870353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.2","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.338177+0000","last_change":"2024-03-30T15:05:13.336923+0000","last_active":"2024-03-30T15:05:13.338177+0000","last_peered":"2024-03-30T15:05:13.338177+0000","last_clean":"2024-03-30T15:05:13.338177+0000","last_became_active":"2024-03-30T15:05:13.336565+0000","last_became_peered":"2024-03-30T15:05:13.336565+0000","last_unstale":"2024-03-30T15:05:13.338177+0000","last_undegraded":"2024-03-30T15:05:13.338177+0000","last_fullsized":"2024-03-30T15:05:13.338177+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:24.209721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335492+0000","last_change":"2024-03-30T15:05:13.335291+0000","last_active":"2024-03-30T15:05:13.335492+0000","last_peered":"2024-03-30T15:05:13.335492+0000","last_clean":"2024-03-30T15:05:13.335492+0000","last_became_active":"2024-03-30T15:05:13.335031+0000","last_became_peered":"2024-03-30T15:05:13.335031+0000","last_unstale":"2024-03-30T15:05:13.335492+0000","last_undegraded":"2024-03-30T15:05:13.335492+0000","last_fullsized":"2024-03-30T15:05:13.335492+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.153064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.0","version":"0'0","reported_seq":10,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.346394+0000","last_change":"2024-03-30T15:05:13.345724+0000","last_active":"2024-03-30T15:05:13.346394+0000","last_peered":"2024-03-30T15:05:13.346394+0000","last_clean":"2024-03-30T15:05:13.346394+0000","last_became_active":"2024-03-30T15:05:13.344827+0000","last_became_peered":"2024-03-30T15:05:13.344827+0000","last_unstale":"2024-03-30T15:05:13.346394+0000","last_undegraded":"2024-03-30T15:05:13.346394+0000","last_fullsized":"2024-03-30T15:05:13.346394+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:26.349960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":983,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319016+0000","last_change":"2024-03-30T15:04:06.344710+0000","last_active":"2024-03-30T15:05:13.319016+0000","last_peered":"2024-03-30T15:05:13.319016+0000","last_clean":"2024-03-30T15:05:13.319016+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:05:13.319016+0000","last_undegraded":"2024-03-30T15:05:13.319016+0000","last_fullsized":"2024-03-30T15:05:13.319016+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":1003,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332004+0000","last_change":"2024-03-30T15:04:06.379921+0000","last_active":"2024-03-30T15:05:13.332004+0000","last_peered":"2024-03-30T15:05:13.332004+0000","last_clean":"2024-03-30T15:05:13.332004+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:05:13.332004+0000","last_undegraded":"2024-03-30T15:05:13.332004+0000","last_fullsized":"2024-03-30T15:05:13.332004+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:31.961438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.961438+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"474'106","reported_seq":1028,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326572+0000","last_change":"2024-03-30T15:04:06.471415+0000","last_active":"2024-03-30T15:05:13.326572+0000","last_peered":"2024-03-30T15:05:13.326572+0000","last_clean":"2024-03-30T15:05:13.326572+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:05:13.326572+0000","last_undegraded":"2024-03-30T15:05:13.326572+0000","last_fullsized":"2024-03-30T15:05:13.326572+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'87","last_scrub_stamp":"2024-03-30T15:02:40.447037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.447037+0000","objects_scrubbed":7,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":98,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":1040,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323398+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:05:13.323398+0000","last_peered":"2024-03-30T15:05:13.323398+0000","last_clean":"2024-03-30T15:05:13.323398+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:05:13.323398+0000","last_undegraded":"2024-03-30T15:05:13.323398+0000","last_fullsized":"2024-03-30T15:05:13.323398+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.992843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"491'43","reported_seq":987,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.320263+0000","last_change":"2024-03-30T15:04:06.346222+0000","last_active":"2024-03-30T15:05:13.320263+0000","last_peered":"2024-03-30T15:05:13.320263+0000","last_clean":"2024-03-30T15:05:13.320263+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:05:13.320263+0000","last_undegraded":"2024-03-30T15:05:13.320263+0000","last_fullsized":"2024-03-30T15:05:13.320263+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:03:19.020165+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.020165+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":1075,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.320817+0000","last_change":"2024-03-30T15:04:06.347583+0000","last_active":"2024-03-30T15:05:13.320817+0000","last_peered":"2024-03-30T15:05:13.320817+0000","last_clean":"2024-03-30T15:05:13.320817+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:05:13.320817+0000","last_undegraded":"2024-03-30T15:05:13.320817+0000","last_fullsized":"2024-03-30T15:05:13.320817+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":965,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331999+0000","last_change":"2024-03-30T15:04:06.381417+0000","last_active":"2024-03-30T15:05:13.331999+0000","last_peered":"2024-03-30T15:05:13.331999+0000","last_clean":"2024-03-30T15:05:13.331999+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:05:13.331999+0000","last_undegraded":"2024-03-30T15:05:13.331999+0000","last_fullsized":"2024-03-30T15:05:13.331999+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":985,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333070+0000","last_change":"2024-03-30T15:04:06.473326+0000","last_active":"2024-03-30T15:05:13.333070+0000","last_peered":"2024-03-30T15:05:13.333070+0000","last_clean":"2024-03-30T15:05:13.333070+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:05:13.333070+0000","last_undegraded":"2024-03-30T15:05:13.333070+0000","last_fullsized":"2024-03-30T15:05:13.333070+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.599669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":985,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324111+0000","last_change":"2024-03-30T15:04:34.489872+0000","last_active":"2024-03-30T15:05:13.324111+0000","last_peered":"2024-03-30T15:05:13.324111+0000","last_clean":"2024-03-30T15:05:13.324111+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:05:13.324111+0000","last_undegraded":"2024-03-30T15:05:13.324111+0000","last_fullsized":"2024-03-30T15:05:13.324111+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:04:34.489778+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.489778+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.956502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":969,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324429+0000","last_change":"2024-03-30T15:04:06.474203+0000","last_active":"2024-03-30T15:05:13.324429+0000","last_peered":"2024-03-30T15:05:13.324429+0000","last_clean":"2024-03-30T15:05:13.324429+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:05:13.324429+0000","last_undegraded":"2024-03-30T15:05:13.324429+0000","last_fullsized":"2024-03-30T15:05:13.324429+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:02:00.557079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:00.557079+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"494'22","reported_seq":955,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.344698+0000","last_change":"2024-03-30T15:04:06.471477+0000","last_active":"2024-03-30T15:05:13.344698+0000","last_peered":"2024-03-30T15:05:13.344698+0000","last_clean":"2024-03-30T15:05:13.344698+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:05:13.344698+0000","last_undegraded":"2024-03-30T15:05:13.344698+0000","last_fullsized":"2024-03-30T15:05:13.344698+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'5","last_scrub_stamp":"2024-03-30T15:02:29.567113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:29.567113+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"428'21","reported_seq":955,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327463+0000","last_change":"2024-03-30T15:04:06.472183+0000","last_active":"2024-03-30T15:05:13.327463+0000","last_peered":"2024-03-30T15:05:13.327463+0000","last_clean":"2024-03-30T15:05:13.327463+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:05:13.327463+0000","last_undegraded":"2024-03-30T15:05:13.327463+0000","last_fullsized":"2024-03-30T15:05:13.327463+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'17","last_scrub_stamp":"2024-03-30T15:02:35.464703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.464703+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":948,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334489+0000","last_change":"2024-03-30T15:04:06.381760+0000","last_active":"2024-03-30T15:05:13.334489+0000","last_peered":"2024-03-30T15:05:13.334489+0000","last_clean":"2024-03-30T15:05:13.334489+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:05:13.334489+0000","last_undegraded":"2024-03-30T15:05:13.334489+0000","last_fullsized":"2024-03-30T15:05:13.334489+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":1012,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326017+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:05:13.326017+0000","last_peered":"2024-03-30T15:05:13.326017+0000","last_clean":"2024-03-30T15:05:13.326017+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:05:13.326017+0000","last_undegraded":"2024-03-30T15:05:13.326017+0000","last_fullsized":"2024-03-30T15:05:13.326017+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"460'60","reported_seq":994,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325230+0000","last_change":"2024-03-30T15:04:06.474098+0000","last_active":"2024-03-30T15:05:13.325230+0000","last_peered":"2024-03-30T15:05:13.325230+0000","last_clean":"2024-03-30T15:05:13.325230+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:05:13.325230+0000","last_undegraded":"2024-03-30T15:05:13.325230+0000","last_fullsized":"2024-03-30T15:05:13.325230+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":1019,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.320567+0000","last_change":"2024-03-30T15:04:06.346253+0000","last_active":"2024-03-30T15:05:13.320567+0000","last_peered":"2024-03-30T15:05:13.320567+0000","last_clean":"2024-03-30T15:05:13.320567+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:05:13.320567+0000","last_undegraded":"2024-03-30T15:05:13.320567+0000","last_fullsized":"2024-03-30T15:05:13.320567+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"494'36","reported_seq":966,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327528+0000","last_change":"2024-03-30T15:04:06.473836+0000","last_active":"2024-03-30T15:05:13.327528+0000","last_peered":"2024-03-30T15:05:13.327528+0000","last_clean":"2024-03-30T15:05:13.327528+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:05:13.327528+0000","last_undegraded":"2024-03-30T15:05:13.327528+0000","last_fullsized":"2024-03-30T15:05:13.327528+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'35","last_scrub_stamp":"2024-03-30T15:02:42.436904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.436904+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327061+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"238'104","reported_seq":1035,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328523+0000","last_change":"2024-03-30T15:04:06.383533+0000","last_active":"2024-03-30T15:05:13.328523+0000","last_peered":"2024-03-30T15:05:13.328523+0000","last_clean":"2024-03-30T15:05:13.328523+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:05:13.328523+0000","last_undegraded":"2024-03-30T15:05:13.328523+0000","last_fullsized":"2024-03-30T15:05:13.328523+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'104","last_scrub_stamp":"2024-03-30T15:02:39.411331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.411331+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"444'55","reported_seq":997,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324206+0000","last_change":"2024-03-30T15:04:07.016310+0000","last_active":"2024-03-30T15:05:13.324206+0000","last_peered":"2024-03-30T15:05:13.324206+0000","last_clean":"2024-03-30T15:05:13.324206+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:05:13.324206+0000","last_undegraded":"2024-03-30T15:05:13.324206+0000","last_fullsized":"2024-03-30T15:05:13.324206+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":986,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330490+0000","last_change":"2024-03-30T15:04:55.161018+0000","last_active":"2024-03-30T15:05:13.330490+0000","last_peered":"2024-03-30T15:05:13.330490+0000","last_clean":"2024-03-30T15:05:13.330490+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:05:13.330490+0000","last_undegraded":"2024-03-30T15:05:13.330490+0000","last_fullsized":"2024-03-30T15:05:13.330490+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:04:55.160916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.160916+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.035072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":1044,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331004+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:05:13.331004+0000","last_peered":"2024-03-30T15:05:13.331004+0000","last_clean":"2024-03-30T15:05:13.331004+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:05:13.331004+0000","last_undegraded":"2024-03-30T15:05:13.331004+0000","last_fullsized":"2024-03-30T15:05:13.331004+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.731859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":1005,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326472+0000","last_change":"2024-03-30T15:04:35.492607+0000","last_active":"2024-03-30T15:05:13.326472+0000","last_peered":"2024-03-30T15:05:13.326472+0000","last_clean":"2024-03-30T15:05:13.326472+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:05:13.326472+0000","last_undegraded":"2024-03-30T15:05:13.326472+0000","last_fullsized":"2024-03-30T15:05:13.326472+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'60","last_scrub_stamp":"2024-03-30T15:04:35.492526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.492526+0000","objects_scrubbed":6,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:39.249210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":996,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333476+0000","last_change":"2024-03-30T15:04:06.473834+0000","last_active":"2024-03-30T15:05:13.333476+0000","last_peered":"2024-03-30T15:05:13.333476+0000","last_clean":"2024-03-30T15:05:13.333476+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:05:13.333476+0000","last_undegraded":"2024-03-30T15:05:13.333476+0000","last_fullsized":"2024-03-30T15:05:13.333476+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"400'89","reported_seq":1008,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334065+0000","last_change":"2024-03-30T15:04:06.381269+0000","last_active":"2024-03-30T15:05:13.334065+0000","last_peered":"2024-03-30T15:05:13.334065+0000","last_clean":"2024-03-30T15:05:13.334065+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:05:13.334065+0000","last_undegraded":"2024-03-30T15:05:13.334065+0000","last_fullsized":"2024-03-30T15:05:13.334065+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'87","last_scrub_stamp":"2024-03-30T15:02:01.656924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:01.656924+0000","objects_scrubbed":6,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":1027,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324699+0000","last_change":"2024-03-30T15:04:07.014089+0000","last_active":"2024-03-30T15:05:13.324699+0000","last_peered":"2024-03-30T15:05:13.324699+0000","last_clean":"2024-03-30T15:05:13.324699+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:05:13.324699+0000","last_undegraded":"2024-03-30T15:05:13.324699+0000","last_fullsized":"2024-03-30T15:05:13.324699+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.102197+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":989,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319451+0000","last_change":"2024-03-30T15:04:35.992306+0000","last_active":"2024-03-30T15:05:13.319451+0000","last_peered":"2024-03-30T15:05:13.319451+0000","last_clean":"2024-03-30T15:05:13.319451+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:05:13.319451+0000","last_undegraded":"2024-03-30T15:05:13.319451+0000","last_fullsized":"2024-03-30T15:05:13.319451+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:04:35.992167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.992167+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.226654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"452'42","reported_seq":993,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318418+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:05:13.318418+0000","last_peered":"2024-03-30T15:05:13.318418+0000","last_clean":"2024-03-30T15:05:13.318418+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:05:13.318418+0000","last_undegraded":"2024-03-30T15:05:13.318418+0000","last_fullsized":"2024-03-30T15:05:13.318418+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"487'119","reported_seq":1062,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331552+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:05:13.331552+0000","last_peered":"2024-03-30T15:05:13.331552+0000","last_clean":"2024-03-30T15:05:13.331552+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:05:13.331552+0000","last_undegraded":"2024-03-30T15:05:13.331552+0000","last_fullsized":"2024-03-30T15:05:13.331552+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000104,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":37,"num_write_kb":3940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":959,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335118+0000","last_change":"2024-03-30T15:04:06.474417+0000","last_active":"2024-03-30T15:05:13.335118+0000","last_peered":"2024-03-30T15:05:13.335118+0000","last_clean":"2024-03-30T15:05:13.335118+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:05:13.335118+0000","last_undegraded":"2024-03-30T15:05:13.335118+0000","last_fullsized":"2024-03-30T15:05:13.335118+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:02:36.478360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.478360+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327041+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":958,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.321070+0000","last_change":"2024-03-30T15:05:13.117385+0000","last_active":"2024-03-30T15:05:13.321070+0000","last_peered":"2024-03-30T15:05:13.321070+0000","last_clean":"2024-03-30T15:05:13.321070+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:05:13.321070+0000","last_undegraded":"2024-03-30T15:05:13.321070+0000","last_fullsized":"2024-03-30T15:05:13.321070+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:05:13.117303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.117303+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.803622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":994,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325290+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:05:13.325290+0000","last_peered":"2024-03-30T15:05:13.325290+0000","last_clean":"2024-03-30T15:05:13.325290+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:05:13.325290+0000","last_undegraded":"2024-03-30T15:05:13.325290+0000","last_fullsized":"2024-03-30T15:05:13.325290+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":1038,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325646+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:05:13.325646+0000","last_peered":"2024-03-30T15:05:13.325646+0000","last_clean":"2024-03-30T15:05:13.325646+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:05:13.325646+0000","last_undegraded":"2024-03-30T15:05:13.325646+0000","last_fullsized":"2024-03-30T15:05:13.325646+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324853+0000","last_change":"2024-03-30T15:04:42.056061+0000","last_active":"2024-03-30T15:05:13.324853+0000","last_peered":"2024-03-30T15:05:13.324853+0000","last_clean":"2024-03-30T15:05:13.324853+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:05:13.324853+0000","last_undegraded":"2024-03-30T15:05:13.324853+0000","last_fullsized":"2024-03-30T15:05:13.324853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.055955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.055955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.618053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":793,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329448+0000","last_change":"2024-03-30T15:04:34.266981+0000","last_active":"2024-03-30T15:05:13.329448+0000","last_peered":"2024-03-30T15:05:13.329448+0000","last_clean":"2024-03-30T15:05:13.329448+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:05:13.329448+0000","last_undegraded":"2024-03-30T15:05:13.329448+0000","last_fullsized":"2024-03-30T15:05:13.329448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:34.266865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.266865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:49.944697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335727+0000","last_change":"2024-03-30T15:04:42.277788+0000","last_active":"2024-03-30T15:05:13.335727+0000","last_peered":"2024-03-30T15:05:13.335727+0000","last_clean":"2024-03-30T15:05:13.335727+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:05:13.335727+0000","last_undegraded":"2024-03-30T15:05:13.335727+0000","last_fullsized":"2024-03-30T15:05:13.335727+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.712874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329650+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:05:13.329650+0000","last_peered":"2024-03-30T15:05:13.329650+0000","last_clean":"2024-03-30T15:05:13.329650+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:05:13.329650+0000","last_undegraded":"2024-03-30T15:05:13.329650+0000","last_fullsized":"2024-03-30T15:05:13.329650+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.852443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":780,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323340+0000","last_change":"2024-03-30T15:01:57.530926+0000","last_active":"2024-03-30T15:05:13.323340+0000","last_peered":"2024-03-30T15:05:13.323340+0000","last_clean":"2024-03-30T15:05:13.323340+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:05:13.323340+0000","last_undegraded":"2024-03-30T15:05:13.323340+0000","last_fullsized":"2024-03-30T15:05:13.323340+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.530827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.530827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":788,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318454+0000","last_change":"2024-03-30T15:02:41.928550+0000","last_active":"2024-03-30T15:05:13.318454+0000","last_peered":"2024-03-30T15:05:13.318454+0000","last_clean":"2024-03-30T15:05:13.318454+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:05:13.318454+0000","last_undegraded":"2024-03-30T15:05:13.318454+0000","last_fullsized":"2024-03-30T15:05:13.318454+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.928469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.928469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":776,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332274+0000","last_change":"2024-03-30T15:01:57.663288+0000","last_active":"2024-03-30T15:05:13.332274+0000","last_peered":"2024-03-30T15:05:13.332274+0000","last_clean":"2024-03-30T15:05:13.332274+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:05:13.332274+0000","last_undegraded":"2024-03-30T15:05:13.332274+0000","last_fullsized":"2024-03-30T15:05:13.332274+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:57.663204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:57.663204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323353+0000","last_change":"2024-03-30T15:04:38.084291+0000","last_active":"2024-03-30T15:05:13.323353+0000","last_peered":"2024-03-30T15:05:13.323353+0000","last_clean":"2024-03-30T15:05:13.323353+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:05:13.323353+0000","last_undegraded":"2024-03-30T15:05:13.323353+0000","last_fullsized":"2024-03-30T15:05:13.323353+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:38.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:40.487265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":775,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333281+0000","last_change":"2024-03-30T15:02:34.464153+0000","last_active":"2024-03-30T15:05:13.333281+0000","last_peered":"2024-03-30T15:05:13.333281+0000","last_clean":"2024-03-30T15:05:13.333281+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:05:13.333281+0000","last_undegraded":"2024-03-30T15:05:13.333281+0000","last_fullsized":"2024-03-30T15:05:13.333281+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:34.464067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.464067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327660+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:05:13.327660+0000","last_peered":"2024-03-30T15:05:13.327660+0000","last_clean":"2024-03-30T15:05:13.327660+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:05:13.327660+0000","last_undegraded":"2024-03-30T15:05:13.327660+0000","last_fullsized":"2024-03-30T15:05:13.327660+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":776,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335567+0000","last_change":"2024-03-30T15:01:12.319523+0000","last_active":"2024-03-30T15:05:13.335567+0000","last_peered":"2024-03-30T15:05:13.335567+0000","last_clean":"2024-03-30T15:05:13.335567+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:05:13.335567+0000","last_undegraded":"2024-03-30T15:05:13.335567+0000","last_fullsized":"2024-03-30T15:05:13.335567+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:12.319449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:12.319449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.291979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":788,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.320300+0000","last_change":"2024-03-30T15:02:38.936916+0000","last_active":"2024-03-30T15:05:13.320300+0000","last_peered":"2024-03-30T15:05:13.320300+0000","last_clean":"2024-03-30T15:05:13.320300+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:05:13.320300+0000","last_undegraded":"2024-03-30T15:05:13.320300+0000","last_fullsized":"2024-03-30T15:05:13.320300+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:38.936820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.936820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":776,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330551+0000","last_change":"2024-03-30T15:01:51.636327+0000","last_active":"2024-03-30T15:05:13.330551+0000","last_peered":"2024-03-30T15:05:13.330551+0000","last_clean":"2024-03-30T15:05:13.330551+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:05:13.330551+0000","last_undegraded":"2024-03-30T15:05:13.330551+0000","last_fullsized":"2024-03-30T15:05:13.330551+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:51.636231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:01:51.636231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324759+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:05:13.324759+0000","last_peered":"2024-03-30T15:05:13.324759+0000","last_clean":"2024-03-30T15:05:13.324759+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:05:13.324759+0000","last_undegraded":"2024-03-30T15:05:13.324759+0000","last_fullsized":"2024-03-30T15:05:13.324759+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":795,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319558+0000","last_change":"2024-03-30T15:04:37.008749+0000","last_active":"2024-03-30T15:05:13.319558+0000","last_peered":"2024-03-30T15:05:13.319558+0000","last_clean":"2024-03-30T15:05:13.319558+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:05:13.319558+0000","last_undegraded":"2024-03-30T15:05:13.319558+0000","last_fullsized":"2024-03-30T15:05:13.319558+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.008666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.008666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.100396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":795,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319120+0000","last_change":"2024-03-30T15:04:54.955662+0000","last_active":"2024-03-30T15:05:13.319120+0000","last_peered":"2024-03-30T15:05:13.319120+0000","last_clean":"2024-03-30T15:05:13.319120+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:05:13.319120+0000","last_undegraded":"2024-03-30T15:05:13.319120+0000","last_fullsized":"2024-03-30T15:05:13.319120+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.955541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.955541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.235351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334268+0000","last_change":"2024-03-30T15:04:55.271526+0000","last_active":"2024-03-30T15:05:13.334268+0000","last_peered":"2024-03-30T15:05:13.334268+0000","last_clean":"2024-03-30T15:05:13.334268+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:05:13.334268+0000","last_undegraded":"2024-03-30T15:05:13.334268+0000","last_fullsized":"2024-03-30T15:05:13.334268+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.271452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.271452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.970909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334342+0000","last_change":"2024-03-30T15:04:45.608747+0000","last_active":"2024-03-30T15:05:13.334342+0000","last_peered":"2024-03-30T15:05:13.334342+0000","last_clean":"2024-03-30T15:05:13.334342+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:05:13.334342+0000","last_undegraded":"2024-03-30T15:05:13.334342+0000","last_fullsized":"2024-03-30T15:05:13.334342+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.608652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.608652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:05.611823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324543+0000","last_change":"2024-03-30T15:04:36.136344+0000","last_active":"2024-03-30T15:05:13.324543+0000","last_peered":"2024-03-30T15:05:13.324543+0000","last_clean":"2024-03-30T15:05:13.324543+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:05:13.324543+0000","last_undegraded":"2024-03-30T15:05:13.324543+0000","last_fullsized":"2024-03-30T15:05:13.324543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.136228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.136228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:36.176305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334619+0000","last_change":"2024-03-30T15:04:50.298381+0000","last_active":"2024-03-30T15:05:13.334619+0000","last_peered":"2024-03-30T15:05:13.334619+0000","last_clean":"2024-03-30T15:05:13.334619+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:05:13.334619+0000","last_undegraded":"2024-03-30T15:05:13.334619+0000","last_fullsized":"2024-03-30T15:05:13.334619+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.298282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.756120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323576+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:05:13.323576+0000","last_peered":"2024-03-30T15:05:13.323576+0000","last_clean":"2024-03-30T15:05:13.323576+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:05:13.323576+0000","last_undegraded":"2024-03-30T15:05:13.323576+0000","last_fullsized":"2024-03-30T15:05:13.323576+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062368+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330089+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:05:13.330089+0000","last_peered":"2024-03-30T15:05:13.330089+0000","last_clean":"2024-03-30T15:05:13.330089+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:05:13.330089+0000","last_undegraded":"2024-03-30T15:05:13.330089+0000","last_fullsized":"2024-03-30T15:05:13.330089+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:20.116140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328680+0000","last_change":"2024-03-30T15:02:40.426121+0000","last_active":"2024-03-30T15:05:13.328680+0000","last_peered":"2024-03-30T15:05:13.328680+0000","last_clean":"2024-03-30T15:05:13.328680+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:05:13.328680+0000","last_undegraded":"2024-03-30T15:05:13.328680+0000","last_fullsized":"2024-03-30T15:05:13.328680+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.426031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.426031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":796,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318279+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:05:13.318279+0000","last_peered":"2024-03-30T15:05:13.318279+0000","last_clean":"2024-03-30T15:05:13.318279+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:05:13.318279+0000","last_undegraded":"2024-03-30T15:05:13.318279+0000","last_fullsized":"2024-03-30T15:05:13.318279+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.425095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330639+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:05:13.330639+0000","last_peered":"2024-03-30T15:05:13.330639+0000","last_clean":"2024-03-30T15:05:13.330639+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:05:13.330639+0000","last_undegraded":"2024-03-30T15:05:13.330639+0000","last_fullsized":"2024-03-30T15:05:13.330639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327758+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:05:13.327758+0000","last_peered":"2024-03-30T15:05:13.327758+0000","last_clean":"2024-03-30T15:05:13.327758+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:05:13.327758+0000","last_undegraded":"2024-03-30T15:05:13.327758+0000","last_fullsized":"2024-03-30T15:05:13.327758+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325048+0000","last_change":"2024-03-30T15:04:55.123255+0000","last_active":"2024-03-30T15:05:13.325048+0000","last_peered":"2024-03-30T15:05:13.325048+0000","last_clean":"2024-03-30T15:05:13.325048+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:05:13.325048+0000","last_undegraded":"2024-03-30T15:05:13.325048+0000","last_fullsized":"2024-03-30T15:05:13.325048+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.123176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.123176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.501538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324288+0000","last_change":"2024-03-30T15:04:45.130399+0000","last_active":"2024-03-30T15:05:13.324288+0000","last_peered":"2024-03-30T15:05:13.324288+0000","last_clean":"2024-03-30T15:05:13.324288+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:05:13.324288+0000","last_undegraded":"2024-03-30T15:05:13.324288+0000","last_fullsized":"2024-03-30T15:05:13.324288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.130322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.130322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.109084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":793,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330586+0000","last_change":"2024-03-30T15:04:44.264561+0000","last_active":"2024-03-30T15:05:13.330586+0000","last_peered":"2024-03-30T15:05:13.330586+0000","last_clean":"2024-03-30T15:05:13.330586+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:05:13.330586+0000","last_undegraded":"2024-03-30T15:05:13.330586+0000","last_fullsized":"2024-03-30T15:05:13.330586+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.264430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.264430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.388512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323459+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:05:13.323459+0000","last_peered":"2024-03-30T15:05:13.323459+0000","last_clean":"2024-03-30T15:05:13.323459+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:05:13.323459+0000","last_undegraded":"2024-03-30T15:05:13.323459+0000","last_fullsized":"2024-03-30T15:05:13.323459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.836391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332152+0000","last_change":"2024-03-30T15:04:54.465626+0000","last_active":"2024-03-30T15:05:13.332152+0000","last_peered":"2024-03-30T15:05:13.332152+0000","last_clean":"2024-03-30T15:05:13.332152+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:05:13.332152+0000","last_undegraded":"2024-03-30T15:05:13.332152+0000","last_fullsized":"2024-03-30T15:05:13.332152+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.465543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.465543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:57.501311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":784,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332122+0000","last_change":"2024-03-30T15:04:53.311876+0000","last_active":"2024-03-30T15:05:13.332122+0000","last_peered":"2024-03-30T15:05:13.332122+0000","last_clean":"2024-03-30T15:05:13.332122+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:05:13.332122+0000","last_undegraded":"2024-03-30T15:05:13.332122+0000","last_fullsized":"2024-03-30T15:05:13.332122+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.311797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.311797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.989454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329545+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:05:13.329545+0000","last_peered":"2024-03-30T15:05:13.329545+0000","last_clean":"2024-03-30T15:05:13.329545+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:05:13.329545+0000","last_undegraded":"2024-03-30T15:05:13.329545+0000","last_fullsized":"2024-03-30T15:05:13.329545+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":910,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334439+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:05:13.334439+0000","last_peered":"2024-03-30T15:05:13.334439+0000","last_clean":"2024-03-30T15:05:13.334439+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:05:13.334439+0000","last_undegraded":"2024-03-30T15:05:13.334439+0000","last_fullsized":"2024-03-30T15:05:13.334439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":790,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324819+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:05:13.324819+0000","last_peered":"2024-03-30T15:05:13.324819+0000","last_clean":"2024-03-30T15:05:13.324819+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:05:13.324819+0000","last_undegraded":"2024-03-30T15:05:13.324819+0000","last_fullsized":"2024-03-30T15:05:13.324819+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":34,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330547+0000","last_change":"2024-03-30T15:05:06.360304+0000","last_active":"2024-03-30T15:05:13.330547+0000","last_peered":"2024-03-30T15:05:13.330547+0000","last_clean":"2024-03-30T15:05:13.330547+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:05:13.330547+0000","last_undegraded":"2024-03-30T15:05:13.330547+0000","last_fullsized":"2024-03-30T15:05:13.330547+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.920515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":872,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.336048+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:05:13.336048+0000","last_peered":"2024-03-30T15:05:13.336048+0000","last_clean":"2024-03-30T15:05:13.336048+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:05:13.336048+0000","last_undegraded":"2024-03-30T15:05:13.336048+0000","last_fullsized":"2024-03-30T15:05:13.336048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.569612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330796+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:05:13.330796+0000","last_peered":"2024-03-30T15:05:13.330796+0000","last_clean":"2024-03-30T15:05:13.330796+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:05:13.330796+0000","last_undegraded":"2024-03-30T15:05:13.330796+0000","last_fullsized":"2024-03-30T15:05:13.330796+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:26.821252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":937,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333878+0000","last_change":"2024-03-30T15:04:34.512958+0000","last_active":"2024-03-30T15:05:13.333878+0000","last_peered":"2024-03-30T15:05:13.333878+0000","last_clean":"2024-03-30T15:05:13.333878+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:05:13.333878+0000","last_undegraded":"2024-03-30T15:05:13.333878+0000","last_fullsized":"2024-03-30T15:05:13.333878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:04:34.512838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.512838+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.123121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":900,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329788+0000","last_change":"2024-03-30T15:04:48.217127+0000","last_active":"2024-03-30T15:05:13.329788+0000","last_peered":"2024-03-30T15:05:13.329788+0000","last_clean":"2024-03-30T15:05:13.329788+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:05:13.329788+0000","last_undegraded":"2024-03-30T15:05:13.329788+0000","last_fullsized":"2024-03-30T15:05:13.329788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.217046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.217046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.053751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":893,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328508+0000","last_change":"2024-03-30T15:04:48.997109+0000","last_active":"2024-03-30T15:05:13.328508+0000","last_peered":"2024-03-30T15:05:13.328508+0000","last_clean":"2024-03-30T15:05:13.328508+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:05:13.328508+0000","last_undegraded":"2024-03-30T15:05:13.328508+0000","last_fullsized":"2024-03-30T15:05:13.328508+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.997021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.997021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.181602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":890,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.321139+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:05:13.321139+0000","last_peered":"2024-03-30T15:05:13.321139+0000","last_clean":"2024-03-30T15:05:13.321139+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:05:13.321139+0000","last_undegraded":"2024-03-30T15:05:13.321139+0000","last_fullsized":"2024-03-30T15:05:13.321139+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:33.116494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":791,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329853+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:05:13.329853+0000","last_peered":"2024-03-30T15:05:13.329853+0000","last_clean":"2024-03-30T15:05:13.329853+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:05:13.329853+0000","last_undegraded":"2024-03-30T15:05:13.329853+0000","last_fullsized":"2024-03-30T15:05:13.329853+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062283+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":916,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335847+0000","last_change":"2024-03-30T15:02:03.449650+0000","last_active":"2024-03-30T15:05:13.335847+0000","last_peered":"2024-03-30T15:05:13.335847+0000","last_clean":"2024-03-30T15:05:13.335847+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:05:13.335847+0000","last_undegraded":"2024-03-30T15:05:13.335847+0000","last_fullsized":"2024-03-30T15:05:13.335847+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:03.449549+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.449549+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":884,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328142+0000","last_change":"2024-03-30T15:01:50.503242+0000","last_active":"2024-03-30T15:05:13.328142+0000","last_peered":"2024-03-30T15:05:13.328142+0000","last_clean":"2024-03-30T15:05:13.328142+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:05:13.328142+0000","last_undegraded":"2024-03-30T15:05:13.328142+0000","last_fullsized":"2024-03-30T15:05:13.328142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:50.503156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:50.503156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":875,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.337696+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:05:13.337696+0000","last_peered":"2024-03-30T15:05:13.337696+0000","last_clean":"2024-03-30T15:05:13.337696+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:05:13.337696+0000","last_undegraded":"2024-03-30T15:05:13.337696+0000","last_fullsized":"2024-03-30T15:05:13.337696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":794,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329697+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:05:13.329697+0000","last_peered":"2024-03-30T15:05:13.329697+0000","last_clean":"2024-03-30T15:05:13.329697+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:05:13.329697+0000","last_undegraded":"2024-03-30T15:05:13.329697+0000","last_fullsized":"2024-03-30T15:05:13.329697+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":791,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329344+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:05:13.329344+0000","last_peered":"2024-03-30T15:05:13.329344+0000","last_clean":"2024-03-30T15:05:13.329344+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:05:13.329344+0000","last_undegraded":"2024-03-30T15:05:13.329344+0000","last_fullsized":"2024-03-30T15:05:13.329344+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":931,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329452+0000","last_change":"2024-03-30T15:01:46.424704+0000","last_active":"2024-03-30T15:05:13.329452+0000","last_peered":"2024-03-30T15:05:13.329452+0000","last_clean":"2024-03-30T15:05:13.329452+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:05:13.329452+0000","last_undegraded":"2024-03-30T15:05:13.329452+0000","last_fullsized":"2024-03-30T15:05:13.329452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'22","last_scrub_stamp":"2024-03-30T15:01:46.424546+0000","last_deep_scrub":"94'22","last_deep_scrub_stamp":"2024-03-30T14:56:15.296989+0000","last_clean_scrub_stamp":"2024-03-30T15:01:46.424546+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":795,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325537+0000","last_change":"2024-03-30T15:04:50.960049+0000","last_active":"2024-03-30T15:05:13.325537+0000","last_peered":"2024-03-30T15:05:13.325537+0000","last_clean":"2024-03-30T15:05:13.325537+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:05:13.325537+0000","last_undegraded":"2024-03-30T15:05:13.325537+0000","last_fullsized":"2024-03-30T15:05:13.325537+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.959968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.959968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.339138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1012,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335951+0000","last_change":"2024-03-30T15:03:22.567558+0000","last_active":"2024-03-30T15:05:13.335951+0000","last_peered":"2024-03-30T15:05:13.335951+0000","last_clean":"2024-03-30T15:05:13.335951+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:05:13.335951+0000","last_undegraded":"2024-03-30T15:05:13.335951+0000","last_fullsized":"2024-03-30T15:05:13.335951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:22.567465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.567465+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":871,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334569+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:05:13.334569+0000","last_peered":"2024-03-30T15:05:13.334569+0000","last_clean":"2024-03-30T15:05:13.334569+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:05:13.334569+0000","last_undegraded":"2024-03-30T15:05:13.334569+0000","last_fullsized":"2024-03-30T15:05:13.334569+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":788,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318762+0000","last_change":"2024-03-30T15:02:30.902281+0000","last_active":"2024-03-30T15:05:13.318762+0000","last_peered":"2024-03-30T15:05:13.318762+0000","last_clean":"2024-03-30T15:05:13.318762+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:05:13.318762+0000","last_undegraded":"2024-03-30T15:05:13.318762+0000","last_fullsized":"2024-03-30T15:05:13.318762+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.902199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.902199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":877,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327990+0000","last_change":"2024-03-30T15:02:39.473396+0000","last_active":"2024-03-30T15:05:13.327990+0000","last_peered":"2024-03-30T15:05:13.327990+0000","last_clean":"2024-03-30T15:05:13.327990+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:05:13.327990+0000","last_undegraded":"2024-03-30T15:05:13.327990+0000","last_fullsized":"2024-03-30T15:05:13.327990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:39.473279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.473279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":894,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325627+0000","last_change":"2024-03-30T15:02:34.958123+0000","last_active":"2024-03-30T15:05:13.325627+0000","last_peered":"2024-03-30T15:05:13.325627+0000","last_clean":"2024-03-30T15:05:13.325627+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:05:13.325627+0000","last_undegraded":"2024-03-30T15:05:13.325627+0000","last_fullsized":"2024-03-30T15:05:13.325627+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-30T15:02:34.958001+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:02:34.958001+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":860,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324635+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:05:13.324635+0000","last_peered":"2024-03-30T15:05:13.324635+0000","last_clean":"2024-03-30T15:05:13.324635+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:05:13.324635+0000","last_undegraded":"2024-03-30T15:05:13.324635+0000","last_fullsized":"2024-03-30T15:05:13.324635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":873,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327879+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:05:13.327879+0000","last_peered":"2024-03-30T15:05:13.327879+0000","last_clean":"2024-03-30T15:05:13.327879+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:05:13.327879+0000","last_undegraded":"2024-03-30T15:05:13.327879+0000","last_fullsized":"2024-03-30T15:05:13.327879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":803,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318201+0000","last_change":"2024-03-30T15:04:48.979522+0000","last_active":"2024-03-30T15:05:13.318201+0000","last_peered":"2024-03-30T15:05:13.318201+0000","last_clean":"2024-03-30T15:05:13.318201+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:05:13.318201+0000","last_undegraded":"2024-03-30T15:05:13.318201+0000","last_fullsized":"2024-03-30T15:05:13.318201+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.979438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.432525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":899,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325059+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:05:13.325059+0000","last_peered":"2024-03-30T15:05:13.325059+0000","last_clean":"2024-03-30T15:05:13.325059+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:05:13.325059+0000","last_undegraded":"2024-03-30T15:05:13.325059+0000","last_fullsized":"2024-03-30T15:05:13.325059+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":931,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328284+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:05:13.328284+0000","last_peered":"2024-03-30T15:05:13.328284+0000","last_clean":"2024-03-30T15:05:13.328284+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:05:13.328284+0000","last_undegraded":"2024-03-30T15:05:13.328284+0000","last_fullsized":"2024-03-30T15:05:13.328284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":890,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326580+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:05:13.326580+0000","last_peered":"2024-03-30T15:05:13.326580+0000","last_clean":"2024-03-30T15:05:13.326580+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:05:13.326580+0000","last_undegraded":"2024-03-30T15:05:13.326580+0000","last_fullsized":"2024-03-30T15:05:13.326580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":882,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323750+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:05:13.323750+0000","last_peered":"2024-03-30T15:05:13.323750+0000","last_clean":"2024-03-30T15:05:13.323750+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:05:13.323750+0000","last_undegraded":"2024-03-30T15:05:13.323750+0000","last_fullsized":"2024-03-30T15:05:13.323750+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":803,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318374+0000","last_change":"2024-03-30T15:04:47.018205+0000","last_active":"2024-03-30T15:05:13.318374+0000","last_peered":"2024-03-30T15:05:13.318374+0000","last_clean":"2024-03-30T15:05:13.318374+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:05:13.318374+0000","last_undegraded":"2024-03-30T15:05:13.318374+0000","last_fullsized":"2024-03-30T15:05:13.318374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.018083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.520606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1025,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324715+0000","last_change":"2024-03-30T15:01:48.471009+0000","last_active":"2024-03-30T15:05:13.324715+0000","last_peered":"2024-03-30T15:05:13.324715+0000","last_clean":"2024-03-30T15:05:13.324715+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:05:13.324715+0000","last_undegraded":"2024-03-30T15:05:13.324715+0000","last_fullsized":"2024-03-30T15:05:13.324715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:01:48.470884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:01:48.470884+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":925,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325937+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:05:13.325937+0000","last_peered":"2024-03-30T15:05:13.325937+0000","last_clean":"2024-03-30T15:05:13.325937+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:05:13.325937+0000","last_undegraded":"2024-03-30T15:05:13.325937+0000","last_fullsized":"2024-03-30T15:05:13.325937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":892,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334866+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:05:13.334866+0000","last_peered":"2024-03-30T15:05:13.334866+0000","last_clean":"2024-03-30T15:05:13.334866+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:05:13.334866+0000","last_undegraded":"2024-03-30T15:05:13.334866+0000","last_fullsized":"2024-03-30T15:05:13.334866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":877,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323134+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:05:13.323134+0000","last_peered":"2024-03-30T15:05:13.323134+0000","last_clean":"2024-03-30T15:05:13.323134+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:05:13.323134+0000","last_undegraded":"2024-03-30T15:05:13.323134+0000","last_fullsized":"2024-03-30T15:05:13.323134+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":796,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319880+0000","last_change":"2024-03-30T15:04:52.952256+0000","last_active":"2024-03-30T15:05:13.319880+0000","last_peered":"2024-03-30T15:05:13.319880+0000","last_clean":"2024-03-30T15:05:13.319880+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:05:13.319880+0000","last_undegraded":"2024-03-30T15:05:13.319880+0000","last_fullsized":"2024-03-30T15:05:13.319880+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.952177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.313787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":893,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330212+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:05:13.330212+0000","last_peered":"2024-03-30T15:05:13.330212+0000","last_clean":"2024-03-30T15:05:13.330212+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:05:13.330212+0000","last_undegraded":"2024-03-30T15:05:13.330212+0000","last_fullsized":"2024-03-30T15:05:13.330212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.244262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":944,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327069+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:05:13.327069+0000","last_peered":"2024-03-30T15:05:13.327069+0000","last_clean":"2024-03-30T15:05:13.327069+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:05:13.327069+0000","last_undegraded":"2024-03-30T15:05:13.327069+0000","last_fullsized":"2024-03-30T15:05:13.327069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.338727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":902,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328138+0000","last_change":"2024-03-30T15:04:51.927488+0000","last_active":"2024-03-30T15:05:13.328138+0000","last_peered":"2024-03-30T15:05:13.328138+0000","last_clean":"2024-03-30T15:05:13.328138+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:05:13.328138+0000","last_undegraded":"2024-03-30T15:05:13.328138+0000","last_fullsized":"2024-03-30T15:05:13.328138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.927412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.927412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.838858+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":882,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323675+0000","last_change":"2024-03-30T15:01:54.450014+0000","last_active":"2024-03-30T15:05:13.323675+0000","last_peered":"2024-03-30T15:05:13.323675+0000","last_clean":"2024-03-30T15:05:13.323675+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:05:13.323675+0000","last_undegraded":"2024-03-30T15:05:13.323675+0000","last_fullsized":"2024-03-30T15:05:13.323675+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:54.449931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:01:54.449931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062161+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326443+0000","last_change":"2024-03-30T15:01:56.570174+0000","last_active":"2024-03-30T15:05:13.326443+0000","last_peered":"2024-03-30T15:05:13.326443+0000","last_clean":"2024-03-30T15:05:13.326443+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:05:13.326443+0000","last_undegraded":"2024-03-30T15:05:13.326443+0000","last_fullsized":"2024-03-30T15:05:13.326443+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:56.570063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:56.570063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":883,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.337818+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:05:13.337818+0000","last_peered":"2024-03-30T15:05:13.337818+0000","last_clean":"2024-03-30T15:05:13.337818+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:05:13.337818+0000","last_undegraded":"2024-03-30T15:05:13.337818+0000","last_fullsized":"2024-03-30T15:05:13.337818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":928,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326340+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:05:13.326340+0000","last_peered":"2024-03-30T15:05:13.326340+0000","last_clean":"2024-03-30T15:05:13.326340+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:05:13.326340+0000","last_undegraded":"2024-03-30T15:05:13.326340+0000","last_fullsized":"2024-03-30T15:05:13.326340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.711973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1535,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325204+0000","last_change":"2024-03-30T15:04:50.115333+0000","last_active":"2024-03-30T15:05:13.325204+0000","last_peered":"2024-03-30T15:05:13.325204+0000","last_clean":"2024-03-30T15:05:13.325204+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:05:13.325204+0000","last_undegraded":"2024-03-30T15:05:13.325204+0000","last_fullsized":"2024-03-30T15:05:13.325204+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.115166+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.947111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":889,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.322907+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:05:13.322907+0000","last_peered":"2024-03-30T15:05:13.322907+0000","last_clean":"2024-03-30T15:05:13.322907+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:05:13.322907+0000","last_undegraded":"2024-03-30T15:05:13.322907+0000","last_fullsized":"2024-03-30T15:05:13.322907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":882,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327922+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:05:13.327922+0000","last_peered":"2024-03-30T15:05:13.327922+0000","last_clean":"2024-03-30T15:05:13.327922+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:05:13.327922+0000","last_undegraded":"2024-03-30T15:05:13.327922+0000","last_fullsized":"2024-03-30T15:05:13.327922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.928596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":790,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325519+0000","last_change":"2024-03-30T15:04:37.123232+0000","last_active":"2024-03-30T15:05:13.325519+0000","last_peered":"2024-03-30T15:05:13.325519+0000","last_clean":"2024-03-30T15:05:13.325519+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:05:13.325519+0000","last_undegraded":"2024-03-30T15:05:13.325519+0000","last_fullsized":"2024-03-30T15:05:13.325519+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.957564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":878,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324191+0000","last_change":"2024-03-30T15:02:33.948293+0000","last_active":"2024-03-30T15:05:13.324191+0000","last_peered":"2024-03-30T15:05:13.324191+0000","last_clean":"2024-03-30T15:05:13.324191+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:05:13.324191+0000","last_undegraded":"2024-03-30T15:05:13.324191+0000","last_fullsized":"2024-03-30T15:05:13.324191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:33.948156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.948156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1065,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331003+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:05:13.331003+0000","last_peered":"2024-03-30T15:05:13.331003+0000","last_clean":"2024-03-30T15:05:13.331003+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:05:13.331003+0000","last_undegraded":"2024-03-30T15:05:13.331003+0000","last_fullsized":"2024-03-30T15:05:13.331003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332224+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:05:13.332224+0000","last_peered":"2024-03-30T15:05:13.332224+0000","last_clean":"2024-03-30T15:05:13.332224+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:05:13.332224+0000","last_undegraded":"2024-03-30T15:05:13.332224+0000","last_fullsized":"2024-03-30T15:05:13.332224+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.548386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1147,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325312+0000","last_change":"2024-03-30T15:02:39.890565+0000","last_active":"2024-03-30T15:05:13.325312+0000","last_peered":"2024-03-30T15:05:13.325312+0000","last_clean":"2024-03-30T15:05:13.325312+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:05:13.325312+0000","last_undegraded":"2024-03-30T15:05:13.325312+0000","last_fullsized":"2024-03-30T15:05:13.325312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:02:39.890462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:02:39.890462+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":871,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335992+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:05:13.335992+0000","last_peered":"2024-03-30T15:05:13.335992+0000","last_clean":"2024-03-30T15:05:13.335992+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:05:13.335992+0000","last_undegraded":"2024-03-30T15:05:13.335992+0000","last_fullsized":"2024-03-30T15:05:13.335992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331337+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:05:13.331337+0000","last_peered":"2024-03-30T15:05:13.331337+0000","last_clean":"2024-03-30T15:05:13.331337+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:05:13.331337+0000","last_undegraded":"2024-03-30T15:05:13.331337+0000","last_fullsized":"2024-03-30T15:05:13.331337+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":894,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.320492+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:05:13.320492+0000","last_peered":"2024-03-30T15:05:13.320492+0000","last_clean":"2024-03-30T15:05:13.320492+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:05:13.320492+0000","last_undegraded":"2024-03-30T15:05:13.320492+0000","last_fullsized":"2024-03-30T15:05:13.320492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.873816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":935,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329335+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:05:13.329335+0000","last_peered":"2024-03-30T15:05:13.329335+0000","last_clean":"2024-03-30T15:05:13.329335+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:05:13.329335+0000","last_undegraded":"2024-03-30T15:05:13.329335+0000","last_fullsized":"2024-03-30T15:05:13.329335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":912,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318496+0000","last_change":"2024-03-30T15:04:55.918561+0000","last_active":"2024-03-30T15:05:13.318496+0000","last_peered":"2024-03-30T15:05:13.318496+0000","last_clean":"2024-03-30T15:05:13.318496+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:05:13.318496+0000","last_undegraded":"2024-03-30T15:05:13.318496+0000","last_fullsized":"2024-03-30T15:05:13.318496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.918411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.374429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":902,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325218+0000","last_change":"2024-03-30T15:04:53.983173+0000","last_active":"2024-03-30T15:05:13.325218+0000","last_peered":"2024-03-30T15:05:13.325218+0000","last_clean":"2024-03-30T15:05:13.325218+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:05:13.325218+0000","last_undegraded":"2024-03-30T15:05:13.325218+0000","last_fullsized":"2024-03-30T15:05:13.325218+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.983055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.611127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":882,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325744+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:05:13.325744+0000","last_peered":"2024-03-30T15:05:13.325744+0000","last_clean":"2024-03-30T15:05:13.325744+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:05:13.325744+0000","last_undegraded":"2024-03-30T15:05:13.325744+0000","last_fullsized":"2024-03-30T15:05:13.325744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325758+0000","last_change":"2024-03-30T15:01:59.537303+0000","last_active":"2024-03-30T15:05:13.325758+0000","last_peered":"2024-03-30T15:05:13.325758+0000","last_clean":"2024-03-30T15:05:13.325758+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:05:13.325758+0000","last_undegraded":"2024-03-30T15:05:13.325758+0000","last_fullsized":"2024-03-30T15:05:13.325758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:59.537225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:59.537225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":889,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325683+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:05:13.325683+0000","last_peered":"2024-03-30T15:05:13.325683+0000","last_clean":"2024-03-30T15:05:13.325683+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:05:13.325683+0000","last_undegraded":"2024-03-30T15:05:13.325683+0000","last_fullsized":"2024-03-30T15:05:13.325683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.100823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1028,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332344+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:05:13.332344+0000","last_peered":"2024-03-30T15:05:13.332344+0000","last_clean":"2024-03-30T15:05:13.332344+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:05:13.332344+0000","last_undegraded":"2024-03-30T15:05:13.332344+0000","last_fullsized":"2024-03-30T15:05:13.332344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":893,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332359+0000","last_change":"2024-03-30T15:04:53.505197+0000","last_active":"2024-03-30T15:05:13.332359+0000","last_peered":"2024-03-30T15:05:13.332359+0000","last_clean":"2024-03-30T15:05:13.332359+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:05:13.332359+0000","last_undegraded":"2024-03-30T15:05:13.332359+0000","last_fullsized":"2024-03-30T15:05:13.332359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.505107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.505107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.825117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":895,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332341+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:05:13.332341+0000","last_peered":"2024-03-30T15:05:13.332341+0000","last_clean":"2024-03-30T15:05:13.332341+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:05:13.332341+0000","last_undegraded":"2024-03-30T15:05:13.332341+0000","last_fullsized":"2024-03-30T15:05:13.332341+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335415+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:05:13.335415+0000","last_peered":"2024-03-30T15:05:13.335415+0000","last_clean":"2024-03-30T15:05:13.335415+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:05:13.335415+0000","last_undegraded":"2024-03-30T15:05:13.335415+0000","last_fullsized":"2024-03-30T15:05:13.335415+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325015+0000","last_change":"2024-03-30T15:01:58.563335+0000","last_active":"2024-03-30T15:05:13.325015+0000","last_peered":"2024-03-30T15:05:13.325015+0000","last_clean":"2024-03-30T15:05:13.325015+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:05:13.325015+0000","last_undegraded":"2024-03-30T15:05:13.325015+0000","last_fullsized":"2024-03-30T15:05:13.325015+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:58.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:58.563256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":917,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324917+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:05:13.324917+0000","last_peered":"2024-03-30T15:05:13.324917+0000","last_clean":"2024-03-30T15:05:13.324917+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:05:13.324917+0000","last_undegraded":"2024-03-30T15:05:13.324917+0000","last_fullsized":"2024-03-30T15:05:13.324917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:00.062419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":891,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332870+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:05:13.332870+0000","last_peered":"2024-03-30T15:05:13.332870+0000","last_clean":"2024-03-30T15:05:13.332870+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:05:13.332870+0000","last_undegraded":"2024-03-30T15:05:13.332870+0000","last_fullsized":"2024-03-30T15:05:13.332870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.634280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":894,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318537+0000","last_change":"2024-03-30T15:03:26.133453+0000","last_active":"2024-03-30T15:05:13.318537+0000","last_peered":"2024-03-30T15:05:13.318537+0000","last_clean":"2024-03-30T15:05:13.318537+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:05:13.318537+0000","last_undegraded":"2024-03-30T15:05:13.318537+0000","last_fullsized":"2024-03-30T15:05:13.318537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:26.133342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:26.133342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":876,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323247+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:05:13.323247+0000","last_peered":"2024-03-30T15:05:13.323247+0000","last_clean":"2024-03-30T15:05:13.323247+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:05:13.323247+0000","last_undegraded":"2024-03-30T15:05:13.323247+0000","last_fullsized":"2024-03-30T15:05:13.323247+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328295+0000","last_change":"2024-03-30T15:04:46.590243+0000","last_active":"2024-03-30T15:05:13.328295+0000","last_peered":"2024-03-30T15:05:13.328295+0000","last_clean":"2024-03-30T15:05:13.328295+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:05:13.328295+0000","last_undegraded":"2024-03-30T15:05:13.328295+0000","last_fullsized":"2024-03-30T15:05:13.328295+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:46.590171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.590171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.642800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":977,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330109+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:05:13.330109+0000","last_peered":"2024-03-30T15:05:13.330109+0000","last_clean":"2024-03-30T15:05:13.330109+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:05:13.330109+0000","last_undegraded":"2024-03-30T15:05:13.330109+0000","last_fullsized":"2024-03-30T15:05:13.330109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":885,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325610+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:05:13.325610+0000","last_peered":"2024-03-30T15:05:13.325610+0000","last_clean":"2024-03-30T15:05:13.325610+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:05:13.325610+0000","last_undegraded":"2024-03-30T15:05:13.325610+0000","last_fullsized":"2024-03-30T15:05:13.325610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":894,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319773+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:05:13.319773+0000","last_peered":"2024-03-30T15:05:13.319773+0000","last_clean":"2024-03-30T15:05:13.319773+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:05:13.319773+0000","last_undegraded":"2024-03-30T15:05:13.319773+0000","last_fullsized":"2024-03-30T15:05:13.319773+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.803620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":873,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324145+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:05:13.324145+0000","last_peered":"2024-03-30T15:05:13.324145+0000","last_clean":"2024-03-30T15:05:13.324145+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:05:13.324145+0000","last_undegraded":"2024-03-30T15:05:13.324145+0000","last_fullsized":"2024-03-30T15:05:13.324145+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":811,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330670+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:05:13.330670+0000","last_peered":"2024-03-30T15:05:13.330670+0000","last_clean":"2024-03-30T15:05:13.330670+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:05:13.330670+0000","last_undegraded":"2024-03-30T15:05:13.330670+0000","last_fullsized":"2024-03-30T15:05:13.330670+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":894,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318868+0000","last_change":"2024-03-30T15:04:49.959125+0000","last_active":"2024-03-30T15:05:13.318868+0000","last_peered":"2024-03-30T15:05:13.318868+0000","last_clean":"2024-03-30T15:05:13.318868+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:05:13.318868+0000","last_undegraded":"2024-03-30T15:05:13.318868+0000","last_fullsized":"2024-03-30T15:05:13.318868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.959020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.959020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.221503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":920,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328349+0000","last_change":"2024-03-30T15:03:19.519526+0000","last_active":"2024-03-30T15:05:13.328349+0000","last_peered":"2024-03-30T15:05:13.328349+0000","last_clean":"2024-03-30T15:05:13.328349+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:05:13.328349+0000","last_undegraded":"2024-03-30T15:05:13.328349+0000","last_fullsized":"2024-03-30T15:05:13.328349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:03:19.519440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.519440+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":893,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324549+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:05:13.324549+0000","last_peered":"2024-03-30T15:05:13.324549+0000","last_clean":"2024-03-30T15:05:13.324549+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:05:13.324549+0000","last_undegraded":"2024-03-30T15:05:13.324549+0000","last_fullsized":"2024-03-30T15:05:13.324549+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.756007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323899+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:05:13.323899+0000","last_peered":"2024-03-30T15:05:13.323899+0000","last_clean":"2024-03-30T15:05:13.323899+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:05:13.323899+0000","last_undegraded":"2024-03-30T15:05:13.323899+0000","last_fullsized":"2024-03-30T15:05:13.323899+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.943192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326887+0000","last_change":"2024-03-30T15:04:37.589562+0000","last_active":"2024-03-30T15:05:13.326887+0000","last_peered":"2024-03-30T15:05:13.326887+0000","last_clean":"2024-03-30T15:05:13.326887+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:05:13.326887+0000","last_undegraded":"2024-03-30T15:05:13.326887+0000","last_fullsized":"2024-03-30T15:05:13.326887+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:50.917173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1013,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324377+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:05:13.324377+0000","last_peered":"2024-03-30T15:05:13.324377+0000","last_clean":"2024-03-30T15:05:13.324377+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:05:13.324377+0000","last_undegraded":"2024-03-30T15:05:13.324377+0000","last_fullsized":"2024-03-30T15:05:13.324377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":922,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330188+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:05:13.330188+0000","last_peered":"2024-03-30T15:05:13.330188+0000","last_clean":"2024-03-30T15:05:13.330188+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:05:13.330188+0000","last_undegraded":"2024-03-30T15:05:13.330188+0000","last_fullsized":"2024-03-30T15:05:13.330188+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":905,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324775+0000","last_change":"2024-03-30T15:04:44.117459+0000","last_active":"2024-03-30T15:05:13.324775+0000","last_peered":"2024-03-30T15:05:13.324775+0000","last_clean":"2024-03-30T15:05:13.324775+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:05:13.324775+0000","last_undegraded":"2024-03-30T15:05:13.324775+0000","last_fullsized":"2024-03-30T15:05:13.324775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:04:44.117358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.117358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.327955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":871,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332616+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:05:13.332616+0000","last_peered":"2024-03-30T15:05:13.332616+0000","last_clean":"2024-03-30T15:05:13.332616+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:05:13.332616+0000","last_undegraded":"2024-03-30T15:05:13.332616+0000","last_fullsized":"2024-03-30T15:05:13.332616+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331231+0000","last_change":"2024-03-30T15:04:49.244698+0000","last_active":"2024-03-30T15:05:13.331231+0000","last_peered":"2024-03-30T15:05:13.331231+0000","last_clean":"2024-03-30T15:05:13.331231+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:05:13.331231+0000","last_undegraded":"2024-03-30T15:05:13.331231+0000","last_fullsized":"2024-03-30T15:05:13.331231+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.244616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.244616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:56.099737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":481,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.602967+0000","last_change":"2024-03-30T15:05:13.602996+0000","last_active":"2024-03-30T15:05:13.602967+0000","last_peered":"2024-03-30T15:05:13.602967+0000","last_clean":"2024-03-30T15:05:13.602967+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:05:13.602967+0000","last_undegraded":"2024-03-30T15:05:13.602967+0000","last_fullsized":"2024-03-30T15:05:13.602967+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:05:13.602902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.602902+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.234443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":900,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330062+0000","last_change":"2024-03-30T15:02:45.427403+0000","last_active":"2024-03-30T15:05:13.330062+0000","last_peered":"2024-03-30T15:05:13.330062+0000","last_clean":"2024-03-30T15:05:13.330062+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:05:13.330062+0000","last_undegraded":"2024-03-30T15:05:13.330062+0000","last_fullsized":"2024-03-30T15:05:13.330062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:02:45.427234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":875,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334714+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:05:13.334714+0000","last_peered":"2024-03-30T15:05:13.334714+0000","last_clean":"2024-03-30T15:05:13.334714+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:05:13.334714+0000","last_undegraded":"2024-03-30T15:05:13.334714+0000","last_fullsized":"2024-03-30T15:05:13.334714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":873,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324912+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:05:13.324912+0000","last_peered":"2024-03-30T15:05:13.324912+0000","last_clean":"2024-03-30T15:05:13.324912+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:05:13.324912+0000","last_undegraded":"2024-03-30T15:05:13.324912+0000","last_fullsized":"2024-03-30T15:05:13.324912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335263+0000","last_change":"2024-03-30T15:04:44.618218+0000","last_active":"2024-03-30T15:05:13.335263+0000","last_peered":"2024-03-30T15:05:13.335263+0000","last_clean":"2024-03-30T15:05:13.335263+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:05:13.335263+0000","last_undegraded":"2024-03-30T15:05:13.335263+0000","last_fullsized":"2024-03-30T15:05:13.335263+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.618135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.618135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.110883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":909,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326784+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:05:13.326784+0000","last_peered":"2024-03-30T15:05:13.326784+0000","last_clean":"2024-03-30T15:05:13.326784+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:05:13.326784+0000","last_undegraded":"2024-03-30T15:05:13.326784+0000","last_fullsized":"2024-03-30T15:05:13.326784+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":893,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323597+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:05:13.323597+0000","last_peered":"2024-03-30T15:05:13.323597+0000","last_clean":"2024-03-30T15:05:13.323597+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:05:13.323597+0000","last_undegraded":"2024-03-30T15:05:13.323597+0000","last_fullsized":"2024-03-30T15:05:13.323597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:30.940258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":794,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.329594+0000","last_change":"2024-03-30T15:04:03.988041+0000","last_active":"2024-03-30T15:05:13.329594+0000","last_peered":"2024-03-30T15:05:13.329594+0000","last_clean":"2024-03-30T15:05:13.329594+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:05:13.329594+0000","last_undegraded":"2024-03-30T15:05:13.329594+0000","last_fullsized":"2024-03-30T15:05:13.329594+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:18.333733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":890,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319991+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:05:13.319991+0000","last_peered":"2024-03-30T15:05:13.319991+0000","last_clean":"2024-03-30T15:05:13.319991+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:05:13.319991+0000","last_undegraded":"2024-03-30T15:05:13.319991+0000","last_fullsized":"2024-03-30T15:05:13.319991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.018619+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325383+0000","last_change":"2024-03-30T15:00:47.573554+0000","last_active":"2024-03-30T15:05:13.325383+0000","last_peered":"2024-03-30T15:05:13.325383+0000","last_clean":"2024-03-30T15:05:13.325383+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:05:13.325383+0000","last_undegraded":"2024-03-30T15:05:13.325383+0000","last_fullsized":"2024-03-30T15:05:13.325383+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:00:47.573474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:00:47.573474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":904,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335083+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:05:13.335083+0000","last_peered":"2024-03-30T15:05:13.335083+0000","last_clean":"2024-03-30T15:05:13.335083+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:05:13.335083+0000","last_undegraded":"2024-03-30T15:05:13.335083+0000","last_fullsized":"2024-03-30T15:05:13.335083+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":896,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324404+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:05:13.324404+0000","last_peered":"2024-03-30T15:05:13.324404+0000","last_clean":"2024-03-30T15:05:13.324404+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:05:13.324404+0000","last_undegraded":"2024-03-30T15:05:13.324404+0000","last_fullsized":"2024-03-30T15:05:13.324404+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.659468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":899,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323952+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:05:13.323952+0000","last_peered":"2024-03-30T15:05:13.323952+0000","last_clean":"2024-03-30T15:05:13.323952+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:05:13.323952+0000","last_undegraded":"2024-03-30T15:05:13.323952+0000","last_fullsized":"2024-03-30T15:05:13.323952+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":872,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.337945+0000","last_change":"2024-03-30T15:04:35.553690+0000","last_active":"2024-03-30T15:05:13.337945+0000","last_peered":"2024-03-30T15:05:13.337945+0000","last_clean":"2024-03-30T15:05:13.337945+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:05:13.337945+0000","last_undegraded":"2024-03-30T15:05:13.337945+0000","last_fullsized":"2024-03-30T15:05:13.337945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.553605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.553605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.879462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325444+0000","last_change":"2024-03-30T15:01:52.471709+0000","last_active":"2024-03-30T15:05:13.325444+0000","last_peered":"2024-03-30T15:05:13.325444+0000","last_clean":"2024-03-30T15:05:13.325444+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:05:13.325444+0000","last_undegraded":"2024-03-30T15:05:13.325444+0000","last_fullsized":"2024-03-30T15:05:13.325444+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:52.471603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:52.471603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":843,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328491+0000","last_change":"2024-03-30T15:04:38.199333+0000","last_active":"2024-03-30T15:05:13.328491+0000","last_peered":"2024-03-30T15:05:13.328491+0000","last_clean":"2024-03-30T15:05:13.328491+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:05:13.328491+0000","last_undegraded":"2024-03-30T15:05:13.328491+0000","last_fullsized":"2024-03-30T15:05:13.328491+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:04:38.199239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.199239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.783575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":885,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323255+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:05:13.323255+0000","last_peered":"2024-03-30T15:05:13.323255+0000","last_clean":"2024-03-30T15:05:13.323255+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:05:13.323255+0000","last_undegraded":"2024-03-30T15:05:13.323255+0000","last_fullsized":"2024-03-30T15:05:13.323255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1014,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334768+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:05:13.334768+0000","last_peered":"2024-03-30T15:05:13.334768+0000","last_clean":"2024-03-30T15:05:13.334768+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:05:13.334768+0000","last_undegraded":"2024-03-30T15:05:13.334768+0000","last_fullsized":"2024-03-30T15:05:13.334768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":878,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326219+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:05:13.326219+0000","last_peered":"2024-03-30T15:05:13.326219+0000","last_clean":"2024-03-30T15:05:13.326219+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:05:13.326219+0000","last_undegraded":"2024-03-30T15:05:13.326219+0000","last_fullsized":"2024-03-30T15:05:13.326219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.473225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":788,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332655+0000","last_change":"2024-03-30T15:04:41.620132+0000","last_active":"2024-03-30T15:05:13.332655+0000","last_peered":"2024-03-30T15:05:13.332655+0000","last_clean":"2024-03-30T15:05:13.332655+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:05:13.332655+0000","last_undegraded":"2024-03-30T15:05:13.332655+0000","last_fullsized":"2024-03-30T15:05:13.332655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:41.620027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:41.620027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.996471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":916,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332742+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:05:13.332742+0000","last_peered":"2024-03-30T15:05:13.332742+0000","last_clean":"2024-03-30T15:05:13.332742+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:05:13.332742+0000","last_undegraded":"2024-03-30T15:05:13.332742+0000","last_fullsized":"2024-03-30T15:05:13.332742+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.342982+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":892,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333950+0000","last_change":"2024-03-30T15:04:43.577081+0000","last_active":"2024-03-30T15:05:13.333950+0000","last_peered":"2024-03-30T15:05:13.333950+0000","last_clean":"2024-03-30T15:05:13.333950+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:05:13.333950+0000","last_undegraded":"2024-03-30T15:05:13.333950+0000","last_fullsized":"2024-03-30T15:05:13.333950+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.576993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.576993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.972808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":885,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325834+0000","last_change":"2024-03-30T15:02:03.477727+0000","last_active":"2024-03-30T15:05:13.325834+0000","last_peered":"2024-03-30T15:05:13.325834+0000","last_clean":"2024-03-30T15:05:13.325834+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:05:13.325834+0000","last_undegraded":"2024-03-30T15:05:13.325834+0000","last_fullsized":"2024-03-30T15:05:13.325834+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:03.477653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.477653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.322992+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:05:13.322992+0000","last_peered":"2024-03-30T15:05:13.322992+0000","last_clean":"2024-03-30T15:05:13.322992+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:05:13.322992+0000","last_undegraded":"2024-03-30T15:05:13.322992+0000","last_fullsized":"2024-03-30T15:05:13.322992+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.074178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":788,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327641+0000","last_change":"2024-03-30T15:04:40.601679+0000","last_active":"2024-03-30T15:05:13.327641+0000","last_peered":"2024-03-30T15:05:13.327641+0000","last_clean":"2024-03-30T15:05:13.327641+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:05:13.327641+0000","last_undegraded":"2024-03-30T15:05:13.327641+0000","last_fullsized":"2024-03-30T15:05:13.327641+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.601541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.601541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":914,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326045+0000","last_change":"2024-03-30T15:04:54.952409+0000","last_active":"2024-03-30T15:05:13.326045+0000","last_peered":"2024-03-30T15:05:13.326045+0000","last_clean":"2024-03-30T15:05:13.326045+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:05:13.326045+0000","last_undegraded":"2024-03-30T15:05:13.326045+0000","last_fullsized":"2024-03-30T15:05:13.326045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:54.952263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.952263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.078655+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":908,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325015+0000","last_change":"2024-03-30T15:04:52.183278+0000","last_active":"2024-03-30T15:05:13.325015+0000","last_peered":"2024-03-30T15:05:13.325015+0000","last_clean":"2024-03-30T15:05:13.325015+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:05:13.325015+0000","last_undegraded":"2024-03-30T15:05:13.325015+0000","last_fullsized":"2024-03-30T15:05:13.325015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:04:52.183194+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.183194+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.754133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331890+0000","last_change":"2024-03-30T15:04:37.298232+0000","last_active":"2024-03-30T15:05:13.331890+0000","last_peered":"2024-03-30T15:05:13.331890+0000","last_clean":"2024-03-30T15:05:13.331890+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:05:13.331890+0000","last_undegraded":"2024-03-30T15:05:13.331890+0000","last_fullsized":"2024-03-30T15:05:13.331890+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":876,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.322835+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:05:13.322835+0000","last_peered":"2024-03-30T15:05:13.322835+0000","last_clean":"2024-03-30T15:05:13.322835+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:05:13.322835+0000","last_undegraded":"2024-03-30T15:05:13.322835+0000","last_fullsized":"2024-03-30T15:05:13.322835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":794,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.322748+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:05:13.322748+0000","last_peered":"2024-03-30T15:05:13.322748+0000","last_clean":"2024-03-30T15:05:13.322748+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:05:13.322748+0000","last_undegraded":"2024-03-30T15:05:13.322748+0000","last_fullsized":"2024-03-30T15:05:13.322748+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.076034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":836,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324231+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:05:13.324231+0000","last_peered":"2024-03-30T15:05:13.324231+0000","last_clean":"2024-03-30T15:05:13.324231+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:05:13.324231+0000","last_undegraded":"2024-03-30T15:05:13.324231+0000","last_fullsized":"2024-03-30T15:05:13.324231+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":883,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334736+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:05:13.334736+0000","last_peered":"2024-03-30T15:05:13.334736+0000","last_clean":"2024-03-30T15:05:13.334736+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:05:13.334736+0000","last_undegraded":"2024-03-30T15:05:13.334736+0000","last_fullsized":"2024-03-30T15:05:13.334736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":877,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325537+0000","last_change":"2024-03-30T15:02:41.903716+0000","last_active":"2024-03-30T15:05:13.325537+0000","last_peered":"2024-03-30T15:05:13.325537+0000","last_clean":"2024-03-30T15:05:13.325537+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:05:13.325537+0000","last_undegraded":"2024-03-30T15:05:13.325537+0000","last_fullsized":"2024-03-30T15:05:13.325537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.903609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.903609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324543+0000","last_change":"2024-03-30T15:04:40.631484+0000","last_active":"2024-03-30T15:05:13.324543+0000","last_peered":"2024-03-30T15:05:13.324543+0000","last_clean":"2024-03-30T15:05:13.324543+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:05:13.324543+0000","last_undegraded":"2024-03-30T15:05:13.324543+0000","last_fullsized":"2024-03-30T15:05:13.324543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.631340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.631340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.585112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":791,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325377+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:05:13.325377+0000","last_peered":"2024-03-30T15:05:13.325377+0000","last_clean":"2024-03-30T15:05:13.325377+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:05:13.325377+0000","last_undegraded":"2024-03-30T15:05:13.325377+0000","last_fullsized":"2024-03-30T15:05:13.325377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":459,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.343504+0000","last_change":"2024-03-30T15:04:01.043987+0000","last_active":"2024-03-30T15:05:13.343504+0000","last_peered":"2024-03-30T15:05:13.343504+0000","last_clean":"2024-03-30T15:05:13.343504+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:05:13.343504+0000","last_undegraded":"2024-03-30T15:05:13.343504+0000","last_fullsized":"2024-03-30T15:05:13.343504+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:04:01.043885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.043885+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.133940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":896,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326712+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:05:13.326712+0000","last_peered":"2024-03-30T15:05:13.326712+0000","last_clean":"2024-03-30T15:05:13.326712+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:05:13.326712+0000","last_undegraded":"2024-03-30T15:05:13.326712+0000","last_fullsized":"2024-03-30T15:05:13.326712+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.517062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":892,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331688+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:05:13.331688+0000","last_peered":"2024-03-30T15:05:13.331688+0000","last_clean":"2024-03-30T15:05:13.331688+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:05:13.331688+0000","last_undegraded":"2024-03-30T15:05:13.331688+0000","last_fullsized":"2024-03-30T15:05:13.331688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":873,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323265+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:05:13.323265+0000","last_peered":"2024-03-30T15:05:13.323265+0000","last_clean":"2024-03-30T15:05:13.323265+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:05:13.323265+0000","last_undegraded":"2024-03-30T15:05:13.323265+0000","last_fullsized":"2024-03-30T15:05:13.323265+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331447+0000","last_change":"2024-03-30T15:04:51.176640+0000","last_active":"2024-03-30T15:05:13.331447+0000","last_peered":"2024-03-30T15:05:13.331447+0000","last_clean":"2024-03-30T15:05:13.331447+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:05:13.331447+0000","last_undegraded":"2024-03-30T15:05:13.331447+0000","last_fullsized":"2024-03-30T15:05:13.331447+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.176549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.176549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.079483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":925,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331484+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:05:13.331484+0000","last_peered":"2024-03-30T15:05:13.331484+0000","last_clean":"2024-03-30T15:05:13.331484+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:05:13.331484+0000","last_undegraded":"2024-03-30T15:05:13.331484+0000","last_fullsized":"2024-03-30T15:05:13.331484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":790,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323337+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:05:13.323337+0000","last_peered":"2024-03-30T15:05:13.323337+0000","last_clean":"2024-03-30T15:05:13.323337+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:05:13.323337+0000","last_undegraded":"2024-03-30T15:05:13.323337+0000","last_fullsized":"2024-03-30T15:05:13.323337+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.782926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":881,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326886+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:05:13.326886+0000","last_peered":"2024-03-30T15:05:13.326886+0000","last_clean":"2024-03-30T15:05:13.326886+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:05:13.326886+0000","last_undegraded":"2024-03-30T15:05:13.326886+0000","last_fullsized":"2024-03-30T15:05:13.326886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":881,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328256+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:05:13.328256+0000","last_peered":"2024-03-30T15:05:13.328256+0000","last_clean":"2024-03-30T15:05:13.328256+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:05:13.328256+0000","last_undegraded":"2024-03-30T15:05:13.328256+0000","last_fullsized":"2024-03-30T15:05:13.328256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331044+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:05:13.331044+0000","last_peered":"2024-03-30T15:05:13.331044+0000","last_clean":"2024-03-30T15:05:13.331044+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:05:13.331044+0000","last_undegraded":"2024-03-30T15:05:13.331044+0000","last_fullsized":"2024-03-30T15:05:13.331044+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":920,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326728+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:05:13.326728+0000","last_peered":"2024-03-30T15:05:13.326728+0000","last_clean":"2024-03-30T15:05:13.326728+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:05:13.326728+0000","last_undegraded":"2024-03-30T15:05:13.326728+0000","last_fullsized":"2024-03-30T15:05:13.326728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":886,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323775+0000","last_change":"2024-03-30T15:01:47.383230+0000","last_active":"2024-03-30T15:05:13.323775+0000","last_peered":"2024-03-30T15:05:13.323775+0000","last_clean":"2024-03-30T15:05:13.323775+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:05:13.323775+0000","last_undegraded":"2024-03-30T15:05:13.323775+0000","last_fullsized":"2024-03-30T15:05:13.323775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:01:47.383148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:01:47.383148+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1032,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331352+0000","last_change":"2024-03-30T15:02:51.451850+0000","last_active":"2024-03-30T15:05:13.331352+0000","last_peered":"2024-03-30T15:05:13.331352+0000","last_clean":"2024-03-30T15:05:13.331352+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:05:13.331352+0000","last_undegraded":"2024-03-30T15:05:13.331352+0000","last_fullsized":"2024-03-30T15:05:13.331352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:51.451690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:51.451690+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":883,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324064+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:05:13.324064+0000","last_peered":"2024-03-30T15:05:13.324064+0000","last_clean":"2024-03-30T15:05:13.324064+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:05:13.324064+0000","last_undegraded":"2024-03-30T15:05:13.324064+0000","last_fullsized":"2024-03-30T15:05:13.324064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":798,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330717+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:05:13.330717+0000","last_peered":"2024-03-30T15:05:13.330717+0000","last_clean":"2024-03-30T15:05:13.330717+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:05:13.330717+0000","last_undegraded":"2024-03-30T15:05:13.330717+0000","last_fullsized":"2024-03-30T15:05:13.330717+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:17.458537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":844,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327838+0000","last_change":"2024-03-30T15:04:40.214721+0000","last_active":"2024-03-30T15:05:13.327838+0000","last_peered":"2024-03-30T15:05:13.327838+0000","last_clean":"2024-03-30T15:05:13.327838+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:05:13.327838+0000","last_undegraded":"2024-03-30T15:05:13.327838+0000","last_fullsized":"2024-03-30T15:05:13.327838+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:40.214633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.214633+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:09.122719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":884,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328087+0000","last_change":"2024-03-30T15:03:21.522112+0000","last_active":"2024-03-30T15:05:13.328087+0000","last_peered":"2024-03-30T15:05:13.328087+0000","last_clean":"2024-03-30T15:05:13.328087+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:05:13.328087+0000","last_undegraded":"2024-03-30T15:05:13.328087+0000","last_fullsized":"2024-03-30T15:05:13.328087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.522038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.522038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":876,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334540+0000","last_change":"2024-03-30T15:03:23.551230+0000","last_active":"2024-03-30T15:05:13.334540+0000","last_peered":"2024-03-30T15:05:13.334540+0000","last_clean":"2024-03-30T15:05:13.334540+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:05:13.334540+0000","last_undegraded":"2024-03-30T15:05:13.334540+0000","last_fullsized":"2024-03-30T15:05:13.334540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:23.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":794,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.331576+0000","last_change":"2024-03-30T15:02:49.425842+0000","last_active":"2024-03-30T15:05:13.331576+0000","last_peered":"2024-03-30T15:05:13.331576+0000","last_clean":"2024-03-30T15:05:13.331576+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:05:13.331576+0000","last_undegraded":"2024-03-30T15:05:13.331576+0000","last_fullsized":"2024-03-30T15:05:13.331576+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:49.425762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.425762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.333023+0000","last_change":"2024-03-30T15:04:36.274480+0000","last_active":"2024-03-30T15:05:13.333023+0000","last_peered":"2024-03-30T15:05:13.333023+0000","last_clean":"2024-03-30T15:05:13.333023+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:05:13.333023+0000","last_undegraded":"2024-03-30T15:05:13.333023+0000","last_fullsized":"2024-03-30T15:05:13.333023+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.274391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.274391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:57.565878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":948,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.319472+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:05:13.319472+0000","last_peered":"2024-03-30T15:05:13.319472+0000","last_clean":"2024-03-30T15:05:13.319472+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:05:13.319472+0000","last_undegraded":"2024-03-30T15:05:13.319472+0000","last_fullsized":"2024-03-30T15:05:13.319472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":896,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326241+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:05:13.326241+0000","last_peered":"2024-03-30T15:05:13.326241+0000","last_clean":"2024-03-30T15:05:13.326241+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:05:13.326241+0000","last_undegraded":"2024-03-30T15:05:13.326241+0000","last_fullsized":"2024-03-30T15:05:13.326241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.088353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":885,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328123+0000","last_change":"2024-03-30T15:04:07.011782+0000","last_active":"2024-03-30T15:05:13.328123+0000","last_peered":"2024-03-30T15:05:13.328123+0000","last_clean":"2024-03-30T15:05:13.328123+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:05:13.328123+0000","last_undegraded":"2024-03-30T15:05:13.328123+0000","last_fullsized":"2024-03-30T15:05:13.328123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.011661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.011661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":873,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.335539+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:05:13.335539+0000","last_peered":"2024-03-30T15:05:13.335539+0000","last_clean":"2024-03-30T15:05:13.335539+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:05:13.335539+0000","last_undegraded":"2024-03-30T15:05:13.335539+0000","last_fullsized":"2024-03-30T15:05:13.335539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":791,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.324743+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:05:13.324743+0000","last_peered":"2024-03-30T15:05:13.324743+0000","last_clean":"2024-03-30T15:05:13.324743+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:05:13.324743+0000","last_undegraded":"2024-03-30T15:05:13.324743+0000","last_fullsized":"2024-03-30T15:05:13.324743+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062267+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":917,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332504+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:05:13.332504+0000","last_peered":"2024-03-30T15:05:13.332504+0000","last_clean":"2024-03-30T15:05:13.332504+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:05:13.332504+0000","last_undegraded":"2024-03-30T15:05:13.332504+0000","last_fullsized":"2024-03-30T15:05:13.332504+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:19.236100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":883,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332775+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:05:13.332775+0000","last_peered":"2024-03-30T15:05:13.332775+0000","last_clean":"2024-03-30T15:05:13.332775+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:05:13.332775+0000","last_undegraded":"2024-03-30T15:05:13.332775+0000","last_fullsized":"2024-03-30T15:05:13.332775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":877,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323167+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:05:13.323167+0000","last_peered":"2024-03-30T15:05:13.323167+0000","last_clean":"2024-03-30T15:05:13.323167+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:05:13.323167+0000","last_undegraded":"2024-03-30T15:05:13.323167+0000","last_fullsized":"2024-03-30T15:05:13.323167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":879,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330882+0000","last_change":"2024-03-30T15:04:35.253973+0000","last_active":"2024-03-30T15:05:13.330882+0000","last_peered":"2024-03-30T15:05:13.330882+0000","last_clean":"2024-03-30T15:05:13.330882+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:05:13.330882+0000","last_undegraded":"2024-03-30T15:05:13.330882+0000","last_fullsized":"2024-03-30T15:05:13.330882+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.253856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.253856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.131487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":785,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.325444+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:05:13.325444+0000","last_peered":"2024-03-30T15:05:13.325444+0000","last_clean":"2024-03-30T15:05:13.325444+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:05:13.325444+0000","last_undegraded":"2024-03-30T15:05:13.325444+0000","last_fullsized":"2024-03-30T15:05:13.325444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":905,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.328035+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:05:13.328035+0000","last_peered":"2024-03-30T15:05:13.328035+0000","last_clean":"2024-03-30T15:05:13.328035+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:05:13.328035+0000","last_undegraded":"2024-03-30T15:05:13.328035+0000","last_fullsized":"2024-03-30T15:05:13.328035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.139552+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":883,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.334672+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:05:13.334672+0000","last_peered":"2024-03-30T15:05:13.334672+0000","last_clean":"2024-03-30T15:05:13.334672+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:05:13.334672+0000","last_undegraded":"2024-03-30T15:05:13.334672+0000","last_fullsized":"2024-03-30T15:05:13.334672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":875,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.332122+0000","last_change":"2024-03-30T15:02:44.998020+0000","last_active":"2024-03-30T15:05:13.332122+0000","last_peered":"2024-03-30T15:05:13.332122+0000","last_clean":"2024-03-30T15:05:13.332122+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:05:13.332122+0000","last_undegraded":"2024-03-30T15:05:13.332122+0000","last_fullsized":"2024-03-30T15:05:13.332122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.997898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:14.292179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":884,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323391+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:05:13.323391+0000","last_peered":"2024-03-30T15:05:13.323391+0000","last_clean":"2024-03-30T15:05:13.323391+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:05:13.323391+0000","last_undegraded":"2024-03-30T15:05:13.323391+0000","last_fullsized":"2024-03-30T15:05:13.323391+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.231315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":789,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330682+0000","last_change":"2024-03-30T15:04:52.290105+0000","last_active":"2024-03-30T15:05:13.330682+0000","last_peered":"2024-03-30T15:05:13.330682+0000","last_clean":"2024-03-30T15:05:13.330682+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:05:13.330682+0000","last_undegraded":"2024-03-30T15:05:13.330682+0000","last_fullsized":"2024-03-30T15:05:13.330682+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.289998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.289998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.214437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":898,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.318656+0000","last_change":"2024-03-30T15:02:33.875911+0000","last_active":"2024-03-30T15:05:13.318656+0000","last_peered":"2024-03-30T15:05:13.318656+0000","last_clean":"2024-03-30T15:05:13.318656+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:05:13.318656+0000","last_undegraded":"2024-03-30T15:05:13.318656+0000","last_fullsized":"2024-03-30T15:05:13.318656+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'22","last_scrub_stamp":"2024-03-30T15:02:33.875778+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:02:33.875778+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":795,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.330902+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:05:13.330902+0000","last_peered":"2024-03-30T15:05:13.330902+0000","last_clean":"2024-03-30T15:05:13.330902+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:05:13.330902+0000","last_undegraded":"2024-03-30T15:05:13.330902+0000","last_fullsized":"2024-03-30T15:05:13.330902+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":876,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326836+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:05:13.326836+0000","last_peered":"2024-03-30T15:05:13.326836+0000","last_clean":"2024-03-30T15:05:13.326836+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:05:13.326836+0000","last_undegraded":"2024-03-30T15:05:13.326836+0000","last_fullsized":"2024-03-30T15:05:13.326836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:16.327150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":447,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.347111+0000","last_change":"2024-03-30T15:04:39.209738+0000","last_active":"2024-03-30T15:05:13.347111+0000","last_peered":"2024-03-30T15:05:13.347111+0000","last_clean":"2024-03-30T15:05:13.347111+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:05:13.347111+0000","last_undegraded":"2024-03-30T15:05:13.347111+0000","last_fullsized":"2024-03-30T15:05:13.347111+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:39.209655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.209655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.749896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":786,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323778+0000","last_change":"2024-03-30T15:01:17.584059+0000","last_active":"2024-03-30T15:05:13.323778+0000","last_peered":"2024-03-30T15:05:13.323778+0000","last_clean":"2024-03-30T15:05:13.323778+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:05:13.323778+0000","last_undegraded":"2024-03-30T15:05:13.323778+0000","last_fullsized":"2024-03-30T15:05:13.323778+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:17.583984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:01:17.583984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":929,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326933+0000","last_change":"2024-03-30T15:04:39.592435+0000","last_active":"2024-03-30T15:05:13.326933+0000","last_peered":"2024-03-30T15:05:13.326933+0000","last_clean":"2024-03-30T15:05:13.326933+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:05:13.326933+0000","last_undegraded":"2024-03-30T15:05:13.326933+0000","last_fullsized":"2024-03-30T15:05:13.326933+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:39.592324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.592324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.690325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":897,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.326577+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:05:13.326577+0000","last_peered":"2024-03-30T15:05:13.326577+0000","last_clean":"2024-03-30T15:05:13.326577+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:05:13.326577+0000","last_undegraded":"2024-03-30T15:05:13.326577+0000","last_fullsized":"2024-03-30T15:05:13.326577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:13.406410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":877,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.323644+0000","last_change":"2024-03-30T15:01:53.452032+0000","last_active":"2024-03-30T15:05:13.323644+0000","last_peered":"2024-03-30T15:05:13.323644+0000","last_clean":"2024-03-30T15:05:13.323644+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:05:13.323644+0000","last_undegraded":"2024-03-30T15:05:13.323644+0000","last_fullsized":"2024-03-30T15:05:13.323644+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:01:53.451927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:28.281774+0000","last_clean_scrub_stamp":"2024-03-30T15:01:53.451927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:15.062215+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":872,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-03-30T15:05:13.327788+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:05:13.327788+0000","last_peered":"2024-03-30T15:05:13.327788+0000","last_clean":"2024-03-30T15:05:13.327788+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:05:13.327788+0000","last_undegraded":"2024-03-30T15:05:13.327788+0000","last_fullsized":"2024-03-30T15:05:13.327788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":304,"num_pg":32,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195937,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":115,"num_read_kb":11,"num_write":630,"num_write_kb":12716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26677248,"data_stored":24587787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1969,"ondisk_log_size":1969,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":628,"num_read_kb":623,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005188,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76116,"kb_used_data":12856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669036,"statfs":{"total":95995035648,"available":95917092864,"internally_reserved":0,"allocated":13164544,"data_stored":9420442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299529,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76040,"kb_used_data":12780,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669112,"statfs":{"total":95995035648,"available":95917170688,"internally_reserved":0,"allocated":13086720,"data_stored":9417975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88244,"kb_used_data":8332,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93656908,"statfs":{"total":95995035648,"available":95904673792,"internally_reserved":0,"allocated":8531968,"data_stored":4828829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70740,"kb_used_data":12856,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93674412,"statfs":{"total":95995035648,"available":95922597888,"internally_reserved":0,"allocated":13164544,"data_stored":9529339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83780,"kb_used_data":16360,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93661372,"statfs":{"total":95995035648,"available":95909244928,"internally_reserved":0,"allocated":16752640,"data_stored":13022361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117145,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75676,"kb_used_data":12372,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93669476,"statfs":{"total":95995035648,"available":95917543424,"internally_reserved":0,"allocated":12668928,"data_stored":8939643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953619,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75816,"kb_used_data":8396,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93669336,"statfs":{"total":95995035648,"available":95917400064,"internally_reserved":0,"allocated":8597504,"data_stored":4828791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659278,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71652,"kb_used_data":8392,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93673500,"statfs":{"total":95995035648,"available":95921664000,"internally_reserved":0,"allocated":8593408,"data_stored":4828492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":4194827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8470528,"data_stored":8194915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4505600,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4255744,"data_stored":4000620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:05:16.002 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-30T15:05:16.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:05:16.025 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:16.026 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:16.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:16.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:16.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:16.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:16.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:16.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- 172.21.15.28:0/2380596959 >> v1:172.21.15.28:6790/0 conn(0x7f67a810d960 legacy=0x7f67a810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- 172.21.15.28:0/2380596959 shutdown_connections 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- 172.21.15.28:0/2380596959 >> 172.21.15.28:0/2380596959 conn(0x7f67a80fc9e0 msgr2=0x7f67a80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- 172.21.15.28:0/2380596959 shutdown_connections 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- 172.21.15.28:0/2380596959 wait complete. 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 librados: starting msgr at 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 librados: starting objecter 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 Processor -- start 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 -- start start 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 librados: setting wanted keys 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.418+0000 7f67aee28700 1 librados: calling monclient init 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67aee28700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67a8106880 con 0x7f67a8109dc0 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67aee28700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67a81a8540 con 0x7f67a8101130 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67aee28700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67a81a9720 con 0x7f67a810d960 2024-03-30T15:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a7fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f67a8109dc0 0x7f67a8104db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51936/0 (socket says 172.21.15.28:51936) 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a7fff700 1 -- 172.21.15.28:0/1955431408 learned_addr learned my addr 172.21.15.28:0/1955431408 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67ad3c5700 1 --1- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6790/0 conn(0x7f67a810d960 0x7f67a81a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56344/0 (socket says 172.21.15.28:56344) 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter ms_handle_connect 0x7f67a8109dc0 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter ms_handle_connect 0x7f67a810d960 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter ms_handle_connect 0x7f67a8101130 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790556649 0 0) 0x7f67a81a8540 con 0x7f67a8101130 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6790003aa0 con 0x7f67a8101130 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 352386010 0 0) 0x7f67a81a9720 con 0x7f67a810d960 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67a81a8540 con 0x7f67a810d960 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995935500 0 0) 0x7f67a8106880 con 0x7f67a8109dc0 2024-03-30T15:05:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67a81a9720 con 0x7f67a8109dc0 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3187622736 0 0) 0x7f6790003aa0 con 0x7f67a8101130 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67a8106880 con 0x7f67a8101130 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6798001830 con 0x7f67a8101130 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3984426652 0 0) 0x7f67a81a8540 con 0x7f67a810d960 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6790003aa0 con 0x7f67a810d960 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f67a00048b0 con 0x7f67a810d960 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.419+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2800570462 0 0) 0x7f67a81a9720 con 0x7f67a8109dc0 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67a81a8540 con 0x7f67a8109dc0 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f679c0013d0 con 0x7f67a8109dc0 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2863188488 0 0) 0x7f67a8106880 con 0x7f67a8101130 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6790/0 conn(0x7f67a810d960 legacy=0x7f67a81a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6789/0 conn(0x7f67a8109dc0 legacy=0x7f67a8104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a81aa900 con 0x7f67a8101130 2024-03-30T15:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67a81a9950 con 0x7f67a8101130 2024-03-30T15:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67aee28700 10 client.50215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67a81a9e10 con 0x7f67a8101130 2024-03-30T15:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67aee28700 1 librados: init done 2024-03-30T15:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6798003b20 con 0x7f67a8101130 2024-03-30T15:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.420+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6798004a60 con 0x7f67a8101130 2024-03-30T15:05:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.421+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6794005100 con 0x7f67a8101130 2024-03-30T15:05:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6798003680 con 0x7f67a8101130 2024-03-30T15:05:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(496..496 src has 442..496) ==== 8589+0+0 (unknown 442855878 0 0) 0x7f6798088830 con 0x7f67a8101130 2024-03-30T15:05:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 osd_map(496..496 src has 442..496) 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 3 client.50215.objecter handle_osd_map got epochs [496,496] > 0 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 3 client.50215.objecter handle_osd_map decoding full epoch 496 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 20 client.50215.objecter dump_active .. 0 homeless 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v496) ==== 54+0+16 (unknown 2453855458 0 2420000192) 0x7f6798088c20 con 0x7f67a8101130 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6790074880 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 10 client.50215.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 1 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 1 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 0 1 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 4 2024-03-30T15:05:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 10 client.50215.objecter _send_command 1 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.422+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6794004b50 con 0x7f6794004730 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6794004730 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794004b50 con 0x7f6794004730 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 1 = system:0 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 4 2024-03-30T15:05:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.423+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 0 1 2024-03-30T15:05:16.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:05:16.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 12.10 scrub starts 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 12.10 scrub ok 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: pgmap v712: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 304.11 deep-scrub starts 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[41614]: 304.11 deep-scrub ok 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:05:16.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 12.10 scrub starts 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 12.10 scrub ok 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: pgmap v712: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 304.11 deep-scrub starts 2024-03-30T15:05:16.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:16 smithi028 ceph-mon[29876]: 304.11 deep-scrub ok 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 10 client.50215.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 2 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 2 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f67940053e0 osd=0 3 2024-03-30T15:05:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 0 2 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 4 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 10 client.50215.objecter _send_command 2 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.444+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794007b00 con 0x7f6794004730 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794007b00 con 0x7f6794004730 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 4 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 0 2 2024-03-30T15:05:16.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 10 client.50215.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 3 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 3 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 1 3 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 4 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 10 client.50215.objecter _send_command 3 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6794002130 con 0x7f67940042a0 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.447+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f67940042a0 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.448+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794002130 con 0x7f67940042a0 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.448+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.448+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 3 = system:0 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.448+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 4 2024-03-30T15:05:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.448+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 1 3 2024-03-30T15:05:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:16.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 10 client.50215.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 4 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 4 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794004130 osd=1 3 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 1 4 2024-03-30T15:05:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 4 2024-03-30T15:05:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 10 client.50215.objecter _send_command 4 2024-03-30T15:05:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.466+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794001600 con 0x7f67940042a0 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794001600 con 0x7f67940042a0 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 4 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 1 4 2024-03-30T15:05:16.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 10 client.50215.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 5 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 5 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 2 5 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 4 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 10 client.50215.objecter _send_command 5 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6794010f70 con 0x7f6794008990 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.468+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6794008990 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.469+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794010f70 con 0x7f6794008990 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.469+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.469+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 5 = system:0 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.469+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 4 2024-03-30T15:05:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.469+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 2 5 2024-03-30T15:05:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 10 client.50215.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 6 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 6 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794001a00 osd=2 3 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 2 6 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 4 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 10 client.50215.objecter _send_command 6 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.488+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794007fc0 con 0x7f6794008990 2024-03-30T15:05:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794007fc0 con 0x7f6794008990 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 4 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 2 6 2024-03-30T15:05:16.491 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 10 client.50215.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 7 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 7 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 3 7 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 4 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 10 client.50215.objecter _send_command 7 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6794011cc0 con 0x7f6794011150 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.490+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6794011150 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.491+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794011cc0 con 0x7f6794011150 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.491+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:16.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.492+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 7 = system:0 2024-03-30T15:05:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.492+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 4 2024-03-30T15:05:16.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.492+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 3 7 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 10 client.50215.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 8 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 8 2024-03-30T15:05:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794008250 osd=3 3 2024-03-30T15:05:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 3 8 2024-03-30T15:05:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 4 2024-03-30T15:05:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 10 client.50215.objecter _send_command 8 2024-03-30T15:05:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.519+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794012670 con 0x7f6794011150 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794012670 con 0x7f6794011150 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 4 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 3 8 2024-03-30T15:05:16.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 10 client.50215.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 9 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 9 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 4 9 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 4 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 10 client.50215.objecter _send_command 9 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f679401a730 con 0x7f6794016b30 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.521+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6794016b30 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.522+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f679401a730 con 0x7f6794016b30 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.522+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.522+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 9 = system:0 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.522+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 4 2024-03-30T15:05:16.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.522+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 4 9 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 10 client.50215.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 10 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 10 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f67940137c0 osd=4 3 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 4 10 2024-03-30T15:05:16.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 4 2024-03-30T15:05:16.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 10 client.50215.objecter _send_command 10 2024-03-30T15:05:16.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.539+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f679400c870 con 0x7f6794016b30 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f679400c870 con 0x7f6794016b30 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 4 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 4 10 2024-03-30T15:05:16.542 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 10 client.50215.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 11 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 11 2024-03-30T15:05:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 5 11 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 4 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 10 client.50215.objecter _send_command 11 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6794013590 con 0x7f679401cdb0 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.541+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f679401cdb0 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.542+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794013590 con 0x7f679401cdb0 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.542+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.542+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 11 = system:0 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.542+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 4 2024-03-30T15:05:16.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.542+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 5 11 2024-03-30T15:05:16.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 10 client.50215.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 12 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 12 2024-03-30T15:05:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f679400c870 osd=5 3 2024-03-30T15:05:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 5 12 2024-03-30T15:05:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 4 2024-03-30T15:05:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 10 client.50215.objecter _send_command 12 2024-03-30T15:05:16.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.559+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794013590 con 0x7f679401cdb0 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794013590 con 0x7f679401cdb0 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 4 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 5 12 2024-03-30T15:05:16.562 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 10 client.50215.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 13 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 13 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 6 13 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 4 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 10 client.50215.objecter _send_command 13 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.561+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6794023300 con 0x7f679401f7f0 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f679401f7f0 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6794023300 con 0x7f679401f7f0 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 13 = system:0 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 4 2024-03-30T15:05:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.562+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 6 13 2024-03-30T15:05:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.579+0000 7f67aee28700 10 client.50215.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.579+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 14 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 14 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f679401c9d0 osd=6 3 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 6 14 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 4 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 10 client.50215.objecter _send_command 14 2024-03-30T15:05:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.580+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6794023300 con 0x7f679401f7f0 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6794023300 con 0x7f679401f7f0 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 4 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 6 14 2024-03-30T15:05:16.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 10 client.50215.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 15 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 15 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 7 15 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 4 2024-03-30T15:05:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 10 client.50215.objecter _send_command 15 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f67940256b0 con 0x7f6794023a20 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.582+0000 7f67a5ffb700 10 client.50215.objecter ms_handle_connect 0x7f6794023a20 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.583+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f67940256b0 con 0x7f6794023a20 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.583+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.583+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 15 = system:0 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.583+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 4 2024-03-30T15:05:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.583+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 7 15 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 10 client.50215.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign -1 16 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 20 client.50215.objecter _get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 15 client.50215.objecter _session_command_op_remove -1 16 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 20 client.50215.objecter get_session s=0x7f6794023300 osd=7 3 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 15 client.50215.objecter _session_command_op_assign 7 16 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 4 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 10 client.50215.objecter _send_command 16 2024-03-30T15:05:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.600+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f67940256b0 con 0x7f6794023a20 2024-03-30T15:05:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.601+0000 7f67a5ffb700 1 -- 172.21.15.28:0/1955431408 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f67940256b0 con 0x7f6794023a20 2024-03-30T15:05:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.601+0000 7f67a5ffb700 10 client.50215.objecter ms_dispatch 0x7f67a8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.601+0000 7f67a5ffb700 10 client.50215.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.601+0000 7f67a5ffb700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 4 2024-03-30T15:05:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.601+0000 7f67a5ffb700 15 client.50215.objecter _session_command_op_remove 7 16 2024-03-30T15:05:16.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 librados: watch_flush enter 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 librados: watch_flush exit 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.0 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6794004730 legacy=0x7f6794002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940053e0 osd=0 1 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.1 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6805/1850313547 conn(0x7f67940042a0 legacy=0x7f679400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794004130 osd=1 1 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.2 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6794008990 legacy=0x7f679400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794001a00 osd=2 1 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.3 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6794011150 legacy=0x7f67940115b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794008250 osd=3 1 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.4 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6794016b30 legacy=0x7f6794018f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f67940137c0 osd=4 1 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67ad3c5700 1 -- 172.21.15.28:0/1955431408 reap_dead start 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.5 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.134:6804/3668151179 conn(0x7f679401cdb0 legacy=0x7f6794011e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67ad3c5700 1 -- 172.21.15.28:0/1955431408 reap_dead start 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679400c870 osd=5 1 2024-03-30T15:05:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.6 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.134:6808/2135224194 conn(0x7f679401f7f0 legacy=0x7f6794021be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f679401c9d0 osd=6 1 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter close_session for osd.7 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6794023a20 legacy=0x7f6794023e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67ad3c5700 1 -- 172.21.15.28:0/1955431408 reap_dead start 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter put_session s=0x7f6794023300 osd=7 1 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 20 client.50215.objecter shutdown clearing up homeless session... 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 10 client.50215.objecter successfully canceled tick 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6790074880 legacy=0x7f6790076d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> v1:172.21.15.134:6789/0 conn(0x7f67a8101130 legacy=0x7f67a81046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 shutdown_connections 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 >> 172.21.15.28:0/1955431408 conn(0x7f67a80fc9e0 msgr2=0x7f67a8069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 shutdown_connections 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 -- 172.21.15.28:0/1955431408 wait complete. 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:16.602+0000 7f67aee28700 1 librados: shutdown 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: osdmap e496: 8 total, 8 up, 8 in 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.615 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.11"}]: dispatch 2024-03-30T15:05:16.616 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 12.10 scrub starts 2024-03-30T15:05:16.616 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 12.10 scrub ok 2024-03-30T15:05:16.616 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: pgmap v712: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-30T15:05:16.616 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 304.11 deep-scrub starts 2024-03-30T15:05:16.616 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:16 smithi134 ceph-mon[33714]: 304.11 deep-scrub ok 2024-03-30T15:05:16.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:05:16.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:16.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:16.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:16.876 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:16.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:16.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:16.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:17.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:17.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:17.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:17.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:17.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:17.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:17.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:17.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1955431408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:17.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T15:05:17.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T15:05:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1955431408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T15:05:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:17 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T15:05:17.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:17.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1955431408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:17 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T15:05:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:17 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T15:05:17.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:17.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:17.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:17.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:17.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:17.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:17.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:18.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.088 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:18.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:18.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:18.213 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:18.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:18.336 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:18.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:05:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:05:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T15:05:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T15:05:18.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 42.1f scrub starts 2024-03-30T15:05:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 42.1f scrub ok 2024-03-30T15:05:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 12.e scrub starts 2024-03-30T15:05:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: 12.e scrub ok 2024-03-30T15:05:18.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:18 smithi134 ceph-mon[33714]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 897 B/s wr, 2 op/s 2024-03-30T15:05:18.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:18.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:18.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:18.587 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:05:18.587 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:05:18.587 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:05:18.587 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:05:18.588 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:05:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:18.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:05:18.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:05:18.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T15:05:18.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 42.1f scrub starts 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 42.1f scrub ok 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 12.e scrub starts 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: 12.e scrub ok 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[29876]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 897 B/s wr, 2 op/s 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 42.1f scrub starts 2024-03-30T15:05:18.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 42.1f scrub ok 2024-03-30T15:05:18.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 12.e scrub starts 2024-03-30T15:05:18.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: 12.e scrub ok 2024-03-30T15:05:18.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:18 smithi028 ceph-mon[41614]: pgmap v713: 260 pgs: 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 897 B/s wr, 2 op/s 2024-03-30T15:05:18.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:18.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:18.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:18.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:18.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:18.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:18.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:19.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:19.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.057+0000 7f15361d0700 1 -- 172.21.15.28:0/515943474 >> v1:172.21.15.28:6790/0 conn(0x7f153010ea80 legacy=0x7f1530110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.057+0000 7f15361d0700 1 -- 172.21.15.28:0/515943474 shutdown_connections 2024-03-30T15:05:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.057+0000 7f15361d0700 1 -- 172.21.15.28:0/515943474 >> 172.21.15.28:0/515943474 conn(0x7f15300fdb20 msgr2=0x7f15300fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.057+0000 7f15361d0700 1 -- 172.21.15.28:0/515943474 shutdown_connections 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.057+0000 7f15361d0700 1 -- 172.21.15.28:0/515943474 wait complete. 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 librados: starting msgr at 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 librados: starting objecter 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 Processor -- start 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 -- start start 2024-03-30T15:05:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 librados: setting wanted keys 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 librados: calling monclient init 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15301a70f0 con 0x7f153010aee0 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15301a8310 con 0x7f1530102250 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f15361d0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15301a9550 con 0x7f153010ea80 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f153010ea80 0x7f15301a57b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36242/0 (socket says 172.21.15.28:36242) 2024-03-30T15:05:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152effd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f153010aee0 0x7f15301a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59006/0 (socket says 172.21.15.28:59006) 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152ffff700 1 -- 172.21.15.28:0/1228607428 learned_addr learned my addr 172.21.15.28:0/1228607428 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152cff9700 10 client.?.objecter ms_handle_connect 0x7f153010ea80 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152cff9700 10 client.?.objecter ms_handle_connect 0x7f153010aee0 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.058+0000 7f152cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306195866 0 0) 0x7f15301a9550 con 0x7f153010ea80 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15180031f0 con 0x7f153010ea80 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818419834 0 0) 0x7f15301a70f0 con 0x7f153010aee0 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15301a9550 con 0x7f153010aee0 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 10 client.?.objecter ms_handle_connect 0x7f1530102250 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4090001093 0 0) 0x7f15180031f0 con 0x7f153010ea80 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15301a70f0 con 0x7f153010ea80 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1528004920 con 0x7f153010ea80 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1930509731 0 0) 0x7f15301a9550 con 0x7f153010aee0 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15180031f0 con 0x7f153010aee0 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439821532 0 0) 0x7f15301a8310 con 0x7f1530102250 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15301a9550 con 0x7f1530102250 2024-03-30T15:05:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1524001830 con 0x7f153010aee0 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2630785771 0 0) 0x7f15301a70f0 con 0x7f153010ea80 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 >> v1:172.21.15.134:6789/0 conn(0x7f1530102250 legacy=0x7f1530107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 >> v1:172.21.15.28:6789/0 conn(0x7f153010aee0 legacy=0x7f15301a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15301aa790 con 0x7f153010ea80 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15301a7320 con 0x7f153010ea80 2024-03-30T15:05:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f15361d0700 10 client.58211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f15301a77e0 con 0x7f153010ea80 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f15361d0700 1 librados: init done 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1528001420 con 0x7f153010ea80 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.059+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1528005090 con 0x7f153010ea80 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.060+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c005120 con 0x7f153010ea80 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1528005310 con 0x7f153010ea80 2024-03-30T15:05:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(497..497 src has 442..497) ==== 8589+0+0 (unknown 3403226000 0 0) 0x7f1528089d10 con 0x7f153010ea80 2024-03-30T15:05:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 10 client.58211.objecter ms_dispatch 0x7f1530001550 osd_map(497..497 src has 442..497) 2024-03-30T15:05:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 3 client.58211.objecter handle_osd_map got epochs [497,497] > 0 2024-03-30T15:05:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 3 client.58211.objecter handle_osd_map decoding full epoch 497 2024-03-30T15:05:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 20 client.58211.objecter dump_active .. 0 homeless 2024-03-30T15:05:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.061+0000 7f152cff9700 10 client.58211.objecter ms_handle_connect 0x7f1518070400 2024-03-30T15:05:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.066+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1528056d70 con 0x7f153010ea80 2024-03-30T15:05:19.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:19.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:19.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:05:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.280+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f151c005410 con 0x7f153010ea80 2024-03-30T15:05:19.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:19.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.344+0000 7f152cff9700 1 -- 172.21.15.28:0/1228607428 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v498) ==== 101+0+0 (unknown 4119118472 0 0) 0x7f152805a7c0 con 0x7f153010ea80 2024-03-30T15:05:19.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.346+0000 7f15361d0700 10 librados: watch_flush enter 2024-03-30T15:05:19.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 10 librados: watch_flush exit 2024-03-30T15:05:19.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 20 client.58211.objecter shutdown clearing up homeless session... 2024-03-30T15:05:19.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 10 client.58211.objecter successfully canceled tick 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1518070400 legacy=0x7f15180728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 >> v1:172.21.15.28:6790/0 conn(0x7f153010ea80 legacy=0x7f15301a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 shutdown_connections 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 >> 172.21.15.28:0/1228607428 conn(0x7f15300fdb20 msgr2=0x7f1530107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 shutdown_connections 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 -- 172.21.15.28:0/1228607428 wait complete. 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:19.347+0000 7f15361d0700 1 librados: shutdown 2024-03-30T15:05:19.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.350 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:05:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:19.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:19.543 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:19.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:19.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T15:05:19.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: 12.1c scrub starts 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: 12.1c scrub ok 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1228607428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:19 smithi134 ceph-mon[33714]: from='client.58211 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:05:19.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: 12.1c scrub starts 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: 12.1c scrub ok 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1228607428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[29876]: from='client.58211 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: osdmap e497: 8 total, 8 up, 8 in 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: 12.1c scrub starts 2024-03-30T15:05:19.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: 12.1c scrub ok 2024-03-30T15:05:19.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1228607428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:19 smithi028 ceph-mon[41614]: from='client.58211 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:19.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:19.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:19.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:19.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:20.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:20.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:20.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:20.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:20.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:20.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:05:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:05:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T15:05:20.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.3 scrub starts 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.3 scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.1d scrub starts 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.1d scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.19 scrub starts 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 42.19 scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: from='client.58211 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 12.3 scrub starts 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: 12.3 scrub ok 2024-03-30T15:05:20.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[29876]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 359 B/s wr, 2 op/s 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.3 scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.3 scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.1d scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.1d scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.19 scrub starts 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 42.19 scrub ok 2024-03-30T15:05:20.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:05:20.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: from='client.58211 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:20.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T15:05:20.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 12.3 scrub starts 2024-03-30T15:05:20.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: 12.3 scrub ok 2024-03-30T15:05:20.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:20 smithi028 ceph-mon[41614]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 359 B/s wr, 2 op/s 2024-03-30T15:05:20.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:20.653 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:20.653 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.3 scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.3 scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.1d scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.1d scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.19 scrub starts 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 42.19 scrub ok 2024-03-30T15:05:20.654 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:05:20.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: from='client.58211 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:20.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: osdmap e498: 8 total, 8 up, 8 in 2024-03-30T15:05:20.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 12.3 scrub starts 2024-03-30T15:05:20.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: 12.3 scrub ok 2024-03-30T15:05:20.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:20 smithi134 ceph-mon[33714]: pgmap v716: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 359 B/s wr, 2 op/s 2024-03-30T15:05:20.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:20.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:20.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:20.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:20.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:20.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:21.003 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:05:21.003 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:05:21.003 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T15:05:21.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:05:21.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:21.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.226 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24838.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 454.888721369, 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8658181240000005, 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.227 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24838", 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3970114250", 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334484+0000", 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334486+0000", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.784e-06 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334493+0000", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8900000000000001e-06 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334501+0000", 2024-03-30T15:05:21.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5739999999999996e-06 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.334512+0000", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0152e-05 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.342482+0000", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079708320000000006 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:46.342492+0000", 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3489999999999998e-06 2024-03-30T15:05:21.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111380+0000", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7688878199999998 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111508+0000", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012799400000000001 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.111645+0000", 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013785199999999999 2024-03-30T15:05:21.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.119441+0000", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0077957549999999997 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.184335+0000", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.064893748000000001 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200217+0000", 2024-03-30T15:05:21.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015881663000000001 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200263+0000", 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6193999999999999e-05 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200302+0000", 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9517000000000001e-05 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24877.0:2 18.16 18:6ceb5ccc:test-rados-api-smithi028-93253-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 446.12848096200003, 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.190799929, 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24877", 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2268236432", 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.235 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094725+0000", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094727+0000", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1579999999999999e-06 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094733+0000", 2024-03-30T15:05:21.236 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0410000000000003e-06 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094744+0000", 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1019000000000001e-05 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.094755+0000", 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0852e-05 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.237 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.095307+0000", 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055243399999999998 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.095317+0000", 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0013000000000001e-05 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.238 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.231293+0000", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1359758339999999 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.231407+0000", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011398 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285525+0000", 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.054117598000000003 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.239 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24964.0:2 30.1e 30:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 446.06785432800001, 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069668294000000006, 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24964", 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2019932805", 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.240 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.241 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155351+0000", 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155353+0000", 2024-03-30T15:05:21.242 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.669e-06 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.155363+0000", 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9850000000000001e-06 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158533+0000", 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031701979999999999 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.243 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158546+0000", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3341e-05 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158606+0000", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9194999999999998e-05 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.158724+0000", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011867 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:05:21.244 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160432+0000", 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017071880000000001 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.171186+0000", 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01075456 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.171405+0000", 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021920300000000001 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.245 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.224916+0000", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053511156999999997 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.224980+0000", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3842000000000003e-05 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.225020+0000", 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9286000000000003e-05 2024-03-30T15:05:21.246 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24853.0:2 14.d 14:b0550ca8:test-rados-api-smithi028-93116-2::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 446.062385339, 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092514707000000002, 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24853", 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:05:21.247 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160820+0000", 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.248 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160822+0000", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.857e-06 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160829+0000", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.781e-06 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160837+0000", 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9640000000000003e-06 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.249 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160847+0000", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0117e-05 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160901+0000", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4135999999999998e-05 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.160909+0000", 2024-03-30T15:05:21.250 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8890000000000002e-06 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175479+0000", 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014569598 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175602+0000", 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000122986 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:05:21.251 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.175791+0000", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000189159 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.185216+0000", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0094253619999999996 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.223454+0000", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038237762000000002 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253285+0000", 2024-03-30T15:05:21.252 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.029830768000000001 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253311+0000", 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6393e-05 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.253335+0000", 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3935e-05 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24958.0:3 29.f 29:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.96780888699999, 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.040191584, 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24958", 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/925060148", 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.254 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255397+0000", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255398+0000", 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.251e-06 2024-03-30T15:05:21.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255406+0000", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3119999999999994e-06 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255415+0000", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3489999999999992e-06 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255422+0000", 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1019999999999999e-06 2024-03-30T15:05:21.256 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255533+0000", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011099600000000001 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.255540+0000", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4030000000000001e-06 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267675+0000", 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0121351670000001 2024-03-30T15:05:21.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267969+0000", 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029406199999999999 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.268118+0000", 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014865700000000001 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.258 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.270968+0000", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028503249999999999 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274869+0000", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0039004489999999998 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295496+0000", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020626938000000001 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.259 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295547+0000", 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1697000000000003e-05 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.295588+0000", 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0876e-05 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:14 11.a 11:56696d74:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:05:21.260 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.95551232399998, 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0178625910000001, 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.261 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267693+0000", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267694+0000", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0100000000000004e-07 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267698+0000", 2024-03-30T15:05:21.262 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0899999999999998e-06 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267706+0000", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7330000000000003e-06 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267711+0000", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0710000000000002e-06 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267749+0000", 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7834999999999999e-05 2024-03-30T15:05:21.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.267754+0000", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.13e-06 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.226737+0000", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95898334299999999 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.226877+0000", 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014019100000000001 2024-03-30T15:05:21.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.227059+0000", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018114199999999999 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.254281+0000", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027221973 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.254892+0000", 2024-03-30T15:05:21.265 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00061162599999999999 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285470+0000", 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030577887000000002 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285516+0000", 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5532999999999999e-05 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.266 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.285556+0000", 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0235999999999999e-05 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24940.0:86 26.11 26:8f09172f:test-rados-api-smithi028-93750-14::testobj:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:05:21.267 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.89101662000002, 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94252751899999998, 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24940", 2024-03-30T15:05:21.268 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 86 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332189+0000", 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332191+0000", 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0600000000000002e-06 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332196+0000", 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.977e-06 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.270 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332204+0000", 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3489999999999992e-06 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332220+0000", 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5622999999999999e-05 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332276+0000", 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5619999999999999e-05 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.271 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.332282+0000", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1589999999999998e-06 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274618+0000", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94233667499999996 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274716+0000", 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8055999999999998e-05 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.272 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24856.0:7 15.1 15:8386110d:test-rados-api-smithi028-93086-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.05673132599998, 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.100957805, 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24856", 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/13297926", 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 7 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.273 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166474+0000", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166476+0000", 2024-03-30T15:05:21.274 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.967e-06 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166483+0000", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8109999999999997e-06 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166493+0000", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0135e-05 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166503+0000", 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0305999999999999e-05 2024-03-30T15:05:21.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166635+0000", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013131800000000001 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.166735+0000", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010037799999999999 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.267432+0000", 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.10069689 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:1323 11.a 11:561c3777:test-rados-api-smithi028-93257-15::foo:head [setxattr a637 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 443.88035229899998, 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080122802000000007, 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1323 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.277 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342853+0000", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342854+0000", 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1230000000000001e-06 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.278 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342860+0000", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8989999999999998e-06 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.344152+0000", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001291883 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.344661+0000", 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00050922000000000001 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.279 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.354837+0000", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01017559 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.354946+0000", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010925900000000001 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.355269+0000", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032268200000000003 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.280 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.360362+0000", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050931029999999999 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.405152+0000", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044789735999999997 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422873+0000", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017721447000000001 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422931+0000", 2024-03-30T15:05:21.281 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8053999999999997e-05 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.422976+0000", 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4805999999999998e-05 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:8 13.1 13:848d865e:test-rados-api-smithi028-93525-4::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 439.43388053299998, 2024-03-30T15:05:21.282 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092471127, 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 8 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.283 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789325+0000", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789327+0000", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7349999999999999e-06 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789333+0000", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5749999999999997e-06 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789343+0000", 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9259999999999995e-06 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789353+0000", 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4460000000000007e-06 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789406+0000", 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3009e-05 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789412+0000", 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9329999999999998e-06 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.827838+0000", 2024-03-30T15:05:21.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038426244999999998 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.827978+0000", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014053500000000001 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.828134+0000", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015600799999999999 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.832326+0000", 2024-03-30T15:05:21.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041916180000000003 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.867355+0000", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.035029326999999999 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881695+0000", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014340031 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881750+0000", 2024-03-30T15:05:21.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4085999999999997e-05 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.881796+0000", 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6653000000000002e-05 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:10 13.1 13:848d865e:test-rados-api-smithi028-93525-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 438.41923344700001, 2024-03-30T15:05:21.289 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14530184500000001, 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 10 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:05:21.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803972+0000", 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803975+0000", 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.52e-06 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.803985+0000", 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.08e-05 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804006+0000", 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0316e-05 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804019+0000", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2860000000000001e-05 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804138+0000", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118913 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.804151+0000", 2024-03-30T15:05:21.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3857e-05 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.843765+0000", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.039613576999999997 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.843864+0000", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9179999999999996e-05 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.844054+0000", 2024-03-30T15:05:21.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019031300000000001 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.850640+0000", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065853350000000003 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.896075+0000", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.045435206999999998 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949167+0000", 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053091987 2024-03-30T15:05:21.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949224+0000", 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6543999999999999e-05 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:21.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.949274+0000", 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0436000000000003e-05 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.32242.0:2 120.6 120:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:05:21.298 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 419.17323769199999, 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.23603611299999999, 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.32242", 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/680296349", 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049968+0000", 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049970+0000", 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.154e-06 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.049978+0000", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3329999999999997e-06 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.050672+0000", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00069341499999999996 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.053009+0000", 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023367040000000002 2024-03-30T15:05:21.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.059224+0000", 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062157419999999998 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.059235+0000", 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0664e-05 2024-03-30T15:05:21.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.078870+0000", 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.019634743 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.303 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.078992+0000", 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012259900000000001 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.079132+0000", 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013983100000000001 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.081146+0000", 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020137979999999998 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.111454+0000", 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030308234999999999 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285896+0000", 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.17444207 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.285955+0000", 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8797e-05 2024-03-30T15:05:21.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:22.286004+0000", 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9027999999999999e-05 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34738.0:2 124.0 124:0384ab8e:::oid:head [writefull 0~3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 417.93112030999998, 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1976773839999999, 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.308 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34738", 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3778949751", 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.309 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292085+0000", 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292088+0000", 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3590000000000002e-06 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292095+0000", 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9450000000000004e-06 2024-03-30T15:05:21.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292106+0000", 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1911e-05 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292115+0000", 2024-03-30T15:05:21.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6910000000000003e-06 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292174+0000", 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8742000000000001e-05 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:23.292182+0000", 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7950000000000008e-06 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.480809+0000", 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1886268090000001 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.480965+0000", 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000156176 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481066+0000", 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101111 2024-03-30T15:05:21.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481431+0000", 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036508800000000001 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.481637+0000", 2024-03-30T15:05:21.316 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000206221 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489590+0000", 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079524929999999997 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489668+0000", 2024-03-30T15:05:21.317 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8608000000000004e-05 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.489763+0000", 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4435e-05 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.318 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.36958.0:2 131.0 131:0384ab8e:::oid:head [write 0~0,setxattr key1 (5) in=4b,omap-clear] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 414.373764908, 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.144649171, 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.319 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.36958", 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/962681329", 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.320 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849441+0000", 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849442+0000", 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2240000000000001e-06 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849450+0000", 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6520000000000006e-06 2024-03-30T15:05:21.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849459+0000", 2024-03-30T15:05:21.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7499999999999998e-06 2024-03-30T15:05:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849465+0000", 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5010000000000004e-06 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849520+0000", 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5136999999999997e-05 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:26.849529+0000", 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1339999999999993e-06 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967025+0000", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.117495906 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967148+0000", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012341300000000001 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967295+0000", 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146317 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.967911+0000", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00061612600000000004 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.968081+0000", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017055600000000001 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994027+0000", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025945373000000001 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994060+0000", 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3176999999999998e-05 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:27.994090+0000", 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9904999999999999e-05 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2079 11.a 11:5269a482:test-rados-api-smithi028-93257-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 412.32014442899998, 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13445506500000001, 2024-03-30T15:05:21.329 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2079 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903061+0000", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903063+0000", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.747e-06 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903070+0000", 2024-03-30T15:05:21.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6710000000000002e-06 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903080+0000", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0465999999999999e-05 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903087+0000", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5150000000000003e-06 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903142+0000", 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5466000000000001e-05 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.903275+0000", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000132578 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.972361+0000", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069086094000000001 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.983211+0000", 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010850741000000001 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:28.984309+0000", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010978019999999999 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037413+0000", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.053103996000000001 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037473+0000", 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9632000000000001e-05 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.037516+0000", 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3356999999999998e-05 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.42298.0:2 137.0 137:0384ab8e:::oid:head [append 0~64] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 409.91227983900001, 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.91052688900000001, 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.42298", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3149698539", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310926+0000", 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310928+0000", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8840000000000001e-06 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310934+0000", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2400000000000004e-06 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310962+0000", 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7712000000000002e-05 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.310971+0000", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3829999999999998e-06 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.338059+0000", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027088421000000001 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.338069+0000", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8430000000000005e-06 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213014+0000", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.87494432600000005 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213132+0000", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118268 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.213274+0000", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014212200000000001 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.214845+0000", 2024-03-30T15:05:21.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015707869999999999 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.216399+0000", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001554366 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221386+0000", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049865179999999997 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221423+0000", 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7002000000000002e-05 2024-03-30T15:05:21.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:32.221453+0000", 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0017000000000001e-05 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35600.0:110 127.e 127:7056b563:test-rados-api-smithi028-93253-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 409.84088813400001, 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.082170402000000003, 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.341 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35600", 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 110 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382317+0000", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382321+0000", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5540000000000002e-06 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382332+0000", 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0859999999999999e-05 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382359+0000", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6655000000000001e-05 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382380+0000", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1384000000000001e-05 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382527+0000", 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014746599999999999 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.382828+0000", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030051500000000001 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.384285+0000", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001456609 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.384354+0000", 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9276999999999994e-05 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464212+0000", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.079858083999999996 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464285+0000", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3312999999999994e-05 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.464488+0000", 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000202685 2024-03-30T15:05:21.346 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.53870.0:2 144.0 144:0384ab8e:::oid:head [write 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 406.88004902300003, 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.105832438, 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.53870", 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2684867265", 2024-03-30T15:05:21.347 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343157+0000", 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343158+0000", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.72e-06 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343165+0000", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.72e-06 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343174+0000", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1290000000000002e-06 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343184+0000", 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8700000000000004e-06 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343329+0000", 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000145489 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:34.343340+0000", 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0159e-05 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427351+0000", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.084011002 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427475+0000", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012393300000000001 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.427615+0000", 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014084500000000001 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.428264+0000", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064877700000000001 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.440421+0000", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012156372 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448863+0000", 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084419430000000004 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448933+0000", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0883999999999994e-05 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:35.448989+0000", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5594999999999999e-05 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49366.0:2 226.13 226:cbb9903e:::test_obj:head [writefull 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 329.02795460800002, 2024-03-30T15:05:21.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.161443155, 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49366", 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/796261573", 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195251+0000", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195254+0000", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8040000000000002e-06 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195266+0000", 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2408999999999999e-05 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195279+0000", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2907e-05 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195292+0000", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3298999999999999e-05 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195630+0000", 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033764299999999999 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.195637+0000", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6730000000000003e-06 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.202689+0000", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070526920000000002 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.202701+0000", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1103e-05 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354226+0000", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.151525983 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354342+0000", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115528 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354464+0000", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012247000000000001 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:05:21.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.354895+0000", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043055200000000001 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.355425+0000", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052977600000000003 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356598+0000", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011735140000000001 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356653+0000", 2024-03-30T15:05:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4456000000000002e-05 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:52.356694+0000", 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1346e-05 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49447.0:1 237.6s0 237:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 309.98885613499999, 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012543746, 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:05:21.360 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49447", 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1989601426", 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:05:21.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234349+0000", 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234352+0000", 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3429999999999998e-06 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234359+0000", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5120000000000002e-06 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234376+0000", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6331999999999999e-05 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.234387+0000", 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1476e-05 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.244898+0000", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010510455 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:11.244901+0000", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.382e-06 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.242735+0000", 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.997833796 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.242865+0000", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000130321 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.243228+0000", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036260699999999998 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.243478+0000", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025050199999999999 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:05:21.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.246817+0000", 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033390640000000001 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:12.246893+0000", 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5956000000000003e-05 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:05:21.366 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:05:21.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:21.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:21.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:21.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:21.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 42.13 scrub starts 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 42.13 scrub ok 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 12.c scrub starts 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 12.c scrub ok 2024-03-30T15:05:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:05:21.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:21 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 42.13 scrub starts 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 42.13 scrub ok 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 12.c scrub starts 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 12.c scrub ok 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:05:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 42.13 scrub starts 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 42.13 scrub ok 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 12.c scrub starts 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 12.c scrub ok 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:05:21.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:21 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:05:21.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:21.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:21.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:21.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:22.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:05:22.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:22.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.111+0000 7f8ecca32700 1 -- 172.21.15.28:0/243519347 >> v1:172.21.15.28:6790/0 conn(0x7f8ec8109dc0 legacy=0x7f8ec810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 -- 172.21.15.28:0/243519347 shutdown_connections 2024-03-30T15:05:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 -- 172.21.15.28:0/243519347 >> 172.21.15.28:0/243519347 conn(0x7f8ec80fc9e0 msgr2=0x7f8ec80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 -- 172.21.15.28:0/243519347 shutdown_connections 2024-03-30T15:05:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 -- 172.21.15.28:0/243519347 wait complete. 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 librados: starting msgr at 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 librados: starting objecter 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 Processor -- start 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 -- start start 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 librados: setting wanted keys 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.112+0000 7f8ecca32700 1 librados: calling monclient init 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8ecca32700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ec81a70f0 con 0x7f8ec8109dc0 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8ecca32700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ec81a8310 con 0x7f8ec810d960 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8ecca32700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ec81a9550 con 0x7f8ec8101130 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8ec659c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8ec8101130 0x7f8ec81064c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36264/0 (socket says 172.21.15.28:36264) 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8ec659c700 1 -- 172.21.15.28:0/1777665309 learned_addr learned my addr 172.21.15.28:0/1777665309 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter ms_handle_connect 0x7f8ec8101130 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter ms_handle_connect 0x7f8ec8109dc0 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter ms_handle_connect 0x7f8ec810d960 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3206968783 0 0) 0x7f8ec81a9550 con 0x7f8ec8101130 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb00039c0 con 0x7f8ec8101130 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901715680 0 0) 0x7f8ec81a70f0 con 0x7f8ec8109dc0 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec81a9550 con 0x7f8ec8109dc0 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520038929 0 0) 0x7f8ec81a8310 con 0x7f8ec810d960 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec81a70f0 con 0x7f8ec810d960 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 176246797 0 0) 0x7f8eb00039c0 con 0x7f8ec8101130 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ec81a8310 con 0x7f8ec8101130 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8eb8001930 con 0x7f8ec8101130 2024-03-30T15:05:22.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3991356868 0 0) 0x7f8ec81a9550 con 0x7f8ec8109dc0 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb00039c0 con 0x7f8ec8109dc0 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ebc001a00 con 0x7f8ec8109dc0 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 122858928 0 0) 0x7f8ec81a70f0 con 0x7f8ec810d960 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ec81a9550 con 0x7f8ec810d960 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ec00013f0 con 0x7f8ec810d960 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1567804844 0 0) 0x7f8ec81a8310 con 0x7f8ec8101130 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.134:6789/0 conn(0x7f8ec810d960 legacy=0x7f8ec81a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.113+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6789/0 conn(0x7f8ec8109dc0 legacy=0x7f8ec81a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec81aa790 con 0x7f8ec8101130 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec81a7320 con 0x7f8ec8101130 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8ecca32700 10 client.58217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ec81a77e0 con 0x7f8ec8101130 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8ecca32700 1 librados: init done 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8eb8003b40 con 0x7f8ec8101130 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.114+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8eb8005d30 con 0x7f8ec8101130 2024-03-30T15:05:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.115+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8ea8005130 con 0x7f8ec8101130 2024-03-30T15:05:22.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.115+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8eb801c650 con 0x7f8ec8101130 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(499..499 src has 442..499) ==== 8589+0+0 (unknown 1850587597 0 0) 0x7f8eb8089980 con 0x7f8ec8101130 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 osd_map(499..499 src has 442..499) 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 3 client.58217.objecter handle_osd_map got epochs [499,499] > 0 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 3 client.58217.objecter handle_osd_map decoding full epoch 499 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 20 client.58217.objecter dump_active .. 0 homeless 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v499) ==== 54+0+16 (unknown 2373191449 0 2420000192) 0x7f8eb8089d60 con 0x7f8ec8101130 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8eb0074970 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 10 client.58217.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 1 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 1 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 0 1 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 4 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 10 client.58217.objecter _send_command 1 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.116+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ea8004b80 con 0x7f8ea8004760 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea8004760 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea8004b80 con 0x7f8ea8004760 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 1 = system:0 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 4 2024-03-30T15:05:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.117+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 0 1 2024-03-30T15:05:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 10 client.58217.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 2 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 2 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8005410 osd=0 3 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 0 2 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 4 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 10 client.58217.objecter _send_command 2 2024-03-30T15:05:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.136+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea8007b30 con 0x7f8ea8004760 2024-03-30T15:05:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.137+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea8007b30 con 0x7f8ea8004760 2024-03-30T15:05:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.137+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.137+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.137+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 4 2024-03-30T15:05:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.137+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 0 2 2024-03-30T15:05:22.140 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 10 client.58217.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 1 3 2024-03-30T15:05:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 4 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 10 client.58217.objecter _send_command 3 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8ea8002190 con 0x7f8ea80042d0 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.138+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea80042d0 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.139+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8eb8035e50 con 0x7f8ea80042d0 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.139+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.139+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 3 = system:0 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.139+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 4 2024-03-30T15:05:22.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.139+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 1 3 2024-03-30T15:05:22.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 10 client.58217.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 4 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 4 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8004160 osd=1 3 2024-03-30T15:05:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 1 4 2024-03-30T15:05:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 4 2024-03-30T15:05:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 10 client.58217.objecter _send_command 4 2024-03-30T15:05:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.158+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea8001630 con 0x7f8ea80042d0 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea8001630 con 0x7f8ea80042d0 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 4 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 1 4 2024-03-30T15:05:22.161 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:22.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 10 client.58217.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 5 2024-03-30T15:05:22.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 5 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 2 5 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 4 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 10 client.58217.objecter _send_command 5 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8ea8010fe0 con 0x7f8ea80089c0 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.160+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea80089c0 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.161+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea8010fe0 con 0x7f8ea80089c0 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.161+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.161+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 5 = system:0 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.161+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 4 2024-03-30T15:05:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.161+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 2 5 2024-03-30T15:05:22.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 10 client.58217.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 6 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 6 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8001a30 osd=2 3 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 2 6 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 4 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 10 client.58217.objecter _send_command 6 2024-03-30T15:05:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.179+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea8008050 con 0x7f8ea80089c0 2024-03-30T15:05:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea8008050 con 0x7f8ea80089c0 2024-03-30T15:05:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 4 2024-03-30T15:05:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 2 6 2024-03-30T15:05:22.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 10 client.58217.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 7 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 7 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 3 7 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 4 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 10 client.58217.objecter _send_command 7 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.180+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8ea8011d10 con 0x7f8ea80111c0 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea80111c0 2024-03-30T15:05:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea8011d10 con 0x7f8ea80111c0 2024-03-30T15:05:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 7 = system:0 2024-03-30T15:05:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 4 2024-03-30T15:05:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.181+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 3 7 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 10 client.58217.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 8 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 8 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea80082e0 osd=3 3 2024-03-30T15:05:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 3 8 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 4 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 10 client.58217.objecter _send_command 8 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.199+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea80126c0 con 0x7f8ea80111c0 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea80126c0 con 0x7f8ea80111c0 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 4 2024-03-30T15:05:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 3 8 2024-03-30T15:05:22.202 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 10 client.58217.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 9 2024-03-30T15:05:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 9 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 4 9 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 4 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 10 client.58217.objecter _send_command 9 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.200+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8ea801a820 con 0x7f8ea8016ba0 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.201+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea8016ba0 2024-03-30T15:05:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.202+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea801a820 con 0x7f8ea8016ba0 2024-03-30T15:05:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.202+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.202+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 9 = system:0 2024-03-30T15:05:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.202+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 4 2024-03-30T15:05:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.202+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 4 9 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 10 client.58217.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 10 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 10 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8013840 osd=4 3 2024-03-30T15:05:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 4 10 2024-03-30T15:05:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 4 2024-03-30T15:05:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 10 client.58217.objecter _send_command 10 2024-03-30T15:05:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.219+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea801cbf0 con 0x7f8ea8016ba0 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea801cbf0 con 0x7f8ea8016ba0 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 4 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 4 10 2024-03-30T15:05:22.222 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 10 client.58217.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 11 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 11 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 5 11 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 4 2024-03-30T15:05:22.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 10 client.58217.objecter _send_command 11 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8ea801bf40 con 0x7f8ea801c9a0 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.221+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea801c9a0 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.222+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea801bf40 con 0x7f8ea801c9a0 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.222+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.222+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 11 = system:0 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.222+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 4 2024-03-30T15:05:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.222+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 5 11 2024-03-30T15:05:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 10 client.58217.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 12 2024-03-30T15:05:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 12 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea801d050 osd=5 3 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 5 12 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 4 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 10 client.58217.objecter _send_command 12 2024-03-30T15:05:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.239+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea801bf40 con 0x7f8ea801c9a0 2024-03-30T15:05:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea801bf40 con 0x7f8ea801c9a0 2024-03-30T15:05:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 4 2024-03-30T15:05:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 5 12 2024-03-30T15:05:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 10 client.58217.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 13 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 13 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 6 13 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 4 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 10 client.58217.objecter _send_command 13 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.241+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8ea8013670 con 0x7f8ea801f8e0 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea801f8e0 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea8013670 con 0x7f8ea801f8e0 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 13 = system:0 2024-03-30T15:05:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 4 2024-03-30T15:05:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.242+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 6 13 2024-03-30T15:05:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 10 client.58217.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 14 2024-03-30T15:05:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 14 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea801bf40 osd=6 3 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 6 14 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 4 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 10 client.58217.objecter _send_command 14 2024-03-30T15:05:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.259+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea8024bf0 con 0x7f8ea801f8e0 2024-03-30T15:05:22.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8ea8024bf0 con 0x7f8ea801f8e0 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 4 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 6 14 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 10 client.58217.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:22.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 15 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 15 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 7 15 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 4 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 10 client.58217.objecter _send_command 15 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.261+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8ea80257d0 con 0x7f8ea8025310 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.262+0000 7f8eb77fe700 10 client.58217.objecter ms_handle_connect 0x7f8ea8025310 2024-03-30T15:05:22.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.263+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ea80257d0 con 0x7f8ea8025310 2024-03-30T15:05:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.263+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.263+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 15 = system:0 2024-03-30T15:05:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.263+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 4 2024-03-30T15:05:22.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.263+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 7 15 2024-03-30T15:05:22.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:05:22.272 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:22.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 10 client.58217.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign -1 16 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 20 client.58217.objecter _get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_remove -1 16 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 20 client.58217.objecter get_session s=0x7f8ea8024bf0 osd=7 3 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 15 client.58217.objecter _session_command_op_assign 7 16 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 4 2024-03-30T15:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 10 client.58217.objecter _send_command 16 2024-03-30T15:05:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.281+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8ea8025790 con 0x7f8ea8025310 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.282+0000 7f8eb77fe700 1 -- 172.21.15.28:0/1777665309 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8eb801fe50 con 0x7f8ea8025310 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.282+0000 7f8eb77fe700 10 client.58217.objecter ms_dispatch 0x7f8ec8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.282+0000 7f8eb77fe700 10 client.58217.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.282+0000 7f8eb77fe700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 4 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.282+0000 7f8eb77fe700 15 client.58217.objecter _session_command_op_remove 7 16 2024-03-30T15:05:22.284 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 librados: watch_flush enter 2024-03-30T15:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 librados: watch_flush exit 2024-03-30T15:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.0 2024-03-30T15:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6801/3394323575 conn(0x7f8ea8004760 legacy=0x7f8ea8002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8005410 osd=0 1 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.1 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6805/1850313547 conn(0x7f8ea80042d0 legacy=0x7f8ea800ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8004160 osd=1 1 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.2 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6809/3451325043 conn(0x7f8ea80089c0 legacy=0x7f8ea800f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ec6d9d700 1 -- 172.21.15.28:0/1777665309 reap_dead start 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8001a30 osd=2 1 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.3 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6813/1506157250 conn(0x7f8ea80111c0 legacy=0x7f8ea8011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea80082e0 osd=3 1 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.4 2024-03-30T15:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.134:6800/3063775514 conn(0x7f8ea8016ba0 legacy=0x7f8ea8018f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8013840 osd=4 1 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.5 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.134:6804/3668151179 conn(0x7f8ea801c9a0 legacy=0x7f8ea801a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801d050 osd=5 1 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.6 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.134:6808/2135224194 conn(0x7f8ea801f8e0 legacy=0x7f8ea8021cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea801bf40 osd=6 1 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter close_session for osd.7 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.134:6812/3758190298 conn(0x7f8ea8025310 legacy=0x7f8ea8023420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter put_session s=0x7f8ea8024bf0 osd=7 1 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 20 client.58217.objecter shutdown clearing up homeless session... 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 10 client.58217.objecter successfully canceled tick 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8eb0074970 legacy=0x7f8eb0076e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> v1:172.21.15.28:6790/0 conn(0x7f8ec8101130 legacy=0x7f8ec81064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ec6d9d700 1 -- 172.21.15.28:0/1777665309 reap_dead start 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 shutdown_connections 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.283+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 >> 172.21.15.28:0/1777665309 conn(0x7f8ec80fc9e0 msgr2=0x7f8ec81105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.284+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 shutdown_connections 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.284+0000 7f8ecca32700 1 -- 172.21.15.28:0/1777665309 wait complete. 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:22.284+0000 7f8ecca32700 1 librados: shutdown 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:22.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:22.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:22.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:22.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:22.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:22.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 101.d deep-scrub starts 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 101.6 scrub starts 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 101.6 scrub ok 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 101.d deep-scrub ok 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1777665309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:22.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 101.d deep-scrub starts 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 101.6 scrub starts 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 101.6 scrub ok 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 101.d deep-scrub ok 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-03-30T15:05:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1777665309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 101.d deep-scrub starts 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 101.6 scrub starts 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 101.6 scrub ok 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 101.d deep-scrub ok 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: osdmap e499: 8 total, 8 up, 8 in 2024-03-30T15:05:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-03-30T15:05:22.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1777665309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:22.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:05:22.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:05:22.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:22.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:22.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:22.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:22.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:22.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:23.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:23.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:23.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:23.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:23.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:23.410 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:23 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T15:05:23.411 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:23 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T15:05:23.411 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:23 smithi134 ceph-mon[33714]: 101.a scrub starts 2024-03-30T15:05:23.411 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:23 smithi134 ceph-mon[33714]: 101.a scrub ok 2024-03-30T15:05:23.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:23.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:23.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:23.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:23.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T15:05:23.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[41614]: 101.a scrub starts 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[41614]: 101.a scrub ok 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[29876]: 101.a scrub starts 2024-03-30T15:05:23.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:23 smithi028 ceph-mon[29876]: 101.a scrub ok 2024-03-30T15:05:23.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:23.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:23.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:23.922 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:23.922 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:23.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:23.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:24.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:24.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:24.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:24.319 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:24.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:24.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:24.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:24 smithi134 ceph-mon[33714]: 6.1 deep-scrub starts 2024-03-30T15:05:24.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:24 smithi134 ceph-mon[33714]: 6.1 deep-scrub ok 2024-03-30T15:05:24.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:24 smithi134 ceph-mon[33714]: 101.18 scrub starts 2024-03-30T15:05:24.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:24 smithi134 ceph-mon[33714]: 101.18 scrub ok 2024-03-30T15:05:24.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:24 smithi134 ceph-mon[33714]: pgmap v719: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-03-30T15:05:24.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:24.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:05:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[41614]: 6.1 deep-scrub starts 2024-03-30T15:05:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[41614]: 6.1 deep-scrub ok 2024-03-30T15:05:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[41614]: 101.18 scrub starts 2024-03-30T15:05:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[41614]: 101.18 scrub ok 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[41614]: pgmap v719: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[29876]: 6.1 deep-scrub starts 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[29876]: 6.1 deep-scrub ok 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[29876]: 101.18 scrub starts 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[29876]: 101.18 scrub ok 2024-03-30T15:05:24.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:24 smithi028 ceph-mon[29876]: pgmap v719: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-03-30T15:05:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:24.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:24.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:24.956 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:24.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:24.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:24.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:25.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:25.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:25.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:25.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:25.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:25.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:05:25.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:05:25.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 101.10 scrub starts 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 101.10 scrub ok 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 101.13 scrub starts 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[41614]: 101.13 scrub ok 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 101.10 scrub starts 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 101.10 scrub ok 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 101.13 scrub starts 2024-03-30T15:05:25.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:25 smithi028 ceph-mon[29876]: 101.13 scrub ok 2024-03-30T15:05:25.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:25.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 -- 172.21.15.28:0/2132820809 >> v1:172.21.15.28:6789/0 conn(0x7f6068102250 legacy=0x7f6068102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 -- 172.21.15.28:0/2132820809 shutdown_connections 2024-03-30T15:05:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 -- 172.21.15.28:0/2132820809 >> 172.21.15.28:0/2132820809 conn(0x7f60680fdb20 msgr2=0x7f60680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 -- 172.21.15.28:0/2132820809 shutdown_connections 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 -- 172.21.15.28:0/2132820809 wait complete. 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 librados: starting msgr at 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.505+0000 7f606e6b6700 1 librados: starting objecter 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 Processor -- start 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 -- start start 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 librados: setting wanted keys 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 librados: calling monclient init 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f60681078a0 con 0x7f606810ea80 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6068107ab0 con 0x7f6068102250 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606e6b6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f60681a9660 con 0x7f606810aee0 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60677fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f606810aee0 0x7f6068104e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36272/0 (socket says 172.21.15.28:36272) 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60677fe700 1 -- 172.21.15.28:0/2699511250 learned_addr learned my addr 172.21.15.28:0/2699511250 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f606cc53700 1 --1- 172.21.15.28:0/2699511250 >> v1:172.21.15.28:6789/0 conn(0x7f606810ea80 0x7f60681a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59036/0 (socket says 172.21.15.28:59036) 2024-03-30T15:05:25.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter ms_handle_connect 0x7f606810ea80 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter ms_handle_connect 0x7f606810aee0 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter ms_handle_connect 0x7f6068102250 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007491700 0 0) 0x7f60681a9660 con 0x7f606810aee0 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60500039c0 con 0x7f606810aee0 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065581960 0 0) 0x7f60681078a0 con 0x7f606810ea80 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60681a9660 con 0x7f606810ea80 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450116978 0 0) 0x7f6068107ab0 con 0x7f6068102250 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.506+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60681078a0 con 0x7f6068102250 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 521735499 0 0) 0x7f60681a9660 con 0x7f606810ea80 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6068107ab0 con 0x7f606810ea80 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3296544069 0 0) 0x7f60500039c0 con 0x7f606810aee0 2024-03-30T15:05:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60681a9660 con 0x7f606810aee0 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6060001db0 con 0x7f606810ea80 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 648111692 0 0) 0x7f60681078a0 con 0x7f6068102250 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60500039c0 con 0x7f6068102250 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f605c001930 con 0x7f606810aee0 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f60580015b0 con 0x7f6068102250 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 456175932 0 0) 0x7f6068107ab0 con 0x7f606810ea80 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 >> v1:172.21.15.28:6790/0 conn(0x7f606810aee0 legacy=0x7f6068104e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 >> v1:172.21.15.134:6789/0 conn(0x7f6068102250 legacy=0x7f60681046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60681aa840 con 0x7f606810ea80 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60681a9890 con 0x7f606810ea80 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f606e6b6700 10 client.25023.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f60681a9d50 con 0x7f606810ea80 2024-03-30T15:05:25.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f606e6b6700 1 librados: init done 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6060002020 con 0x7f606810ea80 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.507+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6060005f70 con 0x7f606810ea80 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.508+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6054005120 con 0x7f606810ea80 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6060003cb0 con 0x7f606810ea80 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(499..499 src has 442..499) ==== 8589+0+0 (unknown 1850587597 0 0) 0x7f6060089930 con 0x7f606810ea80 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 10 client.25023.objecter ms_dispatch 0x7f6068001550 osd_map(499..499 src has 442..499) 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 3 client.25023.objecter handle_osd_map got epochs [499,499] > 0 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 3 client.25023.objecter handle_osd_map decoding full epoch 499 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 20 client.25023.objecter dump_active .. 0 homeless 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.509+0000 7f60657fa700 10 client.25023.objecter ms_handle_connect 0x7f6050074840 2024-03-30T15:05:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.513+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6060056990 con 0x7f606810ea80 2024-03-30T15:05:25.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:25.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:05:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:05:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 101.10 scrub starts 2024-03-30T15:05:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 101.10 scrub ok 2024-03-30T15:05:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 101.13 scrub starts 2024-03-30T15:05:25.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:25 smithi134 ceph-mon[33714]: 101.13 scrub ok 2024-03-30T15:05:25.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:05:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.723+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f6054005410 con 0x7f606810ea80 2024-03-30T15:05:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.726+0000 7f60657fa700 1 -- 172.21.15.28:0/2699511250 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v499) ==== 81+0+56289 (unknown 1188292947 0 4173724482) 0x7f6060040e20 con 0x7f606810ea80 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 499, 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:05:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:05:21.343600+0000", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 112, 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:05:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 304, 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:05:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "437", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:05:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:05:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "491", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "411", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "411", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "411", 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 26, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 491, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 304, 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-79", 2024-03-30T15:05:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:05:11.572750+0000", 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "496", 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 3, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 496, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 10 librados: watch_flush enter 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 10 librados: watch_flush exit 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 20 client.25023.objecter shutdown clearing up homeless session... 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 10 client.25023.objecter successfully canceled tick 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6050074840 legacy=0x7f6050076d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 >> v1:172.21.15.28:6789/0 conn(0x7f606810ea80 legacy=0x7f60681a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 shutdown_connections 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 >> 172.21.15.28:0/2699511250 conn(0x7f60680fdb20 msgr2=0x7f6068110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 shutdown_connections 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 -- 172.21.15.28:0/2699511250 wait complete. 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:25.729+0000 7f606e6b6700 1 librados: shutdown 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 417, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:05:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:05:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:05:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:05:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 493, 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:05:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:34.941959+0000", 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:31.463076+0000", 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:14.412793+0000", 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:21.119206+0000", 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:07.577028+0000", 2024-03-30T15:05:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:19.162669+0000", 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:05:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:18.056823+0000", 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:27.766874+0000", 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:05:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:05:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:05:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:05:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:05:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:05:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:05:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:05:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:05:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:05:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:05:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:05:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:25.857 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:05:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:25.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:25.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:25.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:25.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:26.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:26.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:26.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:26.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:26.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:26.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:26.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:26.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: 101.1b scrub starts 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: 101.1b scrub ok 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: 101.11 scrub starts 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: 101.11 scrub ok 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2699511250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[29876]: pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: 101.1b scrub starts 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: 101.1b scrub ok 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: 101.11 scrub starts 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: 101.11 scrub ok 2024-03-30T15:05:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2699511250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:26.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:26 smithi028 ceph-mon[41614]: pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-30T15:05:26.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: 101.1b scrub starts 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: 101.1b scrub ok 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: 101.11 scrub starts 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: 101.11 scrub ok 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2699511250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:26.553 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:26 smithi134 ceph-mon[33714]: pgmap v720: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 266 B/s wr, 2 op/s 2024-03-30T15:05:26.581 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:26.581 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:26.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:26.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:26.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:26.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:26.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:26.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:26.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:26.984 INFO:tasks.thrashosds.thrasher:j is {'epoch': 499, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:05:21.343600+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 112, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 304, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '437', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '491', 'last_force_op_resend': '411', 'last_force_op_resend_prenautilus': '411', 'last_force_op_resend_preluminous': '411', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 491, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 304, 'pool_name': 'test-rados-api-smithi028-93253-79', 'create_time': '2024-03-30T15:05:11.572750+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '496', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 496, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 493, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:34.941959+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:31.463076+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:14.412793+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:21.119206+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:07.577028+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:19.162669+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:18.056823+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:27.766874+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}, {'pgid': '5.13', 'mappings': [{'from': 3, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:05:26.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:05:26.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:26.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:27.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:27.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:27.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:05:27.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:27.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:27.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:27.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:27.419 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 4.f deep-scrub starts 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 4.f deep-scrub ok 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 5.1b scrub starts 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 5.1b scrub ok 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:05:27.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:27 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:05:27.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:27.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/1019120583 >> v1:172.21.15.28:6789/0 conn(0x7f6b94101140 legacy=0x7f6b94101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/1019120583 shutdown_connections 2024-03-30T15:05:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/1019120583 >> 172.21.15.28:0/1019120583 conn(0x7f6b940fc9f0 msgr2=0x7f6b940fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/1019120583 shutdown_connections 2024-03-30T15:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/1019120583 wait complete. 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 4.f deep-scrub starts 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 4.f deep-scrub ok 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 5.1b scrub starts 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 5.1b scrub ok 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:05:27.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 4.f deep-scrub starts 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 4.f deep-scrub ok 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 5.1b scrub starts 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 5.1b scrub ok 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:05:27.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:27 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:05:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 librados: starting msgr at 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 librados: starting objecter 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 Processor -- start 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 -- start start 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 librados: setting wanted keys 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.504+0000 7f6b9bf3e700 1 librados: calling monclient init 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b9bf3e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b940fe9e0 con 0x7f6b9410d970 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b9bf3e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b940febf0 con 0x7f6b94109dd0 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b9bf3e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b940fee00 con 0x7f6b94101140 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b99cda700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6b94101140 0x7f6b940fcea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36294/0 (socket says 172.21.15.28:36294) 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b99cda700 1 -- 172.21.15.28:0/4213565773 learned_addr learned my addr 172.21.15.28:0/4213565773 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter ms_handle_connect 0x7f6b94101140 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter ms_handle_connect 0x7f6b9410d970 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2787039951 0 0) 0x7f6b940fe9e0 con 0x7f6b9410d970 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b800039c0 con 0x7f6b9410d970 2024-03-30T15:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861399419 0 0) 0x7f6b94114df0 con 0x7f6b94101140 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b940fe9e0 con 0x7f6b94101140 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3556994830 0 0) 0x7f6b800039c0 con 0x7f6b9410d970 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b94114df0 con 0x7f6b9410d970 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6b8c001db0 con 0x7f6b9410d970 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter ms_handle_connect 0x7f6b94109dd0 2024-03-30T15:05:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.505+0000 7f6b8affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2116240166 0 0) 0x7f6b940fe9e0 con 0x7f6b94101140 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b800039c0 con 0x7f6b94101140 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6b90001980 con 0x7f6b94101140 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3305053679 0 0) 0x7f6b94114df0 con 0x7f6b9410d970 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 >> v1:172.21.15.28:6790/0 conn(0x7f6b94101140 legacy=0x7f6b940fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 >> v1:172.21.15.134:6789/0 conn(0x7f6b94109dd0 legacy=0x7f6b940fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:27.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b941ac9c0 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b941a9990 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b9bf3e700 10 client.25029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b941a9e50 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b9bf3e700 1 librados: init done 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6b8c0019b0 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.506+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6b8c004d70 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.507+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b78005120 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.507+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6b8c004f10 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(499..499 src has 442..499) ==== 8589+0+0 (unknown 1850587597 0 0) 0x7f6b8c088940 con 0x7f6b9410d970 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 10 client.25029.objecter ms_dispatch 0x7f6b94001550 osd_map(499..499 src has 442..499) 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 3 client.25029.objecter handle_osd_map got epochs [499,499] > 0 2024-03-30T15:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 3 client.25029.objecter handle_osd_map decoding full epoch 499 2024-03-30T15:05:27.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 20 client.25029.objecter dump_active .. 0 homeless 2024-03-30T15:05:27.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.508+0000 7f6b8affd700 10 client.25029.objecter ms_handle_connect 0x7f6b800702d0 2024-03-30T15:05:27.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.513+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6b8c0559a0 con 0x7f6b9410d970 2024-03-30T15:05:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:27.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:27.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:27.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:27.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:27.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:27.743+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f6b78005410 con 0x7f6b9410d970 2024-03-30T15:05:27.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:27.872 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:27.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:27.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:28.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:28.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:28.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:28.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:28.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.369+0000 7f6b8affd700 1 -- 172.21.15.28:0/4213565773 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v500) ==== 87+0+0 (unknown 3081676702 0 0) 0x7f6b8c0593f0 con 0x7f6b9410d970 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 10 librados: watch_flush enter 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 10 librados: watch_flush exit 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 20 client.25029.objecter shutdown clearing up homeless session... 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 10 client.25029.objecter successfully canceled tick 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6b800702d0 legacy=0x7f6b80072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:28.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 >> v1:172.21.15.28:6789/0 conn(0x7f6b9410d970 legacy=0x7f6b941a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:28.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:28.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 shutdown_connections 2024-03-30T15:05:28.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.372+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 >> 172.21.15.28:0/4213565773 conn(0x7f6b940fc9f0 msgr2=0x7f6b9406ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.373+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 shutdown_connections 2024-03-30T15:05:28.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.373+0000 7f6b9bf3e700 1 -- 172.21.15.28:0/4213565773 wait complete. 2024-03-30T15:05:28.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:28.373+0000 7f6b9bf3e700 1 librados: shutdown 2024-03-30T15:05:28.376 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:05:28.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:28.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:28.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 42.10 deep-scrub starts 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 42.10 deep-scrub ok 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 5.19 deep-scrub starts 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 5.19 deep-scrub ok 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 42.11 scrub starts 2024-03-30T15:05:28.587 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: 42.11 scrub ok 2024-03-30T15:05:28.588 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:05:28.588 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:28 smithi134 ceph-mon[33714]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-30T15:05:28.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:28.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 42.10 deep-scrub starts 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 42.10 deep-scrub ok 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 5.19 deep-scrub starts 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 5.19 deep-scrub ok 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 42.11 scrub starts 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: 42.11 scrub ok 2024-03-30T15:05:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[41614]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 42.10 deep-scrub starts 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 42.10 deep-scrub ok 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 5.19 deep-scrub starts 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 5.19 deep-scrub ok 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 42.11 scrub starts 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: 42.11 scrub ok 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-mon[29876]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-30T15:05:28.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:28.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:28.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:28.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:28.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:28.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:28.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:29.091 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:29.092 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:29.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:29.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:29.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:29.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:29.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:29.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 101.b deep-scrub starts 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 101.b deep-scrub ok 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 42.d scrub starts 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 42.d scrub ok 2024-03-30T15:05:29.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 101.b deep-scrub starts 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 101.b deep-scrub ok 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 42.d scrub starts 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 42.d scrub ok 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:05:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:29 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:05:29.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:29.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:29.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 101.b deep-scrub starts 2024-03-30T15:05:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 101.b deep-scrub ok 2024-03-30T15:05:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4213565773' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:05:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: osdmap e500: 8 total, 8 up, 8 in 2024-03-30T15:05:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 42.d scrub starts 2024-03-30T15:05:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 42.d scrub ok 2024-03-30T15:05:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:05:29.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:29 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:05:29.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:29.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:05:29.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:29.910 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:29.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:30.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:30.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:30.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:30.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:30.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:30.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.598+0000 7f615131b700 1 -- 172.21.15.28:0/2463735246 >> v1:172.21.15.28:6789/0 conn(0x7f614c100780 legacy=0x7f614c100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.598+0000 7f615131b700 1 -- 172.21.15.28:0/2463735246 shutdown_connections 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.598+0000 7f615131b700 1 -- 172.21.15.28:0/2463735246 >> 172.21.15.28:0/2463735246 conn(0x7f614c077370 msgr2=0x7f614c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.598+0000 7f615131b700 1 -- 172.21.15.28:0/2463735246 shutdown_connections 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.598+0000 7f615131b700 1 -- 172.21.15.28:0/2463735246 wait complete. 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 librados: starting msgr at 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 librados: starting objecter 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 Processor -- start 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 -- start start 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 librados: setting wanted keys 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.599+0000 7f615131b700 1 librados: calling monclient init 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f614affd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f614c100780 0x7f614c107410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:43870/0 (socket says 172.21.15.28:43870) 2024-03-30T15:05:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f614affd700 1 -- 172.21.15.28:0/97562125 learned_addr learned my addr 172.21.15.28:0/97562125 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f614c1a9220 con 0x7f614c10ea70 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f614c1aa440 con 0x7f614c10af00 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f614c1ab660 con 0x7f614c100780 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter ms_handle_connect 0x7f614c10ea70 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter ms_handle_connect 0x7f614c100780 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter ms_handle_connect 0x7f614c10af00 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517566247 0 0) 0x7f614c1a9220 con 0x7f614c10ea70 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61340031f0 con 0x7f614c10ea70 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384002911 0 0) 0x7f614c1ab660 con 0x7f614c100780 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f614c1a9220 con 0x7f614c100780 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057350787 0 0) 0x7f614c1aa440 con 0x7f614c10af00 2024-03-30T15:05:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f614c1ab660 con 0x7f614c10af00 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2678412503 0 0) 0x7f61340031f0 con 0x7f614c10ea70 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f614c1aa440 con 0x7f614c10ea70 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 458188258 0 0) 0x7f614c1a9220 con 0x7f614c100780 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61340031f0 con 0x7f614c100780 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3038805833 0 0) 0x7f614c1ab660 con 0x7f614c10af00 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f614c1a9220 con 0x7f614c10af00 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.600+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6144001590 con 0x7f614c10ea70 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f613c004500 con 0x7f614c100780 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f61400017b0 con 0x7f614c10af00 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3456965723 0 0) 0x7f614c1aa440 con 0x7f614c10ea70 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6790/0 conn(0x7f614c100780 legacy=0x7f614c107410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.134:6789/0 conn(0x7f614c10af00 legacy=0x7f614c1a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c1ac8a0 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f614c1ab890 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f615131b700 10 client.25035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f614c1abd50 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f615131b700 1 librados: init done 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6144003ea0 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.601+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6144004d80 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.602+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6138005100 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.603+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f614401b5c0 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.603+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(500..500 src has 442..500) ==== 8589+0+0 (unknown 1453343629 0 0) 0x7f61440889d0 con 0x7f614c10ea70 2024-03-30T15:05:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.603+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 osd_map(500..500 src has 442..500) 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.603+0000 7f6133fff700 3 client.25035.objecter handle_osd_map got epochs [500,500] > 0 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.603+0000 7f6133fff700 3 client.25035.objecter handle_osd_map decoding full epoch 500 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f6133fff700 20 client.25035.objecter dump_active .. 0 homeless 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f6134070400 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v500) ==== 54+0+16 (unknown 3119777350 0 2420000192) 0x7f6144088dc0 con 0x7f614c10ea70 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 10 client.25035.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 1 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 1 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 0 1 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 4 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 10 client.25035.objecter _send_command 1 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6138004b70 con 0x7f6138004730 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.604+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f6138004730 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.605+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6138004b70 con 0x7f6138004730 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.605+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.605+0000 7f6133fff700 10 client.25035.objecter _finish_command 1 = system:0 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.605+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 4 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.605+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 0 1 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 10 client.25035.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 2 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 2 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f61380053e0 osd=0 3 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 0 2 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 4 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 10 client.25035.objecter _send_command 2 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.623+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f61380079f0 con 0x7f6138004730 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f61380079f0 con 0x7f6138004730 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 10 client.25035.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 4 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 0 2 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 10 client.25035.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 3 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 3 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 1 3 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 4 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 10 client.25035.objecter _send_command 3 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f61380022a0 con 0x7f6138004310 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.625+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f6138004310 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.626+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f61380022a0 con 0x7f6138004310 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.626+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:30.316 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.626+0000 7f6133fff700 10 client.25035.objecter _finish_command 3 = system:0 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.626+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 4 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.626+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 1 3 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 10 client.25035.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 4 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 4 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138004130 osd=1 3 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 1 4 2024-03-30T15:05:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 4 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 10 client.25035.objecter _send_command 4 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.643+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6138001720 con 0x7f6138004310 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.644+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6138001720 con 0x7f6138004310 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.644+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.644+0000 7f6133fff700 10 client.25035.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.644+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 4 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.644+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 1 4 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 10 client.25035.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 5 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 5 2024-03-30T15:05:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 2 5 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 4 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 10 client.25035.objecter _send_command 5 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6138011170 con 0x7f613800d5d0 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.645+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f613800d5d0 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.646+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6138011170 con 0x7f613800d5d0 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.646+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.646+0000 7f6133fff700 10 client.25035.objecter _finish_command 5 = system:0 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.646+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 4 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.646+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 2 5 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 10 client.25035.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 6 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 6 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138001980 osd=2 3 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 2 6 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 4 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 10 client.25035.objecter _send_command 6 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.662+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6138008100 con 0x7f613800d5d0 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6138008100 con 0x7f613800d5d0 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 10 client.25035.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 4 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 2 6 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 10 client.25035.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 7 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 7 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 3 7 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 4 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 10 client.25035.objecter _send_command 7 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6138015fb0 con 0x7f61380124a0 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.664+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f61380124a0 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.665+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6138015fb0 con 0x7f61380124a0 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.665+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.665+0000 7f6133fff700 10 client.25035.objecter _finish_command 7 = system:0 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.665+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 4 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.665+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 3 7 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 10 client.25035.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 8 2024-03-30T15:05:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 8 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613800cc20 osd=3 3 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 3 8 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 4 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 10 client.25035.objecter _send_command 8 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.681+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f613800c700 con 0x7f61380124a0 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f613800c700 con 0x7f61380124a0 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f6133fff700 10 client.25035.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 4 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 3 8 2024-03-30T15:05:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 10 client.25035.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 9 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 9 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 4 9 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 4 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 10 client.25035.objecter _send_command 9 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.683+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f613800c740 con 0x7f6138012000 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f6138012000 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f613800c740 con 0x7f6138012000 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:30.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 10 client.25035.objecter _finish_command 9 = system:0 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 4 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.684+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 4 9 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 10 client.25035.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 10 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:30.324 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 10 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613800c4b0 osd=4 3 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 4 10 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 4 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 10 client.25035.objecter _send_command 10 2024-03-30T15:05:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.701+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f613801c030 con 0x7f6138012000 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f613801c030 con 0x7f6138012000 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 10 client.25035.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 4 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 4 10 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 10 client.25035.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 11 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 11 2024-03-30T15:05:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 5 11 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 4 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 10 client.25035.objecter _send_command 11 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f613801cbe0 con 0x7f613801c750 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.703+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f613801c750 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.704+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f613801cbe0 con 0x7f613801c750 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.704+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.704+0000 7f6133fff700 10 client.25035.objecter _finish_command 11 = system:0 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.704+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 4 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.704+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 5 11 2024-03-30T15:05:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 10 client.25035.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 12 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 12 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613801c030 osd=5 3 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 5 12 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 4 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 10 client.25035.objecter _send_command 12 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.721+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f613801cbe0 con 0x7f613801c750 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.722+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f613801cbe0 con 0x7f613801c750 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.722+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.722+0000 7f6133fff700 10 client.25035.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.722+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 4 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.722+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 5 12 2024-03-30T15:05:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 10 client.25035.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 13 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 13 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 6 13 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 4 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 10 client.25035.objecter _send_command 13 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6138023230 con 0x7f613801f720 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.723+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f613801f720 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.724+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6138023230 con 0x7f613801f720 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.724+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.724+0000 7f6133fff700 10 client.25035.objecter _finish_command 13 = system:0 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.724+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 4 2024-03-30T15:05:30.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.724+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 6 13 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 10 client.25035.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 14 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 14 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f613801cbe0 osd=6 3 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 6 14 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 4 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 10 client.25035.objecter _send_command 14 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.740+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6138023230 con 0x7f613801f720 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6138023230 con 0x7f613801f720 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f6133fff700 10 client.25035.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 4 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 6 14 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 10 client.25035.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 15 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 15 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 7 15 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.742+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 4 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.743+0000 7f615131b700 10 client.25035.objecter _send_command 15 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.743+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6138027600 con 0x7f61380238b0 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.743+0000 7f6133fff700 10 client.25035.objecter ms_handle_connect 0x7f61380238b0 2024-03-30T15:05:30.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.744+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6138027600 con 0x7f61380238b0 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.744+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.744+0000 7f6133fff700 10 client.25035.objecter _finish_command 15 = system:0 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.744+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 4 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.744+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 7 15 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 10 client.25035.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign -1 16 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 20 client.25035.objecter _get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 15 client.25035.objecter _session_command_op_remove -1 16 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 20 client.25035.objecter get_session s=0x7f6138023230 osd=7 3 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 15 client.25035.objecter _session_command_op_assign 7 16 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 4 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 10 client.25035.objecter _send_command 16 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.760+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6138027600 con 0x7f61380238b0 2024-03-30T15:05:30.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.762+0000 7f6133fff700 1 -- 172.21.15.28:0/97562125 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6138027600 con 0x7f61380238b0 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.762+0000 7f6133fff700 10 client.25035.objecter ms_dispatch 0x7f614c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.762+0000 7f6133fff700 10 client.25035.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.762+0000 7f6133fff700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 4 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.762+0000 7f6133fff700 15 client.25035.objecter _session_command_op_remove 7 16 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 librados: watch_flush enter 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 librados: watch_flush exit 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.0 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6138004730 legacy=0x7f6138002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f61380053e0 osd=0 1 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.1 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6138004310 legacy=0x7f613800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138004130 osd=1 1 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.2 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6809/3451325043 conn(0x7f613800d5d0 legacy=0x7f613800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f614b7fe700 1 -- 172.21.15.28:0/97562125 reap_dead start 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138001980 osd=2 1 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.3 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6813/1506157250 conn(0x7f61380124a0 legacy=0x7f6138014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800cc20 osd=3 1 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.4 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6138012000 legacy=0x7f6138019140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613800c4b0 osd=4 1 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.5 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.134:6804/3668151179 conn(0x7f613801c750 legacy=0x7f613801a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801c030 osd=5 1 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.6 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.134:6808/2135224194 conn(0x7f613801f720 legacy=0x7f6138021b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f613801cbe0 osd=6 1 2024-03-30T15:05:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter close_session for osd.7 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.134:6812/3758190298 conn(0x7f61380238b0 legacy=0x7f6138025d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter put_session s=0x7f6138023230 osd=7 1 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 20 client.25035.objecter shutdown clearing up homeless session... 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 10 client.25035.objecter successfully canceled tick 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f614b7fe700 1 -- 172.21.15.28:0/97562125 reap_dead start 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6134070400 legacy=0x7f61340728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> v1:172.21.15.28:6789/0 conn(0x7f614c10ea70 legacy=0x7f614c1a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 shutdown_connections 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 >> 172.21.15.28:0/97562125 conn(0x7f614c077370 msgr2=0x7f614c110ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 shutdown_connections 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 -- 172.21.15.28:0/97562125 wait complete. 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:29.763+0000 7f615131b700 1 librados: shutdown 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:30.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:30.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:30.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:30.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:30.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/97562125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:30.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:30 smithi028 ceph-mon[29876]: pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:30.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/97562125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:30.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:30 smithi028 ceph-mon[41614]: pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:30.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:30.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/97562125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:30 smithi134 ceph-mon[33714]: pgmap v723: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:30.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:30.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:30.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:30.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:30.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:31.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:31.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:31.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:31.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:31.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:31.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:31.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:31.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:31.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:31.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:31.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:31.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:31.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:31.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:32.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:32.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:32.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:32.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:32.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:32.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.493+0000 7f63492a7700 1 -- 172.21.15.28:0/219576091 >> v1:172.21.15.28:6789/0 conn(0x7f634410eb70 legacy=0x7f6344111010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.493+0000 7f63492a7700 1 -- 172.21.15.28:0/219576091 shutdown_connections 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.493+0000 7f63492a7700 1 -- 172.21.15.28:0/219576091 >> 172.21.15.28:0/219576091 conn(0x7f6344077370 msgr2=0x7f63440fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.493+0000 7f63492a7700 1 -- 172.21.15.28:0/219576091 shutdown_connections 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.493+0000 7f63492a7700 1 -- 172.21.15.28:0/219576091 wait complete. 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 librados: starting msgr at 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 librados: starting objecter 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 Processor -- start 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 -- start start 2024-03-30T15:05:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 librados: setting wanted keys 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 librados: calling monclient init 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63441a71e0 con 0x7f6344106fd0 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63441a8400 con 0x7f634410eb70 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.494+0000 7f63492a7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f63441a9640 con 0x7f634410ae70 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f6342ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6344106fd0 0x7f63440ff630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33588/0 (socket says 172.21.15.28:33588) 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f6342ffd700 1 -- 172.21.15.28:0/972271089 learned_addr learned my addr 172.21.15.28:0/972271089 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter ms_handle_connect 0x7f6344106fd0 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter ms_handle_connect 0x7f634410ae70 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter ms_handle_connect 0x7f634410eb70 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289374880 0 0) 0x7f63441a71e0 con 0x7f6344106fd0 2024-03-30T15:05:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63300031f0 con 0x7f6344106fd0 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661880034 0 0) 0x7f63441a9640 con 0x7f634410ae70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63441a71e0 con 0x7f634410ae70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420922846 0 0) 0x7f63441a8400 con 0x7f634410eb70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63441a9640 con 0x7f634410eb70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2161292783 0 0) 0x7f63300031f0 con 0x7f6344106fd0 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63441a8400 con 0x7f6344106fd0 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6334001cb0 con 0x7f6344106fd0 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877486122 0 0) 0x7f63441a9640 con 0x7f634410eb70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63300031f0 con 0x7f634410eb70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f633c004400 con 0x7f634410eb70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3473349906 0 0) 0x7f63441a71e0 con 0x7f634410ae70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63441a9640 con 0x7f634410ae70 2024-03-30T15:05:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f632c001890 con 0x7f634410ae70 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2317937547 0 0) 0x7f63441a8400 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 >> v1:172.21.15.28:6790/0 conn(0x7f634410ae70 legacy=0x7f63441a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 >> v1:172.21.15.134:6789/0 conn(0x7f634410eb70 legacy=0x7f63441a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.495+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63441aa880 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f63492a7700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63441a9870 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f63492a7700 10 client.25041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f63492a7700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63441a9d30 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f63492a7700 1 librados: init done 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6334003ec0 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.496+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6334006090 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.497+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6328005120 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.497+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6334003a20 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(500..500 src has 442..500) ==== 8589+0+0 (unknown 1453343629 0 0) 0x7f6334089c40 con 0x7f6344106fd0 2024-03-30T15:05:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 10 client.25041.objecter ms_dispatch 0x7f6344001550 osd_map(500..500 src has 442..500) 2024-03-30T15:05:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 3 client.25041.objecter handle_osd_map got epochs [500,500] > 0 2024-03-30T15:05:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 3 client.25041.objecter handle_osd_map decoding full epoch 500 2024-03-30T15:05:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 20 client.25041.objecter dump_active .. 0 homeless 2024-03-30T15:05:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.498+0000 7f633bfff700 10 client.25041.objecter ms_handle_connect 0x7f63300704c0 2024-03-30T15:05:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.502+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6334056ca0 con 0x7f6344106fd0 2024-03-30T15:05:32.534 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:32.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:32.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:32 smithi028 ceph-mon[29876]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:05:32.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:32 smithi028 ceph-mon[41614]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:05:32.679 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:32 smithi134 ceph-mon[33714]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:05:32.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:32.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.709+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6328000b90 con 0x7f63300704c0 2024-03-30T15:05:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.727+0000 7f633bfff700 1 -- 172.21.15.28:0/972271089 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+666679 (unknown 2965378022 0 463309182) 0x7f6328000b90 con 0x7f63300704c0 2024-03-30T15:05:32.737 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.732+0000 7f6339ffb700 10 librados: watch_flush enter 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.732+0000 7f6339ffb700 10 librados: watch_flush exit 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.732+0000 7f6339ffb700 20 client.25041.objecter shutdown clearing up homeless session... 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.732+0000 7f6339ffb700 10 client.25041.objecter successfully canceled tick 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 >> v1:172.21.15.28:6800/3011431093 conn(0x7f63300704c0 legacy=0x7f6330072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 >> v1:172.21.15.28:6789/0 conn(0x7f6344106fd0 legacy=0x7f63440ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 shutdown_connections 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 >> 172.21.15.28:0/972271089 conn(0x7f6344077370 msgr2=0x7f6344111f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 shutdown_connections 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 -- 172.21.15.28:0/972271089 wait complete. 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:32.733+0000 7f6339ffb700 1 librados: shutdown 2024-03-30T15:05:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:05:32.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:32.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:32.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:32.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:32.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:33.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:05:33.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:33.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:33.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:33.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:33.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:33.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:33.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:33.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:33.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:33 smithi028 ceph-mon[41614]: from='client.25041 v1:172.21.15.28:0/972271089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:33.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:33 smithi028 ceph-mon[29876]: from='client.25041 v1:172.21.15.28:0/972271089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:33.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:33.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:05:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:33 smithi134 ceph-mon[33714]: from='client.25041 v1:172.21.15.28:0/972271089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:05:33.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:33.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:33.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:33.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:33.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:33.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:34.024 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2024-03-30T15:05:32.018975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792290,"num_objects":378,"num_object_clones":12,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":2126,"num_read_kb":1866,"num_write":1544,"num_write_kb":15187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2964,"ondisk_log_size":2964,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":780,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":609912,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":528371,"kb_avail":749351304,"statfs":{"total":767960285184,"available":767335735296,"internally_reserved":0,"allocated":82870272,"data_stored":56077691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":541052464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003742"},"pg_stats":[{"pgid":"304.1f","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034251+0000","last_change":"2024-03-30T15:05:13.343866+0000","last_active":"2024-03-30T15:05:22.034251+0000","last_peered":"2024-03-30T15:05:22.034251+0000","last_clean":"2024-03-30T15:05:22.034251+0000","last_became_active":"2024-03-30T15:05:13.343591+0000","last_became_peered":"2024-03-30T15:05:13.343591+0000","last_unstale":"2024-03-30T15:05:22.034251+0000","last_undegraded":"2024-03-30T15:05:22.034251+0000","last_fullsized":"2024-03-30T15:05:22.034251+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:24.287286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1e","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033857+0000","last_change":"2024-03-30T15:05:13.344170+0000","last_active":"2024-03-30T15:05:22.033857+0000","last_peered":"2024-03-30T15:05:22.033857+0000","last_clean":"2024-03-30T15:05:22.033857+0000","last_became_active":"2024-03-30T15:05:13.343841+0000","last_became_peered":"2024-03-30T15:05:13.343841+0000","last_unstale":"2024-03-30T15:05:22.033857+0000","last_undegraded":"2024-03-30T15:05:22.033857+0000","last_fullsized":"2024-03-30T15:05:22.033857+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1d","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379852+0000","last_change":"2024-03-30T15:05:13.337980+0000","last_active":"2024-03-30T15:05:28.379852+0000","last_peered":"2024-03-30T15:05:28.379852+0000","last_clean":"2024-03-30T15:05:28.379852+0000","last_became_active":"2024-03-30T15:05:13.337694+0000","last_became_peered":"2024-03-30T15:05:13.337694+0000","last_unstale":"2024-03-30T15:05:28.379852+0000","last_undegraded":"2024-03-30T15:05:28.379852+0000","last_fullsized":"2024-03-30T15:05:28.379852+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:23.773050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"304.1c","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063808+0000","last_change":"2024-03-30T15:05:13.332109+0000","last_active":"2024-03-30T15:05:22.063808+0000","last_peered":"2024-03-30T15:05:22.063808+0000","last_clean":"2024-03-30T15:05:22.063808+0000","last_became_active":"2024-03-30T15:05:13.331788+0000","last_became_peered":"2024-03-30T15:05:13.331788+0000","last_unstale":"2024-03-30T15:05:22.063808+0000","last_undegraded":"2024-03-30T15:05:22.063808+0000","last_fullsized":"2024-03-30T15:05:22.063808+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.161059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.1b","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033100+0000","last_change":"2024-03-30T15:05:13.344037+0000","last_active":"2024-03-30T15:05:22.033100+0000","last_peered":"2024-03-30T15:05:22.033100+0000","last_clean":"2024-03-30T15:05:22.033100+0000","last_became_active":"2024-03-30T15:05:13.343558+0000","last_became_peered":"2024-03-30T15:05:13.343558+0000","last_unstale":"2024-03-30T15:05:22.033100+0000","last_undegraded":"2024-03-30T15:05:22.033100+0000","last_fullsized":"2024-03-30T15:05:22.033100+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.510657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1a","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967833+0000","last_change":"2024-03-30T15:05:13.336866+0000","last_active":"2024-03-30T15:05:21.967833+0000","last_peered":"2024-03-30T15:05:21.967833+0000","last_clean":"2024-03-30T15:05:21.967833+0000","last_became_active":"2024-03-30T15:05:13.336536+0000","last_became_peered":"2024-03-30T15:05:13.336536+0000","last_unstale":"2024-03-30T15:05:21.967833+0000","last_undegraded":"2024-03-30T15:05:21.967833+0000","last_fullsized":"2024-03-30T15:05:21.967833+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.200069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.19","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967737+0000","last_change":"2024-03-30T15:05:13.347183+0000","last_active":"2024-03-30T15:05:21.967737+0000","last_peered":"2024-03-30T15:05:21.967737+0000","last_clean":"2024-03-30T15:05:21.967737+0000","last_became_active":"2024-03-30T15:05:13.346878+0000","last_became_peered":"2024-03-30T15:05:13.346878+0000","last_unstale":"2024-03-30T15:05:21.967737+0000","last_undegraded":"2024-03-30T15:05:21.967737+0000","last_fullsized":"2024-03-30T15:05:21.967737+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.962908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"304.18","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143200+0000","last_change":"2024-03-30T15:05:13.336581+0000","last_active":"2024-03-30T15:05:22.143200+0000","last_peered":"2024-03-30T15:05:22.143200+0000","last_clean":"2024-03-30T15:05:22.143200+0000","last_became_active":"2024-03-30T15:05:13.336207+0000","last_became_peered":"2024-03-30T15:05:13.336207+0000","last_unstale":"2024-03-30T15:05:22.143200+0000","last_undegraded":"2024-03-30T15:05:22.143200+0000","last_fullsized":"2024-03-30T15:05:22.143200+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:26.293181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.17","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380241+0000","last_change":"2024-03-30T15:05:13.339981+0000","last_active":"2024-03-30T15:05:28.380241+0000","last_peered":"2024-03-30T15:05:28.380241+0000","last_clean":"2024-03-30T15:05:28.380241+0000","last_became_active":"2024-03-30T15:05:13.339642+0000","last_became_peered":"2024-03-30T15:05:13.339642+0000","last_unstale":"2024-03-30T15:05:28.380241+0000","last_undegraded":"2024-03-30T15:05:28.380241+0000","last_fullsized":"2024-03-30T15:05:28.380241+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.994337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"304.16","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405356+0000","last_change":"2024-03-30T15:05:13.343874+0000","last_active":"2024-03-30T15:05:28.405356+0000","last_peered":"2024-03-30T15:05:28.405356+0000","last_clean":"2024-03-30T15:05:28.405356+0000","last_became_active":"2024-03-30T15:05:13.343568+0000","last_became_peered":"2024-03-30T15:05:13.343568+0000","last_unstale":"2024-03-30T15:05:28.405356+0000","last_undegraded":"2024-03-30T15:05:28.405356+0000","last_fullsized":"2024-03-30T15:05:28.405356+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:23.210771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.15","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143900+0000","last_change":"2024-03-30T15:05:13.334764+0000","last_active":"2024-03-30T15:05:22.143900+0000","last_peered":"2024-03-30T15:05:22.143900+0000","last_clean":"2024-03-30T15:05:22.143900+0000","last_became_active":"2024-03-30T15:05:13.334353+0000","last_became_peered":"2024-03-30T15:05:13.334353+0000","last_unstale":"2024-03-30T15:05:22.143900+0000","last_undegraded":"2024-03-30T15:05:22.143900+0000","last_fullsized":"2024-03-30T15:05:22.143900+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:14.457460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.14","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063880+0000","last_change":"2024-03-30T15:05:13.329452+0000","last_active":"2024-03-30T15:05:22.063880+0000","last_peered":"2024-03-30T15:05:22.063880+0000","last_clean":"2024-03-30T15:05:22.063880+0000","last_became_active":"2024-03-30T15:05:13.329130+0000","last_became_peered":"2024-03-30T15:05:13.329130+0000","last_unstale":"2024-03-30T15:05:22.063880+0000","last_undegraded":"2024-03-30T15:05:22.063880+0000","last_fullsized":"2024-03-30T15:05:22.063880+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.783055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.13","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063722+0000","last_change":"2024-03-30T15:05:13.344097+0000","last_active":"2024-03-30T15:05:22.063722+0000","last_peered":"2024-03-30T15:05:22.063722+0000","last_clean":"2024-03-30T15:05:22.063722+0000","last_became_active":"2024-03-30T15:05:13.343627+0000","last_became_peered":"2024-03-30T15:05:13.343627+0000","last_unstale":"2024-03-30T15:05:22.063722+0000","last_undegraded":"2024-03-30T15:05:22.063722+0000","last_fullsized":"2024-03-30T15:05:22.063722+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.482850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.12","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032803+0000","last_change":"2024-03-30T15:05:13.336998+0000","last_active":"2024-03-30T15:05:22.032803+0000","last_peered":"2024-03-30T15:05:22.032803+0000","last_clean":"2024-03-30T15:05:22.032803+0000","last_became_active":"2024-03-30T15:05:13.336699+0000","last_became_peered":"2024-03-30T15:05:13.336699+0000","last_unstale":"2024-03-30T15:05:22.032803+0000","last_undegraded":"2024-03-30T15:05:22.032803+0000","last_fullsized":"2024-03-30T15:05:22.032803+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.413130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.11","version":"496'8","reported_seq":37,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143272+0000","last_change":"2024-03-30T15:05:16.138802+0000","last_active":"2024-03-30T15:05:22.143272+0000","last_peered":"2024-03-30T15:05:22.143272+0000","last_clean":"2024-03-30T15:05:22.143272+0000","last_became_active":"2024-03-30T15:05:13.329076+0000","last_became_peered":"2024-03-30T15:05:13.329076+0000","last_unstale":"2024-03-30T15:05:22.143272+0000","last_undegraded":"2024-03-30T15:05:22.143272+0000","last_fullsized":"2024-03-30T15:05:22.143272+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'8","last_scrub_stamp":"2024-03-30T15:05:16.138717+0000","last_deep_scrub":"496'8","last_deep_scrub_stamp":"2024-03-30T15:05:16.138717+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.138717+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.039578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.10","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032420+0000","last_change":"2024-03-30T15:05:13.345632+0000","last_active":"2024-03-30T15:05:22.032420+0000","last_peered":"2024-03-30T15:05:22.032420+0000","last_clean":"2024-03-30T15:05:22.032420+0000","last_became_active":"2024-03-30T15:05:13.345171+0000","last_became_peered":"2024-03-30T15:05:13.345171+0000","last_unstale":"2024-03-30T15:05:22.032420+0000","last_undegraded":"2024-03-30T15:05:22.032420+0000","last_fullsized":"2024-03-30T15:05:22.032420+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.888536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.f","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144129+0000","last_change":"2024-03-30T15:05:13.336249+0000","last_active":"2024-03-30T15:05:22.144129+0000","last_peered":"2024-03-30T15:05:22.144129+0000","last_clean":"2024-03-30T15:05:22.144129+0000","last_became_active":"2024-03-30T15:05:13.335669+0000","last_became_peered":"2024-03-30T15:05:13.335669+0000","last_unstale":"2024-03-30T15:05:22.144129+0000","last_undegraded":"2024-03-30T15:05:22.144129+0000","last_fullsized":"2024-03-30T15:05:22.144129+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:23.258437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.e","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375068+0000","last_change":"2024-03-30T15:05:13.338019+0000","last_active":"2024-03-30T15:05:28.375068+0000","last_peered":"2024-03-30T15:05:28.375068+0000","last_clean":"2024-03-30T15:05:28.375068+0000","last_became_active":"2024-03-30T15:05:13.337705+0000","last_became_peered":"2024-03-30T15:05:13.337705+0000","last_unstale":"2024-03-30T15:05:28.375068+0000","last_undegraded":"2024-03-30T15:05:28.375068+0000","last_fullsized":"2024-03-30T15:05:28.375068+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.256364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.d","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144388+0000","last_change":"2024-03-30T15:05:13.333258+0000","last_active":"2024-03-30T15:05:22.144388+0000","last_peered":"2024-03-30T15:05:22.144388+0000","last_clean":"2024-03-30T15:05:22.144388+0000","last_became_active":"2024-03-30T15:05:13.332983+0000","last_became_peered":"2024-03-30T15:05:13.332983+0000","last_unstale":"2024-03-30T15:05:22.144388+0000","last_undegraded":"2024-03-30T15:05:22.144388+0000","last_fullsized":"2024-03-30T15:05:22.144388+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.148410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"304.c","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374253+0000","last_change":"2024-03-30T15:05:13.343804+0000","last_active":"2024-03-30T15:05:28.374253+0000","last_peered":"2024-03-30T15:05:28.374253+0000","last_clean":"2024-03-30T15:05:28.374253+0000","last_became_active":"2024-03-30T15:05:13.343461+0000","last_became_peered":"2024-03-30T15:05:13.343461+0000","last_unstale":"2024-03-30T15:05:28.374253+0000","last_undegraded":"2024-03-30T15:05:28.374253+0000","last_fullsized":"2024-03-30T15:05:28.374253+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.905194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.b","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404782+0000","last_change":"2024-03-30T15:05:13.343804+0000","last_active":"2024-03-30T15:05:28.404782+0000","last_peered":"2024-03-30T15:05:28.404782+0000","last_clean":"2024-03-30T15:05:28.404782+0000","last_became_active":"2024-03-30T15:05:13.343512+0000","last_became_peered":"2024-03-30T15:05:13.343512+0000","last_unstale":"2024-03-30T15:05:28.404782+0000","last_undegraded":"2024-03-30T15:05:28.404782+0000","last_fullsized":"2024-03-30T15:05:28.404782+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.141771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.a","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373310+0000","last_change":"2024-03-30T15:05:13.332286+0000","last_active":"2024-03-30T15:05:28.373310+0000","last_peered":"2024-03-30T15:05:28.373310+0000","last_clean":"2024-03-30T15:05:28.373310+0000","last_became_active":"2024-03-30T15:05:13.331872+0000","last_became_peered":"2024-03-30T15:05:13.331872+0000","last_unstale":"2024-03-30T15:05:28.373310+0000","last_undegraded":"2024-03-30T15:05:28.373310+0000","last_fullsized":"2024-03-30T15:05:28.373310+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.338548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.9","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062830+0000","last_change":"2024-03-30T15:05:13.344837+0000","last_active":"2024-03-30T15:05:22.062830+0000","last_peered":"2024-03-30T15:05:22.062830+0000","last_clean":"2024-03-30T15:05:22.062830+0000","last_became_active":"2024-03-30T15:05:13.343863+0000","last_became_peered":"2024-03-30T15:05:13.343863+0000","last_unstale":"2024-03-30T15:05:22.062830+0000","last_undegraded":"2024-03-30T15:05:22.062830+0000","last_fullsized":"2024-03-30T15:05:22.062830+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.889855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.8","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062816+0000","last_change":"2024-03-30T15:05:13.345256+0000","last_active":"2024-03-30T15:05:22.062816+0000","last_peered":"2024-03-30T15:05:22.062816+0000","last_clean":"2024-03-30T15:05:22.062816+0000","last_became_active":"2024-03-30T15:05:13.344921+0000","last_became_peered":"2024-03-30T15:05:13.344921+0000","last_unstale":"2024-03-30T15:05:22.062816+0000","last_undegraded":"2024-03-30T15:05:22.062816+0000","last_fullsized":"2024-03-30T15:05:22.062816+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.038504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.7","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064336+0000","last_change":"2024-03-30T15:05:13.326046+0000","last_active":"2024-03-30T15:05:22.064336+0000","last_peered":"2024-03-30T15:05:22.064336+0000","last_clean":"2024-03-30T15:05:22.064336+0000","last_became_active":"2024-03-30T15:05:13.325739+0000","last_became_peered":"2024-03-30T15:05:13.325739+0000","last_unstale":"2024-03-30T15:05:22.064336+0000","last_undegraded":"2024-03-30T15:05:22.064336+0000","last_fullsized":"2024-03-30T15:05:22.064336+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.360244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.6","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405481+0000","last_change":"2024-03-30T15:05:13.339237+0000","last_active":"2024-03-30T15:05:28.405481+0000","last_peered":"2024-03-30T15:05:28.405481+0000","last_clean":"2024-03-30T15:05:28.405481+0000","last_became_active":"2024-03-30T15:05:13.338947+0000","last_became_peered":"2024-03-30T15:05:13.338947+0000","last_unstale":"2024-03-30T15:05:28.405481+0000","last_undegraded":"2024-03-30T15:05:28.405481+0000","last_fullsized":"2024-03-30T15:05:28.405481+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:15.233289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.5","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405173+0000","last_change":"2024-03-30T15:05:13.338870+0000","last_active":"2024-03-30T15:05:28.405173+0000","last_peered":"2024-03-30T15:05:28.405173+0000","last_clean":"2024-03-30T15:05:28.405173+0000","last_became_active":"2024-03-30T15:05:13.338587+0000","last_became_peered":"2024-03-30T15:05:13.338587+0000","last_unstale":"2024-03-30T15:05:28.405173+0000","last_undegraded":"2024-03-30T15:05:28.405173+0000","last_fullsized":"2024-03-30T15:05:28.405173+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.863271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"304.4","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063947+0000","last_change":"2024-03-30T15:05:13.344393+0000","last_active":"2024-03-30T15:05:22.063947+0000","last_peered":"2024-03-30T15:05:22.063947+0000","last_clean":"2024-03-30T15:05:22.063947+0000","last_became_active":"2024-03-30T15:05:13.343541+0000","last_became_peered":"2024-03-30T15:05:13.343541+0000","last_unstale":"2024-03-30T15:05:22.063947+0000","last_undegraded":"2024-03-30T15:05:22.063947+0000","last_fullsized":"2024-03-30T15:05:22.063947+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:41.869519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"304.3","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373516+0000","last_change":"2024-03-30T15:05:13.338045+0000","last_active":"2024-03-30T15:05:28.373516+0000","last_peered":"2024-03-30T15:05:28.373516+0000","last_clean":"2024-03-30T15:05:28.373516+0000","last_became_active":"2024-03-30T15:05:13.337722+0000","last_became_peered":"2024-03-30T15:05:13.337722+0000","last_unstale":"2024-03-30T15:05:28.373516+0000","last_undegraded":"2024-03-30T15:05:28.373516+0000","last_fullsized":"2024-03-30T15:05:28.373516+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.537321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.2","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032929+0000","last_change":"2024-03-30T15:05:13.336923+0000","last_active":"2024-03-30T15:05:22.032929+0000","last_peered":"2024-03-30T15:05:22.032929+0000","last_clean":"2024-03-30T15:05:22.032929+0000","last_became_active":"2024-03-30T15:05:13.336565+0000","last_became_peered":"2024-03-30T15:05:13.336565+0000","last_unstale":"2024-03-30T15:05:22.032929+0000","last_undegraded":"2024-03-30T15:05:22.032929+0000","last_fullsized":"2024-03-30T15:05:22.032929+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.352473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"304.1","version":"0'0","reported_seq":24,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373097+0000","last_change":"2024-03-30T15:05:13.335291+0000","last_active":"2024-03-30T15:05:28.373097+0000","last_peered":"2024-03-30T15:05:28.373097+0000","last_clean":"2024-03-30T15:05:28.373097+0000","last_became_active":"2024-03-30T15:05:13.335031+0000","last_became_peered":"2024-03-30T15:05:13.335031+0000","last_unstale":"2024-03-30T15:05:28.373097+0000","last_undegraded":"2024-03-30T15:05:28.373097+0000","last_fullsized":"2024-03-30T15:05:28.373097+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.680177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"304.0","version":"0'0","reported_seq":22,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062905+0000","last_change":"2024-03-30T15:05:13.345724+0000","last_active":"2024-03-30T15:05:22.062905+0000","last_peered":"2024-03-30T15:05:22.062905+0000","last_clean":"2024-03-30T15:05:22.062905+0000","last_became_active":"2024-03-30T15:05:13.344827+0000","last_became_peered":"2024-03-30T15:05:13.344827+0000","last_unstale":"2024-03-30T15:05:22.062905+0000","last_undegraded":"2024-03-30T15:05:22.062905+0000","last_fullsized":"2024-03-30T15:05:22.062905+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":493,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:12.304412+0000","last_clean_scrub_stamp":"2024-03-30T15:05:12.304412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.584413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1a","version":"267'39","reported_seq":993,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063635+0000","last_change":"2024-03-30T15:04:06.344710+0000","last_active":"2024-03-30T15:05:22.063635+0000","last_peered":"2024-03-30T15:05:22.063635+0000","last_clean":"2024-03-30T15:05:22.063635+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:05:22.063635+0000","last_undegraded":"2024-03-30T15:05:22.063635+0000","last_fullsized":"2024-03-30T15:05:22.063635+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":1023,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379545+0000","last_change":"2024-03-30T15:05:25.203009+0000","last_active":"2024-03-30T15:05:28.379545+0000","last_peered":"2024-03-30T15:05:28.379545+0000","last_clean":"2024-03-30T15:05:28.379545+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:05:28.379545+0000","last_undegraded":"2024-03-30T15:05:28.379545+0000","last_fullsized":"2024-03-30T15:05:28.379545+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:05:25.202851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.202851+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:35.901031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"474'106","reported_seq":1049,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404395+0000","last_change":"2024-03-30T15:05:23.274570+0000","last_active":"2024-03-30T15:05:28.404395+0000","last_peered":"2024-03-30T15:05:28.404395+0000","last_clean":"2024-03-30T15:05:28.404395+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:05:28.404395+0000","last_undegraded":"2024-03-30T15:05:28.404395+0000","last_fullsized":"2024-03-30T15:05:28.404395+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'106","last_scrub_stamp":"2024-03-30T15:05:23.274464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.274464+0000","objects_scrubbed":11,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.451911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":98,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"400'105","reported_seq":1051,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967817+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:05:21.967817+0000","last_peered":"2024-03-30T15:05:21.967817+0000","last_clean":"2024-03-30T15:05:21.967817+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:05:21.967817+0000","last_undegraded":"2024-03-30T15:05:21.967817+0000","last_fullsized":"2024-03-30T15:05:21.967817+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"491'43","reported_seq":997,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064126+0000","last_change":"2024-03-30T15:04:06.346222+0000","last_active":"2024-03-30T15:05:22.064126+0000","last_peered":"2024-03-30T15:05:22.064126+0000","last_clean":"2024-03-30T15:05:22.064126+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:05:22.064126+0000","last_undegraded":"2024-03-30T15:05:22.064126+0000","last_fullsized":"2024-03-30T15:05:22.064126+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'36","last_scrub_stamp":"2024-03-30T15:03:19.020165+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.020165+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"415'82","reported_seq":1085,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064823+0000","last_change":"2024-03-30T15:04:06.347583+0000","last_active":"2024-03-30T15:05:22.064823+0000","last_peered":"2024-03-30T15:05:22.064823+0000","last_clean":"2024-03-30T15:05:22.064823+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:05:22.064823+0000","last_undegraded":"2024-03-30T15:05:22.064823+0000","last_fullsized":"2024-03-30T15:05:22.064823+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'80","last_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.929964+0000","objects_scrubbed":9,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4194351,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":27,"num_write_kb":8216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"418'38","reported_seq":977,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379618+0000","last_change":"2024-03-30T15:04:06.381417+0000","last_active":"2024-03-30T15:05:28.379618+0000","last_peered":"2024-03-30T15:05:28.379618+0000","last_clean":"2024-03-30T15:05:28.379618+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:05:28.379618+0000","last_undegraded":"2024-03-30T15:05:28.379618+0000","last_fullsized":"2024-03-30T15:05:28.379618+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'34","last_scrub_stamp":"2024-03-30T15:02:30.913015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.913015+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"368'53","reported_seq":997,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405220+0000","last_change":"2024-03-30T15:04:06.473326+0000","last_active":"2024-03-30T15:05:28.405220+0000","last_peered":"2024-03-30T15:05:28.405220+0000","last_clean":"2024-03-30T15:05:28.405220+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:05:28.405220+0000","last_undegraded":"2024-03-30T15:05:28.405220+0000","last_fullsized":"2024-03-30T15:05:28.405220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":997,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373406+0000","last_change":"2024-03-30T15:04:34.489872+0000","last_active":"2024-03-30T15:05:28.373406+0000","last_peered":"2024-03-30T15:05:28.373406+0000","last_clean":"2024-03-30T15:05:28.373406+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:05:28.373406+0000","last_undegraded":"2024-03-30T15:05:28.373406+0000","last_fullsized":"2024-03-30T15:05:28.373406+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:04:34.489778+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.489778+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.956502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"237'34","reported_seq":989,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373667+0000","last_change":"2024-03-30T15:05:24.551301+0000","last_active":"2024-03-30T15:05:28.373667+0000","last_peered":"2024-03-30T15:05:28.373667+0000","last_clean":"2024-03-30T15:05:28.373667+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:05:28.373667+0000","last_undegraded":"2024-03-30T15:05:28.373667+0000","last_fullsized":"2024-03-30T15:05:28.373667+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:05:24.551158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.551158+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.999934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"496'23","reported_seq":976,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404370+0000","last_change":"2024-03-30T15:05:24.227205+0000","last_active":"2024-03-30T15:05:28.404370+0000","last_peered":"2024-03-30T15:05:28.404370+0000","last_clean":"2024-03-30T15:05:28.404370+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:05:28.404370+0000","last_undegraded":"2024-03-30T15:05:28.404370+0000","last_fullsized":"2024-03-30T15:05:28.404370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'23","last_scrub_stamp":"2024-03-30T15:05:24.227122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.227122+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.963448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":66,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"428'21","reported_seq":975,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404445+0000","last_change":"2024-03-30T15:05:25.269379+0000","last_active":"2024-03-30T15:05:28.404445+0000","last_peered":"2024-03-30T15:05:28.404445+0000","last_clean":"2024-03-30T15:05:28.404445+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:05:28.404445+0000","last_undegraded":"2024-03-30T15:05:28.404445+0000","last_fullsized":"2024-03-30T15:05:28.404445+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'21","last_scrub_stamp":"2024-03-30T15:05:25.269264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.269264+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.898384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"368'19","reported_seq":961,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380449+0000","last_change":"2024-03-30T15:04:06.381760+0000","last_active":"2024-03-30T15:05:28.380449+0000","last_peered":"2024-03-30T15:05:28.380449+0000","last_clean":"2024-03-30T15:05:28.380449+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:05:28.380449+0000","last_undegraded":"2024-03-30T15:05:28.380449+0000","last_fullsized":"2024-03-30T15:05:28.380449+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'19","last_scrub_stamp":"2024-03-30T15:02:28.859481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:28.859481+0000","objects_scrubbed":3,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"418'78","reported_seq":1025,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374807+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:05:28.374807+0000","last_peered":"2024-03-30T15:05:28.374807+0000","last_clean":"2024-03-30T15:05:28.374807+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:05:28.374807+0000","last_undegraded":"2024-03-30T15:05:28.374807+0000","last_fullsized":"2024-03-30T15:05:28.374807+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":67,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"460'60","reported_seq":1006,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374075+0000","last_change":"2024-03-30T15:04:06.474098+0000","last_active":"2024-03-30T15:05:28.374075+0000","last_peered":"2024-03-30T15:05:28.374075+0000","last_clean":"2024-03-30T15:05:28.374075+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:05:28.374075+0000","last_undegraded":"2024-03-30T15:05:28.374075+0000","last_fullsized":"2024-03-30T15:05:28.374075+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'53","last_scrub_stamp":"2024-03-30T15:02:03.544522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:03.544522+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"368'91","reported_seq":1030,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064224+0000","last_change":"2024-03-30T15:04:06.346253+0000","last_active":"2024-03-30T15:05:22.064224+0000","last_peered":"2024-03-30T15:05:22.064224+0000","last_clean":"2024-03-30T15:05:22.064224+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:05:22.064224+0000","last_undegraded":"2024-03-30T15:05:22.064224+0000","last_fullsized":"2024-03-30T15:05:22.064224+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'91","last_scrub_stamp":"2024-03-30T15:02:50.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.933002+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"496'37","reported_seq":987,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404633+0000","last_change":"2024-03-30T15:05:22.315909+0000","last_active":"2024-03-30T15:05:28.404633+0000","last_peered":"2024-03-30T15:05:28.404633+0000","last_clean":"2024-03-30T15:05:28.404633+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:05:28.404633+0000","last_undegraded":"2024-03-30T15:05:28.404633+0000","last_fullsized":"2024-03-30T15:05:28.404633+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'37","last_scrub_stamp":"2024-03-30T15:05:22.315812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.315812+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.650200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"499'111","reported_seq":1060,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.105949+0000","last_change":"2024-03-30T15:05:28.105979+0000","last_active":"2024-03-30T15:05:28.105949+0000","last_peered":"2024-03-30T15:05:28.105949+0000","last_clean":"2024-03-30T15:05:28.105949+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:05:28.105949+0000","last_undegraded":"2024-03-30T15:05:28.105949+0000","last_fullsized":"2024-03-30T15:05:28.105949+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'111","last_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_deep_scrub":"499'111","last_deep_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_clean_scrub_stamp":"2024-03-30T15:05:28.105889+0000","objects_scrubbed":7,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.276671+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"444'55","reported_seq":1007,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143379+0000","last_change":"2024-03-30T15:04:07.016310+0000","last_active":"2024-03-30T15:05:22.143379+0000","last_peered":"2024-03-30T15:05:22.143379+0000","last_clean":"2024-03-30T15:05:22.143379+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:05:22.143379+0000","last_undegraded":"2024-03-30T15:05:22.143379+0000","last_fullsized":"2024-03-30T15:05:22.143379+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":996,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032674+0000","last_change":"2024-03-30T15:04:55.161018+0000","last_active":"2024-03-30T15:05:22.032674+0000","last_peered":"2024-03-30T15:05:22.032674+0000","last_clean":"2024-03-30T15:05:22.032674+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:05:22.032674+0000","last_undegraded":"2024-03-30T15:05:22.032674+0000","last_fullsized":"2024-03-30T15:05:22.032674+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:04:55.160916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.160916+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.035072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"204'108","reported_seq":1054,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034119+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:05:22.034119+0000","last_peered":"2024-03-30T15:05:22.034119+0000","last_clean":"2024-03-30T15:05:22.034119+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:05:22.034119+0000","last_undegraded":"2024-03-30T15:05:22.034119+0000","last_fullsized":"2024-03-30T15:05:22.034119+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.731859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":1017,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374872+0000","last_change":"2024-03-30T15:04:35.492607+0000","last_active":"2024-03-30T15:05:28.374872+0000","last_peered":"2024-03-30T15:05:28.374872+0000","last_clean":"2024-03-30T15:05:28.374872+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:05:28.374872+0000","last_undegraded":"2024-03-30T15:05:28.374872+0000","last_fullsized":"2024-03-30T15:05:28.374872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'60","last_scrub_stamp":"2024-03-30T15:04:35.492526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.492526+0000","objects_scrubbed":6,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:39.249210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":1008,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405110+0000","last_change":"2024-03-30T15:04:06.473834+0000","last_active":"2024-03-30T15:05:28.405110+0000","last_peered":"2024-03-30T15:05:28.405110+0000","last_clean":"2024-03-30T15:05:28.405110+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:05:28.405110+0000","last_undegraded":"2024-03-30T15:05:28.405110+0000","last_fullsized":"2024-03-30T15:05:28.405110+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"498'97","reported_seq":1036,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379934+0000","last_change":"2024-03-30T15:05:21.308139+0000","last_active":"2024-03-30T15:05:28.379934+0000","last_peered":"2024-03-30T15:05:28.379934+0000","last_clean":"2024-03-30T15:05:28.379934+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:05:28.379934+0000","last_undegraded":"2024-03-30T15:05:28.379934+0000","last_fullsized":"2024-03-30T15:05:28.379934+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'97","last_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_deep_scrub":"498'97","last_deep_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.308003+0000","objects_scrubbed":8,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:25.060048+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":1037,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967853+0000","last_change":"2024-03-30T15:04:07.014089+0000","last_active":"2024-03-30T15:05:21.967853+0000","last_peered":"2024-03-30T15:05:21.967853+0000","last_clean":"2024-03-30T15:05:21.967853+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:05:21.967853+0000","last_undegraded":"2024-03-30T15:05:21.967853+0000","last_fullsized":"2024-03-30T15:05:21.967853+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:02:27.539670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:27.539670+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"248'39","reported_seq":999,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063755+0000","last_change":"2024-03-30T15:04:35.992306+0000","last_active":"2024-03-30T15:05:22.063755+0000","last_peered":"2024-03-30T15:05:22.063755+0000","last_clean":"2024-03-30T15:05:22.063755+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:05:22.063755+0000","last_undegraded":"2024-03-30T15:05:22.063755+0000","last_fullsized":"2024-03-30T15:05:22.063755+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:04:35.992167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.992167+0000","objects_scrubbed":3,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.226654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"452'42","reported_seq":1003,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063028+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:05:22.063028+0000","last_peered":"2024-03-30T15:05:22.063028+0000","last_clean":"2024-03-30T15:05:22.063028+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:05:22.063028+0000","last_undegraded":"2024-03-30T15:05:22.063028+0000","last_fullsized":"2024-03-30T15:05:22.063028+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"487'119","reported_seq":1072,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032944+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:05:22.032944+0000","last_peered":"2024-03-30T15:05:22.032944+0000","last_clean":"2024-03-30T15:05:22.032944+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:05:22.032944+0000","last_undegraded":"2024-03-30T15:05:22.032944+0000","last_fullsized":"2024-03-30T15:05:22.032944+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000104,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":37,"num_write_kb":3940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"368'23","reported_seq":979,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405718+0000","last_change":"2024-03-30T15:05:21.307427+0000","last_active":"2024-03-30T15:05:28.405718+0000","last_peered":"2024-03-30T15:05:28.405718+0000","last_clean":"2024-03-30T15:05:28.405718+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:05:28.405718+0000","last_undegraded":"2024-03-30T15:05:28.405718+0000","last_fullsized":"2024-03-30T15:05:28.405718+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:05:21.307305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.307305+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.716318+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"368'2","reported_seq":968,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064944+0000","last_change":"2024-03-30T15:05:13.117385+0000","last_active":"2024-03-30T15:05:22.064944+0000","last_peered":"2024-03-30T15:05:22.064944+0000","last_clean":"2024-03-30T15:05:22.064944+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:05:22.064944+0000","last_undegraded":"2024-03-30T15:05:22.064944+0000","last_fullsized":"2024-03-30T15:05:22.064944+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:05:13.117303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.117303+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.803622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":1006,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374098+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:05:28.374098+0000","last_peered":"2024-03-30T15:05:28.374098+0000","last_clean":"2024-03-30T15:05:28.374098+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:05:28.374098+0000","last_undegraded":"2024-03-30T15:05:28.374098+0000","last_fullsized":"2024-03-30T15:05:28.374098+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"368'103","reported_seq":1048,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145115+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:05:22.145115+0000","last_peered":"2024-03-30T15:05:22.145115+0000","last_clean":"2024-03-30T15:05:22.145115+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:05:22.145115+0000","last_undegraded":"2024-03-30T15:05:22.145115+0000","last_fullsized":"2024-03-30T15:05:22.145115+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144065+0000","last_change":"2024-03-30T15:04:42.056061+0000","last_active":"2024-03-30T15:05:22.144065+0000","last_peered":"2024-03-30T15:05:22.144065+0000","last_clean":"2024-03-30T15:05:22.144065+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:05:22.144065+0000","last_undegraded":"2024-03-30T15:05:22.144065+0000","last_fullsized":"2024-03-30T15:05:22.144065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.055955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.055955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.618053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":803,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033687+0000","last_change":"2024-03-30T15:04:34.266981+0000","last_active":"2024-03-30T15:05:22.033687+0000","last_peered":"2024-03-30T15:05:22.033687+0000","last_clean":"2024-03-30T15:05:22.033687+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:05:22.033687+0000","last_undegraded":"2024-03-30T15:05:22.033687+0000","last_fullsized":"2024-03-30T15:05:22.033687+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:34.266865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.266865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:49.944697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380359+0000","last_change":"2024-03-30T15:04:42.277788+0000","last_active":"2024-03-30T15:05:28.380359+0000","last_peered":"2024-03-30T15:05:28.380359+0000","last_clean":"2024-03-30T15:05:28.380359+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:05:28.380359+0000","last_undegraded":"2024-03-30T15:05:28.380359+0000","last_fullsized":"2024-03-30T15:05:28.380359+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.712874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968716+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:05:21.968716+0000","last_peered":"2024-03-30T15:05:21.968716+0000","last_clean":"2024-03-30T15:05:21.968716+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:05:21.968716+0000","last_undegraded":"2024-03-30T15:05:21.968716+0000","last_fullsized":"2024-03-30T15:05:21.968716+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:22.852443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":800,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373713+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:05:28.373713+0000","last_peered":"2024-03-30T15:05:28.373713+0000","last_clean":"2024-03-30T15:05:28.373713+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:05:28.373713+0000","last_undegraded":"2024-03-30T15:05:28.373713+0000","last_fullsized":"2024-03-30T15:05:28.373713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.284980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:27.005951+0000","last_change":"2024-03-30T15:05:27.005973+0000","last_active":"2024-03-30T15:05:27.005951+0000","last_peered":"2024-03-30T15:05:27.005951+0000","last_clean":"2024-03-30T15:05:27.005951+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:05:27.005951+0000","last_undegraded":"2024-03-30T15:05:27.005951+0000","last_fullsized":"2024-03-30T15:05:27.005951+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.005887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.577288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379897+0000","last_change":"2024-03-30T15:05:20.323134+0000","last_active":"2024-03-30T15:05:28.379897+0000","last_peered":"2024-03-30T15:05:28.379897+0000","last_clean":"2024-03-30T15:05:28.379897+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:05:28.379897+0000","last_undegraded":"2024-03-30T15:05:28.379897+0000","last_fullsized":"2024-03-30T15:05:28.379897+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.323015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.323015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.231176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143787+0000","last_change":"2024-03-30T15:04:38.084291+0000","last_active":"2024-03-30T15:05:22.143787+0000","last_peered":"2024-03-30T15:05:22.143787+0000","last_clean":"2024-03-30T15:05:22.143787+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:05:22.143787+0000","last_undegraded":"2024-03-30T15:05:22.143787+0000","last_fullsized":"2024-03-30T15:05:22.143787+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:38.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:40.487265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405463+0000","last_change":"2024-03-30T15:05:19.276266+0000","last_active":"2024-03-30T15:05:28.405463+0000","last_peered":"2024-03-30T15:05:28.405463+0000","last_clean":"2024-03-30T15:05:28.405463+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:05:28.405463+0000","last_undegraded":"2024-03-30T15:05:28.405463+0000","last_fullsized":"2024-03-30T15:05:28.405463+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.276185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.276185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.789727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968415+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:05:21.968415+0000","last_peered":"2024-03-30T15:05:21.968415+0000","last_clean":"2024-03-30T15:05:21.968415+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:05:21.968415+0000","last_undegraded":"2024-03-30T15:05:21.968415+0000","last_fullsized":"2024-03-30T15:05:21.968415+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380352+0000","last_change":"2024-03-30T15:05:17.336662+0000","last_active":"2024-03-30T15:05:28.380352+0000","last_peered":"2024-03-30T15:05:28.380352+0000","last_clean":"2024-03-30T15:05:28.380352+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:05:28.380352+0000","last_undegraded":"2024-03-30T15:05:28.380352+0000","last_fullsized":"2024-03-30T15:05:28.380352+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.336577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.336577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.490620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064202+0000","last_change":"2024-03-30T15:05:19.069973+0000","last_active":"2024-03-30T15:05:22.064202+0000","last_peered":"2024-03-30T15:05:22.064202+0000","last_clean":"2024-03-30T15:05:22.064202+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:05:22.064202+0000","last_undegraded":"2024-03-30T15:05:22.064202+0000","last_fullsized":"2024-03-30T15:05:22.064202+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.069896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.069896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:35.468169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379160+0000","last_change":"2024-03-30T15:05:19.327818+0000","last_active":"2024-03-30T15:05:28.379160+0000","last_peered":"2024-03-30T15:05:28.379160+0000","last_clean":"2024-03-30T15:05:28.379160+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:05:28.379160+0000","last_undegraded":"2024-03-30T15:05:28.379160+0000","last_fullsized":"2024-03-30T15:05:28.379160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.327720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.327720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.657218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967635+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:05:21.967635+0000","last_peered":"2024-03-30T15:05:21.967635+0000","last_clean":"2024-03-30T15:05:21.967635+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:05:21.967635+0000","last_undegraded":"2024-03-30T15:05:21.967635+0000","last_fullsized":"2024-03-30T15:05:21.967635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063869+0000","last_change":"2024-03-30T15:04:37.008749+0000","last_active":"2024-03-30T15:05:22.063869+0000","last_peered":"2024-03-30T15:05:22.063869+0000","last_clean":"2024-03-30T15:05:22.063869+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:05:22.063869+0000","last_undegraded":"2024-03-30T15:05:22.063869+0000","last_fullsized":"2024-03-30T15:05:22.063869+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.008666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.008666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.100396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063713+0000","last_change":"2024-03-30T15:04:54.955662+0000","last_active":"2024-03-30T15:05:22.063713+0000","last_peered":"2024-03-30T15:05:22.063713+0000","last_clean":"2024-03-30T15:05:22.063713+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:05:22.063713+0000","last_undegraded":"2024-03-30T15:05:22.063713+0000","last_fullsized":"2024-03-30T15:05:22.063713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.955541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.955541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.235351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380040+0000","last_change":"2024-03-30T15:04:55.271526+0000","last_active":"2024-03-30T15:05:28.380040+0000","last_peered":"2024-03-30T15:05:28.380040+0000","last_clean":"2024-03-30T15:05:28.380040+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:05:28.380040+0000","last_undegraded":"2024-03-30T15:05:28.380040+0000","last_fullsized":"2024-03-30T15:05:28.380040+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.271452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.271452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.970909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405484+0000","last_change":"2024-03-30T15:04:45.608747+0000","last_active":"2024-03-30T15:05:28.405484+0000","last_peered":"2024-03-30T15:05:28.405484+0000","last_clean":"2024-03-30T15:05:28.405484+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:05:28.405484+0000","last_undegraded":"2024-03-30T15:05:28.405484+0000","last_fullsized":"2024-03-30T15:05:28.405484+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.608652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.608652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:05.611823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144850+0000","last_change":"2024-03-30T15:04:36.136344+0000","last_active":"2024-03-30T15:05:22.144850+0000","last_peered":"2024-03-30T15:05:22.144850+0000","last_clean":"2024-03-30T15:05:22.144850+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:05:22.144850+0000","last_undegraded":"2024-03-30T15:05:22.144850+0000","last_fullsized":"2024-03-30T15:05:22.144850+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.136228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.136228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:36.176305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380319+0000","last_change":"2024-03-30T15:04:50.298381+0000","last_active":"2024-03-30T15:05:28.380319+0000","last_peered":"2024-03-30T15:05:28.380319+0000","last_clean":"2024-03-30T15:05:28.380319+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:05:28.380319+0000","last_undegraded":"2024-03-30T15:05:28.380319+0000","last_fullsized":"2024-03-30T15:05:28.380319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.298282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.756120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967893+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:05:21.967893+0000","last_peered":"2024-03-30T15:05:21.967893+0000","last_clean":"2024-03-30T15:05:21.967893+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:05:21.967893+0000","last_undegraded":"2024-03-30T15:05:21.967893+0000","last_fullsized":"2024-03-30T15:05:21.967893+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379122+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:05:28.379122+0000","last_peered":"2024-03-30T15:05:28.379122+0000","last_clean":"2024-03-30T15:05:28.379122+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:05:28.379122+0000","last_undegraded":"2024-03-30T15:05:28.379122+0000","last_fullsized":"2024-03-30T15:05:28.379122+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":803,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033260+0000","last_change":"2024-03-30T15:05:19.111999+0000","last_active":"2024-03-30T15:05:22.033260+0000","last_peered":"2024-03-30T15:05:22.033260+0000","last_clean":"2024-03-30T15:05:22.033260+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:05:22.033260+0000","last_undegraded":"2024-03-30T15:05:22.033260+0000","last_fullsized":"2024-03-30T15:05:22.033260+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.111909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.111909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.337658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063192+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:05:22.063192+0000","last_peered":"2024-03-30T15:05:22.063192+0000","last_clean":"2024-03-30T15:05:22.063192+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:05:22.063192+0000","last_undegraded":"2024-03-30T15:05:22.063192+0000","last_fullsized":"2024-03-30T15:05:22.063192+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034202+0000","last_change":"2024-03-30T15:02:37.380106+0000","last_active":"2024-03-30T15:05:22.034202+0000","last_peered":"2024-03-30T15:05:22.034202+0000","last_clean":"2024-03-30T15:05:22.034202+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:05:22.034202+0000","last_undegraded":"2024-03-30T15:05:22.034202+0000","last_fullsized":"2024-03-30T15:05:22.034202+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.379986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.379986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968368+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:05:21.968368+0000","last_peered":"2024-03-30T15:05:21.968368+0000","last_clean":"2024-03-30T15:05:21.968368+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:05:21.968368+0000","last_undegraded":"2024-03-30T15:05:21.968368+0000","last_fullsized":"2024-03-30T15:05:21.968368+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145151+0000","last_change":"2024-03-30T15:04:55.123255+0000","last_active":"2024-03-30T15:05:22.145151+0000","last_peered":"2024-03-30T15:05:22.145151+0000","last_clean":"2024-03-30T15:05:22.145151+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:05:22.145151+0000","last_undegraded":"2024-03-30T15:05:22.145151+0000","last_fullsized":"2024-03-30T15:05:22.145151+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.123176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.123176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.501538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144133+0000","last_change":"2024-03-30T15:04:45.130399+0000","last_active":"2024-03-30T15:05:22.144133+0000","last_peered":"2024-03-30T15:05:22.144133+0000","last_clean":"2024-03-30T15:05:22.144133+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:05:22.144133+0000","last_undegraded":"2024-03-30T15:05:22.144133+0000","last_fullsized":"2024-03-30T15:05:22.144133+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.130322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.130322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.109084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":803,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032672+0000","last_change":"2024-03-30T15:04:44.264561+0000","last_active":"2024-03-30T15:05:22.032672+0000","last_peered":"2024-03-30T15:05:22.032672+0000","last_clean":"2024-03-30T15:05:22.032672+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:05:22.032672+0000","last_undegraded":"2024-03-30T15:05:22.032672+0000","last_fullsized":"2024-03-30T15:05:22.032672+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.264430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.264430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.388512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143369+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:05:22.143369+0000","last_peered":"2024-03-30T15:05:22.143369+0000","last_clean":"2024-03-30T15:05:22.143369+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:05:22.143369+0000","last_undegraded":"2024-03-30T15:05:22.143369+0000","last_fullsized":"2024-03-30T15:05:22.143369+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:24.836391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405004+0000","last_change":"2024-03-30T15:04:54.465626+0000","last_active":"2024-03-30T15:05:28.405004+0000","last_peered":"2024-03-30T15:05:28.405004+0000","last_clean":"2024-03-30T15:05:28.405004+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:05:28.405004+0000","last_undegraded":"2024-03-30T15:05:28.405004+0000","last_fullsized":"2024-03-30T15:05:28.405004+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.465543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.465543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:57.501311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":796,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379459+0000","last_change":"2024-03-30T15:04:53.311876+0000","last_active":"2024-03-30T15:05:28.379459+0000","last_peered":"2024-03-30T15:05:28.379459+0000","last_clean":"2024-03-30T15:05:28.379459+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:05:28.379459+0000","last_undegraded":"2024-03-30T15:05:28.379459+0000","last_fullsized":"2024-03-30T15:05:28.379459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.311797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.311797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.989454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033735+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:05:22.033735+0000","last_peered":"2024-03-30T15:05:22.033735+0000","last_clean":"2024-03-30T15:05:22.033735+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:05:22.033735+0000","last_undegraded":"2024-03-30T15:05:22.033735+0000","last_fullsized":"2024-03-30T15:05:22.033735+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":922,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405633+0000","last_change":"2024-03-30T15:02:49.428683+0000","last_active":"2024-03-30T15:05:28.405633+0000","last_peered":"2024-03-30T15:05:28.405633+0000","last_clean":"2024-03-30T15:05:28.405633+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:05:28.405633+0000","last_undegraded":"2024-03-30T15:05:28.405633+0000","last_fullsized":"2024-03-30T15:05:28.405633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:02:49.428597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.428597+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":802,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374053+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:05:28.374053+0000","last_peered":"2024-03-30T15:05:28.374053+0000","last_clean":"2024-03-30T15:05:28.374053+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:05:28.374053+0000","last_undegraded":"2024-03-30T15:05:28.374053+0000","last_fullsized":"2024-03-30T15:05:28.374053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":47,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629617+0000","last_change":"2024-03-30T15:05:06.360304+0000","last_active":"2024-03-30T15:05:22.629617+0000","last_peered":"2024-03-30T15:05:22.629617+0000","last_clean":"2024-03-30T15:05:22.629617+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:05:22.629617+0000","last_undegraded":"2024-03-30T15:05:22.629617+0000","last_fullsized":"2024-03-30T15:05:22.629617+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":884,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.406127+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:05:28.406127+0000","last_peered":"2024-03-30T15:05:28.406127+0000","last_clean":"2024-03-30T15:05:28.406127+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:05:28.406127+0000","last_undegraded":"2024-03-30T15:05:28.406127+0000","last_fullsized":"2024-03-30T15:05:28.406127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034326+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:05:22.034326+0000","last_peered":"2024-03-30T15:05:22.034326+0000","last_clean":"2024-03-30T15:05:22.034326+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:05:22.034326+0000","last_undegraded":"2024-03-30T15:05:22.034326+0000","last_fullsized":"2024-03-30T15:05:22.034326+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:26.821252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":949,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405677+0000","last_change":"2024-03-30T15:04:34.512958+0000","last_active":"2024-03-30T15:05:28.405677+0000","last_peered":"2024-03-30T15:05:28.405677+0000","last_clean":"2024-03-30T15:05:28.405677+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:05:28.405677+0000","last_undegraded":"2024-03-30T15:05:28.405677+0000","last_fullsized":"2024-03-30T15:05:28.405677+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:04:34.512838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.512838+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.123121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":911,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033214+0000","last_change":"2024-03-30T15:04:48.217127+0000","last_active":"2024-03-30T15:05:22.033214+0000","last_peered":"2024-03-30T15:05:22.033214+0000","last_clean":"2024-03-30T15:05:22.033214+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:05:22.033214+0000","last_undegraded":"2024-03-30T15:05:22.033214+0000","last_fullsized":"2024-03-30T15:05:22.033214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.217046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.217046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.053751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967243+0000","last_change":"2024-03-30T15:04:48.997109+0000","last_active":"2024-03-30T15:05:21.967243+0000","last_peered":"2024-03-30T15:05:21.967243+0000","last_clean":"2024-03-30T15:05:21.967243+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:05:21.967243+0000","last_undegraded":"2024-03-30T15:05:21.967243+0000","last_fullsized":"2024-03-30T15:05:21.967243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.997021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.997021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.181602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":900,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064997+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:05:22.064997+0000","last_peered":"2024-03-30T15:05:22.064997+0000","last_clean":"2024-03-30T15:05:22.064997+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:05:22.064997+0000","last_undegraded":"2024-03-30T15:05:22.064997+0000","last_fullsized":"2024-03-30T15:05:22.064997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:33.116494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":801,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968896+0000","last_change":"2024-03-30T15:02:37.491651+0000","last_active":"2024-03-30T15:05:21.968896+0000","last_peered":"2024-03-30T15:05:21.968896+0000","last_clean":"2024-03-30T15:05:21.968896+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:05:21.968896+0000","last_undegraded":"2024-03-30T15:05:21.968896+0000","last_fullsized":"2024-03-30T15:05:21.968896+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.491566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.491566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":936,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.406028+0000","last_change":"2024-03-30T15:05:17.276827+0000","last_active":"2024-03-30T15:05:28.406028+0000","last_peered":"2024-03-30T15:05:28.406028+0000","last_clean":"2024-03-30T15:05:28.406028+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:05:28.406028+0000","last_undegraded":"2024-03-30T15:05:28.406028+0000","last_fullsized":"2024-03-30T15:05:28.406028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:17.276735+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.276735+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.270995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":904,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404798+0000","last_change":"2024-03-30T15:05:14.260362+0000","last_active":"2024-03-30T15:05:28.404798+0000","last_peered":"2024-03-30T15:05:28.404798+0000","last_clean":"2024-03-30T15:05:28.404798+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:05:28.404798+0000","last_undegraded":"2024-03-30T15:05:28.404798+0000","last_fullsized":"2024-03-30T15:05:28.404798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.260283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.260283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:14.479226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":887,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380987+0000","last_change":"2024-03-30T15:02:37.010981+0000","last_active":"2024-03-30T15:05:28.380987+0000","last_peered":"2024-03-30T15:05:28.380987+0000","last_clean":"2024-03-30T15:05:28.380987+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:05:28.380987+0000","last_undegraded":"2024-03-30T15:05:28.380987+0000","last_fullsized":"2024-03-30T15:05:28.380987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:37.010838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.010838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":804,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033835+0000","last_change":"2024-03-30T15:02:42.407612+0000","last_active":"2024-03-30T15:05:22.033835+0000","last_peered":"2024-03-30T15:05:22.033835+0000","last_clean":"2024-03-30T15:05:22.033835+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:05:22.033835+0000","last_undegraded":"2024-03-30T15:05:22.033835+0000","last_fullsized":"2024-03-30T15:05:22.033835+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.407489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.407489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":801,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968886+0000","last_change":"2024-03-30T15:02:31.491323+0000","last_active":"2024-03-30T15:05:21.968886+0000","last_peered":"2024-03-30T15:05:21.968886+0000","last_clean":"2024-03-30T15:05:21.968886+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:05:21.968886+0000","last_undegraded":"2024-03-30T15:05:21.968886+0000","last_fullsized":"2024-03-30T15:05:21.968886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.491236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.491236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":949,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:26.068568+0000","last_change":"2024-03-30T15:05:26.068603+0000","last_active":"2024-03-30T15:05:26.068568+0000","last_peered":"2024-03-30T15:05:26.068568+0000","last_clean":"2024-03-30T15:05:26.068568+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:05:26.068568+0000","last_undegraded":"2024-03-30T15:05:26.068568+0000","last_fullsized":"2024-03-30T15:05:26.068568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.068493+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.425413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":805,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967753+0000","last_change":"2024-03-30T15:04:50.960049+0000","last_active":"2024-03-30T15:05:21.967753+0000","last_peered":"2024-03-30T15:05:21.967753+0000","last_clean":"2024-03-30T15:05:21.967753+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:05:21.967753+0000","last_undegraded":"2024-03-30T15:05:21.967753+0000","last_fullsized":"2024-03-30T15:05:21.967753+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.959968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.959968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.339138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1027,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.406059+0000","last_change":"2024-03-30T15:03:22.567558+0000","last_active":"2024-03-30T15:05:28.406059+0000","last_peered":"2024-03-30T15:05:28.406059+0000","last_clean":"2024-03-30T15:05:28.406059+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:05:28.406059+0000","last_undegraded":"2024-03-30T15:05:28.406059+0000","last_fullsized":"2024-03-30T15:05:28.406059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:22.567465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.567465+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":883,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380105+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:05:28.380105+0000","last_peered":"2024-03-30T15:05:28.380105+0000","last_clean":"2024-03-30T15:05:28.380105+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:05:28.380105+0000","last_undegraded":"2024-03-30T15:05:28.380105+0000","last_fullsized":"2024-03-30T15:05:28.380105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063252+0000","last_change":"2024-03-30T15:05:15.068570+0000","last_active":"2024-03-30T15:05:22.063252+0000","last_peered":"2024-03-30T15:05:22.063252+0000","last_clean":"2024-03-30T15:05:22.063252+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:05:22.063252+0000","last_undegraded":"2024-03-30T15:05:22.063252+0000","last_fullsized":"2024-03-30T15:05:22.063252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.068496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.068496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.114011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":897,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404718+0000","last_change":"2024-03-30T15:05:18.317376+0000","last_active":"2024-03-30T15:05:28.404718+0000","last_peered":"2024-03-30T15:05:28.404718+0000","last_clean":"2024-03-30T15:05:28.404718+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:05:28.404718+0000","last_undegraded":"2024-03-30T15:05:28.404718+0000","last_fullsized":"2024-03-30T15:05:28.404718+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.317302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.949240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":912,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145898+0000","last_change":"2024-03-30T15:05:22.145932+0000","last_active":"2024-03-30T15:05:22.145898+0000","last_peered":"2024-03-30T15:05:22.145898+0000","last_clean":"2024-03-30T15:05:22.145898+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:05:22.145898+0000","last_undegraded":"2024-03-30T15:05:22.145898+0000","last_fullsized":"2024-03-30T15:05:22.145898+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:05:22.145836+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.145836+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.469421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":870,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144914+0000","last_change":"2024-03-30T15:02:44.977008+0000","last_active":"2024-03-30T15:05:22.144914+0000","last_peered":"2024-03-30T15:05:22.144914+0000","last_clean":"2024-03-30T15:05:22.144914+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:05:22.144914+0000","last_undegraded":"2024-03-30T15:05:22.144914+0000","last_fullsized":"2024-03-30T15:05:22.144914+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:02:44.976870+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.976870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404845+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:05:28.404845+0000","last_peered":"2024-03-30T15:05:28.404845+0000","last_clean":"2024-03-30T15:05:28.404845+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:05:28.404845+0000","last_undegraded":"2024-03-30T15:05:28.404845+0000","last_fullsized":"2024-03-30T15:05:28.404845+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":813,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063248+0000","last_change":"2024-03-30T15:04:48.979522+0000","last_active":"2024-03-30T15:05:22.063248+0000","last_peered":"2024-03-30T15:05:22.063248+0000","last_clean":"2024-03-30T15:05:22.063248+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:05:22.063248+0000","last_undegraded":"2024-03-30T15:05:22.063248+0000","last_fullsized":"2024-03-30T15:05:22.063248+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.979438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.432525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":912,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629428+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:05:22.629428+0000","last_peered":"2024-03-30T15:05:22.629428+0000","last_clean":"2024-03-30T15:05:22.629428+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:05:22.629428+0000","last_undegraded":"2024-03-30T15:05:22.629428+0000","last_fullsized":"2024-03-30T15:05:22.629428+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":943,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404898+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:05:28.404898+0000","last_peered":"2024-03-30T15:05:28.404898+0000","last_clean":"2024-03-30T15:05:28.404898+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:05:28.404898+0000","last_undegraded":"2024-03-30T15:05:28.404898+0000","last_fullsized":"2024-03-30T15:05:28.404898+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":903,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375323+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:05:28.375323+0000","last_peered":"2024-03-30T15:05:28.375323+0000","last_clean":"2024-03-30T15:05:28.375323+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:05:28.375323+0000","last_undegraded":"2024-03-30T15:05:28.375323+0000","last_fullsized":"2024-03-30T15:05:28.375323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":892,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143754+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:05:22.143754+0000","last_peered":"2024-03-30T15:05:22.143754+0000","last_clean":"2024-03-30T15:05:22.143754+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:05:22.143754+0000","last_undegraded":"2024-03-30T15:05:22.143754+0000","last_fullsized":"2024-03-30T15:05:22.143754+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":813,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063287+0000","last_change":"2024-03-30T15:04:47.018205+0000","last_active":"2024-03-30T15:05:22.063287+0000","last_peered":"2024-03-30T15:05:22.063287+0000","last_clean":"2024-03-30T15:05:22.063287+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:05:22.063287+0000","last_undegraded":"2024-03-30T15:05:22.063287+0000","last_fullsized":"2024-03-30T15:05:22.063287+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.018083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.520606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1048,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374002+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:05:28.374002+0000","last_peered":"2024-03-30T15:05:28.374002+0000","last_clean":"2024-03-30T15:05:28.374002+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:05:28.374002+0000","last_undegraded":"2024-03-30T15:05:28.374002+0000","last_fullsized":"2024-03-30T15:05:28.374002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:36.047732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":935,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968322+0000","last_change":"2024-03-30T15:02:07.498993+0000","last_active":"2024-03-30T15:05:21.968322+0000","last_peered":"2024-03-30T15:05:21.968322+0000","last_clean":"2024-03-30T15:05:21.968322+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:05:21.968322+0000","last_undegraded":"2024-03-30T15:05:21.968322+0000","last_fullsized":"2024-03-30T15:05:21.968322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:02:07.498902+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:02:07.498902+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":904,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380211+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:05:28.380211+0000","last_peered":"2024-03-30T15:05:28.380211+0000","last_clean":"2024-03-30T15:05:28.380211+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:05:28.380211+0000","last_undegraded":"2024-03-30T15:05:28.380211+0000","last_fullsized":"2024-03-30T15:05:28.380211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":889,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.372868+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:05:28.372868+0000","last_peered":"2024-03-30T15:05:28.372868+0000","last_clean":"2024-03-30T15:05:28.372868+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:05:28.372868+0000","last_undegraded":"2024-03-30T15:05:28.372868+0000","last_fullsized":"2024-03-30T15:05:28.372868+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":806,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063950+0000","last_change":"2024-03-30T15:04:52.952256+0000","last_active":"2024-03-30T15:05:22.063950+0000","last_peered":"2024-03-30T15:05:22.063950+0000","last_clean":"2024-03-30T15:05:22.063950+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:05:22.063950+0000","last_undegraded":"2024-03-30T15:05:22.063950+0000","last_fullsized":"2024-03-30T15:05:22.063950+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.952177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.313787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032307+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:05:22.032307+0000","last_peered":"2024-03-30T15:05:22.032307+0000","last_clean":"2024-03-30T15:05:22.032307+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:05:22.032307+0000","last_undegraded":"2024-03-30T15:05:22.032307+0000","last_fullsized":"2024-03-30T15:05:22.032307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":956,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374745+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:05:28.374745+0000","last_peered":"2024-03-30T15:05:28.374745+0000","last_clean":"2024-03-30T15:05:28.374745+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:05:28.374745+0000","last_undegraded":"2024-03-30T15:05:28.374745+0000","last_fullsized":"2024-03-30T15:05:28.374745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.338727+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":912,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968611+0000","last_change":"2024-03-30T15:04:51.927488+0000","last_active":"2024-03-30T15:05:21.968611+0000","last_peered":"2024-03-30T15:05:21.968611+0000","last_clean":"2024-03-30T15:05:21.968611+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:05:21.968611+0000","last_undegraded":"2024-03-30T15:05:21.968611+0000","last_fullsized":"2024-03-30T15:05:21.968611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.927412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.927412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:53.838858+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":900,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:23.007594+0000","last_change":"2024-03-30T15:05:23.007617+0000","last_active":"2024-03-30T15:05:23.007594+0000","last_peered":"2024-03-30T15:05:23.007594+0000","last_clean":"2024-03-30T15:05:23.007594+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:05:23.007594+0000","last_undegraded":"2024-03-30T15:05:23.007594+0000","last_fullsized":"2024-03-30T15:05:23.007594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.007527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.808237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375240+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:05:28.375240+0000","last_peered":"2024-03-30T15:05:28.375240+0000","last_clean":"2024-03-30T15:05:28.375240+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:05:28.375240+0000","last_undegraded":"2024-03-30T15:05:28.375240+0000","last_fullsized":"2024-03-30T15:05:28.375240+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.839469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":896,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380990+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:05:28.380990+0000","last_peered":"2024-03-30T15:05:28.380990+0000","last_clean":"2024-03-30T15:05:28.380990+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:05:28.380990+0000","last_undegraded":"2024-03-30T15:05:28.380990+0000","last_fullsized":"2024-03-30T15:05:28.380990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":941,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375225+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:05:28.375225+0000","last_peered":"2024-03-30T15:05:28.375225+0000","last_clean":"2024-03-30T15:05:28.375225+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:05:28.375225+0000","last_undegraded":"2024-03-30T15:05:28.375225+0000","last_fullsized":"2024-03-30T15:05:28.375225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1560,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:26.836508+0000","last_change":"2024-03-30T15:04:50.115333+0000","last_active":"2024-03-30T15:05:26.836508+0000","last_peered":"2024-03-30T15:05:26.836508+0000","last_clean":"2024-03-30T15:05:26.836508+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:05:26.836508+0000","last_undegraded":"2024-03-30T15:05:26.836508+0000","last_fullsized":"2024-03-30T15:05:26.836508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.115166+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.947111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":643,"num_read_kb":638,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":901,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373404+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:05:28.373404+0000","last_peered":"2024-03-30T15:05:28.373404+0000","last_clean":"2024-03-30T15:05:28.373404+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:05:28.373404+0000","last_undegraded":"2024-03-30T15:05:28.373404+0000","last_fullsized":"2024-03-30T15:05:28.373404+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":892,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968488+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:05:21.968488+0000","last_peered":"2024-03-30T15:05:21.968488+0000","last_clean":"2024-03-30T15:05:21.968488+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:05:21.968488+0000","last_undegraded":"2024-03-30T15:05:21.968488+0000","last_fullsized":"2024-03-30T15:05:21.968488+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":800,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144973+0000","last_change":"2024-03-30T15:04:37.123232+0000","last_active":"2024-03-30T15:05:22.144973+0000","last_peered":"2024-03-30T15:05:22.144973+0000","last_clean":"2024-03-30T15:05:22.144973+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:05:22.144973+0000","last_undegraded":"2024-03-30T15:05:22.144973+0000","last_fullsized":"2024-03-30T15:05:22.144973+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:43.957564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":896,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144333+0000","last_change":"2024-03-30T15:05:15.118857+0000","last_active":"2024-03-30T15:05:22.144333+0000","last_peered":"2024-03-30T15:05:22.144333+0000","last_clean":"2024-03-30T15:05:22.144333+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:05:22.144333+0000","last_undegraded":"2024-03-30T15:05:22.144333+0000","last_fullsized":"2024-03-30T15:05:22.144333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.118778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.118778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.638794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1078,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629790+0000","last_change":"2024-03-30T15:02:38.384839+0000","last_active":"2024-03-30T15:05:22.629790+0000","last_peered":"2024-03-30T15:05:22.629790+0000","last_clean":"2024-03-30T15:05:22.629790+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:05:22.629790+0000","last_undegraded":"2024-03-30T15:05:22.629790+0000","last_fullsized":"2024-03-30T15:05:22.629790+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'16","last_scrub_stamp":"2024-03-30T15:02:38.384693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.384693+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":798,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379537+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:05:28.379537+0000","last_peered":"2024-03-30T15:05:28.379537+0000","last_clean":"2024-03-30T15:05:28.379537+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:05:28.379537+0000","last_undegraded":"2024-03-30T15:05:28.379537+0000","last_fullsized":"2024-03-30T15:05:28.379537+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1165,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:27.137532+0000","last_change":"2024-03-30T15:05:27.137562+0000","last_active":"2024-03-30T15:05:27.137532+0000","last_peered":"2024-03-30T15:05:27.137532+0000","last_clean":"2024-03-30T15:05:27.137532+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:05:27.137532+0000","last_undegraded":"2024-03-30T15:05:27.137532+0000","last_fullsized":"2024-03-30T15:05:27.137532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:05:27.137472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.137472+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.903904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":883,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380795+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:05:28.380795+0000","last_peered":"2024-03-30T15:05:28.380795+0000","last_clean":"2024-03-30T15:05:28.380795+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:05:28.380795+0000","last_undegraded":"2024-03-30T15:05:28.380795+0000","last_fullsized":"2024-03-30T15:05:28.380795+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034411+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:05:22.034411+0000","last_peered":"2024-03-30T15:05:22.034411+0000","last_clean":"2024-03-30T15:05:22.034411+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:05:22.034411+0000","last_undegraded":"2024-03-30T15:05:22.034411+0000","last_fullsized":"2024-03-30T15:05:22.034411+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":905,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064830+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:05:22.064830+0000","last_peered":"2024-03-30T15:05:22.064830+0000","last_clean":"2024-03-30T15:05:22.064830+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:05:22.064830+0000","last_undegraded":"2024-03-30T15:05:22.064830+0000","last_fullsized":"2024-03-30T15:05:22.064830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.873816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":946,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968840+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:05:21.968840+0000","last_peered":"2024-03-30T15:05:21.968840+0000","last_clean":"2024-03-30T15:05:21.968840+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:05:21.968840+0000","last_undegraded":"2024-03-30T15:05:21.968840+0000","last_fullsized":"2024-03-30T15:05:21.968840+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":922,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063389+0000","last_change":"2024-03-30T15:04:55.918561+0000","last_active":"2024-03-30T15:05:22.063389+0000","last_peered":"2024-03-30T15:05:22.063389+0000","last_clean":"2024-03-30T15:05:22.063389+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:05:22.063389+0000","last_undegraded":"2024-03-30T15:05:22.063389+0000","last_fullsized":"2024-03-30T15:05:22.063389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.918411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.374429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":912,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967580+0000","last_change":"2024-03-30T15:04:53.983173+0000","last_active":"2024-03-30T15:05:21.967580+0000","last_peered":"2024-03-30T15:05:21.967580+0000","last_clean":"2024-03-30T15:05:21.967580+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:05:21.967580+0000","last_undegraded":"2024-03-30T15:05:21.967580+0000","last_fullsized":"2024-03-30T15:05:21.967580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.983055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.611127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":892,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145211+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:05:22.145211+0000","last_peered":"2024-03-30T15:05:22.145211+0000","last_clean":"2024-03-30T15:05:22.145211+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:05:22.145211+0000","last_undegraded":"2024-03-30T15:05:22.145211+0000","last_fullsized":"2024-03-30T15:05:22.145211+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374294+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:05:28.374294+0000","last_peered":"2024-03-30T15:05:28.374294+0000","last_clean":"2024-03-30T15:05:28.374294+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:05:28.374294+0000","last_undegraded":"2024-03-30T15:05:28.374294+0000","last_fullsized":"2024-03-30T15:05:28.374294+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.687947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":901,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374643+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:05:28.374643+0000","last_peered":"2024-03-30T15:05:28.374643+0000","last_clean":"2024-03-30T15:05:28.374643+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:05:28.374643+0000","last_undegraded":"2024-03-30T15:05:28.374643+0000","last_fullsized":"2024-03-30T15:05:28.374643+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:03.100823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1044,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379679+0000","last_change":"2024-03-30T15:02:49.026258+0000","last_active":"2024-03-30T15:05:28.379679+0000","last_peered":"2024-03-30T15:05:28.379679+0000","last_clean":"2024-03-30T15:05:28.379679+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:05:28.379679+0000","last_undegraded":"2024-03-30T15:05:28.379679+0000","last_fullsized":"2024-03-30T15:05:28.379679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'29","last_scrub_stamp":"2024-03-30T15:02:49.026125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:49.026125+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":905,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405193+0000","last_change":"2024-03-30T15:04:53.505197+0000","last_active":"2024-03-30T15:05:28.405193+0000","last_peered":"2024-03-30T15:05:28.405193+0000","last_clean":"2024-03-30T15:05:28.405193+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:05:28.405193+0000","last_undegraded":"2024-03-30T15:05:28.405193+0000","last_fullsized":"2024-03-30T15:05:28.405193+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.505107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.505107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.825117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":907,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379553+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:05:28.379553+0000","last_peered":"2024-03-30T15:05:28.379553+0000","last_clean":"2024-03-30T15:05:28.379553+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:05:28.379553+0000","last_undegraded":"2024-03-30T15:05:28.379553+0000","last_fullsized":"2024-03-30T15:05:28.379553+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":797,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405835+0000","last_change":"2024-03-30T15:03:11.498906+0000","last_active":"2024-03-30T15:05:28.405835+0000","last_peered":"2024-03-30T15:05:28.405835+0000","last_clean":"2024-03-30T15:05:28.405835+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:05:28.405835+0000","last_undegraded":"2024-03-30T15:05:28.405835+0000","last_fullsized":"2024-03-30T15:05:28.405835+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.498760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.498760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374220+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:05:28.374220+0000","last_peered":"2024-03-30T15:05:28.374220+0000","last_clean":"2024-03-30T15:05:28.374220+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:05:28.374220+0000","last_undegraded":"2024-03-30T15:05:28.374220+0000","last_fullsized":"2024-03-30T15:05:28.374220+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.060229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":929,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374117+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:05:28.374117+0000","last_peered":"2024-03-30T15:05:28.374117+0000","last_clean":"2024-03-30T15:05:28.374117+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:05:28.374117+0000","last_undegraded":"2024-03-30T15:05:28.374117+0000","last_fullsized":"2024-03-30T15:05:28.374117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:00.062419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":903,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379889+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:05:28.379889+0000","last_peered":"2024-03-30T15:05:28.379889+0000","last_clean":"2024-03-30T15:05:28.379889+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:05:28.379889+0000","last_undegraded":"2024-03-30T15:05:28.379889+0000","last_fullsized":"2024-03-30T15:05:28.379889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:35.634280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":904,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062612+0000","last_change":"2024-03-30T15:03:26.133453+0000","last_active":"2024-03-30T15:05:22.062612+0000","last_peered":"2024-03-30T15:05:22.062612+0000","last_clean":"2024-03-30T15:05:22.062612+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:05:22.062612+0000","last_undegraded":"2024-03-30T15:05:22.062612+0000","last_fullsized":"2024-03-30T15:05:22.062612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:26.133342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:26.133342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373617+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:05:28.373617+0000","last_peered":"2024-03-30T15:05:28.373617+0000","last_clean":"2024-03-30T15:05:28.373617+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:05:28.373617+0000","last_undegraded":"2024-03-30T15:05:28.373617+0000","last_fullsized":"2024-03-30T15:05:28.373617+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405194+0000","last_change":"2024-03-30T15:04:46.590243+0000","last_active":"2024-03-30T15:05:28.405194+0000","last_peered":"2024-03-30T15:05:28.405194+0000","last_clean":"2024-03-30T15:05:28.405194+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:05:28.405194+0000","last_undegraded":"2024-03-30T15:05:28.405194+0000","last_fullsized":"2024-03-30T15:05:28.405194+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:46.590171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.590171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.642800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":987,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.033365+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:05:22.033365+0000","last_peered":"2024-03-30T15:05:22.033365+0000","last_clean":"2024-03-30T15:05:22.033365+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:05:22.033365+0000","last_undegraded":"2024-03-30T15:05:22.033365+0000","last_fullsized":"2024-03-30T15:05:22.033365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":895,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145064+0000","last_change":"2024-03-30T15:02:35.936293+0000","last_active":"2024-03-30T15:05:22.145064+0000","last_peered":"2024-03-30T15:05:22.145064+0000","last_clean":"2024-03-30T15:05:22.145064+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:05:22.145064+0000","last_undegraded":"2024-03-30T15:05:22.145064+0000","last_fullsized":"2024-03-30T15:05:22.145064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":904,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062624+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:05:22.062624+0000","last_peered":"2024-03-30T15:05:22.062624+0000","last_clean":"2024-03-30T15:05:22.062624+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:05:22.062624+0000","last_undegraded":"2024-03-30T15:05:22.062624+0000","last_fullsized":"2024-03-30T15:05:22.062624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":883,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143489+0000","last_change":"2024-03-30T15:02:50.847363+0000","last_active":"2024-03-30T15:05:22.143489+0000","last_peered":"2024-03-30T15:05:22.143489+0000","last_clean":"2024-03-30T15:05:22.143489+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:05:22.143489+0000","last_undegraded":"2024-03-30T15:05:22.143489+0000","last_fullsized":"2024-03-30T15:05:22.143489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.847286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.847286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":821,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032762+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:05:22.032762+0000","last_peered":"2024-03-30T15:05:22.032762+0000","last_clean":"2024-03-30T15:05:22.032762+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:05:22.032762+0000","last_undegraded":"2024-03-30T15:05:22.032762+0000","last_fullsized":"2024-03-30T15:05:22.032762+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":904,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.062611+0000","last_change":"2024-03-30T15:04:49.959125+0000","last_active":"2024-03-30T15:05:22.062611+0000","last_peered":"2024-03-30T15:05:22.062611+0000","last_clean":"2024-03-30T15:05:22.062611+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:05:22.062611+0000","last_undegraded":"2024-03-30T15:05:22.062611+0000","last_fullsized":"2024-03-30T15:05:22.062611+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.959020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.959020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.221503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405073+0000","last_change":"2024-03-30T15:03:19.519526+0000","last_active":"2024-03-30T15:05:28.405073+0000","last_peered":"2024-03-30T15:05:28.405073+0000","last_clean":"2024-03-30T15:05:28.405073+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:05:28.405073+0000","last_undegraded":"2024-03-30T15:05:28.405073+0000","last_fullsized":"2024-03-30T15:05:28.405073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:03:19.519440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.519440+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144328+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:05:22.144328+0000","last_peered":"2024-03-30T15:05:22.144328+0000","last_clean":"2024-03-30T15:05:22.144328+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:05:22.144328+0000","last_undegraded":"2024-03-30T15:05:22.144328+0000","last_fullsized":"2024-03-30T15:05:22.144328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.756007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":796,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143971+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:05:22.143971+0000","last_peered":"2024-03-30T15:05:22.143971+0000","last_clean":"2024-03-30T15:05:22.143971+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:05:22.143971+0000","last_undegraded":"2024-03-30T15:05:22.143971+0000","last_fullsized":"2024-03-30T15:05:22.143971+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404489+0000","last_change":"2024-03-30T15:04:37.589562+0000","last_active":"2024-03-30T15:05:28.404489+0000","last_peered":"2024-03-30T15:05:28.404489+0000","last_clean":"2024-03-30T15:05:28.404489+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:05:28.404489+0000","last_undegraded":"2024-03-30T15:05:28.404489+0000","last_fullsized":"2024-03-30T15:05:28.404489+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:50.917173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1026,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629847+0000","last_change":"2024-03-30T15:02:37.923842+0000","last_active":"2024-03-30T15:05:22.629847+0000","last_peered":"2024-03-30T15:05:22.629847+0000","last_clean":"2024-03-30T15:05:22.629847+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:05:22.629847+0000","last_undegraded":"2024-03-30T15:05:22.629847+0000","last_fullsized":"2024-03-30T15:05:22.629847+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:02:37.923752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:37.923752+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":935,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379182+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:05:28.379182+0000","last_peered":"2024-03-30T15:05:28.379182+0000","last_clean":"2024-03-30T15:05:28.379182+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:05:28.379182+0000","last_undegraded":"2024-03-30T15:05:28.379182+0000","last_fullsized":"2024-03-30T15:05:28.379182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":915,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.145042+0000","last_change":"2024-03-30T15:04:44.117459+0000","last_active":"2024-03-30T15:05:22.145042+0000","last_peered":"2024-03-30T15:05:22.145042+0000","last_clean":"2024-03-30T15:05:22.145042+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:05:22.145042+0000","last_undegraded":"2024-03-30T15:05:22.145042+0000","last_fullsized":"2024-03-30T15:05:22.145042+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:04:44.117358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.117358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.327955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":883,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379733+0000","last_change":"2024-03-30T15:02:47.030283+0000","last_active":"2024-03-30T15:05:28.379733+0000","last_peered":"2024-03-30T15:05:28.379733+0000","last_clean":"2024-03-30T15:05:28.379733+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:05:28.379733+0000","last_undegraded":"2024-03-30T15:05:28.379733+0000","last_fullsized":"2024-03-30T15:05:28.379733+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.030057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034344+0000","last_change":"2024-03-30T15:04:49.244698+0000","last_active":"2024-03-30T15:05:22.034344+0000","last_peered":"2024-03-30T15:05:22.034344+0000","last_clean":"2024-03-30T15:05:22.034344+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:05:22.034344+0000","last_undegraded":"2024-03-30T15:05:22.034344+0000","last_fullsized":"2024-03-30T15:05:22.034344+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.244616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.244616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:56.099737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":491,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.584751+0000","last_change":"2024-03-30T15:05:13.602996+0000","last_active":"2024-03-30T15:05:21.584751+0000","last_peered":"2024-03-30T15:05:21.584751+0000","last_clean":"2024-03-30T15:05:21.584751+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:05:21.584751+0000","last_undegraded":"2024-03-30T15:05:21.584751+0000","last_fullsized":"2024-03-30T15:05:21.584751+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:05:13.602902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.602902+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.234443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":918,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034454+0000","last_change":"2024-03-30T15:05:22.034492+0000","last_active":"2024-03-30T15:05:22.034454+0000","last_peered":"2024-03-30T15:05:22.034454+0000","last_clean":"2024-03-30T15:05:22.034454+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:05:22.034454+0000","last_undegraded":"2024-03-30T15:05:22.034454+0000","last_fullsized":"2024-03-30T15:05:22.034454+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:22.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.034372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.488283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":887,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380431+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:05:28.380431+0000","last_peered":"2024-03-30T15:05:28.380431+0000","last_clean":"2024-03-30T15:05:28.380431+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:05:28.380431+0000","last_undegraded":"2024-03-30T15:05:28.380431+0000","last_fullsized":"2024-03-30T15:05:28.380431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":883,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144063+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:05:22.144063+0000","last_peered":"2024-03-30T15:05:22.144063+0000","last_clean":"2024-03-30T15:05:22.144063+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:05:22.144063+0000","last_undegraded":"2024-03-30T15:05:22.144063+0000","last_fullsized":"2024-03-30T15:05:22.144063+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405832+0000","last_change":"2024-03-30T15:04:44.618218+0000","last_active":"2024-03-30T15:05:28.405832+0000","last_peered":"2024-03-30T15:05:28.405832+0000","last_clean":"2024-03-30T15:05:28.405832+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:05:28.405832+0000","last_undegraded":"2024-03-30T15:05:28.405832+0000","last_fullsized":"2024-03-30T15:05:28.405832+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.618135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.618135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:06.110883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":921,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375741+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:05:28.375741+0000","last_peered":"2024-03-30T15:05:28.375741+0000","last_clean":"2024-03-30T15:05:28.375741+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:05:28.375741+0000","last_undegraded":"2024-03-30T15:05:28.375741+0000","last_fullsized":"2024-03-30T15:05:28.375741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143698+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:05:22.143698+0000","last_peered":"2024-03-30T15:05:22.143698+0000","last_clean":"2024-03-30T15:05:22.143698+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:05:22.143698+0000","last_undegraded":"2024-03-30T15:05:22.143698+0000","last_fullsized":"2024-03-30T15:05:22.143698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:30.940258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":804,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.969250+0000","last_change":"2024-03-30T15:04:03.988041+0000","last_active":"2024-03-30T15:05:21.969250+0000","last_peered":"2024-03-30T15:05:21.969250+0000","last_clean":"2024-03-30T15:05:21.969250+0000","last_became_active":"2024-03-30T14:57:56.267594+0000","last_became_peered":"2024-03-30T14:57:56.267594+0000","last_unstale":"2024-03-30T15:05:21.969250+0000","last_undegraded":"2024-03-30T15:05:21.969250+0000","last_fullsized":"2024-03-30T15:05:21.969250+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":900,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064326+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:05:22.064326+0000","last_peered":"2024-03-30T15:05:22.064326+0000","last_clean":"2024-03-30T15:05:22.064326+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:05:22.064326+0000","last_undegraded":"2024-03-30T15:05:22.064326+0000","last_fullsized":"2024-03-30T15:05:22.064326+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374491+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:05:28.374491+0000","last_peered":"2024-03-30T15:05:28.374491+0000","last_clean":"2024-03-30T15:05:28.374491+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:05:28.374491+0000","last_undegraded":"2024-03-30T15:05:28.374491+0000","last_fullsized":"2024-03-30T15:05:28.374491+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:20.103043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":916,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380351+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:05:28.380351+0000","last_peered":"2024-03-30T15:05:28.380351+0000","last_clean":"2024-03-30T15:05:28.380351+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:05:28.380351+0000","last_undegraded":"2024-03-30T15:05:28.380351+0000","last_fullsized":"2024-03-30T15:05:28.380351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373634+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:05:28.373634+0000","last_peered":"2024-03-30T15:05:28.373634+0000","last_clean":"2024-03-30T15:05:28.373634+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:05:28.373634+0000","last_undegraded":"2024-03-30T15:05:28.373634+0000","last_fullsized":"2024-03-30T15:05:28.373634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.659468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":912,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629542+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:05:22.629542+0000","last_peered":"2024-03-30T15:05:22.629542+0000","last_clean":"2024-03-30T15:05:22.629542+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:05:22.629542+0000","last_undegraded":"2024-03-30T15:05:22.629542+0000","last_fullsized":"2024-03-30T15:05:22.629542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":884,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.406026+0000","last_change":"2024-03-30T15:04:35.553690+0000","last_active":"2024-03-30T15:05:28.406026+0000","last_peered":"2024-03-30T15:05:28.406026+0000","last_clean":"2024-03-30T15:05:28.406026+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:05:28.406026+0000","last_undegraded":"2024-03-30T15:05:28.406026+0000","last_fullsized":"2024-03-30T15:05:28.406026+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.553605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.553605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.879462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374201+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:05:28.374201+0000","last_peered":"2024-03-30T15:05:28.374201+0000","last_clean":"2024-03-30T15:05:28.374201+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:05:28.374201+0000","last_undegraded":"2024-03-30T15:05:28.374201+0000","last_fullsized":"2024-03-30T15:05:28.374201+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.969152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":853,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968686+0000","last_change":"2024-03-30T15:04:38.199333+0000","last_active":"2024-03-30T15:05:21.968686+0000","last_peered":"2024-03-30T15:05:21.968686+0000","last_clean":"2024-03-30T15:05:21.968686+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:05:21.968686+0000","last_undegraded":"2024-03-30T15:05:21.968686+0000","last_fullsized":"2024-03-30T15:05:21.968686+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:04:38.199239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.199239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.783575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":895,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143647+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:05:22.143647+0000","last_peered":"2024-03-30T15:05:22.143647+0000","last_clean":"2024-03-30T15:05:22.143647+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:05:22.143647+0000","last_undegraded":"2024-03-30T15:05:22.143647+0000","last_fullsized":"2024-03-30T15:05:22.143647+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1029,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380178+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:05:28.380178+0000","last_peered":"2024-03-30T15:05:28.380178+0000","last_clean":"2024-03-30T15:05:28.380178+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:05:28.380178+0000","last_undegraded":"2024-03-30T15:05:28.380178+0000","last_fullsized":"2024-03-30T15:05:28.380178+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":890,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375018+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:05:28.375018+0000","last_peered":"2024-03-30T15:05:28.375018+0000","last_clean":"2024-03-30T15:05:28.375018+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:05:28.375018+0000","last_undegraded":"2024-03-30T15:05:28.375018+0000","last_fullsized":"2024-03-30T15:05:28.375018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405463+0000","last_change":"2024-03-30T15:04:41.620132+0000","last_active":"2024-03-30T15:05:28.405463+0000","last_peered":"2024-03-30T15:05:28.405463+0000","last_clean":"2024-03-30T15:05:28.405463+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:05:28.405463+0000","last_undegraded":"2024-03-30T15:05:28.405463+0000","last_fullsized":"2024-03-30T15:05:28.405463+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:41.620027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:41.620027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.996471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":928,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405474+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:05:28.405474+0000","last_peered":"2024-03-30T15:05:28.405474+0000","last_clean":"2024-03-30T15:05:28.405474+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:05:28.405474+0000","last_undegraded":"2024-03-30T15:05:28.405474+0000","last_fullsized":"2024-03-30T15:05:28.405474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":904,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405242+0000","last_change":"2024-03-30T15:04:43.577081+0000","last_active":"2024-03-30T15:05:28.405242+0000","last_peered":"2024-03-30T15:05:28.405242+0000","last_clean":"2024-03-30T15:05:28.405242+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:05:28.405242+0000","last_undegraded":"2024-03-30T15:05:28.405242+0000","last_fullsized":"2024-03-30T15:05:28.405242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.576993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.576993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.972808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968314+0000","last_change":"2024-03-30T15:05:19.043801+0000","last_active":"2024-03-30T15:05:21.968314+0000","last_peered":"2024-03-30T15:05:21.968314+0000","last_clean":"2024-03-30T15:05:21.968314+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:05:21.968314+0000","last_undegraded":"2024-03-30T15:05:21.968314+0000","last_fullsized":"2024-03-30T15:05:21.968314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.043661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.043661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.583679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373408+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:05:28.373408+0000","last_peered":"2024-03-30T15:05:28.373408+0000","last_clean":"2024-03-30T15:05:28.373408+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:05:28.373408+0000","last_undegraded":"2024-03-30T15:05:28.373408+0000","last_fullsized":"2024-03-30T15:05:28.373408+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:48.074178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404755+0000","last_change":"2024-03-30T15:04:40.601679+0000","last_active":"2024-03-30T15:05:28.404755+0000","last_peered":"2024-03-30T15:05:28.404755+0000","last_clean":"2024-03-30T15:05:28.404755+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:05:28.404755+0000","last_undegraded":"2024-03-30T15:05:28.404755+0000","last_fullsized":"2024-03-30T15:05:28.404755+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.601541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.601541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:01.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":924,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968433+0000","last_change":"2024-03-30T15:04:54.952409+0000","last_active":"2024-03-30T15:05:21.968433+0000","last_peered":"2024-03-30T15:05:21.968433+0000","last_clean":"2024-03-30T15:05:21.968433+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:05:21.968433+0000","last_undegraded":"2024-03-30T15:05:21.968433+0000","last_fullsized":"2024-03-30T15:05:21.968433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:54.952263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.952263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:59.078655+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":918,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144178+0000","last_change":"2024-03-30T15:04:52.183278+0000","last_active":"2024-03-30T15:05:22.144178+0000","last_peered":"2024-03-30T15:05:22.144178+0000","last_clean":"2024-03-30T15:05:22.144178+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:05:22.144178+0000","last_undegraded":"2024-03-30T15:05:22.144178+0000","last_fullsized":"2024-03-30T15:05:22.144178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:04:52.183194+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.183194+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.754133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":63,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404787+0000","last_change":"2024-03-30T15:04:37.298232+0000","last_active":"2024-03-30T15:05:28.404787+0000","last_peered":"2024-03-30T15:05:28.404787+0000","last_clean":"2024-03-30T15:05:28.404787+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:05:28.404787+0000","last_undegraded":"2024-03-30T15:05:28.404787+0000","last_fullsized":"2024-03-30T15:05:28.404787+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.372859+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:05:28.372859+0000","last_peered":"2024-03-30T15:05:28.372859+0000","last_clean":"2024-03-30T15:05:28.372859+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:05:28.372859+0000","last_undegraded":"2024-03-30T15:05:28.372859+0000","last_fullsized":"2024-03-30T15:05:28.372859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373195+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:05:28.373195+0000","last_peered":"2024-03-30T15:05:28.373195+0000","last_clean":"2024-03-30T15:05:28.373195+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:05:28.373195+0000","last_undegraded":"2024-03-30T15:05:28.373195+0000","last_fullsized":"2024-03-30T15:05:28.373195+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.076034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":846,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143957+0000","last_change":"2024-03-30T15:02:32.992286+0000","last_active":"2024-03-30T15:05:22.143957+0000","last_peered":"2024-03-30T15:05:22.143957+0000","last_clean":"2024-03-30T15:05:22.143957+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:05:22.143957+0000","last_undegraded":"2024-03-30T15:05:22.143957+0000","last_fullsized":"2024-03-30T15:05:22.143957+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'11","last_scrub_stamp":"2024-03-30T15:02:32.992159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.992159+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":895,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380800+0000","last_change":"2024-03-30T15:02:35.972849+0000","last_active":"2024-03-30T15:05:28.380800+0000","last_peered":"2024-03-30T15:05:28.380800+0000","last_clean":"2024-03-30T15:05:28.380800+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:05:28.380800+0000","last_undegraded":"2024-03-30T15:05:28.380800+0000","last_fullsized":"2024-03-30T15:05:28.380800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:35.972760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:35.972760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":895,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.142699+0000","last_change":"2024-03-30T15:05:21.130956+0000","last_active":"2024-03-30T15:05:22.142699+0000","last_peered":"2024-03-30T15:05:22.142699+0000","last_clean":"2024-03-30T15:05:22.142699+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:05:22.142699+0000","last_undegraded":"2024-03-30T15:05:22.142699+0000","last_fullsized":"2024-03-30T15:05:22.142699+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:21.130862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.130862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:24.342532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373819+0000","last_change":"2024-03-30T15:04:40.631484+0000","last_active":"2024-03-30T15:05:28.373819+0000","last_peered":"2024-03-30T15:05:28.373819+0000","last_clean":"2024-03-30T15:05:28.373819+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:05:28.373819+0000","last_undegraded":"2024-03-30T15:05:28.373819+0000","last_fullsized":"2024-03-30T15:05:28.373819+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.631340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.631340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:54.585112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":801,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967687+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:05:21.967687+0000","last_peered":"2024-03-30T15:05:21.967687+0000","last_clean":"2024-03-30T15:05:21.967687+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:05:21.967687+0000","last_undegraded":"2024-03-30T15:05:21.967687+0000","last_fullsized":"2024-03-30T15:05:21.967687+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":469,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.582672+0000","last_change":"2024-03-30T15:04:01.043987+0000","last_active":"2024-03-30T15:05:21.582672+0000","last_peered":"2024-03-30T15:05:21.582672+0000","last_clean":"2024-03-30T15:05:21.582672+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:05:21.582672+0000","last_undegraded":"2024-03-30T15:05:21.582672+0000","last_fullsized":"2024-03-30T15:05:21.582672+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'4","last_scrub_stamp":"2024-03-30T15:04:01.043885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.043885+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:29.133940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375184+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:05:28.375184+0000","last_peered":"2024-03-30T15:05:28.375184+0000","last_clean":"2024-03-30T15:05:28.375184+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:05:28.375184+0000","last_undegraded":"2024-03-30T15:05:28.375184+0000","last_fullsized":"2024-03-30T15:05:28.375184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:44.517062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":902,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.032300+0000","last_change":"2024-03-30T15:03:14.483793+0000","last_active":"2024-03-30T15:05:22.032300+0000","last_peered":"2024-03-30T15:05:22.032300+0000","last_clean":"2024-03-30T15:05:22.032300+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:05:22.032300+0000","last_undegraded":"2024-03-30T15:05:22.032300+0000","last_fullsized":"2024-03-30T15:05:22.032300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.483716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.483716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":883,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.143729+0000","last_change":"2024-03-30T15:02:40.922302+0000","last_active":"2024-03-30T15:05:22.143729+0000","last_peered":"2024-03-30T15:05:22.143729+0000","last_clean":"2024-03-30T15:05:22.143729+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:05:22.143729+0000","last_undegraded":"2024-03-30T15:05:22.143729+0000","last_fullsized":"2024-03-30T15:05:22.143729+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.922214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.922214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034510+0000","last_change":"2024-03-30T15:04:51.176640+0000","last_active":"2024-03-30T15:05:22.034510+0000","last_peered":"2024-03-30T15:05:22.034510+0000","last_clean":"2024-03-30T15:05:22.034510+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:05:22.034510+0000","last_undegraded":"2024-03-30T15:05:22.034510+0000","last_fullsized":"2024-03-30T15:05:22.034510+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.176549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.176549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:04.079483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":935,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034518+0000","last_change":"2024-03-30T15:02:43.386204+0000","last_active":"2024-03-30T15:05:22.034518+0000","last_peered":"2024-03-30T15:05:22.034518+0000","last_clean":"2024-03-30T15:05:22.034518+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:05:22.034518+0000","last_undegraded":"2024-03-30T15:05:22.034518+0000","last_fullsized":"2024-03-30T15:05:22.034518+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-03-30T15:02:43.386045+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.386045+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":802,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373080+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:05:28.373080+0000","last_peered":"2024-03-30T15:05:28.373080+0000","last_clean":"2024-03-30T15:05:28.373080+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:05:28.373080+0000","last_undegraded":"2024-03-30T15:05:28.373080+0000","last_fullsized":"2024-03-30T15:05:28.373080+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":893,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375489+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:05:28.375489+0000","last_peered":"2024-03-30T15:05:28.375489+0000","last_clean":"2024-03-30T15:05:28.375489+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:05:28.375489+0000","last_undegraded":"2024-03-30T15:05:28.375489+0000","last_fullsized":"2024-03-30T15:05:28.375489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":891,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968772+0000","last_change":"2024-03-30T15:02:30.505583+0000","last_active":"2024-03-30T15:05:21.968772+0000","last_peered":"2024-03-30T15:05:21.968772+0000","last_clean":"2024-03-30T15:05:21.968772+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:05:21.968772+0000","last_undegraded":"2024-03-30T15:05:21.968772+0000","last_fullsized":"2024-03-30T15:05:21.968772+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:30.505443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:30.505443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034190+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:05:22.034190+0000","last_peered":"2024-03-30T15:05:22.034190+0000","last_clean":"2024-03-30T15:05:22.034190+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:05:22.034190+0000","last_undegraded":"2024-03-30T15:05:22.034190+0000","last_fullsized":"2024-03-30T15:05:22.034190+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.375332+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:05:28.375332+0000","last_peered":"2024-03-30T15:05:28.375332+0000","last_clean":"2024-03-30T15:05:28.375332+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:05:28.375332+0000","last_undegraded":"2024-03-30T15:05:28.375332+0000","last_fullsized":"2024-03-30T15:05:28.375332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":904,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:24.043261+0000","last_change":"2024-03-30T15:05:24.043308+0000","last_active":"2024-03-30T15:05:24.043261+0000","last_peered":"2024-03-30T15:05:24.043261+0000","last_clean":"2024-03-30T15:05:24.043261+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:05:24.043261+0000","last_undegraded":"2024-03-30T15:05:24.043261+0000","last_fullsized":"2024-03-30T15:05:24.043261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:05:24.043163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.043163+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.017816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1053,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.629551+0000","last_change":"2024-03-30T15:05:20.066039+0000","last_active":"2024-03-30T15:05:22.629551+0000","last_peered":"2024-03-30T15:05:22.629551+0000","last_clean":"2024-03-30T15:05:22.629551+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:05:22.629551+0000","last_undegraded":"2024-03-30T15:05:22.629551+0000","last_fullsized":"2024-03-30T15:05:22.629551+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:20.065886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.065886+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.126952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":893,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.144285+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:05:22.144285+0000","last_peered":"2024-03-30T15:05:22.144285+0000","last_clean":"2024-03-30T15:05:22.144285+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:05:22.144285+0000","last_undegraded":"2024-03-30T15:05:22.144285+0000","last_fullsized":"2024-03-30T15:05:22.144285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":808,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034052+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:05:22.034052+0000","last_peered":"2024-03-30T15:05:22.034052+0000","last_clean":"2024-03-30T15:05:22.034052+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:05:22.034052+0000","last_undegraded":"2024-03-30T15:05:22.034052+0000","last_fullsized":"2024-03-30T15:05:22.034052+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":855,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968624+0000","last_change":"2024-03-30T15:04:40.214721+0000","last_active":"2024-03-30T15:05:21.968624+0000","last_peered":"2024-03-30T15:05:21.968624+0000","last_clean":"2024-03-30T15:05:21.968624+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:05:21.968624+0000","last_undegraded":"2024-03-30T15:05:21.968624+0000","last_fullsized":"2024-03-30T15:05:21.968624+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:40.214633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.214633+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:09.122719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":896,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404978+0000","last_change":"2024-03-30T15:03:21.522112+0000","last_active":"2024-03-30T15:05:28.404978+0000","last_peered":"2024-03-30T15:05:28.404978+0000","last_clean":"2024-03-30T15:05:28.404978+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:05:28.404978+0000","last_undegraded":"2024-03-30T15:05:28.404978+0000","last_fullsized":"2024-03-30T15:05:28.404978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.522038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.522038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405638+0000","last_change":"2024-03-30T15:03:23.551230+0000","last_active":"2024-03-30T15:05:28.405638+0000","last_peered":"2024-03-30T15:05:28.405638+0000","last_clean":"2024-03-30T15:05:28.405638+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:05:28.405638+0000","last_undegraded":"2024-03-30T15:05:28.405638+0000","last_fullsized":"2024-03-30T15:05:28.405638+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:23.551149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.551149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":812,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034622+0000","last_change":"2024-03-30T15:05:15.145381+0000","last_active":"2024-03-30T15:05:22.034622+0000","last_peered":"2024-03-30T15:05:22.034622+0000","last_clean":"2024-03-30T15:05:22.034622+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:05:22.034622+0000","last_undegraded":"2024-03-30T15:05:22.034622+0000","last_fullsized":"2024-03-30T15:05:22.034622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.145314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.145314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.542525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379988+0000","last_change":"2024-03-30T15:04:36.274480+0000","last_active":"2024-03-30T15:05:28.379988+0000","last_peered":"2024-03-30T15:05:28.379988+0000","last_clean":"2024-03-30T15:05:28.379988+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:05:28.379988+0000","last_undegraded":"2024-03-30T15:05:28.379988+0000","last_fullsized":"2024-03-30T15:05:28.379988+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.274391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.274391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:57.565878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":958,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.064148+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:05:22.064148+0000","last_peered":"2024-03-30T15:05:22.064148+0000","last_clean":"2024-03-30T15:05:22.064148+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:05:22.064148+0000","last_undegraded":"2024-03-30T15:05:22.064148+0000","last_fullsized":"2024-03-30T15:05:22.064148+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374634+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:05:28.374634+0000","last_peered":"2024-03-30T15:05:28.374634+0000","last_clean":"2024-03-30T15:05:28.374634+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:05:28.374634+0000","last_undegraded":"2024-03-30T15:05:28.374634+0000","last_fullsized":"2024-03-30T15:05:28.374634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.088353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":903,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.967250+0000","last_change":"2024-03-30T15:05:15.051014+0000","last_active":"2024-03-30T15:05:21.967250+0000","last_peered":"2024-03-30T15:05:21.967250+0000","last_clean":"2024-03-30T15:05:21.967250+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:05:21.967250+0000","last_undegraded":"2024-03-30T15:05:21.967250+0000","last_fullsized":"2024-03-30T15:05:21.967250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.050918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.050918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.102896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405933+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:05:28.405933+0000","last_peered":"2024-03-30T15:05:28.405933+0000","last_clean":"2024-03-30T15:05:28.405933+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:05:28.405933+0000","last_undegraded":"2024-03-30T15:05:28.405933+0000","last_fullsized":"2024-03-30T15:05:28.405933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":801,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968162+0000","last_change":"2024-03-30T15:02:32.469819+0000","last_active":"2024-03-30T15:05:21.968162+0000","last_peered":"2024-03-30T15:05:21.968162+0000","last_clean":"2024-03-30T15:05:21.968162+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:05:21.968162+0000","last_undegraded":"2024-03-30T15:05:21.968162+0000","last_fullsized":"2024-03-30T15:05:21.968162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:32.469699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:32.469699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":929,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.405264+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:05:28.405264+0000","last_peered":"2024-03-30T15:05:28.405264+0000","last_clean":"2024-03-30T15:05:28.405264+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:05:28.405264+0000","last_undegraded":"2024-03-30T15:05:28.405264+0000","last_fullsized":"2024-03-30T15:05:28.405264+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":895,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379866+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:05:28.379866+0000","last_peered":"2024-03-30T15:05:28.379866+0000","last_clean":"2024-03-30T15:05:28.379866+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:05:28.379866+0000","last_undegraded":"2024-03-30T15:05:28.379866+0000","last_fullsized":"2024-03-30T15:05:28.379866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":887,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.142754+0000","last_change":"2024-03-30T15:02:31.010377+0000","last_active":"2024-03-30T15:05:22.142754+0000","last_peered":"2024-03-30T15:05:22.142754+0000","last_clean":"2024-03-30T15:05:22.142754+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:05:22.142754+0000","last_undegraded":"2024-03-30T15:05:22.142754+0000","last_fullsized":"2024-03-30T15:05:22.142754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:31.010277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:31.010277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":891,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379338+0000","last_change":"2024-03-30T15:04:35.253973+0000","last_active":"2024-03-30T15:05:28.379338+0000","last_peered":"2024-03-30T15:05:28.379338+0000","last_clean":"2024-03-30T15:05:28.379338+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:05:28.379338+0000","last_undegraded":"2024-03-30T15:05:28.379338+0000","last_fullsized":"2024-03-30T15:05:28.379338+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.253856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.253856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:46.131487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.968090+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:05:21.968090+0000","last_peered":"2024-03-30T15:05:21.968090+0000","last_clean":"2024-03-30T15:05:21.968090+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:05:21.968090+0000","last_undegraded":"2024-03-30T15:05:21.968090+0000","last_fullsized":"2024-03-30T15:05:21.968090+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":917,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404975+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:05:28.404975+0000","last_peered":"2024-03-30T15:05:28.404975+0000","last_clean":"2024-03-30T15:05:28.404975+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:05:28.404975+0000","last_undegraded":"2024-03-30T15:05:28.404975+0000","last_fullsized":"2024-03-30T15:05:28.404975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":895,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.380113+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:05:28.380113+0000","last_peered":"2024-03-30T15:05:28.380113+0000","last_clean":"2024-03-30T15:05:28.380113+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:05:28.380113+0000","last_undegraded":"2024-03-30T15:05:28.380113+0000","last_fullsized":"2024-03-30T15:05:28.380113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":895,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379830+0000","last_change":"2024-03-30T15:05:26.176781+0000","last_active":"2024-03-30T15:05:28.379830+0000","last_peered":"2024-03-30T15:05:28.379830+0000","last_clean":"2024-03-30T15:05:28.379830+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:05:28.379830+0000","last_undegraded":"2024-03-30T15:05:28.379830+0000","last_fullsized":"2024-03-30T15:05:28.379830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:26.176703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.176703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:49.697924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":896,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373072+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:05:28.373072+0000","last_peered":"2024-03-30T15:05:28.373072+0000","last_clean":"2024-03-30T15:05:28.373072+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:05:28.373072+0000","last_undegraded":"2024-03-30T15:05:28.373072+0000","last_fullsized":"2024-03-30T15:05:28.373072+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.231315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":801,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.379255+0000","last_change":"2024-03-30T15:04:52.290105+0000","last_active":"2024-03-30T15:05:28.379255+0000","last_peered":"2024-03-30T15:05:28.379255+0000","last_clean":"2024-03-30T15:05:28.379255+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:05:28.379255+0000","last_undegraded":"2024-03-30T15:05:28.379255+0000","last_fullsized":"2024-03-30T15:05:28.379255+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.289998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.289998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:58.214437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":916,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.063476+0000","last_change":"2024-03-30T15:05:17.006278+0000","last_active":"2024-03-30T15:05:22.063476+0000","last_peered":"2024-03-30T15:05:22.063476+0000","last_clean":"2024-03-30T15:05:22.063476+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:05:22.063476+0000","last_undegraded":"2024-03-30T15:05:22.063476+0000","last_fullsized":"2024-03-30T15:05:22.063476+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:17.006186+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.006186+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.567542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":805,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:22.034101+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:05:22.034101+0000","last_peered":"2024-03-30T15:05:22.034101+0000","last_clean":"2024-03-30T15:05:22.034101+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:05:22.034101+0000","last_undegraded":"2024-03-30T15:05:22.034101+0000","last_fullsized":"2024-03-30T15:05:22.034101+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404447+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:05:28.404447+0000","last_peered":"2024-03-30T15:05:28.404447+0000","last_clean":"2024-03-30T15:05:28.404447+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:05:28.404447+0000","last_undegraded":"2024-03-30T15:05:28.404447+0000","last_fullsized":"2024-03-30T15:05:28.404447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":457,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:21.583346+0000","last_change":"2024-03-30T15:04:39.209738+0000","last_active":"2024-03-30T15:05:21.583346+0000","last_peered":"2024-03-30T15:05:21.583346+0000","last_clean":"2024-03-30T15:05:21.583346+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:05:21.583346+0000","last_undegraded":"2024-03-30T15:05:21.583346+0000","last_fullsized":"2024-03-30T15:05:21.583346+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:39.209655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.209655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:05:47.749896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":806,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.373486+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:05:28.373486+0000","last_peered":"2024-03-30T15:05:28.373486+0000","last_clean":"2024-03-30T15:05:28.373486+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:05:28.373486+0000","last_undegraded":"2024-03-30T15:05:28.373486+0000","last_fullsized":"2024-03-30T15:05:28.373486+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.065353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":941,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404527+0000","last_change":"2024-03-30T15:04:39.592435+0000","last_active":"2024-03-30T15:05:28.404527+0000","last_peered":"2024-03-30T15:05:28.404527+0000","last_clean":"2024-03-30T15:05:28.404527+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:05:28.404527+0000","last_undegraded":"2024-03-30T15:05:28.404527+0000","last_fullsized":"2024-03-30T15:05:28.404527+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:39.592324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.592324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.690325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":909,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.374969+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:05:28.374969+0000","last_peered":"2024-03-30T15:05:28.374969+0000","last_clean":"2024-03-30T15:05:28.374969+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:05:28.374969+0000","last_undegraded":"2024-03-30T15:05:28.374969+0000","last_fullsized":"2024-03-30T15:05:28.374969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":895,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-30T15:05:27.054751+0000","last_change":"2024-03-30T15:05:27.054772+0000","last_active":"2024-03-30T15:05:27.054751+0000","last_peered":"2024-03-30T15:05:27.054751+0000","last_clean":"2024-03-30T15:05:27.054751+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:05:27.054751+0000","last_undegraded":"2024-03-30T15:05:27.054751+0000","last_fullsized":"2024-03-30T15:05:27.054751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.054691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.277274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":884,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-30T15:05:28.404799+0000","last_change":"2024-03-30T15:03:18.547558+0000","last_active":"2024-03-30T15:05:28.404799+0000","last_peered":"2024-03-30T15:05:28.404799+0000","last_clean":"2024-03-30T15:05:28.404799+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:05:28.404799+0000","last_undegraded":"2024-03-30T15:05:28.404799+0000","last_fullsized":"2024-03-30T15:05:28.404799+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:18.547481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:18.547481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":304,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8195937,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":117,"num_read_kb":11,"num_write":632,"num_write_kb":12716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26677248,"data_stored":24587787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1986,"ondisk_log_size":1986,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":643,"num_read_kb":638,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005191,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74532,"kb_used_data":11420,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670620,"statfs":{"total":95995035648,"available":95918714880,"internally_reserved":0,"allocated":11694080,"data_stored":8331292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299532,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74736,"kb_used_data":11340,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670416,"statfs":{"total":95995035648,"available":95918505984,"internally_reserved":0,"allocated":11612160,"data_stored":8328815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575015,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86652,"kb_used_data":6900,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93658500,"statfs":{"total":95995035648,"available":95906304000,"internally_reserved":0,"allocated":7065600,"data_stored":3739689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69148,"kb_used_data":11352,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93676004,"statfs":{"total":95995035648,"available":95924228096,"internally_reserved":0,"allocated":11624448,"data_stored":8330609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":82188,"kb_used_data":14916,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93662964,"statfs":{"total":95995035648,"available":95910875136,"internally_reserved":0,"allocated":15273984,"data_stored":11933191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117148,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78364,"kb_used_data":11092,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666788,"statfs":{"total":95995035648,"available":95914790912,"internally_reserved":0,"allocated":11358208,"data_stored":7935132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953622,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74232,"kb_used_data":6960,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670920,"statfs":{"total":95995035648,"available":95919022080,"internally_reserved":0,"allocated":7127040,"data_stored":3739641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659281,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70060,"kb_used_data":6948,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93675092,"statfs":{"total":95995035648,"available":95923294208,"internally_reserved":0,"allocated":7114752,"data_stored":3739322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4423680,"data_stored":4194827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8470528,"data_stored":8194915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4505600,"data_stored":4195009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4255744,"data_stored":4000620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":304,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:05:34.035 INFO:tasks.thrashosds.thrasher:Setting 5.17 to [6, 7, 1] 2024-03-30T15:05:34.035 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.17', '6', '7', '1'] 2024-03-30T15:05:34.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 5.17 6 7 1 2024-03-30T15:05:34.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:34.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:34.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:34.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:34.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:34.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:34.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:34.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:34.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[29876]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[29876]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[41614]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T15:05:34.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:34 smithi028 ceph-mon[41614]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:34 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:34 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:34 smithi134 ceph-mon[33714]: osdmap e501: 8 total, 8 up, 8 in 2024-03-30T15:05:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:34 smithi134 ceph-mon[33714]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:05:34.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:34.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:34.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:34.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:34.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:34.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:35.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:35.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:35.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:35.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:35.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:35.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:35.473 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T15:05:35.360+0000 7f2e52536700 -1 snap_mapper.add_oid found existing snaps mapped on 304:8a0bdb89:test-rados-api-smithi028-93253-80::foo:2, removing 2024-03-30T15:05:35.473 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T15:05:35.360+0000 7f46abe79700 -1 snap_mapper.add_oid found existing snaps mapped on 304:8a0bdb89:test-rados-api-smithi028-93253-80::foo:2, removing 2024-03-30T15:05:35.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:35.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:35.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:35.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:35.653 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:05:35 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:05:35.360+0000 7feca2000700 -1 snap_mapper.add_oid found existing snaps mapped on 304:8a0bdb89:test-rados-api-smithi028-93253-80::foo:2, removing 2024-03-30T15:05:35.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:35.742 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:05:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:05:35.742 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:05:35.742 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:05:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:35.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:35.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:35.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:35.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:35 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:35.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:35 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-79"}]: dispatch 2024-03-30T15:05:35.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:36.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:36.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:36.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:36.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:36.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:36.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:36.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:36.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:36.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:36.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:36.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:36.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:36.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:36.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:36.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:36 smithi134 ceph-mon[33714]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T15:05:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:36 smithi134 ceph-mon[33714]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:05:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:36 smithi134 ceph-mon[33714]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T15:05:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:36 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[29876]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T15:05:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[29876]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:05:36.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[29876]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T15:05:36.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:36.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[41614]: osdmap e502: 8 total, 8 up, 8 in 2024-03-30T15:05:36.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[41614]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:05:36.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[41614]: osdmap e503: 8 total, 8 up, 8 in 2024-03-30T15:05:36.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:36.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:36 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:37.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:37.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:37.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:05:37.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:37.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:37.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:37.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:37.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:37.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:37.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:37.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:37.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:37.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:37.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:37.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:37.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:38.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:38.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:38.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:38.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:38.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:38.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:38.523 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:38.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:38.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: osdmap e504: 8 total, 8 up, 8 in 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:38 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:38.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:38.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:38.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:38.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:38.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:38.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:38.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:39.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:39.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:39.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:39.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:39.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:39.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:39 smithi134 ceph-mon[33714]: pgmap v731: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:05:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:39 smithi134 ceph-mon[33714]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T15:05:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:39 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:39.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[29876]: pgmap v731: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[29876]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[41614]: pgmap v731: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:39.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[41614]: osdmap e505: 8 total, 8 up, 8 in 2024-03-30T15:05:39.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:39 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:39.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:39.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:39.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.725+0000 7f854c0d0700 1 -- 172.21.15.28:0/2746644150 >> v1:172.21.15.28:6789/0 conn(0x7f854410b070 legacy=0x7f854410d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.725+0000 7f854c0d0700 1 -- 172.21.15.28:0/2746644150 shutdown_connections 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.725+0000 7f854c0d0700 1 -- 172.21.15.28:0/2746644150 >> 172.21.15.28:0/2746644150 conn(0x7f85440fca30 msgr2=0x7f85440fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.725+0000 7f854c0d0700 1 -- 172.21.15.28:0/2746644150 shutdown_connections 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.725+0000 7f854c0d0700 1 -- 172.21.15.28:0/2746644150 wait complete. 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 librados: starting msgr at 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 librados: starting objecter 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 Processor -- start 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 -- start start 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 librados: setting wanted keys 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 librados: calling monclient init 2024-03-30T15:05:39.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85441a91d0 con 0x7f854410b070 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85441aa3f0 con 0x7f8544103260 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854c0d0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f85441ab610 con 0x7f854410eb80 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854966b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f854410b070 0x7f85441a42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58750/0 (socket says 172.21.15.28:58750) 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854a66d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f854410eb80 0x7f85441a7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:46448/0 (socket says 172.21.15.28:46448) 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f854966b700 1 -- 172.21.15.28:0/3118322022 learned_addr learned my addr 172.21.15.28:0/3118322022 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f853affd700 10 client.?.objecter ms_handle_connect 0x7f854410eb80 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f853affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f853affd700 10 client.?.objecter ms_handle_connect 0x7f854410b070 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.726+0000 7f853affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 10 client.?.objecter ms_handle_connect 0x7f8544103260 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357240274 0 0) 0x7f85441ab610 con 0x7f854410eb80 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85300039c0 con 0x7f854410eb80 2024-03-30T15:05:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879604365 0 0) 0x7f85441a91d0 con 0x7f854410b070 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85441ab610 con 0x7f854410b070 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674178666 0 0) 0x7f85441aa3f0 con 0x7f8544103260 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85441a91d0 con 0x7f8544103260 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3892710011 0 0) 0x7f85441a91d0 con 0x7f8544103260 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85441aa3f0 con 0x7f8544103260 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8540001830 con 0x7f8544103260 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1397980000 0 0) 0x7f85441ab610 con 0x7f854410b070 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85441a91d0 con 0x7f854410b070 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1435241312 0 0) 0x7f85300039c0 con 0x7f854410eb80 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85441ab610 con 0x7f854410eb80 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8534001350 con 0x7f854410b070 2024-03-30T15:05:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f853c001c10 con 0x7f854410eb80 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3819597199 0 0) 0x7f85441aa3f0 con 0x7f8544103260 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6790/0 conn(0x7f854410eb80 legacy=0x7f85441a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6789/0 conn(0x7f854410b070 legacy=0x7f85441a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85441ac850 con 0x7f8544103260 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85441a9400 con 0x7f8544103260 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f854c0d0700 10 client.50257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f85441a98c0 con 0x7f8544103260 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f854c0d0700 1 librados: init done 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8540003b20 con 0x7f8544103260 2024-03-30T15:05:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.727+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8540004b10 con 0x7f8544103260 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.728+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8528005100 con 0x7f8544103260 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.729+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8540003cd0 con 0x7f8544103260 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.729+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(506..506 src has 442..506) ==== 8621+0+0 (unknown 2903405535 0 0) 0x7f8540088600 con 0x7f8544103260 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.729+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 osd_map(506..506 src has 442..506) 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.729+0000 7f853affd700 3 client.50257.objecter handle_osd_map got epochs [506,506] > 0 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.729+0000 7f853affd700 3 client.50257.objecter handle_osd_map decoding full epoch 506 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f853affd700 20 client.50257.objecter dump_active .. 0 homeless 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v506) ==== 54+0+16 (unknown 3505652984 0 2420000192) 0x7f8540088a00 con 0x7f8544103260 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f85300703f0 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 10 client.50257.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 1 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 1 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 0 1 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 4 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 10 client.50257.objecter _send_command 1 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8528004b50 con 0x7f8528004730 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.730+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528004730 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.731+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528004b50 con 0x7f8528004730 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.731+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.731+0000 7f853affd700 10 client.50257.objecter _finish_command 1 = system:0 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.731+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 4 2024-03-30T15:05:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.731+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 0 1 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 10 client.50257.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 2 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 2 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f85280053e0 osd=0 3 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 0 2 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 4 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 10 client.50257.objecter _send_command 2 2024-03-30T15:05:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.748+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f85280079f0 con 0x7f8528004730 2024-03-30T15:05:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f85280079f0 con 0x7f8528004730 2024-03-30T15:05:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f853affd700 10 client.50257.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 4 2024-03-30T15:05:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 0 2 2024-03-30T15:05:39.752 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 10 client.50257.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 1 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 4 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 10 client.50257.objecter _send_command 3 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.750+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8528002180 con 0x7f8528004280 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528004280 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528002180 con 0x7f8528004280 2024-03-30T15:05:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 10 client.50257.objecter _finish_command 3 = system:0 2024-03-30T15:05:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 4 2024-03-30T15:05:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.751+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 1 3 2024-03-30T15:05:39.769 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.773+0000 7f854c0d0700 10 client.50257.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 4 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 4 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528004110 osd=1 3 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 1 4 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 4 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 10 client.50257.objecter _send_command 4 2024-03-30T15:05:39.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.774+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8528001600 con 0x7f8528004280 2024-03-30T15:05:39.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.775+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8528001600 con 0x7f8528004280 2024-03-30T15:05:39.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.775+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.775+0000 7f853affd700 10 client.50257.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.775+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 4 2024-03-30T15:05:39.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.775+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 1 4 2024-03-30T15:05:39.777 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 10 client.50257.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 5 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 5 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 2 5 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 4 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 10 client.50257.objecter _send_command 5 2024-03-30T15:05:39.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8528010f20 con 0x7f8528008880 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.776+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528008880 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.777+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528010f20 con 0x7f8528008880 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.777+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.777+0000 7f853affd700 10 client.50257.objecter _finish_command 5 = system:0 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.777+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 4 2024-03-30T15:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.777+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 2 5 2024-03-30T15:05:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 10 client.50257.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 6 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 6 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528001a00 osd=2 3 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 2 6 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 4 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 10 client.50257.objecter _send_command 6 2024-03-30T15:05:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.794+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8528007f10 con 0x7f8528008880 2024-03-30T15:05:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8528007f10 con 0x7f8528008880 2024-03-30T15:05:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 10 client.50257.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 4 2024-03-30T15:05:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 2 6 2024-03-30T15:05:39.798 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 10 client.50257.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 7 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 7 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 3 7 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 4 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 10 client.50257.objecter _send_command 7 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8528011c70 con 0x7f8528011100 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.796+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528011100 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.797+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528011c70 con 0x7f8528011100 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.797+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.797+0000 7f853affd700 10 client.50257.objecter _finish_command 7 = system:0 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.797+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 4 2024-03-30T15:05:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.797+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 3 7 2024-03-30T15:05:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 10 client.50257.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 8 2024-03-30T15:05:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 8 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f85280081a0 osd=3 3 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 3 8 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 4 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 10 client.50257.objecter _send_command 8 2024-03-30T15:05:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.814+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8528012620 con 0x7f8528011100 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8528012620 con 0x7f8528011100 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 10 client.50257.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 4 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 3 8 2024-03-30T15:05:39.817 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 10 client.50257.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 9 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 9 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 4 9 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 4 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 10 client.50257.objecter _send_command 9 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f852801a7a0 con 0x7f8528016ae0 2024-03-30T15:05:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.816+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528016ae0 2024-03-30T15:05:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.817+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f852801a7a0 con 0x7f8528016ae0 2024-03-30T15:05:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.817+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.817+0000 7f853affd700 10 client.50257.objecter _finish_command 9 = system:0 2024-03-30T15:05:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.817+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 4 2024-03-30T15:05:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.817+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 4 9 2024-03-30T15:05:39.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 10 client.50257.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 10 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 10 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528012620 osd=4 3 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 4 10 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 4 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 10 client.50257.objecter _send_command 10 2024-03-30T15:05:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.834+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f85280120a0 con 0x7f8528016ae0 2024-03-30T15:05:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.835+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f85280120a0 con 0x7f8528016ae0 2024-03-30T15:05:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.835+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.835+0000 7f853affd700 10 client.50257.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.835+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 4 2024-03-30T15:05:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.835+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 4 10 2024-03-30T15:05:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 10 client.50257.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 11 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 11 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 5 11 2024-03-30T15:05:39.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 4 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 10 client.50257.objecter _send_command 11 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8528013830 con 0x7f8528011e50 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.836+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f8528011e50 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.837+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528013830 con 0x7f8528011e50 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.837+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.837+0000 7f853affd700 10 client.50257.objecter _finish_command 11 = system:0 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.837+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 4 2024-03-30T15:05:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.837+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 5 11 2024-03-30T15:05:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 10 client.50257.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 12 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 12 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f852800c760 osd=5 3 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 5 12 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 4 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 10 client.50257.objecter _send_command 12 2024-03-30T15:05:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.854+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8528013830 con 0x7f8528011e50 2024-03-30T15:05:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8528013830 con 0x7f8528011e50 2024-03-30T15:05:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f853affd700 10 client.50257.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 4 2024-03-30T15:05:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 5 12 2024-03-30T15:05:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 10 client.50257.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 13 2024-03-30T15:05:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 13 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 6 13 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 4 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 10 client.50257.objecter _send_command 13 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.855+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8528023190 con 0x7f852801f680 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.856+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f852801f680 2024-03-30T15:05:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.857+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8528023190 con 0x7f852801f680 2024-03-30T15:05:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.857+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.857+0000 7f853affd700 10 client.50257.objecter _finish_command 13 = system:0 2024-03-30T15:05:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.857+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 4 2024-03-30T15:05:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.857+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 6 13 2024-03-30T15:05:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 10 client.50257.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 14 2024-03-30T15:05:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 14 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528013830 osd=6 3 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 6 14 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 4 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 10 client.50257.objecter _send_command 14 2024-03-30T15:05:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.873+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8528023190 con 0x7f852801f680 2024-03-30T15:05:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8528023190 con 0x7f852801f680 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 10 client.50257.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 4 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 6 14 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 10 client.50257.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 15 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 15 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 7 15 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 4 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 10 client.50257.objecter _send_command 15 2024-03-30T15:05:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f85280256f0 con 0x7f85280251d0 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.875+0000 7f853affd700 10 client.50257.objecter ms_handle_connect 0x7f85280251d0 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.876+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f85280256f0 con 0x7f85280251d0 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.876+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.876+0000 7f853affd700 10 client.50257.objecter _finish_command 15 = system:0 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.876+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 4 2024-03-30T15:05:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.876+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 7 15 2024-03-30T15:05:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 10 client.50257.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign -1 16 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 20 client.50257.objecter _get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_remove -1 16 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 20 client.50257.objecter get_session s=0x7f8528024b10 osd=7 3 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 15 client.50257.objecter _session_command_op_assign 7 16 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 4 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 10 client.50257.objecter _send_command 16 2024-03-30T15:05:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.893+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f85280256f0 con 0x7f85280251d0 2024-03-30T15:05:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.895+0000 7f853affd700 1 -- 172.21.15.28:0/3118322022 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f85280256f0 con 0x7f85280251d0 2024-03-30T15:05:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.895+0000 7f853affd700 10 client.50257.objecter ms_dispatch 0x7f8544001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.895+0000 7f853affd700 10 client.50257.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.895+0000 7f853affd700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 4 2024-03-30T15:05:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.895+0000 7f853affd700 15 client.50257.objecter _session_command_op_remove 7 16 2024-03-30T15:05:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 librados: watch_flush enter 2024-03-30T15:05:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 librados: watch_flush exit 2024-03-30T15:05:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.0 2024-03-30T15:05:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6801/3394323575 conn(0x7f8528004730 legacy=0x7f8528002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280053e0 osd=0 1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6805/1850313547 conn(0x7f8528004280 legacy=0x7f852800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528004110 osd=1 1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.2 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6809/3451325043 conn(0x7f8528008880 legacy=0x7f852800f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528001a00 osd=2 1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.3 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6813/1506157250 conn(0x7f8528011100 legacy=0x7f8528011560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f85280081a0 osd=3 1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.4 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.134:6800/3063775514 conn(0x7f8528016ae0 legacy=0x7f8528018ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528012620 osd=4 1 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.5 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.134:6804/3668151179 conn(0x7f8528011e50 legacy=0x7f8528013050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854a66d700 1 -- 172.21.15.28:0/3118322022 reap_dead start 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854a66d700 1 -- 172.21.15.28:0/3118322022 reap_dead start 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f852800c760 osd=5 1 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.6 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.134:6808/2135224194 conn(0x7f852801f680 legacy=0x7f8528021a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854a66d700 1 -- 172.21.15.28:0/3118322022 reap_dead start 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528013830 osd=6 1 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter close_session for osd.7 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.134:6812/3758190298 conn(0x7f85280251d0 legacy=0x7f8528023190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter put_session s=0x7f8528024b10 osd=7 1 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 20 client.50257.objecter shutdown clearing up homeless session... 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 10 client.50257.objecter successfully canceled tick 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.28:6800/3011431093 conn(0x7f85300703f0 legacy=0x7f85300728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> v1:172.21.15.134:6789/0 conn(0x7f8544103260 legacy=0x7f854410a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 shutdown_connections 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 >> 172.21.15.28:0/3118322022 conn(0x7f85440fca30 msgr2=0x7f85440fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 shutdown_connections 2024-03-30T15:05:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 -- 172.21.15.28:0/3118322022 wait complete. 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:39.896+0000 7f854c0d0700 1 librados: shutdown 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:39.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:39.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:39.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:40.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:40.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:40.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.392+0000 7fe76ded2700 1 -- 172.21.15.28:0/1495673466 >> v1:172.21.15.28:6790/0 conn(0x7fe768102230 legacy=0x7fe768102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.392+0000 7fe76ded2700 1 -- 172.21.15.28:0/1495673466 shutdown_connections 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 -- 172.21.15.28:0/1495673466 >> 172.21.15.28:0/1495673466 conn(0x7fe7680fdb00 msgr2=0x7fe7680fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 -- 172.21.15.28:0/1495673466 shutdown_connections 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 -- 172.21.15.28:0/1495673466 wait complete. 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 librados: starting msgr at 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 librados: starting objecter 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 Processor -- start 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 -- start start 2024-03-30T15:05:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 librados: setting wanted keys 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.393+0000 7fe76ded2700 1 librados: calling monclient init 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe76ded2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe768076040 con 0x7fe76810ea60 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe76ded2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe768076250 con 0x7fe76810aec0 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe76ded2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe768076460 con 0x7fe768102230 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe7677fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe768102230 0x7fe768074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:46464/0 (socket says 172.21.15.28:46464) 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe7677fe700 1 -- 172.21.15.28:0/1282085462 learned_addr learned my addr 172.21.15.28:0/1282085462 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter ms_handle_connect 0x7fe768102230 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter ms_handle_connect 0x7fe76810ea60 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter ms_handle_connect 0x7fe76810aec0 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61808589 0 0) 0x7fe768076460 con 0x7fe768102230 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7500031f0 con 0x7fe768102230 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967471077 0 0) 0x7fe768076040 con 0x7fe76810ea60 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7500040e0 con 0x7fe76810ea60 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955494979 0 0) 0x7fe768076250 con 0x7fe76810aec0 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe768076040 con 0x7fe76810aec0 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2586500633 0 0) 0x7fe7500031f0 con 0x7fe768102230 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe768076250 con 0x7fe768102230 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe758004650 con 0x7fe768102230 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 175697202 0 0) 0x7fe7500040e0 con 0x7fe76810ea60 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7500031f0 con 0x7fe76810ea60 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4144573679 0 0) 0x7fe768076040 con 0x7fe76810aec0 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7500040e0 con 0x7fe76810aec0 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe760001c30 con 0x7fe76810ea60 2024-03-30T15:05:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe75c0017b0 con 0x7fe76810aec0 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1665809788 0 0) 0x7fe768076250 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 >> v1:172.21.15.134:6789/0 conn(0x7fe76810aec0 legacy=0x7fe768074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 >> v1:172.21.15.28:6789/0 conn(0x7fe76810ea60 legacy=0x7fe7681a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7681aca80 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.394+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7681a9a70 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.395+0000 7fe76ded2700 10 client.58253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.395+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7681a9f30 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.395+0000 7fe76ded2700 1 librados: init done 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.395+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe758001af0 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.395+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe758004e90 con 0x7fe768102230 2024-03-30T15:05:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe754005120 con 0x7fe768102230 2024-03-30T15:05:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe7580050d0 con 0x7fe768102230 2024-03-30T15:05:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(506..506 src has 442..506) ==== 8621+0+0 (unknown 2903405535 0 0) 0x7fe758089910 con 0x7fe768102230 2024-03-30T15:05:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe764ff9700 10 client.58253.objecter ms_dispatch 0x7fe768001550 osd_map(506..506 src has 442..506) 2024-03-30T15:05:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe764ff9700 3 client.58253.objecter handle_osd_map got epochs [506,506] > 0 2024-03-30T15:05:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.396+0000 7fe764ff9700 3 client.58253.objecter handle_osd_map decoding full epoch 506 2024-03-30T15:05:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.397+0000 7fe764ff9700 20 client.58253.objecter dump_active .. 0 homeless 2024-03-30T15:05:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.397+0000 7fe764ff9700 10 client.58253.objecter ms_handle_connect 0x7fe7500707c0 2024-03-30T15:05:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.404+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe758056950 con 0x7fe768102230 2024-03-30T15:05:40.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:40.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3118322022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[41614]: pgmap v734: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3118322022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:40.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:40 smithi028 ceph-mon[29876]: pgmap v734: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:40.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:40.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:40.632+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]} v 0) -- 0x7fe754005c20 con 0x7fe768102230 2024-03-30T15:05:40.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: osdmap e506: 8 total, 8 up, 8 in 2024-03-30T15:05:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3118322022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:40.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:40 smithi134 ceph-mon[33714]: pgmap v734: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:40.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:40.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:40.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:40.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:40.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:40.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:41.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:41.102 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:41.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.136+0000 7f4556553700 1 -- 172.21.15.28:0/153082751 >> v1:172.21.15.28:6789/0 conn(0x7f4550109db0 legacy=0x7f455010c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.136+0000 7f4556553700 1 -- 172.21.15.28:0/153082751 shutdown_connections 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.136+0000 7f4556553700 1 -- 172.21.15.28:0/153082751 >> 172.21.15.28:0/153082751 conn(0x7f45500fc9f0 msgr2=0x7f45500fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.136+0000 7f4556553700 1 -- 172.21.15.28:0/153082751 shutdown_connections 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.136+0000 7f4556553700 1 -- 172.21.15.28:0/153082751 wait complete. 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 librados: starting msgr at 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 librados: starting objecter 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 Processor -- start 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 -- start start 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 librados: setting wanted keys 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 librados: calling monclient init 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4550106770 con 0x7f455010d950 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4550106980 con 0x7f4550109db0 2024-03-30T15:05:41.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f4556553700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45501a8530 con 0x7f4550101120 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f454ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f4550101120 0x7f45501035c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:46476/0 (socket says 172.21.15.28:46476) 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.137+0000 7f454ffff700 1 -- 172.21.15.28:0/1318190832 learned_addr learned my addr 172.21.15.28:0/1318190832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter ms_handle_connect 0x7f455010d950 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter ms_handle_connect 0x7f4550101120 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter ms_handle_connect 0x7f4550109db0 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891331892 0 0) 0x7f4550106770 con 0x7f455010d950 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45380039c0 con 0x7f455010d950 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567869307 0 0) 0x7f45501a8530 con 0x7f4550101120 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4550106770 con 0x7f4550101120 2024-03-30T15:05:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931458139 0 0) 0x7f4550106980 con 0x7f4550109db0 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45501a8530 con 0x7f4550109db0 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3881565554 0 0) 0x7f4550106770 con 0x7f4550101120 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4550106980 con 0x7f4550101120 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1196812663 0 0) 0x7f45380039c0 con 0x7f455010d950 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4550106770 con 0x7f455010d950 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f45440018f0 con 0x7f4550101120 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2720261124 0 0) 0x7f45501a8530 con 0x7f4550109db0 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45380039c0 con 0x7f4550109db0 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4548001590 con 0x7f455010d950 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4540001860 con 0x7f4550109db0 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3068591200 0 0) 0x7f4550106980 con 0x7f4550101120 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 >> v1:172.21.15.134:6789/0 conn(0x7f4550109db0 legacy=0x7f4550103cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 >> v1:172.21.15.28:6789/0 conn(0x7f455010d950 legacy=0x7f45501a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45501a9710 con 0x7f4550101120 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45501a8760 con 0x7f4550101120 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f4556553700 10 client.58259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f45501a8c20 con 0x7f4550101120 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f4556553700 1 librados: init done 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4544001d00 con 0x7f4550101120 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.138+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4544005d80 con 0x7f4550101120 2024-03-30T15:05:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.139+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c005120 con 0x7f4550101120 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.140+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f454401c6a0 con 0x7f4550101120 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.140+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(507..507 src has 442..507) ==== 8640+0+0 (unknown 1042561127 0 0) 0x7f4544089b40 con 0x7f4550101120 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.140+0000 7f454d7fa700 10 client.58259.objecter ms_dispatch 0x7f4550001550 osd_map(507..507 src has 442..507) 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.140+0000 7f454d7fa700 3 client.58259.objecter handle_osd_map got epochs [507,507] > 0 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.140+0000 7f454d7fa700 3 client.58259.objecter handle_osd_map decoding full epoch 507 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.141+0000 7f454d7fa700 20 client.58259.objecter dump_active .. 0 homeless 2024-03-30T15:05:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.141+0000 7f454d7fa700 10 client.58259.objecter ms_handle_connect 0x7f4538070490 2024-03-30T15:05:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.145+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4544056b50 con 0x7f4550101120 2024-03-30T15:05:41.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:41.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:41.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:41.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.368+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f453c005410 con 0x7f4550101120 2024-03-30T15:05:41.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.412+0000 7f454d7fa700 1 -- 172.21.15.28:0/1318190832 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v508) ==== 97+0+0 (unknown 328441465 0 0) 0x7f454405a5a0 con 0x7f4550101120 2024-03-30T15:05:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.413+0000 7fe764ff9700 1 -- 172.21.15.28:0/1282085462 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]=0 set 5.17 pg_upmap mapping to [6,7,1] v508) ==== 129+0+0 (unknown 742429813 0 0) 0x7fe758004e70 con 0x7fe768102230 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 10 librados: watch_flush enter 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 10 librados: watch_flush exit 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 20 client.58259.objecter shutdown clearing up homeless session... 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 10 client.58259.objecter successfully canceled tick 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4538070490 legacy=0x7f4538072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 >> v1:172.21.15.28:6790/0 conn(0x7f4550101120 legacy=0x7f45501035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 shutdown_connections 2024-03-30T15:05:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 >> 172.21.15.28:0/1318190832 conn(0x7f45500fc9f0 msgr2=0x7f4550106c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 shutdown_connections 2024-03-30T15:05:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.415+0000 7f4556553700 1 -- 172.21.15.28:0/1318190832 wait complete. 2024-03-30T15:05:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7f4556553700 1 librados: shutdown 2024-03-30T15:05:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 10 librados: watch_flush enter 2024-03-30T15:05:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 10 librados: watch_flush exit 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 20 client.58253.objecter shutdown clearing up homeless session... 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 10 client.58253.objecter successfully canceled tick 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe7500707c0 legacy=0x7fe750072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.416+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 >> v1:172.21.15.28:6790/0 conn(0x7fe768102230 legacy=0x7fe768074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.417+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 shutdown_connections 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.417+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 >> 172.21.15.28:0/1282085462 conn(0x7fe7680fdb00 msgr2=0x7fe7681100b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.417+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 shutdown_connections 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.417+0000 7fe76ded2700 1 -- 172.21.15.28:0/1282085462 wait complete. 2024-03-30T15:05:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:41.417+0000 7fe76ded2700 1 librados: shutdown 2024-03-30T15:05:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:05:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:set 5.17 pg_upmap mapping to [6,7,1] 2024-03-30T15:05:41.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:41.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:41.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:41.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T15:05:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1282085462' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.58253 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1318190832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:41 smithi134 ceph-mon[33714]: from='client.58259 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1282085462' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.58253 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1318190832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[41614]: from='client.58259 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: osdmap e507: 8 total, 8 up, 8 in 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1282085462' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.58253 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1318190832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:41 smithi028 ceph-mon[29876]: from='client.58259 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:05:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:41.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:41.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:41.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:41.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:41.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:42.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:42.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:42.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:42.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:42.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:42.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:42.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:42.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:42.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:42.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:42.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.58253 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]': finished 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.58259 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: pgmap v737: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.58253 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]': finished 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.58259 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: pgmap v737: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:42 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.58253 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.17", "id": [6, 7, 1]}]': finished 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.58259 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: osdmap e508: 8 total, 8 up, 8 in 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-30T15:05:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: pgmap v737: 260 pgs: 260 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-30T15:05:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-30T15:05:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: osdmap e509: 8 total, 8 up, 8 in 2024-03-30T15:05:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:42 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-30T15:05:42.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:42.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:42.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:42.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:42.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:42.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:42.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:43.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:43.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:43.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:43.244 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:43.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:43.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:43.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:43.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:43.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:43.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:43 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T15:05:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T15:05:43.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-30T15:05:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: osdmap e510: 8 total, 8 up, 8 in 2024-03-30T15:05:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:43 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:43.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:43.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:43.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:43.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:43.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:43.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:44.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:44.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:44.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:44.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:44.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:44.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:44.629 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:44.629 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: pgmap v740: 260 pgs: 260 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: pgmap v740: 260 pgs: 260 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:44 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: pgmap v740: 260 pgs: 260 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-03-30T15:05:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: osdmap e511: 8 total, 8 up, 8 in 2024-03-30T15:05:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:44 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-81"}]: dispatch 2024-03-30T15:05:44.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.725 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:44.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:44.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:44.849 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:44.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:44.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:45.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:45.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:45.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:05:45.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:45.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:45.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:45.526 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:45.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:45.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:45.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:05:45.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:45.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:45.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:45.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:45.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:45.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:45.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:46.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:46.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:05:46.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:46.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:46.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:46.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:46.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:46.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:46.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:46.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:46 smithi134 ceph-mon[33714]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T15:05:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:46 smithi134 ceph-mon[33714]: pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T15:05:46.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:46 smithi028 ceph-mon[29876]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T15:05:46.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:46 smithi028 ceph-mon[29876]: pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T15:05:46.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:46 smithi028 ceph-mon[41614]: osdmap e512: 8 total, 8 up, 8 in 2024-03-30T15:05:46.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:46 smithi028 ceph-mon[41614]: pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 9.0 KiB/s wr, 28 op/s 2024-03-30T15:05:46.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:46.792 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:46.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:46.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:46.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:46.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:47.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:47.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:47.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:47.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:47.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:47.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:47.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:47.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[41614]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T15:05:47.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[29876]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T15:05:47.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:47 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:47.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:47 smithi134 ceph-mon[33714]: osdmap e513: 8 total, 8 up, 8 in 2024-03-30T15:05:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:47 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:47.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:47.780 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:47.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:47.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:47.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:48.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:48.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:48.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:48.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:05:48.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:05:48.039 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T15:05:48.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:05:48.064 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:05:48.065 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:05:48.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:48.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:48.330 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:48.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:48.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:48.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:48.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:48 smithi134 ceph-mon[33714]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[29876]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: osdmap e514: 8 total, 8 up, 8 in 2024-03-30T15:05:48.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:48.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:48.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:48 smithi028 ceph-mon[41614]: pgmap v746: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:48.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:48.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:05:48.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:48.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:48.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:48.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:49.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:49.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:49.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:49.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:49.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:49.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:49.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:49.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:49.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:49 smithi134 ceph-mon[33714]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T15:05:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:49.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:49.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[29876]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T15:05:49.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:49.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[41614]: osdmap e515: 8 total, 8 up, 8 in 2024-03-30T15:05:49.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:49.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:49.873 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:49.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:49.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:49.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:50.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:50.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:50.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:50.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:50.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:50.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:50.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:50.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T15:05:50.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: pgmap v749: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2649022070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:50.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-mon[33714]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.139+0000 7f62bc80a700 1 -- 172.21.15.28:0/1547424615 >> v1:172.21.15.28:6790/0 conn(0x7f62b810d970 legacy=0x7f62b810fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.139+0000 7f62bc80a700 1 -- 172.21.15.28:0/1547424615 shutdown_connections 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.139+0000 7f62bc80a700 1 -- 172.21.15.28:0/1547424615 >> 172.21.15.28:0/1547424615 conn(0x7f62b80fc9f0 msgr2=0x7f62b80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.139+0000 7f62bc80a700 1 -- 172.21.15.28:0/1547424615 shutdown_connections 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.139+0000 7f62bc80a700 1 -- 172.21.15.28:0/1547424615 wait complete. 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 librados: starting msgr at 2024-03-30T15:05:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 librados: starting objecter 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 Processor -- start 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 -- start start 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 librados: setting wanted keys 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 librados: calling monclient init 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62b81a92b0 con 0x7f62b810d970 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62b81aa4d0 con 0x7f62b8109dd0 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62bc80a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f62b81ab710 con 0x7f62b8101140 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.140+0000 7f62b659c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f62b8101140 0x7f62b81062c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56370/0 (socket says 172.21.15.28:56370) 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62b659c700 1 -- 172.21.15.28:0/2649022070 learned_addr learned my addr 172.21.15.28:0/2649022070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter ms_handle_connect 0x7f62b8101140 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter ms_handle_connect 0x7f62b810d970 2024-03-30T15:05:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter ms_handle_connect 0x7f62b8109dd0 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747350071 0 0) 0x7f62b81ab710 con 0x7f62b8101140 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a40031f0 con 0x7f62b8101140 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130903147 0 0) 0x7f62b81a92b0 con 0x7f62b810d970 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62b81ab710 con 0x7f62b810d970 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604436404 0 0) 0x7f62b81aa4d0 con 0x7f62b8109dd0 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62b81a92b0 con 0x7f62b8109dd0 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 426222527 0 0) 0x7f62a40031f0 con 0x7f62b8101140 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62b81aa4d0 con 0x7f62b8101140 2024-03-30T15:05:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62a8001cb0 con 0x7f62b8101140 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3822899519 0 0) 0x7f62b81a92b0 con 0x7f62b8109dd0 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62a40031f0 con 0x7f62b8109dd0 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62a0001110 con 0x7f62b8109dd0 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1459271233 0 0) 0x7f62b81ab710 con 0x7f62b810d970 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62b81a92b0 con 0x7f62b810d970 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62b0004540 con 0x7f62b810d970 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1159438413 0 0) 0x7f62b81aa4d0 con 0x7f62b8101140 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.134:6789/0 conn(0x7f62b8109dd0 legacy=0x7f62b81a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6789/0 conn(0x7f62b810d970 legacy=0x7f62b81a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b81ac950 con 0x7f62b8101140 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62b81a94e0 con 0x7f62b8101140 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62bc80a700 10 client.58265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f62b81a99a0 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62bc80a700 1 librados: init done 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f62a8002080 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.141+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62a8006170 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.142+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6298005100 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f62a8003cd0 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(516..516 src has 442..516) ==== 8654+0+0 (unknown 3100275588 0 0) 0x7f62a8089e10 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 osd_map(516..516 src has 442..516) 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 3 client.58265.objecter handle_osd_map got epochs [516,516] > 0 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 3 client.58265.objecter handle_osd_map decoding full epoch 516 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 20 client.58265.objecter dump_active .. 0 homeless 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.143+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v516) ==== 54+0+16 (unknown 3317437023 0 2420000192) 0x7f62a808c270 con 0x7f62b8101140 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f62a40703d0 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 10 client.58265.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 1 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 1 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 0 1 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 4 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 10 client.58265.objecter _send_command 1 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6298004b50 con 0x7f6298004730 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.144+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f6298004730 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.145+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298004b50 con 0x7f6298004730 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.145+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 1: 0 ) 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.145+0000 7f62af7fe700 10 client.58265.objecter _finish_command 1 = system:0 2024-03-30T15:05:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.145+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 4 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.145+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 0 1 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 10 client.58265.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 2 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 2 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f62980053e0 osd=0 3 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 0 2 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 4 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 10 client.58265.objecter _send_command 2 2024-03-30T15:05:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.161+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6298007b00 con 0x7f6298004730 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6298007b00 con 0x7f6298004730 2024-03-30T15:05:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 10 client.58265.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 4 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 0 2 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 10 client.58265.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 1 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 4 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 10 client.58265.objecter _send_command 3 2024-03-30T15:05:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6298002160 con 0x7f62980042a0 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.163+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f62980042a0 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.164+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298002160 con 0x7f62980042a0 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.164+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 3: 0 ) 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.164+0000 7f62af7fe700 10 client.58265.objecter _finish_command 3 = system:0 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.164+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 4 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.164+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 1 3 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 10 client.58265.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 4 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 4 2024-03-30T15:05:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298004130 osd=1 3 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 1 4 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 4 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 10 client.58265.objecter _send_command 4 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.181+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6298001600 con 0x7f62980042a0 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6298001600 con 0x7f62980042a0 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62af7fe700 10 client.58265.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 4 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 1 4 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 10 client.58265.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 5 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 5 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 2 5 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 4 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 10 client.58265.objecter _send_command 5 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.182+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6298010fb0 con 0x7f6298008990 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f6298008990 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298010fb0 con 0x7f6298008990 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 5: 0 ) 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 10 client.58265.objecter _finish_command 5 = system:0 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 4 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.183+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 2 5 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 10 client.58265.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 6 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 6 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298001a00 osd=2 3 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 2 6 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 4 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 10 client.58265.objecter _send_command 6 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.200+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6298008000 con 0x7f6298008990 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6298008000 con 0x7f6298008990 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 10 client.58265.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 4 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 2 6 2024-03-30T15:05:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 10 client.58265.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 7 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 7 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 3 7 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 4 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 10 client.58265.objecter _send_command 7 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6298011d00 con 0x7f6298011190 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.202+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f6298011190 2024-03-30T15:05:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.203+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298011d00 con 0x7f6298011190 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.203+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 7: 0 ) 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.203+0000 7f62af7fe700 10 client.58265.objecter _finish_command 7 = system:0 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.203+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 4 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.203+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 3 7 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 10 client.58265.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 8 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 8 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298008290 osd=3 3 2024-03-30T15:05:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 3 8 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 4 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 10 client.58265.objecter _send_command 8 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.220+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62980126b0 con 0x7f6298011190 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f62980126b0 con 0x7f6298011190 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62af7fe700 10 client.58265.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 4 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 3 8 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 10 client.58265.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 9 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 9 2024-03-30T15:05:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 4 9 2024-03-30T15:05:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 4 2024-03-30T15:05:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 10 client.58265.objecter _send_command 9 2024-03-30T15:05:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.221+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f629801a680 con 0x7f6298016b70 2024-03-30T15:05:50.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:05:50.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:05:50.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:05:50.450 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:05:50.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.222+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f6298016b70 2024-03-30T15:05:50.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.223+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f629801a680 con 0x7f6298016b70 2024-03-30T15:05:50.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.223+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 9: 0 ) 2024-03-30T15:05:50.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.223+0000 7f62af7fe700 10 client.58265.objecter _finish_command 9 = system:0 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.223+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 4 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.223+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 4 9 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 10 client.58265.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 10 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 10 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298012530 osd=4 3 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 4 10 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 4 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 10 client.58265.objecter _send_command 10 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.239+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f629800c810 con 0x7f6298016b70 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f629800c810 con 0x7f6298016b70 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62af7fe700 10 client.58265.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 4 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 4 10 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 10 client.58265.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 11 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 11 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 5 11 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 4 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 10 client.58265.objecter _send_command 11 2024-03-30T15:05:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.241+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f629801bc90 con 0x7f629801b800 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f629801b800 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f629801bc90 con 0x7f629801b800 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 11: 0 ) 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 10 client.58265.objecter _finish_command 11 = system:0 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 4 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.242+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 5 11 2024-03-30T15:05:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 10 client.58265.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 12 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 12 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f629800c810 osd=5 3 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 5 12 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 4 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 10 client.58265.objecter _send_command 12 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.259+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f629801bc90 con 0x7f629801b800 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f629801bc90 con 0x7f629801b800 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62af7fe700 10 client.58265.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 4 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 5 12 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 10 client.58265.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 13 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 13 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 6 13 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 4 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 10 client.58265.objecter _send_command 13 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.261+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6298023040 con 0x7f629801f530 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.262+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f629801f530 2024-03-30T15:05:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.263+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298023040 con 0x7f629801f530 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.263+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 13: 0 ) 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.263+0000 7f62af7fe700 10 client.58265.objecter _finish_command 13 = system:0 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.263+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 4 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.263+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 6 13 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 10 client.58265.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 14 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 14 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298013920 osd=6 3 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 6 14 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 4 2024-03-30T15:05:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 10 client.58265.objecter _send_command 14 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.280+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6298023040 con 0x7f629801f530 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6298023040 con 0x7f629801f530 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 10 client.58265.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 4 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 6 14 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 10 client.58265.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 15 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 15 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 7 15 2024-03-30T15:05:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 4 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 10 client.58265.objecter _send_command 15 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6298024270 con 0x7f6298024f00 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.282+0000 7f62af7fe700 10 client.58265.objecter ms_handle_connect 0x7f6298024f00 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.283+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6298024270 con 0x7f6298024f00 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.283+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 15: 0 ) 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.283+0000 7f62af7fe700 10 client.58265.objecter _finish_command 15 = system:0 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.283+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 4 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.283+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 7 15 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 10 client.58265.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign -1 16 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 20 client.58265.objecter _get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_remove -1 16 2024-03-30T15:05:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 20 client.58265.objecter get_session s=0x7f6298023040 osd=7 3 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 15 client.58265.objecter _session_command_op_assign 7 16 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 4 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 10 client.58265.objecter _send_command 16 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.300+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62980244d0 con 0x7f6298024f00 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.301+0000 7f62af7fe700 1 -- 172.21.15.28:0/2649022070 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f62980244d0 con 0x7f6298024f00 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.301+0000 7f62af7fe700 10 client.58265.objecter ms_dispatch 0x7f62b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.301+0000 7f62af7fe700 10 client.58265.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.301+0000 7f62af7fe700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 4 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.301+0000 7f62af7fe700 15 client.58265.objecter _session_command_op_remove 7 16 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 librados: watch_flush enter 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 librados: watch_flush exit 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.0 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6801/3394323575 conn(0x7f6298004730 legacy=0x7f6298002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f62980053e0 osd=0 1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6805/1850313547 conn(0x7f62980042a0 legacy=0x7f629800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298004130 osd=1 1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.2 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6809/3451325043 conn(0x7f6298008990 legacy=0x7f629800f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298001a00 osd=2 1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.3 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6813/1506157250 conn(0x7f6298011190 legacy=0x7f62980115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298008290 osd=3 1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.4 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.134:6800/3063775514 conn(0x7f6298016b70 legacy=0x7f6298018f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298012530 osd=4 1 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.5 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.134:6804/3668151179 conn(0x7f629801b800 legacy=0x7f62980130e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f629800c810 osd=5 1 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.6 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62b6d9d700 1 -- 172.21.15.28:0/2649022070 reap_dead start 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.134:6808/2135224194 conn(0x7f629801f530 legacy=0x7f6298021920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62b6d9d700 1 -- 172.21.15.28:0/2649022070 reap_dead start 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298013920 osd=6 1 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter close_session for osd.7 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.134:6812/3758190298 conn(0x7f6298024f00 legacy=0x7f6298025360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter put_session s=0x7f6298023040 osd=7 1 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 20 client.58265.objecter shutdown clearing up homeless session... 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 10 client.58265.objecter successfully canceled tick 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62b6d9d700 1 -- 172.21.15.28:0/2649022070 reap_dead start 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6800/3011431093 conn(0x7f62a40703d0 legacy=0x7f62a4072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> v1:172.21.15.28:6790/0 conn(0x7f62b8101140 legacy=0x7f62b81062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 shutdown_connections 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 >> 172.21.15.28:0/2649022070 conn(0x7f62b80fc9f0 msgr2=0x7f62b80fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 shutdown_connections 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 -- 172.21.15.28:0/2649022070 wait complete. 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:50.302+0000 7f62bc80a700 1 librados: shutdown 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:05:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:05:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:50.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:50.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:50.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:50.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: pgmap v749: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2649022070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:50.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[29876]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: osdmap e516: 8 total, 8 up, 8 in 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:50.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: pgmap v749: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:50.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2649022070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:05:50.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:05:50.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:50 smithi028 ceph-mon[41614]: osdmap e517: 8 total, 8 up, 8 in 2024-03-30T15:05:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:50.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:50.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:05:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:05:50.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:50.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:50.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:51.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:51.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:51.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:51.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:51.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:51.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:51.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:51.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:51.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:51.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:05:51.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:51.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:51.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:51.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:51.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:52.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:52.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:52.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:52.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:52.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:52.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:52.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:52.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[29876]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[29876]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[29876]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[41614]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[41614]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:52.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:52 smithi028 ceph-mon[41614]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T15:05:52.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:52.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:52 smithi134 ceph-mon[33714]: osdmap e518: 8 total, 8 up, 8 in 2024-03-30T15:05:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:52 smithi134 ceph-mon[33714]: pgmap v752: 260 pgs: 260 active+clean; 8.4 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-30T15:05:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:05:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:52 smithi134 ceph-mon[33714]: osdmap e519: 8 total, 8 up, 8 in 2024-03-30T15:05:52.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:52.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:52.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:52.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:52.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:53.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:53.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:53.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:53.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.255 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:53.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:53.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:53.382 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:53.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:53.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:53.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:53.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:05:53.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:53 smithi028 ceph-mon[41614]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T15:05:53.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:05:53.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:53 smithi028 ceph-mon[29876]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T15:05:53.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:05:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:53 smithi134 ceph-mon[33714]: osdmap e520: 8 total, 8 up, 8 in 2024-03-30T15:05:53.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:53.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:53.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:53.886 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:53.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 -- 172.21.15.28:0/1549157297 >> v1:172.21.15.28:6789/0 conn(0x7fe650101130 legacy=0x7fe650101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 -- 172.21.15.28:0/1549157297 shutdown_connections 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 -- 172.21.15.28:0/1549157297 >> 172.21.15.28:0/1549157297 conn(0x7fe6500fc9a0 msgr2=0x7fe6500fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 -- 172.21.15.28:0/1549157297 shutdown_connections 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 -- 172.21.15.28:0/1549157297 wait complete. 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 librados: starting msgr at 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.897+0000 7fe655fc5700 1 librados: starting objecter 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 Processor -- start 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 -- start start 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 librados: setting wanted keys 2024-03-30T15:05:53.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 librados: calling monclient init 2024-03-30T15:05:53.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6501068b0 con 0x7fe650101130 2024-03-30T15:05:53.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6501a8540 con 0x7fe650109dc0 2024-03-30T15:05:53.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe655fc5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6501a9720 con 0x7fe65010d960 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64f7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe650101130 0x7fe650104680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33640/0 (socket says 172.21.15.28:33640) 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64f7fe700 1 -- 172.21.15.28:0/421748506 learned_addr learned my addr 172.21.15.28:0/421748506 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter ms_handle_connect 0x7fe65010d960 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter ms_handle_connect 0x7fe650101130 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter ms_handle_connect 0x7fe650109dc0 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519020670 0 0) 0x7fe6501068b0 con 0x7fe650101130 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6380031f0 con 0x7fe650101130 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497287607 0 0) 0x7fe6501a9720 con 0x7fe65010d960 2024-03-30T15:05:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6501068b0 con 0x7fe65010d960 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.898+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838435013 0 0) 0x7fe6501a8540 con 0x7fe650109dc0 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6501a9720 con 0x7fe650109dc0 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1189746029 0 0) 0x7fe6380031f0 con 0x7fe650101130 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6501a8540 con 0x7fe650101130 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe640004580 con 0x7fe650101130 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2450591548 0 0) 0x7fe6501a9720 con 0x7fe650109dc0 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6380031f0 con 0x7fe650109dc0 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2412196230 0 0) 0x7fe6501068b0 con 0x7fe65010d960 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6501a9720 con 0x7fe65010d960 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe6440017f0 con 0x7fe650109dc0 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe648001470 con 0x7fe65010d960 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3665697843 0 0) 0x7fe6501a8540 con 0x7fe650101130 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 >> v1:172.21.15.28:6790/0 conn(0x7fe65010d960 legacy=0x7fe6501a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 >> v1:172.21.15.134:6789/0 conn(0x7fe650109dc0 legacy=0x7fe650104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6501aa900 con 0x7fe650101130 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6501a7590 con 0x7fe650101130 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe655fc5700 10 client.25071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6501a7a50 con 0x7fe650101130 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe655fc5700 1 librados: init done 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe640001ac0 con 0x7fe650101130 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.899+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe640004e00 con 0x7fe650101130 2024-03-30T15:05:53.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.900+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe63c005120 con 0x7fe650101130 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.901+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe640003680 con 0x7fe650101130 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.901+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(520..520 src has 442..520) ==== 8689+0+0 (unknown 3744318779 0 0) 0x7fe6400899c0 con 0x7fe650101130 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.901+0000 7fe64cff9700 10 client.25071.objecter ms_dispatch 0x7fe650001550 osd_map(520..520 src has 442..520) 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.901+0000 7fe64cff9700 3 client.25071.objecter handle_osd_map got epochs [520,520] > 0 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.901+0000 7fe64cff9700 3 client.25071.objecter handle_osd_map decoding full epoch 520 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.902+0000 7fe64cff9700 20 client.25071.objecter dump_active .. 0 homeless 2024-03-30T15:05:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.902+0000 7fe64cff9700 10 client.25071.objecter ms_handle_connect 0x7fe638070430 2024-03-30T15:05:53.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:53.906+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe64008be20 con 0x7fe650101130 2024-03-30T15:05:53.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:54.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:54.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.121+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fe63c005410 con 0x7fe650101130 2024-03-30T15:05:54.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:54.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:54.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:54.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.349+0000 7fe64cff9700 1 -- 172.21.15.28:0/421748506 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v521) ==== 91+0+0 (unknown 4027394053 0 0) 0x7fe6400569c0 con 0x7fe650101130 2024-03-30T15:05:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.352+0000 7fe655fc5700 10 librados: watch_flush enter 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.352+0000 7fe655fc5700 10 librados: watch_flush exit 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.352+0000 7fe655fc5700 20 client.25071.objecter shutdown clearing up homeless session... 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.352+0000 7fe655fc5700 10 client.25071.objecter successfully canceled tick 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.353+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe638070430 legacy=0x7fe6380728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.353+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 >> v1:172.21.15.28:6789/0 conn(0x7fe650101130 legacy=0x7fe650104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.354+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 shutdown_connections 2024-03-30T15:05:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.354+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 >> 172.21.15.28:0/421748506 conn(0x7fe6500fc9a0 msgr2=0x7fe650110710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.355+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 shutdown_connections 2024-03-30T15:05:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.356+0000 7fe655fc5700 1 -- 172.21.15.28:0/421748506 wait complete. 2024-03-30T15:05:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:54.356+0000 7fe655fc5700 1 librados: shutdown 2024-03-30T15:05:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:05:54.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:54.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:54.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:54.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:54.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi134 to 3248M 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-03-30T15:05:54.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi134 to 3248M 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-83"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi134 to 3248M 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: pgmap v755: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-03-30T15:05:54.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:05:54.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:54.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:05:54.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:54.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:54.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:54.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:55.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:55.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:55.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:55.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:55.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:55.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: Adjusting osd_memory_target on smithi028 to 3249M 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:55.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: Adjusting osd_memory_target on smithi028 to 3249M 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:05:55.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:55 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:55.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/421748506' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: osdmap e521: 8 total, 8 up, 8 in 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: Adjusting osd_memory_target on smithi028 to 3249M 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:05:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:55 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:05:55.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:55.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:55.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:56.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:56.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:56.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:05:56.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:56.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:56.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:56.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:56.507 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:05:56.507 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:05:56.507 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:05:56.507 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:05:56.508 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[41614]: pgmap v758: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:05:56.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:56 smithi028 ceph-mon[29876]: pgmap v758: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T15:05:56.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:56.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:56.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: osdmap e522: 8 total, 8 up, 8 in 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:05:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:56 smithi134 ceph-mon[33714]: pgmap v758: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-30T15:05:56.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:56.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:56.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:56.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:57.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:57.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 -- 172.21.15.28:0/3427273048 >> v1:172.21.15.28:6790/0 conn(0x7f1fdc10d960 legacy=0x7f1fdc10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 -- 172.21.15.28:0/3427273048 shutdown_connections 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 -- 172.21.15.28:0/3427273048 >> 172.21.15.28:0/3427273048 conn(0x7f1fdc0fc9e0 msgr2=0x7f1fdc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 -- 172.21.15.28:0/3427273048 shutdown_connections 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 -- 172.21.15.28:0/3427273048 wait complete. 2024-03-30T15:05:57.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 librados: starting msgr at 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.145+0000 7f1fe3e75700 1 librados: starting objecter 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 Processor -- start 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 -- start start 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 librados: setting wanted keys 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 librados: calling monclient init 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fdc1069c0 con 0x7f1fdc101130 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fdc1a8540 con 0x7f1fdc109dc0 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe3e75700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fdc1a9760 con 0x7f1fdc10d960 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe1c11700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1fdc101130 0x7f1fdc1047e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33670/0 (socket says 172.21.15.28:33670) 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fe1c11700 1 -- 172.21.15.28:0/752293055 learned_addr learned my addr 172.21.15.28:0/752293055 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter ms_handle_connect 0x7f1fdc101130 2024-03-30T15:05:57.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter ms_handle_connect 0x7f1fdc10d960 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter ms_handle_connect 0x7f1fdc109dc0 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371998584 0 0) 0x7f1fdc1069c0 con 0x7f1fdc101130 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc40031f0 con 0x7f1fdc101130 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630679157 0 0) 0x7f1fdc1a9760 con 0x7f1fdc10d960 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fdc1069c0 con 0x7f1fdc10d960 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 545049938 0 0) 0x7f1fdc1a8540 con 0x7f1fdc109dc0 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fdc1a9760 con 0x7f1fdc109dc0 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4215645467 0 0) 0x7f1fc40031f0 con 0x7f1fdc101130 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fdc1a8540 con 0x7f1fdc101130 2024-03-30T15:05:57.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.146+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fcc001d30 con 0x7f1fdc101130 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1007496194 0 0) 0x7f1fdc1069c0 con 0x7f1fdc10d960 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fc40031f0 con 0x7f1fdc10d960 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fd4004500 con 0x7f1fdc10d960 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3536911891 0 0) 0x7f1fdc1a9760 con 0x7f1fdc109dc0 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fdc1069c0 con 0x7f1fdc109dc0 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fd0001830 con 0x7f1fdc109dc0 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 340556313 0 0) 0x7f1fdc1a8540 con 0x7f1fdc101130 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 >> v1:172.21.15.28:6790/0 conn(0x7f1fdc10d960 legacy=0x7f1fdc1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 >> v1:172.21.15.134:6789/0 conn(0x7f1fdc109dc0 legacy=0x7f1fdc104ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fdc1aa980 con 0x7f1fdc101130 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fdc1a9990 con 0x7f1fdc101130 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fe3e75700 10 client.25077.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:05:57.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fdc1a9e50 con 0x7f1fdc101130 2024-03-30T15:05:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fe3e75700 1 librados: init done 2024-03-30T15:05:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1fcc003ea0 con 0x7f1fdc101130 2024-03-30T15:05:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.147+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fcc006050 con 0x7f1fdc101130 2024-03-30T15:05:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.148+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc8005120 con 0x7f1fdc101130 2024-03-30T15:05:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1fcc003a00 con 0x7f1fdc101130 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(523..523 src has 442..523) ==== 8622+0+0 (unknown 142900242 0 0) 0x7f1fcc089cc0 con 0x7f1fdc101130 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 10 client.25077.objecter ms_dispatch 0x7f1fdc001550 osd_map(523..523 src has 442..523) 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 3 client.25077.objecter handle_osd_map got epochs [523,523] > 0 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 3 client.25077.objecter handle_osd_map decoding full epoch 523 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 20 client.25077.objecter dump_active .. 0 homeless 2024-03-30T15:05:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.149+0000 7f1fdaffd700 10 client.25077.objecter ms_handle_connect 0x7f1fc40704e0 2024-03-30T15:05:57.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.154+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1fcc056d00 con 0x7f1fdc101130 2024-03-30T15:05:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:57.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.374+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f1fc8005410 con 0x7f1fdc101130 2024-03-30T15:05:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.376+0000 7f1fdaffd700 1 -- 172.21.15.28:0/752293055 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v524) ==== 81+0+56513 (unknown 1017269087 0 2743429802) 0x7f1fcc05a750 con 0x7f1fdc101130 2024-03-30T15:05:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:05:57.378 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:05:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 524, 2024-03-30T15:05:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:05:57.378 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:05:57.359045+0000", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T14:54:22.403545+0000", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 112, 2024-03-30T15:05:57.379 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 307, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:05:57.380 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-03-30T15:05:57.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "437", 2024-03-30T15:05:57.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.383 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.384 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.385 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.389 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.390 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:05:57.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:05:57.392 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.394 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.396 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.397 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.398 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.399 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.400 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.401 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.402 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.404 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:05:57.405 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.407 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.410 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.25, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.25, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.25, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.25, 2024-03-30T15:05:57.411 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:05:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:05:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:05:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.429 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 101, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-6", 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:58:12.352170+0000", 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.430 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.431 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "491", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "411", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "411", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "411", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.432 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 26, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 491, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.434 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.435 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 307, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-85", 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:05:54.624042+0000", 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:05:57.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:05:57.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "524", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:05:57.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:05:57.439 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:05:57.440 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "fingerprint_algorithm": "sha1" 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:05:57.441 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.442 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.443 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 10 librados: watch_flush enter 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 10 librados: watch_flush exit 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 20 client.25077.objecter shutdown clearing up homeless session... 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 10 client.25077.objecter successfully canceled tick 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1fc40704e0 legacy=0x7f1fc40729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 >> v1:172.21.15.28:6789/0 conn(0x7f1fdc101130 legacy=0x7f1fdc1047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 shutdown_connections 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 >> 172.21.15.28:0/752293055 conn(0x7f1fdc0fc9e0 msgr2=0x7f1fdc110a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 shutdown_connections 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 -- 172.21.15.28:0/752293055 wait complete. 2024-03-30T15:05:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:05:57.379+0000 7f1fe3e75700 1 librados: shutdown 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:05:57.447 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.449 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:05:57.450 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.453 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.455 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.456 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.457 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 417, 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.460 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:05:57.461 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.463 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.465 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:05:57.468 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:05:57.469 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.470 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:57.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.472 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.473 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:05:57.474 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 522, 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:05:57.475 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:57.476 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:05:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:04:34.941959+0000", 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.479 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:42.551928+0000", 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:14.412793+0000", 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:21.119206+0000", 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.481 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:33.347903+0000", 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:19.162669+0000", 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:18.056823+0000", 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:05:50.488310+0000", 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.484 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.17", 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:05:57.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:05:57.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.487 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:05:57.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.489 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:05:57.490 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.491 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.492 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.494 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:05:57.501 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.502 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:05:57.503 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.504 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.505 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.506 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:05:57.507 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:05:57.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:05:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:05:57.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:57 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:05:57.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:57.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T15:05:57.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: osdmap e523: 8 total, 8 up, 8 in 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.712 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:57 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:05:57.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:57.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:57.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:05:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:57.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:05:58.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:58.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:58.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:58.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:58.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:05:58.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:58.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:58.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:58.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:05:58.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:05:58.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 101.16 scrub starts 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 101.16 scrub ok 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/752293055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:05:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:05:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:58 smithi134 ceph-mon[33714]: pgmap v761: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 101.16 scrub starts 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 101.16 scrub ok 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/752293055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:05:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[29876]: pgmap v761: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:05:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 101.16 scrub starts 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 101.16 scrub ok 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: osdmap e524: 8 total, 8 up, 8 in 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:05:58.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/752293055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:05:58.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:05:58.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:05:58.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:58 smithi028 ceph-mon[41614]: pgmap v761: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:05:58.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:05:58.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:05:58.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:05:58.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:58.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:58.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:59.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:59.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:59.163 INFO:tasks.thrashosds.thrasher:j is {'epoch': 524, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:05:57.359045+0000', 'last_up_change': '2024-03-30T14:54:22.403545+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 112, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 307, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '437', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 101, 'pool_name': 'test-rados-api-smithi028-93253-6', 'create_time': '2024-03-30T14:58:12.352170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '491', 'last_force_op_resend': '411', 'last_force_op_resend_prenautilus': '411', 'last_force_op_resend_preluminous': '411', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 491, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 307, 'pool_name': 'test-rados-api-smithi028-93253-85', 'create_time': '2024-03-30T15:05:54.624042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '524', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:04:34.941959+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:42.551928+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:14.412793+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:21.119206+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:33.347903+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:19.162669+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:18.056823+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:05:50.488310+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.17', 'osds': [6, 7, 1]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}, {'pgid': '5.13', 'mappings': [{'from': 3, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:05:59.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:05:59.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:05:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:05:59.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:59.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:05:59.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T15:05:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:05:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:05:59 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:05:59.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:05:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:05:59.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:05:59.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:05:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:05:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:05:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_tier","val": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:05:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: osdmap e525: 8 total, 8 up, 8 in 2024-03-30T15:05:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:05:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:05:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:05:59 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:05:59.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:59.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:59.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:05:59.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:05:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:05:59.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:05:59.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:00.002 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:00.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:00.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:00.151 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.170+0000 7fe735ffb700 1 -- 172.21.15.28:0/764724801 <== mon.2 v1:172.21.15.28:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe7280046a0 con 0x7fe738101870 2024-03-30T15:06:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.172+0000 7fe73d656700 1 -- 172.21.15.28:0/764724801 >> v1:172.21.15.28:6790/0 conn(0x7fe738101870 legacy=0x7fe738101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.173+0000 7fe73d656700 1 -- 172.21.15.28:0/764724801 shutdown_connections 2024-03-30T15:06:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.173+0000 7fe73d656700 1 -- 172.21.15.28:0/764724801 >> 172.21.15.28:0/764724801 conn(0x7fe738077370 msgr2=0x7fe7380fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.173+0000 7fe73d656700 1 -- 172.21.15.28:0/764724801 shutdown_connections 2024-03-30T15:06:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.173+0000 7fe73d656700 1 -- 172.21.15.28:0/764724801 wait complete. 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 librados: starting msgr at 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 librados: starting objecter 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 Processor -- start 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 -- start start 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 librados: setting wanted keys 2024-03-30T15:06:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.174+0000 7fe73d656700 1 librados: calling monclient init 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe73d656700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7381a5180 con 0x7fe738109fe0 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe73d656700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7381a63a0 con 0x7fe738101870 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe73d656700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7381a75e0 con 0x7fe73810db80 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe7377fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe73810db80 0x7fe7381a3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54890/0 (socket says 172.21.15.28:54890) 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe7377fe700 1 -- 172.21.15.28:0/678729043 learned_addr learned my addr 172.21.15.28:0/678729043 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter ms_handle_connect 0x7fe738109fe0 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter ms_handle_connect 0x7fe73810db80 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298160135 0 0) 0x7fe7381a5180 con 0x7fe738109fe0 2024-03-30T15:06:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7200035b0 con 0x7fe738109fe0 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter ms_handle_connect 0x7fe738101870 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2024149064 0 0) 0x7fe7381a75e0 con 0x7fe73810db80 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7381a5180 con 0x7fe73810db80 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4234138539 0 0) 0x7fe7381a63a0 con 0x7fe738101870 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7381a75e0 con 0x7fe738101870 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4033866932 0 0) 0x7fe7200035b0 con 0x7fe738109fe0 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7381a63a0 con 0x7fe738109fe0 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe72c0017e0 con 0x7fe738109fe0 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1104921641 0 0) 0x7fe7381a5180 con 0x7fe73810db80 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7200035b0 con 0x7fe73810db80 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.175+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe730001bb0 con 0x7fe73810db80 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3603835081 0 0) 0x7fe7381a63a0 con 0x7fe738109fe0 2024-03-30T15:06:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6790/0 conn(0x7fe73810db80 legacy=0x7fe7381a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.134:6789/0 conn(0x7fe738101870 legacy=0x7fe738106560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7381a8820 con 0x7fe738109fe0 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7381a7810 con 0x7fe738109fe0 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe73d656700 10 client.25083.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7381a7cd0 con 0x7fe738109fe0 2024-03-30T15:06:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe73d656700 1 librados: init done 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe72c003b20 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.176+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe72c004a00 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.177+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe724005100 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.178+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe72c003680 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.178+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(526..526 src has 442..526) ==== 8673+0+0 (unknown 3844904181 0 0) 0x7fe72c088830 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.178+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 osd_map(526..526 src has 442..526) 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.178+0000 7fe71ffff700 3 client.25083.objecter handle_osd_map got epochs [526,526] > 0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.178+0000 7fe71ffff700 3 client.25083.objecter handle_osd_map decoding full epoch 526 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe71ffff700 20 client.25083.objecter dump_active .. 0 homeless 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v526) ==== 54+0+16 (unknown 2265631301 0 2420000192) 0x7fe72c088c20 con 0x7fe738109fe0 2024-03-30T15:06:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe720074860 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 10 client.25083.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 1 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 1 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 0 1 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 4 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 10 client.25083.objecter _send_command 1 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe724004b50 con 0x7fe724004730 2024-03-30T15:06:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.179+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe724004730 2024-03-30T15:06:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.180+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe724004b50 con 0x7fe724004730 2024-03-30T15:06:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.180+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.180+0000 7fe71ffff700 10 client.25083.objecter _finish_command 1 = system:0 2024-03-30T15:06:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.180+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 4 2024-03-30T15:06:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.180+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 0 1 2024-03-30T15:06:00.194 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 10 client.25083.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 2 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 2 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe7240053e0 osd=0 3 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 0 2 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 4 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 10 client.25083.objecter _send_command 2 2024-03-30T15:06:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.204+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe724007b00 con 0x7fe724004730 2024-03-30T15:06:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe724007b00 con 0x7fe724004730 2024-03-30T15:06:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe71ffff700 10 client.25083.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 4 2024-03-30T15:06:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 0 2 2024-03-30T15:06:00.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 10 client.25083.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 1 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 4 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 10 client.25083.objecter _send_command 3 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.206+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe724002160 con 0x7fe7240042a0 2024-03-30T15:06:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe7240042a0 2024-03-30T15:06:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe724002160 con 0x7fe7240042a0 2024-03-30T15:06:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 10 client.25083.objecter _finish_command 3 = system:0 2024-03-30T15:06:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 4 2024-03-30T15:06:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.207+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 1 3 2024-03-30T15:06:00.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 10 client.25083.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 4 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 4 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724004130 osd=1 3 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 1 4 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 4 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 10 client.25083.objecter _send_command 4 2024-03-30T15:06:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.225+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe724001600 con 0x7fe7240042a0 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe724001600 con 0x7fe7240042a0 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 10 client.25083.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 4 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 1 4 2024-03-30T15:06:00.228 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 10 client.25083.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 5 2024-03-30T15:06:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 5 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 2 5 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 4 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 10 client.25083.objecter _send_command 5 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe724011010 con 0x7fe724008990 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.227+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe724008990 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.228+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe724011010 con 0x7fe724008990 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.228+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.228+0000 7fe71ffff700 10 client.25083.objecter _finish_command 5 = system:0 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.228+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 4 2024-03-30T15:06:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.228+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 2 5 2024-03-30T15:06:00.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:00.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 10 client.25083.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 6 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 6 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724001a00 osd=2 3 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 2 6 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 4 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 10 client.25083.objecter _send_command 6 2024-03-30T15:06:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.245+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe724008020 con 0x7fe724008990 2024-03-30T15:06:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe724008020 con 0x7fe724008990 2024-03-30T15:06:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe71ffff700 10 client.25083.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 4 2024-03-30T15:06:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 2 6 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 10 client.25083.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 7 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 7 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 3 7 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 4 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 10 client.25083.objecter _send_command 7 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.247+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe724011d10 con 0x7fe7240111c0 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe7240111c0 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe724011d10 con 0x7fe7240111c0 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 10 client.25083.objecter _finish_command 7 = system:0 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 4 2024-03-30T15:06:00.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.248+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 3 7 2024-03-30T15:06:00.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 10 client.25083.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 8 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 8 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724008270 osd=3 3 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 3 8 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 4 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 10 client.25083.objecter _send_command 8 2024-03-30T15:06:00.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.266+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7240126c0 con 0x7fe7240111c0 2024-03-30T15:06:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7240126c0 con 0x7fe7240111c0 2024-03-30T15:06:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe71ffff700 10 client.25083.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 4 2024-03-30T15:06:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 3 8 2024-03-30T15:06:00.269 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 10 client.25083.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 9 2024-03-30T15:06:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 9 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 4 9 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 4 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 10 client.25083.objecter _send_command 9 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.267+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe72401a830 con 0x7fe724016bd0 2024-03-30T15:06:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.268+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe724016bd0 2024-03-30T15:06:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.269+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe72401a830 con 0x7fe724016bd0 2024-03-30T15:06:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.269+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.269+0000 7fe71ffff700 10 client.25083.objecter _finish_command 9 = system:0 2024-03-30T15:06:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.269+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 4 2024-03-30T15:06:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.269+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 4 9 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 10 client.25083.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 10 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 10 2024-03-30T15:06:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe7240126c0 osd=4 3 2024-03-30T15:06:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 4 10 2024-03-30T15:06:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 4 2024-03-30T15:06:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 10 client.25083.objecter _send_command 10 2024-03-30T15:06:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.286+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe72401aa10 con 0x7fe724016bd0 2024-03-30T15:06:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe72401aa10 con 0x7fe724016bd0 2024-03-30T15:06:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe71ffff700 10 client.25083.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 4 2024-03-30T15:06:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 4 10 2024-03-30T15:06:00.291 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 10 client.25083.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 11 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 11 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 5 11 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 4 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 10 client.25083.objecter _send_command 11 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.289+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe72401bdd0 con 0x7fe72401b130 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe72401b130 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe72401bdd0 con 0x7fe72401b130 2024-03-30T15:06:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 10 client.25083.objecter _finish_command 11 = system:0 2024-03-30T15:06:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 4 2024-03-30T15:06:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.290+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 5 11 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 10 client.25083.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 12 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 12 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe72401aa10 osd=5 3 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 5 12 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 4 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 10 client.25083.objecter _send_command 12 2024-03-30T15:06:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.307+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe72401c030 con 0x7fe72401b130 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe72401c030 con 0x7fe72401b130 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 10 client.25083.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 4 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 5 12 2024-03-30T15:06:00.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 10 client.25083.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 13 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 13 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 6 13 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 4 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 10 client.25083.objecter _send_command 13 2024-03-30T15:06:00.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe7240232c0 con 0x7fe72401f710 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.310+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe72401f710 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.311+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7240232c0 con 0x7fe72401f710 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.311+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.311+0000 7fe71ffff700 10 client.25083.objecter _finish_command 13 = system:0 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.311+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 4 2024-03-30T15:06:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.311+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 6 13 2024-03-30T15:06:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 10 client.25083.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 14 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 14 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe72401c070 osd=6 3 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 6 14 2024-03-30T15:06:00.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 4 2024-03-30T15:06:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 10 client.25083.objecter _send_command 14 2024-03-30T15:06:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.328+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7240232c0 con 0x7fe72401f710 2024-03-30T15:06:00.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7240232c0 con 0x7fe72401f710 2024-03-30T15:06:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe71ffff700 10 client.25083.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 4 2024-03-30T15:06:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 6 14 2024-03-30T15:06:00.332 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 10 client.25083.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 15 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 15 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 7 15 2024-03-30T15:06:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 4 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 10 client.25083.objecter _send_command 15 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.330+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe724024580 con 0x7fe7240251c0 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.331+0000 7fe71ffff700 10 client.25083.objecter ms_handle_connect 0x7fe7240251c0 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.332+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe724024580 con 0x7fe7240251c0 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.332+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.332+0000 7fe71ffff700 10 client.25083.objecter _finish_command 15 = system:0 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.332+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 4 2024-03-30T15:06:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.332+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 7 15 2024-03-30T15:06:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 10 client.25083.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign -1 16 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 20 client.25083.objecter _get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 15 client.25083.objecter _session_command_op_remove -1 16 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 20 client.25083.objecter get_session s=0x7fe724023300 osd=7 3 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 15 client.25083.objecter _session_command_op_assign 7 16 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 4 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 10 client.25083.objecter _send_command 16 2024-03-30T15:06:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.348+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe724024580 con 0x7fe7240251c0 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe71ffff700 1 -- 172.21.15.28:0/678729043 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe724024580 con 0x7fe7240251c0 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe71ffff700 10 client.25083.objecter ms_dispatch 0x7fe738001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe71ffff700 10 client.25083.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe71ffff700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 4 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe71ffff700 15 client.25083.objecter _session_command_op_remove 7 16 2024-03-30T15:06:00.352 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 librados: watch_flush enter 2024-03-30T15:06:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 librados: watch_flush exit 2024-03-30T15:06:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.0 2024-03-30T15:06:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6801/3394323575 conn(0x7fe724004730 legacy=0x7fe724002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240053e0 osd=0 1 2024-03-30T15:06:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.1 2024-03-30T15:06:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6805/1850313547 conn(0x7fe7240042a0 legacy=0x7fe72400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724004130 osd=1 1 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.2 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6809/3451325043 conn(0x7fe724008990 legacy=0x7fe72400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724001a00 osd=2 1 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.3 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6813/1506157250 conn(0x7fe7240111c0 legacy=0x7fe724011600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724008270 osd=3 1 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.4 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.134:6800/3063775514 conn(0x7fe724016bd0 legacy=0x7fe724018fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe7240126c0 osd=4 1 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.5 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.134:6804/3668151179 conn(0x7fe72401b130 legacy=0x7fe72401b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401aa10 osd=5 1 2024-03-30T15:06:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.6 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.134:6808/2135224194 conn(0x7fe72401f710 legacy=0x7fe724021b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe7377fe700 1 -- 172.21.15.28:0/678729043 reap_dead start 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe72401c070 osd=6 1 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter close_session for osd.7 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.134:6812/3758190298 conn(0x7fe7240251c0 legacy=0x7fe724025620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe7377fe700 1 -- 172.21.15.28:0/678729043 reap_dead start 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe7377fe700 1 -- 172.21.15.28:0/678729043 reap_dead start 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe7377fe700 1 -- 172.21.15.28:0/678729043 reap_dead start 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter put_session s=0x7fe724023300 osd=7 1 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 20 client.25083.objecter shutdown clearing up homeless session... 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 10 client.25083.objecter successfully canceled tick 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe720074860 legacy=0x7fe720076d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> v1:172.21.15.28:6789/0 conn(0x7fe738109fe0 legacy=0x7fe73819ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 shutdown_connections 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 >> 172.21.15.28:0/678729043 conn(0x7fe738077370 msgr2=0x7fe738110d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.351+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 shutdown_connections 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.352+0000 7fe73d656700 1 -- 172.21.15.28:0/678729043 wait complete. 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:00.352+0000 7fe73d656700 1 librados: shutdown 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:00.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:00.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:00.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:06:00.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:06:00.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 101.2 scrub starts 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 101.2 scrub ok 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:06:00.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/678729043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 101.2 scrub starts 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 101.2 scrub ok 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T15:06:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/678729043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:00.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:00.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:00.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:00.649 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:00.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:06:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 101.2 scrub starts 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 101.2 scrub ok 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: osdmap e526: 8 total, 8 up, 8 in 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: pgmap v764: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/678729043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:00.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:00.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:00.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:00.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:01.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:01.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:01.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:01.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:01.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:01.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:06:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:06:01.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T15:06:01.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T15:06:01.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:01 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T15:06:01.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:01.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: osdmap e527: 8 total, 8 up, 8 in 2024-03-30T15:06:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:06:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:06:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T15:06:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:01 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T15:06:01.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:01.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:01.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:01.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:01.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:02.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:02.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:02.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:02.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:02.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:02.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:02.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:02.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:02.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:02.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 101.15 scrub starts 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 101.15 scrub ok 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 101.1c scrub starts 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 101.1c scrub ok 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:06:02.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 101.15 scrub starts 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 101.15 scrub ok 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:06:02.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 101.1c scrub starts 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 101.1c scrub ok 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:02 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 101.15 scrub starts 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 101.15 scrub ok 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: osdmap e528: 8 total, 8 up, 8 in 2024-03-30T15:06:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 101.1c scrub starts 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 101.1c scrub ok 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: pgmap v767: 260 pgs: 260 active+clean; 8.4 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: osdmap e529: 8 total, 8 up, 8 in 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:02 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-85"}]: dispatch 2024-03-30T15:06:02.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:02.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:02.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:02.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:02.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:03.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:03.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:03.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:03.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:03.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:03.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:03.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:03.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:03.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:03.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:03.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:03.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:03.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:06:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:03 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:06:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:03 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:06:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:03 smithi134 ceph-mon[33714]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[41614]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:06:03.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:03 smithi028 ceph-mon[29876]: osdmap e530: 8 total, 8 up, 8 in 2024-03-30T15:06:03.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:03.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:03.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:04.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:04.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:04.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:04.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:04.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:04.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:04.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:04.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:04.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:04.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:04.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:04.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:04.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: 101.1f scrub starts 2024-03-30T15:06:04.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: 101.1f scrub ok 2024-03-30T15:06:04.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: 101.1e scrub starts 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: 101.1e scrub ok 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: 101.1f scrub starts 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: 101.1f scrub ok 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: 101.1e scrub starts 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: 101.1e scrub ok 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T15:06:04.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.836 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:04 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:04.882 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:06:04.882 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: 101.1f scrub starts 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: 101.1f scrub ok 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: 101.1e scrub starts 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: 101.1e scrub ok 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: osdmap e531: 8 total, 8 up, 8 in 2024-03-30T15:06:04.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:04 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:04.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:04.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:05.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:05.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:05.327 INFO:tasks.workunit.client.0.smithi028.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17998 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22069 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2937 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3011 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3019 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3003 ms) 2024-03-30T15:06:05.328 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3980 ms) 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40035 ms) 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16368 ms) 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3614 ms) 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3005 ms) 2024-03-30T15:06:05.329 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25079 ms) 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14701 ms) 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4216 ms) 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4020 ms) 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T15:06:05.330 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: done waiting 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24085 ms) 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9920 ms) 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9041 ms) 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8962 ms) 2024-03-30T15:06:05.331 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-30T15:06:05.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:05.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:05.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:05.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:05.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:05.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:05.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:05.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:05.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:05.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T15:06:05.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T15:06:05.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:05.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[41614]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[29876]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T15:06:05.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:05.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:06:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:05 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T15:06:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:05 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T15:06:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:05 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:05 smithi134 ceph-mon[33714]: osdmap e532: 8 total, 8 up, 8 in 2024-03-30T15:06:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:05.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:06.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:06.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:06.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:06.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:06.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:06.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 -- 172.21.15.28:0/1220798778 >> v1:172.21.15.28:6790/0 conn(0x7f0268100da0 legacy=0x7f0268101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 -- 172.21.15.28:0/1220798778 shutdown_connections 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 -- 172.21.15.28:0/1220798778 >> 172.21.15.28:0/1220798778 conn(0x7f02680fc9e0 msgr2=0x7f02680fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 -- 172.21.15.28:0/1220798778 shutdown_connections 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 -- 172.21.15.28:0/1220798778 wait complete. 2024-03-30T15:06:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 librados: starting msgr at 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.498+0000 7f026f549700 1 librados: starting objecter 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 Processor -- start 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 -- start start 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 librados: setting wanted keys 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 librados: calling monclient init 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f026810f7a0 con 0x7f0268110c70 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02681ae900 con 0x7f0268100da0 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026f549700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f02681afae0 con 0x7f0268101980 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026cae4700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f0268101980 0x7f0268105fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54910/0 (socket says 172.21.15.28:54910) 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f026cae4700 1 -- 172.21.15.28:0/923288305 learned_addr learned my addr 172.21.15.28:0/923288305 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter ms_handle_connect 0x7f0268101980 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter ms_handle_connect 0x7f0268110c70 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902050234 0 0) 0x7f02681afae0 con 0x7f0268101980 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02500031f0 con 0x7f0268101980 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter ms_handle_connect 0x7f0268100da0 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159203544 0 0) 0x7f026810f7a0 con 0x7f0268110c70 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02681afae0 con 0x7f0268110c70 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 499771292 0 0) 0x7f02681ae900 con 0x7f0268100da0 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026810f7a0 con 0x7f0268100da0 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3945718218 0 0) 0x7f02500031f0 con 0x7f0268101980 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02681ae900 con 0x7f0268101980 2024-03-30T15:06:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.499+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f025c001a30 con 0x7f0268101980 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1772302984 0 0) 0x7f02681afae0 con 0x7f0268110c70 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02500031f0 con 0x7f0268110c70 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0258001850 con 0x7f0268110c70 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3669800150 0 0) 0x7f026810f7a0 con 0x7f0268100da0 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02681afae0 con 0x7f0268100da0 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0260004540 con 0x7f0268100da0 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 177400783 0 0) 0x7f02681ae900 con 0x7f0268101980 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 >> v1:172.21.15.134:6789/0 conn(0x7f0268100da0 legacy=0x7f026810ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 >> v1:172.21.15.28:6789/0 conn(0x7f0268110c70 legacy=0x7f02681066f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02681b0cc0 con 0x7f0268101980 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02681ad8f0 con 0x7f0268101980 2024-03-30T15:06:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f026f549700 10 client.58289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f02681addb0 con 0x7f0268101980 2024-03-30T15:06:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f026f549700 1 librados: init done 2024-03-30T15:06:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f025c001bd0 con 0x7f0268101980 2024-03-30T15:06:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.500+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f025c005cc0 con 0x7f0268101980 2024-03-30T15:06:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.501+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0254005150 con 0x7f0268101980 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.501+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f025c004140 con 0x7f0268101980 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(533..533 src has 442..533) ==== 9005+0+0 (unknown 4187053256 0 0) 0x7f025c089a80 con 0x7f0268101980 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 10 client.58289.objecter ms_dispatch 0x7f0268001550 osd_map(533..533 src has 442..533) 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 3 client.58289.objecter handle_osd_map got epochs [533,533] > 0 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 3 client.58289.objecter handle_osd_map decoding full epoch 533 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 20 client.58289.objecter dump_active .. 0 homeless 2024-03-30T15:06:06.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.502+0000 7f02667fc700 10 client.58289.objecter ms_handle_connect 0x7f02500703d0 2024-03-30T15:06:06.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.506+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f025c056920 con 0x7f0268101980 2024-03-30T15:06:06.534 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:06.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:06.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:06.728+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0254005440 con 0x7f0268101980 2024-03-30T15:06:06.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:06.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:06.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:06.918 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:06.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T15:06:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T15:06:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:06:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:06:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: pgmap v773: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:06.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: pgmap v773: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: pgmap v773: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: osdmap e533: 8 total, 8 up, 8 in 2024-03-30T15:06:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:06.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:07.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:07.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:07.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:07.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.322+0000 7f02667fc700 1 -- 172.21.15.28:0/923288305 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v534) ==== 101+0+0 (unknown 1903834320 0 0) 0x7f025c05a370 con 0x7f0268101980 2024-03-30T15:06:07.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.326+0000 7f026f549700 10 librados: watch_flush enter 2024-03-30T15:06:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.331+0000 7f026f549700 10 librados: watch_flush exit 2024-03-30T15:06:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.331+0000 7f026f549700 20 client.58289.objecter shutdown clearing up homeless session... 2024-03-30T15:06:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.331+0000 7f026f549700 10 client.58289.objecter successfully canceled tick 2024-03-30T15:06:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.331+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 >> v1:172.21.15.28:6800/3011431093 conn(0x7f02500703d0 legacy=0x7f0250072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.331+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 >> v1:172.21.15.28:6790/0 conn(0x7f0268101980 legacy=0x7f0268105fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:07.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.332+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 shutdown_connections 2024-03-30T15:06:07.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.332+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 >> 172.21.15.28:0/923288305 conn(0x7f02680fc9e0 msgr2=0x7f02680fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:07.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.332+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 shutdown_connections 2024-03-30T15:06:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.333+0000 7f026f549700 1 -- 172.21.15.28:0/923288305 wait complete. 2024-03-30T15:06:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:07.334+0000 7f026f549700 1 librados: shutdown 2024-03-30T15:06:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:06:07.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:07.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:07.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:07.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:07.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:07.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:07.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:07.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/923288305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.58289 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:06:07.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.58289 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:07.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:07.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:07.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/923288305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.58289 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.58289 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/923288305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.58289 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.58289 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:07.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: osdmap e534: 8 total, 8 up, 8 in 2024-03-30T15:06:07.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:07.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]: dispatch 2024-03-30T15:06:08.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:08.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:08.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:08.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:08.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:08.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:08.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:08.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:08.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:08.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.397+0000 7f63a0850700 1 -- 172.21.15.28:0/3331217764 >> v1:172.21.15.28:6790/0 conn(0x7f639c10aed0 legacy=0x7f639c10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.397+0000 7f63a0850700 1 -- 172.21.15.28:0/3331217764 shutdown_connections 2024-03-30T15:06:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.397+0000 7f63a0850700 1 -- 172.21.15.28:0/3331217764 >> 172.21.15.28:0/3331217764 conn(0x7f639c0fdaf0 msgr2=0x7f639c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.397+0000 7f63a0850700 1 -- 172.21.15.28:0/3331217764 shutdown_connections 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.397+0000 7f63a0850700 1 -- 172.21.15.28:0/3331217764 wait complete. 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 librados: starting msgr at 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 librados: starting objecter 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 Processor -- start 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 -- start start 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 librados: setting wanted keys 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 librados: calling monclient init 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f639c1a70e0 con 0x7f639c10ea70 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f639c1a8300 con 0x7f639c102240 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63a0850700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f639c1a9540 con 0x7f639c10aed0 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f6399d9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f639c10aed0 0x7f639c1a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54920/0 (socket says 172.21.15.28:54920) 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f6399d9b700 1 -- 172.21.15.28:0/2819613688 learned_addr learned my addr 172.21.15.28:0/2819613688 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63937fe700 10 client.?.objecter ms_handle_connect 0x7f639c10aed0 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63937fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63937fe700 10 client.?.objecter ms_handle_connect 0x7f639c10ea70 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.398+0000 7f63937fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 10 client.?.objecter ms_handle_connect 0x7f639c102240 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1166623909 0 0) 0x7f639c1a9540 con 0x7f639c10aed0 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6388003aa0 con 0x7f639c10aed0 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895470535 0 0) 0x7f639c1a70e0 con 0x7f639c10ea70 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f639c1a9540 con 0x7f639c10ea70 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311858700 0 0) 0x7f639c1a8300 con 0x7f639c102240 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f639c1a70e0 con 0x7f639c102240 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2368435068 0 0) 0x7f6388003aa0 con 0x7f639c10aed0 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f639c1a8300 con 0x7f639c10aed0 2024-03-30T15:06:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f63840046c0 con 0x7f639c10aed0 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1319752680 0 0) 0x7f639c1a9540 con 0x7f639c10ea70 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6388003aa0 con 0x7f639c10ea70 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6394001070 con 0x7f639c10ea70 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3203870709 0 0) 0x7f639c1a70e0 con 0x7f639c102240 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f639c1a9540 con 0x7f639c102240 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f638c001bb0 con 0x7f639c102240 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 528674024 0 0) 0x7f639c1a8300 con 0x7f639c10aed0 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.134:6789/0 conn(0x7f639c102240 legacy=0x7f639c107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6789/0 conn(0x7f639c10ea70 legacy=0x7f639c1a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c1aa780 con 0x7f639c10aed0 2024-03-30T15:06:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f639c1a7310 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63a0850700 10 client.58295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f639c1a77d0 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63a0850700 1 librados: init done 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6384001ca0 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.399+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6384004f20 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.400+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f637c005100 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6384001800 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(535..535 src has 442..535) ==== 9037+0+0 (unknown 548771734 0 0) 0x7f6384089b20 con 0x7f639c10aed0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 osd_map(535..535 src has 442..535) 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 3 client.58295.objecter handle_osd_map got epochs [535,535] > 0 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 3 client.58295.objecter handle_osd_map decoding full epoch 535 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 20 client.58295.objecter dump_active .. 0 homeless 2024-03-30T15:06:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v535) ==== 54+0+16 (unknown 1951507252 0 2420000192) 0x7f6384089f60 con 0x7f639c10aed0 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.401+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f63880703f0 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 10 client.58295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 1 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 1 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 0 1 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 4 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 10 client.58295.objecter _send_command 1 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f637c004b50 con 0x7f637c004730 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.402+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c004730 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.403+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c004b50 con 0x7f637c004730 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.403+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.403+0000 7f63937fe700 10 client.58295.objecter _finish_command 1 = system:0 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.403+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 4 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.403+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 0 1 2024-03-30T15:06:08.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 10 client.58295.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 2 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 2 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0053e0 osd=0 3 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 0 2 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 4 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 10 client.58295.objecter _send_command 2 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.420+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c007b00 con 0x7f637c004730 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c007b00 con 0x7f637c004730 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 10 client.58295.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 4 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 0 2 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 10 client.58295.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 1 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 4 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 10 client.58295.objecter _send_command 3 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f637c002130 con 0x7f637c0042f0 2024-03-30T15:06:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.422+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c0042f0 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.423+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c002130 con 0x7f637c0042f0 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.423+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.423+0000 7f63937fe700 10 client.58295.objecter _finish_command 3 = system:0 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.423+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 4 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.423+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 1 3 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.440+0000 7f63a0850700 10 client.58295.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 4 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 4 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c004180 osd=1 3 2024-03-30T15:06:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 1 4 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 4 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 10 client.58295.objecter _send_command 4 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.441+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c001600 con 0x7f637c0042f0 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:08.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c001600 con 0x7f637c0042f0 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63937fe700 10 client.58295.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 4 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 1 4 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 10 client.58295.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 5 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 5 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 2 5 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 4 2024-03-30T15:06:08.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 10 client.58295.objecter _send_command 5 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.442+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f637c015540 con 0x7f637c0089c0 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c0089c0 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c015540 con 0x7f637c0089c0 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 10 client.58295.objecter _finish_command 5 = system:0 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 4 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.443+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 2 5 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 10 client.58295.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 6 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 6 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c001a00 osd=2 3 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 2 6 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 4 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 10 client.58295.objecter _send_command 6 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.460+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c008020 con 0x7f637c0089c0 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c008020 con 0x7f637c0089c0 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 10 client.58295.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 4 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 2 6 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 10 client.58295.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 7 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 7 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 3 7 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 4 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 10 client.58295.objecter _send_command 7 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f637c011940 con 0x7f637c011470 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.462+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c011470 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.463+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c011940 con 0x7f637c011470 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.463+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.463+0000 7f63937fe700 10 client.58295.objecter _finish_command 7 = system:0 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.463+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 4 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.463+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 3 7 2024-03-30T15:06:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 10 client.58295.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 8 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 8 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0082d0 osd=3 3 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 3 8 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 4 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 10 client.58295.objecter _send_command 8 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.480+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c010fb0 con 0x7f637c011470 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c010fb0 con 0x7f637c011470 2024-03-30T15:06:08.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63937fe700 10 client.58295.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 4 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 3 8 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 10 client.58295.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 9 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 9 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 4 9 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 4 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 10 client.58295.objecter _send_command 9 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.482+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f637c0112c0 con 0x7f637c0163d0 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.483+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c0163d0 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.483+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c0112c0 con 0x7f637c0163d0 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.483+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.483+0000 7f63937fe700 10 client.58295.objecter _finish_command 9 = system:0 2024-03-30T15:06:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.484+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 4 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.484+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 4 9 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 10 client.58295.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 10 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 10 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c010fb0 osd=4 3 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 4 10 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 4 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 10 client.58295.objecter _send_command 10 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.501+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c01ef00 con 0x7f637c0163d0 2024-03-30T15:06:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c01ef00 con 0x7f637c0163d0 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63937fe700 10 client.58295.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 4 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 4 10 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 10 client.58295.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 11 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 11 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 5 11 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 4 2024-03-30T15:06:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 10 client.58295.objecter _send_command 11 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.503+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f637c020210 con 0x7f637c01f620 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.504+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c01f620 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.505+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c020210 con 0x7f637c01f620 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.505+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.505+0000 7f63937fe700 10 client.58295.objecter _finish_command 11 = system:0 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.505+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 4 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.505+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 5 11 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 10 client.58295.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 12 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 12 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c01ef00 osd=5 3 2024-03-30T15:06:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 5 12 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 4 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 10 client.58295.objecter _send_command 12 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.525+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c020210 con 0x7f637c01f620 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c020210 con 0x7f637c01f620 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 10 client.58295.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 4 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 5 12 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 10 client.58295.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 13 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 13 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 6 13 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 4 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 10 client.58295.objecter _send_command 13 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f637c027710 con 0x7f637c023c00 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.527+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c023c00 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.528+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c027710 con 0x7f637c023c00 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.528+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.528+0000 7f63937fe700 10 client.58295.objecter _finish_command 13 = system:0 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.528+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 4 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.528+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 6 13 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 10 client.58295.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 14 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 14 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c020210 osd=6 3 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 6 14 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 4 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 10 client.58295.objecter _send_command 14 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.547+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c0278f0 con 0x7f637c023c00 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c0278f0 con 0x7f637c023c00 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63937fe700 10 client.58295.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 4 2024-03-30T15:06:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 6 14 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 10 client.58295.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 15 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 15 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 7 15 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 4 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 10 client.58295.objecter _send_command 15 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.550+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f637c028ba0 con 0x7f637c028010 2024-03-30T15:06:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.551+0000 7f63937fe700 10 client.58295.objecter ms_handle_connect 0x7f637c028010 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.552+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f637c028ba0 con 0x7f637c028010 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.552+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.552+0000 7f63937fe700 10 client.58295.objecter _finish_command 15 = system:0 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.552+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 4 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.552+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 7 15 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 10 client.58295.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign -1 16 2024-03-30T15:06:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 20 client.58295.objecter _get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 15 client.58295.objecter _session_command_op_remove -1 16 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 20 client.58295.objecter get_session s=0x7f637c0278f0 osd=7 3 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 15 client.58295.objecter _session_command_op_assign 7 16 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 4 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 10 client.58295.objecter _send_command 16 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.574+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f637c028ba0 con 0x7f637c028010 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.576+0000 7f63937fe700 1 -- 172.21.15.28:0/2819613688 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f637c028ba0 con 0x7f637c028010 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.576+0000 7f63937fe700 10 client.58295.objecter ms_dispatch 0x7f639c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.576+0000 7f63937fe700 10 client.58295.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.576+0000 7f63937fe700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 4 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.576+0000 7f63937fe700 15 client.58295.objecter _session_command_op_remove 7 16 2024-03-30T15:06:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 librados: watch_flush enter 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 librados: watch_flush exit 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.0 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6801/3394323575 conn(0x7f637c004730 legacy=0x7f637c002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0053e0 osd=0 1 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.1 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6805/1850313547 conn(0x7f637c0042f0 legacy=0x7f637c008e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c004180 osd=1 1 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.2 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6809/3451325043 conn(0x7f637c0089c0 legacy=0x7f637c013c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c001a00 osd=2 1 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.3 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6813/1506157250 conn(0x7f637c011470 legacy=0x7f637c018f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0082d0 osd=3 1 2024-03-30T15:06:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.4 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.134:6800/3063775514 conn(0x7f637c0163d0 legacy=0x7f637c01d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c010fb0 osd=4 1 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.5 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.134:6804/3668151179 conn(0x7f637c01f620 legacy=0x7f637c01fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c01ef00 osd=5 1 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.6 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.134:6808/2135224194 conn(0x7f637c023c00 legacy=0x7f637c025ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f639ad9d700 1 -- 172.21.15.28:0/2819613688 reap_dead start 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c020210 osd=6 1 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter close_session for osd.7 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.134:6812/3758190298 conn(0x7f637c028010 legacy=0x7f637c028490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f639ad9d700 1 -- 172.21.15.28:0/2819613688 reap_dead start 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter put_session s=0x7f637c0278f0 osd=7 1 2024-03-30T15:06:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 20 client.58295.objecter shutdown clearing up homeless session... 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 10 client.58295.objecter successfully canceled tick 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f639ad9d700 1 -- 172.21.15.28:0/2819613688 reap_dead start 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6800/3011431093 conn(0x7f63880703f0 legacy=0x7f63880728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f639ad9d700 1 -- 172.21.15.28:0/2819613688 reap_dead start 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> v1:172.21.15.28:6790/0 conn(0x7f639c10aed0 legacy=0x7f639c1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 shutdown_connections 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 >> 172.21.15.28:0/2819613688 conn(0x7f639c0fdaf0 msgr2=0x7f639c111c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 shutdown_connections 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 -- 172.21.15.28:0/2819613688 wait complete. 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:08.577+0000 7f63a0850700 1 librados: shutdown 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:08.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: pgmap v776: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]': finished 2024-03-30T15:06:08.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2819613688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: pgmap v776: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]': finished 2024-03-30T15:06:08.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T15:06:08.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2819613688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:08.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:08.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:08.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: pgmap v776: 292 pgs: 64 unknown, 228 active+clean; 8.4 MiB data, 556 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T15:06:08.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_tier","val": "test-rados-api-smithi028-93253-89-test-flush"}]': finished 2024-03-30T15:06:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: osdmap e535: 8 total, 8 up, 8 in 2024-03-30T15:06:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:06:08.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2819613688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:08.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:09.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:09.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:09.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:09.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.536+0000 7fec6ae6d700 1 -- 172.21.15.28:0/2270729748 >> v1:172.21.15.28:6789/0 conn(0x7fec64101140 legacy=0x7fec64101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- 172.21.15.28:0/2270729748 shutdown_connections 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- 172.21.15.28:0/2270729748 >> 172.21.15.28:0/2270729748 conn(0x7fec640fc9d0 msgr2=0x7fec640fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- 172.21.15.28:0/2270729748 shutdown_connections 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- 172.21.15.28:0/2270729748 wait complete. 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 librados: starting msgr at 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 librados: starting objecter 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 Processor -- start 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- start start 2024-03-30T15:06:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 librados: setting wanted keys 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 librados: calling monclient init 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec641a93c0 con 0x7fec64109dd0 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec641aa5e0 con 0x7fec64101140 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec6ae6d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec641ab820 con 0x7fec6410d970 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec63fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fec64109dd0 0x7fec641a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41606/0 (socket says 172.21.15.28:41606) 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.537+0000 7fec63fff700 1 -- 172.21.15.28:0/1215796485 learned_addr learned my addr 172.21.15.28:0/1215796485 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter ms_handle_connect 0x7fec6410d970 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter ms_handle_connect 0x7fec64109dd0 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080352104 0 0) 0x7fec641ab820 con 0x7fec6410d970 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec540031f0 con 0x7fec6410d970 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter ms_handle_connect 0x7fec64101140 2024-03-30T15:06:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669222365 0 0) 0x7fec641a93c0 con 0x7fec64109dd0 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec641ab820 con 0x7fec64109dd0 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477306077 0 0) 0x7fec641aa5e0 con 0x7fec64101140 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec641a93c0 con 0x7fec64101140 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1016643648 0 0) 0x7fec540031f0 con 0x7fec6410d970 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec641aa5e0 con 0x7fec6410d970 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec5c0013f0 con 0x7fec6410d970 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2958593235 0 0) 0x7fec641ab820 con 0x7fec64109dd0 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec540031f0 con 0x7fec64109dd0 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec50001830 con 0x7fec64109dd0 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1071197405 0 0) 0x7fec641a93c0 con 0x7fec64101140 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec641ab820 con 0x7fec64101140 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec580045e0 con 0x7fec64101140 2024-03-30T15:06:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1590533887 0 0) 0x7fec641aa5e0 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 >> v1:172.21.15.134:6789/0 conn(0x7fec64101140 legacy=0x7fec640fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 >> v1:172.21.15.28:6789/0 conn(0x7fec64109dd0 legacy=0x7fec641a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec641aca60 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec641a95f0 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fec5c003ea0 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.538+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec5c004d80 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.539+0000 7fec6ae6d700 10 client.58301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.539+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fec641a9ab0 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.539+0000 7fec6ae6d700 1 librados: init done 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec64102620 con 0x7fec6410d970 2024-03-30T15:06:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fec5c003a00 con 0x7fec6410d970 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(536..536 src has 442..536) ==== 9056+0+0 (unknown 601042099 0 0) 0x7fec5c088b30 con 0x7fec6410d970 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 10 client.58301.objecter ms_dispatch 0x7fec64001550 osd_map(536..536 src has 442..536) 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 3 client.58301.objecter handle_osd_map got epochs [536,536] > 0 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 3 client.58301.objecter handle_osd_map decoding full epoch 536 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.540+0000 7fec61ffb700 20 client.58301.objecter dump_active .. 0 homeless 2024-03-30T15:06:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.541+0000 7fec61ffb700 10 client.58301.objecter ms_handle_connect 0x7fec54070400 2024-03-30T15:06:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.545+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fec5c0559c0 con 0x7fec6410d970 2024-03-30T15:06:09.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:09.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:09.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 101.14 scrub starts 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 101.14 scrub ok 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 3.1a deep-scrub starts 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 3.1a deep-scrub ok 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 42.c scrub starts 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: 42.c scrub ok 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 101.14 scrub starts 2024-03-30T15:06:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 101.14 scrub ok 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 3.1a deep-scrub starts 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 3.1a deep-scrub ok 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 42.c scrub starts 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: 42.c scrub ok 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.750+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec640fe360 con 0x7fec54070400 2024-03-30T15:06:09.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.771+0000 7fec61ffb700 1 -- 172.21.15.28:0/1215796485 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+746774 (unknown 2965378022 0 3886977140) 0x7fec640fe360 con 0x7fec54070400 2024-03-30T15:06:09.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:06:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.775+0000 7fec6ae6d700 10 librados: watch_flush enter 2024-03-30T15:06:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 10 librados: watch_flush exit 2024-03-30T15:06:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 20 client.58301.objecter shutdown clearing up homeless session... 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 10 client.58301.objecter successfully canceled tick 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 >> v1:172.21.15.28:6800/3011431093 conn(0x7fec54070400 legacy=0x7fec540728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 >> v1:172.21.15.28:6790/0 conn(0x7fec6410d970 legacy=0x7fec641a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 shutdown_connections 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 >> 172.21.15.28:0/1215796485 conn(0x7fec640fc9d0 msgr2=0x7fec641109b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 shutdown_connections 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 -- 172.21.15.28:0/1215796485 wait complete. 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:09.776+0000 7fec6ae6d700 1 librados: shutdown 2024-03-30T15:06:09.790 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:06:09.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:09.827 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 101.14 scrub starts 2024-03-30T15:06:09.827 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 101.14 scrub ok 2024-03-30T15:06:09.827 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 3.1a deep-scrub starts 2024-03-30T15:06:09.827 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 3.1a deep-scrub ok 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 42.c scrub starts 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: 42.c scrub ok 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: osdmap e536: 8 total, 8 up, 8 in 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.828 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:06:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:09.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:09.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:09.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:10.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:10.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:10.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:10.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:10.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:10.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:10.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:10.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:10.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:10.771 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: from='client.58301 v1:172.21.15.28:0/1215796485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 101.1a deep-scrub starts 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 101.1a deep-scrub ok 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:10.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 101.8 deep-scrub starts 2024-03-30T15:06:10.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: 101.8 deep-scrub ok 2024-03-30T15:06:10.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:10.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:10 smithi134 ceph-mon[33714]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T15:06:10.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:10.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T15:06:10.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: from='client.58301 v1:172.21.15.28:0/1215796485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 101.1a deep-scrub starts 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 101.1a deep-scrub ok 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 101.8 deep-scrub starts 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: 101.8 deep-scrub ok 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[41614]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T15:06:10.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: from='client.58301 v1:172.21.15.28:0/1215796485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 101.1a deep-scrub starts 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 101.1a deep-scrub ok 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 101.8 deep-scrub starts 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: 101.8 deep-scrub ok 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:06:10.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:10 smithi028 ceph-mon[29876]: osdmap e537: 8 total, 8 up, 8 in 2024-03-30T15:06:10.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:10.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:10.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:11.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:11.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:11.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:11.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:11.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:11.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:11.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:11.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:11.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:11.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:11.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:11.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 101.c scrub starts 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 101.c scrub ok 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 101.c scrub starts 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 101.c scrub ok 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T15:06:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:11.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:11.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:11.907 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2024-03-30T15:06:09.323249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811064,"num_objects":390,"num_object_clones":10,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":2264,"num_read_kb":1954,"num_write":1679,"num_write_kb":15269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3049,"ondisk_log_size":3049,"up":876,"acting":876,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":807,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":569948,"kb_used_data":49760,"kb_used_omap":12,"kb_used_meta":520051,"kb_avail":749391268,"statfs":{"total":767960285184,"available":767376658432,"internally_reserved":0,"allocated":50954240,"data_stored":34190656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":532532784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006323"},"pg_stats":[{"pgid":"309.1e","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.495926+0000","last_change":"2024-03-30T15:06:07.495480+0000","last_active":"2024-03-30T15:06:07.495926+0000","last_peered":"2024-03-30T15:06:07.495926+0000","last_clean":"2024-03-30T15:06:07.495926+0000","last_became_active":"2024-03-30T15:06:07.494191+0000","last_became_peered":"2024-03-30T15:06:07.494191+0000","last_unstale":"2024-03-30T15:06:07.495926+0000","last_undegraded":"2024-03-30T15:06:07.495926+0000","last_fullsized":"2024-03-30T15:06:07.495926+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:24.382792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"308.1f","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339297+0000","last_change":"2024-03-30T15:06:05.337809+0000","last_active":"2024-03-30T15:06:07.339297+0000","last_peered":"2024-03-30T15:06:07.339297+0000","last_clean":"2024-03-30T15:06:07.339297+0000","last_became_active":"2024-03-30T15:06:05.337209+0000","last_became_peered":"2024-03-30T15:06:05.337209+0000","last_unstale":"2024-03-30T15:06:07.339297+0000","last_undegraded":"2024-03-30T15:06:07.339297+0000","last_fullsized":"2024-03-30T15:06:07.339297+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:11.135518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.1f","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325607+0000","last_change":"2024-03-30T15:06:07.348282+0000","last_active":"2024-03-30T15:06:08.325607+0000","last_peered":"2024-03-30T15:06:08.325607+0000","last_clean":"2024-03-30T15:06:08.325607+0000","last_became_active":"2024-03-30T15:06:07.347929+0000","last_became_peered":"2024-03-30T15:06:07.347929+0000","last_unstale":"2024-03-30T15:06:08.325607+0000","last_undegraded":"2024-03-30T15:06:08.325607+0000","last_fullsized":"2024-03-30T15:06:08.325607+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:06.792813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"308.1e","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341879+0000","last_change":"2024-03-30T15:06:05.442187+0000","last_active":"2024-03-30T15:06:07.341879+0000","last_peered":"2024-03-30T15:06:07.341879+0000","last_clean":"2024-03-30T15:06:07.341879+0000","last_became_active":"2024-03-30T15:06:05.441871+0000","last_became_peered":"2024-03-30T15:06:05.441871+0000","last_unstale":"2024-03-30T15:06:07.341879+0000","last_undegraded":"2024-03-30T15:06:07.341879+0000","last_fullsized":"2024-03-30T15:06:07.341879+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.290489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.1c","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.496399+0000","last_change":"2024-03-30T15:06:07.495673+0000","last_active":"2024-03-30T15:06:07.496399+0000","last_peered":"2024-03-30T15:06:07.496399+0000","last_clean":"2024-03-30T15:06:07.496399+0000","last_became_active":"2024-03-30T15:06:07.495339+0000","last_became_peered":"2024-03-30T15:06:07.495339+0000","last_unstale":"2024-03-30T15:06:07.496399+0000","last_undegraded":"2024-03-30T15:06:07.496399+0000","last_fullsized":"2024-03-30T15:06:07.496399+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.032702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"308.1d","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549621+0000","last_change":"2024-03-30T15:06:05.443250+0000","last_active":"2024-03-30T15:06:08.549621+0000","last_peered":"2024-03-30T15:06:08.549621+0000","last_clean":"2024-03-30T15:06:08.549621+0000","last_became_active":"2024-03-30T15:06:05.442898+0000","last_became_peered":"2024-03-30T15:06:05.442898+0000","last_unstale":"2024-03-30T15:06:08.549621+0000","last_undegraded":"2024-03-30T15:06:08.549621+0000","last_fullsized":"2024-03-30T15:06:08.549621+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.771696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.1d","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562239+0000","last_change":"2024-03-30T15:06:07.497134+0000","last_active":"2024-03-30T15:06:08.562239+0000","last_peered":"2024-03-30T15:06:08.562239+0000","last_clean":"2024-03-30T15:06:08.562239+0000","last_became_active":"2024-03-30T15:06:07.496797+0000","last_became_peered":"2024-03-30T15:06:07.496797+0000","last_unstale":"2024-03-30T15:06:08.562239+0000","last_undegraded":"2024-03-30T15:06:08.562239+0000","last_fullsized":"2024-03-30T15:06:08.562239+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.037559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"308.1c","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324159+0000","last_change":"2024-03-30T15:06:05.443552+0000","last_active":"2024-03-30T15:06:08.324159+0000","last_peered":"2024-03-30T15:06:08.324159+0000","last_clean":"2024-03-30T15:06:08.324159+0000","last_became_active":"2024-03-30T15:06:05.443201+0000","last_became_peered":"2024-03-30T15:06:05.443201+0000","last_unstale":"2024-03-30T15:06:08.324159+0000","last_undegraded":"2024-03-30T15:06:08.324159+0000","last_fullsized":"2024-03-30T15:06:08.324159+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:30.401294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"309.1a","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323119+0000","last_change":"2024-03-30T15:06:07.337463+0000","last_active":"2024-03-30T15:06:08.323119+0000","last_peered":"2024-03-30T15:06:08.323119+0000","last_clean":"2024-03-30T15:06:08.323119+0000","last_became_active":"2024-03-30T15:06:07.337156+0000","last_became_peered":"2024-03-30T15:06:07.337156+0000","last_unstale":"2024-03-30T15:06:08.323119+0000","last_undegraded":"2024-03-30T15:06:08.323119+0000","last_fullsized":"2024-03-30T15:06:08.323119+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.947739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.1b","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339338+0000","last_change":"2024-03-30T15:06:05.335743+0000","last_active":"2024-03-30T15:06:07.339338+0000","last_peered":"2024-03-30T15:06:07.339338+0000","last_clean":"2024-03-30T15:06:07.339338+0000","last_became_active":"2024-03-30T15:06:05.335403+0000","last_became_peered":"2024-03-30T15:06:05.335403+0000","last_unstale":"2024-03-30T15:06:07.339338+0000","last_undegraded":"2024-03-30T15:06:07.339338+0000","last_fullsized":"2024-03-30T15:06:07.339338+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:17.687218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.1b","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323900+0000","last_change":"2024-03-30T15:06:07.340444+0000","last_active":"2024-03-30T15:06:08.323900+0000","last_peered":"2024-03-30T15:06:08.323900+0000","last_clean":"2024-03-30T15:06:08.323900+0000","last_became_active":"2024-03-30T15:06:07.339872+0000","last_became_peered":"2024-03-30T15:06:07.339872+0000","last_unstale":"2024-03-30T15:06:08.323900+0000","last_undegraded":"2024-03-30T15:06:08.323900+0000","last_fullsized":"2024-03-30T15:06:08.323900+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:06.645897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.1a","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323352+0000","last_change":"2024-03-30T15:06:05.334909+0000","last_active":"2024-03-30T15:06:08.323352+0000","last_peered":"2024-03-30T15:06:08.323352+0000","last_clean":"2024-03-30T15:06:08.323352+0000","last_became_active":"2024-03-30T15:06:05.334568+0000","last_became_peered":"2024-03-30T15:06:05.334568+0000","last_unstale":"2024-03-30T15:06:08.323352+0000","last_undegraded":"2024-03-30T15:06:08.323352+0000","last_fullsized":"2024-03-30T15:06:08.323352+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.191946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"309.18","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.322764+0000","last_change":"2024-03-30T15:06:07.338032+0000","last_active":"2024-03-30T15:06:08.322764+0000","last_peered":"2024-03-30T15:06:08.322764+0000","last_clean":"2024-03-30T15:06:08.322764+0000","last_became_active":"2024-03-30T15:06:07.337544+0000","last_became_peered":"2024-03-30T15:06:07.337544+0000","last_unstale":"2024-03-30T15:06:08.322764+0000","last_undegraded":"2024-03-30T15:06:08.322764+0000","last_fullsized":"2024-03-30T15:06:08.322764+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.018220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.19","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560942+0000","last_change":"2024-03-30T15:06:05.328287+0000","last_active":"2024-03-30T15:06:08.560942+0000","last_peered":"2024-03-30T15:06:08.560942+0000","last_clean":"2024-03-30T15:06:08.560942+0000","last_became_active":"2024-03-30T15:06:05.328005+0000","last_became_peered":"2024-03-30T15:06:05.328005+0000","last_unstale":"2024-03-30T15:06:08.560942+0000","last_undegraded":"2024-03-30T15:06:08.560942+0000","last_fullsized":"2024-03-30T15:06:08.560942+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.148812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"309.19","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.547891+0000","last_change":"2024-03-30T15:06:07.496743+0000","last_active":"2024-03-30T15:06:08.547891+0000","last_peered":"2024-03-30T15:06:08.547891+0000","last_clean":"2024-03-30T15:06:08.547891+0000","last_became_active":"2024-03-30T15:06:07.496407+0000","last_became_peered":"2024-03-30T15:06:07.496407+0000","last_unstale":"2024-03-30T15:06:08.547891+0000","last_undegraded":"2024-03-30T15:06:08.547891+0000","last_fullsized":"2024-03-30T15:06:08.547891+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.001171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"308.18","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.547990+0000","last_change":"2024-03-30T15:06:05.444087+0000","last_active":"2024-03-30T15:06:08.547990+0000","last_peered":"2024-03-30T15:06:08.547990+0000","last_clean":"2024-03-30T15:06:08.547990+0000","last_became_active":"2024-03-30T15:06:05.443775+0000","last_became_peered":"2024-03-30T15:06:05.443775+0000","last_unstale":"2024-03-30T15:06:08.547990+0000","last_undegraded":"2024-03-30T15:06:08.547990+0000","last_fullsized":"2024-03-30T15:06:08.547990+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.281000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.16","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343137+0000","last_change":"2024-03-30T15:06:07.341357+0000","last_active":"2024-03-30T15:06:07.343137+0000","last_peered":"2024-03-30T15:06:07.343137+0000","last_clean":"2024-03-30T15:06:07.343137+0000","last_became_active":"2024-03-30T15:06:07.341024+0000","last_became_peered":"2024-03-30T15:06:07.341024+0000","last_unstale":"2024-03-30T15:06:07.343137+0000","last_undegraded":"2024-03-30T15:06:07.343137+0000","last_fullsized":"2024-03-30T15:06:07.343137+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.588617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"308.17","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.550975+0000","last_change":"2024-03-30T15:06:05.336209+0000","last_active":"2024-03-30T15:06:08.550975+0000","last_peered":"2024-03-30T15:06:08.550975+0000","last_clean":"2024-03-30T15:06:08.550975+0000","last_became_active":"2024-03-30T15:06:05.335858+0000","last_became_peered":"2024-03-30T15:06:05.335858+0000","last_unstale":"2024-03-30T15:06:08.550975+0000","last_undegraded":"2024-03-30T15:06:08.550975+0000","last_fullsized":"2024-03-30T15:06:08.550975+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:17.202233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.17","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560492+0000","last_change":"2024-03-30T15:06:07.341318+0000","last_active":"2024-03-30T15:06:08.560492+0000","last_peered":"2024-03-30T15:06:08.560492+0000","last_clean":"2024-03-30T15:06:08.560492+0000","last_became_active":"2024-03-30T15:06:07.340905+0000","last_became_peered":"2024-03-30T15:06:07.340905+0000","last_unstale":"2024-03-30T15:06:08.560492+0000","last_undegraded":"2024-03-30T15:06:08.560492+0000","last_fullsized":"2024-03-30T15:06:08.560492+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.290731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"308.16","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339378+0000","last_change":"2024-03-30T15:06:05.444570+0000","last_active":"2024-03-30T15:06:07.339378+0000","last_peered":"2024-03-30T15:06:07.339378+0000","last_clean":"2024-03-30T15:06:07.339378+0000","last_became_active":"2024-03-30T15:06:05.442827+0000","last_became_peered":"2024-03-30T15:06:05.442827+0000","last_unstale":"2024-03-30T15:06:07.339378+0000","last_undegraded":"2024-03-30T15:06:07.339378+0000","last_fullsized":"2024-03-30T15:06:07.339378+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.558929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.14","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343166+0000","last_change":"2024-03-30T15:06:07.341320+0000","last_active":"2024-03-30T15:06:07.343166+0000","last_peered":"2024-03-30T15:06:07.343166+0000","last_clean":"2024-03-30T15:06:07.343166+0000","last_became_active":"2024-03-30T15:06:07.340998+0000","last_became_peered":"2024-03-30T15:06:07.340998+0000","last_unstale":"2024-03-30T15:06:07.343166+0000","last_undegraded":"2024-03-30T15:06:07.343166+0000","last_fullsized":"2024-03-30T15:06:07.343166+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:17.525564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"308.15","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549691+0000","last_change":"2024-03-30T15:06:05.339150+0000","last_active":"2024-03-30T15:06:08.549691+0000","last_peered":"2024-03-30T15:06:08.549691+0000","last_clean":"2024-03-30T15:06:08.549691+0000","last_became_active":"2024-03-30T15:06:05.336587+0000","last_became_peered":"2024-03-30T15:06:05.336587+0000","last_unstale":"2024-03-30T15:06:08.549691+0000","last_undegraded":"2024-03-30T15:06:08.549691+0000","last_fullsized":"2024-03-30T15:06:08.549691+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.487695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.15","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.559546+0000","last_change":"2024-03-30T15:06:07.496506+0000","last_active":"2024-03-30T15:06:08.559546+0000","last_peered":"2024-03-30T15:06:08.559546+0000","last_clean":"2024-03-30T15:06:08.559546+0000","last_became_active":"2024-03-30T15:06:07.495042+0000","last_became_peered":"2024-03-30T15:06:07.495042+0000","last_unstale":"2024-03-30T15:06:08.559546+0000","last_undegraded":"2024-03-30T15:06:08.559546+0000","last_fullsized":"2024-03-30T15:06:08.559546+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.104838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"308.14","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339129+0000","last_change":"2024-03-30T15:06:05.338098+0000","last_active":"2024-03-30T15:06:07.339129+0000","last_peered":"2024-03-30T15:06:07.339129+0000","last_clean":"2024-03-30T15:06:07.339129+0000","last_became_active":"2024-03-30T15:06:05.337694+0000","last_became_peered":"2024-03-30T15:06:05.337694+0000","last_unstale":"2024-03-30T15:06:07.339129+0000","last_undegraded":"2024-03-30T15:06:07.339129+0000","last_fullsized":"2024-03-30T15:06:07.339129+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:30.320280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.12","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548044+0000","last_change":"2024-03-30T15:06:07.494509+0000","last_active":"2024-03-30T15:06:08.548044+0000","last_peered":"2024-03-30T15:06:08.548044+0000","last_clean":"2024-03-30T15:06:08.548044+0000","last_became_active":"2024-03-30T15:06:07.494061+0000","last_became_peered":"2024-03-30T15:06:07.494061+0000","last_unstale":"2024-03-30T15:06:08.548044+0000","last_undegraded":"2024-03-30T15:06:08.548044+0000","last_fullsized":"2024-03-30T15:06:08.548044+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:10.794116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"308.13","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324169+0000","last_change":"2024-03-30T15:06:05.329472+0000","last_active":"2024-03-30T15:06:08.324169+0000","last_peered":"2024-03-30T15:06:08.324169+0000","last_clean":"2024-03-30T15:06:08.324169+0000","last_became_active":"2024-03-30T15:06:05.329132+0000","last_became_peered":"2024-03-30T15:06:05.329132+0000","last_unstale":"2024-03-30T15:06:08.324169+0000","last_undegraded":"2024-03-30T15:06:08.324169+0000","last_fullsized":"2024-03-30T15:06:08.324169+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.758429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"309.13","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.495776+0000","last_change":"2024-03-30T15:06:07.494475+0000","last_active":"2024-03-30T15:06:07.495776+0000","last_peered":"2024-03-30T15:06:07.495776+0000","last_clean":"2024-03-30T15:06:07.495776+0000","last_became_active":"2024-03-30T15:06:07.494133+0000","last_became_peered":"2024-03-30T15:06:07.494133+0000","last_unstale":"2024-03-30T15:06:07.495776+0000","last_undegraded":"2024-03-30T15:06:07.495776+0000","last_fullsized":"2024-03-30T15:06:07.495776+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:30.249098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"308.12","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557340+0000","last_change":"2024-03-30T15:06:05.333315+0000","last_active":"2024-03-30T15:06:08.557340+0000","last_peered":"2024-03-30T15:06:08.557340+0000","last_clean":"2024-03-30T15:06:08.557340+0000","last_became_active":"2024-03-30T15:06:05.333069+0000","last_became_peered":"2024-03-30T15:06:05.333069+0000","last_unstale":"2024-03-30T15:06:08.557340+0000","last_undegraded":"2024-03-30T15:06:08.557340+0000","last_fullsized":"2024-03-30T15:06:08.557340+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.482245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"309.10","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.547741+0000","last_change":"2024-03-30T15:06:07.345005+0000","last_active":"2024-03-30T15:06:08.547741+0000","last_peered":"2024-03-30T15:06:08.547741+0000","last_clean":"2024-03-30T15:06:08.547741+0000","last_became_active":"2024-03-30T15:06:07.344617+0000","last_became_peered":"2024-03-30T15:06:07.344617+0000","last_unstale":"2024-03-30T15:06:08.547741+0000","last_undegraded":"2024-03-30T15:06:08.547741+0000","last_fullsized":"2024-03-30T15:06:08.547741+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:18.664162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"308.11","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323573+0000","last_change":"2024-03-30T15:06:05.443266+0000","last_active":"2024-03-30T15:06:08.323573+0000","last_peered":"2024-03-30T15:06:08.323573+0000","last_clean":"2024-03-30T15:06:08.323573+0000","last_became_active":"2024-03-30T15:06:05.442897+0000","last_became_peered":"2024-03-30T15:06:05.442897+0000","last_unstale":"2024-03-30T15:06:08.323573+0000","last_undegraded":"2024-03-30T15:06:08.323573+0000","last_fullsized":"2024-03-30T15:06:08.323573+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.540818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"309.11","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.495928+0000","last_change":"2024-03-30T15:06:07.495237+0000","last_active":"2024-03-30T15:06:07.495928+0000","last_peered":"2024-03-30T15:06:07.495928+0000","last_clean":"2024-03-30T15:06:07.495928+0000","last_became_active":"2024-03-30T15:06:07.494164+0000","last_became_peered":"2024-03-30T15:06:07.494164+0000","last_unstale":"2024-03-30T15:06:07.495928+0000","last_undegraded":"2024-03-30T15:06:07.495928+0000","last_fullsized":"2024-03-30T15:06:07.495928+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.520467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"308.10","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556602+0000","last_change":"2024-03-30T15:06:05.336413+0000","last_active":"2024-03-30T15:06:08.556602+0000","last_peered":"2024-03-30T15:06:08.556602+0000","last_clean":"2024-03-30T15:06:08.556602+0000","last_became_active":"2024-03-30T15:06:05.336108+0000","last_became_peered":"2024-03-30T15:06:05.336108+0000","last_unstale":"2024-03-30T15:06:08.556602+0000","last_undegraded":"2024-03-30T15:06:08.556602+0000","last_fullsized":"2024-03-30T15:06:08.556602+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.601341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"309.e","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325200+0000","last_change":"2024-03-30T15:06:07.338298+0000","last_active":"2024-03-30T15:06:08.325200+0000","last_peered":"2024-03-30T15:06:08.325200+0000","last_clean":"2024-03-30T15:06:08.325200+0000","last_became_active":"2024-03-30T15:06:07.337682+0000","last_became_peered":"2024-03-30T15:06:07.337682+0000","last_unstale":"2024-03-30T15:06:08.325200+0000","last_undegraded":"2024-03-30T15:06:08.325200+0000","last_fullsized":"2024-03-30T15:06:08.325200+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.566972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.f","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324843+0000","last_change":"2024-03-30T15:06:05.340322+0000","last_active":"2024-03-30T15:06:08.324843+0000","last_peered":"2024-03-30T15:06:08.324843+0000","last_clean":"2024-03-30T15:06:08.324843+0000","last_became_active":"2024-03-30T15:06:05.339993+0000","last_became_peered":"2024-03-30T15:06:05.339993+0000","last_unstale":"2024-03-30T15:06:08.324843+0000","last_undegraded":"2024-03-30T15:06:08.324843+0000","last_fullsized":"2024-03-30T15:06:08.324843+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.172539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"309.f","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560556+0000","last_change":"2024-03-30T15:06:07.496148+0000","last_active":"2024-03-30T15:06:08.560556+0000","last_peered":"2024-03-30T15:06:08.560556+0000","last_clean":"2024-03-30T15:06:08.560556+0000","last_became_active":"2024-03-30T15:06:07.495660+0000","last_became_peered":"2024-03-30T15:06:07.495660+0000","last_unstale":"2024-03-30T15:06:08.560556+0000","last_undegraded":"2024-03-30T15:06:08.560556+0000","last_fullsized":"2024-03-30T15:06:08.560556+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.556898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"308.e","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340472+0000","last_change":"2024-03-30T15:06:05.333971+0000","last_active":"2024-03-30T15:06:07.340472+0000","last_peered":"2024-03-30T15:06:07.340472+0000","last_clean":"2024-03-30T15:06:07.340472+0000","last_became_active":"2024-03-30T15:06:05.333557+0000","last_became_peered":"2024-03-30T15:06:05.333557+0000","last_unstale":"2024-03-30T15:06:07.340472+0000","last_undegraded":"2024-03-30T15:06:07.340472+0000","last_fullsized":"2024-03-30T15:06:07.340472+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.559840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.c","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324268+0000","last_change":"2024-03-30T15:06:07.339836+0000","last_active":"2024-03-30T15:06:08.324268+0000","last_peered":"2024-03-30T15:06:08.324268+0000","last_clean":"2024-03-30T15:06:08.324268+0000","last_became_active":"2024-03-30T15:06:07.339264+0000","last_became_peered":"2024-03-30T15:06:07.339264+0000","last_unstale":"2024-03-30T15:06:08.324268+0000","last_undegraded":"2024-03-30T15:06:08.324268+0000","last_fullsized":"2024-03-30T15:06:08.324268+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:07.775545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.d","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.559756+0000","last_change":"2024-03-30T15:06:05.442353+0000","last_active":"2024-03-30T15:06:08.559756+0000","last_peered":"2024-03-30T15:06:08.559756+0000","last_clean":"2024-03-30T15:06:08.559756+0000","last_became_active":"2024-03-30T15:06:05.441874+0000","last_became_peered":"2024-03-30T15:06:05.441874+0000","last_unstale":"2024-03-30T15:06:08.559756+0000","last_undegraded":"2024-03-30T15:06:08.559756+0000","last_fullsized":"2024-03-30T15:06:08.559756+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.438414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"309.d","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324276+0000","last_change":"2024-03-30T15:06:07.348277+0000","last_active":"2024-03-30T15:06:08.324276+0000","last_peered":"2024-03-30T15:06:08.324276+0000","last_clean":"2024-03-30T15:06:08.324276+0000","last_became_active":"2024-03-30T15:06:07.347995+0000","last_became_peered":"2024-03-30T15:06:07.347995+0000","last_unstale":"2024-03-30T15:06:08.324276+0000","last_undegraded":"2024-03-30T15:06:08.324276+0000","last_fullsized":"2024-03-30T15:06:08.324276+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.859380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"308.c","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342717+0000","last_change":"2024-03-30T15:06:05.333244+0000","last_active":"2024-03-30T15:06:07.342717+0000","last_peered":"2024-03-30T15:06:07.342717+0000","last_clean":"2024-03-30T15:06:07.342717+0000","last_became_active":"2024-03-30T15:06:05.332879+0000","last_became_peered":"2024-03-30T15:06:05.332879+0000","last_unstale":"2024-03-30T15:06:07.342717+0000","last_undegraded":"2024-03-30T15:06:07.342717+0000","last_fullsized":"2024-03-30T15:06:07.342717+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.174572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.a","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340437+0000","last_change":"2024-03-30T15:06:07.339873+0000","last_active":"2024-03-30T15:06:07.340437+0000","last_peered":"2024-03-30T15:06:07.340437+0000","last_clean":"2024-03-30T15:06:07.340437+0000","last_became_active":"2024-03-30T15:06:07.339550+0000","last_became_peered":"2024-03-30T15:06:07.339550+0000","last_unstale":"2024-03-30T15:06:07.340437+0000","last_undegraded":"2024-03-30T15:06:07.340437+0000","last_fullsized":"2024-03-30T15:06:07.340437+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.791563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"308.b","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339088+0000","last_change":"2024-03-30T15:06:05.334919+0000","last_active":"2024-03-30T15:06:07.339088+0000","last_peered":"2024-03-30T15:06:07.339088+0000","last_clean":"2024-03-30T15:06:07.339088+0000","last_became_active":"2024-03-30T15:06:05.334374+0000","last_became_peered":"2024-03-30T15:06:05.334374+0000","last_unstale":"2024-03-30T15:06:07.339088+0000","last_undegraded":"2024-03-30T15:06:07.339088+0000","last_fullsized":"2024-03-30T15:06:07.339088+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.034279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.b","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323906+0000","last_change":"2024-03-30T15:06:07.494765+0000","last_active":"2024-03-30T15:06:08.323906+0000","last_peered":"2024-03-30T15:06:08.323906+0000","last_clean":"2024-03-30T15:06:08.323906+0000","last_became_active":"2024-03-30T15:06:07.494100+0000","last_became_peered":"2024-03-30T15:06:07.494100+0000","last_unstale":"2024-03-30T15:06:08.323906+0000","last_undegraded":"2024-03-30T15:06:08.323906+0000","last_fullsized":"2024-03-30T15:06:08.323906+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.074992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.a","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548526+0000","last_change":"2024-03-30T15:06:05.444484+0000","last_active":"2024-03-30T15:06:08.548526+0000","last_peered":"2024-03-30T15:06:08.548526+0000","last_clean":"2024-03-30T15:06:08.548526+0000","last_became_active":"2024-03-30T15:06:05.443553+0000","last_became_peered":"2024-03-30T15:06:05.443553+0000","last_unstale":"2024-03-30T15:06:08.548526+0000","last_undegraded":"2024-03-30T15:06:08.548526+0000","last_fullsized":"2024-03-30T15:06:08.548526+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.374526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.8","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.547875+0000","last_change":"2024-03-30T15:06:07.343902+0000","last_active":"2024-03-30T15:06:08.547875+0000","last_peered":"2024-03-30T15:06:08.547875+0000","last_clean":"2024-03-30T15:06:08.547875+0000","last_became_active":"2024-03-30T15:06:07.343595+0000","last_became_peered":"2024-03-30T15:06:07.343595+0000","last_unstale":"2024-03-30T15:06:08.547875+0000","last_undegraded":"2024-03-30T15:06:08.547875+0000","last_fullsized":"2024-03-30T15:06:08.547875+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:24.247057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"308.9","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556644+0000","last_change":"2024-03-30T15:06:05.327087+0000","last_active":"2024-03-30T15:06:08.556644+0000","last_peered":"2024-03-30T15:06:08.556644+0000","last_clean":"2024-03-30T15:06:08.556644+0000","last_became_active":"2024-03-30T15:06:05.326812+0000","last_became_peered":"2024-03-30T15:06:05.326812+0000","last_unstale":"2024-03-30T15:06:08.556644+0000","last_undegraded":"2024-03-30T15:06:08.556644+0000","last_fullsized":"2024-03-30T15:06:08.556644+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.990333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"309.9","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323147+0000","last_change":"2024-03-30T15:06:07.495332+0000","last_active":"2024-03-30T15:06:08.323147+0000","last_peered":"2024-03-30T15:06:08.323147+0000","last_clean":"2024-03-30T15:06:08.323147+0000","last_became_active":"2024-03-30T15:06:07.495007+0000","last_became_peered":"2024-03-30T15:06:07.495007+0000","last_unstale":"2024-03-30T15:06:08.323147+0000","last_undegraded":"2024-03-30T15:06:08.323147+0000","last_fullsized":"2024-03-30T15:06:08.323147+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"308.8","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548112+0000","last_change":"2024-03-30T15:06:05.336169+0000","last_active":"2024-03-30T15:06:08.548112+0000","last_peered":"2024-03-30T15:06:08.548112+0000","last_clean":"2024-03-30T15:06:08.548112+0000","last_became_active":"2024-03-30T15:06:05.335803+0000","last_became_peered":"2024-03-30T15:06:05.335803+0000","last_unstale":"2024-03-30T15:06:08.548112+0000","last_undegraded":"2024-03-30T15:06:08.548112+0000","last_fullsized":"2024-03-30T15:06:08.548112+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:07.345302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"309.6","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.559561+0000","last_change":"2024-03-30T15:06:07.341527+0000","last_active":"2024-03-30T15:06:08.559561+0000","last_peered":"2024-03-30T15:06:08.559561+0000","last_clean":"2024-03-30T15:06:08.559561+0000","last_became_active":"2024-03-30T15:06:07.341227+0000","last_became_peered":"2024-03-30T15:06:07.341227+0000","last_unstale":"2024-03-30T15:06:08.559561+0000","last_undegraded":"2024-03-30T15:06:08.559561+0000","last_fullsized":"2024-03-30T15:06:08.559561+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.222581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"308.7","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560794+0000","last_change":"2024-03-30T15:06:05.337696+0000","last_active":"2024-03-30T15:06:08.560794+0000","last_peered":"2024-03-30T15:06:08.560794+0000","last_clean":"2024-03-30T15:06:08.560794+0000","last_became_active":"2024-03-30T15:06:05.337385+0000","last_became_peered":"2024-03-30T15:06:05.337385+0000","last_unstale":"2024-03-30T15:06:08.560794+0000","last_undegraded":"2024-03-30T15:06:08.560794+0000","last_fullsized":"2024-03-30T15:06:08.560794+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:13.047354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"309.7","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325382+0000","last_change":"2024-03-30T15:06:07.497234+0000","last_active":"2024-03-30T15:06:08.325382+0000","last_peered":"2024-03-30T15:06:08.325382+0000","last_clean":"2024-03-30T15:06:08.325382+0000","last_became_active":"2024-03-30T15:06:07.496945+0000","last_became_peered":"2024-03-30T15:06:07.496945+0000","last_unstale":"2024-03-30T15:06:08.325382+0000","last_undegraded":"2024-03-30T15:06:08.325382+0000","last_fullsized":"2024-03-30T15:06:08.325382+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.314000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.6","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564157+0000","last_change":"2024-03-30T15:06:05.329082+0000","last_active":"2024-03-30T15:06:08.564157+0000","last_peered":"2024-03-30T15:06:08.564157+0000","last_clean":"2024-03-30T15:06:08.564157+0000","last_became_active":"2024-03-30T15:06:05.328785+0000","last_became_peered":"2024-03-30T15:06:05.328785+0000","last_unstale":"2024-03-30T15:06:08.564157+0000","last_undegraded":"2024-03-30T15:06:08.564157+0000","last_fullsized":"2024-03-30T15:06:08.564157+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.937443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"309.4","version":"0'0","reported_seq":10,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.495822+0000","last_change":"2024-03-30T15:06:07.495233+0000","last_active":"2024-03-30T15:06:07.495822+0000","last_peered":"2024-03-30T15:06:07.495822+0000","last_clean":"2024-03-30T15:06:07.495822+0000","last_became_active":"2024-03-30T15:06:07.494839+0000","last_became_peered":"2024-03-30T15:06:07.494839+0000","last_unstale":"2024-03-30T15:06:07.495822+0000","last_undegraded":"2024-03-30T15:06:07.495822+0000","last_fullsized":"2024-03-30T15:06:07.495822+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.700439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"308.5","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342064+0000","last_change":"2024-03-30T15:06:05.445357+0000","last_active":"2024-03-30T15:06:07.342064+0000","last_peered":"2024-03-30T15:06:07.342064+0000","last_clean":"2024-03-30T15:06:07.342064+0000","last_became_active":"2024-03-30T15:06:05.445022+0000","last_became_peered":"2024-03-30T15:06:05.445022+0000","last_unstale":"2024-03-30T15:06:07.342064+0000","last_undegraded":"2024-03-30T15:06:07.342064+0000","last_fullsized":"2024-03-30T15:06:07.342064+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:13.151496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.5","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562395+0000","last_change":"2024-03-30T15:06:07.496579+0000","last_active":"2024-03-30T15:06:08.562395+0000","last_peered":"2024-03-30T15:06:08.562395+0000","last_clean":"2024-03-30T15:06:08.562395+0000","last_became_active":"2024-03-30T15:06:07.496234+0000","last_became_peered":"2024-03-30T15:06:07.496234+0000","last_unstale":"2024-03-30T15:06:08.562395+0000","last_undegraded":"2024-03-30T15:06:08.562395+0000","last_fullsized":"2024-03-30T15:06:08.562395+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:24.043636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"308.4","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343697+0000","last_change":"2024-03-30T15:06:05.444945+0000","last_active":"2024-03-30T15:06:07.343697+0000","last_peered":"2024-03-30T15:06:07.343697+0000","last_clean":"2024-03-30T15:06:07.343697+0000","last_became_active":"2024-03-30T15:06:05.444435+0000","last_became_peered":"2024-03-30T15:06:05.444435+0000","last_unstale":"2024-03-30T15:06:07.343697+0000","last_undegraded":"2024-03-30T15:06:07.343697+0000","last_fullsized":"2024-03-30T15:06:07.343697+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.748369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.2","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323388+0000","last_change":"2024-03-30T15:06:07.340866+0000","last_active":"2024-03-30T15:06:08.323388+0000","last_peered":"2024-03-30T15:06:08.323388+0000","last_clean":"2024-03-30T15:06:08.323388+0000","last_became_active":"2024-03-30T15:06:07.340538+0000","last_became_peered":"2024-03-30T15:06:07.340538+0000","last_unstale":"2024-03-30T15:06:08.323388+0000","last_undegraded":"2024-03-30T15:06:08.323388+0000","last_fullsized":"2024-03-30T15:06:08.323388+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.110161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"308.3","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339311+0000","last_change":"2024-03-30T15:06:05.443837+0000","last_active":"2024-03-30T15:06:07.339311+0000","last_peered":"2024-03-30T15:06:07.339311+0000","last_clean":"2024-03-30T15:06:07.339311+0000","last_became_active":"2024-03-30T15:06:05.443126+0000","last_became_peered":"2024-03-30T15:06:05.443126+0000","last_unstale":"2024-03-30T15:06:07.339311+0000","last_undegraded":"2024-03-30T15:06:07.339311+0000","last_fullsized":"2024-03-30T15:06:07.339311+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:13.673854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.3","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548403+0000","last_change":"2024-03-30T15:06:07.496794+0000","last_active":"2024-03-30T15:06:08.548403+0000","last_peered":"2024-03-30T15:06:08.548403+0000","last_clean":"2024-03-30T15:06:08.548403+0000","last_became_active":"2024-03-30T15:06:07.496481+0000","last_became_peered":"2024-03-30T15:06:07.496481+0000","last_unstale":"2024-03-30T15:06:08.548403+0000","last_undegraded":"2024-03-30T15:06:08.548403+0000","last_fullsized":"2024-03-30T15:06:08.548403+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:32.352169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"308.2","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562385+0000","last_change":"2024-03-30T15:06:05.331430+0000","last_active":"2024-03-30T15:06:08.562385+0000","last_peered":"2024-03-30T15:06:08.562385+0000","last_clean":"2024-03-30T15:06:08.562385+0000","last_became_active":"2024-03-30T15:06:05.331135+0000","last_became_peered":"2024-03-30T15:06:05.331135+0000","last_unstale":"2024-03-30T15:06:08.562385+0000","last_undegraded":"2024-03-30T15:06:08.562385+0000","last_fullsized":"2024-03-30T15:06:08.562385+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:21.344013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"309.0","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.322789+0000","last_change":"2024-03-30T15:06:07.494376+0000","last_active":"2024-03-30T15:06:08.322789+0000","last_peered":"2024-03-30T15:06:08.322789+0000","last_clean":"2024-03-30T15:06:08.322789+0000","last_became_active":"2024-03-30T15:06:07.493952+0000","last_became_peered":"2024-03-30T15:06:07.493952+0000","last_unstale":"2024-03-30T15:06:08.322789+0000","last_undegraded":"2024-03-30T15:06:08.322789+0000","last_fullsized":"2024-03-30T15:06:08.322789+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.882726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"308.1","version":"0'0","reported_seq":16,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.332697+0000","last_change":"2024-03-30T15:06:05.443561+0000","last_active":"2024-03-30T15:06:07.332697+0000","last_peered":"2024-03-30T15:06:07.332697+0000","last_clean":"2024-03-30T15:06:07.332697+0000","last_became_active":"2024-03-30T15:06:05.443197+0000","last_became_peered":"2024-03-30T15:06:05.443197+0000","last_unstale":"2024-03-30T15:06:07.332697+0000","last_undegraded":"2024-03-30T15:06:07.332697+0000","last_fullsized":"2024-03-30T15:06:07.332697+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.594528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"309.1","version":"0'0","reported_seq":13,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323197+0000","last_change":"2024-03-30T15:06:07.341787+0000","last_active":"2024-03-30T15:06:08.323197+0000","last_peered":"2024-03-30T15:06:08.323197+0000","last_clean":"2024-03-30T15:06:08.323197+0000","last_became_active":"2024-03-30T15:06:07.341449+0000","last_became_peered":"2024-03-30T15:06:07.341449+0000","last_unstale":"2024-03-30T15:06:08.323197+0000","last_undegraded":"2024-03-30T15:06:08.323197+0000","last_fullsized":"2024-03-30T15:06:08.323197+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:06.316258+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.316258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.938929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"308.0","version":"0'0","reported_seq":19,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548191+0000","last_change":"2024-03-30T15:06:05.330602+0000","last_active":"2024-03-30T15:06:08.548191+0000","last_peered":"2024-03-30T15:06:08.548191+0000","last_clean":"2024-03-30T15:06:08.548191+0000","last_became_active":"2024-03-30T15:06:05.330252+0000","last_became_peered":"2024-03-30T15:06:05.330252+0000","last_unstale":"2024-03-30T15:06:08.548191+0000","last_undegraded":"2024-03-30T15:06:08.548191+0000","last_fullsized":"2024-03-30T15:06:08.548191+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":531,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:04.312749+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.312749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:08.444119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.1a","version":"511'43","reported_seq":1070,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564144+0000","last_change":"2024-03-30T15:04:06.344710+0000","last_active":"2024-03-30T15:06:08.564144+0000","last_peered":"2024-03-30T15:06:08.564144+0000","last_clean":"2024-03-30T15:06:08.564144+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:06:08.564144+0000","last_undegraded":"2024-03-30T15:06:08.564144+0000","last_fullsized":"2024-03-30T15:06:08.564144+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'39","last_scrub_stamp":"2024-03-30T15:03:22.058552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.058552+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":18,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":1092,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339413+0000","last_change":"2024-03-30T15:05:25.203009+0000","last_active":"2024-03-30T15:06:07.339413+0000","last_peered":"2024-03-30T15:06:07.339413+0000","last_clean":"2024-03-30T15:06:07.339413+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:06:07.339413+0000","last_undegraded":"2024-03-30T15:06:07.339413+0000","last_fullsized":"2024-03-30T15:06:07.339413+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:05:25.202851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.202851+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:35.901031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"474'106","reported_seq":1118,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548181+0000","last_change":"2024-03-30T15:05:23.274570+0000","last_active":"2024-03-30T15:06:08.548181+0000","last_peered":"2024-03-30T15:06:08.548181+0000","last_clean":"2024-03-30T15:06:08.548181+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:06:08.548181+0000","last_undegraded":"2024-03-30T15:06:08.548181+0000","last_fullsized":"2024-03-30T15:06:08.548181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'106","last_scrub_stamp":"2024-03-30T15:05:23.274464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.274464+0000","objects_scrubbed":11,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.451911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":98,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"529'107","reported_seq":1125,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556640+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:06:08.556640+0000","last_peered":"2024-03-30T15:06:08.556640+0000","last_clean":"2024-03-30T15:06:08.556640+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:06:08.556640+0000","last_undegraded":"2024-03-30T15:06:08.556640+0000","last_fullsized":"2024-03-30T15:06:08.556640+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":2127,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"491'43","reported_seq":1076,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564096+0000","last_change":"2024-03-30T15:06:04.015679+0000","last_active":"2024-03-30T15:06:08.564096+0000","last_peered":"2024-03-30T15:06:08.564096+0000","last_clean":"2024-03-30T15:06:08.564096+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:06:08.564096+0000","last_undegraded":"2024-03-30T15:06:08.564096+0000","last_fullsized":"2024-03-30T15:06:08.564096+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'43","last_scrub_stamp":"2024-03-30T15:06:04.015550+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.015550+0000","objects_scrubbed":4,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.566952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"529'85","reported_seq":1167,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564478+0000","last_change":"2024-03-30T15:06:03.059243+0000","last_active":"2024-03-30T15:06:08.564478+0000","last_peered":"2024-03-30T15:06:08.564478+0000","last_clean":"2024-03-30T15:06:08.564478+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:06:08.564478+0000","last_undegraded":"2024-03-30T15:06:08.564478+0000","last_fullsized":"2024-03-30T15:06:08.564478+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'85","last_scrub_stamp":"2024-03-30T15:06:03.059121+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:06:03.059121+0000","objects_scrubbed":11,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.051044+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4195566,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":32,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"529'41","reported_seq":1057,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342781+0000","last_change":"2024-03-30T15:06:02.001529+0000","last_active":"2024-03-30T15:06:07.342781+0000","last_peered":"2024-03-30T15:06:07.342781+0000","last_clean":"2024-03-30T15:06:07.342781+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:06:07.342781+0000","last_undegraded":"2024-03-30T15:06:07.342781+0000","last_fullsized":"2024-03-30T15:06:07.342781+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'38","last_scrub_stamp":"2024-03-30T15:06:02.001426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.001426+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.654365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":594,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"520'57","reported_seq":1071,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549744+0000","last_change":"2024-03-30T15:04:06.473326+0000","last_active":"2024-03-30T15:06:08.549744+0000","last_peered":"2024-03-30T15:06:08.549744+0000","last_clean":"2024-03-30T15:06:08.549744+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:06:08.549744+0000","last_undegraded":"2024-03-30T15:06:08.549744+0000","last_fullsized":"2024-03-30T15:06:08.549744+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'53","last_scrub_stamp":"2024-03-30T15:03:50.417663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.417663+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":1067,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323219+0000","last_change":"2024-03-30T15:04:34.489872+0000","last_active":"2024-03-30T15:06:08.323219+0000","last_peered":"2024-03-30T15:06:08.323219+0000","last_clean":"2024-03-30T15:06:08.323219+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:06:08.323219+0000","last_undegraded":"2024-03-30T15:06:08.323219+0000","last_fullsized":"2024-03-30T15:06:08.323219+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:04:34.489778+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.489778+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"511'44","reported_seq":1071,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324108+0000","last_change":"2024-03-30T15:05:24.551301+0000","last_active":"2024-03-30T15:06:08.324108+0000","last_peered":"2024-03-30T15:06:08.324108+0000","last_clean":"2024-03-30T15:06:08.324108+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:06:08.324108+0000","last_undegraded":"2024-03-30T15:06:08.324108+0000","last_fullsized":"2024-03-30T15:06:08.324108+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:05:24.551158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.551158+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.999934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":24,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"520'30","reported_seq":1054,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548255+0000","last_change":"2024-03-30T15:05:24.227205+0000","last_active":"2024-03-30T15:06:08.548255+0000","last_peered":"2024-03-30T15:06:08.548255+0000","last_clean":"2024-03-30T15:06:08.548255+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:06:08.548255+0000","last_undegraded":"2024-03-30T15:06:08.548255+0000","last_fullsized":"2024-03-30T15:06:08.548255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'23","last_scrub_stamp":"2024-03-30T15:05:24.227122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.227122+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.963448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":2376,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"520'22","reported_seq":1045,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.547856+0000","last_change":"2024-03-30T15:05:25.269379+0000","last_active":"2024-03-30T15:06:08.547856+0000","last_peered":"2024-03-30T15:06:08.547856+0000","last_clean":"2024-03-30T15:06:08.547856+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:06:08.547856+0000","last_undegraded":"2024-03-30T15:06:08.547856+0000","last_fullsized":"2024-03-30T15:06:08.547856+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'21","last_scrub_stamp":"2024-03-30T15:05:25.269264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.269264+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.898384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":586,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"529'25","reported_seq":1043,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339433+0000","last_change":"2024-03-30T15:05:56.954018+0000","last_active":"2024-03-30T15:06:07.339433+0000","last_peered":"2024-03-30T15:06:07.339433+0000","last_clean":"2024-03-30T15:06:07.339433+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:06:07.339433+0000","last_undegraded":"2024-03-30T15:06:07.339433+0000","last_fullsized":"2024-03-30T15:06:07.339433+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'23","last_scrub_stamp":"2024-03-30T15:05:56.953844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:56.953844+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:57.561630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"529'83","reported_seq":1100,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325486+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:06:08.325486+0000","last_peered":"2024-03-30T15:06:08.325486+0000","last_clean":"2024-03-30T15:06:08.325486+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:06:08.325486+0000","last_undegraded":"2024-03-30T15:06:08.325486+0000","last_fullsized":"2024-03-30T15:06:08.325486+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":2377,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":3,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"510'62","reported_seq":1086,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.553247+0000","last_change":"2024-03-30T15:06:08.553305+0000","last_active":"2024-03-30T15:06:08.553247+0000","last_peered":"2024-03-30T15:06:08.553247+0000","last_clean":"2024-03-30T15:06:08.553247+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:06:08.553247+0000","last_undegraded":"2024-03-30T15:06:08.553247+0000","last_fullsized":"2024-03-30T15:06:08.553247+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'62","last_scrub_stamp":"2024-03-30T15:06:08.553141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.553141+0000","objects_scrubbed":6,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.407235+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"520'94","reported_seq":1112,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562520+0000","last_change":"2024-03-30T15:06:01.014001+0000","last_active":"2024-03-30T15:06:08.562520+0000","last_peered":"2024-03-30T15:06:08.562520+0000","last_clean":"2024-03-30T15:06:08.562520+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:06:08.562520+0000","last_undegraded":"2024-03-30T15:06:08.562520+0000","last_fullsized":"2024-03-30T15:06:08.562520+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'94","last_scrub_stamp":"2024-03-30T15:06:01.013913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.013913+0000","objects_scrubbed":7,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:14.917306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":1605,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"520'39","reported_seq":1058,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548645+0000","last_change":"2024-03-30T15:05:22.315909+0000","last_active":"2024-03-30T15:06:08.548645+0000","last_peered":"2024-03-30T15:06:08.548645+0000","last_clean":"2024-03-30T15:06:08.548645+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:06:08.548645+0000","last_undegraded":"2024-03-30T15:06:08.548645+0000","last_fullsized":"2024-03-30T15:06:08.548645+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'37","last_scrub_stamp":"2024-03-30T15:05:22.315812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.315812+0000","objects_scrubbed":4,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.650200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":1245,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":16,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"520'113","reported_seq":1131,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.337731+0000","last_change":"2024-03-30T15:05:28.105979+0000","last_active":"2024-03-30T15:06:07.337731+0000","last_peered":"2024-03-30T15:06:07.337731+0000","last_clean":"2024-03-30T15:06:07.337731+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:06:07.337731+0000","last_undegraded":"2024-03-30T15:06:07.337731+0000","last_fullsized":"2024-03-30T15:06:07.337731+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'111","last_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_deep_scrub":"499'111","last_deep_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_clean_scrub_stamp":"2024-03-30T15:05:28.105889+0000","objects_scrubbed":7,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.276671+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":1932,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":36,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"529'61","reported_seq":1085,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323168+0000","last_change":"2024-03-30T15:04:07.016310+0000","last_active":"2024-03-30T15:06:08.323168+0000","last_peered":"2024-03-30T15:06:08.323168+0000","last_clean":"2024-03-30T15:06:08.323168+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:06:08.323168+0000","last_undegraded":"2024-03-30T15:06:08.323168+0000","last_fullsized":"2024-03-30T15:06:08.323168+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'37","last_scrub_stamp":"2024-03-30T15:03:24.014830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.014830+0000","objects_scrubbed":3,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":551,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":25,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":1065,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.335873+0000","last_change":"2024-03-30T15:04:55.161018+0000","last_active":"2024-03-30T15:06:07.335873+0000","last_peered":"2024-03-30T15:06:07.335873+0000","last_clean":"2024-03-30T15:06:07.335873+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:06:07.335873+0000","last_undegraded":"2024-03-30T15:06:07.335873+0000","last_fullsized":"2024-03-30T15:06:07.335873+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:04:55.160916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.160916+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.035072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"529'120","reported_seq":1135,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342227+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:06:07.342227+0000","last_peered":"2024-03-30T15:06:07.342227+0000","last_clean":"2024-03-30T15:06:07.342227+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:06:07.342227+0000","last_undegraded":"2024-03-30T15:06:07.342227+0000","last_fullsized":"2024-03-30T15:06:07.342227+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":1621,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":15,"num_read_kb":5,"num_write":46,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"424'60","reported_seq":1087,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325529+0000","last_change":"2024-03-30T15:04:35.492607+0000","last_active":"2024-03-30T15:06:08.325529+0000","last_peered":"2024-03-30T15:06:08.325529+0000","last_clean":"2024-03-30T15:06:08.325529+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:06:08.325529+0000","last_undegraded":"2024-03-30T15:06:08.325529+0000","last_fullsized":"2024-03-30T15:06:08.325529+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'60","last_scrub_stamp":"2024-03-30T15:04:35.492526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.492526+0000","objects_scrubbed":6,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":1077,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549054+0000","last_change":"2024-03-30T15:04:06.473834+0000","last_active":"2024-03-30T15:06:08.549054+0000","last_peered":"2024-03-30T15:06:08.549054+0000","last_clean":"2024-03-30T15:06:08.549054+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:06:08.549054+0000","last_undegraded":"2024-03-30T15:06:08.549054+0000","last_fullsized":"2024-03-30T15:06:08.549054+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'56","last_scrub_stamp":"2024-03-30T15:03:49.463501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:49.463501+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"520'98","reported_seq":1105,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342135+0000","last_change":"2024-03-30T15:05:21.308139+0000","last_active":"2024-03-30T15:06:07.342135+0000","last_peered":"2024-03-30T15:06:07.342135+0000","last_clean":"2024-03-30T15:06:07.342135+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:06:07.342135+0000","last_undegraded":"2024-03-30T15:06:07.342135+0000","last_fullsized":"2024-03-30T15:06:07.342135+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'97","last_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_deep_scrub":"498'97","last_deep_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.308003+0000","objects_scrubbed":8,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:25.060048+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":1294,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":1116,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557264+0000","last_change":"2024-03-30T15:05:59.137056+0000","last_active":"2024-03-30T15:06:08.557264+0000","last_peered":"2024-03-30T15:06:08.557264+0000","last_clean":"2024-03-30T15:06:08.557264+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:06:08.557264+0000","last_undegraded":"2024-03-30T15:06:08.557264+0000","last_fullsized":"2024-03-30T15:06:08.557264+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'102","last_scrub_stamp":"2024-03-30T15:05:59.136916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.136916+0000","objects_scrubbed":8,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:00.982736+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"511'43","reported_seq":1074,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561671+0000","last_change":"2024-03-30T15:04:35.992306+0000","last_active":"2024-03-30T15:06:08.561671+0000","last_peered":"2024-03-30T15:06:08.561671+0000","last_clean":"2024-03-30T15:06:08.561671+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:06:08.561671+0000","last_undegraded":"2024-03-30T15:06:08.561671+0000","last_fullsized":"2024-03-30T15:06:08.561671+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:04:35.992167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.992167+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"510'44","reported_seq":1077,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560916+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:06:08.560916+0000","last_peered":"2024-03-30T15:06:08.560916+0000","last_clean":"2024-03-30T15:06:08.560916+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:06:08.560916+0000","last_undegraded":"2024-03-30T15:06:08.560916+0000","last_fullsized":"2024-03-30T15:06:08.560916+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"511'123","reported_seq":1145,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.336428+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:06:07.336428+0000","last_peered":"2024-03-30T15:06:07.336428+0000","last_clean":"2024-03-30T15:06:07.336428+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:06:07.336428+0000","last_undegraded":"2024-03-30T15:06:07.336428+0000","last_fullsized":"2024-03-30T15:06:07.336428+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000104,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":2,"num_write":43,"num_write_kb":3944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"529'30","reported_seq":1056,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551096+0000","last_change":"2024-03-30T15:05:21.307427+0000","last_active":"2024-03-30T15:06:08.551096+0000","last_peered":"2024-03-30T15:06:08.551096+0000","last_clean":"2024-03-30T15:06:08.551096+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:06:08.551096+0000","last_undegraded":"2024-03-30T15:06:08.551096+0000","last_fullsized":"2024-03-30T15:06:08.551096+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:05:21.307305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.307305+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.716318+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":1234,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":19,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"529'3","reported_seq":1041,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564556+0000","last_change":"2024-03-30T15:05:13.117385+0000","last_active":"2024-03-30T15:06:08.564556+0000","last_peered":"2024-03-30T15:06:08.564556+0000","last_clean":"2024-03-30T15:06:08.564556+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:06:08.564556+0000","last_undegraded":"2024-03-30T15:06:08.564556+0000","last_fullsized":"2024-03-30T15:06:08.564556+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:05:13.117303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.117303+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:28.803622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":716,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":1076,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324104+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:06:08.324104+0000","last_peered":"2024-03-30T15:06:08.324104+0000","last_clean":"2024-03-30T15:06:08.324104+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:06:08.324104+0000","last_undegraded":"2024-03-30T15:06:08.324104+0000","last_fullsized":"2024-03-30T15:06:08.324104+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"510'105","reported_seq":1122,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324843+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:06:08.324843+0000","last_peered":"2024-03-30T15:06:08.324843+0000","last_clean":"2024-03-30T15:06:08.324843+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:06:08.324843+0000","last_undegraded":"2024-03-30T15:06:08.324843+0000","last_fullsized":"2024-03-30T15:06:08.324843+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324337+0000","last_change":"2024-03-30T15:04:42.056061+0000","last_active":"2024-03-30T15:06:08.324337+0000","last_peered":"2024-03-30T15:06:08.324337+0000","last_clean":"2024-03-30T15:06:08.324337+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:06:08.324337+0000","last_undegraded":"2024-03-30T15:06:08.324337+0000","last_fullsized":"2024-03-30T15:06:08.324337+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.055955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.055955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":872,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340078+0000","last_change":"2024-03-30T15:04:34.266981+0000","last_active":"2024-03-30T15:06:07.340078+0000","last_peered":"2024-03-30T15:06:07.340078+0000","last_clean":"2024-03-30T15:06:07.340078+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:06:07.340078+0000","last_undegraded":"2024-03-30T15:06:07.340078+0000","last_fullsized":"2024-03-30T15:06:07.340078+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:34.266865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.266865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339724+0000","last_change":"2024-03-30T15:04:42.277788+0000","last_active":"2024-03-30T15:06:07.339724+0000","last_peered":"2024-03-30T15:06:07.339724+0000","last_clean":"2024-03-30T15:06:07.339724+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:06:07.339724+0000","last_undegraded":"2024-03-30T15:06:07.339724+0000","last_fullsized":"2024-03-30T15:06:07.339724+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.712874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.559880+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:06:08.559880+0000","last_peered":"2024-03-30T15:06:08.559880+0000","last_clean":"2024-03-30T15:06:08.559880+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:06:08.559880+0000","last_undegraded":"2024-03-30T15:06:08.559880+0000","last_fullsized":"2024-03-30T15:06:08.559880+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":870,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323668+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:06:08.323668+0000","last_peered":"2024-03-30T15:06:08.323668+0000","last_clean":"2024-03-30T15:06:08.323668+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:06:08.323668+0000","last_undegraded":"2024-03-30T15:06:08.323668+0000","last_fullsized":"2024-03-30T15:06:08.323668+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.284980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561031+0000","last_change":"2024-03-30T15:05:27.005973+0000","last_active":"2024-03-30T15:06:08.561031+0000","last_peered":"2024-03-30T15:06:08.561031+0000","last_clean":"2024-03-30T15:06:08.561031+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:06:08.561031+0000","last_undegraded":"2024-03-30T15:06:08.561031+0000","last_fullsized":"2024-03-30T15:06:08.561031+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.005887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.577288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339722+0000","last_change":"2024-03-30T15:05:20.323134+0000","last_active":"2024-03-30T15:06:07.339722+0000","last_peered":"2024-03-30T15:06:07.339722+0000","last_clean":"2024-03-30T15:06:07.339722+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:06:07.339722+0000","last_undegraded":"2024-03-30T15:06:07.339722+0000","last_fullsized":"2024-03-30T15:06:07.339722+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.323015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.323015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.231176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323688+0000","last_change":"2024-03-30T15:04:38.084291+0000","last_active":"2024-03-30T15:06:08.323688+0000","last_peered":"2024-03-30T15:06:08.323688+0000","last_clean":"2024-03-30T15:06:08.323688+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:06:08.323688+0000","last_undegraded":"2024-03-30T15:06:08.323688+0000","last_fullsized":"2024-03-30T15:06:08.323688+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:38.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":865,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549815+0000","last_change":"2024-03-30T15:05:19.276266+0000","last_active":"2024-03-30T15:06:08.549815+0000","last_peered":"2024-03-30T15:06:08.549815+0000","last_clean":"2024-03-30T15:06:08.549815+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:06:08.549815+0000","last_undegraded":"2024-03-30T15:06:08.549815+0000","last_fullsized":"2024-03-30T15:06:08.549815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.276185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.276185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.789727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557743+0000","last_change":"2024-03-30T15:03:14.727384+0000","last_active":"2024-03-30T15:06:08.557743+0000","last_peered":"2024-03-30T15:06:08.557743+0000","last_clean":"2024-03-30T15:06:08.557743+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:06:08.557743+0000","last_undegraded":"2024-03-30T15:06:08.557743+0000","last_fullsized":"2024-03-30T15:06:08.557743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.727306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.727306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339634+0000","last_change":"2024-03-30T15:05:17.336662+0000","last_active":"2024-03-30T15:06:07.339634+0000","last_peered":"2024-03-30T15:06:07.339634+0000","last_clean":"2024-03-30T15:06:07.339634+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:06:07.339634+0000","last_undegraded":"2024-03-30T15:06:07.339634+0000","last_fullsized":"2024-03-30T15:06:07.339634+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.336577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.336577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:21.490620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":878,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564200+0000","last_change":"2024-03-30T15:05:19.069973+0000","last_active":"2024-03-30T15:06:08.564200+0000","last_peered":"2024-03-30T15:06:08.564200+0000","last_clean":"2024-03-30T15:06:08.564200+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:06:08.564200+0000","last_undegraded":"2024-03-30T15:06:08.564200+0000","last_fullsized":"2024-03-30T15:06:08.564200+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.069896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.069896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:35.468169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.332761+0000","last_change":"2024-03-30T15:05:19.327818+0000","last_active":"2024-03-30T15:06:07.332761+0000","last_peered":"2024-03-30T15:06:07.332761+0000","last_clean":"2024-03-30T15:06:07.332761+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:06:07.332761+0000","last_undegraded":"2024-03-30T15:06:07.332761+0000","last_fullsized":"2024-03-30T15:06:07.332761+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.327720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.327720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.657218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556586+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:06:08.556586+0000","last_peered":"2024-03-30T15:06:08.556586+0000","last_clean":"2024-03-30T15:06:08.556586+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:06:08.556586+0000","last_undegraded":"2024-03-30T15:06:08.556586+0000","last_fullsized":"2024-03-30T15:06:08.556586+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561673+0000","last_change":"2024-03-30T15:04:37.008749+0000","last_active":"2024-03-30T15:06:08.561673+0000","last_peered":"2024-03-30T15:06:08.561673+0000","last_clean":"2024-03-30T15:06:08.561673+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:06:08.561673+0000","last_undegraded":"2024-03-30T15:06:08.561673+0000","last_fullsized":"2024-03-30T15:06:08.561673+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.008666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.008666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564242+0000","last_change":"2024-03-30T15:04:54.955662+0000","last_active":"2024-03-30T15:06:08.564242+0000","last_peered":"2024-03-30T15:06:08.564242+0000","last_clean":"2024-03-30T15:06:08.564242+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:06:08.564242+0000","last_undegraded":"2024-03-30T15:06:08.564242+0000","last_fullsized":"2024-03-30T15:06:08.564242+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.955541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.955541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:16.235351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342413+0000","last_change":"2024-03-30T15:04:55.271526+0000","last_active":"2024-03-30T15:06:07.342413+0000","last_peered":"2024-03-30T15:06:07.342413+0000","last_clean":"2024-03-30T15:06:07.342413+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:06:07.342413+0000","last_undegraded":"2024-03-30T15:06:07.342413+0000","last_fullsized":"2024-03-30T15:06:07.342413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.271452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.271452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":865,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549448+0000","last_change":"2024-03-30T15:04:45.608747+0000","last_active":"2024-03-30T15:06:08.549448+0000","last_peered":"2024-03-30T15:06:08.549448+0000","last_clean":"2024-03-30T15:06:08.549448+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:06:08.549448+0000","last_undegraded":"2024-03-30T15:06:08.549448+0000","last_fullsized":"2024-03-30T15:06:08.549448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.608652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.608652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325572+0000","last_change":"2024-03-30T15:04:36.136344+0000","last_active":"2024-03-30T15:06:08.325572+0000","last_peered":"2024-03-30T15:06:08.325572+0000","last_clean":"2024-03-30T15:06:08.325572+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:06:08.325572+0000","last_undegraded":"2024-03-30T15:06:08.325572+0000","last_fullsized":"2024-03-30T15:06:08.325572+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.136228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.136228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340672+0000","last_change":"2024-03-30T15:04:50.298381+0000","last_active":"2024-03-30T15:06:07.340672+0000","last_peered":"2024-03-30T15:06:07.340672+0000","last_clean":"2024-03-30T15:06:07.340672+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:06:07.340672+0000","last_undegraded":"2024-03-30T15:06:07.340672+0000","last_fullsized":"2024-03-30T15:06:07.340672+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.298282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556793+0000","last_change":"2024-03-30T15:02:40.580207+0000","last_active":"2024-03-30T15:06:08.556793+0000","last_peered":"2024-03-30T15:06:08.556793+0000","last_clean":"2024-03-30T15:06:08.556793+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:06:08.556793+0000","last_undegraded":"2024-03-30T15:06:08.556793+0000","last_fullsized":"2024-03-30T15:06:08.556793+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.580130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.580130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":865,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.331209+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:06:07.331209+0000","last_peered":"2024-03-30T15:06:07.331209+0000","last_clean":"2024-03-30T15:06:07.331209+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:06:07.331209+0000","last_undegraded":"2024-03-30T15:06:07.331209+0000","last_fullsized":"2024-03-30T15:06:07.331209+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":872,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.337887+0000","last_change":"2024-03-30T15:05:19.111999+0000","last_active":"2024-03-30T15:06:07.337887+0000","last_peered":"2024-03-30T15:06:07.337887+0000","last_clean":"2024-03-30T15:06:07.337887+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:06:07.337887+0000","last_undegraded":"2024-03-30T15:06:07.337887+0000","last_fullsized":"2024-03-30T15:06:07.337887+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.111909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.111909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.337658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562292+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:06:08.562292+0000","last_peered":"2024-03-30T15:06:08.562292+0000","last_clean":"2024-03-30T15:06:08.562292+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:06:08.562292+0000","last_undegraded":"2024-03-30T15:06:08.562292+0000","last_fullsized":"2024-03-30T15:06:08.562292+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":872,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341111+0000","last_change":"2024-03-30T15:05:29.140639+0000","last_active":"2024-03-30T15:06:07.341111+0000","last_peered":"2024-03-30T15:06:07.341111+0000","last_clean":"2024-03-30T15:06:07.341111+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:06:07.341111+0000","last_undegraded":"2024-03-30T15:06:07.341111+0000","last_fullsized":"2024-03-30T15:06:07.341111+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.140508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.140508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:57.419807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557926+0000","last_change":"2024-03-30T15:02:43.564201+0000","last_active":"2024-03-30T15:06:08.557926+0000","last_peered":"2024-03-30T15:06:08.557926+0000","last_clean":"2024-03-30T15:06:08.557926+0000","last_became_active":"2024-03-30T14:57:56.256466+0000","last_became_peered":"2024-03-30T14:57:56.256466+0000","last_unstale":"2024-03-30T15:06:08.557926+0000","last_undegraded":"2024-03-30T15:06:08.557926+0000","last_fullsized":"2024-03-30T15:06:08.557926+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:02:43.564123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325781+0000","last_change":"2024-03-30T15:04:55.123255+0000","last_active":"2024-03-30T15:06:08.325781+0000","last_peered":"2024-03-30T15:06:08.325781+0000","last_clean":"2024-03-30T15:06:08.325781+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:06:08.325781+0000","last_undegraded":"2024-03-30T15:06:08.325781+0000","last_fullsized":"2024-03-30T15:06:08.325781+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.123176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.123176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.501538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324331+0000","last_change":"2024-03-30T15:04:45.130399+0000","last_active":"2024-03-30T15:06:08.324331+0000","last_peered":"2024-03-30T15:06:08.324331+0000","last_clean":"2024-03-30T15:06:08.324331+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:06:08.324331+0000","last_undegraded":"2024-03-30T15:06:08.324331+0000","last_fullsized":"2024-03-30T15:06:08.324331+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.130322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.130322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":872,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.335997+0000","last_change":"2024-03-30T15:04:44.264561+0000","last_active":"2024-03-30T15:06:07.335997+0000","last_peered":"2024-03-30T15:06:07.335997+0000","last_clean":"2024-03-30T15:06:07.335997+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:06:07.335997+0000","last_undegraded":"2024-03-30T15:06:07.335997+0000","last_fullsized":"2024-03-30T15:06:07.335997+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.264430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.264430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":867,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323101+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:06:08.323101+0000","last_peered":"2024-03-30T15:06:08.323101+0000","last_clean":"2024-03-30T15:06:08.323101+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:06:08.323101+0000","last_undegraded":"2024-03-30T15:06:08.323101+0000","last_fullsized":"2024-03-30T15:06:08.323101+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":865,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548686+0000","last_change":"2024-03-30T15:04:54.465626+0000","last_active":"2024-03-30T15:06:08.548686+0000","last_peered":"2024-03-30T15:06:08.548686+0000","last_clean":"2024-03-30T15:06:08.548686+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:06:08.548686+0000","last_undegraded":"2024-03-30T15:06:08.548686+0000","last_fullsized":"2024-03-30T15:06:08.548686+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.465543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.465543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":864,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.338229+0000","last_change":"2024-03-30T15:04:53.311876+0000","last_active":"2024-03-30T15:06:07.338229+0000","last_peered":"2024-03-30T15:06:07.338229+0000","last_clean":"2024-03-30T15:06:07.338229+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:06:07.338229+0000","last_undegraded":"2024-03-30T15:06:07.338229+0000","last_fullsized":"2024-03-30T15:06:07.338229+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.311797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.311797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":878,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340260+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:06:07.340260+0000","last_peered":"2024-03-30T15:06:07.340260+0000","last_clean":"2024-03-30T15:06:07.340260+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:06:07.340260+0000","last_undegraded":"2024-03-30T15:06:07.340260+0000","last_fullsized":"2024-03-30T15:06:07.340260+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1000,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549416+0000","last_change":"2024-03-30T15:05:57.752605+0000","last_active":"2024-03-30T15:06:08.549416+0000","last_peered":"2024-03-30T15:06:08.549416+0000","last_clean":"2024-03-30T15:06:08.549416+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:06:08.549416+0000","last_undegraded":"2024-03-30T15:06:08.549416+0000","last_fullsized":"2024-03-30T15:06:08.549416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:05:57.752440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:57.752440+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:08.206404+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":872,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324397+0000","last_change":"2024-03-30T15:02:40.645645+0000","last_active":"2024-03-30T15:06:08.324397+0000","last_peered":"2024-03-30T15:06:08.324397+0000","last_clean":"2024-03-30T15:06:08.324397+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:06:08.324397+0000","last_undegraded":"2024-03-30T15:06:08.324397+0000","last_fullsized":"2024-03-30T15:06:08.324397+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:40.645507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:40.645507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":124,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341044+0000","last_change":"2024-03-30T15:05:06.360304+0000","last_active":"2024-03-30T15:06:07.341044+0000","last_peered":"2024-03-30T15:06:07.341044+0000","last_clean":"2024-03-30T15:06:07.341044+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:06:07.341044+0000","last_undegraded":"2024-03-30T15:06:07.341044+0000","last_fullsized":"2024-03-30T15:06:07.341044+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":954,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551505+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:06:08.551505+0000","last_peered":"2024-03-30T15:06:08.551505+0000","last_clean":"2024-03-30T15:06:08.551505+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:06:08.551505+0000","last_undegraded":"2024-03-30T15:06:08.551505+0000","last_fullsized":"2024-03-30T15:06:08.551505+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":878,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341796+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:06:07.341796+0000","last_peered":"2024-03-30T15:06:07.341796+0000","last_clean":"2024-03-30T15:06:07.341796+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:06:07.341796+0000","last_undegraded":"2024-03-30T15:06:07.341796+0000","last_fullsized":"2024-03-30T15:06:07.341796+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1019,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.550055+0000","last_change":"2024-03-30T15:04:34.512958+0000","last_active":"2024-03-30T15:06:08.550055+0000","last_peered":"2024-03-30T15:06:08.550055+0000","last_clean":"2024-03-30T15:06:08.550055+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:06:08.550055+0000","last_undegraded":"2024-03-30T15:06:08.550055+0000","last_fullsized":"2024-03-30T15:06:08.550055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:04:34.512838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.512838+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":980,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.336740+0000","last_change":"2024-03-30T15:04:48.217127+0000","last_active":"2024-03-30T15:06:07.336740+0000","last_peered":"2024-03-30T15:06:07.336740+0000","last_clean":"2024-03-30T15:06:07.336740+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:06:07.336740+0000","last_undegraded":"2024-03-30T15:06:07.336740+0000","last_fullsized":"2024-03-30T15:06:07.336740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.217046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.217046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":975,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.558762+0000","last_change":"2024-03-30T15:04:48.997109+0000","last_active":"2024-03-30T15:06:08.558762+0000","last_peered":"2024-03-30T15:06:08.558762+0000","last_clean":"2024-03-30T15:06:08.558762+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:06:08.558762+0000","last_undegraded":"2024-03-30T15:06:08.558762+0000","last_fullsized":"2024-03-30T15:06:08.558762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.997021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.997021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":972,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564616+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:06:08.564616+0000","last_peered":"2024-03-30T15:06:08.564616+0000","last_clean":"2024-03-30T15:06:08.564616+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:06:08.564616+0000","last_undegraded":"2024-03-30T15:06:08.564616+0000","last_fullsized":"2024-03-30T15:06:08.564616+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":880,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560224+0000","last_change":"2024-03-30T15:06:05.089656+0000","last_active":"2024-03-30T15:06:08.560224+0000","last_peered":"2024-03-30T15:06:08.560224+0000","last_clean":"2024-03-30T15:06:08.560224+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:06:08.560224+0000","last_undegraded":"2024-03-30T15:06:08.560224+0000","last_fullsized":"2024-03-30T15:06:08.560224+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:05.089580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.089580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:10.423897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1006,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551406+0000","last_change":"2024-03-30T15:05:17.276827+0000","last_active":"2024-03-30T15:06:08.551406+0000","last_peered":"2024-03-30T15:06:08.551406+0000","last_clean":"2024-03-30T15:06:08.551406+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:06:08.551406+0000","last_undegraded":"2024-03-30T15:06:08.551406+0000","last_fullsized":"2024-03-30T15:06:08.551406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:17.276735+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.276735+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.270995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":974,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548516+0000","last_change":"2024-03-30T15:05:14.260362+0000","last_active":"2024-03-30T15:06:08.548516+0000","last_peered":"2024-03-30T15:06:08.548516+0000","last_clean":"2024-03-30T15:06:08.548516+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:06:08.548516+0000","last_undegraded":"2024-03-30T15:06:08.548516+0000","last_fullsized":"2024-03-30T15:06:08.548516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.260283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.260283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:14.479226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":964,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340667+0000","last_change":"2024-03-30T15:05:58.982494+0000","last_active":"2024-03-30T15:06:07.340667+0000","last_peered":"2024-03-30T15:06:07.340667+0000","last_clean":"2024-03-30T15:06:07.340667+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:06:07.340667+0000","last_undegraded":"2024-03-30T15:06:07.340667+0000","last_fullsized":"2024-03-30T15:06:07.340667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.982407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.982407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.069312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":881,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340438+0000","last_change":"2024-03-30T15:06:01.027884+0000","last_active":"2024-03-30T15:06:07.340438+0000","last_peered":"2024-03-30T15:06:07.340438+0000","last_clean":"2024-03-30T15:06:07.340438+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:06:07.340438+0000","last_undegraded":"2024-03-30T15:06:07.340438+0000","last_fullsized":"2024-03-30T15:06:07.340438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.027770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.027770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:06.480899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":880,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560724+0000","last_change":"2024-03-30T15:06:08.169797+0000","last_active":"2024-03-30T15:06:08.560724+0000","last_peered":"2024-03-30T15:06:08.560724+0000","last_clean":"2024-03-30T15:06:08.560724+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:06:08.560724+0000","last_undegraded":"2024-03-30T15:06:08.560724+0000","last_fullsized":"2024-03-30T15:06:08.560724+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.169669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.169669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.236873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1021,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560819+0000","last_change":"2024-03-30T15:05:26.068603+0000","last_active":"2024-03-30T15:06:08.560819+0000","last_peered":"2024-03-30T15:06:08.560819+0000","last_clean":"2024-03-30T15:06:08.560819+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:06:08.560819+0000","last_undegraded":"2024-03-30T15:06:08.560819+0000","last_fullsized":"2024-03-30T15:06:08.560819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.068493+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.425413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557016+0000","last_change":"2024-03-30T15:04:50.960049+0000","last_active":"2024-03-30T15:06:08.557016+0000","last_peered":"2024-03-30T15:06:08.557016+0000","last_clean":"2024-03-30T15:06:08.557016+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:06:08.557016+0000","last_undegraded":"2024-03-30T15:06:08.557016+0000","last_fullsized":"2024-03-30T15:06:08.557016+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.959968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.959968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.339138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1113,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551456+0000","last_change":"2024-03-30T15:06:06.615931+0000","last_active":"2024-03-30T15:06:08.551456+0000","last_peered":"2024-03-30T15:06:08.551456+0000","last_clean":"2024-03-30T15:06:08.551456+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:06:08.551456+0000","last_undegraded":"2024-03-30T15:06:08.551456+0000","last_fullsized":"2024-03-30T15:06:08.551456+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:06.615780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.615780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.931187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":952,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342613+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:06:07.342613+0000","last_peered":"2024-03-30T15:06:07.342613+0000","last_clean":"2024-03-30T15:06:07.342613+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:06:07.342613+0000","last_undegraded":"2024-03-30T15:06:07.342613+0000","last_fullsized":"2024-03-30T15:06:07.342613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561577+0000","last_change":"2024-03-30T15:05:15.068570+0000","last_active":"2024-03-30T15:06:08.561577+0000","last_peered":"2024-03-30T15:06:08.561577+0000","last_clean":"2024-03-30T15:06:08.561577+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:06:08.561577+0000","last_undegraded":"2024-03-30T15:06:08.561577+0000","last_fullsized":"2024-03-30T15:06:08.561577+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.068496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.068496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.114011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":966,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548259+0000","last_change":"2024-03-30T15:05:18.317376+0000","last_active":"2024-03-30T15:06:08.548259+0000","last_peered":"2024-03-30T15:06:08.548259+0000","last_clean":"2024-03-30T15:06:08.548259+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:06:08.548259+0000","last_undegraded":"2024-03-30T15:06:08.548259+0000","last_fullsized":"2024-03-30T15:06:08.548259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.317302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.949240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":985,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323751+0000","last_change":"2024-03-30T15:05:22.145932+0000","last_active":"2024-03-30T15:06:08.323751+0000","last_peered":"2024-03-30T15:06:08.323751+0000","last_clean":"2024-03-30T15:06:08.323751+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:06:08.323751+0000","last_undegraded":"2024-03-30T15:06:08.323751+0000","last_fullsized":"2024-03-30T15:06:08.323751+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:05:22.145836+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.145836+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.469421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":950,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325627+0000","last_change":"2024-03-30T15:05:59.324853+0000","last_active":"2024-03-30T15:06:08.325627+0000","last_peered":"2024-03-30T15:06:08.325627+0000","last_clean":"2024-03-30T15:06:08.325627+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:06:08.325627+0000","last_undegraded":"2024-03-30T15:06:08.325627+0000","last_fullsized":"2024-03-30T15:06:08.325627+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:05:59.324757+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.324757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:05.068206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":954,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548894+0000","last_change":"2024-03-30T15:03:25.563148+0000","last_active":"2024-03-30T15:06:08.548894+0000","last_peered":"2024-03-30T15:06:08.548894+0000","last_clean":"2024-03-30T15:06:08.548894+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:06:08.548894+0000","last_undegraded":"2024-03-30T15:06:08.548894+0000","last_fullsized":"2024-03-30T15:06:08.548894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:25.563070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.563070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":884,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561066+0000","last_change":"2024-03-30T15:04:48.979522+0000","last_active":"2024-03-30T15:06:08.561066+0000","last_peered":"2024-03-30T15:06:08.561066+0000","last_clean":"2024-03-30T15:06:08.561066+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:06:08.561066+0000","last_undegraded":"2024-03-30T15:06:08.561066+0000","last_fullsized":"2024-03-30T15:06:08.561066+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.979438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.432525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":992,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556794+0000","last_change":"2024-03-30T15:02:44.572769+0000","last_active":"2024-03-30T15:06:08.556794+0000","last_peered":"2024-03-30T15:06:08.556794+0000","last_clean":"2024-03-30T15:06:08.556794+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:06:08.556794+0000","last_undegraded":"2024-03-30T15:06:08.556794+0000","last_fullsized":"2024-03-30T15:06:08.556794+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:02:44.572673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:02:44.572673+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1012,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548654+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:06:08.548654+0000","last_peered":"2024-03-30T15:06:08.548654+0000","last_clean":"2024-03-30T15:06:08.548654+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:06:08.548654+0000","last_undegraded":"2024-03-30T15:06:08.548654+0000","last_fullsized":"2024-03-30T15:06:08.548654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":973,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325505+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:06:08.325505+0000","last_peered":"2024-03-30T15:06:08.325505+0000","last_clean":"2024-03-30T15:06:08.325505+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:06:08.325505+0000","last_undegraded":"2024-03-30T15:06:08.325505+0000","last_fullsized":"2024-03-30T15:06:08.325505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":964,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323768+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:06:08.323768+0000","last_peered":"2024-03-30T15:06:08.323768+0000","last_clean":"2024-03-30T15:06:08.323768+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:06:08.323768+0000","last_undegraded":"2024-03-30T15:06:08.323768+0000","last_fullsized":"2024-03-30T15:06:08.323768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":884,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562392+0000","last_change":"2024-03-30T15:04:47.018205+0000","last_active":"2024-03-30T15:06:08.562392+0000","last_peered":"2024-03-30T15:06:08.562392+0000","last_clean":"2024-03-30T15:06:08.562392+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:06:08.562392+0000","last_undegraded":"2024-03-30T15:06:08.562392+0000","last_fullsized":"2024-03-30T15:06:08.562392+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.018083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.520606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1126,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324357+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:06:08.324357+0000","last_peered":"2024-03-30T15:06:08.324357+0000","last_clean":"2024-03-30T15:06:08.324357+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:06:08.324357+0000","last_undegraded":"2024-03-30T15:06:08.324357+0000","last_fullsized":"2024-03-30T15:06:08.324357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:36.047732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1014,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557768+0000","last_change":"2024-03-30T15:05:57.179353+0000","last_active":"2024-03-30T15:06:08.557768+0000","last_peered":"2024-03-30T15:06:08.557768+0000","last_clean":"2024-03-30T15:06:08.557768+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:06:08.557768+0000","last_undegraded":"2024-03-30T15:06:08.557768+0000","last_fullsized":"2024-03-30T15:06:08.557768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:05:57.179258+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:05:57.179258+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:58.467151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":972,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342963+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:06:07.342963+0000","last_peered":"2024-03-30T15:06:07.342963+0000","last_clean":"2024-03-30T15:06:07.342963+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:06:07.342963+0000","last_undegraded":"2024-03-30T15:06:07.342963+0000","last_fullsized":"2024-03-30T15:06:07.342963+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":959,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.322982+0000","last_change":"2024-03-30T15:02:36.686202+0000","last_active":"2024-03-30T15:06:08.322982+0000","last_peered":"2024-03-30T15:06:08.322982+0000","last_clean":"2024-03-30T15:06:08.322982+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:06:08.322982+0000","last_undegraded":"2024-03-30T15:06:08.322982+0000","last_fullsized":"2024-03-30T15:06:08.322982+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:36.686088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:36.686088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":877,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561954+0000","last_change":"2024-03-30T15:04:52.952256+0000","last_active":"2024-03-30T15:06:08.561954+0000","last_peered":"2024-03-30T15:06:08.561954+0000","last_clean":"2024-03-30T15:06:08.561954+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:06:08.561954+0000","last_undegraded":"2024-03-30T15:06:08.561954+0000","last_fullsized":"2024-03-30T15:06:08.561954+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.952177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":972,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.337386+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:06:07.337386+0000","last_peered":"2024-03-30T15:06:07.337386+0000","last_clean":"2024-03-30T15:06:07.337386+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:06:07.337386+0000","last_undegraded":"2024-03-30T15:06:07.337386+0000","last_fullsized":"2024-03-30T15:06:07.337386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1026,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324800+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:06:08.324800+0000","last_peered":"2024-03-30T15:06:08.324800+0000","last_clean":"2024-03-30T15:06:08.324800+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:06:08.324800+0000","last_undegraded":"2024-03-30T15:06:08.324800+0000","last_fullsized":"2024-03-30T15:06:08.324800+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":983,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.558648+0000","last_change":"2024-03-30T15:04:51.927488+0000","last_active":"2024-03-30T15:06:08.558648+0000","last_peered":"2024-03-30T15:06:08.558648+0000","last_clean":"2024-03-30T15:06:08.558648+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:06:08.558648+0000","last_undegraded":"2024-03-30T15:06:08.558648+0000","last_fullsized":"2024-03-30T15:06:08.558648+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.927412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.927412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":971,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556909+0000","last_change":"2024-03-30T15:05:23.007617+0000","last_active":"2024-03-30T15:06:08.556909+0000","last_peered":"2024-03-30T15:06:08.556909+0000","last_clean":"2024-03-30T15:06:08.556909+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:06:08.556909+0000","last_undegraded":"2024-03-30T15:06:08.556909+0000","last_fullsized":"2024-03-30T15:06:08.556909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.007527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.808237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325458+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:06:08.325458+0000","last_peered":"2024-03-30T15:06:08.325458+0000","last_clean":"2024-03-30T15:06:08.325458+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:06:08.325458+0000","last_undegraded":"2024-03-30T15:06:08.325458+0000","last_fullsized":"2024-03-30T15:06:08.325458+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.839469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":964,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.340765+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:06:07.340765+0000","last_peered":"2024-03-30T15:06:07.340765+0000","last_clean":"2024-03-30T15:06:07.340765+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:06:07.340765+0000","last_undegraded":"2024-03-30T15:06:07.340765+0000","last_fullsized":"2024-03-30T15:06:07.340765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1011,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325405+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:06:08.325405+0000","last_peered":"2024-03-30T15:06:08.325405+0000","last_clean":"2024-03-30T15:06:08.325405+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:06:08.325405+0000","last_undegraded":"2024-03-30T15:06:08.325405+0000","last_fullsized":"2024-03-30T15:06:08.325405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1672,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323324+0000","last_change":"2024-03-30T15:04:50.115333+0000","last_active":"2024-03-30T15:06:08.323324+0000","last_peered":"2024-03-30T15:06:08.323324+0000","last_clean":"2024-03-30T15:06:08.323324+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:06:08.323324+0000","last_undegraded":"2024-03-30T15:06:08.323324+0000","last_fullsized":"2024-03-30T15:06:08.323324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.115166+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":683,"num_read_kb":678,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":971,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323345+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:06:08.323345+0000","last_peered":"2024-03-30T15:06:08.323345+0000","last_clean":"2024-03-30T15:06:08.323345+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:06:08.323345+0000","last_undegraded":"2024-03-30T15:06:08.323345+0000","last_fullsized":"2024-03-30T15:06:08.323345+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":963,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557994+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:06:08.557994+0000","last_peered":"2024-03-30T15:06:08.557994+0000","last_clean":"2024-03-30T15:06:08.557994+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:06:08.557994+0000","last_undegraded":"2024-03-30T15:06:08.557994+0000","last_fullsized":"2024-03-30T15:06:08.557994+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":872,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324692+0000","last_change":"2024-03-30T15:04:37.123232+0000","last_active":"2024-03-30T15:06:08.324692+0000","last_peered":"2024-03-30T15:06:08.324692+0000","last_clean":"2024-03-30T15:06:08.324692+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:06:08.324692+0000","last_undegraded":"2024-03-30T15:06:08.324692+0000","last_fullsized":"2024-03-30T15:06:08.324692+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":968,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324653+0000","last_change":"2024-03-30T15:05:15.118857+0000","last_active":"2024-03-30T15:06:08.324653+0000","last_peered":"2024-03-30T15:06:08.324653+0000","last_clean":"2024-03-30T15:06:08.324653+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:06:08.324653+0000","last_undegraded":"2024-03-30T15:06:08.324653+0000","last_fullsized":"2024-03-30T15:06:08.324653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.118778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.118778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.638794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1163,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341534+0000","last_change":"2024-03-30T15:06:02.076986+0000","last_active":"2024-03-30T15:06:07.341534+0000","last_peered":"2024-03-30T15:06:07.341534+0000","last_clean":"2024-03-30T15:06:07.341534+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:06:07.341534+0000","last_undegraded":"2024-03-30T15:06:07.341534+0000","last_fullsized":"2024-03-30T15:06:07.341534+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:02.076864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.076864+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.210052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":866,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.338332+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:06:07.338332+0000","last_peered":"2024-03-30T15:06:07.338332+0000","last_clean":"2024-03-30T15:06:07.338332+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:06:07.338332+0000","last_undegraded":"2024-03-30T15:06:07.338332+0000","last_fullsized":"2024-03-30T15:06:07.338332+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1237,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323382+0000","last_change":"2024-03-30T15:05:27.137562+0000","last_active":"2024-03-30T15:06:08.323382+0000","last_peered":"2024-03-30T15:06:08.323382+0000","last_clean":"2024-03-30T15:06:08.323382+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:06:08.323382+0000","last_undegraded":"2024-03-30T15:06:08.323382+0000","last_fullsized":"2024-03-30T15:06:08.323382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:05:27.137472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.137472+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.903904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":952,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339899+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:06:07.339899+0000","last_peered":"2024-03-30T15:06:07.339899+0000","last_clean":"2024-03-30T15:06:07.339899+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:06:07.339899+0000","last_undegraded":"2024-03-30T15:06:07.339899+0000","last_fullsized":"2024-03-30T15:06:07.339899+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":877,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343456+0000","last_change":"2024-03-30T15:03:20.415455+0000","last_active":"2024-03-30T15:06:07.343456+0000","last_peered":"2024-03-30T15:06:07.343456+0000","last_clean":"2024-03-30T15:06:07.343456+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:06:07.343456+0000","last_undegraded":"2024-03-30T15:06:07.343456+0000","last_fullsized":"2024-03-30T15:06:07.343456+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:20.415377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.415377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":976,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.564343+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:06:08.564343+0000","last_peered":"2024-03-30T15:06:08.564343+0000","last_clean":"2024-03-30T15:06:08.564343+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:06:08.564343+0000","last_undegraded":"2024-03-30T15:06:08.564343+0000","last_fullsized":"2024-03-30T15:06:08.564343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1017,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560622+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:06:08.560622+0000","last_peered":"2024-03-30T15:06:08.560622+0000","last_clean":"2024-03-30T15:06:08.560622+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:06:08.560622+0000","last_undegraded":"2024-03-30T15:06:08.560622+0000","last_fullsized":"2024-03-30T15:06:08.560622+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":993,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561160+0000","last_change":"2024-03-30T15:04:55.918561+0000","last_active":"2024-03-30T15:06:08.561160+0000","last_peered":"2024-03-30T15:06:08.561160+0000","last_clean":"2024-03-30T15:06:08.561160+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:06:08.561160+0000","last_undegraded":"2024-03-30T15:06:08.561160+0000","last_fullsized":"2024-03-30T15:06:08.561160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.918411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":983,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556894+0000","last_change":"2024-03-30T15:04:53.983173+0000","last_active":"2024-03-30T15:06:08.556894+0000","last_peered":"2024-03-30T15:06:08.556894+0000","last_clean":"2024-03-30T15:06:08.556894+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:06:08.556894+0000","last_undegraded":"2024-03-30T15:06:08.556894+0000","last_fullsized":"2024-03-30T15:06:08.556894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.983055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:13.611127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":964,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324971+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:06:08.324971+0000","last_peered":"2024-03-30T15:06:08.324971+0000","last_clean":"2024-03-30T15:06:08.324971+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:06:08.324971+0000","last_undegraded":"2024-03-30T15:06:08.324971+0000","last_fullsized":"2024-03-30T15:06:08.324971+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324384+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:06:08.324384+0000","last_peered":"2024-03-30T15:06:08.324384+0000","last_clean":"2024-03-30T15:06:08.324384+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:06:08.324384+0000","last_undegraded":"2024-03-30T15:06:08.324384+0000","last_fullsized":"2024-03-30T15:06:08.324384+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.687947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":971,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324811+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:06:08.324811+0000","last_peered":"2024-03-30T15:06:08.324811+0000","last_clean":"2024-03-30T15:06:08.324811+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:06:08.324811+0000","last_undegraded":"2024-03-30T15:06:08.324811+0000","last_fullsized":"2024-03-30T15:06:08.324811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1128,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342920+0000","last_change":"2024-03-30T15:06:05.010246+0000","last_active":"2024-03-30T15:06:07.342920+0000","last_peered":"2024-03-30T15:06:07.342920+0000","last_clean":"2024-03-30T15:06:07.342920+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:06:07.342920+0000","last_undegraded":"2024-03-30T15:06:07.342920+0000","last_fullsized":"2024-03-30T15:06:07.342920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:06:05.010128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.010128+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.104222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":974,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548900+0000","last_change":"2024-03-30T15:04:53.505197+0000","last_active":"2024-03-30T15:06:08.548900+0000","last_peered":"2024-03-30T15:06:08.548900+0000","last_clean":"2024-03-30T15:06:08.548900+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:06:08.548900+0000","last_undegraded":"2024-03-30T15:06:08.548900+0000","last_fullsized":"2024-03-30T15:06:08.548900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.505107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.505107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:11.825117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":975,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.338425+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:06:07.338425+0000","last_peered":"2024-03-30T15:06:07.338425+0000","last_clean":"2024-03-30T15:06:07.338425+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:06:07.338425+0000","last_undegraded":"2024-03-30T15:06:07.338425+0000","last_fullsized":"2024-03-30T15:06:07.338425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":874,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551219+0000","last_change":"2024-03-30T15:05:59.777166+0000","last_active":"2024-03-30T15:06:08.551219+0000","last_peered":"2024-03-30T15:06:08.551219+0000","last_clean":"2024-03-30T15:06:08.551219+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:06:08.551219+0000","last_undegraded":"2024-03-30T15:06:08.551219+0000","last_fullsized":"2024-03-30T15:06:08.551219+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:59.776966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.776966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:01.928817+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324877+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:06:08.324877+0000","last_peered":"2024-03-30T15:06:08.324877+0000","last_clean":"2024-03-30T15:06:08.324877+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:06:08.324877+0000","last_undegraded":"2024-03-30T15:06:08.324877+0000","last_fullsized":"2024-03-30T15:06:08.324877+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.060229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1000,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324460+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:06:08.324460+0000","last_peered":"2024-03-30T15:06:08.324460+0000","last_clean":"2024-03-30T15:06:08.324460+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:06:08.324460+0000","last_undegraded":"2024-03-30T15:06:08.324460+0000","last_fullsized":"2024-03-30T15:06:08.324460+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":972,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343366+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:06:07.343366+0000","last_peered":"2024-03-30T15:06:07.343366+0000","last_clean":"2024-03-30T15:06:07.343366+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:06:07.343366+0000","last_undegraded":"2024-03-30T15:06:07.343366+0000","last_fullsized":"2024-03-30T15:06:07.343366+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":984,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562486+0000","last_change":"2024-03-30T15:06:06.028818+0000","last_active":"2024-03-30T15:06:08.562486+0000","last_peered":"2024-03-30T15:06:08.562486+0000","last_clean":"2024-03-30T15:06:08.562486+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:06:08.562486+0000","last_undegraded":"2024-03-30T15:06:08.562486+0000","last_fullsized":"2024-03-30T15:06:08.562486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.028733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.028733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.118957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":959,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323625+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:06:08.323625+0000","last_peered":"2024-03-30T15:06:08.323625+0000","last_clean":"2024-03-30T15:06:08.323625+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:06:08.323625+0000","last_undegraded":"2024-03-30T15:06:08.323625+0000","last_fullsized":"2024-03-30T15:06:08.323625+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":871,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549300+0000","last_change":"2024-03-30T15:04:46.590243+0000","last_active":"2024-03-30T15:06:08.549300+0000","last_peered":"2024-03-30T15:06:08.549300+0000","last_clean":"2024-03-30T15:06:08.549300+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:06:08.549300+0000","last_undegraded":"2024-03-30T15:06:08.549300+0000","last_fullsized":"2024-03-30T15:06:08.549300+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:46.590171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.590171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1057,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.337338+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:06:07.337338+0000","last_peered":"2024-03-30T15:06:07.337338+0000","last_clean":"2024-03-30T15:06:07.337338+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:06:07.337338+0000","last_undegraded":"2024-03-30T15:06:07.337338+0000","last_fullsized":"2024-03-30T15:06:07.337338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":976,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324720+0000","last_change":"2024-03-30T15:06:00.363324+0000","last_active":"2024-03-30T15:06:08.324720+0000","last_peered":"2024-03-30T15:06:08.324720+0000","last_clean":"2024-03-30T15:06:08.324720+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:06:08.324720+0000","last_undegraded":"2024-03-30T15:06:08.324720+0000","last_fullsized":"2024-03-30T15:06:08.324720+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.363175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.363175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:05.991392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":976,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561856+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:06:08.561856+0000","last_peered":"2024-03-30T15:06:08.561856+0000","last_clean":"2024-03-30T15:06:08.561856+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:06:08.561856+0000","last_undegraded":"2024-03-30T15:06:08.561856+0000","last_fullsized":"2024-03-30T15:06:08.561856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":964,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323270+0000","last_change":"2024-03-30T15:05:29.180863+0000","last_active":"2024-03-30T15:06:08.323270+0000","last_peered":"2024-03-30T15:06:08.323270+0000","last_clean":"2024-03-30T15:06:08.323270+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:06:08.323270+0000","last_undegraded":"2024-03-30T15:06:08.323270+0000","last_fullsized":"2024-03-30T15:06:08.323270+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.180786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.180786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.154763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":891,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.336139+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:06:07.336139+0000","last_peered":"2024-03-30T15:06:07.336139+0000","last_clean":"2024-03-30T15:06:07.336139+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:06:07.336139+0000","last_undegraded":"2024-03-30T15:06:07.336139+0000","last_fullsized":"2024-03-30T15:06:07.336139+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":976,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561186+0000","last_change":"2024-03-30T15:04:49.959125+0000","last_active":"2024-03-30T15:06:08.561186+0000","last_peered":"2024-03-30T15:06:08.561186+0000","last_clean":"2024-03-30T15:06:08.561186+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:06:08.561186+0000","last_undegraded":"2024-03-30T15:06:08.561186+0000","last_fullsized":"2024-03-30T15:06:08.561186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.959020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.959020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1010,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548803+0000","last_change":"2024-03-30T15:06:01.783446+0000","last_active":"2024-03-30T15:06:08.548803+0000","last_peered":"2024-03-30T15:06:08.548803+0000","last_clean":"2024-03-30T15:06:08.548803+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:06:08.548803+0000","last_undegraded":"2024-03-30T15:06:08.548803+0000","last_fullsized":"2024-03-30T15:06:08.548803+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:06:01.783327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.783327+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.764475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":976,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324653+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:06:08.324653+0000","last_peered":"2024-03-30T15:06:08.324653+0000","last_clean":"2024-03-30T15:06:08.324653+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:06:08.324653+0000","last_undegraded":"2024-03-30T15:06:08.324653+0000","last_fullsized":"2024-03-30T15:06:08.324653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:10.756007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":868,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323880+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:06:08.323880+0000","last_peered":"2024-03-30T15:06:08.323880+0000","last_clean":"2024-03-30T15:06:08.323880+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:06:08.323880+0000","last_undegraded":"2024-03-30T15:06:08.323880+0000","last_fullsized":"2024-03-30T15:06:08.323880+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":871,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548434+0000","last_change":"2024-03-30T15:04:37.589562+0000","last_active":"2024-03-30T15:06:08.548434+0000","last_peered":"2024-03-30T15:06:08.548434+0000","last_clean":"2024-03-30T15:06:08.548434+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:06:08.548434+0000","last_undegraded":"2024-03-30T15:06:08.548434+0000","last_fullsized":"2024-03-30T15:06:08.548434+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1115,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324628+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:06:08.324628+0000","last_peered":"2024-03-30T15:06:08.324628+0000","last_clean":"2024-03-30T15:06:08.324628+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:06:08.324628+0000","last_undegraded":"2024-03-30T15:06:08.324628+0000","last_fullsized":"2024-03-30T15:06:08.324628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.089310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1003,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.331883+0000","last_change":"2024-03-30T15:03:23.214326+0000","last_active":"2024-03-30T15:06:07.331883+0000","last_peered":"2024-03-30T15:06:07.331883+0000","last_clean":"2024-03-30T15:06:07.331883+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:06:07.331883+0000","last_undegraded":"2024-03-30T15:06:07.331883+0000","last_fullsized":"2024-03-30T15:06:07.331883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'15","last_scrub_stamp":"2024-03-30T15:03:23.214178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:23.214178+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":988,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325686+0000","last_change":"2024-03-30T15:04:44.117459+0000","last_active":"2024-03-30T15:06:08.325686+0000","last_peered":"2024-03-30T15:06:08.325686+0000","last_clean":"2024-03-30T15:06:08.325686+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:06:08.325686+0000","last_undegraded":"2024-03-30T15:06:08.325686+0000","last_fullsized":"2024-03-30T15:06:08.325686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:04:44.117358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.117358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":960,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.338683+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:06:07.338683+0000","last_peered":"2024-03-30T15:06:07.338683+0000","last_clean":"2024-03-30T15:06:07.338683+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:06:07.338683+0000","last_undegraded":"2024-03-30T15:06:07.338683+0000","last_fullsized":"2024-03-30T15:06:07.338683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.272716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":878,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343318+0000","last_change":"2024-03-30T15:04:49.244698+0000","last_active":"2024-03-30T15:06:07.343318+0000","last_peered":"2024-03-30T15:06:07.343318+0000","last_clean":"2024-03-30T15:06:07.343318+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:06:07.343318+0000","last_undegraded":"2024-03-30T15:06:07.343318+0000","last_fullsized":"2024-03-30T15:06:07.343318+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.244616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.244616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":561,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.499769+0000","last_change":"2024-03-30T15:05:13.602996+0000","last_active":"2024-03-30T15:06:07.499769+0000","last_peered":"2024-03-30T15:06:07.499769+0000","last_clean":"2024-03-30T15:06:07.499769+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:06:07.499769+0000","last_undegraded":"2024-03-30T15:06:07.499769+0000","last_fullsized":"2024-03-30T15:06:07.499769+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:05:13.602902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.602902+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.234443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":987,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.334112+0000","last_change":"2024-03-30T15:05:22.034492+0000","last_active":"2024-03-30T15:06:07.334112+0000","last_peered":"2024-03-30T15:06:07.334112+0000","last_clean":"2024-03-30T15:06:07.334112+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:06:07.334112+0000","last_undegraded":"2024-03-30T15:06:07.334112+0000","last_fullsized":"2024-03-30T15:06:07.334112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:22.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.034372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.488283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":955,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341481+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:06:07.341481+0000","last_peered":"2024-03-30T15:06:07.341481+0000","last_clean":"2024-03-30T15:06:07.341481+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:06:07.341481+0000","last_undegraded":"2024-03-30T15:06:07.341481+0000","last_fullsized":"2024-03-30T15:06:07.341481+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":955,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324481+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:06:08.324481+0000","last_peered":"2024-03-30T15:06:08.324481+0000","last_clean":"2024-03-30T15:06:08.324481+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:06:08.324481+0000","last_undegraded":"2024-03-30T15:06:08.324481+0000","last_fullsized":"2024-03-30T15:06:08.324481+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":871,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551185+0000","last_change":"2024-03-30T15:04:44.618218+0000","last_active":"2024-03-30T15:06:08.551185+0000","last_peered":"2024-03-30T15:06:08.551185+0000","last_clean":"2024-03-30T15:06:08.551185+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:06:08.551185+0000","last_undegraded":"2024-03-30T15:06:08.551185+0000","last_fullsized":"2024-03-30T15:06:08.551185+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.618135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.618135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":991,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325603+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:06:08.325603+0000","last_peered":"2024-03-30T15:06:08.325603+0000","last_clean":"2024-03-30T15:06:08.325603+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:06:08.325603+0000","last_undegraded":"2024-03-30T15:06:08.325603+0000","last_fullsized":"2024-03-30T15:06:08.325603+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":975,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324115+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:06:08.324115+0000","last_peered":"2024-03-30T15:06:08.324115+0000","last_clean":"2024-03-30T15:06:08.324115+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:06:08.324115+0000","last_undegraded":"2024-03-30T15:06:08.324115+0000","last_fullsized":"2024-03-30T15:06:08.324115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":60,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343174+0000","last_change":"2024-03-30T15:05:42.322974+0000","last_active":"2024-03-30T15:06:07.343174+0000","last_peered":"2024-03-30T15:06:07.343174+0000","last_clean":"2024-03-30T15:06:07.343174+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:06:07.343174+0000","last_undegraded":"2024-03-30T15:06:07.343174+0000","last_fullsized":"2024-03-30T15:06:07.343174+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":971,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.562034+0000","last_change":"2024-03-30T15:03:53.114040+0000","last_active":"2024-03-30T15:06:08.562034+0000","last_peered":"2024-03-30T15:06:08.562034+0000","last_clean":"2024-03-30T15:06:08.562034+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:06:08.562034+0000","last_undegraded":"2024-03-30T15:06:08.562034+0000","last_fullsized":"2024-03-30T15:06:08.562034+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.113916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.113916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324690+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:06:08.324690+0000","last_peered":"2024-03-30T15:06:08.324690+0000","last_clean":"2024-03-30T15:06:08.324690+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:06:08.324690+0000","last_undegraded":"2024-03-30T15:06:08.324690+0000","last_fullsized":"2024-03-30T15:06:08.324690+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:20.103043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":984,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343165+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:06:07.343165+0000","last_peered":"2024-03-30T15:06:07.343165+0000","last_clean":"2024-03-30T15:06:07.343165+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:06:07.343165+0000","last_undegraded":"2024-03-30T15:06:07.343165+0000","last_fullsized":"2024-03-30T15:06:07.343165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":978,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323532+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:06:08.323532+0000","last_peered":"2024-03-30T15:06:08.323532+0000","last_clean":"2024-03-30T15:06:08.323532+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:06:08.323532+0000","last_undegraded":"2024-03-30T15:06:08.323532+0000","last_fullsized":"2024-03-30T15:06:08.323532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":992,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324463+0000","last_change":"2024-03-30T15:03:20.993970+0000","last_active":"2024-03-30T15:06:08.324463+0000","last_peered":"2024-03-30T15:06:08.324463+0000","last_clean":"2024-03-30T15:06:08.324463+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:06:08.324463+0000","last_undegraded":"2024-03-30T15:06:08.324463+0000","last_fullsized":"2024-03-30T15:06:08.324463+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:03:20.993848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:20.993848+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":953,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551334+0000","last_change":"2024-03-30T15:04:35.553690+0000","last_active":"2024-03-30T15:06:08.551334+0000","last_peered":"2024-03-30T15:06:08.551334+0000","last_clean":"2024-03-30T15:06:08.551334+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:06:08.551334+0000","last_undegraded":"2024-03-30T15:06:08.551334+0000","last_fullsized":"2024-03-30T15:06:08.551334+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.553605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.553605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324262+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:06:08.324262+0000","last_peered":"2024-03-30T15:06:08.324262+0000","last_clean":"2024-03-30T15:06:08.324262+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:06:08.324262+0000","last_undegraded":"2024-03-30T15:06:08.324262+0000","last_fullsized":"2024-03-30T15:06:08.324262+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.969152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":924,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.558711+0000","last_change":"2024-03-30T15:04:38.199333+0000","last_active":"2024-03-30T15:06:08.558711+0000","last_peered":"2024-03-30T15:06:08.558711+0000","last_clean":"2024-03-30T15:06:08.558711+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:06:08.558711+0000","last_undegraded":"2024-03-30T15:06:08.558711+0000","last_fullsized":"2024-03-30T15:06:08.558711+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:04:38.199239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.199239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":967,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323632+0000","last_change":"2024-03-30T15:03:13.928414+0000","last_active":"2024-03-30T15:06:08.323632+0000","last_peered":"2024-03-30T15:06:08.323632+0000","last_clean":"2024-03-30T15:06:08.323632+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:06:08.323632+0000","last_undegraded":"2024-03-30T15:06:08.323632+0000","last_fullsized":"2024-03-30T15:06:08.323632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:13.928301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:13.928301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1105,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342906+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:06:07.342906+0000","last_peered":"2024-03-30T15:06:07.342906+0000","last_clean":"2024-03-30T15:06:07.342906+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:06:07.342906+0000","last_undegraded":"2024-03-30T15:06:07.342906+0000","last_fullsized":"2024-03-30T15:06:07.342906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":960,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325350+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:06:08.325350+0000","last_peered":"2024-03-30T15:06:08.325350+0000","last_clean":"2024-03-30T15:06:08.325350+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:06:08.325350+0000","last_undegraded":"2024-03-30T15:06:08.325350+0000","last_fullsized":"2024-03-30T15:06:08.325350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":870,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549168+0000","last_change":"2024-03-30T15:04:41.620132+0000","last_active":"2024-03-30T15:06:08.549168+0000","last_peered":"2024-03-30T15:06:08.549168+0000","last_clean":"2024-03-30T15:06:08.549168+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:06:08.549168+0000","last_undegraded":"2024-03-30T15:06:08.549168+0000","last_fullsized":"2024-03-30T15:06:08.549168+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:41.620027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:41.620027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":997,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549240+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:06:08.549240+0000","last_peered":"2024-03-30T15:06:08.549240+0000","last_clean":"2024-03-30T15:06:08.549240+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:06:08.549240+0000","last_undegraded":"2024-03-30T15:06:08.549240+0000","last_fullsized":"2024-03-30T15:06:08.549240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":973,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549268+0000","last_change":"2024-03-30T15:04:43.577081+0000","last_active":"2024-03-30T15:06:08.549268+0000","last_peered":"2024-03-30T15:06:08.549268+0000","last_clean":"2024-03-30T15:06:08.549268+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:06:08.549268+0000","last_undegraded":"2024-03-30T15:06:08.549268+0000","last_fullsized":"2024-03-30T15:06:08.549268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.576993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.576993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":974,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557694+0000","last_change":"2024-03-30T15:05:19.043801+0000","last_active":"2024-03-30T15:06:08.557694+0000","last_peered":"2024-03-30T15:06:08.557694+0000","last_clean":"2024-03-30T15:06:08.557694+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:06:08.557694+0000","last_undegraded":"2024-03-30T15:06:08.557694+0000","last_fullsized":"2024-03-30T15:06:08.557694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.043661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.043661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.583679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":871,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323438+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:06:08.323438+0000","last_peered":"2024-03-30T15:06:08.323438+0000","last_clean":"2024-03-30T15:06:08.323438+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:06:08.323438+0000","last_undegraded":"2024-03-30T15:06:08.323438+0000","last_fullsized":"2024-03-30T15:06:08.323438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":870,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548691+0000","last_change":"2024-03-30T15:04:40.601679+0000","last_active":"2024-03-30T15:06:08.548691+0000","last_peered":"2024-03-30T15:06:08.548691+0000","last_clean":"2024-03-30T15:06:08.548691+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:06:08.548691+0000","last_undegraded":"2024-03-30T15:06:08.548691+0000","last_fullsized":"2024-03-30T15:06:08.548691+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.601541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.601541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":995,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557839+0000","last_change":"2024-03-30T15:04:54.952409+0000","last_active":"2024-03-30T15:06:08.557839+0000","last_peered":"2024-03-30T15:06:08.557839+0000","last_clean":"2024-03-30T15:06:08.557839+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:06:08.557839+0000","last_undegraded":"2024-03-30T15:06:08.557839+0000","last_fullsized":"2024-03-30T15:06:08.557839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:54.952263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.952263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":990,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324453+0000","last_change":"2024-03-30T15:04:52.183278+0000","last_active":"2024-03-30T15:06:08.324453+0000","last_peered":"2024-03-30T15:06:08.324453+0000","last_clean":"2024-03-30T15:06:08.324453+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:06:08.324453+0000","last_undegraded":"2024-03-30T15:06:08.324453+0000","last_fullsized":"2024-03-30T15:06:08.324453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:04:52.183194+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.183194+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":132,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548420+0000","last_change":"2024-03-30T15:04:37.298232+0000","last_active":"2024-03-30T15:06:08.548420+0000","last_peered":"2024-03-30T15:06:08.548420+0000","last_clean":"2024-03-30T15:06:08.548420+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:06:08.548420+0000","last_undegraded":"2024-03-30T15:06:08.548420+0000","last_fullsized":"2024-03-30T15:06:08.548420+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.603115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.603115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":958,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.322933+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:06:08.322933+0000","last_peered":"2024-03-30T15:06:08.322933+0000","last_clean":"2024-03-30T15:06:08.322933+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:06:08.322933+0000","last_undegraded":"2024-03-30T15:06:08.322933+0000","last_fullsized":"2024-03-30T15:06:08.322933+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323184+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:06:08.323184+0000","last_peered":"2024-03-30T15:06:08.323184+0000","last_clean":"2024-03-30T15:06:08.323184+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:06:08.323184+0000","last_undegraded":"2024-03-30T15:06:08.323184+0000","last_fullsized":"2024-03-30T15:06:08.323184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.076034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":926,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324445+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:06:08.324445+0000","last_peered":"2024-03-30T15:06:08.324445+0000","last_clean":"2024-03-30T15:06:08.324445+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:06:08.324445+0000","last_undegraded":"2024-03-30T15:06:08.324445+0000","last_fullsized":"2024-03-30T15:06:08.324445+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.309540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":971,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341607+0000","last_change":"2024-03-30T15:06:00.028025+0000","last_active":"2024-03-30T15:06:07.341607+0000","last_peered":"2024-03-30T15:06:07.341607+0000","last_clean":"2024-03-30T15:06:07.341607+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:06:07.341607+0000","last_undegraded":"2024-03-30T15:06:07.341607+0000","last_fullsized":"2024-03-30T15:06:07.341607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.027914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.027914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.091022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":967,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323588+0000","last_change":"2024-03-30T15:05:21.130956+0000","last_active":"2024-03-30T15:06:08.323588+0000","last_peered":"2024-03-30T15:06:08.323588+0000","last_clean":"2024-03-30T15:06:08.323588+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:06:08.323588+0000","last_undegraded":"2024-03-30T15:06:08.323588+0000","last_fullsized":"2024-03-30T15:06:08.323588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:21.130862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.130862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:24.342532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":879,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324249+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:06:08.324249+0000","last_peered":"2024-03-30T15:06:08.324249+0000","last_clean":"2024-03-30T15:06:08.324249+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:06:08.324249+0000","last_undegraded":"2024-03-30T15:06:08.324249+0000","last_fullsized":"2024-03-30T15:06:08.324249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:00.791475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":872,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556933+0000","last_change":"2024-03-30T15:03:11.725558+0000","last_active":"2024-03-30T15:06:08.556933+0000","last_peered":"2024-03-30T15:06:08.556933+0000","last_clean":"2024-03-30T15:06:08.556933+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:06:08.556933+0000","last_undegraded":"2024-03-30T15:06:08.556933+0000","last_fullsized":"2024-03-30T15:06:08.556933+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:11.725443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:11.725443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":547,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.495387+0000","last_change":"2024-03-30T15:05:59.397565+0000","last_active":"2024-03-30T15:06:07.495387+0000","last_peered":"2024-03-30T15:06:07.495387+0000","last_clean":"2024-03-30T15:06:07.495387+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:06:07.495387+0000","last_undegraded":"2024-03-30T15:06:07.495387+0000","last_fullsized":"2024-03-30T15:06:07.495387+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:05:59.397484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.397484+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:13.734849+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":978,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325733+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:06:08.325733+0000","last_peered":"2024-03-30T15:06:08.325733+0000","last_clean":"2024-03-30T15:06:08.325733+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:06:08.325733+0000","last_undegraded":"2024-03-30T15:06:08.325733+0000","last_fullsized":"2024-03-30T15:06:08.325733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":979,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.336729+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:06:07.336729+0000","last_peered":"2024-03-30T15:06:07.336729+0000","last_clean":"2024-03-30T15:06:07.336729+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:06:07.336729+0000","last_undegraded":"2024-03-30T15:06:07.336729+0000","last_fullsized":"2024-03-30T15:06:07.336729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:32.616611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":963,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323597+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:06:08.323597+0000","last_peered":"2024-03-30T15:06:08.323597+0000","last_clean":"2024-03-30T15:06:08.323597+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:06:08.323597+0000","last_undegraded":"2024-03-30T15:06:08.323597+0000","last_fullsized":"2024-03-30T15:06:08.323597+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.275230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":877,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343562+0000","last_change":"2024-03-30T15:04:51.176640+0000","last_active":"2024-03-30T15:06:07.343562+0000","last_peered":"2024-03-30T15:06:07.343562+0000","last_clean":"2024-03-30T15:06:07.343562+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:06:07.343562+0000","last_undegraded":"2024-03-30T15:06:07.343562+0000","last_fullsized":"2024-03-30T15:06:07.343562+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.176549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.176549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1012,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343580+0000","last_change":"2024-03-30T15:05:59.994488+0000","last_active":"2024-03-30T15:06:07.343580+0000","last_peered":"2024-03-30T15:06:07.343580+0000","last_clean":"2024-03-30T15:06:07.343580+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:06:07.343580+0000","last_undegraded":"2024-03-30T15:06:07.343580+0000","last_fullsized":"2024-03-30T15:06:07.343580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:59.994304+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.994304+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.366647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":872,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323146+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:06:08.323146+0000","last_peered":"2024-03-30T15:06:08.323146+0000","last_clean":"2024-03-30T15:06:08.323146+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:06:08.323146+0000","last_undegraded":"2024-03-30T15:06:08.323146+0000","last_fullsized":"2024-03-30T15:06:08.323146+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":963,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325724+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:06:08.325724+0000","last_peered":"2024-03-30T15:06:08.325724+0000","last_clean":"2024-03-30T15:06:08.325724+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:06:08.325724+0000","last_undegraded":"2024-03-30T15:06:08.325724+0000","last_fullsized":"2024-03-30T15:06:08.325724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":970,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560182+0000","last_change":"2024-03-30T15:06:01.137816+0000","last_active":"2024-03-30T15:06:08.560182+0000","last_peered":"2024-03-30T15:06:08.560182+0000","last_clean":"2024-03-30T15:06:08.560182+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:06:08.560182+0000","last_undegraded":"2024-03-30T15:06:08.560182+0000","last_fullsized":"2024-03-30T15:06:08.560182+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.137663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.137663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.649632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":877,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342330+0000","last_change":"2024-03-30T15:03:15.456555+0000","last_active":"2024-03-30T15:06:07.342330+0000","last_peered":"2024-03-30T15:06:07.342330+0000","last_clean":"2024-03-30T15:06:07.342330+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:06:07.342330+0000","last_undegraded":"2024-03-30T15:06:07.342330+0000","last_fullsized":"2024-03-30T15:06:07.342330+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:15.456442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:15.456442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1002,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325555+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:06:08.325555+0000","last_peered":"2024-03-30T15:06:08.325555+0000","last_clean":"2024-03-30T15:06:08.325555+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:06:08.325555+0000","last_undegraded":"2024-03-30T15:06:08.325555+0000","last_fullsized":"2024-03-30T15:06:08.325555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":975,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557023+0000","last_change":"2024-03-30T15:05:24.043308+0000","last_active":"2024-03-30T15:06:08.557023+0000","last_peered":"2024-03-30T15:06:08.557023+0000","last_clean":"2024-03-30T15:06:08.557023+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:06:08.557023+0000","last_undegraded":"2024-03-30T15:06:08.557023+0000","last_fullsized":"2024-03-30T15:06:08.557023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:05:24.043163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.043163+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.017816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1130,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343495+0000","last_change":"2024-03-30T15:05:20.066039+0000","last_active":"2024-03-30T15:06:07.343495+0000","last_peered":"2024-03-30T15:06:07.343495+0000","last_clean":"2024-03-30T15:06:07.343495+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:06:07.343495+0000","last_undegraded":"2024-03-30T15:06:07.343495+0000","last_fullsized":"2024-03-30T15:06:07.343495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:20.065886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.065886+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.126952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":965,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324446+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:06:08.324446+0000","last_peered":"2024-03-30T15:06:08.324446+0000","last_clean":"2024-03-30T15:06:08.324446+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:06:08.324446+0000","last_undegraded":"2024-03-30T15:06:08.324446+0000","last_fullsized":"2024-03-30T15:06:08.324446+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":877,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.341216+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:06:07.341216+0000","last_peered":"2024-03-30T15:06:07.341216+0000","last_clean":"2024-03-30T15:06:07.341216+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:06:07.341216+0000","last_undegraded":"2024-03-30T15:06:07.341216+0000","last_fullsized":"2024-03-30T15:06:07.341216+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":926,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.559938+0000","last_change":"2024-03-30T15:04:40.214721+0000","last_active":"2024-03-30T15:06:08.559938+0000","last_peered":"2024-03-30T15:06:08.559938+0000","last_clean":"2024-03-30T15:06:08.559938+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:06:08.559938+0000","last_undegraded":"2024-03-30T15:06:08.559938+0000","last_fullsized":"2024-03-30T15:06:08.559938+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:40.214633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.214633+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:09.122719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":973,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.565821+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:06:08.565821+0000","last_peered":"2024-03-30T15:06:08.565821+0000","last_clean":"2024-03-30T15:06:08.565821+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:06:08.565821+0000","last_undegraded":"2024-03-30T15:06:08.565821+0000","last_fullsized":"2024-03-30T15:06:08.565821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:29.176862+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":965,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549549+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:06:08.549549+0000","last_peered":"2024-03-30T15:06:08.549549+0000","last_clean":"2024-03-30T15:06:08.549549+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:06:08.549549+0000","last_undegraded":"2024-03-30T15:06:08.549549+0000","last_fullsized":"2024-03-30T15:06:08.549549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.497561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":881,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343683+0000","last_change":"2024-03-30T15:05:15.145381+0000","last_active":"2024-03-30T15:06:07.343683+0000","last_peered":"2024-03-30T15:06:07.343683+0000","last_clean":"2024-03-30T15:06:07.343683+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:06:07.343683+0000","last_undegraded":"2024-03-30T15:06:07.343683+0000","last_fullsized":"2024-03-30T15:06:07.343683+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.145314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.145314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.542525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":869,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343522+0000","last_change":"2024-03-30T15:04:36.274480+0000","last_active":"2024-03-30T15:06:07.343522+0000","last_peered":"2024-03-30T15:06:07.343522+0000","last_clean":"2024-03-30T15:06:07.343522+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:06:07.343522+0000","last_undegraded":"2024-03-30T15:06:07.343522+0000","last_fullsized":"2024-03-30T15:06:07.343522+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.274391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.274391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1029,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561856+0000","last_change":"2024-03-30T15:03:25.102434+0000","last_active":"2024-03-30T15:06:08.561856+0000","last_peered":"2024-03-30T15:06:08.561856+0000","last_clean":"2024-03-30T15:06:08.561856+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:06:08.561856+0000","last_undegraded":"2024-03-30T15:06:08.561856+0000","last_fullsized":"2024-03-30T15:06:08.561856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'22","last_scrub_stamp":"2024-03-30T15:03:25.102272+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.102272+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":978,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.324708+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:06:08.324708+0000","last_peered":"2024-03-30T15:06:08.324708+0000","last_clean":"2024-03-30T15:06:08.324708+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:06:08.324708+0000","last_undegraded":"2024-03-30T15:06:08.324708+0000","last_fullsized":"2024-03-30T15:06:08.324708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":974,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.560137+0000","last_change":"2024-03-30T15:05:15.051014+0000","last_active":"2024-03-30T15:06:08.560137+0000","last_peered":"2024-03-30T15:06:08.560137+0000","last_clean":"2024-03-30T15:06:08.560137+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:06:08.560137+0000","last_undegraded":"2024-03-30T15:06:08.560137+0000","last_fullsized":"2024-03-30T15:06:08.560137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.050918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.050918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.102896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":954,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.551281+0000","last_change":"2024-03-30T15:03:52.434349+0000","last_active":"2024-03-30T15:06:08.551281+0000","last_peered":"2024-03-30T15:06:08.551281+0000","last_clean":"2024-03-30T15:06:08.551281+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:06:08.551281+0000","last_undegraded":"2024-03-30T15:06:08.551281+0000","last_fullsized":"2024-03-30T15:06:08.551281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.434269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":880,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557604+0000","last_change":"2024-03-30T15:06:04.136327+0000","last_active":"2024-03-30T15:06:08.557604+0000","last_peered":"2024-03-30T15:06:08.557604+0000","last_clean":"2024-03-30T15:06:08.557604+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:06:08.557604+0000","last_undegraded":"2024-03-30T15:06:08.557604+0000","last_fullsized":"2024-03-30T15:06:08.557604+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.136246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.136246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:29.092415+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":998,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549029+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:06:08.549029+0000","last_peered":"2024-03-30T15:06:08.549029+0000","last_clean":"2024-03-30T15:06:08.549029+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:06:08.549029+0000","last_undegraded":"2024-03-30T15:06:08.549029+0000","last_fullsized":"2024-03-30T15:06:08.549029+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":963,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.343278+0000","last_change":"2024-03-30T15:02:41.955737+0000","last_active":"2024-03-30T15:06:07.343278+0000","last_peered":"2024-03-30T15:06:07.343278+0000","last_clean":"2024-03-30T15:06:07.343278+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:06:07.343278+0000","last_undegraded":"2024-03-30T15:06:07.343278+0000","last_fullsized":"2024-03-30T15:06:07.343278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:41.955658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:02:41.955658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":967,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323567+0000","last_change":"2024-03-30T15:05:58.374684+0000","last_active":"2024-03-30T15:06:08.323567+0000","last_peered":"2024-03-30T15:06:08.323567+0000","last_clean":"2024-03-30T15:06:08.323567+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:06:08.323567+0000","last_undegraded":"2024-03-30T15:06:08.323567+0000","last_fullsized":"2024-03-30T15:06:08.323567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.374590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.374590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:14.674868+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":959,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.335095+0000","last_change":"2024-03-30T15:04:35.253973+0000","last_active":"2024-03-30T15:06:07.335095+0000","last_peered":"2024-03-30T15:06:07.335095+0000","last_clean":"2024-03-30T15:06:07.335095+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:06:07.335095+0000","last_undegraded":"2024-03-30T15:06:07.335095+0000","last_fullsized":"2024-03-30T15:06:07.335095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.253856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.253856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":866,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.557532+0000","last_change":"2024-03-30T15:02:42.582984+0000","last_active":"2024-03-30T15:06:08.557532+0000","last_peered":"2024-03-30T15:06:08.557532+0000","last_clean":"2024-03-30T15:06:08.557532+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:06:08.557532+0000","last_undegraded":"2024-03-30T15:06:08.557532+0000","last_fullsized":"2024-03-30T15:06:08.557532+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:42.582905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.582905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":986,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.549112+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:06:08.549112+0000","last_peered":"2024-03-30T15:06:08.549112+0000","last_clean":"2024-03-30T15:06:08.549112+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:06:08.549112+0000","last_undegraded":"2024-03-30T15:06:08.549112+0000","last_fullsized":"2024-03-30T15:06:08.549112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":963,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342763+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:06:07.342763+0000","last_peered":"2024-03-30T15:06:07.342763+0000","last_clean":"2024-03-30T15:06:07.342763+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:06:07.342763+0000","last_undegraded":"2024-03-30T15:06:07.342763+0000","last_fullsized":"2024-03-30T15:06:07.342763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":963,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.339550+0000","last_change":"2024-03-30T15:05:26.176781+0000","last_active":"2024-03-30T15:06:07.339550+0000","last_peered":"2024-03-30T15:06:07.339550+0000","last_clean":"2024-03-30T15:06:07.339550+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:06:07.339550+0000","last_undegraded":"2024-03-30T15:06:07.339550+0000","last_fullsized":"2024-03-30T15:06:07.339550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:26.176703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.176703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:49.697924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":966,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323146+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:06:08.323146+0000","last_peered":"2024-03-30T15:06:08.323146+0000","last_clean":"2024-03-30T15:06:08.323146+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:06:08.323146+0000","last_undegraded":"2024-03-30T15:06:08.323146+0000","last_fullsized":"2024-03-30T15:06:08.323146+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:17.231315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":869,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.334855+0000","last_change":"2024-03-30T15:04:52.290105+0000","last_active":"2024-03-30T15:06:07.334855+0000","last_peered":"2024-03-30T15:06:07.334855+0000","last_clean":"2024-03-30T15:06:07.334855+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:06:07.334855+0000","last_undegraded":"2024-03-30T15:06:07.334855+0000","last_fullsized":"2024-03-30T15:06:07.334855+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.289998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.289998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":987,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.561557+0000","last_change":"2024-03-30T15:05:17.006278+0000","last_active":"2024-03-30T15:06:08.561557+0000","last_peered":"2024-03-30T15:06:08.561557+0000","last_clean":"2024-03-30T15:06:08.561557+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:06:08.561557+0000","last_undegraded":"2024-03-30T15:06:08.561557+0000","last_fullsized":"2024-03-30T15:06:08.561557+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:17.006186+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.006186+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:19.567542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":875,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.342132+0000","last_change":"2024-03-30T15:03:19.435023+0000","last_active":"2024-03-30T15:06:07.342132+0000","last_peered":"2024-03-30T15:06:07.342132+0000","last_clean":"2024-03-30T15:06:07.342132+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:06:07.342132+0000","last_undegraded":"2024-03-30T15:06:07.342132+0000","last_fullsized":"2024-03-30T15:06:07.342132+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.434932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.434932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":957,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548478+0000","last_change":"2024-03-30T15:03:54.410901+0000","last_active":"2024-03-30T15:06:08.548478+0000","last_peered":"2024-03-30T15:06:08.548478+0000","last_clean":"2024-03-30T15:06:08.548478+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:06:08.548478+0000","last_undegraded":"2024-03-30T15:06:08.548478+0000","last_fullsized":"2024-03-30T15:06:08.548478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":535,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-30T15:06:07.498955+0000","last_change":"2024-03-30T15:05:55.432567+0000","last_active":"2024-03-30T15:06:07.498955+0000","last_peered":"2024-03-30T15:06:07.498955+0000","last_clean":"2024-03-30T15:06:07.498955+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:06:07.498955+0000","last_undegraded":"2024-03-30T15:06:07.498955+0000","last_fullsized":"2024-03-30T15:06:07.498955+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.432437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.432437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:18.685438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":876,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.323427+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:06:08.323427+0000","last_peered":"2024-03-30T15:06:08.323427+0000","last_clean":"2024-03-30T15:06:08.323427+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:06:08.323427+0000","last_undegraded":"2024-03-30T15:06:08.323427+0000","last_fullsized":"2024-03-30T15:06:08.323427+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.065353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1010,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548569+0000","last_change":"2024-03-30T15:04:39.592435+0000","last_active":"2024-03-30T15:06:08.548569+0000","last_peered":"2024-03-30T15:06:08.548569+0000","last_clean":"2024-03-30T15:06:08.548569+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:06:08.548569+0000","last_undegraded":"2024-03-30T15:06:08.548569+0000","last_fullsized":"2024-03-30T15:06:08.548569+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:39.592324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.592324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:08.690325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":979,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.325595+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:06:08.325595+0000","last_peered":"2024-03-30T15:06:08.325595+0000","last_clean":"2024-03-30T15:06:08.325595+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:06:08.325595+0000","last_undegraded":"2024-03-30T15:06:08.325595+0000","last_fullsized":"2024-03-30T15:06:08.325595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":966,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.556881+0000","last_change":"2024-03-30T15:05:27.054772+0000","last_active":"2024-03-30T15:06:08.556881+0000","last_peered":"2024-03-30T15:06:08.556881+0000","last_clean":"2024-03-30T15:06:08.556881+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:06:08.556881+0000","last_undegraded":"2024-03-30T15:06:08.556881+0000","last_fullsized":"2024-03-30T15:06:08.556881+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.054691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.277274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":961,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-30T15:06:08.548813+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:06:08.548813+0000","last_peered":"2024-03-30T15:06:08.548813+0000","last_clean":"2024-03-30T15:06:08.548813+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:06:08.548813+0000","last_undegraded":"2024-03-30T15:06:08.548813+0000","last_fullsized":"2024-03-30T15:06:08.548813+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.368798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":309,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":308,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8214741,"num_objects":191,"num_object_clones":10,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":215,"num_read_kb":59,"num_write":771,"num_write_kb":12801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26861568,"data_stored":24644199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2079,"ondisk_log_size":2079,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":683,"num_read_kb":678,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005199,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70604,"kb_used_data":7480,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93674548,"statfs":{"total":95995035648,"available":95922737152,"internally_reserved":0,"allocated":7659520,"data_stored":5566079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299540,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74692,"kb_used_data":7412,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670460,"statfs":{"total":95995035648,"available":95918551040,"internally_reserved":0,"allocated":7589888,"data_stored":5564955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66092,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93679060,"statfs":{"total":95995035648,"available":95927357440,"internally_reserved":0,"allocated":3047424,"data_stored":978997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65208,"kb_used_data":7660,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93679944,"statfs":{"total":95995035648,"available":95928262656,"internally_reserved":0,"allocated":7843840,"data_stored":5771666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411496,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78288,"kb_used_data":11004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666864,"statfs":{"total":95995035648,"available":95914868736,"internally_reserved":0,"allocated":11268096,"data_stored":9172964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117156,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74444,"kb_used_data":7168,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670708,"statfs":{"total":95995035648,"available":95918804992,"internally_reserved":0,"allocated":7340032,"data_stored":5175810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953630,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70324,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93674828,"statfs":{"total":95995035648,"available":95923023872,"internally_reserved":0,"allocated":3121152,"data_stored":987225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659289,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70296,"kb_used_data":3012,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93674856,"statfs":{"total":95995035648,"available":95923052544,"internally_reserved":0,"allocated":3084288,"data_stored":972960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":4201425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":9535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8204374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4530176,"data_stored":4205373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":8031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":4021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":4006327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4005113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":308,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:06:11.919 INFO:tasks.thrashosds.thrasher:Setting 42.c to [3, 7, 1] 2024-03-30T15:06:11.919 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '42.c', '3', '7', '1'] 2024-03-30T15:06:11.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 42.c 3 7 1 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 101.c scrub starts 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 101.c scrub ok 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T15:06:11.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: osdmap e538: 8 total, 8 up, 8 in 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-87"}]: dispatch 2024-03-30T15:06:11.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:12.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:12.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:12.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:12.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:12.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:06:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:12.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:12.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:12.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:12.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:12.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:12.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:12.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:12.845 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: 3.b deep-scrub starts 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: 3.b deep-scrub ok 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: 42.1 deep-scrub starts 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: 42.1 deep-scrub ok 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[41614]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T15:06:12.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: 3.b deep-scrub starts 2024-03-30T15:06:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: 3.b deep-scrub ok 2024-03-30T15:06:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: 42.1 deep-scrub starts 2024-03-30T15:06:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: 42.1 deep-scrub ok 2024-03-30T15:06:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:12 smithi028 ceph-mon[29876]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T15:06:12.882 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:06:12.882 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:06:12.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:06:12.882 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24907.0:2 24.b 24:d3fd6930:test-rados-api-smithi028-93417-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 497.91189661599998, 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27101285899999999, 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24907", 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1522502118", 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.883 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966645+0000", 2024-03-30T15:06:12.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966647+0000", 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.165e-06 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966656+0000", 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4060000000000005e-06 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966667+0000", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.118e-05 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966679+0000", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.202e-05 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967482+0000", 2024-03-30T15:06:12.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080304200000000001 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.967489+0000", 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8129999999999998e-06 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093584+0000", 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12609579200000001 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.093596+0000", 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1165000000000001e-05 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224181+0000", 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13058547400000001 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224307+0000", 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125637 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.224451+0000", 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014380600000000001 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226643+0000", 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021921750000000002 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227566+0000", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092316999999999996 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237564+0000", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0099978580000000001 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237612+0000", 2024-03-30T15:06:12.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8216000000000001e-05 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237658+0000", 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5939999999999997e-05 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:2 23.10 23:0f4c2fcb:test-rados-api-smithi028-93408-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 497.90725081199997, 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26655647300000002, 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971291+0000", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971294+0000", 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.242e-06 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971305+0000", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0713999999999999e-05 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971327+0000", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2089000000000001e-05 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971340+0000", 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2721999999999999e-05 2024-03-30T15:06:12.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971419+0000", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9756e-05 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.971426+0000", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5610000000000004e-06 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072900+0000", 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.101473934 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.072909+0000", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5359999999999998e-06 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.226905+0000", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153995785 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227020+0000", 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011438300000000001 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227154+0000", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013489 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.227892+0000", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00073729500000000001 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.235325+0000", 2024-03-30T15:06:12.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007433762 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237751+0000", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024258859999999999 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237804+0000", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2366e-05 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.237847+0000", 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3551999999999999e-05 2024-03-30T15:06:12.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24940.0:2 26.5 26:a21fad64:test-rados-api-smithi028-93750-3::testobj:head [write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 497.89910108599997, 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.25955751599999999, 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24940", 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1333607810", 2024-03-30T15:06:12.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979441+0000", 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979443+0000", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8150000000000002e-06 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979453+0000", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3689999999999996e-06 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979464+0000", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1061e-05 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.979475+0000", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1506e-05 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057216+0000", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.077740857999999996 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.057225+0000", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210726+0000", 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.153500514 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.210858+0000", 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132724 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.211010+0000", 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015139900000000001 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.215902+0000", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048923489999999998 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.230960+0000", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015057615 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238899+0000", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079398179999999995 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238954+0000", 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4842000000000001e-05 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.238998+0000", 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3739000000000003e-05 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34880.0:2 13.16 13:6e692731:test-rados-api-smithi028-93525-2::foo:head [write 0~128] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:06:12.906 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 497.73091516699998, 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.148417239, 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34880", 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147626+0000", 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147629+0000", 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147636+0000", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6860000000000006e-06 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147651+0000", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5071e-05 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147662+0000", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1103e-05 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147747+0000", 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4765000000000006e-05 2024-03-30T15:06:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.147757+0000", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4099999999999997e-06 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280327+0000", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.132570166 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280452+0000", 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125182 2024-03-30T15:06:12.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:06:12.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280591+0000", 2024-03-30T15:06:12.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013946 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282110+0000", 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015181719999999999 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282802+0000", 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069276799999999996 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295940+0000", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013137729000000001 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295998+0000", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.766e-05 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.296044+0000", 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5887000000000003e-05 2024-03-30T15:06:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35207.0:2 45.6 45:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 496.74462457800001, 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.151675741, 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.914 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35207", 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/517891268", 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133917+0000", 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133919+0000", 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.902e-06 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133926+0000", 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4460000000000004e-06 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133942+0000", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5692000000000002e-05 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.133954+0000", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.169e-05 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134023+0000", 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8934000000000001e-05 2024-03-30T15:06:12.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.134029+0000", 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6760000000000001e-06 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223013+0000", 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088983584000000004 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.223022+0000", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5620000000000004e-06 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264315+0000", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041293273999999998 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264431+0000", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116506 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:06:12.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.264556+0000", 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124237 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.279474+0000", 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014917977000000001 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280282+0000", 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080855300000000005 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285498+0000", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052162550000000004 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285549+0000", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0924999999999998e-05 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.285593+0000", 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3528000000000001e-05 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2059 11.7 e116/110 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head v 116'1, mlcod=0'0)", 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 494.908433699, 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061806397999999999, 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970108+0000", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.970112+0000", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0609999999999997e-06 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013285+0000", 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.04317315 2024-03-30T15:06:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013310+0000", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4615000000000001e-05 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013320+0000", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0107e-05 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013400+0000", 2024-03-30T15:06:12.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0039999999999999e-05 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.013425+0000", 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5494e-05 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031865+0000", 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01844001 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.031914+0000", 2024-03-30T15:06:12.926 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8921000000000001e-05 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2074 11.15 e117/110 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head v 117'1, mlcod=0'0)", 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 494.20109828199998, 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044884778, 2024-03-30T15:06:12.927 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677443+0000", 2024-03-30T15:06:12.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.677447+0000", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.811e-06 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689208+0000", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011761303000000001 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689235+0000", 2024-03-30T15:06:12.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6809e-05 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689248+0000", 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2918000000000001e-05 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689313+0000", 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4999999999999994e-05 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.689337+0000", 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3526000000000001e-05 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722259+0000", 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032922117000000001 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.722328+0000", 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9294000000000004e-05 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32236.0:2 121.0 121:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 471.86386050700003, 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2733807989999999, 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32236", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/520874703", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014685+0000", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4340000000000001e-06 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014695+0000", 2024-03-30T15:06:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0513000000000001e-05 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014709+0000", 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3791000000000001e-05 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014721+0000", 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2165e-05 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075744+0000", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061023012000000001 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075753+0000", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2e-06 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162599+0000", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0868460719999999 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162721+0000", 2024-03-30T15:06:12.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121662 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.171761+0000", 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090404189999999992 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.176616+0000", 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004854934 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.272952+0000", 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.096335658000000005 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.287962+0000", 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015010364 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288013+0000", 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0917000000000001e-05 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288062+0000", 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8658000000000001e-05 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1598 124.0 e147/140 124:0384ab8e:::oid:1 v 147'6, mlcod=144'3)", 2024-03-30T15:06:12.938 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 461.48433257300002, 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050952077999999998, 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394210+0000", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.102e-06 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394226+0000", 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554e-05 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394235+0000", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394244+0000", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2120000000000003e-06 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396824+0000", 2024-03-30T15:06:12.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025802059999999998 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396852+0000", 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8252999999999999e-05 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445122+0000", 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048270362999999997 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445161+0000", 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8736999999999999e-05 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:35 111.1 e149/134 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head v 149'7, mlcod=148'4)", 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 459.65359126200002, 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043878153000000003, 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.943 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224951+0000", 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.91e-07 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224962+0000", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0858e-05 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224967+0000", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9699999999999996e-06 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224977+0000", 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1079999999999999e-06 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235114+0000", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010137607999999999 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235139+0000", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4437e-05 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268792+0000", 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033653216 2024-03-30T15:06:12.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268829+0000", 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6665000000000002e-05 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.48856.0:9 157.2s0 157:523311ec:test-rados-api-smithi028-93441-13::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 446.692851754, 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053529637999999997, 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.947 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.48856", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1328251834", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185693+0000", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1939999999999998e-06 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185705+0000", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1446e-05 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185727+0000", 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2249e-05 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185739+0000", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185814+0000", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5013999999999997e-05 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185822+0000", 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5329999999999997e-06 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198631+0000", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012809375 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198727+0000", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5884000000000006e-05 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.199305+0000", 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057763400000000005 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.211644+0000", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012338905000000001 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239140+0000", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027495908999999999 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239220+0000", 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9802999999999995e-05 2024-03-30T15:06:12.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.56915.0:2 173.1b 173:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 443.69296978199998, 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.049005913, 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.56915", 2024-03-30T15:06:12.953 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1224869641", 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185574+0000", 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9039999999999999e-06 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185581+0000", 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7090000000000002e-06 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185595+0000", 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3257e-05 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185606+0000", 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1419e-05 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193710+0000", 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081039819999999992 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193719+0000", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1670000000000005e-06 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223661+0000", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299414410000001 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223739+0000", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8771999999999997e-05 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223867+0000", 2024-03-30T15:06:12.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705999999999999 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.225806+0000", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019391670000000001 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.227421+0000", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00161532 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234490+0000", 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070693969999999998 2024-03-30T15:06:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234537+0000", 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6709999999999998e-05 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234578+0000", 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0608000000000002e-05 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.959 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57314.0:2 223.9 223:94beac24:::oid_RoundTripCmpExtPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 383.67774327299998, 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16945650800000001, 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57314", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2222872349", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:06:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200800+0000", 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1320000000000001e-06 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200808+0000", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2590000000000003e-06 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200817+0000", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200827+0000", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0472000000000001e-05 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200895+0000", 2024-03-30T15:06:12.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7424999999999999e-05 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200900+0000", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8420000000000001e-06 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362513+0000", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16161309700000001 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362526+0000", 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2933e-05 2024-03-30T15:06:12.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.368887+0000", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063609690000000002 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369004+0000", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116635 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369145+0000", 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014146499999999999 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370068+0000", 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092285399999999997 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370108+0000", 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9529e-05 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370170+0000", 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1995999999999994e-05 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370216+0000", 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5865000000000002e-05 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370255+0000", 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9128000000000002e-05 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.966 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.2.0:1538 101.3 e248/130 101:c3bfc9e7:test-rados-api-smithi028-93253-24::bar:8 v 248'39, mlcod=243'35)", 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 360.673248113, 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050164583999999998, 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:06:12.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6100000000000001e-07 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205300+0000", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2219999999999999e-06 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227707+0000", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022406780000000001 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227716+0000", 2024-03-30T15:06:12.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5339999999999999e-06 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231237+0000", 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035216029999999999 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231261+0000", 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3981999999999998e-05 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255425+0000", 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024164396000000001 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255458+0000", 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2605999999999999e-05 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24402.0:1 256.7s0 256:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:06:12.971 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 322.64898889599999, 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005296856, 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24402", 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2485876767", 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:06:12.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229555+0000", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229563+0000", 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9759999999999995e-06 2024-03-30T15:06:12.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229573+0000", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0592e-05 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229584+0000", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0533000000000001e-05 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229647+0000", 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3118000000000001e-05 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229651+0000", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8680000000000001e-06 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232864+0000", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0032131120000001 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232980+0000", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011602100000000001 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.233406+0000", 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042549999999999999 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234188+0000", 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078191600000000001 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234777+0000", 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00058978600000000002 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234850+0000", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2254000000000003e-05 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:4660 275.0 275:00000000::::head [pgnls start_epoch 344] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e344)", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 264.63345390400002, 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.266339783, 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:06:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4660 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245090+0000", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.667e-06 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245101+0000", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0541e-05 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245116+0000", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5091e-05 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245129+0000", 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.278e-05 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245194+0000", 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5180999999999998e-05 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245199+0000", 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.118e-06 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511197+0000", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26599746200000002 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511236+0000", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9444999999999999e-05 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511428+0000", 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019149799999999999 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57803.0:1 280.4s0 280:384b83b2:::oid_RoundTripWriteFullPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 249.640455081, 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0069688640000001, 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57803", 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2168501096", 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238088+0000", 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9439999999999999e-06 2024-03-30T15:06:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238096+0000", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8110000000000003e-06 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238109+0000", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3004e-05 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238121+0000", 2024-03-30T15:06:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1703e-05 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.246334+0000", 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082125289999999997 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.246337+0000", 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.219e-06 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.242417+0000", 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99608009799999997 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.242549+0000", 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132533 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.243002+0000", 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045222099999999999 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.243973+0000", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00097163700000000002 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.244983+0000", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001009355 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.245055+0000", 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2810000000000003e-05 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35600.0:6785 101.c e463/130 101:33efc025:test-rados-api-smithi028-93253-70::foo:head v 463'57, mlcod=424'55)", 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 110.57335183000001, 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010387468, 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3799999999999998e-07 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305197+0000", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4060000000000002e-06 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309914+0000", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047169780000000001 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309931+0000", 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7027000000000001e-05 2024-03-30T15:06:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309985+0000", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4101999999999998e-05 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.310010+0000", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4870000000000001e-05 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315540+0000", 2024-03-30T15:06:12.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005530559 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315577+0000", 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7187999999999997e-05 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(osd.6.52:32 101.19 101:9ac2b7c9:test-rados-api-smithi028-93253-84::4a1b15efcf64c48b2771596f22c4392a83f101f1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e520)", 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:05:53.320995+0000", 2024-03-30T15:06:12.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 19.557546935000001, 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060927339999999998, 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.320995+0000", 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:12.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.320995+0000", 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.320996+0000", 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7889999999999999e-06 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.321018+0000", 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1824999999999999e-05 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.321029+0000", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0893000000000001e-05 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.321039+0000", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.482e-06 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.321108+0000", 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8925999999999997e-05 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.321114+0000", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.178e-06 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.324498+0000", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033843990000000002 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.324641+0000", 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014291 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.324888+0000", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024729299999999999 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.325396+0000", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050715799999999996 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.326919+0000", 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015229880000000001 2024-03-30T15:06:12.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.326982+0000", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3759999999999999e-05 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.327045+0000", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3144000000000004e-05 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:05:53.327087+0000", 2024-03-30T15:06:12.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1989000000000001e-05 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(3.1a,QREQUEST,e:535,from:4,reservation_nonce:8)", 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:06:08.555159+0000", 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.3233830930000003, 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029832610000000001, 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:06:12.999 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555159+0000", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555159+0000", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555162+0000", 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0709999999999999e-06 2024-03-30T15:06:13.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555170+0000", 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.439e-06 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555188+0000", 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8176e-05 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555200+0000", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1435e-05 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555268+0000", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8583000000000004e-05 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.555275+0000", 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7900000000000002e-06 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.557725+0000", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002449943 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.557740+0000", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.526e-05 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:08.558142+0000", 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040156399999999998 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:13.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:13.004 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:13.004 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:06:13.004 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: 3.b deep-scrub starts 2024-03-30T15:06:13.004 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: 3.b deep-scrub ok 2024-03-30T15:06:13.004 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: pgmap v782: 260 pgs: 260 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:06:13.004 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: 42.1 deep-scrub starts 2024-03-30T15:06:13.005 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: 42.1 deep-scrub ok 2024-03-30T15:06:13.005 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:12 smithi134 ceph-mon[33714]: osdmap e539: 8 total, 8 up, 8 in 2024-03-30T15:06:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:13.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:13.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:13.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:13.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:13.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:13.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=cleanup t=2024-03-30T15:06:13.074843754Z level=info msg="Completed cleanup jobs" duration=5.855137ms 2024-03-30T15:06:13.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:13.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:13.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:13.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:13.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:13.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:13.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:13.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:13.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:06:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:06:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-mon[33714]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T15:06:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:13.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:13.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[29876]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[41614]: osdmap e540: 8 total, 8 up, 8 in 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:13.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:14.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:14.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:14.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:14.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:14.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:06:14.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:14.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:14.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:06:14.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:14.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:14.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:14.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:14.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: pgmap v785: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 12.16 deep-scrub starts 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 12.16 deep-scrub ok 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:06:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:06:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: pgmap v785: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 12.16 deep-scrub starts 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 12.16 deep-scrub ok 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.827 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:14.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:14.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: pgmap v785: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 12.16 deep-scrub starts 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 12.16 deep-scrub ok 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:06:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: osdmap e541: 8 total, 8 up, 8 in 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:06:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:14.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:15.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:15.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:15.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:15.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:15.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:15.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:15.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:15.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:15.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:15.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:15.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:15.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[29876]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:15.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:06:15.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:06:15.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:06:15.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:06:15.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:15.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:15 smithi028 ceph-mon[41614]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:06:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:15 smithi134 ceph-mon[33714]: osdmap e542: 8 total, 8 up, 8 in 2024-03-30T15:06:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:15.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:16.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:16.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:16.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:16.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:16.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:16.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:16.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:16.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:16.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.702 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:16.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.775+0000 7f597eed3700 1 -- 172.21.15.28:0/322276860 >> v1:172.21.15.28:6789/0 conn(0x7f597810aee0 legacy=0x7f597810d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.775+0000 7f597eed3700 1 -- 172.21.15.28:0/322276860 shutdown_connections 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.775+0000 7f597eed3700 1 -- 172.21.15.28:0/322276860 >> 172.21.15.28:0/322276860 conn(0x7f59780fdb20 msgr2=0x7f59780fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- 172.21.15.28:0/322276860 shutdown_connections 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- 172.21.15.28:0/322276860 wait complete. 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 librados: starting msgr at 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 librados: starting objecter 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 Processor -- start 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- start start 2024-03-30T15:06:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 librados: setting wanted keys 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 librados: calling monclient init 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59781a9450 con 0x7f5978102250 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59781aa670 con 0x7f597810aee0 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597eed3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59781ab8b0 con 0x7f597810ea80 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597cc6f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5978102250 0x7f5978107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41614/0 (socket says 172.21.15.28:41614) 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f597cc6f700 1 -- 172.21.15.28:0/172336418 learned_addr learned my addr 172.21.15.28:0/172336418 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f5975ffb700 10 client.?.objecter ms_handle_connect 0x7f5978102250 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f5975ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f5975ffb700 10 client.?.objecter ms_handle_connect 0x7f597810ea80 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.776+0000 7f5975ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4164704989 0 0) 0x7f59781ab8b0 con 0x7f597810ea80 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5968003aa0 con 0x7f597810ea80 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 10 client.?.objecter ms_handle_connect 0x7f597810aee0 2024-03-30T15:06:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919405472 0 0) 0x7f59781a9450 con 0x7f5978102250 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59781ab8b0 con 0x7f5978102250 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797921927 0 0) 0x7f59781aa670 con 0x7f597810aee0 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59781a9450 con 0x7f597810aee0 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1105019275 0 0) 0x7f5968003aa0 con 0x7f597810ea80 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59781aa670 con 0x7f597810ea80 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5970001b70 con 0x7f597810ea80 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3556732022 0 0) 0x7f59781ab8b0 con 0x7f5978102250 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5968003aa0 con 0x7f5978102250 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f596c001890 con 0x7f5978102250 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3939960912 0 0) 0x7f59781a9450 con 0x7f597810aee0 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59781ab8b0 con 0x7f597810aee0 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1765299295 0 0) 0x7f59781aa670 con 0x7f597810ea80 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 >> v1:172.21.15.134:6789/0 conn(0x7f597810aee0 legacy=0x7f59781a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 >> v1:172.21.15.28:6789/0 conn(0x7f5978102250 legacy=0x7f5978107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59781acaf0 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59781abae0 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5970003ea0 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5970004e30 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f597eed3700 10 client.58307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f59781abfa0 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.777+0000 7f597eed3700 1 librados: init done 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.778+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5978103730 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5970003a00 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(543..543 src has 442..543) ==== 8622+0+0 (unknown 1784855694 0 0) 0x7f5970088a00 con 0x7f597810ea80 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 10 client.58307.objecter ms_dispatch 0x7f5978001550 osd_map(543..543 src has 442..543) 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 3 client.58307.objecter handle_osd_map got epochs [543,543] > 0 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 3 client.58307.objecter handle_osd_map decoding full epoch 543 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.779+0000 7f5975ffb700 20 client.58307.objecter dump_active .. 0 homeless 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.780+0000 7f5975ffb700 10 client.58307.objecter ms_handle_connect 0x7f5968070400 2024-03-30T15:06:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:16.784+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5970055a40 con 0x7f597810ea80 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: pgmap v788: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:06:16.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[29876]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: pgmap v788: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:06:16.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:16 smithi028 ceph-mon[41614]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T15:06:16.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:16.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:06:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:06:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:06:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:06:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: pgmap v788: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:06:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:06:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:06:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:16 smithi134 ceph-mon[33714]: osdmap e543: 8 total, 8 up, 8 in 2024-03-30T15:06:16.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:16.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.014+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]} v 0) -- 0x7f5978075a70 con 0x7f597810ea80 2024-03-30T15:06:17.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:17.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:17.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:17.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:17.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.326+0000 7f5975ffb700 1 -- 172.21.15.28:0/172336418 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]=0 set 42.c pg_upmap mapping to [3,7,1] v544) ==== 129+0+0 (unknown 2177486899 0 0) 0x7f5970059490 con 0x7f597810ea80 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 10 librados: watch_flush enter 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 10 librados: watch_flush exit 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 20 client.58307.objecter shutdown clearing up homeless session... 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 10 client.58307.objecter successfully canceled tick 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5968070400 legacy=0x7f59680728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 >> v1:172.21.15.28:6790/0 conn(0x7f597810ea80 legacy=0x7f59781a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:17.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 shutdown_connections 2024-03-30T15:06:17.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.330+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 >> 172.21.15.28:0/172336418 conn(0x7f59780fdb20 msgr2=0x7f5978111c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.331+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 shutdown_connections 2024-03-30T15:06:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.333+0000 7f597eed3700 1 -- 172.21.15.28:0/172336418 wait complete. 2024-03-30T15:06:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.333+0000 7f597eed3700 1 librados: shutdown 2024-03-30T15:06:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:set 42.c pg_upmap mapping to [3,7,1] 2024-03-30T15:06:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:17.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:17.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:17.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:17.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.470+0000 7f38c1f10700 1 -- 172.21.15.28:0/2989008559 >> v1:172.21.15.28:6790/0 conn(0x7f38bc10d960 legacy=0x7f38bc10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.470+0000 7f38c1f10700 1 -- 172.21.15.28:0/2989008559 shutdown_connections 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.470+0000 7f38c1f10700 1 -- 172.21.15.28:0/2989008559 >> 172.21.15.28:0/2989008559 conn(0x7f38bc0fc9e0 msgr2=0x7f38bc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.470+0000 7f38c1f10700 1 -- 172.21.15.28:0/2989008559 shutdown_connections 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.470+0000 7f38c1f10700 1 -- 172.21.15.28:0/2989008559 wait complete. 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 librados: starting msgr at 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 librados: starting objecter 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 Processor -- start 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 -- start start 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 librados: setting wanted keys 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 librados: calling monclient init 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38bc0fe9d0 con 0x7f38bc101130 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38bc0febe0 con 0x7f38bc109dc0 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38c1f10700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f38bc0fedf0 con 0x7f38bc10d960 2024-03-30T15:06:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38bb7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f38bc101130 0x7f38bc0fce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41642/0 (socket says 172.21.15.28:41642) 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38bb7fe700 1 -- 172.21.15.28:0/1526546833 learned_addr learned my addr 172.21.15.28:0/1526546833 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38bbfff700 1 --1- 172.21.15.28:0/1526546833 >> v1:172.21.15.28:6790/0 conn(0x7f38bc10d960 0x7f38bc1a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34644/0 (socket says 172.21.15.28:34644) 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter ms_handle_connect 0x7f38bc10d960 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter ms_handle_connect 0x7f38bc101130 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter ms_handle_connect 0x7f38bc109dc0 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.471+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333750585 0 0) 0x7f38bc114de0 con 0x7f38bc10d960 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38a40031f0 con 0x7f38bc10d960 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617892500 0 0) 0x7f38bc0fe9d0 con 0x7f38bc101130 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38bc114de0 con 0x7f38bc101130 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437710708 0 0) 0x7f38bc0febe0 con 0x7f38bc109dc0 2024-03-30T15:06:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38bc0fe9d0 con 0x7f38bc109dc0 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 782877856 0 0) 0x7f38a40031f0 con 0x7f38bc10d960 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38bc0febe0 con 0x7f38bc10d960 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38b40016f0 con 0x7f38bc10d960 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 181635260 0 0) 0x7f38bc0fe9d0 con 0x7f38bc109dc0 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38a40031f0 con 0x7f38bc109dc0 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1418370695 0 0) 0x7f38bc114de0 con 0x7f38bc101130 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38bc0fe9d0 con 0x7f38bc101130 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38b0001830 con 0x7f38bc109dc0 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38ac001940 con 0x7f38bc101130 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4139456483 0 0) 0x7f38bc0febe0 con 0x7f38bc10d960 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 >> v1:172.21.15.134:6789/0 conn(0x7f38bc109dc0 legacy=0x7f38bc0fd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 >> v1:172.21.15.28:6789/0 conn(0x7f38bc101130 legacy=0x7f38bc0fce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38bc1ac9b0 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f38bc1a9980 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38c1f10700 10 client.58313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f38bc1a9e40 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38c1f10700 1 librados: init done 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f38b4001e70 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.472+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f38b4005050 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.473+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a8005120 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.474+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f38b4003be0 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.474+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(544..544 src has 442..544) ==== 8655+0+0 (unknown 700539876 0 0) 0x7f38b4089b80 con 0x7f38bc10d960 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.474+0000 7f38b8ff9700 10 client.58313.objecter ms_dispatch 0x7f38bc001550 osd_map(544..544 src has 442..544) 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.474+0000 7f38b8ff9700 3 client.58313.objecter handle_osd_map got epochs [544,544] > 0 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.474+0000 7f38b8ff9700 3 client.58313.objecter handle_osd_map decoding full epoch 544 2024-03-30T15:06:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.475+0000 7f38b8ff9700 20 client.58313.objecter dump_active .. 0 homeless 2024-03-30T15:06:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.475+0000 7f38b8ff9700 10 client.58313.objecter ms_handle_connect 0x7f38a4070400 2024-03-30T15:06:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.479+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f38bc0fedf0 con 0x7f38bc10d960 2024-03-30T15:06:17.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:17.711+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f38a8005410 con 0x7f38bc10d960 2024-03-30T15:06:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:17.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:17.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/172336418' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.58307 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:06:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:06:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.58307 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]': finished 2024-03-30T15:06:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T15:06:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/172336418' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.58307 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.58307 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]': finished 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T15:06:17.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/172336418' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.58307 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.58307 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "42.c", "id": [3, 7, 1]}]': finished 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: osdmap e544: 8 total, 8 up, 8 in 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-90"}]: dispatch 2024-03-30T15:06:17.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:17.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:18.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:18.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:18.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:18.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.327+0000 7f38b8ff9700 1 -- 172.21.15.28:0/1526546833 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v545) ==== 87+0+0 (unknown 4280989892 0 0) 0x7f38b4056b80 con 0x7f38bc10d960 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 10 librados: watch_flush enter 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 10 librados: watch_flush exit 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 20 client.58313.objecter shutdown clearing up homeless session... 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 10 client.58313.objecter successfully canceled tick 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 >> v1:172.21.15.28:6800/3011431093 conn(0x7f38a4070400 legacy=0x7f38a40728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 >> v1:172.21.15.28:6790/0 conn(0x7f38bc10d960 legacy=0x7f38bc1a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.330+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 shutdown_connections 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.331+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 >> 172.21.15.28:0/1526546833 conn(0x7f38bc0fc9e0 msgr2=0x7f38bc06a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.331+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 shutdown_connections 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.331+0000 7f38c1f10700 1 -- 172.21.15.28:0/1526546833 wait complete. 2024-03-30T15:06:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:18.331+0000 7f38c1f10700 1 librados: shutdown 2024-03-30T15:06:18.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:06:18.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:18.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:18.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:18.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:18.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:18.626 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:18.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:18.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:18.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:18.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: 101.1d deep-scrub starts 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: 101.1d deep-scrub ok 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1526546833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: from='client.58313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: pgmap v791: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: 42.1c scrub starts 2024-03-30T15:06:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: 42.1c scrub ok 2024-03-30T15:06:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: from='client.58313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:06:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:18 smithi134 ceph-mon[33714]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T15:06:18.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:18.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: 101.1d deep-scrub starts 2024-03-30T15:06:18.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: 101.1d deep-scrub ok 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1526546833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: from='client.58313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: pgmap v791: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: 42.1c scrub starts 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: 42.1c scrub ok 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: from='client.58313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[29876]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T15:06:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: 101.1d deep-scrub starts 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: 101.1d deep-scrub ok 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1526546833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: from='client.58313 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: pgmap v791: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 557 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: 42.1c scrub starts 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: 42.1c scrub ok 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: from='client.58313 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:06:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:18 smithi028 ceph-mon[41614]: osdmap e545: 8 total, 8 up, 8 in 2024-03-30T15:06:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:18.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:19.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:19.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:19.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.239 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:19.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:19.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:19.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:19.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:19.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:19.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:19.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:19.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:19.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:19.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:19.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:19 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:06:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:19 smithi134 ceph-mon[33714]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T15:06:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:06:19.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[41614]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T15:06:19.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:06:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[29876]: osdmap e546: 8 total, 8 up, 8 in 2024-03-30T15:06:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:19.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:20.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:20.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:20.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:20.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:20.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:20.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:20.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:20.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:20.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:20.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:20.646 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:20.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:20.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:20.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:20.886 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:20.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:20 smithi134 ceph-mon[33714]: pgmap v794: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:20 smithi134 ceph-mon[33714]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T15:06:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[29876]: pgmap v794: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[29876]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[41614]: pgmap v794: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[41614]: osdmap e547: 8 total, 8 up, 8 in 2024-03-30T15:06:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:20.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:21.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:21.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:06:21.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:21.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:21.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:21.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:21.274 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:06:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:06:21.274 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:06:21.274 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:06:21.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:21.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:21.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:21.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:21.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:21.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:21.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:21 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:21 smithi134 ceph-mon[33714]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T15:06:21.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[41614]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T15:06:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:21 smithi028 ceph-mon[29876]: osdmap e548: 8 total, 8 up, 8 in 2024-03-30T15:06:21.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:22.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:06:22.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:22.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:22.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:22.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:22.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:22.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:22.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: pgmap v797: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: pgmap v797: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:22.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:22.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:22.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:22.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:22.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: pgmap v797: 260 pgs: 32 unknown, 1 active+clean+scrubbing, 227 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:06:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: osdmap e549: 8 total, 8 up, 8 in 2024-03-30T15:06:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-92"}]: dispatch 2024-03-30T15:06:23.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.004+0000 7f2605ed4700 1 -- 172.21.15.28:0/298888478 >> v1:172.21.15.28:6790/0 conn(0x7f260010d960 legacy=0x7f260010fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 -- 172.21.15.28:0/298888478 shutdown_connections 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 -- 172.21.15.28:0/298888478 >> 172.21.15.28:0/298888478 conn(0x7f26000fc9e0 msgr2=0x7f26000fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 -- 172.21.15.28:0/298888478 shutdown_connections 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 -- 172.21.15.28:0/298888478 wait complete. 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 librados: starting msgr at 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 librados: starting objecter 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 Processor -- start 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 -- start start 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 librados: setting wanted keys 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.005+0000 7f2605ed4700 1 librados: calling monclient init 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f2605ed4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26001a9380 con 0x7f2600101130 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f2605ed4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26001aa5a0 con 0x7f2600109dc0 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f2605ed4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f26001ab7e0 con 0x7f260010d960 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25ff7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2600101130 0x7f2600106420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53788/0 (socket says 172.21.15.28:53788) 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25ff7fe700 1 -- 172.21.15.28:0/4169439348 learned_addr learned my addr 172.21.15.28:0/4169439348 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter ms_handle_connect 0x7f2600101130 2024-03-30T15:06:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter ms_handle_connect 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19527806 0 0) 0x7f26001ab7e0 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25e80031f0 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter ms_handle_connect 0x7f2600109dc0 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3127163562 0 0) 0x7f26001a9380 con 0x7f2600101130 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26001ab7e0 con 0x7f2600101130 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1080180713 0 0) 0x7f25e80031f0 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26001a9380 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25f8001c50 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252753871 0 0) 0x7f26001aa5a0 con 0x7f2600109dc0 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25e80031f0 con 0x7f2600109dc0 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2343545011 0 0) 0x7f26001a9380 con 0x7f260010d960 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.134:6789/0 conn(0x7f2600109dc0 legacy=0x7f26001a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6789/0 conn(0x7f2600101130 legacy=0x7f2600106420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.006+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26001aca20 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26001aba10 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f2605ed4700 10 client.58319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f26001abed0 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f2605ed4700 1 librados: init done 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f25f8004a10 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.007+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f25f8004f80 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.008+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f25ec005100 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.008+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f25f8003a00 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(549..549 src has 442..549) ==== 8655+0+0 (unknown 1327238346 0 0) 0x7f25f8088b60 con 0x7f260010d960 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 osd_map(549..549 src has 442..549) 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 3 client.58319.objecter handle_osd_map got epochs [549,549] > 0 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 3 client.58319.objecter handle_osd_map decoding full epoch 549 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 20 client.58319.objecter dump_active .. 0 homeless 2024-03-30T15:06:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v549) ==== 54+0+16 (unknown 2534082343 0 2420000192) 0x7f25f8088fa0 con 0x7f260010d960 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25e8070120 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 10 client.58319.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 1 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 1 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 0 1 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 4 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 10 client.58319.objecter _send_command 1 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25ec004b70 con 0x7f25ec004730 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.009+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec004730 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.010+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec004b70 con 0x7f25ec004730 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.010+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.010+0000 7f25fcff9700 10 client.58319.objecter _finish_command 1 = system:0 2024-03-30T15:06:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.010+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 4 2024-03-30T15:06:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.010+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 0 1 2024-03-30T15:06:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:23.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 10 client.58319.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 2 2024-03-30T15:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 2 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec0053e0 osd=0 3 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 0 2 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 4 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 10 client.58319.objecter _send_command 2 2024-03-30T15:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.029+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec0079f0 con 0x7f25ec004730 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec0079f0 con 0x7f25ec004730 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f25fcff9700 10 client.58319.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 4 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 0 2 2024-03-30T15:06:23.032 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 10 client.58319.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 1 3 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 4 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 10 client.58319.objecter _send_command 3 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.031+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f25ec0022a0 con 0x7f25ec004310 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec004310 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec0022a0 con 0x7f25ec004310 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 10 client.58319.objecter _finish_command 3 = system:0 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 4 2024-03-30T15:06:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.032+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 1 3 2024-03-30T15:06:23.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 10 client.58319.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 4 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 4 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec004130 osd=1 3 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 1 4 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 4 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 10 client.58319.objecter _send_command 4 2024-03-30T15:06:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.053+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec001720 con 0x7f25ec004310 2024-03-30T15:06:23.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec001720 con 0x7f25ec004310 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f25fcff9700 10 client.58319.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 4 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 1 4 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 10 client.58319.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 5 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 5 2024-03-30T15:06:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 2 5 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 4 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 10 client.58319.objecter _send_command 5 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.055+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f25ec011170 con 0x7f25ec00d5d0 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec00d5d0 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec011170 con 0x7f25ec00d5d0 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 10 client.58319.objecter _finish_command 5 = system:0 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 4 2024-03-30T15:06:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.056+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 2 5 2024-03-30T15:06:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 10 client.58319.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 6 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 6 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec001980 osd=2 3 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 2 6 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 4 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 10 client.58319.objecter _send_command 6 2024-03-30T15:06:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.076+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec008100 con 0x7f25ec00d5d0 2024-03-30T15:06:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.078+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec008100 con 0x7f25ec00d5d0 2024-03-30T15:06:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.078+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.078+0000 7f25fcff9700 10 client.58319.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.078+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 4 2024-03-30T15:06:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.078+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 2 6 2024-03-30T15:06:23.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 10 client.58319.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 7 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 7 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 3 7 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 4 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 10 client.58319.objecter _send_command 7 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f25ec015fb0 con 0x7f25ec0124a0 2024-03-30T15:06:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.079+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec0124a0 2024-03-30T15:06:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.080+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec015fb0 con 0x7f25ec0124a0 2024-03-30T15:06:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.080+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.080+0000 7f25fcff9700 10 client.58319.objecter _finish_command 7 = system:0 2024-03-30T15:06:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.080+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 4 2024-03-30T15:06:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.080+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 3 7 2024-03-30T15:06:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 10 client.58319.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 8 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 8 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec00cc20 osd=3 3 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 3 8 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 4 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 10 client.58319.objecter _send_command 8 2024-03-30T15:06:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.097+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec00c700 con 0x7f25ec0124a0 2024-03-30T15:06:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec00c700 con 0x7f25ec0124a0 2024-03-30T15:06:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f25fcff9700 10 client.58319.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 4 2024-03-30T15:06:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 3 8 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 10 client.58319.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 9 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 9 2024-03-30T15:06:23.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 4 9 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 4 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 10 client.58319.objecter _send_command 9 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.099+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f25ec00c6e0 con 0x7f25ec012000 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.100+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec012000 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.101+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec00c6e0 con 0x7f25ec012000 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.101+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.101+0000 7f25fcff9700 10 client.58319.objecter _finish_command 9 = system:0 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.101+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 4 2024-03-30T15:06:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.101+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 4 9 2024-03-30T15:06:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 10 client.58319.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 10 2024-03-30T15:06:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 10 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec00c4b0 osd=4 3 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 4 10 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 4 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 10 client.58319.objecter _send_command 10 2024-03-30T15:06:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.117+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec01c210 con 0x7f25ec012000 2024-03-30T15:06:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec01c210 con 0x7f25ec012000 2024-03-30T15:06:23.121 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 10 client.58319.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 4 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 4 10 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 10 client.58319.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 11 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 11 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 5 11 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 4 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 10 client.58319.objecter _send_command 11 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f25ec01cdc0 con 0x7f25ec01c930 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.120+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec01c930 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.121+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec01cdc0 con 0x7f25ec01c930 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.121+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.121+0000 7f25fcff9700 10 client.58319.objecter _finish_command 11 = system:0 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.121+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 4 2024-03-30T15:06:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.121+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 5 11 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 10 client.58319.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 12 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 12 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec01c210 osd=5 3 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 5 12 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 4 2024-03-30T15:06:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 10 client.58319.objecter _send_command 12 2024-03-30T15:06:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.139+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec01cdc0 con 0x7f25ec01c930 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec01cdc0 con 0x7f25ec01c930 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 10 client.58319.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 4 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 5 12 2024-03-30T15:06:23.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 10 client.58319.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 13 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 13 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 6 13 2024-03-30T15:06:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 4 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 10 client.58319.objecter _send_command 13 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f25ec01d1c0 con 0x7f25ec01f900 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.141+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec01f900 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.142+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec01d1c0 con 0x7f25ec01f900 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.142+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.142+0000 7f25fcff9700 10 client.58319.objecter _finish_command 13 = system:0 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.142+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 4 2024-03-30T15:06:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.142+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 6 13 2024-03-30T15:06:23.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 10 client.58319.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 14 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 14 2024-03-30T15:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec01cdc0 osd=6 3 2024-03-30T15:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 6 14 2024-03-30T15:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 4 2024-03-30T15:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 10 client.58319.objecter _send_command 14 2024-03-30T15:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.159+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec023410 con 0x7f25ec01f900 2024-03-30T15:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec023410 con 0x7f25ec01f900 2024-03-30T15:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f25fcff9700 10 client.58319.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 4 2024-03-30T15:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 6 14 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 10 client.58319.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 15 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 15 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 7 15 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 4 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 10 client.58319.objecter _send_command 15 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.161+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f25ec024720 con 0x7f25ec023b30 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.162+0000 7f25fcff9700 10 client.58319.objecter ms_handle_connect 0x7f25ec023b30 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.163+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f25ec024720 con 0x7f25ec023b30 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.163+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.163+0000 7f25fcff9700 10 client.58319.objecter _finish_command 15 = system:0 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.163+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 4 2024-03-30T15:06:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.163+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 7 15 2024-03-30T15:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 10 client.58319.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign -1 16 2024-03-30T15:06:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 20 client.58319.objecter _get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_remove -1 16 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 20 client.58319.objecter get_session s=0x7f25ec023410 osd=7 3 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 15 client.58319.objecter _session_command_op_assign 7 16 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 4 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 10 client.58319.objecter _send_command 16 2024-03-30T15:06:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.180+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f25ec024720 con 0x7f25ec023b30 2024-03-30T15:06:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.182+0000 7f25fcff9700 1 -- 172.21.15.28:0/4169439348 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f25ec024720 con 0x7f25ec023b30 2024-03-30T15:06:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.182+0000 7f25fcff9700 10 client.58319.objecter ms_dispatch 0x7f2600001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.182+0000 7f25fcff9700 10 client.58319.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.182+0000 7f25fcff9700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 4 2024-03-30T15:06:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.182+0000 7f25fcff9700 15 client.58319.objecter _session_command_op_remove 7 16 2024-03-30T15:06:23.184 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 librados: watch_flush enter 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 librados: watch_flush exit 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.0 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6801/3394323575 conn(0x7f25ec004730 legacy=0x7f25ec002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec0053e0 osd=0 1 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.1 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6805/1850313547 conn(0x7f25ec004310 legacy=0x7f25ec00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec004130 osd=1 1 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.2 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6809/3451325043 conn(0x7f25ec00d5d0 legacy=0x7f25ec00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec001980 osd=2 1 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.3 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6813/1506157250 conn(0x7f25ec0124a0 legacy=0x7f25ec014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00cc20 osd=3 1 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.4 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.134:6800/3063775514 conn(0x7f25ec012000 legacy=0x7f25ec0192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f25fffff700 1 -- 172.21.15.28:0/4169439348 reap_dead start 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f25fffff700 1 -- 172.21.15.28:0/4169439348 reap_dead start 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec00c4b0 osd=4 1 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.5 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.134:6804/3668151179 conn(0x7f25ec01c930 legacy=0x7f25ec01aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01c210 osd=5 1 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.6 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.134:6808/2135224194 conn(0x7f25ec01f900 legacy=0x7f25ec021cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec01cdc0 osd=6 1 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter close_session for osd.7 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.134:6812/3758190298 conn(0x7f25ec023b30 legacy=0x7f25ec023fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter put_session s=0x7f25ec023410 osd=7 1 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 20 client.58319.objecter shutdown clearing up homeless session... 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 10 client.58319.objecter successfully canceled tick 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6800/3011431093 conn(0x7f25e8070120 legacy=0x7f25e80725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f25fffff700 1 -- 172.21.15.28:0/4169439348 reap_dead start 2024-03-30T15:06:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> v1:172.21.15.28:6790/0 conn(0x7f260010d960 legacy=0x7f26001a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 shutdown_connections 2024-03-30T15:06:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 >> 172.21.15.28:0/4169439348 conn(0x7f26000fc9e0 msgr2=0x7f2600110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 shutdown_connections 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 -- 172.21.15.28:0/4169439348 wait complete. 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:23.183+0000 7f2605ed4700 1 librados: shutdown 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:23.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:23.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:23.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:23.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:23.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:23.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:23.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:23.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:23.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4169439348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:23 smithi134 ceph-mon[33714]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T15:06:23.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:23.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4169439348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:23 smithi028 ceph-mon[29876]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T15:06:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4169439348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:23.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:23 smithi028 ceph-mon[41614]: osdmap e550: 8 total, 8 up, 8 in 2024-03-30T15:06:24.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:24.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:24.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:24.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:24.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:24.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:24.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:24.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:24.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:24.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:24.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:24.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:24 smithi134 ceph-mon[33714]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:24 smithi134 ceph-mon[33714]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T15:06:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:24 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:24.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:24.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:24.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[29876]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[29876]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[41614]: pgmap v800: 228 pgs: 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[41614]: osdmap e551: 8 total, 8 up, 8 in 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:24.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:24 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:25.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:25.085 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:25.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:25.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:25.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:25.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:25.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:25.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:25.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:25.651 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:06:25.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:06:25.651 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-30T15:06:25.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:06:25.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:25.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:25.864 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:25.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:25.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:26.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:26.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:26.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:26.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:26.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:06:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:26.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:26.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:26.455 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[41614]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[41614]: pgmap v803: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[29876]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:26.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:26 smithi028 ceph-mon[29876]: pgmap v803: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:26.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:26.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:26.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:26 smithi134 ceph-mon[33714]: osdmap e552: 8 total, 8 up, 8 in 2024-03-30T15:06:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:26 smithi134 ceph-mon[33714]: pgmap v803: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- 172.21.15.28:0/329740733 >> v1:172.21.15.28:6789/0 conn(0x7f7aa0109dc0 legacy=0x7f7aa010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- 172.21.15.28:0/329740733 shutdown_connections 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- 172.21.15.28:0/329740733 >> 172.21.15.28:0/329740733 conn(0x7f7aa00fc9e0 msgr2=0x7f7aa00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- 172.21.15.28:0/329740733 shutdown_connections 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- 172.21.15.28:0/329740733 wait complete. 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 librados: starting msgr at 2024-03-30T15:06:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 librados: starting objecter 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 Processor -- start 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 -- start start 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 librados: setting wanted keys 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.691+0000 7f7aa69ef700 1 librados: calling monclient init 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7aa69ef700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7aa01a7010 con 0x7f7aa0101130 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7aa69ef700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7aa01a8230 con 0x7f7aa0109dc0 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7aa69ef700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7aa01a9470 con 0x7f7aa010d960 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7aa0101130 0x7f7aa01063c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53794/0 (socket says 172.21.15.28:53794) 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7aa4f8c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7aa010d960 0x7f7aa01a56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52740/0 (socket says 172.21.15.28:52740) 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9ffff700 1 -- 172.21.15.28:0/570499416 learned_addr learned my addr 172.21.15.28:0/570499416 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter ms_handle_connect 0x7f7aa0101130 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter ms_handle_connect 0x7f7aa010d960 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter ms_handle_connect 0x7f7aa0109dc0 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236361981 0 0) 0x7f7aa01a7010 con 0x7f7aa0101130 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a88003aa0 con 0x7f7aa0101130 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3903290034 0 0) 0x7f7aa01a9470 con 0x7f7aa010d960 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa01a7010 con 0x7f7aa010d960 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287141738 0 0) 0x7f7aa01a8230 con 0x7f7aa0109dc0 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa01a9470 con 0x7f7aa0109dc0 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4080436590 0 0) 0x7f7a88003aa0 con 0x7f7aa0101130 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7aa01a8230 con 0x7f7aa0101130 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.692+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a90001910 con 0x7f7aa0101130 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1701881937 0 0) 0x7f7aa01a7010 con 0x7f7aa010d960 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a88003aa0 con 0x7f7aa010d960 2024-03-30T15:06:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a980015b0 con 0x7f7aa010d960 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 936757993 0 0) 0x7f7aa01a9470 con 0x7f7aa0109dc0 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7aa01a7010 con 0x7f7aa0109dc0 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a94004520 con 0x7f7aa0109dc0 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1700629980 0 0) 0x7f7aa01a8230 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 >> v1:172.21.15.28:6790/0 conn(0x7f7aa010d960 legacy=0x7f7aa01a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 >> v1:172.21.15.134:6789/0 conn(0x7f7aa0109dc0 legacy=0x7f7aa01a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa01aa6b0 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa01a96a0 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7aa69ef700 10 client.25125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7aa01a9b60 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7aa69ef700 1 librados: init done 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7a90001c60 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.693+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a90005d60 con 0x7f7aa0101130 2024-03-30T15:06:26.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.694+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c005120 con 0x7f7aa0101130 2024-03-30T15:06:26.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7a90003950 con 0x7f7aa0101130 2024-03-30T15:06:26.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(553..553 src has 442..553) ==== 8655+0+0 (unknown 3325073570 0 0) 0x7f7a90089860 con 0x7f7aa0101130 2024-03-30T15:06:26.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 10 client.25125.objecter ms_dispatch 0x7f7aa0001550 osd_map(553..553 src has 442..553) 2024-03-30T15:06:26.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 3 client.25125.objecter handle_osd_map got epochs [553,553] > 0 2024-03-30T15:06:26.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 3 client.25125.objecter handle_osd_map decoding full epoch 553 2024-03-30T15:06:26.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 20 client.25125.objecter dump_active .. 0 homeless 2024-03-30T15:06:26.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.695+0000 7f7a9d7fa700 10 client.25125.objecter ms_handle_connect 0x7f7a88070420 2024-03-30T15:06:26.701 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.700+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7a90005d60 con 0x7f7aa0101130 2024-03-30T15:06:26.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:26.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:26.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:26.918+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f7a8c005410 con 0x7f7aa0101130 2024-03-30T15:06:26.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:27.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:27.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:27.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:27.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:27.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:27.283 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:06:27.283 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:06:27.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.327+0000 7f7a9d7fa700 1 -- 172.21.15.28:0/570499416 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v554) ==== 97+0+0 (unknown 2258180491 0 0) 0x7f7a90056880 con 0x7f7aa0101130 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 10 librados: watch_flush enter 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 10 librados: watch_flush exit 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 20 client.25125.objecter shutdown clearing up homeless session... 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 10 client.25125.objecter successfully canceled tick 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7a88070420 legacy=0x7f7a880728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.330+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 >> v1:172.21.15.28:6789/0 conn(0x7f7aa0101130 legacy=0x7f7aa01063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.331+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 shutdown_connections 2024-03-30T15:06:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.331+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 >> 172.21.15.28:0/570499416 conn(0x7f7aa00fc9e0 msgr2=0x7f7aa00ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.332+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 shutdown_connections 2024-03-30T15:06:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.332+0000 7f7aa69ef700 1 -- 172.21.15.28:0/570499416 wait complete. 2024-03-30T15:06:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:27.333+0000 7f7aa69ef700 1 librados: shutdown 2024-03-30T15:06:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:06:27.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:27.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[29876]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[41614]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:27.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:27 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:27.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:27.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:27.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:27 smithi134 ceph-mon[33714]: osdmap e553: 8 total, 8 up, 8 in 2024-03-30T15:06:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:27 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:06:27.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:27.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:27.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:27.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:28.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:28.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:28.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:28.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:28.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:28.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:28.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:28.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:28.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:28 smithi134 ceph-mon[33714]: pgmap v806: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[41614]: pgmap v806: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/570499416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: osdmap e554: 8 total, 8 up, 8 in 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-mon[29876]: pgmap v806: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:28.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:28.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:28.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:28.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:28.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:28.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:29.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:29.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:29.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:29.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:29.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:29.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:29 smithi028 ceph-mon[29876]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T15:06:29.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:29 smithi028 ceph-mon[41614]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T15:06:29.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:29.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:29.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:06:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:29.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:29 smithi134 ceph-mon[33714]: osdmap e555: 8 total, 8 up, 8 in 2024-03-30T15:06:29.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:29.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:29.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:30.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:30.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:30.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:30.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:30.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:30.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:30.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:06:30.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:30.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:30 smithi028 ceph-mon[29876]: pgmap v808: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T15:06:30.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:30 smithi028 ceph-mon[41614]: pgmap v808: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T15:06:30.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:30 smithi134 ceph-mon[33714]: pgmap v808: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-30T15:06:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:30.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:30.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:30.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:30.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:30.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:31.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:31.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:31.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:31.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:31.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:31.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:31.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.470+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2504125261 >> v1:172.21.15.28:6790/0 conn(0x7fc5b8101140 legacy=0x7fc5b8101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2504125261 shutdown_connections 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2504125261 >> 172.21.15.28:0/2504125261 conn(0x7fc5b80fc9d0 msgr2=0x7fc5b80fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2504125261 shutdown_connections 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2504125261 wait complete. 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 librados: starting msgr at 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 librados: starting objecter 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 Processor -- start 2024-03-30T15:06:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- start start 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 librados: setting wanted keys 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 librados: calling monclient init 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5b81a7110 con 0x7fc5b810d970 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5b81a8330 con 0x7fc5b8101140 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bf3a8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5b81a9570 con 0x7fc5b8109dd0 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bc943700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc5b8109dd0 0x7fc5b81a1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56474/0 (socket says 172.21.15.28:56474) 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.471+0000 7fc5bc943700 1 -- 172.21.15.28:0/2884002106 learned_addr learned my addr 172.21.15.28:0/2884002106 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc5b8109dd0 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc5b810d970 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter ms_handle_connect 0x7fc5b8101140 2024-03-30T15:06:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342943601 0 0) 0x7fc5b81a9570 con 0x7fc5b8109dd0 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a40031f0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387623156 0 0) 0x7fc5b81a7110 con 0x7fc5b810d970 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b81a9570 con 0x7fc5b810d970 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109472071 0 0) 0x7fc5b81a8330 con 0x7fc5b8101140 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b81a7110 con 0x7fc5b8101140 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 874436212 0 0) 0x7fc5a40031f0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5b81a8330 con 0x7fc5b8109dd0 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5a80018f0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3166172127 0 0) 0x7fc5b81a9570 con 0x7fc5b810d970 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a40031f0 con 0x7fc5b810d970 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5b0001c10 con 0x7fc5b810d970 2024-03-30T15:06:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2923491675 0 0) 0x7fc5b81a7110 con 0x7fc5b8101140 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5b81a9570 con 0x7fc5b8101140 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5b4001800 con 0x7fc5b8101140 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2797208836 0 0) 0x7fc5b81a8330 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 >> v1:172.21.15.134:6789/0 conn(0x7fc5b8101140 legacy=0x7fc5b806af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 >> v1:172.21.15.28:6789/0 conn(0x7fc5b810d970 legacy=0x7fc5b81a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b81aa7b0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b81a97a0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5bf3a8700 10 client.58331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5b81a9c60 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5bf3a8700 1 librados: init done 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc5a8001cc0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.472+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5a8005da0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.473+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c005120 con 0x7fc5b8109dd0 2024-03-30T15:06:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc5a8004160 con 0x7fc5b8109dd0 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(555..555 src has 442..555) ==== 8655+0+0 (unknown 115772641 0 0) 0x7fc5a8089a70 con 0x7fc5b8109dd0 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 10 client.58331.objecter ms_dispatch 0x7fc5b8001550 osd_map(555..555 src has 442..555) 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 3 client.58331.objecter handle_osd_map got epochs [555,555] > 0 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 3 client.58331.objecter handle_osd_map decoding full epoch 555 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.474+0000 7fc5ae7fc700 20 client.58331.objecter dump_active .. 0 homeless 2024-03-30T15:06:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.475+0000 7fc5ae7fc700 10 client.58331.objecter ms_handle_connect 0x7fc5a4070430 2024-03-30T15:06:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.480+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc5a80044d0 con 0x7fc5b8109dd0 2024-03-30T15:06:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:31.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:31.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:31.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:31.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.693+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc59c000b90 con 0x7fc5a4070430 2024-03-30T15:06:31.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:31.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.710+0000 7fc5ae7fc700 1 -- 172.21.15.28:0/2884002106 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+668603 (unknown 2965378022 0 3235071806) 0x7fc59c000b90 con 0x7fc5a4070430 2024-03-30T15:06:31.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 10 librados: watch_flush enter 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 10 librados: watch_flush exit 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 20 client.58331.objecter shutdown clearing up homeless session... 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 10 client.58331.objecter successfully canceled tick 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc5a4070430 legacy=0x7fc5a40728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 >> v1:172.21.15.28:6790/0 conn(0x7fc5b8109dd0 legacy=0x7fc5b81a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 shutdown_connections 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 >> 172.21.15.28:0/2884002106 conn(0x7fc5b80fc9d0 msgr2=0x7fc5b8110ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 shutdown_connections 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 -- 172.21.15.28:0/2884002106 wait complete. 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:31.715+0000 7fc5bf3a8700 1 librados: shutdown 2024-03-30T15:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:06:31.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:31.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:31.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:32.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:32.088 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:32.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:32.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:32.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:32.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:32.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:32.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:32.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:32 smithi134 ceph-mon[33714]: from='client.58331 v1:172.21.15.28:0/2884002106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:32 smithi134 ceph-mon[33714]: pgmap v809: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T15:06:32.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:32.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:32 smithi028 ceph-mon[29876]: from='client.58331 v1:172.21.15.28:0/2884002106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:32.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:32 smithi028 ceph-mon[29876]: pgmap v809: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T15:06:32.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:32 smithi028 ceph-mon[41614]: from='client.58331 v1:172.21.15.28:0/2884002106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:06:32.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:32 smithi028 ceph-mon[41614]: pgmap v809: 260 pgs: 260 active+clean; 8.4 MiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-30T15:06:32.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:32.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:32.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:32.878 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":808,"stamp":"2024-03-30T15:06:30.035073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811167,"num_objects":402,"num_object_clones":16,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":2297,"num_read_kb":1981,"num_write":1708,"num_write_kb":15287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1771902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3112,"ondisk_log_size":3112,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":767,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":581060,"kb_used_data":52552,"kb_used_omap":12,"kb_used_meta":528371,"kb_avail":749380156,"statfs":{"total":767960285184,"available":767365279744,"internally_reserved":0,"allocated":53813248,"data_stored":35668321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":541052464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.712490"},"pg_stats":[{"pgid":"312.1f","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347892+0000","last_change":"2024-03-30T15:06:25.339958+0000","last_active":"2024-03-30T15:06:27.347892+0000","last_peered":"2024-03-30T15:06:27.347892+0000","last_clean":"2024-03-30T15:06:27.347892+0000","last_became_active":"2024-03-30T15:06:25.339591+0000","last_became_peered":"2024-03-30T15:06:25.339591+0000","last_unstale":"2024-03-30T15:06:27.347892+0000","last_undegraded":"2024-03-30T15:06:27.347892+0000","last_fullsized":"2024-03-30T15:06:27.347892+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:36.224926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"312.1e","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360647+0000","last_change":"2024-03-30T15:06:25.348093+0000","last_active":"2024-03-30T15:06:27.360647+0000","last_peered":"2024-03-30T15:06:27.360647+0000","last_clean":"2024-03-30T15:06:27.360647+0000","last_became_active":"2024-03-30T15:06:25.347550+0000","last_became_peered":"2024-03-30T15:06:25.347550+0000","last_unstale":"2024-03-30T15:06:27.360647+0000","last_undegraded":"2024-03-30T15:06:27.360647+0000","last_fullsized":"2024-03-30T15:06:27.360647+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:49.511478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"312.1d","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360003+0000","last_change":"2024-03-30T15:06:25.345107+0000","last_active":"2024-03-30T15:06:27.360003+0000","last_peered":"2024-03-30T15:06:27.360003+0000","last_clean":"2024-03-30T15:06:27.360003+0000","last_became_active":"2024-03-30T15:06:25.344770+0000","last_became_peered":"2024-03-30T15:06:25.344770+0000","last_unstale":"2024-03-30T15:06:27.360003+0000","last_undegraded":"2024-03-30T15:06:27.360003+0000","last_fullsized":"2024-03-30T15:06:27.360003+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.926882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"312.1c","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351258+0000","last_change":"2024-03-30T15:06:25.347373+0000","last_active":"2024-03-30T15:06:27.351258+0000","last_peered":"2024-03-30T15:06:27.351258+0000","last_clean":"2024-03-30T15:06:27.351258+0000","last_became_active":"2024-03-30T15:06:25.346464+0000","last_became_peered":"2024-03-30T15:06:25.346464+0000","last_unstale":"2024-03-30T15:06:27.351258+0000","last_undegraded":"2024-03-30T15:06:27.351258+0000","last_fullsized":"2024-03-30T15:06:27.351258+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:53.882259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"312.1b","version":"554'3","reported_seq":18,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.375958+0000","last_change":"2024-03-30T15:06:25.342663+0000","last_active":"2024-03-30T15:06:27.375958+0000","last_peered":"2024-03-30T15:06:27.375958+0000","last_clean":"2024-03-30T15:06:27.375958+0000","last_became_active":"2024-03-30T15:06:25.342346+0000","last_became_peered":"2024-03-30T15:06:25.342346+0000","last_unstale":"2024-03-30T15:06:27.375958+0000","last_undegraded":"2024-03-30T15:06:27.375958+0000","last_fullsized":"2024-03-30T15:06:27.375958+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:39.673075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.1a","version":"0'0","reported_seq":17,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346927+0000","last_change":"2024-03-30T15:06:25.346749+0000","last_active":"2024-03-30T15:06:28.346927+0000","last_peered":"2024-03-30T15:06:28.346927+0000","last_clean":"2024-03-30T15:06:28.346927+0000","last_became_active":"2024-03-30T15:06:25.346286+0000","last_became_peered":"2024-03-30T15:06:25.346286+0000","last_unstale":"2024-03-30T15:06:28.346927+0000","last_undegraded":"2024-03-30T15:06:28.346927+0000","last_fullsized":"2024-03-30T15:06:28.346927+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:25.264956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"312.19","version":"554'3","reported_seq":18,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.379575+0000","last_change":"2024-03-30T15:06:25.342118+0000","last_active":"2024-03-30T15:06:27.379575+0000","last_peered":"2024-03-30T15:06:27.379575+0000","last_clean":"2024-03-30T15:06:27.379575+0000","last_became_active":"2024-03-30T15:06:25.341559+0000","last_became_peered":"2024-03-30T15:06:25.341559+0000","last_unstale":"2024-03-30T15:06:27.379575+0000","last_undegraded":"2024-03-30T15:06:27.379575+0000","last_fullsized":"2024-03-30T15:06:27.379575+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.941700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.18","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.342860+0000","last_change":"2024-03-30T15:06:25.348838+0000","last_active":"2024-03-30T15:06:27.342860+0000","last_peered":"2024-03-30T15:06:27.342860+0000","last_clean":"2024-03-30T15:06:27.342860+0000","last_became_active":"2024-03-30T15:06:25.348543+0000","last_became_peered":"2024-03-30T15:06:25.348543+0000","last_unstale":"2024-03-30T15:06:27.342860+0000","last_undegraded":"2024-03-30T15:06:27.342860+0000","last_fullsized":"2024-03-30T15:06:27.342860+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:40.690167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.17","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340145+0000","last_change":"2024-03-30T15:06:25.338810+0000","last_active":"2024-03-30T15:06:27.340145+0000","last_peered":"2024-03-30T15:06:27.340145+0000","last_clean":"2024-03-30T15:06:27.340145+0000","last_became_active":"2024-03-30T15:06:25.338422+0000","last_became_peered":"2024-03-30T15:06:25.338422+0000","last_unstale":"2024-03-30T15:06:27.340145+0000","last_undegraded":"2024-03-30T15:06:27.340145+0000","last_fullsized":"2024-03-30T15:06:27.340145+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.968112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.16","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344776+0000","last_change":"2024-03-30T15:06:25.336310+0000","last_active":"2024-03-30T15:06:27.344776+0000","last_peered":"2024-03-30T15:06:27.344776+0000","last_clean":"2024-03-30T15:06:27.344776+0000","last_became_active":"2024-03-30T15:06:25.336023+0000","last_became_peered":"2024-03-30T15:06:25.336023+0000","last_unstale":"2024-03-30T15:06:27.344776+0000","last_undegraded":"2024-03-30T15:06:27.344776+0000","last_fullsized":"2024-03-30T15:06:27.344776+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:41.054364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.15","version":"0'0","reported_seq":17,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347782+0000","last_change":"2024-03-30T15:06:25.346372+0000","last_active":"2024-03-30T15:06:28.347782+0000","last_peered":"2024-03-30T15:06:28.347782+0000","last_clean":"2024-03-30T15:06:28.347782+0000","last_became_active":"2024-03-30T15:06:25.346016+0000","last_became_peered":"2024-03-30T15:06:25.346016+0000","last_unstale":"2024-03-30T15:06:28.347782+0000","last_undegraded":"2024-03-30T15:06:28.347782+0000","last_fullsized":"2024-03-30T15:06:28.347782+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:53.581427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"312.14","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360449+0000","last_change":"2024-03-30T15:06:25.343203+0000","last_active":"2024-03-30T15:06:27.360449+0000","last_peered":"2024-03-30T15:06:27.360449+0000","last_clean":"2024-03-30T15:06:27.360449+0000","last_became_active":"2024-03-30T15:06:25.342853+0000","last_became_peered":"2024-03-30T15:06:25.342853+0000","last_unstale":"2024-03-30T15:06:27.360449+0000","last_undegraded":"2024-03-30T15:06:27.360449+0000","last_fullsized":"2024-03-30T15:06:27.360449+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:42.434171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"312.13","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360517+0000","last_change":"2024-03-30T15:06:25.334129+0000","last_active":"2024-03-30T15:06:27.360517+0000","last_peered":"2024-03-30T15:06:27.360517+0000","last_clean":"2024-03-30T15:06:27.360517+0000","last_became_active":"2024-03-30T15:06:25.333788+0000","last_became_peered":"2024-03-30T15:06:25.333788+0000","last_unstale":"2024-03-30T15:06:27.360517+0000","last_undegraded":"2024-03-30T15:06:27.360517+0000","last_fullsized":"2024-03-30T15:06:27.360517+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:42.778328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"312.12","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338419+0000","last_change":"2024-03-30T15:06:25.339402+0000","last_active":"2024-03-30T15:06:27.338419+0000","last_peered":"2024-03-30T15:06:27.338419+0000","last_clean":"2024-03-30T15:06:27.338419+0000","last_became_active":"2024-03-30T15:06:25.339103+0000","last_became_peered":"2024-03-30T15:06:25.339103+0000","last_unstale":"2024-03-30T15:06:27.338419+0000","last_undegraded":"2024-03-30T15:06:27.338419+0000","last_fullsized":"2024-03-30T15:06:27.338419+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:24.517098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.11","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338288+0000","last_change":"2024-03-30T15:06:25.343783+0000","last_active":"2024-03-30T15:06:27.338288+0000","last_peered":"2024-03-30T15:06:27.338288+0000","last_clean":"2024-03-30T15:06:27.338288+0000","last_became_active":"2024-03-30T15:06:25.343255+0000","last_became_peered":"2024-03-30T15:06:25.343255+0000","last_unstale":"2024-03-30T15:06:27.338288+0000","last_undegraded":"2024-03-30T15:06:27.338288+0000","last_fullsized":"2024-03-30T15:06:27.338288+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:39.787240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.10","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.349325+0000","last_change":"2024-03-30T15:06:25.341509+0000","last_active":"2024-03-30T15:06:27.349325+0000","last_peered":"2024-03-30T15:06:27.349325+0000","last_clean":"2024-03-30T15:06:27.349325+0000","last_became_active":"2024-03-30T15:06:25.341215+0000","last_became_peered":"2024-03-30T15:06:25.341215+0000","last_unstale":"2024-03-30T15:06:27.349325+0000","last_undegraded":"2024-03-30T15:06:27.349325+0000","last_fullsized":"2024-03-30T15:06:27.349325+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:43.177538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"312.f","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340319+0000","last_change":"2024-03-30T15:06:25.348002+0000","last_active":"2024-03-30T15:06:27.340319+0000","last_peered":"2024-03-30T15:06:27.340319+0000","last_clean":"2024-03-30T15:06:27.340319+0000","last_became_active":"2024-03-30T15:06:25.347680+0000","last_became_peered":"2024-03-30T15:06:25.347680+0000","last_unstale":"2024-03-30T15:06:27.340319+0000","last_undegraded":"2024-03-30T15:06:27.340319+0000","last_fullsized":"2024-03-30T15:06:27.340319+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:41.873202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.e","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352040+0000","last_change":"2024-03-30T15:06:25.339105+0000","last_active":"2024-03-30T15:06:27.352040+0000","last_peered":"2024-03-30T15:06:27.352040+0000","last_clean":"2024-03-30T15:06:27.352040+0000","last_became_active":"2024-03-30T15:06:25.337872+0000","last_became_peered":"2024-03-30T15:06:25.337872+0000","last_unstale":"2024-03-30T15:06:27.352040+0000","last_undegraded":"2024-03-30T15:06:27.352040+0000","last_fullsized":"2024-03-30T15:06:27.352040+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:27.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"312.d","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346969+0000","last_change":"2024-03-30T15:06:25.346431+0000","last_active":"2024-03-30T15:06:27.346969+0000","last_peered":"2024-03-30T15:06:27.346969+0000","last_clean":"2024-03-30T15:06:27.346969+0000","last_became_active":"2024-03-30T15:06:25.345957+0000","last_became_peered":"2024-03-30T15:06:25.345957+0000","last_unstale":"2024-03-30T15:06:27.346969+0000","last_undegraded":"2024-03-30T15:06:27.346969+0000","last_fullsized":"2024-03-30T15:06:27.346969+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:48.714767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"312.c","version":"554'2","reported_seq":17,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.367211+0000","last_change":"2024-03-30T15:06:25.343248+0000","last_active":"2024-03-30T15:06:27.367211+0000","last_peered":"2024-03-30T15:06:27.367211+0000","last_clean":"2024-03-30T15:06:27.367211+0000","last_became_active":"2024-03-30T15:06:25.342207+0000","last_became_peered":"2024-03-30T15:06:25.342207+0000","last_unstale":"2024-03-30T15:06:27.367211+0000","last_undegraded":"2024-03-30T15:06:27.367211+0000","last_fullsized":"2024-03-30T15:06:27.367211+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.609057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.b","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343708+0000","last_change":"2024-03-30T15:06:25.346488+0000","last_active":"2024-03-30T15:06:27.343708+0000","last_peered":"2024-03-30T15:06:27.343708+0000","last_clean":"2024-03-30T15:06:27.343708+0000","last_became_active":"2024-03-30T15:06:25.346165+0000","last_became_peered":"2024-03-30T15:06:25.346165+0000","last_unstale":"2024-03-30T15:06:27.343708+0000","last_undegraded":"2024-03-30T15:06:27.343708+0000","last_fullsized":"2024-03-30T15:06:27.343708+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:30.649347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.a","version":"554'3","reported_seq":18,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.379765+0000","last_change":"2024-03-30T15:06:25.348024+0000","last_active":"2024-03-30T15:06:27.379765+0000","last_peered":"2024-03-30T15:06:27.379765+0000","last_clean":"2024-03-30T15:06:27.379765+0000","last_became_active":"2024-03-30T15:06:25.347669+0000","last_became_peered":"2024-03-30T15:06:25.347669+0000","last_unstale":"2024-03-30T15:06:27.379765+0000","last_undegraded":"2024-03-30T15:06:27.379765+0000","last_fullsized":"2024-03-30T15:06:27.379765+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:39.179291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"312.9","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345991+0000","last_change":"2024-03-30T15:06:25.348037+0000","last_active":"2024-03-30T15:06:27.345991+0000","last_peered":"2024-03-30T15:06:27.345991+0000","last_clean":"2024-03-30T15:06:27.345991+0000","last_became_active":"2024-03-30T15:06:25.347714+0000","last_became_peered":"2024-03-30T15:06:25.347714+0000","last_unstale":"2024-03-30T15:06:27.345991+0000","last_undegraded":"2024-03-30T15:06:27.345991+0000","last_fullsized":"2024-03-30T15:06:27.345991+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:39.361376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"312.8","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359206+0000","last_change":"2024-03-30T15:06:25.341528+0000","last_active":"2024-03-30T15:06:27.359206+0000","last_peered":"2024-03-30T15:06:27.359206+0000","last_clean":"2024-03-30T15:06:27.359206+0000","last_became_active":"2024-03-30T15:06:25.341199+0000","last_became_peered":"2024-03-30T15:06:25.341199+0000","last_unstale":"2024-03-30T15:06:27.359206+0000","last_undegraded":"2024-03-30T15:06:27.359206+0000","last_fullsized":"2024-03-30T15:06:27.359206+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:35.122478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"312.7","version":"554'3","reported_seq":18,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.377055+0000","last_change":"2024-03-30T15:06:25.335509+0000","last_active":"2024-03-30T15:06:27.377055+0000","last_peered":"2024-03-30T15:06:27.377055+0000","last_clean":"2024-03-30T15:06:27.377055+0000","last_became_active":"2024-03-30T15:06:25.335198+0000","last_became_peered":"2024-03-30T15:06:25.335198+0000","last_unstale":"2024-03-30T15:06:27.377055+0000","last_undegraded":"2024-03-30T15:06:27.377055+0000","last_fullsized":"2024-03-30T15:06:27.377055+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:29.668718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.6","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344733+0000","last_change":"2024-03-30T15:06:25.348076+0000","last_active":"2024-03-30T15:06:27.344733+0000","last_peered":"2024-03-30T15:06:27.344733+0000","last_clean":"2024-03-30T15:06:27.344733+0000","last_became_active":"2024-03-30T15:06:25.347774+0000","last_became_peered":"2024-03-30T15:06:25.347774+0000","last_unstale":"2024-03-30T15:06:27.344733+0000","last_undegraded":"2024-03-30T15:06:27.344733+0000","last_fullsized":"2024-03-30T15:06:27.344733+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:36.874465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.5","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360551+0000","last_change":"2024-03-30T15:06:25.343659+0000","last_active":"2024-03-30T15:06:27.360551+0000","last_peered":"2024-03-30T15:06:27.360551+0000","last_clean":"2024-03-30T15:06:27.360551+0000","last_became_active":"2024-03-30T15:06:25.343361+0000","last_became_peered":"2024-03-30T15:06:25.343361+0000","last_unstale":"2024-03-30T15:06:27.360551+0000","last_undegraded":"2024-03-30T15:06:27.360551+0000","last_fullsized":"2024-03-30T15:06:27.360551+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:37.176071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"312.4","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343977+0000","last_change":"2024-03-30T15:06:25.342620+0000","last_active":"2024-03-30T15:06:27.343977+0000","last_peered":"2024-03-30T15:06:27.343977+0000","last_clean":"2024-03-30T15:06:27.343977+0000","last_became_active":"2024-03-30T15:06:25.341657+0000","last_became_peered":"2024-03-30T15:06:25.341657+0000","last_unstale":"2024-03-30T15:06:27.343977+0000","last_undegraded":"2024-03-30T15:06:27.343977+0000","last_fullsized":"2024-03-30T15:06:27.343977+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.333289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"312.3","version":"0'0","reported_seq":17,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346836+0000","last_change":"2024-03-30T15:06:25.340409+0000","last_active":"2024-03-30T15:06:28.346836+0000","last_peered":"2024-03-30T15:06:28.346836+0000","last_clean":"2024-03-30T15:06:28.346836+0000","last_became_active":"2024-03-30T15:06:25.339576+0000","last_became_peered":"2024-03-30T15:06:25.339576+0000","last_unstale":"2024-03-30T15:06:28.346836+0000","last_undegraded":"2024-03-30T15:06:28.346836+0000","last_fullsized":"2024-03-30T15:06:28.346836+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:39.876760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"312.2","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338445+0000","last_change":"2024-03-30T15:06:25.344130+0000","last_active":"2024-03-30T15:06:27.338445+0000","last_peered":"2024-03-30T15:06:27.338445+0000","last_clean":"2024-03-30T15:06:27.338445+0000","last_became_active":"2024-03-30T15:06:25.342557+0000","last_became_peered":"2024-03-30T15:06:25.342557+0000","last_unstale":"2024-03-30T15:06:27.338445+0000","last_undegraded":"2024-03-30T15:06:27.338445+0000","last_fullsized":"2024-03-30T15:06:27.338445+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.860073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"312.1","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.349554+0000","last_change":"2024-03-30T15:06:25.337261+0000","last_active":"2024-03-30T15:06:27.349554+0000","last_peered":"2024-03-30T15:06:27.349554+0000","last_clean":"2024-03-30T15:06:27.349554+0000","last_became_active":"2024-03-30T15:06:25.336950+0000","last_became_peered":"2024-03-30T15:06:25.336950+0000","last_unstale":"2024-03-30T15:06:27.349554+0000","last_undegraded":"2024-03-30T15:06:27.349554+0000","last_fullsized":"2024-03-30T15:06:27.349554+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:53.878583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"312.0","version":"0'0","reported_seq":15,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.342860+0000","last_change":"2024-03-30T15:06:25.345240+0000","last_active":"2024-03-30T15:06:27.342860+0000","last_peered":"2024-03-30T15:06:27.342860+0000","last_clean":"2024-03-30T15:06:27.342860+0000","last_became_active":"2024-03-30T15:06:25.344680+0000","last_became_peered":"2024-03-30T15:06:25.344680+0000","last_unstale":"2024-03-30T15:06:27.342860+0000","last_undegraded":"2024-03-30T15:06:27.342860+0000","last_fullsized":"2024-03-30T15:06:27.342860+0000","mapping_epoch":551,"log_start":"0'0","ondisk_log_start":"0'0","created":551,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:24.318414+0000","last_clean_scrub_stamp":"2024-03-30T15:06:24.318414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.405982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.1a","version":"536'46","reported_seq":1128,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347367+0000","last_change":"2024-03-30T15:06:09.990673+0000","last_active":"2024-03-30T15:06:28.347367+0000","last_peered":"2024-03-30T15:06:28.347367+0000","last_clean":"2024-03-30T15:06:28.347367+0000","last_became_active":"2024-03-30T14:58:14.231887+0000","last_became_peered":"2024-03-30T14:58:14.231887+0000","last_unstale":"2024-03-30T15:06:28.347367+0000","last_undegraded":"2024-03-30T15:06:28.347367+0000","last_fullsized":"2024-03-30T15:06:28.347367+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'46","last_scrub_stamp":"2024-03-30T15:06:09.990583+0000","last_deep_scrub":"536'46","last_deep_scrub_stamp":"2024-03-30T15:06:09.990583+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.990583+0000","objects_scrubbed":3,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:27.390331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001239029,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":18,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1b","version":"368'86","reported_seq":1139,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346458+0000","last_change":"2024-03-30T15:05:25.203009+0000","last_active":"2024-03-30T15:06:27.346458+0000","last_peered":"2024-03-30T15:06:27.346458+0000","last_clean":"2024-03-30T15:06:27.346458+0000","last_became_active":"2024-03-30T14:58:14.219923+0000","last_became_peered":"2024-03-30T14:58:14.219923+0000","last_unstale":"2024-03-30T15:06:27.346458+0000","last_undegraded":"2024-03-30T15:06:27.346458+0000","last_fullsized":"2024-03-30T15:06:27.346458+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'86","last_scrub_stamp":"2024-03-30T15:05:25.202851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.202851+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.231027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00073085800000000003,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.18","version":"474'106","reported_seq":1163,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338217+0000","last_change":"2024-03-30T15:05:23.274570+0000","last_active":"2024-03-30T15:06:27.338217+0000","last_peered":"2024-03-30T15:06:27.338217+0000","last_clean":"2024-03-30T15:06:27.338217+0000","last_became_active":"2024-03-30T14:58:14.195836+0000","last_became_peered":"2024-03-30T14:58:14.195836+0000","last_unstale":"2024-03-30T15:06:27.338217+0000","last_undegraded":"2024-03-30T15:06:27.338217+0000","last_fullsized":"2024-03-30T15:06:27.338217+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'106","last_scrub_stamp":"2024-03-30T15:05:23.274464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.274464+0000","objects_scrubbed":11,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:50.009435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011393239999999999,"stat_sum":{"num_bytes":98,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.19","version":"529'107","reported_seq":1170,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350246+0000","last_change":"2024-03-30T15:04:07.012228+0000","last_active":"2024-03-30T15:06:27.350246+0000","last_peered":"2024-03-30T15:06:27.350246+0000","last_clean":"2024-03-30T15:06:27.350246+0000","last_became_active":"2024-03-30T14:58:14.228803+0000","last_became_peered":"2024-03-30T14:58:14.228803+0000","last_unstale":"2024-03-30T15:06:27.350246+0000","last_undegraded":"2024-03-30T15:06:27.350246+0000","last_fullsized":"2024-03-30T15:06:27.350246+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'105","last_scrub_stamp":"2024-03-30T15:04:01.987391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:01.987391+0000","objects_scrubbed":9,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00077776899999999999,"stat_sum":{"num_bytes":2127,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":37,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.1e","version":"491'43","reported_seq":1124,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347602+0000","last_change":"2024-03-30T15:06:04.015679+0000","last_active":"2024-03-30T15:06:28.347602+0000","last_peered":"2024-03-30T15:06:28.347602+0000","last_clean":"2024-03-30T15:06:28.347602+0000","last_became_active":"2024-03-30T14:58:14.248453+0000","last_became_peered":"2024-03-30T14:58:14.248453+0000","last_unstale":"2024-03-30T15:06:28.347602+0000","last_undegraded":"2024-03-30T15:06:28.347602+0000","last_fullsized":"2024-03-30T15:06:28.347602+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'43","last_scrub_stamp":"2024-03-30T15:06:04.015550+0000","last_deep_scrub":"286'36","last_deep_scrub_stamp":"2024-03-30T15:00:50.725810+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.015550+0000","objects_scrubbed":4,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:27.201277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048671400000000002,"stat_sum":{"num_bytes":37,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1f","version":"544'99","reported_seq":1230,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347855+0000","last_change":"2024-03-30T15:06:03.059243+0000","last_active":"2024-03-30T15:06:28.347855+0000","last_peered":"2024-03-30T15:06:28.347855+0000","last_clean":"2024-03-30T15:06:28.347855+0000","last_became_active":"2024-03-30T14:58:14.265476+0000","last_became_peered":"2024-03-30T14:58:14.265476+0000","last_unstale":"2024-03-30T15:06:28.347855+0000","last_undegraded":"2024-03-30T15:06:28.347855+0000","last_fullsized":"2024-03-30T15:06:28.347855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'85","last_scrub_stamp":"2024-03-30T15:06:03.059121+0000","last_deep_scrub":"403'80","last_deep_scrub_stamp":"2024-03-30T15:02:47.929964+0000","last_clean_scrub_stamp":"2024-03-30T15:06:03.059121+0000","objects_scrubbed":11,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.560811+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00066287099999999999,"stat_sum":{"num_bytes":4195579,"num_objects":14,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":38,"num_write_kb":8226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1c","version":"529'41","reported_seq":1104,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346534+0000","last_change":"2024-03-30T15:06:02.001529+0000","last_active":"2024-03-30T15:06:27.346534+0000","last_peered":"2024-03-30T15:06:27.346534+0000","last_clean":"2024-03-30T15:06:27.346534+0000","last_became_active":"2024-03-30T14:58:14.181402+0000","last_became_peered":"2024-03-30T14:58:14.181402+0000","last_unstale":"2024-03-30T15:06:27.346534+0000","last_undegraded":"2024-03-30T15:06:27.346534+0000","last_fullsized":"2024-03-30T15:06:27.346534+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'38","last_scrub_stamp":"2024-03-30T15:06:02.001426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.001426+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.059493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001952826,"stat_sum":{"num_bytes":594,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.1d","version":"544'62","reported_seq":1129,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339517+0000","last_change":"2024-03-30T15:06:17.496610+0000","last_active":"2024-03-30T15:06:27.339517+0000","last_peered":"2024-03-30T15:06:27.339517+0000","last_clean":"2024-03-30T15:06:27.339517+0000","last_became_active":"2024-03-30T14:58:14.203147+0000","last_became_peered":"2024-03-30T14:58:14.203147+0000","last_unstale":"2024-03-30T15:06:27.339517+0000","last_undegraded":"2024-03-30T15:06:27.339517+0000","last_fullsized":"2024-03-30T15:06:27.339517+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"544'62","last_scrub_stamp":"2024-03-30T15:06:17.496499+0000","last_deep_scrub":"544'62","last_deep_scrub_stamp":"2024-03-30T15:06:17.496499+0000","last_clean_scrub_stamp":"2024-03-30T15:06:17.496499+0000","objects_scrubbed":5,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:32.938620+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001981488,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.12","version":"358'44","reported_seq":1112,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359556+0000","last_change":"2024-03-30T15:04:34.489872+0000","last_active":"2024-03-30T15:06:27.359556+0000","last_peered":"2024-03-30T15:06:27.359556+0000","last_clean":"2024-03-30T15:06:27.359556+0000","last_became_active":"2024-03-30T14:58:14.208597+0000","last_became_peered":"2024-03-30T14:58:14.208597+0000","last_unstale":"2024-03-30T15:06:27.359556+0000","last_undegraded":"2024-03-30T15:06:27.359556+0000","last_fullsized":"2024-03-30T15:06:27.359556+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-30T15:04:34.489778+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-30T15:02:02.557627+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.489778+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072970900000000004,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.13","version":"511'44","reported_seq":1116,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360807+0000","last_change":"2024-03-30T15:05:24.551301+0000","last_active":"2024-03-30T15:06:27.360807+0000","last_peered":"2024-03-30T15:06:27.360807+0000","last_clean":"2024-03-30T15:06:27.360807+0000","last_became_active":"2024-03-30T14:58:14.236817+0000","last_became_peered":"2024-03-30T14:58:14.236817+0000","last_unstale":"2024-03-30T15:06:27.360807+0000","last_undegraded":"2024-03-30T15:06:27.360807+0000","last_fullsized":"2024-03-30T15:06:27.360807+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-30T15:05:24.551158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.551158+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.310396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010675319999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":24,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.10","version":"520'30","reported_seq":1099,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338306+0000","last_change":"2024-03-30T15:05:24.227205+0000","last_active":"2024-03-30T15:06:27.338306+0000","last_peered":"2024-03-30T15:06:27.338306+0000","last_clean":"2024-03-30T15:06:27.338306+0000","last_became_active":"2024-03-30T14:58:14.195867+0000","last_became_peered":"2024-03-30T14:58:14.195867+0000","last_unstale":"2024-03-30T15:06:27.338306+0000","last_undegraded":"2024-03-30T15:06:27.338306+0000","last_fullsized":"2024-03-30T15:06:27.338306+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'23","last_scrub_stamp":"2024-03-30T15:05:24.227122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.227122+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:40.603804+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010795309999999999,"stat_sum":{"num_bytes":2376,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":23,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.11","version":"520'22","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338447+0000","last_change":"2024-03-30T15:05:25.269379+0000","last_active":"2024-03-30T15:06:27.338447+0000","last_peered":"2024-03-30T15:06:27.338447+0000","last_clean":"2024-03-30T15:06:27.338447+0000","last_became_active":"2024-03-30T14:58:14.188341+0000","last_became_peered":"2024-03-30T14:58:14.188341+0000","last_unstale":"2024-03-30T15:06:27.338447+0000","last_undegraded":"2024-03-30T15:06:27.338447+0000","last_fullsized":"2024-03-30T15:06:27.338447+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'21","last_scrub_stamp":"2024-03-30T15:05:25.269264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:25.269264+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:54.222997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016992859999999999,"stat_sum":{"num_bytes":586,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.16","version":"529'25","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347336+0000","last_change":"2024-03-30T15:05:56.954018+0000","last_active":"2024-03-30T15:06:27.347336+0000","last_peered":"2024-03-30T15:06:27.347336+0000","last_clean":"2024-03-30T15:06:27.347336+0000","last_became_active":"2024-03-30T14:58:14.110970+0000","last_became_peered":"2024-03-30T14:58:14.110970+0000","last_unstale":"2024-03-30T15:06:27.347336+0000","last_undegraded":"2024-03-30T15:06:27.347336+0000","last_fullsized":"2024-03-30T15:06:27.347336+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'23","last_scrub_stamp":"2024-03-30T15:05:56.953844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:56.953844+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:03.699456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0013481890000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.17","version":"529'83","reported_seq":1145,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361442+0000","last_change":"2024-03-30T15:04:06.474824+0000","last_active":"2024-03-30T15:06:27.361442+0000","last_peered":"2024-03-30T15:06:27.361442+0000","last_clean":"2024-03-30T15:06:27.361442+0000","last_became_active":"2024-03-30T14:58:14.286995+0000","last_became_peered":"2024-03-30T14:58:14.286995+0000","last_unstale":"2024-03-30T15:06:27.361442+0000","last_undegraded":"2024-03-30T15:06:27.361442+0000","last_fullsized":"2024-03-30T15:06:27.361442+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'74","last_scrub_stamp":"2024-03-30T15:02:38.712248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:38.712248+0000","objects_scrubbed":6,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069842899999999998,"stat_sum":{"num_bytes":2377,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":11,"num_read_kb":3,"num_write":33,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.14","version":"510'62","reported_seq":1131,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360511+0000","last_change":"2024-03-30T15:06:08.553305+0000","last_active":"2024-03-30T15:06:27.360511+0000","last_peered":"2024-03-30T15:06:27.360511+0000","last_clean":"2024-03-30T15:06:27.360511+0000","last_became_active":"2024-03-30T14:58:14.225691+0000","last_became_peered":"2024-03-30T14:58:14.225691+0000","last_unstale":"2024-03-30T15:06:27.360511+0000","last_undegraded":"2024-03-30T15:06:27.360511+0000","last_fullsized":"2024-03-30T15:06:27.360511+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'62","last_scrub_stamp":"2024-03-30T15:06:08.553141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.553141+0000","objects_scrubbed":6,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.452312+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00090324199999999996,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.15","version":"520'94","reported_seq":1159,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.348051+0000","last_change":"2024-03-30T15:06:01.014001+0000","last_active":"2024-03-30T15:06:28.348051+0000","last_peered":"2024-03-30T15:06:28.348051+0000","last_clean":"2024-03-30T15:06:28.348051+0000","last_became_active":"2024-03-30T14:58:14.257693+0000","last_became_peered":"2024-03-30T14:58:14.257693+0000","last_unstale":"2024-03-30T15:06:28.348051+0000","last_undegraded":"2024-03-30T15:06:28.348051+0000","last_fullsized":"2024-03-30T15:06:28.348051+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'94","last_scrub_stamp":"2024-03-30T15:06:01.013913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.013913+0000","objects_scrubbed":7,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.513918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00096732999999999997,"stat_sum":{"num_bytes":1605,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.a","version":"554'60","reported_seq":1122,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.373968+0000","last_change":"2024-03-30T15:05:22.315909+0000","last_active":"2024-03-30T15:06:27.373968+0000","last_peered":"2024-03-30T15:06:27.373968+0000","last_clean":"2024-03-30T15:06:27.373968+0000","last_became_active":"2024-03-30T14:58:14.235403+0000","last_became_peered":"2024-03-30T14:58:14.235403+0000","last_unstale":"2024-03-30T15:06:27.373968+0000","last_undegraded":"2024-03-30T15:06:27.373968+0000","last_fullsized":"2024-03-30T15:06:27.373968+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'37","last_scrub_stamp":"2024-03-30T15:05:22.315812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.315812+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:51.897161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029388449999999998,"stat_sum":{"num_bytes":1305,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":22,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.b","version":"520'113","reported_seq":1178,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343738+0000","last_change":"2024-03-30T15:05:28.105979+0000","last_active":"2024-03-30T15:06:27.343738+0000","last_peered":"2024-03-30T15:06:27.343738+0000","last_clean":"2024-03-30T15:06:27.343738+0000","last_became_active":"2024-03-30T14:58:14.109986+0000","last_became_peered":"2024-03-30T14:58:14.109986+0000","last_unstale":"2024-03-30T15:06:27.343738+0000","last_undegraded":"2024-03-30T15:06:27.343738+0000","last_fullsized":"2024-03-30T15:06:27.343738+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'111","last_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_deep_scrub":"499'111","last_deep_scrub_stamp":"2024-03-30T15:05:28.105889+0000","last_clean_scrub_stamp":"2024-03-30T15:05:28.105889+0000","objects_scrubbed":7,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:51.857504+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011471140000000001,"stat_sum":{"num_bytes":1932,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":36,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.8","version":"536'64","reported_seq":1141,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359530+0000","last_change":"2024-03-30T15:06:10.223409+0000","last_active":"2024-03-30T15:06:27.359530+0000","last_peered":"2024-03-30T15:06:27.359530+0000","last_clean":"2024-03-30T15:06:27.359530+0000","last_became_active":"2024-03-30T14:58:14.227244+0000","last_became_peered":"2024-03-30T14:58:14.227244+0000","last_unstale":"2024-03-30T15:06:27.359530+0000","last_undegraded":"2024-03-30T15:06:27.359530+0000","last_fullsized":"2024-03-30T15:06:27.359530+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'64","last_scrub_stamp":"2024-03-30T15:06:10.223307+0000","last_deep_scrub":"536'64","last_deep_scrub_stamp":"2024-03-30T15:06:10.223307+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.223307+0000","objects_scrubbed":4,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.987753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001423587,"stat_sum":{"num_bytes":551,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":25,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"101.9","version":"254'37","reported_seq":1112,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343189+0000","last_change":"2024-03-30T15:04:55.161018+0000","last_active":"2024-03-30T15:06:27.343189+0000","last_peered":"2024-03-30T15:06:27.343189+0000","last_clean":"2024-03-30T15:06:27.343189+0000","last_became_active":"2024-03-30T14:58:14.180574+0000","last_became_peered":"2024-03-30T14:58:14.180574+0000","last_unstale":"2024-03-30T15:06:27.343189+0000","last_undegraded":"2024-03-30T15:06:27.343189+0000","last_fullsized":"2024-03-30T15:06:27.343189+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'37","last_scrub_stamp":"2024-03-30T15:04:55.160916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.160916+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001171655,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.e","version":"529'120","reported_seq":1182,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345052+0000","last_change":"2024-03-30T15:04:06.383765+0000","last_active":"2024-03-30T15:06:27.345052+0000","last_peered":"2024-03-30T15:06:27.345052+0000","last_clean":"2024-03-30T15:06:27.345052+0000","last_became_active":"2024-03-30T14:58:14.118019+0000","last_became_peered":"2024-03-30T14:58:14.118019+0000","last_unstale":"2024-03-30T15:06:27.345052+0000","last_undegraded":"2024-03-30T15:06:27.345052+0000","last_fullsized":"2024-03-30T15:06:27.345052+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'108","last_scrub_stamp":"2024-03-30T15:03:53.395375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.395375+0000","objects_scrubbed":8,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00077213800000000001,"stat_sum":{"num_bytes":1621,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":15,"num_read_kb":5,"num_write":46,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.f","version":"538'63","reported_seq":1135,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361720+0000","last_change":"2024-03-30T15:04:35.492607+0000","last_active":"2024-03-30T15:06:27.361720+0000","last_peered":"2024-03-30T15:06:27.361720+0000","last_clean":"2024-03-30T15:06:27.361720+0000","last_became_active":"2024-03-30T14:58:14.219763+0000","last_became_peered":"2024-03-30T14:58:14.219763+0000","last_unstale":"2024-03-30T15:06:27.361720+0000","last_undegraded":"2024-03-30T15:06:27.361720+0000","last_fullsized":"2024-03-30T15:06:27.361720+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'60","last_scrub_stamp":"2024-03-30T15:04:35.492526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.492526+0000","objects_scrubbed":6,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000710639,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.c","version":"463'61","reported_seq":1130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339180+0000","last_change":"2024-03-30T15:06:10.529904+0000","last_active":"2024-03-30T15:06:27.339180+0000","last_peered":"2024-03-30T15:06:27.339180+0000","last_clean":"2024-03-30T15:06:27.339180+0000","last_became_active":"2024-03-30T14:58:14.105660+0000","last_became_peered":"2024-03-30T14:58:14.105660+0000","last_unstale":"2024-03-30T15:06:27.339180+0000","last_undegraded":"2024-03-30T15:06:27.339180+0000","last_fullsized":"2024-03-30T15:06:27.339180+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'61","last_scrub_stamp":"2024-03-30T15:06:10.529776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.529776+0000","objects_scrubbed":7,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.242612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020687829999999998,"stat_sum":{"num_bytes":59,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.d","version":"520'98","reported_seq":1152,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347234+0000","last_change":"2024-03-30T15:05:21.308139+0000","last_active":"2024-03-30T15:06:27.347234+0000","last_peered":"2024-03-30T15:06:27.347234+0000","last_clean":"2024-03-30T15:06:27.347234+0000","last_became_active":"2024-03-30T14:58:14.220478+0000","last_became_peered":"2024-03-30T14:58:14.220478+0000","last_unstale":"2024-03-30T15:06:27.347234+0000","last_undegraded":"2024-03-30T15:06:27.347234+0000","last_fullsized":"2024-03-30T15:06:27.347234+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'97","last_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_deep_scrub":"498'97","last_deep_scrub_stamp":"2024-03-30T15:05:21.308003+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.308003+0000","objects_scrubbed":8,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.152616+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015048329999999999,"stat_sum":{"num_bytes":1294,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"101.2","version":"453'102","reported_seq":1161,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350485+0000","last_change":"2024-03-30T15:05:59.137056+0000","last_active":"2024-03-30T15:06:27.350485+0000","last_peered":"2024-03-30T15:06:27.350485+0000","last_clean":"2024-03-30T15:06:27.350485+0000","last_became_active":"2024-03-30T14:58:14.192499+0000","last_became_peered":"2024-03-30T14:58:14.192499+0000","last_unstale":"2024-03-30T15:06:27.350485+0000","last_undegraded":"2024-03-30T15:06:27.350485+0000","last_fullsized":"2024-03-30T15:06:27.350485+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'102","last_scrub_stamp":"2024-03-30T15:05:59.136916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.136916+0000","objects_scrubbed":8,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.038485+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0025419280000000002,"stat_sum":{"num_bytes":81,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"101.3","version":"511'43","reported_seq":1121,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346912+0000","last_change":"2024-03-30T15:04:35.992306+0000","last_active":"2024-03-30T15:06:28.346912+0000","last_peered":"2024-03-30T15:06:28.346912+0000","last_clean":"2024-03-30T15:06:28.346912+0000","last_became_active":"2024-03-30T14:58:14.227063+0000","last_became_peered":"2024-03-30T14:58:14.227063+0000","last_unstale":"2024-03-30T15:06:28.346912+0000","last_undegraded":"2024-03-30T15:06:28.346912+0000","last_fullsized":"2024-03-30T15:06:28.346912+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'39","last_scrub_stamp":"2024-03-30T15:04:35.992167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.992167+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053779299999999995,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.0","version":"510'44","reported_seq":1125,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346538+0000","last_change":"2024-03-30T15:04:06.343774+0000","last_active":"2024-03-30T15:06:28.346538+0000","last_peered":"2024-03-30T15:06:28.346538+0000","last_clean":"2024-03-30T15:06:28.346538+0000","last_became_active":"2024-03-30T14:58:14.272880+0000","last_became_peered":"2024-03-30T14:58:14.272880+0000","last_unstale":"2024-03-30T15:06:28.346538+0000","last_undegraded":"2024-03-30T15:06:28.346538+0000","last_fullsized":"2024-03-30T15:06:28.346538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'42","last_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_deep_scrub":"452'42","last_deep_scrub_stamp":"2024-03-30T15:04:06.077392+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.077392+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013898669999999999,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.1","version":"511'123","reported_seq":1192,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343591+0000","last_change":"2024-03-30T15:04:06.383650+0000","last_active":"2024-03-30T15:06:27.343591+0000","last_peered":"2024-03-30T15:06:27.343591+0000","last_clean":"2024-03-30T15:06:27.343591+0000","last_became_active":"2024-03-30T14:58:14.155803+0000","last_became_peered":"2024-03-30T14:58:14.155803+0000","last_unstale":"2024-03-30T15:06:27.343591+0000","last_undegraded":"2024-03-30T15:06:27.343591+0000","last_fullsized":"2024-03-30T15:06:27.343591+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'104","last_scrub_stamp":"2024-03-30T15:03:50.351355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.351355+0000","objects_scrubbed":8,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001214545,"stat_sum":{"num_bytes":4000104,"num_objects":11,"num_object_clones":2,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":2,"num_write":43,"num_write_kb":3944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"101.6","version":"529'30","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340197+0000","last_change":"2024-03-30T15:05:21.307427+0000","last_active":"2024-03-30T15:06:27.340197+0000","last_peered":"2024-03-30T15:06:27.340197+0000","last_clean":"2024-03-30T15:06:27.340197+0000","last_became_active":"2024-03-30T14:58:14.186606+0000","last_became_peered":"2024-03-30T14:58:14.186606+0000","last_unstale":"2024-03-30T15:06:27.340197+0000","last_undegraded":"2024-03-30T15:06:27.340197+0000","last_fullsized":"2024-03-30T15:06:27.340197+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'23","last_scrub_stamp":"2024-03-30T15:05:21.307305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.307305+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.823122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0023482160000000002,"stat_sum":{"num_bytes":1234,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":19,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"101.7","version":"529'3","reported_seq":1088,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.348297+0000","last_change":"2024-03-30T15:05:13.117385+0000","last_active":"2024-03-30T15:06:28.348297+0000","last_peered":"2024-03-30T15:06:28.348297+0000","last_clean":"2024-03-30T15:06:28.348297+0000","last_became_active":"2024-03-30T14:58:14.278668+0000","last_became_peered":"2024-03-30T14:58:14.278668+0000","last_unstale":"2024-03-30T15:06:28.348297+0000","last_undegraded":"2024-03-30T15:06:28.348297+0000","last_fullsized":"2024-03-30T15:06:28.348297+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'2","last_scrub_stamp":"2024-03-30T15:05:13.117303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.117303+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.886298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054311500000000005,"stat_sum":{"num_bytes":716,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"101.4","version":"235'68","reported_seq":1121,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360565+0000","last_change":"2024-03-30T15:04:06.474470+0000","last_active":"2024-03-30T15:06:27.360565+0000","last_peered":"2024-03-30T15:06:27.360565+0000","last_clean":"2024-03-30T15:06:27.360565+0000","last_became_active":"2024-03-30T14:58:14.249751+0000","last_became_peered":"2024-03-30T14:58:14.249751+0000","last_unstale":"2024-03-30T15:06:27.360565+0000","last_undegraded":"2024-03-30T15:06:27.360565+0000","last_fullsized":"2024-03-30T15:06:27.360565+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'68","last_scrub_stamp":"2024-03-30T15:02:42.619737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:02:42.619737+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001027107,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"101.5","version":"510'105","reported_seq":1167,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361129+0000","last_change":"2024-03-30T15:04:07.017263+0000","last_active":"2024-03-30T15:06:27.361129+0000","last_peered":"2024-03-30T15:06:27.361129+0000","last_clean":"2024-03-30T15:06:27.361129+0000","last_became_active":"2024-03-30T14:58:14.251391+0000","last_became_peered":"2024-03-30T14:58:14.251391+0000","last_unstale":"2024-03-30T15:06:27.361129+0000","last_undegraded":"2024-03-30T15:06:27.361129+0000","last_fullsized":"2024-03-30T15:06:27.361129+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'103","last_scrub_stamp":"2024-03-30T15:03:51.152307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:13.034736+0000","last_clean_scrub_stamp":"2024-03-30T15:03:51.152307+0000","objects_scrubbed":7,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000387501,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360296+0000","last_change":"2024-03-30T15:04:42.056061+0000","last_active":"2024-03-30T15:06:27.360296+0000","last_peered":"2024-03-30T15:06:27.360296+0000","last_clean":"2024-03-30T15:06:27.360296+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:06:27.360296+0000","last_undegraded":"2024-03-30T15:06:27.360296+0000","last_fullsized":"2024-03-30T15:06:27.360296+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.055955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.055955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344292+0000","last_change":"2024-03-30T15:04:34.266981+0000","last_active":"2024-03-30T15:06:27.344292+0000","last_peered":"2024-03-30T15:06:27.344292+0000","last_clean":"2024-03-30T15:06:27.344292+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:06:27.344292+0000","last_undegraded":"2024-03-30T15:06:27.344292+0000","last_fullsized":"2024-03-30T15:06:27.344292+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:34.266865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.266865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347949+0000","last_change":"2024-03-30T15:04:42.277788+0000","last_active":"2024-03-30T15:06:27.347949+0000","last_peered":"2024-03-30T15:06:27.347949+0000","last_clean":"2024-03-30T15:06:27.347949+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:06:27.347949+0000","last_undegraded":"2024-03-30T15:06:27.347949+0000","last_fullsized":"2024-03-30T15:06:27.347949+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352191+0000","last_change":"2024-03-30T15:03:54.882274+0000","last_active":"2024-03-30T15:06:27.352191+0000","last_peered":"2024-03-30T15:06:27.352191+0000","last_clean":"2024-03-30T15:06:27.352191+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:06:27.352191+0000","last_undegraded":"2024-03-30T15:06:27.352191+0000","last_fullsized":"2024-03-30T15:06:27.352191+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.882183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.882183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":908,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359658+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:06:27.359658+0000","last_peered":"2024-03-30T15:06:27.359658+0000","last_clean":"2024-03-30T15:06:27.359658+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:06:27.359658+0000","last_undegraded":"2024-03-30T15:06:27.359658+0000","last_fullsized":"2024-03-30T15:06:27.359658+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.284980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":917,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346563+0000","last_change":"2024-03-30T15:05:27.005973+0000","last_active":"2024-03-30T15:06:28.346563+0000","last_peered":"2024-03-30T15:06:28.346563+0000","last_clean":"2024-03-30T15:06:28.346563+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:06:28.346563+0000","last_undegraded":"2024-03-30T15:06:28.346563+0000","last_fullsized":"2024-03-30T15:06:28.346563+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.005887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.577288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346842+0000","last_change":"2024-03-30T15:05:20.323134+0000","last_active":"2024-03-30T15:06:27.346842+0000","last_peered":"2024-03-30T15:06:27.346842+0000","last_clean":"2024-03-30T15:06:27.346842+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:06:27.346842+0000","last_undegraded":"2024-03-30T15:06:27.346842+0000","last_fullsized":"2024-03-30T15:06:27.346842+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.323015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.323015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.231176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359881+0000","last_change":"2024-03-30T15:04:38.084291+0000","last_active":"2024-03-30T15:06:27.359881+0000","last_peered":"2024-03-30T15:06:27.359881+0000","last_clean":"2024-03-30T15:06:27.359881+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:06:27.359881+0000","last_undegraded":"2024-03-30T15:06:27.359881+0000","last_fullsized":"2024-03-30T15:06:27.359881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:38.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":903,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339540+0000","last_change":"2024-03-30T15:05:19.276266+0000","last_active":"2024-03-30T15:06:27.339540+0000","last_peered":"2024-03-30T15:06:27.339540+0000","last_clean":"2024-03-30T15:06:27.339540+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:06:27.339540+0000","last_undegraded":"2024-03-30T15:06:27.339540+0000","last_fullsized":"2024-03-30T15:06:27.339540+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.276185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.276185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.789727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351491+0000","last_change":"2024-03-30T15:06:18.054438+0000","last_active":"2024-03-30T15:06:27.351491+0000","last_peered":"2024-03-30T15:06:27.351491+0000","last_clean":"2024-03-30T15:06:27.351491+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:06:27.351491+0000","last_undegraded":"2024-03-30T15:06:27.351491+0000","last_fullsized":"2024-03-30T15:06:27.351491+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:18.054358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:06:18.054358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.537680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347947+0000","last_change":"2024-03-30T15:05:17.336662+0000","last_active":"2024-03-30T15:06:27.347947+0000","last_peered":"2024-03-30T15:06:27.347947+0000","last_clean":"2024-03-30T15:06:27.347947+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:06:27.347947+0000","last_undegraded":"2024-03-30T15:06:27.347947+0000","last_fullsized":"2024-03-30T15:06:27.347947+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.336577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.336577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":918,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347713+0000","last_change":"2024-03-30T15:05:19.069973+0000","last_active":"2024-03-30T15:06:28.347713+0000","last_peered":"2024-03-30T15:06:28.347713+0000","last_clean":"2024-03-30T15:06:28.347713+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:06:28.347713+0000","last_undegraded":"2024-03-30T15:06:28.347713+0000","last_fullsized":"2024-03-30T15:06:28.347713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.069896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.069896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:35.468169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345968+0000","last_change":"2024-03-30T15:05:19.327818+0000","last_active":"2024-03-30T15:06:27.345968+0000","last_peered":"2024-03-30T15:06:27.345968+0000","last_clean":"2024-03-30T15:06:27.345968+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:06:27.345968+0000","last_undegraded":"2024-03-30T15:06:27.345968+0000","last_fullsized":"2024-03-30T15:06:27.345968+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.327720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.327720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:43.657218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.349379+0000","last_change":"2024-03-30T15:03:22.969871+0000","last_active":"2024-03-30T15:06:27.349379+0000","last_peered":"2024-03-30T15:06:27.349379+0000","last_clean":"2024-03-30T15:06:27.349379+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:06:27.349379+0000","last_undegraded":"2024-03-30T15:06:27.349379+0000","last_fullsized":"2024-03-30T15:06:27.349379+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.969776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.969776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":917,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346980+0000","last_change":"2024-03-30T15:04:37.008749+0000","last_active":"2024-03-30T15:06:28.346980+0000","last_peered":"2024-03-30T15:06:28.346980+0000","last_clean":"2024-03-30T15:06:28.346980+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:06:28.346980+0000","last_undegraded":"2024-03-30T15:06:28.346980+0000","last_fullsized":"2024-03-30T15:06:28.346980+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.008666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.008666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":917,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347423+0000","last_change":"2024-03-30T15:04:54.955662+0000","last_active":"2024-03-30T15:06:28.347423+0000","last_peered":"2024-03-30T15:06:28.347423+0000","last_clean":"2024-03-30T15:06:28.347423+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:06:28.347423+0000","last_undegraded":"2024-03-30T15:06:28.347423+0000","last_fullsized":"2024-03-30T15:06:28.347423+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.955541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.955541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347273+0000","last_change":"2024-03-30T15:04:55.271526+0000","last_active":"2024-03-30T15:06:27.347273+0000","last_peered":"2024-03-30T15:06:27.347273+0000","last_clean":"2024-03-30T15:06:27.347273+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:06:27.347273+0000","last_undegraded":"2024-03-30T15:06:27.347273+0000","last_fullsized":"2024-03-30T15:06:27.347273+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.271452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.271452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339468+0000","last_change":"2024-03-30T15:04:45.608747+0000","last_active":"2024-03-30T15:06:27.339468+0000","last_peered":"2024-03-30T15:06:27.339468+0000","last_clean":"2024-03-30T15:06:27.339468+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:06:27.339468+0000","last_undegraded":"2024-03-30T15:06:27.339468+0000","last_fullsized":"2024-03-30T15:06:27.339468+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.608652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.608652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361092+0000","last_change":"2024-03-30T15:04:36.136344+0000","last_active":"2024-03-30T15:06:27.361092+0000","last_peered":"2024-03-30T15:06:27.361092+0000","last_clean":"2024-03-30T15:06:27.361092+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:06:27.361092+0000","last_undegraded":"2024-03-30T15:06:27.361092+0000","last_fullsized":"2024-03-30T15:06:27.361092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.136228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.136228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347459+0000","last_change":"2024-03-30T15:04:50.298381+0000","last_active":"2024-03-30T15:06:27.347459+0000","last_peered":"2024-03-30T15:06:27.347459+0000","last_clean":"2024-03-30T15:06:27.347459+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:06:27.347459+0000","last_undegraded":"2024-03-30T15:06:27.347459+0000","last_fullsized":"2024-03-30T15:06:27.347459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.298282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":913,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350422+0000","last_change":"2024-03-30T15:06:12.137521+0000","last_active":"2024-03-30T15:06:27.350422+0000","last_peered":"2024-03-30T15:06:27.350422+0000","last_clean":"2024-03-30T15:06:27.350422+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:06:27.350422+0000","last_undegraded":"2024-03-30T15:06:27.350422+0000","last_fullsized":"2024-03-30T15:06:27.350422+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.137432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:27.797807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345906+0000","last_change":"2024-03-30T15:04:08.370413+0000","last_active":"2024-03-30T15:06:27.345906+0000","last_peered":"2024-03-30T15:06:27.345906+0000","last_clean":"2024-03-30T15:06:27.345906+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:06:27.345906+0000","last_undegraded":"2024-03-30T15:06:27.345906+0000","last_fullsized":"2024-03-30T15:06:27.345906+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.370328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.370328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062717+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":913,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343836+0000","last_change":"2024-03-30T15:05:19.111999+0000","last_active":"2024-03-30T15:06:27.343836+0000","last_peered":"2024-03-30T15:06:27.343836+0000","last_clean":"2024-03-30T15:06:27.343836+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:06:27.343836+0000","last_undegraded":"2024-03-30T15:06:27.343836+0000","last_fullsized":"2024-03-30T15:06:27.343836+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.111909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.111909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.337658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":918,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346962+0000","last_change":"2024-03-30T15:03:57.148819+0000","last_active":"2024-03-30T15:06:28.346962+0000","last_peered":"2024-03-30T15:06:28.346962+0000","last_clean":"2024-03-30T15:06:28.346962+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:06:28.346962+0000","last_undegraded":"2024-03-30T15:06:28.346962+0000","last_fullsized":"2024-03-30T15:06:28.346962+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:57.148742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:03:57.148742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344512+0000","last_change":"2024-03-30T15:05:29.140639+0000","last_active":"2024-03-30T15:06:27.344512+0000","last_peered":"2024-03-30T15:06:27.344512+0000","last_clean":"2024-03-30T15:06:27.344512+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:06:27.344512+0000","last_undegraded":"2024-03-30T15:06:27.344512+0000","last_fullsized":"2024-03-30T15:06:27.344512+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.140508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.140508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:57.419807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":29,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359818+0000","last_change":"2024-03-30T15:06:18.488509+0000","last_active":"2024-03-30T15:06:27.359818+0000","last_peered":"2024-03-30T15:06:27.359818+0000","last_clean":"2024-03-30T15:06:27.359818+0000","last_became_active":"2024-03-30T15:06:18.488207+0000","last_became_peered":"2024-03-30T15:06:18.488207+0000","last_unstale":"2024-03-30T15:06:27.359818+0000","last_undegraded":"2024-03-30T15:06:27.359818+0000","last_fullsized":"2024-03-30T15:06:27.359818+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.176687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.176687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.561502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361475+0000","last_change":"2024-03-30T15:04:55.123255+0000","last_active":"2024-03-30T15:06:27.361475+0000","last_peered":"2024-03-30T15:06:27.361475+0000","last_clean":"2024-03-30T15:06:27.361475+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:06:27.361475+0000","last_undegraded":"2024-03-30T15:06:27.361475+0000","last_fullsized":"2024-03-30T15:06:27.361475+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.123176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.123176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360711+0000","last_change":"2024-03-30T15:04:45.130399+0000","last_active":"2024-03-30T15:06:27.360711+0000","last_peered":"2024-03-30T15:06:27.360711+0000","last_clean":"2024-03-30T15:06:27.360711+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:06:27.360711+0000","last_undegraded":"2024-03-30T15:06:27.360711+0000","last_fullsized":"2024-03-30T15:06:27.360711+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:45.130322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:45.130322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343262+0000","last_change":"2024-03-30T15:04:44.264561+0000","last_active":"2024-03-30T15:06:27.343262+0000","last_peered":"2024-03-30T15:06:27.343262+0000","last_clean":"2024-03-30T15:06:27.343262+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:06:27.343262+0000","last_undegraded":"2024-03-30T15:06:27.343262+0000","last_fullsized":"2024-03-30T15:06:27.343262+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.264430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.264430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":905,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359316+0000","last_change":"2024-03-30T15:04:03.083845+0000","last_active":"2024-03-30T15:06:27.359316+0000","last_peered":"2024-03-30T15:06:27.359316+0000","last_clean":"2024-03-30T15:06:27.359316+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:06:27.359316+0000","last_undegraded":"2024-03-30T15:06:27.359316+0000","last_fullsized":"2024-03-30T15:06:27.359316+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.083707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.083707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":903,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339008+0000","last_change":"2024-03-30T15:04:54.465626+0000","last_active":"2024-03-30T15:06:27.339008+0000","last_peered":"2024-03-30T15:06:27.339008+0000","last_clean":"2024-03-30T15:06:27.339008+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:06:27.339008+0000","last_undegraded":"2024-03-30T15:06:27.339008+0000","last_fullsized":"2024-03-30T15:06:27.339008+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:54.465543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.465543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":904,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346650+0000","last_change":"2024-03-30T15:04:53.311876+0000","last_active":"2024-03-30T15:06:27.346650+0000","last_peered":"2024-03-30T15:06:27.346650+0000","last_clean":"2024-03-30T15:06:27.346650+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:06:27.346650+0000","last_undegraded":"2024-03-30T15:06:27.346650+0000","last_fullsized":"2024-03-30T15:06:27.346650+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.311797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.311797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344353+0000","last_change":"2024-03-30T15:03:21.465702+0000","last_active":"2024-03-30T15:06:27.344353+0000","last_peered":"2024-03-30T15:06:27.344353+0000","last_clean":"2024-03-30T15:06:27.344353+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:06:27.344353+0000","last_undegraded":"2024-03-30T15:06:27.344353+0000","last_fullsized":"2024-03-30T15:06:27.344353+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:21.465628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:21.465628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1038,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339517+0000","last_change":"2024-03-30T15:05:57.752605+0000","last_active":"2024-03-30T15:06:27.339517+0000","last_peered":"2024-03-30T15:06:27.339517+0000","last_clean":"2024-03-30T15:06:27.339517+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:06:27.339517+0000","last_undegraded":"2024-03-30T15:06:27.339517+0000","last_fullsized":"2024-03-30T15:06:27.339517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:05:57.752440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:57.752440+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:08.206404+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361338+0000","last_change":"2024-03-30T15:06:11.536378+0000","last_active":"2024-03-30T15:06:27.361338+0000","last_peered":"2024-03-30T15:06:27.361338+0000","last_clean":"2024-03-30T15:06:27.361338+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:06:27.361338+0000","last_undegraded":"2024-03-30T15:06:27.361338+0000","last_fullsized":"2024-03-30T15:06:27.361338+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.302034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":168,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344426+0000","last_change":"2024-03-30T15:05:06.360304+0000","last_active":"2024-03-30T15:06:27.344426+0000","last_peered":"2024-03-30T15:06:27.344426+0000","last_clean":"2024-03-30T15:06:27.344426+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:06:27.344426+0000","last_undegraded":"2024-03-30T15:06:27.344426+0000","last_fullsized":"2024-03-30T15:06:27.344426+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:04:03.157352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.157352+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":992,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.341005+0000","last_change":"2024-03-30T15:04:00.522178+0000","last_active":"2024-03-30T15:06:27.341005+0000","last_peered":"2024-03-30T15:06:27.341005+0000","last_clean":"2024-03-30T15:06:27.341005+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:06:27.341005+0000","last_undegraded":"2024-03-30T15:06:27.341005+0000","last_fullsized":"2024-03-30T15:06:27.341005+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:00.522082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:00.522082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344645+0000","last_change":"2024-03-30T15:04:08.363047+0000","last_active":"2024-03-30T15:06:27.344645+0000","last_peered":"2024-03-30T15:06:27.344645+0000","last_clean":"2024-03-30T15:06:27.344645+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:06:27.344645+0000","last_undegraded":"2024-03-30T15:06:27.344645+0000","last_fullsized":"2024-03-30T15:06:27.344645+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:08.362897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:08.362897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1057,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340084+0000","last_change":"2024-03-30T15:04:34.512958+0000","last_active":"2024-03-30T15:06:27.340084+0000","last_peered":"2024-03-30T15:06:27.340084+0000","last_clean":"2024-03-30T15:06:27.340084+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:06:27.340084+0000","last_undegraded":"2024-03-30T15:06:27.340084+0000","last_fullsized":"2024-03-30T15:06:27.340084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'17","last_scrub_stamp":"2024-03-30T15:04:34.512838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:34.512838+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1020,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343613+0000","last_change":"2024-03-30T15:04:48.217127+0000","last_active":"2024-03-30T15:06:27.343613+0000","last_peered":"2024-03-30T15:06:27.343613+0000","last_clean":"2024-03-30T15:06:27.343613+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:06:27.343613+0000","last_undegraded":"2024-03-30T15:06:27.343613+0000","last_fullsized":"2024-03-30T15:06:27.343613+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.217046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.217046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1013,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351837+0000","last_change":"2024-03-30T15:04:48.997109+0000","last_active":"2024-03-30T15:06:27.351837+0000","last_peered":"2024-03-30T15:06:27.351837+0000","last_clean":"2024-03-30T15:06:27.351837+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:06:27.351837+0000","last_undegraded":"2024-03-30T15:06:27.351837+0000","last_fullsized":"2024-03-30T15:06:27.351837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.997021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.997021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1012,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.348461+0000","last_change":"2024-03-30T15:04:10.080847+0000","last_active":"2024-03-30T15:06:28.348461+0000","last_peered":"2024-03-30T15:06:28.348461+0000","last_clean":"2024-03-30T15:06:28.348461+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:06:28.348461+0000","last_undegraded":"2024-03-30T15:06:28.348461+0000","last_fullsized":"2024-03-30T15:06:28.348461+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.080718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.080718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352603+0000","last_change":"2024-03-30T15:06:05.089656+0000","last_active":"2024-03-30T15:06:27.352603+0000","last_peered":"2024-03-30T15:06:27.352603+0000","last_clean":"2024-03-30T15:06:27.352603+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:06:27.352603+0000","last_undegraded":"2024-03-30T15:06:27.352603+0000","last_fullsized":"2024-03-30T15:06:27.352603+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:05.089580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.089580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:10.423897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1044,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340865+0000","last_change":"2024-03-30T15:05:17.276827+0000","last_active":"2024-03-30T15:06:27.340865+0000","last_peered":"2024-03-30T15:06:27.340865+0000","last_clean":"2024-03-30T15:06:27.340865+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:06:27.340865+0000","last_undegraded":"2024-03-30T15:06:27.340865+0000","last_fullsized":"2024-03-30T15:06:27.340865+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:17.276735+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.276735+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.270995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339097+0000","last_change":"2024-03-30T15:05:14.260362+0000","last_active":"2024-03-30T15:06:27.339097+0000","last_peered":"2024-03-30T15:06:27.339097+0000","last_clean":"2024-03-30T15:06:27.339097+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:06:27.339097+0000","last_undegraded":"2024-03-30T15:06:27.339097+0000","last_fullsized":"2024-03-30T15:06:27.339097+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.260283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.260283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1004,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.348339+0000","last_change":"2024-03-30T15:05:58.982494+0000","last_active":"2024-03-30T15:06:27.348339+0000","last_peered":"2024-03-30T15:06:27.348339+0000","last_clean":"2024-03-30T15:06:27.348339+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:06:27.348339+0000","last_undegraded":"2024-03-30T15:06:27.348339+0000","last_fullsized":"2024-03-30T15:06:27.348339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.982407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.982407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.069312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":921,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344450+0000","last_change":"2024-03-30T15:06:01.027884+0000","last_active":"2024-03-30T15:06:27.344450+0000","last_peered":"2024-03-30T15:06:27.344450+0000","last_clean":"2024-03-30T15:06:27.344450+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:06:27.344450+0000","last_undegraded":"2024-03-30T15:06:27.344450+0000","last_fullsized":"2024-03-30T15:06:27.344450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.027770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.027770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:06.480899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352154+0000","last_change":"2024-03-30T15:06:08.169797+0000","last_active":"2024-03-30T15:06:27.352154+0000","last_peered":"2024-03-30T15:06:27.352154+0000","last_clean":"2024-03-30T15:06:27.352154+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:06:27.352154+0000","last_undegraded":"2024-03-30T15:06:27.352154+0000","last_fullsized":"2024-03-30T15:06:27.352154+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.169669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.169669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:12.236873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1059,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352262+0000","last_change":"2024-03-30T15:05:26.068603+0000","last_active":"2024-03-30T15:06:27.352262+0000","last_peered":"2024-03-30T15:06:27.352262+0000","last_clean":"2024-03-30T15:06:27.352262+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:06:27.352262+0000","last_undegraded":"2024-03-30T15:06:27.352262+0000","last_fullsized":"2024-03-30T15:06:27.352262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.068493+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.425413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350102+0000","last_change":"2024-03-30T15:04:50.960049+0000","last_active":"2024-03-30T15:06:27.350102+0000","last_peered":"2024-03-30T15:06:27.350102+0000","last_clean":"2024-03-30T15:06:27.350102+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:06:27.350102+0000","last_undegraded":"2024-03-30T15:06:27.350102+0000","last_fullsized":"2024-03-30T15:06:27.350102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:50.959968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.959968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1155,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340930+0000","last_change":"2024-03-30T15:06:06.615931+0000","last_active":"2024-03-30T15:06:27.340930+0000","last_peered":"2024-03-30T15:06:27.340930+0000","last_clean":"2024-03-30T15:06:27.340930+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:06:27.340930+0000","last_undegraded":"2024-03-30T15:06:27.340930+0000","last_fullsized":"2024-03-30T15:06:27.340930+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:06.615780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.615780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.931187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":992,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347406+0000","last_change":"2024-03-30T15:04:04.318622+0000","last_active":"2024-03-30T15:06:27.347406+0000","last_peered":"2024-03-30T15:06:27.347406+0000","last_clean":"2024-03-30T15:06:27.347406+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:06:27.347406+0000","last_undegraded":"2024-03-30T15:06:27.347406+0000","last_fullsized":"2024-03-30T15:06:27.347406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.318541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.318541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":917,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347484+0000","last_change":"2024-03-30T15:05:15.068570+0000","last_active":"2024-03-30T15:06:28.347484+0000","last_peered":"2024-03-30T15:06:28.347484+0000","last_clean":"2024-03-30T15:06:28.347484+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:06:28.347484+0000","last_undegraded":"2024-03-30T15:06:28.347484+0000","last_fullsized":"2024-03-30T15:06:28.347484+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.068496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.068496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.114011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1004,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338762+0000","last_change":"2024-03-30T15:05:18.317376+0000","last_active":"2024-03-30T15:06:27.338762+0000","last_peered":"2024-03-30T15:06:27.338762+0000","last_clean":"2024-03-30T15:06:27.338762+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:06:27.338762+0000","last_undegraded":"2024-03-30T15:06:27.338762+0000","last_fullsized":"2024-03-30T15:06:27.338762+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.317302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:33.949240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1023,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360361+0000","last_change":"2024-03-30T15:05:22.145932+0000","last_active":"2024-03-30T15:06:27.360361+0000","last_peered":"2024-03-30T15:06:27.360361+0000","last_clean":"2024-03-30T15:06:27.360361+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:06:27.360361+0000","last_undegraded":"2024-03-30T15:06:27.360361+0000","last_fullsized":"2024-03-30T15:06:27.360361+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:05:22.145836+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.145836+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.469421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":988,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361151+0000","last_change":"2024-03-30T15:05:59.324853+0000","last_active":"2024-03-30T15:06:27.361151+0000","last_peered":"2024-03-30T15:06:27.361151+0000","last_clean":"2024-03-30T15:06:27.361151+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:06:27.361151+0000","last_undegraded":"2024-03-30T15:06:27.361151+0000","last_fullsized":"2024-03-30T15:06:27.361151+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:05:59.324757+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.324757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:05.068206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1000,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339139+0000","last_change":"2024-03-30T15:06:09.562654+0000","last_active":"2024-03-30T15:06:27.339139+0000","last_peered":"2024-03-30T15:06:27.339139+0000","last_clean":"2024-03-30T15:06:27.339139+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:06:27.339139+0000","last_undegraded":"2024-03-30T15:06:27.339139+0000","last_fullsized":"2024-03-30T15:06:27.339139+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.562567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.562567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.410427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":924,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346748+0000","last_change":"2024-03-30T15:04:48.979522+0000","last_active":"2024-03-30T15:06:28.346748+0000","last_peered":"2024-03-30T15:06:28.346748+0000","last_clean":"2024-03-30T15:06:28.346748+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:06:28.346748+0000","last_undegraded":"2024-03-30T15:06:28.346748+0000","last_fullsized":"2024-03-30T15:06:28.346748+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.979438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1042,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.349749+0000","last_change":"2024-03-30T15:06:11.179123+0000","last_active":"2024-03-30T15:06:27.349749+0000","last_peered":"2024-03-30T15:06:27.349749+0000","last_clean":"2024-03-30T15:06:27.349749+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:06:27.349749+0000","last_undegraded":"2024-03-30T15:06:27.349749+0000","last_fullsized":"2024-03-30T15:06:27.349749+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.179008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.179008+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:37.124302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1050,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339250+0000","last_change":"2024-03-30T15:04:07.440182+0000","last_active":"2024-03-30T15:06:27.339250+0000","last_peered":"2024-03-30T15:06:27.339250+0000","last_clean":"2024-03-30T15:06:27.339250+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:06:27.339250+0000","last_undegraded":"2024-03-30T15:06:27.339250+0000","last_fullsized":"2024-03-30T15:06:27.339250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'14","last_scrub_stamp":"2024-03-30T15:04:07.440071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.440071+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1011,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361465+0000","last_change":"2024-03-30T15:03:14.468245+0000","last_active":"2024-03-30T15:06:27.361465+0000","last_peered":"2024-03-30T15:06:27.361465+0000","last_clean":"2024-03-30T15:06:27.361465+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:06:27.361465+0000","last_undegraded":"2024-03-30T15:06:27.361465+0000","last_fullsized":"2024-03-30T15:06:27.361465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:03:14.468127+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.468127+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1002,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360930+0000","last_change":"2024-03-30T15:03:19.985599+0000","last_active":"2024-03-30T15:06:27.360930+0000","last_peered":"2024-03-30T15:06:27.360930+0000","last_clean":"2024-03-30T15:06:27.360930+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:06:27.360930+0000","last_undegraded":"2024-03-30T15:06:27.360930+0000","last_fullsized":"2024-03-30T15:06:27.360930+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.985516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.985516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":924,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347065+0000","last_change":"2024-03-30T15:04:47.018205+0000","last_active":"2024-03-30T15:06:28.347065+0000","last_peered":"2024-03-30T15:06:28.347065+0000","last_clean":"2024-03-30T15:06:28.347065+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:06:28.347065+0000","last_undegraded":"2024-03-30T15:06:28.347065+0000","last_fullsized":"2024-03-30T15:06:28.347065+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.018083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1168,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361274+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:06:27.361274+0000","last_peered":"2024-03-30T15:06:27.361274+0000","last_clean":"2024-03-30T15:06:27.361274+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:06:27.361274+0000","last_undegraded":"2024-03-30T15:06:27.361274+0000","last_fullsized":"2024-03-30T15:06:27.361274+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:36.047732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1052,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351438+0000","last_change":"2024-03-30T15:05:57.179353+0000","last_active":"2024-03-30T15:06:27.351438+0000","last_peered":"2024-03-30T15:06:27.351438+0000","last_clean":"2024-03-30T15:06:27.351438+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:06:27.351438+0000","last_undegraded":"2024-03-30T15:06:27.351438+0000","last_fullsized":"2024-03-30T15:06:27.351438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:05:57.179258+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:05:57.179258+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:58.467151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347659+0000","last_change":"2024-03-30T15:04:07.410272+0000","last_active":"2024-03-30T15:06:27.347659+0000","last_peered":"2024-03-30T15:06:27.347659+0000","last_clean":"2024-03-30T15:06:27.347659+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:06:27.347659+0000","last_undegraded":"2024-03-30T15:06:27.347659+0000","last_fullsized":"2024-03-30T15:06:27.347659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:07.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:04:07.410188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1005,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359033+0000","last_change":"2024-03-30T15:06:10.554067+0000","last_active":"2024-03-30T15:06:27.359033+0000","last_peered":"2024-03-30T15:06:27.359033+0000","last_clean":"2024-03-30T15:06:27.359033+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:06:27.359033+0000","last_undegraded":"2024-03-30T15:06:27.359033+0000","last_fullsized":"2024-03-30T15:06:27.359033+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.918377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":917,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347128+0000","last_change":"2024-03-30T15:04:52.952256+0000","last_active":"2024-03-30T15:06:28.347128+0000","last_peered":"2024-03-30T15:06:28.347128+0000","last_clean":"2024-03-30T15:06:28.347128+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:06:28.347128+0000","last_undegraded":"2024-03-30T15:06:28.347128+0000","last_fullsized":"2024-03-30T15:06:28.347128+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.952177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343927+0000","last_change":"2024-03-30T15:03:56.376911+0000","last_active":"2024-03-30T15:06:27.343927+0000","last_peered":"2024-03-30T15:06:27.343927+0000","last_clean":"2024-03-30T15:06:27.343927+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:06:27.343927+0000","last_undegraded":"2024-03-30T15:06:27.343927+0000","last_fullsized":"2024-03-30T15:06:27.343927+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:56.376812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.376812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1064,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361890+0000","last_change":"2024-03-30T15:04:05.567808+0000","last_active":"2024-03-30T15:06:27.361890+0000","last_peered":"2024-03-30T15:06:27.361890+0000","last_clean":"2024-03-30T15:06:27.361890+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:06:27.361890+0000","last_undegraded":"2024-03-30T15:06:27.361890+0000","last_fullsized":"2024-03-30T15:06:27.361890+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'27","last_scrub_stamp":"2024-03-30T15:04:05.567707+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:04:05.567707+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1021,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351747+0000","last_change":"2024-03-30T15:04:51.927488+0000","last_active":"2024-03-30T15:06:27.351747+0000","last_peered":"2024-03-30T15:06:27.351747+0000","last_clean":"2024-03-30T15:06:27.351747+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:06:27.351747+0000","last_undegraded":"2024-03-30T15:06:27.351747+0000","last_fullsized":"2024-03-30T15:06:27.351747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.927412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.927412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1009,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350552+0000","last_change":"2024-03-30T15:05:23.007617+0000","last_active":"2024-03-30T15:06:27.350552+0000","last_peered":"2024-03-30T15:06:27.350552+0000","last_clean":"2024-03-30T15:06:27.350552+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:06:27.350552+0000","last_undegraded":"2024-03-30T15:06:27.350552+0000","last_fullsized":"2024-03-30T15:06:27.350552+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:05:23.007527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:37.808237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361374+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:06:27.361374+0000","last_peered":"2024-03-30T15:06:27.361374+0000","last_clean":"2024-03-30T15:06:27.361374+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:06:27.361374+0000","last_undegraded":"2024-03-30T15:06:27.361374+0000","last_fullsized":"2024-03-30T15:06:27.361374+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.839469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1004,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.348341+0000","last_change":"2024-03-30T15:03:55.352398+0000","last_active":"2024-03-30T15:06:27.348341+0000","last_peered":"2024-03-30T15:06:27.348341+0000","last_clean":"2024-03-30T15:06:27.348341+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:06:27.348341+0000","last_undegraded":"2024-03-30T15:06:27.348341+0000","last_fullsized":"2024-03-30T15:06:27.348341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.352304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.352304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1049,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361267+0000","last_change":"2024-03-30T15:04:09.532624+0000","last_active":"2024-03-30T15:06:27.361267+0000","last_peered":"2024-03-30T15:06:27.361267+0000","last_clean":"2024-03-30T15:06:27.361267+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:06:27.361267+0000","last_undegraded":"2024-03-30T15:06:27.361267+0000","last_fullsized":"2024-03-30T15:06:27.361267+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:09.532532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.532532+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1730,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360015+0000","last_change":"2024-03-30T15:04:50.115333+0000","last_active":"2024-03-30T15:06:27.360015+0000","last_peered":"2024-03-30T15:06:27.360015+0000","last_clean":"2024-03-30T15:06:27.360015+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:06:27.360015+0000","last_undegraded":"2024-03-30T15:06:27.360015+0000","last_fullsized":"2024-03-30T15:06:27.360015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:04:50.115166+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":703,"num_read_kb":698,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1009,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359209+0000","last_change":"2024-03-30T15:03:19.430970+0000","last_active":"2024-03-30T15:06:27.359209+0000","last_peered":"2024-03-30T15:06:27.359209+0000","last_clean":"2024-03-30T15:06:27.359209+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:06:27.359209+0000","last_undegraded":"2024-03-30T15:06:27.359209+0000","last_fullsized":"2024-03-30T15:06:27.359209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:19.430883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.430883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1001,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351642+0000","last_change":"2024-03-30T15:04:09.068397+0000","last_active":"2024-03-30T15:06:27.351642+0000","last_peered":"2024-03-30T15:06:27.351642+0000","last_clean":"2024-03-30T15:06:27.351642+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:06:27.351642+0000","last_undegraded":"2024-03-30T15:06:27.351642+0000","last_fullsized":"2024-03-30T15:06:27.351642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:09.068310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:04:09.068310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":910,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360778+0000","last_change":"2024-03-30T15:04:37.123232+0000","last_active":"2024-03-30T15:06:27.360778+0000","last_peered":"2024-03-30T15:06:27.360778+0000","last_clean":"2024-03-30T15:06:27.360778+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:06:27.360778+0000","last_undegraded":"2024-03-30T15:06:27.360778+0000","last_fullsized":"2024-03-30T15:06:27.360778+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1006,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360352+0000","last_change":"2024-03-30T15:05:15.118857+0000","last_active":"2024-03-30T15:06:27.360352+0000","last_peered":"2024-03-30T15:06:27.360352+0000","last_clean":"2024-03-30T15:06:27.360352+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:06:27.360352+0000","last_undegraded":"2024-03-30T15:06:27.360352+0000","last_fullsized":"2024-03-30T15:06:27.360352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.118778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.118778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1207,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344905+0000","last_change":"2024-03-30T15:06:02.076986+0000","last_active":"2024-03-30T15:06:27.344905+0000","last_peered":"2024-03-30T15:06:27.344905+0000","last_clean":"2024-03-30T15:06:27.344905+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:06:27.344905+0000","last_undegraded":"2024-03-30T15:06:27.344905+0000","last_fullsized":"2024-03-30T15:06:27.344905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:02.076864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.076864+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.210052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":907,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346839+0000","last_change":"2024-03-30T15:04:06.381125+0000","last_active":"2024-03-30T15:06:27.346839+0000","last_peered":"2024-03-30T15:06:27.346839+0000","last_clean":"2024-03-30T15:06:27.346839+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:06:27.346839+0000","last_undegraded":"2024-03-30T15:06:27.346839+0000","last_fullsized":"2024-03-30T15:06:27.346839+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.381005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.381005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1276,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360141+0000","last_change":"2024-03-30T15:05:27.137562+0000","last_active":"2024-03-30T15:06:27.360141+0000","last_peered":"2024-03-30T15:06:27.360141+0000","last_clean":"2024-03-30T15:06:27.360141+0000","last_became_active":"2024-03-30T14:54:26.433540+0000","last_became_peered":"2024-03-30T14:54:26.433540+0000","last_unstale":"2024-03-30T15:06:27.360141+0000","last_undegraded":"2024-03-30T15:06:27.360141+0000","last_fullsized":"2024-03-30T15:06:27.360141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:05:27.137472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.137472+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:38.903904+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":992,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.348106+0000","last_change":"2024-03-30T15:03:59.448350+0000","last_active":"2024-03-30T15:06:27.348106+0000","last_peered":"2024-03-30T15:06:27.348106+0000","last_clean":"2024-03-30T15:06:27.348106+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:06:27.348106+0000","last_undegraded":"2024-03-30T15:06:27.348106+0000","last_fullsized":"2024-03-30T15:06:27.348106+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.448232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.448232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":925,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345217+0000","last_change":"2024-03-30T15:06:10.008324+0000","last_active":"2024-03-30T15:06:27.345217+0000","last_peered":"2024-03-30T15:06:27.345217+0000","last_clean":"2024-03-30T15:06:27.345217+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:06:27.345217+0000","last_undegraded":"2024-03-30T15:06:27.345217+0000","last_fullsized":"2024-03-30T15:06:27.345217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.008246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.008246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.193561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1016,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347948+0000","last_change":"2024-03-30T15:04:04.113202+0000","last_active":"2024-03-30T15:06:28.347948+0000","last_peered":"2024-03-30T15:06:28.347948+0000","last_clean":"2024-03-30T15:06:28.347948+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:06:28.347948+0000","last_undegraded":"2024-03-30T15:06:28.347948+0000","last_fullsized":"2024-03-30T15:06:28.347948+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:04.113109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:04.113109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1055,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352303+0000","last_change":"2024-03-30T15:03:24.925822+0000","last_active":"2024-03-30T15:06:27.352303+0000","last_peered":"2024-03-30T15:06:27.352303+0000","last_clean":"2024-03-30T15:06:27.352303+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:06:27.352303+0000","last_undegraded":"2024-03-30T15:06:27.352303+0000","last_fullsized":"2024-03-30T15:06:27.352303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'18","last_scrub_stamp":"2024-03-30T15:03:24.925732+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.925732+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1033,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346868+0000","last_change":"2024-03-30T15:04:55.918561+0000","last_active":"2024-03-30T15:06:28.346868+0000","last_peered":"2024-03-30T15:06:28.346868+0000","last_clean":"2024-03-30T15:06:28.346868+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:06:28.346868+0000","last_undegraded":"2024-03-30T15:06:28.346868+0000","last_fullsized":"2024-03-30T15:06:28.346868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:04:55.918411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1021,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.349860+0000","last_change":"2024-03-30T15:04:53.983173+0000","last_active":"2024-03-30T15:06:27.349860+0000","last_peered":"2024-03-30T15:06:27.349860+0000","last_clean":"2024-03-30T15:06:27.349860+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:06:27.349860+0000","last_undegraded":"2024-03-30T15:06:27.349860+0000","last_fullsized":"2024-03-30T15:06:27.349860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.983055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1002,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361240+0000","last_change":"2024-03-30T15:03:53.135430+0000","last_active":"2024-03-30T15:06:27.361240+0000","last_peered":"2024-03-30T15:06:27.361240+0000","last_clean":"2024-03-30T15:06:27.361240+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:06:27.361240+0000","last_undegraded":"2024-03-30T15:06:27.361240+0000","last_fullsized":"2024-03-30T15:06:27.361240+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:53.135360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.135360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360708+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:06:27.360708+0000","last_peered":"2024-03-30T15:06:27.360708+0000","last_clean":"2024-03-30T15:06:27.360708+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:06:27.360708+0000","last_undegraded":"2024-03-30T15:06:27.360708+0000","last_fullsized":"2024-03-30T15:06:27.360708+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.687947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1009,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360697+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:06:27.360697+0000","last_peered":"2024-03-30T15:06:27.360697+0000","last_clean":"2024-03-30T15:06:27.360697+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:06:27.360697+0000","last_undegraded":"2024-03-30T15:06:27.360697+0000","last_fullsized":"2024-03-30T15:06:27.360697+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1172,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346632+0000","last_change":"2024-03-30T15:06:05.010246+0000","last_active":"2024-03-30T15:06:27.346632+0000","last_peered":"2024-03-30T15:06:27.346632+0000","last_clean":"2024-03-30T15:06:27.346632+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:06:27.346632+0000","last_undegraded":"2024-03-30T15:06:27.346632+0000","last_fullsized":"2024-03-30T15:06:27.346632+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:06:05.010128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.010128+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.104222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339459+0000","last_change":"2024-03-30T15:04:53.505197+0000","last_active":"2024-03-30T15:06:27.339459+0000","last_peered":"2024-03-30T15:06:27.339459+0000","last_clean":"2024-03-30T15:06:27.339459+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:06:27.339459+0000","last_undegraded":"2024-03-30T15:06:27.339459+0000","last_fullsized":"2024-03-30T15:06:27.339459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.505107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.505107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1015,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346886+0000","last_change":"2024-03-30T15:03:24.211705+0000","last_active":"2024-03-30T15:06:27.346886+0000","last_peered":"2024-03-30T15:06:27.346886+0000","last_clean":"2024-03-30T15:06:27.346886+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:06:27.346886+0000","last_undegraded":"2024-03-30T15:06:27.346886+0000","last_fullsized":"2024-03-30T15:06:27.346886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:03:24.211619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:03:24.211619+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340233+0000","last_change":"2024-03-30T15:05:59.777166+0000","last_active":"2024-03-30T15:06:27.340233+0000","last_peered":"2024-03-30T15:06:27.340233+0000","last_clean":"2024-03-30T15:06:27.340233+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:06:27.340233+0000","last_undegraded":"2024-03-30T15:06:27.340233+0000","last_fullsized":"2024-03-30T15:06:27.340233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:59.776966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.776966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:01.928817+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":915,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361573+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:06:27.361573+0000","last_peered":"2024-03-30T15:06:27.361573+0000","last_clean":"2024-03-30T15:06:27.361573+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:06:27.361573+0000","last_undegraded":"2024-03-30T15:06:27.361573+0000","last_fullsized":"2024-03-30T15:06:27.361573+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.060229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1038,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361466+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:06:27.361466+0000","last_peered":"2024-03-30T15:06:27.361466+0000","last_clean":"2024-03-30T15:06:27.361466+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:06:27.361466+0000","last_undegraded":"2024-03-30T15:06:27.361466+0000","last_fullsized":"2024-03-30T15:06:27.361466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347039+0000","last_change":"2024-03-30T15:04:12.303761+0000","last_active":"2024-03-30T15:06:27.347039+0000","last_peered":"2024-03-30T15:06:27.347039+0000","last_clean":"2024-03-30T15:06:27.347039+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:06:27.347039+0000","last_undegraded":"2024-03-30T15:06:27.347039+0000","last_fullsized":"2024-03-30T15:06:27.347039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:12.303640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:04:12.303640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1024,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347160+0000","last_change":"2024-03-30T15:06:06.028818+0000","last_active":"2024-03-30T15:06:28.347160+0000","last_peered":"2024-03-30T15:06:28.347160+0000","last_clean":"2024-03-30T15:06:28.347160+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:06:28.347160+0000","last_undegraded":"2024-03-30T15:06:28.347160+0000","last_fullsized":"2024-03-30T15:06:28.347160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.028733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.028733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.118957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":997,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359556+0000","last_change":"2024-03-30T15:02:50.473533+0000","last_active":"2024-03-30T15:06:27.359556+0000","last_peered":"2024-03-30T15:06:27.359556+0000","last_clean":"2024-03-30T15:06:27.359556+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:06:27.359556+0000","last_undegraded":"2024-03-30T15:06:27.359556+0000","last_fullsized":"2024-03-30T15:06:27.359556+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:50.473456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:50.473456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339786+0000","last_change":"2024-03-30T15:04:46.590243+0000","last_active":"2024-03-30T15:06:27.339786+0000","last_peered":"2024-03-30T15:06:27.339786+0000","last_clean":"2024-03-30T15:06:27.339786+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:06:27.339786+0000","last_undegraded":"2024-03-30T15:06:27.339786+0000","last_fullsized":"2024-03-30T15:06:27.339786+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:46.590171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.590171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1097,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343826+0000","last_change":"2024-03-30T15:03:16.431130+0000","last_active":"2024-03-30T15:06:27.343826+0000","last_peered":"2024-03-30T15:06:27.343826+0000","last_clean":"2024-03-30T15:06:27.343826+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:06:27.343826+0000","last_undegraded":"2024-03-30T15:06:27.343826+0000","last_fullsized":"2024-03-30T15:06:27.343826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'19","last_scrub_stamp":"2024-03-30T15:03:16.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.430934+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1014,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361029+0000","last_change":"2024-03-30T15:06:00.363324+0000","last_active":"2024-03-30T15:06:27.361029+0000","last_peered":"2024-03-30T15:06:27.361029+0000","last_clean":"2024-03-30T15:06:27.361029+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:06:27.361029+0000","last_undegraded":"2024-03-30T15:06:27.361029+0000","last_fullsized":"2024-03-30T15:06:27.361029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.363175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.363175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:05.991392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1016,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347120+0000","last_change":"2024-03-30T15:03:59.164919+0000","last_active":"2024-03-30T15:06:28.347120+0000","last_peered":"2024-03-30T15:06:28.347120+0000","last_clean":"2024-03-30T15:06:28.347120+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:06:28.347120+0000","last_undegraded":"2024-03-30T15:06:28.347120+0000","last_fullsized":"2024-03-30T15:06:28.347120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:59.164826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.164826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1002,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359726+0000","last_change":"2024-03-30T15:05:29.180863+0000","last_active":"2024-03-30T15:06:27.359726+0000","last_peered":"2024-03-30T15:06:27.359726+0000","last_clean":"2024-03-30T15:06:27.359726+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:06:27.359726+0000","last_undegraded":"2024-03-30T15:06:27.359726+0000","last_fullsized":"2024-03-30T15:06:27.359726+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.180786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.180786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:34.154763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":931,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343320+0000","last_change":"2024-03-30T15:03:17.422472+0000","last_active":"2024-03-30T15:06:27.343320+0000","last_peered":"2024-03-30T15:06:27.343320+0000","last_clean":"2024-03-30T15:06:27.343320+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:06:27.343320+0000","last_undegraded":"2024-03-30T15:06:27.343320+0000","last_fullsized":"2024-03-30T15:06:27.343320+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:03:17.422345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:03:17.422345+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1016,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346692+0000","last_change":"2024-03-30T15:04:49.959125+0000","last_active":"2024-03-30T15:06:28.346692+0000","last_peered":"2024-03-30T15:06:28.346692+0000","last_clean":"2024-03-30T15:06:28.346692+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:06:28.346692+0000","last_undegraded":"2024-03-30T15:06:28.346692+0000","last_fullsized":"2024-03-30T15:06:28.346692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.959020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.959020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1048,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339375+0000","last_change":"2024-03-30T15:06:01.783446+0000","last_active":"2024-03-30T15:06:27.339375+0000","last_peered":"2024-03-30T15:06:27.339375+0000","last_clean":"2024-03-30T15:06:27.339375+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:06:27.339375+0000","last_undegraded":"2024-03-30T15:06:27.339375+0000","last_fullsized":"2024-03-30T15:06:27.339375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:06:01.783327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.783327+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:19.764475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1014,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360989+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:06:27.360989+0000","last_peered":"2024-03-30T15:06:27.360989+0000","last_clean":"2024-03-30T15:06:27.360989+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:06:27.360989+0000","last_undegraded":"2024-03-30T15:06:27.360989+0000","last_fullsized":"2024-03-30T15:06:27.360989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":906,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361046+0000","last_change":"2024-03-30T15:03:52.168282+0000","last_active":"2024-03-30T15:06:27.361046+0000","last_peered":"2024-03-30T15:06:27.361046+0000","last_clean":"2024-03-30T15:06:27.361046+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:06:27.361046+0000","last_undegraded":"2024-03-30T15:06:27.361046+0000","last_fullsized":"2024-03-30T15:06:27.361046+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:52.168194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.168194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338462+0000","last_change":"2024-03-30T15:04:37.589562+0000","last_active":"2024-03-30T15:06:27.338462+0000","last_peered":"2024-03-30T15:06:27.338462+0000","last_clean":"2024-03-30T15:06:27.338462+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:06:27.338462+0000","last_undegraded":"2024-03-30T15:06:27.338462+0000","last_fullsized":"2024-03-30T15:06:27.338462+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.589464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.589464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1157,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360239+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:06:27.360239+0000","last_peered":"2024-03-30T15:06:27.360239+0000","last_clean":"2024-03-30T15:06:27.360239+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:06:27.360239+0000","last_undegraded":"2024-03-30T15:06:27.360239+0000","last_fullsized":"2024-03-30T15:06:27.360239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:23.089310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1051,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345967+0000","last_change":"2024-03-30T15:06:16.030806+0000","last_active":"2024-03-30T15:06:27.345967+0000","last_peered":"2024-03-30T15:06:27.345967+0000","last_clean":"2024-03-30T15:06:27.345967+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:06:27.345967+0000","last_undegraded":"2024-03-30T15:06:27.345967+0000","last_fullsized":"2024-03-30T15:06:27.345967+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:06:16.030694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.030694+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.402151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1026,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361316+0000","last_change":"2024-03-30T15:04:44.117459+0000","last_active":"2024-03-30T15:06:27.361316+0000","last_peered":"2024-03-30T15:06:27.361316+0000","last_clean":"2024-03-30T15:06:27.361316+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:06:27.361316+0000","last_undegraded":"2024-03-30T15:06:27.361316+0000","last_fullsized":"2024-03-30T15:06:27.361316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:04:44.117358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.117358+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1000,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347140+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:06:27.347140+0000","last_peered":"2024-03-30T15:06:27.347140+0000","last_clean":"2024-03-30T15:06:27.347140+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:06:27.347140+0000","last_undegraded":"2024-03-30T15:06:27.347140+0000","last_fullsized":"2024-03-30T15:06:27.347140+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.272716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345110+0000","last_change":"2024-03-30T15:04:49.244698+0000","last_active":"2024-03-30T15:06:27.345110+0000","last_peered":"2024-03-30T15:06:27.345110+0000","last_clean":"2024-03-30T15:06:27.345110+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:06:27.345110+0000","last_undegraded":"2024-03-30T15:06:27.345110+0000","last_fullsized":"2024-03-30T15:06:27.345110+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.244616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.244616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":603,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.351251+0000","last_change":"2024-03-30T15:05:13.602996+0000","last_active":"2024-03-30T15:06:28.351251+0000","last_peered":"2024-03-30T15:06:28.351251+0000","last_clean":"2024-03-30T15:06:28.351251+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:06:28.351251+0000","last_undegraded":"2024-03-30T15:06:28.351251+0000","last_fullsized":"2024-03-30T15:06:28.351251+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:05:13.602902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:13.602902+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.234443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1028,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.343046+0000","last_change":"2024-03-30T15:05:22.034492+0000","last_active":"2024-03-30T15:06:27.343046+0000","last_peered":"2024-03-30T15:06:27.343046+0000","last_clean":"2024-03-30T15:06:27.343046+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:06:27.343046+0000","last_undegraded":"2024-03-30T15:06:27.343046+0000","last_fullsized":"2024-03-30T15:06:27.343046+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:22.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.034372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:31.488283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":995,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347613+0000","last_change":"2024-03-30T15:03:14.211264+0000","last_active":"2024-03-30T15:06:27.347613+0000","last_peered":"2024-03-30T15:06:27.347613+0000","last_clean":"2024-03-30T15:06:27.347613+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:06:27.347613+0000","last_undegraded":"2024-03-30T15:06:27.347613+0000","last_fullsized":"2024-03-30T15:06:27.347613+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:14.211146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:14.211146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":993,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360390+0000","last_change":"2024-03-30T15:03:55.119587+0000","last_active":"2024-03-30T15:06:27.360390+0000","last_peered":"2024-03-30T15:06:27.360390+0000","last_clean":"2024-03-30T15:06:27.360390+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:06:27.360390+0000","last_undegraded":"2024-03-30T15:06:27.360390+0000","last_fullsized":"2024-03-30T15:06:27.360390+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:55.119470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:03:55.119470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340119+0000","last_change":"2024-03-30T15:04:44.618218+0000","last_active":"2024-03-30T15:06:27.340119+0000","last_peered":"2024-03-30T15:06:27.340119+0000","last_clean":"2024-03-30T15:06:27.340119+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:06:27.340119+0000","last_undegraded":"2024-03-30T15:06:27.340119+0000","last_fullsized":"2024-03-30T15:06:27.340119+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:44.618135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:44.618135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1029,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361820+0000","last_change":"2024-03-30T15:03:56.575602+0000","last_active":"2024-03-30T15:06:27.361820+0000","last_peered":"2024-03-30T15:06:27.361820+0000","last_clean":"2024-03-30T15:06:27.361820+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:06:27.361820+0000","last_undegraded":"2024-03-30T15:06:27.361820+0000","last_fullsized":"2024-03-30T15:06:27.361820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:56.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.575506+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1014,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359733+0000","last_change":"2024-03-30T15:04:02.050735+0000","last_active":"2024-03-30T15:06:27.359733+0000","last_peered":"2024-03-30T15:06:27.359733+0000","last_clean":"2024-03-30T15:06:27.359733+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:06:27.359733+0000","last_undegraded":"2024-03-30T15:06:27.359733+0000","last_fullsized":"2024-03-30T15:06:27.359733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:02.050599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:02.050599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":100,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344813+0000","last_change":"2024-03-30T15:05:42.322974+0000","last_active":"2024-03-30T15:06:27.344813+0000","last_peered":"2024-03-30T15:06:27.344813+0000","last_clean":"2024-03-30T15:06:27.344813+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:06:27.344813+0000","last_undegraded":"2024-03-30T15:06:27.344813+0000","last_fullsized":"2024-03-30T15:06:27.344813+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:03.987956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.987956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1020,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.348234+0000","last_change":"2024-03-30T15:06:14.925991+0000","last_active":"2024-03-30T15:06:28.348234+0000","last_peered":"2024-03-30T15:06:28.348234+0000","last_clean":"2024-03-30T15:06:28.348234+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:06:28.348234+0000","last_undegraded":"2024-03-30T15:06:28.348234+0000","last_fullsized":"2024-03-30T15:06:28.348234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.925904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.925904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:28.092815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360615+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:06:27.360615+0000","last_peered":"2024-03-30T15:06:27.360615+0000","last_clean":"2024-03-30T15:06:27.360615+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:06:27.360615+0000","last_undegraded":"2024-03-30T15:06:27.360615+0000","last_fullsized":"2024-03-30T15:06:27.360615+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.612170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1024,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347759+0000","last_change":"2024-03-30T15:03:19.177749+0000","last_active":"2024-03-30T15:06:27.347759+0000","last_peered":"2024-03-30T15:06:27.347759+0000","last_clean":"2024-03-30T15:06:27.347759+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:06:27.347759+0000","last_undegraded":"2024-03-30T15:06:27.347759+0000","last_fullsized":"2024-03-30T15:06:27.347759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:03:19.177603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:03:19.177603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1017,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359986+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:06:27.359986+0000","last_peered":"2024-03-30T15:06:27.359986+0000","last_clean":"2024-03-30T15:06:27.359986+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:06:27.359986+0000","last_undegraded":"2024-03-30T15:06:27.359986+0000","last_fullsized":"2024-03-30T15:06:27.359986+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1042,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360887+0000","last_change":"2024-03-30T15:06:11.222500+0000","last_active":"2024-03-30T15:06:27.360887+0000","last_peered":"2024-03-30T15:06:27.360887+0000","last_clean":"2024-03-30T15:06:27.360887+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:06:27.360887+0000","last_undegraded":"2024-03-30T15:06:27.360887+0000","last_fullsized":"2024-03-30T15:06:27.360887+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.222382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.222382+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.574504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":991,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.341015+0000","last_change":"2024-03-30T15:04:35.553690+0000","last_active":"2024-03-30T15:06:27.341015+0000","last_peered":"2024-03-30T15:06:27.341015+0000","last_clean":"2024-03-30T15:06:27.341015+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:06:27.341015+0000","last_undegraded":"2024-03-30T15:06:27.341015+0000","last_fullsized":"2024-03-30T15:06:27.341015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.553605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.553605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360673+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:06:27.360673+0000","last_peered":"2024-03-30T15:06:27.360673+0000","last_clean":"2024-03-30T15:06:27.360673+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:06:27.360673+0000","last_undegraded":"2024-03-30T15:06:27.360673+0000","last_fullsized":"2024-03-30T15:06:27.360673+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:47.969152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":962,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351808+0000","last_change":"2024-03-30T15:04:38.199333+0000","last_active":"2024-03-30T15:06:27.351808+0000","last_peered":"2024-03-30T15:06:27.351808+0000","last_clean":"2024-03-30T15:06:27.351808+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:06:27.351808+0000","last_undegraded":"2024-03-30T15:06:27.351808+0000","last_fullsized":"2024-03-30T15:06:27.351808+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'10","last_scrub_stamp":"2024-03-30T15:04:38.199239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:38.199239+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1014,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360822+0000","last_change":"2024-03-30T15:06:14.188494+0000","last_active":"2024-03-30T15:06:27.360822+0000","last_peered":"2024-03-30T15:06:27.360822+0000","last_clean":"2024-03-30T15:06:27.360822+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:06:27.360822+0000","last_undegraded":"2024-03-30T15:06:27.360822+0000","last_fullsized":"2024-03-30T15:06:27.360822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.188400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.188400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:14.495631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1149,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347591+0000","last_change":"2024-03-30T15:03:53.401495+0000","last_active":"2024-03-30T15:06:27.347591+0000","last_peered":"2024-03-30T15:06:27.347591+0000","last_clean":"2024-03-30T15:06:27.347591+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:06:27.347591+0000","last_undegraded":"2024-03-30T15:06:27.347591+0000","last_fullsized":"2024-03-30T15:06:27.347591+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:03:53.401392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:53.401392+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":998,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361159+0000","last_change":"2024-03-30T15:03:52.512735+0000","last_active":"2024-03-30T15:06:27.361159+0000","last_peered":"2024-03-30T15:06:27.361159+0000","last_clean":"2024-03-30T15:06:27.361159+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:06:27.361159+0000","last_undegraded":"2024-03-30T15:06:27.361159+0000","last_fullsized":"2024-03-30T15:06:27.361159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:03:52.512650+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:03:52.512650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":908,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340210+0000","last_change":"2024-03-30T15:04:41.620132+0000","last_active":"2024-03-30T15:06:27.340210+0000","last_peered":"2024-03-30T15:06:27.340210+0000","last_clean":"2024-03-30T15:06:27.340210+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:06:27.340210+0000","last_undegraded":"2024-03-30T15:06:27.340210+0000","last_fullsized":"2024-03-30T15:06:27.340210+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:41.620027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:41.620027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1036,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340026+0000","last_change":"2024-03-30T15:03:56.461129+0000","last_active":"2024-03-30T15:06:27.340026+0000","last_peered":"2024-03-30T15:06:27.340026+0000","last_clean":"2024-03-30T15:06:27.340026+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:06:27.340026+0000","last_undegraded":"2024-03-30T15:06:27.340026+0000","last_fullsized":"2024-03-30T15:06:27.340026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2024-03-30T15:03:56.461021+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:03:56.461021+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339338+0000","last_change":"2024-03-30T15:04:43.577081+0000","last_active":"2024-03-30T15:06:27.339338+0000","last_peered":"2024-03-30T15:06:27.339338+0000","last_clean":"2024-03-30T15:06:27.339338+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:06:27.339338+0000","last_undegraded":"2024-03-30T15:06:27.339338+0000","last_fullsized":"2024-03-30T15:06:27.339338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.576993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.576993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1013,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351315+0000","last_change":"2024-03-30T15:05:19.043801+0000","last_active":"2024-03-30T15:06:27.351315+0000","last_peered":"2024-03-30T15:06:27.351315+0000","last_clean":"2024-03-30T15:06:27.351315+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:06:27.351315+0000","last_undegraded":"2024-03-30T15:06:27.351315+0000","last_fullsized":"2024-03-30T15:06:27.351315+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.043661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.043661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.583679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359307+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:06:27.359307+0000","last_peered":"2024-03-30T15:06:27.359307+0000","last_clean":"2024-03-30T15:06:27.359307+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:06:27.359307+0000","last_undegraded":"2024-03-30T15:06:27.359307+0000","last_fullsized":"2024-03-30T15:06:27.359307+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":908,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338745+0000","last_change":"2024-03-30T15:04:40.601679+0000","last_active":"2024-03-30T15:06:27.338745+0000","last_peered":"2024-03-30T15:06:27.338745+0000","last_clean":"2024-03-30T15:06:27.338745+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:06:27.338745+0000","last_undegraded":"2024-03-30T15:06:27.338745+0000","last_fullsized":"2024-03-30T15:06:27.338745+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:40.601541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.601541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1034,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351548+0000","last_change":"2024-03-30T15:04:54.952409+0000","last_active":"2024-03-30T15:06:27.351548+0000","last_peered":"2024-03-30T15:06:27.351548+0000","last_clean":"2024-03-30T15:06:27.351548+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:06:27.351548+0000","last_undegraded":"2024-03-30T15:06:27.351548+0000","last_fullsized":"2024-03-30T15:06:27.351548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:54.952263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:54.952263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1028,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360654+0000","last_change":"2024-03-30T15:04:52.183278+0000","last_active":"2024-03-30T15:06:27.360654+0000","last_peered":"2024-03-30T15:06:27.360654+0000","last_clean":"2024-03-30T15:06:27.360654+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:06:27.360654+0000","last_undegraded":"2024-03-30T15:06:27.360654+0000","last_fullsized":"2024-03-30T15:06:27.360654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:04:52.183194+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T14:56:03.093218+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.183194+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":178,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338773+0000","last_change":"2024-03-30T15:06:15.461799+0000","last_active":"2024-03-30T15:06:27.338773+0000","last_peered":"2024-03-30T15:06:27.338773+0000","last_clean":"2024-03-30T15:06:27.338773+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:06:27.338773+0000","last_undegraded":"2024-03-30T15:06:27.338773+0000","last_fullsized":"2024-03-30T15:06:27.338773+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:15.461699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:15.461699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:24.206642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":997,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359031+0000","last_change":"2024-03-30T15:02:47.485250+0000","last_active":"2024-03-30T15:06:27.359031+0000","last_peered":"2024-03-30T15:06:27.359031+0000","last_clean":"2024-03-30T15:06:27.359031+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:06:27.359031+0000","last_undegraded":"2024-03-30T15:06:27.359031+0000","last_fullsized":"2024-03-30T15:06:27.359031+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:02:47.485158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:02:47.485158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359031+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:06:27.359031+0000","last_peered":"2024-03-30T15:06:27.359031+0000","last_clean":"2024-03-30T15:06:27.359031+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:06:27.359031+0000","last_undegraded":"2024-03-30T15:06:27.359031+0000","last_fullsized":"2024-03-30T15:06:27.359031+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":964,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360031+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:06:27.360031+0000","last_peered":"2024-03-30T15:06:27.360031+0000","last_clean":"2024-03-30T15:06:27.360031+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:06:27.360031+0000","last_undegraded":"2024-03-30T15:06:27.360031+0000","last_fullsized":"2024-03-30T15:06:27.360031+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.309540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1011,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347706+0000","last_change":"2024-03-30T15:06:00.028025+0000","last_active":"2024-03-30T15:06:27.347706+0000","last_peered":"2024-03-30T15:06:27.347706+0000","last_clean":"2024-03-30T15:06:27.347706+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:06:27.347706+0000","last_undegraded":"2024-03-30T15:06:27.347706+0000","last_fullsized":"2024-03-30T15:06:27.347706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.027914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.027914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:22.091022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1006,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360609+0000","last_change":"2024-03-30T15:05:21.130956+0000","last_active":"2024-03-30T15:06:27.360609+0000","last_peered":"2024-03-30T15:06:27.360609+0000","last_clean":"2024-03-30T15:06:27.360609+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:06:27.360609+0000","last_undegraded":"2024-03-30T15:06:27.360609+0000","last_fullsized":"2024-03-30T15:06:27.360609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:21.130862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.130862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.084983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":917,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361007+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:06:27.361007+0000","last_peered":"2024-03-30T15:06:27.361007+0000","last_clean":"2024-03-30T15:06:27.361007+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:06:27.361007+0000","last_undegraded":"2024-03-30T15:06:27.361007+0000","last_fullsized":"2024-03-30T15:06:27.361007+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:00.791475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350013+0000","last_change":"2024-03-30T15:06:16.046921+0000","last_active":"2024-03-30T15:06:27.350013+0000","last_peered":"2024-03-30T15:06:27.350013+0000","last_clean":"2024-03-30T15:06:27.350013+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:06:27.350013+0000","last_undegraded":"2024-03-30T15:06:27.350013+0000","last_fullsized":"2024-03-30T15:06:27.350013+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:16.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.046809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.167911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":590,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.350068+0000","last_change":"2024-03-30T15:05:59.397565+0000","last_active":"2024-03-30T15:06:28.350068+0000","last_peered":"2024-03-30T15:06:28.350068+0000","last_clean":"2024-03-30T15:06:28.350068+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:06:28.350068+0000","last_undegraded":"2024-03-30T15:06:28.350068+0000","last_fullsized":"2024-03-30T15:06:28.350068+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:05:59.397484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.397484+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:13.734849+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1016,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.362084+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:06:27.362084+0000","last_peered":"2024-03-30T15:06:27.362084+0000","last_clean":"2024-03-30T15:06:27.362084+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:06:27.362084+0000","last_undegraded":"2024-03-30T15:06:27.362084+0000","last_fullsized":"2024-03-30T15:06:27.362084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1020,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344064+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:06:27.344064+0000","last_peered":"2024-03-30T15:06:27.344064+0000","last_clean":"2024-03-30T15:06:27.344064+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:06:27.344064+0000","last_undegraded":"2024-03-30T15:06:27.344064+0000","last_fullsized":"2024-03-30T15:06:27.344064+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:32.616611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1002,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359789+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:06:27.359789+0000","last_peered":"2024-03-30T15:06:27.359789+0000","last_clean":"2024-03-30T15:06:27.359789+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:06:27.359789+0000","last_undegraded":"2024-03-30T15:06:27.359789+0000","last_fullsized":"2024-03-30T15:06:27.359789+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:16.275230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":917,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345378+0000","last_change":"2024-03-30T15:04:51.176640+0000","last_active":"2024-03-30T15:06:27.345378+0000","last_peered":"2024-03-30T15:06:27.345378+0000","last_clean":"2024-03-30T15:06:27.345378+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:06:27.345378+0000","last_undegraded":"2024-03-30T15:06:27.345378+0000","last_fullsized":"2024-03-30T15:06:27.345378+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:51.176549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:51.176549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1052,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345382+0000","last_change":"2024-03-30T15:05:59.994488+0000","last_active":"2024-03-30T15:06:27.345382+0000","last_peered":"2024-03-30T15:06:27.345382+0000","last_clean":"2024-03-30T15:06:27.345382+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:06:27.345382+0000","last_undegraded":"2024-03-30T15:06:27.345382+0000","last_fullsized":"2024-03-30T15:06:27.345382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:59.994304+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.994304+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.366647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":910,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359215+0000","last_change":"2024-03-30T15:03:50.441279+0000","last_active":"2024-03-30T15:06:27.359215+0000","last_peered":"2024-03-30T15:06:27.359215+0000","last_clean":"2024-03-30T15:06:27.359215+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:06:27.359215+0000","last_undegraded":"2024-03-30T15:06:27.359215+0000","last_fullsized":"2024-03-30T15:06:27.359215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:50.441206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:03:50.441206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1001,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361893+0000","last_change":"2024-03-30T15:03:16.443068+0000","last_active":"2024-03-30T15:06:27.361893+0000","last_peered":"2024-03-30T15:06:27.361893+0000","last_clean":"2024-03-30T15:06:27.361893+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:06:27.361893+0000","last_undegraded":"2024-03-30T15:06:27.361893+0000","last_fullsized":"2024-03-30T15:06:27.361893+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:16.442946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:16.442946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1008,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352312+0000","last_change":"2024-03-30T15:06:01.137816+0000","last_active":"2024-03-30T15:06:27.352312+0000","last_peered":"2024-03-30T15:06:27.352312+0000","last_clean":"2024-03-30T15:06:27.352312+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:06:27.352312+0000","last_undegraded":"2024-03-30T15:06:27.352312+0000","last_fullsized":"2024-03-30T15:06:27.352312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.137663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.137663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:09.649632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":925,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345112+0000","last_change":"2024-03-30T15:06:14.085407+0000","last_active":"2024-03-30T15:06:27.345112+0000","last_peered":"2024-03-30T15:06:27.345112+0000","last_clean":"2024-03-30T15:06:27.345112+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:06:27.345112+0000","last_undegraded":"2024-03-30T15:06:27.345112+0000","last_fullsized":"2024-03-30T15:06:27.345112+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.085299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.857928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1040,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361560+0000","last_change":"2024-03-30T15:03:25.422866+0000","last_active":"2024-03-30T15:06:27.361560+0000","last_peered":"2024-03-30T15:06:27.361560+0000","last_clean":"2024-03-30T15:06:27.361560+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:06:27.361560+0000","last_undegraded":"2024-03-30T15:06:27.361560+0000","last_fullsized":"2024-03-30T15:06:27.361560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'15","last_scrub_stamp":"2024-03-30T15:03:25.422669+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:03:25.422669+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"69'1","reported_seq":1013,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350703+0000","last_change":"2024-03-30T15:05:24.043308+0000","last_active":"2024-03-30T15:06:27.350703+0000","last_peered":"2024-03-30T15:06:27.350703+0000","last_clean":"2024-03-30T15:06:27.350703+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:06:27.350703+0000","last_undegraded":"2024-03-30T15:06:27.350703+0000","last_fullsized":"2024-03-30T15:06:27.350703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'1","last_scrub_stamp":"2024-03-30T15:05:24.043163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:05:24.043163+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.017816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1174,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345314+0000","last_change":"2024-03-30T15:05:20.066039+0000","last_active":"2024-03-30T15:06:27.345314+0000","last_peered":"2024-03-30T15:06:27.345314+0000","last_clean":"2024-03-30T15:06:27.345314+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:06:27.345314+0000","last_undegraded":"2024-03-30T15:06:27.345314+0000","last_fullsized":"2024-03-30T15:06:27.345314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:20.065886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.065886+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:42.126952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1003,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.360198+0000","last_change":"2024-03-30T15:03:59.026312+0000","last_active":"2024-03-30T15:06:27.360198+0000","last_peered":"2024-03-30T15:06:27.360198+0000","last_clean":"2024-03-30T15:06:27.360198+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:06:27.360198+0000","last_undegraded":"2024-03-30T15:06:27.360198+0000","last_fullsized":"2024-03-30T15:06:27.360198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:03:59.026203+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:03:59.026203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":917,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344601+0000","last_change":"2024-03-30T15:04:10.351248+0000","last_active":"2024-03-30T15:06:27.344601+0000","last_peered":"2024-03-30T15:06:27.344601+0000","last_clean":"2024-03-30T15:06:27.344601+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:06:27.344601+0000","last_undegraded":"2024-03-30T15:06:27.344601+0000","last_fullsized":"2024-03-30T15:06:27.344601+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:10.351151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:10.351151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.201376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":964,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352020+0000","last_change":"2024-03-30T15:04:40.214721+0000","last_active":"2024-03-30T15:06:27.352020+0000","last_peered":"2024-03-30T15:06:27.352020+0000","last_clean":"2024-03-30T15:06:27.352020+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:06:27.352020+0000","last_undegraded":"2024-03-30T15:06:27.352020+0000","last_fullsized":"2024-03-30T15:06:27.352020+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'12","last_scrub_stamp":"2024-03-30T15:04:40.214633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:40.214633+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1011,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339556+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:06:27.339556+0000","last_peered":"2024-03-30T15:06:27.339556+0000","last_clean":"2024-03-30T15:06:27.339556+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:06:27.339556+0000","last_undegraded":"2024-03-30T15:06:27.339556+0000","last_fullsized":"2024-03-30T15:06:27.339556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:29.176862+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1003,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339610+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:06:27.339610+0000","last_peered":"2024-03-30T15:06:27.339610+0000","last_clean":"2024-03-30T15:06:27.339610+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:06:27.339610+0000","last_undegraded":"2024-03-30T15:06:27.339610+0000","last_fullsized":"2024-03-30T15:06:27.339610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:31.497561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":922,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.345485+0000","last_change":"2024-03-30T15:05:15.145381+0000","last_active":"2024-03-30T15:06:27.345485+0000","last_peered":"2024-03-30T15:06:27.345485+0000","last_clean":"2024-03-30T15:06:27.345485+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:06:27.345485+0000","last_undegraded":"2024-03-30T15:06:27.345485+0000","last_fullsized":"2024-03-30T15:06:27.345485+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.145314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.145314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:27.542525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347234+0000","last_change":"2024-03-30T15:04:36.274480+0000","last_active":"2024-03-30T15:06:27.347234+0000","last_peered":"2024-03-30T15:06:27.347234+0000","last_clean":"2024-03-30T15:06:27.347234+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:06:27.347234+0000","last_undegraded":"2024-03-30T15:06:27.347234+0000","last_fullsized":"2024-03-30T15:06:27.347234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.274391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.274391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1077,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.347647+0000","last_change":"2024-03-30T15:06:13.899767+0000","last_active":"2024-03-30T15:06:28.347647+0000","last_peered":"2024-03-30T15:06:28.347647+0000","last_clean":"2024-03-30T15:06:28.347647+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:06:28.347647+0000","last_undegraded":"2024-03-30T15:06:28.347647+0000","last_fullsized":"2024-03-30T15:06:28.347647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:06:13.899670+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:06:13.899670+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:15.962443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1016,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361785+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:06:27.361785+0000","last_peered":"2024-03-30T15:06:27.361785+0000","last_clean":"2024-03-30T15:06:27.361785+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:06:27.361785+0000","last_undegraded":"2024-03-30T15:06:27.361785+0000","last_fullsized":"2024-03-30T15:06:27.361785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.352222+0000","last_change":"2024-03-30T15:05:15.051014+0000","last_active":"2024-03-30T15:06:27.352222+0000","last_peered":"2024-03-30T15:06:27.352222+0000","last_clean":"2024-03-30T15:06:27.352222+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:06:27.352222+0000","last_undegraded":"2024-03-30T15:06:27.352222+0000","last_fullsized":"2024-03-30T15:06:27.352222+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.050918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.050918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:44.102896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1000,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.340719+0000","last_change":"2024-03-30T15:06:12.558953+0000","last_active":"2024-03-30T15:06:27.340719+0000","last_peered":"2024-03-30T15:06:27.340719+0000","last_clean":"2024-03-30T15:06:27.340719+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:06:27.340719+0000","last_undegraded":"2024-03-30T15:06:27.340719+0000","last_fullsized":"2024-03-30T15:06:27.340719+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.558834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.558834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:26.552401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":918,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351126+0000","last_change":"2024-03-30T15:06:04.136327+0000","last_active":"2024-03-30T15:06:27.351126+0000","last_peered":"2024-03-30T15:06:27.351126+0000","last_clean":"2024-03-30T15:06:27.351126+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:06:27.351126+0000","last_undegraded":"2024-03-30T15:06:27.351126+0000","last_fullsized":"2024-03-30T15:06:27.351126+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.136246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.136246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:29.092415+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1036,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339703+0000","last_change":"2024-03-30T15:04:03.516462+0000","last_active":"2024-03-30T15:06:27.339703+0000","last_peered":"2024-03-30T15:06:27.339703+0000","last_clean":"2024-03-30T15:06:27.339703+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:06:27.339703+0000","last_undegraded":"2024-03-30T15:06:27.339703+0000","last_fullsized":"2024-03-30T15:06:27.339703+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-30T15:04:03.516368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:03.516368+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1011,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347096+0000","last_change":"2024-03-30T15:06:14.990566+0000","last_active":"2024-03-30T15:06:27.347096+0000","last_peered":"2024-03-30T15:06:27.347096+0000","last_clean":"2024-03-30T15:06:27.347096+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:06:27.347096+0000","last_undegraded":"2024-03-30T15:06:27.347096+0000","last_fullsized":"2024-03-30T15:06:27.347096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.990472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.990472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:36.729092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1005,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359738+0000","last_change":"2024-03-30T15:05:58.374684+0000","last_active":"2024-03-30T15:06:27.359738+0000","last_peered":"2024-03-30T15:06:27.359738+0000","last_clean":"2024-03-30T15:06:27.359738+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:06:27.359738+0000","last_undegraded":"2024-03-30T15:06:27.359738+0000","last_fullsized":"2024-03-30T15:06:27.359738+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.374590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.374590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:14.674868+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":999,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346294+0000","last_change":"2024-03-30T15:04:35.253973+0000","last_active":"2024-03-30T15:06:27.346294+0000","last_peered":"2024-03-30T15:06:27.346294+0000","last_clean":"2024-03-30T15:06:27.346294+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:06:27.346294+0000","last_undegraded":"2024-03-30T15:06:27.346294+0000","last_fullsized":"2024-03-30T15:06:27.346294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:35.253856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:04:35.253856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":912,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.351464+0000","last_change":"2024-03-30T15:06:14.104026+0000","last_active":"2024-03-30T15:06:27.351464+0000","last_peered":"2024-03-30T15:06:27.351464+0000","last_clean":"2024-03-30T15:06:27.351464+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:06:27.351464+0000","last_undegraded":"2024-03-30T15:06:27.351464+0000","last_fullsized":"2024-03-30T15:06:27.351464+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.103910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.103910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:33.547133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1024,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339405+0000","last_change":"2024-03-30T15:04:06.474191+0000","last_active":"2024-03-30T15:06:27.339405+0000","last_peered":"2024-03-30T15:06:27.339405+0000","last_clean":"2024-03-30T15:06:27.339405+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:06:27.339405+0000","last_undegraded":"2024-03-30T15:06:27.339405+0000","last_fullsized":"2024-03-30T15:06:27.339405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:06.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.474049+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1003,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.347548+0000","last_change":"2024-03-30T15:03:22.212803+0000","last_active":"2024-03-30T15:06:27.347548+0000","last_peered":"2024-03-30T15:06:27.347548+0000","last_clean":"2024-03-30T15:06:27.347548+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:06:27.347548+0000","last_undegraded":"2024-03-30T15:06:27.347548+0000","last_fullsized":"2024-03-30T15:06:27.347548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:03:22.212713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1003,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346587+0000","last_change":"2024-03-30T15:05:26.176781+0000","last_active":"2024-03-30T15:06:27.346587+0000","last_peered":"2024-03-30T15:06:27.346587+0000","last_clean":"2024-03-30T15:06:27.346587+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:06:27.346587+0000","last_undegraded":"2024-03-30T15:06:27.346587+0000","last_fullsized":"2024-03-30T15:06:27.346587+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:26.176703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.176703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:49.697924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1004,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359303+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:06:27.359303+0000","last_peered":"2024-03-30T15:06:27.359303+0000","last_clean":"2024-03-30T15:06:27.359303+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:06:27.359303+0000","last_undegraded":"2024-03-30T15:06:27.359303+0000","last_fullsized":"2024-03-30T15:06:27.359303+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":909,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.346107+0000","last_change":"2024-03-30T15:04:52.290105+0000","last_active":"2024-03-30T15:06:27.346107+0000","last_peered":"2024-03-30T15:06:27.346107+0000","last_clean":"2024-03-30T15:06:27.346107+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:06:27.346107+0000","last_undegraded":"2024-03-30T15:06:27.346107+0000","last_fullsized":"2024-03-30T15:06:27.346107+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:52.289998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:04:52.289998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:30.062818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1027,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.346874+0000","last_change":"2024-03-30T15:05:17.006278+0000","last_active":"2024-03-30T15:06:28.346874+0000","last_peered":"2024-03-30T15:06:28.346874+0000","last_clean":"2024-03-30T15:06:28.346874+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:06:28.346874+0000","last_undegraded":"2024-03-30T15:06:28.346874+0000","last_fullsized":"2024-03-30T15:06:28.346874+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:05:17.006186+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.006186+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":923,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.344996+0000","last_change":"2024-03-30T15:06:17.187196+0000","last_active":"2024-03-30T15:06:27.344996+0000","last_peered":"2024-03-30T15:06:27.344996+0000","last_clean":"2024-03-30T15:06:27.344996+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:06:27.344996+0000","last_undegraded":"2024-03-30T15:06:27.344996+0000","last_fullsized":"2024-03-30T15:06:27.344996+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:17.187094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:06:17.187094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:20.281913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":998,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338460+0000","last_change":"2024-03-30T15:06:19.337138+0000","last_active":"2024-03-30T15:06:27.338460+0000","last_peered":"2024-03-30T15:06:27.338460+0000","last_clean":"2024-03-30T15:06:27.338460+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:06:27.338460+0000","last_undegraded":"2024-03-30T15:06:27.338460+0000","last_fullsized":"2024-03-30T15:06:27.338460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:03:54.410815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:03:54.410815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":577,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-30T15:06:28.350369+0000","last_change":"2024-03-30T15:05:55.432567+0000","last_active":"2024-03-30T15:06:28.350369+0000","last_peered":"2024-03-30T15:06:28.350369+0000","last_clean":"2024-03-30T15:06:28.350369+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:06:28.350369+0000","last_undegraded":"2024-03-30T15:06:28.350369+0000","last_fullsized":"2024-03-30T15:06:28.350369+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.432437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.432437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:18.685438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":914,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.359678+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:06:27.359678+0000","last_peered":"2024-03-30T15:06:27.359678+0000","last_clean":"2024-03-30T15:06:27.359678+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:06:27.359678+0000","last_undegraded":"2024-03-30T15:06:27.359678+0000","last_fullsized":"2024-03-30T15:06:27.359678+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:39.065353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1048,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.338632+0000","last_change":"2024-03-30T15:04:39.592435+0000","last_active":"2024-03-30T15:06:27.338632+0000","last_peered":"2024-03-30T15:06:27.338632+0000","last_clean":"2024-03-30T15:06:27.338632+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:06:27.338632+0000","last_undegraded":"2024-03-30T15:06:27.338632+0000","last_fullsized":"2024-03-30T15:06:27.338632+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-03-30T15:04:39.592324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:39.592324+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1017,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.361876+0000","last_change":"2024-03-30T15:04:06.578575+0000","last_active":"2024-03-30T15:06:27.361876+0000","last_peered":"2024-03-30T15:06:27.361876+0000","last_clean":"2024-03-30T15:06:27.361876+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:06:27.361876+0000","last_undegraded":"2024-03-30T15:06:27.361876+0000","last_fullsized":"2024-03-30T15:06:27.361876+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:06.578497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:06.578497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:29.631206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1004,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.350513+0000","last_change":"2024-03-30T15:05:27.054772+0000","last_active":"2024-03-30T15:06:27.350513+0000","last_peered":"2024-03-30T15:06:27.350513+0000","last_clean":"2024-03-30T15:06:27.350513+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:06:27.350513+0000","last_undegraded":"2024-03-30T15:06:27.350513+0000","last_fullsized":"2024-03-30T15:06:27.350513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.054691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:06:32.277274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":999,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-30T15:06:27.339058+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:06:27.339058+0000","last_peered":"2024-03-30T15:06:27.339058+0000","last_clean":"2024-03-30T15:06:27.339058+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:06:27.339058+0000","last_undegraded":"2024-03-30T15:06:27.339058+0000","last_fullsized":"2024-03-30T15:06:27.339058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:07:34.368798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":312,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":96,"acting":96,"num_store_stats":8},{"poolid":101,"num_pg":32,"stat_sum":{"num_bytes":8214814,"num_objects":200,"num_object_clones":16,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":219,"num_read_kb":62,"num_write":786,"num_write_kb":12814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26947584,"data_stored":24644469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2128,"ondisk_log_size":2128,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":703,"num_read_kb":698,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005203,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75100,"kb_used_data":7820,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670052,"statfs":{"total":95995035648,"available":95918133248,"internally_reserved":0,"allocated":8007680,"data_stored":5752123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299544,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75048,"kb_used_data":7772,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670104,"statfs":{"total":95995035648,"available":95918186496,"internally_reserved":0,"allocated":7958528,"data_stored":5751069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575027,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66440,"kb_used_data":3316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93678712,"statfs":{"total":95995035648,"available":95927001088,"internally_reserved":0,"allocated":3395584,"data_stored":1165041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69708,"kb_used_data":7996,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93675444,"statfs":{"total":95995035648,"available":95923654656,"internally_reserved":0,"allocated":8187904,"data_stored":5947804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411500,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78636,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666516,"statfs":{"total":95995035648,"available":95914512384,"internally_reserved":0,"allocated":11628544,"data_stored":9359038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117160,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74824,"kb_used_data":7548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670328,"statfs":{"total":95995035648,"available":95918415872,"internally_reserved":0,"allocated":7729152,"data_stored":5370839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953634,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70672,"kb_used_data":3392,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93674480,"statfs":{"total":95995035648,"available":95922667520,"internally_reserved":0,"allocated":3473408,"data_stored":1163403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659293,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70632,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93674520,"statfs":{"total":95995035648,"available":95922708480,"internally_reserved":0,"allocated":3432448,"data_stored":1159004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":4201455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":9535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8519680,"data_stored":8204404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4558848,"data_stored":4205463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":8091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":4021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":4006387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":101,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4272128,"data_stored":4005113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":312,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:06:32.888 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-30T15:06:32.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pg_num 11 2024-03-30T15:06:32.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:32.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:32.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:33.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:33.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:33.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:33.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:33.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 -- 172.21.15.28:0/1647983738 >> v1:172.21.15.28:6790/0 conn(0x7f1b60101bd0 legacy=0x7f1b60101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 -- 172.21.15.28:0/1647983738 shutdown_connections 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 -- 172.21.15.28:0/1647983738 >> 172.21.15.28:0/1647983738 conn(0x7f1b600761e0 msgr2=0x7f1b600765f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 -- 172.21.15.28:0/1647983738 shutdown_connections 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 -- 172.21.15.28:0/1647983738 wait complete. 2024-03-30T15:06:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 librados: starting msgr at 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.411+0000 7f1b663fc700 1 librados: starting objecter 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 Processor -- start 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 -- start start 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 librados: setting wanted keys 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 librados: calling monclient init 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b60101740 con 0x7f1b60109e30 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b601a85d0 con 0x7f1b60101bd0 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b663fc700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1b601a97f0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5f7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b60109e30 0x7f1b60105f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56568/0 (socket says 172.21.15.28:56568) 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5f7fe700 1 -- 172.21.15.28:0/2221417146 learned_addr learned my addr 172.21.15.28:0/2221417146 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b64999700 1 --1- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6790/0 conn(0x7f1b6010d9d0 0x7f1b601a5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56490/0 (socket says 172.21.15.28:56490) 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b60109e30 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter ms_handle_connect 0x7f1b60101bd0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537952239 0 0) 0x7f1b601a97f0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.412+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b480035b0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 696691678 0 0) 0x7f1b601a85d0 con 0x7f1b60101bd0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b601a97f0 con 0x7f1b60101bd0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 312977253 0 0) 0x7f1b60101740 con 0x7f1b60109e30 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b601a85d0 con 0x7f1b60109e30 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1742353519 0 0) 0x7f1b480035b0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b60101740 con 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b58001bb0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1666471753 0 0) 0x7f1b601a85d0 con 0x7f1b60109e30 2024-03-30T15:06:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b480035b0 con 0x7f1b60109e30 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b54001890 con 0x7f1b60109e30 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2308386486 0 0) 0x7f1b601a97f0 con 0x7f1b60101bd0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b601a85d0 con 0x7f1b60101bd0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b50001430 con 0x7f1b60101bd0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2404946892 0 0) 0x7f1b60101740 con 0x7f1b6010d9d0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.134:6789/0 conn(0x7f1b60101bd0 legacy=0x7f1b60105800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6789/0 conn(0x7f1b60109e30 legacy=0x7f1b60105f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b601aaa30 con 0x7f1b6010d9d0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b601a9a20 con 0x7f1b6010d9d0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b663fc700 10 client.58337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b601a9ee0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b663fc700 1 librados: init done 2024-03-30T15:06:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1b58003ec0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.413+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1b58006010 con 0x7f1b6010d9d0 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.414+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1b4c005100 con 0x7f1b6010d9d0 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.415+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1b5801c8d0 con 0x7f1b6010d9d0 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.415+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(555..555 src has 442..555) ==== 8655+0+0 (unknown 115772641 0 0) 0x7f1b58089b70 con 0x7f1b6010d9d0 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 osd_map(555..555 src has 442..555) 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 3 client.58337.objecter handle_osd_map got epochs [555,555] > 0 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 3 client.58337.objecter handle_osd_map decoding full epoch 555 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 20 client.58337.objecter dump_active .. 0 homeless 2024-03-30T15:06:33.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v555) ==== 54+0+16 (unknown 4262135193 0 2420000192) 0x7f1b5808a020 con 0x7f1b6010d9d0 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b48070420 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 10 client.58337.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 1 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 1 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 0 1 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 4 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 10 client.58337.objecter _send_command 1 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c004b50 con 0x7f1b4c004730 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.416+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c004730 2024-03-30T15:06:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.417+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c004b50 con 0x7f1b4c004730 2024-03-30T15:06:33.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.417+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:33.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.417+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 1 = system:0 2024-03-30T15:06:33.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.417+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 4 2024-03-30T15:06:33.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.417+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 0 1 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 10 client.58337.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 2 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 2 2024-03-30T15:06:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0053e0 osd=0 3 2024-03-30T15:06:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 0 2 2024-03-30T15:06:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 4 2024-03-30T15:06:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 10 client.58337.objecter _send_command 2 2024-03-30T15:06:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.435+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c007b00 con 0x7f1b4c004730 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c007b00 con 0x7f1b4c004730 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 4 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 0 2 2024-03-30T15:06:33.438 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 10 client.58337.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 3 2024-03-30T15:06:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 1 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 4 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 10 client.58337.objecter _send_command 3 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c002160 con 0x7f1b4c0042a0 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.437+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c0042a0 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.438+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c002160 con 0x7f1b4c0042a0 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.438+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.438+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 3 = system:0 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.438+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 4 2024-03-30T15:06:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.438+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 1 3 2024-03-30T15:06:33.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 10 client.58337.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 4 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 4 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c004130 osd=1 3 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 1 4 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 4 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 10 client.58337.objecter _send_command 4 2024-03-30T15:06:33.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.455+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c001600 con 0x7f1b4c0042a0 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.456+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c001600 con 0x7f1b4c0042a0 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.456+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.456+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.456+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 4 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.456+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 1 4 2024-03-30T15:06:33.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 10 client.58337.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 5 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 5 2024-03-30T15:06:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 2 5 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 4 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 10 client.58337.objecter _send_command 5 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c011070 con 0x7f1b4c0089f0 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.457+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c0089f0 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.458+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c011070 con 0x7f1b4c0089f0 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.458+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.458+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 5 = system:0 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.458+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 4 2024-03-30T15:06:33.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.458+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 2 5 2024-03-30T15:06:33.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:33.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 10 client.58337.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 6 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 6 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c001a00 osd=2 3 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 2 6 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 4 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 10 client.58337.objecter _send_command 6 2024-03-30T15:06:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.476+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c008020 con 0x7f1b4c0089f0 2024-03-30T15:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c008020 con 0x7f1b4c0089f0 2024-03-30T15:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 4 2024-03-30T15:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 2 6 2024-03-30T15:06:33.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 10 client.58337.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 7 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 7 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 3 7 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 4 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 10 client.58337.objecter _send_command 7 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.478+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c011dc0 con 0x7f1b4c011250 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c011250 2024-03-30T15:06:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c011dc0 con 0x7f1b4c011250 2024-03-30T15:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 7 = system:0 2024-03-30T15:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 4 2024-03-30T15:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.479+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 3 7 2024-03-30T15:06:33.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 10 client.58337.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 8 2024-03-30T15:06:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 8 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0082c0 osd=3 3 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 3 8 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 4 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 10 client.58337.objecter _send_command 8 2024-03-30T15:06:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.496+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c0138c0 con 0x7f1b4c011250 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c0138c0 con 0x7f1b4c011250 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 4 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 3 8 2024-03-30T15:06:33.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 10 client.58337.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 9 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 9 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 4 9 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 4 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 10 client.58337.objecter _send_command 9 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c01ecc0 con 0x7f1b4c01b040 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.498+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c01b040 2024-03-30T15:06:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.499+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c01ecc0 con 0x7f1b4c01b040 2024-03-30T15:06:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.499+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.499+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 9 = system:0 2024-03-30T15:06:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.499+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 4 2024-03-30T15:06:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.499+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 4 9 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 10 client.58337.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 10 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 10 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0138c0 osd=4 3 2024-03-30T15:06:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 4 10 2024-03-30T15:06:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 4 2024-03-30T15:06:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 10 client.58337.objecter _send_command 10 2024-03-30T15:06:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.518+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c01ecc0 con 0x7f1b4c01b040 2024-03-30T15:06:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c01ecc0 con 0x7f1b4c01b040 2024-03-30T15:06:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 4 2024-03-30T15:06:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 4 10 2024-03-30T15:06:33.522 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:33.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 10 client.58337.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 11 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 11 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 5 11 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 4 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 10 client.58337.objecter _send_command 11 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.520+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c0212a0 con 0x7f1b4c020e40 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c020e40 2024-03-30T15:06:33.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c0212a0 con 0x7f1b4c020e40 2024-03-30T15:06:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 11 = system:0 2024-03-30T15:06:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 4 2024-03-30T15:06:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.521+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 5 11 2024-03-30T15:06:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 10 client.58337.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 12 2024-03-30T15:06:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 12 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c01ecc0 osd=5 3 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 5 12 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 4 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 10 client.58337.objecter _send_command 12 2024-03-30T15:06:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.541+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c0212a0 con 0x7f1b4c020e40 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.542+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c0212a0 con 0x7f1b4c020e40 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.542+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.542+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.542+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 4 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.542+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 5 12 2024-03-30T15:06:33.544 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 10 client.58337.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 13 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 13 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 6 13 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 4 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 10 client.58337.objecter _send_command 13 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c027640 con 0x7f1b4c021410 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.543+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c021410 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.544+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b4c027640 con 0x7f1b4c021410 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.544+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.544+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 13 = system:0 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.544+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 4 2024-03-30T15:06:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.544+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 6 13 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 10 client.58337.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 14 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 14 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c0212a0 osd=6 3 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 6 14 2024-03-30T15:06:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 4 2024-03-30T15:06:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 10 client.58337.objecter _send_command 14 2024-03-30T15:06:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.561+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c020890 con 0x7f1b4c021410 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c020890 con 0x7f1b4c021410 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 4 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 6 14 2024-03-30T15:06:33.564 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 10 client.58337.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 15 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 15 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 7 15 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 4 2024-03-30T15:06:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 10 client.58337.objecter _send_command 15 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1b4c028e00 con 0x7f1b4c020640 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.563+0000 7f1b5d7fa700 10 client.58337.objecter ms_handle_connect 0x7f1b4c020640 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.564+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1b50003ed0 con 0x7f1b4c020640 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.564+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.564+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 15 = system:0 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.564+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 4 2024-03-30T15:06:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.564+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 7 15 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 10 client.58337.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign -1 16 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 20 client.58337.objecter _get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_remove -1 16 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 20 client.58337.objecter get_session s=0x7f1b4c00c8d0 osd=7 3 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 15 client.58337.objecter _session_command_op_assign 7 16 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 4 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 10 client.58337.objecter _send_command 16 2024-03-30T15:06:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.580+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1b4c01fc50 con 0x7f1b4c020640 2024-03-30T15:06:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.582+0000 7f1b5d7fa700 1 -- 172.21.15.28:0/2221417146 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1b4c01fc50 con 0x7f1b4c020640 2024-03-30T15:06:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.582+0000 7f1b5d7fa700 10 client.58337.objecter ms_dispatch 0x7f1b60001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.582+0000 7f1b5d7fa700 10 client.58337.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.582+0000 7f1b5d7fa700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 4 2024-03-30T15:06:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.582+0000 7f1b5d7fa700 15 client.58337.objecter _session_command_op_remove 7 16 2024-03-30T15:06:33.584 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 librados: watch_flush enter 2024-03-30T15:06:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 librados: watch_flush exit 2024-03-30T15:06:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.0 2024-03-30T15:06:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6801/3394323575 conn(0x7f1b4c004730 legacy=0x7f1b4c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0053e0 osd=0 1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6805/1850313547 conn(0x7f1b4c0042a0 legacy=0x7f1b4c00aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c004130 osd=1 1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.2 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6809/3451325043 conn(0x7f1b4c0089f0 legacy=0x7f1b4c00f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c001a00 osd=2 1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.3 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6813/1506157250 conn(0x7f1b4c011250 legacy=0x7f1b4c0116b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0082c0 osd=3 1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.4 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.134:6800/3063775514 conn(0x7f1b4c01b040 legacy=0x7f1b4c01d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0138c0 osd=4 1 2024-03-30T15:06:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.5 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.134:6804/3668151179 conn(0x7f1b4c020e40 legacy=0x7f1b4c011f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b64999700 1 -- 172.21.15.28:0/2221417146 reap_dead start 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b64999700 1 -- 172.21.15.28:0/2221417146 reap_dead start 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c01ecc0 osd=5 1 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.6 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.134:6808/2135224194 conn(0x7f1b4c021410 legacy=0x7f1b4c025d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c0212a0 osd=6 1 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter close_session for osd.7 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.134:6812/3758190298 conn(0x7f1b4c020640 legacy=0x7f1b4c027640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter put_session s=0x7f1b4c00c8d0 osd=7 1 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 20 client.58337.objecter shutdown clearing up homeless session... 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 10 client.58337.objecter successfully canceled tick 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b64999700 1 -- 172.21.15.28:0/2221417146 reap_dead start 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1b48070420 legacy=0x7f1b480728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> v1:172.21.15.28:6790/0 conn(0x7f1b6010d9d0 legacy=0x7f1b601a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 shutdown_connections 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 >> 172.21.15.28:0/2221417146 conn(0x7f1b600761e0 msgr2=0x7f1b6010fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 shutdown_connections 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 -- 172.21.15.28:0/2221417146 wait complete. 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:33.583+0000 7f1b663fc700 1 librados: shutdown 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:33.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:33.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:33.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2221417146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:33.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2221417146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:33.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:33.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:33.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2221417146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:33.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:33.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:34.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:34.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:34.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:34.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:34.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:34.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:34.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:34.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:34.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:34 smithi028 ceph-mon[41614]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-30T15:06:34.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:34 smithi028 ceph-mon[29876]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-30T15:06:34.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:34.645 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:34.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:34.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:34.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:34.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:34.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:34 smithi134 ceph-mon[33714]: pgmap v810: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.0 KiB/s wr, 4 op/s 2024-03-30T15:06:35.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:35.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:35.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:35.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:35.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:35.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:35.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:35.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:35.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:35.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:35.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:35.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:35.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:35.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:35.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:35.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:35.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:36.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:36.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:36.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:36.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:36.312 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:36.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:36.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:36.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:36.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:36.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:36.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:36.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:36.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:36.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:37.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:37.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:37.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:37.265 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:37 smithi028 ceph-mon[41614]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 940 B/s wr, 4 op/s 2024-03-30T15:06:37.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:37 smithi028 ceph-mon[29876]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 940 B/s wr, 4 op/s 2024-03-30T15:06:37.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:37.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:37.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:37 smithi134 ceph-mon[33714]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 940 B/s wr, 4 op/s 2024-03-30T15:06:37.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:37.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- 172.21.15.28:0/1235261309 >> v1:172.21.15.28:6789/0 conn(0x7ff318102230 legacy=0x7ff318102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- 172.21.15.28:0/1235261309 shutdown_connections 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- 172.21.15.28:0/1235261309 >> 172.21.15.28:0/1235261309 conn(0x7ff3180fdb00 msgr2=0x7ff3180fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- 172.21.15.28:0/1235261309 shutdown_connections 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- 172.21.15.28:0/1235261309 wait complete. 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 librados: starting msgr at 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 librados: starting objecter 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 Processor -- start 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 -- start start 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 librados: setting wanted keys 2024-03-30T15:06:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.558+0000 7ff31d90b700 1 librados: calling monclient init 2024-03-30T15:06:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31d90b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3181a93b0 con 0x7ff318102230 2024-03-30T15:06:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31d90b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3181aa5d0 con 0x7ff31810ea60 2024-03-30T15:06:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31d90b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3181ab810 con 0x7ff31810aec0 2024-03-30T15:06:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff316ffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff318102230 0x7ff318107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56596/0 (socket says 172.21.15.28:56596) 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff3167fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff31810aec0 0x7ff3181a42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56502/0 (socket says 172.21.15.28:56502) 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff316ffd700 1 -- 172.21.15.28:0/4017497062 learned_addr learned my addr 172.21.15.28:0/4017497062 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter ms_handle_connect 0x7ff318102230 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter ms_handle_connect 0x7ff31810aec0 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter ms_handle_connect 0x7ff31810ea60 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396837984 0 0) 0x7ff3181a93b0 con 0x7ff318102230 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff300003aa0 con 0x7ff318102230 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298900535 0 0) 0x7ff3181ab810 con 0x7ff31810aec0 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3181a93b0 con 0x7ff31810aec0 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123415798 0 0) 0x7ff3181aa5d0 con 0x7ff31810ea60 2024-03-30T15:06:37.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3181ab810 con 0x7ff31810ea60 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3276020126 0 0) 0x7ff300003aa0 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3181aa5d0 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff308004520 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2197876717 0 0) 0x7ff3181ab810 con 0x7ff31810ea60 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff300003aa0 con 0x7ff31810ea60 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.559+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff310001c10 con 0x7ff31810ea60 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 962607297 0 0) 0x7ff3181a93b0 con 0x7ff31810aec0 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3181ab810 con 0x7ff31810aec0 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff30c001830 con 0x7ff31810aec0 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 378767435 0 0) 0x7ff3181aa5d0 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 >> v1:172.21.15.28:6790/0 conn(0x7ff31810aec0 legacy=0x7ff3181a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 >> v1:172.21.15.134:6789/0 conn(0x7ff31810ea60 legacy=0x7ff3181a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3181aca50 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3181a95e0 con 0x7ff318102230 2024-03-30T15:06:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31d90b700 10 client.25143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3181a9aa0 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31d90b700 1 librados: init done 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff308003b20 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.560+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff308005040 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.561+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff318103710 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.561+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff308003680 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(555..555 src has 442..555) ==== 8655+0+0 (unknown 115772641 0 0) 0x7ff308089a40 con 0x7ff318102230 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 10 client.25143.objecter ms_dispatch 0x7ff318001550 osd_map(555..555 src has 442..555) 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 3 client.25143.objecter handle_osd_map got epochs [555,555] > 0 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 3 client.25143.objecter handle_osd_map decoding full epoch 555 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 20 client.25143.objecter dump_active .. 0 homeless 2024-03-30T15:06:37.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.562+0000 7ff31c909700 10 client.25143.objecter ms_handle_connect 0x7ff300070520 2024-03-30T15:06:37.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.566+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff308005020 con 0x7ff318102230 2024-03-30T15:06:37.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:37.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:37.678 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25081.0:2 37.6 37:602f83fe:::foo:head [create] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 522.276136759, 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.092626624, 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25081", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/317566051", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197524+0000", 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197526+0000", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.86e-06 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197531+0000", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2379999999999997e-06 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197542+0000", 2024-03-30T15:06:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0648e-05 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.197552+0000", 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0217000000000001e-05 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199081+0000", 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001528908 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.199091+0000", 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6239999999999997e-06 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246480+0000", 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.047389642 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246590+0000", 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109821 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,3", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.246710+0000", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120436 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256246+0000", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095358879999999993 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267473+0000", 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011226707000000001 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290057+0000", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022584090000000001 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290109+0000", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2052000000000002e-05 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290151+0000", 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1492999999999997e-05 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24892.0:5 22.12 22:4a393722:test-rados-api-smithi028-93390-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 522.24037297400002, 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.057181524, 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24892", 2024-03-30T15:06:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1207320814", 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233288+0000", 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233289+0000", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.091e-06 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233295+0000", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8749999999999997e-06 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233306+0000", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0744000000000001e-05 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233313+0000", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6140000000000003e-06 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233359+0000", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6215999999999999e-05 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.233366+0000", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0679999999999999e-06 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242316+0000", 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0089497279999999 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242442+0000", 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012600999999999999 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.242608+0000", 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000165451 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259414+0000", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016806394999999998 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.259426+0000", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2312e-05 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290387+0000", 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030960560000000002 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290434+0000", 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6702000000000001e-05 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.290469+0000", 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5757999999999999e-05 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24901.0:4 23.2 23:4029b033:test-rados-api-smithi028-93408-2::nonexistent:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 522.23337378199994, 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99567974800000003, 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24901", 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1012171239", 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240287+0000", 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240288+0000", 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.212e-06 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240293+0000", 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1719999999999998e-06 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240304+0000", 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0149000000000001e-05 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.240312+0000", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1480000000000006e-06 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241020+0000", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070865900000000005 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.241028+0000", 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9510000000000006e-06 2024-03-30T15:06:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235868+0000", 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99483949599999999 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.235967+0000", 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8961000000000003e-05 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25048.0:2 46.0s0 46:00000000::::head [pgnls start_epoch 116] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 520.04793546300004, 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2275935200000001, 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25048", 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425725+0000", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425727+0000", 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8560000000000001e-06 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425733+0000", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6740000000000002e-06 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425742+0000", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9740000000000008e-06 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425750+0000", 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6539999999999999e-06 2024-03-30T15:06:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425815+0000", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5493999999999993e-05 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.425818+0000", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.706e-06 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651104+0000", 2024-03-30T15:06:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2252865639999999 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.651143+0000", 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8489999999999999e-05 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.653319+0000", 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021761079999999999 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2064 11.18 11:1888b8ad:test-rados-api-smithi028-93257-16::big.copy2:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 519.33312625099995, 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085813029999999998, 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2064 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140535+0000", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140536+0000", 2024-03-30T15:06:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.023e-06 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140540+0000", 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5650000000000003e-06 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140549+0000", 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.789e-06 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140555+0000", 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.947e-06 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140611+0000", 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5859999999999997e-05 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.140733+0000", 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121965 2024-03-30T15:06:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.204443+0000", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063710599000000007 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.206091+0000", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001648084 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226061+0000", 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019969810000000001 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226257+0000", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019557099999999999 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226301+0000", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4688000000000003e-05 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.226348+0000", 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6128999999999998e-05 2024-03-30T15:06:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34853.0:2073 11.1e e116/110 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head v 116'8, mlcod=116'6)", 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 519.01054904700004, 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.121009771, 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463112+0000", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.463113+0000", 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5859999999999999e-06 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471534+0000", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084207299999999995 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471550+0000", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5531000000000001e-05 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471557+0000", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5140000000000004e-06 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485095+0000", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013537693 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.485121+0000", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6554999999999999e-05 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584083+0000", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.098961412999999998 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.584122+0000", 2024-03-30T15:06:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8748999999999998e-05 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25336.0:2 56.15 56:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 519.002640423, 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4325735549999998, 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25336", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/642826322", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471020+0000", 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471022+0000", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9400000000000001e-06 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471028+0000", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8690000000000001e-06 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471039+0000", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0738e-05 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.471050+0000", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0791999999999999e-05 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.355117+0000", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.88406750999999995 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.355126+0000", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1200000000000008e-06 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875371+0000", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5202449380000003 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875483+0000", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011174499999999999 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,7", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.875613+0000", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013038500000000001 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.880351+0000", 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047371089999999998 2024-03-30T15:06:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.887399+0000", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070482210000000003 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903507+0000", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016108285999999999 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903554+0000", 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7107999999999997e-05 2024-03-30T15:06:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.903594+0000", 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9793999999999999e-05 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2074 11.15 11:a8ba00dd:test-rados-api-smithi028-93257-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 518.84330449200002, 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.095916847999999999, 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2074 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630356+0000", 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.630358+0000", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.623e-06 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666706+0000", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036348502999999997 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666726+0000", 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9771000000000001e-05 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.666738+0000", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2181000000000001e-05 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669591+0000", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028525410000000001 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669724+0000", 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013316700000000001 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.669879+0000", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015516999999999999 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.723327+0000", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053447407000000002 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.724496+0000", 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001169123 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726138+0000", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016417020000000001 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726191+0000", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3597000000000002e-05 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:58.726273+0000", 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2063000000000004e-05 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35393.0:4 55.1f 55:ff7b1f36:::obj1:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 517.741527507, 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1634292589999999, 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35393", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3403500224", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732133+0000", 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732135+0000", 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1610000000000001e-06 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732140+0000", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9779999999999996e-06 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732152+0000", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2031000000000001e-05 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732160+0000", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5549999999999997e-06 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732204+0000", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3553999999999997e-05 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.732209+0000", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7509999999999997e-06 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895502+0000", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1632922020000001 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895563+0000", 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1026999999999997e-05 2024-03-30T15:06:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25342.0:3 57.f 57:f7483bc4:::src_object.2:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 517.739161835, 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1724189570000001, 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25342", 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2592697076", 2024-03-30T15:06:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734499+0000", 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734500+0000", 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.381e-06 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734509+0000", 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0000000000000002e-06 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734520+0000", 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0373e-05 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734528+0000", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4200000000000007e-06 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734582+0000", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3516000000000001e-05 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:59.734587+0000", 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2009999999999998e-06 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.894986+0000", 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1603986509999999 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895093+0000", 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107148 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:06:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895229+0000", 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000136644 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.899876+0000", 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004646837 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.901346+0000", 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001469459 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906825+0000", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054790519999999999 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906885+0000", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0461000000000003e-05 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906918+0000", 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2814000000000001e-05 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25501.0:2 65.c 65:368a68c5:::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [writefull 0~7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 516.48425218, 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9912078200000001, 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25501", 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3010623157", 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989409+0000", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989412+0000", 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2329999999999998e-06 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989425+0000", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3040999999999999e-05 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989439+0000", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4147e-05 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989452+0000", 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2842e-05 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.516774+0000", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.52732199599999996 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.516784+0000", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0035e-05 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.933959+0000", 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.417175176 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.934110+0000", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015060799999999999 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.934274+0000", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000164468 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.942259+0000", 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079849589999999998 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.958103+0000", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015843731 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980496+0000", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022392689 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980572+0000", 2024-03-30T15:06:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6538000000000005e-05 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.980616+0000", 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4357000000000002e-05 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25450.0:2 60.6 60:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 515.66100842399999, 2024-03-30T15:06:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.093909628999999994, 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25450", 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4259934820", 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:06:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812652+0000", 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812654+0000", 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9939999999999999e-06 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812662+0000", 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1300000000000003e-06 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812671+0000", 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6399999999999992e-06 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.812682+0000", 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0607e-05 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.827604+0000", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014922308 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.827613+0000", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1759999999999999e-06 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.893746+0000", 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.066132370999999995 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.893870+0000", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012424100000000001 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.894005+0000", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013545199999999999 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895511+0000", 2024-03-30T15:06:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015054510000000001 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.895603+0000", 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1797000000000002e-05 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906478+0000", 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010874968 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906523+0000", 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5037000000000003e-05 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.906562+0000", 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9456999999999998e-05 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:104 115.7s0 115:eb850916:::103:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-30T15:06:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 496.476196168, 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085521396999999999, 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997465+0000", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997467+0000", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.65e-06 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997476+0000", 2024-03-30T15:06:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7739999999999993e-06 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997493+0000", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6878999999999999e-05 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997503+0000", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0417e-05 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.997580+0000", 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6157999999999998e-05 2024-03-30T15:06:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.998161+0000", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000581453 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:20.998600+0000", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043924900000000001 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.033062+0000", 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034461749 2024-03-30T15:06:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.082911+0000", 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.049849088 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.082986+0000", 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4980000000000004e-05 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:256 115.6s0 115:7f07515b:::255:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 495.47100473099999, 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27174643399999998, 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 256 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002656+0000", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002658+0000", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2809999999999998e-06 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002767+0000", 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108225 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002785+0000", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7963999999999999e-05 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002796+0000", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1630999999999999e-05 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002903+0000", 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107241 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.002911+0000", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.605e-06 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074440+0000", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.071529408000000003 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074584+0000", 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014313800000000001 2024-03-30T15:06:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.074986+0000", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000402488 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.137353+0000", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.062367140000000001 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.274320+0000", 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.136966596 2024-03-30T15:06:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.274403+0000", 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2717000000000001e-05 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:10 104.a e139/131 104:503bb488:test-rados-api-smithi028-93441-7::foo:head v 139'6, mlcod=136'3)", 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 495.28855209800003, 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.100151402, 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185109+0000", 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0400000000000002e-07 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185114+0000", 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0950000000000003e-06 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185121+0000", 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7469999999999999e-06 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185129+0000", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0949999999999996e-06 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.265981+0000", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080851965999999997 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.266007+0000", 2024-03-30T15:06:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6166999999999999e-05 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285232+0000", 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019224465999999999 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285260+0000", 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8462e-05 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32242.0:2 120.6 e139/138 120:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 495.28829898599997, 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099712396999999994, 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185362+0000", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5600000000000001e-07 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185367+0000", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4159999999999997e-06 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185373+0000", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1179999999999997e-06 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185379+0000", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0480000000000004e-06 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185623+0000", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024430699999999998 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.185641+0000", 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7757999999999998e-05 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285042+0000", 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099401239000000002 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.285074+0000", 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2155e-05 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32200.0:2 118.15 118:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 495.21468195, 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088700135999999999, 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32200", 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4179484271", 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258979+0000", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258981+0000", 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-06 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.258991+0000", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0012e-05 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.259004+0000", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.242e-05 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.259014+0000", 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6150000000000003e-06 2024-03-30T15:06:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.268485+0000", 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0094716160000000004 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.268495+0000", 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.628e-06 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324078+0000", 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055582950999999998 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324172+0000", 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4473000000000002e-05 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.324310+0000", 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013811499999999999 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.331855+0000", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075448010000000003 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.331893+0000", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8108999999999997e-05 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347598+0000", 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.01570473 2024-03-30T15:06:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347641+0000", 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3544000000000002e-05 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.347679+0000", 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7533999999999997e-05 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34853.0:2078 11.18 11:1888ddfb:test-rados-api-smithi028-93257-17::big.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 488.71660084899997, 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14551070799999999, 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34853", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2078 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:06:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757060+0000", 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757062+0000", 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6199999999999999e-06 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757071+0000", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6010000000000001e-06 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757081+0000", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4860000000000003e-06 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757088+0000", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0899999999999999e-06 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757142+0000", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3749000000000003e-05 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.757248+0000", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106406 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.884537+0000", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.127288966 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.887673+0000", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003135826 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902352+0000", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014678862000000001 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902473+0000", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121415 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902531+0000", 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7527e-05 2024-03-30T15:06:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.902571+0000", 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0160000000000002e-05 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57050.0:2 190.6 190:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 450.26396218100001, 2024-03-30T15:06:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075417256000000002, 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57050", 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1148433196", 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209701+0000", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3800000000000001e-06 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209707+0000", 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9460000000000003e-06 2024-03-30T15:06:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209718+0000", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0679000000000001e-05 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209729+0000", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0896999999999999e-05 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213086+0000", 2024-03-30T15:06:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033575290000000002 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213092+0000", 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6489999999999996e-06 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220438+0000", 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073462149999999997 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:06:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220446+0000", 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2770000000000005e-06 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272811+0000", 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052364490999999999 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272945+0000", 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013451899999999999 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273074+0000", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012904 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273476+0000", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000401875 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.274847+0000", 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013706339999999999 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.284902+0000", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010055015 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285053+0000", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015113999999999999 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285116+0000", 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2970000000000002e-05 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:5374 285.4 285:20000000:test-rados-api-smithi028-93253-50:::head [pgnls start_epoch 389] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e389)", 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 244.08764943, 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11848968999999999, 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5374 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386012+0000", 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0109999999999999e-06 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386018+0000", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2329999999999997e-06 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386026+0000", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5250000000000005e-06 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386031+0000", 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2859999999999999e-06 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386087+0000", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5448999999999997e-05 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386091+0000", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.296e-06 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504352+0000", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.118261214 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504392+0000", 2024-03-30T15:06:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9728000000000003e-05 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504501+0000", 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108948 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:37.793 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:06:37.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:37.782+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7ff318075a20 con 0x7ff318102230 2024-03-30T15:06:37.821 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:37.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:37.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:37.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:38.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:38.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:38.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:38.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:38.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.323+0000 7ff31c909700 1 -- 172.21.15.28:0/4017497062 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v556) ==== 91+0+0 (unknown 12228809 0 0) 0x7ff308056a60 con 0x7ff318102230 2024-03-30T15:06:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.326+0000 7ff31d90b700 10 librados: watch_flush enter 2024-03-30T15:06:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.326+0000 7ff31d90b700 10 librados: watch_flush exit 2024-03-30T15:06:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.326+0000 7ff31d90b700 20 client.25143.objecter shutdown clearing up homeless session... 2024-03-30T15:06:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.326+0000 7ff31d90b700 10 client.25143.objecter successfully canceled tick 2024-03-30T15:06:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.327+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff300070520 legacy=0x7ff3000729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.327+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 >> v1:172.21.15.28:6789/0 conn(0x7ff318102230 legacy=0x7ff318107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.327+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 shutdown_connections 2024-03-30T15:06:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.327+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 >> 172.21.15.28:0/4017497062 conn(0x7ff3180fdb00 msgr2=0x7ff3181116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.328+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 shutdown_connections 2024-03-30T15:06:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.328+0000 7ff31d90b700 1 -- 172.21.15.28:0/4017497062 wait complete. 2024-03-30T15:06:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:38.328+0000 7ff31d90b700 1 librados: shutdown 2024-03-30T15:06:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:06:38.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:38.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:38.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:06:38.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:38.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:06:38.344 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:38.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:06:38.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:38.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:06:38.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:38.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:38.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:38.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:38.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:38.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:38.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:38.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:06:38.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:38.978 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:38.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:39.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:39.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:39.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:39.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:39.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:39.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:39 smithi134 ceph-mon[33714]: pgmap v812: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-30T15:06:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:39 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:06:39.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:39 smithi134 ceph-mon[33714]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T15:06:39.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[29876]: pgmap v812: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-30T15:06:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:06:39.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[29876]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T15:06:39.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[41614]: pgmap v812: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-30T15:06:39.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:06:39.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4017497062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:06:39.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:39 smithi028 ceph-mon[41614]: osdmap e556: 8 total, 8 up, 8 in 2024-03-30T15:06:39.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:39.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:39.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:39.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:39.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:39.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:39.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:40.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:40.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 101.19 scrub starts 2024-03-30T15:06:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 101.19 scrub ok 2024-03-30T15:06:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 101.19 scrub starts 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 101.19 scrub ok 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T15:06:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:40 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T15:06:40.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:40.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:40.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 101.19 scrub starts 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 101.19 scrub ok 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T15:06:40.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T15:06:40.490 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:40.490 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:40.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.512+0000 7f559f425700 1 -- 172.21.15.28:0/1487464589 >> v1:172.21.15.28:6789/0 conn(0x7f5598109dc0 legacy=0x7f559810c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.512+0000 7f559f425700 1 -- 172.21.15.28:0/1487464589 shutdown_connections 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.512+0000 7f559f425700 1 -- 172.21.15.28:0/1487464589 >> 172.21.15.28:0/1487464589 conn(0x7f55980fc9e0 msgr2=0x7f55980fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.512+0000 7f559f425700 1 -- 172.21.15.28:0/1487464589 shutdown_connections 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.512+0000 7f559f425700 1 -- 172.21.15.28:0/1487464589 wait complete. 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 librados: starting msgr at 2024-03-30T15:06:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 librados: starting objecter 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 Processor -- start 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 -- start start 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 librados: setting wanted keys 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.513+0000 7f559f425700 1 librados: calling monclient init 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f559f425700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5598106820 con 0x7f5598109dc0 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f559f425700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5598106a30 con 0x7f5598101130 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f559f425700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f55981a7570 con 0x7f559810d960 2024-03-30T15:06:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f559d9c2700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f559810d960 0x7f55981a3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37082/0 (socket says 172.21.15.28:37082) 2024-03-30T15:06:40.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f559d9c2700 1 -- 172.21.15.28:0/2677957353 learned_addr learned my addr 172.21.15.28:0/2677957353 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:40.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter ms_handle_connect 0x7f559810d960 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter ms_handle_connect 0x7f5598109dc0 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355765266 0 0) 0x7f55981a7570 con 0x7f559810d960 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55840035b0 con 0x7f559810d960 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter ms_handle_connect 0x7f5598101130 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705674819 0 0) 0x7f5598106820 con 0x7f5598109dc0 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.514+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55981a7570 con 0x7f5598109dc0 2024-03-30T15:06:40.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3381570529 0 0) 0x7f55840035b0 con 0x7f559810d960 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5598106820 con 0x7f559810d960 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5590001c50 con 0x7f559810d960 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 521127968 0 0) 0x7f55981a7570 con 0x7f5598109dc0 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55840035b0 con 0x7f5598109dc0 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5594001830 con 0x7f5598109dc0 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515168347 0 0) 0x7f5598106a30 con 0x7f5598101130 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55981a7570 con 0x7f5598101130 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 739547770 0 0) 0x7f55840035b0 con 0x7f5598109dc0 2024-03-30T15:06:40.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 >> v1:172.21.15.28:6790/0 conn(0x7f559810d960 legacy=0x7f55981a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 >> v1:172.21.15.134:6789/0 conn(0x7f5598101130 legacy=0x7f5598102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55981acbc0 con 0x7f5598109dc0 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55981a77a0 con 0x7f5598109dc0 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f559f425700 10 client.25149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f55981a7c60 con 0x7f5598109dc0 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f559f425700 1 librados: init done 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f55940011b0 con 0x7f5598109dc0 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.515+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5594004c90 con 0x7f5598109dc0 2024-03-30T15:06:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.516+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557c005120 con 0x7f5598109dc0 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.517+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5594001a90 con 0x7f5598109dc0 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(557..557 src has 442..557) ==== 8683+0+0 (unknown 944114192 0 0) 0x7f5594088990 con 0x7f5598109dc0 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 10 client.25149.objecter ms_dispatch 0x7f5598001550 osd_map(557..557 src has 442..557) 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 3 client.25149.objecter handle_osd_map got epochs [557,557] > 0 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 3 client.25149.objecter handle_osd_map decoding full epoch 557 2024-03-30T15:06:40.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 20 client.25149.objecter dump_active .. 0 homeless 2024-03-30T15:06:40.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.518+0000 7f558a7fc700 10 client.25149.objecter ms_handle_connect 0x7f5584078c00 2024-03-30T15:06:40.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.522+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5594055970 con 0x7f5598109dc0 2024-03-30T15:06:40.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:40.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:40.728+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7f557c005410 con 0x7f5598109dc0 2024-03-30T15:06:40.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:40.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:40.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:40.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:40.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:40.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:41.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:41.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:41.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.104+0000 7f558a7fc700 1 -- 172.21.15.28:0/2677957353 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v558) ==== 129+0+0 (unknown 2791609607 0 0) 0x7f55940593c0 con 0x7f5598109dc0 2024-03-30T15:06:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 10 librados: watch_flush enter 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 10 librados: watch_flush exit 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 20 client.25149.objecter shutdown clearing up homeless session... 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 10 client.25149.objecter successfully canceled tick 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5584078c00 legacy=0x7f558407b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 >> v1:172.21.15.28:6789/0 conn(0x7f5598109dc0 legacy=0x7f5598103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 shutdown_connections 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.107+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 >> 172.21.15.28:0/2677957353 conn(0x7f55980fc9e0 msgr2=0x7f55981105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.108+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 shutdown_connections 2024-03-30T15:06:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.108+0000 7f559f425700 1 -- 172.21.15.28:0/2677957353 wait complete. 2024-03-30T15:06:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:41.108+0000 7f559f425700 1 librados: shutdown 2024-03-30T15:06:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 11 2024-03-30T15:06:41.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:41.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 101.0 scrub starts 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 101.0 scrub ok 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: pgmap v814: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:06:41.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 101.0 scrub starts 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 101.0 scrub ok 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: pgmap v814: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:06:41.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:06:41.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T15:06:41.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:41.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:41.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:41.417 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 101.0 scrub starts 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 101.0 scrub ok 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:41.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: pgmap v814: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:41.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: osdmap e557: 8 total, 8 up, 8 in 2024-03-30T15:06:41.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:06:41.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:06:41.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-30T15:06:41.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:41.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:41.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:41.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:41.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:41.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:41.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:41.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:41.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:06:42.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:42.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:42.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 101.3 scrub starts 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 101.3 scrub ok 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:06:42.188 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 101.3 scrub starts 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 101.3 scrub ok 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T15:06:42.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T15:06:42.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:06:42.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:06:42.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:42 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:42.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:42.319 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:42.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 101.3 scrub starts 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 101.3 scrub ok 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2677957353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-30T15:06:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: osdmap e558: 8 total, 8 up, 8 in 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:42 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:42.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:42.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:42.576 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:42.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:42.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:42.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:42.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:42.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:42.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:43.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:43.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:06:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:06:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:06:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:06:43.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:43.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: pgmap v817: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T15:06:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T15:06:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T15:06:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T15:06:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T15:06:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T15:06:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:43 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T15:06:43.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: pgmap v817: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: pgmap v817: 260 pgs: 260 active+clean; 8.4 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-30T15:06:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: osdmap e559: 8 total, 8 up, 8 in 2024-03-30T15:06:43.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T15:06:43.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T15:06:43.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: osdmap e560: 8 total, 8 up, 8 in 2024-03-30T15:06:43.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T15:06:43.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:43 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T15:06:43.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:43.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:43.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:43.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:43.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:43.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:43.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:43.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:43.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:43.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:44.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:44.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:44.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:44.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:44.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:06:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:06:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T15:06:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:44 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: osdmap e561: 8 total, 8 up, 8 in 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:44 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-30T15:06:44.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:44.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:06:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:44.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:44.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:44.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:44.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:44.872 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:44.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:44.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:45.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:45.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:45.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:45.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.379+0000 7f128e16c700 1 -- 172.21.15.28:0/1759515485 >> v1:172.21.15.28:6789/0 conn(0x7f128810d960 legacy=0x7f128810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 42.18 scrub starts 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 42.18 scrub ok 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: pgmap v821: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:45.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 101.4 scrub starts 2024-03-30T15:06:45.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[29876]: 101.4 scrub ok 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 42.18 scrub starts 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 42.18 scrub ok 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: pgmap v821: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:06:45.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 101.4 scrub starts 2024-03-30T15:06:45.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:45 smithi028 ceph-mon[41614]: 101.4 scrub ok 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.380+0000 7f128e16c700 1 -- 172.21.15.28:0/1759515485 shutdown_connections 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.380+0000 7f128e16c700 1 -- 172.21.15.28:0/1759515485 >> 172.21.15.28:0/1759515485 conn(0x7f12880fc9e0 msgr2=0x7f12880fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.380+0000 7f128e16c700 1 -- 172.21.15.28:0/1759515485 shutdown_connections 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.380+0000 7f128e16c700 1 -- 172.21.15.28:0/1759515485 wait complete. 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.381+0000 7f128e16c700 1 librados: starting msgr at 2024-03-30T15:06:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.381+0000 7f128e16c700 1 librados: starting objecter 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f128e16c700 1 Processor -- start 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f128e16c700 1 -- start start 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f128e16c700 1 librados: setting wanted keys 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f128e16c700 1 librados: calling monclient init 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f1286ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1288109dc0 0x7f1288105d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37100/0 (socket says 172.21.15.28:37100) 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.382+0000 7f1286ffd700 1 -- 172.21.15.28:0/2946563703 learned_addr learned my addr 172.21.15.28:0/2946563703 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter ms_handle_connect 0x7f1288109dc0 2024-03-30T15:06:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter ms_handle_connect 0x7f128810d960 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f128806b1a0 con 0x7f128810d960 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f12881a7530 con 0x7f1288101130 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter ms_handle_connect 0x7f1288101130 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f12881a7700 con 0x7f1288109dc0 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1499866330 0 0) 0x7f128806b1a0 con 0x7f128810d960 2024-03-30T15:06:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12700035b0 con 0x7f128810d960 2024-03-30T15:06:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1305687420 0 0) 0x7f12881a7700 con 0x7f1288109dc0 2024-03-30T15:06:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f128806b1a0 con 0x7f1288109dc0 2024-03-30T15:06:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4034634166 0 0) 0x7f12881a7530 con 0x7f1288101130 2024-03-30T15:06:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.383+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12881a7700 con 0x7f1288101130 2024-03-30T15:06:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 173425565 0 0) 0x7f12700035b0 con 0x7f128810d960 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12881a7530 con 0x7f128810d960 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3303358541 0 0) 0x7f12881a7700 con 0x7f1288101130 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12700035b0 con 0x7f1288101130 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 848667128 0 0) 0x7f128806b1a0 con 0x7f1288109dc0 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12881a7700 con 0x7f1288109dc0 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1280004920 con 0x7f128810d960 2024-03-30T15:06:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12780017b0 con 0x7f1288101130 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.384+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f127c001830 con 0x7f1288109dc0 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2254312980 0 0) 0x7f12881a7530 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6790/0 conn(0x7f1288109dc0 legacy=0x7f1288105d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.134:6789/0 conn(0x7f1288101130 legacy=0x7f1288105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12881a78d0 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1288103420 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f128e16c700 10 client.25155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1288103880 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f128e16c700 1 librados: init done 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1280001a60 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.385+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f12800050b0 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.386+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1274005100 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.387+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f128001b750 con 0x7f128810d960 2024-03-30T15:06:45.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.387+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(563..563 src has 442..563) ==== 8945+0+0 (unknown 2997117752 0 0) 0x7f1280088ab0 con 0x7f128810d960 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.387+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 osd_map(563..563 src has 442..563) 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.387+0000 7f1284ff9700 3 client.25155.objecter handle_osd_map got epochs [563,563] > 0 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.387+0000 7f1284ff9700 3 client.25155.objecter handle_osd_map decoding full epoch 563 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f1284ff9700 20 client.25155.objecter dump_active .. 0 homeless 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v563) ==== 54+0+16 (unknown 129882689 0 2420000192) 0x7f1280088ef0 con 0x7f128810d960 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 10 client.25155.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 1 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f12700703d0 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 1 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 0 1 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 4 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 10 client.25155.objecter _send_command 1 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.388+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1274004b50 con 0x7f1274004730 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.389+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f1274004730 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.390+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274004b50 con 0x7f1274004730 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.390+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.390+0000 7f1284ff9700 10 client.25155.objecter _finish_command 1 = system:0 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.390+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 4 2024-03-30T15:06:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.390+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 0 1 2024-03-30T15:06:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 10 client.25155.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 2 2024-03-30T15:06:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 2 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f12740053e0 osd=0 3 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 0 2 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 4 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 10 client.25155.objecter _send_command 2 2024-03-30T15:06:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.410+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1274007b00 con 0x7f1274004730 2024-03-30T15:06:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1274007b00 con 0x7f1274004730 2024-03-30T15:06:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 10 client.25155.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 4 2024-03-30T15:06:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 0 2 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 10 client.25155.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 3 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 3 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 1 3 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 4 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 10 client.25155.objecter _send_command 3 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1274002160 con 0x7f12740042a0 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.412+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f12740042a0 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.413+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274002160 con 0x7f12740042a0 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.413+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.413+0000 7f1284ff9700 10 client.25155.objecter _finish_command 3 = system:0 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.413+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 4 2024-03-30T15:06:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.413+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 1 3 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 42.18 scrub starts 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 42.18 scrub ok 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T15:06:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: pgmap v821: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: osdmap e562: 8 total, 8 up, 8 in 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 101.4 scrub starts 2024-03-30T15:06:45.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:45 smithi134 ceph-mon[33714]: 101.4 scrub ok 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 10 client.25155.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 4 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 4 2024-03-30T15:06:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274004130 osd=1 3 2024-03-30T15:06:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 1 4 2024-03-30T15:06:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 4 2024-03-30T15:06:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 10 client.25155.objecter _send_command 4 2024-03-30T15:06:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.431+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1274001600 con 0x7f12740042a0 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1274001600 con 0x7f12740042a0 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 10 client.25155.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 4 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 1 4 2024-03-30T15:06:45.434 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 10 client.25155.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 5 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 5 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 2 5 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 4 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 10 client.25155.objecter _send_command 5 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1274011010 con 0x7f1274008990 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.433+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f1274008990 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.434+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274011010 con 0x7f1274008990 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.434+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.434+0000 7f1284ff9700 10 client.25155.objecter _finish_command 5 = system:0 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.434+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 4 2024-03-30T15:06:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.434+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 2 5 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 10 client.25155.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 6 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 6 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274001a00 osd=2 3 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 2 6 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 4 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 10 client.25155.objecter _send_command 6 2024-03-30T15:06:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.451+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1274007fc0 con 0x7f1274008990 2024-03-30T15:06:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1274007fc0 con 0x7f1274008990 2024-03-30T15:06:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f1284ff9700 10 client.25155.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 4 2024-03-30T15:06:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 2 6 2024-03-30T15:06:45.454 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 10 client.25155.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 7 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 7 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 3 7 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 4 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 10 client.25155.objecter _send_command 7 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.452+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1274011d40 con 0x7f12740111f0 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f12740111f0 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274011d40 con 0x7f12740111f0 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 10 client.25155.objecter _finish_command 7 = system:0 2024-03-30T15:06:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 4 2024-03-30T15:06:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.453+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 3 7 2024-03-30T15:06:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 10 client.25155.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 8 2024-03-30T15:06:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 8 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127400cd30 osd=3 3 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 3 8 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 4 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 10 client.25155.objecter _send_command 8 2024-03-30T15:06:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.470+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1274013840 con 0x7f12740111f0 2024-03-30T15:06:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1274013840 con 0x7f12740111f0 2024-03-30T15:06:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f1284ff9700 10 client.25155.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 4 2024-03-30T15:06:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 3 8 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 10 client.25155.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 9 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 9 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 4 9 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 4 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 10 client.25155.objecter _send_command 9 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.472+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f127401a7b0 con 0x7f1274016bd0 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.473+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f1274016bd0 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.474+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f127401a7b0 con 0x7f1274016bd0 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.474+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.474+0000 7f1284ff9700 10 client.25155.objecter _finish_command 9 = system:0 2024-03-30T15:06:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.474+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 4 2024-03-30T15:06:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.474+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 4 9 2024-03-30T15:06:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 10 client.25155.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 10 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 10 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274013840 osd=4 3 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 4 10 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 4 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 10 client.25155.objecter _send_command 10 2024-03-30T15:06:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.490+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f127401bb10 con 0x7f1274016bd0 2024-03-30T15:06:45.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f127401bb10 con 0x7f1274016bd0 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f1284ff9700 10 client.25155.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 4 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 4 10 2024-03-30T15:06:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 10 client.25155.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 11 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 11 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 5 11 2024-03-30T15:06:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 4 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 10 client.25155.objecter _send_command 11 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.492+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f127401ce40 con 0x7f127401bd20 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.493+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f127401bd20 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.494+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f127401ce40 con 0x7f127401bd20 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.494+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.494+0000 7f1284ff9700 10 client.25155.objecter _finish_command 11 = system:0 2024-03-30T15:06:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.494+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 4 2024-03-30T15:06:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.494+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 5 11 2024-03-30T15:06:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 10 client.25155.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 12 2024-03-30T15:06:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 12 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127401bb10 osd=5 3 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 5 12 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 4 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 10 client.25155.objecter _send_command 12 2024-03-30T15:06:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.511+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f127401ce40 con 0x7f127401bd20 2024-03-30T15:06:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f127401ce40 con 0x7f127401bd20 2024-03-30T15:06:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f1284ff9700 10 client.25155.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 4 2024-03-30T15:06:45.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 5 12 2024-03-30T15:06:45.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:45.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 10 client.25155.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 13 2024-03-30T15:06:45.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 13 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 6 13 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 4 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 10 client.25155.objecter _send_command 13 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.513+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1274023380 con 0x7f127401f870 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.514+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f127401f870 2024-03-30T15:06:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.515+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274023380 con 0x7f127401f870 2024-03-30T15:06:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.515+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.515+0000 7f1284ff9700 10 client.25155.objecter _finish_command 13 = system:0 2024-03-30T15:06:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.515+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 4 2024-03-30T15:06:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.515+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 6 13 2024-03-30T15:06:45.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 10 client.25155.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 14 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 14 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f127401ce40 osd=6 3 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 6 14 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 4 2024-03-30T15:06:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 10 client.25155.objecter _send_command 14 2024-03-30T15:06:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.533+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f127400c5c0 con 0x7f127401f870 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f127400c5c0 con 0x7f127401f870 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 10 client.25155.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 4 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 6 14 2024-03-30T15:06:45.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 10 client.25155.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 15 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 15 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 7 15 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 4 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 10 client.25155.objecter _send_command 15 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1274027720 con 0x7f1274023a00 2024-03-30T15:06:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.535+0000 7f1284ff9700 10 client.25155.objecter ms_handle_connect 0x7f1274023a00 2024-03-30T15:06:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.536+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1274027720 con 0x7f1274023a00 2024-03-30T15:06:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.536+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.536+0000 7f1284ff9700 10 client.25155.objecter _finish_command 15 = system:0 2024-03-30T15:06:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.536+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 4 2024-03-30T15:06:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.536+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 7 15 2024-03-30T15:06:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 10 client.25155.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:06:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign -1 16 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 20 client.25155.objecter _get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 15 client.25155.objecter _session_command_op_remove -1 16 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 20 client.25155.objecter get_session s=0x7f1274023380 osd=7 3 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 15 client.25155.objecter _session_command_op_assign 7 16 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 4 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 10 client.25155.objecter _send_command 16 2024-03-30T15:06:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.553+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f127401a960 con 0x7f1274023a00 2024-03-30T15:06:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.555+0000 7f1284ff9700 1 -- 172.21.15.28:0/2946563703 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f127401a960 con 0x7f1274023a00 2024-03-30T15:06:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.555+0000 7f1284ff9700 10 client.25155.objecter ms_dispatch 0x7f1288001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.555+0000 7f1284ff9700 10 client.25155.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.555+0000 7f1284ff9700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 4 2024-03-30T15:06:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.555+0000 7f1284ff9700 15 client.25155.objecter _session_command_op_remove 7 16 2024-03-30T15:06:45.557 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:45.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 librados: watch_flush enter 2024-03-30T15:06:45.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 librados: watch_flush exit 2024-03-30T15:06:45.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.0 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6801/3394323575 conn(0x7f1274004730 legacy=0x7f1274002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f12740053e0 osd=0 1 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.1 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6805/1850313547 conn(0x7f12740042a0 legacy=0x7f127400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274004130 osd=1 1 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.2 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6809/3451325043 conn(0x7f1274008990 legacy=0x7f127400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f1287fff700 1 -- 172.21.15.28:0/2946563703 reap_dead start 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274001a00 osd=2 1 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.3 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6813/1506157250 conn(0x7f12740111f0 legacy=0x7f1274011630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127400cd30 osd=3 1 2024-03-30T15:06:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.4 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.134:6800/3063775514 conn(0x7f1274016bd0 legacy=0x7f1274018fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274013840 osd=4 1 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.5 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.134:6804/3668151179 conn(0x7f127401bd20 legacy=0x7f1274011ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401bb10 osd=5 1 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.6 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.134:6808/2135224194 conn(0x7f127401f870 legacy=0x7f1274021c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f127401ce40 osd=6 1 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter close_session for osd.7 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.134:6812/3758190298 conn(0x7f1274023a00 legacy=0x7f1274025e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f1287fff700 1 -- 172.21.15.28:0/2946563703 reap_dead start 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter put_session s=0x7f1274023380 osd=7 1 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 20 client.25155.objecter shutdown clearing up homeless session... 2024-03-30T15:06:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 10 client.25155.objecter successfully canceled tick 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6800/3011431093 conn(0x7f12700703d0 legacy=0x7f1270072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> v1:172.21.15.28:6789/0 conn(0x7f128810d960 legacy=0x7f12881a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 shutdown_connections 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 >> 172.21.15.28:0/2946563703 conn(0x7f12880fc9e0 msgr2=0x7f1288110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 shutdown_connections 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 -- 172.21.15.28:0/2946563703 wait complete. 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:45.556+0000 7f128e16c700 1 librados: shutdown 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:45.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:45.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:45.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:45.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:46.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:46.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:46.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:46.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:46.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:46.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:06:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:06:46.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T15:06:46.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2946563703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2946563703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: osdmap e563: 8 total, 8 up, 8 in 2024-03-30T15:06:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2946563703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:46.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:46.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:46.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:46.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:46.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:46.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:46.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:47.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:47.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:47.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 42.2 scrub starts 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 42.2 scrub ok 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: pgmap v824: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 101.5 scrub starts 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: 101.5 scrub ok 2024-03-30T15:06:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[29876]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 42.2 scrub starts 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 42.2 scrub ok 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: pgmap v824: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 101.5 scrub starts 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: 101.5 scrub ok 2024-03-30T15:06:47.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:47 smithi028 ceph-mon[41614]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T15:06:47.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:47.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:47.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:47.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 42.2 scrub starts 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 42.2 scrub ok 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: pgmap v824: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 101.5 scrub starts 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: 101.5 scrub ok 2024-03-30T15:06:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:47 smithi134 ceph-mon[33714]: osdmap e564: 8 total, 8 up, 8 in 2024-03-30T15:06:47.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:47.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:47.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:47.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:47.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:47.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:47.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:48.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:48.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:48.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:48.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:48.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:48.330 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:48.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:48.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:48.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:48.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 101.1 scrub starts 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 101.1 scrub ok 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: pgmap v827: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T15:06:48.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 101.1 scrub starts 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 101.1 scrub ok 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: pgmap v827: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T15:06:48.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:48 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T15:06:48.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: osdmap e565: 8 total, 8 up, 8 in 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 101.1 scrub starts 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 101.1 scrub ok 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: pgmap v827: 270 pgs: 270 active+clean; 8.4 MiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T15:06:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:48 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T15:06:48.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:48.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.721+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/2217378609 >> v1:172.21.15.28:6790/0 conn(0x7f0e1410d970 legacy=0x7f0e1410fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.721+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/2217378609 shutdown_connections 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.721+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/2217378609 >> 172.21.15.28:0/2217378609 conn(0x7f0e140fc9f0 msgr2=0x7f0e140fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/2217378609 shutdown_connections 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/2217378609 wait complete. 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 librados: starting msgr at 2024-03-30T15:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 librados: starting objecter 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 Processor -- start 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- start start 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 librados: setting wanted keys 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 librados: calling monclient init 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e140fe9e0 con 0x7f0e14101140 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e140febf0 con 0x7f0e1410d970 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e1ae9a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e141ab950 con 0x7f0e14109dd0 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e18c36700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e14101140 0x7f0e140fcea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57008/0 (socket says 172.21.15.28:57008) 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e13fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f0e14109dd0 0x7f0e140fd5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58326/0 (socket says 172.21.15.28:58326) 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.722+0000 7f0e18c36700 1 -- 172.21.15.28:0/3802353790 learned_addr learned my addr 172.21.15.28:0/3802353790 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter ms_handle_connect 0x7f0e14101140 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter ms_handle_connect 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter ms_handle_connect 0x7f0e1410d970 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503184706 0 0) 0x7f0e141ab950 con 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e00003aa0 con 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889659689 0 0) 0x7f0e140fe9e0 con 0x7f0e14101140 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e141ab950 con 0x7f0e14101140 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932861367 0 0) 0x7f0e140febf0 con 0x7f0e1410d970 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e140fe9e0 con 0x7f0e1410d970 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2023432764 0 0) 0x7f0e00003aa0 con 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e140febf0 con 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0dfc0018b0 con 0x7f0e14109dd0 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1858855008 0 0) 0x7f0e141ab950 con 0x7f0e14101140 2024-03-30T15:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e00003aa0 con 0x7f0e14101140 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e04001db0 con 0x7f0e14101140 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1900915295 0 0) 0x7f0e140fe9e0 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e141ab950 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e0c004650 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3854568259 0 0) 0x7f0e141ab950 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 >> v1:172.21.15.28:6790/0 conn(0x7f0e14109dd0 legacy=0x7f0e140fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 >> v1:172.21.15.28:6789/0 conn(0x7f0e14101140 legacy=0x7f0e140fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e141acb30 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.723+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e141abb20 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0e0c001600 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e0c004ef0 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e1ae9a700 10 client.50368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e141abfe0 con 0x7f0e1410d970 2024-03-30T15:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e1ae9a700 1 librados: init done 2024-03-30T15:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.724+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e14102620 con 0x7f0e1410d970 2024-03-30T15:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.725+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0e0c0037f0 con 0x7f0e1410d970 2024-03-30T15:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.725+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(566..566 src has 442..566) ==== 8655+0+0 (unknown 2005843438 0 0) 0x7f0e0c088750 con 0x7f0e1410d970 2024-03-30T15:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.725+0000 7f0e11ffb700 10 client.50368.objecter ms_dispatch 0x7f0e14001550 osd_map(566..566 src has 442..566) 2024-03-30T15:06:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.725+0000 7f0e11ffb700 3 client.50368.objecter handle_osd_map got epochs [566,566] > 0 2024-03-30T15:06:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.725+0000 7f0e11ffb700 3 client.50368.objecter handle_osd_map decoding full epoch 566 2024-03-30T15:06:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.726+0000 7f0e11ffb700 20 client.50368.objecter dump_active .. 0 homeless 2024-03-30T15:06:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.726+0000 7f0e11ffb700 10 client.50368.objecter ms_handle_connect 0x7f0e000703d0 2024-03-30T15:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.730+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0e0c055750 con 0x7f0e1410d970 2024-03-30T15:06:48.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:48.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:48.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:48.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:48.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:48.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:48.947+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0e140fe3d0 con 0x7f0e1410d970 2024-03-30T15:06:49.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:49.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:49.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:49.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.342+0000 7f0e11ffb700 1 -- 172.21.15.28:0/3802353790 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v567) ==== 101+0+0 (unknown 4144179829 0 0) 0x7f0e0c0591a0 con 0x7f0e1410d970 2024-03-30T15:06:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 10 librados: watch_flush enter 2024-03-30T15:06:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 10 librados: watch_flush exit 2024-03-30T15:06:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 20 client.50368.objecter shutdown clearing up homeless session... 2024-03-30T15:06:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 10 client.50368.objecter successfully canceled tick 2024-03-30T15:06:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0e000703d0 legacy=0x7f0e00072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 >> v1:172.21.15.134:6789/0 conn(0x7f0e1410d970 legacy=0x7f0e141a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 shutdown_connections 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.346+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 >> 172.21.15.28:0/3802353790 conn(0x7f0e140fc9f0 msgr2=0x7f0e141105e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.347+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 shutdown_connections 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.347+0000 7f0e1ae9a700 1 -- 172.21.15.28:0/3802353790 wait complete. 2024-03-30T15:06:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:49.347+0000 7f0e1ae9a700 1 librados: shutdown 2024-03-30T15:06:49.349 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:06:49.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:49.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:49.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: 1.5 scrub starts 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: 1.5 scrub ok 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3802353790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: from='client.50368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: 1.9 scrub starts 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[41614]: 1.9 scrub ok 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T15:06:49.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: 1.5 scrub starts 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: 1.5 scrub ok 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3802353790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: from='client.50368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: 1.9 scrub starts 2024-03-30T15:06:49.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:49 smithi028 ceph-mon[29876]: 1.9 scrub ok 2024-03-30T15:06:49.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:49.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:06:49.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: osdmap e566: 8 total, 8 up, 8 in 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-94"}]: dispatch 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: 1.5 scrub starts 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: 1.5 scrub ok 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3802353790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: from='client.50368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:06:49.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: 1.9 scrub starts 2024-03-30T15:06:49.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:49 smithi134 ceph-mon[33714]: 1.9 scrub ok 2024-03-30T15:06:49.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:49.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:49.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:49.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:50.026 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:06:50.026 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:06:50.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:50.158 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:06:50.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-03-30T15:06:50.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:50.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:50.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:50.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.6 scrub starts 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.6 scrub ok 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.3 scrub starts 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.3 scrub ok 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.1 scrub starts 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: 1.1 scrub ok 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: from='client.50368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[29876]: pgmap v830: 238 pgs: 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:50.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.6 scrub starts 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.6 scrub ok 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.3 scrub starts 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.3 scrub ok 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.1 scrub starts 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: 1.1 scrub ok 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: from='client.50368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T15:06:50.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:50 smithi028 ceph-mon[41614]: pgmap v830: 238 pgs: 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:50.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:50.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:50.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:50.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:06:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.6 scrub starts 2024-03-30T15:06:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.6 scrub ok 2024-03-30T15:06:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:06:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.3 scrub starts 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.3 scrub ok 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.1 scrub starts 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: 1.1 scrub ok 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: from='client.50368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: osdmap e567: 8 total, 8 up, 8 in 2024-03-30T15:06:50.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:50 smithi134 ceph-mon[33714]: pgmap v830: 238 pgs: 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:50.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:50.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:50.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:50.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:50.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:50.993 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:50.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:51.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:51.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:51.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:51.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:51.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:51.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 1.7 scrub starts 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 1.7 scrub ok 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 1.8 scrub starts 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 1.8 scrub ok 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:06:51.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:51 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 1.7 scrub starts 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 1.7 scrub ok 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 1.8 scrub starts 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 1.8 scrub ok 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:06:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 1.7 scrub starts 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 1.7 scrub ok 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 1.8 scrub starts 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 1.8 scrub ok 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: osdmap e568: 8 total, 8 up, 8 in 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:06:51.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:51 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:06:51.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:51.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:51.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:51.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:51.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:52.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:52.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:52.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:52.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:52.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:52.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:52.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:52.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:52.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:52.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: 1.a scrub starts 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: 1.a scrub ok 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: pgmap v833: 270 pgs: 32 unknown, 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:52.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: 1.a scrub starts 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: 1.a scrub ok 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: pgmap v833: 270 pgs: 32 unknown, 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:52.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:52.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: 1.a scrub starts 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: 1.a scrub ok 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: osdmap e569: 8 total, 8 up, 8 in 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: pgmap v833: 270 pgs: 32 unknown, 238 active+clean; 8.2 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:06:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:52.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:52.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:52.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:53.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:53.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:53.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:53.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:53.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:53.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:53.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:53.649 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:53.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:53 smithi134 ceph-mon[33714]: 1.4 scrub starts 2024-03-30T15:06:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:53 smithi134 ceph-mon[33714]: 1.4 scrub ok 2024-03-30T15:06:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:53 smithi134 ceph-mon[33714]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T15:06:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[41614]: 1.4 scrub starts 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[41614]: 1.4 scrub ok 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[41614]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[29876]: 1.4 scrub starts 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[29876]: 1.4 scrub ok 2024-03-30T15:06:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:53 smithi028 ceph-mon[29876]: osdmap e570: 8 total, 8 up, 8 in 2024-03-30T15:06:53.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:53.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:53.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:54.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:54.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.080+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1866768599 >> v1:172.21.15.28:6790/0 conn(0x7f3fc010ea80 legacy=0x7f3fc0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.080+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1866768599 shutdown_connections 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.080+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1866768599 >> 172.21.15.28:0/1866768599 conn(0x7f3fc00761d0 msgr2=0x7f3fc00765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1866768599 shutdown_connections 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1866768599 wait complete. 2024-03-30T15:06:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 librados: starting msgr at 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 librados: starting objecter 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 Processor -- start 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- start start 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 librados: setting wanted keys 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 librados: calling monclient init 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3fc01a71d0 con 0x7f3fc010ae80 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3fc01a83f0 con 0x7f3fc010ea80 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc7ab1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3fc01a9630 con 0x7f3fc0108110 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc584d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3fc0108110 0x7f3fc0107570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58346/0 (socket says 172.21.15.28:58346) 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.081+0000 7f3fc584d700 1 -- 172.21.15.28:0/1971978825 learned_addr learned my addr 172.21.15.28:0/1971978825 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter ms_handle_connect 0x7f3fc0108110 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter ms_handle_connect 0x7f3fc010ae80 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203415431 0 0) 0x7f3fc01a9630 con 0x7f3fc0108110 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fac0031f0 con 0x7f3fc0108110 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter ms_handle_connect 0x7f3fc010ea80 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2912100266 0 0) 0x7f3fc01a71d0 con 0x7f3fc010ae80 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fc01a9630 con 0x7f3fc010ae80 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326036692 0 0) 0x7f3fc01a83f0 con 0x7f3fc010ea80 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fc01a71d0 con 0x7f3fc010ea80 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2387805565 0 0) 0x7f3fac0031f0 con 0x7f3fc0108110 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fc01a83f0 con 0x7f3fc0108110 2024-03-30T15:06:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3fbc0012d0 con 0x7f3fc0108110 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2504679851 0 0) 0x7f3fc01a9630 con 0x7f3fc010ae80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fac0031f0 con 0x7f3fc010ae80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 460942471 0 0) 0x7f3fc01a71d0 con 0x7f3fc010ea80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fc01a9630 con 0x7f3fc010ea80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3fb0001870 con 0x7f3fc010ae80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3fb8001bc0 con 0x7f3fc010ea80 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3653950882 0 0) 0x7f3fc01a83f0 con 0x7f3fc0108110 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.134:6789/0 conn(0x7f3fc010ea80 legacy=0x7f3fc01a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6789/0 conn(0x7f3fc010ae80 legacy=0x7f3fc01a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc01aa870 con 0x7f3fc0108110 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc01a7400 con 0x7f3fc0108110 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fc7ab1700 10 client.58367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3fc01a78c0 con 0x7f3fc0108110 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fc7ab1700 1 librados: init done 2024-03-30T15:06:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3fbc001670 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.082+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3fbc005d40 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.083+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3fa4005100 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3fbc01c3e0 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(570..570 src has 442..570) ==== 8655+0+0 (unknown 4272040326 0 0) 0x7f3fbc089880 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 osd_map(570..570 src has 442..570) 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 3 client.58367.objecter handle_osd_map got epochs [570,570] > 0 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 3 client.58367.objecter handle_osd_map decoding full epoch 570 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 20 client.58367.objecter dump_active .. 0 homeless 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v570) ==== 54+0+16 (unknown 1514002848 0 2420000192) 0x7f3fbc089c70 con 0x7f3fc0108110 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.084+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fac0703c0 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 1 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 1 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 0 1 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 4 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 1 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4004b50 con 0x7f3fa4004730 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4004730 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4004b50 con 0x7f3fa4004730 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 1: 0 ) 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 1 = system:0 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 4 2024-03-30T15:06:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.085+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 0 1 2024-03-30T15:06:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:06:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 2 2024-03-30T15:06:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 2 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa40053e0 osd=0 3 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 0 2 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 4 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 2 2024-03-30T15:06:54.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.105+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4007b00 con 0x7f3fa4004730 2024-03-30T15:06:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.107+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4007b00 con 0x7f3fa4004730 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.107+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.107+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.107+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 4 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.107+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 0 2 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:06:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 1 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 4 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 3 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4002160 con 0x7f3fa40042a0 2024-03-30T15:06:54.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.108+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa40042a0 2024-03-30T15:06:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.109+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4002160 con 0x7f3fa40042a0 2024-03-30T15:06:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.109+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 3: 0 ) 2024-03-30T15:06:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.109+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 3 = system:0 2024-03-30T15:06:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.109+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 4 2024-03-30T15:06:54.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.109+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 1 3 2024-03-30T15:06:54.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 4 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 4 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4004130 osd=1 3 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 1 4 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 4 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 4 2024-03-30T15:06:54.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.128+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4001600 con 0x7f3fa40042a0 2024-03-30T15:06:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4001600 con 0x7f3fa40042a0 2024-03-30T15:06:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 4 2024-03-30T15:06:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 1 4 2024-03-30T15:06:54.132 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 5 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 5 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 2 5 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 4 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 5 2024-03-30T15:06:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.130+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4008c30 con 0x7f3fa4011af0 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4011af0 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4008c30 con 0x7f3fa4011af0 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 5: 0 ) 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 5 = system:0 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 4 2024-03-30T15:06:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.131+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 2 5 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 6 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 6 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4008480 osd=2 3 2024-03-30T15:06:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 2 6 2024-03-30T15:06:54.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 4 2024-03-30T15:06:54.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 6 2024-03-30T15:06:54.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.148+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4007fc0 con 0x7f3fa4011af0 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4007fc0 con 0x7f3fa4011af0 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 4 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 2 6 2024-03-30T15:06:54.151 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 7 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 7 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 3 7 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 4 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 7 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4016170 con 0x7f3fa4015600 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.150+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4015600 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.151+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4016170 con 0x7f3fa4015600 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.151+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 7: 0 ) 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.151+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 7 = system:0 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.151+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 4 2024-03-30T15:06:54.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.151+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 3 7 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 8 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 8 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4008250 osd=3 3 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 3 8 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 4 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 8 2024-03-30T15:06:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.168+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4017ca0 con 0x7f3fa4015600 2024-03-30T15:06:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4017ca0 con 0x7f3fa4015600 2024-03-30T15:06:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 4 2024-03-30T15:06:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 3 8 2024-03-30T15:06:54.171 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:06:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 9 2024-03-30T15:06:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 9 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 4 9 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 4 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 9 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.169+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3fa401eb60 con 0x7f3fa401afc0 2024-03-30T15:06:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.170+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa401afc0 2024-03-30T15:06:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.171+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa401eb60 con 0x7f3fa401afc0 2024-03-30T15:06:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.171+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 9: 0 ) 2024-03-30T15:06:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.171+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 9 = system:0 2024-03-30T15:06:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.171+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 4 2024-03-30T15:06:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.171+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 4 9 2024-03-30T15:06:54.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 10 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 10 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4017c70 osd=4 3 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 4 10 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 4 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 10 2024-03-30T15:06:54.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.188+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4020f30 con 0x7f3fa401afc0 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4020f30 con 0x7f3fa401afc0 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 4 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 4 10 2024-03-30T15:06:54.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 11 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 11 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 5 11 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 4 2024-03-30T15:06:54.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 11 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.190+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4020280 con 0x7f3fa4020ce0 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4020ce0 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4020280 con 0x7f3fa4020ce0 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 11: 0 ) 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 11 = system:0 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 4 2024-03-30T15:06:54.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.191+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 5 11 2024-03-30T15:06:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 12 2024-03-30T15:06:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 12 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4021390 osd=5 3 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 5 12 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 4 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 12 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.208+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4020280 con 0x7f3fa4020ce0 2024-03-30T15:06:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4020280 con 0x7f3fa4020ce0 2024-03-30T15:06:54.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 4 2024-03-30T15:06:54.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 5 12 2024-03-30T15:06:54.212 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 13 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 13 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 6 13 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 4 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 13 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4017a80 con 0x7f3fa4023c00 2024-03-30T15:06:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.211+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4023c00 2024-03-30T15:06:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.212+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4017a80 con 0x7f3fa4023c00 2024-03-30T15:06:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.212+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 13: 0 ) 2024-03-30T15:06:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.212+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 13 = system:0 2024-03-30T15:06:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.212+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 4 2024-03-30T15:06:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.212+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 6 13 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.230+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.230+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 14 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 14 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4020a50 osd=6 3 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 6 14 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 4 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 14 2024-03-30T15:06:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.231+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4028f10 con 0x7f3fa4023c00 2024-03-30T15:06:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4028f10 con 0x7f3fa4023c00 2024-03-30T15:06:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 4 2024-03-30T15:06:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 6 14 2024-03-30T15:06:54.235 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:06:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:06:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 15 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 15 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 7 15 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 4 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 15 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.233+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3fa4029a70 con 0x7f3fa4029630 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 10 client.58367.objecter ms_handle_connect 0x7f3fa4029630 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3fa4029a70 con 0x7f3fa4029630 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 15: 0 ) 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 15 = system:0 2024-03-30T15:06:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 4 2024-03-30T15:06:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.234+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 7 15 2024-03-30T15:06:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 10 client.58367.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:06:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign -1 16 2024-03-30T15:06:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 20 client.58367.objecter _get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_remove -1 16 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 20 client.58367.objecter get_session s=0x7f3fa4028f10 osd=7 3 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 15 client.58367.objecter _session_command_op_assign 7 16 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 4 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 10 client.58367.objecter _send_command 16 2024-03-30T15:06:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.251+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3fa4029a70 con 0x7f3fa4029630 2024-03-30T15:06:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fb6ffd700 1 -- 172.21.15.28:0/1971978825 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3fa4029a70 con 0x7f3fa4029630 2024-03-30T15:06:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fb6ffd700 10 client.58367.objecter ms_dispatch 0x7f3fc0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:06:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fb6ffd700 10 client.58367.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:06:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fb6ffd700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 4 2024-03-30T15:06:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fb6ffd700 15 client.58367.objecter _session_command_op_remove 7 16 2024-03-30T15:06:54.255 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:06:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 librados: watch_flush enter 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 librados: watch_flush exit 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.0 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6801/3394323575 conn(0x7f3fa4004730 legacy=0x7f3fa4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa40053e0 osd=0 1 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.1 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6805/1850313547 conn(0x7f3fa40042a0 legacy=0x7f3fa400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4004130 osd=1 1 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.2 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6809/3451325043 conn(0x7f3fa4011af0 legacy=0x7f3fa4013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008480 osd=2 1 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.3 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6813/1506157250 conn(0x7f3fa4015600 legacy=0x7f3fa4015a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4008250 osd=3 1 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.4 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.134:6800/3063775514 conn(0x7f3fa401afc0 legacy=0x7f3fa401d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4017c70 osd=4 1 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.5 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.134:6804/3668151179 conn(0x7f3fa4020ce0 legacy=0x7f3fa401eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4021390 osd=5 1 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.6 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.134:6808/2135224194 conn(0x7f3fa4023c00 legacy=0x7f3fa4025ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4020a50 osd=6 1 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter close_session for osd.7 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.134:6812/3758190298 conn(0x7f3fa4029630 legacy=0x7f3fa4027740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter put_session s=0x7f3fa4028f10 osd=7 1 2024-03-30T15:06:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 20 client.58367.objecter shutdown clearing up homeless session... 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.253+0000 7f3fc7ab1700 10 client.58367.objecter successfully canceled tick 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3fac0703c0 legacy=0x7f3fac072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> v1:172.21.15.28:6790/0 conn(0x7f3fc0108110 legacy=0x7f3fc0107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc604e700 1 -- 172.21.15.28:0/1971978825 reap_dead start 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 shutdown_connections 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 >> 172.21.15.28:0/1971978825 conn(0x7f3fc00761d0 msgr2=0x7f3fc0109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 shutdown_connections 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 -- 172.21.15.28:0/1971978825 wait complete. 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:54.254+0000 7f3fc7ab1700 1 librados: shutdown 2024-03-30T15:06:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:06:54.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:06:54.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:54.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:54.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 5.6 deep-scrub starts 2024-03-30T15:06:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 5.6 deep-scrub ok 2024-03-30T15:06:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: pgmap v835: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1971978825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:06:54.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:54 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:06:54.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:54.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 5.6 deep-scrub starts 2024-03-30T15:06:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 5.6 deep-scrub ok 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: pgmap v835: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1971978825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 5.6 deep-scrub starts 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 5.6 deep-scrub ok 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T15:06:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: pgmap v835: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1971978825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:06:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:54 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:06:54.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:54.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:54.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:54.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:55.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:55.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:55.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:55.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:55.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:55.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:55.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:55.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:55.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 5.2 deep-scrub starts 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 5.2 deep-scrub ok 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T15:06:55.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 5.2 deep-scrub starts 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 5.2 deep-scrub ok 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T15:06:55.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:55 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:06:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:06:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:06:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T15:06:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T15:06:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 5.2 deep-scrub starts 2024-03-30T15:06:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 5.2 deep-scrub ok 2024-03-30T15:06:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T15:06:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T15:06:55.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:55 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:06:55.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:55.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:55.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:55.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:55.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:55.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:56.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:56.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:56.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:56.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:56.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:56.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:56.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:56.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:56.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 42.1b scrub starts 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 42.1b scrub ok 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 101.e scrub starts 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 101.e scrub ok 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: pgmap v836: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 42.8 scrub starts 2024-03-30T15:06:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:56 smithi134 ceph-mon[33714]: 42.8 scrub ok 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 42.1b scrub starts 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 42.1b scrub ok 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 101.e scrub starts 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 101.e scrub ok 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: pgmap v836: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 42.8 scrub starts 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[41614]: 42.8 scrub ok 2024-03-30T15:06:56.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 42.1b scrub starts 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 42.1b scrub ok 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 101.e scrub starts 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 101.e scrub ok 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: pgmap v836: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 42.8 scrub starts 2024-03-30T15:06:56.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:56 smithi028 ceph-mon[29876]: 42.8 scrub ok 2024-03-30T15:06:56.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:56.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:56.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:06:56.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.940+0000 7f232a443700 1 -- 172.21.15.28:0/808500719 >> v1:172.21.15.28:6789/0 conn(0x7f232410aed0 legacy=0x7f232410d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.940+0000 7f232a443700 1 -- 172.21.15.28:0/808500719 shutdown_connections 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.940+0000 7f232a443700 1 -- 172.21.15.28:0/808500719 >> 172.21.15.28:0/808500719 conn(0x7f23240fdaf0 msgr2=0x7f23240fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.940+0000 7f232a443700 1 -- 172.21.15.28:0/808500719 shutdown_connections 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.940+0000 7f232a443700 1 -- 172.21.15.28:0/808500719 wait complete. 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 librados: starting msgr at 2024-03-30T15:06:56.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 librados: starting objecter 2024-03-30T15:06:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 Processor -- start 2024-03-30T15:06:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 -- start start 2024-03-30T15:06:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 librados: setting wanted keys 2024-03-30T15:06:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 librados: calling monclient init 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23241a91b0 con 0x7f232410ea70 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23241aa3d0 con 0x7f232410aed0 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f232a443700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f23241ab610 con 0x7f2324102240 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f2323fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2324102240 0x7f23241073d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58366/0 (socket says 172.21.15.28:58366) 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f2323fff700 1 -- 172.21.15.28:0/3691219030 learned_addr learned my addr 172.21.15.28:0/3691219030 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f23217fa700 10 client.?.objecter ms_handle_connect 0x7f232410ea70 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f23217fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f23217fa700 10 client.?.objecter ms_handle_connect 0x7f2324102240 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.941+0000 7f23217fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 10 client.?.objecter ms_handle_connect 0x7f232410aed0 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060756683 0 0) 0x7f23241a91b0 con 0x7f232410ea70 2024-03-30T15:06:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f230c0031f0 con 0x7f232410ea70 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995349273 0 0) 0x7f23241ab610 con 0x7f2324102240 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23241a91b0 con 0x7f2324102240 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724050557 0 0) 0x7f23241aa3d0 con 0x7f232410aed0 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23241ab610 con 0x7f232410aed0 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1625334124 0 0) 0x7f23241a91b0 con 0x7f2324102240 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23241aa3d0 con 0x7f2324102240 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f231c001d30 con 0x7f2324102240 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3813974063 0 0) 0x7f230c0031f0 con 0x7f232410ea70 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23241a91b0 con 0x7f232410ea70 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2314001770 con 0x7f232410ea70 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1419006151 0 0) 0x7f23241ab610 con 0x7f232410aed0 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f230c0031f0 con 0x7f232410aed0 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2318004440 con 0x7f232410aed0 2024-03-30T15:06:56.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 308911955 0 0) 0x7f23241aa3d0 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 >> v1:172.21.15.134:6789/0 conn(0x7f232410aed0 legacy=0x7f23241a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 >> v1:172.21.15.28:6789/0 conn(0x7f232410ea70 legacy=0x7f23241a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23241ac850 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23241ab840 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f232a443700 10 client.58373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f23241abd00 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f232a443700 1 librados: init done 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f231c003ea0 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.942+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f231c006020 con 0x7f2324102240 2024-03-30T15:06:56.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.943+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2310005120 con 0x7f2324102240 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f231c003a00 con 0x7f2324102240 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(570..570 src has 442..570) ==== 8655+0+0 (unknown 4272040326 0 0) 0x7f231c089bf0 con 0x7f2324102240 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 10 client.58373.objecter ms_dispatch 0x7f2324001550 osd_map(570..570 src has 442..570) 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 3 client.58373.objecter handle_osd_map got epochs [570,570] > 0 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 3 client.58373.objecter handle_osd_map decoding full epoch 570 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.944+0000 7f23217fa700 20 client.58373.objecter dump_active .. 0 homeless 2024-03-30T15:06:56.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.945+0000 7f23217fa700 10 client.58373.objecter ms_handle_connect 0x7f230c070430 2024-03-30T15:06:56.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:56.949+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f231c056c10 con 0x7f2324102240 2024-03-30T15:06:56.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:56.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:57.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:57.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.171+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f2310005410 con 0x7f2324102240 2024-03-30T15:06:57.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:57.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:57.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.328+0000 7f23217fa700 1 -- 172.21.15.28:0/3691219030 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v571) ==== 113+0+0 (unknown 354161630 0 0) 0x7f231c05a660 con 0x7f2324102240 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.332+0000 7f232a443700 10 librados: watch_flush enter 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.332+0000 7f232a443700 10 librados: watch_flush exit 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.332+0000 7f232a443700 20 client.58373.objecter shutdown clearing up homeless session... 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.332+0000 7f232a443700 10 client.58373.objecter successfully canceled tick 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 >> v1:172.21.15.28:6800/3011431093 conn(0x7f230c070430 legacy=0x7f230c0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 >> v1:172.21.15.28:6790/0 conn(0x7f2324102240 legacy=0x7f23241073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 shutdown_connections 2024-03-30T15:06:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 >> 172.21.15.28:0/3691219030 conn(0x7f23240fdaf0 msgr2=0x7f232410ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:06:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 shutdown_connections 2024-03-30T15:06:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 -- 172.21.15.28:0/3691219030 wait complete. 2024-03-30T15:06:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:06:57.333+0000 7f232a443700 1 librados: shutdown 2024-03-30T15:06:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T15:06:57.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:57.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:57.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:06:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:06:57.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:57.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: 101.17 scrub starts 2024-03-30T15:06:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: 101.17 scrub ok 2024-03-30T15:06:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:06:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3691219030' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='client.58373 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: from='client.58373 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:06:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:57 smithi134 ceph-mon[33714]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T15:06:57.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: 101.17 scrub starts 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: 101.17 scrub ok 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3691219030' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='client.58373 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: from='client.58373 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[29876]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: 101.17 scrub starts 2024-03-30T15:06:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: 101.17 scrub ok 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3691219030' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='client.58373 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: from='client.58373 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:06:57.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:57 smithi028 ceph-mon[41614]: osdmap e571: 8 total, 8 up, 8 in 2024-03-30T15:06:57.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:57.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:57.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:57.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:57.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:57.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:58.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:06:58.056 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:06:58.056 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(14.1d epoch 114/111 rep_tid 87 entries 114'2 (0'0) error 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head by client.24853.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 541.88091052699997, 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023963393999999999, 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173901+0000", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173902+0000", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7300000000000002e-07 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173911+0000", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8720000000000008e-06 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173929+0000", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8666000000000001e-05 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.173946+0000", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6724000000000001e-05 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.193026+0000", 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019079608000000001 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.197865+0000", 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048391509999999999 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(15.1 epoch 114/111 rep_tid 80 entries 114'2 (0'0) error 15:8386110d:test-rados-api-smithi028-93086-4::foo:head by client.24856.0:7 0.000000 -17 [r=-17+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:06:58.060 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 541.84744166300004, 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047719461999999997, 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207370+0000", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207371+0000", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4999999999999998e-07 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207383+0000", 2024-03-30T15:06:58.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2333e-05 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207399+0000", 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6235999999999998e-05 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.207417+0000", 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7303999999999999e-05 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.247826+0000", 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040409395000000001 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.255090+0000", 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072638440000000002 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.34820.0:20 9.8 e115/114 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head v 115'1, mlcod=0'0)", 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:06:58.064 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 541.77366258999996, 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021284207999999999, 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281149+0000", 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281150+0000", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0800000000000001e-07 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281157+0000", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5959999999999998e-06 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281163+0000", 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4249999999999997e-06 2024-03-30T15:06:58.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.281170+0000", 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2110000000000001e-06 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282755+0000", 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001584577 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.282773+0000", 2024-03-30T15:06:58.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8474e-05 2024-03-30T15:06:58.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302395+0000", 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019622245999999999 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302434+0000", 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8371000000000002e-05 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.24853.0:11 14.15 e115/111 14:ada94957:test-rados-api-smithi028-93116-5::foo:head v 115'3, mlcod=115'1)", 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 540.68204360799996, 2024-03-30T15:06:58.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038572226000000001, 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372768+0000", 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372769+0000", 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0950000000000001e-06 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372783+0000", 2024-03-30T15:06:58.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3463e-05 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372792+0000", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.448e-06 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.372802+0000", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7890000000000006e-06 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387040+0000", 2024-03-30T15:06:58.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014237757 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.387062+0000", 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1857000000000001e-05 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411293+0000", 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024231447 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:57.411341+0000", 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7370000000000002e-05 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32242.0:2 120.6 e139/138 120:602f83fe:::foo:head v 139'1, mlcod=0'0)", 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 515.97392589799995, 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029704504999999999, 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.077 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080886+0000", 2024-03-30T15:06:58.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8999999999999998e-07 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080897+0000", 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0572e-05 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080903+0000", 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4659999999999996e-06 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.080913+0000", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.431e-06 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097177+0000", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016263732999999999 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097197+0000", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0520999999999999e-05 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110549+0000", 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013351633 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110591+0000", 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1859e-05 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(115.6s1 139/136 ECSubWrite(tid=562, reqid=client.27400.0:256, at_version=139'28, trim_to=0'0, roll_forward_to=138'27))", 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:06:58.081 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 515.97119817700002, 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026921001, 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083614+0000", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7399999999999999e-07 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083633+0000", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8479999999999999e-05 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083641+0000", 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7190000000000007e-06 2024-03-30T15:06:58.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.083652+0000", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0902e-05 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097409+0000", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013756496 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.097418+0000", 2024-03-30T15:06:58.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110508+0000", 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013089735999999999 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110535+0000", 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7146000000000001e-05 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.32239.0:2 119.1b e139/137 119:d96a01e0:::testobj:head v 139'1, mlcod=0'0)", 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 515.94425452099995, 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024229506000000001, 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110557+0000", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110558+0000", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3299999999999998e-07 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110568+0000", 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0029e-05 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110575+0000", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3660000000000003e-06 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.110585+0000", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3400000000000004e-06 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111135+0000", 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055055900000000005 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.111154+0000", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9269000000000001e-05 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134746+0000", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023591918999999999 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.134787+0000", 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0691000000000001e-05 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:12 104.1c e141/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:head v 141'1, mlcod=0'0)", 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 513.65056712900002, 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041302041999999997, 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:06:58.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404245+0000", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404246+0000", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5799999999999999e-06 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404257+0000", 2024-03-30T15:06:58.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193999999999999e-05 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404267+0000", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0215e-05 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.404276+0000", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4520000000000003e-06 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433295+0000", 2024-03-30T15:06:58.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029018773000000001 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.433316+0000", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0636999999999999e-05 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445508+0000", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012192305000000001 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.445547+0000", 2024-03-30T15:06:58.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8886e-05 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.8 epoch 142/140 rep_tid 1005 entries 142'13 (0'0) error 123:1179ae53:::test_441:head by client.44099.0:448 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 142'11)", 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 511.23620117600001, 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038477480000000001, 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818611+0000", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818612+0000", 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.37e-07 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.818622+0000", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0477e-05 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851833+0000", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033211076999999999 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851867+0000", 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3874999999999997e-05 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.851967+0000", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100182 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:26.857088+0000", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051211319999999996 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(123.1f epoch 143/140 rep_tid 611 entries 143'25 (0'0) error 123:fbee941d:::test_708:head by client.44099.0:715 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 143'23)", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:06:58.097 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 510.21904674000001, 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.114938996, 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835765+0000", 2024-03-30T15:06:58.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835768+0000", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.892e-06 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835782+0000", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3456e-05 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835803+0000", 2024-03-30T15:06:58.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.101e-05 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835820+0000", 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-05 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.935436+0000", 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099616275000000004 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.950704+0000", 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015268273000000001 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1491 104.1c e144/131 104:3950bad6:test-rados-api-smithi028-93441-8::foo:5 v 144'6, mlcod=143'3)", 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 510.15257660200001, 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022363271000000001, 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.101 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902235+0000", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902236+0000", 2024-03-30T15:06:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5999999999999999e-07 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902245+0000", 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9339999999999995e-06 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902253+0000", 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6200000000000005e-06 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.902263+0000", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1749999999999994e-06 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903266+0000", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010038880000000001 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.903291+0000", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4907999999999999e-05 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924561+0000", 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021269507 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.924599+0000", 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7778999999999999e-05 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1533 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:06:58.105 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 509.135948042, 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046703298999999997, 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918864+0000", 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.918865+0000", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3909999999999999e-06 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955732+0000", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036866290000000003 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955761+0000", 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9051e-05 2024-03-30T15:06:58.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955774+0000", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3448999999999999e-05 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955856+0000", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1716000000000002e-05 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.955881+0000", 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5361e-05 2024-03-30T15:06:58.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965519+0000", 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096378399999999999 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.965567+0000", 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8201000000000001e-05 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1535 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 509.09633849099998, 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055252190999999999, 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958473+0000", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:28.958476+0000", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2699999999999999e-06 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004606+0000", 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046129950000000003 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004628+0000", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2775000000000001e-05 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004638+0000", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3659999999999998e-06 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004730+0000", 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2026000000000003e-05 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.004753+0000", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2898e-05 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013677+0000", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089245069999999999 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.013726+0000", 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8399000000000002e-05 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1536 11.a e144/114 -13:5269a482:::temp_11.a_0_24304_1:head v 0'0, mlcod=116'1001)", 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 509.04711698699998, 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029420031999999999, 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007695+0000", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.007696+0000", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4410000000000001e-06 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024530+0000", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016833292 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024541+0000", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1619e-05 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024550+0000", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2840000000000006e-06 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024603+0000", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3566999999999997e-05 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.024628+0000", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4326999999999999e-05 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037063+0000", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012435493000000001 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.037115+0000", 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2009000000000003e-05 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.3 epoch 144/143 rep_tid 1544 entries 144'5 (0'0) error 130:c0c9adc9:::test_234:head by client.46235.0:241 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 144'3)", 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.91781334000001, 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031151346999999999, 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:06:58.118 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.136999+0000", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137001+0000", 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1840000000000002e-06 2024-03-30T15:06:58.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.137017+0000", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5866999999999999e-05 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150695+0000", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013677962 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150725+0000", 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.042e-05 2024-03-30T15:06:58.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.150878+0000", 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015338299999999999 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:29.168150+0000", 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017271531 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1599 127.e e147/142 127:7056b563:test-rados-api-smithi028-93253-10::foo:head v 147'1, mlcod=0'0)", 2024-03-30T15:06:58.121 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 506.66718818800001, 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075902677000000002, 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387624+0000", 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7899999999999999e-07 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387631+0000", 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3709999999999996e-06 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387636+0000", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2689999999999999e-06 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.387641+0000", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2129999999999999e-06 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.388146+0000", 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00050455300000000003 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.388164+0000", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8002000000000001e-05 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463488+0000", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075323946000000003 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463526+0000", 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8943999999999998e-05 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:23 104.7 e147/131 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head v 147'3, mlcod=0'0)", 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 506.66426503399998, 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072716648999999994, 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390547+0000", 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390548+0000", 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1399999999999997e-07 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390561+0000", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3682e-05 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390568+0000", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7220000000000002e-06 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390577+0000", 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7150000000000004e-06 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390748+0000", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000170817 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.390769+0000", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1851000000000001e-05 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463222+0000", 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072453039999999996 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.463264+0000", 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1208000000000003e-05 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.25696.0:25 104.7 e149/131 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head v 149'4, mlcod=146'1)", 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 504.79407755300002, 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027373235999999999, 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260734+0000", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260735+0000", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3999999999999997e-07 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.260741+0000", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4799999999999998e-06 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270493+0000", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097522140000000004 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270511+0000", 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7096e-05 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270773+0000", 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000262188 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.270797+0000", 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4264999999999999e-05 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.288065+0000", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017268426 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.288108+0000", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2227000000000001e-05 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:63 111.18 e162/134 111:1e07d18e:test-rados-api-smithi028-93525-14::foo:head v 162'1, mlcod=0'0)", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 491.85978441999998, 2024-03-30T15:06:58.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020132151000000001, 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195028+0000", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6100000000000002e-07 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195035+0000", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7700000000000004e-06 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195041+0000", 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.964e-06 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195049+0000", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9390000000000003e-06 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195528+0000", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047816500000000001 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.195550+0000", 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1985999999999999e-05 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.215114+0000", 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019564406999999999 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.215160+0000", 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5658999999999998e-05 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.0.0:954 101.4 e201/130 101:24de1dd9:test-rados-api-smithi028-93253-14::foo38:6 v 201'40, mlcod=167'34)", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 452.83619634299998, 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025813787000000001, 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218616+0000", 2024-03-30T15:06:58.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218617+0000", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.336e-06 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218625+0000", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7419999999999997e-06 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218639+0000", 2024-03-30T15:06:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3995999999999999e-05 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.218646+0000", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9009999999999997e-06 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.228251+0000", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096057060000000003 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.228270+0000", 2024-03-30T15:06:58.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8728000000000001e-05 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.244397+0000", 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016126699000000001 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:25.244429+0000", 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2678999999999997e-05 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:06:58.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:06:58.144 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:06:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:58.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:58.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:58.380 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:06:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:06:58.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:58.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:58.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:58.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:58.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:06:58.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: 42.16 scrub starts 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: 42.16 scrub ok 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: pgmap v838: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:58.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:58 smithi134 ceph-mon[33714]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T15:06:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: 42.16 scrub starts 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: 42.16 scrub ok 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: pgmap v838: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:58.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[29876]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: 42.16 scrub starts 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: 42.16 scrub ok 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: pgmap v838: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:06:58.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:58 smithi028 ceph-mon[41614]: osdmap e572: 8 total, 8 up, 8 in 2024-03-30T15:06:58.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:58.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:58.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:06:58.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:58.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:06:59.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:59.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:59.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:59.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:06:59.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:06:59.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.7 scrub starts 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.7 scrub ok 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:06:59.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.0 scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.0 scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.12 scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[41614]: 42.12 scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.7 scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.7 scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:06:59.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:06:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:06:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.0 scrub starts 2024-03-30T15:06:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.0 scrub ok 2024-03-30T15:06:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.12 scrub starts 2024-03-30T15:06:59.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:06:59 smithi028 ceph-mon[29876]: 42.12 scrub ok 2024-03-30T15:06:59.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:06:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:59.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:06:59.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.7 scrub starts 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.7 scrub ok 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:06:59.640 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.0 scrub starts 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.0 scrub ok 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.12 scrub starts 2024-03-30T15:06:59.641 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:06:59 smithi134 ceph-mon[33714]: 42.12 scrub ok 2024-03-30T15:06:59.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:06:59.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:06:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:06:59.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:06:59.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:06:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:06:59.861 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:06:59.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:06:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:06:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:06:59.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:00.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:00.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:00.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:00.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:00.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-03-30T15:07:00.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:00.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:00.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:07:00.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:07:00.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[41614]: pgmap v840: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:00.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:07:00.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:07:00.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:00 smithi028 ceph-mon[29876]: pgmap v840: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:00.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:00.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:00.540 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:00 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:07:00.540 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:00 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:07:00.540 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:00 smithi134 ceph-mon[33714]: pgmap v840: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:00.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:00.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.665+0000 7ff157695700 1 -- 172.21.15.28:0/3207947483 >> v1:172.21.15.28:6789/0 conn(0x7ff150101130 legacy=0x7ff150101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.665+0000 7ff157695700 1 -- 172.21.15.28:0/3207947483 shutdown_connections 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.665+0000 7ff157695700 1 -- 172.21.15.28:0/3207947483 >> 172.21.15.28:0/3207947483 conn(0x7ff1500fc9e0 msgr2=0x7ff1500fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.665+0000 7ff157695700 1 -- 172.21.15.28:0/3207947483 shutdown_connections 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.665+0000 7ff157695700 1 -- 172.21.15.28:0/3207947483 wait complete. 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 librados: starting msgr at 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 librados: starting objecter 2024-03-30T15:07:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 Processor -- start 2024-03-30T15:07:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 -- start start 2024-03-30T15:07:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 librados: setting wanted keys 2024-03-30T15:07:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 librados: calling monclient init 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1501a6e50 con 0x7ff150101130 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1501a8090 con 0x7ff150109dc0 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff157695700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1501a92d0 con 0x7ff15010d960 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff155c32700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff15010d960 0x7ff1501a5510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56260/0 (socket says 172.21.15.28:56260) 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.666+0000 7ff155c32700 1 -- 172.21.15.28:0/1929484783 learned_addr learned my addr 172.21.15.28:0/1929484783 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter ms_handle_connect 0x7ff150109dc0 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter ms_handle_connect 0x7ff15010d960 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187722626 0 0) 0x7ff1501a8090 con 0x7ff150109dc0 2024-03-30T15:07:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13c0032d0 con 0x7ff150109dc0 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236041189 0 0) 0x7ff1501a92d0 con 0x7ff15010d960 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1501a8090 con 0x7ff15010d960 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter ms_handle_connect 0x7ff150101130 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4146122496 0 0) 0x7ff13c0032d0 con 0x7ff150109dc0 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1501a92d0 con 0x7ff150109dc0 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff140001870 con 0x7ff150109dc0 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2554063264 0 0) 0x7ff1501a8090 con 0x7ff15010d960 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.667+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff13c0032d0 con 0x7ff15010d960 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff148001c50 con 0x7ff15010d960 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1226218105 0 0) 0x7ff1501a6e50 con 0x7ff150101130 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1501a8090 con 0x7ff150101130 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3600955984 0 0) 0x7ff1501a92d0 con 0x7ff150109dc0 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 >> v1:172.21.15.28:6790/0 conn(0x7ff15010d960 legacy=0x7ff1501a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 >> v1:172.21.15.28:6789/0 conn(0x7ff150101130 legacy=0x7ff15006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1501aa510 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1501a9500 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff157695700 10 client.50386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1501a99c0 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff157695700 1 librados: init done 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.668+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff140003b20 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.669+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff140004a20 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.670+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff134005120 con 0x7ff150109dc0 2024-03-30T15:07:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.670+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff140003680 con 0x7ff150109dc0 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(572..572 src has 442..572) ==== 8655+0+0 (unknown 3123995570 0 0) 0x7ff1400885d0 con 0x7ff150109dc0 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 10 client.50386.objecter ms_dispatch 0x7ff150001550 osd_map(572..572 src has 442..572) 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 3 client.50386.objecter handle_osd_map got epochs [572,572] > 0 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 3 client.50386.objecter handle_osd_map decoding full epoch 572 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 20 client.50386.objecter dump_active .. 0 homeless 2024-03-30T15:07:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.671+0000 7ff1467fc700 10 client.50386.objecter ms_handle_connect 0x7ff13c0705b0 2024-03-30T15:07:00.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.675+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff1400555d0 con 0x7ff150109dc0 2024-03-30T15:07:00.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:00.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:00.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:00.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:00.911+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff134005410 con 0x7ff150109dc0 2024-03-30T15:07:00.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:00.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:00.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:00.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:01.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:01.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:01.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.406+0000 7ff1467fc700 1 -- 172.21.15.28:0/1929484783 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v573) ==== 87+0+0 (unknown 1760515152 0 0) 0x7ff140059020 con 0x7ff150109dc0 2024-03-30T15:07:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 10 librados: watch_flush enter 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 10 librados: watch_flush exit 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 20 client.50386.objecter shutdown clearing up homeless session... 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 10 client.50386.objecter successfully canceled tick 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff13c0705b0 legacy=0x7ff13c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 >> v1:172.21.15.134:6789/0 conn(0x7ff150109dc0 legacy=0x7ff1501a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 shutdown_connections 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 >> 172.21.15.28:0/1929484783 conn(0x7ff1500fc9e0 msgr2=0x7ff15010ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 shutdown_connections 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 -- 172.21.15.28:0/1929484783 wait complete. 2024-03-30T15:07:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:01.408+0000 7ff157695700 1 librados: shutdown 2024-03-30T15:07:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:07:01.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:01.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:01.634 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:07:01.634 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:07:01.634 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:07:01.635 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1929484783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: from='client.50386 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T15:07:01.635 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:01 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T15:07:01.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:01.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:01.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:07:01.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1929484783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: from='client.50386 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T15:07:01.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T15:07:01.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:07:01.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:07:01.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:07:01.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1929484783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: from='client.50386 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T15:07:01.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:01 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T15:07:01.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:01.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:01.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:01.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:01.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:02.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:02.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:02.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:02.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:07:02.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:02.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:02.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:02.538 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:02.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:02.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:02.659 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:07:02.391+0000 7fec997ef700 -1 snap_mapper.add_oid found existing snaps mapped on 101:2df7255a:test-rados-api-smithi028-93253-97::foo:21, removing 2024-03-30T15:07:02.659 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T15:07:02.391+0000 7f99a6f55700 -1 snap_mapper.add_oid found existing snaps mapped on 101:2df7255a:test-rados-api-smithi028-93253-97::foo:21, removing 2024-03-30T15:07:02.660 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:02.660 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-mon[33714]: from='client.50386 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:02.660 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-mon[33714]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T15:07:02.660 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:02 smithi134 ceph-mon[33714]: pgmap v842: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:07:02.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:02.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:02.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[29876]: from='client.50386 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[29876]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[29876]: pgmap v842: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[41614]: from='client.50386 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[41614]: osdmap e573: 8 total, 8 up, 8 in 2024-03-30T15:07:02.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-mon[41614]: pgmap v842: 270 pgs: 270 active+clean; 8.4 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:07:02.723 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:07:02 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T15:07:02.391+0000 7f2e59544700 -1 snap_mapper.add_oid found existing snaps mapped on 101:2df7255a:test-rados-api-smithi028-93253-97::foo:21, removing 2024-03-30T15:07:02.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:02.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:02.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:03.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:03.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:03.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.392+0000 7f72af676700 1 -- 172.21.15.28:0/1194489603 >> v1:172.21.15.28:6790/0 conn(0x7f72a810b020 legacy=0x7f72a810d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.392+0000 7f72af676700 1 -- 172.21.15.28:0/1194489603 shutdown_connections 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.392+0000 7f72af676700 1 -- 172.21.15.28:0/1194489603 >> 172.21.15.28:0/1194489603 conn(0x7f72a80fc9e0 msgr2=0x7f72a80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 -- 172.21.15.28:0/1194489603 shutdown_connections 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 -- 172.21.15.28:0/1194489603 wait complete. 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 librados: starting msgr at 2024-03-30T15:07:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 librados: starting objecter 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 Processor -- start 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 -- start start 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 librados: setting wanted keys 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.393+0000 7f72af676700 1 librados: calling monclient init 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72af676700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72a81a6fc0 con 0x7f72a810eb30 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72af676700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72a81a81e0 con 0x7f72a8103210 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72af676700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72a81a9420 con 0x7f72a810b020 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72adc13700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72a810eb30 0x7f72a81a5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58214/0 (socket says 172.21.15.28:58214) 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72acc11700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f72a810b020 0x7f72a81a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56282/0 (socket says 172.21.15.28:56282) 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f72adc13700 1 -- 172.21.15.28:0/1669408920 learned_addr learned my addr 172.21.15.28:0/1669408920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter ms_handle_connect 0x7f72a810eb30 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter ms_handle_connect 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter ms_handle_connect 0x7f72a8103210 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331718329 0 0) 0x7f72a81a9420 con 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7294003aa0 con 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783578364 0 0) 0x7f72a81a6fc0 con 0x7f72a810eb30 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a81a9420 con 0x7f72a810eb30 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123818206 0 0) 0x7f72a81a81e0 con 0x7f72a8103210 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a81a6fc0 con 0x7f72a8103210 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158264021 0 0) 0x7f7294003aa0 con 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72a81a81e0 con 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7298004660 con 0x7f72a810b020 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3977228470 0 0) 0x7f72a81a9420 con 0x7f72a810eb30 2024-03-30T15:07:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7294003aa0 con 0x7f72a810eb30 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.394+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72a0001b70 con 0x7f72a810eb30 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4273567504 0 0) 0x7f72a81a6fc0 con 0x7f72a8103210 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72a81a9420 con 0x7f72a8103210 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72a40013d0 con 0x7f72a8103210 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3542649669 0 0) 0x7f72a81a81e0 con 0x7f72a810b020 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.134:6789/0 conn(0x7f72a8103210 legacy=0x7f72a810a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6789/0 conn(0x7f72a810eb30 legacy=0x7f72a81a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a81aa660 con 0x7f72a810b020 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81a9650 con 0x7f72a810b020 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f72af676700 10 client.58385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f72a81a9b10 con 0x7f72a810b020 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f72af676700 1 librados: init done 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7298001b10 con 0x7f72a810b020 2024-03-30T15:07:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.395+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7298004c70 con 0x7f72a810b020 2024-03-30T15:07:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.396+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f728c005100 con 0x7f72a810b020 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.396+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7298003860 con 0x7f72a810b020 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(573..573 src has 442..573) ==== 8655+0+0 (unknown 1487491819 0 0) 0x7f7298089740 con 0x7f72a810b020 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 osd_map(573..573 src has 442..573) 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 3 client.58385.objecter handle_osd_map got epochs [573,573] > 0 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 3 client.58385.objecter handle_osd_map decoding full epoch 573 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 20 client.58385.objecter dump_active .. 0 homeless 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v573) ==== 54+0+16 (unknown 1857935615 0 2420000192) 0x7f729808bba0 con 0x7f72a810b020 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f72940704d0 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 10 client.58385.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 1 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 1 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 0 1 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 4 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 10 client.58385.objecter _send_command 1 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f728c004b50 con 0x7f728c004730 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.397+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c004730 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.398+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c004b50 con 0x7f728c004730 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.398+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.398+0000 7f729e7fc700 10 client.58385.objecter _finish_command 1 = system:0 2024-03-30T15:07:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.398+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 4 2024-03-30T15:07:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.398+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 0 1 2024-03-30T15:07:03.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:07:03.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1669408920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1669408920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 10 client.58385.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 2 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 2 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c0053e0 osd=0 3 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 0 2 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 4 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 10 client.58385.objecter _send_command 2 2024-03-30T15:07:03.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.428+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c00bf10 con 0x7f728c004730 2024-03-30T15:07:03.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c00bf10 con 0x7f728c004730 2024-03-30T15:07:03.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f729e7fc700 10 client.58385.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 4 2024-03-30T15:07:03.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 0 2 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 10 client.58385.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 3 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 3 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 1 3 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 4 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 10 client.58385.objecter _send_command 3 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.430+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f728c002160 con 0x7f728c0042a0 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c0042a0 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c002160 con 0x7f728c0042a0 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 10 client.58385.objecter _finish_command 3 = system:0 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 4 2024-03-30T15:07:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.431+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 1 3 2024-03-30T15:07:03.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 10 client.58385.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 4 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 4 2024-03-30T15:07:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c004130 osd=1 3 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 1 4 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 4 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 10 client.58385.objecter _send_command 4 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.448+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c001600 con 0x7f728c0042a0 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c001600 con 0x7f728c0042a0 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 10 client.58385.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 4 2024-03-30T15:07:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 1 4 2024-03-30T15:07:03.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 10 client.58385.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 5 2024-03-30T15:07:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 5 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 2 5 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 4 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 10 client.58385.objecter _send_command 5 2024-03-30T15:07:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f728c015420 con 0x7f728c00cda0 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.450+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c00cda0 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.451+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c015420 con 0x7f728c00cda0 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.451+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.451+0000 7f729e7fc700 10 client.58385.objecter _finish_command 5 = system:0 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.451+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 4 2024-03-30T15:07:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.451+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 2 5 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 10 client.58385.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 6 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 6 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c001a00 osd=2 3 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 2 6 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 4 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 10 client.58385.objecter _send_command 6 2024-03-30T15:07:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.469+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c00c430 con 0x7f728c00cda0 2024-03-30T15:07:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c00c430 con 0x7f728c00cda0 2024-03-30T15:07:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f729e7fc700 10 client.58385.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 4 2024-03-30T15:07:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 2 6 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 10 client.58385.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 7 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 7 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 3 7 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 4 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 10 client.58385.objecter _send_command 7 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.472+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f728c016120 con 0x7f728c0155d0 2024-03-30T15:07:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c0155d0 2024-03-30T15:07:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:03.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c016120 con 0x7f728c0155d0 2024-03-30T15:07:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:03.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 10 client.58385.objecter _finish_command 7 = system:0 2024-03-30T15:07:03.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 4 2024-03-30T15:07:03.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.473+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 3 7 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 10 client.58385.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 8 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 8 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c00c680 osd=3 3 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 3 8 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 4 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 10 client.58385.objecter _send_command 8 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.502+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c016ad0 con 0x7f728c0155d0 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c016ad0 con 0x7f728c0155d0 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f729e7fc700 10 client.58385.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 4 2024-03-30T15:07:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 3 8 2024-03-30T15:07:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f72af676700 10 client.58385.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.503+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 9 2024-03-30T15:07:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 9 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 4 9 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 4 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 10 client.58385.objecter _send_command 9 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f728c01ec40 con 0x7f728c01afe0 2024-03-30T15:07:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.504+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c01afe0 2024-03-30T15:07:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.505+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c01ec40 con 0x7f728c01afe0 2024-03-30T15:07:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.505+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.505+0000 7f729e7fc700 10 client.58385.objecter _finish_command 9 = system:0 2024-03-30T15:07:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.505+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 4 2024-03-30T15:07:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.505+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 4 9 2024-03-30T15:07:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 10 client.58385.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 10 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 10 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c016ad0 osd=4 3 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 4 10 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 4 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 10 client.58385.objecter _send_command 10 2024-03-30T15:07:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.521+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c01ee20 con 0x7f728c01afe0 2024-03-30T15:07:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.523+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c01ee20 con 0x7f728c01afe0 2024-03-30T15:07:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.523+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.523+0000 7f729e7fc700 10 client.58385.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.523+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 4 2024-03-30T15:07:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.523+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 4 10 2024-03-30T15:07:03.525 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 10 client.58385.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 11 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 11 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 5 11 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 4 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 10 client.58385.objecter _send_command 11 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f728c0201b0 con 0x7f728c01f540 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.524+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c01f540 2024-03-30T15:07:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.525+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c0201b0 con 0x7f728c01f540 2024-03-30T15:07:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.525+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.525+0000 7f729e7fc700 10 client.58385.objecter _finish_command 11 = system:0 2024-03-30T15:07:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.525+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 4 2024-03-30T15:07:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.525+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 5 11 2024-03-30T15:07:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 10 client.58385.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 12 2024-03-30T15:07:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 12 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c01ee20 osd=5 3 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 5 12 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 4 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 10 client.58385.objecter _send_command 12 2024-03-30T15:07:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.543+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c020430 con 0x7f728c01f540 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7298056760 con 0x7f728c01f540 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f729e7fc700 10 client.58385.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 4 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 5 12 2024-03-30T15:07:03.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 10 client.58385.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 13 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 13 2024-03-30T15:07:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 6 13 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 4 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 10 client.58385.objecter _send_command 13 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.545+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f728c027800 con 0x7f728c023b20 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c023b20 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c027800 con 0x7f728c023b20 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 10 client.58385.objecter _finish_command 13 = system:0 2024-03-30T15:07:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 4 2024-03-30T15:07:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.546+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 6 13 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 10 client.58385.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 14 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 14 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c0201b0 osd=6 3 2024-03-30T15:07:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 6 14 2024-03-30T15:07:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 4 2024-03-30T15:07:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 10 client.58385.objecter _send_command 14 2024-03-30T15:07:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.564+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c027800 con 0x7f728c023b20 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c027800 con 0x7f728c023b20 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f729e7fc700 10 client.58385.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 4 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 6 14 2024-03-30T15:07:03.567 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 10 client.58385.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 15 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 15 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 7 15 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 4 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 10 client.58385.objecter _send_command 15 2024-03-30T15:07:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.566+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f728c028b20 con 0x7f728c0297b0 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.567+0000 7f729e7fc700 10 client.58385.objecter ms_handle_connect 0x7f728c0297b0 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.568+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f728c028b20 con 0x7f728c0297b0 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.568+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.568+0000 7f729e7fc700 10 client.58385.objecter _finish_command 15 = system:0 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.568+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 4 2024-03-30T15:07:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.568+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 7 15 2024-03-30T15:07:03.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 10 client.58385.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign -1 16 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 20 client.58385.objecter _get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 15 client.58385.objecter _session_command_op_remove -1 16 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 20 client.58385.objecter get_session s=0x7f728c027840 osd=7 3 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 15 client.58385.objecter _session_command_op_assign 7 16 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 4 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 10 client.58385.objecter _send_command 16 2024-03-30T15:07:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.595+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f728c028b20 con 0x7f728c0297b0 2024-03-30T15:07:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f729e7fc700 1 -- 172.21.15.28:0/1669408920 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f728c028b20 con 0x7f728c0297b0 2024-03-30T15:07:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f729e7fc700 10 client.58385.objecter ms_dispatch 0x7f72a8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f729e7fc700 10 client.58385.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f729e7fc700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 4 2024-03-30T15:07:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f729e7fc700 15 client.58385.objecter _session_command_op_remove 7 16 2024-03-30T15:07:03.599 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 librados: watch_flush enter 2024-03-30T15:07:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 librados: watch_flush exit 2024-03-30T15:07:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.0 2024-03-30T15:07:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6801/3394323575 conn(0x7f728c004730 legacy=0x7f728c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0053e0 osd=0 1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6805/1850313547 conn(0x7f728c0042a0 legacy=0x7f728c00f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c004130 osd=1 1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.2 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6809/3451325043 conn(0x7f728c00cda0 legacy=0x7f728c013b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c001a00 osd=2 1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.3 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6813/1506157250 conn(0x7f728c0155d0 legacy=0x7f728c015a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c00c680 osd=3 1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.4 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.134:6800/3063775514 conn(0x7f728c01afe0 legacy=0x7f728c01d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72adc13700 1 -- 172.21.15.28:0/1669408920 reap_dead start 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c016ad0 osd=4 1 2024-03-30T15:07:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 10 client.58385.objecter close_session for osd.5 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.597+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.134:6804/3668151179 conn(0x7f728c01f540 legacy=0x7f728c01f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72adc13700 1 -- 172.21.15.28:0/1669408920 reap_dead start 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c01ee20 osd=5 1 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 10 client.58385.objecter close_session for osd.6 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72adc13700 1 -- 172.21.15.28:0/1669408920 reap_dead start 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.134:6808/2135224194 conn(0x7f728c023b20 legacy=0x7f728c025f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c0201b0 osd=6 1 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 10 client.58385.objecter close_session for osd.7 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.134:6812/3758190298 conn(0x7f728c0297b0 legacy=0x7f728c029c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 20 client.58385.objecter put_session s=0x7f728c027840 osd=7 1 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 20 client.58385.objecter shutdown clearing up homeless session... 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 10 client.58385.objecter successfully canceled tick 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6800/3011431093 conn(0x7f72940704d0 legacy=0x7f7294072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> v1:172.21.15.28:6790/0 conn(0x7f72a810b020 legacy=0x7f72a81a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 shutdown_connections 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 >> 172.21.15.28:0/1669408920 conn(0x7f72a80fc9e0 msgr2=0x7f72a80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 shutdown_connections 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 -- 172.21.15.28:0/1669408920 wait complete. 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:03.598+0000 7f72af676700 1 librados: shutdown 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:03.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:07:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-96"}]: dispatch 2024-03-30T15:07:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1669408920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:03.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:03.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:03.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:03.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:03.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:03.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:04.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:04.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:04.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:04.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:04.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:04.442 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:04.442 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:04.442 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:04 smithi134 ceph-mon[33714]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T15:07:04.442 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:04 smithi134 ceph-mon[33714]: pgmap v844: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T15:07:04.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:04.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:04.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:04 smithi028 ceph-mon[29876]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T15:07:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:04 smithi028 ceph-mon[29876]: pgmap v844: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T15:07:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:04 smithi028 ceph-mon[41614]: osdmap e574: 8 total, 8 up, 8 in 2024-03-30T15:07:04.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:04 smithi028 ceph-mon[41614]: pgmap v844: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-30T15:07:04.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:04.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:04.885 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:04.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:04.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:05.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:05.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:07:05.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:05.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:05.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:05.304 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:05.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:05.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:05.421 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:05 smithi134 ceph-mon[33714]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T15:07:05.421 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.421 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:05 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:05.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 -- 172.21.15.28:0/2078728032 >> v1:172.21.15.28:6790/0 conn(0x7fc91010ae90 legacy=0x7fc91010d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 -- 172.21.15.28:0/2078728032 shutdown_connections 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 -- 172.21.15.28:0/2078728032 >> 172.21.15.28:0/2078728032 conn(0x7fc9100761d0 msgr2=0x7fc9100765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 -- 172.21.15.28:0/2078728032 shutdown_connections 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 -- 172.21.15.28:0/2078728032 wait complete. 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 librados: starting msgr at 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.559+0000 7fc9148ea700 1 librados: starting objecter 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 Processor -- start 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 -- start start 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 librados: setting wanted keys 2024-03-30T15:07:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 librados: calling monclient init 2024-03-30T15:07:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9101a70d0 con 0x7fc91010ae90 2024-03-30T15:07:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9101a82f0 con 0x7fc91010eb30 2024-03-30T15:07:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc9148ea700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9101a9530 con 0x7fc910107060 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc90e59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc910107060 0x7fc910101010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:56298/0 (socket says 172.21.15.28:56298) 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc90e59c700 1 -- 172.21.15.28:0/2380252352 learned_addr learned my addr 172.21.15.28:0/2380252352 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter ms_handle_connect 0x7fc910107060 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter ms_handle_connect 0x7fc91010ae90 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135154778 0 0) 0x7fc9101a9530 con 0x7fc910107060 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8f80039c0 con 0x7fc910107060 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069218398 0 0) 0x7fc9101a70d0 con 0x7fc91010ae90 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9101a9530 con 0x7fc91010ae90 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter ms_handle_connect 0x7fc91010eb30 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748697221 0 0) 0x7fc9101a82f0 con 0x7fc91010eb30 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9101a70d0 con 0x7fc91010eb30 2024-03-30T15:07:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.560+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3762132521 0 0) 0x7fc8f80039c0 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9101a82f0 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9000018b0 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1573238066 0 0) 0x7fc9101a9530 con 0x7fc91010ae90 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8f80039c0 con 0x7fc91010ae90 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc904001980 con 0x7fc91010ae90 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4224101148 0 0) 0x7fc9101a70d0 con 0x7fc91010eb30 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9101a9530 con 0x7fc91010eb30 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9080015b0 con 0x7fc91010eb30 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 412949799 0 0) 0x7fc9101a82f0 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 >> v1:172.21.15.134:6789/0 conn(0x7fc91010eb30 legacy=0x7fc9101a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 >> v1:172.21.15.28:6789/0 conn(0x7fc91010ae90 legacy=0x7fc9101a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9101aa770 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9101a7300 con 0x7fc910107060 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc9148ea700 10 client.58391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9101a77c0 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc9148ea700 1 librados: init done 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc900003b40 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.561+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc900005c50 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.562+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f0005150 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.562+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc9000036a0 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(576..576 src has 442..576) ==== 8655+0+0 (unknown 1997806432 0 0) 0x7fc900089920 con 0x7fc910107060 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 10 client.58391.objecter ms_dispatch 0x7fc910001550 osd_map(576..576 src has 442..576) 2024-03-30T15:07:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 3 client.58391.objecter handle_osd_map got epochs [576,576] > 0 2024-03-30T15:07:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 3 client.58391.objecter handle_osd_map decoding full epoch 576 2024-03-30T15:07:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 20 client.58391.objecter dump_active .. 0 homeless 2024-03-30T15:07:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.563+0000 7fc8ff7fe700 10 client.58391.objecter ms_handle_connect 0x7fc8f80704d0 2024-03-30T15:07:05.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.568+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc900056920 con 0x7fc910107060 2024-03-30T15:07:05.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:05.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[29876]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T15:07:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[41614]: osdmap e575: 8 total, 8 up, 8 in 2024-03-30T15:07:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:05 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:05.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:05.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:05.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:05.792+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7fc8f0005440 con 0x7fc910107060 2024-03-30T15:07:05.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:05.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:05.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:07:05.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:06.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:06.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:06.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:06.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:06.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:06.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.434+0000 7fc8ff7fe700 1 -- 172.21.15.28:0/2380252352 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v577) ==== 113+0+0 (unknown 2623036061 0 0) 0x7fc90005a370 con 0x7fc910107060 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 10 librados: watch_flush enter 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 10 librados: watch_flush exit 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 20 client.58391.objecter shutdown clearing up homeless session... 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 10 client.58391.objecter successfully canceled tick 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc8f80704d0 legacy=0x7fc8f8072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 >> v1:172.21.15.28:6790/0 conn(0x7fc910107060 legacy=0x7fc910101010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 shutdown_connections 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 >> 172.21.15.28:0/2380252352 conn(0x7fc9100761d0 msgr2=0x7fc910111780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 shutdown_connections 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 -- 172.21.15.28:0/2380252352 wait complete. 2024-03-30T15:07:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:06.437+0000 7fc9148ea700 1 librados: shutdown 2024-03-30T15:07:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T15:07:06.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:06.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:06.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2380252352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: from='client.58391 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:06 smithi134 ceph-mon[33714]: pgmap v847: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2380252352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: from='client.58391 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[41614]: pgmap v847: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: osdmap e576: 8 total, 8 up, 8 in 2024-03-30T15:07:06.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:06.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2380252352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: from='client.58391 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:07:06.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:06 smithi028 ceph-mon[29876]: pgmap v847: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-03-30T15:07:06.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:06.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:06.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:06.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:06.994 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:06.994 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:06.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:07.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:07.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:07.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:07.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:07.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:07.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:07:07.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:07.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:07.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:07.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='client.58391 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:07.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:07.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:07.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:07.555 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:07.555 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='client.58391 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:07:07.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T15:07:07.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:07.556 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:07.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:07:07.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:07.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-03-30T15:07:07.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:07.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='client.58391 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:07:07.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: osdmap e577: 8 total, 8 up, 8 in 2024-03-30T15:07:07.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]: dispatch 2024-03-30T15:07:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:07.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:07.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:07.863 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:07.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:07.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:08.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:08.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:08.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:08.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:08.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:08.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:08.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:08.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]': finished 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[29876]: pgmap v850: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:08.506 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]': finished 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.507 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:08 smithi028 ceph-mon[41614]: pgmap v850: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:08.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:08.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:08.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-98", "mode": "writeback"}]': finished 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: osdmap e578: 8 total, 8 up, 8 in 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:08 smithi134 ceph-mon[33714]: pgmap v850: 270 pgs: 32 unknown, 1 active+clean+scrubbing, 237 active+clean; 8.4 MiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:08.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:08.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:08.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:08.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 -- 172.21.15.28:0/1918494660 >> v1:172.21.15.28:6789/0 conn(0x7f8de810afb0 legacy=0x7f8de810d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 -- 172.21.15.28:0/1918494660 shutdown_connections 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 -- 172.21.15.28:0/1918494660 >> 172.21.15.28:0/1918494660 conn(0x7f8de80fc9f0 msgr2=0x7f8de80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 -- 172.21.15.28:0/1918494660 shutdown_connections 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 -- 172.21.15.28:0/1918494660 wait complete. 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 librados: starting msgr at 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 librados: starting objecter 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.013+0000 7f8dede2d700 1 Processor -- start 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 -- start start 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 librados: setting wanted keys 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 librados: calling monclient init 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8de81a7150 con 0x7f8de810afb0 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8de81a8370 con 0x7f8de81031a0 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8dede2d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8de81a95b0 con 0x7f8de810eac0 2024-03-30T15:07:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de7fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8de810eac0 0x7f8de81a5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41434/0 (socket says 172.21.15.28:41434) 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de7fff700 1 -- 172.21.15.28:0/2978757791 learned_addr learned my addr 172.21.15.28:0/2978757791 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de6ffd700 1 --1- 172.21.15.28:0/2978757791 >> v1:172.21.15.28:6789/0 conn(0x7f8de810afb0 0x7f8de81a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53008/0 (socket says 172.21.15.28:53008) 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter ms_handle_connect 0x7f8de81031a0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter ms_handle_connect 0x7f8de810eac0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter ms_handle_connect 0x7f8de810afb0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931582227 0 0) 0x7f8de81a8370 con 0x7f8de81031a0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd00031f0 con 0x7f8de81031a0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000123371 0 0) 0x7f8de81a95b0 con 0x7f8de810eac0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8de81a8370 con 0x7f8de810eac0 2024-03-30T15:07:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123414585 0 0) 0x7f8de81a7150 con 0x7f8de810afb0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.014+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8de81a95b0 con 0x7f8de810afb0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 695298768 0 0) 0x7f8dd00031f0 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8de81a7150 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8dd8001830 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4064146479 0 0) 0x7f8de81a8370 con 0x7f8de810eac0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd00031f0 con 0x7f8de810eac0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8de0001c60 con 0x7f8de810eac0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3913205645 0 0) 0x7f8de81a7150 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 >> v1:172.21.15.28:6790/0 conn(0x7f8de810eac0 legacy=0x7f8de81a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 >> v1:172.21.15.28:6789/0 conn(0x7f8de810afb0 legacy=0x7f8de81a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de81aa7f0 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8de81a97e0 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8dede2d700 10 client.50404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8de81a9ca0 con 0x7f8de81031a0 2024-03-30T15:07:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8dede2d700 1 librados: init done 2024-03-30T15:07:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8dd8003b20 con 0x7f8de81031a0 2024-03-30T15:07:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.015+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8dd8004ae0 con 0x7f8de81031a0 2024-03-30T15:07:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.016+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd4005120 con 0x7f8de81031a0 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.016+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8dd8003680 con 0x7f8de81031a0 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(579..579 src has 442..579) ==== 8663+0+0 (unknown 4199096438 0 0) 0x7f8dd8088740 con 0x7f8de81031a0 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 10 client.50404.objecter ms_dispatch 0x7f8de8001550 osd_map(579..579 src has 442..579) 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 3 client.50404.objecter handle_osd_map got epochs [579,579] > 0 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 3 client.50404.objecter handle_osd_map decoding full epoch 579 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 20 client.50404.objecter dump_active .. 0 homeless 2024-03-30T15:07:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.017+0000 7f8de4ff9700 10 client.50404.objecter ms_handle_connect 0x7f8dd00701d0 2024-03-30T15:07:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.022+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8dd8055740 con 0x7f8de81031a0 2024-03-30T15:07:09.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:09.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.235+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f8dd4005410 con 0x7f8de81031a0 2024-03-30T15:07:09.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:09.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:09.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.445+0000 7f8de4ff9700 1 -- 172.21.15.28:0/2978757791 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v580) ==== 97+0+0 (unknown 2165026786 0 0) 0x7f8dd8059190 con 0x7f8de81031a0 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 10 librados: watch_flush enter 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 10 librados: watch_flush exit 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 20 client.50404.objecter shutdown clearing up homeless session... 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 10 client.50404.objecter successfully canceled tick 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8dd00701d0 legacy=0x7f8dd0072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 >> v1:172.21.15.134:6789/0 conn(0x7f8de81031a0 legacy=0x7f8de810a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 shutdown_connections 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 >> 172.21.15.28:0/2978757791 conn(0x7f8de80fc9f0 msgr2=0x7f8de8110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.448+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 shutdown_connections 2024-03-30T15:07:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.449+0000 7f8dede2d700 1 -- 172.21.15.28:0/2978757791 wait complete. 2024-03-30T15:07:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:09.449+0000 7f8dede2d700 1 librados: shutdown 2024-03-30T15:07:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:07:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:09.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:09.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:09.601 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:09.601 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2978757791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.602 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:09 smithi134 ceph-mon[33714]: from='client.50404 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2978757791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[29876]: from='client.50404 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: osdmap e579: 8 total, 8 up, 8 in 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:07:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2978757791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:09 smithi028 ceph-mon[41614]: from='client.50404 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:09.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:09.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:09.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:09.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:09.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:10.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:10.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:10.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:10.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:07:10.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:10.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:10.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:10.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:10.542 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: from='client.50404 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:07:10.543 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-mon[33714]: pgmap v853: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:10.549 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:10.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:10.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:10.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: from='client.50404 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:07:10.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[29876]: pgmap v853: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: from='client.50404 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: osdmap e580: 8 total, 8 up, 8 in 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:07:10.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:10 smithi028 ceph-mon[41614]: pgmap v853: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:10.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:10.805 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:10.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:10.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:10.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:10.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:11.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:11.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:11.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:11.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:11.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:11.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:11.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T15:07:11.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:11 smithi134 ceph-mon[33714]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T15:07:11.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:11 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:11.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T15:07:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[29876]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T15:07:11.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_count","val": "1"}]': finished 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[41614]: osdmap e581: 8 total, 8 up, 8 in 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:11.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:11.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:11.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:11.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:11.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:12.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:12.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:12.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:12.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:12.247 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:12.248 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:12.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:07:12.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:12.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:12.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:12.550 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:12.550 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T15:07:12.550 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: pgmap v856: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:12 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: pgmap v856: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: osdmap e582: 8 total, 8 up, 8 in 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: pgmap v856: 270 pgs: 270 active+clean; 8.4 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-98","var": "target_max_objects","val": "250"}]': finished 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: osdmap e583: 8 total, 8 up, 8 in 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:12 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:12.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:12.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:07:12.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:12.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:12.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- 172.21.15.28:0/3009724287 >> v1:172.21.15.28:6789/0 conn(0x7f15a410d960 legacy=0x7f15a410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- 172.21.15.28:0/3009724287 shutdown_connections 2024-03-30T15:07:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- 172.21.15.28:0/3009724287 >> 172.21.15.28:0/3009724287 conn(0x7f15a40fc9e0 msgr2=0x7f15a40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- 172.21.15.28:0/3009724287 shutdown_connections 2024-03-30T15:07:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- 172.21.15.28:0/3009724287 wait complete. 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 librados: starting msgr at 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 librados: starting objecter 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 Processor -- start 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 -- start start 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 librados: setting wanted keys 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.892+0000 7f15abbd1700 1 librados: calling monclient init 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f15abbd1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15a41068c0 con 0x7f15a410d960 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f15abbd1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15a41a8540 con 0x7f15a4109dc0 2024-03-30T15:07:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f15abbd1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15a41a9720 con 0x7f15a4101130 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f15a996d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f15a4101130 0x7f15a41046e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41460/0 (socket says 172.21.15.28:41460) 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f15a996d700 1 -- 172.21.15.28:0/1185430476 learned_addr learned my addr 172.21.15.28:0/1185430476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter ms_handle_connect 0x7f15a4101130 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter ms_handle_connect 0x7f15a410d960 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter ms_handle_connect 0x7f15a4109dc0 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125157743 0 0) 0x7f15a41a9720 con 0x7f15a4101130 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15900039c0 con 0x7f15a4101130 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237180341 0 0) 0x7f15a41a8540 con 0x7f15a4109dc0 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15a41a9720 con 0x7f15a4109dc0 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978418147 0 0) 0x7f15a41068c0 con 0x7f15a410d960 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15a41a8540 con 0x7f15a410d960 2024-03-30T15:07:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1095901431 0 0) 0x7f15a41a9720 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15a41068c0 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1594001870 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3990564906 0 0) 0x7f15900039c0 con 0x7f15a4101130 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15a41a9720 con 0x7f15a4101130 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2583846647 0 0) 0x7f15a41a8540 con 0x7f15a410d960 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15900039c0 con 0x7f15a410d960 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f15a0001230 con 0x7f15a4101130 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f159c001be0 con 0x7f15a410d960 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1646802702 0 0) 0x7f15a41068c0 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6790/0 conn(0x7f15a4101130 legacy=0x7f15a41046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6789/0 conn(0x7f15a410d960 legacy=0x7f15a41a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.893+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a41aa900 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15a41a9950 con 0x7f15a4109dc0 2024-03-30T15:07:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f15abbd1700 10 client.50410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15a41a9e10 con 0x7f15a4109dc0 2024-03-30T15:07:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f15abbd1700 1 librados: init done 2024-03-30T15:07:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1594003b40 con 0x7f15a4109dc0 2024-03-30T15:07:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f15940049f0 con 0x7f15a4109dc0 2024-03-30T15:07:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.894+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1588005100 con 0x7f15a4109dc0 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.895+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f159401b310 con 0x7f15a4109dc0 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(583..583 src has 442..583) ==== 8689+0+0 (unknown 846407502 0 0) 0x7f15940887f0 con 0x7f15a4109dc0 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 osd_map(583..583 src has 442..583) 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 3 client.50410.objecter handle_osd_map got epochs [583,583] > 0 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 3 client.50410.objecter handle_osd_map decoding full epoch 583 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 20 client.50410.objecter dump_active .. 0 homeless 2024-03-30T15:07:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v583) ==== 54+0+16 (unknown 1782550022 0 2420000192) 0x7f1594088be0 con 0x7f15a4109dc0 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1590070590 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 10 client.50410.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 1 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 1 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 0 1 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 4 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 10 client.50410.objecter _send_command 1 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1588004b50 con 0x7f1588004730 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.896+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1588004730 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.897+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1588004b50 con 0x7f1588004730 2024-03-30T15:07:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.897+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.897+0000 7f159affd700 10 client.50410.objecter _finish_command 1 = system:0 2024-03-30T15:07:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.897+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 4 2024-03-30T15:07:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.897+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 0 1 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 10 client.50410.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 2 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 2 2024-03-30T15:07:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880053e0 osd=0 3 2024-03-30T15:07:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 0 2 2024-03-30T15:07:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 4 2024-03-30T15:07:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 10 client.50410.objecter _send_command 2 2024-03-30T15:07:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.914+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588007b00 con 0x7f1588004730 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588007b00 con 0x7f1588004730 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 10 client.50410.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 4 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 0 2 2024-03-30T15:07:12.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 10 client.50410.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 1 3 2024-03-30T15:07:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 4 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 10 client.50410.objecter _send_command 3 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1588002130 con 0x7f15880042a0 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.916+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f15880042a0 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.917+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1588002130 con 0x7f15880042a0 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.917+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.917+0000 7f159affd700 10 client.50410.objecter _finish_command 3 = system:0 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.917+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 4 2024-03-30T15:07:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.917+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 1 3 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 10 client.50410.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 4 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 4 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588004130 osd=1 3 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 1 4 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 4 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 10 client.50410.objecter _send_command 4 2024-03-30T15:07:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.934+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588001600 con 0x7f15880042a0 2024-03-30T15:07:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588001600 con 0x7f15880042a0 2024-03-30T15:07:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f159affd700 10 client.50410.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 4 2024-03-30T15:07:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 1 4 2024-03-30T15:07:12.937 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 10 client.50410.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 5 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 5 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 2 5 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 4 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 10 client.50410.objecter _send_command 5 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.935+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1588008c30 con 0x7f1588011af0 2024-03-30T15:07:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1588011af0 2024-03-30T15:07:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1588008c30 con 0x7f1588011af0 2024-03-30T15:07:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 10 client.50410.objecter _finish_command 5 = system:0 2024-03-30T15:07:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 4 2024-03-30T15:07:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.936+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 2 5 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 10 client.50410.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 6 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 6 2024-03-30T15:07:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588008480 osd=2 3 2024-03-30T15:07:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 2 6 2024-03-30T15:07:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 4 2024-03-30T15:07:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 10 client.50410.objecter _send_command 6 2024-03-30T15:07:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.953+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588007fa0 con 0x7f1588011af0 2024-03-30T15:07:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.954+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588007fa0 con 0x7f1588011af0 2024-03-30T15:07:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.954+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.954+0000 7f159affd700 10 client.50410.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.954+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 4 2024-03-30T15:07:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.954+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 2 6 2024-03-30T15:07:12.956 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 10 client.50410.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 7 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 7 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 3 7 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 4 2024-03-30T15:07:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 10 client.50410.objecter _send_command 7 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f15880161a0 con 0x7f1588015600 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1588015600 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15880161a0 con 0x7f1588015600 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 10 client.50410.objecter _finish_command 7 = system:0 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 4 2024-03-30T15:07:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.955+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 3 7 2024-03-30T15:07:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 10 client.50410.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 8 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 8 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880081e0 osd=3 3 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 3 8 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 4 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 10 client.50410.objecter _send_command 8 2024-03-30T15:07:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.972+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15880161a0 con 0x7f1588015600 2024-03-30T15:07:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15880161a0 con 0x7f1588015600 2024-03-30T15:07:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:12.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f159affd700 10 client.50410.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:12.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 4 2024-03-30T15:07:12.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 3 8 2024-03-30T15:07:12.976 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 10 client.50410.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 9 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 9 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 4 9 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 4 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 10 client.50410.objecter _send_command 9 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.974+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f158801ead0 con 0x7f158801afc0 2024-03-30T15:07:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f158801afc0 2024-03-30T15:07:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f158801ead0 con 0x7f158801afc0 2024-03-30T15:07:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 10 client.50410.objecter _finish_command 9 = system:0 2024-03-30T15:07:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 4 2024-03-30T15:07:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.975+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 4 9 2024-03-30T15:07:12.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 10 client.50410.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 10 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 10 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588017d50 osd=4 3 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 4 10 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 4 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 10 client.50410.objecter _send_command 10 2024-03-30T15:07:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.992+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588016bc0 con 0x7f158801afc0 2024-03-30T15:07:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588016bc0 con 0x7f158801afc0 2024-03-30T15:07:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f159affd700 10 client.50410.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 4 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 4 10 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 10 client.50410.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 11 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 11 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 5 11 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 4 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 10 client.50410.objecter _send_command 11 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.994+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f15880213e0 con 0x7f15880172e0 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.995+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f15880172e0 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.996+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f15880213e0 con 0x7f15880172e0 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.996+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.996+0000 7f159affd700 10 client.50410.objecter _finish_command 11 = system:0 2024-03-30T15:07:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.996+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 4 2024-03-30T15:07:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:12.996+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 5 11 2024-03-30T15:07:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 10 client.50410.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 12 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 12 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588016bc0 osd=5 3 2024-03-30T15:07:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 5 12 2024-03-30T15:07:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 4 2024-03-30T15:07:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 10 client.50410.objecter _send_command 12 2024-03-30T15:07:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.013+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15880213e0 con 0x7f15880172e0 2024-03-30T15:07:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15880213e0 con 0x7f15880172e0 2024-03-30T15:07:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 10 client.50410.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 4 2024-03-30T15:07:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 5 12 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 10 client.50410.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 13 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 13 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 6 13 2024-03-30T15:07:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 4 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 10 client.50410.objecter _send_command 13 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1588027640 con 0x7f1588023b30 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.015+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1588023b30 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.016+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1588027640 con 0x7f1588023b30 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.016+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.016+0000 7f159affd700 10 client.50410.objecter _finish_command 13 = system:0 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.016+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 4 2024-03-30T15:07:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.016+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 6 13 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 10 client.50410.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 14 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 14 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f15880213e0 osd=6 3 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 6 14 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 4 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 10 client.50410.objecter _send_command 14 2024-03-30T15:07:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.034+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588027640 con 0x7f1588023b30 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.035+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588027640 con 0x7f1588023b30 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.035+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.035+0000 7f159affd700 10 client.50410.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.035+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 4 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.035+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 6 14 2024-03-30T15:07:13.037 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 10 client.50410.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 15 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 15 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 7 15 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 4 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 10 client.50410.objecter _send_command 15 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.036+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1588029980 con 0x7f1588027d60 2024-03-30T15:07:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 10 client.50410.objecter ms_handle_connect 0x7f1588027d60 2024-03-30T15:07:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1588029980 con 0x7f1588027d60 2024-03-30T15:07:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 10 client.50410.objecter _finish_command 15 = system:0 2024-03-30T15:07:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 4 2024-03-30T15:07:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.037+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 7 15 2024-03-30T15:07:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 10 client.50410.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign -1 16 2024-03-30T15:07:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 20 client.50410.objecter _get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_remove -1 16 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 20 client.50410.objecter get_session s=0x7f1588027640 osd=7 3 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 15 client.50410.objecter _session_command_op_assign 7 16 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 4 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 10 client.50410.objecter _send_command 16 2024-03-30T15:07:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.054+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1588029980 con 0x7f1588027d60 2024-03-30T15:07:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f159affd700 1 -- 172.21.15.28:0/1185430476 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1588029980 con 0x7f1588027d60 2024-03-30T15:07:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f159affd700 10 client.50410.objecter ms_dispatch 0x7f15a4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f159affd700 10 client.50410.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f159affd700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 4 2024-03-30T15:07:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f159affd700 15 client.50410.objecter _session_command_op_remove 7 16 2024-03-30T15:07:13.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 librados: watch_flush enter 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 librados: watch_flush exit 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.0 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6801/3394323575 conn(0x7f1588004730 legacy=0x7f1588002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880053e0 osd=0 1 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.1 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6805/1850313547 conn(0x7f15880042a0 legacy=0x7f158800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588004130 osd=1 1 2024-03-30T15:07:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.2 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6809/3451325043 conn(0x7f1588011af0 legacy=0x7f1588013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588008480 osd=2 1 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.3 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6813/1506157250 conn(0x7f1588015600 legacy=0x7f1588015a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880081e0 osd=3 1 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.4 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.134:6800/3063775514 conn(0x7f158801afc0 legacy=0x7f158801d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588017d50 osd=4 1 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.5 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.134:6804/3668151179 conn(0x7f15880172e0 legacy=0x7f158801fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588016bc0 osd=5 1 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.6 2024-03-30T15:07:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15aa16e700 1 -- 172.21.15.28:0/1185430476 reap_dead start 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.134:6808/2135224194 conn(0x7f1588023b30 legacy=0x7f1588025f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15aa16e700 1 -- 172.21.15.28:0/1185430476 reap_dead start 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f15880213e0 osd=6 1 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter close_session for osd.7 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.134:6812/3758190298 conn(0x7f1588027d60 legacy=0x7f15880281a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter put_session s=0x7f1588027640 osd=7 1 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 20 client.50410.objecter shutdown clearing up homeless session... 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.056+0000 7f15abbd1700 10 client.50410.objecter successfully canceled tick 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15aa16e700 1 -- 172.21.15.28:0/1185430476 reap_dead start 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15aa16e700 1 -- 172.21.15.28:0/1185430476 reap_dead start 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1590070590 legacy=0x7f1590072a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> v1:172.21.15.134:6789/0 conn(0x7f15a4109dc0 legacy=0x7f15a4104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 shutdown_connections 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 >> 172.21.15.28:0/1185430476 conn(0x7f15a40fc9e0 msgr2=0x7f15a406a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 shutdown_connections 2024-03-30T15:07:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 -- 172.21.15.28:0/1185430476 wait complete. 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.057+0000 7f15abbd1700 1 librados: shutdown 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:13.075 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:13.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.092+0000 7fc37fd12700 1 -- 172.21.15.28:0/155280818 >> v1:172.21.15.28:6789/0 conn(0x7fc37810d9a0 legacy=0x7fc37810fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.092+0000 7fc37fd12700 1 -- 172.21.15.28:0/155280818 shutdown_connections 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.092+0000 7fc37fd12700 1 -- 172.21.15.28:0/155280818 >> 172.21.15.28:0/155280818 conn(0x7fc3780fca20 msgr2=0x7fc3780fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.092+0000 7fc37fd12700 1 -- 172.21.15.28:0/155280818 shutdown_connections 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.092+0000 7fc37fd12700 1 -- 172.21.15.28:0/155280818 wait complete. 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 librados: starting msgr at 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 librados: starting objecter 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 Processor -- start 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 -- start start 2024-03-30T15:07:13.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 librados: setting wanted keys 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 librados: calling monclient init 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3781a7100 con 0x7fc37810d9a0 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3781a8320 con 0x7fc378101170 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37fd12700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc3781a9560 con 0x7fc378109e00 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37d2ad700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc378109e00 0x7fc3781a1e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41478/0 (socket says 172.21.15.28:41478) 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc37d2ad700 1 -- 172.21.15.28:0/1487624486 learned_addr learned my addr 172.21.15.28:0/1487624486 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter ms_handle_connect 0x7fc378109e00 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter ms_handle_connect 0x7fc37810d9a0 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter ms_handle_connect 0x7fc378101170 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872349129 0 0) 0x7fc3781a9560 con 0x7fc378109e00 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3600031f0 con 0x7fc378109e00 2024-03-30T15:07:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968192493 0 0) 0x7fc3781a7100 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3781a9560 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2641681328 0 0) 0x7fc3781a8320 con 0x7fc378101170 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.093+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3781a7100 con 0x7fc378101170 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1276959482 0 0) 0x7fc3781a9560 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3781a8320 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc370001840 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1959022927 0 0) 0x7fc3781a7100 con 0x7fc378101170 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3781a9560 con 0x7fc378101170 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2165327388 0 0) 0x7fc3600031f0 con 0x7fc378109e00 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3781a7100 con 0x7fc378109e00 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc3680014b0 con 0x7fc378101170 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc36c0018b0 con 0x7fc378109e00 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1747715687 0 0) 0x7fc3781a8320 con 0x7fc37810d9a0 2024-03-30T15:07:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 >> v1:172.21.15.28:6790/0 conn(0x7fc378109e00 legacy=0x7fc3781a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 >> v1:172.21.15.134:6789/0 conn(0x7fc378101170 legacy=0x7fc37806af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3781aa7a0 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3781a7330 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc37fd12700 10 client.25209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3781a77f0 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc37fd12700 1 librados: init done 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc370004760 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.094+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc370004fd0 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.095+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc378102650 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.095+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc370003860 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(583..583 src has 442..583) ==== 8689+0+0 (unknown 846407502 0 0) 0x7fc3700887c0 con 0x7fc37810d9a0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 10 client.25209.objecter ms_dispatch 0x7fc378001550 osd_map(583..583 src has 442..583) 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 3 client.25209.objecter handle_osd_map got epochs [583,583] > 0 2024-03-30T15:07:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 3 client.25209.objecter handle_osd_map decoding full epoch 583 2024-03-30T15:07:13.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 20 client.25209.objecter dump_active .. 0 homeless 2024-03-30T15:07:13.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.096+0000 7fc376ffd700 10 client.25209.objecter ms_handle_connect 0x7fc3600703a0 2024-03-30T15:07:13.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.100+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc3700557c0 con 0x7fc37810d9a0 2024-03-30T15:07:13.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:13.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.323+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7fc37806a6c0 con 0x7fc37810d9a0 2024-03-30T15:07:13.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.338+0000 7fc376ffd700 1 -- 172.21.15.28:0/1487624486 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v584) ==== 113+0+0 (unknown 2238221042 0 0) 0x7fc370059210 con 0x7fc37810d9a0 2024-03-30T15:07:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 10 librados: watch_flush enter 2024-03-30T15:07:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 10 librados: watch_flush exit 2024-03-30T15:07:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 20 client.25209.objecter shutdown clearing up homeless session... 2024-03-30T15:07:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 10 client.25209.objecter successfully canceled tick 2024-03-30T15:07:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc3600703a0 legacy=0x7fc360072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.341+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 >> v1:172.21.15.28:6789/0 conn(0x7fc37810d9a0 legacy=0x7fc3781a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.342+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 shutdown_connections 2024-03-30T15:07:13.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.342+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 >> 172.21.15.28:0/1487624486 conn(0x7fc3780fca20 msgr2=0x7fc378110610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.344+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 shutdown_connections 2024-03-30T15:07:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.344+0000 7fc37fd12700 1 -- 172.21.15.28:0/1487624486 wait complete. 2024-03-30T15:07:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:13.344+0000 7fc37fd12700 1 librados: shutdown 2024-03-30T15:07:13.346 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T15:07:13.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:13.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:13.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:13.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1185430476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1185430476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:13 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:13.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:13.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:13.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1185430476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487624486' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: osdmap e584: 8 total, 8 up, 8 in 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.891 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:13 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]: dispatch 2024-03-30T15:07:13.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:14.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:14.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:14.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:14.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:14.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:14.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:14.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:14.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:14.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:14.662 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:14 smithi134 ceph-mon[33714]: pgmap v859: 270 pgs: 270 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:14.663 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:14 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:14.663 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:14 smithi134 ceph-mon[33714]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25069.0:3 35.f 35:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 559.46002035699996, 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0835292080000001, 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.664 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25069", 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3135164836", 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200377+0000", 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200379+0000", 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6729999999999999e-06 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200387+0000", 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1799999999999996e-06 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200395+0000", 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3850000000000002e-06 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200404+0000", 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6219999999999998e-06 2024-03-30T15:07:14.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200468+0000", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4142999999999999e-05 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.200475+0000", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4580000000000004e-06 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259289+0000", 2024-03-30T15:07:14.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0588138250000001 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.259412+0000", 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012236699999999999 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,6", 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.260458+0000", 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010460790000000001 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.261342+0000", 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00088464100000000003 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.268237+0000", 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068943329999999999 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283840+0000", 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015603258 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283874+0000", 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3736999999999999e-05 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.283906+0000", 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2506999999999999e-05 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25048.0:1 46.6s0 46:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 558.52758903699998, 2024-03-30T15:07:14.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2874541669999999, 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25048", 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3576801120", 2024-03-30T15:07:14.673 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132808+0000", 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132810+0000", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0250000000000001e-06 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132815+0000", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9869999999999999e-06 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132823+0000", 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0560000000000005e-06 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.132833+0000", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5200000000000003e-06 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145608+0000", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012775376999999999 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.145613+0000", 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5079999999999999e-06 2024-03-30T15:07:14.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397079+0000", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2514661540000001 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397194+0000", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115407 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.397531+0000", 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000336745 2024-03-30T15:07:14.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.402316+0000", 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047853660000000001 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420194+0000", 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017877613000000001 2024-03-30T15:07:14.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.420262+0000", 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8409000000000002e-05 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34886.0:1 50.0s0 50:0c828f53:test-rados-api-smithi028-93540-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 557.32315682900003, 2024-03-30T15:07:14.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3929710050000002, 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34886", 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4069139007", 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337241+0000", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337242+0000", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.792e-06 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337249+0000", 2024-03-30T15:07:14.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9430000000000003e-06 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337260+0000", 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0535e-05 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.337269+0000", 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8009999999999993e-06 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342443+0000", 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051742710000000003 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.342445+0000", 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.446e-06 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718228+0000", 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3757831299999999 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.718356+0000", 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012730400000000001 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719072+0000", 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071628200000000005 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.720051+0000", 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00097890299999999993 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730124+0000", 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010073343 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.730212+0000", 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7255000000000004e-05 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2059 11.7 11:e6e7235b:test-rados-api-smithi028-93257-16::big:head [writefull 0~25165836] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 556.71461573099998, 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.086346471999999994, 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.686 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2059 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945782+0000", 2024-03-30T15:07:14.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.945784+0000", 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0310000000000001e-06 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969470+0000", 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023686577 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969496+0000", 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6210000000000001e-05 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969507+0000", 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0723000000000001e-05 2024-03-30T15:07:14.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969596+0000", 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8653999999999995e-05 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969758+0000", 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016222 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.969914+0000", 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015554699999999999 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030515+0000", 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060601742 2024-03-30T15:07:14.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.030889+0000", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037352300000000001 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032049+0000", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011604639999999999 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032088+0000", 2024-03-30T15:07:14.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8621000000000002e-05 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.032128+0000", 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0160000000000002e-05 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34880.0:5 13.b 13:d4d78c7e:test-rados-api-smithi028-93525-3::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:07:14.693 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 556.18802109900003, 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15821444200000001, 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34880", 2024-03-30T15:07:14.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4209260695", 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 5 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:07:14.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[41614]: pgmap v859: 270 pgs: 270 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:14.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:14.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[41614]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T15:07:14.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[29876]: pgmap v859: 270 pgs: 270 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:14.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-98"}]': finished 2024-03-30T15:07:14.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:14 smithi028 ceph-mon[29876]: osdmap e585: 8 total, 8 up, 8 in 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472376+0000", 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472378+0000", 2024-03-30T15:07:14.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2000000000000001e-06 2024-03-30T15:07:14.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472386+0000", 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8760000000000005e-06 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472397+0000", 2024-03-30T15:07:14.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0900000000000001e-05 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472409+0000", 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1379999999999999e-05 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472488+0000", 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9796999999999995e-05 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.472495+0000", 2024-03-30T15:07:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5749999999999997e-06 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517331+0000", 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044836187 2024-03-30T15:07:14.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.517472+0000", 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014135100000000001 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.521687+0000", 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042141230000000002 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.534881+0000", 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.01319443 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.553213+0000", 2024-03-30T15:07:14.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018331719 2024-03-30T15:07:14.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630475+0000", 2024-03-30T15:07:14.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.077262443 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630530+0000", 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4910000000000001e-05 2024-03-30T15:07:14.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.630591+0000", 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0550999999999997e-05 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.712 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.713 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.713 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25483.0:2 63.1b 63:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:07:14.713 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 553.67090879399996, 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9747056280000002, 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25483", 2024-03-30T15:07:14.715 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1101472932", 2024-03-30T15:07:14.715 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:14.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.715 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989489+0000", 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989490+0000", 2024-03-30T15:07:14.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5880000000000001e-06 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989498+0000", 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.396e-06 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.718 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989506+0000", 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8459999999999997e-06 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989518+0000", 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1309e-05 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989578+0000", 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0393000000000003e-05 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989589+0000", 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0829000000000001e-05 2024-03-30T15:07:14.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914500+0000", 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9249114629999999 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914631+0000", 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013069899999999999 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914772+0000", 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014078499999999999 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.917517+0000", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027451939999999998 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962191+0000", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.044673933999999998 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964110+0000", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019193019999999999 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964150+0000", 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0212000000000001e-05 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.964194+0000", 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3677999999999998e-05 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35456.0:2 69.0 69:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 552.87112326399995, 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729640450000001, 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35456", 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3389323125", 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789274+0000", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789276+0000", 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.317e-06 2024-03-30T15:07:14.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789282+0000", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0510000000000002e-06 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789297+0000", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4766999999999999e-05 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.789307+0000", 2024-03-30T15:07:14.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8810000000000008e-06 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806392+0000", 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017084881 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.806402+0000", 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9089999999999995e-06 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.728 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885233+0000", 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0788315669999999 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885345+0000", 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111907 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.885469+0000", 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012389899999999999 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.901898+0000", 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016428991 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.902088+0000", 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000189459 2024-03-30T15:07:14.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962138+0000", 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060050234000000001 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962190+0000", 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2432000000000002e-05 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.962238+0000", 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7750000000000002e-05 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25489.0:3 64.f 64:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 551.74619554799995, 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.084413231000000005, 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.732 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25489", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2934336176", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914202+0000", 2024-03-30T15:07:14.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914204+0000", 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.905e-06 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914214+0000", 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9399999999999997e-06 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914224+0000", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0190999999999999e-05 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914235+0000", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0686999999999999e-05 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914298+0000", 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3063000000000003e-05 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.735 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.914308+0000", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0642000000000001e-05 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995802+0000", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.081493828000000004 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995889+0000", 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7015000000000006e-05 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.995993+0000", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104177 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996545+0000", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055223100000000003 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.996864+0000", 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031808999999999999 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998524+0000", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016600040000000001 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998579+0000", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5368000000000003e-05 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.998615+0000", 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6090000000000002e-05 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.738 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49076.0:2 135.1b 135:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 525.48681372399994, 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0617338199999999, 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49076", 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2505490190", 2024-03-30T15:07:14.739 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173584+0000", 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173586+0000", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9530000000000002e-06 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173594+0000", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4069999999999993e-06 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173615+0000", 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0788000000000002e-05 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.741 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.173632+0000", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7589999999999999e-05 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176962+0000", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033293039999999999 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:29.176970+0000", 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9889999999999992e-06 2024-03-30T15:07:14.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226049+0000", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.049079015 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226159+0000", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110021 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.226288+0000", 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012942800000000001 2024-03-30T15:07:14.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.229542+0000", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032538939999999998 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.231623+0000", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020808440000000001 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235241+0000", 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036180539999999999 2024-03-30T15:07:14.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235288+0000", 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7382000000000001e-05 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.235317+0000", 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9150999999999999e-05 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:23 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [write 0~128] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 523.31397491500002, 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12244129300000001, 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 23 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346422+0000", 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346424+0000", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9690000000000001e-06 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346432+0000", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5979999999999999e-06 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346442+0000", 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0431000000000001e-05 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.346453+0000", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0896e-05 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356731+0000", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010277319 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.356836+0000", 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105664 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.357052+0000", 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021593299999999999 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.366074+0000", 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090220760000000004 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.388374+0000", 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022299229 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468758+0000", 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080384786999999999 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.751 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468818+0000", 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9493000000000001e-05 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:31.468864+0000", 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5898e-05 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25696.0:25 104.7 104:e5c65bbd:test-rados-api-smithi028-93441-9::foo:head [rollback 8] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 521.44850348800003, 2024-03-30T15:07:14.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12770631599999999, 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25696", 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1786221605", 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 25 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211894+0000", 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211896+0000", 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0789999999999999e-06 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211903+0000", 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2359999999999998e-06 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211912+0000", 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4109999999999996e-06 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.211923+0000", 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1038e-05 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212228+0000", 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030524600000000001 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212330+0000", 2024-03-30T15:07:14.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102433 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.212488+0000", 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015767099999999999 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.213555+0000", 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001067045 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.292128+0000", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.078572806999999995 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339484+0000", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.047356374999999999 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339548+0000", 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4126000000000002e-05 2024-03-30T15:07:14.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:33.339600+0000", 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1848999999999998e-05 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57281.0:1 221.6s0 221:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 451.45215606199997, 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1519453040000001, 2024-03-30T15:07:14.759 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57281", 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4286010033", 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208243+0000", 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1550000000000001e-06 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208253+0000", 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3139999999999993e-06 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208264+0000", 2024-03-30T15:07:14.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1592e-05 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208277+0000", 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2703e-05 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209635+0000", 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001357638 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209639+0000", 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6949999999999999e-06 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358223+0000", 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1485833839999999 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358319+0000", 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6644999999999994e-05 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358620+0000", 2024-03-30T15:07:14.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030071200000000002 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.359145+0000", 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052535199999999998 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360113+0000", 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096798800000000003 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360187+0000", 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3126000000000004e-05 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.769 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49354.0:1 224.6s1 224:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 444.456142258, 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025966369, 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.770 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49354", 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1904812330", 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204256+0000", 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.39e-06 2024-03-30T15:07:14.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204263+0000", 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0669999999999997e-06 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204271+0000", 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1880000000000002e-06 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204279+0000", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5639999999999997e-06 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228587+0000", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.024308065 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228592+0000", 2024-03-30T15:07:14.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222766+0000", 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99417422799999999 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222903+0000", 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013736600000000001 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.223366+0000", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000463187 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.226776+0000", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003409287 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230143+0000", 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033669149999999998 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230221+0000", 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8844000000000006e-05 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57626.0:1 262.5s0 262:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 370.424827107, 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0934634089999999, 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.778 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57626", 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1398925", 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235572+0000", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9889999999999999e-06 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235581+0000", 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8370000000000003e-06 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235591+0000", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0292000000000001e-05 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235602+0000", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0976e-05 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237121+0000", 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015190379999999999 2024-03-30T15:07:14.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237126+0000", 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1679999999999997e-06 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326125+0000", 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0889992150000001 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326273+0000", 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014821499999999999 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326919+0000", 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064574200000000004 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.327411+0000", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00049205899999999999 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.328990+0000", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001578861 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.329034+0000", 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4017000000000002e-05 2024-03-30T15:07:14.784 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49657.0:1 263.6s0 263:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 367.43194861199999, 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.017319423, 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.785 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49657", 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2923162696", 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228451+0000", 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4700000000000001e-06 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228459+0000", 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8490000000000006e-06 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228469+0000", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5219999999999996e-06 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228478+0000", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2150000000000007e-06 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238328+0000", 2024-03-30T15:07:14.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098497410000000004 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238331+0000", 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9009999999999996e-06 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242049+0000", 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0037178710000001 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242184+0000", 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0001351 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242588+0000", 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040368400000000002 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.244103+0000", 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015145740000000001 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245701+0000", 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015980180000000001 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245768+0000", 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7477999999999993e-05 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.791 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24537.0:1 270.6s0 270:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 346.40354274700002, 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0134851680000001, 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24537", 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2126246388", 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256858+0000", 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6389999999999998e-06 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256868+0000", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0231000000000001e-05 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256886+0000", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7988000000000001e-05 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256901+0000", 2024-03-30T15:07:14.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4817999999999999e-05 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261660+0000", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004759012 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261665+0000", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0649999999999998e-06 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267449+0000", 2024-03-30T15:07:14.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0057839420000001 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267596+0000", 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014701699999999999 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.268682+0000", 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001085325 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.269079+0000", 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039733100000000001 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270298+0000", 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001219354 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270340+0000", 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1446000000000003e-05 2024-03-30T15:07:14.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57869.0:1 286.5s0 286:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 287.38590260299998, 2024-03-30T15:07:14.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0109918019999999, 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57869", 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/714138788", 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274498+0000", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9220000000000001e-06 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274509+0000", 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1678999999999999e-05 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274525+0000", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5781e-05 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274540+0000", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4808e-05 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277431+0000", 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028910849999999998 2024-03-30T15:07:14.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277436+0000", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4589999999999997e-06 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.282966+0000", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0055296760000001 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283097+0000", 2024-03-30T15:07:14.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013078699999999999 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283531+0000", 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043440599999999997 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283992+0000", 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046019300000000002 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285413+0000", 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014217050000000001 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285487+0000", 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3300999999999995e-05 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57923.0:1 290.7s0 290:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:07:14.804 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 266.40000221600002, 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.011345699, 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57923", 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3909045240", 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260397+0000", 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.733e-06 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260404+0000", 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.221e-06 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260415+0000", 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0659000000000001e-05 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260425+0000", 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0703999999999999e-05 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260972+0000", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054698500000000005 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260976+0000", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1020000000000001e-06 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269376+0000", 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.008400816 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269503+0000", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126236 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269916+0000", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000413743 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271218+0000", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013020810000000001 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271674+0000", 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00045557200000000001 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271741+0000", 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6847000000000006e-05 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.6.52:31 101.d 101:b5bd8322:test-rados-api-smithi028-93253-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e520)", 2024-03-30T15:07:14.810 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 81.339438412000007, 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050572109999999998, 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:07:14.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320961+0000", 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7549999999999999e-06 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320966+0000", 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8039999999999998e-06 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320979+0000", 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2858e-05 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320987+0000", 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1110000000000008e-06 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321177+0000", 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018962499999999999 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321306+0000", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012909299999999999 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321547+0000", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024123600000000001 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322529+0000", 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00098205800000000006 2024-03-30T15:07:14.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322707+0000", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000177629 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325919+0000", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032115210000000002 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325973+0000", 2024-03-30T15:07:14.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4014999999999997e-05 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.326016+0000", 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3506e-05 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35600.0:7884 308.4 308:20000000:%01:::head [pgnls start_epoch 538] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e538)", 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:06:11.349265+0000", 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 63.311132163000003, 2024-03-30T15:07:14.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035193070000000002, 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35600", 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 7884 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349265+0000", 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349265+0000", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349268+0000", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.379e-06 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349276+0000", 2024-03-30T15:07:14.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1319999999999994e-06 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349291+0000", 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5647000000000001e-05 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349302+0000", 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0988999999999999e-05 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349392+0000", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9758000000000004e-05 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.349396+0000", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1949999999999996e-06 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.352652+0000", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003255998 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:14.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.352682+0000", 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9499e-05 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:06:11.352784+0000", 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010271 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:14.821 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:07:14.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:14.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-03-30T15:07:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:14.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:14.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:15.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:15.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:15.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:15.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:15.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:15.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:15.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:07:15.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:15.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:15.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:15.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:15.791 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:15.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:15.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:15.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:15.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:07:15.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:16.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:16.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:16.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:16.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:16.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:16.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:16.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:16.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:16.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:16.574 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:16 smithi134 ceph-mon[33714]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T15:07:16.574 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:16 smithi134 ceph-mon[33714]: pgmap v862: 238 pgs: 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:16.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:16 smithi028 ceph-mon[41614]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T15:07:16.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:16 smithi028 ceph-mon[41614]: pgmap v862: 238 pgs: 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:16.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:16 smithi028 ceph-mon[29876]: osdmap e586: 8 total, 8 up, 8 in 2024-03-30T15:07:16.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:16 smithi028 ceph-mon[29876]: pgmap v862: 238 pgs: 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:16.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.587+0000 7f7cec8f1700 1 -- 172.21.15.28:0/1197673810 >> v1:172.21.15.28:6790/0 conn(0x7f7ce810aee0 legacy=0x7f7ce810d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.587+0000 7f7cec8f1700 1 -- 172.21.15.28:0/1197673810 shutdown_connections 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.587+0000 7f7cec8f1700 1 -- 172.21.15.28:0/1197673810 >> 172.21.15.28:0/1197673810 conn(0x7f7ce80fdae0 msgr2=0x7f7ce80fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.587+0000 7f7cec8f1700 1 -- 172.21.15.28:0/1197673810 shutdown_connections 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.587+0000 7f7cec8f1700 1 -- 172.21.15.28:0/1197673810 wait complete. 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 librados: starting msgr at 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 librados: starting objecter 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 Processor -- start 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 -- start start 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 librados: setting wanted keys 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 librados: calling monclient init 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ce81a9330 con 0x7f7ce810aee0 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ce81aa550 con 0x7f7ce810ea80 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cec8f1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ce81ab790 con 0x7f7ce8102250 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7ce659c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7ce8102250 0x7f7ce81075b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41506/0 (socket says 172.21.15.28:41506) 2024-03-30T15:07:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7ce5d9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ce810aee0 0x7f7ce81a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53084/0 (socket says 172.21.15.28:53084) 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7ce659c700 1 -- 172.21.15.28:0/3866256028 learned_addr learned my addr 172.21.15.28:0/3866256028 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f7ce810aee0 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f7ce810ea80 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter ms_handle_connect 0x7f7ce8102250 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.588+0000 7f7cdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118465979 0 0) 0x7f7ce81aa550 con 0x7f7ce810ea80 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cd4003aa0 con 0x7f7ce810ea80 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164150309 0 0) 0x7f7ce81ab790 con 0x7f7ce8102250 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ce81aa550 con 0x7f7ce8102250 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171518660 0 0) 0x7f7ce81a9330 con 0x7f7ce810aee0 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ce81ab790 con 0x7f7ce810aee0 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3656379504 0 0) 0x7f7cd4003aa0 con 0x7f7ce810ea80 2024-03-30T15:07:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ce81a9330 con 0x7f7ce810ea80 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cd8001af0 con 0x7f7ce810ea80 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3442897587 0 0) 0x7f7ce81aa550 con 0x7f7ce8102250 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cd4003aa0 con 0x7f7ce8102250 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ce0001230 con 0x7f7ce8102250 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4064991460 0 0) 0x7f7ce81ab790 con 0x7f7ce810aee0 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ce81aa550 con 0x7f7ce810aee0 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cd00046c0 con 0x7f7ce810aee0 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2907961217 0 0) 0x7f7ce81a9330 con 0x7f7ce810ea80 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 >> v1:172.21.15.28:6790/0 conn(0x7f7ce8102250 legacy=0x7f7ce81075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 >> v1:172.21.15.28:6789/0 conn(0x7f7ce810aee0 legacy=0x7f7ce81a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce81ac9d0 con 0x7f7ce810ea80 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce81a9560 con 0x7f7ce810ea80 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cec8f1700 10 client.50422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:16.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ce81a9a20 con 0x7f7ce810ea80 2024-03-30T15:07:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cec8f1700 1 librados: init done 2024-03-30T15:07:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7cd80020a0 con 0x7f7ce810ea80 2024-03-30T15:07:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.589+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7cd8004d60 con 0x7f7ce810ea80 2024-03-30T15:07:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.590+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc8005120 con 0x7f7ce810ea80 2024-03-30T15:07:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7cd801b400 con 0x7f7ce810ea80 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(587..587 src has 442..587) ==== 8643+0+0 (unknown 616869204 0 0) 0x7f7cd80889a0 con 0x7f7ce810ea80 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 10 client.50422.objecter ms_dispatch 0x7f7ce8001550 osd_map(587..587 src has 442..587) 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 3 client.50422.objecter handle_osd_map got epochs [587,587] > 0 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 3 client.50422.objecter handle_osd_map decoding full epoch 587 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 20 client.50422.objecter dump_active .. 0 homeless 2024-03-30T15:07:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.591+0000 7f7cdf7fe700 10 client.50422.objecter ms_handle_connect 0x7f7cd40705c0 2024-03-30T15:07:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.596+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7cd80559b0 con 0x7f7ce810ea80 2024-03-30T15:07:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:16.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:16.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:16.816+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f7cc8005410 con 0x7f7ce810ea80 2024-03-30T15:07:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:16.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:16.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:17.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:17.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:17.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:17.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.364+0000 7f7cdf7fe700 1 -- 172.21.15.28:0/3866256028 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v588) ==== 91+0+0 (unknown 4139761226 0 0) 0x7f7cd8059400 con 0x7f7ce810ea80 2024-03-30T15:07:17.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.367+0000 7f7cec8f1700 10 librados: watch_flush enter 2024-03-30T15:07:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.368+0000 7f7cec8f1700 10 librados: watch_flush exit 2024-03-30T15:07:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.368+0000 7f7cec8f1700 20 client.50422.objecter shutdown clearing up homeless session... 2024-03-30T15:07:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.368+0000 7f7cec8f1700 10 client.50422.objecter successfully canceled tick 2024-03-30T15:07:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.368+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7cd40705c0 legacy=0x7f7cd4072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.368+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 >> v1:172.21.15.134:6789/0 conn(0x7f7ce810ea80 legacy=0x7f7ce81a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.369+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 shutdown_connections 2024-03-30T15:07:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.369+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 >> 172.21.15.28:0/3866256028 conn(0x7f7ce80fdae0 msgr2=0x7f7ce8111c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.370+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 shutdown_connections 2024-03-30T15:07:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.370+0000 7f7cec8f1700 1 -- 172.21.15.28:0/3866256028 wait complete. 2024-03-30T15:07:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:17.370+0000 7f7cec8f1700 1 librados: shutdown 2024-03-30T15:07:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:07:17.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:17.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:17.529 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:07:17.529 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:07:17.529 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:07:17.529 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:07:17.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:17 smithi134 ceph-mon[33714]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T15:07:17.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.530 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:17 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.530 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3866256028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.530 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:17 smithi134 ceph-mon[33714]: from='client.50422 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:17.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:17.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[29876]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T15:07:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3866256028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[29876]: from='client.50422 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[41614]: osdmap e587: 8 total, 8 up, 8 in 2024-03-30T15:07:17.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:17.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3866256028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:17 smithi028 ceph-mon[41614]: from='client.50422 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:17.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:17.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:17.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:17.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:17.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:18.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:07:18.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:18.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:18.335 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:18.336 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:18.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:18.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: from='client.50422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T15:07:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[41614]: pgmap v865: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: from='client.50422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:18 smithi028 ceph-mon[29876]: pgmap v865: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:18.484 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:18.508 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:18.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: from='client.50422 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:18.617 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: osdmap e588: 8 total, 8 up, 8 in 2024-03-30T15:07:18.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:07:18.618 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:18 smithi134 ceph-mon[33714]: pgmap v865: 270 pgs: 32 unknown, 238 active+clean; 8.4 MiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:18.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:18.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:18.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:18.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:18.885 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:18.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:18.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:19.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:19.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.046+0000 7f8acb4bf700 1 -- 172.21.15.28:0/1934117779 >> v1:172.21.15.28:6790/0 conn(0x7f8ac410eab0 legacy=0x7f8ac4110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:19.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- 172.21.15.28:0/1934117779 shutdown_connections 2024-03-30T15:07:19.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- 172.21.15.28:0/1934117779 >> 172.21.15.28:0/1934117779 conn(0x7f8ac40fdb30 msgr2=0x7f8ac40fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:19.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- 172.21.15.28:0/1934117779 shutdown_connections 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- 172.21.15.28:0/1934117779 wait complete. 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 librados: starting msgr at 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 librados: starting objecter 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 Processor -- start 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- start start 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 librados: setting wanted keys 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 librados: calling monclient init 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.047+0000 7f8acb4bf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ac4076040 con 0x7f8ac410eab0 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8acb4bf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ac4076250 con 0x7f8ac410af10 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8acb4bf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ac4076460 con 0x7f8ac4102280 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8ac925b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8ac4102280 0x7f8ac4074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44656/0 (socket says 172.21.15.28:44656) 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8ac925b700 1 -- 172.21.15.28:0/2213637182 learned_addr learned my addr 172.21.15.28:0/2213637182 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:19.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8ac9a5c700 1 --1- 172.21.15.28:0/2213637182 >> v1:172.21.15.28:6789/0 conn(0x7f8ac410eab0 0x7f8ac41a8340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37134/0 (socket says 172.21.15.28:37134) 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter ms_handle_connect 0x7f8ac4102280 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter ms_handle_connect 0x7f8ac410eab0 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter ms_handle_connect 0x7f8ac410af10 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2508235190 0 0) 0x7f8ac4076460 con 0x7f8ac4102280 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ab00031f0 con 0x7f8ac4102280 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491585818 0 0) 0x7f8ac4076040 con 0x7f8ac410eab0 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ac4076460 con 0x7f8ac410eab0 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506221069 0 0) 0x7f8ac4076250 con 0x7f8ac410af10 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ac4076040 con 0x7f8ac410af10 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2843668030 0 0) 0x7f8ab00031f0 con 0x7f8ac4102280 2024-03-30T15:07:19.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ac4076250 con 0x7f8ac4102280 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ac00013d0 con 0x7f8ac4102280 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 565707956 0 0) 0x7f8ac4076040 con 0x7f8ac410af10 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ab00031f0 con 0x7f8ac410af10 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3050731975 0 0) 0x7f8ac4076460 con 0x7f8ac410eab0 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ac4076040 con 0x7f8ac410eab0 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ab40017b0 con 0x7f8ac410af10 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8abc0047f0 con 0x7f8ac410eab0 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 165618645 0 0) 0x7f8ac4076250 con 0x7f8ac4102280 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 >> v1:172.21.15.134:6789/0 conn(0x7f8ac410af10 legacy=0x7f8ac4074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 >> v1:172.21.15.28:6789/0 conn(0x7f8ac410eab0 legacy=0x7f8ac41a8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.048+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac41acad0 con 0x7f8ac4102280 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac41a9ac0 con 0x7f8ac4102280 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8acb4bf700 10 client.58421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ac41a9f80 con 0x7f8ac4102280 2024-03-30T15:07:19.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8acb4bf700 1 librados: init done 2024-03-30T15:07:19.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8ac0001630 con 0x7f8ac4102280 2024-03-30T15:07:19.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ac0005cf0 con 0x7f8ac4102280 2024-03-30T15:07:19.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.049+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa8005150 con 0x7f8ac4102280 2024-03-30T15:07:19.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.050+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8ac0003930 con 0x7f8ac4102280 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.050+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(589..589 src has 442..589) ==== 8664+0+0 (unknown 1491642838 0 0) 0x7f8ac0089ad0 con 0x7f8ac4102280 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.050+0000 7f8aba7fc700 10 client.58421.objecter ms_dispatch 0x7f8ac4001550 osd_map(589..589 src has 442..589) 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.052+0000 7f8aba7fc700 3 client.58421.objecter handle_osd_map got epochs [589,589] > 0 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.052+0000 7f8aba7fc700 3 client.58421.objecter handle_osd_map decoding full epoch 589 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.052+0000 7f8aba7fc700 20 client.58421.objecter dump_active .. 0 homeless 2024-03-30T15:07:19.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.052+0000 7f8aba7fc700 10 client.58421.objecter ms_handle_connect 0x7f8ab0070580 2024-03-30T15:07:19.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.055+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8ac0056af0 con 0x7f8ac4102280 2024-03-30T15:07:19.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:19.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:19.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:19.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.274+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f8aa8005440 con 0x7f8ac4102280 2024-03-30T15:07:19.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:19.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.372+0000 7f8aba7fc700 1 -- 172.21.15.28:0/2213637182 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v590) ==== 113+0+0 (unknown 1105889237 0 0) 0x7f8ac005a540 con 0x7f8ac4102280 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.375+0000 7f8acb4bf700 10 librados: watch_flush enter 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.375+0000 7f8acb4bf700 10 librados: watch_flush exit 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.375+0000 7f8acb4bf700 20 client.58421.objecter shutdown clearing up homeless session... 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.375+0000 7f8acb4bf700 10 client.58421.objecter successfully canceled tick 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8ab0070580 legacy=0x7f8ab0072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 >> v1:172.21.15.28:6790/0 conn(0x7f8ac4102280 legacy=0x7f8ac4074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:19.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 shutdown_connections 2024-03-30T15:07:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 >> 172.21.15.28:0/2213637182 conn(0x7f8ac40fdb30 msgr2=0x7f8ac4111800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 shutdown_connections 2024-03-30T15:07:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 -- 172.21.15.28:0/2213637182 wait complete. 2024-03-30T15:07:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:19.376+0000 7f8acb4bf700 1 librados: shutdown 2024-03-30T15:07:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T15:07:19.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:19.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:19.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:07:19.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:07:19.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:19.504 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2213637182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.505 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:19 smithi134 ceph-mon[33714]: from='client.58421 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:19.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:07:19.543 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2213637182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[41614]: from='client.58421 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: osdmap e589: 8 total, 8 up, 8 in 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2213637182' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:19 smithi028 ceph-mon[29876]: from='client.58421 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:07:19.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:19.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:19.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:19.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:19.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:20.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:20.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:20.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:20.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:20.153 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:07:20.154 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:07:20.154 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:07:20.154 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:07:20.154 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:07:20.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:20.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:20.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:20.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:20.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 101.7 scrub starts 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 101.7 scrub ok 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: from='client.58421 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:07:20.616 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 42.b scrub starts 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: 42.b scrub ok 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: pgmap v868: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]': finished 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[41614]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:07:20.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 101.7 scrub starts 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 101.7 scrub ok 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: from='client.58421 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 42.b scrub starts 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: 42.b scrub ok 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: pgmap v868: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]': finished 2024-03-30T15:07:20.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:20 smithi028 ceph-mon[29876]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T15:07:20.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:20.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:20.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:20.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:07:20.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:07:20.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 101.7 scrub starts 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 101.7 scrub ok 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-6", "overlaypool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: from='client.58421 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: osdmap e590: 8 total, 8 up, 8 in 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]: dispatch 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 42.b scrub starts 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: 42.b scrub ok 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: pgmap v868: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-100", "mode": "writeback"}]': finished 2024-03-30T15:07:20.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:20 smithi134 ceph-mon[33714]: osdmap e591: 8 total, 8 up, 8 in 2024-03-30T15:07:20.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:07:20.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:20.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:20.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:07:20.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:20.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:20.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:21.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:21.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:21.112 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:21.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:21.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-03-30T15:07:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.186 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:21.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:21.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:21.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:21.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 101.f scrub starts 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 101.f scrub ok 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:07:21.416 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 101.f scrub starts 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 101.f scrub ok 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.417 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:21 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 -- 172.21.15.28:0/3619292642 >> v1:172.21.15.28:6790/0 conn(0x7f181410aed0 legacy=0x7f181410d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 -- 172.21.15.28:0/3619292642 shutdown_connections 2024-03-30T15:07:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 -- 172.21.15.28:0/3619292642 >> 172.21.15.28:0/3619292642 conn(0x7f18140fdaf0 msgr2=0x7f18140fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 -- 172.21.15.28:0/3619292642 shutdown_connections 2024-03-30T15:07:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 -- 172.21.15.28:0/3619292642 wait complete. 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 librados: starting msgr at 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.444+0000 7f181b0ca700 1 librados: starting objecter 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 Processor -- start 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 -- start start 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 librados: setting wanted keys 2024-03-30T15:07:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 librados: calling monclient init 2024-03-30T15:07:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18141a7870 con 0x7f1814107930 2024-03-30T15:07:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18141a8a90 con 0x7f1814102240 2024-03-30T15:07:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f181b0ca700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f18141a9cd0 con 0x7f181410ea70 2024-03-30T15:07:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1813fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f181410ea70 0x7f18141050f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44676/0 (socket says 172.21.15.28:44676) 2024-03-30T15:07:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1813fff700 1 -- 172.21.15.28:0/1149248479 learned_addr learned my addr 172.21.15.28:0/1149248479 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1819667700 1 --1- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6789/0 conn(0x7f1814107930 0x7f18141a5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37154/0 (socket says 172.21.15.28:37154) 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1811ffb700 10 client.?.objecter ms_handle_connect 0x7f1814102240 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1811ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1811ffb700 10 client.?.objecter ms_handle_connect 0x7f181410ea70 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.445+0000 7f1811ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 10 client.?.objecter ms_handle_connect 0x7f1814107930 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 196907020 0 0) 0x7f18141a8a90 con 0x7f1814102240 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17fc0035b0 con 0x7f1814102240 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403327151 0 0) 0x7f18141a9cd0 con 0x7f181410ea70 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18141a8a90 con 0x7f181410ea70 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 633639151 0 0) 0x7f17fc0035b0 con 0x7f1814102240 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18141a9cd0 con 0x7f1814102240 2024-03-30T15:07:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453453873 0 0) 0x7f18141a7870 con 0x7f1814107930 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17fc0035b0 con 0x7f1814107930 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1804001830 con 0x7f1814102240 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1718704956 0 0) 0x7f18141a8a90 con 0x7f181410ea70 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18141a7870 con 0x7f181410ea70 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3404113444 0 0) 0x7f18141a9cd0 con 0x7f1814102240 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6790/0 conn(0x7f181410ea70 legacy=0x7f18141050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6789/0 conn(0x7f1814107930 legacy=0x7f18141a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18141aaf10 con 0x7f1814102240 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.446+0000 7f181b0ca700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18141a9f00 con 0x7f1814102240 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.447+0000 7f181b0ca700 10 client.50434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.447+0000 7f181b0ca700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f18141aa3c0 con 0x7f1814102240 2024-03-30T15:07:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.447+0000 7f181b0ca700 1 librados: init done 2024-03-30T15:07:21.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.447+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1804003b40 con 0x7f1814102240 2024-03-30T15:07:21.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.447+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f18040049c0 con 0x7f1814102240 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.448+0000 7f181b0ca700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1800005100 con 0x7f1814102240 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.448+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f180401b200 con 0x7f1814102240 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(592..592 src has 442..592) ==== 8664+0+0 (unknown 3651511725 0 0) 0x7f18040887c0 con 0x7f1814102240 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 osd_map(592..592 src has 442..592) 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 3 client.50434.objecter handle_osd_map got epochs [592,592] > 0 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 3 client.50434.objecter handle_osd_map decoding full epoch 592 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 20 client.50434.objecter dump_active .. 0 homeless 2024-03-30T15:07:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v592) ==== 54+0+16 (unknown 4033144265 0 2420000192) 0x7f1804088bb0 con 0x7f1814102240 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.449+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17fc0703c0 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 1 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 1 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 0 1 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 4 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 10 client.50434.objecter _send_command 1 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17f4006d40 con 0x7f17f4002fd0 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.450+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f4002fd0 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.451+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4006d40 con 0x7f17f4002fd0 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.451+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:21.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.451+0000 7f1811ffb700 10 client.50434.objecter _finish_command 1 = system:0 2024-03-30T15:07:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.451+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 4 2024-03-30T15:07:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.451+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 0 1 2024-03-30T15:07:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 2 2024-03-30T15:07:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 2 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40028e0 osd=0 3 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 0 2 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 4 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 10 client.50434.objecter _send_command 2 2024-03-30T15:07:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.475+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f40012a0 con 0x7f17f4002fd0 2024-03-30T15:07:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.477+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f40012a0 con 0x7f17f4002fd0 2024-03-30T15:07:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.477+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.477+0000 7f1811ffb700 10 client.50434.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.477+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 4 2024-03-30T15:07:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.477+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 0 2 2024-03-30T15:07:21.479 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 1 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 4 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 10 client.50434.objecter _send_command 3 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f17f4008ca0 con 0x7f17f4008100 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.478+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f4008100 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.479+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4008ca0 con 0x7f17f4008100 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.479+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.479+0000 7f1811ffb700 10 client.50434.objecter _finish_command 3 = system:0 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.479+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 4 2024-03-30T15:07:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.479+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 1 3 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 4 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 4 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4001f70 osd=1 3 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 1 4 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 4 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 10 client.50434.objecter _send_command 4 2024-03-30T15:07:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.498+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f4009620 con 0x7f17f4008100 2024-03-30T15:07:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f4009620 con 0x7f17f4008100 2024-03-30T15:07:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f1811ffb700 10 client.50434.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 4 2024-03-30T15:07:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 1 4 2024-03-30T15:07:21.502 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 5 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 5 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 2 5 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 4 2024-03-30T15:07:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 10 client.50434.objecter _send_command 5 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.500+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f17f4010b20 con 0x7f17f400ce70 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.501+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f400ce70 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.502+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4010b20 con 0x7f17f400ce70 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.502+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.502+0000 7f1811ffb700 10 client.50434.objecter _finish_command 5 = system:0 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.502+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 4 2024-03-30T15:07:21.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.502+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 2 5 2024-03-30T15:07:21.508 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 6 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 6 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4001d00 osd=2 3 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 2 6 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 4 2024-03-30T15:07:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 10 client.50434.objecter _send_command 6 2024-03-30T15:07:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.520+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f40091c0 con 0x7f17f400ce70 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f40091c0 con 0x7f17f400ce70 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 10 client.50434.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 4 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 2 6 2024-03-30T15:07:21.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 7 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 7 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 3 7 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 4 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 10 client.50434.objecter _send_command 7 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f17f4011840 con 0x7f17f4010cd0 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.522+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f4010cd0 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.523+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4011840 con 0x7f17f4010cd0 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.523+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.523+0000 7f1811ffb700 10 client.50434.objecter _finish_command 7 = system:0 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.523+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 4 2024-03-30T15:07:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.523+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 3 7 2024-03-30T15:07:21.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 8 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 8 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4009420 osd=3 3 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 3 8 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 4 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 10 client.50434.objecter _send_command 8 2024-03-30T15:07:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.540+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f40121f0 con 0x7f17f4010cd0 2024-03-30T15:07:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.542+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f40121f0 con 0x7f17f4010cd0 2024-03-30T15:07:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.542+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.542+0000 7f1811ffb700 10 client.50434.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 4 2024-03-30T15:07:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 3 8 2024-03-30T15:07:21.544 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 9 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 9 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 4 9 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 4 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 10 client.50434.objecter _send_command 9 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f17f401a1f0 con 0x7f17f40166e0 2024-03-30T15:07:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.543+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f40166e0 2024-03-30T15:07:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.544+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f401a1f0 con 0x7f17f40166e0 2024-03-30T15:07:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.544+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.544+0000 7f1811ffb700 10 client.50434.objecter _finish_command 9 = system:0 2024-03-30T15:07:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.544+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 4 2024-03-30T15:07:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.544+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 4 9 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 101.f scrub starts 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 101.f scrub ok 2024-03-30T15:07:21.561 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:07:21.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:07:21.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:21.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: osdmap e592: 8 total, 8 up, 8 in 2024-03-30T15:07:21.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:21 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 10 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 10 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40121f0 osd=4 3 2024-03-30T15:07:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 4 10 2024-03-30T15:07:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 4 2024-03-30T15:07:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 10 client.50434.objecter _send_command 10 2024-03-30T15:07:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.563+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f4007230 con 0x7f17f40166e0 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f4007230 con 0x7f17f40166e0 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f1811ffb700 10 client.50434.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 4 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 4 10 2024-03-30T15:07:21.567 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 11 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 11 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 5 11 2024-03-30T15:07:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 4 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 10 client.50434.objecter _send_command 11 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.566+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f17f401cac0 con 0x7f17f4012c20 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f4012c20 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f401cac0 con 0x7f17f4012c20 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 10 client.50434.objecter _finish_command 11 = system:0 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 4 2024-03-30T15:07:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.567+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 5 11 2024-03-30T15:07:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 12 2024-03-30T15:07:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 12 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f4007270 osd=5 3 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 5 12 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 4 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 10 client.50434.objecter _send_command 12 2024-03-30T15:07:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.585+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f401cac0 con 0x7f17f4012c20 2024-03-30T15:07:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f401cac0 con 0x7f17f4012c20 2024-03-30T15:07:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f1811ffb700 10 client.50434.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 4 2024-03-30T15:07:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 5 12 2024-03-30T15:07:21.589 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 13 2024-03-30T15:07:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 13 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 6 13 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 4 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 10 client.50434.objecter _send_command 13 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.587+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f17f4022b30 con 0x7f17f40131b0 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.588+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f40131b0 2024-03-30T15:07:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.589+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4022b30 con 0x7f17f40131b0 2024-03-30T15:07:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.589+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.589+0000 7f1811ffb700 10 client.50434.objecter _finish_command 13 = system:0 2024-03-30T15:07:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.589+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 4 2024-03-30T15:07:21.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.589+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 6 13 2024-03-30T15:07:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 14 2024-03-30T15:07:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 14 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f401cac0 osd=6 3 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 6 14 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 4 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 10 client.50434.objecter _send_command 14 2024-03-30T15:07:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.605+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f40244e0 con 0x7f17f40131b0 2024-03-30T15:07:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f40244e0 con 0x7f17f40131b0 2024-03-30T15:07:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f1811ffb700 10 client.50434.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 4 2024-03-30T15:07:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 6 14 2024-03-30T15:07:21.609 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 15 2024-03-30T15:07:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 15 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 7 15 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 4 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 10 client.50434.objecter _send_command 15 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.607+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f17f4025060 con 0x7f17f4024c00 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 10 client.50434.objecter ms_handle_connect 0x7f17f4024c00 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f17f4025060 con 0x7f17f4024c00 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 10 client.50434.objecter _finish_command 15 = system:0 2024-03-30T15:07:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 4 2024-03-30T15:07:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.608+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 7 15 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 10 client.50434.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign -1 16 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 20 client.50434.objecter _get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_remove -1 16 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 20 client.50434.objecter get_session s=0x7f17f40244e0 osd=7 3 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 15 client.50434.objecter _session_command_op_assign 7 16 2024-03-30T15:07:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 4 2024-03-30T15:07:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 10 client.50434.objecter _send_command 16 2024-03-30T15:07:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.626+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f17f4025060 con 0x7f17f4024c00 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1811ffb700 1 -- 172.21.15.28:0/1149248479 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f17f4025060 con 0x7f17f4024c00 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1811ffb700 10 client.50434.objecter ms_dispatch 0x7f1814001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1811ffb700 10 client.50434.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1811ffb700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 4 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1811ffb700 15 client.50434.objecter _session_command_op_remove 7 16 2024-03-30T15:07:21.629 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 librados: watch_flush enter 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 librados: watch_flush exit 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.0 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6801/3394323575 conn(0x7f17f4002fd0 legacy=0x7f17f4005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40028e0 osd=0 1 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.1 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6805/1850313547 conn(0x7f17f4008100 legacy=0x7f17f4008590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001f70 osd=1 1 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.2 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6809/3451325043 conn(0x7f17f400ce70 legacy=0x7f17f400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4001d00 osd=2 1 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.3 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6813/1506157250 conn(0x7f17f4010cd0 legacy=0x7f17f4011130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4009420 osd=3 1 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.4 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.134:6800/3063775514 conn(0x7f17f40166e0 legacy=0x7f17f4018ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40121f0 osd=4 1 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.5 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.134:6804/3668151179 conn(0x7f17f4012c20 legacy=0x7f17f401b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f4007270 osd=5 1 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.6 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.134:6808/2135224194 conn(0x7f17f40131b0 legacy=0x7f17f4021280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f401cac0 osd=6 1 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1819667700 1 -- 172.21.15.28:0/1149248479 reap_dead start 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter close_session for osd.7 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.134:6812/3758190298 conn(0x7f17f4024c00 legacy=0x7f17f4022d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1819667700 1 -- 172.21.15.28:0/1149248479 reap_dead start 2024-03-30T15:07:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter put_session s=0x7f17f40244e0 osd=7 1 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 20 client.50434.objecter shutdown clearing up homeless session... 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 10 client.50434.objecter successfully canceled tick 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1819667700 1 -- 172.21.15.28:0/1149248479 reap_dead start 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f1819667700 1 -- 172.21.15.28:0/1149248479 reap_dead start 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.28:6800/3011431093 conn(0x7f17fc0703c0 legacy=0x7f17fc072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> v1:172.21.15.134:6789/0 conn(0x7f1814102240 legacy=0x7f18141049e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 shutdown_connections 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.628+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 >> 172.21.15.28:0/1149248479 conn(0x7f18140fdaf0 msgr2=0x7f181410a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.629+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 shutdown_connections 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.629+0000 7f17fb7fe700 1 -- 172.21.15.28:0/1149248479 wait complete. 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:21.629+0000 7f17fb7fe700 1 librados: shutdown 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:21.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:21.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:21.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:21.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:21.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:21.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:21.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:22.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:22.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:22.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:22.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:22.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:22.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:22.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 101.15 scrub starts 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 101.15 scrub ok 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1149248479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: pgmap v871: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 101.15 scrub starts 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 101.15 scrub ok 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1149248479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:07:22.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: pgmap v871: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.475 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:22 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:22.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:22.610 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 101.15 scrub starts 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 101.15 scrub ok 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1149248479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: pgmap v871: 270 pgs: 270 active+clean; 8.4 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:07:22.611 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: osdmap e593: 8 total, 8 up, 8 in 2024-03-30T15:07:22.612 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.612 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:22 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:07:22.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:22.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:22.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:22.892 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:22.892 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:22.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:22.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:22.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:23.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:23.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:23.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:23.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:23.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:23.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:23.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:23.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:23.474 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:07:23.474 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.475 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:23 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:07:23.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:23.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:23.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:23.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:07:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:07:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: osdmap e594: 8 total, 8 up, 8 in 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:23 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-30T15:07:23.756 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:23.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:23.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:23.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:23.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:24.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:24.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:24.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:24.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:24.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:24.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: pgmap v874: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T15:07:24.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:24 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:24.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:24.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:24.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:24.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:07:24.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: pgmap v874: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: pgmap v874: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: osdmap e595: 8 total, 8 up, 8 in 2024-03-30T15:07:24.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:24 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:24.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:24.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.794+0000 7fbcdfd41700 1 -- 172.21.15.28:0/2433494350 >> v1:172.21.15.28:6790/0 conn(0x7fbcd8102240 legacy=0x7fbcd8102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.794+0000 7fbcdfd41700 1 -- 172.21.15.28:0/2433494350 shutdown_connections 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.794+0000 7fbcdfd41700 1 -- 172.21.15.28:0/2433494350 >> 172.21.15.28:0/2433494350 conn(0x7fbcd80fdaf0 msgr2=0x7fbcd80fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.794+0000 7fbcdfd41700 1 -- 172.21.15.28:0/2433494350 shutdown_connections 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.794+0000 7fbcdfd41700 1 -- 172.21.15.28:0/2433494350 wait complete. 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 librados: starting msgr at 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 librados: starting objecter 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 Processor -- start 2024-03-30T15:07:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 -- start start 2024-03-30T15:07:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 librados: setting wanted keys 2024-03-30T15:07:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 librados: calling monclient init 2024-03-30T15:07:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcd81079a0 con 0x7fbcd810aed0 2024-03-30T15:07:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcd81a8640 con 0x7fbcd8102240 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdfd41700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcd81a9820 con 0x7fbcd810ea70 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdd2dc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbcd810aed0 0x7fbcd8104f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37174/0 (socket says 172.21.15.28:37174) 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcde2de700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbcd810ea70 0x7fbcd81a5f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44698/0 (socket says 172.21.15.28:44698) 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcdd2dc700 1 -- 172.21.15.28:0/171550482 learned_addr learned my addr 172.21.15.28:0/171550482 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter ms_handle_connect 0x7fbcd810aed0 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter ms_handle_connect 0x7fbcd810ea70 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter ms_handle_connect 0x7fbcd8102240 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.795+0000 7fbcceffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557233287 0 0) 0x7fbcd81079a0 con 0x7fbcd810aed0 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcc40035b0 con 0x7fbcd810aed0 2024-03-30T15:07:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300043383 0 0) 0x7fbcd81a9820 con 0x7fbcd810ea70 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd81079a0 con 0x7fbcd810ea70 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366460328 0 0) 0x7fbcd81a8640 con 0x7fbcd8102240 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd81a9820 con 0x7fbcd8102240 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2141714178 0 0) 0x7fbcc40035b0 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcd81a8640 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbcc80018d0 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 242033833 0 0) 0x7fbcd81079a0 con 0x7fbcd810ea70 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcc40035b0 con 0x7fbcd810ea70 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbcd0001af0 con 0x7fbcd810ea70 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3692004068 0 0) 0x7fbcd81a8640 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 >> v1:172.21.15.28:6790/0 conn(0x7fbcd810ea70 legacy=0x7fbcd81a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 >> v1:172.21.15.134:6789/0 conn(0x7fbcd8102240 legacy=0x7fbcd8104840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd81aaa00 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd81a9a50 con 0x7fbcd810aed0 2024-03-30T15:07:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcdfd41700 10 client.25233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbcd81a9f10 con 0x7fbcd810aed0 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcdfd41700 1 librados: init done 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbcc8001ca0 con 0x7fbcd810aed0 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.796+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbcc8005d50 con 0x7fbcd810aed0 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.797+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcbc005120 con 0x7fbcd810aed0 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.798+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbcc8003950 con 0x7fbcd810aed0 2024-03-30T15:07:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(595..595 src has 442..595) ==== 8664+0+0 (unknown 1977835232 0 0) 0x7fbcc80898d0 con 0x7fbcd810aed0 2024-03-30T15:07:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 10 client.25233.objecter ms_dispatch 0x7fbcd8001550 osd_map(595..595 src has 442..595) 2024-03-30T15:07:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 3 client.25233.objecter handle_osd_map got epochs [595,595] > 0 2024-03-30T15:07:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 3 client.25233.objecter handle_osd_map decoding full epoch 595 2024-03-30T15:07:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 20 client.25233.objecter dump_active .. 0 homeless 2024-03-30T15:07:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.799+0000 7fbcceffd700 10 client.25233.objecter ms_handle_connect 0x7fbcc4070130 2024-03-30T15:07:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:24.803+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbcc8005d50 con 0x7fbcd810aed0 2024-03-30T15:07:24.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:24.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:24.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:24.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:24.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:24.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:24.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:07:25.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.025+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fbcbc005410 con 0x7fbcd810aed0 2024-03-30T15:07:25.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:25.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:25.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:25.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:25.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.335+0000 7fbcceffd700 1 -- 172.21.15.28:0/171550482 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v596) ==== 101+0+0 (unknown 2019852139 0 0) 0x7fbcc80568f0 con 0x7fbcd810aed0 2024-03-30T15:07:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.339+0000 7fbcdfd41700 10 librados: watch_flush enter 2024-03-30T15:07:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.340+0000 7fbcdfd41700 10 librados: watch_flush exit 2024-03-30T15:07:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.340+0000 7fbcdfd41700 20 client.25233.objecter shutdown clearing up homeless session... 2024-03-30T15:07:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.340+0000 7fbcdfd41700 10 client.25233.objecter successfully canceled tick 2024-03-30T15:07:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.340+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbcc4070130 legacy=0x7fbcc40725f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.340+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 >> v1:172.21.15.28:6789/0 conn(0x7fbcd810aed0 legacy=0x7fbcd8104f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.341+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 shutdown_connections 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.341+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 >> 172.21.15.28:0/171550482 conn(0x7fbcd80fdaf0 msgr2=0x7fbcd810a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.341+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 shutdown_connections 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.341+0000 7fbcdfd41700 1 -- 172.21.15.28:0/171550482 wait complete. 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:25.341+0000 7fbcdfd41700 1 librados: shutdown 2024-03-30T15:07:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:07:25.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:25.422 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T15:07:25.422 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 3.6 scrub starts 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: 3.6 scrub ok 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.423 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:25 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:25.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:25.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:25.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T15:07:25.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 3.6 scrub starts 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: 3.6 scrub ok 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 3.6 scrub starts 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: 3.6 scrub ok 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]': finished 2024-03-30T15:07:25.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/171550482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:25.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: osdmap e596: 8 total, 8 up, 8 in 2024-03-30T15:07:25.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:25 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]: dispatch 2024-03-30T15:07:25.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:25.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:25.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:25.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:25.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:25.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:26.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:07:26.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:26.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:26.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.239+0000 7f925eb07700 1 -- 172.21.15.28:0/2374511196 >> v1:172.21.15.28:6789/0 conn(0x7f925810aee0 legacy=0x7f925810d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.239+0000 7f925eb07700 1 -- 172.21.15.28:0/2374511196 shutdown_connections 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.239+0000 7f925eb07700 1 -- 172.21.15.28:0/2374511196 >> 172.21.15.28:0/2374511196 conn(0x7f92580fdb20 msgr2=0x7f92580fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.239+0000 7f925eb07700 1 -- 172.21.15.28:0/2374511196 shutdown_connections 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.239+0000 7f925eb07700 1 -- 172.21.15.28:0/2374511196 wait complete. 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 librados: starting msgr at 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 librados: starting objecter 2024-03-30T15:07:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 Processor -- start 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 -- start start 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 librados: setting wanted keys 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 librados: calling monclient init 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92581a93d0 con 0x7f925810aee0 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92581aa5f0 con 0x7f925810ea80 2024-03-30T15:07:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f925eb07700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f92581ab830 con 0x7f9258102250 2024-03-30T15:07:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9257fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f925810aee0 0x7f92581a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37200/0 (socket says 172.21.15.28:37200) 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9257fff700 1 -- 172.21.15.28:0/43070878 learned_addr learned my addr 172.21.15.28:0/43070878 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter ms_handle_connect 0x7f925810aee0 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter ms_handle_connect 0x7f9258102250 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter ms_handle_connect 0x7f925810ea80 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575456203 0 0) 0x7f92581ab830 con 0x7f9258102250 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92480031f0 con 0x7f9258102250 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.240+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1597286856 0 0) 0x7f92581a93d0 con 0x7f925810aee0 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92581ab830 con 0x7f925810aee0 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920922590 0 0) 0x7f92581aa5f0 con 0x7f925810ea80 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92581a93d0 con 0x7f925810ea80 2024-03-30T15:07:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 490009002 0 0) 0x7f92480031f0 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92581aa5f0 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1148945468 0 0) 0x7f92581ab830 con 0x7f925810aee0 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92480031f0 con 0x7f925810aee0 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f924c001130 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f92440044e0 con 0x7f925810aee0 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2176031040 0 0) 0x7f92581a93d0 con 0x7f925810ea80 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92581ab830 con 0x7f925810ea80 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9250001bb0 con 0x7f925810ea80 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 901593522 0 0) 0x7f92581aa5f0 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 >> v1:172.21.15.134:6789/0 conn(0x7f925810ea80 legacy=0x7f92581a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 >> v1:172.21.15.28:6789/0 conn(0x7f925810aee0 legacy=0x7f92581a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92581aca70 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92581aba60 con 0x7f9258102250 2024-03-30T15:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f924c003b20 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f924c004a30 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f925eb07700 10 client.58439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f92581abf20 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.241+0000 7f925eb07700 1 librados: init done 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.242+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258103730 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f924c003680 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(596..596 src has 442..596) ==== 8664+0+0 (unknown 2954582559 0 0) 0x7f924c088740 con 0x7f9258102250 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 10 client.58439.objecter ms_dispatch 0x7f9258001550 osd_map(596..596 src has 442..596) 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 3 client.58439.objecter handle_osd_map got epochs [596,596] > 0 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 3 client.58439.objecter handle_osd_map decoding full epoch 596 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 20 client.58439.objecter dump_active .. 0 homeless 2024-03-30T15:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.243+0000 7f9255ffb700 10 client.58439.objecter ms_handle_connect 0x7f92480704a0 2024-03-30T15:07:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.248+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f924c055760 con 0x7f9258102250 2024-03-30T15:07:26.300 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:26.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:26.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:26.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:26.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:26.481+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f9258075a70 con 0x7f9258102250 2024-03-30T15:07:26.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:26.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 42.14 scrub starts 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 42.14 scrub ok 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T15:07:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: pgmap v877: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:26 smithi134 ceph-mon[33714]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 42.14 scrub starts 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 42.14 scrub ok 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T15:07:26.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: pgmap v877: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[41614]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 42.14 scrub starts 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 42.14 scrub ok 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T15:07:26.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T15:07:26.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: pgmap v877: 270 pgs: 270 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:26.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-100"}]': finished 2024-03-30T15:07:26.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:26 smithi028 ceph-mon[29876]: osdmap e597: 8 total, 8 up, 8 in 2024-03-30T15:07:26.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:26.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:26.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:26.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:26.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:27.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:27.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:27.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:27.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:27.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.343+0000 7f9255ffb700 1 -- 172.21.15.28:0/43070878 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v598) ==== 113+0+0 (unknown 4153383681 0 0) 0x7f924c0591b0 con 0x7f9258102250 2024-03-30T15:07:27.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 10 librados: watch_flush enter 2024-03-30T15:07:27.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 10 librados: watch_flush exit 2024-03-30T15:07:27.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 20 client.58439.objecter shutdown clearing up homeless session... 2024-03-30T15:07:27.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 10 client.58439.objecter successfully canceled tick 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 >> v1:172.21.15.28:6800/3011431093 conn(0x7f92480704a0 legacy=0x7f9248072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 >> v1:172.21.15.28:6790/0 conn(0x7f9258102250 legacy=0x7f9258107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 shutdown_connections 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 >> 172.21.15.28:0/43070878 conn(0x7f92580fdb20 msgr2=0x7f9258111bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 shutdown_connections 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 -- 172.21.15.28:0/43070878 wait complete. 2024-03-30T15:07:27.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:27.347+0000 7f925eb07700 1 librados: shutdown 2024-03-30T15:07:27.356 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T15:07:27.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:27.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:27.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:27.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:27.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:27.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:27.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:27.759 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.4 scrub starts 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.4 scrub ok 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.1a scrub starts 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.1a scrub ok 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.e deep-scrub starts 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 42.e deep-scrub ok 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/43070878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: from='client.58439 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: from='client.58439 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:07:27.760 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:27 smithi134 ceph-mon[33714]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T15:07:27.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:27.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:27.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.4 scrub starts 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.4 scrub ok 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.1a scrub starts 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.1a scrub ok 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.e deep-scrub starts 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 42.e deep-scrub ok 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/43070878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: from='client.58439 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:07:27.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: from='client.58439 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[41614]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.4 scrub starts 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.4 scrub ok 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.1a scrub starts 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.1a scrub ok 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.e deep-scrub starts 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 42.e deep-scrub ok 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/43070878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: from='client.58439 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:07:27.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:07:27.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: from='client.58439 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:07:27.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:27 smithi028 ceph-mon[29876]: osdmap e598: 8 total, 8 up, 8 in 2024-03-30T15:07:27.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:28.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:28.017 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:28.017 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:28.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:28.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:28.352 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:28.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:28.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:28.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:28.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:28.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:28.642 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:28.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:07:28.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-03-30T15:07:28.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:28.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:28.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:07:28.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: pgmap v880: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: pgmap v880: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:28.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:07:28.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:07:28.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T15:07:28.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:28 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.885 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:07:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.885+0000 7fca0a0af700 1 -- 172.21.15.28:0/3522289729 >> v1:172.21.15.28:6790/0 conn(0x7fca04101130 legacy=0x7fca04101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.886+0000 7fca0a0af700 1 -- 172.21.15.28:0/3522289729 shutdown_connections 2024-03-30T15:07:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.886+0000 7fca0a0af700 1 -- 172.21.15.28:0/3522289729 >> 172.21.15.28:0/3522289729 conn(0x7fca040fc9e0 msgr2=0x7fca040fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.886+0000 7fca0a0af700 1 -- 172.21.15.28:0/3522289729 shutdown_connections 2024-03-30T15:07:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.886+0000 7fca0a0af700 1 -- 172.21.15.28:0/3522289729 wait complete. 2024-03-30T15:07:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 librados: starting msgr at 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 librados: starting objecter 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 Processor -- start 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 -- start start 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 librados: setting wanted keys 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 librados: calling monclient init 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca041068b0 con 0x7fca04109dc0 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca041a8540 con 0x7fca04101130 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca0a0af700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fca041a9720 con 0x7fca0410d960 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca03fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fca0410d960 0x7fca041a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42016/0 (socket says 172.21.15.28:42016) 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca03fff700 1 -- 172.21.15.28:0/598042650 learned_addr learned my addr 172.21.15.28:0/598042650 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.887+0000 7fca00ff9700 10 client.?.objecter ms_handle_connect 0x7fca0410d960 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 10 client.?.objecter ms_handle_connect 0x7fca04109dc0 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 10 client.?.objecter ms_handle_connect 0x7fca04101130 2024-03-30T15:07:28.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201961722 0 0) 0x7fca041a9720 con 0x7fca0410d960 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f00031f0 con 0x7fca0410d960 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2307523001 0 0) 0x7fca041a8540 con 0x7fca04101130 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca041a9720 con 0x7fca04101130 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665727546 0 0) 0x7fca041068b0 con 0x7fca04109dc0 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca041a8540 con 0x7fca04109dc0 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3084934239 0 0) 0x7fca041a9720 con 0x7fca04101130 2024-03-30T15:07:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca041068b0 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9f4001840 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2548772697 0 0) 0x7fca041a8540 con 0x7fca04109dc0 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca041a9720 con 0x7fca04109dc0 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 781012340 0 0) 0x7fc9f00031f0 con 0x7fca0410d960 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca041a8540 con 0x7fca0410d960 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9ec0017f0 con 0x7fca04109dc0 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9fc001650 con 0x7fca0410d960 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 451632392 0 0) 0x7fca041068b0 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6790/0 conn(0x7fca0410d960 legacy=0x7fca041a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6789/0 conn(0x7fca04109dc0 legacy=0x7fca04104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca041aa900 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.888+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca041a8770 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.889+0000 7fca0a0af700 10 client.50452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.889+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fca041a8c30 con 0x7fca04101130 2024-03-30T15:07:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.889+0000 7fca0a0af700 1 librados: init done 2024-03-30T15:07:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.889+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc9f4001b20 con 0x7fca04101130 2024-03-30T15:07:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.889+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc9f4004ba0 con 0x7fca04101130 2024-03-30T15:07:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.890+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc9e4005100 con 0x7fca04101130 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.890+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc9f4004d40 con 0x7fca04101130 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(599..599 src has 442..599) ==== 8643+0+0 (unknown 1548864794 0 0) 0x7fc9f40885a0 con 0x7fca04101130 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 osd_map(599..599 src has 442..599) 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 3 client.50452.objecter handle_osd_map got epochs [599,599] > 0 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 3 client.50452.objecter handle_osd_map decoding full epoch 599 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 20 client.50452.objecter dump_active .. 0 homeless 2024-03-30T15:07:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v599) ==== 54+0+16 (unknown 3303335062 0 2420000192) 0x7fc9f408a9d0 con 0x7fca04101130 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9f00706d0 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 10 client.50452.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 1 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 1 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 0 1 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 4 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 10 client.50452.objecter _send_command 1 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.891+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4004b50 con 0x7fc9e4004730 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e4004730 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4004b50 con 0x7fc9e4004730 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 10 client.50452.objecter _finish_command 1 = system:0 2024-03-30T15:07:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 4 2024-03-30T15:07:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.892+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 0 1 2024-03-30T15:07:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:28.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:28.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 10 client.50452.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 2 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 2 2024-03-30T15:07:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e40053e0 osd=0 3 2024-03-30T15:07:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 0 2 2024-03-30T15:07:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 4 2024-03-30T15:07:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 10 client.50452.objecter _send_command 2 2024-03-30T15:07:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.917+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4007b00 con 0x7fc9e4004730 2024-03-30T15:07:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.919+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4007b00 con 0x7fc9e4004730 2024-03-30T15:07:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.919+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.919+0000 7fca00ff9700 10 client.50452.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.919+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 4 2024-03-30T15:07:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.919+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 0 2 2024-03-30T15:07:28.921 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 10 client.50452.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 1 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 4 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 10 client.50452.objecter _send_command 3 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4002160 con 0x7fc9e40042a0 2024-03-30T15:07:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.920+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e40042a0 2024-03-30T15:07:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.921+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4002160 con 0x7fc9e40042a0 2024-03-30T15:07:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.921+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.921+0000 7fca00ff9700 10 client.50452.objecter _finish_command 3 = system:0 2024-03-30T15:07:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.921+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 4 2024-03-30T15:07:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.921+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 1 3 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: pgmap v880: 238 pgs: 238 active+clean; 8.4 MiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: osdmap e599: 8 total, 8 up, 8 in 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:28 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 10 client.50452.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 4 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 4 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4004130 osd=1 3 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 1 4 2024-03-30T15:07:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 4 2024-03-30T15:07:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.943+0000 7fca0a0af700 10 client.50452.objecter _send_command 4 2024-03-30T15:07:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.944+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4001630 con 0x7fc9e40042a0 2024-03-30T15:07:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.945+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4001630 con 0x7fc9e40042a0 2024-03-30T15:07:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.945+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.945+0000 7fca00ff9700 10 client.50452.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.945+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 4 2024-03-30T15:07:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.945+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 1 4 2024-03-30T15:07:28.950 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:28.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 10 client.50452.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 5 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 5 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 2 5 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 4 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 10 client.50452.objecter _send_command 5 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc9e40153f0 con 0x7fc9e4011850 2024-03-30T15:07:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.946+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e4011850 2024-03-30T15:07:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.947+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e40153f0 con 0x7fc9e4011850 2024-03-30T15:07:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.947+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.947+0000 7fca00ff9700 10 client.50452.objecter _finish_command 5 = system:0 2024-03-30T15:07:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.947+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 4 2024-03-30T15:07:28.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.947+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 2 5 2024-03-30T15:07:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 10 client.50452.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 6 2024-03-30T15:07:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 6 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4001a30 osd=2 3 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 2 6 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 4 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 10 client.50452.objecter _send_command 6 2024-03-30T15:07:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.970+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4009850 con 0x7fc9e4011850 2024-03-30T15:07:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4009850 con 0x7fc9e4011850 2024-03-30T15:07:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca00ff9700 10 client.50452.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:28.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 4 2024-03-30T15:07:28.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 2 6 2024-03-30T15:07:28.974 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 10 client.50452.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 7 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 7 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 3 7 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 4 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 10 client.50452.objecter _send_command 7 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.972+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4008c10 con 0x7fc9e40155d0 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.973+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e40155d0 2024-03-30T15:07:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.974+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4008c10 con 0x7fc9e40155d0 2024-03-30T15:07:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.974+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.974+0000 7fca00ff9700 10 client.50452.objecter _finish_command 7 = system:0 2024-03-30T15:07:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.974+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 4 2024-03-30T15:07:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.974+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 3 7 2024-03-30T15:07:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 10 client.50452.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 8 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 8 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4009ae0 osd=3 3 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 3 8 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 4 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 10 client.50452.objecter _send_command 8 2024-03-30T15:07:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.996+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4016980 con 0x7fc9e40155d0 2024-03-30T15:07:28.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.998+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4016980 con 0x7fc9e40155d0 2024-03-30T15:07:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.998+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.998+0000 7fca00ff9700 10 client.50452.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.998+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 4 2024-03-30T15:07:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.998+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 3 8 2024-03-30T15:07:29.000 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 10 client.50452.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 9 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 9 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 4 9 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 4 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 10 client.50452.objecter _send_command 9 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc9e401eac0 con 0x7fc9e401afb0 2024-03-30T15:07:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:28.999+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e401afb0 2024-03-30T15:07:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.000+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e401eac0 con 0x7fc9e401afb0 2024-03-30T15:07:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.000+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.000+0000 7fca00ff9700 10 client.50452.objecter _finish_command 9 = system:0 2024-03-30T15:07:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.000+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 4 2024-03-30T15:07:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.000+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 4 9 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 10 client.50452.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 10 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 10 2024-03-30T15:07:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4017b00 osd=4 3 2024-03-30T15:07:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 4 10 2024-03-30T15:07:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 4 2024-03-30T15:07:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 10 client.50452.objecter _send_command 10 2024-03-30T15:07:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.018+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4016bd0 con 0x7fc9e401afb0 2024-03-30T15:07:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4016bd0 con 0x7fc9e401afb0 2024-03-30T15:07:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca00ff9700 10 client.50452.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 4 2024-03-30T15:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 4 10 2024-03-30T15:07:29.022 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 10 client.50452.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 11 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 11 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 5 11 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 4 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 10 client.50452.objecter _send_command 11 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.020+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4017680 con 0x7fc9e40170e0 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e40170e0 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4017680 con 0x7fc9e40170e0 2024-03-30T15:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 10 client.50452.objecter _finish_command 11 = system:0 2024-03-30T15:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 4 2024-03-30T15:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.021+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 5 11 2024-03-30T15:07:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 10 client.50452.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 12 2024-03-30T15:07:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 12 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4007fc0 osd=5 3 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 5 12 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 4 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 10 client.50452.objecter _send_command 12 2024-03-30T15:07:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.038+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e40178e0 con 0x7fc9e40170e0 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e40178e0 con 0x7fc9e40170e0 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 10 client.50452.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 4 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 5 12 2024-03-30T15:07:29.041 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 10 client.50452.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 13 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 13 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 6 13 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 4 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 10 client.50452.objecter _send_command 13 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4027430 con 0x7fc9e4023880 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.040+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e4023880 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.041+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4027430 con 0x7fc9e4023880 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.041+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.041+0000 7fca00ff9700 10 client.50452.objecter _finish_command 13 = system:0 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.041+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 4 2024-03-30T15:07:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.041+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 6 13 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 10 client.50452.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 14 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 14 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e40178e0 osd=6 3 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 6 14 2024-03-30T15:07:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 4 2024-03-30T15:07:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 10 client.50452.objecter _send_command 14 2024-03-30T15:07:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.058+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4028de0 con 0x7fc9e4023880 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4028de0 con 0x7fc9e4023880 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 10 client.50452.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 4 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 6 14 2024-03-30T15:07:29.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 10 client.50452.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 15 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 15 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 7 15 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 4 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 10 client.50452.objecter _send_command 15 2024-03-30T15:07:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc9e4029980 con 0x7fc9e4029500 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.060+0000 7fca00ff9700 10 client.50452.objecter ms_handle_connect 0x7fc9e4029500 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.061+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc9e4029980 con 0x7fc9e4029500 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.061+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.061+0000 7fca00ff9700 10 client.50452.objecter _finish_command 15 = system:0 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.061+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 4 2024-03-30T15:07:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.061+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 7 15 2024-03-30T15:07:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 10 client.50452.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign -1 16 2024-03-30T15:07:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 20 client.50452.objecter _get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_remove -1 16 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 20 client.50452.objecter get_session s=0x7fc9e4028de0 osd=7 3 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 15 client.50452.objecter _session_command_op_assign 7 16 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 4 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 10 client.50452.objecter _send_command 16 2024-03-30T15:07:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.077+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc9e4029980 con 0x7fc9e4029500 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.079+0000 7fca00ff9700 1 -- 172.21.15.28:0/598042650 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc9e4029980 con 0x7fc9e4029500 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.079+0000 7fca00ff9700 10 client.50452.objecter ms_dispatch 0x7fca04001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.079+0000 7fca00ff9700 10 client.50452.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.079+0000 7fca00ff9700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 4 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.079+0000 7fca00ff9700 15 client.50452.objecter _session_command_op_remove 7 16 2024-03-30T15:07:29.081 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 librados: watch_flush enter 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 librados: watch_flush exit 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.0 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6801/3394323575 conn(0x7fc9e4004730 legacy=0x7fc9e4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40053e0 osd=0 1 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.1 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6805/1850313547 conn(0x7fc9e40042a0 legacy=0x7fc9e4008e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4004130 osd=1 1 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.2 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6809/3451325043 conn(0x7fc9e4011850 legacy=0x7fc9e4013c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4001a30 osd=2 1 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.3 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6813/1506157250 conn(0x7fc9e40155d0 legacy=0x7fc9e4015a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4009ae0 osd=3 1 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.4 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.134:6800/3063775514 conn(0x7fc9e401afb0 legacy=0x7fc9e401d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4017b00 osd=4 1 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.5 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.134:6804/3668151179 conn(0x7fc9e40170e0 legacy=0x7fc9e4021c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4007fc0 osd=5 1 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.6 2024-03-30T15:07:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca03fff700 1 -- 172.21.15.28:0/598042650 reap_dead start 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.134:6808/2135224194 conn(0x7fc9e4023880 legacy=0x7fc9e4025c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca03fff700 1 -- 172.21.15.28:0/598042650 reap_dead start 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca03fff700 1 -- 172.21.15.28:0/598042650 reap_dead start 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca03fff700 1 -- 172.21.15.28:0/598042650 reap_dead start 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e40178e0 osd=6 1 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter close_session for osd.7 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.134:6812/3758190298 conn(0x7fc9e4029500 legacy=0x7fc9e4027610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter put_session s=0x7fc9e4028de0 osd=7 1 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 20 client.50452.objecter shutdown clearing up homeless session... 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 10 client.50452.objecter successfully canceled tick 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc9f00706d0 legacy=0x7fc9f0072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> v1:172.21.15.134:6789/0 conn(0x7fca04101130 legacy=0x7fca04104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 shutdown_connections 2024-03-30T15:07:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 >> 172.21.15.28:0/598042650 conn(0x7fca040fc9e0 msgr2=0x7fca04110e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 shutdown_connections 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 -- 172.21.15.28:0/598042650 wait complete. 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:29.080+0000 7fca0a0af700 1 librados: shutdown 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:29.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:29.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:29.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:29.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:29.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:29.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:29.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:29.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:29.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:29.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:29.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 5.1e scrub starts 2024-03-30T15:07:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 5.1e scrub ok 2024-03-30T15:07:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/598042650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T15:07:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:29 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 5.1e scrub starts 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 5.1e scrub ok 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/598042650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T15:07:29.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 5.1e scrub starts 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 5.1e scrub ok 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/598042650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: osdmap e600: 8 total, 8 up, 8 in 2024-03-30T15:07:29.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-30T15:07:29.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:29.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:29 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-30T15:07:30.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:30.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:30.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:30.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:30.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:30.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:30.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:30.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:30.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24865.0:4 17.10 17:08000000::::head [pgnls start_epoch 114] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:30.648 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 575.38647825299995, 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.016273526, 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24865", 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3133651075", 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 4 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259409+0000", 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259411+0000", 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1000000000000001e-06 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259418+0000", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6809999999999998e-06 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259427+0000", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8659999999999995e-06 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259432+0000", 2024-03-30T15:07:30.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2320000000000001e-06 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259486+0000", 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3547000000000001e-05 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:55.259495+0000", 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0040000000000005e-06 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.273975+0000", 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.014480228 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.274008+0000", 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2824000000000003e-05 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:56.275683+0000", 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016750440000000001 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.653 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24853.0:11 14.15 14:ada94957:test-rados-api-smithi028-93116-5::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 573.27459188299997, 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.040512728999999997, 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24853", 2024-03-30T15:07:30.654 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 11 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371296+0000", 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371298+0000", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.973e-06 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371305+0000", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.956e-06 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371315+0000", 2024-03-30T15:07:30.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0227999999999999e-05 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371325+0000", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0009e-05 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371393+0000", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7558000000000006e-05 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371494+0000", 2024-03-30T15:07:30.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010136000000000001 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,4", 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.371687+0000", 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019319699999999999 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.375970+0000", 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042825850000000002 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.401997+0000", 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.026027127000000001 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411745+0000", 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0097477950000000001 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411781+0000", 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6205000000000003e-05 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:57.411809+0000", 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7736e-05 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2070 11.1e 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 572.29031323200002, 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.090460287, 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.660 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2070 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355574+0000", 2024-03-30T15:07:30.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.355577+0000", 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5239999999999999e-06 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381402+0000", 2024-03-30T15:07:30.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025824751 2024-03-30T15:07:30.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381428+0000", 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6114000000000001e-05 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381442+0000", 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3645000000000001e-05 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381556+0000", 2024-03-30T15:07:30.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114195 2024-03-30T15:07:30.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381660+0000", 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104416 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.381768+0000", 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108322 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.423084+0000", 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041315927000000002 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445887+0000", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022802991000000002 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445906+0000", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8759000000000001e-05 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.445976+0000", 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9926999999999996e-05 2024-03-30T15:07:30.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.446035+0000", 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8715999999999998e-05 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34853.0:2073 11.1e 11:79dfa13a:test-rados-api-smithi028-93257-17::big2:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 572.19484513099997, 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16928738600000001, 2024-03-30T15:07:30.668 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34853", 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1734910647", 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2073 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451043+0000", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.451044+0000", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111e-06 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454143+0000", 2024-03-30T15:07:30.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030991600000000001 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454169+0000", 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6114000000000001e-05 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454182+0000", 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3062e-05 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454263+0000", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0641000000000003e-05 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.454373+0000", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109875 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.458049+0000", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036762639999999998 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.474113+0000", 2024-03-30T15:07:30.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016064330000000002 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.533037+0000", 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.058924057000000002 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620236+0000", 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.087198494000000001 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620291+0000", 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4829e-05 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.620330+0000", 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9449000000000001e-05 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25342.0:2 57.1e 57:7a7d4169:::src_object.1:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:07:30.674 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 572.17862377500001, 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2667642749999999, 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25342", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2592697076", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467264+0000", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467266+0000", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9099999999999999e-06 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467277+0000", 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0706999999999999e-05 2024-03-30T15:07:30.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467287+0000", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0508e-05 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467298+0000", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0519999999999999e-05 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467376+0000", 2024-03-30T15:07:30.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8356000000000007e-05 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.467386+0000", 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9620000000000005e-06 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.727985+0000", 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.260599096 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.728092+0000", 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107036 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.728213+0000", 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012053900000000001 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729270+0000", 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001057721 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.729293+0000", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2796999999999999e-05 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733920+0000", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046273019999999998 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.733982+0000", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1978999999999997e-05 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.734028+0000", 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5841999999999999e-05 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35294.0:2 54.6 54:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 572.17255010500003, 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.251836993, 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.681 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35294", 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/806098981", 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473338+0000", 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473339+0000", 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8220000000000001e-06 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473345+0000", 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3179999999999997e-06 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473356+0000", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0954e-05 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473365+0000", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7769999999999997e-06 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473429+0000", 2024-03-30T15:07:30.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3115999999999997e-05 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:58.473438+0000", 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7580000000000004e-06 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719452+0000", 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.246013944 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719691+0000", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000238197 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.719925+0000", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023494400000000001 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.721930+0000", 2024-03-30T15:07:30.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002004329 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.722343+0000", 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00041327999999999997 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725066+0000", 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027224760000000001 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725131+0000", 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5407999999999994e-05 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:57:59.725175+0000", 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.367e-05 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35294.0:6 54.6 54:602f83fe:::foo:head [watch unwatch cookie 93835005080800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:07:30.688 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 569.91575176799995, 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.084326659999999998, 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35294", 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/806098981", 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 6 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730136+0000", 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730139+0000", 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.179e-06 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730148+0000", 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9379999999999998e-06 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730168+0000", 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9602e-05 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730181+0000", 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3796e-05 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:30.692 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T15:07:30.692 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T15:07:30.692 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: 3.19 deep-scrub starts 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: 3.19 deep-scrub ok 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: pgmap v883: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.693 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:30 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:30.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:30.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730418+0000", 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023605800000000001 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730534+0000", 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116564 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.730722+0000", 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000187894 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.731099+0000", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000376993 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.731523+0000", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042433299999999999 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814309+0000", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.082786025999999999 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814379+0000", 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0009e-05 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.814463+0000", 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3268000000000003e-05 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25489.0:2 64.1a 64:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:07:30.697 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 569.65634844800002, 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.91822508, 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25489", 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2934336176", 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989539+0000", 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989541+0000", 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8899999999999999e-06 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989549+0000", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6000000000000001e-06 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989557+0000", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6819999999999992e-06 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989564+0000", 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7649999999999996e-06 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989627+0000", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2780000000000005e-05 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:00.989635+0000", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6029999999999999e-06 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894091+0000", 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.904456081 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894206+0000", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115474 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.894346+0000", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139486 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.896778+0000", 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002432156 2024-03-30T15:07:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907464+0000", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010685741 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907629+0000", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000165055 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907721+0000", 2024-03-30T15:07:30.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2670000000000006e-05 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.907764+0000", 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3096999999999998e-05 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35396.0:1 66.0s0 66:074d3272:test-rados-api-smithi028-93069-23::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 568.79284213999995, 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.114964936, 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.704 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35396", 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/658745307", 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853046+0000", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853048+0000", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9520000000000001e-06 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853055+0000", 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.576e-06 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853065+0000", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9790000000000005e-06 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.853076+0000", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0764999999999999e-05 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.854153+0000", 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010775750000000001 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:01.854157+0000", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.331e-06 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939401+0000", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.085244367 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939520+0000", 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119022 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.939930+0000", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040942499999999999 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.940245+0000", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031576299999999999 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.967952+0000", 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027706232000000001 2024-03-30T15:07:30.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:02.968011+0000", 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8949000000000001e-05 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35956.0:2 125.1b 125:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 546.24883344600005, 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.270631477, 2024-03-30T15:07:30.710 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35956", 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4007703721", 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397054+0000", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397056+0000", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6249999999999999e-06 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397063+0000", 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8850000000000001e-06 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397072+0000", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3989999999999993e-06 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.397082+0000", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0210000000000001e-05 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.404263+0000", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007180484 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:24.404283+0000", 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9973e-05 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.662959+0000", 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.258676382 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663072+0000", 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113008 2024-03-30T15:07:30.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,6", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663217+0000", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014457599999999999 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663678+0000", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046156600000000001 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663884+0000", 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020523500000000001 2024-03-30T15:07:30.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667630+0000", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003746096 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667660+0000", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0332999999999999e-05 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667686+0000", 2024-03-30T15:07:30.716 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5704999999999999e-05 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.35977.0:2 126.16 126:68f45781:::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 545.04916051999999, 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.079154859999999994, 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.35977", 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2482478745", 2024-03-30T15:07:30.717 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596727+0000", 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.718 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596729+0000", 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8959999999999999e-06 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596736+0000", 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.489e-06 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596744+0000", 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7849999999999997e-06 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.719 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596754+0000", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1789999999999997e-06 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596838+0000", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4194000000000001e-05 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.596846+0000", 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6330000000000002e-06 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.720 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.662948+0000", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.066101602999999995 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663060+0000", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112466 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663207+0000", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014695100000000001 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.721 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.663682+0000", 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00047461399999999999 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.667728+0000", 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040456939999999999 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675811+0000", 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080834860000000008 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.722 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675843+0000", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1751000000000001e-05 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:25.675882+0000", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9119000000000003e-05 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.46235.0:515 130.4 130:24299189:::test_508:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e145)", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:07:30.723 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 540.50130594500001, 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062198679999999999, 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.46235", 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 515 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:07:30.724 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144582+0000", 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144583+0000", 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2890000000000001e-06 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.725 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.144591+0000", 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0630000000000006e-06 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.145783+0000", 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011915199999999999 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.145797+0000", 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4464000000000001e-05 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.168993+0000", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023195436 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.169128+0000", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013532799999999999 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:30.206780+0000", 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.037652579999999998 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.727 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.56882.0:2 166.18 166:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 524.44239728499997, 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.030150073999999999, 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.56882", 2024-03-30T15:07:30.728 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1234052616", 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203490+0000", 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.729 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203492+0000", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0470000000000001e-06 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203504+0000", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1810999999999999e-05 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203514+0000", 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0009e-05 2024-03-30T15:07:30.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.203535+0000", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0738e-05 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.213352+0000", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098172729999999996 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.213361+0000", 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1069999999999994e-06 2024-03-30T15:07:30.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226469+0000", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013108066 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226597+0000", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012716999999999999 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.226734+0000", 2024-03-30T15:07:30.732 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013731299999999999 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.227951+0000", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012165450000000001 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.232907+0000", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049565479999999999 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233549+0000", 2024-03-30T15:07:30.733 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064232700000000005 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233599+0000", 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9830000000000002e-05 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:58:46.233641+0000", 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1289999999999999e-05 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.734 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57104.0:13 195.6s0 195:71611c3d:test-rados-api-smithi028-93525-20::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 491.43655132999999, 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036499746999999999, 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57104", 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2817065657", 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 13 2024-03-30T15:07:30.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209336+0000", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209339+0000", 2024-03-30T15:07:30.736 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.322e-06 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209348+0000", 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7499999999999998e-06 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209362+0000", 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3577000000000001e-05 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.737 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.209374+0000", 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2077e-05 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.210150+0000", 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00077631599999999999 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.210159+0000", 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0359999999999995e-06 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219259+0000", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090994409999999998 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219399+0000", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013971400000000001 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.219883+0000", 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00048410599999999999 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.222618+0000", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027352790000000002 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.245760+0000", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023141623 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:19.245836+0000", 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6506000000000002e-05 2024-03-30T15:07:30.740 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57182.0:2 206.11 206:8e43fa19:test-rados-api-smithi028-93257-34::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 487.45453758100001, 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.032742122999999998, 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57182", 2024-03-30T15:07:30.741 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/354569125", 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191350+0000", 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.742 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191353+0000", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3800000000000001e-06 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191368+0000", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5608e-05 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191379+0000", 2024-03-30T15:07:30.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1093e-05 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.191391+0000", 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1999e-05 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.193786+0000", 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023949930000000002 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.193797+0000", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0971999999999999e-05 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214441+0000", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020643470000000001 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214580+0000", 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013977799999999999 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.214734+0000", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000153351 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.218361+0000", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036272399999999999 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.221480+0000", 2024-03-30T15:07:30.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003119442 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.223991+0000", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025103450000000002 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.224045+0000", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4627999999999999e-05 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:23.224092+0000", 2024-03-30T15:07:30.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6823999999999998e-05 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57227.0:2 211.d 211:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 482.44282129200002, 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.040305306999999999, 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.748 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57227", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1866544138", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203066+0000", 2024-03-30T15:07:30.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203068+0000", 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9690000000000001e-06 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203077+0000", 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9230000000000008e-06 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203088+0000", 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0525e-05 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203097+0000", 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6260000000000007e-06 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203155+0000", 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7924999999999999e-05 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.751 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.203160+0000", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9439999999999996e-06 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.210784+0000", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076239929999999999 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.210794+0000", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7180000000000008e-06 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.227995+0000", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017200653999999999 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.228136+0000", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141679 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.228273+0000", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013709700000000001 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:07:30.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.232978+0000", 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047047629999999998 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.241577+0000", 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0085986069999999994 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243117+0000", 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015401970000000001 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243325+0000", 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020754599999999999 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:28.243372+0000", 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7141000000000001e-05 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57224.0:1 212.6s0 212:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-03-30T15:07:30.755 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 481.42805888999999, 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0139440129999999, 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57224", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2535484916", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217829+0000", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217831+0000", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9479999999999998e-06 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217842+0000", 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0769e-05 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217855+0000", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3015e-05 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217863+0000", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3860000000000007e-06 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217925+0000", 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2392999999999997e-05 2024-03-30T15:07:30.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:29.217928+0000", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9780000000000001e-06 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230208+0000", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0122793919999999 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230351+0000", 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014353200000000001 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.230724+0000", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037236099999999998 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231099+0000", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037526500000000001 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231702+0000", 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00060294299999999999 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:30.231773+0000", 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1031000000000005e-05 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.761 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57476.0:1 245.6s0 245:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 425.413880302, 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0238222990000001, 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57476", 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4051625719", 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:30.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232010+0000", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.604e-06 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232018+0000", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3520000000000007e-06 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232031+0000", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2765999999999999e-05 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232043+0000", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1732e-05 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232111+0000", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8403000000000002e-05 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232115+0000", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7220000000000001e-06 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252496+0000", 2024-03-30T15:07:30.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0203810069999999 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252648+0000", 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000152017 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.254117+0000", 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014689320000000001 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255025+0000", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090758899999999999 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255742+0000", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071763700000000003 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255830+0000", 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7538000000000001e-05 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49756.0:1 273.6s0 273:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 349.33199018300002, 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93441624899999998, 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49756", 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1885953997", 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.768 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313900+0000", 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.785e-06 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313911+0000", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0925e-05 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313947+0000", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6205000000000003e-05 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313960+0000", 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2768999999999999e-05 2024-03-30T15:07:30.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525411+0000", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.21145103200000001 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525415+0000", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.014e-06 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.246985+0000", 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.72157016200000001 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247116+0000", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013093499999999999 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247514+0000", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039808699999999999 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247951+0000", 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043615700000000001 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248270+0000", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031954200000000002 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248314+0000", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3636e-05 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49768.0:1 274.5s0 274:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e338)", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:07:30.773 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 348.41366857399998, 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0270852340000001, 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49768", 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3544512235", 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:07:30.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232221+0000", 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3120000000000001e-06 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:07:30.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232228+0000", 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1680000000000001e-06 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232240+0000", 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2248e-05 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232251+0000", 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1559000000000001e-05 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234151+0000", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001899277 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234167+0000", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5872e-05 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259198+0000", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025031091 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259304+0000", 2024-03-30T15:07:30.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010670699999999999 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:07:30.778 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:07:30.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:30.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:30.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:30.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T15:07:30.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T15:07:30.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: 3.19 deep-scrub starts 2024-03-30T15:07:30.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: 3.19 deep-scrub ok 2024-03-30T15:07:30.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: pgmap v883: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: 3.19 deep-scrub starts 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: 3.19 deep-scrub ok 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: pgmap v883: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: osdmap e601: 8 total, 8 up, 8 in 2024-03-30T15:07:30.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:30 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-30T15:07:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:30.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:31.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:31.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:31.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:31.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:31.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:31.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.495+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/2766351525 >> v1:172.21.15.28:6789/0 conn(0x7fdb54109dc0 legacy=0x7fdb5410c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.495+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/2766351525 shutdown_connections 2024-03-30T15:07:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.495+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/2766351525 >> 172.21.15.28:0/2766351525 conn(0x7fdb540fc9e0 msgr2=0x7fdb540fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.495+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/2766351525 shutdown_connections 2024-03-30T15:07:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.495+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/2766351525 wait complete. 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 librados: starting msgr at 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 librados: starting objecter 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 Processor -- start 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 -- start start 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 librados: setting wanted keys 2024-03-30T15:07:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 librados: calling monclient init 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb541a9350 con 0x7fdb54109dc0 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb541aa570 con 0x7fdb5410d960 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb5c6e9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb541ab7b0 con 0x7fdb54101130 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb59c84700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdb54109dc0 0x7fdb541a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40736/0 (socket says 172.21.15.28:40736) 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.496+0000 7fdb59c84700 1 -- 172.21.15.28:0/135526117 learned_addr learned my addr 172.21.15.28:0/135526117 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter ms_handle_connect 0x7fdb54109dc0 2024-03-30T15:07:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter ms_handle_connect 0x7fdb54101130 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter ms_handle_connect 0x7fdb5410d960 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1203480353 0 0) 0x7fdb541a9350 con 0x7fdb54109dc0 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb40003aa0 con 0x7fdb54109dc0 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936611629 0 0) 0x7fdb541aa570 con 0x7fdb5410d960 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb541a9350 con 0x7fdb5410d960 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446190055 0 0) 0x7fdb541ab7b0 con 0x7fdb54101130 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb541aa570 con 0x7fdb54101130 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2492509802 0 0) 0x7fdb40003aa0 con 0x7fdb54109dc0 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb541ab7b0 con 0x7fdb54109dc0 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdb440045a0 con 0x7fdb54109dc0 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3666727408 0 0) 0x7fdb541a9350 con 0x7fdb5410d960 2024-03-30T15:07:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb40003aa0 con 0x7fdb5410d960 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1867780926 0 0) 0x7fdb541aa570 con 0x7fdb54101130 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb541a9350 con 0x7fdb54101130 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdb4c001af0 con 0x7fdb5410d960 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdb50001840 con 0x7fdb54101130 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3170169440 0 0) 0x7fdb541ab7b0 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 >> v1:172.21.15.28:6790/0 conn(0x7fdb54101130 legacy=0x7fdb54106350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 >> v1:172.21.15.134:6789/0 conn(0x7fdb5410d960 legacy=0x7fdb541a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb541ac9f0 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.497+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb541ab9e0 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.498+0000 7fdb5c6e9700 10 client.25251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.498+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb541abea0 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.498+0000 7fdb5c6e9700 1 librados: init done 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.498+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdb44001b10 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.498+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdb44004dd0 con 0x7fdb54109dc0 2024-03-30T15:07:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.499+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb38005120 con 0x7fdb54109dc0 2024-03-30T15:07:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.500+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdb44003680 con 0x7fdb54109dc0 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.500+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(602..602 src has 442..602) ==== 8691+0+0 (unknown 1841150744 0 0) 0x7fdb440899c0 con 0x7fdb54109dc0 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.500+0000 7fdb4b7fe700 10 client.25251.objecter ms_dispatch 0x7fdb54001550 osd_map(602..602 src has 442..602) 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.500+0000 7fdb4b7fe700 3 client.25251.objecter handle_osd_map got epochs [602,602] > 0 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.500+0000 7fdb4b7fe700 3 client.25251.objecter handle_osd_map decoding full epoch 602 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.501+0000 7fdb4b7fe700 20 client.25251.objecter dump_active .. 0 homeless 2024-03-30T15:07:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.501+0000 7fdb4b7fe700 10 client.25251.objecter ms_handle_connect 0x7fdb40070570 2024-03-30T15:07:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.505+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdb44004db0 con 0x7fdb54109dc0 2024-03-30T15:07:31.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:07:31.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:31.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:31.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:31.732+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7fdb38005410 con 0x7fdb54109dc0 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 42.9 scrub starts 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[29876]: 42.9 scrub ok 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:07:31.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T15:07:31.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 42.9 scrub starts 2024-03-30T15:07:31.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:31 smithi028 ceph-mon[41614]: 42.9 scrub ok 2024-03-30T15:07:31.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:07:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:07:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: osdmap e602: 8 total, 8 up, 8 in 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 42.9 scrub starts 2024-03-30T15:07:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:31 smithi134 ceph-mon[33714]: 42.9 scrub ok 2024-03-30T15:07:31.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:31.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:31.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:07:32.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:32.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:32.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:32.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:32.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.334+0000 7fdb4b7fe700 1 -- 172.21.15.28:0/135526117 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v603) ==== 113+0+0 (unknown 3791115336 0 0) 0x7fdb440569a0 con 0x7fdb54109dc0 2024-03-30T15:07:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.337+0000 7fdb5c6e9700 10 librados: watch_flush enter 2024-03-30T15:07:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.337+0000 7fdb5c6e9700 10 librados: watch_flush exit 2024-03-30T15:07:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.337+0000 7fdb5c6e9700 20 client.25251.objecter shutdown clearing up homeless session... 2024-03-30T15:07:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.337+0000 7fdb5c6e9700 10 client.25251.objecter successfully canceled tick 2024-03-30T15:07:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.338+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdb40070570 legacy=0x7fdb40072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.338+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 >> v1:172.21.15.28:6789/0 conn(0x7fdb54109dc0 legacy=0x7fdb541a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.338+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 shutdown_connections 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.338+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 >> 172.21.15.28:0/135526117 conn(0x7fdb540fc9e0 msgr2=0x7fdb541105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.338+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 shutdown_connections 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.339+0000 7fdb5c6e9700 1 -- 172.21.15.28:0/135526117 wait complete. 2024-03-30T15:07:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:32.339+0000 7fdb5c6e9700 1 librados: shutdown 2024-03-30T15:07:32.341 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T15:07:32.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:32.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:32.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:32.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:32.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:32.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:32.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:32.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:32.849 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:07:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: pgmap v886: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:07:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:07:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:32 smithi134 ceph-mon[33714]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: pgmap v886: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T15:07:32.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[41614]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: pgmap v886: 270 pgs: 32 creating+peering, 238 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/135526117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: osdmap e603: 8 total, 8 up, 8 in 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-6"}]: dispatch 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3141701961' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:32 smithi028 ceph-mon[29876]: from='client.35600 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-6", "tierpool": "test-rados-api-smithi028-93253-102"}]: dispatch 2024-03-30T15:07:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:32.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:32.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:33.007 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:33.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:33.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:33.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:33.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:33.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:33.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:33.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:33.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:33.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:33.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:33.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-03-30T15:07:33.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:33.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.3 deep-scrub starts 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.3 deep-scrub ok 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:07:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.15 deep-scrub starts 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: 3.15 deep-scrub ok 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:07:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:33 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.3 deep-scrub starts 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.3 deep-scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.15 deep-scrub starts 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: 3.15 deep-scrub ok 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T15:07:33.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.3 deep-scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.3 deep-scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.15 deep-scrub starts 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: 3.15 deep-scrub ok 2024-03-30T15:07:33.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: osdmap e604: 8 total, 8 up, 8 in 2024-03-30T15:07:33.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:07:33.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:33 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:07:33.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:33.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:33.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:34.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:34.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:34.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:34.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:34.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:34.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:34.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:34.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:34.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 3.9 deep-scrub starts 2024-03-30T15:07:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 3.9 deep-scrub ok 2024-03-30T15:07:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: pgmap v889: 238 pgs: 238 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 3.9 deep-scrub starts 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 3.9 deep-scrub ok 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: pgmap v889: 238 pgs: 238 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 3.9 deep-scrub starts 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 3.9 deep-scrub ok 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: pgmap v889: 238 pgs: 238 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: osdmap e605: 8 total, 8 up, 8 in 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:34.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:34.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:35.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:35.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:35.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:35.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:35.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:35.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:35.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:35.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:35.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- 172.21.15.28:0/2633420549 >> v1:172.21.15.28:6790/0 conn(0x7f1d8010d960 legacy=0x7f1d8010fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- 172.21.15.28:0/2633420549 shutdown_connections 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- 172.21.15.28:0/2633420549 >> 172.21.15.28:0/2633420549 conn(0x7f1d800fc9e0 msgr2=0x7f1d800fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- 172.21.15.28:0/2633420549 shutdown_connections 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- 172.21.15.28:0/2633420549 wait complete. 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 librados: starting msgr at 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 librados: starting objecter 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 Processor -- start 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 -- start start 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 librados: setting wanted keys 2024-03-30T15:07:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.864+0000 7f1d8651e700 1 librados: calling monclient init 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d8651e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d801a9350 con 0x7f1d8010d960 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d8651e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d801aa570 con 0x7f1d80109dc0 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d8651e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1d801ab7b0 con 0x7f1d80101130 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1d80101130 0x7f1d80106360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42050/0 (socket says 172.21.15.28:42050) 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7ffff700 1 -- 172.21.15.28:0/2341847557 learned_addr learned my addr 172.21.15.28:0/2341847557 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter ms_handle_connect 0x7f1d8010d960 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter ms_handle_connect 0x7f1d80101130 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter ms_handle_connect 0x7f1d80109dc0 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226936530 0 0) 0x7f1d801a9350 con 0x7f1d8010d960 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d680039c0 con 0x7f1d8010d960 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134809309 0 0) 0x7f1d801ab7b0 con 0x7f1d80101130 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d801a9350 con 0x7f1d80101130 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524431447 0 0) 0x7f1d801aa570 con 0x7f1d80109dc0 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.865+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d801ab7b0 con 0x7f1d80109dc0 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 971133533 0 0) 0x7f1d680039c0 con 0x7f1d8010d960 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d801aa570 con 0x7f1d8010d960 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1d78001870 con 0x7f1d8010d960 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1274534365 0 0) 0x7f1d801a9350 con 0x7f1d80101130 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d680039c0 con 0x7f1d80101130 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1d70001890 con 0x7f1d80101130 2024-03-30T15:07:35.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2462711007 0 0) 0x7f1d801ab7b0 con 0x7f1d80109dc0 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d801a9350 con 0x7f1d80109dc0 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1d740017f0 con 0x7f1d80109dc0 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1928704514 0 0) 0x7f1d801aa570 con 0x7f1d8010d960 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 >> v1:172.21.15.28:6790/0 conn(0x7f1d80101130 legacy=0x7f1d80106360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 >> v1:172.21.15.134:6789/0 conn(0x7f1d80109dc0 legacy=0x7f1d801a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d801ac9f0 con 0x7f1d8010d960 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d801ab9e0 con 0x7f1d8010d960 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d8651e700 10 client.25263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d801abea0 con 0x7f1d8010d960 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1d78004b10 con 0x7f1d8010d960 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d8651e700 1 librados: init done 2024-03-30T15:07:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.866+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1d780052e0 con 0x7f1d8010d960 2024-03-30T15:07:35.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.867+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d80102610 con 0x7f1d8010d960 2024-03-30T15:07:35.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1d780055f0 con 0x7f1d8010d960 2024-03-30T15:07:35.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(606..606 src has 442..606) ==== 8202+0+0 (unknown 3850944932 0 0) 0x7f1d78088b20 con 0x7f1d8010d960 2024-03-30T15:07:35.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 10 client.25263.objecter ms_dispatch 0x7f1d80001550 osd_map(606..606 src has 442..606) 2024-03-30T15:07:35.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 3 client.25263.objecter handle_osd_map got epochs [606,606] > 0 2024-03-30T15:07:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 3 client.25263.objecter handle_osd_map decoding full epoch 606 2024-03-30T15:07:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 20 client.25263.objecter dump_active .. 0 homeless 2024-03-30T15:07:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.868+0000 7f1d7d7fa700 10 client.25263.objecter ms_handle_connect 0x7f1d68070440 2024-03-30T15:07:35.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:35.873+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1d7805e010 con 0x7f1d8010d960 2024-03-30T15:07:35.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:35.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T15:07:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T15:07:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 42.6 scrub starts 2024-03-30T15:07:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 42.6 scrub ok 2024-03-30T15:07:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:07:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:07:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T15:07:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:35.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 42.6 scrub starts 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 42.6 scrub ok 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 42.6 scrub starts 2024-03-30T15:07:35.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 42.6 scrub ok 2024-03-30T15:07:35.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:07:35.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:07:35.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:35.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: osdmap e606: 8 total, 8 up, 8 in 2024-03-30T15:07:35.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:07:35.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.089+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1d80103b10 con 0x7f1d8010d960 2024-03-30T15:07:36.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:36.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:36.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.317+0000 7f2c25d4a700 1 -- 172.21.15.28:0/1361255576 >> v1:172.21.15.28:6790/0 conn(0x7f2c20103280 legacy=0x7f2c20103660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- 172.21.15.28:0/1361255576 shutdown_connections 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- 172.21.15.28:0/1361255576 >> 172.21.15.28:0/1361255576 conn(0x7f2c200fca30 msgr2=0x7f2c200fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- 172.21.15.28:0/1361255576 shutdown_connections 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- 172.21.15.28:0/1361255576 wait complete. 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 librados: starting msgr at 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 librados: starting objecter 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 Processor -- start 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- start start 2024-03-30T15:07:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 librados: setting wanted keys 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 librados: calling monclient init 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2c2010e410 con 0x7f2c2010b090 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2c201aa570 con 0x7f2c2010eba0 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.318+0000 7f2c25d4a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2c201ab790 con 0x7f2c20103280 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1f7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2c20103280 0x7f2c2010d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:42068/0 (socket says 172.21.15.28:42068) 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1f7fe700 1 -- 172.21.15.28:0/3397437033 learned_addr learned my addr 172.21.15.28:0/3397437033 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter ms_handle_connect 0x7f2c20103280 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter ms_handle_connect 0x7f2c2010b090 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter ms_handle_connect 0x7f2c2010eba0 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952326993 0 0) 0x7f2c2010e410 con 0x7f2c2010b090 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c0c0032d0 con 0x7f2c2010b090 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188507413 0 0) 0x7f2c201ab790 con 0x7f2c20103280 2024-03-30T15:07:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c2010e410 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236223756 0 0) 0x7f2c201aa570 con 0x7f2c2010eba0 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c201ab790 con 0x7f2c2010eba0 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2499097146 0 0) 0x7f2c2010e410 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c201aa570 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2c18001430 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2666141055 0 0) 0x7f2c0c0032d0 con 0x7f2c2010b090 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c2010e410 con 0x7f2c2010b090 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2c08001850 con 0x7f2c2010b090 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2132861504 0 0) 0x7f2c201aa570 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.134:6789/0 conn(0x7f2c2010eba0 legacy=0x7f2c201a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6789/0 conn(0x7f2c2010b090 legacy=0x7f2c201a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c201ac9f0 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c201ab9c0 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c25d4a700 10 client.58469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c201abe80 con 0x7f2c20103280 2024-03-30T15:07:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c25d4a700 1 librados: init done 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.319+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2c18001b30 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.320+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2c18004f50 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.320+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2c00005100 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.321+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2c180051d0 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.321+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(606..606 src has 442..606) ==== 8202+0+0 (unknown 3850944932 0 0) 0x7f2c18089970 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.321+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 osd_map(606..606 src has 442..606) 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.321+0000 7f2c1cff9700 3 client.58469.objecter handle_osd_map got epochs [606,606] > 0 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.321+0000 7f2c1cff9700 3 client.58469.objecter handle_osd_map decoding full epoch 606 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c1cff9700 20 client.58469.objecter dump_active .. 0 homeless 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v606) ==== 54+0+16 (unknown 2573526903 0 2420000192) 0x7f2c1808a060 con 0x7f2c20103280 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2c0c070530 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 10 client.58469.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign -1 1 2024-03-30T15:07:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_remove -1 1 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 20 client.58469.objecter get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign 0 1 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 4 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 10 client.58469.objecter _send_command 1 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c00004b50 con 0x7f2c00004730 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.322+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2c00004730 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.323+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2c00004b50 con 0x7f2c00004730 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.323+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.323+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 1 = system:0 2024-03-30T15:07:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.323+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 4 2024-03-30T15:07:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.323+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 0 1 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.340+0000 7f1d7d7fa700 1 -- 172.21.15.28:0/2341847557 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v607) ==== 87+0+0 (unknown 3758559548 0 0) 0x7f1d780052c0 con 0x7f1d8010d960 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 10 client.58469.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign -1 2 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_remove -1 2 2024-03-30T15:07:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 20 client.58469.objecter get_session s=0x7f2c000053e0 osd=0 3 2024-03-30T15:07:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign 0 2 2024-03-30T15:07:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 4 2024-03-30T15:07:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 10 client.58469.objecter _send_command 2 2024-03-30T15:07:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.341+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c000079f0 con 0x7f2c00004730 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.343+0000 7f1d8651e700 10 librados: watch_flush enter 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.343+0000 7f1d8651e700 10 librados: watch_flush exit 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.343+0000 7f1d8651e700 20 client.25263.objecter shutdown clearing up homeless session... 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.343+0000 7f1d8651e700 10 client.25263.objecter successfully canceled tick 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.344+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1d68070440 legacy=0x7f1d68072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.344+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 >> v1:172.21.15.28:6789/0 conn(0x7f1d8010d960 legacy=0x7f1d801a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 shutdown_connections 2024-03-30T15:07:36.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 >> 172.21.15.28:0/2341847557 conn(0x7f1d800fc9e0 msgr2=0x7f1d8006a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2c000079f0 con 0x7f2c00004730 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 4 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 0 2 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 shutdown_connections 2024-03-30T15:07:36.347 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f1d8651e700 1 -- 172.21.15.28:0/2341847557 wait complete. 2024-03-30T15:07:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f1d8651e700 1 librados: shutdown 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c25d4a700 10 client.58469.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.345+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign -1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_remove -1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 20 client.58469.objecter get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign 1 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 4 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 10 client.58469.objecter _send_command 3 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2c00002180 con 0x7f2c00004280 2024-03-30T15:07:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.346+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2c00004280 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.347+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2c00002180 con 0x7f2c00004280 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.347+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.347+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 3 = system:0 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.347+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 4 2024-03-30T15:07:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.347+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 1 3 2024-03-30T15:07:36.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 10 client.58469.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign -1 4 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 20 client.58469.objecter _get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_remove -1 4 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 20 client.58469.objecter get_session s=0x7f2c00004110 osd=1 3 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 15 client.58469.objecter _session_command_op_assign 1 4 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 4 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 10 client.58469.objecter _send_command 4 2024-03-30T15:07:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.373+0000 7f2c25d4a700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c00001600 con 0x7f2c00004280 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2c00001600 con 0x7f2c00004280 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 4 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 1 4 2024-03-30T15:07:36.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 10 client.58469.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 5 2024-03-30T15:07:36.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 5 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 2 5 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.377+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 4 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.378+0000 7f2c167fc700 10 client.58469.objecter _send_command 5 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.378+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2bf8006dd0 con 0x7f2bf8002fd0 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.378+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf8002fd0 2024-03-30T15:07:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.379+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf8006dd0 con 0x7f2bf8002fd0 2024-03-30T15:07:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.379+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.379+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 5 = system:0 2024-03-30T15:07:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.379+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 4 2024-03-30T15:07:36.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.379+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 2 5 2024-03-30T15:07:36.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 10 client.58469.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 6 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 6 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf80028e0 osd=2 3 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 2 6 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 4 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 10 client.58469.objecter _send_command 6 2024-03-30T15:07:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.398+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf80012a0 con 0x7f2bf8002fd0 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf80012a0 con 0x7f2bf8002fd0 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 4 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 2 6 2024-03-30T15:07:36.402 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 10 client.58469.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 7 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 7 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 3 7 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 4 2024-03-30T15:07:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 10 client.58469.objecter _send_command 7 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2bf8008cd0 con 0x7f2bf8008130 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.401+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf8008130 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.402+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf8008cd0 con 0x7f2bf8008130 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.402+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.402+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 7 = system:0 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.402+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 4 2024-03-30T15:07:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.402+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 3 7 2024-03-30T15:07:36.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 10 client.58469.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 8 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 8 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8001f70 osd=3 3 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 3 8 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 4 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 10 client.58469.objecter _send_command 8 2024-03-30T15:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.419+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf80096c0 con 0x7f2bf8008130 2024-03-30T15:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf80096c0 con 0x7f2bf8008130 2024-03-30T15:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 4 2024-03-30T15:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 3 8 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 10 client.58469.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 9 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 9 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 4 9 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 4 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 10 client.58469.objecter _send_command 9 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.421+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2bf8010710 con 0x7f2bf8001b00 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf8001b00 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf8010710 con 0x7f2bf8001b00 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 9 = system:0 2024-03-30T15:07:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 4 2024-03-30T15:07:36.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.422+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 4 9 2024-03-30T15:07:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 10 client.58469.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 10 2024-03-30T15:07:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 10 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf80096c0 osd=4 3 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 4 10 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 4 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 10 client.58469.objecter _send_command 10 2024-03-30T15:07:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.439+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf8009100 con 0x7f2bf8001b00 2024-03-30T15:07:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf8009100 con 0x7f2bf8001b00 2024-03-30T15:07:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 4 2024-03-30T15:07:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 4 10 2024-03-30T15:07:36.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 10 client.58469.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 11 2024-03-30T15:07:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 11 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 5 11 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 4 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 10 client.58469.objecter _send_command 11 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.441+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2bf8011e30 con 0x7f2bf8007c90 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf8007c90 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf8011e30 con 0x7f2bf8007c90 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 11 = system:0 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 4 2024-03-30T15:07:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.442+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 5 11 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 10 client.58469.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 12 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 12 2024-03-30T15:07:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8008eb0 osd=5 3 2024-03-30T15:07:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 5 12 2024-03-30T15:07:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 4 2024-03-30T15:07:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 10 client.58469.objecter _send_command 12 2024-03-30T15:07:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.459+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf8011e30 con 0x7f2bf8007c90 2024-03-30T15:07:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf8011e30 con 0x7f2bf8007c90 2024-03-30T15:07:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 4 2024-03-30T15:07:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 5 12 2024-03-30T15:07:36.463 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 10 client.58469.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 13 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 13 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 6 13 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 4 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 10 client.58469.objecter _send_command 13 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2bf80192c0 con 0x7f2bf80157b0 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.461+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf80157b0 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.462+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf80192c0 con 0x7f2bf80157b0 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.462+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.462+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 13 = system:0 2024-03-30T15:07:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.462+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 4 2024-03-30T15:07:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.462+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 6 13 2024-03-30T15:07:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 10 client.58469.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 14 2024-03-30T15:07:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 14 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8011e30 osd=6 3 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 6 14 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 4 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 10 client.58469.objecter _send_command 14 2024-03-30T15:07:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.479+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf8009080 con 0x7f2bf80157b0 2024-03-30T15:07:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf8009080 con 0x7f2bf80157b0 2024-03-30T15:07:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 4 2024-03-30T15:07:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 6 14 2024-03-30T15:07:36.483 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 10 client.58469.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:36.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 15 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 15 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 7 15 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 4 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 10 client.58469.objecter _send_command 15 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.481+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2bf8007320 con 0x7f2bf8012d70 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 10 client.58469.objecter ms_handle_connect 0x7f2bf8012d70 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2bf8007320 con 0x7f2bf8012d70 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 15 = system:0 2024-03-30T15:07:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 4 2024-03-30T15:07:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.482+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 7 15 2024-03-30T15:07:36.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 10 client.58469.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign -1 16 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 20 client.58469.objecter _get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_remove -1 16 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 20 client.58469.objecter get_session s=0x7f2bf8006fb0 osd=7 3 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 15 client.58469.objecter _session_command_op_assign 7 16 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 4 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 10 client.58469.objecter _send_command 16 2024-03-30T15:07:36.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.499+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2bf8007530 con 0x7f2bf8012d70 2024-03-30T15:07:36.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.501+0000 7f2c1cff9700 1 -- 172.21.15.28:0/3397437033 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2bf8007530 con 0x7f2bf8012d70 2024-03-30T15:07:36.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.501+0000 7f2c1cff9700 10 client.58469.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.501+0000 7f2c1cff9700 10 client.58469.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.501+0000 7f2c1cff9700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 4 2024-03-30T15:07:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.501+0000 7f2c1cff9700 15 client.58469.objecter _session_command_op_remove 7 16 2024-03-30T15:07:36.503 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:36.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 librados: watch_flush enter 2024-03-30T15:07:36.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 librados: watch_flush exit 2024-03-30T15:07:36.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.0 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6801/3394323575 conn(0x7f2c00004730 legacy=0x7f2c00002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2c000053e0 osd=0 1 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.1 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6805/1850313547 conn(0x7f2c00004280 legacy=0x7f2c0000ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2c00004110 osd=1 1 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.2 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6809/3451325043 conn(0x7f2bf8002fd0 legacy=0x7f2bf8005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c1ffff700 1 -- 172.21.15.28:0/3397437033 reap_dead start 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80028e0 osd=2 1 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.3 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6813/1506157250 conn(0x7f2bf8008130 legacy=0x7f2bf80085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8001f70 osd=3 1 2024-03-30T15:07:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.4 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.134:6800/3063775514 conn(0x7f2bf8001b00 legacy=0x7f2bf800eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf80096c0 osd=4 1 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.5 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.134:6804/3668151179 conn(0x7f2bf8007c90 legacy=0x7f2bf8010710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8008eb0 osd=5 1 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.6 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.134:6808/2135224194 conn(0x7f2bf80157b0 legacy=0x7f2bf8017ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8011e30 osd=6 1 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter close_session for osd.7 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.134:6812/3758190298 conn(0x7f2bf8012d70 legacy=0x7f2bf801c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c1ffff700 1 -- 172.21.15.28:0/3397437033 reap_dead start 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter put_session s=0x7f2bf8006fb0 osd=7 1 2024-03-30T15:07:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 20 client.58469.objecter shutdown clearing up homeless session... 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 10 client.58469.objecter successfully canceled tick 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2c0c070530 legacy=0x7f2c0c0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> v1:172.21.15.28:6790/0 conn(0x7f2c20103280 legacy=0x7f2c2010d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 shutdown_connections 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.502+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 >> 172.21.15.28:0/3397437033 conn(0x7f2c200fca30 msgr2=0x7f2c20110f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.503+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 shutdown_connections 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.503+0000 7f2c167fc700 1 -- 172.21.15.28:0/3397437033 wait complete. 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:36.503+0000 7f2c167fc700 1 librados: shutdown 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:36.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:36.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T15:07:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T15:07:36.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: pgmap v892: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 12.8 deep-scrub starts 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 12.8 deep-scrub ok 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3397437033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[29876]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T15:07:36.755 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: pgmap v892: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 12.8 deep-scrub starts 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 12.8 deep-scrub ok 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3397437033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:36.756 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:36 smithi028 ceph-mon[41614]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T15:07:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:36.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:36.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:36.890 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:36.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: pgmap v892: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 12.8 deep-scrub starts 2024-03-30T15:07:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 12.8 deep-scrub ok 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3397437033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2341847557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:07:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:36 smithi134 ceph-mon[33714]: osdmap e607: 8 total, 8 up, 8 in 2024-03-30T15:07:36.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:37.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:37.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:37.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:37.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:37.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:37.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:37.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:37.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:37.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:37.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:37.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:37.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:37.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: 42.1d scrub starts 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: 42.1d scrub ok 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: 42.10 scrub starts 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:07:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:37 smithi134 ceph-mon[33714]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: 42.1d scrub starts 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: 42.1d scrub ok 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: 42.10 scrub starts 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[29876]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T15:07:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: 42.1d scrub starts 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: 42.1d scrub ok 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: 42.10 scrub starts 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi028-93253-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:07:37.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:37 smithi028 ceph-mon[41614]: osdmap e608: 8 total, 8 up, 8 in 2024-03-30T15:07:38.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:38.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:38.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:38.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:38.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:38.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:38.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:38.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:38.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:38.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:38.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:38.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:38.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:38.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:38.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:38.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:38.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:38 smithi134 ceph-mon[33714]: pgmap v895: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:38.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:38 smithi134 ceph-mon[33714]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T15:07:38.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:38 smithi028 ceph-mon[29876]: pgmap v895: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:38.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:38 smithi028 ceph-mon[29876]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T15:07:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:38 smithi028 ceph-mon[41614]: pgmap v895: 214 pgs: 8 unknown, 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:38.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:38 smithi028 ceph-mon[41614]: osdmap e609: 8 total, 8 up, 8 in 2024-03-30T15:07:39.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:39.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:39.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:39.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:39.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:39.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:39.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:39.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:39.564 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:07:39.565 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:07:39.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:39.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:39.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:39 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:39 smithi134 ceph-mon[33714]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T15:07:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:39 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:39.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:39.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[29876]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[41614]: osdmap e610: 8 total, 8 up, 8 in 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:39.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:39 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:40.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:07:40.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:40.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:40.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:40.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:40.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:40.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:40.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:40.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:40.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:40 smithi134 ceph-mon[33714]: pgmap v898: 246 pgs: 32 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:40 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:40 smithi134 ceph-mon[33714]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T15:07:40.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:40.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:07:40.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[29876]: pgmap v898: 246 pgs: 32 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:40.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:40.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[29876]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T15:07:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[41614]: pgmap v898: 246 pgs: 32 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:40.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:40 smithi028 ceph-mon[41614]: osdmap e611: 8 total, 8 up, 8 in 2024-03-30T15:07:40.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:40.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:40.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:41.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:07:41.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:41.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:41.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:41.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.290+0000 7f0e9359f700 1 -- 172.21.15.28:0/598097679 >> v1:172.21.15.28:6789/0 conn(0x7f0e8c10d960 legacy=0x7f0e8c10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.290+0000 7f0e9359f700 1 -- 172.21.15.28:0/598097679 shutdown_connections 2024-03-30T15:07:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.290+0000 7f0e9359f700 1 -- 172.21.15.28:0/598097679 >> 172.21.15.28:0/598097679 conn(0x7f0e8c0fc9e0 msgr2=0x7f0e8c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 -- 172.21.15.28:0/598097679 shutdown_connections 2024-03-30T15:07:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 -- 172.21.15.28:0/598097679 wait complete. 2024-03-30T15:07:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 librados: starting msgr at 2024-03-30T15:07:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 librados: starting objecter 2024-03-30T15:07:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 Processor -- start 2024-03-30T15:07:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 -- start start 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 librados: setting wanted keys 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.291+0000 7f0e9359f700 1 librados: calling monclient init 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e9359f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e8c1a9310 con 0x7f0e8c101130 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e9359f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e8c1aa530 con 0x7f0e8c10d960 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e9359f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e8c1ab770 con 0x7f0e8c109dc0 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e9133b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e8c101130 0x7f0e8c106310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37664/0 (socket says 172.21.15.28:37664) 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e9133b700 1 -- 172.21.15.28:0/2005075246 learned_addr learned my addr 172.21.15.28:0/2005075246 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter ms_handle_connect 0x7f0e8c109dc0 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter ms_handle_connect 0x7f0e8c101130 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter ms_handle_connect 0x7f0e8c10d960 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124169967 0 0) 0x7f0e8c1ab770 con 0x7f0e8c109dc0 2024-03-30T15:07:41.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e780031f0 con 0x7f0e8c109dc0 2024-03-30T15:07:41.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544948244 0 0) 0x7f0e8c1aa530 con 0x7f0e8c10d960 2024-03-30T15:07:41.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e8c1ab770 con 0x7f0e8c10d960 2024-03-30T15:07:41.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400212479 0 0) 0x7f0e8c1a9310 con 0x7f0e8c101130 2024-03-30T15:07:41.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e8c1aa530 con 0x7f0e8c101130 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.292+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1317265405 0 0) 0x7f0e780031f0 con 0x7f0e8c109dc0 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e8c1a9310 con 0x7f0e8c109dc0 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1238259225 0 0) 0x7f0e8c1ab770 con 0x7f0e8c10d960 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e780031f0 con 0x7f0e8c10d960 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1132234871 0 0) 0x7f0e8c1aa530 con 0x7f0e8c101130 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e8c1ab770 con 0x7f0e8c101130 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e7c001a10 con 0x7f0e8c109dc0 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e840016f0 con 0x7f0e8c10d960 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e880017f0 con 0x7f0e8c101130 2024-03-30T15:07:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4144418336 0 0) 0x7f0e8c1a9310 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 >> v1:172.21.15.134:6789/0 conn(0x7f0e8c10d960 legacy=0x7f0e8c1a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 >> v1:172.21.15.28:6789/0 conn(0x7f0e8c101130 legacy=0x7f0e8c106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c1ac9b0 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0e7c001c70 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.293+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e7c004bb0 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.294+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c1aa760 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.294+0000 7f0e9359f700 10 client.58481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.294+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e8c1aac20 con 0x7f0e8c109dc0 2024-03-30T15:07:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.294+0000 7f0e9359f700 1 librados: init done 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.295+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c102610 con 0x7f0e8c109dc0 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.296+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0e7c003930 con 0x7f0e8c109dc0 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.296+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(611..611 src has 442..611) ==== 9123+0+0 (unknown 675863581 0 0) 0x7f0e7c0888f0 con 0x7f0e8c109dc0 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.296+0000 7f0e827fc700 10 client.58481.objecter ms_dispatch 0x7f0e8c001550 osd_map(611..611 src has 442..611) 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.296+0000 7f0e827fc700 3 client.58481.objecter handle_osd_map got epochs [611,611] > 0 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.296+0000 7f0e827fc700 3 client.58481.objecter handle_osd_map decoding full epoch 611 2024-03-30T15:07:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.297+0000 7f0e827fc700 20 client.58481.objecter dump_active .. 0 homeless 2024-03-30T15:07:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.297+0000 7f0e827fc700 10 client.58481.objecter ms_handle_connect 0x7f0e78070550 2024-03-30T15:07:41.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.305+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0e7c055740 con 0x7f0e8c109dc0 2024-03-30T15:07:41.329 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:41.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:41.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:41.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:41.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:41.554+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f0e8c103b10 con 0x7f0e8c109dc0 2024-03-30T15:07:41.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:41.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:41.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:41.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2005075246' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:41.855 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[41614]: from='client.58481 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2005075246' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:41.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:41 smithi028 ceph-mon[29876]: from='client.58481 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:41.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:41.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: osdmap e612: 8 total, 8 up, 8 in 2024-03-30T15:07:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2005075246' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:41 smithi134 ceph-mon[33714]: from='client.58481 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:07:42.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:42.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:42.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:42.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:42.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:42.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.342+0000 7f0e827fc700 1 -- 172.21.15.28:0/2005075246 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v613) ==== 113+0+0 (unknown 1833780513 0 0) 0x7f0e7c059190 con 0x7f0e8c109dc0 2024-03-30T15:07:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.349+0000 7f0e9359f700 10 librados: watch_flush enter 2024-03-30T15:07:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.350+0000 7f0e9359f700 10 librados: watch_flush exit 2024-03-30T15:07:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.350+0000 7f0e9359f700 20 client.58481.objecter shutdown clearing up homeless session... 2024-03-30T15:07:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.350+0000 7f0e9359f700 10 client.58481.objecter successfully canceled tick 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0e78070550 legacy=0x7f0e78072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 >> v1:172.21.15.28:6790/0 conn(0x7f0e8c109dc0 legacy=0x7f0e8c1a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 shutdown_connections 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 >> 172.21.15.28:0/2005075246 conn(0x7f0e8c0fc9e0 msgr2=0x7f0e8c110d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 shutdown_connections 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.351+0000 7f0e9359f700 1 -- 172.21.15.28:0/2005075246 wait complete. 2024-03-30T15:07:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:42.353+0000 7f0e9359f700 1 librados: shutdown 2024-03-30T15:07:42.355 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T15:07:42.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:42.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:42.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:42.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:42.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:42.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:42.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:42.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:42.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:42.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: pgmap v901: 278 pgs: 64 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: from='client.58481 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:42 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: pgmap v901: 278 pgs: 64 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: from='client.58481 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: pgmap v901: 278 pgs: 64 unknown, 4 creating+activating, 210 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-30T15:07:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: from='client.58481 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:07:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: osdmap e613: 8 total, 8 up, 8 in 2024-03-30T15:07:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:42 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:43.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:43.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:43.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:43.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:43.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:43.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:43.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:43.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:43.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:43.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:43.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:43.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:43.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:43.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:43.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:43.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:44.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:44.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:44.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:44.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.309+0000 7f45d55b7700 1 -- 172.21.15.28:0/3293783072 >> v1:172.21.15.28:6790/0 conn(0x7f45d0109dd0 legacy=0x7f45d010c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.309+0000 7f45d55b7700 1 -- 172.21.15.28:0/3293783072 shutdown_connections 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.309+0000 7f45d55b7700 1 -- 172.21.15.28:0/3293783072 >> 172.21.15.28:0/3293783072 conn(0x7f45d00fc9f0 msgr2=0x7f45d00fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.309+0000 7f45d55b7700 1 -- 172.21.15.28:0/3293783072 shutdown_connections 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.309+0000 7f45d55b7700 1 -- 172.21.15.28:0/3293783072 wait complete. 2024-03-30T15:07:44.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 librados: starting msgr at 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 librados: starting objecter 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 Processor -- start 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 -- start start 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 librados: setting wanted keys 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 librados: calling monclient init 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45d01068a0 con 0x7f45d0101140 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45d01a8550 con 0x7f45d0109dd0 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45d55b7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f45d01a9730 con 0x7f45d010d970 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45ceffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f45d0101140 0x7f45d01046c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37676/0 (socket says 172.21.15.28:37676) 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45ceffd700 1 -- 172.21.15.28:0/657358399 learned_addr learned my addr 172.21.15.28:0/657358399 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.310+0000 7f45cf7fe700 1 --1- 172.21.15.28:0/657358399 >> v1:172.21.15.28:6790/0 conn(0x7f45d010d970 0x7f45d01a5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50344/0 (socket says 172.21.15.28:50344) 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter ms_handle_connect 0x7f45d0101140 2024-03-30T15:07:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter ms_handle_connect 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter ms_handle_connect 0x7f45d0109dd0 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171204195 0 0) 0x7f45d01a9730 con 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45b80031f0 con 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3611117202 0 0) 0x7f45d01068a0 con 0x7f45d0101140 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45d01a9730 con 0x7f45d0101140 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86163479 0 0) 0x7f45d01a8550 con 0x7f45d0109dd0 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45d01068a0 con 0x7f45d0109dd0 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3270942099 0 0) 0x7f45b80031f0 con 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45d01a8550 con 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f45c8001e10 con 0x7f45d010d970 2024-03-30T15:07:44.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1272461206 0 0) 0x7f45d01a9730 con 0x7f45d0101140 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45b80031f0 con 0x7f45d0101140 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f45c0001870 con 0x7f45d0101140 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 498583706 0 0) 0x7f45d01068a0 con 0x7f45d0109dd0 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45d01a9730 con 0x7f45d0109dd0 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f45c40019a0 con 0x7f45d0109dd0 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3551857611 0 0) 0x7f45d01a8550 con 0x7f45d010d970 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 >> v1:172.21.15.134:6789/0 conn(0x7f45d0109dd0 legacy=0x7f45d0104dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 >> v1:172.21.15.28:6789/0 conn(0x7f45d0101140 legacy=0x7f45d01046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d01aa910 con 0x7f45d010d970 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01a75a0 con 0x7f45d010d970 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45d55b7700 10 client.58487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f45d01a7a60 con 0x7f45d010d970 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45d55b7700 1 librados: init done 2024-03-30T15:07:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f45c80020b0 con 0x7f45d010d970 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.311+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f45c8005e60 con 0x7f45d010d970 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.312+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45bc005120 con 0x7f45d010d970 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f45c8004170 con 0x7f45d010d970 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(614..614 src has 442..614) ==== 9523+0+0 (unknown 4221940345 0 0) 0x7f45c8089de0 con 0x7f45d010d970 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 10 client.58487.objecter ms_dispatch 0x7f45d0001550 osd_map(614..614 src has 442..614) 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 3 client.58487.objecter handle_osd_map got epochs [614,614] > 0 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 3 client.58487.objecter handle_osd_map decoding full epoch 614 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 20 client.58487.objecter dump_active .. 0 homeless 2024-03-30T15:07:44.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.313+0000 7f45b7fff700 10 client.58487.objecter ms_handle_connect 0x7f45b8070530 2024-03-30T15:07:44.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:44.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.318+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f45c8005e60 con 0x7f45d010d970 2024-03-30T15:07:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:44.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[29876]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T15:07:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[29876]: pgmap v904: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[41614]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T15:07:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:44 smithi028 ceph-mon[41614]: pgmap v904: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:44.541+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f45bc005410 con 0x7f45d010d970 2024-03-30T15:07:44.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:44 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:44.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:44 smithi134 ceph-mon[33714]: osdmap e614: 8 total, 8 up, 8 in 2024-03-30T15:07:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:44 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:44 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:44.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:44 smithi134 ceph-mon[33714]: pgmap v904: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:44.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:44.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:44.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:44.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:44.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:45.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:45.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:45.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-03-30T15:07:45.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.350+0000 7f45b7fff700 1 -- 172.21.15.28:0/657358399 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v616) ==== 97+0+0 (unknown 441404395 0 0) 0x7f45c8056aa0 con 0x7f45d010d970 2024-03-30T15:07:45.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 10 librados: watch_flush enter 2024-03-30T15:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 10 librados: watch_flush exit 2024-03-30T15:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 20 client.58487.objecter shutdown clearing up homeless session... 2024-03-30T15:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 10 client.58487.objecter successfully canceled tick 2024-03-30T15:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 >> v1:172.21.15.28:6800/3011431093 conn(0x7f45b8070530 legacy=0x7f45b80729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.354+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 >> v1:172.21.15.28:6790/0 conn(0x7f45d010d970 legacy=0x7f45d01a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.355+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 shutdown_connections 2024-03-30T15:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.355+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 >> 172.21.15.28:0/657358399 conn(0x7f45d00fc9f0 msgr2=0x7f45d006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.355+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 shutdown_connections 2024-03-30T15:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.355+0000 7f45d55b7700 1 -- 172.21.15.28:0/657358399 wait complete. 2024-03-30T15:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:45.356+0000 7f45d55b7700 1 librados: shutdown 2024-03-30T15:07:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:07:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/657358399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[29876]: from='client.58487 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/657358399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:45 smithi028 ceph-mon[41614]: from='client.58487 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:45.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-107", "overlaypool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: osdmap e615: 8 total, 8 up, 8 in 2024-03-30T15:07:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/657358399' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:45 smithi134 ceph-mon[33714]: from='client.58487 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:07:45.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:45.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:45.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:45.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:46.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:46.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:46.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.172+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3684754023 >> v1:172.21.15.28:6790/0 conn(0x7f7c1810d960 legacy=0x7f7c1810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.172+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3684754023 shutdown_connections 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.172+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3684754023 >> 172.21.15.28:0/3684754023 conn(0x7f7c180fc9e0 msgr2=0x7f7c180fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.172+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3684754023 shutdown_connections 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.172+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3684754023 wait complete. 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 librados: starting msgr at 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 librados: starting objecter 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 Processor -- start 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 -- start start 2024-03-30T15:07:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 librados: setting wanted keys 2024-03-30T15:07:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 librados: calling monclient init 2024-03-30T15:07:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c18106920 con 0x7f7c18109dc0 2024-03-30T15:07:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c181a8540 con 0x7f7c1810d960 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1f57e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7c181a9720 con 0x7f7c18101130 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1cb19700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7c18109dc0 0x7f7c18104e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37702/0 (socket says 172.21.15.28:37702) 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c1cb19700 1 -- 172.21.15.28:0/3154916810 learned_addr learned my addr 172.21.15.28:0/3154916810 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter ms_handle_connect 0x7f7c18101130 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter ms_handle_connect 0x7f7c18109dc0 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755589459 0 0) 0x7f7c181a9720 con 0x7f7c18101130 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c040031f0 con 0x7f7c18101130 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212882219 0 0) 0x7f7c18106920 con 0x7f7c18109dc0 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c181a9720 con 0x7f7c18109dc0 2024-03-30T15:07:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter ms_handle_connect 0x7f7c1810d960 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.173+0000 7f7c0a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 690901371 0 0) 0x7f7c181a8540 con 0x7f7c1810d960 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c18106920 con 0x7f7c1810d960 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4258536631 0 0) 0x7f7c040031f0 con 0x7f7c18101130 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c181a8540 con 0x7f7c18101130 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7c14001830 con 0x7f7c18101130 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3715882816 0 0) 0x7f7c181a9720 con 0x7f7c18109dc0 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c040031f0 con 0x7f7c18109dc0 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7c0c001890 con 0x7f7c18109dc0 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3300014340 0 0) 0x7f7c181a8540 con 0x7f7c18101130 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.134:6789/0 conn(0x7f7c1810d960 legacy=0x7f7c181a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6789/0 conn(0x7f7c18109dc0 legacy=0x7f7c18104e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c181aa900 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c181a7590 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c1f57e700 10 client.58493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c181a7a50 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c1f57e700 1 librados: init done 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7c14003b20 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.174+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7c14004b50 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.175+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7bfc005130 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.175+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7c1401b1d0 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(616..616 src has 442..616) ==== 9523+0+0 (unknown 101576842 0 0) 0x7f7c14088a70 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 osd_map(616..616 src has 442..616) 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 3 client.58493.objecter handle_osd_map got epochs [616,616] > 0 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 3 client.58493.objecter handle_osd_map decoding full epoch 616 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 20 client.58493.objecter dump_active .. 0 homeless 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v616) ==== 54+0+16 (unknown 1367906752 0 2420000192) 0x7f7c14088e30 con 0x7f7c18101130 2024-03-30T15:07:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7c04070160 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 1 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 1 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 0 1 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 4 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 10 client.58493.objecter _send_command 1 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.176+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc004b80 con 0x7f7bfc004760 2024-03-30T15:07:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc004760 2024-03-30T15:07:46.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc004b80 con 0x7f7bfc004760 2024-03-30T15:07:46.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:46.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 1 = system:0 2024-03-30T15:07:46.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 4 2024-03-30T15:07:46.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.177+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 0 1 2024-03-30T15:07:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 2 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 2 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc005410 osd=0 3 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 0 2 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 4 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 10 client.58493.objecter _send_command 2 2024-03-30T15:07:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.194+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc007b30 con 0x7f7bfc004760 2024-03-30T15:07:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc007b30 con 0x7f7bfc004760 2024-03-30T15:07:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 4 2024-03-30T15:07:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 0 2 2024-03-30T15:07:46.198 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:46.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 1 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 4 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 10 client.58493.objecter _send_command 3 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc002230 con 0x7f7bfc004320 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.196+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc004320 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.197+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc002230 con 0x7f7bfc004320 2024-03-30T15:07:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.197+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.197+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 3 = system:0 2024-03-30T15:07:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.197+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 4 2024-03-30T15:07:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.197+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 1 3 2024-03-30T15:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 4 2024-03-30T15:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 4 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc0041b0 osd=1 3 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 1 4 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 4 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 10 client.58493.objecter _send_command 4 2024-03-30T15:07:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.214+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc001630 con 0x7f7bfc004320 2024-03-30T15:07:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc001630 con 0x7f7bfc004320 2024-03-30T15:07:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 4 2024-03-30T15:07:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 1 4 2024-03-30T15:07:46.218 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 5 2024-03-30T15:07:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 5 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 2 5 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 4 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 10 client.58493.objecter _send_command 5 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc011220 con 0x7f7bfc00d710 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.216+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc00d710 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.217+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc011220 con 0x7f7bfc00d710 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.217+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.217+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 5 = system:0 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.217+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 4 2024-03-30T15:07:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.217+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 2 5 2024-03-30T15:07:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 6 2024-03-30T15:07:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 6 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc0084b0 osd=2 3 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 2 6 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 4 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 10 client.58493.objecter _send_command 6 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.234+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc007ff0 con 0x7f7bfc00d710 2024-03-30T15:07:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.235+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc007ff0 con 0x7f7bfc00d710 2024-03-30T15:07:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.235+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.235+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.235+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 4 2024-03-30T15:07:46.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.235+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 2 6 2024-03-30T15:07:46.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 7 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 7 2024-03-30T15:07:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 3 7 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 4 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 10 client.58493.objecter _send_command 7 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc01a470 con 0x7f7bfc016960 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.236+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc016960 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.237+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc01a470 con 0x7f7bfc016960 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.237+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.237+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 7 = system:0 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.237+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 4 2024-03-30T15:07:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.237+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 3 7 2024-03-30T15:07:46.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 8 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 8 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc008280 osd=3 3 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 3 8 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 4 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 10 client.58493.objecter _send_command 8 2024-03-30T15:07:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.254+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc00caf0 con 0x7f7bfc016960 2024-03-30T15:07:46.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.255+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc00caf0 con 0x7f7bfc016960 2024-03-30T15:07:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.255+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.255+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.255+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 4 2024-03-30T15:07:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.255+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 3 8 2024-03-30T15:07:46.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 9 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 9 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 4 9 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 4 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 10 client.58493.objecter _send_command 9 2024-03-30T15:07:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc01eed0 con 0x7f7bfc0120b0 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.256+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc0120b0 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.257+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc01eed0 con 0x7f7bfc0120b0 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.257+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.257+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 9 = system:0 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.257+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 4 2024-03-30T15:07:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.257+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 4 9 2024-03-30T15:07:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 10 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 10 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc00c8a0 osd=4 3 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 4 10 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 4 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 10 client.58493.objecter _send_command 10 2024-03-30T15:07:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.273+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc011620 con 0x7f7bfc0120b0 2024-03-30T15:07:46.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:46.277 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.275+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc011620 con 0x7f7bfc0120b0 2024-03-30T15:07:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.275+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.275+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.275+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 4 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.275+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 4 10 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 11 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 11 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 5 11 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 4 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 10 client.58493.objecter _send_command 11 2024-03-30T15:07:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc01b460 con 0x7f7bfc011790 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.276+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc011790 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.277+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc01b460 con 0x7f7bfc011790 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.277+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.277+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 11 = system:0 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.277+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 4 2024-03-30T15:07:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.277+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 5 11 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 12 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 12 2024-03-30T15:07:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc011620 osd=5 3 2024-03-30T15:07:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 5 12 2024-03-30T15:07:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 4 2024-03-30T15:07:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 10 client.58493.objecter _send_command 12 2024-03-30T15:07:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.294+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc01b460 con 0x7f7bfc011790 2024-03-30T15:07:46.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc01b460 con 0x7f7bfc011790 2024-03-30T15:07:46.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 4 2024-03-30T15:07:46.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 5 12 2024-03-30T15:07:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:46.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 13 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 13 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 6 13 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 4 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 10 client.58493.objecter _send_command 13 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.296+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc0279f0 con 0x7f7bfc023c90 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc023c90 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc0279f0 con 0x7f7bfc023c90 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 13 = system:0 2024-03-30T15:07:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 4 2024-03-30T15:07:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.297+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 6 13 2024-03-30T15:07:46.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 14 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 14 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc01b460 osd=6 3 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 6 14 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 4 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 10 client.58493.objecter _send_command 14 2024-03-30T15:07:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.314+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc027bd0 con 0x7f7bfc023c90 2024-03-30T15:07:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc027bd0 con 0x7f7bfc023c90 2024-03-30T15:07:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 4 2024-03-30T15:07:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 6 14 2024-03-30T15:07:46.318 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 15 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 15 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 7 15 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 4 2024-03-30T15:07:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 10 client.58493.objecter _send_command 15 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.316+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7bfc028e60 con 0x7f7bfc0282f0 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.317+0000 7f7c0a7fc700 10 client.58493.objecter ms_handle_connect 0x7f7bfc0282f0 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.318+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7bfc028e60 con 0x7f7bfc0282f0 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.318+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.318+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 15 = system:0 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.318+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 4 2024-03-30T15:07:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.318+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 7 15 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 10 client.58493.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign -1 16 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 20 client.58493.objecter _get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_remove -1 16 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 20 client.58493.objecter get_session s=0x7f7bfc027bd0 osd=7 3 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 15 client.58493.objecter _session_command_op_assign 7 16 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 4 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 10 client.58493.objecter _send_command 16 2024-03-30T15:07:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.335+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7bfc028e60 con 0x7f7bfc0282f0 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.336+0000 7f7c0a7fc700 1 -- 172.21.15.28:0/3154916810 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7bfc028e60 con 0x7f7bfc0282f0 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.336+0000 7f7c0a7fc700 10 client.58493.objecter ms_dispatch 0x7f7c18001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.336+0000 7f7c0a7fc700 10 client.58493.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.336+0000 7f7c0a7fc700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 4 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.336+0000 7f7c0a7fc700 15 client.58493.objecter _session_command_op_remove 7 16 2024-03-30T15:07:46.338 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 librados: watch_flush enter 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 librados: watch_flush exit 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.0 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6801/3394323575 conn(0x7f7bfc004760 legacy=0x7f7bfc0027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc005410 osd=0 1 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.1 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6805/1850313547 conn(0x7f7bfc004320 legacy=0x7f7bfc00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0041b0 osd=1 1 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.2 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6809/3451325043 conn(0x7f7bfc00d710 legacy=0x7f7bfc00fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc0084b0 osd=2 1 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.3 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6813/1506157250 conn(0x7f7bfc016960 legacy=0x7f7bfc018d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc008280 osd=3 1 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.4 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.134:6800/3063775514 conn(0x7f7bfc0120b0 legacy=0x7f7bfc01d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc00c8a0 osd=4 1 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.5 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1db1b700 1 -- 172.21.15.28:0/3154916810 reap_dead start 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.134:6804/3668151179 conn(0x7f7bfc011790 legacy=0x7f7bfc022060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1db1b700 1 -- 172.21.15.28:0/3154916810 reap_dead start 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1db1b700 1 -- 172.21.15.28:0/3154916810 reap_dead start 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc011620 osd=5 1 2024-03-30T15:07:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.6 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.134:6808/2135224194 conn(0x7f7bfc023c90 legacy=0x7f7bfc026100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc01b460 osd=6 1 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter close_session for osd.7 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.134:6812/3758190298 conn(0x7f7bfc0282f0 legacy=0x7f7bfc028750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter put_session s=0x7f7bfc027bd0 osd=7 1 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 20 client.58493.objecter shutdown clearing up homeless session... 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 10 client.58493.objecter successfully canceled tick 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7c04070160 legacy=0x7f7c04072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> v1:172.21.15.28:6790/0 conn(0x7f7c18101130 legacy=0x7f7c181046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1db1b700 1 -- 172.21.15.28:0/3154916810 reap_dead start 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 shutdown_connections 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 >> 172.21.15.28:0/3154916810 conn(0x7f7c180fc9e0 msgr2=0x7f7c181105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 shutdown_connections 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 -- 172.21.15.28:0/3154916810 wait complete. 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:46.337+0000 7f7c1f57e700 1 librados: shutdown 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:46.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:46.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: from='client.58487 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: pgmap v907: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3154916810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: from='client.58487 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: pgmap v907: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3154916810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:46.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:46.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:46.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-107-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: from='client.58487 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: osdmap e616: 8 total, 8 up, 8 in 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]: dispatch 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: pgmap v907: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3154916810' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:46.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:46.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:46.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:46.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:47.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:47.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:47.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:47.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:47.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:47.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:47.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]': finished 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[29876]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]': finished 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[41614]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:47 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:47.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:47.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:47.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:47 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-107"}]': finished 2024-03-30T15:07:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:47 smithi134 ceph-mon[33714]: osdmap e617: 8 total, 8 up, 8 in 2024-03-30T15:07:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2454894572' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:47 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]: dispatch 2024-03-30T15:07:47.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:47.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:47.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:47.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:47.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:48.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:48.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:48.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:48.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:48.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:48.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:48.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:07:48.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[29876]: pgmap v910: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:07:48.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:48 smithi028 ceph-mon[41614]: pgmap v910: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:48.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:48.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:48.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: from='client.58475 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-107", "tierpool": "test-rados-api-smithi028-93253-107-cache"}]': finished 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: osdmap e618: 8 total, 8 up, 8 in 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:07:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:48 smithi134 ceph-mon[33714]: pgmap v910: 278 pgs: 278 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:48.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:48.775 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:07:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:07:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:07:48.775 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:07:48.776 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:07:48.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:48.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:48.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:48.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:49.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:49.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:49.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:07:49.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:49.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:49.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:49.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:49.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:49.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:07:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:49.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:49.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:49.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:49 smithi028 ceph-mon[29876]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T15:07:49.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:49 smithi028 ceph-mon[41614]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T15:07:49.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:49 smithi134 ceph-mon[33714]: osdmap e619: 8 total, 8 up, 8 in 2024-03-30T15:07:49.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:49.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:49.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:07:49.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:49.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:49.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:49.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:49.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:50.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:50.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:50.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:50.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:50.286 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:50.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:50.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:50.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:50.588 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-mon[33714]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T15:07:50.588 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.588 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.589 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:50.589 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-mon[33714]: pgmap v913: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:50.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:50.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:50.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[29876]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T15:07:50.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[29876]: pgmap v913: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[41614]: osdmap e620: 8 total, 8 up, 8 in 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:07:50.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:50 smithi028 ceph-mon[41614]: pgmap v913: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:50.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:50.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:50.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:50.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:07:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:07:50.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:51.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:51.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:51.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:51.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:51.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:51.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:51.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:51.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:51.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:51.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.780 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:51.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:51.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:51.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:51 smithi134 ceph-mon[33714]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T15:07:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[29876]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[41614]: osdmap e621: 8 total, 8 up, 8 in 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:07:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:07:52.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:52.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:52.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:52.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:52.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:52.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:52.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:52.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:52.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:52.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:52.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:52.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:52.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[41614]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[41614]: pgmap v916: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[41614]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[29876]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[29876]: pgmap v916: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:07:52.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:52 smithi028 ceph-mon[29876]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T15:07:52.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:52.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:52.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:52 smithi134 ceph-mon[33714]: osdmap e622: 8 total, 8 up, 8 in 2024-03-30T15:07:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:52 smithi134 ceph-mon[33714]: pgmap v916: 214 pgs: 214 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:07:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:07:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:52 smithi134 ceph-mon[33714]: osdmap e623: 8 total, 8 up, 8 in 2024-03-30T15:07:53.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:53.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:53.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:53.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:53.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:53.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:53.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:53.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:53.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.689 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:53.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:53.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:53.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:53.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:53.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:53.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:54.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:54.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:54.245 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24910.0:2 25.17 25:ecb6a661:test-rados-api-smithi028-93441-2::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 599.27627735800002, 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3400814589999999, 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24910", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1422378701", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.247 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966911+0000", 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966913+0000", 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8819999999999999e-06 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966920+0000", 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0570000000000004e-06 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966930+0000", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0156000000000001e-05 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.966940+0000", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0147e-05 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083450+0000", 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.116509813 2024-03-30T15:07:54.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.083460+0000", 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9089999999999995e-06 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.292378+0000", 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2089187990000001 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.250 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294004+0000", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00162598 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.294807+0000", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080235899999999999 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.295802+0000", 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000994834 2024-03-30T15:07:54.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.299106+0000", 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003304202 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306871+0000", 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077654550000000001 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306919+0000", 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7447000000000002e-05 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.306992+0000", 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3418999999999996e-05 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.253 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24961.0:2 27.15 27:ad551702:::test:head [stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 599.27453034400003, 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23402463300000001, 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24961", 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1241392178", 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968658+0000", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968659+0000", 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.73e-06 2024-03-30T15:07:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968665+0000", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8200000000000002e-06 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968676+0000", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0528e-05 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.968685+0000", 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.329e-06 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969119+0000", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043351199999999999 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:54.969125+0000", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2280000000000003e-06 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049864+0000", 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080739092999999998 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.049873+0000", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7539999999999995e-06 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202601+0000", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.152728379 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.202682+0000", 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1260000000000005e-05 2024-03-30T15:07:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35042.0:4 21.1f 21:ff7b1f36:::obj1:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 599.06825686699995, 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0929989090000001, 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35042", 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/917910517", 2024-03-30T15:07:54.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174931+0000", 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174933+0000", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.421e-06 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174937+0000", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4139999999999996e-06 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174945+0000", 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2090000000000005e-06 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.174955+0000", 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5300000000000002e-06 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175014+0000", 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9110000000000002e-05 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.175023+0000", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2920000000000001e-06 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267861+0000", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0928374670000001 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267930+0000", 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9466000000000002e-05 2024-03-30T15:07:54.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34820.0:20 9.8 9:136ae20b:test-rados-api-smithi028-93069-6::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 599.02883923900004, 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.089392117, 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.264 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34820", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4253613591", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 20 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214349+0000", 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214351+0000", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.103e-06 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214358+0000", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5320000000000001e-06 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214370+0000", 2024-03-30T15:07:54.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1377e-05 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214379+0000", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9090000000000006e-06 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214437+0000", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8692e-05 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.214446+0000", 2024-03-30T15:07:54.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8680000000000005e-06 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253200+0000", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.038753864 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253307+0000", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010676299999999999 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.253432+0000", 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124957 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.266525+0000", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013092763 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.268732+0000", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022073129999999998 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303646+0000", 2024-03-30T15:07:54.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034913911999999998 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303698+0000", 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1986e-05 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.303741+0000", 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3078000000000003e-05 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25075.0:2 39.1b 39:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 598.99991316299997, 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0594717039999999, 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25075", 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3988199589", 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243275+0000", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243277+0000", 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.796e-06 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243283+0000", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2740000000000004e-06 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243299+0000", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6532e-05 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243310+0000", 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0463e-05 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243383+0000", 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3207000000000006e-05 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:55.243392+0000", 2024-03-30T15:07:54.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8559999999999996e-06 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267447+0000", 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.024054757 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267568+0000", 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012110199999999999 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-03-30T15:07:54.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.267732+0000", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016446500000000001 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.280028+0000", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012295451000000001 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.284857+0000", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048288619999999997 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302648+0000", 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017791609999999999 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302697+0000", 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9116999999999997e-05 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.302747+0000", 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9212000000000002e-05 2024-03-30T15:07:54.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24853.0:7 14.1d 14:b9b5017e:test-rados-api-smithi028-93116-4::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 598.07866231900005, 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.092228541999999997, 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24853", 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2848446493", 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T15:07:54.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164526+0000", 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164528+0000", 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.328e-06 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164538+0000", 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0189e-05 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164550+0000", 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1353e-05 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.164560+0000", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0828e-05 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167851+0000", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032908939999999999 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.167956+0000", 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010505699999999999 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:57:56.256754+0000", 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088797893000000003 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35405.0:2 62.13 62:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 593.25342397199995, 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.977776094, 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35405", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1251877021", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989764+0000", 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989767+0000", 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.615e-06 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989773+0000", 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2430000000000002e-06 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989784+0000", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1043000000000001e-05 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:00.989793+0000", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5279999999999992e-06 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511344+0000", 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.52155030499999999 2024-03-30T15:07:54.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:01.511352+0000", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4479999999999994e-06 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949501+0000", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.43814895 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949615+0000", 2024-03-30T15:07:54.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114282 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.949747+0000", 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000131564 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956142+0000", 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063947639999999998 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.956280+0000", 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000138496 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967445+0000", 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011165029999999999 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967488+0000", 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2397000000000001e-05 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:02.967540+0000", 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2429000000000002e-05 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.27400.0:483 115.4s0 115:2235e3a1:::482:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 570.98421081799995, 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15220373500000001, 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.27400", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3323927705", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 483 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258977+0000", 2024-03-30T15:07:54.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258980+0000", 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5359999999999999e-06 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.258989+0000", 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1479999999999995e-06 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281859+0000", 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022870352 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.281876+0000", 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6492000000000001e-05 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338663+0000", 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056787694 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.338791+0000", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012725400000000001 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.339139+0000", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034834299999999998 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.368274+0000", 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029135306 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411112+0000", 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.042837274000000002 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.411181+0000", 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9336000000000002e-05 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:2 123.15 123:ad551702:::test:head [create,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-30T15:07:54.295 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 570.87886399499996, 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2153437069999999, 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364324+0000", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364326+0000", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.435e-06 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364336+0000", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6139999999999998e-06 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364348+0000", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2194e-05 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364360+0000", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1511e-05 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364419+0000", 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8956999999999998e-05 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:23.364429+0000", 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0026e-05 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538529+0000", 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.174100744 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538649+0000", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011977900000000001 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.538787+0000", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000137458 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.541812+0000", 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030249740000000002 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.545225+0000", 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034129009999999999 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571213+0000", 2024-03-30T15:07:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025988165000000001 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.571265+0000", 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2385000000000001e-05 2024-03-30T15:07:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:24.579668+0000", 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084025639999999995 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.303 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.304 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.44099.0:715 123.1f 123:fbee941d:::test_708:head [cmpext 0~0,write 0~3] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e143)", 2024-03-30T15:07:54.304 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 566.40841755999998, 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146558462, 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:07:54.305 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.44099", 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/740948191", 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 715 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.307 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834770+0000", 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834773+0000", 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.57e-06 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834784+0000", 2024-03-30T15:07:54.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1277999999999999e-05 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834803+0000", 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8281999999999999e-05 2024-03-30T15:07:54.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834813+0000", 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.078e-05 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.834910+0000", 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6772000000000001e-05 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.835058+0000", 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148066 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:27.981329+0000", 2024-03-30T15:07:54.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146270714 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.314 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "pg_update_log_missing(130.2 epoch 146/143 rep_tid 752 entries 146'28 (0'0) error 130:40466eb8:::test_509:head by client.46235.0:516 0.000000 -4095 [r=-4095+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 145'26)", 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 564.00885814399999, 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024038321000000001, 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234330+0000", 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.72e-07 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234339+0000", 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9590000000000001e-06 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234353+0000", 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3560000000000001e-05 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.234368+0000", 2024-03-30T15:07:54.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5308e-05 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.241947+0000", 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007578904 2024-03-30T15:07:54.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.258368+0000", 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016421318000000001 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.322 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.46235.0:517 130.5 130:a3243b6a:::test_510:head [cmpext 0~0,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e146)", 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:07:54.323 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 563.97305276700001, 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023663910999999999, 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.324 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.46235", 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2275516611", 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 517 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270135+0000", 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270137+0000", 2024-03-30T15:07:54.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.356e-06 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270143+0000", 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5889999999999999e-06 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270151+0000", 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5000000000000002e-06 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270160+0000", 2024-03-30T15:07:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6399999999999992e-06 2024-03-30T15:07:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270522+0000", 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036188600000000001 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.270653+0000", 2024-03-30T15:07:54.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000130791 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:30.293799+0000", 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023146149000000001 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42667.0:2 138.0 138:0384ab8e:::oid:head [write 0~7,setxattr key1 (5) in=4b,truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 562.91358308700001, 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93406830600000001, 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.333 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42667", 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2528295921", 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:07:54.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329605+0000", 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329607+0000", 2024-03-30T15:07:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.874e-06 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329614+0000", 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1169999999999998e-06 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329624+0000", 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0457e-05 2024-03-30T15:07:54.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.329636+0000", 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1382999999999999e-05 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350953+0000", 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021317545 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.350963+0000", 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0193e-05 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.249891+0000", 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89892724199999996 2024-03-30T15:07:54.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250074+0000", 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018359100000000001 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250291+0000", 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021668499999999999 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252632+0000", 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023406579999999998 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257757+0000", 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051254320000000001 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263567+0000", 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058095780000000001 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263628+0000", 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1237999999999999e-05 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.263673+0000", 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5312999999999998e-05 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.42271.0:2 136.6 136:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:07:54.342 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 562.06622968299996, 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.090549882999999998, 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.42271", 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/553852123", 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176958+0000", 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176961+0000", 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-06 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176971+0000", 2024-03-30T15:07:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9720000000000004e-06 2024-03-30T15:07:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.176993+0000", 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2331e-05 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.177006+0000", 2024-03-30T15:07:54.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3244999999999999e-05 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198821+0000", 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021814229000000001 2024-03-30T15:07:54.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.198830+0000", 2024-03-30T15:07:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8740000000000001e-06 2024-03-30T15:07:54.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250126+0000", 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.051296025000000002 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250242+0000", 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116638 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-03-30T15:07:54.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.250389+0000", 2024-03-30T15:07:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014667799999999999 2024-03-30T15:07:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.252485+0000", 2024-03-30T15:07:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020958959999999999 2024-03-30T15:07:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.257681+0000", 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005195987 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267377+0000", 2024-03-30T15:07:54.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0096964649999999996 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267466+0000", 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8781000000000003e-05 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:32.267508+0000", 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2174e-05 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.43429.0:2 140.7 140:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-30T15:07:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:07:54.358 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 560.93223731499995, 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.059809913999999999, 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.43429", 2024-03-30T15:07:54.359 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1777674724", 2024-03-30T15:07:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:07:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310951+0000", 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310953+0000", 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2840000000000001e-06 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.310961+0000", 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7340000000000008e-06 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313256+0000", 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022953600000000002 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313269+0000", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2472e-05 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313347+0000", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8175000000000003e-05 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.313356+0000", 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9029999999999993e-06 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316618+0000", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003262587 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316732+0000", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011378 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.316884+0000", 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015224100000000001 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.332636+0000", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015752226000000001 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.342429+0000", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0097926090000000007 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370655+0000", 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.028226161999999999 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370712+0000", 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6952e-05 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.370761+0000", 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8429e-05 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.366 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24384.0:1 254.6s0 254:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 428.00989356899998, 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.003921587, 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24384", 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3107101299", 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233294+0000", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233296+0000", 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8679999999999999e-06 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233304+0000", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5299999999999999e-06 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233315+0000", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1473e-05 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233327+0000", 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1598e-05 2024-03-30T15:07:54.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233890+0000", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056354599999999995 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:46.233894+0000", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3400000000000002e-06 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.232901+0000", 2024-03-30T15:07:54.370 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99900681199999997 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233056+0000", 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000155811 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233360+0000", 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000303982 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:07:54.371 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.233629+0000", 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000268712 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.237139+0000", 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035102390000000001 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:47.237216+0000", 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6675999999999996e-05 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24865.0:1050 17.c 17:30000000::::head [pgnls start_epoch 300] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e300)", 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 409.73697002400002, 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91512182399999997, 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24865", 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3133651075", 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1050 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506218+0000", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506219+0000", 2024-03-30T15:07:54.374 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5000000000000001e-07 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506223+0000", 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4689999999999999e-06 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506232+0000", 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6540000000000005e-06 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506239+0000", 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.148e-06 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506281+0000", 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1637000000000001e-05 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:04.506287+0000", 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4649999999999999e-06 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421133+0000", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91484526300000002 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421177+0000", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4280000000000003e-05 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:05.421340+0000", 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016295799999999999 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:3042 261.1f 261:f8f7c133:test-rados-api-smithi028-93253-36::foo:head [writefull 0~1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 407.01201176799998, 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027248054000000001, 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:07:54.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3042 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231176+0000", 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231181+0000", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5569999999999997e-06 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231199+0000", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8062000000000001e-05 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231227+0000", 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8402999999999999e-05 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.380 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231242+0000", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4956999999999999e-05 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231319+0000", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6625999999999994e-05 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.231326+0000", 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.832e-06 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.242303+0000", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010977551 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for blocked object", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.242479+0000", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000175878 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243461+0000", 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098175100000000011 2024-03-30T15:07:54.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243576+0000", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000115201 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,7", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.243753+0000", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017675199999999999 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.244990+0000", 2024-03-30T15:07:54.383 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001236975 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.249284+0000", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042942049999999997 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258343+0000", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090588839999999997 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258384+0000", 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1437999999999999e-05 2024-03-30T15:07:54.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:07.258424+0000", 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9981999999999998e-05 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:3240 267.4 267:20000000:%01:::head [pgnls start_epoch 316] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e316)", 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 393.99206211199999, 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21607105099999999, 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3240 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.386 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251126+0000", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251129+0000", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7829999999999999e-06 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251138+0000", 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4029999999999996e-06 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.387 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251152+0000", 2024-03-30T15:07:54.388 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4414e-05 2024-03-30T15:07:54.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251162+0000", 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3789999999999995e-06 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251207+0000", 2024-03-30T15:07:54.389 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5121999999999999e-05 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.251211+0000", 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0770000000000001e-06 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467004+0000", 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21579247800000001 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.390 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467041+0000", 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7539000000000001e-05 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:20.467197+0000", 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015585600000000001 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57695.0:1 268.2s1 268:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-30T15:07:54.391 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 393.01064825999998, 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.026729614, 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57695", 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2330167966", 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:07:54.392 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232540+0000", 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232541+0000", 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5269999999999999e-06 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232555+0000", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3358e-05 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232566+0000", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0951000000000001e-05 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232576+0000", 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0631e-05 2024-03-30T15:07:54.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232958+0000", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038161199999999998 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:21.232962+0000", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8369999999999999e-06 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255130+0000", 2024-03-30T15:07:54.395 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0221682700000001 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255262+0000", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132003 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.255735+0000", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047280999999999999 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.257894+0000", 2024-03-30T15:07:54.396 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021590899999999998 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.258958+0000", 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010644039999999999 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:22.259269+0000", 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031112099999999998 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:07:54.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:07:54.398 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:07:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:54.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[29876]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T15:07:54.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:54.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[29876]: pgmap v919: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:54.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[41614]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T15:07:54.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:54.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:54 smithi028 ceph-mon[41614]: pgmap v919: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:54.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:54.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:54 smithi134 ceph-mon[33714]: osdmap e624: 8 total, 8 up, 8 in 2024-03-30T15:07:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:07:54.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:54 smithi134 ceph-mon[33714]: pgmap v919: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:54.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:54.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:54.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:54.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:54.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:55.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:55.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.209+0000 7fd8d35be700 1 -- 172.21.15.28:0/2260624212 >> v1:172.21.15.28:6790/0 conn(0x7fd8cc102250 legacy=0x7fd8cc102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.209+0000 7fd8d35be700 1 -- 172.21.15.28:0/2260624212 shutdown_connections 2024-03-30T15:07:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.209+0000 7fd8d35be700 1 -- 172.21.15.28:0/2260624212 >> 172.21.15.28:0/2260624212 conn(0x7fd8cc0fdb20 msgr2=0x7fd8cc0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- 172.21.15.28:0/2260624212 shutdown_connections 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- 172.21.15.28:0/2260624212 wait complete. 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 librados: starting msgr at 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 librados: starting objecter 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 Processor -- start 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- start start 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 librados: setting wanted keys 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 librados: calling monclient init 2024-03-30T15:07:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8cc1a93f0 con 0x7fd8cc10aee0 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8cc1aa610 con 0x7fd8cc102250 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d35be700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8cc1ab850 con 0x7fd8cc10ea80 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d0b59700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd8cc10aee0 0x7fd8cc1a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48494/0 (socket says 172.21.15.28:48494) 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.210+0000 7fd8d0b59700 1 -- 172.21.15.28:0/1152170437 learned_addr learned my addr 172.21.15.28:0/1152170437 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter ms_handle_connect 0x7fd8cc10ea80 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter ms_handle_connect 0x7fd8cc10aee0 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062371829 0 0) 0x7fd8cc1ab850 con 0x7fd8cc10ea80 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b80031f0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029752694 0 0) 0x7fd8cc1a93f0 con 0x7fd8cc10aee0 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8cc1ab850 con 0x7fd8cc10aee0 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter ms_handle_connect 0x7fd8cc102250 2024-03-30T15:07:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1376712999 0 0) 0x7fd8b80031f0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8cc1a93f0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8c4001db0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737957044 0 0) 0x7fd8cc1aa610 con 0x7fd8cc102250 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b80031f0 con 0x7fd8cc102250 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 637858769 0 0) 0x7fd8cc1ab850 con 0x7fd8cc10aee0 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8cc1aa610 con 0x7fd8cc10aee0 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8c0001870 con 0x7fd8cc10aee0 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1466517728 0 0) 0x7fd8cc1a93f0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 >> v1:172.21.15.134:6789/0 conn(0x7fd8cc102250 legacy=0x7fd8cc107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 >> v1:172.21.15.28:6789/0 conn(0x7fd8cc10aee0 legacy=0x7fd8cc1a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc1aca90 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8cc1a9620 con 0x7fd8cc10ea80 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8d35be700 10 client.58505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8cc1a9ae0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.211+0000 7fd8d35be700 1 librados: init done 2024-03-30T15:07:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.212+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd8c40019b0 con 0x7fd8cc10ea80 2024-03-30T15:07:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.212+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8c4004d70 con 0x7fd8cc10ea80 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b0005120 con 0x7fd8cc10ea80 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd8c4005080 con 0x7fd8cc10ea80 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(625..625 src has 442..625) ==== 9988+0+0 (unknown 2238593870 0 0) 0x7fd8c4089020 con 0x7fd8cc10ea80 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8be7fc700 10 client.58505.objecter ms_dispatch 0x7fd8cc001550 osd_map(625..625 src has 442..625) 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8be7fc700 3 client.58505.objecter handle_osd_map got epochs [625,625] > 0 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.213+0000 7fd8be7fc700 3 client.58505.objecter handle_osd_map decoding full epoch 625 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.214+0000 7fd8be7fc700 20 client.58505.objecter dump_active .. 0 homeless 2024-03-30T15:07:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.214+0000 7fd8be7fc700 10 client.58505.objecter ms_handle_connect 0x7fd8b8070410 2024-03-30T15:07:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.218+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd8c4055b10 con 0x7fd8cc10ea80 2024-03-30T15:07:55.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:55.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:55.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[41614]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:55.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:55.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[41614]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T15:07:55.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:55.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[29876]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:55.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:55.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[29876]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T15:07:55.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:55.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:55.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.449+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd8b0005410 con 0x7fd8cc10ea80 2024-03-30T15:07:55.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:55.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:55 smithi134 ceph-mon[33714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:07:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:07:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:55 smithi134 ceph-mon[33714]: osdmap e625: 8 total, 8 up, 8 in 2024-03-30T15:07:55.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:55.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 -- 172.21.15.28:0/3167085609 >> v1:172.21.15.28:6790/0 conn(0x7fdf6010ea70 legacy=0x7fdf60110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 -- 172.21.15.28:0/3167085609 shutdown_connections 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 -- 172.21.15.28:0/3167085609 >> 172.21.15.28:0/3167085609 conn(0x7fdf600fdaf0 msgr2=0x7fdf600fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 -- 172.21.15.28:0/3167085609 shutdown_connections 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 -- 172.21.15.28:0/3167085609 wait complete. 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 librados: starting msgr at 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 librados: starting objecter 2024-03-30T15:07:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.762+0000 7fdf65bf1700 1 Processor -- start 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 -- start start 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 librados: setting wanted keys 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 librados: calling monclient init 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf601a6ed0 con 0x7fdf6010aed0 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf601a80f0 con 0x7fdf60102240 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf65bf1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf601a9330 con 0x7fdf6010ea70 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5effd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf6010aed0 0x7fdf601a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48506/0 (socket says 172.21.15.28:48506) 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5effd700 1 -- 172.21.15.28:0/969899839 learned_addr learned my addr 172.21.15.28:0/969899839 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5ffff700 1 --1- 172.21.15.28:0/969899839 >> v1:172.21.15.28:6790/0 conn(0x7fdf6010ea70 0x7fdf601a5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39890/0 (socket says 172.21.15.28:39890) 2024-03-30T15:07:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter ms_handle_connect 0x7fdf6010ea70 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter ms_handle_connect 0x7fdf6010aed0 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter ms_handle_connect 0x7fdf60102240 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859442598 0 0) 0x7fdf601a9330 con 0x7fdf6010ea70 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf4c0031f0 con 0x7fdf6010ea70 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3220813849 0 0) 0x7fdf601a6ed0 con 0x7fdf6010aed0 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf601a9330 con 0x7fdf6010aed0 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083234756 0 0) 0x7fdf601a80f0 con 0x7fdf60102240 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.763+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf601a6ed0 con 0x7fdf60102240 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2411959351 0 0) 0x7fdf4c0031f0 con 0x7fdf6010ea70 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf601a80f0 con 0x7fdf6010ea70 2024-03-30T15:07:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf50004620 con 0x7fdf6010ea70 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 961267973 0 0) 0x7fdf601a9330 con 0x7fdf6010aed0 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf4c0031f0 con 0x7fdf6010aed0 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf480017b0 con 0x7fdf6010aed0 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2970133210 0 0) 0x7fdf601a6ed0 con 0x7fdf60102240 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf601a9330 con 0x7fdf60102240 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf58001cc0 con 0x7fdf60102240 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1929021543 0 0) 0x7fdf601a80f0 con 0x7fdf6010ea70 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 >> v1:172.21.15.134:6789/0 conn(0x7fdf60102240 legacy=0x7fdf60107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 >> v1:172.21.15.28:6789/0 conn(0x7fdf6010aed0 legacy=0x7fdf601a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf601aa570 con 0x7fdf6010ea70 2024-03-30T15:07:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf601a7100 con 0x7fdf6010ea70 2024-03-30T15:07:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf65bf1700 10 client.58511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf601a75c0 con 0x7fdf6010ea70 2024-03-30T15:07:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf65bf1700 1 librados: init done 2024-03-30T15:07:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdf50001b70 con 0x7fdf6010ea70 2024-03-30T15:07:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.764+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf50004d50 con 0x7fdf6010ea70 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.765+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf40005120 con 0x7fdf6010ea70 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.765+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdf50003680 con 0x7fdf6010ea70 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(626..626 src has 442..626) ==== 9983+0+0 (unknown 4219265 0 0) 0x7fdf50089d70 con 0x7fdf6010ea70 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 10 client.58511.objecter ms_dispatch 0x7fdf60001550 osd_map(626..626 src has 442..626) 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 3 client.58511.objecter handle_osd_map got epochs [626,626] > 0 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 3 client.58511.objecter handle_osd_map decoding full epoch 626 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 20 client.58511.objecter dump_active .. 0 homeless 2024-03-30T15:07:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.766+0000 7fdf5cff9700 10 client.58511.objecter ms_handle_connect 0x7fdf4c070490 2024-03-30T15:07:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.770+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdf50056860 con 0x7fdf6010ea70 2024-03-30T15:07:55.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:55.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:55.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:55.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:55.991+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fdf40005410 con 0x7fdf6010ea70 2024-03-30T15:07:56.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:56.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:56.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:56.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:56.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.362+0000 7fdf5cff9700 1 -- 172.21.15.28:0/969899839 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v627) ==== 113+0+0 (unknown 2875382213 0 0) 0x7fdf5005a2b0 con 0x7fdf6010ea70 2024-03-30T15:07:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.362+0000 7fd8be7fc700 1 -- 172.21.15.28:0/1152170437 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v627) ==== 91+0+0 (unknown 3984234325 0 0) 0x7fd8c4059560 con 0x7fd8cc10ea80 2024-03-30T15:07:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 10 librados: watch_flush enter 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 10 librados: watch_flush exit 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 20 client.58511.objecter shutdown clearing up homeless session... 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 10 client.58511.objecter successfully canceled tick 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdf4c070490 legacy=0x7fdf4c072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 >> v1:172.21.15.28:6790/0 conn(0x7fdf6010ea70 legacy=0x7fdf601a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 shutdown_connections 2024-03-30T15:07:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 >> 172.21.15.28:0/969899839 conn(0x7fdf600fdaf0 msgr2=0x7fdf600fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fd8d35be700 10 librados: watch_flush enter 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fd8d35be700 10 librados: watch_flush exit 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fd8d35be700 20 client.58505.objecter shutdown clearing up homeless session... 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fd8d35be700 10 client.58505.objecter successfully canceled tick 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.364+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd8b8070410 legacy=0x7fd8b80728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 >> v1:172.21.15.28:6790/0 conn(0x7fd8cc10ea80 legacy=0x7fd8cc1a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 shutdown_connections 2024-03-30T15:07:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 shutdown_connections 2024-03-30T15:07:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 >> 172.21.15.28:0/1152170437 conn(0x7fd8cc0fdb20 msgr2=0x7fd8cc111bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fdf65bf1700 1 -- 172.21.15.28:0/969899839 wait complete. 2024-03-30T15:07:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fdf65bf1700 1 librados: shutdown 2024-03-30T15:07:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 shutdown_connections 2024-03-30T15:07:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 -- 172.21.15.28:0/1152170437 wait complete. 2024-03-30T15:07:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:56.365+0000 7fd8d35be700 1 librados: shutdown 2024-03-30T15:07:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T15:07:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:07:56.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:56.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:56.523 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1152170437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.58505 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/969899839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: from='client.58511 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[41614]: pgmap v922: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:56.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1152170437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.58505 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/969899839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: from='client.58511 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:56 smithi028 ceph-mon[29876]: pgmap v922: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:56.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:56.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: osdmap e626: 8 total, 8 up, 8 in 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1152170437' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.58505 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:07:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/969899839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: from='client.58511 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:07:56.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:56 smithi134 ceph-mon[33714]: pgmap v922: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:07:56.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:56.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:56.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:56.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:56.962 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:56.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:57.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:07:57.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:57.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:57.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:57.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:57.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:57.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:07:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:07:57.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: from='client.58505 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: from='client.58511 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:07:57.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: from='client.58505 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: from='client.58511 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:07:57.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:57 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-109", "overlaypool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: from='client.58505 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: from='client.58511 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: osdmap e627: 8 total, 8 up, 8 in 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]: dispatch 2024-03-30T15:07:57.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:07:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:07:57.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:57 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:07:57.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:57.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:57.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:57.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:57.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T15:07:57.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:07:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:57.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:57.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:58.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:07:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:07:58.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:58.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:58.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:58.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:58.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:58.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:58.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: 42.10 scrub starts 2024-03-30T15:07:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: 42.10 scrub ok 2024-03-30T15:07:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:07:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: pgmap v925: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:07:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:58 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:58.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:58.710 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:07:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:07:58.710 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: 42.10 scrub starts 2024-03-30T15:07:58.710 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: 42.10 scrub ok 2024-03-30T15:07:58.710 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: pgmap v925: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:07:58.711 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: 42.10 scrub starts 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: 42.10 scrub ok 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:07:58.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-109-cache", "mode": "writeback"}]': finished 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: osdmap e628: 8 total, 8 up, 8 in 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: pgmap v925: 254 pgs: 32 creating+peering, 222 active+clean; 582 KiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:07:58.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:58 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:07:58.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:58.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:58.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:58.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:58.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:58.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.992+0000 7f533e84a700 1 -- 172.21.15.28:0/1423186325 >> v1:172.21.15.134:6789/0 conn(0x7f533810ae80 legacy=0x7f533810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.992+0000 7f533e84a700 1 -- 172.21.15.28:0/1423186325 shutdown_connections 2024-03-30T15:07:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.992+0000 7f533e84a700 1 -- 172.21.15.28:0/1423186325 >> 172.21.15.28:0/1423186325 conn(0x7f5338077370 msgr2=0x7f53380fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.992+0000 7f533e84a700 1 -- 172.21.15.28:0/1423186325 shutdown_connections 2024-03-30T15:07:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.992+0000 7f533e84a700 1 -- 172.21.15.28:0/1423186325 wait complete. 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 librados: starting msgr at 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 librados: starting objecter 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 Processor -- start 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 -- start start 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 librados: setting wanted keys 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 librados: calling monclient init 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.993+0000 7f533e84a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53381a9340 con 0x7f533810ae80 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53381aa560 con 0x7f533810eb50 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53381ab7a0 con 0x7f5338107000 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533cde7700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f533810eb50 0x7f53381a7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:55530/0 (socket says 172.21.15.28:55530) 2024-03-30T15:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533cde7700 1 -- 172.21.15.28:0/384533200 learned_addr learned my addr 172.21.15.28:0/384533200 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter ms_handle_connect 0x7f533810eb50 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter ms_handle_connect 0x7f533810ae80 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter ms_handle_connect 0x7f5338107000 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753462566 0 0) 0x7f53381aa560 con 0x7f533810eb50 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53200031f0 con 0x7f533810eb50 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012010745 0 0) 0x7f53381a9340 con 0x7f533810ae80 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53381aa560 con 0x7f533810ae80 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3066059751 0 0) 0x7f53381ab7a0 con 0x7f5338107000 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53381a9340 con 0x7f5338107000 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2020642798 0 0) 0x7f53200031f0 con 0x7f533810eb50 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53381ab7a0 con 0x7f533810eb50 2024-03-30T15:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5330001cc0 con 0x7f533810eb50 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3716298800 0 0) 0x7f53381ab7a0 con 0x7f533810eb50 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6790/0 conn(0x7f5338107000 legacy=0x7f5338106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6789/0 conn(0x7f533810ae80 legacy=0x7f53381a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53381ac9e0 con 0x7f533810eb50 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53381a9570 con 0x7f533810eb50 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 10 client.50524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53381a9a30 con 0x7f533810eb50 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.994+0000 7f533e84a700 1 librados: init done 2024-03-30T15:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.995+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5330003ec0 con 0x7f533810eb50 2024-03-30T15:07:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.995+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53300060e0 con 0x7f533810eb50 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.995+0000 7f533e84a700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5338103740 con 0x7f533810eb50 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.996+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5330003a20 con 0x7f533810eb50 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.996+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(629..629 src has 442..629) ==== 9983+0+0 (unknown 1671313260 0 0) 0x7f533008a4e0 con 0x7f533810eb50 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 osd_map(629..629 src has 442..629) 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 3 client.50524.objecter handle_osd_map got epochs [629,629] > 0 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 3 client.50524.objecter handle_osd_map decoding full epoch 629 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 20 client.50524.objecter dump_active .. 0 homeless 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v629) ==== 54+0+16 (unknown 2305108501 0 2420000192) 0x7f533008c890 con 0x7f533810eb50 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f5320070080 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 10 client.50524.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 1 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 1 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 0 1 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 4 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 10 client.50524.objecter _send_command 1 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5318006d40 con 0x7f5318002fd0 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.997+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f5318002fd0 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.998+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318006d40 con 0x7f5318002fd0 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.998+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 1: 0 ) 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.998+0000 7f53357fa700 10 client.50524.objecter _finish_command 1 = system:0 2024-03-30T15:07:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.998+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 4 2024-03-30T15:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:58.998+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 0 1 2024-03-30T15:07:59.012 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 10 client.50524.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 2 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 2 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180028e0 osd=0 3 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 0 2 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 4 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 10 client.50524.objecter _send_command 2 2024-03-30T15:07:59.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.016+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318001240 con 0x7f5318002fd0 2024-03-30T15:07:59.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.018+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318001240 con 0x7f5318002fd0 2024-03-30T15:07:59.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.018+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.018+0000 7f53357fa700 10 client.50524.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.018+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 4 2024-03-30T15:07:59.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.018+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 0 2 2024-03-30T15:07:59.020 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 10 client.50524.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 1 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 4 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 10 client.50524.objecter _send_command 3 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5318008cc0 con 0x7f5318008100 2024-03-30T15:07:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.019+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f5318008100 2024-03-30T15:07:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.020+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318008cc0 con 0x7f5318008100 2024-03-30T15:07:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.020+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 3: 0 ) 2024-03-30T15:07:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.020+0000 7f53357fa700 10 client.50524.objecter _finish_command 3 = system:0 2024-03-30T15:07:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.020+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 4 2024-03-30T15:07:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.020+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 1 3 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.043+0000 7f531effd700 10 client.50524.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.043+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 4 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.043+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 4 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318001f50 osd=1 3 2024-03-30T15:07:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 1 4 2024-03-30T15:07:59.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 4 2024-03-30T15:07:59.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 10 client.50524.objecter _send_command 4 2024-03-30T15:07:59.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.044+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53180096e0 con 0x7f5318008100 2024-03-30T15:07:59.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.046+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f53180096e0 con 0x7f5318008100 2024-03-30T15:07:59.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.046+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.046+0000 7f53357fa700 10 client.50524.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.046+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 4 2024-03-30T15:07:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.046+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 1 4 2024-03-30T15:07:59.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:07:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 10 client.50524.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 5 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 5 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 2 5 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 4 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 10 client.50524.objecter _send_command 5 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5318010b20 con 0x7f531800ce70 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.047+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f531800ce70 2024-03-30T15:07:59.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.048+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318010b20 con 0x7f531800ce70 2024-03-30T15:07:59.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.048+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 5: 0 ) 2024-03-30T15:07:59.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.048+0000 7f53357fa700 10 client.50524.objecter _finish_command 5 = system:0 2024-03-30T15:07:59.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.048+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 4 2024-03-30T15:07:59.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.048+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 2 5 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 10 client.50524.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 6 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 6 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318001cc0 osd=2 3 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 2 6 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 4 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 10 client.50524.objecter _send_command 6 2024-03-30T15:07:59.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.066+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318009210 con 0x7f531800ce70 2024-03-30T15:07:59.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318009210 con 0x7f531800ce70 2024-03-30T15:07:59.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 10 client.50524.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 4 2024-03-30T15:07:59.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 2 6 2024-03-30T15:07:59.070 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:07:59.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 10 client.50524.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 7 2024-03-30T15:07:59.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 7 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 3 7 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 4 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 10 client.50524.objecter _send_command 7 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5318007f50 con 0x7f5318010d00 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.068+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f5318010d00 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.069+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318007f50 con 0x7f5318010d00 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.069+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 7: 0 ) 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.069+0000 7f53357fa700 10 client.50524.objecter _finish_command 7 = system:0 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.069+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 4 2024-03-30T15:07:59.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.069+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 3 7 2024-03-30T15:07:59.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 10 client.50524.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 8 2024-03-30T15:07:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 8 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318009420 osd=3 3 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 3 8 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 4 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 10 client.50524.objecter _send_command 8 2024-03-30T15:07:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.086+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318012130 con 0x7f5318010d00 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318012130 con 0x7f5318010d00 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 10 client.50524.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 4 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 3 8 2024-03-30T15:07:59.089 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 10 client.50524.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 9 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 9 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 4 9 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 4 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 10 client.50524.objecter _send_command 9 2024-03-30T15:07:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f531801a320 con 0x7f53180074e0 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.088+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f53180074e0 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.089+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f531801a320 con 0x7f53180074e0 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.089+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 9: 0 ) 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.089+0000 7f53357fa700 10 client.50524.objecter _finish_command 9 = system:0 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.089+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 4 2024-03-30T15:07:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.089+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 4 9 2024-03-30T15:07:59.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 10 client.50524.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 10 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 10 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180132b0 osd=4 3 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 4 10 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 4 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 10 client.50524.objecter _send_command 10 2024-03-30T15:07:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.106+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318007290 con 0x7f53180074e0 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318007290 con 0x7f53180074e0 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 10 client.50524.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 4 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 4 10 2024-03-30T15:07:59.109 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 10 client.50524.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 11 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 11 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 5 11 2024-03-30T15:07:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 4 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 10 client.50524.objecter _send_command 11 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f531801ba40 con 0x7f531801c6c0 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.108+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f531801c6c0 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.109+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f531801ba40 con 0x7f531801c6c0 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.109+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 11: 0 ) 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.109+0000 7f53357fa700 10 client.50524.objecter _finish_command 11 = system:0 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.109+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 4 2024-03-30T15:07:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.109+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 5 11 2024-03-30T15:07:59.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 10 client.50524.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 12 2024-03-30T15:07:59.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 12 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f53180072d0 osd=5 3 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 5 12 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 4 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 10 client.50524.objecter _send_command 12 2024-03-30T15:07:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.126+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f531801ba40 con 0x7f531801c6c0 2024-03-30T15:07:59.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f531801ba40 con 0x7f531801c6c0 2024-03-30T15:07:59.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f53357fa700 10 client.50524.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 4 2024-03-30T15:07:59.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 5 12 2024-03-30T15:07:59.130 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 10 client.50524.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 13 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 13 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 6 13 2024-03-30T15:07:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 4 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 10 client.50524.objecter _send_command 13 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.128+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5318022ec0 con 0x7f531801f3b0 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f531801f3b0 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318022ec0 con 0x7f531801f3b0 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 13: 0 ) 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 10 client.50524.objecter _finish_command 13 = system:0 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 4 2024-03-30T15:07:59.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.129+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 6 13 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 10 client.50524.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 14 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 14 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f531801c210 osd=6 3 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 6 14 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 4 2024-03-30T15:07:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 10 client.50524.objecter _send_command 14 2024-03-30T15:07:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.147+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318022ec0 con 0x7f531801f3b0 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318022ec0 con 0x7f531801f3b0 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 10 client.50524.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 4 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 6 14 2024-03-30T15:07:59.150 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 10 client.50524.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 15 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 15 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 7 15 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 4 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 10 client.50524.objecter _send_command 15 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5318027270 con 0x7f53180235e0 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.149+0000 7f53357fa700 10 client.50524.objecter ms_handle_connect 0x7f53180235e0 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.150+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5318027270 con 0x7f53180235e0 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.150+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 15: 0 ) 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.150+0000 7f53357fa700 10 client.50524.objecter _finish_command 15 = system:0 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.150+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 4 2024-03-30T15:07:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.150+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 7 15 2024-03-30T15:07:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 10 client.50524.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:07:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign -1 16 2024-03-30T15:07:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 20 client.50524.objecter _get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 15 client.50524.objecter _session_command_op_remove -1 16 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 20 client.50524.objecter get_session s=0x7f5318022ec0 osd=7 3 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 15 client.50524.objecter _session_command_op_assign 7 16 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 4 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 10 client.50524.objecter _send_command 16 2024-03-30T15:07:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.167+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5318027270 con 0x7f53180235e0 2024-03-30T15:07:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f53357fa700 1 -- 172.21.15.28:0/384533200 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5318027270 con 0x7f53180235e0 2024-03-30T15:07:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f53357fa700 10 client.50524.objecter ms_dispatch 0x7f5338001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:07:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f53357fa700 10 client.50524.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:07:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f53357fa700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 4 2024-03-30T15:07:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f53357fa700 15 client.50524.objecter _session_command_op_remove 7 16 2024-03-30T15:07:59.172 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:07:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f531effd700 10 librados: watch_flush enter 2024-03-30T15:07:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f531effd700 10 librados: watch_flush exit 2024-03-30T15:07:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f531effd700 10 client.50524.objecter close_session for osd.0 2024-03-30T15:07:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.169+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6801/3394323575 conn(0x7f5318002fd0 legacy=0x7f5318005490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180028e0 osd=0 1 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.1 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6805/1850313547 conn(0x7f5318008100 legacy=0x7f53180085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001f50 osd=1 1 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.2 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6809/3451325043 conn(0x7f531800ce70 legacy=0x7f531800f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318001cc0 osd=2 1 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.3 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6813/1506157250 conn(0x7f5318010d00 legacy=0x7f5318011160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318009420 osd=3 1 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.4 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.134:6800/3063775514 conn(0x7f53180074e0 legacy=0x7f5318018c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180132b0 osd=4 1 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.5 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.134:6804/3668151179 conn(0x7f531801c6c0 legacy=0x7f531801a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f533cde7700 1 -- 172.21.15.28:0/384533200 reap_dead start 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f53180072d0 osd=5 1 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.6 2024-03-30T15:07:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.134:6808/2135224194 conn(0x7f531801f3b0 legacy=0x7f53180217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f533cde7700 1 -- 172.21.15.28:0/384533200 reap_dead start 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f531801c210 osd=6 1 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter close_session for osd.7 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.134:6812/3758190298 conn(0x7f53180235e0 legacy=0x7f53180259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f533cde7700 1 -- 172.21.15.28:0/384533200 reap_dead start 2024-03-30T15:07:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f533cde7700 1 -- 172.21.15.28:0/384533200 reap_dead start 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter put_session s=0x7f5318022ec0 osd=7 1 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 20 client.50524.objecter shutdown clearing up homeless session... 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 10 client.50524.objecter successfully canceled tick 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5320070080 legacy=0x7f5320072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> v1:172.21.15.134:6789/0 conn(0x7f533810eb50 legacy=0x7f53381a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:07:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 shutdown_connections 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 >> 172.21.15.28:0/384533200 conn(0x7f5338077370 msgr2=0x7f533810da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 shutdown_connections 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 -- 172.21.15.28:0/384533200 wait complete. 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:07:59.170+0000 7f531effd700 1 librados: shutdown 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:07:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:07:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:07:59.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:07:59.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:07:59.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:59.347 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:07:59.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:07:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:59.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:07:59.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:07:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:07:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 42.a scrub starts 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 42.a scrub ok 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T15:07:59.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:07:59.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:07:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/384533200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:59.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 42.a scrub starts 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 42.a scrub ok 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:59.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/384533200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 42.a scrub starts 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 42.a scrub ok 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:07:59.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:07:59.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: osdmap e629: 8 total, 8 up, 8 in 2024-03-30T15:07:59.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:07:59.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:07:59.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:07:59.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:07:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/384533200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:07:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:07:59.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:07:59.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:07:59.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:07:59.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:07:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:07:59.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:00.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:00.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:00.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:00.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:00.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:00.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 42.5 scrub starts 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 42.5 scrub ok 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: pgmap v928: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:08:00.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[41614]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 42.5 scrub starts 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 42.5 scrub ok 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: pgmap v928: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:00.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:00.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:08:00.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:00 smithi028 ceph-mon[29876]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T15:08:00.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:00.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:00.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:00.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:00.662 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:00.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 42.5 scrub starts 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 42.5 scrub ok 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:08:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: osdmap e630: 8 total, 8 up, 8 in 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: pgmap v928: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:08:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:00 smithi134 ceph-mon[33714]: osdmap e631: 8 total, 8 up, 8 in 2024-03-30T15:08:00.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:00.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:00.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:08:00.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:00.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:00.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:01.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:01.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:01.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:01.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:01.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:01.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:01.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:01.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:01.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:08:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:08:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 42.17 scrub starts 2024-03-30T15:08:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 42.17 scrub ok 2024-03-30T15:08:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 42.15 scrub starts 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: 42.15 scrub ok 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T15:08:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:01 smithi134 ceph-mon[33714]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T15:08:01.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:01.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 42.17 scrub starts 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 42.17 scrub ok 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 42.15 scrub starts 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: 42.15 scrub ok 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T15:08:01.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[41614]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 42.17 scrub starts 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 42.17 scrub ok 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:08:01.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 42.15 scrub starts 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: 42.15 scrub ok 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-30T15:08:01.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:01 smithi028 ceph-mon[29876]: osdmap e632: 8 total, 8 up, 8 in 2024-03-30T15:08:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:01.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:01.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:01.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:01.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:01.979 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:08:01.979 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:08:01.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:01.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:02.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:02.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:08:02.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:02.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:02.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:02.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:02.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:02.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:02.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: pgmap v931: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:02.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T15:08:02.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: pgmap v931: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:02.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: pgmap v931: 254 pgs: 254 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:02.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:02.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: osdmap e633: 8 total, 8 up, 8 in 2024-03-30T15:08:02.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]: dispatch 2024-03-30T15:08:02.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:02.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:02.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:02.892 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:02.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:02.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:08:02.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:03.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:03.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:03.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:03.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:03.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:03.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 4.3 scrub starts 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 4.3 scrub ok 2024-03-30T15:08:03.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 42.f scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: 42.f scrub ok 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[29876]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 4.3 scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 4.3 scrub ok 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:03.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:03.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 42.f scrub starts 2024-03-30T15:08:03.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: 42.f scrub ok 2024-03-30T15:08:03.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:08:03.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:03 smithi028 ceph-mon[41614]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T15:08:03.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 4.3 scrub starts 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 4.3 scrub ok 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 42.f scrub starts 2024-03-30T15:08:03.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: 42.f scrub ok 2024-03-30T15:08:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-109", "tierpool": "test-rados-api-smithi028-93253-109-cache"}]': finished 2024-03-30T15:08:03.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:03 smithi134 ceph-mon[33714]: osdmap e634: 8 total, 8 up, 8 in 2024-03-30T15:08:03.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:03.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:03.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:03.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:03.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:03.915 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:03.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:04.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 -- 172.21.15.28:0/2296277441 >> v1:172.21.15.28:6790/0 conn(0x7fa1d810afa0 legacy=0x7fa1d810d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 -- 172.21.15.28:0/2296277441 shutdown_connections 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 -- 172.21.15.28:0/2296277441 >> 172.21.15.28:0/2296277441 conn(0x7fa1d80fc9e0 msgr2=0x7fa1d80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 -- 172.21.15.28:0/2296277441 shutdown_connections 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 -- 172.21.15.28:0/2296277441 wait complete. 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 librados: starting msgr at 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.194+0000 7fa1e04e4700 1 librados: starting objecter 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 Processor -- start 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 -- start start 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 librados: setting wanted keys 2024-03-30T15:08:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 librados: calling monclient init 2024-03-30T15:08:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1d81029c0 con 0x7fa1d8103190 2024-03-30T15:08:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1d8102bd0 con 0x7fa1d810afa0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1e04e4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1d81aba30 con 0x7fa1d810eab0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1dea81700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa1d810eab0 0x7fa1d81a82f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50562/0 (socket says 172.21.15.28:50562) 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1dea81700 1 -- 172.21.15.28:0/527815034 learned_addr learned my addr 172.21.15.28:0/527815034 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1de280700 1 --1- 172.21.15.28:0/527815034 >> v1:172.21.15.28:6789/0 conn(0x7fa1d8103190 0x7fa1d8100df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34584/0 (socket says 172.21.15.28:34584) 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 10 client.?.objecter ms_handle_connect 0x7fa1d8103190 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 10 client.?.objecter ms_handle_connect 0x7fa1d810eab0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656007056 0 0) 0x7fa1d81aba30 con 0x7fa1d810eab0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1c40031f0 con 0x7fa1d810eab0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143903048 0 0) 0x7fa1d81029c0 con 0x7fa1d8103190 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.195+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1d81aba30 con 0x7fa1d8103190 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2903398497 0 0) 0x7fa1c40031f0 con 0x7fa1d810eab0 2024-03-30T15:08:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1d81029c0 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa1d0001c30 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 101303911 0 0) 0x7fa1d81aba30 con 0x7fa1d8103190 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1c40031f0 con 0x7fa1d8103190 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa1d40018b0 con 0x7fa1d8103190 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1970685492 0 0) 0x7fa1d81029c0 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 >> v1:172.21.15.134:6789/0 conn(0x7fa1d810afa0 legacy=0x7fa1d8101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 >> v1:172.21.15.28:6789/0 conn(0x7fa1d8103190 legacy=0x7fa1d8100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d81acc10 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d81abc60 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1e04e4700 10 client.58523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1d81ac120 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1e04e4700 1 librados: init done 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa1d0003ec0 con 0x7fa1d810eab0 2024-03-30T15:08:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.196+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa1d00060d0 con 0x7fa1d810eab0 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.197+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1bc005120 con 0x7fa1d810eab0 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa1d001c9d0 con 0x7fa1d810eab0 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(634..634 src has 442..634) ==== 9975+0+0 (unknown 3460037226 0 0) 0x7fa1d008a290 con 0x7fa1d810eab0 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 10 client.58523.objecter ms_dispatch 0x7fa1d8001550 osd_map(634..634 src has 442..634) 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 3 client.58523.objecter handle_osd_map got epochs [634,634] > 0 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 3 client.58523.objecter handle_osd_map decoding full epoch 634 2024-03-30T15:08:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 20 client.58523.objecter dump_active .. 0 homeless 2024-03-30T15:08:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.198+0000 7fa1cf7fe700 10 client.58523.objecter ms_handle_connect 0x7fa1c4070250 2024-03-30T15:08:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.207+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa1d0056d90 con 0x7fa1d810eab0 2024-03-30T15:08:04.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:04.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:04.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.417+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1bc000b90 con 0x7fa1c4070250 2024-03-30T15:08:04.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.434+0000 7fa1cf7fe700 1 -- 172.21.15.28:0/527815034 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+575060 (unknown 2965378022 0 314786823) 0x7fa1bc000b90 con 0x7fa1c4070250 2024-03-30T15:08:04.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:04 smithi028 ceph-mon[41614]: pgmap v934: 254 pgs: 254 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:08:04.436 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:04 smithi028 ceph-mon[41614]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T15:08:04.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:04 smithi028 ceph-mon[29876]: pgmap v934: 254 pgs: 254 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:08:04.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:04 smithi028 ceph-mon[29876]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T15:08:04.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 10 librados: watch_flush enter 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 10 librados: watch_flush exit 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 20 client.58523.objecter shutdown clearing up homeless session... 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 10 client.58523.objecter successfully canceled tick 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa1c4070250 legacy=0x7fa1c4072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:04.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 >> v1:172.21.15.28:6790/0 conn(0x7fa1d810eab0 legacy=0x7fa1d81a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 shutdown_connections 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 >> 172.21.15.28:0/527815034 conn(0x7fa1d80fc9e0 msgr2=0x7fa1d8110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 shutdown_connections 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 -- 172.21.15.28:0/527815034 wait complete. 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:04.440+0000 7fa1e04e4700 1 librados: shutdown 2024-03-30T15:08:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:08:04.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:04.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:04.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:04.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:04 smithi134 ceph-mon[33714]: pgmap v934: 254 pgs: 254 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:08:04.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:04 smithi134 ceph-mon[33714]: osdmap e635: 8 total, 8 up, 8 in 2024-03-30T15:08:04.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:04.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:04.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:04.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:05.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:05.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:05.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:05.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:05.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:05.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:05.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:05.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:05.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:05.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: from='client.58523 v1:172.21.15.28:0/527815034' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:05.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: from='client.58523 v1:172.21.15.28:0/527815034' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:05.617 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:08:05.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:08:05.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:08:05.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:08:05.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T15:08:05.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:05.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:05.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: from='client.58523 v1:172.21.15.28:0/527815034' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:08:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:08:05.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:08:05.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: osdmap e636: 8 total, 8 up, 8 in 2024-03-30T15:08:05.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:05.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.789 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:05.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:05.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:05.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:05.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:06.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:06.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:06.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:06.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:06.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:06.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:06.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:06.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:08:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:08:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: pgmap v937: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T15:08:06.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: pgmap v937: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T15:08:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: pgmap v937: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: osdmap e637: 8 total, 8 up, 8 in 2024-03-30T15:08:06.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]: dispatch 2024-03-30T15:08:06.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:06.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:06.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:06.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:06.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:07.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:07.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:07.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:07.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:07.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:07.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:07.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:07.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:07.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:07.626 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[41614]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T15:08:07.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:07.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:07 smithi028 ceph-mon[29876]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T15:08:07.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:08:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1487369706' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-109"}]': finished 2024-03-30T15:08:07.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:07 smithi134 ceph-mon[33714]: osdmap e638: 8 total, 8 up, 8 in 2024-03-30T15:08:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:07.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:07.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.819+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3183657527 >> v1:172.21.15.28:6790/0 conn(0x7f01b810ead0 legacy=0x7f01b8110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.819+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3183657527 shutdown_connections 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.819+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3183657527 >> 172.21.15.28:0/3183657527 conn(0x7f01b80fdb50 msgr2=0x7f01b80fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3183657527 shutdown_connections 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3183657527 wait complete. 2024-03-30T15:08:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 librados: starting msgr at 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 librados: starting objecter 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 Processor -- start 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- start start 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 librados: setting wanted keys 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 librados: calling monclient init 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01b8107990 con 0x7f01b810ead0 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01b8107ba0 con 0x7f01b81022a0 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bfe9d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01b81a96f0 con 0x7f01b810af30 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bd438700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f01b810af30 0x7f01b8104ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50570/0 (socket says 172.21.15.28:50570) 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.820+0000 7f01bd438700 1 -- 172.21.15.28:0/3060285336 learned_addr learned my addr 172.21.15.28:0/3060285336 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter ms_handle_connect 0x7f01b810ead0 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter ms_handle_connect 0x7f01b810af30 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter ms_handle_connect 0x7f01b81022a0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1053789459 0 0) 0x7f01b8107990 con 0x7f01b810ead0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a40031f0 con 0x7f01b810ead0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567301573 0 0) 0x7f01b81a96f0 con 0x7f01b810af30 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b8107990 con 0x7f01b810af30 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747288276 0 0) 0x7f01b8107ba0 con 0x7f01b81022a0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b81a96f0 con 0x7f01b81022a0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 620590577 0 0) 0x7f01a40031f0 con 0x7f01b810ead0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01b8107ba0 con 0x7f01b810ead0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01b00047f0 con 0x7f01b810ead0 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 752742957 0 0) 0x7f01b8107990 con 0x7f01b810af30 2024-03-30T15:08:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01a40031f0 con 0x7f01b810af30 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01b4001230 con 0x7f01b810af30 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 423173249 0 0) 0x7f01b81a96f0 con 0x7f01b81022a0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01b8107990 con 0x7f01b81022a0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01ac001830 con 0x7f01b81022a0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1981466158 0 0) 0x7f01b8107ba0 con 0x7f01b810ead0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 >> v1:172.21.15.28:6790/0 conn(0x7f01b810af30 legacy=0x7f01b8104ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 >> v1:172.21.15.134:6789/0 conn(0x7f01b81022a0 legacy=0x7f01b81047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b81aa930 con 0x7f01b810ead0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.821+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01b81a9920 con 0x7f01b810ead0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.822+0000 7f01bfe9d700 10 client.25329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.822+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f01b81a9de0 con 0x7f01b810ead0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.822+0000 7f01bfe9d700 1 librados: init done 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.822+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f01b0004a70 con 0x7f01b810ead0 2024-03-30T15:08:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.822+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01b0005000 con 0x7f01b810ead0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.823+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019c005120 con 0x7f01b810ead0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.824+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f01b0001e30 con 0x7f01b810ead0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(638..638 src has 442..638) ==== 8737+0+0 (unknown 1314851735 0 0) 0x7f01b0088b60 con 0x7f01b810ead0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 10 client.25329.objecter ms_dispatch 0x7f01b8001550 osd_map(638..638 src has 442..638) 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 3 client.25329.objecter handle_osd_map got epochs [638,638] > 0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 3 client.25329.objecter handle_osd_map decoding full epoch 638 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 20 client.25329.objecter dump_active .. 0 homeless 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.825+0000 7f01aaffd700 10 client.25329.objecter ms_handle_connect 0x7f01a40747d0 2024-03-30T15:08:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:07.829+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f01b0055b10 con 0x7f01b810ead0 2024-03-30T15:08:07.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:07.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:08.024 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:08:08.024 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:08:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.048+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f019c005410 con 0x7f01b810ead0 2024-03-30T15:08:08.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:08.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:08.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:08.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:08.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.352+0000 7f01aaffd700 1 -- 172.21.15.28:0/3060285336 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v639) ==== 101+0+0 (unknown 1768253131 0 0) 0x7f01b0059560 con 0x7f01b810ead0 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.355+0000 7f01bfe9d700 10 librados: watch_flush enter 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 10 librados: watch_flush exit 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 20 client.25329.objecter shutdown clearing up homeless session... 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 10 client.25329.objecter successfully canceled tick 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 >> v1:172.21.15.28:6800/3011431093 conn(0x7f01a40747d0 legacy=0x7f01a4076c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 >> v1:172.21.15.28:6789/0 conn(0x7f01b810ead0 legacy=0x7f01b81a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 shutdown_connections 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 >> 172.21.15.28:0/3060285336 conn(0x7f01b80fdb50 msgr2=0x7f01b80fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 shutdown_connections 2024-03-30T15:08:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 -- 172.21.15.28:0/3060285336 wait complete. 2024-03-30T15:08:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:08.356+0000 7f01bfe9d700 1 librados: shutdown 2024-03-30T15:08:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:08:08.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:08.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:08.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:08.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:08:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:08:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: pgmap v940: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T15:08:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:08.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:08.720 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:08.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:08:08.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:08:08.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: pgmap v940: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:08:08.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: pgmap v940: 214 pgs: 214 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3060285336' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: osdmap e639: 8 total, 8 up, 8 in 2024-03-30T15:08:08.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:08.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:08.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:08:08.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:08.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:08.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:08.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:09.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:09.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:09.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:09.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:09.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:09.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:09.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:09.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:08:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:08:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:08:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:08:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 42.3 scrub starts 2024-03-30T15:08:09.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: 42.3 scrub ok 2024-03-30T15:08:09.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:09.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T15:08:09.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 42.3 scrub starts 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: 42.3 scrub ok 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:09.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 42.3 scrub starts 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: 42.3 scrub ok 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: osdmap e640: 8 total, 8 up, 8 in 2024-03-30T15:08:09.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]: dispatch 2024-03-30T15:08:09.717 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":935,"stamp":"2024-03-30T15:08:04.348285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596331,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":1956,"num_read_kb":1704,"num_write":731,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":666,"acting":666,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":762,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":627452,"kb_used_data":36496,"kb_used_omap":12,"kb_used_meta":590835,"kb_avail":749333764,"statfs":{"total":767960285184,"available":767317774336,"internally_reserved":0,"allocated":37371904,"data_stored":17050142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":605015600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.967955"},"pg_stats":[{"pgid":"320.7","version":"0'0","reported_seq":35,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489199+0000","last_change":"2024-03-30T15:07:53.359680+0000","last_active":"2024-03-30T15:08:03.489199+0000","last_peered":"2024-03-30T15:08:03.489199+0000","last_clean":"2024-03-30T15:08:03.489199+0000","last_became_active":"2024-03-30T15:07:53.359353+0000","last_became_peered":"2024-03-30T15:07:53.359353+0000","last_unstale":"2024-03-30T15:08:03.489199+0000","last_undegraded":"2024-03-30T15:08:03.489199+0000","last_fullsized":"2024-03-30T15:08:03.489199+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.930121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"320.6","version":"632'1","reported_seq":40,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489259+0000","last_change":"2024-03-30T15:07:53.359567+0000","last_active":"2024-03-30T15:08:03.489259+0000","last_peered":"2024-03-30T15:08:03.489259+0000","last_clean":"2024-03-30T15:08:03.489259+0000","last_became_active":"2024-03-30T15:07:53.359225+0000","last_became_peered":"2024-03-30T15:07:53.359225+0000","last_unstale":"2024-03-30T15:08:03.489259+0000","last_undegraded":"2024-03-30T15:08:03.489259+0000","last_fullsized":"2024-03-30T15:08:03.489259+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.102446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"320.5","version":"0'0","reported_seq":35,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514789+0000","last_change":"2024-03-30T15:07:53.361569+0000","last_active":"2024-03-30T15:08:03.514789+0000","last_peered":"2024-03-30T15:08:03.514789+0000","last_clean":"2024-03-30T15:08:03.514789+0000","last_became_active":"2024-03-30T15:07:53.361078+0000","last_became_peered":"2024-03-30T15:07:53.361078+0000","last_unstale":"2024-03-30T15:08:03.514789+0000","last_undegraded":"2024-03-30T15:08:03.514789+0000","last_fullsized":"2024-03-30T15:08:03.514789+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:56.423646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"320.4","version":"0'0","reported_seq":35,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488687+0000","last_change":"2024-03-30T15:07:53.357804+0000","last_active":"2024-03-30T15:08:03.488687+0000","last_peered":"2024-03-30T15:08:03.488687+0000","last_clean":"2024-03-30T15:08:03.488687+0000","last_became_active":"2024-03-30T15:07:53.357454+0000","last_became_peered":"2024-03-30T15:07:53.357454+0000","last_unstale":"2024-03-30T15:08:03.488687+0000","last_undegraded":"2024-03-30T15:08:03.488687+0000","last_fullsized":"2024-03-30T15:08:03.488687+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:08.519879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"320.3","version":"0'0","reported_seq":32,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707329+0000","last_change":"2024-03-30T15:07:53.359808+0000","last_active":"2024-03-30T15:08:02.707329+0000","last_peered":"2024-03-30T15:08:02.707329+0000","last_clean":"2024-03-30T15:08:02.707329+0000","last_became_active":"2024-03-30T15:07:53.357244+0000","last_became_peered":"2024-03-30T15:07:53.357244+0000","last_unstale":"2024-03-30T15:08:02.707329+0000","last_undegraded":"2024-03-30T15:08:02.707329+0000","last_fullsized":"2024-03-30T15:08:02.707329+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.434214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"320.2","version":"0'0","reported_seq":32,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707134+0000","last_change":"2024-03-30T15:07:53.363013+0000","last_active":"2024-03-30T15:08:02.707134+0000","last_peered":"2024-03-30T15:08:02.707134+0000","last_clean":"2024-03-30T15:08:02.707134+0000","last_became_active":"2024-03-30T15:07:53.358658+0000","last_became_peered":"2024-03-30T15:07:53.358658+0000","last_unstale":"2024-03-30T15:08:02.707134+0000","last_undegraded":"2024-03-30T15:08:02.707134+0000","last_fullsized":"2024-03-30T15:08:02.707134+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.833975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"320.1","version":"0'0","reported_seq":32,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347114+0000","last_change":"2024-03-30T15:07:53.357768+0000","last_active":"2024-03-30T15:08:02.347114+0000","last_peered":"2024-03-30T15:08:02.347114+0000","last_clean":"2024-03-30T15:08:02.347114+0000","last_became_active":"2024-03-30T15:07:53.357462+0000","last_became_peered":"2024-03-30T15:07:53.357462+0000","last_unstale":"2024-03-30T15:08:02.347114+0000","last_undegraded":"2024-03-30T15:08:02.347114+0000","last_fullsized":"2024-03-30T15:08:02.347114+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.959800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"320.0","version":"0'0","reported_seq":32,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.706830+0000","last_change":"2024-03-30T15:07:53.355216+0000","last_active":"2024-03-30T15:08:02.706830+0000","last_peered":"2024-03-30T15:08:02.706830+0000","last_clean":"2024-03-30T15:08:02.706830+0000","last_became_active":"2024-03-30T15:07:53.354897+0000","last_became_peered":"2024-03-30T15:07:53.354897+0000","last_unstale":"2024-03-30T15:08:02.706830+0000","last_undegraded":"2024-03-30T15:08:02.706830+0000","last_fullsized":"2024-03-30T15:08:02.706830+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":623,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:52.339943+0000","last_clean_scrub_stamp":"2024-03-30T15:07:52.339943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:12.409170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"317.7","version":"0'0","reported_seq":60,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.350249+0000","last_change":"2024-03-30T15:07:39.294561+0000","last_active":"2024-03-30T15:08:03.350249+0000","last_peered":"2024-03-30T15:08:03.350249+0000","last_clean":"2024-03-30T15:08:03.350249+0000","last_became_active":"2024-03-30T15:07:39.294234+0000","last_became_peered":"2024-03-30T15:07:39.294234+0000","last_unstale":"2024-03-30T15:08:03.350249+0000","last_undegraded":"2024-03-30T15:08:03.350249+0000","last_fullsized":"2024-03-30T15:08:03.350249+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:50.212078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"317.6","version":"0'0","reported_seq":59,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349439+0000","last_change":"2024-03-30T15:07:39.294234+0000","last_active":"2024-03-30T15:08:02.349439+0000","last_peered":"2024-03-30T15:08:02.349439+0000","last_clean":"2024-03-30T15:08:02.349439+0000","last_became_active":"2024-03-30T15:07:39.293872+0000","last_became_peered":"2024-03-30T15:07:39.293872+0000","last_unstale":"2024-03-30T15:08:02.349439+0000","last_undegraded":"2024-03-30T15:08:02.349439+0000","last_fullsized":"2024-03-30T15:08:02.349439+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.880051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"317.5","version":"0'0","reported_seq":58,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707922+0000","last_change":"2024-03-30T15:07:39.295017+0000","last_active":"2024-03-30T15:08:02.707922+0000","last_peered":"2024-03-30T15:08:02.707922+0000","last_clean":"2024-03-30T15:08:02.707922+0000","last_became_active":"2024-03-30T15:07:39.294594+0000","last_became_peered":"2024-03-30T15:07:39.294594+0000","last_unstale":"2024-03-30T15:08:02.707922+0000","last_undegraded":"2024-03-30T15:08:02.707922+0000","last_fullsized":"2024-03-30T15:08:02.707922+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:49.310383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"317.4","version":"0'0","reported_seq":58,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707555+0000","last_change":"2024-03-30T15:07:39.271516+0000","last_active":"2024-03-30T15:08:02.707555+0000","last_peered":"2024-03-30T15:08:02.707555+0000","last_clean":"2024-03-30T15:08:02.707555+0000","last_became_active":"2024-03-30T15:07:39.271161+0000","last_became_peered":"2024-03-30T15:07:39.271161+0000","last_unstale":"2024-03-30T15:08:02.707555+0000","last_undegraded":"2024-03-30T15:08:02.707555+0000","last_fullsized":"2024-03-30T15:08:02.707555+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.933009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"317.3","version":"0'0","reported_seq":58,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.706676+0000","last_change":"2024-03-30T15:07:38.358876+0000","last_active":"2024-03-30T15:08:02.706676+0000","last_peered":"2024-03-30T15:08:02.706676+0000","last_clean":"2024-03-30T15:08:02.706676+0000","last_became_active":"2024-03-30T15:07:38.358566+0000","last_became_peered":"2024-03-30T15:07:38.358566+0000","last_unstale":"2024-03-30T15:08:02.706676+0000","last_undegraded":"2024-03-30T15:08:02.706676+0000","last_fullsized":"2024-03-30T15:08:02.706676+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.808766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"317.2","version":"0'0","reported_seq":60,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348419+0000","last_change":"2024-03-30T15:07:38.647032+0000","last_active":"2024-03-30T15:08:03.348419+0000","last_peered":"2024-03-30T15:08:03.348419+0000","last_clean":"2024-03-30T15:08:03.348419+0000","last_became_active":"2024-03-30T15:07:38.646504+0000","last_became_peered":"2024-03-30T15:07:38.646504+0000","last_unstale":"2024-03-30T15:08:03.348419+0000","last_undegraded":"2024-03-30T15:08:03.348419+0000","last_fullsized":"2024-03-30T15:08:03.348419+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:44.533687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"317.1","version":"0'0","reported_seq":57,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.515902+0000","last_change":"2024-03-30T15:07:38.358263+0000","last_active":"2024-03-30T15:08:02.515902+0000","last_peered":"2024-03-30T15:08:02.515902+0000","last_clean":"2024-03-30T15:08:02.515902+0000","last_became_active":"2024-03-30T15:07:38.357921+0000","last_became_peered":"2024-03-30T15:07:38.357921+0000","last_unstale":"2024-03-30T15:08:02.515902+0000","last_undegraded":"2024-03-30T15:08:02.515902+0000","last_fullsized":"2024-03-30T15:08:02.515902+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:44.645300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"317.0","version":"609'4","reported_seq":66,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348116+0000","last_change":"2024-03-30T15:07:38.359879+0000","last_active":"2024-03-30T15:08:03.348116+0000","last_peered":"2024-03-30T15:08:03.348116+0000","last_clean":"2024-03-30T15:08:03.348116+0000","last_became_active":"2024-03-30T15:07:38.359091+0000","last_became_peered":"2024-03-30T15:07:38.359091+0000","last_unstale":"2024-03-30T15:08:03.348116+0000","last_undegraded":"2024-03-30T15:08:03.348116+0000","last_fullsized":"2024-03-30T15:08:03.348116+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":608,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:37.341600+0000","last_clean_scrub_stamp":"2024-03-30T15:07:37.341600+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.527316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349091+0000","last_change":"2024-03-30T15:08:00.661723+0000","last_active":"2024-03-30T15:08:03.349091+0000","last_peered":"2024-03-30T15:08:03.349091+0000","last_clean":"2024-03-30T15:08:03.349091+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:08:03.349091+0000","last_undegraded":"2024-03-30T15:08:03.349091+0000","last_fullsized":"2024-03-30T15:08:03.349091+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.661646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.661646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:23.913327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":1077,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514514+0000","last_change":"2024-03-30T15:07:25.409282+0000","last_active":"2024-03-30T15:08:03.514514+0000","last_peered":"2024-03-30T15:08:03.514514+0000","last_clean":"2024-03-30T15:08:03.514514+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:08:03.514514+0000","last_undegraded":"2024-03-30T15:08:03.514514+0000","last_fullsized":"2024-03-30T15:08:03.514514+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.409160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.409160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.520458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":1068,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517472+0000","last_change":"2024-03-30T15:08:00.188376+0000","last_active":"2024-03-30T15:08:02.517472+0000","last_peered":"2024-03-30T15:08:02.517472+0000","last_clean":"2024-03-30T15:08:02.517472+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:08:02.517472+0000","last_undegraded":"2024-03-30T15:08:02.517472+0000","last_fullsized":"2024-03-30T15:08:02.517472+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.188301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.188301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.659221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349605+0000","last_change":"2024-03-30T15:06:56.895628+0000","last_active":"2024-03-30T15:08:02.349605+0000","last_peered":"2024-03-30T15:08:02.349605+0000","last_clean":"2024-03-30T15:08:02.349605+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:08:02.349605+0000","last_undegraded":"2024-03-30T15:08:02.349605+0000","last_fullsized":"2024-03-30T15:08:02.349605+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.895505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.895505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:02.675385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":1064,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488651+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:08:03.488651+0000","last_peered":"2024-03-30T15:08:03.488651+0000","last_clean":"2024-03-30T15:08:03.488651+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:08:03.488651+0000","last_undegraded":"2024-03-30T15:08:03.488651+0000","last_fullsized":"2024-03-30T15:08:03.488651+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":1086,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348253+0000","last_change":"2024-03-30T15:07:57.201822+0000","last_active":"2024-03-30T15:08:03.348253+0000","last_peered":"2024-03-30T15:08:03.348253+0000","last_clean":"2024-03-30T15:08:03.348253+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:08:03.348253+0000","last_undegraded":"2024-03-30T15:08:03.348253+0000","last_fullsized":"2024-03-30T15:08:03.348253+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.201691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.201691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:05.630818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":1060,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517439+0000","last_change":"2024-03-30T15:05:20.323134+0000","last_active":"2024-03-30T15:08:02.517439+0000","last_peered":"2024-03-30T15:08:02.517439+0000","last_clean":"2024-03-30T15:08:02.517439+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:08:02.517439+0000","last_undegraded":"2024-03-30T15:08:02.517439+0000","last_fullsized":"2024-03-30T15:08:02.517439+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.323015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.323015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348639+0000","last_change":"2024-03-30T15:06:59.257326+0000","last_active":"2024-03-30T15:08:03.348639+0000","last_peered":"2024-03-30T15:08:03.348639+0000","last_clean":"2024-03-30T15:08:03.348639+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:08:03.348639+0000","last_undegraded":"2024-03-30T15:08:03.348639+0000","last_fullsized":"2024-03-30T15:08:03.348639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.257230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.257230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:25.024266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":1067,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708963+0000","last_change":"2024-03-30T15:07:36.042362+0000","last_active":"2024-03-30T15:08:02.708963+0000","last_peered":"2024-03-30T15:08:02.708963+0000","last_clean":"2024-03-30T15:08:02.708963+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:08:02.708963+0000","last_undegraded":"2024-03-30T15:08:02.708963+0000","last_fullsized":"2024-03-30T15:08:02.708963+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.042281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.352269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348424+0000","last_change":"2024-03-30T15:06:18.054438+0000","last_active":"2024-03-30T15:08:02.348424+0000","last_peered":"2024-03-30T15:08:02.348424+0000","last_clean":"2024-03-30T15:08:02.348424+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:08:02.348424+0000","last_undegraded":"2024-03-30T15:08:02.348424+0000","last_fullsized":"2024-03-30T15:08:02.348424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:18.054358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:06:18.054358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":1060,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517433+0000","last_change":"2024-03-30T15:05:17.336662+0000","last_active":"2024-03-30T15:08:02.517433+0000","last_peered":"2024-03-30T15:08:02.517433+0000","last_clean":"2024-03-30T15:08:02.517433+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:08:02.517433+0000","last_undegraded":"2024-03-30T15:08:02.517433+0000","last_fullsized":"2024-03-30T15:08:02.517433+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.336577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.336577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":1084,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349633+0000","last_change":"2024-03-30T15:07:32.132841+0000","last_active":"2024-03-30T15:08:03.349633+0000","last_peered":"2024-03-30T15:08:03.349633+0000","last_clean":"2024-03-30T15:08:03.349633+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:08:03.349633+0000","last_undegraded":"2024-03-30T15:08:03.349633+0000","last_fullsized":"2024-03-30T15:08:03.349633+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.132758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.132758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.404108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":1060,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516473+0000","last_change":"2024-03-30T15:05:19.327818+0000","last_active":"2024-03-30T15:08:02.516473+0000","last_peered":"2024-03-30T15:08:02.516473+0000","last_clean":"2024-03-30T15:08:02.516473+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:08:02.516473+0000","last_undegraded":"2024-03-30T15:08:02.516473+0000","last_fullsized":"2024-03-30T15:08:02.516473+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.327720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.327720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.346971+0000","last_change":"2024-03-30T15:06:43.837698+0000","last_active":"2024-03-30T15:08:02.346971+0000","last_peered":"2024-03-30T15:08:02.346971+0000","last_clean":"2024-03-30T15:08:02.346971+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:08:02.346971+0000","last_undegraded":"2024-03-30T15:08:02.346971+0000","last_fullsized":"2024-03-30T15:08:02.346971+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.837619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.837619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":1083,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348831+0000","last_change":"2024-03-30T15:06:54.767293+0000","last_active":"2024-03-30T15:08:03.348831+0000","last_peered":"2024-03-30T15:08:03.348831+0000","last_clean":"2024-03-30T15:08:03.348831+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:08:03.348831+0000","last_undegraded":"2024-03-30T15:08:03.348831+0000","last_fullsized":"2024-03-30T15:08:03.348831+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.767213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.767213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.877720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":1083,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349275+0000","last_change":"2024-03-30T15:07:25.988108+0000","last_active":"2024-03-30T15:08:03.349275+0000","last_peered":"2024-03-30T15:08:03.349275+0000","last_clean":"2024-03-30T15:08:03.349275+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:08:03.349275+0000","last_undegraded":"2024-03-30T15:08:03.349275+0000","last_fullsized":"2024-03-30T15:08:03.349275+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.988025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.988025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.890511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":1068,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517231+0000","last_change":"2024-03-30T15:07:59.150468+0000","last_active":"2024-03-30T15:08:02.517231+0000","last_peered":"2024-03-30T15:08:02.517231+0000","last_clean":"2024-03-30T15:08:02.517231+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:08:02.517231+0000","last_undegraded":"2024-03-30T15:08:02.517231+0000","last_fullsized":"2024-03-30T15:08:02.517231+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:59.150361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.150361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.062144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":1067,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707941+0000","last_change":"2024-03-30T15:07:25.964793+0000","last_active":"2024-03-30T15:08:02.707941+0000","last_peered":"2024-03-30T15:08:02.707941+0000","last_clean":"2024-03-30T15:08:02.707941+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:08:02.707941+0000","last_undegraded":"2024-03-30T15:08:02.707941+0000","last_fullsized":"2024-03-30T15:08:02.707941+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.964714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.964714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:31.307648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349700+0000","last_change":"2024-03-30T15:06:58.308106+0000","last_active":"2024-03-30T15:08:03.349700+0000","last_peered":"2024-03-30T15:08:03.349700+0000","last_clean":"2024-03-30T15:08:03.349700+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:08:03.349700+0000","last_undegraded":"2024-03-30T15:08:03.349700+0000","last_fullsized":"2024-03-30T15:08:03.349700+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.308015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.308015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.528138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":1068,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517313+0000","last_change":"2024-03-30T15:07:35.172101+0000","last_active":"2024-03-30T15:08:02.517313+0000","last_peered":"2024-03-30T15:08:02.517313+0000","last_clean":"2024-03-30T15:08:02.517313+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:08:02.517313+0000","last_undegraded":"2024-03-30T15:08:02.517313+0000","last_fullsized":"2024-03-30T15:08:02.517313+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.171990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.171990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:02.621474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347243+0000","last_change":"2024-03-30T15:06:12.137521+0000","last_active":"2024-03-30T15:08:02.347243+0000","last_peered":"2024-03-30T15:08:02.347243+0000","last_clean":"2024-03-30T15:08:02.347243+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:08:02.347243+0000","last_undegraded":"2024-03-30T15:08:02.347243+0000","last_fullsized":"2024-03-30T15:08:02.347243+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.137432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516516+0000","last_change":"2024-03-30T15:06:58.979164+0000","last_active":"2024-03-30T15:08:02.516516+0000","last_peered":"2024-03-30T15:08:02.516516+0000","last_clean":"2024-03-30T15:08:02.516516+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:08:02.516516+0000","last_undegraded":"2024-03-30T15:08:02.516516+0000","last_fullsized":"2024-03-30T15:08:02.516516+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.979074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.979074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:22.628955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":1069,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514297+0000","last_change":"2024-03-30T15:05:19.111999+0000","last_active":"2024-03-30T15:08:03.514297+0000","last_peered":"2024-03-30T15:08:03.514297+0000","last_clean":"2024-03-30T15:08:03.514297+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:08:03.514297+0000","last_undegraded":"2024-03-30T15:08:03.514297+0000","last_fullsized":"2024-03-30T15:08:03.514297+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.111909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.111909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":1083,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348721+0000","last_change":"2024-03-30T15:06:45.791752+0000","last_active":"2024-03-30T15:08:03.348721+0000","last_peered":"2024-03-30T15:08:03.348721+0000","last_clean":"2024-03-30T15:08:03.348721+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:08:03.348721+0000","last_undegraded":"2024-03-30T15:08:03.348721+0000","last_fullsized":"2024-03-30T15:08:03.348721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.791667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.791667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:04.401445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":1069,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514866+0000","last_change":"2024-03-30T15:05:29.140639+0000","last_active":"2024-03-30T15:08:03.514866+0000","last_peered":"2024-03-30T15:08:03.514866+0000","last_clean":"2024-03-30T15:08:03.514866+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:08:03.514866+0000","last_undegraded":"2024-03-30T15:08:03.514866+0000","last_fullsized":"2024-03-30T15:08:03.514866+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.140508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.140508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":185,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349045+0000","last_change":"2024-03-30T15:06:18.488509+0000","last_active":"2024-03-30T15:08:03.349045+0000","last_peered":"2024-03-30T15:08:03.349045+0000","last_clean":"2024-03-30T15:08:03.349045+0000","last_became_active":"2024-03-30T15:06:18.488207+0000","last_became_peered":"2024-03-30T15:06:18.488207+0000","last_unstale":"2024-03-30T15:08:03.349045+0000","last_undegraded":"2024-03-30T15:08:03.349045+0000","last_fullsized":"2024-03-30T15:08:03.349045+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.176687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.176687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349917+0000","last_change":"2024-03-30T15:08:02.711973+0000","last_active":"2024-03-30T15:08:03.349917+0000","last_peered":"2024-03-30T15:08:03.349917+0000","last_clean":"2024-03-30T15:08:03.349917+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:08:03.349917+0000","last_undegraded":"2024-03-30T15:08:03.349917+0000","last_fullsized":"2024-03-30T15:08:03.349917+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.711881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.711881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.583184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349193+0000","last_change":"2024-03-30T15:07:26.321108+0000","last_active":"2024-03-30T15:08:03.349193+0000","last_peered":"2024-03-30T15:08:03.349193+0000","last_clean":"2024-03-30T15:08:03.349193+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:08:03.349193+0000","last_undegraded":"2024-03-30T15:08:03.349193+0000","last_fullsized":"2024-03-30T15:08:03.349193+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.321030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:38.839544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":1077,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.513951+0000","last_change":"2024-03-30T15:07:31.354747+0000","last_active":"2024-03-30T15:08:03.513951+0000","last_peered":"2024-03-30T15:08:03.513951+0000","last_clean":"2024-03-30T15:08:03.513951+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:08:03.513951+0000","last_undegraded":"2024-03-30T15:08:03.513951+0000","last_fullsized":"2024-03-30T15:08:03.513951+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.354669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.354669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.082266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348050+0000","last_change":"2024-03-30T15:06:56.238490+0000","last_active":"2024-03-30T15:08:03.348050+0000","last_peered":"2024-03-30T15:08:03.348050+0000","last_clean":"2024-03-30T15:08:03.348050+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:08:03.348050+0000","last_undegraded":"2024-03-30T15:08:03.348050+0000","last_fullsized":"2024-03-30T15:08:03.348050+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.238398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.238398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.509153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":1067,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707577+0000","last_change":"2024-03-30T15:07:19.822016+0000","last_active":"2024-03-30T15:08:02.707577+0000","last_peered":"2024-03-30T15:08:02.707577+0000","last_clean":"2024-03-30T15:08:02.707577+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:08:02.707577+0000","last_undegraded":"2024-03-30T15:08:02.707577+0000","last_fullsized":"2024-03-30T15:08:02.707577+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.821886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.821886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.082006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":1068,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516831+0000","last_change":"2024-03-30T15:07:58.173279+0000","last_active":"2024-03-30T15:08:02.516831+0000","last_peered":"2024-03-30T15:08:02.516831+0000","last_clean":"2024-03-30T15:08:02.516831+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:08:02.516831+0000","last_undegraded":"2024-03-30T15:08:02.516831+0000","last_fullsized":"2024-03-30T15:08:02.516831+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.173193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.173193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:11.398411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1457,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348050+0000","last_change":"2024-03-30T15:06:49.183096+0000","last_active":"2024-03-30T15:08:03.348050+0000","last_peered":"2024-03-30T15:08:03.348050+0000","last_clean":"2024-03-30T15:08:03.348050+0000","last_became_active":"2024-03-30T15:06:48.338292+0000","last_became_peered":"2024-03-30T15:06:48.338292+0000","last_unstale":"2024-03-30T15:08:03.348050+0000","last_undegraded":"2024-03-30T15:08:03.348050+0000","last_fullsized":"2024-03-30T15:08:03.348050+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.183009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.183009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:07.453720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514573+0000","last_change":"2024-03-30T15:06:43.474123+0000","last_active":"2024-03-30T15:08:03.514573+0000","last_peered":"2024-03-30T15:08:03.514573+0000","last_clean":"2024-03-30T15:08:03.514573+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:08:03.514573+0000","last_undegraded":"2024-03-30T15:08:03.514573+0000","last_fullsized":"2024-03-30T15:08:03.514573+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.474015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.474015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:12.294266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1193,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707998+0000","last_change":"2024-03-30T15:05:57.752605+0000","last_active":"2024-03-30T15:08:02.707998+0000","last_peered":"2024-03-30T15:08:02.707998+0000","last_clean":"2024-03-30T15:08:02.707998+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:08:02.707998+0000","last_undegraded":"2024-03-30T15:08:02.707998+0000","last_fullsized":"2024-03-30T15:08:02.707998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:05:57.752440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:05:57.752440+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1074,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488829+0000","last_change":"2024-03-30T15:06:11.536378+0000","last_active":"2024-03-30T15:08:03.488829+0000","last_peered":"2024-03-30T15:08:03.488829+0000","last_clean":"2024-03-30T15:08:03.488829+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:08:03.488829+0000","last_undegraded":"2024-03-30T15:08:03.488829+0000","last_fullsized":"2024-03-30T15:08:03.488829+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":352,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514837+0000","last_change":"2024-03-30T15:06:58.353375+0000","last_active":"2024-03-30T15:08:03.514837+0000","last_peered":"2024-03-30T15:08:03.514837+0000","last_clean":"2024-03-30T15:08:03.514837+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:08:03.514837+0000","last_undegraded":"2024-03-30T15:08:03.514837+0000","last_fullsized":"2024-03-30T15:08:03.514837+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'9","last_scrub_stamp":"2024-03-30T15:06:58.353220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.353220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:10.206991+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1155,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.709054+0000","last_change":"2024-03-30T15:06:44.519216+0000","last_active":"2024-03-30T15:08:02.709054+0000","last_peered":"2024-03-30T15:08:02.709054+0000","last_clean":"2024-03-30T15:08:02.709054+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:02.709054+0000","last_undegraded":"2024-03-30T15:08:02.709054+0000","last_fullsized":"2024-03-30T15:08:02.709054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:44.519122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.519122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":454,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516395+0000","last_change":"2024-03-30T15:06:49.951825+0000","last_active":"2024-03-30T15:08:02.516395+0000","last_peered":"2024-03-30T15:08:02.516395+0000","last_clean":"2024-03-30T15:08:02.516395+0000","last_became_active":"2024-03-30T15:06:48.340482+0000","last_became_peered":"2024-03-30T15:06:48.340482+0000","last_unstale":"2024-03-30T15:08:02.516395+0000","last_undegraded":"2024-03-30T15:08:02.516395+0000","last_fullsized":"2024-03-30T15:08:02.516395+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.951714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.951714+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:06.636258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514972+0000","last_change":"2024-03-30T15:07:22.415753+0000","last_active":"2024-03-30T15:08:03.514972+0000","last_peered":"2024-03-30T15:08:03.514972+0000","last_clean":"2024-03-30T15:08:03.514972+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:08:03.514972+0000","last_undegraded":"2024-03-30T15:08:03.514972+0000","last_fullsized":"2024-03-30T15:08:03.514972+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.415581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.415581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:25.352643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1220,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.709155+0000","last_change":"2024-03-30T15:07:20.849551+0000","last_active":"2024-03-30T15:08:02.709155+0000","last_peered":"2024-03-30T15:08:02.709155+0000","last_clean":"2024-03-30T15:08:02.709155+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:08:02.709155+0000","last_undegraded":"2024-03-30T15:08:02.709155+0000","last_fullsized":"2024-03-30T15:08:02.709155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:07:20.849431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.849431+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:22.083359+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1185,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514338+0000","last_change":"2024-03-30T15:07:34.300735+0000","last_active":"2024-03-30T15:08:03.514338+0000","last_peered":"2024-03-30T15:08:03.514338+0000","last_clean":"2024-03-30T15:08:03.514338+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:08:03.514338+0000","last_undegraded":"2024-03-30T15:08:03.514338+0000","last_fullsized":"2024-03-30T15:08:03.514338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:34.300632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.300632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.024216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1177,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348985+0000","last_change":"2024-03-30T15:07:19.878091+0000","last_active":"2024-03-30T15:08:02.348985+0000","last_peered":"2024-03-30T15:08:02.348985+0000","last_clean":"2024-03-30T15:08:02.348985+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:08:02.348985+0000","last_undegraded":"2024-03-30T15:08:02.348985+0000","last_fullsized":"2024-03-30T15:08:02.348985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.878011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.878011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.486305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1185,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.350493+0000","last_change":"2024-03-30T15:08:02.197580+0000","last_active":"2024-03-30T15:08:03.350493+0000","last_peered":"2024-03-30T15:08:03.350493+0000","last_clean":"2024-03-30T15:08:03.350493+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:08:03.350493+0000","last_undegraded":"2024-03-30T15:08:03.350493+0000","last_fullsized":"2024-03-30T15:08:03.350493+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.197498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.197498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.675581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1075,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349902+0000","last_change":"2024-03-30T15:06:05.089656+0000","last_active":"2024-03-30T15:08:02.349902+0000","last_peered":"2024-03-30T15:08:02.349902+0000","last_clean":"2024-03-30T15:08:02.349902+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:08:02.349902+0000","last_undegraded":"2024-03-30T15:08:02.349902+0000","last_fullsized":"2024-03-30T15:08:02.349902+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:05.089580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.089580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1207,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708951+0000","last_change":"2024-03-30T15:07:58.225113+0000","last_active":"2024-03-30T15:08:02.708951+0000","last_peered":"2024-03-30T15:08:02.708951+0000","last_clean":"2024-03-30T15:08:02.708951+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:08:02.708951+0000","last_undegraded":"2024-03-30T15:08:02.708951+0000","last_fullsized":"2024-03-30T15:08:02.708951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:07:58.225002+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.225002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.514446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1176,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707169+0000","last_change":"2024-03-30T15:07:33.014237+0000","last_active":"2024-03-30T15:08:02.707169+0000","last_peered":"2024-03-30T15:08:02.707169+0000","last_clean":"2024-03-30T15:08:02.707169+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:08:02.707169+0000","last_undegraded":"2024-03-30T15:08:02.707169+0000","last_fullsized":"2024-03-30T15:08:02.707169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.014124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.688496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1159,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518327+0000","last_change":"2024-03-30T15:05:58.982494+0000","last_active":"2024-03-30T15:08:02.518327+0000","last_peered":"2024-03-30T15:08:02.518327+0000","last_clean":"2024-03-30T15:08:02.518327+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:08:02.518327+0000","last_undegraded":"2024-03-30T15:08:02.518327+0000","last_fullsized":"2024-03-30T15:08:02.518327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.982407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.982407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1078,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514662+0000","last_change":"2024-03-30T15:06:01.027884+0000","last_active":"2024-03-30T15:08:03.514662+0000","last_peered":"2024-03-30T15:08:03.514662+0000","last_clean":"2024-03-30T15:08:03.514662+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:08:03.514662+0000","last_undegraded":"2024-03-30T15:08:03.514662+0000","last_fullsized":"2024-03-30T15:08:03.514662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.027770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.027770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":454,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517079+0000","last_change":"2024-03-30T15:06:50.920479+0000","last_active":"2024-03-30T15:08:02.517079+0000","last_peered":"2024-03-30T15:08:02.517079+0000","last_clean":"2024-03-30T15:08:02.517079+0000","last_became_active":"2024-03-30T15:06:48.342510+0000","last_became_peered":"2024-03-30T15:06:48.342510+0000","last_unstale":"2024-03-30T15:08:02.517079+0000","last_undegraded":"2024-03-30T15:08:02.517079+0000","last_fullsized":"2024-03-30T15:08:02.517079+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:50.920389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.920389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:06.596356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1075,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349655+0000","last_change":"2024-03-30T15:06:08.169797+0000","last_active":"2024-03-30T15:08:02.349655+0000","last_peered":"2024-03-30T15:08:02.349655+0000","last_clean":"2024-03-30T15:08:02.349655+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:08:02.349655+0000","last_undegraded":"2024-03-30T15:08:02.349655+0000","last_fullsized":"2024-03-30T15:08:02.349655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.169669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.169669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1223,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349711+0000","last_change":"2024-03-30T15:07:28.992483+0000","last_active":"2024-03-30T15:08:02.349711+0000","last_peered":"2024-03-30T15:08:02.349711+0000","last_clean":"2024-03-30T15:08:02.349711+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:08:02.349711+0000","last_undegraded":"2024-03-30T15:08:02.349711+0000","last_fullsized":"2024-03-30T15:08:02.349711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:28.992348+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.992348+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:29.581423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1079,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347562+0000","last_change":"2024-03-30T15:07:22.928759+0000","last_active":"2024-03-30T15:08:02.347562+0000","last_peered":"2024-03-30T15:08:02.347562+0000","last_clean":"2024-03-30T15:08:02.347562+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:08:02.347562+0000","last_undegraded":"2024-03-30T15:08:02.347562+0000","last_fullsized":"2024-03-30T15:08:02.347562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.928677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.928677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.321766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1329,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708980+0000","last_change":"2024-03-30T15:06:06.615931+0000","last_active":"2024-03-30T15:08:02.708980+0000","last_peered":"2024-03-30T15:08:02.708980+0000","last_clean":"2024-03-30T15:08:02.708980+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:08:02.708980+0000","last_undegraded":"2024-03-30T15:08:02.708980+0000","last_fullsized":"2024-03-30T15:08:02.708980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:06.615780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.615780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1155,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517321+0000","last_change":"2024-03-30T15:06:57.951315+0000","last_active":"2024-03-30T15:08:02.517321+0000","last_peered":"2024-03-30T15:08:02.517321+0000","last_clean":"2024-03-30T15:08:02.517321+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:08:02.517321+0000","last_undegraded":"2024-03-30T15:08:02.517321+0000","last_fullsized":"2024-03-30T15:08:02.517321+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.951208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.951208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":455,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708048+0000","last_change":"2024-03-30T15:06:48.534844+0000","last_active":"2024-03-30T15:08:02.708048+0000","last_peered":"2024-03-30T15:08:02.708048+0000","last_clean":"2024-03-30T15:08:02.708048+0000","last_became_active":"2024-03-30T15:06:48.341420+0000","last_became_peered":"2024-03-30T15:06:48.341420+0000","last_unstale":"2024-03-30T15:08:02.708048+0000","last_undegraded":"2024-03-30T15:08:02.708048+0000","last_fullsized":"2024-03-30T15:08:02.708048+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.534763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.534763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1083,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348631+0000","last_change":"2024-03-30T15:07:36.166030+0000","last_active":"2024-03-30T15:08:03.348631+0000","last_peered":"2024-03-30T15:08:03.348631+0000","last_clean":"2024-03-30T15:08:03.348631+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:08:03.348631+0000","last_undegraded":"2024-03-30T15:08:03.348631+0000","last_fullsized":"2024-03-30T15:08:03.348631+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.165947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.824703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1168,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707060+0000","last_change":"2024-03-30T15:07:57.209970+0000","last_active":"2024-03-30T15:08:02.707060+0000","last_peered":"2024-03-30T15:08:02.707060+0000","last_clean":"2024-03-30T15:08:02.707060+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:08:02.707060+0000","last_undegraded":"2024-03-30T15:08:02.707060+0000","last_fullsized":"2024-03-30T15:08:02.707060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.209884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.209884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.514345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1180,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348788+0000","last_change":"2024-03-30T15:05:22.145932+0000","last_active":"2024-03-30T15:08:03.348788+0000","last_peered":"2024-03-30T15:08:03.348788+0000","last_clean":"2024-03-30T15:08:03.348788+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:08:03.348788+0000","last_undegraded":"2024-03-30T15:08:03.348788+0000","last_fullsized":"2024-03-30T15:08:03.348788+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:05:22.145836+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:05:22.145836+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1145,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349698+0000","last_change":"2024-03-30T15:05:59.324853+0000","last_active":"2024-03-30T15:08:03.349698+0000","last_peered":"2024-03-30T15:08:03.349698+0000","last_clean":"2024-03-30T15:08:03.349698+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:08:03.349698+0000","last_undegraded":"2024-03-30T15:08:03.349698+0000","last_fullsized":"2024-03-30T15:08:03.349698+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:05:59.324757+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.324757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1156,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707462+0000","last_change":"2024-03-30T15:06:09.562654+0000","last_active":"2024-03-30T15:08:02.707462+0000","last_peered":"2024-03-30T15:08:02.707462+0000","last_clean":"2024-03-30T15:08:02.707462+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:08:02.707462+0000","last_undegraded":"2024-03-30T15:08:02.707462+0000","last_fullsized":"2024-03-30T15:08:02.707462+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.562567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.562567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":454,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516918+0000","last_change":"2024-03-30T15:06:51.940521+0000","last_active":"2024-03-30T15:08:02.516918+0000","last_peered":"2024-03-30T15:08:02.516918+0000","last_clean":"2024-03-30T15:08:02.516918+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:08:02.516918+0000","last_undegraded":"2024-03-30T15:08:02.516918+0000","last_fullsized":"2024-03-30T15:08:02.516918+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:51.940375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:51.940375+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1093,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348650+0000","last_change":"2024-03-30T15:07:24.023700+0000","last_active":"2024-03-30T15:08:03.348650+0000","last_peered":"2024-03-30T15:08:03.348650+0000","last_clean":"2024-03-30T15:08:03.348650+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:08:03.348650+0000","last_undegraded":"2024-03-30T15:08:03.348650+0000","last_fullsized":"2024-03-30T15:08:03.348650+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.023617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.023617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.066284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1218,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347320+0000","last_change":"2024-03-30T15:06:11.179123+0000","last_active":"2024-03-30T15:08:02.347320+0000","last_peered":"2024-03-30T15:08:02.347320+0000","last_clean":"2024-03-30T15:08:02.347320+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:08:02.347320+0000","last_undegraded":"2024-03-30T15:08:02.347320+0000","last_fullsized":"2024-03-30T15:08:02.347320+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.179008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.179008+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1214,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707241+0000","last_change":"2024-03-30T15:06:41.454580+0000","last_active":"2024-03-30T15:08:02.707241+0000","last_peered":"2024-03-30T15:08:02.707241+0000","last_clean":"2024-03-30T15:08:02.707241+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:08:02.707241+0000","last_undegraded":"2024-03-30T15:08:02.707241+0000","last_fullsized":"2024-03-30T15:08:02.707241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:41.454445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:41.454445+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1175,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489498+0000","last_change":"2024-03-30T15:07:24.562748+0000","last_active":"2024-03-30T15:08:03.489498+0000","last_peered":"2024-03-30T15:08:03.489498+0000","last_clean":"2024-03-30T15:08:03.489498+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:08:03.489498+0000","last_undegraded":"2024-03-30T15:08:03.489498+0000","last_fullsized":"2024-03-30T15:08:03.489498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.710249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1167,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348995+0000","last_change":"2024-03-30T15:06:39.147213+0000","last_active":"2024-03-30T15:08:03.348995+0000","last_peered":"2024-03-30T15:08:03.348995+0000","last_clean":"2024-03-30T15:08:03.348995+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:08:03.348995+0000","last_undegraded":"2024-03-30T15:08:03.348995+0000","last_fullsized":"2024-03-30T15:08:03.348995+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.147113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.147113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":462,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514920+0000","last_change":"2024-03-30T15:06:49.387252+0000","last_active":"2024-03-30T15:08:03.514920+0000","last_peered":"2024-03-30T15:08:03.514920+0000","last_clean":"2024-03-30T15:08:03.514920+0000","last_became_active":"2024-03-30T15:06:48.342151+0000","last_became_peered":"2024-03-30T15:06:48.342151+0000","last_unstale":"2024-03-30T15:08:03.514920+0000","last_undegraded":"2024-03-30T15:08:03.514920+0000","last_fullsized":"2024-03-30T15:08:03.514920+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.387153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.387153+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1090,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348754+0000","last_change":"2024-03-30T15:07:25.015706+0000","last_active":"2024-03-30T15:08:03.348754+0000","last_peered":"2024-03-30T15:08:03.348754+0000","last_clean":"2024-03-30T15:08:03.348754+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:08:03.348754+0000","last_undegraded":"2024-03-30T15:08:03.348754+0000","last_fullsized":"2024-03-30T15:08:03.348754+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.015600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.015600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.857225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1343,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488807+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:08:03.488807+0000","last_peered":"2024-03-30T15:08:03.488807+0000","last_clean":"2024-03-30T15:08:03.488807+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:08:03.488807+0000","last_undegraded":"2024-03-30T15:08:03.488807+0000","last_fullsized":"2024-03-30T15:08:03.488807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1217,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348138+0000","last_change":"2024-03-30T15:08:01.102530+0000","last_active":"2024-03-30T15:08:02.348138+0000","last_peered":"2024-03-30T15:08:02.348138+0000","last_clean":"2024-03-30T15:08:02.348138+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:08:02.348138+0000","last_undegraded":"2024-03-30T15:08:02.348138+0000","last_fullsized":"2024-03-30T15:08:02.348138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:08:01.102431+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.102431+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.685021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1176,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517654+0000","last_change":"2024-03-30T15:06:53.935771+0000","last_active":"2024-03-30T15:08:02.517654+0000","last_peered":"2024-03-30T15:08:02.517654+0000","last_clean":"2024-03-30T15:08:02.517654+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:08:02.517654+0000","last_undegraded":"2024-03-30T15:08:02.517654+0000","last_fullsized":"2024-03-30T15:08:02.517654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.935674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.935674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:19.007131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1161,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488132+0000","last_change":"2024-03-30T15:06:10.554067+0000","last_active":"2024-03-30T15:08:03.488132+0000","last_peered":"2024-03-30T15:08:03.488132+0000","last_clean":"2024-03-30T15:08:03.488132+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:08:03.488132+0000","last_undegraded":"2024-03-30T15:08:03.488132+0000","last_fullsized":"2024-03-30T15:08:03.488132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":462,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514944+0000","last_change":"2024-03-30T15:06:48.409534+0000","last_active":"2024-03-30T15:08:03.514944+0000","last_peered":"2024-03-30T15:08:03.514944+0000","last_clean":"2024-03-30T15:08:03.514944+0000","last_became_active":"2024-03-30T15:06:48.338506+0000","last_became_peered":"2024-03-30T15:06:48.338506+0000","last_unstale":"2024-03-30T15:08:03.514944+0000","last_undegraded":"2024-03-30T15:08:03.514944+0000","last_fullsized":"2024-03-30T15:08:03.514944+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.409455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.409455+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:05.854159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1083,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349070+0000","last_change":"2024-03-30T15:06:59.884862+0000","last_active":"2024-03-30T15:08:03.349070+0000","last_peered":"2024-03-30T15:08:03.349070+0000","last_clean":"2024-03-30T15:08:03.349070+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:08:03.349070+0000","last_undegraded":"2024-03-30T15:08:03.349070+0000","last_fullsized":"2024-03-30T15:08:03.349070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.884709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.884709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:09.616665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1177,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514506+0000","last_change":"2024-03-30T15:06:54.318139+0000","last_active":"2024-03-30T15:08:03.514506+0000","last_peered":"2024-03-30T15:08:03.514506+0000","last_clean":"2024-03-30T15:08:03.514506+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:08:03.514506+0000","last_undegraded":"2024-03-30T15:08:03.514506+0000","last_fullsized":"2024-03-30T15:08:03.514506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.318005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:10.866580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1228,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489224+0000","last_change":"2024-03-30T15:08:02.516526+0000","last_active":"2024-03-30T15:08:03.489224+0000","last_peered":"2024-03-30T15:08:03.489224+0000","last_clean":"2024-03-30T15:08:03.489224+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:08:03.489224+0000","last_undegraded":"2024-03-30T15:08:03.489224+0000","last_fullsized":"2024-03-30T15:08:03.489224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:14.340123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1186,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348836+0000","last_change":"2024-03-30T15:07:17.809348+0000","last_active":"2024-03-30T15:08:02.348836+0000","last_peered":"2024-03-30T15:08:02.348836+0000","last_clean":"2024-03-30T15:08:02.348836+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:08:02.348836+0000","last_undegraded":"2024-03-30T15:08:02.348836+0000","last_fullsized":"2024-03-30T15:08:02.348836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:17.809228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:17.809228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:27.761242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1174,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347475+0000","last_change":"2024-03-30T15:06:39.844760+0000","last_active":"2024-03-30T15:08:02.347475+0000","last_peered":"2024-03-30T15:08:02.347475+0000","last_clean":"2024-03-30T15:08:02.347475+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:08:02.347475+0000","last_undegraded":"2024-03-30T15:08:02.347475+0000","last_fullsized":"2024-03-30T15:08:02.347475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.844668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.844668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":455,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348000+0000","last_change":"2024-03-30T15:06:48.873065+0000","last_active":"2024-03-30T15:08:02.348000+0000","last_peered":"2024-03-30T15:08:02.348000+0000","last_clean":"2024-03-30T15:08:02.348000+0000","last_became_active":"2024-03-30T15:06:48.339258+0000","last_became_peered":"2024-03-30T15:06:48.339258+0000","last_unstale":"2024-03-30T15:08:02.348000+0000","last_undegraded":"2024-03-30T15:08:02.348000+0000","last_fullsized":"2024-03-30T15:08:02.348000+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.872974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.872974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489393+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:08:03.489393+0000","last_peered":"2024-03-30T15:08:03.489393+0000","last_clean":"2024-03-30T15:08:03.489393+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:08:03.489393+0000","last_undegraded":"2024-03-30T15:08:03.489393+0000","last_fullsized":"2024-03-30T15:08:03.489393+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1168,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518387+0000","last_change":"2024-03-30T15:06:52.973865+0000","last_active":"2024-03-30T15:08:02.518387+0000","last_peered":"2024-03-30T15:08:02.518387+0000","last_clean":"2024-03-30T15:08:02.518387+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:08:02.518387+0000","last_undegraded":"2024-03-30T15:08:02.518387+0000","last_fullsized":"2024-03-30T15:08:02.518387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.973770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.973770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:20.699838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1213,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489392+0000","last_change":"2024-03-30T15:08:01.499074+0000","last_active":"2024-03-30T15:08:03.489392+0000","last_peered":"2024-03-30T15:08:03.489392+0000","last_clean":"2024-03-30T15:08:03.489392+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:08:03.489392+0000","last_undegraded":"2024-03-30T15:08:03.489392+0000","last_fullsized":"2024-03-30T15:08:03.489392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.212933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":1990,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348321+0000","last_change":"2024-03-30T15:07:34.264898+0000","last_active":"2024-03-30T15:08:03.348321+0000","last_peered":"2024-03-30T15:08:03.348321+0000","last_clean":"2024-03-30T15:08:03.348321+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:08:03.348321+0000","last_undegraded":"2024-03-30T15:08:03.348321+0000","last_fullsized":"2024-03-30T15:08:03.348321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:07:34.264815+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.264815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.246868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":798,"num_read_kb":793,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1173,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488481+0000","last_change":"2024-03-30T15:07:29.592120+0000","last_active":"2024-03-30T15:08:03.488481+0000","last_peered":"2024-03-30T15:08:03.488481+0000","last_clean":"2024-03-30T15:08:03.488481+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:08:03.488481+0000","last_undegraded":"2024-03-30T15:08:03.488481+0000","last_fullsized":"2024-03-30T15:08:03.488481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.592039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.592039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.178771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1166,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348712+0000","last_change":"2024-03-30T15:06:45.854073+0000","last_active":"2024-03-30T15:08:02.348712+0000","last_peered":"2024-03-30T15:08:02.348712+0000","last_clean":"2024-03-30T15:08:02.348712+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:08:02.348712+0000","last_undegraded":"2024-03-30T15:08:02.348712+0000","last_fullsized":"2024-03-30T15:08:02.348712+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.853998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.853998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1075,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349395+0000","last_change":"2024-03-30T15:07:24.315658+0000","last_active":"2024-03-30T15:08:03.349395+0000","last_peered":"2024-03-30T15:08:03.349395+0000","last_clean":"2024-03-30T15:08:03.349395+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:08:03.349395+0000","last_undegraded":"2024-03-30T15:08:03.349395+0000","last_fullsized":"2024-03-30T15:08:03.349395+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.315584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.315584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:49.961946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1163,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349356+0000","last_change":"2024-03-30T15:05:15.118857+0000","last_active":"2024-03-30T15:08:03.349356+0000","last_peered":"2024-03-30T15:08:03.349356+0000","last_clean":"2024-03-30T15:08:03.349356+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:08:03.349356+0000","last_undegraded":"2024-03-30T15:08:03.349356+0000","last_fullsized":"2024-03-30T15:08:03.349356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.118778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.118778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1383,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515137+0000","last_change":"2024-03-30T15:06:02.076986+0000","last_active":"2024-03-30T15:08:03.515137+0000","last_peered":"2024-03-30T15:08:03.515137+0000","last_clean":"2024-03-30T15:08:03.515137+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:08:03.515137+0000","last_undegraded":"2024-03-30T15:08:03.515137+0000","last_fullsized":"2024-03-30T15:08:03.515137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:02.076864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.076864+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1070,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517038+0000","last_change":"2024-03-30T15:07:00.995237+0000","last_active":"2024-03-30T15:08:02.517038+0000","last_peered":"2024-03-30T15:08:02.517038+0000","last_clean":"2024-03-30T15:08:02.517038+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:08:02.517038+0000","last_undegraded":"2024-03-30T15:08:02.517038+0000","last_fullsized":"2024-03-30T15:08:02.517038+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.995156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.995156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:11.329581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1448,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348398+0000","last_change":"2024-03-30T15:06:45.099476+0000","last_active":"2024-03-30T15:08:03.348398+0000","last_peered":"2024-03-30T15:08:03.348398+0000","last_clean":"2024-03-30T15:08:03.348398+0000","last_became_active":"2024-03-30T15:06:43.342924+0000","last_became_peered":"2024-03-30T15:06:43.342924+0000","last_unstale":"2024-03-30T15:08:03.348398+0000","last_undegraded":"2024-03-30T15:08:03.348398+0000","last_fullsized":"2024-03-30T15:08:03.348398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:45.099372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.099372+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:06.630852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1156,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517667+0000","last_change":"2024-03-30T15:06:45.066707+0000","last_active":"2024-03-30T15:08:02.517667+0000","last_peered":"2024-03-30T15:08:02.517667+0000","last_clean":"2024-03-30T15:08:02.517667+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:08:02.517667+0000","last_undegraded":"2024-03-30T15:08:02.517667+0000","last_fullsized":"2024-03-30T15:08:02.517667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.066574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.066574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1465,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348268+0000","last_change":"2024-03-30T15:07:56.660010+0000","last_active":"2024-03-30T15:08:03.348268+0000","last_peered":"2024-03-30T15:08:03.348268+0000","last_clean":"2024-03-30T15:08:03.348268+0000","last_became_active":"2024-03-30T15:06:48.339784+0000","last_became_peered":"2024-03-30T15:06:48.339784+0000","last_unstale":"2024-03-30T15:08:03.348268+0000","last_undegraded":"2024-03-30T15:08:03.348268+0000","last_fullsized":"2024-03-30T15:08:03.348268+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:07:56.659850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:07:56.659850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.574408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515056+0000","last_change":"2024-03-30T15:06:10.008324+0000","last_active":"2024-03-30T15:08:03.515056+0000","last_peered":"2024-03-30T15:08:03.515056+0000","last_clean":"2024-03-30T15:08:03.515056+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:08:03.515056+0000","last_undegraded":"2024-03-30T15:08:03.515056+0000","last_fullsized":"2024-03-30T15:08:03.515056+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.008246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.008246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1182,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.350209+0000","last_change":"2024-03-30T15:06:52.804319+0000","last_active":"2024-03-30T15:08:03.350209+0000","last_peered":"2024-03-30T15:08:03.350209+0000","last_clean":"2024-03-30T15:08:03.350209+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:08:03.350209+0000","last_undegraded":"2024-03-30T15:08:03.350209+0000","last_fullsized":"2024-03-30T15:08:03.350209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.804238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1220,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349659+0000","last_change":"2024-03-30T15:06:44.835160+0000","last_active":"2024-03-30T15:08:02.349659+0000","last_peered":"2024-03-30T15:08:02.349659+0000","last_clean":"2024-03-30T15:08:02.349659+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:08:02.349659+0000","last_undegraded":"2024-03-30T15:08:02.349659+0000","last_fullsized":"2024-03-30T15:08:02.349659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:06:44.835068+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.835068+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:04.911909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1199,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348754+0000","last_change":"2024-03-30T15:06:57.786762+0000","last_active":"2024-03-30T15:08:03.348754+0000","last_peered":"2024-03-30T15:08:03.348754+0000","last_clean":"2024-03-30T15:08:03.348754+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:08:03.348754+0000","last_undegraded":"2024-03-30T15:08:03.348754+0000","last_fullsized":"2024-03-30T15:08:03.348754+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.786685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.786685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1186,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347355+0000","last_change":"2024-03-30T15:07:23.937183+0000","last_active":"2024-03-30T15:08:02.347355+0000","last_peered":"2024-03-30T15:08:02.347355+0000","last_clean":"2024-03-30T15:08:02.347355+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:08:02.347355+0000","last_undegraded":"2024-03-30T15:08:02.347355+0000","last_fullsized":"2024-03-30T15:08:02.347355+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.937096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.937096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:27.709429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1175,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349563+0000","last_change":"2024-03-30T15:07:58.653866+0000","last_active":"2024-03-30T15:08:03.349563+0000","last_peered":"2024-03-30T15:08:03.349563+0000","last_clean":"2024-03-30T15:08:03.349563+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:08:03.349563+0000","last_undegraded":"2024-03-30T15:08:03.349563+0000","last_fullsized":"2024-03-30T15:08:03.349563+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.653564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.653564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.992993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":454,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516348+0000","last_change":"2024-03-30T15:06:48.962204+0000","last_active":"2024-03-30T15:08:02.516348+0000","last_peered":"2024-03-30T15:08:02.516348+0000","last_clean":"2024-03-30T15:08:02.516348+0000","last_became_active":"2024-03-30T15:06:48.339451+0000","last_became_peered":"2024-03-30T15:06:48.339451+0000","last_unstale":"2024-03-30T15:08:02.516348+0000","last_undegraded":"2024-03-30T15:08:02.516348+0000","last_fullsized":"2024-03-30T15:08:02.516348+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.962124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.962124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488953+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:08:03.488953+0000","last_peered":"2024-03-30T15:08:03.488953+0000","last_clean":"2024-03-30T15:08:03.488953+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:08:03.488953+0000","last_undegraded":"2024-03-30T15:08:03.488953+0000","last_fullsized":"2024-03-30T15:08:03.488953+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1165,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488940+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:08:03.488940+0000","last_peered":"2024-03-30T15:08:03.488940+0000","last_clean":"2024-03-30T15:08:03.488940+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:08:03.488940+0000","last_undegraded":"2024-03-30T15:08:03.488940+0000","last_fullsized":"2024-03-30T15:08:03.488940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1347,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517122+0000","last_change":"2024-03-30T15:06:05.010246+0000","last_active":"2024-03-30T15:08:02.517122+0000","last_peered":"2024-03-30T15:08:02.517122+0000","last_clean":"2024-03-30T15:08:02.517122+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:08:02.517122+0000","last_undegraded":"2024-03-30T15:08:02.517122+0000","last_fullsized":"2024-03-30T15:08:02.517122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:06:05.010128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.010128+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1176,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707670+0000","last_change":"2024-03-30T15:07:31.968746+0000","last_active":"2024-03-30T15:08:02.707670+0000","last_peered":"2024-03-30T15:08:02.707670+0000","last_clean":"2024-03-30T15:08:02.707670+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:08:02.707670+0000","last_undegraded":"2024-03-30T15:08:02.707670+0000","last_fullsized":"2024-03-30T15:08:02.707670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.259707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1179,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517234+0000","last_change":"2024-03-30T15:06:40.992863+0000","last_active":"2024-03-30T15:08:02.517234+0000","last_peered":"2024-03-30T15:08:02.517234+0000","last_clean":"2024-03-30T15:08:02.517234+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:08:02.517234+0000","last_undegraded":"2024-03-30T15:08:02.517234+0000","last_fullsized":"2024-03-30T15:08:02.517234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:06:40.992758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.992758+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:06.700899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1076,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708068+0000","last_change":"2024-03-30T15:08:00.231743+0000","last_active":"2024-03-30T15:08:02.708068+0000","last_peered":"2024-03-30T15:08:02.708068+0000","last_clean":"2024-03-30T15:08:02.708068+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:08:02.708068+0000","last_undegraded":"2024-03-30T15:08:02.708068+0000","last_fullsized":"2024-03-30T15:08:02.708068+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.231663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.231663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.864990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489067+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:08:03.489067+0000","last_peered":"2024-03-30T15:08:03.489067+0000","last_clean":"2024-03-30T15:08:03.489067+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:08:03.489067+0000","last_undegraded":"2024-03-30T15:08:03.489067+0000","last_fullsized":"2024-03-30T15:08:03.489067+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1193,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488972+0000","last_change":"2024-03-30T15:04:46.560749+0000","last_active":"2024-03-30T15:08:03.488972+0000","last_peered":"2024-03-30T15:08:03.488972+0000","last_clean":"2024-03-30T15:08:03.488972+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:08:03.488972+0000","last_undegraded":"2024-03-30T15:08:03.488972+0000","last_fullsized":"2024-03-30T15:08:03.488972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:04:46.560597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:04:46.560597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1175,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518247+0000","last_change":"2024-03-30T15:07:20.139612+0000","last_active":"2024-03-30T15:08:02.518247+0000","last_peered":"2024-03-30T15:08:02.518247+0000","last_clean":"2024-03-30T15:08:02.518247+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:08:02.518247+0000","last_undegraded":"2024-03-30T15:08:02.518247+0000","last_fullsized":"2024-03-30T15:08:02.518247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:20.139530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.139530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.029764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1190,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348924+0000","last_change":"2024-03-30T15:07:58.247377+0000","last_active":"2024-03-30T15:08:03.348924+0000","last_peered":"2024-03-30T15:08:03.348924+0000","last_clean":"2024-03-30T15:08:03.348924+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:08:03.348924+0000","last_undegraded":"2024-03-30T15:08:03.348924+0000","last_fullsized":"2024-03-30T15:08:03.348924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.247271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.247271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.826144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1161,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488129+0000","last_change":"2024-03-30T15:07:21.573597+0000","last_active":"2024-03-30T15:08:03.488129+0000","last_peered":"2024-03-30T15:08:03.488129+0000","last_clean":"2024-03-30T15:08:03.488129+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:08:03.488129+0000","last_undegraded":"2024-03-30T15:08:03.488129+0000","last_fullsized":"2024-03-30T15:08:03.488129+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.573522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.573522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.445672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1072,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707837+0000","last_change":"2024-03-30T15:07:22.909083+0000","last_active":"2024-03-30T15:08:02.707837+0000","last_peered":"2024-03-30T15:08:02.707837+0000","last_clean":"2024-03-30T15:08:02.707837+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:08:02.707837+0000","last_undegraded":"2024-03-30T15:08:02.707837+0000","last_fullsized":"2024-03-30T15:08:02.707837+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.908989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.908989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.320783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1262,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514479+0000","last_change":"2024-03-30T15:06:44.482561+0000","last_active":"2024-03-30T15:08:03.514479+0000","last_peered":"2024-03-30T15:08:03.514479+0000","last_clean":"2024-03-30T15:08:03.514479+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:08:03.514479+0000","last_undegraded":"2024-03-30T15:08:03.514479+0000","last_fullsized":"2024-03-30T15:08:03.514479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:06:44.482442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.482442+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1178,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349474+0000","last_change":"2024-03-30T15:07:18.376275+0000","last_active":"2024-03-30T15:08:03.349474+0000","last_peered":"2024-03-30T15:08:03.349474+0000","last_clean":"2024-03-30T15:08:03.349474+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:08:03.349474+0000","last_undegraded":"2024-03-30T15:08:03.349474+0000","last_fullsized":"2024-03-30T15:08:03.349474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.376197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.376197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.539972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1181,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349040+0000","last_change":"2024-03-30T15:06:48.713525+0000","last_active":"2024-03-30T15:08:03.349040+0000","last_peered":"2024-03-30T15:08:03.349040+0000","last_clean":"2024-03-30T15:08:03.349040+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:08:03.349040+0000","last_undegraded":"2024-03-30T15:08:03.349040+0000","last_fullsized":"2024-03-30T15:08:03.349040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:48.713441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.713441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:16.143415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1159,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348242+0000","last_change":"2024-03-30T15:05:29.180863+0000","last_active":"2024-03-30T15:08:03.348242+0000","last_peered":"2024-03-30T15:08:03.348242+0000","last_clean":"2024-03-30T15:08:03.348242+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:08:03.348242+0000","last_undegraded":"2024-03-30T15:08:03.348242+0000","last_fullsized":"2024-03-30T15:08:03.348242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:29.180786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:29.180786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1095,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514053+0000","last_change":"2024-03-30T15:06:46.411474+0000","last_active":"2024-03-30T15:08:03.514053+0000","last_peered":"2024-03-30T15:08:03.514053+0000","last_clean":"2024-03-30T15:08:03.514053+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:08:03.514053+0000","last_undegraded":"2024-03-30T15:08:03.514053+0000","last_fullsized":"2024-03-30T15:08:03.514053+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:06:46.411383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:46.411383+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:04.579199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1182,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348370+0000","last_change":"2024-03-30T15:06:58.835912+0000","last_active":"2024-03-30T15:08:03.348370+0000","last_peered":"2024-03-30T15:08:03.348370+0000","last_clean":"2024-03-30T15:08:03.348370+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:08:03.348370+0000","last_undegraded":"2024-03-30T15:08:03.348370+0000","last_fullsized":"2024-03-30T15:08:03.348370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.835804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.835804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:12.281130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1204,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707347+0000","last_change":"2024-03-30T15:06:01.783446+0000","last_active":"2024-03-30T15:08:02.707347+0000","last_peered":"2024-03-30T15:08:02.707347+0000","last_clean":"2024-03-30T15:08:02.707347+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:08:02.707347+0000","last_undegraded":"2024-03-30T15:08:02.707347+0000","last_fullsized":"2024-03-30T15:08:02.707347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:06:01.783327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.783327+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1170,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349504+0000","last_change":"2024-03-30T15:04:48.135565+0000","last_active":"2024-03-30T15:08:03.349504+0000","last_peered":"2024-03-30T15:08:03.349504+0000","last_clean":"2024-03-30T15:08:03.349504+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:08:03.349504+0000","last_undegraded":"2024-03-30T15:08:03.349504+0000","last_fullsized":"2024-03-30T15:08:03.349504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:48.135480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:48.135480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1071,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349442+0000","last_change":"2024-03-30T15:06:54.229912+0000","last_active":"2024-03-30T15:08:03.349442+0000","last_peered":"2024-03-30T15:08:03.349442+0000","last_clean":"2024-03-30T15:08:03.349442+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:08:03.349442+0000","last_undegraded":"2024-03-30T15:08:03.349442+0000","last_fullsized":"2024-03-30T15:08:03.349442+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.229818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.229818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1073,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.706982+0000","last_change":"2024-03-30T15:07:18.807780+0000","last_active":"2024-03-30T15:08:02.706982+0000","last_peered":"2024-03-30T15:08:02.706982+0000","last_clean":"2024-03-30T15:08:02.706982+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:08:02.706982+0000","last_undegraded":"2024-03-30T15:08:02.706982+0000","last_fullsized":"2024-03-30T15:08:02.706982+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.807666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.807666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:47.636769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1333,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348580+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:08:03.348580+0000","last_peered":"2024-03-30T15:08:03.348580+0000","last_clean":"2024-03-30T15:08:03.348580+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:08:03.348580+0000","last_undegraded":"2024-03-30T15:08:03.348580+0000","last_fullsized":"2024-03-30T15:08:03.348580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1207,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516552+0000","last_change":"2024-03-30T15:06:16.030806+0000","last_active":"2024-03-30T15:08:02.516552+0000","last_peered":"2024-03-30T15:08:02.516552+0000","last_clean":"2024-03-30T15:08:02.516552+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:08:02.516552+0000","last_undegraded":"2024-03-30T15:08:02.516552+0000","last_fullsized":"2024-03-30T15:08:02.516552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:06:16.030694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.030694+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1190,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349799+0000","last_change":"2024-03-30T15:07:32.333233+0000","last_active":"2024-03-30T15:08:03.349799+0000","last_peered":"2024-03-30T15:08:03.349799+0000","last_clean":"2024-03-30T15:08:03.349799+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:08:03.349799+0000","last_undegraded":"2024-03-30T15:08:03.349799+0000","last_fullsized":"2024-03-30T15:08:03.349799+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:07:32.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.333139+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.507134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1156,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517605+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:08:02.517605+0000","last_peered":"2024-03-30T15:08:02.517605+0000","last_clean":"2024-03-30T15:08:02.517605+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:02.517605+0000","last_undegraded":"2024-03-30T15:08:02.517605+0000","last_fullsized":"2024-03-30T15:08:02.517605+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515058+0000","last_change":"2024-03-30T15:07:32.310952+0000","last_active":"2024-03-30T15:08:03.515058+0000","last_peered":"2024-03-30T15:08:03.515058+0000","last_clean":"2024-03-30T15:08:03.515058+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:08:03.515058+0000","last_undegraded":"2024-03-30T15:08:03.515058+0000","last_fullsized":"2024-03-30T15:08:03.515058+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.310831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.310831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.533764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":767,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.751466+0000","last_change":"2024-03-30T15:06:41.642605+0000","last_active":"2024-03-30T15:08:02.751466+0000","last_peered":"2024-03-30T15:08:02.751466+0000","last_clean":"2024-03-30T15:08:02.751466+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:08:02.751466+0000","last_undegraded":"2024-03-30T15:08:02.751466+0000","last_fullsized":"2024-03-30T15:08:02.751466+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:06:41.642510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:41.642510+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1192,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.513932+0000","last_change":"2024-03-30T15:07:58.498811+0000","last_active":"2024-03-30T15:08:03.513932+0000","last_peered":"2024-03-30T15:08:03.513932+0000","last_clean":"2024-03-30T15:08:03.513932+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:08:03.513932+0000","last_undegraded":"2024-03-30T15:08:03.513932+0000","last_fullsized":"2024-03-30T15:08:03.513932+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.498678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.498678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.745457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1159,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517466+0000","last_change":"2024-03-30T15:06:38.946768+0000","last_active":"2024-03-30T15:08:02.517466+0000","last_peered":"2024-03-30T15:08:02.517466+0000","last_clean":"2024-03-30T15:08:02.517466+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:08:02.517466+0000","last_undegraded":"2024-03-30T15:08:02.517466+0000","last_fullsized":"2024-03-30T15:08:02.517466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:38.946668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:38.946668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1158,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349181+0000","last_change":"2024-03-30T15:06:42.118047+0000","last_active":"2024-03-30T15:08:03.349181+0000","last_peered":"2024-03-30T15:08:03.349181+0000","last_clean":"2024-03-30T15:08:03.349181+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:08:03.349181+0000","last_undegraded":"2024-03-30T15:08:03.349181+0000","last_fullsized":"2024-03-30T15:08:03.349181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:42.117927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.117927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1073,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707925+0000","last_change":"2024-03-30T15:07:29.943796+0000","last_active":"2024-03-30T15:08:02.707925+0000","last_peered":"2024-03-30T15:08:02.707925+0000","last_clean":"2024-03-30T15:08:02.707925+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:08:02.707925+0000","last_undegraded":"2024-03-30T15:08:02.707925+0000","last_fullsized":"2024-03-30T15:08:02.707925+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.943712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.943712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:33.196867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1193,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489603+0000","last_change":"2024-03-30T15:07:34.554038+0000","last_active":"2024-03-30T15:08:03.489603+0000","last_peered":"2024-03-30T15:08:03.489603+0000","last_clean":"2024-03-30T15:08:03.489603+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:08:03.489603+0000","last_undegraded":"2024-03-30T15:08:03.489603+0000","last_fullsized":"2024-03-30T15:08:03.489603+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.654265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1178,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348372+0000","last_change":"2024-03-30T15:06:57.279563+0000","last_active":"2024-03-30T15:08:03.348372+0000","last_peered":"2024-03-30T15:08:03.348372+0000","last_clean":"2024-03-30T15:08:03.348372+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:08:03.348372+0000","last_undegraded":"2024-03-30T15:08:03.348372+0000","last_fullsized":"2024-03-30T15:08:03.348372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.279464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.279464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:26.580401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":264,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514976+0000","last_change":"2024-03-30T15:07:18.427089+0000","last_active":"2024-03-30T15:08:03.514976+0000","last_peered":"2024-03-30T15:08:03.514976+0000","last_clean":"2024-03-30T15:08:03.514976+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:08:03.514976+0000","last_undegraded":"2024-03-30T15:08:03.514976+0000","last_fullsized":"2024-03-30T15:08:03.514976+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1185,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349810+0000","last_change":"2024-03-30T15:08:01.209737+0000","last_active":"2024-03-30T15:08:03.349810+0000","last_peered":"2024-03-30T15:08:03.349810+0000","last_clean":"2024-03-30T15:08:03.349810+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:08:03.349810+0000","last_undegraded":"2024-03-30T15:08:03.349810+0000","last_fullsized":"2024-03-30T15:08:03.349810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:01.209658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.209658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.903658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488778+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:08:03.488778+0000","last_peered":"2024-03-30T15:08:03.488778+0000","last_clean":"2024-03-30T15:08:03.488778+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:08:03.488778+0000","last_undegraded":"2024-03-30T15:08:03.488778+0000","last_fullsized":"2024-03-30T15:08:03.488778+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1188,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518300+0000","last_change":"2024-03-30T15:06:39.943308+0000","last_active":"2024-03-30T15:08:02.518300+0000","last_peered":"2024-03-30T15:08:02.518300+0000","last_clean":"2024-03-30T15:08:02.518300+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:08:02.518300+0000","last_undegraded":"2024-03-30T15:08:02.518300+0000","last_fullsized":"2024-03-30T15:08:02.518300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:39.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.943172+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1172,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488512+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:08:03.488512+0000","last_peered":"2024-03-30T15:08:03.488512+0000","last_clean":"2024-03-30T15:08:03.488512+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:08:03.488512+0000","last_undegraded":"2024-03-30T15:08:03.488512+0000","last_fullsized":"2024-03-30T15:08:03.488512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1218,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349245+0000","last_change":"2024-03-30T15:06:11.222500+0000","last_active":"2024-03-30T15:08:03.349245+0000","last_peered":"2024-03-30T15:08:03.349245+0000","last_clean":"2024-03-30T15:08:03.349245+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:08:03.349245+0000","last_undegraded":"2024-03-30T15:08:03.349245+0000","last_fullsized":"2024-03-30T15:08:03.349245+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.222382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.222382+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1155,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.709183+0000","last_change":"2024-03-30T15:06:58.575629+0000","last_active":"2024-03-30T15:08:02.709183+0000","last_peered":"2024-03-30T15:08:02.709183+0000","last_clean":"2024-03-30T15:08:02.709183+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:08:02.709183+0000","last_undegraded":"2024-03-30T15:08:02.709183+0000","last_fullsized":"2024-03-30T15:08:02.709183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.575544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.575544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:03.553012+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488886+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:08:03.488886+0000","last_peered":"2024-03-30T15:08:03.488886+0000","last_clean":"2024-03-30T15:08:03.488886+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:08:03.488886+0000","last_undegraded":"2024-03-30T15:08:03.488886+0000","last_fullsized":"2024-03-30T15:08:03.488886+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":1127,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348881+0000","last_change":"2024-03-30T15:06:59.844000+0000","last_active":"2024-03-30T15:08:02.348881+0000","last_peered":"2024-03-30T15:08:02.348881+0000","last_clean":"2024-03-30T15:08:02.348881+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:08:02.348881+0000","last_undegraded":"2024-03-30T15:08:02.348881+0000","last_fullsized":"2024-03-30T15:08:02.348881+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:06:59.843847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.843847+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1170,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348873+0000","last_change":"2024-03-30T15:06:14.188494+0000","last_active":"2024-03-30T15:08:03.348873+0000","last_peered":"2024-03-30T15:08:03.348873+0000","last_clean":"2024-03-30T15:08:03.348873+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:08:03.348873+0000","last_undegraded":"2024-03-30T15:08:03.348873+0000","last_fullsized":"2024-03-30T15:08:03.348873+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.188400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.188400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1332,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518269+0000","last_change":"2024-03-30T15:06:44.020947+0000","last_active":"2024-03-30T15:08:02.518269+0000","last_peered":"2024-03-30T15:08:02.518269+0000","last_clean":"2024-03-30T15:08:02.518269+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:08:02.518269+0000","last_undegraded":"2024-03-30T15:08:02.518269+0000","last_fullsized":"2024-03-30T15:08:02.518269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:44.020829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.020829+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1162,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488159+0000","last_change":"2024-03-30T15:07:30.551177+0000","last_active":"2024-03-30T15:08:03.488159+0000","last_peered":"2024-03-30T15:08:03.488159+0000","last_clean":"2024-03-30T15:08:03.488159+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:08:03.488159+0000","last_undegraded":"2024-03-30T15:08:03.488159+0000","last_fullsized":"2024-03-30T15:08:03.488159+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:07:30.551096+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.551096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:33.345129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1072,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707860+0000","last_change":"2024-03-30T15:07:21.866188+0000","last_active":"2024-03-30T15:08:02.707860+0000","last_peered":"2024-03-30T15:08:02.707860+0000","last_clean":"2024-03-30T15:08:02.707860+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:08:02.707860+0000","last_undegraded":"2024-03-30T15:08:02.707860+0000","last_fullsized":"2024-03-30T15:08:02.707860+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.866106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.866106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.392301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1199,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707916+0000","last_change":"2024-03-30T15:06:54.562492+0000","last_active":"2024-03-30T15:08:02.707916+0000","last_peered":"2024-03-30T15:08:02.707916+0000","last_clean":"2024-03-30T15:08:02.707916+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:08:02.707916+0000","last_undegraded":"2024-03-30T15:08:02.707916+0000","last_fullsized":"2024-03-30T15:08:02.707916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:06:54.562347+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.562347+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1175,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707826+0000","last_change":"2024-03-30T15:07:00.555778+0000","last_active":"2024-03-30T15:08:02.707826+0000","last_peered":"2024-03-30T15:08:02.707826+0000","last_clean":"2024-03-30T15:08:02.707826+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:08:02.707826+0000","last_undegraded":"2024-03-30T15:08:02.707826+0000","last_fullsized":"2024-03-30T15:08:02.707826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.555700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.555700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.200782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1177,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348082+0000","last_change":"2024-03-30T15:07:35.184388+0000","last_active":"2024-03-30T15:08:02.348082+0000","last_peered":"2024-03-30T15:08:02.348082+0000","last_clean":"2024-03-30T15:08:02.348082+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:08:02.348082+0000","last_undegraded":"2024-03-30T15:08:02.348082+0000","last_fullsized":"2024-03-30T15:08:02.348082+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.166334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1065,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488484+0000","last_change":"2024-03-30T15:04:43.553488+0000","last_active":"2024-03-30T15:08:03.488484+0000","last_peered":"2024-03-30T15:08:03.488484+0000","last_clean":"2024-03-30T15:08:03.488484+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:08:03.488484+0000","last_undegraded":"2024-03-30T15:08:03.488484+0000","last_fullsized":"2024-03-30T15:08:03.488484+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:43.553407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.518782+0000","last_clean_scrub_stamp":"2024-03-30T15:04:43.553407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1072,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707368+0000","last_change":"2024-03-30T15:07:24.946870+0000","last_active":"2024-03-30T15:08:02.707368+0000","last_peered":"2024-03-30T15:08:02.707368+0000","last_clean":"2024-03-30T15:08:02.707368+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:08:02.707368+0000","last_undegraded":"2024-03-30T15:08:02.707368+0000","last_fullsized":"2024-03-30T15:08:02.707368+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.946762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.946762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.723932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1198,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348168+0000","last_change":"2024-03-30T15:07:20.851744+0000","last_active":"2024-03-30T15:08:02.348168+0000","last_peered":"2024-03-30T15:08:02.348168+0000","last_clean":"2024-03-30T15:08:02.348168+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:08:02.348168+0000","last_undegraded":"2024-03-30T15:08:02.348168+0000","last_fullsized":"2024-03-30T15:08:02.348168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:20.851659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.851659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:28.316096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1193,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349285+0000","last_change":"2024-03-30T15:07:33.297905+0000","last_active":"2024-03-30T15:08:03.349285+0000","last_peered":"2024-03-30T15:08:03.349285+0000","last_clean":"2024-03-30T15:08:03.349285+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:08:03.349285+0000","last_undegraded":"2024-03-30T15:08:03.349285+0000","last_fullsized":"2024-03-30T15:08:03.349285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.297820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.464840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":334,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707474+0000","last_change":"2024-03-30T15:06:15.461799+0000","last_active":"2024-03-30T15:08:02.707474+0000","last_peered":"2024-03-30T15:08:02.707474+0000","last_clean":"2024-03-30T15:08:02.707474+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:08:02.707474+0000","last_undegraded":"2024-03-30T15:08:02.707474+0000","last_fullsized":"2024-03-30T15:08:02.707474+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:15.461699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:15.461699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1160,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488564+0000","last_change":"2024-03-30T15:06:53.604745+0000","last_active":"2024-03-30T15:08:03.488564+0000","last_peered":"2024-03-30T15:08:03.488564+0000","last_clean":"2024-03-30T15:08:03.488564+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:08:03.488564+0000","last_undegraded":"2024-03-30T15:08:03.488564+0000","last_fullsized":"2024-03-30T15:08:03.488564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:11.232804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488367+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:08:03.488367+0000","last_peered":"2024-03-30T15:08:03.488367+0000","last_clean":"2024-03-30T15:08:03.488367+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:08:03.488367+0000","last_undegraded":"2024-03-30T15:08:03.488367+0000","last_fullsized":"2024-03-30T15:08:03.488367+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1121,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348496+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:08:03.348496+0000","last_peered":"2024-03-30T15:08:03.348496+0000","last_clean":"2024-03-30T15:08:03.348496+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:08:03.348496+0000","last_undegraded":"2024-03-30T15:08:03.348496+0000","last_fullsized":"2024-03-30T15:08:03.348496+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1167,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517508+0000","last_change":"2024-03-30T15:06:00.028025+0000","last_active":"2024-03-30T15:08:02.517508+0000","last_peered":"2024-03-30T15:08:02.517508+0000","last_clean":"2024-03-30T15:08:02.517508+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:08:02.517508+0000","last_undegraded":"2024-03-30T15:08:02.517508+0000","last_fullsized":"2024-03-30T15:08:02.517508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.027914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.027914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1162,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348692+0000","last_change":"2024-03-30T15:05:21.130956+0000","last_active":"2024-03-30T15:08:03.348692+0000","last_peered":"2024-03-30T15:08:03.348692+0000","last_clean":"2024-03-30T15:08:03.348692+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:08:03.348692+0000","last_undegraded":"2024-03-30T15:08:03.348692+0000","last_fullsized":"2024-03-30T15:08:03.348692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:21.130862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:21.130862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1073,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488596+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:08:03.488596+0000","last_peered":"2024-03-30T15:08:03.488596+0000","last_clean":"2024-03-30T15:08:03.488596+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:08:03.488596+0000","last_undegraded":"2024-03-30T15:08:03.488596+0000","last_fullsized":"2024-03-30T15:08:03.488596+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1075,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347447+0000","last_change":"2024-03-30T15:06:16.046921+0000","last_active":"2024-03-30T15:08:02.347447+0000","last_peered":"2024-03-30T15:08:02.347447+0000","last_clean":"2024-03-30T15:08:02.347447+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:08:02.347447+0000","last_undegraded":"2024-03-30T15:08:02.347447+0000","last_fullsized":"2024-03-30T15:08:02.347447+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:16.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.046809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":753,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.750301+0000","last_change":"2024-03-30T15:07:22.646408+0000","last_active":"2024-03-30T15:08:02.750301+0000","last_peered":"2024-03-30T15:08:02.750301+0000","last_clean":"2024-03-30T15:08:02.750301+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:08:02.750301+0000","last_undegraded":"2024-03-30T15:08:02.750301+0000","last_fullsized":"2024-03-30T15:08:02.750301+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:07:22.646314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.646314+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:30.332030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1172,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489393+0000","last_change":"2024-03-30T15:04:37.543419+0000","last_active":"2024-03-30T15:08:03.489393+0000","last_peered":"2024-03-30T15:08:03.489393+0000","last_clean":"2024-03-30T15:08:03.489393+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:08:03.489393+0000","last_undegraded":"2024-03-30T15:08:03.489393+0000","last_fullsized":"2024-03-30T15:08:03.489393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:37.543307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:04:37.543307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1176,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514711+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:08:03.514711+0000","last_peered":"2024-03-30T15:08:03.514711+0000","last_clean":"2024-03-30T15:08:03.514711+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:08:03.514711+0000","last_undegraded":"2024-03-30T15:08:03.514711+0000","last_fullsized":"2024-03-30T15:08:03.514711+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1158,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348573+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:08:03.348573+0000","last_peered":"2024-03-30T15:08:03.348573+0000","last_clean":"2024-03-30T15:08:03.348573+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:08:03.348573+0000","last_undegraded":"2024-03-30T15:08:03.348573+0000","last_fullsized":"2024-03-30T15:08:03.348573+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515166+0000","last_change":"2024-03-30T15:07:33.284903+0000","last_active":"2024-03-30T15:08:03.515166+0000","last_peered":"2024-03-30T15:08:03.515166+0000","last_clean":"2024-03-30T15:08:03.515166+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:08:03.515166+0000","last_undegraded":"2024-03-30T15:08:03.515166+0000","last_fullsized":"2024-03-30T15:08:03.515166+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.284814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.284814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.915368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1209,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515494+0000","last_change":"2024-03-30T15:05:59.994488+0000","last_active":"2024-03-30T15:08:03.515494+0000","last_peered":"2024-03-30T15:08:03.515494+0000","last_clean":"2024-03-30T15:08:03.515494+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:08:03.515494+0000","last_undegraded":"2024-03-30T15:08:03.515494+0000","last_fullsized":"2024-03-30T15:08:03.515494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:59.994304+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.994304+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1074,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488568+0000","last_change":"2024-03-30T15:08:00.523833+0000","last_active":"2024-03-30T15:08:03.488568+0000","last_peered":"2024-03-30T15:08:03.488568+0000","last_clean":"2024-03-30T15:08:03.488568+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:08:03.488568+0000","last_undegraded":"2024-03-30T15:08:03.488568+0000","last_fullsized":"2024-03-30T15:08:03.488568+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.174749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1165,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489673+0000","last_change":"2024-03-30T15:07:28.601931+0000","last_active":"2024-03-30T15:08:03.489673+0000","last_peered":"2024-03-30T15:08:03.489673+0000","last_clean":"2024-03-30T15:08:03.489673+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:08:03.489673+0000","last_undegraded":"2024-03-30T15:08:03.489673+0000","last_fullsized":"2024-03-30T15:08:03.489673+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.601815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.601815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:50.989405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1173,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349667+0000","last_change":"2024-03-30T15:08:02.083375+0000","last_active":"2024-03-30T15:08:02.349667+0000","last_peered":"2024-03-30T15:08:02.349667+0000","last_clean":"2024-03-30T15:08:02.349667+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:08:02.349667+0000","last_undegraded":"2024-03-30T15:08:02.349667+0000","last_fullsized":"2024-03-30T15:08:02.349667+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.083294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.083294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.435219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1082,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515075+0000","last_change":"2024-03-30T15:06:14.085407+0000","last_active":"2024-03-30T15:08:03.515075+0000","last_peered":"2024-03-30T15:08:03.515075+0000","last_clean":"2024-03-30T15:08:03.515075+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:08:03.515075+0000","last_undegraded":"2024-03-30T15:08:03.515075+0000","last_fullsized":"2024-03-30T15:08:03.515075+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.085299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1204,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489505+0000","last_change":"2024-03-30T15:07:25.577161+0000","last_active":"2024-03-30T15:08:03.489505+0000","last_peered":"2024-03-30T15:08:03.489505+0000","last_clean":"2024-03-30T15:08:03.489505+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:08:03.489505+0000","last_undegraded":"2024-03-30T15:08:03.489505+0000","last_fullsized":"2024-03-30T15:08:03.489505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:07:25.577074+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.577074+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.741789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1179,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347667+0000","last_change":"2024-03-30T15:07:30.031279+0000","last_active":"2024-03-30T15:08:02.347667+0000","last_peered":"2024-03-30T15:08:02.347667+0000","last_clean":"2024-03-30T15:08:02.347667+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:08:02.347667+0000","last_undegraded":"2024-03-30T15:08:02.347667+0000","last_fullsized":"2024-03-30T15:08:02.347667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.031154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.145354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1350,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515170+0000","last_change":"2024-03-30T15:05:20.066039+0000","last_active":"2024-03-30T15:08:03.515170+0000","last_peered":"2024-03-30T15:08:03.515170+0000","last_clean":"2024-03-30T15:08:03.515170+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:08:03.515170+0000","last_undegraded":"2024-03-30T15:08:03.515170+0000","last_fullsized":"2024-03-30T15:08:03.515170+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:20.065886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.065886+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1168,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349323+0000","last_change":"2024-03-30T15:06:48.193765+0000","last_active":"2024-03-30T15:08:03.349323+0000","last_peered":"2024-03-30T15:08:03.349323+0000","last_clean":"2024-03-30T15:08:03.349323+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:08:03.349323+0000","last_undegraded":"2024-03-30T15:08:03.349323+0000","last_fullsized":"2024-03-30T15:08:03.349323+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:06:48.193663+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.193663+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1090,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.514955+0000","last_change":"2024-03-30T15:07:57.453748+0000","last_active":"2024-03-30T15:08:03.514955+0000","last_peered":"2024-03-30T15:08:03.514955+0000","last_clean":"2024-03-30T15:08:03.514955+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:08:03.514955+0000","last_undegraded":"2024-03-30T15:08:03.514955+0000","last_fullsized":"2024-03-30T15:08:03.514955+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.453629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.453629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.047448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1129,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349155+0000","last_change":"2024-03-30T15:07:21.894935+0000","last_active":"2024-03-30T15:08:02.349155+0000","last_peered":"2024-03-30T15:08:02.349155+0000","last_clean":"2024-03-30T15:08:02.349155+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:08:02.349155+0000","last_undegraded":"2024-03-30T15:08:02.349155+0000","last_fullsized":"2024-03-30T15:08:02.349155+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:07:21.894855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.894855+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.454380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1167,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707581+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:08:02.707581+0000","last_peered":"2024-03-30T15:08:02.707581+0000","last_clean":"2024-03-30T15:08:02.707581+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:08:02.707581+0000","last_undegraded":"2024-03-30T15:08:02.707581+0000","last_fullsized":"2024-03-30T15:08:02.707581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1159,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708036+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:08:02.708036+0000","last_peered":"2024-03-30T15:08:02.708036+0000","last_clean":"2024-03-30T15:08:02.708036+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:08:02.708036+0000","last_undegraded":"2024-03-30T15:08:02.708036+0000","last_fullsized":"2024-03-30T15:08:02.708036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1086,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515550+0000","last_change":"2024-03-30T15:07:36.255445+0000","last_active":"2024-03-30T15:08:03.515550+0000","last_peered":"2024-03-30T15:08:03.515550+0000","last_clean":"2024-03-30T15:08:03.515550+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:08:03.515550+0000","last_undegraded":"2024-03-30T15:08:03.515550+0000","last_fullsized":"2024-03-30T15:08:03.515550+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.255369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.255369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:58.024102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1073,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.518316+0000","last_change":"2024-03-30T15:07:28.134811+0000","last_active":"2024-03-30T15:08:02.518316+0000","last_peered":"2024-03-30T15:08:02.518316+0000","last_clean":"2024-03-30T15:08:02.518316+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:08:02.518316+0000","last_undegraded":"2024-03-30T15:08:02.518316+0000","last_fullsized":"2024-03-30T15:08:02.518316+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.134705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.134705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.521025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1243,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.349578+0000","last_change":"2024-03-30T15:07:59.295056+0000","last_active":"2024-03-30T15:08:03.349578+0000","last_peered":"2024-03-30T15:08:03.349578+0000","last_clean":"2024-03-30T15:08:03.349578+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:08:03.349578+0000","last_undegraded":"2024-03-30T15:08:03.349578+0000","last_fullsized":"2024-03-30T15:08:03.349578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:59.294962+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.294962+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.470863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1172,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489176+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:08:03.489176+0000","last_peered":"2024-03-30T15:08:03.489176+0000","last_clean":"2024-03-30T15:08:03.489176+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:08:03.489176+0000","last_undegraded":"2024-03-30T15:08:03.489176+0000","last_fullsized":"2024-03-30T15:08:03.489176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1177,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.349572+0000","last_change":"2024-03-30T15:08:00.131070+0000","last_active":"2024-03-30T15:08:02.349572+0000","last_peered":"2024-03-30T15:08:02.349572+0000","last_clean":"2024-03-30T15:08:02.349572+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:08:02.349572+0000","last_undegraded":"2024-03-30T15:08:02.349572+0000","last_fullsized":"2024-03-30T15:08:02.349572+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.130991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.130991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.398245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1156,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.708081+0000","last_change":"2024-03-30T15:06:12.558953+0000","last_active":"2024-03-30T15:08:02.708081+0000","last_peered":"2024-03-30T15:08:02.708081+0000","last_clean":"2024-03-30T15:08:02.708081+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:08:02.708081+0000","last_undegraded":"2024-03-30T15:08:02.708081+0000","last_fullsized":"2024-03-30T15:08:02.708081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.558834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.558834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1075,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.348326+0000","last_change":"2024-03-30T15:06:04.136327+0000","last_active":"2024-03-30T15:08:02.348326+0000","last_peered":"2024-03-30T15:08:02.348326+0000","last_clean":"2024-03-30T15:08:02.348326+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:08:02.348326+0000","last_undegraded":"2024-03-30T15:08:02.348326+0000","last_fullsized":"2024-03-30T15:08:02.348326+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.136246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.136246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1200,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.706662+0000","last_change":"2024-03-30T15:06:50.584525+0000","last_active":"2024-03-30T15:08:02.706662+0000","last_peered":"2024-03-30T15:08:02.706662+0000","last_clean":"2024-03-30T15:08:02.706662+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:08:02.706662+0000","last_undegraded":"2024-03-30T15:08:02.706662+0000","last_fullsized":"2024-03-30T15:08:02.706662+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:06:50.584362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.584362+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.995082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1167,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517711+0000","last_change":"2024-03-30T15:06:14.990566+0000","last_active":"2024-03-30T15:08:02.517711+0000","last_peered":"2024-03-30T15:08:02.517711+0000","last_clean":"2024-03-30T15:08:02.517711+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:08:02.517711+0000","last_undegraded":"2024-03-30T15:08:02.517711+0000","last_fullsized":"2024-03-30T15:08:02.517711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.990472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.990472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1162,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348544+0000","last_change":"2024-03-30T15:05:58.374684+0000","last_active":"2024-03-30T15:08:03.348544+0000","last_peered":"2024-03-30T15:08:03.348544+0000","last_clean":"2024-03-30T15:08:03.348544+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:08:03.348544+0000","last_undegraded":"2024-03-30T15:08:03.348544+0000","last_fullsized":"2024-03-30T15:08:03.348544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.374590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.374590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1163,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516669+0000","last_change":"2024-03-30T15:07:23.180715+0000","last_active":"2024-03-30T15:08:02.516669+0000","last_peered":"2024-03-30T15:08:02.516669+0000","last_clean":"2024-03-30T15:08:02.516669+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:08:02.516669+0000","last_undegraded":"2024-03-30T15:08:02.516669+0000","last_fullsized":"2024-03-30T15:08:02.516669+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.180625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.180625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:23.772874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1069,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347877+0000","last_change":"2024-03-30T15:06:14.104026+0000","last_active":"2024-03-30T15:08:02.347877+0000","last_peered":"2024-03-30T15:08:02.347877+0000","last_clean":"2024-03-30T15:08:02.347877+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:08:02.347877+0000","last_undegraded":"2024-03-30T15:08:02.347877+0000","last_fullsized":"2024-03-30T15:08:02.347877+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.103910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.103910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1188,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707564+0000","last_change":"2024-03-30T15:06:42.502254+0000","last_active":"2024-03-30T15:08:02.707564+0000","last_peered":"2024-03-30T15:08:02.707564+0000","last_clean":"2024-03-30T15:08:02.707564+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:08:02.707564+0000","last_undegraded":"2024-03-30T15:08:02.707564+0000","last_fullsized":"2024-03-30T15:08:02.707564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:42.502138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.502138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1167,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517381+0000","last_change":"2024-03-30T15:06:43.006252+0000","last_active":"2024-03-30T15:08:02.517381+0000","last_peered":"2024-03-30T15:08:02.517381+0000","last_clean":"2024-03-30T15:08:02.517381+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:08:02.517381+0000","last_undegraded":"2024-03-30T15:08:02.517381+0000","last_fullsized":"2024-03-30T15:08:02.517381+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.006124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.006124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1159,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.517103+0000","last_change":"2024-03-30T15:05:26.176781+0000","last_active":"2024-03-30T15:08:02.517103+0000","last_peered":"2024-03-30T15:08:02.517103+0000","last_clean":"2024-03-30T15:08:02.517103+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:08:02.517103+0000","last_undegraded":"2024-03-30T15:08:02.517103+0000","last_fullsized":"2024-03-30T15:08:02.517103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:26.176703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.176703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1160,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488129+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:08:03.488129+0000","last_peered":"2024-03-30T15:08:03.488129+0000","last_clean":"2024-03-30T15:08:03.488129+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:08:03.488129+0000","last_undegraded":"2024-03-30T15:08:03.488129+0000","last_fullsized":"2024-03-30T15:08:03.488129+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1073,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.516516+0000","last_change":"2024-03-30T15:07:33.165675+0000","last_active":"2024-03-30T15:08:02.516516+0000","last_peered":"2024-03-30T15:08:02.516516+0000","last_clean":"2024-03-30T15:08:02.516516+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:08:02.516516+0000","last_undegraded":"2024-03-30T15:08:02.516516+0000","last_fullsized":"2024-03-30T15:08:02.516516+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.165599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.165599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.109467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1193,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.348761+0000","last_change":"2024-03-30T15:07:29.102384+0000","last_active":"2024-03-30T15:08:03.348761+0000","last_peered":"2024-03-30T15:08:03.348761+0000","last_clean":"2024-03-30T15:08:03.348761+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:08:03.348761+0000","last_undegraded":"2024-03-30T15:08:03.348761+0000","last_fullsized":"2024-03-30T15:08:03.348761+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:29.102268+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.102268+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.919896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1080,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.515071+0000","last_change":"2024-03-30T15:06:17.187196+0000","last_active":"2024-03-30T15:08:03.515071+0000","last_peered":"2024-03-30T15:08:03.515071+0000","last_clean":"2024-03-30T15:08:03.515071+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:08:03.515071+0000","last_undegraded":"2024-03-30T15:08:03.515071+0000","last_fullsized":"2024-03-30T15:08:03.515071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:17.187094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:06:17.187094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1162,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.706982+0000","last_change":"2024-03-30T15:06:40.495863+0000","last_active":"2024-03-30T15:08:02.706982+0000","last_peered":"2024-03-30T15:08:02.706982+0000","last_clean":"2024-03-30T15:08:02.706982+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:08:02.706982+0000","last_undegraded":"2024-03-30T15:08:02.706982+0000","last_fullsized":"2024-03-30T15:08:02.706982+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:40.495702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.495702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":741,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.750698+0000","last_change":"2024-03-30T15:07:26.673461+0000","last_active":"2024-03-30T15:08:02.750698+0000","last_peered":"2024-03-30T15:08:02.750698+0000","last_clean":"2024-03-30T15:08:02.750698+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:08:02.750698+0000","last_undegraded":"2024-03-30T15:08:02.750698+0000","last_fullsized":"2024-03-30T15:08:02.750698+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.673362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.673362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.939016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1070,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.488909+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:08:03.488909+0000","last_peered":"2024-03-30T15:08:03.488909+0000","last_clean":"2024-03-30T15:08:03.488909+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:08:03.488909+0000","last_undegraded":"2024-03-30T15:08:03.488909+0000","last_fullsized":"2024-03-30T15:08:03.488909+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1212,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707106+0000","last_change":"2024-03-30T15:07:30.921961+0000","last_active":"2024-03-30T15:08:02.707106+0000","last_peered":"2024-03-30T15:08:02.707106+0000","last_clean":"2024-03-30T15:08:02.707106+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:08:02.707106+0000","last_undegraded":"2024-03-30T15:08:02.707106+0000","last_fullsized":"2024-03-30T15:08:02.707106+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:30.921763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.921763+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:31.154514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1181,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-30T15:08:03.489286+0000","last_change":"2024-03-30T15:07:35.571081+0000","last_active":"2024-03-30T15:08:03.489286+0000","last_peered":"2024-03-30T15:08:03.489286+0000","last_clean":"2024-03-30T15:08:03.489286+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:08:03.489286+0000","last_undegraded":"2024-03-30T15:08:03.489286+0000","last_fullsized":"2024-03-30T15:08:03.489286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.970991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1169,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.347453+0000","last_change":"2024-03-30T15:07:32.100519+0000","last_active":"2024-03-30T15:08:02.347453+0000","last_peered":"2024-03-30T15:08:02.347453+0000","last_clean":"2024-03-30T15:08:02.347453+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:08:02.347453+0000","last_undegraded":"2024-03-30T15:08:02.347453+0000","last_fullsized":"2024-03-30T15:08:02.347453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.100436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.100436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.419009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1155,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-30T15:08:02.707429+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:08:02.707429+0000","last_peered":"2024-03-30T15:08:02.707429+0000","last_clean":"2024-03-30T15:08:02.707429+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:08:02.707429+0000","last_undegraded":"2024-03-30T15:08:02.707429+0000","last_fullsized":"2024-03-30T15:08:02.707429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":798,"num_read_kb":793,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":317,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":320,"num_pg":8,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":24,"acting":24,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005222,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71948,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673204,"statfs":{"total":95995035648,"available":95921360896,"internally_reserved":0,"allocated":4784128,"data_stored":2238545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299563,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75768,"kb_used_data":4324,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93669384,"statfs":{"total":95995035648,"available":95917449216,"internally_reserved":0,"allocated":4427776,"data_stored":1908535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71640,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673512,"statfs":{"total":95995035648,"available":95921676288,"internally_reserved":0,"allocated":4464640,"data_stored":1913825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738569,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49420,"kb_used_data":4352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93695732,"statfs":{"total":95995035648,"available":95944429568,"internally_reserved":0,"allocated":4456448,"data_stored":1914300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94068,"kb_used_data":4584,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93651084,"statfs":{"total":95995035648,"available":95898710016,"internally_reserved":0,"allocated":4694016,"data_stored":2171553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117179,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94416,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650736,"statfs":{"total":95995035648,"available":95898353664,"internally_reserved":0,"allocated":5050368,"data_stored":2490181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953653,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93832,"kb_used_data":4348,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93651320,"statfs":{"total":95995035648,"available":95898951680,"internally_reserved":0,"allocated":4452352,"data_stored":1909525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659312,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76360,"kb_used_data":4924,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668792,"statfs":{"total":95995035648,"available":95916843008,"internally_reserved":0,"allocated":5042176,"data_stored":2503678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":317,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":320,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:08:09.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:08:09.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:09.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:09.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:09.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:09.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:10.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:10.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:10.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:10.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:10.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:10.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:10.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:10.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:10.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 -- 172.21.15.28:0/1550000146 >> v1:172.21.15.28:6789/0 conn(0x7efdbc10d960 legacy=0x7efdbc10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 -- 172.21.15.28:0/1550000146 shutdown_connections 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 -- 172.21.15.28:0/1550000146 >> 172.21.15.28:0/1550000146 conn(0x7efdbc0fc9e0 msgr2=0x7efdbc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 -- 172.21.15.28:0/1550000146 shutdown_connections 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 -- 172.21.15.28:0/1550000146 wait complete. 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 librados: starting msgr at 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.632+0000 7efdc0d72700 1 librados: starting objecter 2024-03-30T15:08:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 Processor -- start 2024-03-30T15:08:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 -- start start 2024-03-30T15:08:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 librados: setting wanted keys 2024-03-30T15:08:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 librados: calling monclient init 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efdbc1a93b0 con 0x7efdbc109dc0 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efdbc1aa5d0 con 0x7efdbc10d960 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdc0d72700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efdbc1ab810 con 0x7efdbc101130 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdba59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7efdbc101130 0x7efdbc106440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49188/0 (socket says 172.21.15.28:49188) 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdba59c700 1 -- 172.21.15.28:0/1677980156 learned_addr learned my addr 172.21.15.28:0/1677980156 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdb9d9b700 1 --1- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6789/0 conn(0x7efdbc109dc0 0x7efdbc1a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47154/0 (socket says 172.21.15.28:47154) 2024-03-30T15:08:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdab7fe700 10 client.?.objecter ms_handle_connect 0x7efdbc109dc0 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdab7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdab7fe700 10 client.?.objecter ms_handle_connect 0x7efdbc101130 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.633+0000 7efdab7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 10 client.?.objecter ms_handle_connect 0x7efdbc10d960 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1747283859 0 0) 0x7efdbc1a93b0 con 0x7efdbc109dc0 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efda4003aa0 con 0x7efdbc109dc0 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760585385 0 0) 0x7efdbc1aa5d0 con 0x7efdbc10d960 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdbc1a93b0 con 0x7efdbc10d960 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886481871 0 0) 0x7efdbc1ab810 con 0x7efdbc101130 2024-03-30T15:08:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdbc1aa5d0 con 0x7efdbc101130 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1944156188 0 0) 0x7efda4003aa0 con 0x7efdbc109dc0 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdbc1ab810 con 0x7efdbc109dc0 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 223779617 0 0) 0x7efdbc1a93b0 con 0x7efdbc10d960 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efda4003aa0 con 0x7efdbc10d960 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3596898854 0 0) 0x7efdbc1aa5d0 con 0x7efdbc101130 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdbc1a93b0 con 0x7efdbc101130 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efdb00018b0 con 0x7efdbc109dc0 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efdb4001bd0 con 0x7efdbc10d960 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efdac001890 con 0x7efdbc101130 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2873714209 0 0) 0x7efda4003aa0 con 0x7efdbc10d960 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6790/0 conn(0x7efdbc101130 legacy=0x7efdbc106440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6789/0 conn(0x7efdbc109dc0 legacy=0x7efdbc1a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc1aca50 con 0x7efdbc10d960 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc1aba40 con 0x7efdbc10d960 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdc0d72700 10 client.50545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efdbc1abf00 con 0x7efdbc10d960 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdc0d72700 1 librados: init done 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7efdb4001360 con 0x7efdbc10d960 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.634+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efdb4004e90 con 0x7efdbc10d960 2024-03-30T15:08:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.635+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7efd9c005100 con 0x7efdbc10d960 2024-03-30T15:08:10.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.636+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7efdb4001d70 con 0x7efdbc10d960 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(641..641 src has 442..641) ==== 7870+0+0 (unknown 189132428 0 0) 0x7efdb40884b0 con 0x7efdbc10d960 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 osd_map(641..641 src has 442..641) 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 3 client.50545.objecter handle_osd_map got epochs [641,641] > 0 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 3 client.50545.objecter handle_osd_map decoding full epoch 641 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 20 client.50545.objecter dump_active .. 0 homeless 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v641) ==== 54+0+16 (unknown 2153875431 0 2420000192) 0x7efdb40888b0 con 0x7efdbc10d960 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efda40703e0 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 10 client.50545.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 1 2024-03-30T15:08:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 1 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 0 1 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 4 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 10 client.50545.objecter _send_command 1 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd9c004b50 con 0x7efd9c004730 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.637+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c004730 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.638+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c004b50 con 0x7efd9c004730 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.638+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 1: 0 ) 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.638+0000 7efdab7fe700 10 client.50545.objecter _finish_command 1 = system:0 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.638+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 4 2024-03-30T15:08:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.638+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 0 1 2024-03-30T15:08:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:10.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 10 client.50545.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 2 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 2 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c0053e0 osd=0 3 2024-03-30T15:08:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 0 2 2024-03-30T15:08:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 4 2024-03-30T15:08:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 10 client.50545.objecter _send_command 2 2024-03-30T15:08:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.655+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c0079f0 con 0x7efd9c004730 2024-03-30T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.1 scrub starts 2024-03-30T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.1 scrub ok 2024-03-30T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.d deep-scrub starts 2024-03-30T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.d deep-scrub ok 2024-03-30T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: pgmap v943: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.1f scrub starts 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: 42.1f scrub ok 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.1 scrub starts 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.1 scrub ok 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.d deep-scrub starts 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.d deep-scrub ok 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: pgmap v943: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.1f scrub starts 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: 42.1f scrub ok 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:10 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c0079f0 con 0x7efd9c004730 2024-03-30T15:08:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdab7fe700 10 client.50545.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 4 2024-03-30T15:08:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 0 2 2024-03-30T15:08:10.662 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:08:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdc0d72700 10 client.50545.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.657+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 1 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 4 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 10 client.50545.objecter _send_command 3 2024-03-30T15:08:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7efd9c002180 con 0x7efd9c004280 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.658+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c004280 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.659+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c002180 con 0x7efd9c004280 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.659+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 3: 0 ) 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.659+0000 7efdab7fe700 10 client.50545.objecter _finish_command 3 = system:0 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.659+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 4 2024-03-30T15:08:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.659+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 1 3 2024-03-30T15:08:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 10 client.50545.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 4 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 4 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c004110 osd=1 3 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 1 4 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 4 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 10 client.50545.objecter _send_command 4 2024-03-30T15:08:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.676+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c001600 con 0x7efd9c004280 2024-03-30T15:08:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c001600 con 0x7efd9c004280 2024-03-30T15:08:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 10 client.50545.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 4 2024-03-30T15:08:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 1 4 2024-03-30T15:08:10.680 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:08:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 10 client.50545.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 5 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 5 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 2 5 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 4 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 10 client.50545.objecter _send_command 5 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7efd9c0110e0 con 0x7efd9c00d5d0 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.678+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c00d5d0 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.679+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c0110e0 con 0x7efd9c00d5d0 2024-03-30T15:08:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.679+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 5: 0 ) 2024-03-30T15:08:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.679+0000 7efdab7fe700 10 client.50545.objecter _finish_command 5 = system:0 2024-03-30T15:08:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.679+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 4 2024-03-30T15:08:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.679+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 2 5 2024-03-30T15:08:10.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.1 scrub starts 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.1 scrub ok 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.d deep-scrub starts 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.d deep-scrub ok 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: pgmap v943: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.1f scrub starts 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: 42.1f scrub ok 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025908142' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi028-93253-104"}]': finished 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: osdmap e641: 8 total, 8 up, 8 in 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:10.684 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.684 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:10 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 10 client.50545.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 6 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 6 2024-03-30T15:08:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c008370 osd=2 3 2024-03-30T15:08:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 2 6 2024-03-30T15:08:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 4 2024-03-30T15:08:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 10 client.50545.objecter _send_command 6 2024-03-30T15:08:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.696+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c007e50 con 0x7efd9c00d5d0 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.697+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c007e50 con 0x7efd9c00d5d0 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.697+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.697+0000 7efdab7fe700 10 client.50545.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.697+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 4 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.697+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 2 6 2024-03-30T15:08:10.699 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 10 client.50545.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 7 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 7 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 3 7 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 4 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 10 client.50545.objecter _send_command 7 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7efd9c015f20 con 0x7efd9c012410 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.698+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c012410 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.699+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c015f20 con 0x7efd9c012410 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.699+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 7: 0 ) 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.699+0000 7efdab7fe700 10 client.50545.objecter _finish_command 7 = system:0 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.699+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 4 2024-03-30T15:08:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.699+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 3 7 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 10 client.50545.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 8 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 8 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00cc20 osd=3 3 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 3 8 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 4 2024-03-30T15:08:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 10 client.50545.objecter _send_command 8 2024-03-30T15:08:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.716+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c00c700 con 0x7efd9c012410 2024-03-30T15:08:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c00c700 con 0x7efd9c012410 2024-03-30T15:08:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdab7fe700 10 client.50545.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 4 2024-03-30T15:08:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 3 8 2024-03-30T15:08:10.719 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:08:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 10 client.50545.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 9 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 9 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 4 9 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 4 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 10 client.50545.objecter _send_command 9 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.717+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7efd9c00c6e0 con 0x7efd9c011f70 2024-03-30T15:08:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.718+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c011f70 2024-03-30T15:08:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.719+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c00c6e0 con 0x7efd9c011f70 2024-03-30T15:08:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.719+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 9: 0 ) 2024-03-30T15:08:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.719+0000 7efdab7fe700 10 client.50545.objecter _finish_command 9 = system:0 2024-03-30T15:08:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.719+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 4 2024-03-30T15:08:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.719+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 4 9 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 10 client.50545.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 10 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 10 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c4b0 osd=4 3 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 4 10 2024-03-30T15:08:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 4 2024-03-30T15:08:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 10 client.50545.objecter _send_command 10 2024-03-30T15:08:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.737+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c01c150 con 0x7efd9c011f70 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c01c150 con 0x7efd9c011f70 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdab7fe700 10 client.50545.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 4 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 4 10 2024-03-30T15:08:10.740 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 10 client.50545.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 11 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 11 2024-03-30T15:08:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 5 11 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 4 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 10 client.50545.objecter _send_command 11 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.739+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7efd9c01cd20 con 0x7efd9c01c870 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c01c870 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c01cd20 con 0x7efd9c01c870 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 11: 0 ) 2024-03-30T15:08:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 10 client.50545.objecter _finish_command 11 = system:0 2024-03-30T15:08:10.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 4 2024-03-30T15:08:10.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.740+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 5 11 2024-03-30T15:08:10.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 10 client.50545.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 12 2024-03-30T15:08:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 12 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c740 osd=5 3 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 5 12 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 4 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 10 client.50545.objecter _send_command 12 2024-03-30T15:08:10.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.757+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c01cf00 con 0x7efd9c01c870 2024-03-30T15:08:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c01cf00 con 0x7efd9c01c870 2024-03-30T15:08:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdab7fe700 10 client.50545.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 4 2024-03-30T15:08:10.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 5 12 2024-03-30T15:08:10.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:08:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 10 client.50545.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 13 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 13 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 6 13 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 4 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 10 client.50545.objecter _send_command 13 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.759+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7efd9c023140 con 0x7efd9c01cef0 2024-03-30T15:08:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.760+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c01cef0 2024-03-30T15:08:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.761+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd9c023140 con 0x7efd9c01cef0 2024-03-30T15:08:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.761+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 13: 0 ) 2024-03-30T15:08:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.761+0000 7efdab7fe700 10 client.50545.objecter _finish_command 13 = system:0 2024-03-30T15:08:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.761+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 4 2024-03-30T15:08:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.761+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 6 13 2024-03-30T15:08:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:10.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 10 client.50545.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 14 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 14 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c01cd80 osd=6 3 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 6 14 2024-03-30T15:08:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 4 2024-03-30T15:08:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 10 client.50545.objecter _send_command 14 2024-03-30T15:08:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.778+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c00c9d0 con 0x7efd9c01cef0 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c00c9d0 con 0x7efd9c01cef0 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 10 client.50545.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 4 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 6 14 2024-03-30T15:08:10.781 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:08:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 10 client.50545.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 15 2024-03-30T15:08:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 15 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 7 15 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 4 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 10 client.50545.objecter _send_command 15 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7efd9c024900 con 0x7efd9c0168a0 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.780+0000 7efdab7fe700 10 client.50545.objecter ms_handle_connect 0x7efd9c0168a0 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.781+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efdac006570 con 0x7efd9c0168a0 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.781+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 15: 0 ) 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.781+0000 7efdab7fe700 10 client.50545.objecter _finish_command 15 = system:0 2024-03-30T15:08:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.781+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 4 2024-03-30T15:08:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.781+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 7 15 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 10 client.50545.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign -1 16 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 20 client.50545.objecter _get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_remove -1 16 2024-03-30T15:08:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 20 client.50545.objecter get_session s=0x7efd9c00c9d0 osd=7 3 2024-03-30T15:08:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 15 client.50545.objecter _session_command_op_assign 7 16 2024-03-30T15:08:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 4 2024-03-30T15:08:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 10 client.50545.objecter _send_command 16 2024-03-30T15:08:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.799+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7efd9c016e90 con 0x7efd9c0168a0 2024-03-30T15:08:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdab7fe700 1 -- 172.21.15.28:0/1677980156 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efd9c016e90 con 0x7efd9c0168a0 2024-03-30T15:08:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdab7fe700 10 client.50545.objecter ms_dispatch 0x7efdbc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdab7fe700 10 client.50545.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdab7fe700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 4 2024-03-30T15:08:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdab7fe700 15 client.50545.objecter _session_command_op_remove 7 16 2024-03-30T15:08:10.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:08:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 librados: watch_flush enter 2024-03-30T15:08:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 librados: watch_flush exit 2024-03-30T15:08:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.0 2024-03-30T15:08:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6801/3394323575 conn(0x7efd9c004730 legacy=0x7efd9c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c0053e0 osd=0 1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6805/1850313547 conn(0x7efd9c004280 legacy=0x7efd9c00ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c004110 osd=1 1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.2 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6809/3451325043 conn(0x7efd9c00d5d0 legacy=0x7efd9c00f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c008370 osd=2 1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.3 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6813/1506157250 conn(0x7efd9c012410 legacy=0x7efd9c014800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00cc20 osd=3 1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.4 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.801+0000 7efdbad9d700 1 -- 172.21.15.28:0/1677980156 reap_dead start 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.134:6800/3063775514 conn(0x7efd9c011f70 legacy=0x7efd9c019260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c4b0 osd=4 1 2024-03-30T15:08:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdbad9d700 1 -- 172.21.15.28:0/1677980156 reap_dead start 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.5 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.134:6804/3668151179 conn(0x7efd9c01c870 legacy=0x7efd9c01a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c740 osd=5 1 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.6 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.134:6808/2135224194 conn(0x7efd9c01cef0 legacy=0x7efd9c021890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c01cd80 osd=6 1 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 10 client.50545.objecter close_session for osd.7 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.134:6812/3758190298 conn(0x7efd9c0168a0 legacy=0x7efd9c023140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 20 client.50545.objecter put_session s=0x7efd9c00c9d0 osd=7 1 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 20 client.50545.objecter shutdown clearing up homeless session... 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 10 client.50545.objecter successfully canceled tick 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.28:6800/3011431093 conn(0x7efda40703e0 legacy=0x7efda40728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> v1:172.21.15.134:6789/0 conn(0x7efdbc10d960 legacy=0x7efdbc1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdbad9d700 1 -- 172.21.15.28:0/1677980156 reap_dead start 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 shutdown_connections 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 >> 172.21.15.28:0/1677980156 conn(0x7efdbc0fc9e0 msgr2=0x7efdbc1105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 shutdown_connections 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 -- 172.21.15.28:0/1677980156 wait complete. 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:10.802+0000 7efdc0d72700 1 librados: shutdown 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:08:10.807 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:08:10.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:10.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:10.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:11.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:11.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:11.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:11.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:11.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:11.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:11.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:11.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:11.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1677980156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:08:11.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:08:11.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1677980156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:11 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:11.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1677980156' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:11.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi028-93253-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T15:08:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: osdmap e642: 8 total, 8 up, 8 in 2024-03-30T15:08:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:11 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:11.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:11.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:11.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:11.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:12.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:12.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:12.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:12.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:12.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:12.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:12.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:12.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:12.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:12.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:12.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:12.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:12.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:12.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:12.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: 6.11 deep-scrub starts 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: 6.11 deep-scrub ok 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: pgmap v946: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:12 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:12.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:12.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: 6.11 deep-scrub starts 2024-03-30T15:08:13.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: 6.11 deep-scrub ok 2024-03-30T15:08:13.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:08:13.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:08:13.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: pgmap v946: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:13.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:13.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: 6.11 deep-scrub starts 2024-03-30T15:08:13.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: 6.11 deep-scrub ok 2024-03-30T15:08:13.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:08:13.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:08:13.007 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: pgmap v946: 206 pgs: 206 active+clean; 582 KiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-30T15:08:13.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:12 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:13.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:13.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:13.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:13.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:08:13.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:13.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:13.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:13.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:13.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:13.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:13.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T15:08:13.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T15:08:13.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T15:08:13.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:08:13.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 42.13 scrub starts 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: 42.13 scrub ok 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[29876]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 42.13 scrub starts 2024-03-30T15:08:13.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: 42.13 scrub ok 2024-03-30T15:08:13.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:13.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:13 smithi028 ceph-mon[41614]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: osdmap e643: 8 total, 8 up, 8 in 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 42.13 scrub starts 2024-03-30T15:08:13.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: 42.13 scrub ok 2024-03-30T15:08:13.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi028-93253-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:13.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:13 smithi134 ceph-mon[33714]: osdmap e644: 8 total, 8 up, 8 in 2024-03-30T15:08:13.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:13.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:13.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:14.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:08:14.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 -- 172.21.15.28:0/1230907573 >> v1:172.21.15.28:6790/0 conn(0x7f6430101130 legacy=0x7f6430101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 -- 172.21.15.28:0/1230907573 shutdown_connections 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 -- 172.21.15.28:0/1230907573 >> 172.21.15.28:0/1230907573 conn(0x7f64300fc9e0 msgr2=0x7f64300fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 -- 172.21.15.28:0/1230907573 shutdown_connections 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 -- 172.21.15.28:0/1230907573 wait complete. 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 librados: starting msgr at 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 librados: starting objecter 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.100+0000 7f6437fd0700 1 Processor -- start 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 -- start start 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 librados: setting wanted keys 2024-03-30T15:08:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 librados: calling monclient init 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6430106880 con 0x7f6430101130 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64301a8540 con 0x7f6430109dc0 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6437fd0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64301a9720 con 0x7f643010d960 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6435d6c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6430101130 0x7f64301046a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47176/0 (socket says 172.21.15.28:47176) 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6435d6c700 1 -- 172.21.15.28:0/1573929774 learned_addr learned my addr 172.21.15.28:0/1573929774 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:14.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter ms_handle_connect 0x7f6430101130 2024-03-30T15:08:14.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:14.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter ms_handle_connect 0x7f643010d960 2024-03-30T15:08:14.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:14.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter ms_handle_connect 0x7f6430109dc0 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403549038 0 0) 0x7f6430106880 con 0x7f6430101130 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f641c0031f0 con 0x7f6430101130 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616192432 0 0) 0x7f64301a9720 con 0x7f643010d960 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6430106880 con 0x7f643010d960 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618883404 0 0) 0x7f64301a8540 con 0x7f6430109dc0 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64301a9720 con 0x7f6430109dc0 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546903690 0 0) 0x7f641c0031f0 con 0x7f6430101130 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64301a8540 con 0x7f6430101130 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f642c001370 con 0x7f6430101130 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.101+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3662868036 0 0) 0x7f6430106880 con 0x7f643010d960 2024-03-30T15:08:14.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f641c0031f0 con 0x7f643010d960 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6428001bf0 con 0x7f643010d960 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3961358795 0 0) 0x7f64301a9720 con 0x7f6430109dc0 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6430106880 con 0x7f6430109dc0 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6420001870 con 0x7f6430109dc0 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3407706761 0 0) 0x7f64301a8540 con 0x7f6430101130 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 >> v1:172.21.15.28:6790/0 conn(0x7f643010d960 legacy=0x7f64301a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 >> v1:172.21.15.134:6789/0 conn(0x7f6430109dc0 legacy=0x7f6430104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64301aa900 con 0x7f6430101130 2024-03-30T15:08:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64301a9950 con 0x7f6430101130 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6437fd0700 10 client.25347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64301a9e10 con 0x7f6430101130 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6437fd0700 1 librados: init done 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f642c001af0 con 0x7f6430101130 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.102+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f642c004cd0 con 0x7f6430101130 2024-03-30T15:08:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.103+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6414005120 con 0x7f6430101130 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f642c003860 con 0x7f6430101130 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(644..644 src has 442..644) ==== 8729+0+0 (unknown 1048187277 0 0) 0x7f642c089800 con 0x7f6430101130 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 10 client.25347.objecter ms_dispatch 0x7f6430001550 osd_map(644..644 src has 442..644) 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 3 client.25347.objecter handle_osd_map got epochs [644,644] > 0 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 3 client.25347.objecter handle_osd_map decoding full epoch 644 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.104+0000 7f6426ffd700 20 client.25347.objecter dump_active .. 0 homeless 2024-03-30T15:08:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.105+0000 7f6426ffd700 10 client.25347.objecter ms_handle_connect 0x7f641c070520 2024-03-30T15:08:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.109+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f642c0567d0 con 0x7f6430101130 2024-03-30T15:08:14.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:14.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.252 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:14.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:14.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.324+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6414000b90 con 0x7f641c070520 2024-03-30T15:08:14.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.339+0000 7f6426ffd700 1 -- 172.21.15.28:0/1573929774 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+551240 (unknown 2965378022 0 1097456562) 0x7f6414000b90 con 0x7f641c070520 2024-03-30T15:08:14.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 10 librados: watch_flush enter 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 10 librados: watch_flush exit 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 20 client.25347.objecter shutdown clearing up homeless session... 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 10 client.25347.objecter successfully canceled tick 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 >> v1:172.21.15.28:6800/3011431093 conn(0x7f641c070520 legacy=0x7f641c0729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 >> v1:172.21.15.28:6789/0 conn(0x7f6430101130 legacy=0x7f64301046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 shutdown_connections 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.343+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 >> 172.21.15.28:0/1573929774 conn(0x7f64300fc9e0 msgr2=0x7f6430110e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.344+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 shutdown_connections 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.344+0000 7f6437fd0700 1 -- 172.21.15.28:0/1573929774 wait complete. 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:14.344+0000 7f6437fd0700 1 librados: shutdown 2024-03-30T15:08:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:08:14.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:14.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:14.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:14.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:14.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:14.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:14.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:14.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:14.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: pgmap v949: 214 pgs: 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:14.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: pgmap v949: 214 pgs: 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: pgmap v949: 214 pgs: 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:08:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:08:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: osdmap e645: 8 total, 8 up, 8 in 2024-03-30T15:08:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:15.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:15.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:15.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:15.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:15.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:15.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:15.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:15.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:15.422 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":949,"stamp":"2024-03-30T15:08:14.066118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1962,"num_read_kb":1714,"num_write":729,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2251,"ondisk_log_size":2251,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":618,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":628748,"kb_used_data":37768,"kb_used_omap":12,"kb_used_meta":590835,"kb_avail":749332468,"statfs":{"total":767960285184,"available":767316447232,"internally_reserved":0,"allocated":38674432,"data_stored":17811005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":605015600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000892"},"pg_stats":[{"pgid":"322.7","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.359794+0000","last_change":"2024-03-30T15:08:13.359794+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.359794+0000","last_undegraded":"2024-03-30T15:08:13.359794+0000","last_fullsized":"2024-03-30T15:08:13.359794+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"322.6","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.357319+0000","last_change":"2024-03-30T15:08:13.357319+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.357319+0000","last_undegraded":"2024-03-30T15:08:13.357319+0000","last_fullsized":"2024-03-30T15:08:13.357319+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"322.5","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.355155+0000","last_change":"2024-03-30T15:08:13.355155+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.355155+0000","last_undegraded":"2024-03-30T15:08:13.355155+0000","last_fullsized":"2024-03-30T15:08:13.355155+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"322.4","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.356719+0000","last_change":"2024-03-30T15:08:13.356719+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.356719+0000","last_undegraded":"2024-03-30T15:08:13.356719+0000","last_fullsized":"2024-03-30T15:08:13.356719+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.3","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.355137+0000","last_change":"2024-03-30T15:08:13.355137+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.355137+0000","last_undegraded":"2024-03-30T15:08:13.355137+0000","last_fullsized":"2024-03-30T15:08:13.355137+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"322.2","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.356708+0000","last_change":"2024-03-30T15:08:13.356708+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.356708+0000","last_undegraded":"2024-03-30T15:08:13.356708+0000","last_fullsized":"2024-03-30T15:08:13.356708+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.1","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.356720+0000","last_change":"2024-03-30T15:08:13.356720+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.356720+0000","last_undegraded":"2024-03-30T15:08:13.356720+0000","last_fullsized":"2024-03-30T15:08:13.356720+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.0","version":"0'0","reported_seq":1,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-03-30T15:08:13.356815+0000","last_change":"2024-03-30T15:08:13.356815+0000","last_active":"2024-03-30T15:08:13.347849+0000","last_peered":"2024-03-30T15:08:13.347849+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T15:08:13.356815+0000","last_undegraded":"2024-03-30T15:08:13.356815+0000","last_fullsized":"2024-03-30T15:08:13.356815+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363246+0000","last_change":"2024-03-30T15:08:00.661723+0000","last_active":"2024-03-30T15:08:13.363246+0000","last_peered":"2024-03-30T15:08:13.363246+0000","last_clean":"2024-03-30T15:08:13.363246+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:08:13.363246+0000","last_undegraded":"2024-03-30T15:08:13.363246+0000","last_fullsized":"2024-03-30T15:08:13.363246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.661646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.661646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:23.913327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354045+0000","last_change":"2024-03-30T15:07:25.409282+0000","last_active":"2024-03-30T15:08:13.354045+0000","last_peered":"2024-03-30T15:08:13.354045+0000","last_clean":"2024-03-30T15:08:13.354045+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:08:13.354045+0000","last_undegraded":"2024-03-30T15:08:13.354045+0000","last_fullsized":"2024-03-30T15:08:13.354045+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.409160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.409160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.520458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362181+0000","last_change":"2024-03-30T15:08:00.188376+0000","last_active":"2024-03-30T15:08:13.362181+0000","last_peered":"2024-03-30T15:08:13.362181+0000","last_clean":"2024-03-30T15:08:13.362181+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:08:13.362181+0000","last_undegraded":"2024-03-30T15:08:13.362181+0000","last_fullsized":"2024-03-30T15:08:13.362181+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.188301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.188301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.659221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361734+0000","last_change":"2024-03-30T15:06:56.895628+0000","last_active":"2024-03-30T15:08:13.361734+0000","last_peered":"2024-03-30T15:08:13.361734+0000","last_clean":"2024-03-30T15:08:13.361734+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:08:13.361734+0000","last_undegraded":"2024-03-30T15:08:13.361734+0000","last_fullsized":"2024-03-30T15:08:13.361734+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.895505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.895505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":1083,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353767+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:08:13.353767+0000","last_peered":"2024-03-30T15:08:13.353767+0000","last_clean":"2024-03-30T15:08:13.353767+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:08:13.353767+0000","last_undegraded":"2024-03-30T15:08:13.353767+0000","last_fullsized":"2024-03-30T15:08:13.353767+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":1106,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361026+0000","last_change":"2024-03-30T15:07:57.201822+0000","last_active":"2024-03-30T15:08:13.361026+0000","last_peered":"2024-03-30T15:08:13.361026+0000","last_clean":"2024-03-30T15:08:13.361026+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:08:13.361026+0000","last_undegraded":"2024-03-30T15:08:13.361026+0000","last_fullsized":"2024-03-30T15:08:13.361026+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.201691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.201691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:05.630818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360605+0000","last_change":"2024-03-30T15:08:13.183528+0000","last_active":"2024-03-30T15:08:13.360605+0000","last_peered":"2024-03-30T15:08:13.360605+0000","last_clean":"2024-03-30T15:08:13.360605+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:08:13.360605+0000","last_undegraded":"2024-03-30T15:08:13.360605+0000","last_fullsized":"2024-03-30T15:08:13.360605+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.183373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.183373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.888133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361574+0000","last_change":"2024-03-30T15:06:59.257326+0000","last_active":"2024-03-30T15:08:13.361574+0000","last_peered":"2024-03-30T15:08:13.361574+0000","last_clean":"2024-03-30T15:08:13.361574+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:08:13.361574+0000","last_undegraded":"2024-03-30T15:08:13.361574+0000","last_fullsized":"2024-03-30T15:08:13.361574+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.257230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.257230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:25.024266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":1088,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359006+0000","last_change":"2024-03-30T15:07:36.042362+0000","last_active":"2024-03-30T15:08:13.359006+0000","last_peered":"2024-03-30T15:08:13.359006+0000","last_clean":"2024-03-30T15:08:13.359006+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:08:13.359006+0000","last_undegraded":"2024-03-30T15:08:13.359006+0000","last_fullsized":"2024-03-30T15:08:13.359006+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.042281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.352269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360351+0000","last_change":"2024-03-30T15:06:18.054438+0000","last_active":"2024-03-30T15:08:13.360351+0000","last_peered":"2024-03-30T15:08:13.360351+0000","last_clean":"2024-03-30T15:08:13.360351+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:08:13.360351+0000","last_undegraded":"2024-03-30T15:08:13.360351+0000","last_fullsized":"2024-03-30T15:08:13.360351+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:18.054358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:06:18.054358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362812+0000","last_change":"2024-03-30T15:08:10.138813+0000","last_active":"2024-03-30T15:08:13.362812+0000","last_peered":"2024-03-30T15:08:13.362812+0000","last_clean":"2024-03-30T15:08:13.362812+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:08:13.362812+0000","last_undegraded":"2024-03-30T15:08:13.362812+0000","last_fullsized":"2024-03-30T15:08:13.362812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.138657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.138657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.317256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":1104,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362786+0000","last_change":"2024-03-30T15:07:32.132841+0000","last_active":"2024-03-30T15:08:13.362786+0000","last_peered":"2024-03-30T15:08:13.362786+0000","last_clean":"2024-03-30T15:08:13.362786+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:08:13.362786+0000","last_undegraded":"2024-03-30T15:08:13.362786+0000","last_fullsized":"2024-03-30T15:08:13.362786+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.132758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.132758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.404108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":1081,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358753+0000","last_change":"2024-03-30T15:05:19.327818+0000","last_active":"2024-03-30T15:08:13.358753+0000","last_peered":"2024-03-30T15:08:13.358753+0000","last_clean":"2024-03-30T15:08:13.358753+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:08:13.358753+0000","last_undegraded":"2024-03-30T15:08:13.358753+0000","last_fullsized":"2024-03-30T15:08:13.358753+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.327720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.327720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357617+0000","last_change":"2024-03-30T15:06:43.837698+0000","last_active":"2024-03-30T15:08:13.357617+0000","last_peered":"2024-03-30T15:08:13.357617+0000","last_clean":"2024-03-30T15:08:13.357617+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:08:13.357617+0000","last_undegraded":"2024-03-30T15:08:13.357617+0000","last_fullsized":"2024-03-30T15:08:13.357617+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.837619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.837619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362312+0000","last_change":"2024-03-30T15:06:54.767293+0000","last_active":"2024-03-30T15:08:13.362312+0000","last_peered":"2024-03-30T15:08:13.362312+0000","last_clean":"2024-03-30T15:08:13.362312+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:08:13.362312+0000","last_undegraded":"2024-03-30T15:08:13.362312+0000","last_fullsized":"2024-03-30T15:08:13.362312+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.767213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.767213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.877720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363668+0000","last_change":"2024-03-30T15:07:25.988108+0000","last_active":"2024-03-30T15:08:13.363668+0000","last_peered":"2024-03-30T15:08:13.363668+0000","last_clean":"2024-03-30T15:08:13.363668+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:08:13.363668+0000","last_undegraded":"2024-03-30T15:08:13.363668+0000","last_fullsized":"2024-03-30T15:08:13.363668+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.988025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.988025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.890511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361377+0000","last_change":"2024-03-30T15:07:59.150468+0000","last_active":"2024-03-30T15:08:13.361377+0000","last_peered":"2024-03-30T15:08:13.361377+0000","last_clean":"2024-03-30T15:08:13.361377+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:08:13.361377+0000","last_undegraded":"2024-03-30T15:08:13.361377+0000","last_fullsized":"2024-03-30T15:08:13.361377+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:59.150361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.150361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.062144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":1088,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360366+0000","last_change":"2024-03-30T15:07:25.964793+0000","last_active":"2024-03-30T15:08:13.360366+0000","last_peered":"2024-03-30T15:08:13.360366+0000","last_clean":"2024-03-30T15:08:13.360366+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:08:13.360366+0000","last_undegraded":"2024-03-30T15:08:13.360366+0000","last_fullsized":"2024-03-30T15:08:13.360366+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.964714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.964714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:31.307648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363915+0000","last_change":"2024-03-30T15:06:58.308106+0000","last_active":"2024-03-30T15:08:13.363915+0000","last_peered":"2024-03-30T15:08:13.363915+0000","last_clean":"2024-03-30T15:08:13.363915+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:08:13.363915+0000","last_undegraded":"2024-03-30T15:08:13.363915+0000","last_fullsized":"2024-03-30T15:08:13.363915+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.308015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.308015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.528138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361967+0000","last_change":"2024-03-30T15:07:35.172101+0000","last_active":"2024-03-30T15:08:13.361967+0000","last_peered":"2024-03-30T15:08:13.361967+0000","last_clean":"2024-03-30T15:08:13.361967+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:08:13.361967+0000","last_undegraded":"2024-03-30T15:08:13.361967+0000","last_fullsized":"2024-03-30T15:08:13.361967+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.171990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.171990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:02.621474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":1099,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358129+0000","last_change":"2024-03-30T15:08:09.224705+0000","last_active":"2024-03-30T15:08:13.358129+0000","last_peered":"2024-03-30T15:08:13.358129+0000","last_clean":"2024-03-30T15:08:13.358129+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:08:13.358129+0000","last_undegraded":"2024-03-30T15:08:13.358129+0000","last_fullsized":"2024-03-30T15:08:13.358129+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.224499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.224499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.632314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358746+0000","last_change":"2024-03-30T15:06:58.979164+0000","last_active":"2024-03-30T15:08:13.358746+0000","last_peered":"2024-03-30T15:08:13.358746+0000","last_clean":"2024-03-30T15:08:13.358746+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:08:13.358746+0000","last_undegraded":"2024-03-30T15:08:13.358746+0000","last_fullsized":"2024-03-30T15:08:13.358746+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.979074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.979074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:22.628955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353577+0000","last_change":"2024-03-30T15:08:08.551682+0000","last_active":"2024-03-30T15:08:13.353577+0000","last_peered":"2024-03-30T15:08:13.353577+0000","last_clean":"2024-03-30T15:08:13.353577+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:08:13.353577+0000","last_undegraded":"2024-03-30T15:08:13.353577+0000","last_fullsized":"2024-03-30T15:08:13.353577+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.551547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.551547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.564084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362569+0000","last_change":"2024-03-30T15:06:45.791752+0000","last_active":"2024-03-30T15:08:13.362569+0000","last_peered":"2024-03-30T15:08:13.362569+0000","last_clean":"2024-03-30T15:08:13.362569+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:08:13.362569+0000","last_undegraded":"2024-03-30T15:08:13.362569+0000","last_fullsized":"2024-03-30T15:08:13.362569+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.791667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.791667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354111+0000","last_change":"2024-03-30T15:08:09.548782+0000","last_active":"2024-03-30T15:08:13.354111+0000","last_peered":"2024-03-30T15:08:13.354111+0000","last_clean":"2024-03-30T15:08:13.354111+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:08:13.354111+0000","last_undegraded":"2024-03-30T15:08:13.354111+0000","last_fullsized":"2024-03-30T15:08:13.354111+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.548687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.755962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":205,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362599+0000","last_change":"2024-03-30T15:06:18.488509+0000","last_active":"2024-03-30T15:08:13.362599+0000","last_peered":"2024-03-30T15:08:13.362599+0000","last_clean":"2024-03-30T15:08:13.362599+0000","last_became_active":"2024-03-30T15:06:18.488207+0000","last_became_peered":"2024-03-30T15:06:18.488207+0000","last_unstale":"2024-03-30T15:08:13.362599+0000","last_undegraded":"2024-03-30T15:08:13.362599+0000","last_fullsized":"2024-03-30T15:08:13.362599+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.176687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.176687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.364164+0000","last_change":"2024-03-30T15:08:02.711973+0000","last_active":"2024-03-30T15:08:13.364164+0000","last_peered":"2024-03-30T15:08:13.364164+0000","last_clean":"2024-03-30T15:08:13.364164+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:08:13.364164+0000","last_undegraded":"2024-03-30T15:08:13.364164+0000","last_fullsized":"2024-03-30T15:08:13.364164+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.711881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.711881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.583184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363291+0000","last_change":"2024-03-30T15:07:26.321108+0000","last_active":"2024-03-30T15:08:13.363291+0000","last_peered":"2024-03-30T15:08:13.363291+0000","last_clean":"2024-03-30T15:08:13.363291+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:08:13.363291+0000","last_undegraded":"2024-03-30T15:08:13.363291+0000","last_fullsized":"2024-03-30T15:08:13.363291+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.321030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:38.839544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353082+0000","last_change":"2024-03-30T15:07:31.354747+0000","last_active":"2024-03-30T15:08:13.353082+0000","last_peered":"2024-03-30T15:08:13.353082+0000","last_clean":"2024-03-30T15:08:13.353082+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:08:13.353082+0000","last_undegraded":"2024-03-30T15:08:13.353082+0000","last_fullsized":"2024-03-30T15:08:13.353082+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.354669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.354669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.082266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":1090,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360319+0000","last_change":"2024-03-30T15:06:56.238490+0000","last_active":"2024-03-30T15:08:13.360319+0000","last_peered":"2024-03-30T15:08:13.360319+0000","last_clean":"2024-03-30T15:08:13.360319+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:08:13.360319+0000","last_undegraded":"2024-03-30T15:08:13.360319+0000","last_fullsized":"2024-03-30T15:08:13.360319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.238398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.238398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.509153+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":1088,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358522+0000","last_change":"2024-03-30T15:07:19.822016+0000","last_active":"2024-03-30T15:08:13.358522+0000","last_peered":"2024-03-30T15:08:13.358522+0000","last_clean":"2024-03-30T15:08:13.358522+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:08:13.358522+0000","last_undegraded":"2024-03-30T15:08:13.358522+0000","last_fullsized":"2024-03-30T15:08:13.358522+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.821886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.821886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.082006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358933+0000","last_change":"2024-03-30T15:07:58.173279+0000","last_active":"2024-03-30T15:08:13.358933+0000","last_peered":"2024-03-30T15:08:13.358933+0000","last_clean":"2024-03-30T15:08:13.358933+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:08:13.358933+0000","last_undegraded":"2024-03-30T15:08:13.358933+0000","last_fullsized":"2024-03-30T15:08:13.358933+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.173193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.173193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:11.398411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360843+0000","last_change":"2024-03-30T15:06:49.183096+0000","last_active":"2024-03-30T15:08:13.360843+0000","last_peered":"2024-03-30T15:08:13.360843+0000","last_clean":"2024-03-30T15:08:13.360843+0000","last_became_active":"2024-03-30T15:06:48.338292+0000","last_became_peered":"2024-03-30T15:06:48.338292+0000","last_unstale":"2024-03-30T15:08:13.360843+0000","last_undegraded":"2024-03-30T15:08:13.360843+0000","last_fullsized":"2024-03-30T15:08:13.360843+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.183009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.183009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354200+0000","last_change":"2024-03-30T15:06:43.474123+0000","last_active":"2024-03-30T15:08:13.354200+0000","last_peered":"2024-03-30T15:08:13.354200+0000","last_clean":"2024-03-30T15:08:13.354200+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:08:13.354200+0000","last_undegraded":"2024-03-30T15:08:13.354200+0000","last_fullsized":"2024-03-30T15:08:13.354200+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.474015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.474015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1223,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360471+0000","last_change":"2024-03-30T15:08:06.185130+0000","last_active":"2024-03-30T15:08:13.360471+0000","last_peered":"2024-03-30T15:08:13.360471+0000","last_clean":"2024-03-30T15:08:13.360471+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:08:13.360471+0000","last_undegraded":"2024-03-30T15:08:13.360471+0000","last_fullsized":"2024-03-30T15:08:13.360471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:08:06.185037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.185037+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.018400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1093,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353167+0000","last_change":"2024-03-30T15:06:11.536378+0000","last_active":"2024-03-30T15:08:13.353167+0000","last_peered":"2024-03-30T15:08:13.353167+0000","last_clean":"2024-03-30T15:08:13.353167+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:08:13.353167+0000","last_undegraded":"2024-03-30T15:08:13.353167+0000","last_fullsized":"2024-03-30T15:08:13.353167+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":374,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354027+0000","last_change":"2024-03-30T15:06:58.353375+0000","last_active":"2024-03-30T15:08:13.354027+0000","last_peered":"2024-03-30T15:08:13.354027+0000","last_clean":"2024-03-30T15:08:13.354027+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:08:13.354027+0000","last_undegraded":"2024-03-30T15:08:13.354027+0000","last_fullsized":"2024-03-30T15:08:13.354027+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'9","last_scrub_stamp":"2024-03-30T15:06:58.353220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.353220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1176,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360229+0000","last_change":"2024-03-30T15:06:44.519216+0000","last_active":"2024-03-30T15:08:13.360229+0000","last_peered":"2024-03-30T15:08:13.360229+0000","last_clean":"2024-03-30T15:08:13.360229+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:13.360229+0000","last_undegraded":"2024-03-30T15:08:13.360229+0000","last_fullsized":"2024-03-30T15:08:13.360229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:44.519122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.519122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":475,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358536+0000","last_change":"2024-03-30T15:06:49.951825+0000","last_active":"2024-03-30T15:08:13.358536+0000","last_peered":"2024-03-30T15:08:13.358536+0000","last_clean":"2024-03-30T15:08:13.358536+0000","last_became_active":"2024-03-30T15:06:48.340482+0000","last_became_peered":"2024-03-30T15:06:48.340482+0000","last_unstale":"2024-03-30T15:08:13.358536+0000","last_undegraded":"2024-03-30T15:08:13.358536+0000","last_fullsized":"2024-03-30T15:08:13.358536+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.951714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.951714+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354284+0000","last_change":"2024-03-30T15:07:22.415753+0000","last_active":"2024-03-30T15:08:13.354284+0000","last_peered":"2024-03-30T15:08:13.354284+0000","last_clean":"2024-03-30T15:08:13.354284+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:08:13.354284+0000","last_undegraded":"2024-03-30T15:08:13.354284+0000","last_fullsized":"2024-03-30T15:08:13.354284+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.415581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.415581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:25.352643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1242,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359946+0000","last_change":"2024-03-30T15:07:20.849551+0000","last_active":"2024-03-30T15:08:13.359946+0000","last_peered":"2024-03-30T15:08:13.359946+0000","last_clean":"2024-03-30T15:08:13.359946+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:08:13.359946+0000","last_undegraded":"2024-03-30T15:08:13.359946+0000","last_fullsized":"2024-03-30T15:08:13.359946+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:07:20.849431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.849431+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:22.083359+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1205,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353279+0000","last_change":"2024-03-30T15:07:34.300735+0000","last_active":"2024-03-30T15:08:13.353279+0000","last_peered":"2024-03-30T15:08:13.353279+0000","last_clean":"2024-03-30T15:08:13.353279+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:08:13.353279+0000","last_undegraded":"2024-03-30T15:08:13.353279+0000","last_fullsized":"2024-03-30T15:08:13.353279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:34.300632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.300632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.024216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1200,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360768+0000","last_change":"2024-03-30T15:07:19.878091+0000","last_active":"2024-03-30T15:08:13.360768+0000","last_peered":"2024-03-30T15:08:13.360768+0000","last_clean":"2024-03-30T15:08:13.360768+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:08:13.360768+0000","last_undegraded":"2024-03-30T15:08:13.360768+0000","last_fullsized":"2024-03-30T15:08:13.360768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.878011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.878011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.486305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1205,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363292+0000","last_change":"2024-03-30T15:08:02.197580+0000","last_active":"2024-03-30T15:08:13.363292+0000","last_peered":"2024-03-30T15:08:13.363292+0000","last_clean":"2024-03-30T15:08:13.363292+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:08:13.363292+0000","last_undegraded":"2024-03-30T15:08:13.363292+0000","last_fullsized":"2024-03-30T15:08:13.363292+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.197498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.197498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.675581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1105,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362041+0000","last_change":"2024-03-30T15:08:05.112361+0000","last_active":"2024-03-30T15:08:13.362041+0000","last_peered":"2024-03-30T15:08:13.362041+0000","last_clean":"2024-03-30T15:08:13.362041+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:08:13.362041+0000","last_undegraded":"2024-03-30T15:08:13.362041+0000","last_fullsized":"2024-03-30T15:08:13.362041+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:05.112284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:05.112284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:08.797988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1229,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360173+0000","last_change":"2024-03-30T15:07:58.225113+0000","last_active":"2024-03-30T15:08:13.360173+0000","last_peered":"2024-03-30T15:08:13.360173+0000","last_clean":"2024-03-30T15:08:13.360173+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:08:13.360173+0000","last_undegraded":"2024-03-30T15:08:13.360173+0000","last_fullsized":"2024-03-30T15:08:13.360173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:07:58.225002+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.225002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.514446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1197,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358168+0000","last_change":"2024-03-30T15:07:33.014237+0000","last_active":"2024-03-30T15:08:13.358168+0000","last_peered":"2024-03-30T15:08:13.358168+0000","last_clean":"2024-03-30T15:08:13.358168+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:08:13.358168+0000","last_undegraded":"2024-03-30T15:08:13.358168+0000","last_fullsized":"2024-03-30T15:08:13.358168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.014124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.688496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1181,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363146+0000","last_change":"2024-03-30T15:05:58.982494+0000","last_active":"2024-03-30T15:08:13.363146+0000","last_peered":"2024-03-30T15:08:13.363146+0000","last_clean":"2024-03-30T15:08:13.363146+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:08:13.363146+0000","last_undegraded":"2024-03-30T15:08:13.363146+0000","last_fullsized":"2024-03-30T15:08:13.363146+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.982407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.982407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1106,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354220+0000","last_change":"2024-03-30T15:08:10.524135+0000","last_active":"2024-03-30T15:08:13.354220+0000","last_peered":"2024-03-30T15:08:13.354220+0000","last_clean":"2024-03-30T15:08:13.354220+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:08:13.354220+0000","last_undegraded":"2024-03-30T15:08:13.354220+0000","last_fullsized":"2024-03-30T15:08:13.354220+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.524046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.524046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.049104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":475,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358702+0000","last_change":"2024-03-30T15:06:50.920479+0000","last_active":"2024-03-30T15:08:13.358702+0000","last_peered":"2024-03-30T15:08:13.358702+0000","last_clean":"2024-03-30T15:08:13.358702+0000","last_became_active":"2024-03-30T15:06:48.342510+0000","last_became_peered":"2024-03-30T15:06:48.342510+0000","last_unstale":"2024-03-30T15:08:13.358702+0000","last_undegraded":"2024-03-30T15:08:13.358702+0000","last_fullsized":"2024-03-30T15:08:13.358702+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:50.920389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.920389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1105,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362436+0000","last_change":"2024-03-30T15:08:06.129401+0000","last_active":"2024-03-30T15:08:13.362436+0000","last_peered":"2024-03-30T15:08:13.362436+0000","last_clean":"2024-03-30T15:08:13.362436+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:08:13.362436+0000","last_undegraded":"2024-03-30T15:08:13.362436+0000","last_fullsized":"2024-03-30T15:08:13.362436+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.129319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.129319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:13.993935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1246,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362547+0000","last_change":"2024-03-30T15:07:28.992483+0000","last_active":"2024-03-30T15:08:13.362547+0000","last_peered":"2024-03-30T15:08:13.362547+0000","last_clean":"2024-03-30T15:08:13.362547+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:08:13.362547+0000","last_undegraded":"2024-03-30T15:08:13.362547+0000","last_fullsized":"2024-03-30T15:08:13.362547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:28.992348+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.992348+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:29.581423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1101,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358253+0000","last_change":"2024-03-30T15:07:22.928759+0000","last_active":"2024-03-30T15:08:13.358253+0000","last_peered":"2024-03-30T15:08:13.358253+0000","last_clean":"2024-03-30T15:08:13.358253+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:08:13.358253+0000","last_undegraded":"2024-03-30T15:08:13.358253+0000","last_fullsized":"2024-03-30T15:08:13.358253+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.928677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.928677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.321766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1353,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360201+0000","last_change":"2024-03-30T15:06:06.615931+0000","last_active":"2024-03-30T15:08:13.360201+0000","last_peered":"2024-03-30T15:08:13.360201+0000","last_clean":"2024-03-30T15:08:13.360201+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:08:13.360201+0000","last_undegraded":"2024-03-30T15:08:13.360201+0000","last_fullsized":"2024-03-30T15:08:13.360201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:06.615780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.615780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1177,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361587+0000","last_change":"2024-03-30T15:06:57.951315+0000","last_active":"2024-03-30T15:08:13.361587+0000","last_peered":"2024-03-30T15:08:13.361587+0000","last_clean":"2024-03-30T15:08:13.361587+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:08:13.361587+0000","last_undegraded":"2024-03-30T15:08:13.361587+0000","last_fullsized":"2024-03-30T15:08:13.361587+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.951208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.951208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":476,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358695+0000","last_change":"2024-03-30T15:06:48.534844+0000","last_active":"2024-03-30T15:08:13.358695+0000","last_peered":"2024-03-30T15:08:13.358695+0000","last_clean":"2024-03-30T15:08:13.358695+0000","last_became_active":"2024-03-30T15:06:48.341420+0000","last_became_peered":"2024-03-30T15:06:48.341420+0000","last_unstale":"2024-03-30T15:08:13.358695+0000","last_undegraded":"2024-03-30T15:08:13.358695+0000","last_fullsized":"2024-03-30T15:08:13.358695+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.534763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.534763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362044+0000","last_change":"2024-03-30T15:07:36.166030+0000","last_active":"2024-03-30T15:08:13.362044+0000","last_peered":"2024-03-30T15:08:13.362044+0000","last_clean":"2024-03-30T15:08:13.362044+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:08:13.362044+0000","last_undegraded":"2024-03-30T15:08:13.362044+0000","last_fullsized":"2024-03-30T15:08:13.362044+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.165947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.824703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1189,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357989+0000","last_change":"2024-03-30T15:07:57.209970+0000","last_active":"2024-03-30T15:08:13.357989+0000","last_peered":"2024-03-30T15:08:13.357989+0000","last_clean":"2024-03-30T15:08:13.357989+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:08:13.357989+0000","last_undegraded":"2024-03-30T15:08:13.357989+0000","last_fullsized":"2024-03-30T15:08:13.357989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.209884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.209884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.514345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1208,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361032+0000","last_change":"2024-03-30T15:08:10.679605+0000","last_active":"2024-03-30T15:08:13.361032+0000","last_peered":"2024-03-30T15:08:13.361032+0000","last_clean":"2024-03-30T15:08:13.361032+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:08:13.361032+0000","last_undegraded":"2024-03-30T15:08:13.361032+0000","last_fullsized":"2024-03-30T15:08:13.361032+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:08:10.679491+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.679491+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:39.573167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1165,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363916+0000","last_change":"2024-03-30T15:05:59.324853+0000","last_active":"2024-03-30T15:08:13.363916+0000","last_peered":"2024-03-30T15:08:13.363916+0000","last_clean":"2024-03-30T15:08:13.363916+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:08:13.363916+0000","last_undegraded":"2024-03-30T15:08:13.363916+0000","last_fullsized":"2024-03-30T15:08:13.363916+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:05:59.324757+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.324757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1185,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358450+0000","last_change":"2024-03-30T15:08:10.175409+0000","last_active":"2024-03-30T15:08:13.358450+0000","last_peered":"2024-03-30T15:08:13.358450+0000","last_clean":"2024-03-30T15:08:13.358450+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:08:13.358450+0000","last_undegraded":"2024-03-30T15:08:13.358450+0000","last_fullsized":"2024-03-30T15:08:13.358450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.175319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.192582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":475,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360827+0000","last_change":"2024-03-30T15:06:51.940521+0000","last_active":"2024-03-30T15:08:13.360827+0000","last_peered":"2024-03-30T15:08:13.360827+0000","last_clean":"2024-03-30T15:08:13.360827+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:08:13.360827+0000","last_undegraded":"2024-03-30T15:08:13.360827+0000","last_fullsized":"2024-03-30T15:08:13.360827+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:51.940375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:51.940375+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1113,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362287+0000","last_change":"2024-03-30T15:07:24.023700+0000","last_active":"2024-03-30T15:08:13.362287+0000","last_peered":"2024-03-30T15:08:13.362287+0000","last_clean":"2024-03-30T15:08:13.362287+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:08:13.362287+0000","last_undegraded":"2024-03-30T15:08:13.362287+0000","last_fullsized":"2024-03-30T15:08:13.362287+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.023617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.023617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.066284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1242,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357759+0000","last_change":"2024-03-30T15:06:11.179123+0000","last_active":"2024-03-30T15:08:13.357759+0000","last_peered":"2024-03-30T15:08:13.357759+0000","last_clean":"2024-03-30T15:08:13.357759+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:08:13.357759+0000","last_undegraded":"2024-03-30T15:08:13.357759+0000","last_fullsized":"2024-03-30T15:08:13.357759+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.179008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.179008+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1235,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358277+0000","last_change":"2024-03-30T15:06:41.454580+0000","last_active":"2024-03-30T15:08:13.358277+0000","last_peered":"2024-03-30T15:08:13.358277+0000","last_clean":"2024-03-30T15:08:13.358277+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:08:13.358277+0000","last_undegraded":"2024-03-30T15:08:13.358277+0000","last_fullsized":"2024-03-30T15:08:13.358277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:41.454445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:41.454445+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1194,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354239+0000","last_change":"2024-03-30T15:07:24.562748+0000","last_active":"2024-03-30T15:08:13.354239+0000","last_peered":"2024-03-30T15:08:13.354239+0000","last_clean":"2024-03-30T15:08:13.354239+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:08:13.354239+0000","last_undegraded":"2024-03-30T15:08:13.354239+0000","last_fullsized":"2024-03-30T15:08:13.354239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.710249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1187,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361837+0000","last_change":"2024-03-30T15:06:39.147213+0000","last_active":"2024-03-30T15:08:13.361837+0000","last_peered":"2024-03-30T15:08:13.361837+0000","last_clean":"2024-03-30T15:08:13.361837+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:08:13.361837+0000","last_undegraded":"2024-03-30T15:08:13.361837+0000","last_fullsized":"2024-03-30T15:08:13.361837+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.147113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.147113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":482,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354451+0000","last_change":"2024-03-30T15:06:49.387252+0000","last_active":"2024-03-30T15:08:13.354451+0000","last_peered":"2024-03-30T15:08:13.354451+0000","last_clean":"2024-03-30T15:08:13.354451+0000","last_became_active":"2024-03-30T15:06:48.342151+0000","last_became_peered":"2024-03-30T15:06:48.342151+0000","last_unstale":"2024-03-30T15:08:13.354451+0000","last_undegraded":"2024-03-30T15:08:13.354451+0000","last_fullsized":"2024-03-30T15:08:13.354451+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.387153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.387153+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1110,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363506+0000","last_change":"2024-03-30T15:07:25.015706+0000","last_active":"2024-03-30T15:08:13.363506+0000","last_peered":"2024-03-30T15:08:13.363506+0000","last_clean":"2024-03-30T15:08:13.363506+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:08:13.363506+0000","last_undegraded":"2024-03-30T15:08:13.363506+0000","last_fullsized":"2024-03-30T15:08:13.363506+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.015600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.015600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.857225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1364,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353125+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:08:13.353125+0000","last_peered":"2024-03-30T15:08:13.353125+0000","last_clean":"2024-03-30T15:08:13.353125+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:08:13.353125+0000","last_undegraded":"2024-03-30T15:08:13.353125+0000","last_fullsized":"2024-03-30T15:08:13.353125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1239,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359265+0000","last_change":"2024-03-30T15:08:01.102530+0000","last_active":"2024-03-30T15:08:13.359265+0000","last_peered":"2024-03-30T15:08:13.359265+0000","last_clean":"2024-03-30T15:08:13.359265+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:08:13.359265+0000","last_undegraded":"2024-03-30T15:08:13.359265+0000","last_fullsized":"2024-03-30T15:08:13.359265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:08:01.102431+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.102431+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.685021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1197,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361849+0000","last_change":"2024-03-30T15:06:53.935771+0000","last_active":"2024-03-30T15:08:13.361849+0000","last_peered":"2024-03-30T15:08:13.361849+0000","last_clean":"2024-03-30T15:08:13.361849+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:08:13.361849+0000","last_undegraded":"2024-03-30T15:08:13.361849+0000","last_fullsized":"2024-03-30T15:08:13.361849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.935674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.935674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:19.007131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1180,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352803+0000","last_change":"2024-03-30T15:06:10.554067+0000","last_active":"2024-03-30T15:08:13.352803+0000","last_peered":"2024-03-30T15:08:13.352803+0000","last_clean":"2024-03-30T15:08:13.352803+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:08:13.352803+0000","last_undegraded":"2024-03-30T15:08:13.352803+0000","last_fullsized":"2024-03-30T15:08:13.352803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":482,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354129+0000","last_change":"2024-03-30T15:06:48.409534+0000","last_active":"2024-03-30T15:08:13.354129+0000","last_peered":"2024-03-30T15:08:13.354129+0000","last_clean":"2024-03-30T15:08:13.354129+0000","last_became_active":"2024-03-30T15:06:48.338506+0000","last_became_peered":"2024-03-30T15:06:48.338506+0000","last_unstale":"2024-03-30T15:08:13.354129+0000","last_undegraded":"2024-03-30T15:08:13.354129+0000","last_fullsized":"2024-03-30T15:08:13.354129+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.409455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.409455+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362668+0000","last_change":"2024-03-30T15:06:59.884862+0000","last_active":"2024-03-30T15:08:13.362668+0000","last_peered":"2024-03-30T15:08:13.362668+0000","last_clean":"2024-03-30T15:08:13.362668+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:08:13.362668+0000","last_undegraded":"2024-03-30T15:08:13.362668+0000","last_fullsized":"2024-03-30T15:08:13.362668+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.884709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.884709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1197,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353617+0000","last_change":"2024-03-30T15:06:54.318139+0000","last_active":"2024-03-30T15:08:13.353617+0000","last_peered":"2024-03-30T15:08:13.353617+0000","last_clean":"2024-03-30T15:08:13.353617+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:08:13.353617+0000","last_undegraded":"2024-03-30T15:08:13.353617+0000","last_fullsized":"2024-03-30T15:08:13.353617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.318005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1247,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354134+0000","last_change":"2024-03-30T15:08:02.516526+0000","last_active":"2024-03-30T15:08:13.354134+0000","last_peered":"2024-03-30T15:08:13.354134+0000","last_clean":"2024-03-30T15:08:13.354134+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:08:13.354134+0000","last_undegraded":"2024-03-30T15:08:13.354134+0000","last_fullsized":"2024-03-30T15:08:13.354134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:14.340123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1208,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360615+0000","last_change":"2024-03-30T15:07:17.809348+0000","last_active":"2024-03-30T15:08:13.360615+0000","last_peered":"2024-03-30T15:08:13.360615+0000","last_clean":"2024-03-30T15:08:13.360615+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:08:13.360615+0000","last_undegraded":"2024-03-30T15:08:13.360615+0000","last_fullsized":"2024-03-30T15:08:13.360615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:17.809228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:17.809228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:27.761242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1196,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358338+0000","last_change":"2024-03-30T15:06:39.844760+0000","last_active":"2024-03-30T15:08:13.358338+0000","last_peered":"2024-03-30T15:08:13.358338+0000","last_clean":"2024-03-30T15:08:13.358338+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:08:13.358338+0000","last_undegraded":"2024-03-30T15:08:13.358338+0000","last_fullsized":"2024-03-30T15:08:13.358338+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.844668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.844668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359477+0000","last_change":"2024-03-30T15:06:48.873065+0000","last_active":"2024-03-30T15:08:13.359477+0000","last_peered":"2024-03-30T15:08:13.359477+0000","last_clean":"2024-03-30T15:08:13.359477+0000","last_became_active":"2024-03-30T15:06:48.339258+0000","last_became_peered":"2024-03-30T15:06:48.339258+0000","last_unstale":"2024-03-30T15:08:13.359477+0000","last_undegraded":"2024-03-30T15:08:13.359477+0000","last_fullsized":"2024-03-30T15:08:13.359477+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.872974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.872974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353767+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:08:13.353767+0000","last_peered":"2024-03-30T15:08:13.353767+0000","last_clean":"2024-03-30T15:08:13.353767+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:08:13.353767+0000","last_undegraded":"2024-03-30T15:08:13.353767+0000","last_fullsized":"2024-03-30T15:08:13.353767+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1189,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363204+0000","last_change":"2024-03-30T15:06:52.973865+0000","last_active":"2024-03-30T15:08:13.363204+0000","last_peered":"2024-03-30T15:08:13.363204+0000","last_clean":"2024-03-30T15:08:13.363204+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:08:13.363204+0000","last_undegraded":"2024-03-30T15:08:13.363204+0000","last_fullsized":"2024-03-30T15:08:13.363204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.973770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.973770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:20.699838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1232,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353714+0000","last_change":"2024-03-30T15:08:01.499074+0000","last_active":"2024-03-30T15:08:13.353714+0000","last_peered":"2024-03-30T15:08:13.353714+0000","last_clean":"2024-03-30T15:08:13.353714+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:08:13.353714+0000","last_undegraded":"2024-03-30T15:08:13.353714+0000","last_fullsized":"2024-03-30T15:08:13.353714+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.212933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2020,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360662+0000","last_change":"2024-03-30T15:07:34.264898+0000","last_active":"2024-03-30T15:08:13.360662+0000","last_peered":"2024-03-30T15:08:13.360662+0000","last_clean":"2024-03-30T15:08:13.360662+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:08:13.360662+0000","last_undegraded":"2024-03-30T15:08:13.360662+0000","last_fullsized":"2024-03-30T15:08:13.360662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:07:34.264815+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.264815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.246868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":808,"num_read_kb":803,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1192,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353405+0000","last_change":"2024-03-30T15:07:29.592120+0000","last_active":"2024-03-30T15:08:13.353405+0000","last_peered":"2024-03-30T15:08:13.353405+0000","last_clean":"2024-03-30T15:08:13.353405+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:08:13.353405+0000","last_undegraded":"2024-03-30T15:08:13.353405+0000","last_fullsized":"2024-03-30T15:08:13.353405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.592039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.592039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.178771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360500+0000","last_change":"2024-03-30T15:06:45.854073+0000","last_active":"2024-03-30T15:08:13.360500+0000","last_peered":"2024-03-30T15:08:13.360500+0000","last_clean":"2024-03-30T15:08:13.360500+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:08:13.360500+0000","last_undegraded":"2024-03-30T15:08:13.360500+0000","last_fullsized":"2024-03-30T15:08:13.360500+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.853998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.853998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1095,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363538+0000","last_change":"2024-03-30T15:07:24.315658+0000","last_active":"2024-03-30T15:08:13.363538+0000","last_peered":"2024-03-30T15:08:13.363538+0000","last_clean":"2024-03-30T15:08:13.363538+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:08:13.363538+0000","last_undegraded":"2024-03-30T15:08:13.363538+0000","last_fullsized":"2024-03-30T15:08:13.363538+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.315584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.315584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:49.961946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1191,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363077+0000","last_change":"2024-03-30T15:08:06.704248+0000","last_active":"2024-03-30T15:08:13.363077+0000","last_peered":"2024-03-30T15:08:13.363077+0000","last_clean":"2024-03-30T15:08:13.363077+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:08:13.363077+0000","last_undegraded":"2024-03-30T15:08:13.363077+0000","last_fullsized":"2024-03-30T15:08:13.363077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.704123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.704123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.001588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1405,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354339+0000","last_change":"2024-03-30T15:06:02.076986+0000","last_active":"2024-03-30T15:08:13.354339+0000","last_peered":"2024-03-30T15:08:13.354339+0000","last_clean":"2024-03-30T15:08:13.354339+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:08:13.354339+0000","last_undegraded":"2024-03-30T15:08:13.354339+0000","last_fullsized":"2024-03-30T15:08:13.354339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:02.076864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.076864+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359160+0000","last_change":"2024-03-30T15:07:00.995237+0000","last_active":"2024-03-30T15:08:13.359160+0000","last_peered":"2024-03-30T15:08:13.359160+0000","last_clean":"2024-03-30T15:08:13.359160+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:08:13.359160+0000","last_undegraded":"2024-03-30T15:08:13.359160+0000","last_fullsized":"2024-03-30T15:08:13.359160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.995156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.995156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1468,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362512+0000","last_change":"2024-03-30T15:06:45.099476+0000","last_active":"2024-03-30T15:08:13.362512+0000","last_peered":"2024-03-30T15:08:13.362512+0000","last_clean":"2024-03-30T15:08:13.362512+0000","last_became_active":"2024-03-30T15:06:43.342924+0000","last_became_peered":"2024-03-30T15:06:43.342924+0000","last_unstale":"2024-03-30T15:08:13.362512+0000","last_undegraded":"2024-03-30T15:08:13.362512+0000","last_fullsized":"2024-03-30T15:08:13.362512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:45.099372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.099372+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1177,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362460+0000","last_change":"2024-03-30T15:06:45.066707+0000","last_active":"2024-03-30T15:08:13.362460+0000","last_peered":"2024-03-30T15:08:13.362460+0000","last_clean":"2024-03-30T15:08:13.362460+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:08:13.362460+0000","last_undegraded":"2024-03-30T15:08:13.362460+0000","last_fullsized":"2024-03-30T15:08:13.362460+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.066574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.066574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1485,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360924+0000","last_change":"2024-03-30T15:07:56.660010+0000","last_active":"2024-03-30T15:08:13.360924+0000","last_peered":"2024-03-30T15:08:13.360924+0000","last_clean":"2024-03-30T15:08:13.360924+0000","last_became_active":"2024-03-30T15:06:48.339784+0000","last_became_peered":"2024-03-30T15:06:48.339784+0000","last_unstale":"2024-03-30T15:08:13.360924+0000","last_undegraded":"2024-03-30T15:08:13.360924+0000","last_fullsized":"2024-03-30T15:08:13.360924+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:07:56.659850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:07:56.659850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.574408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1102,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354624+0000","last_change":"2024-03-30T15:06:10.008324+0000","last_active":"2024-03-30T15:08:13.354624+0000","last_peered":"2024-03-30T15:08:13.354624+0000","last_clean":"2024-03-30T15:08:13.354624+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:08:13.354624+0000","last_undegraded":"2024-03-30T15:08:13.354624+0000","last_fullsized":"2024-03-30T15:08:13.354624+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.008246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.008246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1202,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362946+0000","last_change":"2024-03-30T15:06:52.804319+0000","last_active":"2024-03-30T15:08:13.362946+0000","last_peered":"2024-03-30T15:08:13.362946+0000","last_clean":"2024-03-30T15:08:13.362946+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:08:13.362946+0000","last_undegraded":"2024-03-30T15:08:13.362946+0000","last_fullsized":"2024-03-30T15:08:13.362946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.804238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1242,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362320+0000","last_change":"2024-03-30T15:06:44.835160+0000","last_active":"2024-03-30T15:08:13.362320+0000","last_peered":"2024-03-30T15:08:13.362320+0000","last_clean":"2024-03-30T15:08:13.362320+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:08:13.362320+0000","last_undegraded":"2024-03-30T15:08:13.362320+0000","last_fullsized":"2024-03-30T15:08:13.362320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:06:44.835068+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.835068+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1219,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362383+0000","last_change":"2024-03-30T15:06:57.786762+0000","last_active":"2024-03-30T15:08:13.362383+0000","last_peered":"2024-03-30T15:08:13.362383+0000","last_clean":"2024-03-30T15:08:13.362383+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:08:13.362383+0000","last_undegraded":"2024-03-30T15:08:13.362383+0000","last_fullsized":"2024-03-30T15:08:13.362383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.786685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.786685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1208,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357840+0000","last_change":"2024-03-30T15:07:23.937183+0000","last_active":"2024-03-30T15:08:13.357840+0000","last_peered":"2024-03-30T15:08:13.357840+0000","last_clean":"2024-03-30T15:08:13.357840+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:08:13.357840+0000","last_undegraded":"2024-03-30T15:08:13.357840+0000","last_fullsized":"2024-03-30T15:08:13.357840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.937096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.937096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:27.709429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1195,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363811+0000","last_change":"2024-03-30T15:07:58.653866+0000","last_active":"2024-03-30T15:08:13.363811+0000","last_peered":"2024-03-30T15:08:13.363811+0000","last_clean":"2024-03-30T15:08:13.363811+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:08:13.363811+0000","last_undegraded":"2024-03-30T15:08:13.363811+0000","last_fullsized":"2024-03-30T15:08:13.363811+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.653564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.653564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.992993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":475,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358627+0000","last_change":"2024-03-30T15:06:48.962204+0000","last_active":"2024-03-30T15:08:13.358627+0000","last_peered":"2024-03-30T15:08:13.358627+0000","last_clean":"2024-03-30T15:08:13.358627+0000","last_became_active":"2024-03-30T15:06:48.339451+0000","last_became_peered":"2024-03-30T15:06:48.339451+0000","last_unstale":"2024-03-30T15:08:13.358627+0000","last_undegraded":"2024-03-30T15:08:13.358627+0000","last_fullsized":"2024-03-30T15:08:13.358627+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.962124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.962124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352686+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:08:13.352686+0000","last_peered":"2024-03-30T15:08:13.352686+0000","last_clean":"2024-03-30T15:08:13.352686+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:08:13.352686+0000","last_undegraded":"2024-03-30T15:08:13.352686+0000","last_fullsized":"2024-03-30T15:08:13.352686+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1184,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353366+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:08:13.353366+0000","last_peered":"2024-03-30T15:08:13.353366+0000","last_clean":"2024-03-30T15:08:13.353366+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:08:13.353366+0000","last_undegraded":"2024-03-30T15:08:13.353366+0000","last_fullsized":"2024-03-30T15:08:13.353366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1370,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360914+0000","last_change":"2024-03-30T15:06:05.010246+0000","last_active":"2024-03-30T15:08:13.360914+0000","last_peered":"2024-03-30T15:08:13.360914+0000","last_clean":"2024-03-30T15:08:13.360914+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:08:13.360914+0000","last_undegraded":"2024-03-30T15:08:13.360914+0000","last_fullsized":"2024-03-30T15:08:13.360914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:06:05.010128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.010128+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1197,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358738+0000","last_change":"2024-03-30T15:07:31.968746+0000","last_active":"2024-03-30T15:08:13.358738+0000","last_peered":"2024-03-30T15:08:13.358738+0000","last_clean":"2024-03-30T15:08:13.358738+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:08:13.358738+0000","last_undegraded":"2024-03-30T15:08:13.358738+0000","last_fullsized":"2024-03-30T15:08:13.358738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.259707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1200,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359288+0000","last_change":"2024-03-30T15:06:40.992863+0000","last_active":"2024-03-30T15:08:13.359288+0000","last_peered":"2024-03-30T15:08:13.359288+0000","last_clean":"2024-03-30T15:08:13.359288+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:08:13.359288+0000","last_undegraded":"2024-03-30T15:08:13.359288+0000","last_fullsized":"2024-03-30T15:08:13.359288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:06:40.992758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.992758+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359885+0000","last_change":"2024-03-30T15:08:00.231743+0000","last_active":"2024-03-30T15:08:13.359885+0000","last_peered":"2024-03-30T15:08:13.359885+0000","last_clean":"2024-03-30T15:08:13.359885+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:08:13.359885+0000","last_undegraded":"2024-03-30T15:08:13.359885+0000","last_fullsized":"2024-03-30T15:08:13.359885+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.231663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.231663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.864990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353254+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:08:13.353254+0000","last_peered":"2024-03-30T15:08:13.353254+0000","last_clean":"2024-03-30T15:08:13.353254+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:08:13.353254+0000","last_undegraded":"2024-03-30T15:08:13.353254+0000","last_fullsized":"2024-03-30T15:08:13.353254+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1221,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353229+0000","last_change":"2024-03-30T15:08:12.601879+0000","last_active":"2024-03-30T15:08:13.353229+0000","last_peered":"2024-03-30T15:08:13.353229+0000","last_clean":"2024-03-30T15:08:13.353229+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:08:13.353229+0000","last_undegraded":"2024-03-30T15:08:13.353229+0000","last_fullsized":"2024-03-30T15:08:13.353229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.593442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1197,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361165+0000","last_change":"2024-03-30T15:07:20.139612+0000","last_active":"2024-03-30T15:08:13.361165+0000","last_peered":"2024-03-30T15:08:13.361165+0000","last_clean":"2024-03-30T15:08:13.361165+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:08:13.361165+0000","last_undegraded":"2024-03-30T15:08:13.361165+0000","last_fullsized":"2024-03-30T15:08:13.361165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:20.139530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.139530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.029764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1210,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363400+0000","last_change":"2024-03-30T15:07:58.247377+0000","last_active":"2024-03-30T15:08:13.363400+0000","last_peered":"2024-03-30T15:08:13.363400+0000","last_clean":"2024-03-30T15:08:13.363400+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:08:13.363400+0000","last_undegraded":"2024-03-30T15:08:13.363400+0000","last_fullsized":"2024-03-30T15:08:13.363400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.247271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.247271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.826144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1180,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353661+0000","last_change":"2024-03-30T15:07:21.573597+0000","last_active":"2024-03-30T15:08:13.353661+0000","last_peered":"2024-03-30T15:08:13.353661+0000","last_clean":"2024-03-30T15:08:13.353661+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:08:13.353661+0000","last_undegraded":"2024-03-30T15:08:13.353661+0000","last_fullsized":"2024-03-30T15:08:13.353661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.573522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.573522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.445672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1094,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359518+0000","last_change":"2024-03-30T15:07:22.909083+0000","last_active":"2024-03-30T15:08:13.359518+0000","last_peered":"2024-03-30T15:08:13.359518+0000","last_clean":"2024-03-30T15:08:13.359518+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:08:13.359518+0000","last_undegraded":"2024-03-30T15:08:13.359518+0000","last_fullsized":"2024-03-30T15:08:13.359518+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.908989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.908989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.320783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1282,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353506+0000","last_change":"2024-03-30T15:06:44.482561+0000","last_active":"2024-03-30T15:08:13.353506+0000","last_peered":"2024-03-30T15:08:13.353506+0000","last_clean":"2024-03-30T15:08:13.353506+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:08:13.353506+0000","last_undegraded":"2024-03-30T15:08:13.353506+0000","last_fullsized":"2024-03-30T15:08:13.353506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:06:44.482442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.482442+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1199,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363693+0000","last_change":"2024-03-30T15:07:18.376275+0000","last_active":"2024-03-30T15:08:13.363693+0000","last_peered":"2024-03-30T15:08:13.363693+0000","last_clean":"2024-03-30T15:08:13.363693+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:08:13.363693+0000","last_undegraded":"2024-03-30T15:08:13.363693+0000","last_fullsized":"2024-03-30T15:08:13.363693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.376197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.376197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.539972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1202,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362670+0000","last_change":"2024-03-30T15:06:48.713525+0000","last_active":"2024-03-30T15:08:13.362670+0000","last_peered":"2024-03-30T15:08:13.362670+0000","last_clean":"2024-03-30T15:08:13.362670+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:08:13.362670+0000","last_undegraded":"2024-03-30T15:08:13.362670+0000","last_fullsized":"2024-03-30T15:08:13.362670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:48.713441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.713441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:16.143415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1187,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360553+0000","last_change":"2024-03-30T15:08:11.701750+0000","last_active":"2024-03-30T15:08:13.360553+0000","last_peered":"2024-03-30T15:08:13.360553+0000","last_clean":"2024-03-30T15:08:13.360553+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:08:13.360553+0000","last_undegraded":"2024-03-30T15:08:13.360553+0000","last_fullsized":"2024-03-30T15:08:13.360553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.701619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.701619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.937509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1116,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353186+0000","last_change":"2024-03-30T15:06:46.411474+0000","last_active":"2024-03-30T15:08:13.353186+0000","last_peered":"2024-03-30T15:08:13.353186+0000","last_clean":"2024-03-30T15:08:13.353186+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:08:13.353186+0000","last_undegraded":"2024-03-30T15:08:13.353186+0000","last_fullsized":"2024-03-30T15:08:13.353186+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:06:46.411383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:46.411383+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1202,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361504+0000","last_change":"2024-03-30T15:06:58.835912+0000","last_active":"2024-03-30T15:08:13.361504+0000","last_peered":"2024-03-30T15:08:13.361504+0000","last_clean":"2024-03-30T15:08:13.361504+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:08:13.361504+0000","last_undegraded":"2024-03-30T15:08:13.361504+0000","last_fullsized":"2024-03-30T15:08:13.361504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.835804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.835804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1233,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358421+0000","last_change":"2024-03-30T15:08:08.213331+0000","last_active":"2024-03-30T15:08:13.358421+0000","last_peered":"2024-03-30T15:08:13.358421+0000","last_clean":"2024-03-30T15:08:13.358421+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:08:13.358421+0000","last_undegraded":"2024-03-30T15:08:13.358421+0000","last_fullsized":"2024-03-30T15:08:13.358421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:08:08.213219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.213219+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.230614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1199,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363629+0000","last_change":"2024-03-30T15:08:04.783312+0000","last_active":"2024-03-30T15:08:13.363629+0000","last_peered":"2024-03-30T15:08:13.363629+0000","last_clean":"2024-03-30T15:08:13.363629+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:08:13.363629+0000","last_undegraded":"2024-03-30T15:08:13.363629+0000","last_fullsized":"2024-03-30T15:08:13.363629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:04.783237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.783237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.511079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362089+0000","last_change":"2024-03-30T15:06:54.229912+0000","last_active":"2024-03-30T15:08:13.362089+0000","last_peered":"2024-03-30T15:08:13.362089+0000","last_clean":"2024-03-30T15:08:13.362089+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:08:13.362089+0000","last_undegraded":"2024-03-30T15:08:13.362089+0000","last_fullsized":"2024-03-30T15:08:13.362089+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.229818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.229818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1094,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357415+0000","last_change":"2024-03-30T15:07:18.807780+0000","last_active":"2024-03-30T15:08:13.357415+0000","last_peered":"2024-03-30T15:08:13.357415+0000","last_clean":"2024-03-30T15:08:13.357415+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:08:13.357415+0000","last_undegraded":"2024-03-30T15:08:13.357415+0000","last_fullsized":"2024-03-30T15:08:13.357415+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.807666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.807666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:47.636769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1355,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361884+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:08:13.361884+0000","last_peered":"2024-03-30T15:08:13.361884+0000","last_clean":"2024-03-30T15:08:13.361884+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:08:13.361884+0000","last_undegraded":"2024-03-30T15:08:13.361884+0000","last_fullsized":"2024-03-30T15:08:13.361884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1228,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358860+0000","last_change":"2024-03-30T15:06:16.030806+0000","last_active":"2024-03-30T15:08:13.358860+0000","last_peered":"2024-03-30T15:08:13.358860+0000","last_clean":"2024-03-30T15:08:13.358860+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:08:13.358860+0000","last_undegraded":"2024-03-30T15:08:13.358860+0000","last_fullsized":"2024-03-30T15:08:13.358860+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:06:16.030694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.030694+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1211,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.364046+0000","last_change":"2024-03-30T15:07:32.333233+0000","last_active":"2024-03-30T15:08:13.364046+0000","last_peered":"2024-03-30T15:08:13.364046+0000","last_clean":"2024-03-30T15:08:13.364046+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:08:13.364046+0000","last_undegraded":"2024-03-30T15:08:13.364046+0000","last_fullsized":"2024-03-30T15:08:13.364046+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:07:32.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.333139+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.507134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1177,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359844+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:08:13.359844+0000","last_peered":"2024-03-30T15:08:13.359844+0000","last_clean":"2024-03-30T15:08:13.359844+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:13.359844+0000","last_undegraded":"2024-03-30T15:08:13.359844+0000","last_fullsized":"2024-03-30T15:08:13.359844+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1103,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354621+0000","last_change":"2024-03-30T15:07:32.310952+0000","last_active":"2024-03-30T15:08:13.354621+0000","last_peered":"2024-03-30T15:08:13.354621+0000","last_clean":"2024-03-30T15:08:13.354621+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:08:13.354621+0000","last_undegraded":"2024-03-30T15:08:13.354621+0000","last_fullsized":"2024-03-30T15:08:13.354621+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.310831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.310831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.533764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":797,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360875+0000","last_change":"2024-03-30T15:08:12.833026+0000","last_active":"2024-03-30T15:08:13.360875+0000","last_peered":"2024-03-30T15:08:13.360875+0000","last_clean":"2024-03-30T15:08:13.360875+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:08:13.360875+0000","last_undegraded":"2024-03-30T15:08:13.360875+0000","last_fullsized":"2024-03-30T15:08:13.360875+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:08:12.832934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.832934+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.769346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1212,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353020+0000","last_change":"2024-03-30T15:07:58.498811+0000","last_active":"2024-03-30T15:08:13.353020+0000","last_peered":"2024-03-30T15:08:13.353020+0000","last_clean":"2024-03-30T15:08:13.353020+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:08:13.353020+0000","last_undegraded":"2024-03-30T15:08:13.353020+0000","last_fullsized":"2024-03-30T15:08:13.353020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.498678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.498678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.745457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1180,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362171+0000","last_change":"2024-03-30T15:06:38.946768+0000","last_active":"2024-03-30T15:08:13.362171+0000","last_peered":"2024-03-30T15:08:13.362171+0000","last_clean":"2024-03-30T15:08:13.362171+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:08:13.362171+0000","last_undegraded":"2024-03-30T15:08:13.362171+0000","last_fullsized":"2024-03-30T15:08:13.362171+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:38.946668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:38.946668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1178,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363347+0000","last_change":"2024-03-30T15:06:42.118047+0000","last_active":"2024-03-30T15:08:13.363347+0000","last_peered":"2024-03-30T15:08:13.363347+0000","last_clean":"2024-03-30T15:08:13.363347+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:08:13.363347+0000","last_undegraded":"2024-03-30T15:08:13.363347+0000","last_fullsized":"2024-03-30T15:08:13.363347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:42.117927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.117927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1094,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359769+0000","last_change":"2024-03-30T15:07:29.943796+0000","last_active":"2024-03-30T15:08:13.359769+0000","last_peered":"2024-03-30T15:08:13.359769+0000","last_clean":"2024-03-30T15:08:13.359769+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:08:13.359769+0000","last_undegraded":"2024-03-30T15:08:13.359769+0000","last_fullsized":"2024-03-30T15:08:13.359769+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.943712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.943712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:33.196867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1212,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354397+0000","last_change":"2024-03-30T15:07:34.554038+0000","last_active":"2024-03-30T15:08:13.354397+0000","last_peered":"2024-03-30T15:08:13.354397+0000","last_clean":"2024-03-30T15:08:13.354397+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:08:13.354397+0000","last_undegraded":"2024-03-30T15:08:13.354397+0000","last_fullsized":"2024-03-30T15:08:13.354397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.654265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1198,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361603+0000","last_change":"2024-03-30T15:06:57.279563+0000","last_active":"2024-03-30T15:08:13.361603+0000","last_peered":"2024-03-30T15:08:13.361603+0000","last_clean":"2024-03-30T15:08:13.361603+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:08:13.361603+0000","last_undegraded":"2024-03-30T15:08:13.361603+0000","last_fullsized":"2024-03-30T15:08:13.361603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.279464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.279464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:26.580401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":285,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354506+0000","last_change":"2024-03-30T15:07:18.427089+0000","last_active":"2024-03-30T15:08:13.354506+0000","last_peered":"2024-03-30T15:08:13.354506+0000","last_clean":"2024-03-30T15:08:13.354506+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:08:13.354506+0000","last_undegraded":"2024-03-30T15:08:13.354506+0000","last_fullsized":"2024-03-30T15:08:13.354506+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1205,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363543+0000","last_change":"2024-03-30T15:08:01.209737+0000","last_active":"2024-03-30T15:08:13.363543+0000","last_peered":"2024-03-30T15:08:13.363543+0000","last_clean":"2024-03-30T15:08:13.363543+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:08:13.363543+0000","last_undegraded":"2024-03-30T15:08:13.363543+0000","last_fullsized":"2024-03-30T15:08:13.363543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:01.209658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.209658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.903658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352858+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:08:13.352858+0000","last_peered":"2024-03-30T15:08:13.352858+0000","last_clean":"2024-03-30T15:08:13.352858+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:08:13.352858+0000","last_undegraded":"2024-03-30T15:08:13.352858+0000","last_fullsized":"2024-03-30T15:08:13.352858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1209,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362177+0000","last_change":"2024-03-30T15:06:39.943308+0000","last_active":"2024-03-30T15:08:13.362177+0000","last_peered":"2024-03-30T15:08:13.362177+0000","last_clean":"2024-03-30T15:08:13.362177+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:08:13.362177+0000","last_undegraded":"2024-03-30T15:08:13.362177+0000","last_fullsized":"2024-03-30T15:08:13.362177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:39.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.943172+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1191,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352536+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:08:13.352536+0000","last_peered":"2024-03-30T15:08:13.352536+0000","last_clean":"2024-03-30T15:08:13.352536+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:08:13.352536+0000","last_undegraded":"2024-03-30T15:08:13.352536+0000","last_fullsized":"2024-03-30T15:08:13.352536+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1240,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362909+0000","last_change":"2024-03-30T15:06:11.222500+0000","last_active":"2024-03-30T15:08:13.362909+0000","last_peered":"2024-03-30T15:08:13.362909+0000","last_clean":"2024-03-30T15:08:13.362909+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:08:13.362909+0000","last_undegraded":"2024-03-30T15:08:13.362909+0000","last_fullsized":"2024-03-30T15:08:13.362909+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.222382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.222382+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1176,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360754+0000","last_change":"2024-03-30T15:06:58.575629+0000","last_active":"2024-03-30T15:08:13.360754+0000","last_peered":"2024-03-30T15:08:13.360754+0000","last_clean":"2024-03-30T15:08:13.360754+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:08:13.360754+0000","last_undegraded":"2024-03-30T15:08:13.360754+0000","last_fullsized":"2024-03-30T15:08:13.360754+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.575544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.575544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352564+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:08:13.352564+0000","last_peered":"2024-03-30T15:08:13.352564+0000","last_clean":"2024-03-30T15:08:13.352564+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:08:13.352564+0000","last_undegraded":"2024-03-30T15:08:13.352564+0000","last_fullsized":"2024-03-30T15:08:13.352564+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":1149,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360632+0000","last_change":"2024-03-30T15:06:59.844000+0000","last_active":"2024-03-30T15:08:13.360632+0000","last_peered":"2024-03-30T15:08:13.360632+0000","last_clean":"2024-03-30T15:08:13.360632+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:08:13.360632+0000","last_undegraded":"2024-03-30T15:08:13.360632+0000","last_fullsized":"2024-03-30T15:08:13.360632+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:06:59.843847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.843847+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1190,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361447+0000","last_change":"2024-03-30T15:06:14.188494+0000","last_active":"2024-03-30T15:08:13.361447+0000","last_peered":"2024-03-30T15:08:13.361447+0000","last_clean":"2024-03-30T15:08:13.361447+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:08:13.361447+0000","last_undegraded":"2024-03-30T15:08:13.361447+0000","last_fullsized":"2024-03-30T15:08:13.361447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.188400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.188400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1355,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361741+0000","last_change":"2024-03-30T15:06:44.020947+0000","last_active":"2024-03-30T15:08:13.361741+0000","last_peered":"2024-03-30T15:08:13.361741+0000","last_clean":"2024-03-30T15:08:13.361741+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:08:13.361741+0000","last_undegraded":"2024-03-30T15:08:13.361741+0000","last_fullsized":"2024-03-30T15:08:13.361741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:44.020829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.020829+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1181,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353677+0000","last_change":"2024-03-30T15:07:30.551177+0000","last_active":"2024-03-30T15:08:13.353677+0000","last_peered":"2024-03-30T15:08:13.353677+0000","last_clean":"2024-03-30T15:08:13.353677+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:08:13.353677+0000","last_undegraded":"2024-03-30T15:08:13.353677+0000","last_fullsized":"2024-03-30T15:08:13.353677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:07:30.551096+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.551096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:33.345129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1093,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359205+0000","last_change":"2024-03-30T15:07:21.866188+0000","last_active":"2024-03-30T15:08:13.359205+0000","last_peered":"2024-03-30T15:08:13.359205+0000","last_clean":"2024-03-30T15:08:13.359205+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:08:13.359205+0000","last_undegraded":"2024-03-30T15:08:13.359205+0000","last_fullsized":"2024-03-30T15:08:13.359205+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.866106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.866106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.392301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1220,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359313+0000","last_change":"2024-03-30T15:06:54.562492+0000","last_active":"2024-03-30T15:08:13.359313+0000","last_peered":"2024-03-30T15:08:13.359313+0000","last_clean":"2024-03-30T15:08:13.359313+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:08:13.359313+0000","last_undegraded":"2024-03-30T15:08:13.359313+0000","last_fullsized":"2024-03-30T15:08:13.359313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:06:54.562347+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.562347+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1196,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359352+0000","last_change":"2024-03-30T15:07:00.555778+0000","last_active":"2024-03-30T15:08:13.359352+0000","last_peered":"2024-03-30T15:08:13.359352+0000","last_clean":"2024-03-30T15:08:13.359352+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:08:13.359352+0000","last_undegraded":"2024-03-30T15:08:13.359352+0000","last_fullsized":"2024-03-30T15:08:13.359352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.555700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.555700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.200782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1199,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359180+0000","last_change":"2024-03-30T15:07:35.184388+0000","last_active":"2024-03-30T15:08:13.359180+0000","last_peered":"2024-03-30T15:08:13.359180+0000","last_clean":"2024-03-30T15:08:13.359180+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:08:13.359180+0000","last_undegraded":"2024-03-30T15:08:13.359180+0000","last_fullsized":"2024-03-30T15:08:13.359180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.166334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1092,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353497+0000","last_change":"2024-03-30T15:08:11.553825+0000","last_active":"2024-03-30T15:08:13.353497+0000","last_peered":"2024-03-30T15:08:13.353497+0000","last_clean":"2024-03-30T15:08:13.353497+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:08:13.353497+0000","last_undegraded":"2024-03-30T15:08:13.353497+0000","last_fullsized":"2024-03-30T15:08:13.353497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.326045+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1093,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358101+0000","last_change":"2024-03-30T15:07:24.946870+0000","last_active":"2024-03-30T15:08:13.358101+0000","last_peered":"2024-03-30T15:08:13.358101+0000","last_clean":"2024-03-30T15:08:13.358101+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:08:13.358101+0000","last_undegraded":"2024-03-30T15:08:13.358101+0000","last_fullsized":"2024-03-30T15:08:13.358101+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.946762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.946762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.723932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1220,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359358+0000","last_change":"2024-03-30T15:07:20.851744+0000","last_active":"2024-03-30T15:08:13.359358+0000","last_peered":"2024-03-30T15:08:13.359358+0000","last_clean":"2024-03-30T15:08:13.359358+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:08:13.359358+0000","last_undegraded":"2024-03-30T15:08:13.359358+0000","last_fullsized":"2024-03-30T15:08:13.359358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:20.851659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.851659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:28.316096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1213,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363354+0000","last_change":"2024-03-30T15:07:33.297905+0000","last_active":"2024-03-30T15:08:13.363354+0000","last_peered":"2024-03-30T15:08:13.363354+0000","last_clean":"2024-03-30T15:08:13.363354+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:08:13.363354+0000","last_undegraded":"2024-03-30T15:08:13.363354+0000","last_fullsized":"2024-03-30T15:08:13.363354+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.297820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.464840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":355,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358549+0000","last_change":"2024-03-30T15:06:15.461799+0000","last_active":"2024-03-30T15:08:13.358549+0000","last_peered":"2024-03-30T15:08:13.358549+0000","last_clean":"2024-03-30T15:08:13.358549+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:08:13.358549+0000","last_undegraded":"2024-03-30T15:08:13.358549+0000","last_fullsized":"2024-03-30T15:08:13.358549+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:15.461699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:15.461699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1179,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352335+0000","last_change":"2024-03-30T15:06:53.604745+0000","last_active":"2024-03-30T15:08:13.352335+0000","last_peered":"2024-03-30T15:08:13.352335+0000","last_clean":"2024-03-30T15:08:13.352335+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:08:13.352335+0000","last_undegraded":"2024-03-30T15:08:13.352335+0000","last_fullsized":"2024-03-30T15:08:13.352335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353244+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:08:13.353244+0000","last_peered":"2024-03-30T15:08:13.353244+0000","last_clean":"2024-03-30T15:08:13.353244+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:08:13.353244+0000","last_undegraded":"2024-03-30T15:08:13.353244+0000","last_fullsized":"2024-03-30T15:08:13.353244+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1141,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361772+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:08:13.361772+0000","last_peered":"2024-03-30T15:08:13.361772+0000","last_clean":"2024-03-30T15:08:13.361772+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:08:13.361772+0000","last_undegraded":"2024-03-30T15:08:13.361772+0000","last_fullsized":"2024-03-30T15:08:13.361772+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362348+0000","last_change":"2024-03-30T15:06:00.028025+0000","last_active":"2024-03-30T15:08:13.362348+0000","last_peered":"2024-03-30T15:08:13.362348+0000","last_clean":"2024-03-30T15:08:13.362348+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:08:13.362348+0000","last_undegraded":"2024-03-30T15:08:13.362348+0000","last_fullsized":"2024-03-30T15:08:13.362348+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.027914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.027914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1190,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360918+0000","last_change":"2024-03-30T15:08:07.697978+0000","last_active":"2024-03-30T15:08:13.360918+0000","last_peered":"2024-03-30T15:08:13.360918+0000","last_clean":"2024-03-30T15:08:13.360918+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:08:13.360918+0000","last_undegraded":"2024-03-30T15:08:13.360918+0000","last_fullsized":"2024-03-30T15:08:13.360918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:07.697832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:07.697832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.589928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1092,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352918+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:08:13.352918+0000","last_peered":"2024-03-30T15:08:13.352918+0000","last_clean":"2024-03-30T15:08:13.352918+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:08:13.352918+0000","last_undegraded":"2024-03-30T15:08:13.352918+0000","last_fullsized":"2024-03-30T15:08:13.352918+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1105,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358100+0000","last_change":"2024-03-30T15:08:08.177938+0000","last_active":"2024-03-30T15:08:13.358100+0000","last_peered":"2024-03-30T15:08:13.358100+0000","last_clean":"2024-03-30T15:08:13.358100+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:08:13.358100+0000","last_undegraded":"2024-03-30T15:08:13.358100+0000","last_fullsized":"2024-03-30T15:08:13.358100+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.177858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.177858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.773779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":775,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358118+0000","last_change":"2024-03-30T15:07:22.646408+0000","last_active":"2024-03-30T15:08:13.358118+0000","last_peered":"2024-03-30T15:08:13.358118+0000","last_clean":"2024-03-30T15:08:13.358118+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:08:13.358118+0000","last_undegraded":"2024-03-30T15:08:13.358118+0000","last_fullsized":"2024-03-30T15:08:13.358118+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:07:22.646314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.646314+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:30.332030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1199,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353793+0000","last_change":"2024-03-30T15:08:10.508709+0000","last_active":"2024-03-30T15:08:13.353793+0000","last_peered":"2024-03-30T15:08:13.353793+0000","last_clean":"2024-03-30T15:08:13.353793+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:08:13.353793+0000","last_undegraded":"2024-03-30T15:08:13.353793+0000","last_fullsized":"2024-03-30T15:08:13.353793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.166959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1196,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353734+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:08:13.353734+0000","last_peered":"2024-03-30T15:08:13.353734+0000","last_clean":"2024-03-30T15:08:13.353734+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:08:13.353734+0000","last_undegraded":"2024-03-30T15:08:13.353734+0000","last_fullsized":"2024-03-30T15:08:13.353734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1178,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361457+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:08:13.361457+0000","last_peered":"2024-03-30T15:08:13.361457+0000","last_clean":"2024-03-30T15:08:13.361457+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:08:13.361457+0000","last_undegraded":"2024-03-30T15:08:13.361457+0000","last_fullsized":"2024-03-30T15:08:13.361457+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1102,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354714+0000","last_change":"2024-03-30T15:07:33.284903+0000","last_active":"2024-03-30T15:08:13.354714+0000","last_peered":"2024-03-30T15:08:13.354714+0000","last_clean":"2024-03-30T15:08:13.354714+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:08:13.354714+0000","last_undegraded":"2024-03-30T15:08:13.354714+0000","last_fullsized":"2024-03-30T15:08:13.354714+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.284814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.284814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.915368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1229,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354782+0000","last_change":"2024-03-30T15:05:59.994488+0000","last_active":"2024-03-30T15:08:13.354782+0000","last_peered":"2024-03-30T15:08:13.354782+0000","last_clean":"2024-03-30T15:08:13.354782+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:08:13.354782+0000","last_undegraded":"2024-03-30T15:08:13.354782+0000","last_fullsized":"2024-03-30T15:08:13.354782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:05:59.994304+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:05:59.994304+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1093,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352684+0000","last_change":"2024-03-30T15:08:00.523833+0000","last_active":"2024-03-30T15:08:13.352684+0000","last_peered":"2024-03-30T15:08:13.352684+0000","last_clean":"2024-03-30T15:08:13.352684+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:08:13.352684+0000","last_undegraded":"2024-03-30T15:08:13.352684+0000","last_fullsized":"2024-03-30T15:08:13.352684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.174749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1184,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354500+0000","last_change":"2024-03-30T15:07:28.601931+0000","last_active":"2024-03-30T15:08:13.354500+0000","last_peered":"2024-03-30T15:08:13.354500+0000","last_clean":"2024-03-30T15:08:13.354500+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:08:13.354500+0000","last_undegraded":"2024-03-30T15:08:13.354500+0000","last_fullsized":"2024-03-30T15:08:13.354500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.601815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.601815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:50.989405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1195,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361543+0000","last_change":"2024-03-30T15:08:02.083375+0000","last_active":"2024-03-30T15:08:13.361543+0000","last_peered":"2024-03-30T15:08:13.361543+0000","last_clean":"2024-03-30T15:08:13.361543+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:08:13.361543+0000","last_undegraded":"2024-03-30T15:08:13.361543+0000","last_fullsized":"2024-03-30T15:08:13.361543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.083294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.083294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.435219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1102,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354536+0000","last_change":"2024-03-30T15:06:14.085407+0000","last_active":"2024-03-30T15:08:13.354536+0000","last_peered":"2024-03-30T15:08:13.354536+0000","last_clean":"2024-03-30T15:08:13.354536+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:08:13.354536+0000","last_undegraded":"2024-03-30T15:08:13.354536+0000","last_fullsized":"2024-03-30T15:08:13.354536+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.085299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1223,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354338+0000","last_change":"2024-03-30T15:07:25.577161+0000","last_active":"2024-03-30T15:08:13.354338+0000","last_peered":"2024-03-30T15:08:13.354338+0000","last_clean":"2024-03-30T15:08:13.354338+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:08:13.354338+0000","last_undegraded":"2024-03-30T15:08:13.354338+0000","last_fullsized":"2024-03-30T15:08:13.354338+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:07:25.577074+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.577074+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.741789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1201,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358447+0000","last_change":"2024-03-30T15:07:30.031279+0000","last_active":"2024-03-30T15:08:13.358447+0000","last_peered":"2024-03-30T15:08:13.358447+0000","last_clean":"2024-03-30T15:08:13.358447+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:08:13.358447+0000","last_undegraded":"2024-03-30T15:08:13.358447+0000","last_fullsized":"2024-03-30T15:08:13.358447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.031154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.145354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1380,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354679+0000","last_change":"2024-03-30T15:08:04.561718+0000","last_active":"2024-03-30T15:08:13.354679+0000","last_peered":"2024-03-30T15:08:13.354679+0000","last_clean":"2024-03-30T15:08:13.354679+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:08:13.354679+0000","last_undegraded":"2024-03-30T15:08:13.354679+0000","last_fullsized":"2024-03-30T15:08:13.354679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:04.561610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.561610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:28.847750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363052+0000","last_change":"2024-03-30T15:06:48.193765+0000","last_active":"2024-03-30T15:08:13.363052+0000","last_peered":"2024-03-30T15:08:13.363052+0000","last_clean":"2024-03-30T15:08:13.363052+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:08:13.363052+0000","last_undegraded":"2024-03-30T15:08:13.363052+0000","last_fullsized":"2024-03-30T15:08:13.363052+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:06:48.193663+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.193663+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1110,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354142+0000","last_change":"2024-03-30T15:07:57.453748+0000","last_active":"2024-03-30T15:08:13.354142+0000","last_peered":"2024-03-30T15:08:13.354142+0000","last_clean":"2024-03-30T15:08:13.354142+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:08:13.354142+0000","last_undegraded":"2024-03-30T15:08:13.354142+0000","last_fullsized":"2024-03-30T15:08:13.354142+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.453629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.453629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.047448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1151,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361303+0000","last_change":"2024-03-30T15:07:21.894935+0000","last_active":"2024-03-30T15:08:13.361303+0000","last_peered":"2024-03-30T15:08:13.361303+0000","last_clean":"2024-03-30T15:08:13.361303+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:08:13.361303+0000","last_undegraded":"2024-03-30T15:08:13.361303+0000","last_fullsized":"2024-03-30T15:08:13.361303+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:07:21.894855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.894855+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.454380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358753+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:08:13.358753+0000","last_peered":"2024-03-30T15:08:13.358753+0000","last_clean":"2024-03-30T15:08:13.358753+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:08:13.358753+0000","last_undegraded":"2024-03-30T15:08:13.358753+0000","last_fullsized":"2024-03-30T15:08:13.358753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1180,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360575+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:08:13.360575+0000","last_peered":"2024-03-30T15:08:13.360575+0000","last_clean":"2024-03-30T15:08:13.360575+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:08:13.360575+0000","last_undegraded":"2024-03-30T15:08:13.360575+0000","last_fullsized":"2024-03-30T15:08:13.360575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1106,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354880+0000","last_change":"2024-03-30T15:07:36.255445+0000","last_active":"2024-03-30T15:08:13.354880+0000","last_peered":"2024-03-30T15:08:13.354880+0000","last_clean":"2024-03-30T15:08:13.354880+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:08:13.354880+0000","last_undegraded":"2024-03-30T15:08:13.354880+0000","last_fullsized":"2024-03-30T15:08:13.354880+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.255369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.255369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:58.024102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1094,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362028+0000","last_change":"2024-03-30T15:07:28.134811+0000","last_active":"2024-03-30T15:08:13.362028+0000","last_peered":"2024-03-30T15:08:13.362028+0000","last_clean":"2024-03-30T15:08:13.362028+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:08:13.362028+0000","last_undegraded":"2024-03-30T15:08:13.362028+0000","last_fullsized":"2024-03-30T15:08:13.362028+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.134705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.134705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.521025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1263,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.363293+0000","last_change":"2024-03-30T15:07:59.295056+0000","last_active":"2024-03-30T15:08:13.363293+0000","last_peered":"2024-03-30T15:08:13.363293+0000","last_clean":"2024-03-30T15:08:13.363293+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:08:13.363293+0000","last_undegraded":"2024-03-30T15:08:13.363293+0000","last_fullsized":"2024-03-30T15:08:13.363293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:59.294962+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.294962+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.470863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1191,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353412+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:08:13.353412+0000","last_peered":"2024-03-30T15:08:13.353412+0000","last_clean":"2024-03-30T15:08:13.353412+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:08:13.353412+0000","last_undegraded":"2024-03-30T15:08:13.353412+0000","last_fullsized":"2024-03-30T15:08:13.353412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1199,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361438+0000","last_change":"2024-03-30T15:08:00.131070+0000","last_active":"2024-03-30T15:08:13.361438+0000","last_peered":"2024-03-30T15:08:13.361438+0000","last_clean":"2024-03-30T15:08:13.361438+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:08:13.361438+0000","last_undegraded":"2024-03-30T15:08:13.361438+0000","last_fullsized":"2024-03-30T15:08:13.361438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.130991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.130991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.398245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1177,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360002+0000","last_change":"2024-03-30T15:06:12.558953+0000","last_active":"2024-03-30T15:08:13.360002+0000","last_peered":"2024-03-30T15:08:13.360002+0000","last_clean":"2024-03-30T15:08:13.360002+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:08:13.360002+0000","last_undegraded":"2024-03-30T15:08:13.360002+0000","last_fullsized":"2024-03-30T15:08:13.360002+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.558834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.558834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1097,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359829+0000","last_change":"2024-03-30T15:06:04.136327+0000","last_active":"2024-03-30T15:08:13.359829+0000","last_peered":"2024-03-30T15:08:13.359829+0000","last_clean":"2024-03-30T15:08:13.359829+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:08:13.359829+0000","last_undegraded":"2024-03-30T15:08:13.359829+0000","last_fullsized":"2024-03-30T15:08:13.359829+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:04.136246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:04.136246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1221,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359021+0000","last_change":"2024-03-30T15:06:50.584525+0000","last_active":"2024-03-30T15:08:13.359021+0000","last_peered":"2024-03-30T15:08:13.359021+0000","last_clean":"2024-03-30T15:08:13.359021+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:08:13.359021+0000","last_undegraded":"2024-03-30T15:08:13.359021+0000","last_fullsized":"2024-03-30T15:08:13.359021+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:06:50.584362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.584362+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:17.995082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361127+0000","last_change":"2024-03-30T15:06:14.990566+0000","last_active":"2024-03-30T15:08:13.361127+0000","last_peered":"2024-03-30T15:08:13.361127+0000","last_clean":"2024-03-30T15:08:13.361127+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:08:13.361127+0000","last_undegraded":"2024-03-30T15:08:13.361127+0000","last_fullsized":"2024-03-30T15:08:13.361127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.990472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.990472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1182,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361339+0000","last_change":"2024-03-30T15:05:58.374684+0000","last_active":"2024-03-30T15:08:13.361339+0000","last_peered":"2024-03-30T15:08:13.361339+0000","last_clean":"2024-03-30T15:08:13.361339+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:08:13.361339+0000","last_undegraded":"2024-03-30T15:08:13.361339+0000","last_fullsized":"2024-03-30T15:08:13.361339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:58.374590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:58.374590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1184,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359193+0000","last_change":"2024-03-30T15:07:23.180715+0000","last_active":"2024-03-30T15:08:13.359193+0000","last_peered":"2024-03-30T15:08:13.359193+0000","last_clean":"2024-03-30T15:08:13.359193+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:08:13.359193+0000","last_undegraded":"2024-03-30T15:08:13.359193+0000","last_fullsized":"2024-03-30T15:08:13.359193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.180625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.180625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:23.772874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1091,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358774+0000","last_change":"2024-03-30T15:06:14.104026+0000","last_active":"2024-03-30T15:08:13.358774+0000","last_peered":"2024-03-30T15:08:13.358774+0000","last_clean":"2024-03-30T15:08:13.358774+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:08:13.358774+0000","last_undegraded":"2024-03-30T15:08:13.358774+0000","last_fullsized":"2024-03-30T15:08:13.358774+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.103910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.103910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1209,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358637+0000","last_change":"2024-03-30T15:06:42.502254+0000","last_active":"2024-03-30T15:08:13.358637+0000","last_peered":"2024-03-30T15:08:13.358637+0000","last_clean":"2024-03-30T15:08:13.358637+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:08:13.358637+0000","last_undegraded":"2024-03-30T15:08:13.358637+0000","last_fullsized":"2024-03-30T15:08:13.358637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:42.502138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.502138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1188,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.361646+0000","last_change":"2024-03-30T15:06:43.006252+0000","last_active":"2024-03-30T15:08:13.361646+0000","last_peered":"2024-03-30T15:08:13.361646+0000","last_clean":"2024-03-30T15:08:13.361646+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:08:13.361646+0000","last_undegraded":"2024-03-30T15:08:13.361646+0000","last_fullsized":"2024-03-30T15:08:13.361646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.006124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.006124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1180,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.360138+0000","last_change":"2024-03-30T15:05:26.176781+0000","last_active":"2024-03-30T15:08:13.360138+0000","last_peered":"2024-03-30T15:08:13.360138+0000","last_clean":"2024-03-30T15:08:13.360138+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:08:13.360138+0000","last_undegraded":"2024-03-30T15:08:13.360138+0000","last_fullsized":"2024-03-30T15:08:13.360138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:26.176703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:44.997898+0000","last_clean_scrub_stamp":"2024-03-30T15:05:26.176703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1179,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.352898+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:08:13.352898+0000","last_peered":"2024-03-30T15:08:13.352898+0000","last_clean":"2024-03-30T15:08:13.352898+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:08:13.352898+0000","last_undegraded":"2024-03-30T15:08:13.352898+0000","last_fullsized":"2024-03-30T15:08:13.352898+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1094,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358858+0000","last_change":"2024-03-30T15:07:33.165675+0000","last_active":"2024-03-30T15:08:13.358858+0000","last_peered":"2024-03-30T15:08:13.358858+0000","last_clean":"2024-03-30T15:08:13.358858+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:08:13.358858+0000","last_undegraded":"2024-03-30T15:08:13.358858+0000","last_fullsized":"2024-03-30T15:08:13.358858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.165599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.165599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.109467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1213,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.362391+0000","last_change":"2024-03-30T15:07:29.102384+0000","last_active":"2024-03-30T15:08:13.362391+0000","last_peered":"2024-03-30T15:08:13.362391+0000","last_clean":"2024-03-30T15:08:13.362391+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:08:13.362391+0000","last_undegraded":"2024-03-30T15:08:13.362391+0000","last_fullsized":"2024-03-30T15:08:13.362391+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:29.102268+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.102268+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.919896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1100,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.354400+0000","last_change":"2024-03-30T15:06:17.187196+0000","last_active":"2024-03-30T15:08:13.354400+0000","last_peered":"2024-03-30T15:08:13.354400+0000","last_clean":"2024-03-30T15:08:13.354400+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:08:13.354400+0000","last_undegraded":"2024-03-30T15:08:13.354400+0000","last_fullsized":"2024-03-30T15:08:13.354400+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:17.187094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:06:17.187094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1183,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357388+0000","last_change":"2024-03-30T15:06:40.495863+0000","last_active":"2024-03-30T15:08:13.357388+0000","last_peered":"2024-03-30T15:08:13.357388+0000","last_clean":"2024-03-30T15:08:13.357388+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:08:13.357388+0000","last_undegraded":"2024-03-30T15:08:13.357388+0000","last_fullsized":"2024-03-30T15:08:13.357388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:40.495702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.495702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":763,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.359730+0000","last_change":"2024-03-30T15:07:26.673461+0000","last_active":"2024-03-30T15:08:13.359730+0000","last_peered":"2024-03-30T15:08:13.359730+0000","last_clean":"2024-03-30T15:08:13.359730+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:08:13.359730+0000","last_undegraded":"2024-03-30T15:08:13.359730+0000","last_fullsized":"2024-03-30T15:08:13.359730+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.673362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.673362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:32.939016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1089,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353088+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:08:13.353088+0000","last_peered":"2024-03-30T15:08:13.353088+0000","last_clean":"2024-03-30T15:08:13.353088+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:08:13.353088+0000","last_undegraded":"2024-03-30T15:08:13.353088+0000","last_fullsized":"2024-03-30T15:08:13.353088+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1233,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.357502+0000","last_change":"2024-03-30T15:07:30.921961+0000","last_active":"2024-03-30T15:08:13.357502+0000","last_peered":"2024-03-30T15:08:13.357502+0000","last_clean":"2024-03-30T15:08:13.357502+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:08:13.357502+0000","last_undegraded":"2024-03-30T15:08:13.357502+0000","last_fullsized":"2024-03-30T15:08:13.357502+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:30.921763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.921763+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:31.154514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1200,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.353480+0000","last_change":"2024-03-30T15:07:35.571081+0000","last_active":"2024-03-30T15:08:13.353480+0000","last_peered":"2024-03-30T15:08:13.353480+0000","last_clean":"2024-03-30T15:08:13.353480+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:08:13.353480+0000","last_undegraded":"2024-03-30T15:08:13.353480+0000","last_fullsized":"2024-03-30T15:08:13.353480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.970991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1191,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358263+0000","last_change":"2024-03-30T15:07:32.100519+0000","last_active":"2024-03-30T15:08:13.358263+0000","last_peered":"2024-03-30T15:08:13.358263+0000","last_clean":"2024-03-30T15:08:13.358263+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:08:13.358263+0000","last_undegraded":"2024-03-30T15:08:13.358263+0000","last_fullsized":"2024-03-30T15:08:13.358263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.100436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.100436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.419009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1176,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T15:08:13.358330+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:08:13.358330+0000","last_peered":"2024-03-30T15:08:13.358330+0000","last_clean":"2024-03-30T15:08:13.358330+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:08:13.358330+0000","last_undegraded":"2024-03-30T15:08:13.358330+0000","last_fullsized":"2024-03-30T15:08:13.358330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":808,"num_read_kb":803,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":322,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005224,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72092,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673060,"statfs":{"total":95995035648,"available":95921213440,"internally_reserved":0,"allocated":4931584,"data_stored":2324290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299565,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75936,"kb_used_data":4496,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93669216,"statfs":{"total":95995035648,"available":95917277184,"internally_reserved":0,"allocated":4603904,"data_stored":2013238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71800,"kb_used_data":4516,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673352,"statfs":{"total":95995035648,"available":95921512448,"internally_reserved":0,"allocated":4624384,"data_stored":2005072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49600,"kb_used_data":4524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93695552,"statfs":{"total":95995035648,"available":95944245248,"internally_reserved":0,"allocated":4632576,"data_stored":2014915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411521,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94228,"kb_used_data":4740,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650924,"statfs":{"total":95995035648,"available":95898546176,"internally_reserved":0,"allocated":4853760,"data_stored":2266888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117181,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94580,"kb_used_data":5092,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650572,"statfs":{"total":95995035648,"available":95898185728,"internally_reserved":0,"allocated":5214208,"data_stored":2586817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953655,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93996,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93651156,"statfs":{"total":95995035648,"available":95898783744,"internally_reserved":0,"allocated":4616192,"data_stored":2004868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659314,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76516,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668636,"statfs":{"total":95995035648,"available":95916683264,"internally_reserved":0,"allocated":5197824,"data_stored":2594917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:08:15.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph status --format=json 2024-03-30T15:08:15.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:15.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:15.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: from='client.25347 v1:172.21.15.28:0/1573929774' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: 42.19 deep-scrub starts 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: 42.19 deep-scrub ok 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T15:08:15.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: from='client.25347 v1:172.21.15.28:0/1573929774' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: 42.19 deep-scrub starts 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: 42.19 deep-scrub ok 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:15.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:15.797 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:15.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:15.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: from='client.25347 v1:172.21.15.28:0/1573929774' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: 42.19 deep-scrub starts 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: 42.19 deep-scrub ok 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: osdmap e646: 8 total, 8 up, 8 in 2024-03-30T15:08:15.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:15.934 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:15.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:16.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:16.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:16.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:16.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:16.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:16.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:16.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:16.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:16.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:16.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:16.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:16.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:16.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:16.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:08:16.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: pgmap v952: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:16.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:16 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:16.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:16.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: pgmap v952: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:08:16.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: pgmap v952: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: osdmap e647: 8 total, 8 up, 8 in 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:16 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:17.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:17.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:17.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:17.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:17.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:17.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:17.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:17.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:17.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:17.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T15:08:17.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:17.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: osdmap e648: 8 total, 8 up, 8 in 2024-03-30T15:08:17.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:17.975 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:18.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:18.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:18.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:18.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:18.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:18.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:18.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:18.491 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:18.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:18.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:18.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:18.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:08:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:08:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:08:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:08:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: pgmap v955: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T15:08:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:18 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:18.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: pgmap v955: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:08:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: pgmap v955: 246 pgs: 32 unknown, 8 creating+peering, 206 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: osdmap e649: 8 total, 8 up, 8 in 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:18 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]: dispatch 2024-03-30T15:08:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:18.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:18.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:19.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:19.123 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:19.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.32236.0:2 121.0 121:0384ab8e:::oid:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 598.23186412400003, 2024-03-30T15:08:19.249 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2733807989999999, 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.32236", 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/520874703", 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014681+0000", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014685+0000", 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4340000000000001e-06 2024-03-30T15:08:19.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014695+0000", 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0513000000000001e-05 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014709+0000", 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3791000000000001e-05 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.252 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.014721+0000", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2165e-05 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075744+0000", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.061023012000000001 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:21.075753+0000", 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2e-06 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162599+0000", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0868460719999999 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.162721+0000", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121662 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.171761+0000", 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0090404189999999992 2024-03-30T15:08:19.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.176616+0000", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004854934 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.272952+0000", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.096335658000000005 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.287962+0000", 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015010364 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.255 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288013+0000", 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0917000000000001e-05 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:22.288062+0000", 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8658000000000001e-05 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:1598 124.0 e147/140 124:0384ab8e:::oid:1 v 147'6, mlcod=144'3)", 2024-03-30T15:08:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 587.85233618999996, 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050952077999999998, 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394209+0000", 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394210+0000", 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.102e-06 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394226+0000", 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5554e-05 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394235+0000", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6509999999999996e-06 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.394244+0000", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2120000000000003e-06 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396824+0000", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025802059999999998 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.259 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.396852+0000", 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8252999999999999e-05 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445122+0000", 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048270362999999997 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:31.445161+0000", 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8736999999999999e-05 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.27442.0:35 111.1 e149/134 111:861d4e6f:test-rados-api-smithi028-93525-10::foo:head v 149'7, mlcod=148'4)", 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 586.02159487899996, 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043878153000000003, 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:08:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224950+0000", 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224951+0000", 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.91e-07 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.262 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224962+0000", 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0858e-05 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224967+0000", 2024-03-30T15:08:19.263 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9699999999999996e-06 2024-03-30T15:08:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.224977+0000", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1079999999999999e-06 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235114+0000", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010137607999999999 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.235139+0000", 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4437e-05 2024-03-30T15:08:19.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268792+0000", 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033653216 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:33.268829+0000", 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6665000000000002e-05 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.48856.0:9 157.2s0 157:523311ec:test-rados-api-smithi028-93441-13::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 573.060855371, 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.053529637999999997, 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.48856", 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1328251834", 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 9 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185690+0000", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185693+0000", 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1939999999999998e-06 2024-03-30T15:08:19.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185705+0000", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1446e-05 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185727+0000", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2249e-05 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185739+0000", 2024-03-30T15:08:19.269 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2692e-05 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185814+0000", 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5013999999999997e-05 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.185822+0000", 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5329999999999997e-06 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.270 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198631+0000", 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012809375 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.198727+0000", 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5884000000000006e-05 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.199305+0000", 2024-03-30T15:08:19.271 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057763400000000005 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.211644+0000", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012338905000000001 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239140+0000", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027495908999999999 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:46.239220+0000", 2024-03-30T15:08:19.272 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9802999999999995e-05 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.56915.0:2 173.1b 173:d96a01e0:::testobj:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 570.06097339899998, 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.049005913, 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.273 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.56915", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1224869641", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185572+0000", 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185574+0000", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9039999999999999e-06 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185581+0000", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7090000000000002e-06 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185595+0000", 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3257e-05 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.275 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.185606+0000", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1419e-05 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193710+0000", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081039819999999992 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:49.193719+0000", 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1670000000000005e-06 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.276 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223661+0000", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0299414410000001 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223739+0000", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8771999999999997e-05 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.223867+0000", 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012705999999999999 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.277 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.225806+0000", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019391670000000001 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.227421+0000", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00161532 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234490+0000", 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070693969999999998 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234537+0000", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6709999999999998e-05 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:58:50.234578+0000", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0608000000000002e-05 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57314.0:2 223.9 223:94beac24:::oid_RoundTripCmpExtPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:08:19.279 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 510.04574688999998, 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16945650800000001, 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57314", 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2222872349", 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.280 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200798+0000", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200800+0000", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1320000000000001e-06 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200808+0000", 2024-03-30T15:08:19.281 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2590000000000003e-06 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200817+0000", 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9069999999999996e-06 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200827+0000", 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0472000000000001e-05 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.282 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200895+0000", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7424999999999999e-05 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.200900+0000", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8420000000000001e-06 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362513+0000", 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16161309700000001 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.362526+0000", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2933e-05 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.368887+0000", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063609690000000002 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369004+0000", 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116635 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.284 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.369145+0000", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014146499999999999 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370068+0000", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092285399999999997 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370108+0000", 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9529e-05 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370170+0000", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1995999999999994e-05 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370216+0000", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5865000000000002e-05 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:49.370255+0000", 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9128000000000002e-05 2024-03-30T15:08:19.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.2.0:1538 101.3 e248/130 101:c3bfc9e7:test-rados-api-smithi028-93253-24::bar:8 v 248'39, mlcod=243'35)", 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 487.04125173, 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.050164583999999998, 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:08:19.287 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205294+0000", 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6100000000000001e-07 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.288 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.205300+0000", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2219999999999999e-06 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227707+0000", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.022406780000000001 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.227716+0000", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5339999999999999e-06 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.289 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231237+0000", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035216029999999999 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.231261+0000", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3981999999999998e-05 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255425+0000", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024164396000000001 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.290 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:12.255458+0000", 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2605999999999999e-05 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24402.0:1 256.7s0 256:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 449.01699251299999, 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005296856, 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.291 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24402", 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2485876767", 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.292 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229553+0000", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229555+0000", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1799999999999999e-06 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229563+0000", 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9759999999999995e-06 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229573+0000", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0592e-05 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229584+0000", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0533000000000001e-05 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229647+0000", 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3118000000000001e-05 2024-03-30T15:08:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:50.229651+0000", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8680000000000001e-06 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232864+0000", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0032131120000001 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.232980+0000", 2024-03-30T15:08:19.295 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011602100000000001 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.233406+0000", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042549999999999999 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234188+0000", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00078191600000000001 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234777+0000", 2024-03-30T15:08:19.296 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00058978600000000002 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:00:51.234850+0000", 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2254000000000003e-05 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:4660 275.0 275:00000000::::head [pgnls start_epoch 344] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e344)", 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 391.00145752100002, 2024-03-30T15:08:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.266339783, 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 4660 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245088+0000", 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245090+0000", 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.667e-06 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.299 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245101+0000", 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0541e-05 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245116+0000", 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5091e-05 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.300 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245129+0000", 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.278e-05 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245194+0000", 2024-03-30T15:08:19.301 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5180999999999998e-05 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.245199+0000", 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.118e-06 2024-03-30T15:08:19.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511197+0000", 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.26599746200000002 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.303 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511236+0000", 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9444999999999999e-05 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.304 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:01:48.511428+0000", 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019149799999999999 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.305 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57803.0:1 280.4s0 280:384b83b2:::oid_RoundTripWriteFullPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 376.00845869800003, 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0069688640000001, 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57803", 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2168501096", 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.307 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238087+0000", 2024-03-30T15:08:19.308 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238088+0000", 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9439999999999999e-06 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.309 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238096+0000", 2024-03-30T15:08:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8110000000000003e-06 2024-03-30T15:08:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.310 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238109+0000", 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3004e-05 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.238121+0000", 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1703e-05 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.246334+0000", 2024-03-30T15:08:19.312 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082125289999999997 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:03.246337+0000", 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.219e-06 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.242417+0000", 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99608009799999997 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.313 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.242549+0000", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132533 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.243002+0000", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045222099999999999 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.243973+0000", 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00097163700000000002 2024-03-30T15:08:19.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.244983+0000", 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001009355 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:04.245055+0000", 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2810000000000003e-05 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35600.0:6785 101.c e463/130 101:33efc025:test-rados-api-smithi028-93253-70::foo:head v 463'57, mlcod=424'55)", 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 236.94135544700001, 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010387468, 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:08:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3799999999999998e-07 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.317 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305197+0000", 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4060000000000002e-06 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309914+0000", 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047169780000000001 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309931+0000", 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7027000000000001e-05 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309985+0000", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4101999999999998e-05 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.310010+0000", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4870000000000001e-05 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315540+0000", 2024-03-30T15:08:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005530559 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315577+0000", 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7187999999999997e-05 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:7982 310.1f 310:fc57b2b2:test-rados-api-smithi028-93253-91::602c57ffb51af99d6f3b54c0ee9587bb110fb990:head [writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 123.921113323, 2024-03-30T15:08:19.320 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063536169999999998, 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7982 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325435+0000", 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.982e-06 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.322 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325446+0000", 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1218e-05 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325469+0000", 2024-03-30T15:08:19.323 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2940000000000001e-05 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325481+0000", 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1826e-05 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325563+0000", 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1933000000000003e-05 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.324 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325569+0000", 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2530000000000001e-06 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330393+0000", 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048238509999999997 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330531+0000", 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013782599999999999 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330675+0000", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144227 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331230+0000", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055465600000000005 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331394+0000", 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016427999999999999 2024-03-30T15:08:19.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331696+0000", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030202799999999998 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331745+0000", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8900999999999998e-05 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331786+0000", 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0695999999999998e-05 2024-03-30T15:08:19.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(5.18,RELEASE,e:545,from:4,reservation_nonce:0)", 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 120.80024392599999, 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037887667999999999, 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.328 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446302+0000", 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1600000000000001e-07 2024-03-30T15:08:19.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446305+0000", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.779e-06 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446312+0000", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5499999999999997e-06 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446317+0000", 2024-03-30T15:08:19.330 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1740000000000002e-06 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446358+0000", 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1288000000000002e-05 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446363+0000", 2024-03-30T15:08:19.331 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.361e-06 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484147+0000", 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037783543000000003 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484166+0000", 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9171999999999999e-05 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.332 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484189+0000", 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3085e-05 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:2937 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94326471853056 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e624)", 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:07:57.655221+0000", 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.591324003, 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032664499999999997, 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2937 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655221+0000", 2024-03-30T15:08:19.334 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655221+0000", 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655223+0000", 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0320000000000002e-06 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.335 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655228+0000", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9609999999999996e-06 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655244+0000", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5358000000000001e-05 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655253+0000", 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5130000000000002e-06 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655339+0000", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5514999999999997e-05 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655433+0000", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4281000000000001e-05 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.655548+0000", 2024-03-30T15:08:19.337 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114985 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:2967 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94326471853056 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:12.657578+0000", 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 6.5889677300000002, 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038676200000000002, 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2967 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.339 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657578+0000", 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657578+0000", 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657581+0000", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6729999999999998e-06 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657593+0000", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2159e-05 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657613+0000", 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9826999999999998e-05 2024-03-30T15:08:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657626+0000", 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.242e-05 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657756+0000", 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013026699999999999 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.342 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657864+0000", 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108488 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:12.657964+0000", 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9927999999999996e-05 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.58535.0:3 323.9 e646/645 323:96df10bd:test-rados-api-smithi028-93253-113::foo:head v 646'1, mlcod=0'0)", 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:15.379656+0000", 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.8668889330000003, 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044392599999999998, 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379656+0000", 2024-03-30T15:08:19.344 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379656+0000", 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.345 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379657+0000", 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1209999999999999e-06 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379665+0000", 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0460000000000006e-06 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.346 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379680+0000", 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4932e-05 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379689+0000", 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4030000000000007e-06 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379739+0000", 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9747999999999999e-05 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.347 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379761+0000", 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2255e-05 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.380050+0000", 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028896899999999998 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.348 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.380100+0000", 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0451999999999997e-05 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:7 323.0 323:00000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.8732310729999999, 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010003161, 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373316+0000", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6390000000000001e-06 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373322+0000", 2024-03-30T15:08:19.351 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9900000000000002e-06 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373331+0000", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8210000000000008e-06 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373341+0000", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0428000000000001e-05 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373391+0000", 2024-03-30T15:08:19.352 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9799999999999998e-05 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373395+0000", 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.228e-06 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.381593+0000", 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081977409999999997 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.353 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.381643+0000", 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0346999999999998e-05 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.383317+0000", 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016741670000000001 2024-03-30T15:08:19.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:2977 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94326471853056 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.659121+0000", 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.5874245769999999, 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00040307899999999998, 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.355 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2977 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659121+0000", 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.356 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659121+0000", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659124+0000", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0280000000000001e-06 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659133+0000", 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.867e-06 2024-03-30T15:08:19.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659168+0000", 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4947000000000003e-05 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659182+0000", 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4917e-05 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.358 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659344+0000", 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016113799999999999 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659417+0000", 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3280000000000003e-05 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.659524+0000", 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106902 2024-03-30T15:08:19.359 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(5.1a,QREQUEST,e:648,from:7,reservation_nonce:10)", 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:18.685121+0000", 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.56142472700000001, 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034612999999999999, 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.360 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685121+0000", 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685121+0000", 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.361 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685125+0000", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0600000000000001e-06 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685137+0000", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2634e-05 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685167+0000", 2024-03-30T15:08:19.362 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9345000000000001e-05 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685184+0000", 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7e-05 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685350+0000", 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016644500000000001 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.363 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685380+0000", 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9839e-05 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.685467+0000", 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6806999999999997e-05 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "replica_scrub(pg: 5.1a,from:0'0,to:0'0,epoch:649/72,start:5:58000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:18.687433+0000", 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.55911194500000005, 2024-03-30T15:08:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00026819499999999999, 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687433+0000", 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687433+0000", 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687434+0000", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9299999999999996e-07 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687443+0000", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1630000000000002e-06 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687483+0000", 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9489000000000001e-05 2024-03-30T15:08:19.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687508+0000", 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5378999999999999e-05 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687631+0000", 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012268000000000001 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:19.367 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687649+0000", 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8223000000000002e-05 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:18.687701+0000", 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2367999999999997e-05 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:19.368 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:08:19.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:19.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- 172.21.15.28:0/1592710419 >> v1:172.21.15.28:6789/0 conn(0x7fd80810aec0 legacy=0x7fd80810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- 172.21.15.28:0/1592710419 shutdown_connections 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- 172.21.15.28:0/1592710419 >> 172.21.15.28:0/1592710419 conn(0x7fd8080fdb00 msgr2=0x7fd8080fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- 172.21.15.28:0/1592710419 shutdown_connections 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- 172.21.15.28:0/1592710419 wait complete. 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 librados: starting msgr at 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 librados: starting objecter 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 Processor -- start 2024-03-30T15:08:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- start start 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 librados: setting wanted keys 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 librados: calling monclient init 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8081a7150 con 0x7fd80810aec0 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8081a8370 con 0x7fd808102230 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.480+0000 7fd80fb64700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8081a95b0 con 0x7fd80810ea60 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80d0ff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd80810aec0 0x7fd8081a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53742/0 (socket says 172.21.15.28:53742) 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80d0ff700 1 -- 172.21.15.28:0/3818761600 learned_addr learned my addr 172.21.15.28:0/3818761600 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80e101700 1 --1- 172.21.15.28:0/3818761600 >> v1:172.21.15.28:6790/0 conn(0x7fd80810ea60 0x7fd8081a5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53644/0 (socket says 172.21.15.28:53644) 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter ms_handle_connect 0x7fd80810aec0 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter ms_handle_connect 0x7fd80810ea60 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter ms_handle_connect 0x7fd808102230 2024-03-30T15:08:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159865307 0 0) 0x7fd8081a7150 con 0x7fd80810aec0 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f4003aa0 con 0x7fd80810aec0 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252230942 0 0) 0x7fd8081a95b0 con 0x7fd80810ea60 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8081a7150 con 0x7fd80810ea60 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944548013 0 0) 0x7fd8081a8370 con 0x7fd808102230 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8081a95b0 con 0x7fd808102230 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3226365277 0 0) 0x7fd8081a7150 con 0x7fd80810ea60 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8081a8370 con 0x7fd80810ea60 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd800001b70 con 0x7fd80810ea60 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 212993161 0 0) 0x7fd7f4003aa0 con 0x7fd80810aec0 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8081a7150 con 0x7fd80810aec0 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd7f8004550 con 0x7fd80810aec0 2024-03-30T15:08:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2263092939 0 0) 0x7fd8081a95b0 con 0x7fd808102230 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7f4003aa0 con 0x7fd808102230 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd804001a30 con 0x7fd808102230 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 363099874 0 0) 0x7fd8081a8370 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 >> v1:172.21.15.134:6789/0 conn(0x7fd808102230 legacy=0x7fd8081075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 >> v1:172.21.15.28:6789/0 conn(0x7fd80810aec0 legacy=0x7fd8081a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081aa7f0 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8081a97e0 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80fb64700 10 client.58553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8081a9ca0 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.481+0000 7fd80fb64700 1 librados: init done 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.482+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd800003ec0 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.482+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd800006010 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.482+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ec005120 con 0x7fd80810ea60 2024-03-30T15:08:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.483+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd800003a20 con 0x7fd80810ea60 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(650..650 src has 442..650) ==== 9102+0+0 (unknown 2461218597 0 0) 0x7fd800089cf0 con 0x7fd80810ea60 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 10 client.58553.objecter ms_dispatch 0x7fd808001550 osd_map(650..650 src has 442..650) 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 3 client.58553.objecter handle_osd_map got epochs [650,650] > 0 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 3 client.58553.objecter handle_osd_map decoding full epoch 650 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 20 client.58553.objecter dump_active .. 0 homeless 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.484+0000 7fd7feffd700 10 client.58553.objecter ms_handle_connect 0x7fd7f40703d0 2024-03-30T15:08:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.488+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd800056b50 con 0x7fd80810ea60 2024-03-30T15:08:19.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:19.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:19.711+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd7ec005410 con 0x7fd80810ea60 2024-03-30T15:08:19.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:19.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[41614]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:19.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:19 smithi028 ceph-mon[29876]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T15:08:19.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:19.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:19.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:19 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:08:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:19 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:08:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:19 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-112"}]': finished 2024-03-30T15:08:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:19 smithi134 ceph-mon[33714]: osdmap e650: 8 total, 8 up, 8 in 2024-03-30T15:08:20.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:20.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:20.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:20.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:20.241 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:20.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:20.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.365+0000 7fd7feffd700 1 -- 172.21.15.28:0/3818761600 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v651) ==== 87+0+0 (unknown 814403685 0 0) 0x7fd80005a5a0 con 0x7fd80810ea60 2024-03-30T15:08:20.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.368+0000 7fd80fb64700 10 librados: watch_flush enter 2024-03-30T15:08:20.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.368+0000 7fd80fb64700 10 librados: watch_flush exit 2024-03-30T15:08:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.368+0000 7fd80fb64700 20 client.58553.objecter shutdown clearing up homeless session... 2024-03-30T15:08:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.368+0000 7fd80fb64700 10 client.58553.objecter successfully canceled tick 2024-03-30T15:08:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd7f40703d0 legacy=0x7fd7f4072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 >> v1:172.21.15.28:6790/0 conn(0x7fd80810ea60 legacy=0x7fd8081a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 shutdown_connections 2024-03-30T15:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 >> 172.21.15.28:0/3818761600 conn(0x7fd8080fdb00 msgr2=0x7fd808111bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 shutdown_connections 2024-03-30T15:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 -- 172.21.15.28:0/3818761600 wait complete. 2024-03-30T15:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.369+0000 7fd80fb64700 1 librados: shutdown 2024-03-30T15:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:08:20.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:20.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:20.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:20.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:20.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:20.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:20.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3818761600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: from='client.58553 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: pgmap v958: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3204518250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: 5.1b deep-scrub starts 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: 5.1b deep-scrub ok 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: from='client.58553 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[29876]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3818761600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: from='client.58553 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: pgmap v958: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3204518250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:20.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: 5.1b deep-scrub starts 2024-03-30T15:08:20.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: 5.1b deep-scrub ok 2024-03-30T15:08:20.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:20.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: from='client.58553 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:08:20.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:20 smithi028 ceph-mon[41614]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T15:08:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:20.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:20.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3818761600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: from='client.58553 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: pgmap v958: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3204518250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: 5.1b deep-scrub starts 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: 5.1b deep-scrub ok 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: from='client.58553 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:08:20.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:20 smithi134 ceph-mon[33714]: osdmap e651: 8 total, 8 up, 8 in 2024-03-30T15:08:20.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.118+0000 7f1bf42f7700 1 -- 172.21.15.28:0/754817641 >> v1:172.21.15.28:6789/0 conn(0x7f1bec101130 legacy=0x7f1bec101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.118+0000 7f1bf42f7700 1 -- 172.21.15.28:0/754817641 shutdown_connections 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.118+0000 7f1bf42f7700 1 -- 172.21.15.28:0/754817641 >> 172.21.15.28:0/754817641 conn(0x7f1bec0fc9e0 msgr2=0x7f1bec0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.118+0000 7f1bf42f7700 1 -- 172.21.15.28:0/754817641 shutdown_connections 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.118+0000 7f1bf42f7700 1 -- 172.21.15.28:0/754817641 wait complete. 2024-03-30T15:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 librados: starting msgr at 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 librados: starting objecter 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 Processor -- start 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 -- start start 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 librados: setting wanted keys 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 librados: calling monclient init 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1bec1a5df0 con 0x7f1bec109dc0 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1bec1a7010 con 0x7f1bec101130 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf42f7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1bec1a8250 con 0x7f1bec10d960 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf1892700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1bec109dc0 0x7f1bec1a0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53762/0 (socket says 172.21.15.28:53762) 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf1892700 1 -- 172.21.15.28:0/3204518250 learned_addr learned my addr 172.21.15.28:0/3204518250 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1bf2894700 1 --1- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6790/0 conn(0x7f1bec10d960 0x7f1bec1a4500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53656/0 (socket says 172.21.15.28:53656) 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1beb7fe700 10 client.?.objecter ms_handle_connect 0x7f1bec109dc0 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1beb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1beb7fe700 10 client.?.objecter ms_handle_connect 0x7f1bec10d960 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.119+0000 7f1beb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 10 client.?.objecter ms_handle_connect 0x7f1bec101130 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172214619 0 0) 0x7f1bec1a8250 con 0x7f1bec10d960 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd40031f0 con 0x7f1bec10d960 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063297779 0 0) 0x7f1bec1a5df0 con 0x7f1bec109dc0 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bec1a8250 con 0x7f1bec109dc0 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110758784 0 0) 0x7f1bec1a7010 con 0x7f1bec101130 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bec1a5df0 con 0x7f1bec101130 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2280848000 0 0) 0x7f1bd40031f0 con 0x7f1bec10d960 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bec1a7010 con 0x7f1bec10d960 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3714632207 0 0) 0x7f1bec1a8250 con 0x7f1bec109dc0 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bd40031f0 con 0x7f1bec109dc0 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1bdc001890 con 0x7f1bec10d960 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1be0001230 con 0x7f1bec109dc0 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2835639799 0 0) 0x7f1bec1a5df0 con 0x7f1bec101130 2024-03-30T15:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bec1a8250 con 0x7f1bec101130 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1be40047f0 con 0x7f1bec101130 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 267086672 0 0) 0x7f1bec1a7010 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.134:6789/0 conn(0x7f1bec101130 legacy=0x7f1bec106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6789/0 conn(0x7f1bec109dc0 legacy=0x7f1bec1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec1a9490 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec1a8480 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1bf42f7700 10 client.58559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1bec1a8940 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1bf42f7700 1 librados: init done 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1bdc001c80 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.120+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1bdc005c10 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.121+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1bd8005100 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1bdc003950 con 0x7f1bec10d960 2024-03-30T15:08:21.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(650..650 src has 442..650) ==== 9102+0+0 (unknown 2461218597 0 0) 0x7f1bdc0899c0 con 0x7f1bec10d960 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 osd_map(650..650 src has 442..650) 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 3 client.58559.objecter handle_osd_map got epochs [650,650] > 0 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 3 client.58559.objecter handle_osd_map decoding full epoch 650 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 20 client.58559.objecter dump_active .. 0 homeless 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v650) ==== 54+0+16 (unknown 3359304020 0 2420000192) 0x7f1bdc089d80 con 0x7f1bec10d960 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.122+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd40703c0 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 1 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 1 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 0 1 2024-03-30T15:08:21.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 4 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 10 client.58559.objecter _send_command 1 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1bd8004b50 con 0x7f1bd8004730 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd8004730 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd8004b50 con 0x7f1bd8004730 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 1: 0 ) 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 1 = system:0 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 4 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.123+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 0 1 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:08:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 2 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 2 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80053e0 osd=0 3 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 0 2 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 4 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 10 client.58559.objecter _send_command 2 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.141+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd8007b00 con 0x7f1bd8004730 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd8007b00 con 0x7f1bd8004730 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 4 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 0 2 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 1 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 4 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 10 client.58559.objecter _send_command 3 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1bd8002200 con 0x7f1bd80042f0 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.143+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd80042f0 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.144+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd8002200 con 0x7f1bd80042f0 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.144+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 3: 0 ) 2024-03-30T15:08:21.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.144+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 3 = system:0 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.144+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.144+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 1 3 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8004180 osd=1 3 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 1 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 10 client.58559.objecter _send_command 4 2024-03-30T15:08:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.161+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd8001660 con 0x7f1bd80042f0 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd8001660 con 0x7f1bd80042f0 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 4 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 1 4 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 5 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 5 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 2 5 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 4 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 10 client.58559.objecter _send_command 5 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1bd800c250 con 0x7f1bd800a6c0 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.163+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd800a6c0 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.164+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd800c250 con 0x7f1bd800a6c0 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.164+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 5: 0 ) 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.164+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 5 = system:0 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.164+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 4 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.164+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 2 5 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 6 2024-03-30T15:08:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 6 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8008480 osd=2 3 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 2 6 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 4 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 10 client.58559.objecter _send_command 6 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.181+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd8007f00 con 0x7f1bd800a6c0 2024-03-30T15:08:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd8007f00 con 0x7f1bd800a6c0 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 4 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 2 6 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 7 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 7 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 3 7 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 4 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 10 client.58559.objecter _send_command 7 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.182+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1bd80082d0 con 0x7f1bd800a220 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd800a220 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd80082d0 con 0x7f1bd800a220 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 7: 0 ) 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 7 = system:0 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 4 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.183+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 3 7 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 8 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 8 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd800c490 osd=3 3 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 3 8 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 4 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 10 client.58559.objecter _send_command 8 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.201+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd8009790 con 0x7f1bd800a220 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd8009790 con 0x7f1bd800a220 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 4 2024-03-30T15:08:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 3 8 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 9 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 9 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 4 9 2024-03-30T15:08:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 4 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 10 client.58559.objecter _send_command 9 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.202+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1bd80096b0 con 0x7f1bd8016240 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.203+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd8016240 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.204+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd80096b0 con 0x7f1bd8016240 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.204+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 9: 0 ) 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.204+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 9 = system:0 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.204+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 4 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.204+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 4 9 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 10 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 10 2024-03-30T15:08:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8009540 osd=4 3 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 4 10 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 4 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 10 client.58559.objecter _send_command 10 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.221+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd80096b0 con 0x7f1bd8016240 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd80096b0 con 0x7f1bd8016240 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 4 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:08:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 4 10 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 11 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 11 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 5 11 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 4 2024-03-30T15:08:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 10 client.58559.objecter _send_command 11 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.223+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1bd8022e00 con 0x7f1bd801f250 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd801f250 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd8022e00 con 0x7f1bd801f250 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 11: 0 ) 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 11 = system:0 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 4 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.224+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 5 11 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 12 2024-03-30T15:08:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.051 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:08:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 12 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80096b0 osd=5 3 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 5 12 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 4 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 10 client.58559.objecter _send_command 12 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.245+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd80247b0 con 0x7f1bd801f250 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.247+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd80247b0 con 0x7f1bd801f250 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.247+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.247+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.247+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 4 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.247+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 5 12 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 13 2024-03-30T15:08:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 13 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 6 13 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 4 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 10 client.58559.objecter _send_command 13 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1bd8025350 con 0x7f1bd8024ed0 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.248+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd8024ed0 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.249+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd8025350 con 0x7f1bd8024ed0 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.249+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 13: 0 ) 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.249+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 13 = system:0 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.249+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 4 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.249+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 6 13 2024-03-30T15:08:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 14 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 14 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd80247b0 osd=6 3 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 6 14 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 4 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 10 client.58559.objecter _send_command 14 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.274+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd8025350 con 0x7f1bd8024ed0 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd8025350 con 0x7f1bd8024ed0 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 4 2024-03-30T15:08:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 6 14 2024-03-30T15:08:21.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:21.055 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:08:21.055 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 15 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 15 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 7 15 2024-03-30T15:08:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 4 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 10 client.58559.objecter _send_command 15 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1bd8025770 con 0x7f1bd8027ec0 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.276+0000 7f1beb7fe700 10 client.58559.objecter ms_handle_connect 0x7f1bd8027ec0 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.277+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1bd8025770 con 0x7f1bd8027ec0 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.277+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 15: 0 ) 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.277+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 15 = system:0 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.277+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 4 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.277+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 7 15 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 10 client.58559.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign -1 16 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 20 client.58559.objecter _get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_remove -1 16 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 20 client.58559.objecter get_session s=0x7f1bd8025350 osd=7 3 2024-03-30T15:08:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 15 client.58559.objecter _session_command_op_assign 7 16 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 4 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 10 client.58559.objecter _send_command 16 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.299+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1bd802d1a0 con 0x7f1bd8027ec0 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1beb7fe700 1 -- 172.21.15.28:0/3204518250 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1bd802d1a0 con 0x7f1bd8027ec0 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1beb7fe700 10 client.58559.objecter ms_dispatch 0x7f1bec001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1beb7fe700 10 client.58559.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1beb7fe700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 4 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1beb7fe700 15 client.58559.objecter _session_command_op_remove 7 16 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 librados: watch_flush enter 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 librados: watch_flush exit 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.0 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6801/3394323575 conn(0x7f1bd8004730 legacy=0x7f1bd8002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80053e0 osd=0 1 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.1 2024-03-30T15:08:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6805/1850313547 conn(0x7f1bd80042f0 legacy=0x7f1bd8008e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8004180 osd=1 1 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.2 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6809/3451325043 conn(0x7f1bd800a6c0 legacy=0x7f1bd800ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8008480 osd=2 1 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.3 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6813/1506157250 conn(0x7f1bd800a220 legacy=0x7f1bd8018c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd800c490 osd=3 1 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.4 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf2894700 1 -- 172.21.15.28:0/3204518250 reap_dead start 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.134:6800/3063775514 conn(0x7f1bd8016240 legacy=0x7f1bd801db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf2894700 1 -- 172.21.15.28:0/3204518250 reap_dead start 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8009540 osd=4 1 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.5 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.134:6804/3668151179 conn(0x7f1bd801f250 legacy=0x7f1bd8021640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80096b0 osd=5 1 2024-03-30T15:08:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.6 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.134:6808/2135224194 conn(0x7f1bd8024ed0 legacy=0x7f1bd8022fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd80247b0 osd=6 1 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 10 client.58559.objecter close_session for osd.7 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.301+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.134:6812/3758190298 conn(0x7f1bd8027ec0 legacy=0x7f1bd802a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 20 client.58559.objecter put_session s=0x7f1bd8025350 osd=7 1 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 20 client.58559.objecter shutdown clearing up homeless session... 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 10 client.58559.objecter successfully canceled tick 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1bd40703c0 legacy=0x7f1bd4072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> v1:172.21.15.28:6790/0 conn(0x7f1bec10d960 legacy=0x7f1bec1a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf2894700 1 -- 172.21.15.28:0/3204518250 reap_dead start 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 shutdown_connections 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 >> 172.21.15.28:0/3204518250 conn(0x7f1bec0fc9e0 msgr2=0x7f1bec10ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 shutdown_connections 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 -- 172.21.15.28:0/3204518250 wait complete. 2024-03-30T15:08:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:20.302+0000 7f1bf42f7700 1 librados: shutdown 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:08:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:08:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:21.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:21.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:21.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:21.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:21.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:21.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:21.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:21.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:21.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:21.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:21.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:21.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: 12.13 deep-scrub starts 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: 12.13 deep-scrub ok 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:08:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T15:08:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: 12.13 deep-scrub starts 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: 12.13 deep-scrub ok 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: 12.13 deep-scrub starts 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: 12.13 deep-scrub ok 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: osdmap e652: 8 total, 8 up, 8 in 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:21.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:22.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:22.111 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:22.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:22.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:22.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:22.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:22.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:22.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:22.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:22.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: pgmap v961: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: pgmap v961: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T15:08:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:22.757 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:22.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:22.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: pgmap v961: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: osdmap e653: 8 total, 8 up, 8 in 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:22.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:23.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:23.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:23.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:23.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:23.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:23.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:23.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:23.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:23.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:23.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:08:23.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:23.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:23.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:23.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:24.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:24.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:24.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:24.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:24.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:24.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:24.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:08:24.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[41614]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[41614]: pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:08:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[29876]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T15:08:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:24 smithi028 ceph-mon[29876]: pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:08:24.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:24.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:24 smithi134 ceph-mon[33714]: osdmap e654: 8 total, 8 up, 8 in 2024-03-30T15:08:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:24 smithi134 ceph-mon[33714]: pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-30T15:08:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:24.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:24.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:24.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:24.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:24.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:25.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:25.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:25.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:25.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:25.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:25.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:25.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:25.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:25.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:25.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[41614]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T15:08:25.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.585 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[29876]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T15:08:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:25 smithi134 ceph-mon[33714]: osdmap e655: 8 total, 8 up, 8 in 2024-03-30T15:08:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]: dispatch 2024-03-30T15:08:25.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:25.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.802+0000 7ff38b9b6700 1 -- 172.21.15.28:0/3747302821 >> v1:172.21.15.28:6790/0 conn(0x7ff384101140 legacy=0x7ff384101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.802+0000 7ff38b9b6700 1 -- 172.21.15.28:0/3747302821 shutdown_connections 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.802+0000 7ff38b9b6700 1 -- 172.21.15.28:0/3747302821 >> 172.21.15.28:0/3747302821 conn(0x7ff3840fc9d0 msgr2=0x7ff3840fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- 172.21.15.28:0/3747302821 shutdown_connections 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- 172.21.15.28:0/3747302821 wait complete. 2024-03-30T15:08:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 librados: starting msgr at 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 librados: starting objecter 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 Processor -- start 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- start start 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 librados: setting wanted keys 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 librados: calling monclient init 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff384106880 con 0x7ff384101140 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3841a8550 con 0x7ff384109dd0 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff38b9b6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3841a9730 con 0x7ff38410d970 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff389f53700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff38410d970 0x7ff3841a5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53676/0 (socket says 172.21.15.28:53676) 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff389752700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff384101140 0x7ff384104650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53768/0 (socket says 172.21.15.28:53768) 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.803+0000 7ff389f53700 1 -- 172.21.15.28:0/2961850997 learned_addr learned my addr 172.21.15.28:0/2961850997 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter ms_handle_connect 0x7ff384101140 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter ms_handle_connect 0x7ff38410d970 2024-03-30T15:08:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter ms_handle_connect 0x7ff384109dd0 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102917692 0 0) 0x7ff384106880 con 0x7ff384101140 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff370003480 con 0x7ff384101140 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942345309 0 0) 0x7ff3841a9730 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff384106880 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008830982 0 0) 0x7ff3841a8550 con 0x7ff384109dd0 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3841a9730 con 0x7ff384109dd0 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3659491936 0 0) 0x7ff384106880 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3841a8550 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff37c001db0 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2581566113 0 0) 0x7ff3841a8550 con 0x7ff38410d970 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 >> v1:172.21.15.134:6789/0 conn(0x7ff384109dd0 legacy=0x7ff384104d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.804+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 >> v1:172.21.15.28:6789/0 conn(0x7ff384101140 legacy=0x7ff384104650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3841aa910 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3841a9900 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff38b9b6700 10 client.58565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3841a9dc0 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff38b9b6700 1 librados: init done 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff37c002010 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.805+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff37c006000 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.806+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff368005120 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.806+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff37c0072d0 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(656..656 src has 442..656) ==== 9110+0+0 (unknown 750170860 0 0) 0x7ff37c089d80 con 0x7ff38410d970 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 10 client.58565.objecter ms_dispatch 0x7ff384001550 osd_map(656..656 src has 442..656) 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 3 client.58565.objecter handle_osd_map got epochs [656,656] > 0 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 3 client.58565.objecter handle_osd_map decoding full epoch 656 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 20 client.58565.objecter dump_active .. 0 homeless 2024-03-30T15:08:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.807+0000 7ff37a7fc700 10 client.58565.objecter ms_handle_connect 0x7ff37006fe70 2024-03-30T15:08:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:25.812+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff37c0ba270 con 0x7ff38410d970 2024-03-30T15:08:25.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:25.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:26.043 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:08:26.044 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:08:26.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:26.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.117+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7ff368005410 con 0x7ff38410d970 2024-03-30T15:08:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.117+0000 7ff37a7fc700 1 -- 172.21.15.28:0/2961850997 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+2582 (unknown 3226119948 0 1444883358) 0x7ff37c056bc0 con 0x7ff38410d970 2024-03-30T15:08:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:08:26.120 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","health":{"status":"HEALTH_WARN","checks":{"CACHE_POOL_NO_HIT_SET":{"severity":"HEALTH_WARN","summary":{"message":"1 cache pools are missing hit_sets","count":1},"muted":false},"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"3 pool(s) do not have an application enabled","count":3},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1057,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":656,"num_osds":8,"num_up_osds":8,"osd_up_since":1711810462,"num_in_osds":8,"osd_in_since":1711810445,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":246}],"num_pgs":246,"num_pools":10,"num_objects":201,"data_bytes":596335,"bytes_used":649469952,"bytes_avail":767310815232,"bytes_total":767960285184,"read_bytes_sec":767,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.28:8443/","prometheus":"http://172.21.15.28:9283/"}},"servicemap":{"epoch":20,"modified":"2024-03-30T14:58:17.631614+0000","services":{"rgw":{"daemons":{"summary":"","24394":{"start_epoch":12,"start_stamp":"2024-03-30T14:54:44.439972+0000","gid":24394,"addr":"172.21.15.28:0/2248569927","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1967-g4e50bbcd (4e50bbcdd1630c7760b708f3a5b4d38a121e66cb) squid (dev)","ceph_version_short":"19.0.0-1967-g4e50bbcd","container_hostname":"smithi028","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi028","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024","kernel_version":"5.14.0-434.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32494656","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"df97c51e-80b0-40c1-8746-45f996cee991","zone_name":"default","zonegroup_id":"ac105772-b6a2-47ca-98b4-17fcbbaa36c6","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 10 librados: watch_flush enter 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 10 librados: watch_flush exit 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 20 client.58565.objecter shutdown clearing up homeless session... 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 10 client.58565.objecter successfully canceled tick 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff37006fe70 legacy=0x7ff370072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.119+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 >> v1:172.21.15.28:6790/0 conn(0x7ff38410d970 legacy=0x7ff3841a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.120+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 shutdown_connections 2024-03-30T15:08:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.120+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 >> 172.21.15.28:0/2961850997 conn(0x7ff3840fc9d0 msgr2=0x7ff3841105e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.120+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 shutdown_connections 2024-03-30T15:08:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.120+0000 7ff38b9b6700 1 -- 172.21.15.28:0/2961850997 wait complete. 2024-03-30T15:08:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:26.120+0000 7ff38b9b6700 1 librados: shutdown 2024-03-30T15:08:26.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:26.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:26.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:26.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:26.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:26.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:26.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:26.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:26.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]': finished 2024-03-30T15:08:26.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T15:08:26.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2961850997' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]': finished 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:26.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2961850997' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-30T15:08:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-114", "mode": "writeback"}]': finished 2024-03-30T15:08:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: osdmap e656: 8 total, 8 up, 8 in 2024-03-30T15:08:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:08:26.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2961850997' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-30T15:08:26.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:26.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:26.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:26.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:26.975 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:26.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:26.991 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-30T15:08:26.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:08:27.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:27.103 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:27.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:27.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:27.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:27.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:27.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:27.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:08:27.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:27.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:27 smithi134 ceph-mon[33714]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T15:08:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:27 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:27.704 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[41614]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[29876]: osdmap e657: 8 total, 8 up, 8 in 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]: dispatch 2024-03-30T15:08:27.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:27 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:27.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:27.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:27.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:27.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:27.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:28.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:28.122 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:28.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:28.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:28.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:28.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:28.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:28.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:28.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:28.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T15:08:28.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:28 smithi134 ceph-mon[33714]: pgmap v970: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T15:08:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[29876]: pgmap v970: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-114"}]': finished 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: osdmap e658: 8 total, 8 up, 8 in 2024-03-30T15:08:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:28 smithi028 ceph-mon[41614]: pgmap v970: 246 pgs: 246 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:08:28.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:28.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:28.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:28.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:29.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:29.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:29.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:29.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:29.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:29.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:29.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:29.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:29.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:29.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:29 smithi134 ceph-mon[33714]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T15:08:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:29 smithi028 ceph-mon[41614]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T15:08:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:29 smithi028 ceph-mon[29876]: osdmap e659: 8 total, 8 up, 8 in 2024-03-30T15:08:29.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:29.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:29.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:30.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:30.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:30.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:30.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:30.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:30.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:30.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[29876]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[29876]: pgmap v973: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[41614]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:30 smithi028 ceph-mon[41614]: pgmap v973: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:30.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:30.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:30.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:30 smithi134 ceph-mon[33714]: osdmap e660: 8 total, 8 up, 8 in 2024-03-30T15:08:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:30.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:30 smithi134 ceph-mon[33714]: pgmap v973: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:30.784 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:30.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:30.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:30.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:30.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:30.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:31.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:31.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:31.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:31.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:31.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:31.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:31.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:31.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:31.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:31.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[41614]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T15:08:31.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:31.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[29876]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T15:08:31.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:31.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 -- 172.21.15.28:0/546984125 >> v1:172.21.15.28:6789/0 conn(0x7fbeec101130 legacy=0x7fbeec101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 -- 172.21.15.28:0/546984125 shutdown_connections 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 -- 172.21.15.28:0/546984125 >> 172.21.15.28:0/546984125 conn(0x7fbeec0fc9e0 msgr2=0x7fbeec0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 -- 172.21.15.28:0/546984125 shutdown_connections 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 -- 172.21.15.28:0/546984125 wait complete. 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 librados: starting msgr at 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 librados: starting objecter 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.474+0000 7fbef12c2700 1 Processor -- start 2024-03-30T15:08:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 -- start start 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 librados: setting wanted keys 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 librados: calling monclient init 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbeec1a5df0 con 0x7fbeec10d960 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbeec1a7010 con 0x7fbeec109dc0 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbef12c2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbeec1a8250 con 0x7fbeec101130 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbeeb7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbeec10d960 0x7fbeec1a4500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34750/0 (socket says 172.21.15.28:34750) 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbeeb7fe700 1 -- 172.21.15.28:0/696048921 learned_addr learned my addr 172.21.15.28:0/696048921 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbeeaffd700 1 --1- 172.21.15.28:0/696048921 >> v1:172.21.15.28:6790/0 conn(0x7fbeec101130 0x7fbeec106290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47694/0 (socket says 172.21.15.28:47694) 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter ms_handle_connect 0x7fbeec101130 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter ms_handle_connect 0x7fbeec10d960 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter ms_handle_connect 0x7fbeec109dc0 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451896657 0 0) 0x7fbeec1a8250 con 0x7fbeec101130 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbed4003aa0 con 0x7fbeec101130 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464103143 0 0) 0x7fbeec1a5df0 con 0x7fbeec10d960 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeec1a8250 con 0x7fbeec10d960 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449348859 0 0) 0x7fbeec1a7010 con 0x7fbeec109dc0 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeec1a5df0 con 0x7fbeec109dc0 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1847805639 0 0) 0x7fbed4003aa0 con 0x7fbeec101130 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbeec1a7010 con 0x7fbeec101130 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbedc004490 con 0x7fbeec101130 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1520933470 0 0) 0x7fbeec1a8250 con 0x7fbeec10d960 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbed4003aa0 con 0x7fbeec10d960 2024-03-30T15:08:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.475+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbee4001bb0 con 0x7fbeec10d960 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3648623070 0 0) 0x7fbeec1a5df0 con 0x7fbeec109dc0 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbeec1a8250 con 0x7fbeec109dc0 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbee0001050 con 0x7fbeec109dc0 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2724560088 0 0) 0x7fbeec1a7010 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 >> v1:172.21.15.134:6789/0 conn(0x7fbeec109dc0 legacy=0x7fbeec1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 >> v1:172.21.15.28:6789/0 conn(0x7fbeec10d960 legacy=0x7fbeec1a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec1a9490 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec1a6020 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbef12c2700 10 client.58571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbeec1a64e0 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbef12c2700 1 librados: init done 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbedc001a40 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.476+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbedc004c70 con 0x7fbeec101130 2024-03-30T15:08:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.477+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed8005120 con 0x7fbeec101130 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.477+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbedc003680 con 0x7fbeec101130 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(662..662 src has 442..662) ==== 9115+0+0 (unknown 3888506600 0 0) 0x7fbedc089990 con 0x7fbeec101130 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 10 client.58571.objecter ms_dispatch 0x7fbeec001550 osd_map(662..662 src has 442..662) 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 3 client.58571.objecter handle_osd_map got epochs [662,662] > 0 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 3 client.58571.objecter handle_osd_map decoding full epoch 662 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 20 client.58571.objecter dump_active .. 0 homeless 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.478+0000 7fbed3fff700 10 client.58571.objecter ms_handle_connect 0x7fbed4070470 2024-03-30T15:08:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.482+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbedc0567c0 con 0x7fbeec101130 2024-03-30T15:08:31.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:31 smithi134 ceph-mon[33714]: osdmap e661: 8 total, 8 up, 8 in 2024-03-30T15:08:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:31.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:31.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:31.710+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fbed8005410 con 0x7fbeec101130 2024-03-30T15:08:31.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:31.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.852 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:31.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:31.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:31.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:32.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:32.096 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:08:32.097 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:08:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:32.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:32.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:32.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.357+0000 7fbed3fff700 1 -- 172.21.15.28:0/696048921 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v663) ==== 97+0+0 (unknown 897427828 0 0) 0x7fbedc05a210 con 0x7fbeec101130 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 10 librados: watch_flush enter 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 10 librados: watch_flush exit 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 20 client.58571.objecter shutdown clearing up homeless session... 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 10 client.58571.objecter successfully canceled tick 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbed4070470 legacy=0x7fbed4072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.361+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 >> v1:172.21.15.28:6790/0 conn(0x7fbeec101130 legacy=0x7fbeec106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.362+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 shutdown_connections 2024-03-30T15:08:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.362+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 >> 172.21.15.28:0/696048921 conn(0x7fbeec0fc9e0 msgr2=0x7fbeec10ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.362+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 shutdown_connections 2024-03-30T15:08:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.362+0000 7fbef12c2700 1 -- 172.21.15.28:0/696048921 wait complete. 2024-03-30T15:08:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:32.362+0000 7fbef12c2700 1 librados: shutdown 2024-03-30T15:08:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:32.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:08:32.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:32.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/696048921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.58571 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: pgmap v976: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:32.607 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.58571 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/696048921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.58571 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: pgmap v976: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.58571 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: osdmap e662: 8 total, 8 up, 8 in 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/696048921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.58571 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:32.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: pgmap v976: 246 pgs: 32 unknown, 214 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.58571 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: osdmap e663: 8 total, 8 up, 8 in 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:32.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:32.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:32.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:32.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:32.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:33.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:33.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:33.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:33.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 -- 172.21.15.28:0/1967023988 >> v1:172.21.15.28:6789/0 conn(0x7f543c109dc0 legacy=0x7f543c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 -- 172.21.15.28:0/1967023988 shutdown_connections 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 -- 172.21.15.28:0/1967023988 >> 172.21.15.28:0/1967023988 conn(0x7f543c0fc9e0 msgr2=0x7f543c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 -- 172.21.15.28:0/1967023988 shutdown_connections 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 -- 172.21.15.28:0/1967023988 wait complete. 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 librados: starting msgr at 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 librados: starting objecter 2024-03-30T15:08:33.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.179+0000 7f54420d1700 1 Processor -- start 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 -- start start 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 librados: setting wanted keys 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 librados: calling monclient init 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f543c10fa00 con 0x7f543c10d960 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f543c10fc10 con 0x7f543c109dc0 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f54420d1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f543c10fe20 con 0x7f543c101130 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f543bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f543c10d960 0x7f543c1a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34766/0 (socket says 172.21.15.28:34766) 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f543bfff700 1 -- 172.21.15.28:0/3044382936 learned_addr learned my addr 172.21.15.28:0/3044382936 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:33.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f543b7fe700 1 --1- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6790/0 conn(0x7f543c101130 0x7f543c10dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47716/0 (socket says 172.21.15.28:47716) 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter ms_handle_connect 0x7f543c10d960 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter ms_handle_connect 0x7f543c101130 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter ms_handle_connect 0x7f543c109dc0 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494559816 0 0) 0x7f543c10fa00 con 0x7f543c10d960 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54240031f0 con 0x7f543c10d960 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523784669 0 0) 0x7f542c0018f0 con 0x7f543c101130 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f543c10fa00 con 0x7f543c101130 2024-03-30T15:08:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244923336 0 0) 0x7f543c10fc10 con 0x7f543c109dc0 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.180+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f542c0018f0 con 0x7f543c109dc0 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1266980266 0 0) 0x7f54240031f0 con 0x7f543c10d960 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f543c10fc10 con 0x7f543c10d960 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5434001c90 con 0x7f543c10d960 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3429926349 0 0) 0x7f543c10fa00 con 0x7f543c101130 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54240031f0 con 0x7f543c101130 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4181396748 0 0) 0x7f542c0018f0 con 0x7f543c109dc0 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f543c10fa00 con 0x7f543c109dc0 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f542c0012d0 con 0x7f543c101130 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5430001910 con 0x7f543c109dc0 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1302836346 0 0) 0x7f54240031f0 con 0x7f543c101130 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.134:6789/0 conn(0x7f543c109dc0 legacy=0x7f543c10e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6789/0 conn(0x7f543c10d960 legacy=0x7f543c1a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c1ac9b0 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f543c1a9980 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f54420d1700 10 client.58577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f543c1a9e40 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f54420d1700 1 librados: init done 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f542c003b20 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.181+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f542c005cd0 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.182+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5428005100 con 0x7f543c101130 2024-03-30T15:08:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f542c005f60 con 0x7f543c101130 2024-03-30T15:08:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(663..663 src has 442..663) ==== 9110+0+0 (unknown 4085275653 0 0) 0x7f542c0899e0 con 0x7f543c101130 2024-03-30T15:08:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 osd_map(663..663 src has 442..663) 2024-03-30T15:08:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 3 client.58577.objecter handle_osd_map got epochs [663,663] > 0 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 3 client.58577.objecter handle_osd_map decoding full epoch 663 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 20 client.58577.objecter dump_active .. 0 homeless 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v663) ==== 54+0+16 (unknown 282548865 0 2420000192) 0x7f543c10fe20 con 0x7f543c101130 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f5424070450 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 10 client.58577.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 1 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 1 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 0 1 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 4 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 10 client.58577.objecter _send_command 1 2024-03-30T15:08:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.183+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5428004b50 con 0x7f5428004730 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f5428004730 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5428004b50 con 0x7f5428004730 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 1: 0 ) 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 10 client.58577.objecter _finish_command 1 = system:0 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 4 2024-03-30T15:08:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.184+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 0 1 2024-03-30T15:08:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 10 client.58577.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 2 2024-03-30T15:08:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 2 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f54280053e0 osd=0 3 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 0 2 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 4 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 10 client.58577.objecter _send_command 2 2024-03-30T15:08:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.201+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5428007b00 con 0x7f5428004730 2024-03-30T15:08:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5428007b00 con 0x7f5428004730 2024-03-30T15:08:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 10 client.58577.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 4 2024-03-30T15:08:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 0 2 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 10 client.58577.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 3 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 3 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 1 3 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 4 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 10 client.58577.objecter _send_command 3 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5428002200 con 0x7f54280042f0 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.203+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f54280042f0 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.204+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5428002200 con 0x7f54280042f0 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.204+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 3: 0 ) 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.204+0000 7f5438ff9700 10 client.58577.objecter _finish_command 3 = system:0 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.204+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 4 2024-03-30T15:08:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.204+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 1 3 2024-03-30T15:08:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 10 client.58577.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 4 2024-03-30T15:08:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 4 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428004180 osd=1 3 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 1 4 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 4 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 10 client.58577.objecter _send_command 4 2024-03-30T15:08:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.221+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5428001600 con 0x7f54280042f0 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.222+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5428001600 con 0x7f54280042f0 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f5438ff9700 10 client.58577.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 4 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 1 4 2024-03-30T15:08:33.224 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 10 client.58577.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 5 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 5 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 2 5 2024-03-30T15:08:33.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 4 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 10 client.58577.objecter _send_command 5 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f54280111f0 con 0x7f542800d6e0 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.223+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f542800d6e0 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.224+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f54280111f0 con 0x7f542800d6e0 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.224+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 5: 0 ) 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.224+0000 7f5438ff9700 10 client.58577.objecter _finish_command 5 = system:0 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.224+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 4 2024-03-30T15:08:33.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.224+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 2 5 2024-03-30T15:08:33.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 10 client.58577.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 6 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 6 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428008480 osd=2 3 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 2 6 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 4 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 10 client.58577.objecter _send_command 6 2024-03-30T15:08:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.240+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5428007fc0 con 0x7f542800d6e0 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5428007fc0 con 0x7f542800d6e0 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 10 client.58577.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 4 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 2 6 2024-03-30T15:08:33.243 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 10 client.58577.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 7 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 7 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 3 7 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 4 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 10 client.58577.objecter _send_command 7 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5428016030 con 0x7f5428012520 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.242+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f5428012520 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.243+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5428016030 con 0x7f5428012520 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.243+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 7: 0 ) 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.243+0000 7f5438ff9700 10 client.58577.objecter _finish_command 7 = system:0 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.243+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 4 2024-03-30T15:08:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.243+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 3 7 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 10 client.58577.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 8 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 8 2024-03-30T15:08:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428008250 osd=3 3 2024-03-30T15:08:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 3 8 2024-03-30T15:08:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 4 2024-03-30T15:08:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 10 client.58577.objecter _send_command 8 2024-03-30T15:08:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.260+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542800ca00 con 0x7f5428012520 2024-03-30T15:08:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:33.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:33.263 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:08:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542800ca00 con 0x7f5428012520 2024-03-30T15:08:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f5438ff9700 10 client.58577.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 4 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 3 8 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 10 client.58577.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 9 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 9 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 4 9 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 4 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 10 client.58577.objecter _send_command 9 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.261+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f542800c990 con 0x7f5428012080 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.262+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f5428012080 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.263+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542800c990 con 0x7f5428012080 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.263+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 9: 0 ) 2024-03-30T15:08:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.263+0000 7f5438ff9700 10 client.58577.objecter _finish_command 9 = system:0 2024-03-30T15:08:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.263+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 4 2024-03-30T15:08:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.263+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 4 9 2024-03-30T15:08:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 10 client.58577.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 10 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 10 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542800c7b0 osd=4 3 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 4 10 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 4 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 10 client.58577.objecter _send_command 10 2024-03-30T15:08:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.279+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542801aa90 con 0x7f5428012080 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.281+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542801aa90 con 0x7f5428012080 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.281+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.281+0000 7f5438ff9700 10 client.58577.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.281+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 4 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.281+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 4 10 2024-03-30T15:08:33.283 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 10 client.58577.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 11 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 11 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 5 11 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 4 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 10 client.58577.objecter _send_command 11 2024-03-30T15:08:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f542801cdf0 con 0x7f542801b1b0 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.282+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f542801b1b0 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.283+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542801cdf0 con 0x7f542801b1b0 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.283+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 11: 0 ) 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.283+0000 7f5438ff9700 10 client.58577.objecter _finish_command 11 = system:0 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.283+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 4 2024-03-30T15:08:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.283+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 5 11 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.299+0000 7f54420d1700 10 client.58577.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 12 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 12 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542801aa90 osd=5 3 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 5 12 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 4 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 10 client.58577.objecter _send_command 12 2024-03-30T15:08:33.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.300+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542801cdf0 con 0x7f542801b1b0 2024-03-30T15:08:33.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.301+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542801cdf0 con 0x7f542801b1b0 2024-03-30T15:08:33.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f5438ff9700 10 client.58577.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 4 2024-03-30T15:08:33.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 5 12 2024-03-30T15:08:33.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 10 client.58577.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 13 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 13 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 6 13 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 4 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 10 client.58577.objecter _send_command 13 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.302+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f542801d200 con 0x7f542801f940 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f542801f940 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542801d200 con 0x7f542801f940 2024-03-30T15:08:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 13: 0 ) 2024-03-30T15:08:33.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 10 client.58577.objecter _finish_command 13 = system:0 2024-03-30T15:08:33.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 4 2024-03-30T15:08:33.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.303+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 6 13 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 10 client.58577.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 14 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 14 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f542801cdf0 osd=6 3 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 6 14 2024-03-30T15:08:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 4 2024-03-30T15:08:33.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 10 client.58577.objecter _send_command 14 2024-03-30T15:08:33.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.320+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5428024c80 con 0x7f542801f940 2024-03-30T15:08:33.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5428024c80 con 0x7f542801f940 2024-03-30T15:08:33.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f5438ff9700 10 client.58577.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 4 2024-03-30T15:08:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 6 14 2024-03-30T15:08:33.324 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:08:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 10 client.58577.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 15 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 15 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 7 15 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 4 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 10 client.58577.objecter _send_command 15 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.322+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5428025820 con 0x7f54280253a0 2024-03-30T15:08:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.323+0000 7f5438ff9700 10 client.58577.objecter ms_handle_connect 0x7f54280253a0 2024-03-30T15:08:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.324+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5428025820 con 0x7f54280253a0 2024-03-30T15:08:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.324+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 15: 0 ) 2024-03-30T15:08:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.324+0000 7f5438ff9700 10 client.58577.objecter _finish_command 15 = system:0 2024-03-30T15:08:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.324+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 4 2024-03-30T15:08:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.324+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 7 15 2024-03-30T15:08:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 10 client.58577.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign -1 16 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 20 client.58577.objecter _get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 15 client.58577.objecter _session_command_op_remove -1 16 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 20 client.58577.objecter get_session s=0x7f5428024c80 osd=7 3 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 15 client.58577.objecter _session_command_op_assign 7 16 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 4 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 10 client.58577.objecter _send_command 16 2024-03-30T15:08:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.340+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5428025820 con 0x7f54280253a0 2024-03-30T15:08:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f5438ff9700 1 -- 172.21.15.28:0/3044382936 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5428025820 con 0x7f54280253a0 2024-03-30T15:08:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f5438ff9700 10 client.58577.objecter ms_dispatch 0x7f543c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f5438ff9700 10 client.58577.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f5438ff9700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 4 2024-03-30T15:08:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f5438ff9700 15 client.58577.objecter _session_command_op_remove 7 16 2024-03-30T15:08:33.344 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:08:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 librados: watch_flush enter 2024-03-30T15:08:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 librados: watch_flush exit 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.0 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6801/3394323575 conn(0x7f5428004730 legacy=0x7f5428002790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f54280053e0 osd=0 1 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.1 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6805/1850313547 conn(0x7f54280042f0 legacy=0x7f542800ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428004180 osd=1 1 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.2 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6809/3451325043 conn(0x7f542800d6e0 legacy=0x7f542800fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008480 osd=2 1 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.3 2024-03-30T15:08:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6813/1506157250 conn(0x7f5428012520 legacy=0x7f5428014910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428008250 osd=3 1 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.4 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.134:6800/3063775514 conn(0x7f5428012080 legacy=0x7f5428019370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542800c7b0 osd=4 1 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.5 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.134:6804/3668151179 conn(0x7f542801b1b0 legacy=0x7f542801b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801aa90 osd=5 1 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 10 client.58577.objecter close_session for osd.6 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.134:6808/2135224194 conn(0x7f542801f940 legacy=0x7f5428021d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f543bfff700 1 -- 172.21.15.28:0/3044382936 reap_dead start 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f543bfff700 1 -- 172.21.15.28:0/3044382936 reap_dead start 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.342+0000 7f543bfff700 1 -- 172.21.15.28:0/3044382936 reap_dead start 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f542801cdf0 osd=6 1 2024-03-30T15:08:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 10 client.58577.objecter close_session for osd.7 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.134:6812/3758190298 conn(0x7f54280253a0 legacy=0x7f54280234b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 20 client.58577.objecter put_session s=0x7f5428024c80 osd=7 1 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 20 client.58577.objecter shutdown clearing up homeless session... 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 10 client.58577.objecter successfully canceled tick 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5424070450 legacy=0x7f5424072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> v1:172.21.15.28:6790/0 conn(0x7f543c101130 legacy=0x7f543c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 shutdown_connections 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 >> 172.21.15.28:0/3044382936 conn(0x7f543c0fc9e0 msgr2=0x7f543c069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 shutdown_connections 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 -- 172.21.15.28:0/3044382936 wait complete. 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:33.343+0000 7f54420d1700 1 librados: shutdown 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:08:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:08:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:08:33.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:33.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3044382936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[41614]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T15:08:33.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3044382936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:33.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:33.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[29876]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T15:08:33.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3044382936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:33 smithi134 ceph-mon[33714]: osdmap e664: 8 total, 8 up, 8 in 2024-03-30T15:08:33.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]: dispatch 2024-03-30T15:08:33.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:33.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:33.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:33.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:33.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:33.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:34.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:34.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:34.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:34.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:34.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:34.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:34.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:08:34.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:34.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:34.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:34.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:34 smithi134 ceph-mon[33714]: pgmap v979: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:08:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:34 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]': finished 2024-03-30T15:08:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:34 smithi134 ceph-mon[33714]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T15:08:34.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[29876]: pgmap v979: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:08:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]': finished 2024-03-30T15:08:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[29876]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T15:08:34.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:34.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[41614]: pgmap v979: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-30T15:08:34.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:34.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-116", "mode": "writeback"}]': finished 2024-03-30T15:08:34.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[41614]: osdmap e665: 8 total, 8 up, 8 in 2024-03-30T15:08:34.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:34.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:34.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:34.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:34.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:34.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:35.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:35.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:35.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:35.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:35.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:35.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:35.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:35.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:35.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:35.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:35.662 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:35 smithi134 ceph-mon[33714]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:35 smithi134 ceph-mon[33714]: 325.6 scrub starts 2024-03-30T15:08:35.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:35 smithi134 ceph-mon[33714]: 325.6 scrub ok 2024-03-30T15:08:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:35.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[29876]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[29876]: 325.6 scrub starts 2024-03-30T15:08:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[29876]: 325.6 scrub ok 2024-03-30T15:08:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[41614]: from='mon.? v1:172.21.15.28:0/2745634723' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.6"}]: dispatch 2024-03-30T15:08:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[41614]: 325.6 scrub starts 2024-03-30T15:08:35.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:35 smithi028 ceph-mon[41614]: 325.6 scrub ok 2024-03-30T15:08:35.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:35.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:35.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:35.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:36.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:08:36.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:36.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:36.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.243+0000 7f7849c4e700 1 -- 172.21.15.28:0/1808340254 >> v1:172.21.15.28:6790/0 conn(0x7f784410d950 legacy=0x7f784410fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.243+0000 7f7849c4e700 1 -- 172.21.15.28:0/1808340254 shutdown_connections 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.243+0000 7f7849c4e700 1 -- 172.21.15.28:0/1808340254 >> 172.21.15.28:0/1808340254 conn(0x7f7844077370 msgr2=0x7f78440fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.243+0000 7f7849c4e700 1 -- 172.21.15.28:0/1808340254 shutdown_connections 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.243+0000 7f7849c4e700 1 -- 172.21.15.28:0/1808340254 wait complete. 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 librados: starting msgr at 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 librados: starting objecter 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 Processor -- start 2024-03-30T15:08:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 -- start start 2024-03-30T15:08:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 librados: setting wanted keys 2024-03-30T15:08:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 librados: calling monclient init 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78441a5fa0 con 0x7f784410d950 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78441a71c0 con 0x7f7844109d80 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7849c4e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78441a8400 con 0x7f7844105f00 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f78437fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7844105f00 0x7f784410d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47734/0 (socket says 172.21.15.28:47734) 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f78437fe700 1 -- 172.21.15.28:0/3138483631 learned_addr learned my addr 172.21.15.28:0/3138483631 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7840ff9700 10 client.?.objecter ms_handle_connect 0x7f7844105f00 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.244+0000 7f7840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 10 client.?.objecter ms_handle_connect 0x7f784410d950 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 10 client.?.objecter ms_handle_connect 0x7f7844109d80 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254170120 0 0) 0x7f78441a8400 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f782c0031f0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203606892 0 0) 0x7f78441a5fa0 con 0x7f784410d950 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78441a8400 con 0x7f784410d950 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765924543 0 0) 0x7f78441a71c0 con 0x7f7844109d80 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78441a5fa0 con 0x7f7844109d80 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3968166974 0 0) 0x7f782c0031f0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78441a71c0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.245+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78340018e0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 349433331 0 0) 0x7f78441a71c0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 >> v1:172.21.15.134:6789/0 conn(0x7f7844109d80 legacy=0x7f78441a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 >> v1:172.21.15.28:6789/0 conn(0x7f784410d950 legacy=0x7f78441a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78441a9640 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78441a61d0 con 0x7f7844105f00 2024-03-30T15:08:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7849c4e700 10 client.58583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f78441a6710 con 0x7f7844105f00 2024-03-30T15:08:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7849c4e700 1 librados: init done 2024-03-30T15:08:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7834003b20 con 0x7f7844105f00 2024-03-30T15:08:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.246+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7834005cd0 con 0x7f7844105f00 2024-03-30T15:08:36.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.247+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7830005120 con 0x7f7844105f00 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.247+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7834003680 con 0x7f7844105f00 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.248+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(665..665 src has 442..665) ==== 9110+0+0 (unknown 1208088358 0 0) 0x7f7834089c10 con 0x7f7844105f00 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.248+0000 7f7840ff9700 10 client.58583.objecter ms_dispatch 0x7f7844001550 osd_map(665..665 src has 442..665) 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.248+0000 7f7840ff9700 3 client.58583.objecter handle_osd_map got epochs [665,665] > 0 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.248+0000 7f7840ff9700 3 client.58583.objecter handle_osd_map decoding full epoch 665 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.248+0000 7f7840ff9700 20 client.58583.objecter dump_active .. 0 homeless 2024-03-30T15:08:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.249+0000 7f7840ff9700 10 client.58583.objecter ms_handle_connect 0x7f782c0703c0 2024-03-30T15:08:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.254+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7834056a70 con 0x7f7844105f00 2024-03-30T15:08:36.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:36.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:36.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:36.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.461+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7830000b90 con 0x7f782c0703c0 2024-03-30T15:08:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.479+0000 7f7840ff9700 1 -- 172.21.15.28:0/3138483631 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+635212 (unknown 2965378022 0 2730234481) 0x7f7830000b90 con 0x7f782c0703c0 2024-03-30T15:08:36.488 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 10 librados: watch_flush enter 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 10 librados: watch_flush exit 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 20 client.58583.objecter shutdown clearing up homeless session... 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 10 client.58583.objecter successfully canceled tick 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 >> v1:172.21.15.28:6800/3011431093 conn(0x7f782c0703c0 legacy=0x7f782c072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 >> v1:172.21.15.28:6790/0 conn(0x7f7844105f00 legacy=0x7f784410d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 shutdown_connections 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 >> 172.21.15.28:0/3138483631 conn(0x7f7844077370 msgr2=0x7f7844110b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 shutdown_connections 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 -- 172.21.15.28:0/3138483631 wait complete. 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:36.483+0000 7f7849c4e700 1 librados: shutdown 2024-03-30T15:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:08:36.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:36.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:36 smithi134 ceph-mon[33714]: pgmap v981: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:08:36.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:36.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:36.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:36 smithi028 ceph-mon[29876]: pgmap v981: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:08:36.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:36 smithi028 ceph-mon[41614]: pgmap v981: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-30T15:08:36.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:36.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:36.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:36.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:37.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:08:37.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:37.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:37.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:37.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:37.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:37.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":981,"stamp":"2024-03-30T15:08:36.072729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596382,"num_objects":209,"num_object_clones":4,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":1,"num_read":1984,"num_read_kb":1735,"num_write":739,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2265,"ondisk_log_size":2265,"up":738,"acting":738,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":729,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":639796,"kb_used_data":40516,"kb_used_omap":12,"kb_used_meta":599155,"kb_avail":749321420,"statfs":{"total":767960285184,"available":767305134080,"internally_reserved":0,"allocated":41488384,"data_stored":19503851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":613535280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.675399"},"pg_stats":[{"pgid":"325.1f","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529350+0000","last_change":"2024-03-30T15:08:30.420585+0000","last_active":"2024-03-30T15:08:32.529350+0000","last_peered":"2024-03-30T15:08:32.529350+0000","last_clean":"2024-03-30T15:08:32.529350+0000","last_became_active":"2024-03-30T15:08:30.420034+0000","last_became_peered":"2024-03-30T15:08:30.420034+0000","last_unstale":"2024-03-30T15:08:32.529350+0000","last_undegraded":"2024-03-30T15:08:32.529350+0000","last_fullsized":"2024-03-30T15:08:32.529350+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:37.808862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1e","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569744+0000","last_change":"2024-03-30T15:08:30.415672+0000","last_active":"2024-03-30T15:08:33.569744+0000","last_peered":"2024-03-30T15:08:33.569744+0000","last_clean":"2024-03-30T15:08:33.569744+0000","last_became_active":"2024-03-30T15:08:30.415362+0000","last_became_peered":"2024-03-30T15:08:30.415362+0000","last_unstale":"2024-03-30T15:08:33.569744+0000","last_undegraded":"2024-03-30T15:08:33.569744+0000","last_fullsized":"2024-03-30T15:08:33.569744+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:56.314619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.1d","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368281+0000","last_change":"2024-03-30T15:08:30.416291+0000","last_active":"2024-03-30T15:08:33.368281+0000","last_peered":"2024-03-30T15:08:33.368281+0000","last_clean":"2024-03-30T15:08:33.368281+0000","last_became_active":"2024-03-30T15:08:30.415933+0000","last_became_peered":"2024-03-30T15:08:30.415933+0000","last_unstale":"2024-03-30T15:08:33.368281+0000","last_undegraded":"2024-03-30T15:08:33.368281+0000","last_fullsized":"2024-03-30T15:08:33.368281+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.671831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.1c","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529175+0000","last_change":"2024-03-30T15:08:30.412089+0000","last_active":"2024-03-30T15:08:32.529175+0000","last_peered":"2024-03-30T15:08:32.529175+0000","last_clean":"2024-03-30T15:08:32.529175+0000","last_became_active":"2024-03-30T15:08:30.411756+0000","last_became_peered":"2024-03-30T15:08:30.411756+0000","last_unstale":"2024-03-30T15:08:32.529175+0000","last_undegraded":"2024-03-30T15:08:32.529175+0000","last_fullsized":"2024-03-30T15:08:32.529175+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.778863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1b","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568842+0000","last_change":"2024-03-30T15:08:30.416787+0000","last_active":"2024-03-30T15:08:33.568842+0000","last_peered":"2024-03-30T15:08:33.568842+0000","last_clean":"2024-03-30T15:08:33.568842+0000","last_became_active":"2024-03-30T15:08:30.416046+0000","last_became_peered":"2024-03-30T15:08:30.416046+0000","last_unstale":"2024-03-30T15:08:33.568842+0000","last_undegraded":"2024-03-30T15:08:33.568842+0000","last_fullsized":"2024-03-30T15:08:33.568842+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.694672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.1a","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810992+0000","last_change":"2024-03-30T15:08:30.405554+0000","last_active":"2024-03-30T15:08:33.810992+0000","last_peered":"2024-03-30T15:08:33.810992+0000","last_clean":"2024-03-30T15:08:33.810992+0000","last_became_active":"2024-03-30T15:08:30.404955+0000","last_became_peered":"2024-03-30T15:08:30.404955+0000","last_unstale":"2024-03-30T15:08:33.810992+0000","last_undegraded":"2024-03-30T15:08:33.810992+0000","last_fullsized":"2024-03-30T15:08:33.810992+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.397764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"325.19","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810445+0000","last_change":"2024-03-30T15:08:30.403318+0000","last_active":"2024-03-30T15:08:33.810445+0000","last_peered":"2024-03-30T15:08:33.810445+0000","last_clean":"2024-03-30T15:08:33.810445+0000","last_became_active":"2024-03-30T15:08:30.402785+0000","last_became_peered":"2024-03-30T15:08:30.402785+0000","last_unstale":"2024-03-30T15:08:33.810445+0000","last_undegraded":"2024-03-30T15:08:33.810445+0000","last_fullsized":"2024-03-30T15:08:33.810445+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:48.040265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"325.18","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364159+0000","last_change":"2024-03-30T15:08:30.399469+0000","last_active":"2024-03-30T15:08:33.364159+0000","last_peered":"2024-03-30T15:08:33.364159+0000","last_clean":"2024-03-30T15:08:33.364159+0000","last_became_active":"2024-03-30T15:08:30.399167+0000","last_became_peered":"2024-03-30T15:08:30.399167+0000","last_unstale":"2024-03-30T15:08:33.364159+0000","last_undegraded":"2024-03-30T15:08:33.364159+0000","last_fullsized":"2024-03-30T15:08:33.364159+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:51.289833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"325.17","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360910+0000","last_change":"2024-03-30T15:08:30.416722+0000","last_active":"2024-03-30T15:08:32.360910+0000","last_peered":"2024-03-30T15:08:32.360910+0000","last_clean":"2024-03-30T15:08:32.360910+0000","last_became_active":"2024-03-30T15:08:30.416408+0000","last_became_peered":"2024-03-30T15:08:30.416408+0000","last_unstale":"2024-03-30T15:08:32.360910+0000","last_undegraded":"2024-03-30T15:08:32.360910+0000","last_fullsized":"2024-03-30T15:08:32.360910+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.737755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.16","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528983+0000","last_change":"2024-03-30T15:08:30.419992+0000","last_active":"2024-03-30T15:08:32.528983+0000","last_peered":"2024-03-30T15:08:32.528983+0000","last_clean":"2024-03-30T15:08:32.528983+0000","last_became_active":"2024-03-30T15:08:30.419649+0000","last_became_peered":"2024-03-30T15:08:30.419649+0000","last_unstale":"2024-03-30T15:08:32.528983+0000","last_undegraded":"2024-03-30T15:08:32.528983+0000","last_fullsized":"2024-03-30T15:08:32.528983+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:56.192929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.15","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359528+0000","last_change":"2024-03-30T15:08:30.413730+0000","last_active":"2024-03-30T15:08:32.359528+0000","last_peered":"2024-03-30T15:08:32.359528+0000","last_clean":"2024-03-30T15:08:32.359528+0000","last_became_active":"2024-03-30T15:08:30.412082+0000","last_became_peered":"2024-03-30T15:08:30.412082+0000","last_unstale":"2024-03-30T15:08:32.359528+0000","last_undegraded":"2024-03-30T15:08:32.359528+0000","last_fullsized":"2024-03-30T15:08:32.359528+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.356218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.14","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358763+0000","last_change":"2024-03-30T15:08:30.412650+0000","last_active":"2024-03-30T15:08:32.358763+0000","last_peered":"2024-03-30T15:08:32.358763+0000","last_clean":"2024-03-30T15:08:32.358763+0000","last_became_active":"2024-03-30T15:08:30.409803+0000","last_became_peered":"2024-03-30T15:08:30.409803+0000","last_unstale":"2024-03-30T15:08:32.358763+0000","last_undegraded":"2024-03-30T15:08:32.358763+0000","last_fullsized":"2024-03-30T15:08:32.358763+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:43.340493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.13","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358653+0000","last_change":"2024-03-30T15:08:30.414624+0000","last_active":"2024-03-30T15:08:32.358653+0000","last_peered":"2024-03-30T15:08:32.358653+0000","last_clean":"2024-03-30T15:08:32.358653+0000","last_became_active":"2024-03-30T15:08:30.414233+0000","last_became_peered":"2024-03-30T15:08:30.414233+0000","last_unstale":"2024-03-30T15:08:32.358653+0000","last_undegraded":"2024-03-30T15:08:32.358653+0000","last_fullsized":"2024-03-30T15:08:32.358653+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:50.653845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.12","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527488+0000","last_change":"2024-03-30T15:08:30.411919+0000","last_active":"2024-03-30T15:08:32.527488+0000","last_peered":"2024-03-30T15:08:32.527488+0000","last_clean":"2024-03-30T15:08:32.527488+0000","last_became_active":"2024-03-30T15:08:30.411212+0000","last_became_peered":"2024-03-30T15:08:30.411212+0000","last_unstale":"2024-03-30T15:08:32.527488+0000","last_undegraded":"2024-03-30T15:08:32.527488+0000","last_fullsized":"2024-03-30T15:08:32.527488+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:31.613910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.11","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357927+0000","last_change":"2024-03-30T15:08:30.412176+0000","last_active":"2024-03-30T15:08:32.357927+0000","last_peered":"2024-03-30T15:08:32.357927+0000","last_clean":"2024-03-30T15:08:32.357927+0000","last_became_active":"2024-03-30T15:08:30.411812+0000","last_became_peered":"2024-03-30T15:08:30.411812+0000","last_unstale":"2024-03-30T15:08:32.357927+0000","last_undegraded":"2024-03-30T15:08:32.357927+0000","last_fullsized":"2024-03-30T15:08:32.357927+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.229735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.10","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356055+0000","last_change":"2024-03-30T15:08:30.408932+0000","last_active":"2024-03-30T15:08:32.356055+0000","last_peered":"2024-03-30T15:08:32.356055+0000","last_clean":"2024-03-30T15:08:32.356055+0000","last_became_active":"2024-03-30T15:08:30.407936+0000","last_became_peered":"2024-03-30T15:08:30.407936+0000","last_unstale":"2024-03-30T15:08:32.356055+0000","last_undegraded":"2024-03-30T15:08:32.356055+0000","last_fullsized":"2024-03-30T15:08:32.356055+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:53.301087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.f","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529533+0000","last_change":"2024-03-30T15:08:30.417310+0000","last_active":"2024-03-30T15:08:32.529533+0000","last_peered":"2024-03-30T15:08:32.529533+0000","last_clean":"2024-03-30T15:08:32.529533+0000","last_became_active":"2024-03-30T15:08:30.416848+0000","last_became_peered":"2024-03-30T15:08:30.416848+0000","last_unstale":"2024-03-30T15:08:32.529533+0000","last_undegraded":"2024-03-30T15:08:32.529533+0000","last_fullsized":"2024-03-30T15:08:32.529533+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:48.211685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.e","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569789+0000","last_change":"2024-03-30T15:08:30.410219+0000","last_active":"2024-03-30T15:08:33.569789+0000","last_peered":"2024-03-30T15:08:33.569789+0000","last_clean":"2024-03-30T15:08:33.569789+0000","last_became_active":"2024-03-30T15:08:30.408510+0000","last_became_peered":"2024-03-30T15:08:30.408510+0000","last_unstale":"2024-03-30T15:08:33.569789+0000","last_undegraded":"2024-03-30T15:08:33.569789+0000","last_fullsized":"2024-03-30T15:08:33.569789+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:56.662774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.d","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526887+0000","last_change":"2024-03-30T15:08:30.409944+0000","last_active":"2024-03-30T15:08:32.526887+0000","last_peered":"2024-03-30T15:08:32.526887+0000","last_clean":"2024-03-30T15:08:32.526887+0000","last_became_active":"2024-03-30T15:08:30.409590+0000","last_became_peered":"2024-03-30T15:08:30.409590+0000","last_unstale":"2024-03-30T15:08:32.526887+0000","last_undegraded":"2024-03-30T15:08:32.526887+0000","last_fullsized":"2024-03-30T15:08:32.526887+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.724082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"325.c","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358972+0000","last_change":"2024-03-30T15:08:30.414353+0000","last_active":"2024-03-30T15:08:32.358972+0000","last_peered":"2024-03-30T15:08:32.358972+0000","last_clean":"2024-03-30T15:08:32.358972+0000","last_became_active":"2024-03-30T15:08:30.413835+0000","last_became_peered":"2024-03-30T15:08:30.413835+0000","last_unstale":"2024-03-30T15:08:32.358972+0000","last_undegraded":"2024-03-30T15:08:32.358972+0000","last_fullsized":"2024-03-30T15:08:32.358972+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.068364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.b","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525440+0000","last_change":"2024-03-30T15:08:30.411594+0000","last_active":"2024-03-30T15:08:32.525440+0000","last_peered":"2024-03-30T15:08:32.525440+0000","last_clean":"2024-03-30T15:08:32.525440+0000","last_became_active":"2024-03-30T15:08:30.411294+0000","last_became_peered":"2024-03-30T15:08:30.411294+0000","last_unstale":"2024-03-30T15:08:32.525440+0000","last_undegraded":"2024-03-30T15:08:32.525440+0000","last_fullsized":"2024-03-30T15:08:32.525440+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:48.186259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"325.a","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358195+0000","last_change":"2024-03-30T15:08:30.407079+0000","last_active":"2024-03-30T15:08:32.358195+0000","last_peered":"2024-03-30T15:08:32.358195+0000","last_clean":"2024-03-30T15:08:32.358195+0000","last_became_active":"2024-03-30T15:08:30.406728+0000","last_became_peered":"2024-03-30T15:08:30.406728+0000","last_unstale":"2024-03-30T15:08:32.358195+0000","last_undegraded":"2024-03-30T15:08:32.358195+0000","last_fullsized":"2024-03-30T15:08:32.358195+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:43.665711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.9","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356285+0000","last_change":"2024-03-30T15:08:30.408447+0000","last_active":"2024-03-30T15:08:32.356285+0000","last_peered":"2024-03-30T15:08:32.356285+0000","last_clean":"2024-03-30T15:08:32.356285+0000","last_became_active":"2024-03-30T15:08:30.407675+0000","last_became_peered":"2024-03-30T15:08:30.407675+0000","last_unstale":"2024-03-30T15:08:32.356285+0000","last_undegraded":"2024-03-30T15:08:32.356285+0000","last_fullsized":"2024-03-30T15:08:32.356285+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.553199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.8","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568380+0000","last_change":"2024-03-30T15:08:30.407530+0000","last_active":"2024-03-30T15:08:33.568380+0000","last_peered":"2024-03-30T15:08:33.568380+0000","last_clean":"2024-03-30T15:08:33.568380+0000","last_became_active":"2024-03-30T15:08:30.407185+0000","last_became_peered":"2024-03-30T15:08:30.407185+0000","last_unstale":"2024-03-30T15:08:33.568380+0000","last_undegraded":"2024-03-30T15:08:33.568380+0000","last_fullsized":"2024-03-30T15:08:33.568380+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:47.592350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.0","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357789+0000","last_change":"2024-03-30T15:08:30.410339+0000","last_active":"2024-03-30T15:08:32.357789+0000","last_peered":"2024-03-30T15:08:32.357789+0000","last_clean":"2024-03-30T15:08:32.357789+0000","last_became_active":"2024-03-30T15:08:30.409418+0000","last_became_peered":"2024-03-30T15:08:30.409418+0000","last_unstale":"2024-03-30T15:08:32.357789+0000","last_undegraded":"2024-03-30T15:08:32.357789+0000","last_fullsized":"2024-03-30T15:08:32.357789+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:49.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.7","version":"0'0","reported_seq":57,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529817+0000","last_change":"2024-03-30T15:08:14.381160+0000","last_active":"2024-03-30T15:08:32.529817+0000","last_peered":"2024-03-30T15:08:32.529817+0000","last_clean":"2024-03-30T15:08:32.529817+0000","last_became_active":"2024-03-30T15:08:14.380692+0000","last_became_peered":"2024-03-30T15:08:14.380692+0000","last_unstale":"2024-03-30T15:08:32.529817+0000","last_undegraded":"2024-03-30T15:08:32.529817+0000","last_fullsized":"2024-03-30T15:08:32.529817+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:28.848649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1","version":"0'0","reported_seq":19,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810653+0000","last_change":"2024-03-30T15:08:30.407137+0000","last_active":"2024-03-30T15:08:33.810653+0000","last_peered":"2024-03-30T15:08:33.810653+0000","last_clean":"2024-03-30T15:08:33.810653+0000","last_became_active":"2024-03-30T15:08:30.406525+0000","last_became_peered":"2024-03-30T15:08:30.406525+0000","last_unstale":"2024-03-30T15:08:33.810653+0000","last_undegraded":"2024-03-30T15:08:33.810653+0000","last_fullsized":"2024-03-30T15:08:33.810653+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:44.784193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"322.6","version":"662'3","reported_seq":59,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360136+0000","last_change":"2024-03-30T15:08:14.374104+0000","last_active":"2024-03-30T15:08:32.360136+0000","last_peered":"2024-03-30T15:08:32.360136+0000","last_clean":"2024-03-30T15:08:32.360136+0000","last_became_active":"2024-03-30T15:08:14.373744+0000","last_became_peered":"2024-03-30T15:08:14.373744+0000","last_unstale":"2024-03-30T15:08:32.360136+0000","last_undegraded":"2024-03-30T15:08:32.360136+0000","last_fullsized":"2024-03-30T15:08:32.360136+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.104735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.2","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525450+0000","last_change":"2024-03-30T15:08:30.404313+0000","last_active":"2024-03-30T15:08:32.525450+0000","last_peered":"2024-03-30T15:08:32.525450+0000","last_clean":"2024-03-30T15:08:32.525450+0000","last_became_active":"2024-03-30T15:08:30.404017+0000","last_became_peered":"2024-03-30T15:08:30.404017+0000","last_unstale":"2024-03-30T15:08:32.525450+0000","last_undegraded":"2024-03-30T15:08:32.525450+0000","last_fullsized":"2024-03-30T15:08:32.525450+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:29.968773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.5","version":"0'0","reported_seq":60,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368432+0000","last_change":"2024-03-30T15:08:14.368479+0000","last_active":"2024-03-30T15:08:33.368432+0000","last_peered":"2024-03-30T15:08:33.368432+0000","last_clean":"2024-03-30T15:08:33.368432+0000","last_became_active":"2024-03-30T15:08:14.368163+0000","last_became_peered":"2024-03-30T15:08:14.368163+0000","last_unstale":"2024-03-30T15:08:33.368432+0000","last_undegraded":"2024-03-30T15:08:33.368432+0000","last_fullsized":"2024-03-30T15:08:33.368432+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.681841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.3","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525660+0000","last_change":"2024-03-30T15:08:30.414252+0000","last_active":"2024-03-30T15:08:32.525660+0000","last_peered":"2024-03-30T15:08:32.525660+0000","last_clean":"2024-03-30T15:08:32.525660+0000","last_became_active":"2024-03-30T15:08:30.413598+0000","last_became_peered":"2024-03-30T15:08:30.413598+0000","last_unstale":"2024-03-30T15:08:32.525660+0000","last_undegraded":"2024-03-30T15:08:32.525660+0000","last_fullsized":"2024-03-30T15:08:32.525660+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:44.199283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.4","version":"662'3","reported_seq":59,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358987+0000","last_change":"2024-03-30T15:08:14.374633+0000","last_active":"2024-03-30T15:08:32.358987+0000","last_peered":"2024-03-30T15:08:32.358987+0000","last_clean":"2024-03-30T15:08:32.358987+0000","last_became_active":"2024-03-30T15:08:14.374287+0000","last_became_peered":"2024-03-30T15:08:14.374287+0000","last_unstale":"2024-03-30T15:08:32.358987+0000","last_undegraded":"2024-03-30T15:08:32.358987+0000","last_fullsized":"2024-03-30T15:08:32.358987+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:37.837037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.4","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359279+0000","last_change":"2024-03-30T15:08:30.409940+0000","last_active":"2024-03-30T15:08:32.359279+0000","last_peered":"2024-03-30T15:08:32.359279+0000","last_clean":"2024-03-30T15:08:32.359279+0000","last_became_active":"2024-03-30T15:08:30.409504+0000","last_became_peered":"2024-03-30T15:08:30.409504+0000","last_unstale":"2024-03-30T15:08:32.359279+0000","last_undegraded":"2024-03-30T15:08:32.359279+0000","last_fullsized":"2024-03-30T15:08:32.359279+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:46.510455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"322.3","version":"0'0","reported_seq":60,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.365512+0000","last_change":"2024-03-30T15:08:14.369066+0000","last_active":"2024-03-30T15:08:33.365512+0000","last_peered":"2024-03-30T15:08:33.365512+0000","last_clean":"2024-03-30T15:08:33.365512+0000","last_became_active":"2024-03-30T15:08:14.367491+0000","last_became_peered":"2024-03-30T15:08:14.367491+0000","last_unstale":"2024-03-30T15:08:33.365512+0000","last_undegraded":"2024-03-30T15:08:33.365512+0000","last_fullsized":"2024-03-30T15:08:33.365512+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:42.954042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.5","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358876+0000","last_change":"2024-03-30T15:08:30.418292+0000","last_active":"2024-03-30T15:08:32.358876+0000","last_peered":"2024-03-30T15:08:32.358876+0000","last_clean":"2024-03-30T15:08:32.358876+0000","last_became_active":"2024-03-30T15:08:30.417980+0000","last_became_peered":"2024-03-30T15:08:30.417980+0000","last_unstale":"2024-03-30T15:08:32.358876+0000","last_undegraded":"2024-03-30T15:08:32.358876+0000","last_fullsized":"2024-03-30T15:08:32.358876+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:49.701269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.2","version":"0'0","reported_seq":57,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358317+0000","last_change":"2024-03-30T15:08:14.378059+0000","last_active":"2024-03-30T15:08:32.358317+0000","last_peered":"2024-03-30T15:08:32.358317+0000","last_clean":"2024-03-30T15:08:32.358317+0000","last_became_active":"2024-03-30T15:08:14.377722+0000","last_became_peered":"2024-03-30T15:08:14.377722+0000","last_unstale":"2024-03-30T15:08:32.358317+0000","last_undegraded":"2024-03-30T15:08:32.358317+0000","last_fullsized":"2024-03-30T15:08:32.358317+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:39.532846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.6","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526703+0000","last_change":"2024-03-30T15:08:30.408788+0000","last_active":"2024-03-30T15:08:32.526703+0000","last_peered":"2024-03-30T15:08:32.526703+0000","last_clean":"2024-03-30T15:08:32.526703+0000","last_became_active":"2024-03-30T15:08:30.408474+0000","last_became_peered":"2024-03-30T15:08:30.408474+0000","last_unstale":"2024-03-30T15:08:32.526703+0000","last_undegraded":"2024-03-30T15:08:32.526703+0000","last_fullsized":"2024-03-30T15:08:32.526703+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.048705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.1","version":"646'1","reported_seq":59,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357841+0000","last_change":"2024-03-30T15:08:14.375816+0000","last_active":"2024-03-30T15:08:32.357841+0000","last_peered":"2024-03-30T15:08:32.357841+0000","last_clean":"2024-03-30T15:08:32.357841+0000","last_became_active":"2024-03-30T15:08:14.375417+0000","last_became_peered":"2024-03-30T15:08:14.375417+0000","last_unstale":"2024-03-30T15:08:32.357841+0000","last_undegraded":"2024-03-30T15:08:32.357841+0000","last_fullsized":"2024-03-30T15:08:32.357841+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.108834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.7","version":"0'0","reported_seq":16,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.361248+0000","last_change":"2024-03-30T15:08:30.411515+0000","last_active":"2024-03-30T15:08:32.361248+0000","last_peered":"2024-03-30T15:08:32.361248+0000","last_clean":"2024-03-30T15:08:32.361248+0000","last_became_active":"2024-03-30T15:08:30.410333+0000","last_became_peered":"2024-03-30T15:08:30.410333+0000","last_unstale":"2024-03-30T15:08:32.361248+0000","last_undegraded":"2024-03-30T15:08:32.361248+0000","last_fullsized":"2024-03-30T15:08:32.361248+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.707059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"322.0","version":"662'7","reported_seq":64,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356987+0000","last_change":"2024-03-30T15:08:14.374886+0000","last_active":"2024-03-30T15:08:32.356987+0000","last_peered":"2024-03-30T15:08:32.356987+0000","last_clean":"2024-03-30T15:08:32.356987+0000","last_became_active":"2024-03-30T15:08:14.374433+0000","last_became_peered":"2024-03-30T15:08:14.374433+0000","last_unstale":"2024-03-30T15:08:32.356987+0000","last_undegraded":"2024-03-30T15:08:32.356987+0000","last_fullsized":"2024-03-30T15:08:32.356987+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:29.444628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569545+0000","last_change":"2024-03-30T15:08:00.661723+0000","last_active":"2024-03-30T15:08:33.569545+0000","last_peered":"2024-03-30T15:08:33.569545+0000","last_clean":"2024-03-30T15:08:33.569545+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:08:33.569545+0000","last_undegraded":"2024-03-30T15:08:33.569545+0000","last_fullsized":"2024-03-30T15:08:33.569545+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.661646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.661646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:23.913327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":1135,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526982+0000","last_change":"2024-03-30T15:07:25.409282+0000","last_active":"2024-03-30T15:08:32.526982+0000","last_peered":"2024-03-30T15:08:32.526982+0000","last_clean":"2024-03-30T15:08:32.526982+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:08:32.526982+0000","last_undegraded":"2024-03-30T15:08:32.526982+0000","last_fullsized":"2024-03-30T15:08:32.526982+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.409160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.409160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.520458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529891+0000","last_change":"2024-03-30T15:08:00.188376+0000","last_active":"2024-03-30T15:08:32.529891+0000","last_peered":"2024-03-30T15:08:32.529891+0000","last_clean":"2024-03-30T15:08:32.529891+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:08:32.529891+0000","last_undegraded":"2024-03-30T15:08:32.529891+0000","last_fullsized":"2024-03-30T15:08:32.529891+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.188301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.188301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.659221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":1129,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360385+0000","last_change":"2024-03-30T15:06:56.895628+0000","last_active":"2024-03-30T15:08:32.360385+0000","last_peered":"2024-03-30T15:08:32.360385+0000","last_clean":"2024-03-30T15:08:32.360385+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:08:32.360385+0000","last_undegraded":"2024-03-30T15:08:32.360385+0000","last_fullsized":"2024-03-30T15:08:32.360385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.895505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.895505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":1123,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364713+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:08:33.364713+0000","last_peered":"2024-03-30T15:08:33.364713+0000","last_clean":"2024-03-30T15:08:33.364713+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:08:33.364713+0000","last_undegraded":"2024-03-30T15:08:33.364713+0000","last_fullsized":"2024-03-30T15:08:33.364713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860136+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":1146,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810261+0000","last_change":"2024-03-30T15:07:57.201822+0000","last_active":"2024-03-30T15:08:33.810261+0000","last_peered":"2024-03-30T15:08:33.810261+0000","last_clean":"2024-03-30T15:08:33.810261+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:08:33.810261+0000","last_undegraded":"2024-03-30T15:08:33.810261+0000","last_fullsized":"2024-03-30T15:08:33.810261+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.201691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.201691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:05.630818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528956+0000","last_change":"2024-03-30T15:08:13.183528+0000","last_active":"2024-03-30T15:08:32.528956+0000","last_peered":"2024-03-30T15:08:32.528956+0000","last_clean":"2024-03-30T15:08:32.528956+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:08:32.528956+0000","last_undegraded":"2024-03-30T15:08:32.528956+0000","last_fullsized":"2024-03-30T15:08:32.528956+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.183373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.183373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.888133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569088+0000","last_change":"2024-03-30T15:06:59.257326+0000","last_active":"2024-03-30T15:08:33.569088+0000","last_peered":"2024-03-30T15:08:33.569088+0000","last_clean":"2024-03-30T15:08:33.569088+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:08:33.569088+0000","last_undegraded":"2024-03-30T15:08:33.569088+0000","last_fullsized":"2024-03-30T15:08:33.569088+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.257230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.257230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.596445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":1126,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359369+0000","last_change":"2024-03-30T15:07:36.042362+0000","last_active":"2024-03-30T15:08:32.359369+0000","last_peered":"2024-03-30T15:08:32.359369+0000","last_clean":"2024-03-30T15:08:32.359369+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:08:32.359369+0000","last_undegraded":"2024-03-30T15:08:32.359369+0000","last_fullsized":"2024-03-30T15:08:32.359369+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.042281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.352269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":1129,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359093+0000","last_change":"2024-03-30T15:06:18.054438+0000","last_active":"2024-03-30T15:08:32.359093+0000","last_peered":"2024-03-30T15:08:32.359093+0000","last_clean":"2024-03-30T15:08:32.359093+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:08:32.359093+0000","last_undegraded":"2024-03-30T15:08:32.359093+0000","last_fullsized":"2024-03-30T15:08:32.359093+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:18.054358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:06:18.054358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":1128,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529795+0000","last_change":"2024-03-30T15:08:10.138813+0000","last_active":"2024-03-30T15:08:32.529795+0000","last_peered":"2024-03-30T15:08:32.529795+0000","last_clean":"2024-03-30T15:08:32.529795+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:08:32.529795+0000","last_undegraded":"2024-03-30T15:08:32.529795+0000","last_fullsized":"2024-03-30T15:08:32.529795+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.138657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.138657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.317256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":1142,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525057+0000","last_change":"2024-03-30T15:07:32.132841+0000","last_active":"2024-03-30T15:08:32.525057+0000","last_peered":"2024-03-30T15:08:32.525057+0000","last_clean":"2024-03-30T15:08:32.525057+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:08:32.525057+0000","last_undegraded":"2024-03-30T15:08:32.525057+0000","last_fullsized":"2024-03-30T15:08:32.525057+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.132758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.132758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526163+0000","last_change":"2024-03-30T15:08:15.202644+0000","last_active":"2024-03-30T15:08:32.526163+0000","last_peered":"2024-03-30T15:08:32.526163+0000","last_clean":"2024-03-30T15:08:32.526163+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:08:32.526163+0000","last_undegraded":"2024-03-30T15:08:32.526163+0000","last_fullsized":"2024-03-30T15:08:32.526163+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.202555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:37.887939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":1129,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356121+0000","last_change":"2024-03-30T15:06:43.837698+0000","last_active":"2024-03-30T15:08:32.356121+0000","last_peered":"2024-03-30T15:08:32.356121+0000","last_clean":"2024-03-30T15:08:32.356121+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:08:32.356121+0000","last_undegraded":"2024-03-30T15:08:32.356121+0000","last_fullsized":"2024-03-30T15:08:32.356121+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.837619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.837619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":1143,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810955+0000","last_change":"2024-03-30T15:06:54.767293+0000","last_active":"2024-03-30T15:08:33.810955+0000","last_peered":"2024-03-30T15:08:33.810955+0000","last_clean":"2024-03-30T15:08:33.810955+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:08:33.810955+0000","last_undegraded":"2024-03-30T15:08:33.810955+0000","last_fullsized":"2024-03-30T15:08:33.810955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.767213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.767213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":1143,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811510+0000","last_change":"2024-03-30T15:07:25.988108+0000","last_active":"2024-03-30T15:08:33.811510+0000","last_peered":"2024-03-30T15:08:33.811510+0000","last_clean":"2024-03-30T15:08:33.811510+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:08:33.811510+0000","last_undegraded":"2024-03-30T15:08:33.811510+0000","last_fullsized":"2024-03-30T15:08:33.811510+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.988025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.988025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":1128,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529205+0000","last_change":"2024-03-30T15:07:59.150468+0000","last_active":"2024-03-30T15:08:32.529205+0000","last_peered":"2024-03-30T15:08:32.529205+0000","last_clean":"2024-03-30T15:08:32.529205+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:08:32.529205+0000","last_undegraded":"2024-03-30T15:08:32.529205+0000","last_fullsized":"2024-03-30T15:08:32.529205+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:59.150361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.150361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.062144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359366+0000","last_change":"2024-03-30T15:07:25.964793+0000","last_active":"2024-03-30T15:08:32.359366+0000","last_peered":"2024-03-30T15:08:32.359366+0000","last_clean":"2024-03-30T15:08:32.359366+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:08:32.359366+0000","last_undegraded":"2024-03-30T15:08:32.359366+0000","last_fullsized":"2024-03-30T15:08:32.359366+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.964714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.964714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570073+0000","last_change":"2024-03-30T15:06:58.308106+0000","last_active":"2024-03-30T15:08:33.570073+0000","last_peered":"2024-03-30T15:08:33.570073+0000","last_clean":"2024-03-30T15:08:33.570073+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:08:33.570073+0000","last_undegraded":"2024-03-30T15:08:33.570073+0000","last_fullsized":"2024-03-30T15:08:33.570073+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.308015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.308015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529071+0000","last_change":"2024-03-30T15:07:35.172101+0000","last_active":"2024-03-30T15:08:32.529071+0000","last_peered":"2024-03-30T15:08:32.529071+0000","last_clean":"2024-03-30T15:08:32.529071+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:08:32.529071+0000","last_undegraded":"2024-03-30T15:08:32.529071+0000","last_fullsized":"2024-03-30T15:08:32.529071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.171990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.171990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:02.621474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":1138,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356450+0000","last_change":"2024-03-30T15:08:09.224705+0000","last_active":"2024-03-30T15:08:32.356450+0000","last_peered":"2024-03-30T15:08:32.356450+0000","last_clean":"2024-03-30T15:08:32.356450+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:08:32.356450+0000","last_undegraded":"2024-03-30T15:08:32.356450+0000","last_fullsized":"2024-03-30T15:08:32.356450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.224499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.224499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.632314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":1128,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526309+0000","last_change":"2024-03-30T15:06:58.979164+0000","last_active":"2024-03-30T15:08:32.526309+0000","last_peered":"2024-03-30T15:08:32.526309+0000","last_clean":"2024-03-30T15:08:32.526309+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:08:32.526309+0000","last_undegraded":"2024-03-30T15:08:32.526309+0000","last_fullsized":"2024-03-30T15:08:32.526309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.979074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.979074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":1136,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526372+0000","last_change":"2024-03-30T15:08:08.551682+0000","last_active":"2024-03-30T15:08:32.526372+0000","last_peered":"2024-03-30T15:08:32.526372+0000","last_clean":"2024-03-30T15:08:32.526372+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:08:32.526372+0000","last_undegraded":"2024-03-30T15:08:32.526372+0000","last_fullsized":"2024-03-30T15:08:32.526372+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.551547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.551547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.564084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":1144,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811123+0000","last_change":"2024-03-30T15:06:45.791752+0000","last_active":"2024-03-30T15:08:33.811123+0000","last_peered":"2024-03-30T15:08:33.811123+0000","last_clean":"2024-03-30T15:08:33.811123+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:08:33.811123+0000","last_undegraded":"2024-03-30T15:08:33.811123+0000","last_fullsized":"2024-03-30T15:08:33.811123+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.791667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.791667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":1135,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527092+0000","last_change":"2024-03-30T15:08:09.548782+0000","last_active":"2024-03-30T15:08:32.527092+0000","last_peered":"2024-03-30T15:08:32.527092+0000","last_clean":"2024-03-30T15:08:32.527092+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:08:32.527092+0000","last_undegraded":"2024-03-30T15:08:32.527092+0000","last_fullsized":"2024-03-30T15:08:32.527092+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.548687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.755962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":246,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569266+0000","last_change":"2024-03-30T15:06:18.488509+0000","last_active":"2024-03-30T15:08:33.569266+0000","last_peered":"2024-03-30T15:08:33.569266+0000","last_clean":"2024-03-30T15:08:33.569266+0000","last_became_active":"2024-03-30T15:06:18.488207+0000","last_became_peered":"2024-03-30T15:06:18.488207+0000","last_unstale":"2024-03-30T15:08:33.569266+0000","last_undegraded":"2024-03-30T15:08:33.569266+0000","last_fullsized":"2024-03-30T15:08:33.569266+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.176687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.176687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570367+0000","last_change":"2024-03-30T15:08:02.711973+0000","last_active":"2024-03-30T15:08:33.570367+0000","last_peered":"2024-03-30T15:08:33.570367+0000","last_clean":"2024-03-30T15:08:33.570367+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:08:33.570367+0000","last_undegraded":"2024-03-30T15:08:33.570367+0000","last_fullsized":"2024-03-30T15:08:33.570367+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.711881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.711881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.583184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569806+0000","last_change":"2024-03-30T15:07:26.321108+0000","last_active":"2024-03-30T15:08:33.569806+0000","last_peered":"2024-03-30T15:08:33.569806+0000","last_clean":"2024-03-30T15:08:33.569806+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:08:33.569806+0000","last_undegraded":"2024-03-30T15:08:33.569806+0000","last_fullsized":"2024-03-30T15:08:33.569806+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.321030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:38.839544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":1135,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.524981+0000","last_change":"2024-03-30T15:07:31.354747+0000","last_active":"2024-03-30T15:08:32.524981+0000","last_peered":"2024-03-30T15:08:32.524981+0000","last_clean":"2024-03-30T15:08:32.524981+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:08:32.524981+0000","last_undegraded":"2024-03-30T15:08:32.524981+0000","last_fullsized":"2024-03-30T15:08:32.524981+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.354669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.354669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.082266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568294+0000","last_change":"2024-03-30T15:06:56.238490+0000","last_active":"2024-03-30T15:08:33.568294+0000","last_peered":"2024-03-30T15:08:33.568294+0000","last_clean":"2024-03-30T15:08:33.568294+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:08:33.568294+0000","last_undegraded":"2024-03-30T15:08:33.568294+0000","last_fullsized":"2024-03-30T15:08:33.568294+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.238398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.238398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":1126,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358241+0000","last_change":"2024-03-30T15:07:19.822016+0000","last_active":"2024-03-30T15:08:32.358241+0000","last_peered":"2024-03-30T15:08:32.358241+0000","last_clean":"2024-03-30T15:08:32.358241+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:08:32.358241+0000","last_undegraded":"2024-03-30T15:08:32.358241+0000","last_fullsized":"2024-03-30T15:08:32.358241+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.821886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.821886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":1127,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527845+0000","last_change":"2024-03-30T15:07:58.173279+0000","last_active":"2024-03-30T15:08:32.527845+0000","last_peered":"2024-03-30T15:08:32.527845+0000","last_clean":"2024-03-30T15:08:32.527845+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:08:32.527845+0000","last_undegraded":"2024-03-30T15:08:32.527845+0000","last_fullsized":"2024-03-30T15:08:32.527845+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.173193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.173193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:11.398411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1517,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568411+0000","last_change":"2024-03-30T15:06:49.183096+0000","last_active":"2024-03-30T15:08:33.568411+0000","last_peered":"2024-03-30T15:08:33.568411+0000","last_clean":"2024-03-30T15:08:33.568411+0000","last_became_active":"2024-03-30T15:06:48.338292+0000","last_became_peered":"2024-03-30T15:06:48.338292+0000","last_unstale":"2024-03-30T15:08:33.568411+0000","last_undegraded":"2024-03-30T15:08:33.568411+0000","last_fullsized":"2024-03-30T15:08:33.568411+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.183009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.183009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1141,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527089+0000","last_change":"2024-03-30T15:06:43.474123+0000","last_active":"2024-03-30T15:08:32.527089+0000","last_peered":"2024-03-30T15:08:32.527089+0000","last_clean":"2024-03-30T15:08:32.527089+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:08:32.527089+0000","last_undegraded":"2024-03-30T15:08:32.527089+0000","last_fullsized":"2024-03-30T15:08:32.527089+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.474015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.474015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1261,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359478+0000","last_change":"2024-03-30T15:08:06.185130+0000","last_active":"2024-03-30T15:08:32.359478+0000","last_peered":"2024-03-30T15:08:32.359478+0000","last_clean":"2024-03-30T15:08:32.359478+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:08:32.359478+0000","last_undegraded":"2024-03-30T15:08:32.359478+0000","last_fullsized":"2024-03-30T15:08:32.359478+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:08:06.185037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.185037+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.018400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1133,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367007+0000","last_change":"2024-03-30T15:06:11.536378+0000","last_active":"2024-03-30T15:08:33.367007+0000","last_peered":"2024-03-30T15:08:33.367007+0000","last_clean":"2024-03-30T15:08:33.367007+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:08:33.367007+0000","last_undegraded":"2024-03-30T15:08:33.367007+0000","last_fullsized":"2024-03-30T15:08:33.367007+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":416,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527099+0000","last_change":"2024-03-30T15:06:58.353375+0000","last_active":"2024-03-30T15:08:32.527099+0000","last_peered":"2024-03-30T15:08:32.527099+0000","last_clean":"2024-03-30T15:08:32.527099+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:08:32.527099+0000","last_undegraded":"2024-03-30T15:08:32.527099+0000","last_fullsized":"2024-03-30T15:08:32.527099+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'9","last_scrub_stamp":"2024-03-30T15:06:58.353220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.353220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867788+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1215,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360179+0000","last_change":"2024-03-30T15:06:44.519216+0000","last_active":"2024-03-30T15:08:32.360179+0000","last_peered":"2024-03-30T15:08:32.360179+0000","last_clean":"2024-03-30T15:08:32.360179+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:32.360179+0000","last_undegraded":"2024-03-30T15:08:32.360179+0000","last_fullsized":"2024-03-30T15:08:32.360179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:44.519122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.519122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":513,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.524553+0000","last_change":"2024-03-30T15:06:49.951825+0000","last_active":"2024-03-30T15:08:32.524553+0000","last_peered":"2024-03-30T15:08:32.524553+0000","last_clean":"2024-03-30T15:08:32.524553+0000","last_became_active":"2024-03-30T15:06:48.340482+0000","last_became_peered":"2024-03-30T15:06:48.340482+0000","last_unstale":"2024-03-30T15:08:32.524553+0000","last_undegraded":"2024-03-30T15:08:32.524553+0000","last_fullsized":"2024-03-30T15:08:32.524553+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.951714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.951714+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1141,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527190+0000","last_change":"2024-03-30T15:07:22.415753+0000","last_active":"2024-03-30T15:08:32.527190+0000","last_peered":"2024-03-30T15:08:32.527190+0000","last_clean":"2024-03-30T15:08:32.527190+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:08:32.527190+0000","last_undegraded":"2024-03-30T15:08:32.527190+0000","last_fullsized":"2024-03-30T15:08:32.527190+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.415581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.415581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1280,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359981+0000","last_change":"2024-03-30T15:07:20.849551+0000","last_active":"2024-03-30T15:08:32.359981+0000","last_peered":"2024-03-30T15:08:32.359981+0000","last_clean":"2024-03-30T15:08:32.359981+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:08:32.359981+0000","last_undegraded":"2024-03-30T15:08:32.359981+0000","last_fullsized":"2024-03-30T15:08:32.359981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:07:20.849431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.849431+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1243,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525513+0000","last_change":"2024-03-30T15:07:34.300735+0000","last_active":"2024-03-30T15:08:32.525513+0000","last_peered":"2024-03-30T15:08:32.525513+0000","last_clean":"2024-03-30T15:08:32.525513+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:08:32.525513+0000","last_undegraded":"2024-03-30T15:08:32.525513+0000","last_fullsized":"2024-03-30T15:08:32.525513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:34.300632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.300632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.024216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1238,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359498+0000","last_change":"2024-03-30T15:07:19.878091+0000","last_active":"2024-03-30T15:08:32.359498+0000","last_peered":"2024-03-30T15:08:32.359498+0000","last_clean":"2024-03-30T15:08:32.359498+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:08:32.359498+0000","last_undegraded":"2024-03-30T15:08:32.359498+0000","last_fullsized":"2024-03-30T15:08:32.359498+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.878011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.878011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.486305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1244,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527100+0000","last_change":"2024-03-30T15:08:02.197580+0000","last_active":"2024-03-30T15:08:32.527100+0000","last_peered":"2024-03-30T15:08:32.527100+0000","last_clean":"2024-03-30T15:08:32.527100+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:08:32.527100+0000","last_undegraded":"2024-03-30T15:08:32.527100+0000","last_fullsized":"2024-03-30T15:08:32.527100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.197498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.197498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.675581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1143,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360615+0000","last_change":"2024-03-30T15:08:05.112361+0000","last_active":"2024-03-30T15:08:32.360615+0000","last_peered":"2024-03-30T15:08:32.360615+0000","last_clean":"2024-03-30T15:08:32.360615+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:08:32.360615+0000","last_undegraded":"2024-03-30T15:08:32.360615+0000","last_fullsized":"2024-03-30T15:08:32.360615+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:05.112284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:05.112284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:08.797988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1267,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360059+0000","last_change":"2024-03-30T15:07:58.225113+0000","last_active":"2024-03-30T15:08:32.360059+0000","last_peered":"2024-03-30T15:08:32.360059+0000","last_clean":"2024-03-30T15:08:32.360059+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:08:32.360059+0000","last_undegraded":"2024-03-30T15:08:32.360059+0000","last_fullsized":"2024-03-30T15:08:32.360059+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:07:58.225002+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.225002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.514446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358511+0000","last_change":"2024-03-30T15:07:33.014237+0000","last_active":"2024-03-30T15:08:32.358511+0000","last_peered":"2024-03-30T15:08:32.358511+0000","last_clean":"2024-03-30T15:08:32.358511+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:08:32.358511+0000","last_undegraded":"2024-03-30T15:08:32.358511+0000","last_fullsized":"2024-03-30T15:08:32.358511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.014124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.688496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1227,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.530208+0000","last_change":"2024-03-30T15:08:21.357777+0000","last_active":"2024-03-30T15:08:32.530208+0000","last_peered":"2024-03-30T15:08:32.530208+0000","last_clean":"2024-03-30T15:08:32.530208+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:08:32.530208+0000","last_undegraded":"2024-03-30T15:08:32.530208+0000","last_fullsized":"2024-03-30T15:08:32.530208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.357646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.357646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.045428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1144,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527266+0000","last_change":"2024-03-30T15:08:10.524135+0000","last_active":"2024-03-30T15:08:32.527266+0000","last_peered":"2024-03-30T15:08:32.527266+0000","last_clean":"2024-03-30T15:08:32.527266+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:08:32.527266+0000","last_undegraded":"2024-03-30T15:08:32.527266+0000","last_fullsized":"2024-03-30T15:08:32.527266+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.524046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.524046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.049104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":513,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527624+0000","last_change":"2024-03-30T15:06:50.920479+0000","last_active":"2024-03-30T15:08:32.527624+0000","last_peered":"2024-03-30T15:08:32.527624+0000","last_clean":"2024-03-30T15:08:32.527624+0000","last_became_active":"2024-03-30T15:06:48.342510+0000","last_became_peered":"2024-03-30T15:06:48.342510+0000","last_unstale":"2024-03-30T15:08:32.527624+0000","last_undegraded":"2024-03-30T15:08:32.527624+0000","last_fullsized":"2024-03-30T15:08:32.527624+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:50.920389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.920389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1143,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.361492+0000","last_change":"2024-03-30T15:08:06.129401+0000","last_active":"2024-03-30T15:08:32.361492+0000","last_peered":"2024-03-30T15:08:32.361492+0000","last_clean":"2024-03-30T15:08:32.361492+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:08:32.361492+0000","last_undegraded":"2024-03-30T15:08:32.361492+0000","last_fullsized":"2024-03-30T15:08:32.361492+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.129319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.129319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:13.993935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1284,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.361595+0000","last_change":"2024-03-30T15:07:28.992483+0000","last_active":"2024-03-30T15:08:32.361595+0000","last_peered":"2024-03-30T15:08:32.361595+0000","last_clean":"2024-03-30T15:08:32.361595+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:08:32.361595+0000","last_undegraded":"2024-03-30T15:08:32.361595+0000","last_fullsized":"2024-03-30T15:08:32.361595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:28.992348+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.992348+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1139,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356977+0000","last_change":"2024-03-30T15:07:22.928759+0000","last_active":"2024-03-30T15:08:32.356977+0000","last_peered":"2024-03-30T15:08:32.356977+0000","last_clean":"2024-03-30T15:08:32.356977+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:08:32.356977+0000","last_undegraded":"2024-03-30T15:08:32.356977+0000","last_fullsized":"2024-03-30T15:08:32.356977+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.928677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.928677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.321766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1403,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360149+0000","last_change":"2024-03-30T15:08:15.143675+0000","last_active":"2024-03-30T15:08:32.360149+0000","last_peered":"2024-03-30T15:08:32.360149+0000","last_clean":"2024-03-30T15:08:32.360149+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:08:32.360149+0000","last_undegraded":"2024-03-30T15:08:32.360149+0000","last_fullsized":"2024-03-30T15:08:32.360149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:15.143526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.143526+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.962882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1215,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529279+0000","last_change":"2024-03-30T15:06:57.951315+0000","last_active":"2024-03-30T15:08:32.529279+0000","last_peered":"2024-03-30T15:08:32.529279+0000","last_clean":"2024-03-30T15:08:32.529279+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:08:32.529279+0000","last_undegraded":"2024-03-30T15:08:32.529279+0000","last_fullsized":"2024-03-30T15:08:32.529279+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.951208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.951208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":514,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358982+0000","last_change":"2024-03-30T15:06:48.534844+0000","last_active":"2024-03-30T15:08:32.358982+0000","last_peered":"2024-03-30T15:08:32.358982+0000","last_clean":"2024-03-30T15:08:32.358982+0000","last_became_active":"2024-03-30T15:06:48.341420+0000","last_became_peered":"2024-03-30T15:06:48.341420+0000","last_unstale":"2024-03-30T15:08:32.358982+0000","last_undegraded":"2024-03-30T15:08:32.358982+0000","last_fullsized":"2024-03-30T15:08:32.358982+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.534763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.534763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1143,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810622+0000","last_change":"2024-03-30T15:07:36.166030+0000","last_active":"2024-03-30T15:08:33.810622+0000","last_peered":"2024-03-30T15:08:33.810622+0000","last_clean":"2024-03-30T15:08:33.810622+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:08:33.810622+0000","last_undegraded":"2024-03-30T15:08:33.810622+0000","last_fullsized":"2024-03-30T15:08:33.810622+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.165947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.824703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1227,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358484+0000","last_change":"2024-03-30T15:07:57.209970+0000","last_active":"2024-03-30T15:08:32.358484+0000","last_peered":"2024-03-30T15:08:32.358484+0000","last_clean":"2024-03-30T15:08:32.358484+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:08:32.358484+0000","last_undegraded":"2024-03-30T15:08:32.358484+0000","last_fullsized":"2024-03-30T15:08:32.358484+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.209884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.209884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.514345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1248,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569169+0000","last_change":"2024-03-30T15:08:10.679605+0000","last_active":"2024-03-30T15:08:33.569169+0000","last_peered":"2024-03-30T15:08:33.569169+0000","last_clean":"2024-03-30T15:08:33.569169+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:08:33.569169+0000","last_undegraded":"2024-03-30T15:08:33.569169+0000","last_fullsized":"2024-03-30T15:08:33.569169+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:08:10.679491+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.679491+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:39.573167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1213,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570149+0000","last_change":"2024-03-30T15:08:15.656962+0000","last_active":"2024-03-30T15:08:33.570149+0000","last_peered":"2024-03-30T15:08:33.570149+0000","last_clean":"2024-03-30T15:08:33.570149+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:08:33.570149+0000","last_undegraded":"2024-03-30T15:08:33.570149+0000","last_fullsized":"2024-03-30T15:08:33.570149+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:08:15.656821+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.656821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.329037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1223,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359216+0000","last_change":"2024-03-30T15:08:10.175409+0000","last_active":"2024-03-30T15:08:32.359216+0000","last_peered":"2024-03-30T15:08:32.359216+0000","last_clean":"2024-03-30T15:08:32.359216+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:08:32.359216+0000","last_undegraded":"2024-03-30T15:08:32.359216+0000","last_fullsized":"2024-03-30T15:08:32.359216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.175319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.192582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":513,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528815+0000","last_change":"2024-03-30T15:06:51.940521+0000","last_active":"2024-03-30T15:08:32.528815+0000","last_peered":"2024-03-30T15:08:32.528815+0000","last_clean":"2024-03-30T15:08:32.528815+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:08:32.528815+0000","last_undegraded":"2024-03-30T15:08:32.528815+0000","last_fullsized":"2024-03-30T15:08:32.528815+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:51.940375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:51.940375+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1153,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810702+0000","last_change":"2024-03-30T15:07:24.023700+0000","last_active":"2024-03-30T15:08:33.810702+0000","last_peered":"2024-03-30T15:08:33.810702+0000","last_clean":"2024-03-30T15:08:33.810702+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:08:33.810702+0000","last_undegraded":"2024-03-30T15:08:33.810702+0000","last_fullsized":"2024-03-30T15:08:33.810702+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.023617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.023617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.066284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1284,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356446+0000","last_change":"2024-03-30T15:06:11.179123+0000","last_active":"2024-03-30T15:08:32.356446+0000","last_peered":"2024-03-30T15:08:32.356446+0000","last_clean":"2024-03-30T15:08:32.356446+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:08:32.356446+0000","last_undegraded":"2024-03-30T15:08:32.356446+0000","last_fullsized":"2024-03-30T15:08:32.356446+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.179008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.179008+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1273,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358633+0000","last_change":"2024-03-30T15:06:41.454580+0000","last_active":"2024-03-30T15:08:32.358633+0000","last_peered":"2024-03-30T15:08:32.358633+0000","last_clean":"2024-03-30T15:08:32.358633+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:08:32.358633+0000","last_undegraded":"2024-03-30T15:08:32.358633+0000","last_fullsized":"2024-03-30T15:08:32.358633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:41.454445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:41.454445+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1234,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369421+0000","last_change":"2024-03-30T15:07:24.562748+0000","last_active":"2024-03-30T15:08:33.369421+0000","last_peered":"2024-03-30T15:08:33.369421+0000","last_clean":"2024-03-30T15:08:33.369421+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:08:33.369421+0000","last_undegraded":"2024-03-30T15:08:33.369421+0000","last_fullsized":"2024-03-30T15:08:33.369421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.710249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1227,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569162+0000","last_change":"2024-03-30T15:06:39.147213+0000","last_active":"2024-03-30T15:08:33.569162+0000","last_peered":"2024-03-30T15:08:33.569162+0000","last_clean":"2024-03-30T15:08:33.569162+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:08:33.569162+0000","last_undegraded":"2024-03-30T15:08:33.569162+0000","last_fullsized":"2024-03-30T15:08:33.569162+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.147113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.147113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":520,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528039+0000","last_change":"2024-03-30T15:06:49.387252+0000","last_active":"2024-03-30T15:08:32.528039+0000","last_peered":"2024-03-30T15:08:32.528039+0000","last_clean":"2024-03-30T15:08:32.528039+0000","last_became_active":"2024-03-30T15:06:48.342151+0000","last_became_peered":"2024-03-30T15:06:48.342151+0000","last_unstale":"2024-03-30T15:08:32.528039+0000","last_undegraded":"2024-03-30T15:08:32.528039+0000","last_fullsized":"2024-03-30T15:08:32.528039+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.387153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.387153+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867705+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1150,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811167+0000","last_change":"2024-03-30T15:07:25.015706+0000","last_active":"2024-03-30T15:08:33.811167+0000","last_peered":"2024-03-30T15:08:33.811167+0000","last_clean":"2024-03-30T15:08:33.811167+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:08:33.811167+0000","last_undegraded":"2024-03-30T15:08:33.811167+0000","last_fullsized":"2024-03-30T15:08:33.811167+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.015600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.015600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.857225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1408,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.366903+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:08:33.366903+0000","last_peered":"2024-03-30T15:08:33.366903+0000","last_clean":"2024-03-30T15:08:33.366903+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:08:33.366903+0000","last_undegraded":"2024-03-30T15:08:33.366903+0000","last_fullsized":"2024-03-30T15:08:33.366903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1277,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357839+0000","last_change":"2024-03-30T15:08:01.102530+0000","last_active":"2024-03-30T15:08:32.357839+0000","last_peered":"2024-03-30T15:08:32.357839+0000","last_clean":"2024-03-30T15:08:32.357839+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:08:32.357839+0000","last_undegraded":"2024-03-30T15:08:32.357839+0000","last_fullsized":"2024-03-30T15:08:32.357839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:08:01.102431+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.102431+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.685021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529589+0000","last_change":"2024-03-30T15:06:53.935771+0000","last_active":"2024-03-30T15:08:32.529589+0000","last_peered":"2024-03-30T15:08:32.529589+0000","last_clean":"2024-03-30T15:08:32.529589+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:08:32.529589+0000","last_undegraded":"2024-03-30T15:08:32.529589+0000","last_fullsized":"2024-03-30T15:08:32.529589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.935674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.935674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1220,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364175+0000","last_change":"2024-03-30T15:06:10.554067+0000","last_active":"2024-03-30T15:08:33.364175+0000","last_peered":"2024-03-30T15:08:33.364175+0000","last_clean":"2024-03-30T15:08:33.364175+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:08:33.364175+0000","last_undegraded":"2024-03-30T15:08:33.364175+0000","last_fullsized":"2024-03-30T15:08:33.364175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":520,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527447+0000","last_change":"2024-03-30T15:06:48.409534+0000","last_active":"2024-03-30T15:08:32.527447+0000","last_peered":"2024-03-30T15:08:32.527447+0000","last_clean":"2024-03-30T15:08:32.527447+0000","last_became_active":"2024-03-30T15:06:48.338506+0000","last_became_peered":"2024-03-30T15:06:48.338506+0000","last_unstale":"2024-03-30T15:08:32.527447+0000","last_undegraded":"2024-03-30T15:08:32.527447+0000","last_fullsized":"2024-03-30T15:08:32.527447+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.409455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.409455+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1143,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811137+0000","last_change":"2024-03-30T15:06:59.884862+0000","last_active":"2024-03-30T15:08:33.811137+0000","last_peered":"2024-03-30T15:08:33.811137+0000","last_clean":"2024-03-30T15:08:33.811137+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:08:33.811137+0000","last_undegraded":"2024-03-30T15:08:33.811137+0000","last_fullsized":"2024-03-30T15:08:33.811137+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.884709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.884709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525829+0000","last_change":"2024-03-30T15:06:54.318139+0000","last_active":"2024-03-30T15:08:32.525829+0000","last_peered":"2024-03-30T15:08:32.525829+0000","last_clean":"2024-03-30T15:08:32.525829+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:08:32.525829+0000","last_undegraded":"2024-03-30T15:08:32.525829+0000","last_fullsized":"2024-03-30T15:08:32.525829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.318005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1287,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367300+0000","last_change":"2024-03-30T15:08:02.516526+0000","last_active":"2024-03-30T15:08:33.367300+0000","last_peered":"2024-03-30T15:08:33.367300+0000","last_clean":"2024-03-30T15:08:33.367300+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:08:33.367300+0000","last_undegraded":"2024-03-30T15:08:33.367300+0000","last_fullsized":"2024-03-30T15:08:33.367300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:14.340123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1246,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359415+0000","last_change":"2024-03-30T15:07:17.809348+0000","last_active":"2024-03-30T15:08:32.359415+0000","last_peered":"2024-03-30T15:08:32.359415+0000","last_clean":"2024-03-30T15:08:32.359415+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:08:32.359415+0000","last_undegraded":"2024-03-30T15:08:32.359415+0000","last_fullsized":"2024-03-30T15:08:32.359415+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:17.809228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:17.809228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356717+0000","last_change":"2024-03-30T15:06:39.844760+0000","last_active":"2024-03-30T15:08:32.356717+0000","last_peered":"2024-03-30T15:08:32.356717+0000","last_clean":"2024-03-30T15:08:32.356717+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:08:32.356717+0000","last_undegraded":"2024-03-30T15:08:32.356717+0000","last_fullsized":"2024-03-30T15:08:32.356717+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.844668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.844668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":515,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358853+0000","last_change":"2024-03-30T15:06:48.873065+0000","last_active":"2024-03-30T15:08:32.358853+0000","last_peered":"2024-03-30T15:08:32.358853+0000","last_clean":"2024-03-30T15:08:32.358853+0000","last_became_active":"2024-03-30T15:06:48.339258+0000","last_became_peered":"2024-03-30T15:06:48.339258+0000","last_unstale":"2024-03-30T15:08:32.358853+0000","last_undegraded":"2024-03-30T15:08:32.358853+0000","last_fullsized":"2024-03-30T15:08:32.358853+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.872974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.872974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369351+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:08:33.369351+0000","last_peered":"2024-03-30T15:08:33.369351+0000","last_clean":"2024-03-30T15:08:33.369351+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:08:33.369351+0000","last_undegraded":"2024-03-30T15:08:33.369351+0000","last_fullsized":"2024-03-30T15:08:33.369351+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1227,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.530188+0000","last_change":"2024-03-30T15:06:52.973865+0000","last_active":"2024-03-30T15:08:32.530188+0000","last_peered":"2024-03-30T15:08:32.530188+0000","last_clean":"2024-03-30T15:08:32.530188+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:08:32.530188+0000","last_undegraded":"2024-03-30T15:08:32.530188+0000","last_fullsized":"2024-03-30T15:08:32.530188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.973770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.973770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1272,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369316+0000","last_change":"2024-03-30T15:08:01.499074+0000","last_active":"2024-03-30T15:08:33.369316+0000","last_peered":"2024-03-30T15:08:33.369316+0000","last_clean":"2024-03-30T15:08:33.369316+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:08:33.369316+0000","last_undegraded":"2024-03-30T15:08:33.369316+0000","last_fullsized":"2024-03-30T15:08:33.369316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.212933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2080,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568560+0000","last_change":"2024-03-30T15:07:34.264898+0000","last_active":"2024-03-30T15:08:33.568560+0000","last_peered":"2024-03-30T15:08:33.568560+0000","last_clean":"2024-03-30T15:08:33.568560+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:08:33.568560+0000","last_undegraded":"2024-03-30T15:08:33.568560+0000","last_fullsized":"2024-03-30T15:08:33.568560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:07:34.264815+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.264815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.246868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":828,"num_read_kb":823,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1232,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364217+0000","last_change":"2024-03-30T15:07:29.592120+0000","last_active":"2024-03-30T15:08:33.364217+0000","last_peered":"2024-03-30T15:08:33.364217+0000","last_clean":"2024-03-30T15:08:33.364217+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:08:33.364217+0000","last_undegraded":"2024-03-30T15:08:33.364217+0000","last_fullsized":"2024-03-30T15:08:33.364217+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.592039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.592039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.178771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359330+0000","last_change":"2024-03-30T15:06:45.854073+0000","last_active":"2024-03-30T15:08:32.359330+0000","last_peered":"2024-03-30T15:08:32.359330+0000","last_clean":"2024-03-30T15:08:32.359330+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:08:32.359330+0000","last_undegraded":"2024-03-30T15:08:32.359330+0000","last_fullsized":"2024-03-30T15:08:32.359330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.853998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.853998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1135,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569868+0000","last_change":"2024-03-30T15:07:24.315658+0000","last_active":"2024-03-30T15:08:33.569868+0000","last_peered":"2024-03-30T15:08:33.569868+0000","last_clean":"2024-03-30T15:08:33.569868+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:08:33.569868+0000","last_undegraded":"2024-03-30T15:08:33.569868+0000","last_fullsized":"2024-03-30T15:08:33.569868+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.315584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.315584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:49.961946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1231,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569611+0000","last_change":"2024-03-30T15:08:06.704248+0000","last_active":"2024-03-30T15:08:33.569611+0000","last_peered":"2024-03-30T15:08:33.569611+0000","last_clean":"2024-03-30T15:08:33.569611+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:08:33.569611+0000","last_undegraded":"2024-03-30T15:08:33.569611+0000","last_fullsized":"2024-03-30T15:08:33.569611+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.704123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.704123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.001588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1447,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528171+0000","last_change":"2024-03-30T15:06:02.076986+0000","last_active":"2024-03-30T15:08:32.528171+0000","last_peered":"2024-03-30T15:08:32.528171+0000","last_clean":"2024-03-30T15:08:32.528171+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:08:32.528171+0000","last_undegraded":"2024-03-30T15:08:32.528171+0000","last_fullsized":"2024-03-30T15:08:32.528171+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:02.076864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.076864+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1130,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528021+0000","last_change":"2024-03-30T15:07:00.995237+0000","last_active":"2024-03-30T15:08:32.528021+0000","last_peered":"2024-03-30T15:08:32.528021+0000","last_clean":"2024-03-30T15:08:32.528021+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:08:32.528021+0000","last_undegraded":"2024-03-30T15:08:32.528021+0000","last_fullsized":"2024-03-30T15:08:32.528021+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.995156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.995156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1509,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568610+0000","last_change":"2024-03-30T15:06:45.099476+0000","last_active":"2024-03-30T15:08:33.568610+0000","last_peered":"2024-03-30T15:08:33.568610+0000","last_clean":"2024-03-30T15:08:33.568610+0000","last_became_active":"2024-03-30T15:06:43.342924+0000","last_became_peered":"2024-03-30T15:06:43.342924+0000","last_unstale":"2024-03-30T15:08:33.568610+0000","last_undegraded":"2024-03-30T15:08:33.568610+0000","last_fullsized":"2024-03-30T15:08:33.568610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:45.099372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.099372+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1215,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.530014+0000","last_change":"2024-03-30T15:06:45.066707+0000","last_active":"2024-03-30T15:08:32.530014+0000","last_peered":"2024-03-30T15:08:32.530014+0000","last_clean":"2024-03-30T15:08:32.530014+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:08:32.530014+0000","last_undegraded":"2024-03-30T15:08:32.530014+0000","last_fullsized":"2024-03-30T15:08:32.530014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.066574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.066574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1525,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568542+0000","last_change":"2024-03-30T15:07:56.660010+0000","last_active":"2024-03-30T15:08:33.568542+0000","last_peered":"2024-03-30T15:08:33.568542+0000","last_clean":"2024-03-30T15:08:33.568542+0000","last_became_active":"2024-03-30T15:06:48.339784+0000","last_became_peered":"2024-03-30T15:06:48.339784+0000","last_unstale":"2024-03-30T15:08:33.568542+0000","last_undegraded":"2024-03-30T15:08:33.568542+0000","last_fullsized":"2024-03-30T15:08:33.568542+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:07:56.659850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:07:56.659850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.574408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1140,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528291+0000","last_change":"2024-03-30T15:06:10.008324+0000","last_active":"2024-03-30T15:08:32.528291+0000","last_peered":"2024-03-30T15:08:32.528291+0000","last_clean":"2024-03-30T15:08:32.528291+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:08:32.528291+0000","last_undegraded":"2024-03-30T15:08:32.528291+0000","last_fullsized":"2024-03-30T15:08:32.528291+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.008246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.008246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1248,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525620+0000","last_change":"2024-03-30T15:08:14.240324+0000","last_active":"2024-03-30T15:08:32.525620+0000","last_peered":"2024-03-30T15:08:32.525620+0000","last_clean":"2024-03-30T15:08:32.525620+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:08:32.525620+0000","last_undegraded":"2024-03-30T15:08:32.525620+0000","last_fullsized":"2024-03-30T15:08:32.525620+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:14.240226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:08:14.240226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.396355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1280,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.361378+0000","last_change":"2024-03-30T15:06:44.835160+0000","last_active":"2024-03-30T15:08:32.361378+0000","last_peered":"2024-03-30T15:08:32.361378+0000","last_clean":"2024-03-30T15:08:32.361378+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:08:32.361378+0000","last_undegraded":"2024-03-30T15:08:32.361378+0000","last_fullsized":"2024-03-30T15:08:32.361378+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:06:44.835068+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.835068+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1259,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810763+0000","last_change":"2024-03-30T15:06:57.786762+0000","last_active":"2024-03-30T15:08:33.810763+0000","last_peered":"2024-03-30T15:08:33.810763+0000","last_clean":"2024-03-30T15:08:33.810763+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:08:33.810763+0000","last_undegraded":"2024-03-30T15:08:33.810763+0000","last_fullsized":"2024-03-30T15:08:33.810763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.786685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.786685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1246,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356553+0000","last_change":"2024-03-30T15:07:23.937183+0000","last_active":"2024-03-30T15:08:32.356553+0000","last_peered":"2024-03-30T15:08:32.356553+0000","last_clean":"2024-03-30T15:08:32.356553+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:08:32.356553+0000","last_undegraded":"2024-03-30T15:08:32.356553+0000","last_fullsized":"2024-03-30T15:08:32.356553+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.937096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.937096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1235,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570087+0000","last_change":"2024-03-30T15:07:58.653866+0000","last_active":"2024-03-30T15:08:33.570087+0000","last_peered":"2024-03-30T15:08:33.570087+0000","last_clean":"2024-03-30T15:08:33.570087+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:08:33.570087+0000","last_undegraded":"2024-03-30T15:08:33.570087+0000","last_fullsized":"2024-03-30T15:08:33.570087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.653564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.653564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.992993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":513,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526012+0000","last_change":"2024-03-30T15:06:48.962204+0000","last_active":"2024-03-30T15:08:32.526012+0000","last_peered":"2024-03-30T15:08:32.526012+0000","last_clean":"2024-03-30T15:08:32.526012+0000","last_became_active":"2024-03-30T15:06:48.339451+0000","last_became_peered":"2024-03-30T15:06:48.339451+0000","last_unstale":"2024-03-30T15:08:32.526012+0000","last_undegraded":"2024-03-30T15:08:32.526012+0000","last_fullsized":"2024-03-30T15:08:32.526012+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.962124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.962124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368141+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:08:33.368141+0000","last_peered":"2024-03-30T15:08:33.368141+0000","last_clean":"2024-03-30T15:08:33.368141+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:08:33.368141+0000","last_undegraded":"2024-03-30T15:08:33.368141+0000","last_fullsized":"2024-03-30T15:08:33.368141+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1224,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368518+0000","last_change":"2024-03-30T15:04:53.572060+0000","last_active":"2024-03-30T15:08:33.368518+0000","last_peered":"2024-03-30T15:08:33.368518+0000","last_clean":"2024-03-30T15:08:33.368518+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:08:33.368518+0000","last_undegraded":"2024-03-30T15:08:33.368518+0000","last_fullsized":"2024-03-30T15:08:33.368518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:53.571952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:04:53.571952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1412,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528998+0000","last_change":"2024-03-30T15:06:05.010246+0000","last_active":"2024-03-30T15:08:32.528998+0000","last_peered":"2024-03-30T15:08:32.528998+0000","last_clean":"2024-03-30T15:08:32.528998+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:08:32.528998+0000","last_undegraded":"2024-03-30T15:08:32.528998+0000","last_fullsized":"2024-03-30T15:08:32.528998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:06:05.010128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:05.010128+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358403+0000","last_change":"2024-03-30T15:07:31.968746+0000","last_active":"2024-03-30T15:08:32.358403+0000","last_peered":"2024-03-30T15:08:32.358403+0000","last_clean":"2024-03-30T15:08:32.358403+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:08:32.358403+0000","last_undegraded":"2024-03-30T15:08:32.358403+0000","last_fullsized":"2024-03-30T15:08:32.358403+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.259707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1238,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528124+0000","last_change":"2024-03-30T15:06:40.992863+0000","last_active":"2024-03-30T15:08:32.528124+0000","last_peered":"2024-03-30T15:08:32.528124+0000","last_clean":"2024-03-30T15:08:32.528124+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:08:32.528124+0000","last_undegraded":"2024-03-30T15:08:32.528124+0000","last_fullsized":"2024-03-30T15:08:32.528124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:06:40.992758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.992758+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1135,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359744+0000","last_change":"2024-03-30T15:08:00.231743+0000","last_active":"2024-03-30T15:08:32.359744+0000","last_peered":"2024-03-30T15:08:32.359744+0000","last_clean":"2024-03-30T15:08:32.359744+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:08:32.359744+0000","last_undegraded":"2024-03-30T15:08:32.359744+0000","last_fullsized":"2024-03-30T15:08:32.359744+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.231663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.231663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.864990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1130,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367132+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:08:33.367132+0000","last_peered":"2024-03-30T15:08:33.367132+0000","last_clean":"2024-03-30T15:08:33.367132+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:08:33.367132+0000","last_undegraded":"2024-03-30T15:08:33.367132+0000","last_fullsized":"2024-03-30T15:08:33.367132+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1261,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367095+0000","last_change":"2024-03-30T15:08:12.601879+0000","last_active":"2024-03-30T15:08:33.367095+0000","last_peered":"2024-03-30T15:08:33.367095+0000","last_clean":"2024-03-30T15:08:33.367095+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:08:33.367095+0000","last_undegraded":"2024-03-30T15:08:33.367095+0000","last_fullsized":"2024-03-30T15:08:33.367095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.593442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529696+0000","last_change":"2024-03-30T15:07:20.139612+0000","last_active":"2024-03-30T15:08:32.529696+0000","last_peered":"2024-03-30T15:08:32.529696+0000","last_clean":"2024-03-30T15:08:32.529696+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:08:32.529696+0000","last_undegraded":"2024-03-30T15:08:32.529696+0000","last_fullsized":"2024-03-30T15:08:32.529696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:20.139530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.139530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.029764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1250,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.809917+0000","last_change":"2024-03-30T15:07:58.247377+0000","last_active":"2024-03-30T15:08:33.809917+0000","last_peered":"2024-03-30T15:08:33.809917+0000","last_clean":"2024-03-30T15:08:33.809917+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:08:33.809917+0000","last_undegraded":"2024-03-30T15:08:33.809917+0000","last_fullsized":"2024-03-30T15:08:33.809917+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.247271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.247271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.826144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1220,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364591+0000","last_change":"2024-03-30T15:07:21.573597+0000","last_active":"2024-03-30T15:08:33.364591+0000","last_peered":"2024-03-30T15:08:33.364591+0000","last_clean":"2024-03-30T15:08:33.364591+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:08:33.364591+0000","last_undegraded":"2024-03-30T15:08:33.364591+0000","last_fullsized":"2024-03-30T15:08:33.364591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.573522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.573522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:36.445672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1132,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.360108+0000","last_change":"2024-03-30T15:07:22.909083+0000","last_active":"2024-03-30T15:08:32.360108+0000","last_peered":"2024-03-30T15:08:32.360108+0000","last_clean":"2024-03-30T15:08:32.360108+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:08:32.360108+0000","last_undegraded":"2024-03-30T15:08:32.360108+0000","last_fullsized":"2024-03-30T15:08:32.360108+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.908989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.908989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.320783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1320,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525710+0000","last_change":"2024-03-30T15:06:44.482561+0000","last_active":"2024-03-30T15:08:32.525710+0000","last_peered":"2024-03-30T15:08:32.525710+0000","last_clean":"2024-03-30T15:08:32.525710+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:08:32.525710+0000","last_undegraded":"2024-03-30T15:08:32.525710+0000","last_fullsized":"2024-03-30T15:08:32.525710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:06:44.482442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.482442+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1239,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569990+0000","last_change":"2024-03-30T15:07:18.376275+0000","last_active":"2024-03-30T15:08:33.569990+0000","last_peered":"2024-03-30T15:08:33.569990+0000","last_clean":"2024-03-30T15:08:33.569990+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:08:33.569990+0000","last_undegraded":"2024-03-30T15:08:33.569990+0000","last_fullsized":"2024-03-30T15:08:33.569990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.376197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.376197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.539972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1242,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.809923+0000","last_change":"2024-03-30T15:06:48.713525+0000","last_active":"2024-03-30T15:08:33.809923+0000","last_peered":"2024-03-30T15:08:33.809923+0000","last_clean":"2024-03-30T15:08:33.809923+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:08:33.809923+0000","last_undegraded":"2024-03-30T15:08:33.809923+0000","last_fullsized":"2024-03-30T15:08:33.809923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:48.713441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.713441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1227,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568470+0000","last_change":"2024-03-30T15:08:11.701750+0000","last_active":"2024-03-30T15:08:33.568470+0000","last_peered":"2024-03-30T15:08:33.568470+0000","last_clean":"2024-03-30T15:08:33.568470+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:08:33.568470+0000","last_undegraded":"2024-03-30T15:08:33.568470+0000","last_fullsized":"2024-03-30T15:08:33.568470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.701619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.701619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.937509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1154,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525094+0000","last_change":"2024-03-30T15:06:46.411474+0000","last_active":"2024-03-30T15:08:32.525094+0000","last_peered":"2024-03-30T15:08:32.525094+0000","last_clean":"2024-03-30T15:08:32.525094+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:08:32.525094+0000","last_undegraded":"2024-03-30T15:08:32.525094+0000","last_fullsized":"2024-03-30T15:08:32.525094+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:06:46.411383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:46.411383+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1242,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.809884+0000","last_change":"2024-03-30T15:06:58.835912+0000","last_active":"2024-03-30T15:08:33.809884+0000","last_peered":"2024-03-30T15:08:33.809884+0000","last_clean":"2024-03-30T15:08:33.809884+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:08:33.809884+0000","last_undegraded":"2024-03-30T15:08:33.809884+0000","last_fullsized":"2024-03-30T15:08:33.809884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.835804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.835804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.247232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1271,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358769+0000","last_change":"2024-03-30T15:08:08.213331+0000","last_active":"2024-03-30T15:08:32.358769+0000","last_peered":"2024-03-30T15:08:32.358769+0000","last_clean":"2024-03-30T15:08:32.358769+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:08:32.358769+0000","last_undegraded":"2024-03-30T15:08:32.358769+0000","last_fullsized":"2024-03-30T15:08:32.358769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:08:08.213219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.213219+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.230614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1239,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570079+0000","last_change":"2024-03-30T15:08:04.783312+0000","last_active":"2024-03-30T15:08:33.570079+0000","last_peered":"2024-03-30T15:08:33.570079+0000","last_clean":"2024-03-30T15:08:33.570079+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:08:33.570079+0000","last_undegraded":"2024-03-30T15:08:33.570079+0000","last_fullsized":"2024-03-30T15:08:33.570079+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:04.783237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.783237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.511079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1131,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569330+0000","last_change":"2024-03-30T15:06:54.229912+0000","last_active":"2024-03-30T15:08:33.569330+0000","last_peered":"2024-03-30T15:08:33.569330+0000","last_clean":"2024-03-30T15:08:33.569330+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:08:33.569330+0000","last_undegraded":"2024-03-30T15:08:33.569330+0000","last_fullsized":"2024-03-30T15:08:33.569330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.229818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.229818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1132,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357866+0000","last_change":"2024-03-30T15:07:18.807780+0000","last_active":"2024-03-30T15:08:32.357866+0000","last_peered":"2024-03-30T15:08:32.357866+0000","last_clean":"2024-03-30T15:08:32.357866+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:08:32.357866+0000","last_undegraded":"2024-03-30T15:08:32.357866+0000","last_fullsized":"2024-03-30T15:08:32.357866+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.807666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.807666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:47.636769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1399,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569318+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:08:33.569318+0000","last_peered":"2024-03-30T15:08:33.569318+0000","last_clean":"2024-03-30T15:08:33.569318+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:08:33.569318+0000","last_undegraded":"2024-03-30T15:08:33.569318+0000","last_fullsized":"2024-03-30T15:08:33.569318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1266,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526325+0000","last_change":"2024-03-30T15:06:16.030806+0000","last_active":"2024-03-30T15:08:32.526325+0000","last_peered":"2024-03-30T15:08:32.526325+0000","last_clean":"2024-03-30T15:08:32.526325+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:08:32.526325+0000","last_undegraded":"2024-03-30T15:08:32.526325+0000","last_fullsized":"2024-03-30T15:08:32.526325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:06:16.030694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.030694+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1251,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.570204+0000","last_change":"2024-03-30T15:07:32.333233+0000","last_active":"2024-03-30T15:08:33.570204+0000","last_peered":"2024-03-30T15:08:33.570204+0000","last_clean":"2024-03-30T15:08:33.570204+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:08:33.570204+0000","last_undegraded":"2024-03-30T15:08:33.570204+0000","last_fullsized":"2024-03-30T15:08:33.570204+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:07:32.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.333139+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:38.566901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1215,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528423+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:08:32.528423+0000","last_peered":"2024-03-30T15:08:32.528423+0000","last_clean":"2024-03-30T15:08:32.528423+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:32.528423+0000","last_undegraded":"2024-03-30T15:08:32.528423+0000","last_fullsized":"2024-03-30T15:08:32.528423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1141,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528157+0000","last_change":"2024-03-30T15:07:32.310952+0000","last_active":"2024-03-30T15:08:32.528157+0000","last_peered":"2024-03-30T15:08:32.528157+0000","last_clean":"2024-03-30T15:08:32.528157+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:08:32.528157+0000","last_undegraded":"2024-03-30T15:08:32.528157+0000","last_fullsized":"2024-03-30T15:08:32.528157+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.310831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.310831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.533764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":836,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368105+0000","last_change":"2024-03-30T15:08:12.833026+0000","last_active":"2024-03-30T15:08:33.368105+0000","last_peered":"2024-03-30T15:08:33.368105+0000","last_clean":"2024-03-30T15:08:33.368105+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:08:33.368105+0000","last_undegraded":"2024-03-30T15:08:33.368105+0000","last_fullsized":"2024-03-30T15:08:33.368105+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:08:12.832934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.832934+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.769346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1250,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.524771+0000","last_change":"2024-03-30T15:07:58.498811+0000","last_active":"2024-03-30T15:08:32.524771+0000","last_peered":"2024-03-30T15:08:32.524771+0000","last_clean":"2024-03-30T15:08:32.524771+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:08:32.524771+0000","last_undegraded":"2024-03-30T15:08:32.524771+0000","last_fullsized":"2024-03-30T15:08:32.524771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.498678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.498678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.745457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1218,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529204+0000","last_change":"2024-03-30T15:06:38.946768+0000","last_active":"2024-03-30T15:08:32.529204+0000","last_peered":"2024-03-30T15:08:32.529204+0000","last_clean":"2024-03-30T15:08:32.529204+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:08:32.529204+0000","last_undegraded":"2024-03-30T15:08:32.529204+0000","last_fullsized":"2024-03-30T15:08:32.529204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:38.946668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:38.946668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1218,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569572+0000","last_change":"2024-03-30T15:06:42.118047+0000","last_active":"2024-03-30T15:08:33.569572+0000","last_peered":"2024-03-30T15:08:33.569572+0000","last_clean":"2024-03-30T15:08:33.569572+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:08:33.569572+0000","last_undegraded":"2024-03-30T15:08:33.569572+0000","last_fullsized":"2024-03-30T15:08:33.569572+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:42.117927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.117927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1132,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359346+0000","last_change":"2024-03-30T15:07:29.943796+0000","last_active":"2024-03-30T15:08:32.359346+0000","last_peered":"2024-03-30T15:08:32.359346+0000","last_clean":"2024-03-30T15:08:32.359346+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:08:32.359346+0000","last_undegraded":"2024-03-30T15:08:32.359346+0000","last_fullsized":"2024-03-30T15:08:32.359346+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.943712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.943712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:33.196867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1252,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369522+0000","last_change":"2024-03-30T15:07:34.554038+0000","last_active":"2024-03-30T15:08:33.369522+0000","last_peered":"2024-03-30T15:08:33.369522+0000","last_clean":"2024-03-30T15:08:33.369522+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:08:33.369522+0000","last_undegraded":"2024-03-30T15:08:33.369522+0000","last_fullsized":"2024-03-30T15:08:33.369522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.654265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1238,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569007+0000","last_change":"2024-03-30T15:06:57.279563+0000","last_active":"2024-03-30T15:08:33.569007+0000","last_peered":"2024-03-30T15:08:33.569007+0000","last_clean":"2024-03-30T15:08:33.569007+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:08:33.569007+0000","last_undegraded":"2024-03-30T15:08:33.569007+0000","last_fullsized":"2024-03-30T15:08:33.569007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.279464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.279464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:38.566878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":323,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528022+0000","last_change":"2024-03-30T15:07:18.427089+0000","last_active":"2024-03-30T15:08:32.528022+0000","last_peered":"2024-03-30T15:08:32.528022+0000","last_clean":"2024-03-30T15:08:32.528022+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:08:32.528022+0000","last_undegraded":"2024-03-30T15:08:32.528022+0000","last_fullsized":"2024-03-30T15:08:32.528022+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1245,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811555+0000","last_change":"2024-03-30T15:08:01.209737+0000","last_active":"2024-03-30T15:08:33.811555+0000","last_peered":"2024-03-30T15:08:33.811555+0000","last_clean":"2024-03-30T15:08:33.811555+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:08:33.811555+0000","last_undegraded":"2024-03-30T15:08:33.811555+0000","last_fullsized":"2024-03-30T15:08:33.811555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:01.209658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.209658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.903658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.368391+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:08:33.368391+0000","last_peered":"2024-03-30T15:08:33.368391+0000","last_clean":"2024-03-30T15:08:33.368391+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:08:33.368391+0000","last_undegraded":"2024-03-30T15:08:33.368391+0000","last_fullsized":"2024-03-30T15:08:33.368391+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860105+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1247,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529745+0000","last_change":"2024-03-30T15:06:39.943308+0000","last_active":"2024-03-30T15:08:32.529745+0000","last_peered":"2024-03-30T15:08:32.529745+0000","last_clean":"2024-03-30T15:08:32.529745+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:08:32.529745+0000","last_undegraded":"2024-03-30T15:08:32.529745+0000","last_fullsized":"2024-03-30T15:08:32.529745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:39.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.943172+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1231,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.365231+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:08:33.365231+0000","last_peered":"2024-03-30T15:08:33.365231+0000","last_clean":"2024-03-30T15:08:33.365231+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:08:33.365231+0000","last_undegraded":"2024-03-30T15:08:33.365231+0000","last_fullsized":"2024-03-30T15:08:33.365231+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1284,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569497+0000","last_change":"2024-03-30T15:06:11.222500+0000","last_active":"2024-03-30T15:08:33.569497+0000","last_peered":"2024-03-30T15:08:33.569497+0000","last_clean":"2024-03-30T15:08:33.569497+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:08:33.569497+0000","last_undegraded":"2024-03-30T15:08:33.569497+0000","last_fullsized":"2024-03-30T15:08:33.569497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.222382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.222382+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1214,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359596+0000","last_change":"2024-03-30T15:06:58.575629+0000","last_active":"2024-03-30T15:08:32.359596+0000","last_peered":"2024-03-30T15:08:32.359596+0000","last_clean":"2024-03-30T15:08:32.359596+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:08:32.359596+0000","last_undegraded":"2024-03-30T15:08:32.359596+0000","last_fullsized":"2024-03-30T15:08:32.359596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.575544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.575544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367728+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:08:33.367728+0000","last_peered":"2024-03-30T15:08:33.367728+0000","last_clean":"2024-03-30T15:08:33.367728+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:08:33.367728+0000","last_undegraded":"2024-03-30T15:08:33.367728+0000","last_fullsized":"2024-03-30T15:08:33.367728+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":1187,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359459+0000","last_change":"2024-03-30T15:06:59.844000+0000","last_active":"2024-03-30T15:08:32.359459+0000","last_peered":"2024-03-30T15:08:32.359459+0000","last_clean":"2024-03-30T15:08:32.359459+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:08:32.359459+0000","last_undegraded":"2024-03-30T15:08:32.359459+0000","last_fullsized":"2024-03-30T15:08:32.359459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:06:59.843847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.843847+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1238,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568978+0000","last_change":"2024-03-30T15:08:17.645966+0000","last_active":"2024-03-30T15:08:33.568978+0000","last_peered":"2024-03-30T15:08:33.568978+0000","last_clean":"2024-03-30T15:08:33.568978+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:08:33.568978+0000","last_undegraded":"2024-03-30T15:08:33.568978+0000","last_fullsized":"2024-03-30T15:08:33.568978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.645871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.645871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.597570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1397,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529525+0000","last_change":"2024-03-30T15:06:44.020947+0000","last_active":"2024-03-30T15:08:32.529525+0000","last_peered":"2024-03-30T15:08:32.529525+0000","last_clean":"2024-03-30T15:08:32.529525+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:08:32.529525+0000","last_undegraded":"2024-03-30T15:08:32.529525+0000","last_fullsized":"2024-03-30T15:08:32.529525+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:44.020829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.020829+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1221,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369234+0000","last_change":"2024-03-30T15:07:30.551177+0000","last_active":"2024-03-30T15:08:33.369234+0000","last_peered":"2024-03-30T15:08:33.369234+0000","last_clean":"2024-03-30T15:08:33.369234+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:08:33.369234+0000","last_undegraded":"2024-03-30T15:08:33.369234+0000","last_fullsized":"2024-03-30T15:08:33.369234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:07:30.551096+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.551096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1131,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358719+0000","last_change":"2024-03-30T15:07:21.866188+0000","last_active":"2024-03-30T15:08:32.358719+0000","last_peered":"2024-03-30T15:08:32.358719+0000","last_clean":"2024-03-30T15:08:32.358719+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:08:32.358719+0000","last_undegraded":"2024-03-30T15:08:32.358719+0000","last_fullsized":"2024-03-30T15:08:32.358719+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.866106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.866106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.392301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1258,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358823+0000","last_change":"2024-03-30T15:06:54.562492+0000","last_active":"2024-03-30T15:08:32.358823+0000","last_peered":"2024-03-30T15:08:32.358823+0000","last_clean":"2024-03-30T15:08:32.358823+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:08:32.358823+0000","last_undegraded":"2024-03-30T15:08:32.358823+0000","last_fullsized":"2024-03-30T15:08:32.358823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:06:54.562347+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.562347+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358984+0000","last_change":"2024-03-30T15:07:00.555778+0000","last_active":"2024-03-30T15:08:32.358984+0000","last_peered":"2024-03-30T15:08:32.358984+0000","last_clean":"2024-03-30T15:08:32.358984+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:08:32.358984+0000","last_undegraded":"2024-03-30T15:08:32.358984+0000","last_fullsized":"2024-03-30T15:08:32.358984+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.555700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.555700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357664+0000","last_change":"2024-03-30T15:07:35.184388+0000","last_active":"2024-03-30T15:08:32.357664+0000","last_peered":"2024-03-30T15:08:32.357664+0000","last_clean":"2024-03-30T15:08:32.357664+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:08:32.357664+0000","last_undegraded":"2024-03-30T15:08:32.357664+0000","last_fullsized":"2024-03-30T15:08:32.357664+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:41.166334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1132,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364373+0000","last_change":"2024-03-30T15:08:11.553825+0000","last_active":"2024-03-30T15:08:33.364373+0000","last_peered":"2024-03-30T15:08:33.364373+0000","last_clean":"2024-03-30T15:08:33.364373+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:08:33.364373+0000","last_undegraded":"2024-03-30T15:08:33.364373+0000","last_fullsized":"2024-03-30T15:08:33.364373+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.326045+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1131,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358461+0000","last_change":"2024-03-30T15:07:24.946870+0000","last_active":"2024-03-30T15:08:32.358461+0000","last_peered":"2024-03-30T15:08:32.358461+0000","last_clean":"2024-03-30T15:08:32.358461+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:08:32.358461+0000","last_undegraded":"2024-03-30T15:08:32.358461+0000","last_fullsized":"2024-03-30T15:08:32.358461+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.946762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.946762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.723932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1258,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358055+0000","last_change":"2024-03-30T15:07:20.851744+0000","last_active":"2024-03-30T15:08:32.358055+0000","last_peered":"2024-03-30T15:08:32.358055+0000","last_clean":"2024-03-30T15:08:32.358055+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:08:32.358055+0000","last_undegraded":"2024-03-30T15:08:32.358055+0000","last_fullsized":"2024-03-30T15:08:32.358055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:20.851659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.851659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1253,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569664+0000","last_change":"2024-03-30T15:07:33.297905+0000","last_active":"2024-03-30T15:08:33.569664+0000","last_peered":"2024-03-30T15:08:33.569664+0000","last_clean":"2024-03-30T15:08:33.569664+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:08:33.569664+0000","last_undegraded":"2024-03-30T15:08:33.569664+0000","last_fullsized":"2024-03-30T15:08:33.569664+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.297820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.464840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":393,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358127+0000","last_change":"2024-03-30T15:06:15.461799+0000","last_active":"2024-03-30T15:08:32.358127+0000","last_peered":"2024-03-30T15:08:32.358127+0000","last_clean":"2024-03-30T15:08:32.358127+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:08:32.358127+0000","last_undegraded":"2024-03-30T15:08:32.358127+0000","last_fullsized":"2024-03-30T15:08:32.358127+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:15.461699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:15.461699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1219,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364071+0000","last_change":"2024-03-30T15:06:53.604745+0000","last_active":"2024-03-30T15:08:33.364071+0000","last_peered":"2024-03-30T15:08:33.364071+0000","last_clean":"2024-03-30T15:08:33.364071+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:08:33.364071+0000","last_undegraded":"2024-03-30T15:08:33.364071+0000","last_fullsized":"2024-03-30T15:08:33.364071+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364008+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:08:33.364008+0000","last_peered":"2024-03-30T15:08:33.364008+0000","last_clean":"2024-03-30T15:08:33.364008+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:08:33.364008+0000","last_undegraded":"2024-03-30T15:08:33.364008+0000","last_fullsized":"2024-03-30T15:08:33.364008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1181,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569162+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:08:33.569162+0000","last_peered":"2024-03-30T15:08:33.569162+0000","last_clean":"2024-03-30T15:08:33.569162+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:08:33.569162+0000","last_undegraded":"2024-03-30T15:08:33.569162+0000","last_fullsized":"2024-03-30T15:08:33.569162+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529285+0000","last_change":"2024-03-30T15:06:00.028025+0000","last_active":"2024-03-30T15:08:32.529285+0000","last_peered":"2024-03-30T15:08:32.529285+0000","last_clean":"2024-03-30T15:08:32.529285+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:08:32.529285+0000","last_undegraded":"2024-03-30T15:08:32.529285+0000","last_fullsized":"2024-03-30T15:08:32.529285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:00.027914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:00.027914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1230,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568976+0000","last_change":"2024-03-30T15:08:07.697978+0000","last_active":"2024-03-30T15:08:33.568976+0000","last_peered":"2024-03-30T15:08:33.568976+0000","last_clean":"2024-03-30T15:08:33.568976+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:08:33.568976+0000","last_undegraded":"2024-03-30T15:08:33.568976+0000","last_fullsized":"2024-03-30T15:08:33.568976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:07.697832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:07.697832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.589928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1132,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.366361+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:08:33.366361+0000","last_peered":"2024-03-30T15:08:33.366361+0000","last_clean":"2024-03-30T15:08:33.366361+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:08:33.366361+0000","last_undegraded":"2024-03-30T15:08:33.366361+0000","last_fullsized":"2024-03-30T15:08:33.366361+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1143,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356697+0000","last_change":"2024-03-30T15:08:08.177938+0000","last_active":"2024-03-30T15:08:32.356697+0000","last_peered":"2024-03-30T15:08:32.356697+0000","last_clean":"2024-03-30T15:08:32.356697+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:08:32.356697+0000","last_undegraded":"2024-03-30T15:08:32.356697+0000","last_fullsized":"2024-03-30T15:08:32.356697+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.177858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.177858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.773779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":814,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364569+0000","last_change":"2024-03-30T15:07:22.646408+0000","last_active":"2024-03-30T15:08:33.364569+0000","last_peered":"2024-03-30T15:08:33.364569+0000","last_clean":"2024-03-30T15:08:33.364569+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:08:33.364569+0000","last_undegraded":"2024-03-30T15:08:33.364569+0000","last_fullsized":"2024-03-30T15:08:33.364569+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:07:22.646314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.646314+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.671690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1239,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.370197+0000","last_change":"2024-03-30T15:08:10.508709+0000","last_active":"2024-03-30T15:08:33.370197+0000","last_peered":"2024-03-30T15:08:33.370197+0000","last_clean":"2024-03-30T15:08:33.370197+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:08:33.370197+0000","last_undegraded":"2024-03-30T15:08:33.370197+0000","last_fullsized":"2024-03-30T15:08:33.370197+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.166959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.525804+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:08:32.525804+0000","last_peered":"2024-03-30T15:08:32.525804+0000","last_clean":"2024-03-30T15:08:32.525804+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:08:32.525804+0000","last_undegraded":"2024-03-30T15:08:32.525804+0000","last_fullsized":"2024-03-30T15:08:32.525804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1218,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568961+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:08:33.568961+0000","last_peered":"2024-03-30T15:08:33.568961+0000","last_clean":"2024-03-30T15:08:33.568961+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:08:33.568961+0000","last_undegraded":"2024-03-30T15:08:33.568961+0000","last_fullsized":"2024-03-30T15:08:33.568961+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1140,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528390+0000","last_change":"2024-03-30T15:07:33.284903+0000","last_active":"2024-03-30T15:08:32.528390+0000","last_peered":"2024-03-30T15:08:32.528390+0000","last_clean":"2024-03-30T15:08:32.528390+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:08:32.528390+0000","last_undegraded":"2024-03-30T15:08:32.528390+0000","last_fullsized":"2024-03-30T15:08:32.528390+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.284814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.284814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.915368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1275,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528431+0000","last_change":"2024-03-30T15:08:16.603568+0000","last_active":"2024-03-30T15:08:32.528431+0000","last_peered":"2024-03-30T15:08:32.528431+0000","last_clean":"2024-03-30T15:08:32.528431+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:08:32.528431+0000","last_undegraded":"2024-03-30T15:08:32.528431+0000","last_fullsized":"2024-03-30T15:08:32.528431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:16.603468+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:08:16.603468+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.969813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1133,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364350+0000","last_change":"2024-03-30T15:08:00.523833+0000","last_active":"2024-03-30T15:08:33.364350+0000","last_peered":"2024-03-30T15:08:33.364350+0000","last_clean":"2024-03-30T15:08:33.364350+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:08:33.364350+0000","last_undegraded":"2024-03-30T15:08:33.364350+0000","last_fullsized":"2024-03-30T15:08:33.364350+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.174749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1224,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369620+0000","last_change":"2024-03-30T15:07:28.601931+0000","last_active":"2024-03-30T15:08:33.369620+0000","last_peered":"2024-03-30T15:08:33.369620+0000","last_clean":"2024-03-30T15:08:33.369620+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:08:33.369620+0000","last_undegraded":"2024-03-30T15:08:33.369620+0000","last_fullsized":"2024-03-30T15:08:33.369620+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.601815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.601815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:50.989405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1233,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359982+0000","last_change":"2024-03-30T15:08:02.083375+0000","last_active":"2024-03-30T15:08:32.359982+0000","last_peered":"2024-03-30T15:08:32.359982+0000","last_clean":"2024-03-30T15:08:32.359982+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:08:32.359982+0000","last_undegraded":"2024-03-30T15:08:32.359982+0000","last_fullsized":"2024-03-30T15:08:32.359982+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.083294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.083294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.435219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1140,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527758+0000","last_change":"2024-03-30T15:06:14.085407+0000","last_active":"2024-03-30T15:08:32.527758+0000","last_peered":"2024-03-30T15:08:32.527758+0000","last_clean":"2024-03-30T15:08:32.527758+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:08:32.527758+0000","last_undegraded":"2024-03-30T15:08:32.527758+0000","last_fullsized":"2024-03-30T15:08:32.527758+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.085299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:35.867591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1263,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.369448+0000","last_change":"2024-03-30T15:07:25.577161+0000","last_active":"2024-03-30T15:08:33.369448+0000","last_peered":"2024-03-30T15:08:33.369448+0000","last_clean":"2024-03-30T15:08:33.369448+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:08:33.369448+0000","last_undegraded":"2024-03-30T15:08:33.369448+0000","last_fullsized":"2024-03-30T15:08:33.369448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:07:25.577074+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.577074+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.741789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1239,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356877+0000","last_change":"2024-03-30T15:07:30.031279+0000","last_active":"2024-03-30T15:08:32.356877+0000","last_peered":"2024-03-30T15:08:32.356877+0000","last_clean":"2024-03-30T15:08:32.356877+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:08:32.356877+0000","last_undegraded":"2024-03-30T15:08:32.356877+0000","last_fullsized":"2024-03-30T15:08:32.356877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.031154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.145354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1422,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528374+0000","last_change":"2024-03-30T15:08:04.561718+0000","last_active":"2024-03-30T15:08:32.528374+0000","last_peered":"2024-03-30T15:08:32.528374+0000","last_clean":"2024-03-30T15:08:32.528374+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:08:32.528374+0000","last_undegraded":"2024-03-30T15:08:32.528374+0000","last_fullsized":"2024-03-30T15:08:32.528374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:04.561610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.561610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:28.847750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1228,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.569489+0000","last_change":"2024-03-30T15:06:48.193765+0000","last_active":"2024-03-30T15:08:33.569489+0000","last_peered":"2024-03-30T15:08:33.569489+0000","last_clean":"2024-03-30T15:08:33.569489+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:08:33.569489+0000","last_undegraded":"2024-03-30T15:08:33.569489+0000","last_fullsized":"2024-03-30T15:08:33.569489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:06:48.193663+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.193663+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.541749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1148,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527162+0000","last_change":"2024-03-30T15:07:57.453748+0000","last_active":"2024-03-30T15:08:32.527162+0000","last_peered":"2024-03-30T15:08:32.527162+0000","last_clean":"2024-03-30T15:08:32.527162+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:08:32.527162+0000","last_undegraded":"2024-03-30T15:08:32.527162+0000","last_fullsized":"2024-03-30T15:08:32.527162+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.453629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.453629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.047448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1189,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359743+0000","last_change":"2024-03-30T15:07:21.894935+0000","last_active":"2024-03-30T15:08:32.359743+0000","last_peered":"2024-03-30T15:08:32.359743+0000","last_clean":"2024-03-30T15:08:32.359743+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:08:32.359743+0000","last_undegraded":"2024-03-30T15:08:32.359743+0000","last_fullsized":"2024-03-30T15:08:32.359743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:07:21.894855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.894855+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:37.454380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359526+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:08:32.359526+0000","last_peered":"2024-03-30T15:08:32.359526+0000","last_clean":"2024-03-30T15:08:32.359526+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:08:32.359526+0000","last_undegraded":"2024-03-30T15:08:32.359526+0000","last_fullsized":"2024-03-30T15:08:32.359526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1218,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359599+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:08:32.359599+0000","last_peered":"2024-03-30T15:08:32.359599+0000","last_clean":"2024-03-30T15:08:32.359599+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:08:32.359599+0000","last_undegraded":"2024-03-30T15:08:32.359599+0000","last_fullsized":"2024-03-30T15:08:32.359599+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1145,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528548+0000","last_change":"2024-03-30T15:07:36.255445+0000","last_active":"2024-03-30T15:08:32.528548+0000","last_peered":"2024-03-30T15:08:32.528548+0000","last_clean":"2024-03-30T15:08:32.528548+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:08:32.528548+0000","last_undegraded":"2024-03-30T15:08:32.528548+0000","last_fullsized":"2024-03-30T15:08:32.528548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.255369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.255369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:58.024102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1132,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529833+0000","last_change":"2024-03-30T15:07:28.134811+0000","last_active":"2024-03-30T15:08:32.529833+0000","last_peered":"2024-03-30T15:08:32.529833+0000","last_clean":"2024-03-30T15:08:32.529833+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:08:32.529833+0000","last_undegraded":"2024-03-30T15:08:32.529833+0000","last_fullsized":"2024-03-30T15:08:32.529833+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.134705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.134705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.521025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1303,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.811184+0000","last_change":"2024-03-30T15:07:59.295056+0000","last_active":"2024-03-30T15:08:33.811184+0000","last_peered":"2024-03-30T15:08:33.811184+0000","last_clean":"2024-03-30T15:08:33.811184+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:08:33.811184+0000","last_undegraded":"2024-03-30T15:08:33.811184+0000","last_fullsized":"2024-03-30T15:08:33.811184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:59.294962+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.294962+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.470863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1231,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.367244+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:08:33.367244+0000","last_peered":"2024-03-30T15:08:33.367244+0000","last_clean":"2024-03-30T15:08:33.367244+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:08:33.367244+0000","last_undegraded":"2024-03-30T15:08:33.367244+0000","last_fullsized":"2024-03-30T15:08:33.367244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359885+0000","last_change":"2024-03-30T15:08:00.131070+0000","last_active":"2024-03-30T15:08:32.359885+0000","last_peered":"2024-03-30T15:08:32.359885+0000","last_clean":"2024-03-30T15:08:32.359885+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:08:32.359885+0000","last_undegraded":"2024-03-30T15:08:32.359885+0000","last_fullsized":"2024-03-30T15:08:32.359885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.130991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.130991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.398245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1215,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359800+0000","last_change":"2024-03-30T15:06:12.558953+0000","last_active":"2024-03-30T15:08:32.359800+0000","last_peered":"2024-03-30T15:08:32.359800+0000","last_clean":"2024-03-30T15:08:32.359800+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:08:32.359800+0000","last_undegraded":"2024-03-30T15:08:32.359800+0000","last_fullsized":"2024-03-30T15:08:32.359800+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.558834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.558834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1143,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358951+0000","last_change":"2024-03-30T15:08:21.207838+0000","last_active":"2024-03-30T15:08:32.358951+0000","last_peered":"2024-03-30T15:08:32.358951+0000","last_clean":"2024-03-30T15:08:32.358951+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:08:32.358951+0000","last_undegraded":"2024-03-30T15:08:32.358951+0000","last_fullsized":"2024-03-30T15:08:32.358951+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.207705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:29.807245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1259,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359086+0000","last_change":"2024-03-30T15:06:50.584525+0000","last_active":"2024-03-30T15:08:32.359086+0000","last_peered":"2024-03-30T15:08:32.359086+0000","last_clean":"2024-03-30T15:08:32.359086+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:08:32.359086+0000","last_undegraded":"2024-03-30T15:08:32.359086+0000","last_fullsized":"2024-03-30T15:08:32.359086+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:06:50.584362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.584362+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529645+0000","last_change":"2024-03-30T15:06:14.990566+0000","last_active":"2024-03-30T15:08:32.529645+0000","last_peered":"2024-03-30T15:08:32.529645+0000","last_clean":"2024-03-30T15:08:32.529645+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:08:32.529645+0000","last_undegraded":"2024-03-30T15:08:32.529645+0000","last_fullsized":"2024-03-30T15:08:32.529645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.990472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.990472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1230,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.568908+0000","last_change":"2024-03-30T15:08:18.689134+0000","last_active":"2024-03-30T15:08:33.568908+0000","last_peered":"2024-03-30T15:08:33.568908+0000","last_clean":"2024-03-30T15:08:33.568908+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:08:33.568908+0000","last_undegraded":"2024-03-30T15:08:33.568908+0000","last_fullsized":"2024-03-30T15:08:33.568908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:18.688995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:18.688995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:47.478630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1222,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526765+0000","last_change":"2024-03-30T15:07:23.180715+0000","last_active":"2024-03-30T15:08:32.526765+0000","last_peered":"2024-03-30T15:08:32.526765+0000","last_clean":"2024-03-30T15:08:32.526765+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:08:32.526765+0000","last_undegraded":"2024-03-30T15:08:32.526765+0000","last_fullsized":"2024-03-30T15:08:32.526765+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.180625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.180625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1129,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356725+0000","last_change":"2024-03-30T15:06:14.104026+0000","last_active":"2024-03-30T15:08:32.356725+0000","last_peered":"2024-03-30T15:08:32.356725+0000","last_clean":"2024-03-30T15:08:32.356725+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:08:32.356725+0000","last_undegraded":"2024-03-30T15:08:32.356725+0000","last_fullsized":"2024-03-30T15:08:32.356725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.103910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.103910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1247,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.359480+0000","last_change":"2024-03-30T15:06:42.502254+0000","last_active":"2024-03-30T15:08:32.359480+0000","last_peered":"2024-03-30T15:08:32.359480+0000","last_clean":"2024-03-30T15:08:32.359480+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:08:32.359480+0000","last_undegraded":"2024-03-30T15:08:32.359480+0000","last_fullsized":"2024-03-30T15:08:32.359480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:42.502138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.502138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.529409+0000","last_change":"2024-03-30T15:06:43.006252+0000","last_active":"2024-03-30T15:08:32.529409+0000","last_peered":"2024-03-30T15:08:32.529409+0000","last_clean":"2024-03-30T15:08:32.529409+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:08:32.529409+0000","last_undegraded":"2024-03-30T15:08:32.529409+0000","last_fullsized":"2024-03-30T15:08:32.529409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.006124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.006124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.528752+0000","last_change":"2024-03-30T15:08:20.313597+0000","last_active":"2024-03-30T15:08:32.528752+0000","last_peered":"2024-03-30T15:08:32.528752+0000","last_clean":"2024-03-30T15:08:32.528752+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:08:32.528752+0000","last_undegraded":"2024-03-30T15:08:32.528752+0000","last_fullsized":"2024-03-30T15:08:32.528752+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_clean_scrub_stamp":"2024-03-30T15:08:20.313498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.565879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1219,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364437+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:08:33.364437+0000","last_peered":"2024-03-30T15:08:33.364437+0000","last_clean":"2024-03-30T15:08:33.364437+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:08:33.364437+0000","last_undegraded":"2024-03-30T15:08:33.364437+0000","last_fullsized":"2024-03-30T15:08:33.364437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1132,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.526317+0000","last_change":"2024-03-30T15:07:33.165675+0000","last_active":"2024-03-30T15:08:32.526317+0000","last_peered":"2024-03-30T15:08:32.526317+0000","last_clean":"2024-03-30T15:08:32.526317+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:08:32.526317+0000","last_undegraded":"2024-03-30T15:08:32.526317+0000","last_fullsized":"2024-03-30T15:08:32.526317+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.165599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.165599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:39.109467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1253,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.810800+0000","last_change":"2024-03-30T15:07:29.102384+0000","last_active":"2024-03-30T15:08:33.810800+0000","last_peered":"2024-03-30T15:08:33.810800+0000","last_clean":"2024-03-30T15:08:33.810800+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:08:33.810800+0000","last_undegraded":"2024-03-30T15:08:33.810800+0000","last_fullsized":"2024-03-30T15:08:33.810800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:29.102268+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.102268+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.919896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1146,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.527623+0000","last_change":"2024-03-30T15:08:17.607910+0000","last_active":"2024-03-30T15:08:32.527623+0000","last_peered":"2024-03-30T15:08:32.527623+0000","last_clean":"2024-03-30T15:08:32.527623+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:08:32.527623+0000","last_undegraded":"2024-03-30T15:08:32.527623+0000","last_fullsized":"2024-03-30T15:08:32.527623+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.607819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.607819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:44.662694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1221,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357735+0000","last_change":"2024-03-30T15:06:40.495863+0000","last_active":"2024-03-30T15:08:32.357735+0000","last_peered":"2024-03-30T15:08:32.357735+0000","last_clean":"2024-03-30T15:08:32.357735+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:08:32.357735+0000","last_undegraded":"2024-03-30T15:08:32.357735+0000","last_fullsized":"2024-03-30T15:08:32.357735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:40.495702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.495702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":802,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.366322+0000","last_change":"2024-03-30T15:07:26.673461+0000","last_active":"2024-03-30T15:08:33.366322+0000","last_peered":"2024-03-30T15:08:33.366322+0000","last_clean":"2024-03-30T15:08:33.366322+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:08:33.366322+0000","last_undegraded":"2024-03-30T15:08:33.366322+0000","last_fullsized":"2024-03-30T15:08:33.366322+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.673362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.673362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1129,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.364921+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:08:33.364921+0000","last_peered":"2024-03-30T15:08:33.364921+0000","last_clean":"2024-03-30T15:08:33.364921+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:08:33.364921+0000","last_undegraded":"2024-03-30T15:08:33.364921+0000","last_fullsized":"2024-03-30T15:08:33.364921+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:34.860185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1271,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.357885+0000","last_change":"2024-03-30T15:07:30.921961+0000","last_active":"2024-03-30T15:08:32.357885+0000","last_peered":"2024-03-30T15:08:32.357885+0000","last_clean":"2024-03-30T15:08:32.357885+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:08:32.357885+0000","last_undegraded":"2024-03-30T15:08:32.357885+0000","last_fullsized":"2024-03-30T15:08:32.357885+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:30.921763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.921763+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1240,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T15:08:33.370058+0000","last_change":"2024-03-30T15:07:35.571081+0000","last_active":"2024-03-30T15:08:33.370058+0000","last_peered":"2024-03-30T15:08:33.370058+0000","last_clean":"2024-03-30T15:08:33.370058+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:08:33.370058+0000","last_undegraded":"2024-03-30T15:08:33.370058+0000","last_fullsized":"2024-03-30T15:08:33.370058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.970991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1229,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.356620+0000","last_change":"2024-03-30T15:07:32.100519+0000","last_active":"2024-03-30T15:08:32.356620+0000","last_peered":"2024-03-30T15:08:32.356620+0000","last_clean":"2024-03-30T15:08:32.356620+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:08:32.356620+0000","last_undegraded":"2024-03-30T15:08:32.356620+0000","last_fullsized":"2024-03-30T15:08:32.356620+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.100436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.100436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.419009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1214,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T15:08:32.358887+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:08:32.358887+0000","last_peered":"2024-03-30T15:08:32.358887+0000","last_clean":"2024-03-30T15:08:32.358887+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:08:32.358887+0000","last_undegraded":"2024-03-30T15:08:32.358887+0000","last_fullsized":"2024-03-30T15:08:32.358887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":325,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":828,"num_read_kb":823,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":322,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005228,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76600,"kb_used_data":5164,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668552,"statfs":{"total":95995035648,"available":95916597248,"internally_reserved":0,"allocated":5287936,"data_stored":2534456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299569,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76256,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668896,"statfs":{"total":95995035648,"available":95916949504,"internally_reserved":0,"allocated":4931584,"data_stored":2208534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72148,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673004,"statfs":{"total":95995035648,"available":95921156096,"internally_reserved":0,"allocated":4984832,"data_stored":2226120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54064,"kb_used_data":4836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93691088,"statfs":{"total":95995035648,"available":95939674112,"internally_reserved":0,"allocated":4952064,"data_stored":2202019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411525,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94568,"kb_used_data":5084,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650584,"statfs":{"total":95995035648,"available":95898198016,"internally_reserved":0,"allocated":5206016,"data_stored":2479744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117185,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94980,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650172,"statfs":{"total":95995035648,"available":95897776128,"internally_reserved":0,"allocated":5619712,"data_stored":2839769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953659,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94332,"kb_used_data":4848,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650820,"statfs":{"total":95995035648,"available":95898439680,"internally_reserved":0,"allocated":4964352,"data_stored":2213628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659318,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76848,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668304,"statfs":{"total":95995035648,"available":95916343296,"internally_reserved":0,"allocated":5541888,"data_stored":2799581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:08:37.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:37.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:37.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:37.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[41614]: from='client.58583 v1:172.21.15.28:0/3138483631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[29876]: from='client.58583 v1:172.21.15.28:0/3138483631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:37.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:37.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:37.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:37 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:37 smithi134 ceph-mon[33714]: from='client.58583 v1:172.21.15.28:0/3138483631' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:37.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:37 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:37.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:37.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:37.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:37.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:37.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:38.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:38.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:38.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:38.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:38.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:38.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:38.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:38.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:38 smithi028 ceph-mon[29876]: pgmap v982: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:08:38.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:38 smithi028 ceph-mon[41614]: pgmap v982: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:08:38.509 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:38.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:38.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:38.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:38 smithi134 ceph-mon[33714]: pgmap v982: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-30T15:08:38.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:38.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:38.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:38.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:38.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:39.016 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:39.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:39.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:39.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:39.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:39.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:39.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:39.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:39.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:39.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:39.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:39.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:39.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:39.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:40.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:40.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:40.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:40.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:40.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:40.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:08:40.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:40.451 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:08:40.451 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:08:40.451 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.57050.0:2 190.6 190:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 573.23947275499995, 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075417256000000002, 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.57050", 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/1148433196", 2024-03-30T15:08:40.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209699+0000", 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209701+0000", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3800000000000001e-06 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209707+0000", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9460000000000003e-06 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209718+0000", 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0679000000000001e-05 2024-03-30T15:08:40.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.209729+0000", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0896999999999999e-05 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213086+0000", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033575290000000002 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.213092+0000", 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6489999999999996e-06 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220438+0000", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073462149999999997 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.220446+0000", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2770000000000005e-06 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272811+0000", 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052364490999999999 2024-03-30T15:08:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.272945+0000", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013451899999999999 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273074+0000", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012904 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.273476+0000", 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000401875 2024-03-30T15:08:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.274847+0000", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013706339999999999 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.284902+0000", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010055015 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285053+0000", 2024-03-30T15:08:40.458 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015113999999999999 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T14:59:07.285116+0000", 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2970000000000002e-05 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:5374 285.4 285:20000000:test-rados-api-smithi028-93253-50:::head [pgnls start_epoch 389] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e389)", 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 367.063160004, 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11848968999999999, 2024-03-30T15:08:40.459 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 5374 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.460 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386011+0000", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386012+0000", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0109999999999999e-06 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386018+0000", 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2329999999999997e-06 2024-03-30T15:08:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386026+0000", 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5250000000000005e-06 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386031+0000", 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2859999999999999e-06 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.462 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386087+0000", 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5448999999999997e-05 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.386091+0000", 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.296e-06 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504352+0000", 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.118261214 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.463 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504392+0000", 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9728000000000003e-05 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:02:33.504501+0000", 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108948 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(6.8,QREQUEST,e:641,from:7,reservation_nonce:10)", 2024-03-30T15:08:40.464 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:11.697119+0000", 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 28.752052365000001, 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047981, 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.465 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697119+0000", 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697119+0000", 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697120+0000", 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.423e-06 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697126+0000", 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3789999999999997e-06 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697134+0000", 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3129999999999999e-06 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.467 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697143+0000", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1339999999999993e-06 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697535+0000", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00039204399999999998 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697553+0000", 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7450000000000001e-05 2024-03-30T15:08:40.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:11.697599+0000", 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6066999999999998e-05 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(5.6,GRANT,e:644,from:7,reservation_nonce:0)", 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:14.238984+0000", 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.210187190999999, 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021760799999999999, 2024-03-30T15:08:40.469 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.238984+0000", 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.238984+0000", 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.238986+0000", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5039999999999999e-06 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.238989+0000", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2490000000000001e-06 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.238997+0000", 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8669999999999994e-06 2024-03-30T15:08:40.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.239003+0000", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9630000000000003e-06 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.239044+0000", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1437000000000003e-05 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.239054+0000", 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5510000000000005e-06 2024-03-30T15:08:40.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:14.239202+0000", 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014803700000000001 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:8 323.10 323:08000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.382535+0000", 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.066636550999998, 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000313491, 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 8 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382535+0000", 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.474 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382535+0000", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382537+0000", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4849999999999999e-06 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382545+0000", 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4989999999999997e-06 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.475 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382558+0000", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2830000000000001e-05 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382569+0000", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1343e-05 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382636+0000", 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6643000000000006e-05 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.476 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382675+0000", 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9456999999999998e-05 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.382848+0000", 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173234 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:17 323.a 323:50000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:08:40.477 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.390314+0000", 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.058857092, 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033356299999999999, 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 17 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.478 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390314+0000", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390314+0000", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390318+0000", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3230000000000002e-06 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.479 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390327+0000", 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5200000000000003e-06 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390344+0000", 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7221999999999999e-05 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390355+0000", 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0166999999999999e-05 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.480 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390428+0000", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3769999999999993e-05 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390469+0000", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0528000000000002e-05 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.390648+0000", 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017903299999999999 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.481 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:22 323.1e 323:78000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.392179+0000", 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.056992878999999, 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030084300000000002, 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 22 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392179+0000", 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392179+0000", 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.483 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392180+0000", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9259999999999999e-06 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392192+0000", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1493e-05 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392204+0000", 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2522e-05 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392212+0000", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2450000000000001e-06 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392280+0000", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8034000000000006e-05 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392320+0000", 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9956000000000002e-05 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.392479+0000", 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000159667 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:37 323.f 323:f0000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.402048+0000", 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 23.047123224, 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000327001, 2024-03-30T15:08:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402048+0000", 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402048+0000", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402051+0000", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4420000000000001e-06 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402060+0000", 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0000000000000002e-06 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402076+0000", 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5967000000000001e-05 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402083+0000", 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7500000000000003e-06 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.489 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402139+0000", 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5637999999999998e-05 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402163+0000", 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3736000000000001e-05 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.402375+0000", 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000212468 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(3.1e,QREQUEST,e:648,from:6,reservation_nonce:10)", 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.605729+0000", 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.843442556999999, 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035920199999999998, 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605729+0000", 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605729+0000", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605733+0000", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9299999999999996e-06 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605745+0000", 2024-03-30T15:08:40.492 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1960000000000001e-05 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605770+0000", 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5401999999999998e-05 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605786+0000", 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6033e-05 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605951+0000", 2024-03-30T15:08:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:40.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:40.495 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000164709 2024-03-30T15:08:40.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.605999+0000", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8389e-05 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.606088+0000", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8778999999999998e-05 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.34511.0:29 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e576)", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.766317+0000", 2024-03-30T15:08:40.496 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.682854035999998, 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00082124800000000001, 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.34511", 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3837265254", 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 29 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.497 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766317+0000", 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766317+0000", 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766321+0000", 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.253e-06 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766333+0000", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2816000000000001e-05 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766351+0000", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7203000000000001e-05 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.766364+0000", 2024-03-30T15:08:40.499 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2927e-05 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.767060+0000", 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00069630699999999996 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.767139+0000", 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8741999999999999e-05 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:42 323.10 323:08000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:19.372913+0000", 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.076258257999999, 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027134799999999999, 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 42 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.501 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372913+0000", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372913+0000", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372917+0000", 2024-03-30T15:08:40.502 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6890000000000002e-06 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372931+0000", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4304000000000001e-05 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372958+0000", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6568999999999999e-05 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372972+0000", 2024-03-30T15:08:40.503 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3961999999999999e-05 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.373086+0000", 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114827 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.373097+0000", 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0874e-05 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.504 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.375390+0000", 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022928979999999998 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.375428+0000", 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7465999999999997e-05 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.375627+0000", 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000198891 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:51 323.a 323:50000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:19.390217+0000", 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.058954689, 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00022356000000000001, 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51 2024-03-30T15:08:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390217+0000", 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390217+0000", 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.507 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390220+0000", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8310000000000001e-06 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390229+0000", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2280000000000004e-06 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390245+0000", 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6701000000000002e-05 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.508 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390254+0000", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2290000000000003e-06 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390308+0000", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4030000000000003e-05 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390329+0000", 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1684000000000001e-05 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.390440+0000", 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110857 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:71 323.f 323:f0000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:19.400807+0000", 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.048364293999999, 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032462000000000002, 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 71 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400807+0000", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400807+0000", 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400810+0000", 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4650000000000001e-06 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400818+0000", 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1359999999999997e-06 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400834+0000", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5974e-05 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400849+0000", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4885000000000001e-05 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400902+0000", 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3292999999999999e-05 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.400926+0000", 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4173999999999999e-05 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.401132+0000", 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020569300000000001 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:103 323.f 323:f0000000:test-rados-api-smithi028-93253-113:::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:19.411352+0000", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 21.037819732999999, 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023595199999999999, 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 103 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411352+0000", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411352+0000", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411353+0000", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3349999999999999e-06 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411359+0000", 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5219999999999998e-06 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411369+0000", 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0462000000000001e-05 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411375+0000", 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8869999999999997e-06 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.517 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411419+0000", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4478999999999999e-05 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411440+0000", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0199000000000001e-05 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.411588+0000", 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014806800000000001 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.518 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.7.0:1051 325.e e665/660 325:70ec8fab:test-rados-api-smithi028-93253-117::foo:head v 665'1, mlcod=0'0)", 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:34.373751+0000", 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 6.0754206630000001, 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00116442, 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373751+0000", 2024-03-30T15:08:40.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373751+0000", 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373752+0000", 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1179999999999999e-06 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:08:40.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373765+0000", 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2865e-05 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373773+0000", 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3450000000000006e-06 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373782+0000", 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8629999999999997e-06 2024-03-30T15:08:40.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373870+0000", 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8534000000000003e-05 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.373891+0000", 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0511999999999999e-05 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:08:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.374869+0000", 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0009783420000000001 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:34.374915+0000", 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5840999999999997e-05 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:08:40.523 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:08:40.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:40.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:40.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:40.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:40.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:40.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:40.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:41.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:41.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:41.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:41.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:41.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:41.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:41.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:41 smithi134 ceph-mon[33714]: pgmap v983: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 928 B/s wr, 2 op/s 2024-03-30T15:08:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3734184632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:41 smithi134 ceph-mon[33714]: from='client.58589 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[41614]: pgmap v983: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 928 B/s wr, 2 op/s 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3734184632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[41614]: from='client.58589 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[29876]: pgmap v983: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 928 B/s wr, 2 op/s 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3734184632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:41 smithi028 ceph-mon[29876]: from='client.58589 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:08:41.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:41.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:41.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:41.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:08:41.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:41.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:41.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:41.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:41.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:42.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:42.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.649+0000 7fdce449f700 1 -- 172.21.15.28:0/970132382 >> v1:172.21.15.28:6790/0 conn(0x7fdcdc109dd0 legacy=0x7fdcdc10c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- 172.21.15.28:0/970132382 shutdown_connections 2024-03-30T15:08:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- 172.21.15.28:0/970132382 >> 172.21.15.28:0/970132382 conn(0x7fdcdc0fc9f0 msgr2=0x7fdcdc0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- 172.21.15.28:0/970132382 shutdown_connections 2024-03-30T15:08:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- 172.21.15.28:0/970132382 wait complete. 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 librados: starting msgr at 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 librados: starting objecter 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 Processor -- start 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- start start 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 librados: setting wanted keys 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 librados: calling monclient init 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcdc06b370 con 0x7fdcdc10d970 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcdc1a8550 con 0x7fdcdc109dd0 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce449f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdcdc1a9730 con 0x7fdcdc101140 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce2a3c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdcdc10d970 0x7fdcdc1a5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46498/0 (socket says 172.21.15.28:46498) 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce223b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdcdc101140 0x7fdcdc1056b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57674/0 (socket says 172.21.15.28:57674) 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.650+0000 7fdce2a3c700 1 -- 172.21.15.28:0/3734184632 learned_addr learned my addr 172.21.15.28:0/3734184632 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter ms_handle_connect 0x7fdcdc101140 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter ms_handle_connect 0x7fdcdc10d970 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434968070 0 0) 0x7fdcdc1a9730 con 0x7fdcdc101140 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcc40031f0 con 0x7fdcdc101140 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter ms_handle_connect 0x7fdcdc109dd0 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052626877 0 0) 0x7fdcdc06b370 con 0x7fdcdc10d970 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcdc1a9730 con 0x7fdcdc10d970 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525136912 0 0) 0x7fdcdc1a8550 con 0x7fdcdc109dd0 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcdc06b370 con 0x7fdcdc109dd0 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3260224616 0 0) 0x7fdcc40031f0 con 0x7fdcdc101140 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcdc1a8550 con 0x7fdcdc101140 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdccc001c30 con 0x7fdcdc101140 2024-03-30T15:08:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1772620361 0 0) 0x7fdcdc1a9730 con 0x7fdcdc10d970 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcc40031f0 con 0x7fdcdc10d970 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdcd4001990 con 0x7fdcdc10d970 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3534990100 0 0) 0x7fdcdc06b370 con 0x7fdcdc109dd0 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcdc1a9730 con 0x7fdcdc109dd0 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdcd0003820 con 0x7fdcdc109dd0 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2462782980 0 0) 0x7fdcdc1a8550 con 0x7fdcdc101140 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 >> v1:172.21.15.134:6789/0 conn(0x7fdcdc109dd0 legacy=0x7fdcdc105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 >> v1:172.21.15.28:6789/0 conn(0x7fdcdc10d970 legacy=0x7fdcdc1a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc1aa910 con 0x7fdcdc101140 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc1a75a0 con 0x7fdcdc101140 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdce449f700 10 client.58589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdcdc1a7a60 con 0x7fdcdc101140 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdce449f700 1 librados: init done 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdccc003ec0 con 0x7fdcdc101140 2024-03-30T15:08:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.651+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdccc006090 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.652+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc8005120 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdccc003a20 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(665..665 src has 442..665) ==== 9110+0+0 (unknown 1208088358 0 0) 0x7fdccc089da0 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 10 client.58589.objecter ms_dispatch 0x7fdcdc001550 osd_map(665..665 src has 442..665) 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 3 client.58589.objecter handle_osd_map got epochs [665,665] > 0 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 3 client.58589.objecter handle_osd_map decoding full epoch 665 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 20 client.58589.objecter dump_active .. 0 homeless 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.653+0000 7fdcdb7fe700 10 client.58589.objecter ms_handle_connect 0x7fdcc40703b0 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.658+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdccc056c00 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:40.878+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fdcc8005410 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.130+0000 7fdcdb7fe700 1 -- 172.21.15.28:0/3734184632 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v666) ==== 91+0+0 (unknown 2334993372 0 0) 0x7fdccc05a650 con 0x7fdcdc101140 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 10 librados: watch_flush enter 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 10 librados: watch_flush exit 2024-03-30T15:08:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 20 client.58589.objecter shutdown clearing up homeless session... 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 10 client.58589.objecter successfully canceled tick 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdcc40703b0 legacy=0x7fdcc4072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 >> v1:172.21.15.28:6790/0 conn(0x7fdcdc101140 legacy=0x7fdcdc1056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 shutdown_connections 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 >> 172.21.15.28:0/3734184632 conn(0x7fdcdc0fc9f0 msgr2=0x7fdcdc110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 shutdown_connections 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 -- 172.21.15.28:0/3734184632 wait complete. 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:41.134+0000 7fdce449f700 1 librados: shutdown 2024-03-30T15:08:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:08:42.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:42.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:42.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:42.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[41614]: from='client.58589 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[41614]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[29876]: from='client.58589 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:08:42.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:42 smithi028 ceph-mon[29876]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T15:08:42.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:42 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:42 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:08:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:42 smithi134 ceph-mon[33714]: from='client.58589 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:08:42.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:42 smithi134 ceph-mon[33714]: osdmap e666: 8 total, 8 up, 8 in 2024-03-30T15:08:42.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:42.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:42.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:42.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:42.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:42.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:42.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:42.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:42.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:43.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:43.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:43.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:43.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:43.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:43.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: pgmap v985: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: 42.1c scrub starts 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: 42.1c scrub ok 2024-03-30T15:08:43.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[41614]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: pgmap v985: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: 42.1c scrub starts 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: 42.1c scrub ok 2024-03-30T15:08:43.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:43 smithi028 ceph-mon[29876]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T15:08:43.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:08:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:08:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: pgmap v985: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-03-30T15:08:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: 42.1c scrub starts 2024-03-30T15:08:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: 42.1c scrub ok 2024-03-30T15:08:43.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:43 smithi134 ceph-mon[33714]: osdmap e667: 8 total, 8 up, 8 in 2024-03-30T15:08:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:43.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:43.561 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:43.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:43.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:43.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.821 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:43.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:43.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:43.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:43.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:44.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:44.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:44.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:44.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:44.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:08:44.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:08:44.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:44 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:08:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T15:08:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T15:08:44.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:08:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:08:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:08:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:08:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:08:44.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:44 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:08:44.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:44.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:44.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:44.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:44.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:44.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:44.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:44.848 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:44.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:08:44.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:44.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.969+0000 7f696b1bd700 1 -- 172.21.15.28:0/433020406 >> v1:172.21.15.28:6790/0 conn(0x7f696410eab0 legacy=0x7f6964110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 -- 172.21.15.28:0/433020406 shutdown_connections 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 -- 172.21.15.28:0/433020406 >> 172.21.15.28:0/433020406 conn(0x7f69640fc9e0 msgr2=0x7f69640fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 -- 172.21.15.28:0/433020406 shutdown_connections 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 -- 172.21.15.28:0/433020406 wait complete. 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 librados: starting msgr at 2024-03-30T15:08:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 librados: starting objecter 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 Processor -- start 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 -- start start 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 librados: setting wanted keys 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.970+0000 7f696b1bd700 1 librados: calling monclient init 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f696b1bd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f696419f1d0 con 0x7f696410afa0 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f696b1bd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69641aa480 con 0x7f6964103190 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f696b1bd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f69641ab6c0 con 0x7f696410eab0 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6963fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f696410afa0 0x7f69641a45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46512/0 (socket says 172.21.15.28:46512) 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6963fff700 1 -- 172.21.15.28:0/1223603784 learned_addr learned my addr 172.21.15.28:0/1223603784 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter ms_handle_connect 0x7f696410afa0 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter ms_handle_connect 0x7f6964103190 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661234331 0 0) 0x7f696419f1d0 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954003aa0 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter ms_handle_connect 0x7f696410eab0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2707874735 0 0) 0x7f69641aa480 con 0x7f6964103190 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696419f1d0 con 0x7f6964103190 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4184162191 0 0) 0x7f6954003aa0 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69641aa480 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6950001a30 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581130099 0 0) 0x7f69641ab6c0 con 0x7f696410eab0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954003aa0 con 0x7f696410eab0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2917135177 0 0) 0x7f69641aa480 con 0x7f696410afa0 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.971+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6790/0 conn(0x7f696410eab0 legacy=0x7f69641a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.134:6789/0 conn(0x7f6964103190 legacy=0x7f696419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69641ac900 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69641aa6b0 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6950001c90 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6950005d50 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f696b1bd700 10 client.25395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69641aab70 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.972+0000 7f696b1bd700 1 librados: init done 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.973+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f696406ad30 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.973+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f69500039b0 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(667..667 src has 442..667) ==== 9110+0+0 (unknown 437629792 0 0) 0x7f6950089d90 con 0x7f696410afa0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 osd_map(667..667 src has 442..667) 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 3 client.25395.objecter handle_osd_map got epochs [667,667] > 0 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 3 client.25395.objecter handle_osd_map decoding full epoch 667 2024-03-30T15:08:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 20 client.25395.objecter dump_active .. 0 homeless 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v667) ==== 54+0+16 (unknown 1812739437 0 2420000192) 0x7f695008c1d0 con 0x7f696410afa0 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 10 client.25395.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 1 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 1 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 0 1 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 4 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 10 client.25395.objecter _send_command 1 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.974+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6801/3394323575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f696419ae80 con 0x7f69641abda0 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.975+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f6954070480 2024-03-30T15:08:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.975+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f69641abda0 2024-03-30T15:08:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.976+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.0 v1:172.21.15.28:6801/3394323575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f696419ae80 con 0x7f69641abda0 2024-03-30T15:08:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.976+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 1: 0 ) 2024-03-30T15:08:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.976+0000 7f6961ffb700 10 client.25395.objecter _finish_command 1 = system:0 2024-03-30T15:08:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.976+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 4 2024-03-30T15:08:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.976+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 0 1 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 10 client.25395.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 2 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 2 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f6964068b30 osd=0 3 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 0 2 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 4 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 10 client.25395.objecter _send_command 2 2024-03-30T15:08:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.995+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6801/3394323575 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641ab0b0 con 0x7f69641abda0 2024-03-30T15:08:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.0 v1:172.21.15.28:6801/3394323575 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641ab0b0 con 0x7f69641abda0 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 10 client.25395.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 4 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 0 2 2024-03-30T15:08:44.998 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:08:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 10 client.25395.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 3 2024-03-30T15:08:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 3 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 1 3 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 4 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 10 client.25395.objecter _send_command 3 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6964110610 con 0x7f6964062e50 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.997+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f6964062e50 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.998+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6964110610 con 0x7f6964062e50 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.998+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 3: 0 ) 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.998+0000 7f6961ffb700 10 client.25395.objecter _finish_command 3 = system:0 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.998+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 4 2024-03-30T15:08:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:44.998+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 1 3 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 10 client.25395.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 4 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 4 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641ab4b0 osd=1 3 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 1 4 2024-03-30T15:08:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 4 2024-03-30T15:08:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 10 client.25395.objecter _send_command 4 2024-03-30T15:08:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.018+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69640634c0 con 0x7f6964062e50 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.019+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69640634c0 con 0x7f6964062e50 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.019+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.019+0000 7f6961ffb700 10 client.25395.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.019+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 4 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.019+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 1 4 2024-03-30T15:08:45.021 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 10 client.25395.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 5 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 5 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 2 5 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 4 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 10 client.25395.objecter _send_command 5 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f696402d5f0 con 0x7f696402d180 2024-03-30T15:08:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.020+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f696402d180 2024-03-30T15:08:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.021+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f696402d5f0 con 0x7f696402d180 2024-03-30T15:08:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.021+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 5: 0 ) 2024-03-30T15:08:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.021+0000 7f6961ffb700 10 client.25395.objecter _finish_command 5 = system:0 2024-03-30T15:08:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.021+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 4 2024-03-30T15:08:45.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.021+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 2 5 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 10 client.25395.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 6 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 6 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f6964063670 osd=2 3 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 2 6 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 4 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 10 client.25395.objecter _send_command 6 2024-03-30T15:08:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.039+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6809/3451325043 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641b5460 con 0x7f696402d180 2024-03-30T15:08:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.040+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641b5460 con 0x7f696402d180 2024-03-30T15:08:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.040+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.040+0000 7f6961ffb700 10 client.25395.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.040+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 4 2024-03-30T15:08:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.040+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 2 6 2024-03-30T15:08:45.042 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 10 client.25395.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 7 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 7 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 3 7 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 4 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 10 client.25395.objecter _send_command 7 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f69641b1930 con 0x7f696402cd70 2024-03-30T15:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.041+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f696402cd70 2024-03-30T15:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.042+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69641b1930 con 0x7f696402cd70 2024-03-30T15:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.042+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 7: 0 ) 2024-03-30T15:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.042+0000 7f6961ffb700 10 client.25395.objecter _finish_command 7 = system:0 2024-03-30T15:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.042+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 4 2024-03-30T15:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.042+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 3 7 2024-03-30T15:08:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 10 client.25395.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 8 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 8 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f696402cc00 osd=3 3 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 3 8 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 4 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 10 client.25395.objecter _send_command 8 2024-03-30T15:08:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.059+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641b0e00 con 0x7f696402cd70 2024-03-30T15:08:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641b0e00 con 0x7f696402cd70 2024-03-30T15:08:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f6961ffb700 10 client.25395.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 4 2024-03-30T15:08:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 3 8 2024-03-30T15:08:45.063 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:08:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 10 client.25395.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 9 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 9 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 4 9 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 4 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 10 client.25395.objecter _send_command 9 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.061+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f69641b6420 con 0x7f69641b0f70 2024-03-30T15:08:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.062+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f69641b0f70 2024-03-30T15:08:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.063+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69641b6420 con 0x7f69641b0f70 2024-03-30T15:08:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.063+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 9: 0 ) 2024-03-30T15:08:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.063+0000 7f6961ffb700 10 client.25395.objecter _finish_command 9 = system:0 2024-03-30T15:08:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.063+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 4 2024-03-30T15:08:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.063+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 4 9 2024-03-30T15:08:45.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 10 client.25395.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 10 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 10 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b0e00 osd=4 3 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 4 10 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 4 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 10 client.25395.objecter _send_command 10 2024-03-30T15:08:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.079+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6800/3063775514 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641be770 con 0x7f69641b0f70 2024-03-30T15:08:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:08:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641be770 con 0x7f69641b0f70 2024-03-30T15:08:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 10 client.25395.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 4 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 4 10 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 10 client.25395.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 11 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 11 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 5 11 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 4 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 10 client.25395.objecter _send_command 11 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f69641c2960 con 0x7f69641bec80 2024-03-30T15:08:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.082+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f69641bec80 2024-03-30T15:08:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.083+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69641c2960 con 0x7f69641bec80 2024-03-30T15:08:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.083+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 11: 0 ) 2024-03-30T15:08:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.083+0000 7f6961ffb700 10 client.25395.objecter _finish_command 11 = system:0 2024-03-30T15:08:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.083+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 4 2024-03-30T15:08:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.083+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 5 11 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 10 client.25395.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 12 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 12 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b6420 osd=5 3 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 5 12 2024-03-30T15:08:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 4 2024-03-30T15:08:45.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 10 client.25395.objecter _send_command 12 2024-03-30T15:08:45.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.100+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641b5860 con 0x7f69641bec80 2024-03-30T15:08:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641b5860 con 0x7f69641bec80 2024-03-30T15:08:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f6961ffb700 10 client.25395.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 4 2024-03-30T15:08:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 5 12 2024-03-30T15:08:45.104 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 10 client.25395.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 13 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 13 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 6 13 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 4 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 10 client.25395.objecter _send_command 13 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.102+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f69641c4080 con 0x7f69641baba0 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.103+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f69641baba0 2024-03-30T15:08:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.104+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69641c4080 con 0x7f69641baba0 2024-03-30T15:08:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.104+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 13: 0 ) 2024-03-30T15:08:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.104+0000 7f6961ffb700 10 client.25395.objecter _finish_command 13 = system:0 2024-03-30T15:08:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.104+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 4 2024-03-30T15:08:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.104+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 6 13 2024-03-30T15:08:45.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 10 client.25395.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 14 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 14 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641b5610 osd=6 3 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 6 14 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 4 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 10 client.25395.objecter _send_command 14 2024-03-30T15:08:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.121+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641c4080 con 0x7f69641baba0 2024-03-30T15:08:45.124 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.123+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641c4080 con 0x7f69641baba0 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.123+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.123+0000 7f6961ffb700 10 client.25395.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.123+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 4 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.123+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 6 14 2024-03-30T15:08:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 10 client.25395.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 15 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 15 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 7 15 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 4 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 10 client.25395.objecter _send_command 15 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f69641b5ae0 con 0x7f69641c7a20 2024-03-30T15:08:45.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.124+0000 7f6961ffb700 10 client.25395.objecter ms_handle_connect 0x7f69641c7a20 2024-03-30T15:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.125+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69641b5ae0 con 0x7f69641c7a20 2024-03-30T15:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.125+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 15: 0 ) 2024-03-30T15:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.125+0000 7f6961ffb700 10 client.25395.objecter _finish_command 15 = system:0 2024-03-30T15:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.125+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 4 2024-03-30T15:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.125+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 7 15 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 10 client.25395.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign -1 16 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 20 client.25395.objecter _get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.142+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_remove -1 16 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.143+0000 7f696b1bd700 20 client.25395.objecter get_session s=0x7f69641c4080 osd=7 3 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.143+0000 7f696b1bd700 15 client.25395.objecter _session_command_op_assign 7 16 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.143+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 4 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.143+0000 7f696b1bd700 10 client.25395.objecter _send_command 16 2024-03-30T15:08:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.143+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f69641cb530 con 0x7f69641c7a20 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f6961ffb700 1 -- 172.21.15.28:0/1223603784 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f69641cb530 con 0x7f69641c7a20 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f6961ffb700 10 client.25395.objecter ms_dispatch 0x7f6964001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f6961ffb700 10 client.25395.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f6961ffb700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 4 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f6961ffb700 15 client.25395.objecter _session_command_op_remove 7 16 2024-03-30T15:08:45.146 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:08:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 librados: watch_flush enter 2024-03-30T15:08:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 librados: watch_flush exit 2024-03-30T15:08:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.0 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6801/3394323575 conn(0x7f69641abda0 legacy=0x7f69641ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964068b30 osd=0 1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6805/1850313547 conn(0x7f6964062e50 legacy=0x7f696410eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641ab4b0 osd=1 1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.2 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6809/3451325043 conn(0x7f696402d180 legacy=0x7f69641b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f6964063670 osd=2 1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.3 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6813/1506157250 conn(0x7f696402cd70 legacy=0x7f69641b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f696402cc00 osd=3 1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.4 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.134:6800/3063775514 conn(0x7f69641b0f70 legacy=0x7f69641bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b0e00 osd=4 1 2024-03-30T15:08:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.5 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.134:6804/3668151179 conn(0x7f69641bec80 legacy=0x7f69641c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b6420 osd=5 1 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.6 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.134:6808/2135224194 conn(0x7f69641baba0 legacy=0x7f69641c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641b5610 osd=6 1 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter close_session for osd.7 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.134:6812/3758190298 conn(0x7f69641c7a20 legacy=0x7f69641c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter put_session s=0x7f69641c4080 osd=7 1 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 20 client.25395.objecter shutdown clearing up homeless session... 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 10 client.25395.objecter successfully canceled tick 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696975a700 1 -- 172.21.15.28:0/1223603784 reap_dead start 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696975a700 1 -- 172.21.15.28:0/1223603784 reap_dead start 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6954070480 legacy=0x7f6954072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.145+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> v1:172.21.15.28:6789/0 conn(0x7f696410afa0 legacy=0x7f69641a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696975a700 1 -- 172.21.15.28:0/1223603784 reap_dead start 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696975a700 1 -- 172.21.15.28:0/1223603784 reap_dead start 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696975a700 1 -- 172.21.15.28:0/1223603784 reap_dead start 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 shutdown_connections 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 >> 172.21.15.28:0/1223603784 conn(0x7f69640fc9e0 msgr2=0x7f69640fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 shutdown_connections 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696b1bd700 1 -- 172.21.15.28:0/1223603784 wait complete. 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:45.146+0000 7f696b1bd700 1 librados: shutdown 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:08:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:08:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:08:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:08:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:08:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:08:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:08:45.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:45.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:45 smithi028 ceph-mon[29876]: pgmap v987: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:45.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:45 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1223603784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:45.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:45 smithi028 ceph-mon[41614]: pgmap v987: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:45.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:45 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1223603784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:45.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:45.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:45 smithi134 ceph-mon[33714]: pgmap v987: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:45 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1223603784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:45.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:45.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:45.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:45.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:45.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:45.872 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:45.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:45.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:45.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:46.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:46.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:46.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:46.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:46.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:46.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:08:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:08:46.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:08:46.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:08:46.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T15:08:46.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:46 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:08:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:08:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:08:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T15:08:46.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:46 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T15:08:46.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:46.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:46.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:46.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:46.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:46.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:46.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:46.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:08:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:47.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:08:47.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:47.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- 172.21.15.28:0/2471038823 >> v1:172.21.15.28:6790/0 conn(0x7fc180100da0 legacy=0x7fc180101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- 172.21.15.28:0/2471038823 shutdown_connections 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- 172.21.15.28:0/2471038823 >> 172.21.15.28:0/2471038823 conn(0x7fc1800fc9e0 msgr2=0x7fc1800fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- 172.21.15.28:0/2471038823 shutdown_connections 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- 172.21.15.28:0/2471038823 wait complete. 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 librados: starting msgr at 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 librados: starting objecter 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 Processor -- start 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 -- start start 2024-03-30T15:08:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 librados: setting wanted keys 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.248+0000 7fc186420700 1 librados: calling monclient init 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc186420700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1801a9100 con 0x7fc180110c70 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc186420700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1801aa320 con 0x7fc180101980 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc186420700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc1801ab560 con 0x7fc180100da0 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc180100da0 0x7fc18010f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57700/0 (socket says 172.21.15.28:57700) 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17ffff700 1 -- 172.21.15.28:0/3693915463 learned_addr learned my addr 172.21.15.28:0/3693915463 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter ms_handle_connect 0x7fc180100da0 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter ms_handle_connect 0x7fc180110c70 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954096847 0 0) 0x7fc1801a9100 con 0x7fc180110c70 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1680031f0 con 0x7fc180110c70 2024-03-30T15:08:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter ms_handle_connect 0x7fc180101980 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506910411 0 0) 0x7fc1801ab560 con 0x7fc180100da0 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1801a9100 con 0x7fc180100da0 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609544998 0 0) 0x7fc1801aa320 con 0x7fc180101980 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1801ab560 con 0x7fc180101980 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3366455029 0 0) 0x7fc1680031f0 con 0x7fc180110c70 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1801aa320 con 0x7fc180110c70 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.249+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc178001470 con 0x7fc180110c70 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2839978066 0 0) 0x7fc1801a9100 con 0x7fc180100da0 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1680031f0 con 0x7fc180100da0 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc1700038b0 con 0x7fc180100da0 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1231239038 0 0) 0x7fc1801ab560 con 0x7fc180101980 2024-03-30T15:08:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1801a9100 con 0x7fc180101980 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc174001830 con 0x7fc180101980 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3873260701 0 0) 0x7fc1801aa320 con 0x7fc180110c70 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 >> v1:172.21.15.28:6790/0 conn(0x7fc180100da0 legacy=0x7fc18010f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 >> v1:172.21.15.134:6789/0 conn(0x7fc180101980 legacy=0x7fc1801a40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1801ac7a0 con 0x7fc180110c70 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1801a9330 con 0x7fc180110c70 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc186420700 10 client.25401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1801a97f0 con 0x7fc180110c70 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc186420700 1 librados: init done 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc178003ea0 con 0x7fc180110c70 2024-03-30T15:08:47.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.250+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc178004de0 con 0x7fc180110c70 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.251+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c005120 con 0x7fc180110c70 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.251+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc17801b620 con 0x7fc180110c70 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(667..667 src has 442..667) ==== 9110+0+0 (unknown 437629792 0 0) 0x7fc178088c20 con 0x7fc180110c70 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 10 client.25401.objecter ms_dispatch 0x7fc180001550 osd_map(667..667 src has 442..667) 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 3 client.25401.objecter handle_osd_map got epochs [667,667] > 0 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 3 client.25401.objecter handle_osd_map decoding full epoch 667 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 20 client.25401.objecter dump_active .. 0 homeless 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.252+0000 7fc17d7fa700 10 client.25401.objecter ms_handle_connect 0x7fc168070430 2024-03-30T15:08:47.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.256+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc178055a80 con 0x7fc180110c70 2024-03-30T15:08:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:47.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:47.300 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:47.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:47.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:47.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:47 smithi028 ceph-mon[29876]: pgmap v988: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:47.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:47 smithi028 ceph-mon[41614]: pgmap v988: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:47 smithi134 ceph-mon[33714]: pgmap v988: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:08:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.470+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc16c000b90 con 0x7fc168070430 2024-03-30T15:08:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.487+0000 7fc17d7fa700 1 -- 172.21.15.28:0/3693915463 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+633460 (unknown 2965378022 0 1200795696) 0x7fc16c000b90 con 0x7fc168070430 2024-03-30T15:08:47.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:08:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 10 librados: watch_flush enter 2024-03-30T15:08:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 10 librados: watch_flush exit 2024-03-30T15:08:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 20 client.25401.objecter shutdown clearing up homeless session... 2024-03-30T15:08:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 10 client.25401.objecter successfully canceled tick 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc168070430 legacy=0x7fc1680728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 >> v1:172.21.15.28:6789/0 conn(0x7fc180110c70 legacy=0x7fc1801a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 shutdown_connections 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 >> 172.21.15.28:0/3693915463 conn(0x7fc1800fc9e0 msgr2=0x7fc1800fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 shutdown_connections 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 -- 172.21.15.28:0/3693915463 wait complete. 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:47.491+0000 7fc186420700 1 librados: shutdown 2024-03-30T15:08:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:08:47.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:47.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:47.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:47.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:47.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:47.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:47.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:48.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.079 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:48.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:48.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:48.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:48.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":988,"stamp":"2024-03-30T15:08:46.075830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":211,"num_object_clones":4,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":1,"num_read":1998,"num_read_kb":1747,"num_write":739,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2267,"ondisk_log_size":2267,"up":738,"acting":738,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":738,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":640304,"kb_used_data":41020,"kb_used_omap":12,"kb_used_meta":599155,"kb_avail":749320912,"statfs":{"total":767960285184,"available":767304613888,"internally_reserved":0,"allocated":42004480,"data_stored":19790753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":613535280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002707"},"pg_stats":[{"pgid":"325.1f","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360887+0000","last_change":"2024-03-30T15:08:30.420585+0000","last_active":"2024-03-30T15:08:42.360887+0000","last_peered":"2024-03-30T15:08:42.360887+0000","last_clean":"2024-03-30T15:08:42.360887+0000","last_became_active":"2024-03-30T15:08:30.420034+0000","last_became_peered":"2024-03-30T15:08:30.420034+0000","last_unstale":"2024-03-30T15:08:42.360887+0000","last_undegraded":"2024-03-30T15:08:42.360887+0000","last_fullsized":"2024-03-30T15:08:42.360887+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:59.178217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1e","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642154+0000","last_change":"2024-03-30T15:08:30.415672+0000","last_active":"2024-03-30T15:08:42.642154+0000","last_peered":"2024-03-30T15:08:42.642154+0000","last_clean":"2024-03-30T15:08:42.642154+0000","last_became_active":"2024-03-30T15:08:30.415362+0000","last_became_peered":"2024-03-30T15:08:30.415362+0000","last_unstale":"2024-03-30T15:08:42.642154+0000","last_undegraded":"2024-03-30T15:08:42.642154+0000","last_fullsized":"2024-03-30T15:08:42.642154+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:51.838022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.1d","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358741+0000","last_change":"2024-03-30T15:08:30.416291+0000","last_active":"2024-03-30T15:08:42.358741+0000","last_peered":"2024-03-30T15:08:42.358741+0000","last_clean":"2024-03-30T15:08:42.358741+0000","last_became_active":"2024-03-30T15:08:30.415933+0000","last_became_peered":"2024-03-30T15:08:30.415933+0000","last_unstale":"2024-03-30T15:08:42.358741+0000","last_undegraded":"2024-03-30T15:08:42.358741+0000","last_fullsized":"2024-03-30T15:08:42.358741+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.247261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.1c","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360174+0000","last_change":"2024-03-30T15:08:30.412089+0000","last_active":"2024-03-30T15:08:42.360174+0000","last_peered":"2024-03-30T15:08:42.360174+0000","last_clean":"2024-03-30T15:08:42.360174+0000","last_became_active":"2024-03-30T15:08:30.411756+0000","last_became_peered":"2024-03-30T15:08:30.411756+0000","last_unstale":"2024-03-30T15:08:42.360174+0000","last_undegraded":"2024-03-30T15:08:42.360174+0000","last_fullsized":"2024-03-30T15:08:42.360174+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.698083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1b","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641528+0000","last_change":"2024-03-30T15:08:30.416787+0000","last_active":"2024-03-30T15:08:42.641528+0000","last_peered":"2024-03-30T15:08:42.641528+0000","last_clean":"2024-03-30T15:08:42.641528+0000","last_became_active":"2024-03-30T15:08:30.416046+0000","last_became_peered":"2024-03-30T15:08:30.416046+0000","last_unstale":"2024-03-30T15:08:42.641528+0000","last_undegraded":"2024-03-30T15:08:42.641528+0000","last_fullsized":"2024-03-30T15:08:42.641528+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.675100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.1a","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416448+0000","last_change":"2024-03-30T15:08:30.405554+0000","last_active":"2024-03-30T15:08:42.416448+0000","last_peered":"2024-03-30T15:08:42.416448+0000","last_clean":"2024-03-30T15:08:42.416448+0000","last_became_active":"2024-03-30T15:08:30.404955+0000","last_became_peered":"2024-03-30T15:08:30.404955+0000","last_unstale":"2024-03-30T15:08:42.416448+0000","last_undegraded":"2024-03-30T15:08:42.416448+0000","last_fullsized":"2024-03-30T15:08:42.416448+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:50.105611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"325.19","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416435+0000","last_change":"2024-03-30T15:08:30.403318+0000","last_active":"2024-03-30T15:08:42.416435+0000","last_peered":"2024-03-30T15:08:42.416435+0000","last_clean":"2024-03-30T15:08:42.416435+0000","last_became_active":"2024-03-30T15:08:30.402785+0000","last_became_peered":"2024-03-30T15:08:30.402785+0000","last_unstale":"2024-03-30T15:08:42.416435+0000","last_undegraded":"2024-03-30T15:08:42.416435+0000","last_fullsized":"2024-03-30T15:08:42.416435+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.985944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"325.18","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.418880+0000","last_change":"2024-03-30T15:08:30.399469+0000","last_active":"2024-03-30T15:08:42.418880+0000","last_peered":"2024-03-30T15:08:42.418880+0000","last_clean":"2024-03-30T15:08:42.418880+0000","last_became_active":"2024-03-30T15:08:30.399167+0000","last_became_peered":"2024-03-30T15:08:30.399167+0000","last_unstale":"2024-03-30T15:08:42.418880+0000","last_undegraded":"2024-03-30T15:08:42.418880+0000","last_fullsized":"2024-03-30T15:08:42.418880+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.225171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"325.17","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809196+0000","last_change":"2024-03-30T15:08:30.416722+0000","last_active":"2024-03-30T15:08:42.809196+0000","last_peered":"2024-03-30T15:08:42.809196+0000","last_clean":"2024-03-30T15:08:42.809196+0000","last_became_active":"2024-03-30T15:08:30.416408+0000","last_became_peered":"2024-03-30T15:08:30.416408+0000","last_unstale":"2024-03-30T15:08:42.809196+0000","last_undegraded":"2024-03-30T15:08:42.809196+0000","last_fullsized":"2024-03-30T15:08:42.809196+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:37.387429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.16","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360820+0000","last_change":"2024-03-30T15:08:30.419992+0000","last_active":"2024-03-30T15:08:42.360820+0000","last_peered":"2024-03-30T15:08:42.360820+0000","last_clean":"2024-03-30T15:08:42.360820+0000","last_became_active":"2024-03-30T15:08:30.419649+0000","last_became_peered":"2024-03-30T15:08:30.419649+0000","last_unstale":"2024-03-30T15:08:42.360820+0000","last_undegraded":"2024-03-30T15:08:42.360820+0000","last_fullsized":"2024-03-30T15:08:42.360820+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:56.133212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.15","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808583+0000","last_change":"2024-03-30T15:08:30.413730+0000","last_active":"2024-03-30T15:08:42.808583+0000","last_peered":"2024-03-30T15:08:42.808583+0000","last_clean":"2024-03-30T15:08:42.808583+0000","last_became_active":"2024-03-30T15:08:30.412082+0000","last_became_peered":"2024-03-30T15:08:30.412082+0000","last_unstale":"2024-03-30T15:08:42.808583+0000","last_undegraded":"2024-03-30T15:08:42.808583+0000","last_fullsized":"2024-03-30T15:08:42.808583+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:52.431996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.14","version":"665'1","reported_seq":28,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811851+0000","last_change":"2024-03-30T15:08:30.412650+0000","last_active":"2024-03-30T15:08:42.811851+0000","last_peered":"2024-03-30T15:08:42.811851+0000","last_clean":"2024-03-30T15:08:42.811851+0000","last_became_active":"2024-03-30T15:08:30.409803+0000","last_became_peered":"2024-03-30T15:08:30.409803+0000","last_unstale":"2024-03-30T15:08:42.811851+0000","last_undegraded":"2024-03-30T15:08:42.811851+0000","last_fullsized":"2024-03-30T15:08:42.811851+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.519431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.13","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808346+0000","last_change":"2024-03-30T15:08:30.414624+0000","last_active":"2024-03-30T15:08:42.808346+0000","last_peered":"2024-03-30T15:08:42.808346+0000","last_clean":"2024-03-30T15:08:42.808346+0000","last_became_active":"2024-03-30T15:08:30.414233+0000","last_became_peered":"2024-03-30T15:08:30.414233+0000","last_unstale":"2024-03-30T15:08:42.808346+0000","last_undegraded":"2024-03-30T15:08:42.808346+0000","last_fullsized":"2024-03-30T15:08:42.808346+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:58.658542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.12","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359686+0000","last_change":"2024-03-30T15:08:30.411919+0000","last_active":"2024-03-30T15:08:42.359686+0000","last_peered":"2024-03-30T15:08:42.359686+0000","last_clean":"2024-03-30T15:08:42.359686+0000","last_became_active":"2024-03-30T15:08:30.411212+0000","last_became_peered":"2024-03-30T15:08:30.411212+0000","last_unstale":"2024-03-30T15:08:42.359686+0000","last_undegraded":"2024-03-30T15:08:42.359686+0000","last_fullsized":"2024-03-30T15:08:42.359686+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.361219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.11","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811350+0000","last_change":"2024-03-30T15:08:30.412176+0000","last_active":"2024-03-30T15:08:42.811350+0000","last_peered":"2024-03-30T15:08:42.811350+0000","last_clean":"2024-03-30T15:08:42.811350+0000","last_became_active":"2024-03-30T15:08:30.411812+0000","last_became_peered":"2024-03-30T15:08:30.411812+0000","last_unstale":"2024-03-30T15:08:42.811350+0000","last_undegraded":"2024-03-30T15:08:42.811350+0000","last_fullsized":"2024-03-30T15:08:42.811350+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:58.500033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.10","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.807976+0000","last_change":"2024-03-30T15:08:30.408932+0000","last_active":"2024-03-30T15:08:42.807976+0000","last_peered":"2024-03-30T15:08:42.807976+0000","last_clean":"2024-03-30T15:08:42.807976+0000","last_became_active":"2024-03-30T15:08:30.407936+0000","last_became_peered":"2024-03-30T15:08:30.407936+0000","last_unstale":"2024-03-30T15:08:42.807976+0000","last_undegraded":"2024-03-30T15:08:42.807976+0000","last_fullsized":"2024-03-30T15:08:42.807976+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:31.526670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.f","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361007+0000","last_change":"2024-03-30T15:08:30.417310+0000","last_active":"2024-03-30T15:08:42.361007+0000","last_peered":"2024-03-30T15:08:42.361007+0000","last_clean":"2024-03-30T15:08:42.361007+0000","last_became_active":"2024-03-30T15:08:30.416848+0000","last_became_peered":"2024-03-30T15:08:30.416848+0000","last_unstale":"2024-03-30T15:08:42.361007+0000","last_undegraded":"2024-03-30T15:08:42.361007+0000","last_fullsized":"2024-03-30T15:08:42.361007+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:53.971469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.e","version":"665'1","reported_seq":28,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642180+0000","last_change":"2024-03-30T15:08:30.410219+0000","last_active":"2024-03-30T15:08:42.642180+0000","last_peered":"2024-03-30T15:08:42.642180+0000","last_clean":"2024-03-30T15:08:42.642180+0000","last_became_active":"2024-03-30T15:08:30.408510+0000","last_became_peered":"2024-03-30T15:08:30.408510+0000","last_unstale":"2024-03-30T15:08:42.642180+0000","last_undegraded":"2024-03-30T15:08:42.642180+0000","last_fullsized":"2024-03-30T15:08:42.642180+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:33.969138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.d","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902904+0000","last_change":"2024-03-30T15:08:30.409944+0000","last_active":"2024-03-30T15:08:42.902904+0000","last_peered":"2024-03-30T15:08:42.902904+0000","last_clean":"2024-03-30T15:08:42.902904+0000","last_became_active":"2024-03-30T15:08:30.409590+0000","last_became_peered":"2024-03-30T15:08:30.409590+0000","last_unstale":"2024-03-30T15:08:42.902904+0000","last_undegraded":"2024-03-30T15:08:42.902904+0000","last_fullsized":"2024-03-30T15:08:42.902904+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.678080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"325.c","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808447+0000","last_change":"2024-03-30T15:08:30.414353+0000","last_active":"2024-03-30T15:08:42.808447+0000","last_peered":"2024-03-30T15:08:42.808447+0000","last_clean":"2024-03-30T15:08:42.808447+0000","last_became_active":"2024-03-30T15:08:30.413835+0000","last_became_peered":"2024-03-30T15:08:30.413835+0000","last_unstale":"2024-03-30T15:08:42.808447+0000","last_undegraded":"2024-03-30T15:08:42.808447+0000","last_fullsized":"2024-03-30T15:08:42.808447+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:50.673214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.b","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902217+0000","last_change":"2024-03-30T15:08:30.411594+0000","last_active":"2024-03-30T15:08:42.902217+0000","last_peered":"2024-03-30T15:08:42.902217+0000","last_clean":"2024-03-30T15:08:42.902217+0000","last_became_active":"2024-03-30T15:08:30.411294+0000","last_became_peered":"2024-03-30T15:08:30.411294+0000","last_unstale":"2024-03-30T15:08:42.902217+0000","last_undegraded":"2024-03-30T15:08:42.902217+0000","last_fullsized":"2024-03-30T15:08:42.902217+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:57.728526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"325.a","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811482+0000","last_change":"2024-03-30T15:08:30.407079+0000","last_active":"2024-03-30T15:08:42.811482+0000","last_peered":"2024-03-30T15:08:42.811482+0000","last_clean":"2024-03-30T15:08:42.811482+0000","last_became_active":"2024-03-30T15:08:30.406728+0000","last_became_peered":"2024-03-30T15:08:30.406728+0000","last_unstale":"2024-03-30T15:08:42.811482+0000","last_undegraded":"2024-03-30T15:08:42.811482+0000","last_fullsized":"2024-03-30T15:08:42.811482+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:52.646600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.9","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808089+0000","last_change":"2024-03-30T15:08:30.408447+0000","last_active":"2024-03-30T15:08:42.808089+0000","last_peered":"2024-03-30T15:08:42.808089+0000","last_clean":"2024-03-30T15:08:42.808089+0000","last_became_active":"2024-03-30T15:08:30.407675+0000","last_became_peered":"2024-03-30T15:08:30.407675+0000","last_unstale":"2024-03-30T15:08:42.808089+0000","last_undegraded":"2024-03-30T15:08:42.808089+0000","last_fullsized":"2024-03-30T15:08:42.808089+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:53.423565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.8","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641053+0000","last_change":"2024-03-30T15:08:30.407530+0000","last_active":"2024-03-30T15:08:42.641053+0000","last_peered":"2024-03-30T15:08:42.641053+0000","last_clean":"2024-03-30T15:08:42.641053+0000","last_became_active":"2024-03-30T15:08:30.407185+0000","last_became_peered":"2024-03-30T15:08:30.407185+0000","last_unstale":"2024-03-30T15:08:42.641053+0000","last_undegraded":"2024-03-30T15:08:42.641053+0000","last_fullsized":"2024-03-30T15:08:42.641053+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:42.403299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"325.0","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811277+0000","last_change":"2024-03-30T15:08:30.410339+0000","last_active":"2024-03-30T15:08:42.811277+0000","last_peered":"2024-03-30T15:08:42.811277+0000","last_clean":"2024-03-30T15:08:42.811277+0000","last_became_active":"2024-03-30T15:08:30.409418+0000","last_became_peered":"2024-03-30T15:08:30.409418+0000","last_unstale":"2024-03-30T15:08:42.811277+0000","last_undegraded":"2024-03-30T15:08:42.811277+0000","last_fullsized":"2024-03-30T15:08:42.811277+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.534405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.7","version":"0'0","reported_seq":66,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361069+0000","last_change":"2024-03-30T15:08:14.381160+0000","last_active":"2024-03-30T15:08:42.361069+0000","last_peered":"2024-03-30T15:08:42.361069+0000","last_clean":"2024-03-30T15:08:42.361069+0000","last_became_active":"2024-03-30T15:08:14.380692+0000","last_became_peered":"2024-03-30T15:08:14.380692+0000","last_unstale":"2024-03-30T15:08:42.361069+0000","last_undegraded":"2024-03-30T15:08:42.361069+0000","last_fullsized":"2024-03-30T15:08:42.361069+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.346409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"325.1","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416504+0000","last_change":"2024-03-30T15:08:30.407137+0000","last_active":"2024-03-30T15:08:42.416504+0000","last_peered":"2024-03-30T15:08:42.416504+0000","last_clean":"2024-03-30T15:08:42.416504+0000","last_became_active":"2024-03-30T15:08:30.406525+0000","last_became_peered":"2024-03-30T15:08:30.406525+0000","last_unstale":"2024-03-30T15:08:42.416504+0000","last_undegraded":"2024-03-30T15:08:42.416504+0000","last_fullsized":"2024-03-30T15:08:42.416504+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:44.580939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"322.6","version":"662'3","reported_seq":70,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809287+0000","last_change":"2024-03-30T15:08:14.374104+0000","last_active":"2024-03-30T15:08:42.809287+0000","last_peered":"2024-03-30T15:08:42.809287+0000","last_clean":"2024-03-30T15:08:42.809287+0000","last_became_active":"2024-03-30T15:08:14.373744+0000","last_became_peered":"2024-03-30T15:08:14.373744+0000","last_unstale":"2024-03-30T15:08:42.809287+0000","last_undegraded":"2024-03-30T15:08:42.809287+0000","last_fullsized":"2024-03-30T15:08:42.809287+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:29.484167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"325.2","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902007+0000","last_change":"2024-03-30T15:08:30.404313+0000","last_active":"2024-03-30T15:08:42.902007+0000","last_peered":"2024-03-30T15:08:42.902007+0000","last_clean":"2024-03-30T15:08:42.902007+0000","last_became_active":"2024-03-30T15:08:30.404017+0000","last_became_peered":"2024-03-30T15:08:30.404017+0000","last_unstale":"2024-03-30T15:08:42.902007+0000","last_undegraded":"2024-03-30T15:08:42.902007+0000","last_fullsized":"2024-03-30T15:08:42.902007+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:56.886987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.5","version":"0'0","reported_seq":66,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358772+0000","last_change":"2024-03-30T15:08:14.368479+0000","last_active":"2024-03-30T15:08:42.358772+0000","last_peered":"2024-03-30T15:08:42.358772+0000","last_clean":"2024-03-30T15:08:42.358772+0000","last_became_active":"2024-03-30T15:08:14.368163+0000","last_became_peered":"2024-03-30T15:08:14.368163+0000","last_unstale":"2024-03-30T15:08:42.358772+0000","last_undegraded":"2024-03-30T15:08:42.358772+0000","last_fullsized":"2024-03-30T15:08:42.358772+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.681841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.3","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902369+0000","last_change":"2024-03-30T15:08:30.414252+0000","last_active":"2024-03-30T15:08:42.902369+0000","last_peered":"2024-03-30T15:08:42.902369+0000","last_clean":"2024-03-30T15:08:42.902369+0000","last_became_active":"2024-03-30T15:08:30.413598+0000","last_became_peered":"2024-03-30T15:08:30.413598+0000","last_unstale":"2024-03-30T15:08:42.902369+0000","last_undegraded":"2024-03-30T15:08:42.902369+0000","last_fullsized":"2024-03-30T15:08:42.902369+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:42.782581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.4","version":"662'3","reported_seq":70,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811863+0000","last_change":"2024-03-30T15:08:14.374633+0000","last_active":"2024-03-30T15:08:42.811863+0000","last_peered":"2024-03-30T15:08:42.811863+0000","last_clean":"2024-03-30T15:08:42.811863+0000","last_became_active":"2024-03-30T15:08:14.374287+0000","last_became_peered":"2024-03-30T15:08:14.374287+0000","last_unstale":"2024-03-30T15:08:42.811863+0000","last_undegraded":"2024-03-30T15:08:42.811863+0000","last_fullsized":"2024-03-30T15:08:42.811863+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:23.052896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.4","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808589+0000","last_change":"2024-03-30T15:08:30.409940+0000","last_active":"2024-03-30T15:08:42.808589+0000","last_peered":"2024-03-30T15:08:42.808589+0000","last_clean":"2024-03-30T15:08:42.808589+0000","last_became_active":"2024-03-30T15:08:30.409504+0000","last_became_peered":"2024-03-30T15:08:30.409504+0000","last_unstale":"2024-03-30T15:08:42.808589+0000","last_undegraded":"2024-03-30T15:08:42.808589+0000","last_fullsized":"2024-03-30T15:08:42.808589+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:40.259355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"322.3","version":"0'0","reported_seq":66,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358293+0000","last_change":"2024-03-30T15:08:14.369066+0000","last_active":"2024-03-30T15:08:42.358293+0000","last_peered":"2024-03-30T15:08:42.358293+0000","last_clean":"2024-03-30T15:08:42.358293+0000","last_became_active":"2024-03-30T15:08:14.367491+0000","last_became_peered":"2024-03-30T15:08:14.367491+0000","last_unstale":"2024-03-30T15:08:42.358293+0000","last_undegraded":"2024-03-30T15:08:42.358293+0000","last_fullsized":"2024-03-30T15:08:42.358293+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:42.954042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"325.5","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813086+0000","last_change":"2024-03-30T15:08:30.418292+0000","last_active":"2024-03-30T15:08:42.813086+0000","last_peered":"2024-03-30T15:08:42.813086+0000","last_clean":"2024-03-30T15:08:42.813086+0000","last_became_active":"2024-03-30T15:08:30.417980+0000","last_became_peered":"2024-03-30T15:08:30.417980+0000","last_unstale":"2024-03-30T15:08:42.813086+0000","last_undegraded":"2024-03-30T15:08:42.813086+0000","last_fullsized":"2024-03-30T15:08:42.813086+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.724583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.2","version":"0'0","reported_seq":66,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811558+0000","last_change":"2024-03-30T15:08:14.378059+0000","last_active":"2024-03-30T15:08:42.811558+0000","last_peered":"2024-03-30T15:08:42.811558+0000","last_clean":"2024-03-30T15:08:42.811558+0000","last_became_active":"2024-03-30T15:08:14.377722+0000","last_became_peered":"2024-03-30T15:08:14.377722+0000","last_unstale":"2024-03-30T15:08:42.811558+0000","last_undegraded":"2024-03-30T15:08:42.811558+0000","last_fullsized":"2024-03-30T15:08:42.811558+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.958461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.6","version":"0'0","reported_seq":35,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903197+0000","last_change":"2024-03-30T15:08:34.844696+0000","last_active":"2024-03-30T15:08:42.903197+0000","last_peered":"2024-03-30T15:08:42.903197+0000","last_clean":"2024-03-30T15:08:42.903197+0000","last_became_active":"2024-03-30T15:08:30.408474+0000","last_became_peered":"2024-03-30T15:08:30.408474+0000","last_unstale":"2024-03-30T15:08:42.903197+0000","last_undegraded":"2024-03-30T15:08:42.903197+0000","last_fullsized":"2024-03-30T15:08:42.903197+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:34.844566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:34.844566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.019138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.1","version":"646'1","reported_seq":68,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811364+0000","last_change":"2024-03-30T15:08:14.375816+0000","last_active":"2024-03-30T15:08:42.811364+0000","last_peered":"2024-03-30T15:08:42.811364+0000","last_clean":"2024-03-30T15:08:42.811364+0000","last_became_active":"2024-03-30T15:08:14.375417+0000","last_became_peered":"2024-03-30T15:08:14.375417+0000","last_unstale":"2024-03-30T15:08:42.811364+0000","last_undegraded":"2024-03-30T15:08:42.811364+0000","last_fullsized":"2024-03-30T15:08:42.811364+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.087240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"325.7","version":"0'0","reported_seq":26,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809191+0000","last_change":"2024-03-30T15:08:30.411515+0000","last_active":"2024-03-30T15:08:42.809191+0000","last_peered":"2024-03-30T15:08:42.809191+0000","last_clean":"2024-03-30T15:08:42.809191+0000","last_became_active":"2024-03-30T15:08:30.410333+0000","last_became_peered":"2024-03-30T15:08:30.410333+0000","last_unstale":"2024-03-30T15:08:42.809191+0000","last_undegraded":"2024-03-30T15:08:42.809191+0000","last_fullsized":"2024-03-30T15:08:42.809191+0000","mapping_epoch":660,"log_start":"0'0","ondisk_log_start":"0'0","created":660,"last_epoch_clean":661,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:29.380558+0000","last_clean_scrub_stamp":"2024-03-30T15:08:29.380558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:53.899890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"322.0","version":"662'7","reported_seq":73,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808050+0000","last_change":"2024-03-30T15:08:14.374886+0000","last_active":"2024-03-30T15:08:42.808050+0000","last_peered":"2024-03-30T15:08:42.808050+0000","last_clean":"2024-03-30T15:08:42.808050+0000","last_became_active":"2024-03-30T15:08:14.374433+0000","last_became_peered":"2024-03-30T15:08:14.374433+0000","last_unstale":"2024-03-30T15:08:42.808050+0000","last_undegraded":"2024-03-30T15:08:42.808050+0000","last_fullsized":"2024-03-30T15:08:42.808050+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.420371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.15","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642129+0000","last_change":"2024-03-30T15:08:00.661723+0000","last_active":"2024-03-30T15:08:42.642129+0000","last_peered":"2024-03-30T15:08:42.642129+0000","last_clean":"2024-03-30T15:08:42.642129+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:08:42.642129+0000","last_undegraded":"2024-03-30T15:08:42.642129+0000","last_fullsized":"2024-03-30T15:08:42.642129+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.661646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.661646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:23.913327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":1143,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902994+0000","last_change":"2024-03-30T15:07:25.409282+0000","last_active":"2024-03-30T15:08:42.902994+0000","last_peered":"2024-03-30T15:08:42.902994+0000","last_clean":"2024-03-30T15:08:42.902994+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:08:42.902994+0000","last_undegraded":"2024-03-30T15:08:42.902994+0000","last_fullsized":"2024-03-30T15:08:42.902994+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.409160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.409160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361321+0000","last_change":"2024-03-30T15:08:00.188376+0000","last_active":"2024-03-30T15:08:42.361321+0000","last_peered":"2024-03-30T15:08:42.361321+0000","last_clean":"2024-03-30T15:08:42.361321+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:08:42.361321+0000","last_undegraded":"2024-03-30T15:08:42.361321+0000","last_fullsized":"2024-03-30T15:08:42.361321+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.188301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.188301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.659221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":1137,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809436+0000","last_change":"2024-03-30T15:06:56.895628+0000","last_active":"2024-03-30T15:08:42.809436+0000","last_peered":"2024-03-30T15:08:42.809436+0000","last_clean":"2024-03-30T15:08:42.809436+0000","last_became_active":"2024-03-30T14:57:56.237906+0000","last_became_peered":"2024-03-30T14:57:56.237906+0000","last_unstale":"2024-03-30T15:08:42.809436+0000","last_undegraded":"2024-03-30T15:08:42.809436+0000","last_fullsized":"2024-03-30T15:08:42.809436+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.895505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.895505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":1129,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358371+0000","last_change":"2024-03-30T15:05:27.559147+0000","last_active":"2024-03-30T15:08:42.358371+0000","last_peered":"2024-03-30T15:08:42.358371+0000","last_clean":"2024-03-30T15:08:42.358371+0000","last_became_active":"2024-03-30T14:57:56.258059+0000","last_became_peered":"2024-03-30T14:57:56.258059+0000","last_unstale":"2024-03-30T15:08:42.358371+0000","last_undegraded":"2024-03-30T15:08:42.358371+0000","last_fullsized":"2024-03-30T15:08:42.358371+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":1152,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416390+0000","last_change":"2024-03-30T15:07:57.201822+0000","last_active":"2024-03-30T15:08:42.416390+0000","last_peered":"2024-03-30T15:08:42.416390+0000","last_clean":"2024-03-30T15:08:42.416390+0000","last_became_active":"2024-03-30T14:57:56.220921+0000","last_became_peered":"2024-03-30T14:57:56.220921+0000","last_unstale":"2024-03-30T15:08:42.416390+0000","last_undegraded":"2024-03-30T15:08:42.416390+0000","last_fullsized":"2024-03-30T15:08:42.416390+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.201691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.201691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:05.630818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360376+0000","last_change":"2024-03-30T15:08:13.183528+0000","last_active":"2024-03-30T15:08:42.360376+0000","last_peered":"2024-03-30T15:08:42.360376+0000","last_clean":"2024-03-30T15:08:42.360376+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:08:42.360376+0000","last_undegraded":"2024-03-30T15:08:42.360376+0000","last_fullsized":"2024-03-30T15:08:42.360376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.183373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.183373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.888133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641685+0000","last_change":"2024-03-30T15:06:59.257326+0000","last_active":"2024-03-30T15:08:42.641685+0000","last_peered":"2024-03-30T15:08:42.641685+0000","last_clean":"2024-03-30T15:08:42.641685+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:08:42.641685+0000","last_undegraded":"2024-03-30T15:08:42.641685+0000","last_fullsized":"2024-03-30T15:08:42.641685+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.257230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.257230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":1134,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813276+0000","last_change":"2024-03-30T15:07:36.042362+0000","last_active":"2024-03-30T15:08:42.813276+0000","last_peered":"2024-03-30T15:08:42.813276+0000","last_clean":"2024-03-30T15:08:42.813276+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:08:42.813276+0000","last_undegraded":"2024-03-30T15:08:42.813276+0000","last_fullsized":"2024-03-30T15:08:42.813276+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.042281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.042281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.352269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":1145,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808602+0000","last_change":"2024-03-30T15:08:42.145040+0000","last_active":"2024-03-30T15:08:42.808602+0000","last_peered":"2024-03-30T15:08:42.808602+0000","last_clean":"2024-03-30T15:08:42.808602+0000","last_became_active":"2024-03-30T14:57:56.239496+0000","last_became_peered":"2024-03-30T14:57:56.239496+0000","last_unstale":"2024-03-30T15:08:42.808602+0000","last_undegraded":"2024-03-30T15:08:42.808602+0000","last_fullsized":"2024-03-30T15:08:42.808602+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:42.144953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.144953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:48.021481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361323+0000","last_change":"2024-03-30T15:08:10.138813+0000","last_active":"2024-03-30T15:08:42.361323+0000","last_peered":"2024-03-30T15:08:42.361323+0000","last_clean":"2024-03-30T15:08:42.361323+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:08:42.361323+0000","last_undegraded":"2024-03-30T15:08:42.361323+0000","last_fullsized":"2024-03-30T15:08:42.361323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.138657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.138657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.317256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":1150,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.417208+0000","last_change":"2024-03-30T15:07:32.132841+0000","last_active":"2024-03-30T15:08:42.417208+0000","last_peered":"2024-03-30T15:08:42.417208+0000","last_clean":"2024-03-30T15:08:42.417208+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:08:42.417208+0000","last_undegraded":"2024-03-30T15:08:42.417208+0000","last_fullsized":"2024-03-30T15:08:42.417208+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.132758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.132758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359632+0000","last_change":"2024-03-30T15:08:15.202644+0000","last_active":"2024-03-30T15:08:42.359632+0000","last_peered":"2024-03-30T15:08:42.359632+0000","last_clean":"2024-03-30T15:08:42.359632+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:08:42.359632+0000","last_undegraded":"2024-03-30T15:08:42.359632+0000","last_fullsized":"2024-03-30T15:08:42.359632+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.202555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:37.887939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":1137,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808056+0000","last_change":"2024-03-30T15:06:43.837698+0000","last_active":"2024-03-30T15:08:42.808056+0000","last_peered":"2024-03-30T15:08:42.808056+0000","last_clean":"2024-03-30T15:08:42.808056+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:08:42.808056+0000","last_undegraded":"2024-03-30T15:08:42.808056+0000","last_fullsized":"2024-03-30T15:08:42.808056+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.837619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.837619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416576+0000","last_change":"2024-03-30T15:06:54.767293+0000","last_active":"2024-03-30T15:08:42.416576+0000","last_peered":"2024-03-30T15:08:42.416576+0000","last_clean":"2024-03-30T15:08:42.416576+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:08:42.416576+0000","last_undegraded":"2024-03-30T15:08:42.416576+0000","last_fullsized":"2024-03-30T15:08:42.416576+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.767213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.767213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416941+0000","last_change":"2024-03-30T15:07:25.988108+0000","last_active":"2024-03-30T15:08:42.416941+0000","last_peered":"2024-03-30T15:08:42.416941+0000","last_clean":"2024-03-30T15:08:42.416941+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:08:42.416941+0000","last_undegraded":"2024-03-30T15:08:42.416941+0000","last_fullsized":"2024-03-30T15:08:42.416941+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.988025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:07.597700+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.988025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360533+0000","last_change":"2024-03-30T15:07:59.150468+0000","last_active":"2024-03-30T15:08:42.360533+0000","last_peered":"2024-03-30T15:08:42.360533+0000","last_clean":"2024-03-30T15:08:42.360533+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:08:42.360533+0000","last_undegraded":"2024-03-30T15:08:42.360533+0000","last_fullsized":"2024-03-30T15:08:42.360533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:59.150361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.150361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.062144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812210+0000","last_change":"2024-03-30T15:07:25.964793+0000","last_active":"2024-03-30T15:08:42.812210+0000","last_peered":"2024-03-30T15:08:42.812210+0000","last_clean":"2024-03-30T15:08:42.812210+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:08:42.812210+0000","last_undegraded":"2024-03-30T15:08:42.812210+0000","last_fullsized":"2024-03-30T15:08:42.812210+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.964714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.964714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642531+0000","last_change":"2024-03-30T15:06:58.308106+0000","last_active":"2024-03-30T15:08:42.642531+0000","last_peered":"2024-03-30T15:08:42.642531+0000","last_clean":"2024-03-30T15:08:42.642531+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:08:42.642531+0000","last_undegraded":"2024-03-30T15:08:42.642531+0000","last_fullsized":"2024-03-30T15:08:42.642531+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.308015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.308015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360871+0000","last_change":"2024-03-30T15:07:35.172101+0000","last_active":"2024-03-30T15:08:42.360871+0000","last_peered":"2024-03-30T15:08:42.360871+0000","last_clean":"2024-03-30T15:08:42.360871+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:08:42.360871+0000","last_undegraded":"2024-03-30T15:08:42.360871+0000","last_fullsized":"2024-03-30T15:08:42.360871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.171990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.171990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:02.621474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":1146,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808227+0000","last_change":"2024-03-30T15:08:09.224705+0000","last_active":"2024-03-30T15:08:42.808227+0000","last_peered":"2024-03-30T15:08:42.808227+0000","last_clean":"2024-03-30T15:08:42.808227+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:08:42.808227+0000","last_undegraded":"2024-03-30T15:08:42.808227+0000","last_fullsized":"2024-03-30T15:08:42.808227+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.224499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.224499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.632314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359609+0000","last_change":"2024-03-30T15:06:58.979164+0000","last_active":"2024-03-30T15:08:42.359609+0000","last_peered":"2024-03-30T15:08:42.359609+0000","last_clean":"2024-03-30T15:08:42.359609+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:08:42.359609+0000","last_undegraded":"2024-03-30T15:08:42.359609+0000","last_fullsized":"2024-03-30T15:08:42.359609+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.979074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.979074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":1144,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902599+0000","last_change":"2024-03-30T15:08:08.551682+0000","last_active":"2024-03-30T15:08:42.902599+0000","last_peered":"2024-03-30T15:08:42.902599+0000","last_clean":"2024-03-30T15:08:42.902599+0000","last_became_active":"2024-03-30T14:57:56.232718+0000","last_became_peered":"2024-03-30T14:57:56.232718+0000","last_unstale":"2024-03-30T15:08:42.902599+0000","last_undegraded":"2024-03-30T15:08:42.902599+0000","last_fullsized":"2024-03-30T15:08:42.902599+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.551547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.551547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:35.564084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":1150,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416576+0000","last_change":"2024-03-30T15:06:45.791752+0000","last_active":"2024-03-30T15:08:42.416576+0000","last_peered":"2024-03-30T15:08:42.416576+0000","last_clean":"2024-03-30T15:08:42.416576+0000","last_became_active":"2024-03-30T14:57:56.222928+0000","last_became_peered":"2024-03-30T14:57:56.222928+0000","last_unstale":"2024-03-30T15:08:42.416576+0000","last_undegraded":"2024-03-30T15:08:42.416576+0000","last_fullsized":"2024-03-30T15:08:42.416576+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.791667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.791667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":1143,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903116+0000","last_change":"2024-03-30T15:08:09.548782+0000","last_active":"2024-03-30T15:08:42.903116+0000","last_peered":"2024-03-30T15:08:42.903116+0000","last_clean":"2024-03-30T15:08:42.903116+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:08:42.903116+0000","last_undegraded":"2024-03-30T15:08:42.903116+0000","last_fullsized":"2024-03-30T15:08:42.903116+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.548687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.755962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":252,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.640964+0000","last_change":"2024-03-30T15:06:18.488509+0000","last_active":"2024-03-30T15:08:42.640964+0000","last_peered":"2024-03-30T15:08:42.640964+0000","last_clean":"2024-03-30T15:08:42.640964+0000","last_became_active":"2024-03-30T15:06:18.488207+0000","last_became_peered":"2024-03-30T15:06:18.488207+0000","last_unstale":"2024-03-30T15:08:42.640964+0000","last_undegraded":"2024-03-30T15:08:42.640964+0000","last_fullsized":"2024-03-30T15:08:42.640964+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:09.176687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:06:09.176687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642771+0000","last_change":"2024-03-30T15:08:02.711973+0000","last_active":"2024-03-30T15:08:42.642771+0000","last_peered":"2024-03-30T15:08:42.642771+0000","last_clean":"2024-03-30T15:08:42.642771+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:08:42.642771+0000","last_undegraded":"2024-03-30T15:08:42.642771+0000","last_fullsized":"2024-03-30T15:08:42.642771+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.711881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.711881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.583184+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642256+0000","last_change":"2024-03-30T15:07:26.321108+0000","last_active":"2024-03-30T15:08:42.642256+0000","last_peered":"2024-03-30T15:08:42.642256+0000","last_clean":"2024-03-30T15:08:42.642256+0000","last_became_active":"2024-03-30T14:57:56.249105+0000","last_became_peered":"2024-03-30T14:57:56.249105+0000","last_unstale":"2024-03-30T15:08:42.642256+0000","last_undegraded":"2024-03-30T15:08:42.642256+0000","last_fullsized":"2024-03-30T15:08:42.642256+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.321030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":1143,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.901731+0000","last_change":"2024-03-30T15:07:31.354747+0000","last_active":"2024-03-30T15:08:42.901731+0000","last_peered":"2024-03-30T15:08:42.901731+0000","last_clean":"2024-03-30T15:08:42.901731+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:08:42.901731+0000","last_undegraded":"2024-03-30T15:08:42.901731+0000","last_fullsized":"2024-03-30T15:08:42.901731+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.354669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.354669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641160+0000","last_change":"2024-03-30T15:06:56.238490+0000","last_active":"2024-03-30T15:08:42.641160+0000","last_peered":"2024-03-30T15:08:42.641160+0000","last_clean":"2024-03-30T15:08:42.641160+0000","last_became_active":"2024-03-30T14:57:56.271489+0000","last_became_peered":"2024-03-30T14:57:56.271489+0000","last_unstale":"2024-03-30T15:08:42.641160+0000","last_undegraded":"2024-03-30T15:08:42.641160+0000","last_fullsized":"2024-03-30T15:08:42.641160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.238398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.238398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":1134,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811828+0000","last_change":"2024-03-30T15:07:19.822016+0000","last_active":"2024-03-30T15:08:42.811828+0000","last_peered":"2024-03-30T15:08:42.811828+0000","last_clean":"2024-03-30T15:08:42.811828+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:08:42.811828+0000","last_undegraded":"2024-03-30T15:08:42.811828+0000","last_fullsized":"2024-03-30T15:08:42.811828+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.821886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.821886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.157019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360045+0000","last_change":"2024-03-30T15:07:58.173279+0000","last_active":"2024-03-30T15:08:42.360045+0000","last_peered":"2024-03-30T15:08:42.360045+0000","last_clean":"2024-03-30T15:08:42.360045+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:08:42.360045+0000","last_undegraded":"2024-03-30T15:08:42.360045+0000","last_fullsized":"2024-03-30T15:08:42.360045+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.173193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.173193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:11.398411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1523,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641240+0000","last_change":"2024-03-30T15:06:49.183096+0000","last_active":"2024-03-30T15:08:42.641240+0000","last_peered":"2024-03-30T15:08:42.641240+0000","last_clean":"2024-03-30T15:08:42.641240+0000","last_became_active":"2024-03-30T15:06:48.338292+0000","last_became_peered":"2024-03-30T15:06:48.338292+0000","last_unstale":"2024-03-30T15:08:42.641240+0000","last_undegraded":"2024-03-30T15:08:42.641240+0000","last_fullsized":"2024-03-30T15:08:42.641240+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.183009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.183009+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903090+0000","last_change":"2024-03-30T15:06:43.474123+0000","last_active":"2024-03-30T15:08:42.903090+0000","last_peered":"2024-03-30T15:08:42.903090+0000","last_clean":"2024-03-30T15:08:42.903090+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:08:42.903090+0000","last_undegraded":"2024-03-30T15:08:42.903090+0000","last_fullsized":"2024-03-30T15:08:42.903090+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.474015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.474015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1269,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812245+0000","last_change":"2024-03-30T15:08:06.185130+0000","last_active":"2024-03-30T15:08:42.812245+0000","last_peered":"2024-03-30T15:08:42.812245+0000","last_clean":"2024-03-30T15:08:42.812245+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:08:42.812245+0000","last_undegraded":"2024-03-30T15:08:42.812245+0000","last_fullsized":"2024-03-30T15:08:42.812245+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:08:06.185037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.185037+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.018400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1139,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358708+0000","last_change":"2024-03-30T15:06:11.536378+0000","last_active":"2024-03-30T15:08:42.358708+0000","last_peered":"2024-03-30T15:08:42.358708+0000","last_clean":"2024-03-30T15:08:42.358708+0000","last_became_active":"2024-03-30T14:57:56.219600+0000","last_became_peered":"2024-03-30T14:57:56.219600+0000","last_unstale":"2024-03-30T15:08:42.358708+0000","last_undegraded":"2024-03-30T15:08:42.358708+0000","last_fullsized":"2024-03-30T15:08:42.358708+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":426,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903094+0000","last_change":"2024-03-30T15:06:58.353375+0000","last_active":"2024-03-30T15:08:42.903094+0000","last_peered":"2024-03-30T15:08:42.903094+0000","last_clean":"2024-03-30T15:08:42.903094+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:08:42.903094+0000","last_undegraded":"2024-03-30T15:08:42.903094+0000","last_fullsized":"2024-03-30T15:08:42.903094+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'9","last_scrub_stamp":"2024-03-30T15:06:58.353220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.353220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1223,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813539+0000","last_change":"2024-03-30T15:06:44.519216+0000","last_active":"2024-03-30T15:08:42.813539+0000","last_peered":"2024-03-30T15:08:42.813539+0000","last_clean":"2024-03-30T15:08:42.813539+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:42.813539+0000","last_undegraded":"2024-03-30T15:08:42.813539+0000","last_fullsized":"2024-03-30T15:08:42.813539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:44.519122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.519122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":521,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359486+0000","last_change":"2024-03-30T15:06:49.951825+0000","last_active":"2024-03-30T15:08:42.359486+0000","last_peered":"2024-03-30T15:08:42.359486+0000","last_clean":"2024-03-30T15:08:42.359486+0000","last_became_active":"2024-03-30T15:06:48.340482+0000","last_became_peered":"2024-03-30T15:06:48.340482+0000","last_unstale":"2024-03-30T15:08:42.359486+0000","last_undegraded":"2024-03-30T15:08:42.359486+0000","last_fullsized":"2024-03-30T15:08:42.359486+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.951714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.951714+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903258+0000","last_change":"2024-03-30T15:07:22.415753+0000","last_active":"2024-03-30T15:08:42.903258+0000","last_peered":"2024-03-30T15:08:42.903258+0000","last_clean":"2024-03-30T15:08:42.903258+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:08:42.903258+0000","last_undegraded":"2024-03-30T15:08:42.903258+0000","last_fullsized":"2024-03-30T15:08:42.903258+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.415581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.415581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1288,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813562+0000","last_change":"2024-03-30T15:07:20.849551+0000","last_active":"2024-03-30T15:08:42.813562+0000","last_peered":"2024-03-30T15:08:42.813562+0000","last_clean":"2024-03-30T15:08:42.813562+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:08:42.813562+0000","last_undegraded":"2024-03-30T15:08:42.813562+0000","last_fullsized":"2024-03-30T15:08:42.813562+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:07:20.849431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.849431+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1251,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902153+0000","last_change":"2024-03-30T15:07:34.300735+0000","last_active":"2024-03-30T15:08:42.902153+0000","last_peered":"2024-03-30T15:08:42.902153+0000","last_clean":"2024-03-30T15:08:42.902153+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:08:42.902153+0000","last_undegraded":"2024-03-30T15:08:42.902153+0000","last_fullsized":"2024-03-30T15:08:42.902153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:34.300632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.300632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1246,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809042+0000","last_change":"2024-03-30T15:07:19.878091+0000","last_active":"2024-03-30T15:08:42.809042+0000","last_peered":"2024-03-30T15:08:42.809042+0000","last_clean":"2024-03-30T15:08:42.809042+0000","last_became_active":"2024-03-30T14:54:41.352388+0000","last_became_peered":"2024-03-30T14:54:41.352388+0000","last_unstale":"2024-03-30T15:08:42.809042+0000","last_undegraded":"2024-03-30T15:08:42.809042+0000","last_fullsized":"2024-03-30T15:08:42.809042+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.878011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.878011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.486305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1252,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.417385+0000","last_change":"2024-03-30T15:08:02.197580+0000","last_active":"2024-03-30T15:08:42.417385+0000","last_peered":"2024-03-30T15:08:42.417385+0000","last_clean":"2024-03-30T15:08:42.417385+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:08:42.417385+0000","last_undegraded":"2024-03-30T15:08:42.417385+0000","last_fullsized":"2024-03-30T15:08:42.417385+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.197498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.197498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.675581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1152,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809989+0000","last_change":"2024-03-30T15:08:05.112361+0000","last_active":"2024-03-30T15:08:42.809989+0000","last_peered":"2024-03-30T15:08:42.809989+0000","last_clean":"2024-03-30T15:08:42.809989+0000","last_became_active":"2024-03-30T14:57:54.077112+0000","last_became_peered":"2024-03-30T14:57:54.077112+0000","last_unstale":"2024-03-30T15:08:42.809989+0000","last_undegraded":"2024-03-30T15:08:42.809989+0000","last_fullsized":"2024-03-30T15:08:42.809989+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:05.112284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:05.112284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:08.797988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1275,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813435+0000","last_change":"2024-03-30T15:07:58.225113+0000","last_active":"2024-03-30T15:08:42.813435+0000","last_peered":"2024-03-30T15:08:42.813435+0000","last_clean":"2024-03-30T15:08:42.813435+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:08:42.813435+0000","last_undegraded":"2024-03-30T15:08:42.813435+0000","last_fullsized":"2024-03-30T15:08:42.813435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:07:58.225002+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.225002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.514446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811735+0000","last_change":"2024-03-30T15:07:33.014237+0000","last_active":"2024-03-30T15:08:42.811735+0000","last_peered":"2024-03-30T15:08:42.811735+0000","last_clean":"2024-03-30T15:08:42.811735+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:08:42.811735+0000","last_undegraded":"2024-03-30T15:08:42.811735+0000","last_fullsized":"2024-03-30T15:08:42.811735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.014124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.688496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1235,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361596+0000","last_change":"2024-03-30T15:08:21.357777+0000","last_active":"2024-03-30T15:08:42.361596+0000","last_peered":"2024-03-30T15:08:42.361596+0000","last_clean":"2024-03-30T15:08:42.361596+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:08:42.361596+0000","last_undegraded":"2024-03-30T15:08:42.361596+0000","last_fullsized":"2024-03-30T15:08:42.361596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.357646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.357646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.045428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1152,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903272+0000","last_change":"2024-03-30T15:08:10.524135+0000","last_active":"2024-03-30T15:08:42.903272+0000","last_peered":"2024-03-30T15:08:42.903272+0000","last_clean":"2024-03-30T15:08:42.903272+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:08:42.903272+0000","last_undegraded":"2024-03-30T15:08:42.903272+0000","last_fullsized":"2024-03-30T15:08:42.903272+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.524046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.524046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.049104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":521,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359892+0000","last_change":"2024-03-30T15:06:50.920479+0000","last_active":"2024-03-30T15:08:42.359892+0000","last_peered":"2024-03-30T15:08:42.359892+0000","last_clean":"2024-03-30T15:08:42.359892+0000","last_became_active":"2024-03-30T15:06:48.342510+0000","last_became_peered":"2024-03-30T15:06:48.342510+0000","last_unstale":"2024-03-30T15:08:42.359892+0000","last_undegraded":"2024-03-30T15:08:42.359892+0000","last_fullsized":"2024-03-30T15:08:42.359892+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:50.920389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.920389+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1152,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809460+0000","last_change":"2024-03-30T15:08:06.129401+0000","last_active":"2024-03-30T15:08:42.809460+0000","last_peered":"2024-03-30T15:08:42.809460+0000","last_clean":"2024-03-30T15:08:42.809460+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:08:42.809460+0000","last_undegraded":"2024-03-30T15:08:42.809460+0000","last_fullsized":"2024-03-30T15:08:42.809460+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.129319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.129319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:13.993935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1292,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809453+0000","last_change":"2024-03-30T15:07:28.992483+0000","last_active":"2024-03-30T15:08:42.809453+0000","last_peered":"2024-03-30T15:08:42.809453+0000","last_clean":"2024-03-30T15:08:42.809453+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:08:42.809453+0000","last_undegraded":"2024-03-30T15:08:42.809453+0000","last_fullsized":"2024-03-30T15:08:42.809453+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:28.992348+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.992348+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1147,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809324+0000","last_change":"2024-03-30T15:07:22.928759+0000","last_active":"2024-03-30T15:08:42.809324+0000","last_peered":"2024-03-30T15:08:42.809324+0000","last_clean":"2024-03-30T15:08:42.809324+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:08:42.809324+0000","last_undegraded":"2024-03-30T15:08:42.809324+0000","last_fullsized":"2024-03-30T15:08:42.809324+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.928677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.928677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"76'8","reported_seq":1413,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813494+0000","last_change":"2024-03-30T15:08:15.143675+0000","last_active":"2024-03-30T15:08:42.813494+0000","last_peered":"2024-03-30T15:08:42.813494+0000","last_clean":"2024-03-30T15:08:42.813494+0000","last_became_active":"2024-03-30T14:54:41.354195+0000","last_became_peered":"2024-03-30T14:54:41.354195+0000","last_unstale":"2024-03-30T15:08:42.813494+0000","last_undegraded":"2024-03-30T15:08:42.813494+0000","last_fullsized":"2024-03-30T15:08:42.813494+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:15.143526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.143526+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.962882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1223,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360634+0000","last_change":"2024-03-30T15:06:57.951315+0000","last_active":"2024-03-30T15:08:42.360634+0000","last_peered":"2024-03-30T15:08:42.360634+0000","last_clean":"2024-03-30T15:08:42.360634+0000","last_became_active":"2024-03-30T14:54:43.351040+0000","last_became_peered":"2024-03-30T14:54:43.351040+0000","last_unstale":"2024-03-30T15:08:42.360634+0000","last_undegraded":"2024-03-30T15:08:42.360634+0000","last_fullsized":"2024-03-30T15:08:42.360634+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.951208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.951208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":522,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813109+0000","last_change":"2024-03-30T15:06:48.534844+0000","last_active":"2024-03-30T15:08:42.813109+0000","last_peered":"2024-03-30T15:08:42.813109+0000","last_clean":"2024-03-30T15:08:42.813109+0000","last_became_active":"2024-03-30T15:06:48.341420+0000","last_became_peered":"2024-03-30T15:06:48.341420+0000","last_unstale":"2024-03-30T15:08:42.813109+0000","last_undegraded":"2024-03-30T15:08:42.813109+0000","last_fullsized":"2024-03-30T15:08:42.813109+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.534763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.534763+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416677+0000","last_change":"2024-03-30T15:07:36.166030+0000","last_active":"2024-03-30T15:08:42.416677+0000","last_peered":"2024-03-30T15:08:42.416677+0000","last_clean":"2024-03-30T15:08:42.416677+0000","last_became_active":"2024-03-30T14:57:56.220479+0000","last_became_peered":"2024-03-30T14:57:56.220479+0000","last_unstale":"2024-03-30T15:08:42.416677+0000","last_undegraded":"2024-03-30T15:08:42.416677+0000","last_fullsized":"2024-03-30T15:08:42.416677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.165947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.824703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1235,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811641+0000","last_change":"2024-03-30T15:07:57.209970+0000","last_active":"2024-03-30T15:08:42.811641+0000","last_peered":"2024-03-30T15:08:42.811641+0000","last_clean":"2024-03-30T15:08:42.811641+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:08:42.811641+0000","last_undegraded":"2024-03-30T15:08:42.811641+0000","last_fullsized":"2024-03-30T15:08:42.811641+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.209884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.209884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:25.514345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1254,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641676+0000","last_change":"2024-03-30T15:08:10.679605+0000","last_active":"2024-03-30T15:08:42.641676+0000","last_peered":"2024-03-30T15:08:42.641676+0000","last_clean":"2024-03-30T15:08:42.641676+0000","last_became_active":"2024-03-30T14:57:56.275261+0000","last_became_peered":"2024-03-30T14:57:56.275261+0000","last_unstale":"2024-03-30T15:08:42.641676+0000","last_undegraded":"2024-03-30T15:08:42.641676+0000","last_fullsized":"2024-03-30T15:08:42.641676+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:08:10.679491+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.679491+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:39.573167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1219,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642558+0000","last_change":"2024-03-30T15:08:15.656962+0000","last_active":"2024-03-30T15:08:42.642558+0000","last_peered":"2024-03-30T15:08:42.642558+0000","last_clean":"2024-03-30T15:08:42.642558+0000","last_became_active":"2024-03-30T14:57:56.275730+0000","last_became_peered":"2024-03-30T14:57:56.275730+0000","last_unstale":"2024-03-30T15:08:42.642558+0000","last_undegraded":"2024-03-30T15:08:42.642558+0000","last_fullsized":"2024-03-30T15:08:42.642558+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:08:15.656821+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.656821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.329037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1231,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811834+0000","last_change":"2024-03-30T15:08:10.175409+0000","last_active":"2024-03-30T15:08:42.811834+0000","last_peered":"2024-03-30T15:08:42.811834+0000","last_clean":"2024-03-30T15:08:42.811834+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:08:42.811834+0000","last_undegraded":"2024-03-30T15:08:42.811834+0000","last_fullsized":"2024-03-30T15:08:42.811834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.175319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.192582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":521,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360145+0000","last_change":"2024-03-30T15:06:51.940521+0000","last_active":"2024-03-30T15:08:42.360145+0000","last_peered":"2024-03-30T15:08:42.360145+0000","last_clean":"2024-03-30T15:08:42.360145+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:08:42.360145+0000","last_undegraded":"2024-03-30T15:08:42.360145+0000","last_fullsized":"2024-03-30T15:08:42.360145+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:51.940375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:51.940375+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1160,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416801+0000","last_change":"2024-03-30T15:07:24.023700+0000","last_active":"2024-03-30T15:08:42.416801+0000","last_peered":"2024-03-30T15:08:42.416801+0000","last_clean":"2024-03-30T15:08:42.416801+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:08:42.416801+0000","last_undegraded":"2024-03-30T15:08:42.416801+0000","last_fullsized":"2024-03-30T15:08:42.416801+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.023617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.023617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1294,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808302+0000","last_change":"2024-03-30T15:06:11.179123+0000","last_active":"2024-03-30T15:08:42.808302+0000","last_peered":"2024-03-30T15:08:42.808302+0000","last_clean":"2024-03-30T15:08:42.808302+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:08:42.808302+0000","last_undegraded":"2024-03-30T15:08:42.808302+0000","last_fullsized":"2024-03-30T15:08:42.808302+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.179008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.179008+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1281,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811867+0000","last_change":"2024-03-30T15:06:41.454580+0000","last_active":"2024-03-30T15:08:42.811867+0000","last_peered":"2024-03-30T15:08:42.811867+0000","last_clean":"2024-03-30T15:08:42.811867+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:08:42.811867+0000","last_undegraded":"2024-03-30T15:08:42.811867+0000","last_fullsized":"2024-03-30T15:08:42.811867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:06:41.454445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:41.454445+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1240,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359432+0000","last_change":"2024-03-30T15:07:24.562748+0000","last_active":"2024-03-30T15:08:42.359432+0000","last_peered":"2024-03-30T15:08:42.359432+0000","last_clean":"2024-03-30T15:08:42.359432+0000","last_became_active":"2024-03-30T14:54:37.325852+0000","last_became_peered":"2024-03-30T14:54:37.325852+0000","last_unstale":"2024-03-30T15:08:42.359432+0000","last_undegraded":"2024-03-30T15:08:42.359432+0000","last_fullsized":"2024-03-30T15:08:42.359432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.710249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1233,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641859+0000","last_change":"2024-03-30T15:06:39.147213+0000","last_active":"2024-03-30T15:08:42.641859+0000","last_peered":"2024-03-30T15:08:42.641859+0000","last_clean":"2024-03-30T15:08:42.641859+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:08:42.641859+0000","last_undegraded":"2024-03-30T15:08:42.641859+0000","last_fullsized":"2024-03-30T15:08:42.641859+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.147113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.147113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":528,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903307+0000","last_change":"2024-03-30T15:06:49.387252+0000","last_active":"2024-03-30T15:08:42.903307+0000","last_peered":"2024-03-30T15:08:42.903307+0000","last_clean":"2024-03-30T15:08:42.903307+0000","last_became_active":"2024-03-30T15:06:48.342151+0000","last_became_peered":"2024-03-30T15:06:48.342151+0000","last_unstale":"2024-03-30T15:08:42.903307+0000","last_undegraded":"2024-03-30T15:08:42.903307+0000","last_fullsized":"2024-03-30T15:08:42.903307+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:49.387153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:49.387153+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1157,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416646+0000","last_change":"2024-03-30T15:07:25.015706+0000","last_active":"2024-03-30T15:08:42.416646+0000","last_peered":"2024-03-30T15:08:42.416646+0000","last_clean":"2024-03-30T15:08:42.416646+0000","last_became_active":"2024-03-30T14:57:50.298588+0000","last_became_peered":"2024-03-30T14:57:50.298588+0000","last_unstale":"2024-03-30T15:08:42.416646+0000","last_undegraded":"2024-03-30T15:08:42.416646+0000","last_fullsized":"2024-03-30T15:08:42.416646+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.015600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.015600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.857225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"76'8","reported_seq":1416,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358744+0000","last_change":"2024-03-30T15:05:16.606082+0000","last_active":"2024-03-30T15:08:42.358744+0000","last_peered":"2024-03-30T15:08:42.358744+0000","last_clean":"2024-03-30T15:08:42.358744+0000","last_became_active":"2024-03-30T14:54:41.344263+0000","last_became_peered":"2024-03-30T14:54:41.344263+0000","last_unstale":"2024-03-30T15:08:42.358744+0000","last_undegraded":"2024-03-30T15:08:42.358744+0000","last_fullsized":"2024-03-30T15:08:42.358744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1285,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808922+0000","last_change":"2024-03-30T15:08:01.102530+0000","last_active":"2024-03-30T15:08:42.808922+0000","last_peered":"2024-03-30T15:08:42.808922+0000","last_clean":"2024-03-30T15:08:42.808922+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:08:42.808922+0000","last_undegraded":"2024-03-30T15:08:42.808922+0000","last_fullsized":"2024-03-30T15:08:42.808922+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:08:01.102431+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.102431+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.685021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360866+0000","last_change":"2024-03-30T15:06:53.935771+0000","last_active":"2024-03-30T15:08:42.360866+0000","last_peered":"2024-03-30T15:08:42.360866+0000","last_clean":"2024-03-30T15:08:42.360866+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:08:42.360866+0000","last_undegraded":"2024-03-30T15:08:42.360866+0000","last_fullsized":"2024-03-30T15:08:42.360866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.935674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.935674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1226,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358017+0000","last_change":"2024-03-30T15:06:10.554067+0000","last_active":"2024-03-30T15:08:42.358017+0000","last_peered":"2024-03-30T15:08:42.358017+0000","last_clean":"2024-03-30T15:08:42.358017+0000","last_became_active":"2024-03-30T14:54:43.354837+0000","last_became_peered":"2024-03-30T14:54:43.354837+0000","last_unstale":"2024-03-30T15:08:42.358017+0000","last_undegraded":"2024-03-30T15:08:42.358017+0000","last_fullsized":"2024-03-30T15:08:42.358017+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":528,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903204+0000","last_change":"2024-03-30T15:06:48.409534+0000","last_active":"2024-03-30T15:08:42.903204+0000","last_peered":"2024-03-30T15:08:42.903204+0000","last_clean":"2024-03-30T15:08:42.903204+0000","last_became_active":"2024-03-30T15:06:48.338506+0000","last_became_peered":"2024-03-30T15:06:48.338506+0000","last_unstale":"2024-03-30T15:08:42.903204+0000","last_undegraded":"2024-03-30T15:08:42.903204+0000","last_fullsized":"2024-03-30T15:08:42.903204+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.409455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.409455+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416880+0000","last_change":"2024-03-30T15:06:59.884862+0000","last_active":"2024-03-30T15:08:42.416880+0000","last_peered":"2024-03-30T15:08:42.416880+0000","last_clean":"2024-03-30T15:08:42.416880+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:08:42.416880+0000","last_undegraded":"2024-03-30T15:08:42.416880+0000","last_fullsized":"2024-03-30T15:08:42.416880+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:59.884709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.884709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902323+0000","last_change":"2024-03-30T15:06:54.318139+0000","last_active":"2024-03-30T15:08:42.902323+0000","last_peered":"2024-03-30T15:08:42.902323+0000","last_clean":"2024-03-30T15:08:42.902323+0000","last_became_active":"2024-03-30T14:54:41.344789+0000","last_became_peered":"2024-03-30T14:54:41.344789+0000","last_unstale":"2024-03-30T15:08:42.902323+0000","last_undegraded":"2024-03-30T15:08:42.902323+0000","last_fullsized":"2024-03-30T15:08:42.902323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.318005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1293,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358981+0000","last_change":"2024-03-30T15:08:02.516526+0000","last_active":"2024-03-30T15:08:42.358981+0000","last_peered":"2024-03-30T15:08:42.358981+0000","last_clean":"2024-03-30T15:08:42.358981+0000","last_became_active":"2024-03-30T14:54:39.641815+0000","last_became_peered":"2024-03-30T14:54:39.641815+0000","last_unstale":"2024-03-30T15:08:42.358981+0000","last_undegraded":"2024-03-30T15:08:42.358981+0000","last_fullsized":"2024-03-30T15:08:42.358981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:14.340123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1254,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808913+0000","last_change":"2024-03-30T15:07:17.809348+0000","last_active":"2024-03-30T15:08:42.808913+0000","last_peered":"2024-03-30T15:08:42.808913+0000","last_clean":"2024-03-30T15:08:42.808913+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:08:42.808913+0000","last_undegraded":"2024-03-30T15:08:42.808913+0000","last_fullsized":"2024-03-30T15:08:42.808913+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:17.809228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:17.809228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1242,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808361+0000","last_change":"2024-03-30T15:06:39.844760+0000","last_active":"2024-03-30T15:08:42.808361+0000","last_peered":"2024-03-30T15:08:42.808361+0000","last_clean":"2024-03-30T15:08:42.808361+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:08:42.808361+0000","last_undegraded":"2024-03-30T15:08:42.808361+0000","last_fullsized":"2024-03-30T15:08:42.808361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:39.844668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:23.007527+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.844668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":523,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808475+0000","last_change":"2024-03-30T15:06:48.873065+0000","last_active":"2024-03-30T15:08:42.808475+0000","last_peered":"2024-03-30T15:08:42.808475+0000","last_clean":"2024-03-30T15:08:42.808475+0000","last_became_active":"2024-03-30T15:06:48.339258+0000","last_became_peered":"2024-03-30T15:06:48.339258+0000","last_unstale":"2024-03-30T15:08:42.808475+0000","last_undegraded":"2024-03-30T15:08:42.808475+0000","last_fullsized":"2024-03-30T15:08:42.808475+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.872974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.872974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359387+0000","last_change":"2024-03-30T15:05:17.575071+0000","last_active":"2024-03-30T15:08:42.359387+0000","last_peered":"2024-03-30T15:08:42.359387+0000","last_clean":"2024-03-30T15:08:42.359387+0000","last_became_active":"2024-03-30T14:57:54.015083+0000","last_became_peered":"2024-03-30T14:57:54.015083+0000","last_unstale":"2024-03-30T15:08:42.359387+0000","last_undegraded":"2024-03-30T15:08:42.359387+0000","last_fullsized":"2024-03-30T15:08:42.359387+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1235,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361631+0000","last_change":"2024-03-30T15:06:52.973865+0000","last_active":"2024-03-30T15:08:42.361631+0000","last_peered":"2024-03-30T15:08:42.361631+0000","last_clean":"2024-03-30T15:08:42.361631+0000","last_became_active":"2024-03-30T14:54:41.350968+0000","last_became_peered":"2024-03-30T14:54:41.350968+0000","last_unstale":"2024-03-30T15:08:42.361631+0000","last_undegraded":"2024-03-30T15:08:42.361631+0000","last_fullsized":"2024-03-30T15:08:42.361631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.973770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.973770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1278,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359262+0000","last_change":"2024-03-30T15:08:01.499074+0000","last_active":"2024-03-30T15:08:42.359262+0000","last_peered":"2024-03-30T15:08:42.359262+0000","last_clean":"2024-03-30T15:08:42.359262+0000","last_became_active":"2024-03-30T14:54:39.335673+0000","last_became_peered":"2024-03-30T14:54:39.335673+0000","last_unstale":"2024-03-30T15:08:42.359262+0000","last_undegraded":"2024-03-30T15:08:42.359262+0000","last_fullsized":"2024-03-30T15:08:42.359262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.212933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2097,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641458+0000","last_change":"2024-03-30T15:07:34.264898+0000","last_active":"2024-03-30T15:08:42.641458+0000","last_peered":"2024-03-30T15:08:42.641458+0000","last_clean":"2024-03-30T15:08:42.641458+0000","last_became_active":"2024-03-30T14:54:36.318086+0000","last_became_peered":"2024-03-30T14:54:36.318086+0000","last_unstale":"2024-03-30T15:08:42.641458+0000","last_undegraded":"2024-03-30T15:08:42.641458+0000","last_fullsized":"2024-03-30T15:08:42.641458+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:07:34.264815+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.264815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:04.246868+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":838,"num_read_kb":833,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1238,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358108+0000","last_change":"2024-03-30T15:07:29.592120+0000","last_active":"2024-03-30T15:08:42.358108+0000","last_peered":"2024-03-30T15:08:42.358108+0000","last_clean":"2024-03-30T15:08:42.358108+0000","last_became_active":"2024-03-30T14:54:37.325948+0000","last_became_peered":"2024-03-30T14:54:37.325948+0000","last_unstale":"2024-03-30T15:08:42.358108+0000","last_undegraded":"2024-03-30T15:08:42.358108+0000","last_fullsized":"2024-03-30T15:08:42.358108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.592039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.592039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:53.178771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808773+0000","last_change":"2024-03-30T15:06:45.854073+0000","last_active":"2024-03-30T15:08:42.808773+0000","last_peered":"2024-03-30T15:08:42.808773+0000","last_clean":"2024-03-30T15:08:42.808773+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:08:42.808773+0000","last_undegraded":"2024-03-30T15:08:42.808773+0000","last_fullsized":"2024-03-30T15:08:42.808773+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.853998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.853998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1141,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642343+0000","last_change":"2024-03-30T15:07:24.315658+0000","last_active":"2024-03-30T15:08:42.642343+0000","last_peered":"2024-03-30T15:08:42.642343+0000","last_clean":"2024-03-30T15:08:42.642343+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:08:42.642343+0000","last_undegraded":"2024-03-30T15:08:42.642343+0000","last_fullsized":"2024-03-30T15:08:42.642343+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.315584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.315584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:49.961946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1237,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642232+0000","last_change":"2024-03-30T15:08:06.704248+0000","last_active":"2024-03-30T15:08:42.642232+0000","last_peered":"2024-03-30T15:08:42.642232+0000","last_clean":"2024-03-30T15:08:42.642232+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:08:42.642232+0000","last_undegraded":"2024-03-30T15:08:42.642232+0000","last_fullsized":"2024-03-30T15:08:42.642232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.704123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.704123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:15.001588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1465,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.905141+0000","last_change":"2024-03-30T15:08:42.905186+0000","last_active":"2024-03-30T15:08:42.905141+0000","last_peered":"2024-03-30T15:08:42.905141+0000","last_clean":"2024-03-30T15:08:42.905141+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:08:42.905141+0000","last_undegraded":"2024-03-30T15:08:42.905141+0000","last_fullsized":"2024-03-30T15:08:42.905141+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:08:42.905060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.905060+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.005091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1138,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360162+0000","last_change":"2024-03-30T15:07:00.995237+0000","last_active":"2024-03-30T15:08:42.360162+0000","last_peered":"2024-03-30T15:08:42.360162+0000","last_clean":"2024-03-30T15:08:42.360162+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:08:42.360162+0000","last_undegraded":"2024-03-30T15:08:42.360162+0000","last_fullsized":"2024-03-30T15:08:42.360162+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.995156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.995156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1515,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641391+0000","last_change":"2024-03-30T15:06:45.099476+0000","last_active":"2024-03-30T15:08:42.641391+0000","last_peered":"2024-03-30T15:08:42.641391+0000","last_clean":"2024-03-30T15:08:42.641391+0000","last_became_active":"2024-03-30T15:06:43.342924+0000","last_became_peered":"2024-03-30T15:06:43.342924+0000","last_unstale":"2024-03-30T15:08:42.641391+0000","last_undegraded":"2024-03-30T15:08:42.641391+0000","last_fullsized":"2024-03-30T15:08:42.641391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:45.099372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.099372+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1223,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361450+0000","last_change":"2024-03-30T15:06:45.066707+0000","last_active":"2024-03-30T15:08:42.361450+0000","last_peered":"2024-03-30T15:08:42.361450+0000","last_clean":"2024-03-30T15:08:42.361450+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:08:42.361450+0000","last_undegraded":"2024-03-30T15:08:42.361450+0000","last_fullsized":"2024-03-30T15:08:42.361450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:45.066574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.066574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1531,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641293+0000","last_change":"2024-03-30T15:07:56.660010+0000","last_active":"2024-03-30T15:08:42.641293+0000","last_peered":"2024-03-30T15:08:42.641293+0000","last_clean":"2024-03-30T15:08:42.641293+0000","last_became_active":"2024-03-30T15:06:48.339784+0000","last_became_peered":"2024-03-30T15:06:48.339784+0000","last_unstale":"2024-03-30T15:08:42.641293+0000","last_undegraded":"2024-03-30T15:08:42.641293+0000","last_fullsized":"2024-03-30T15:08:42.641293+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:07:56.659850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:07:56.659850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.574408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1148,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903565+0000","last_change":"2024-03-30T15:06:10.008324+0000","last_active":"2024-03-30T15:08:42.903565+0000","last_peered":"2024-03-30T15:08:42.903565+0000","last_clean":"2024-03-30T15:08:42.903565+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:08:42.903565+0000","last_undegraded":"2024-03-30T15:08:42.903565+0000","last_fullsized":"2024-03-30T15:08:42.903565+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.008246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.008246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1256,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.417476+0000","last_change":"2024-03-30T15:08:14.240324+0000","last_active":"2024-03-30T15:08:42.417476+0000","last_peered":"2024-03-30T15:08:42.417476+0000","last_clean":"2024-03-30T15:08:42.417476+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:08:42.417476+0000","last_undegraded":"2024-03-30T15:08:42.417476+0000","last_fullsized":"2024-03-30T15:08:42.417476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:14.240226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:08:14.240226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.396355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1288,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809344+0000","last_change":"2024-03-30T15:06:44.835160+0000","last_active":"2024-03-30T15:08:42.809344+0000","last_peered":"2024-03-30T15:08:42.809344+0000","last_clean":"2024-03-30T15:08:42.809344+0000","last_became_active":"2024-03-30T14:54:39.337240+0000","last_became_peered":"2024-03-30T14:54:39.337240+0000","last_unstale":"2024-03-30T15:08:42.809344+0000","last_undegraded":"2024-03-30T15:08:42.809344+0000","last_fullsized":"2024-03-30T15:08:42.809344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:06:44.835068+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.835068+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1265,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416861+0000","last_change":"2024-03-30T15:06:57.786762+0000","last_active":"2024-03-30T15:08:42.416861+0000","last_peered":"2024-03-30T15:08:42.416861+0000","last_clean":"2024-03-30T15:08:42.416861+0000","last_became_active":"2024-03-30T14:54:36.316640+0000","last_became_peered":"2024-03-30T14:54:36.316640+0000","last_unstale":"2024-03-30T15:08:42.416861+0000","last_undegraded":"2024-03-30T15:08:42.416861+0000","last_fullsized":"2024-03-30T15:08:42.416861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.786685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.786685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1255,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808558+0000","last_change":"2024-03-30T15:07:23.937183+0000","last_active":"2024-03-30T15:08:42.808558+0000","last_peered":"2024-03-30T15:08:42.808558+0000","last_clean":"2024-03-30T15:08:42.808558+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:08:42.808558+0000","last_undegraded":"2024-03-30T15:08:42.808558+0000","last_fullsized":"2024-03-30T15:08:42.808558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.937096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.937096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1241,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642555+0000","last_change":"2024-03-30T15:07:58.653866+0000","last_active":"2024-03-30T15:08:42.642555+0000","last_peered":"2024-03-30T15:08:42.642555+0000","last_clean":"2024-03-30T15:08:42.642555+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:08:42.642555+0000","last_undegraded":"2024-03-30T15:08:42.642555+0000","last_fullsized":"2024-03-30T15:08:42.642555+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.653564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.653564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:19.992993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":521,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359572+0000","last_change":"2024-03-30T15:06:48.962204+0000","last_active":"2024-03-30T15:08:42.359572+0000","last_peered":"2024-03-30T15:08:42.359572+0000","last_clean":"2024-03-30T15:08:42.359572+0000","last_became_active":"2024-03-30T15:06:48.339451+0000","last_became_peered":"2024-03-30T15:06:48.339451+0000","last_unstale":"2024-03-30T15:08:42.359572+0000","last_undegraded":"2024-03-30T15:08:42.359572+0000","last_fullsized":"2024-03-30T15:08:42.359572+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.962124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.962124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358816+0000","last_change":"2024-03-30T15:05:20.548755+0000","last_active":"2024-03-30T15:08:42.358816+0000","last_peered":"2024-03-30T15:08:42.358816+0000","last_clean":"2024-03-30T15:08:42.358816+0000","last_became_active":"2024-03-30T14:57:54.024598+0000","last_became_peered":"2024-03-30T14:57:54.024598+0000","last_unstale":"2024-03-30T15:08:42.358816+0000","last_undegraded":"2024-03-30T15:08:42.358816+0000","last_fullsized":"2024-03-30T15:08:42.358816+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1238,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.814733+0000","last_change":"2024-03-30T15:08:42.814755+0000","last_active":"2024-03-30T15:08:42.814733+0000","last_peered":"2024-03-30T15:08:42.814733+0000","last_clean":"2024-03-30T15:08:42.814733+0000","last_became_active":"2024-03-30T14:54:41.349567+0000","last_became_peered":"2024-03-30T14:54:41.349567+0000","last_unstale":"2024-03-30T15:08:42.814733+0000","last_undegraded":"2024-03-30T15:08:42.814733+0000","last_fullsized":"2024-03-30T15:08:42.814733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:42.814674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.814674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:47.049446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1430,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.423012+0000","last_change":"2024-03-30T15:08:42.423012+0000","last_active":"2024-03-30T15:08:42.423012+0000","last_peered":"2024-03-30T15:08:42.423012+0000","last_clean":"2024-03-30T15:08:42.423012+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:08:42.423012+0000","last_undegraded":"2024-03-30T15:08:42.423012+0000","last_fullsized":"2024-03-30T15:08:42.423012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:08:42.422879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.422879+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:05.890471+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811997+0000","last_change":"2024-03-30T15:07:31.968746+0000","last_active":"2024-03-30T15:08:42.811997+0000","last_peered":"2024-03-30T15:08:42.811997+0000","last_clean":"2024-03-30T15:08:42.811997+0000","last_became_active":"2024-03-30T14:54:37.325382+0000","last_became_peered":"2024-03-30T14:54:37.325382+0000","last_unstale":"2024-03-30T15:08:42.811997+0000","last_undegraded":"2024-03-30T15:08:42.811997+0000","last_fullsized":"2024-03-30T15:08:42.811997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.259707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1246,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360360+0000","last_change":"2024-03-30T15:06:40.992863+0000","last_active":"2024-03-30T15:08:42.360360+0000","last_peered":"2024-03-30T15:08:42.360360+0000","last_clean":"2024-03-30T15:08:42.360360+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:08:42.360360+0000","last_undegraded":"2024-03-30T15:08:42.360360+0000","last_fullsized":"2024-03-30T15:08:42.360360+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:06:40.992758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.992758+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1143,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813234+0000","last_change":"2024-03-30T15:08:00.231743+0000","last_active":"2024-03-30T15:08:42.813234+0000","last_peered":"2024-03-30T15:08:42.813234+0000","last_clean":"2024-03-30T15:08:42.813234+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:08:42.813234+0000","last_undegraded":"2024-03-30T15:08:42.813234+0000","last_fullsized":"2024-03-30T15:08:42.813234+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.231663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.231663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:18.864990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1136,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358804+0000","last_change":"2024-03-30T15:05:19.578423+0000","last_active":"2024-03-30T15:08:42.358804+0000","last_peered":"2024-03-30T15:08:42.358804+0000","last_clean":"2024-03-30T15:08:42.358804+0000","last_became_active":"2024-03-30T14:57:54.045243+0000","last_became_peered":"2024-03-30T14:57:54.045243+0000","last_unstale":"2024-03-30T15:08:42.358804+0000","last_undegraded":"2024-03-30T15:08:42.358804+0000","last_fullsized":"2024-03-30T15:08:42.358804+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1267,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358795+0000","last_change":"2024-03-30T15:08:12.601879+0000","last_active":"2024-03-30T15:08:42.358795+0000","last_peered":"2024-03-30T15:08:42.358795+0000","last_clean":"2024-03-30T15:08:42.358795+0000","last_became_active":"2024-03-30T14:54:39.336625+0000","last_became_peered":"2024-03-30T14:54:39.336625+0000","last_unstale":"2024-03-30T15:08:42.358795+0000","last_undegraded":"2024-03-30T15:08:42.358795+0000","last_fullsized":"2024-03-30T15:08:42.358795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:16.593442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360874+0000","last_change":"2024-03-30T15:07:20.139612+0000","last_active":"2024-03-30T15:08:42.360874+0000","last_peered":"2024-03-30T15:08:42.360874+0000","last_clean":"2024-03-30T15:08:42.360874+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:08:42.360874+0000","last_undegraded":"2024-03-30T15:08:42.360874+0000","last_fullsized":"2024-03-30T15:08:42.360874+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:20.139530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.139530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.393853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1256,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416715+0000","last_change":"2024-03-30T15:07:58.247377+0000","last_active":"2024-03-30T15:08:42.416715+0000","last_peered":"2024-03-30T15:08:42.416715+0000","last_clean":"2024-03-30T15:08:42.416715+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:08:42.416715+0000","last_undegraded":"2024-03-30T15:08:42.416715+0000","last_fullsized":"2024-03-30T15:08:42.416715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.247271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.247271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:24.826144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1226,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358272+0000","last_change":"2024-03-30T15:07:21.573597+0000","last_active":"2024-03-30T15:08:42.358272+0000","last_peered":"2024-03-30T15:08:42.358272+0000","last_clean":"2024-03-30T15:08:42.358272+0000","last_became_active":"2024-03-30T14:54:43.351173+0000","last_became_peered":"2024-03-30T14:54:43.351173+0000","last_unstale":"2024-03-30T15:08:42.358272+0000","last_undegraded":"2024-03-30T15:08:42.358272+0000","last_fullsized":"2024-03-30T15:08:42.358272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.573522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.573522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1140,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812170+0000","last_change":"2024-03-30T15:07:22.909083+0000","last_active":"2024-03-30T15:08:42.812170+0000","last_peered":"2024-03-30T15:08:42.812170+0000","last_clean":"2024-03-30T15:08:42.812170+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:08:42.812170+0000","last_undegraded":"2024-03-30T15:08:42.812170+0000","last_fullsized":"2024-03-30T15:08:42.812170+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.908989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.908989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.320783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1328,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.901556+0000","last_change":"2024-03-30T15:06:44.482561+0000","last_active":"2024-03-30T15:08:42.901556+0000","last_peered":"2024-03-30T15:08:42.901556+0000","last_clean":"2024-03-30T15:08:42.901556+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:08:42.901556+0000","last_undegraded":"2024-03-30T15:08:42.901556+0000","last_fullsized":"2024-03-30T15:08:42.901556+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:06:44.482442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.482442+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642462+0000","last_change":"2024-03-30T15:07:18.376275+0000","last_active":"2024-03-30T15:08:42.642462+0000","last_peered":"2024-03-30T15:08:42.642462+0000","last_clean":"2024-03-30T15:08:42.642462+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:08:42.642462+0000","last_undegraded":"2024-03-30T15:08:42.642462+0000","last_fullsized":"2024-03-30T15:08:42.642462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.376197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.376197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1248,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416754+0000","last_change":"2024-03-30T15:06:48.713525+0000","last_active":"2024-03-30T15:08:42.416754+0000","last_peered":"2024-03-30T15:08:42.416754+0000","last_clean":"2024-03-30T15:08:42.416754+0000","last_became_active":"2024-03-30T14:54:41.341098+0000","last_became_peered":"2024-03-30T14:54:41.341098+0000","last_unstale":"2024-03-30T15:08:42.416754+0000","last_undegraded":"2024-03-30T15:08:42.416754+0000","last_fullsized":"2024-03-30T15:08:42.416754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:48.713441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.713441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1233,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641280+0000","last_change":"2024-03-30T15:08:11.701750+0000","last_active":"2024-03-30T15:08:42.641280+0000","last_peered":"2024-03-30T15:08:42.641280+0000","last_clean":"2024-03-30T15:08:42.641280+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:08:42.641280+0000","last_undegraded":"2024-03-30T15:08:42.641280+0000","last_fullsized":"2024-03-30T15:08:42.641280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.701619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.701619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.937509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1162,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.901801+0000","last_change":"2024-03-30T15:06:46.411474+0000","last_active":"2024-03-30T15:08:42.901801+0000","last_peered":"2024-03-30T15:08:42.901801+0000","last_clean":"2024-03-30T15:08:42.901801+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:08:42.901801+0000","last_undegraded":"2024-03-30T15:08:42.901801+0000","last_fullsized":"2024-03-30T15:08:42.901801+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:06:46.411383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:46.411383+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1248,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416512+0000","last_change":"2024-03-30T15:06:58.835912+0000","last_active":"2024-03-30T15:08:42.416512+0000","last_peered":"2024-03-30T15:08:42.416512+0000","last_clean":"2024-03-30T15:08:42.416512+0000","last_became_active":"2024-03-30T14:54:41.350167+0000","last_became_peered":"2024-03-30T14:54:41.350167+0000","last_unstale":"2024-03-30T15:08:42.416512+0000","last_undegraded":"2024-03-30T15:08:42.416512+0000","last_fullsized":"2024-03-30T15:08:42.416512+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.835804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.835804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1279,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812066+0000","last_change":"2024-03-30T15:08:08.213331+0000","last_active":"2024-03-30T15:08:42.812066+0000","last_peered":"2024-03-30T15:08:42.812066+0000","last_clean":"2024-03-30T15:08:42.812066+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:08:42.812066+0000","last_undegraded":"2024-03-30T15:08:42.812066+0000","last_fullsized":"2024-03-30T15:08:42.812066+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:08:08.213219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.213219+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.230614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642453+0000","last_change":"2024-03-30T15:08:04.783312+0000","last_active":"2024-03-30T15:08:42.642453+0000","last_peered":"2024-03-30T15:08:42.642453+0000","last_clean":"2024-03-30T15:08:42.642453+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:08:42.642453+0000","last_undegraded":"2024-03-30T15:08:42.642453+0000","last_fullsized":"2024-03-30T15:08:42.642453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:04.783237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.783237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:26.511079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1137,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641983+0000","last_change":"2024-03-30T15:06:54.229912+0000","last_active":"2024-03-30T15:08:42.641983+0000","last_peered":"2024-03-30T15:08:42.641983+0000","last_clean":"2024-03-30T15:08:42.641983+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:08:42.641983+0000","last_undegraded":"2024-03-30T15:08:42.641983+0000","last_fullsized":"2024-03-30T15:08:42.641983+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.229818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.229818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1140,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811376+0000","last_change":"2024-03-30T15:07:18.807780+0000","last_active":"2024-03-30T15:08:42.811376+0000","last_peered":"2024-03-30T15:08:42.811376+0000","last_clean":"2024-03-30T15:08:42.811376+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:08:42.811376+0000","last_undegraded":"2024-03-30T15:08:42.811376+0000","last_fullsized":"2024-03-30T15:08:42.811376+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.807666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.807666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:47.636769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1407,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641826+0000","last_change":"2024-03-30T15:06:08.222175+0000","last_active":"2024-03-30T15:08:42.641826+0000","last_peered":"2024-03-30T15:08:42.641826+0000","last_clean":"2024-03-30T15:08:42.641826+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:08:42.641826+0000","last_undegraded":"2024-03-30T15:08:42.641826+0000","last_fullsized":"2024-03-30T15:08:42.641826+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:08.222087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.222087+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1274,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359630+0000","last_change":"2024-03-30T15:06:16.030806+0000","last_active":"2024-03-30T15:08:42.359630+0000","last_peered":"2024-03-30T15:08:42.359630+0000","last_clean":"2024-03-30T15:08:42.359630+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:08:42.359630+0000","last_undegraded":"2024-03-30T15:08:42.359630+0000","last_fullsized":"2024-03-30T15:08:42.359630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:06:16.030694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:16.030694+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1257,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642675+0000","last_change":"2024-03-30T15:07:32.333233+0000","last_active":"2024-03-30T15:08:42.642675+0000","last_peered":"2024-03-30T15:08:42.642675+0000","last_clean":"2024-03-30T15:08:42.642675+0000","last_became_active":"2024-03-30T14:54:37.330458+0000","last_became_peered":"2024-03-30T14:54:37.330458+0000","last_unstale":"2024-03-30T15:08:42.642675+0000","last_undegraded":"2024-03-30T15:08:42.642675+0000","last_fullsized":"2024-03-30T15:08:42.642675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:07:32.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.333139+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1223,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360618+0000","last_change":"2024-03-30T15:06:01.048668+0000","last_active":"2024-03-30T15:08:42.360618+0000","last_peered":"2024-03-30T15:08:42.360618+0000","last_clean":"2024-03-30T15:08:42.360618+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:08:42.360618+0000","last_undegraded":"2024-03-30T15:08:42.360618+0000","last_fullsized":"2024-03-30T15:08:42.360618+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:01.048577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:06:01.048577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1149,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903459+0000","last_change":"2024-03-30T15:07:32.310952+0000","last_active":"2024-03-30T15:08:42.903459+0000","last_peered":"2024-03-30T15:08:42.903459+0000","last_clean":"2024-03-30T15:08:42.903459+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:08:42.903459+0000","last_undegraded":"2024-03-30T15:08:42.903459+0000","last_fullsized":"2024-03-30T15:08:42.903459+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.310831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.310831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":842,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.420429+0000","last_change":"2024-03-30T15:08:12.833026+0000","last_active":"2024-03-30T15:08:42.420429+0000","last_peered":"2024-03-30T15:08:42.420429+0000","last_clean":"2024-03-30T15:08:42.420429+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:08:42.420429+0000","last_undegraded":"2024-03-30T15:08:42.420429+0000","last_fullsized":"2024-03-30T15:08:42.420429+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:08:12.832934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.832934+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:41.769346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1259,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.901798+0000","last_change":"2024-03-30T15:07:58.498811+0000","last_active":"2024-03-30T15:08:42.901798+0000","last_peered":"2024-03-30T15:08:42.901798+0000","last_clean":"2024-03-30T15:08:42.901798+0000","last_became_active":"2024-03-30T14:54:37.326194+0000","last_became_peered":"2024-03-30T14:54:37.326194+0000","last_unstale":"2024-03-30T15:08:42.901798+0000","last_undegraded":"2024-03-30T15:08:42.901798+0000","last_fullsized":"2024-03-30T15:08:42.901798+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.498678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.498678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.745457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1226,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361006+0000","last_change":"2024-03-30T15:06:38.946768+0000","last_active":"2024-03-30T15:08:42.361006+0000","last_peered":"2024-03-30T15:08:42.361006+0000","last_clean":"2024-03-30T15:08:42.361006+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:08:42.361006+0000","last_undegraded":"2024-03-30T15:08:42.361006+0000","last_fullsized":"2024-03-30T15:08:42.361006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:38.946668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:38.946668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1224,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642164+0000","last_change":"2024-03-30T15:06:42.118047+0000","last_active":"2024-03-30T15:08:42.642164+0000","last_peered":"2024-03-30T15:08:42.642164+0000","last_clean":"2024-03-30T15:08:42.642164+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:08:42.642164+0000","last_undegraded":"2024-03-30T15:08:42.642164+0000","last_fullsized":"2024-03-30T15:08:42.642164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:42.117927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.117927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1140,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813224+0000","last_change":"2024-03-30T15:07:29.943796+0000","last_active":"2024-03-30T15:08:42.813224+0000","last_peered":"2024-03-30T15:08:42.813224+0000","last_clean":"2024-03-30T15:08:42.813224+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:08:42.813224+0000","last_undegraded":"2024-03-30T15:08:42.813224+0000","last_fullsized":"2024-03-30T15:08:42.813224+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.943712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.943712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.157038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1258,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359576+0000","last_change":"2024-03-30T15:07:34.554038+0000","last_active":"2024-03-30T15:08:42.359576+0000","last_peered":"2024-03-30T15:08:42.359576+0000","last_clean":"2024-03-30T15:08:42.359576+0000","last_became_active":"2024-03-30T14:54:39.641901+0000","last_became_peered":"2024-03-30T14:54:39.641901+0000","last_unstale":"2024-03-30T15:08:42.359576+0000","last_undegraded":"2024-03-30T15:08:42.359576+0000","last_fullsized":"2024-03-30T15:08:42.359576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:44.835409+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641612+0000","last_change":"2024-03-30T15:06:57.279563+0000","last_active":"2024-03-30T15:08:42.641612+0000","last_peered":"2024-03-30T15:08:42.641612+0000","last_clean":"2024-03-30T15:08:42.641612+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:08:42.641612+0000","last_undegraded":"2024-03-30T15:08:42.641612+0000","last_fullsized":"2024-03-30T15:08:42.641612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.279464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.279464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":331,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903435+0000","last_change":"2024-03-30T15:07:18.427089+0000","last_active":"2024-03-30T15:08:42.903435+0000","last_peered":"2024-03-30T15:08:42.903435+0000","last_clean":"2024-03-30T15:08:42.903435+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:08:42.903435+0000","last_undegraded":"2024-03-30T15:08:42.903435+0000","last_fullsized":"2024-03-30T15:08:42.903435+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:18.426974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:18.426974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.846808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1251,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.417239+0000","last_change":"2024-03-30T15:08:01.209737+0000","last_active":"2024-03-30T15:08:42.417239+0000","last_peered":"2024-03-30T15:08:42.417239+0000","last_clean":"2024-03-30T15:08:42.417239+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:08:42.417239+0000","last_undegraded":"2024-03-30T15:08:42.417239+0000","last_fullsized":"2024-03-30T15:08:42.417239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:01.209658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.209658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.903658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358916+0000","last_change":"2024-03-30T15:05:14.548003+0000","last_active":"2024-03-30T15:08:42.358916+0000","last_peered":"2024-03-30T15:08:42.358916+0000","last_clean":"2024-03-30T15:08:42.358916+0000","last_became_active":"2024-03-30T14:57:54.070117+0000","last_became_peered":"2024-03-30T14:57:54.070117+0000","last_unstale":"2024-03-30T15:08:42.358916+0000","last_undegraded":"2024-03-30T15:08:42.358916+0000","last_fullsized":"2024-03-30T15:08:42.358916+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1255,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361100+0000","last_change":"2024-03-30T15:06:39.943308+0000","last_active":"2024-03-30T15:08:42.361100+0000","last_peered":"2024-03-30T15:08:42.361100+0000","last_clean":"2024-03-30T15:08:42.361100+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:08:42.361100+0000","last_undegraded":"2024-03-30T15:08:42.361100+0000","last_fullsized":"2024-03-30T15:08:42.361100+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:39.943172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:39.943172+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1238,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358445+0000","last_change":"2024-03-30T15:04:36.536999+0000","last_active":"2024-03-30T15:08:42.358445+0000","last_peered":"2024-03-30T15:08:42.358445+0000","last_clean":"2024-03-30T15:08:42.358445+0000","last_became_active":"2024-03-30T14:54:37.323431+0000","last_became_peered":"2024-03-30T14:54:37.323431+0000","last_unstale":"2024-03-30T15:08:42.358445+0000","last_undegraded":"2024-03-30T15:08:42.358445+0000","last_fullsized":"2024-03-30T15:08:42.358445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1292,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642168+0000","last_change":"2024-03-30T15:06:11.222500+0000","last_active":"2024-03-30T15:08:42.642168+0000","last_peered":"2024-03-30T15:08:42.642168+0000","last_clean":"2024-03-30T15:08:42.642168+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:08:42.642168+0000","last_undegraded":"2024-03-30T15:08:42.642168+0000","last_fullsized":"2024-03-30T15:08:42.642168+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:06:11.222382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.222382+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1222,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813441+0000","last_change":"2024-03-30T15:06:58.575629+0000","last_active":"2024-03-30T15:08:42.813441+0000","last_peered":"2024-03-30T15:08:42.813441+0000","last_clean":"2024-03-30T15:08:42.813441+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:08:42.813441+0000","last_undegraded":"2024-03-30T15:08:42.813441+0000","last_fullsized":"2024-03-30T15:08:42.813441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.575544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.575544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156858+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358732+0000","last_change":"2024-03-30T15:05:18.568817+0000","last_active":"2024-03-30T15:08:42.358732+0000","last_peered":"2024-03-30T15:08:42.358732+0000","last_clean":"2024-03-30T15:08:42.358732+0000","last_became_active":"2024-03-30T14:57:54.020195+0000","last_became_peered":"2024-03-30T14:57:54.020195+0000","last_unstale":"2024-03-30T15:08:42.358732+0000","last_undegraded":"2024-03-30T15:08:42.358732+0000","last_fullsized":"2024-03-30T15:08:42.358732+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":1195,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808985+0000","last_change":"2024-03-30T15:06:59.844000+0000","last_active":"2024-03-30T15:08:42.808985+0000","last_peered":"2024-03-30T15:08:42.808985+0000","last_clean":"2024-03-30T15:08:42.808985+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:08:42.808985+0000","last_undegraded":"2024-03-30T15:08:42.808985+0000","last_fullsized":"2024-03-30T15:08:42.808985+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:06:59.843847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:59.843847+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641713+0000","last_change":"2024-03-30T15:08:17.645966+0000","last_active":"2024-03-30T15:08:42.641713+0000","last_peered":"2024-03-30T15:08:42.641713+0000","last_clean":"2024-03-30T15:08:42.641713+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:08:42.641713+0000","last_undegraded":"2024-03-30T15:08:42.641713+0000","last_fullsized":"2024-03-30T15:08:42.641713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.645871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.645871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:45.597570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"76'8","reported_seq":1407,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361308+0000","last_change":"2024-03-30T15:06:44.020947+0000","last_active":"2024-03-30T15:08:42.361308+0000","last_peered":"2024-03-30T15:08:42.361308+0000","last_clean":"2024-03-30T15:08:42.361308+0000","last_became_active":"2024-03-30T14:54:41.350177+0000","last_became_peered":"2024-03-30T14:54:41.350177+0000","last_unstale":"2024-03-30T15:08:42.361308+0000","last_undegraded":"2024-03-30T15:08:42.361308+0000","last_fullsized":"2024-03-30T15:08:42.361308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:44.020829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.020829+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1227,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359242+0000","last_change":"2024-03-30T15:07:30.551177+0000","last_active":"2024-03-30T15:08:42.359242+0000","last_peered":"2024-03-30T15:08:42.359242+0000","last_clean":"2024-03-30T15:08:42.359242+0000","last_became_active":"2024-03-30T14:54:43.354050+0000","last_became_peered":"2024-03-30T14:54:43.354050+0000","last_unstale":"2024-03-30T15:08:42.359242+0000","last_undegraded":"2024-03-30T15:08:42.359242+0000","last_fullsized":"2024-03-30T15:08:42.359242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:07:30.551096+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.551096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1139,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812151+0000","last_change":"2024-03-30T15:07:21.866188+0000","last_active":"2024-03-30T15:08:42.812151+0000","last_peered":"2024-03-30T15:08:42.812151+0000","last_clean":"2024-03-30T15:08:42.812151+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:08:42.812151+0000","last_undegraded":"2024-03-30T15:08:42.812151+0000","last_fullsized":"2024-03-30T15:08:42.812151+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.866106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.866106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:48.392301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1267,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812237+0000","last_change":"2024-03-30T15:06:54.562492+0000","last_active":"2024-03-30T15:08:42.812237+0000","last_peered":"2024-03-30T15:08:42.812237+0000","last_clean":"2024-03-30T15:08:42.812237+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:08:42.812237+0000","last_undegraded":"2024-03-30T15:08:42.812237+0000","last_fullsized":"2024-03-30T15:08:42.812237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:06:54.562347+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.562347+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812181+0000","last_change":"2024-03-30T15:07:00.555778+0000","last_active":"2024-03-30T15:08:42.812181+0000","last_peered":"2024-03-30T15:08:42.812181+0000","last_clean":"2024-03-30T15:08:42.812181+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:08:42.812181+0000","last_undegraded":"2024-03-30T15:08:42.812181+0000","last_fullsized":"2024-03-30T15:08:42.812181+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:00.555700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:00.555700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1246,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808840+0000","last_change":"2024-03-30T15:07:35.184388+0000","last_active":"2024-03-30T15:08:42.808840+0000","last_peered":"2024-03-30T15:08:42.808840+0000","last_clean":"2024-03-30T15:08:42.808840+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:08:42.808840+0000","last_undegraded":"2024-03-30T15:08:42.808840+0000","last_fullsized":"2024-03-30T15:08:42.808840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1138,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358146+0000","last_change":"2024-03-30T15:08:11.553825+0000","last_active":"2024-03-30T15:08:42.358146+0000","last_peered":"2024-03-30T15:08:42.358146+0000","last_clean":"2024-03-30T15:08:42.358146+0000","last_became_active":"2024-03-30T14:57:56.293047+0000","last_became_peered":"2024-03-30T14:57:56.293047+0000","last_unstale":"2024-03-30T15:08:42.358146+0000","last_undegraded":"2024-03-30T15:08:42.358146+0000","last_fullsized":"2024-03-30T15:08:42.358146+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:22.326045+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1139,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811676+0000","last_change":"2024-03-30T15:07:24.946870+0000","last_active":"2024-03-30T15:08:42.811676+0000","last_peered":"2024-03-30T15:08:42.811676+0000","last_clean":"2024-03-30T15:08:42.811676+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:08:42.811676+0000","last_undegraded":"2024-03-30T15:08:42.811676+0000","last_fullsized":"2024-03-30T15:08:42.811676+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:24.946762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.946762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:54.723932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1267,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808961+0000","last_change":"2024-03-30T15:07:20.851744+0000","last_active":"2024-03-30T15:08:42.808961+0000","last_peered":"2024-03-30T15:08:42.808961+0000","last_clean":"2024-03-30T15:08:42.808961+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:08:42.808961+0000","last_undegraded":"2024-03-30T15:08:42.808961+0000","last_fullsized":"2024-03-30T15:08:42.808961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:20.851659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:20.851659+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1259,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642241+0000","last_change":"2024-03-30T15:07:33.297905+0000","last_active":"2024-03-30T15:08:42.642241+0000","last_peered":"2024-03-30T15:08:42.642241+0000","last_clean":"2024-03-30T15:08:42.642241+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:08:42.642241+0000","last_undegraded":"2024-03-30T15:08:42.642241+0000","last_fullsized":"2024-03-30T15:08:42.642241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.297820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:46.464840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":401,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811697+0000","last_change":"2024-03-30T15:06:15.461799+0000","last_active":"2024-03-30T15:08:42.811697+0000","last_peered":"2024-03-30T15:08:42.811697+0000","last_clean":"2024-03-30T15:08:42.811697+0000","last_became_active":"2024-03-30T15:04:37.297765+0000","last_became_peered":"2024-03-30T15:04:37.297765+0000","last_unstale":"2024-03-30T15:08:42.811697+0000","last_undegraded":"2024-03-30T15:08:42.811697+0000","last_fullsized":"2024-03-30T15:08:42.811697+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:15.461699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:15.461699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1226,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358046+0000","last_change":"2024-03-30T15:06:53.604745+0000","last_active":"2024-03-30T15:08:42.358046+0000","last_peered":"2024-03-30T15:08:42.358046+0000","last_clean":"2024-03-30T15:08:42.358046+0000","last_became_active":"2024-03-30T14:54:43.351017+0000","last_became_peered":"2024-03-30T14:54:43.351017+0000","last_unstale":"2024-03-30T15:08:42.358046+0000","last_undegraded":"2024-03-30T15:08:42.358046+0000","last_fullsized":"2024-03-30T15:08:42.358046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358038+0000","last_change":"2024-03-30T15:04:49.538900+0000","last_active":"2024-03-30T15:08:42.358038+0000","last_peered":"2024-03-30T15:08:42.358038+0000","last_clean":"2024-03-30T15:08:42.358038+0000","last_became_active":"2024-03-30T14:57:54.070456+0000","last_became_peered":"2024-03-30T14:57:54.070456+0000","last_unstale":"2024-03-30T15:08:42.358038+0000","last_undegraded":"2024-03-30T15:08:42.358038+0000","last_fullsized":"2024-03-30T15:08:42.358038+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1187,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641740+0000","last_change":"2024-03-30T15:06:06.233170+0000","last_active":"2024-03-30T15:08:42.641740+0000","last_peered":"2024-03-30T15:08:42.641740+0000","last_clean":"2024-03-30T15:08:42.641740+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:08:42.641740+0000","last_undegraded":"2024-03-30T15:08:42.641740+0000","last_fullsized":"2024-03-30T15:08:42.641740+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:06:06.233057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.233057+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1242,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:43.367857+0000","last_change":"2024-03-30T15:08:43.367882+0000","last_active":"2024-03-30T15:08:43.367857+0000","last_peered":"2024-03-30T15:08:43.367857+0000","last_clean":"2024-03-30T15:08:43.367857+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:08:43.367857+0000","last_undegraded":"2024-03-30T15:08:43.367857+0000","last_fullsized":"2024-03-30T15:08:43.367857+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:43.367786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:43.367786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:07.779164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1237,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641589+0000","last_change":"2024-03-30T15:08:07.697978+0000","last_active":"2024-03-30T15:08:42.641589+0000","last_peered":"2024-03-30T15:08:42.641589+0000","last_clean":"2024-03-30T15:08:42.641589+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:08:42.641589+0000","last_undegraded":"2024-03-30T15:08:42.641589+0000","last_fullsized":"2024-03-30T15:08:42.641589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:07.697832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:07.697832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:09.589928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1138,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358459+0000","last_change":"2024-03-30T15:05:55.573946+0000","last_active":"2024-03-30T15:08:42.358459+0000","last_peered":"2024-03-30T15:08:42.358459+0000","last_clean":"2024-03-30T15:08:42.358459+0000","last_became_active":"2024-03-30T14:57:56.220372+0000","last_became_peered":"2024-03-30T14:57:56.220372+0000","last_unstale":"2024-03-30T15:08:42.358459+0000","last_undegraded":"2024-03-30T15:08:42.358459+0000","last_fullsized":"2024-03-30T15:08:42.358459+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1151,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809054+0000","last_change":"2024-03-30T15:08:08.177938+0000","last_active":"2024-03-30T15:08:42.809054+0000","last_peered":"2024-03-30T15:08:42.809054+0000","last_clean":"2024-03-30T15:08:42.809054+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:08:42.809054+0000","last_undegraded":"2024-03-30T15:08:42.809054+0000","last_fullsized":"2024-03-30T15:08:42.809054+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.177858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.177858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.773779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":821,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.419229+0000","last_change":"2024-03-30T15:07:22.646408+0000","last_active":"2024-03-30T15:08:42.419229+0000","last_peered":"2024-03-30T15:08:42.419229+0000","last_clean":"2024-03-30T15:08:42.419229+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:08:42.419229+0000","last_undegraded":"2024-03-30T15:08:42.419229+0000","last_fullsized":"2024-03-30T15:08:42.419229+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:07:22.646314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.646314+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359636+0000","last_change":"2024-03-30T15:08:10.508709+0000","last_active":"2024-03-30T15:08:42.359636+0000","last_peered":"2024-03-30T15:08:42.359636+0000","last_clean":"2024-03-30T15:08:42.359636+0000","last_became_active":"2024-03-30T14:54:37.323684+0000","last_became_peered":"2024-03-30T14:54:37.323684+0000","last_unstale":"2024-03-30T15:08:42.359636+0000","last_undegraded":"2024-03-30T15:08:42.359636+0000","last_fullsized":"2024-03-30T15:08:42.359636+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:36.166959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1243,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.902175+0000","last_change":"2024-03-30T15:06:06.027362+0000","last_active":"2024-03-30T15:08:42.902175+0000","last_peered":"2024-03-30T15:08:42.902175+0000","last_clean":"2024-03-30T15:08:42.902175+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:08:42.902175+0000","last_undegraded":"2024-03-30T15:08:42.902175+0000","last_fullsized":"2024-03-30T15:08:42.902175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:06.027259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:06.027259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1225,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641610+0000","last_change":"2024-03-30T15:06:07.190512+0000","last_active":"2024-03-30T15:08:42.641610+0000","last_peered":"2024-03-30T15:08:42.641610+0000","last_clean":"2024-03-30T15:08:42.641610+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:08:42.641610+0000","last_undegraded":"2024-03-30T15:08:42.641610+0000","last_fullsized":"2024-03-30T15:08:42.641610+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.190363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.190363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1148,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903674+0000","last_change":"2024-03-30T15:07:33.284903+0000","last_active":"2024-03-30T15:08:42.903674+0000","last_peered":"2024-03-30T15:08:42.903674+0000","last_clean":"2024-03-30T15:08:42.903674+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:08:42.903674+0000","last_undegraded":"2024-03-30T15:08:42.903674+0000","last_fullsized":"2024-03-30T15:08:42.903674+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.284814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.284814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:00.915368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1283,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903680+0000","last_change":"2024-03-30T15:08:16.603568+0000","last_active":"2024-03-30T15:08:42.903680+0000","last_peered":"2024-03-30T15:08:42.903680+0000","last_clean":"2024-03-30T15:08:42.903680+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:08:42.903680+0000","last_undegraded":"2024-03-30T15:08:42.903680+0000","last_fullsized":"2024-03-30T15:08:42.903680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:16.603468+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:08:16.603468+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:21.969813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1139,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358121+0000","last_change":"2024-03-30T15:08:00.523833+0000","last_active":"2024-03-30T15:08:42.358121+0000","last_peered":"2024-03-30T15:08:42.358121+0000","last_clean":"2024-03-30T15:08:42.358121+0000","last_became_active":"2024-03-30T14:57:56.242275+0000","last_became_peered":"2024-03-30T14:57:56.242275+0000","last_unstale":"2024-03-30T15:08:42.358121+0000","last_undegraded":"2024-03-30T15:08:42.358121+0000","last_fullsized":"2024-03-30T15:08:42.358121+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:20.174749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1230,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359629+0000","last_change":"2024-03-30T15:07:28.601931+0000","last_active":"2024-03-30T15:08:42.359629+0000","last_peered":"2024-03-30T15:08:42.359629+0000","last_clean":"2024-03-30T15:08:42.359629+0000","last_became_active":"2024-03-30T14:54:41.346957+0000","last_became_peered":"2024-03-30T14:54:41.346957+0000","last_unstale":"2024-03-30T15:08:42.359629+0000","last_undegraded":"2024-03-30T15:08:42.359629+0000","last_fullsized":"2024-03-30T15:08:42.359629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.601815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.601815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:50.989405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1241,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809197+0000","last_change":"2024-03-30T15:08:02.083375+0000","last_active":"2024-03-30T15:08:42.809197+0000","last_peered":"2024-03-30T15:08:42.809197+0000","last_clean":"2024-03-30T15:08:42.809197+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:08:42.809197+0000","last_undegraded":"2024-03-30T15:08:42.809197+0000","last_fullsized":"2024-03-30T15:08:42.809197+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.083294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.083294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:10.435219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1156,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903460+0000","last_change":"2024-03-30T15:08:41.886052+0000","last_active":"2024-03-30T15:08:42.903460+0000","last_peered":"2024-03-30T15:08:42.903460+0000","last_clean":"2024-03-30T15:08:42.903460+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:08:42.903460+0000","last_undegraded":"2024-03-30T15:08:42.903460+0000","last_fullsized":"2024-03-30T15:08:42.903460+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:41.885921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:08:41.885921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:42.761764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1270,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359530+0000","last_change":"2024-03-30T15:07:25.577161+0000","last_active":"2024-03-30T15:08:42.359530+0000","last_peered":"2024-03-30T15:08:42.359530+0000","last_clean":"2024-03-30T15:08:42.359530+0000","last_became_active":"2024-03-30T14:54:39.641944+0000","last_became_peered":"2024-03-30T14:54:39.641944+0000","last_unstale":"2024-03-30T15:08:42.359530+0000","last_undegraded":"2024-03-30T15:08:42.359530+0000","last_fullsized":"2024-03-30T15:08:42.359530+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:07:25.577074+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.577074+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1247,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808539+0000","last_change":"2024-03-30T15:07:30.031279+0000","last_active":"2024-03-30T15:08:42.808539+0000","last_peered":"2024-03-30T15:08:42.808539+0000","last_clean":"2024-03-30T15:08:42.808539+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:08:42.808539+0000","last_undegraded":"2024-03-30T15:08:42.808539+0000","last_fullsized":"2024-03-30T15:08:42.808539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.031154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1432,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903565+0000","last_change":"2024-03-30T15:08:04.561718+0000","last_active":"2024-03-30T15:08:42.903565+0000","last_peered":"2024-03-30T15:08:42.903565+0000","last_clean":"2024-03-30T15:08:42.903565+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:08:42.903565+0000","last_undegraded":"2024-03-30T15:08:42.903565+0000","last_fullsized":"2024-03-30T15:08:42.903565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:04.561610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.561610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:28.847750+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.642137+0000","last_change":"2024-03-30T15:06:48.193765+0000","last_active":"2024-03-30T15:08:42.642137+0000","last_peered":"2024-03-30T15:08:42.642137+0000","last_clean":"2024-03-30T15:08:42.642137+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:08:42.642137+0000","last_undegraded":"2024-03-30T15:08:42.642137+0000","last_fullsized":"2024-03-30T15:08:42.642137+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:06:48.193663+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.193663+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1156,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903196+0000","last_change":"2024-03-30T15:07:57.453748+0000","last_active":"2024-03-30T15:08:42.903196+0000","last_peered":"2024-03-30T15:08:42.903196+0000","last_clean":"2024-03-30T15:08:42.903196+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:08:42.903196+0000","last_undegraded":"2024-03-30T15:08:42.903196+0000","last_fullsized":"2024-03-30T15:08:42.903196+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.453629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.453629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.047448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1197,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808926+0000","last_change":"2024-03-30T15:07:21.894935+0000","last_active":"2024-03-30T15:08:42.808926+0000","last_peered":"2024-03-30T15:08:42.808926+0000","last_clean":"2024-03-30T15:08:42.808926+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:08:42.808926+0000","last_undegraded":"2024-03-30T15:08:42.808926+0000","last_fullsized":"2024-03-30T15:08:42.808926+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:07:21.894855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.894855+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811995+0000","last_change":"2024-03-30T15:06:08.565843+0000","last_active":"2024-03-30T15:08:42.811995+0000","last_peered":"2024-03-30T15:08:42.811995+0000","last_clean":"2024-03-30T15:08:42.811995+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:08:42.811995+0000","last_undegraded":"2024-03-30T15:08:42.811995+0000","last_fullsized":"2024-03-30T15:08:42.811995+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:08.565745+0000","last_clean_scrub_stamp":"2024-03-30T15:06:08.565745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156688+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1226,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.812286+0000","last_change":"2024-03-30T15:06:02.773008+0000","last_active":"2024-03-30T15:08:42.812286+0000","last_peered":"2024-03-30T15:08:42.812286+0000","last_clean":"2024-03-30T15:08:42.812286+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:08:42.812286+0000","last_undegraded":"2024-03-30T15:08:42.812286+0000","last_fullsized":"2024-03-30T15:08:42.812286+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:02.772909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:02.772909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1153,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903765+0000","last_change":"2024-03-30T15:07:36.255445+0000","last_active":"2024-03-30T15:08:42.903765+0000","last_peered":"2024-03-30T15:08:42.903765+0000","last_clean":"2024-03-30T15:08:42.903765+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:08:42.903765+0000","last_undegraded":"2024-03-30T15:08:42.903765+0000","last_fullsized":"2024-03-30T15:08:42.903765+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.255369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.255369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:58.024102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1140,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.361192+0000","last_change":"2024-03-30T15:07:28.134811+0000","last_active":"2024-03-30T15:08:42.361192+0000","last_peered":"2024-03-30T15:08:42.361192+0000","last_clean":"2024-03-30T15:08:42.361192+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:08:42.361192+0000","last_undegraded":"2024-03-30T15:08:42.361192+0000","last_fullsized":"2024-03-30T15:08:42.361192+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.134705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.134705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.521025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1309,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.417022+0000","last_change":"2024-03-30T15:07:59.295056+0000","last_active":"2024-03-30T15:08:42.417022+0000","last_peered":"2024-03-30T15:08:42.417022+0000","last_clean":"2024-03-30T15:08:42.417022+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:08:42.417022+0000","last_undegraded":"2024-03-30T15:08:42.417022+0000","last_fullsized":"2024-03-30T15:08:42.417022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:59.294962+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.294962+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:27.470863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1237,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358935+0000","last_change":"2024-03-30T15:04:42.537870+0000","last_active":"2024-03-30T15:08:42.358935+0000","last_peered":"2024-03-30T15:08:42.358935+0000","last_clean":"2024-03-30T15:08:42.358935+0000","last_became_active":"2024-03-30T14:54:37.325909+0000","last_became_peered":"2024-03-30T14:54:37.325909+0000","last_unstale":"2024-03-30T15:08:42.358935+0000","last_undegraded":"2024-03-30T15:08:42.358935+0000","last_fullsized":"2024-03-30T15:08:42.358935+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1245,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.809058+0000","last_change":"2024-03-30T15:08:00.131070+0000","last_active":"2024-03-30T15:08:42.809058+0000","last_peered":"2024-03-30T15:08:42.809058+0000","last_clean":"2024-03-30T15:08:42.809058+0000","last_became_active":"2024-03-30T14:54:41.347409+0000","last_became_peered":"2024-03-30T14:54:41.347409+0000","last_unstale":"2024-03-30T15:08:42.809058+0000","last_undegraded":"2024-03-30T15:08:42.809058+0000","last_fullsized":"2024-03-30T15:08:42.809058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.130991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.130991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:06.398245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1223,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.813331+0000","last_change":"2024-03-30T15:06:12.558953+0000","last_active":"2024-03-30T15:08:42.813331+0000","last_peered":"2024-03-30T15:08:42.813331+0000","last_clean":"2024-03-30T15:08:42.813331+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:08:42.813331+0000","last_undegraded":"2024-03-30T15:08:42.813331+0000","last_fullsized":"2024-03-30T15:08:42.813331+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:12.558834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:12.558834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1151,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808674+0000","last_change":"2024-03-30T15:08:21.207838+0000","last_active":"2024-03-30T15:08:42.808674+0000","last_peered":"2024-03-30T15:08:42.808674+0000","last_clean":"2024-03-30T15:08:42.808674+0000","last_became_active":"2024-03-30T14:57:54.079838+0000","last_became_peered":"2024-03-30T14:57:54.079838+0000","last_unstale":"2024-03-30T15:08:42.808674+0000","last_undegraded":"2024-03-30T15:08:42.808674+0000","last_fullsized":"2024-03-30T15:08:42.808674+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.207705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:29.807245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1267,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811179+0000","last_change":"2024-03-30T15:06:50.584525+0000","last_active":"2024-03-30T15:08:42.811179+0000","last_peered":"2024-03-30T15:08:42.811179+0000","last_clean":"2024-03-30T15:08:42.811179+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:08:42.811179+0000","last_undegraded":"2024-03-30T15:08:42.811179+0000","last_fullsized":"2024-03-30T15:08:42.811179+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:06:50.584362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:50.584362+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360754+0000","last_change":"2024-03-30T15:06:14.990566+0000","last_active":"2024-03-30T15:08:42.360754+0000","last_peered":"2024-03-30T15:08:42.360754+0000","last_clean":"2024-03-30T15:08:42.360754+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:08:42.360754+0000","last_undegraded":"2024-03-30T15:08:42.360754+0000","last_fullsized":"2024-03-30T15:08:42.360754+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.990472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.990472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1236,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.641554+0000","last_change":"2024-03-30T15:08:18.689134+0000","last_active":"2024-03-30T15:08:42.641554+0000","last_peered":"2024-03-30T15:08:42.641554+0000","last_clean":"2024-03-30T15:08:42.641554+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:08:42.641554+0000","last_undegraded":"2024-03-30T15:08:42.641554+0000","last_fullsized":"2024-03-30T15:08:42.641554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:18.688995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:18.688995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:47.478630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1230,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359855+0000","last_change":"2024-03-30T15:07:23.180715+0000","last_active":"2024-03-30T15:08:42.359855+0000","last_peered":"2024-03-30T15:08:42.359855+0000","last_clean":"2024-03-30T15:08:42.359855+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:08:42.359855+0000","last_undegraded":"2024-03-30T15:08:42.359855+0000","last_fullsized":"2024-03-30T15:08:42.359855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:23.180625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:07:23.180625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1137,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808851+0000","last_change":"2024-03-30T15:06:14.104026+0000","last_active":"2024-03-30T15:08:42.808851+0000","last_peered":"2024-03-30T15:08:42.808851+0000","last_clean":"2024-03-30T15:08:42.808851+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:08:42.808851+0000","last_undegraded":"2024-03-30T15:08:42.808851+0000","last_fullsized":"2024-03-30T15:08:42.808851+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:14.103910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:06:14.103910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1255,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811962+0000","last_change":"2024-03-30T15:06:42.502254+0000","last_active":"2024-03-30T15:08:42.811962+0000","last_peered":"2024-03-30T15:08:42.811962+0000","last_clean":"2024-03-30T15:08:42.811962+0000","last_became_active":"2024-03-30T14:54:39.329505+0000","last_became_peered":"2024-03-30T14:54:39.329505+0000","last_unstale":"2024-03-30T15:08:42.811962+0000","last_undegraded":"2024-03-30T15:08:42.811962+0000","last_fullsized":"2024-03-30T15:08:42.811962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:42.502138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.502138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156744+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360716+0000","last_change":"2024-03-30T15:06:43.006252+0000","last_active":"2024-03-30T15:08:42.360716+0000","last_peered":"2024-03-30T15:08:42.360716+0000","last_clean":"2024-03-30T15:08:42.360716+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:08:42.360716+0000","last_undegraded":"2024-03-30T15:08:42.360716+0000","last_fullsized":"2024-03-30T15:08:42.360716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:43.006124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:06:43.006124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1234,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.360182+0000","last_change":"2024-03-30T15:08:20.313597+0000","last_active":"2024-03-30T15:08:42.360182+0000","last_peered":"2024-03-30T15:08:42.360182+0000","last_clean":"2024-03-30T15:08:42.360182+0000","last_became_active":"2024-03-30T14:54:41.348204+0000","last_became_peered":"2024-03-30T14:54:41.348204+0000","last_unstale":"2024-03-30T15:08:42.360182+0000","last_undegraded":"2024-03-30T15:08:42.360182+0000","last_fullsized":"2024-03-30T15:08:42.360182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_clean_scrub_stamp":"2024-03-30T15:08:20.313498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:34.565879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1225,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358214+0000","last_change":"2024-03-30T15:04:47.569089+0000","last_active":"2024-03-30T15:08:42.358214+0000","last_peered":"2024-03-30T15:08:42.358214+0000","last_clean":"2024-03-30T15:08:42.358214+0000","last_became_active":"2024-03-30T14:54:43.351578+0000","last_became_peered":"2024-03-30T14:54:43.351578+0000","last_unstale":"2024-03-30T15:08:42.358214+0000","last_undegraded":"2024-03-30T15:08:42.358214+0000","last_fullsized":"2024-03-30T15:08:42.358214+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1140,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359710+0000","last_change":"2024-03-30T15:07:33.165675+0000","last_active":"2024-03-30T15:08:42.359710+0000","last_peered":"2024-03-30T15:08:42.359710+0000","last_clean":"2024-03-30T15:08:42.359710+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:08:42.359710+0000","last_undegraded":"2024-03-30T15:08:42.359710+0000","last_fullsized":"2024-03-30T15:08:42.359710+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.165599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.165599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1259,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.416871+0000","last_change":"2024-03-30T15:07:29.102384+0000","last_active":"2024-03-30T15:08:42.416871+0000","last_peered":"2024-03-30T15:08:42.416871+0000","last_clean":"2024-03-30T15:08:42.416871+0000","last_became_active":"2024-03-30T14:57:56.234694+0000","last_became_peered":"2024-03-30T14:57:56.234694+0000","last_unstale":"2024-03-30T15:08:42.416871+0000","last_undegraded":"2024-03-30T15:08:42.416871+0000","last_fullsized":"2024-03-30T15:08:42.416871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:29.102268+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.102268+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:52.919896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1154,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.903539+0000","last_change":"2024-03-30T15:08:17.607910+0000","last_active":"2024-03-30T15:08:42.903539+0000","last_peered":"2024-03-30T15:08:42.903539+0000","last_clean":"2024-03-30T15:08:42.903539+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:08:42.903539+0000","last_undegraded":"2024-03-30T15:08:42.903539+0000","last_fullsized":"2024-03-30T15:08:42.903539+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.607819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.607819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:44.662694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1229,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811354+0000","last_change":"2024-03-30T15:06:40.495863+0000","last_active":"2024-03-30T15:08:42.811354+0000","last_peered":"2024-03-30T15:08:42.811354+0000","last_clean":"2024-03-30T15:08:42.811354+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:08:42.811354+0000","last_undegraded":"2024-03-30T15:08:42.811354+0000","last_fullsized":"2024-03-30T15:08:42.811354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:40.495702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:40.495702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":808,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.419553+0000","last_change":"2024-03-30T15:07:26.673461+0000","last_active":"2024-03-30T15:08:42.419553+0000","last_peered":"2024-03-30T15:08:42.419553+0000","last_clean":"2024-03-30T15:08:42.419553+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:08:42.419553+0000","last_undegraded":"2024-03-30T15:08:42.419553+0000","last_fullsized":"2024-03-30T15:08:42.419553+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.673362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.673362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:42.689823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1135,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.358578+0000","last_change":"2024-03-30T15:05:15.581201+0000","last_active":"2024-03-30T15:08:42.358578+0000","last_peered":"2024-03-30T15:08:42.358578+0000","last_clean":"2024-03-30T15:08:42.358578+0000","last_became_active":"2024-03-30T14:57:54.060257+0000","last_became_peered":"2024-03-30T14:57:54.060257+0000","last_unstale":"2024-03-30T15:08:42.358578+0000","last_undegraded":"2024-03-30T15:08:42.358578+0000","last_fullsized":"2024-03-30T15:08:42.358578+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1279,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811462+0000","last_change":"2024-03-30T15:07:30.921961+0000","last_active":"2024-03-30T15:08:42.811462+0000","last_peered":"2024-03-30T15:08:42.811462+0000","last_clean":"2024-03-30T15:08:42.811462+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:08:42.811462+0000","last_undegraded":"2024-03-30T15:08:42.811462+0000","last_fullsized":"2024-03-30T15:08:42.811462+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:30.921763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.921763+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1246,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.359479+0000","last_change":"2024-03-30T15:07:35.571081+0000","last_active":"2024-03-30T15:08:42.359479+0000","last_peered":"2024-03-30T15:08:42.359479+0000","last_clean":"2024-03-30T15:08:42.359479+0000","last_became_active":"2024-03-30T14:54:37.474209+0000","last_became_peered":"2024-03-30T14:54:37.474209+0000","last_unstale":"2024-03-30T15:08:42.359479+0000","last_undegraded":"2024-03-30T15:08:42.359479+0000","last_fullsized":"2024-03-30T15:08:42.359479+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:43.970991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1237,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.808334+0000","last_change":"2024-03-30T15:07:32.100519+0000","last_active":"2024-03-30T15:08:42.808334+0000","last_peered":"2024-03-30T15:08:42.808334+0000","last_clean":"2024-03-30T15:08:42.808334+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:08:42.808334+0000","last_undegraded":"2024-03-30T15:08:42.808334+0000","last_fullsized":"2024-03-30T15:08:42.808334+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.100436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.100436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:09:01.419009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1222,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T15:08:42.811785+0000","last_change":"2024-03-30T15:06:07.577645+0000","last_active":"2024-03-30T15:08:42.811785+0000","last_peered":"2024-03-30T15:08:42.811785+0000","last_clean":"2024-03-30T15:08:42.811785+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:08:42.811785+0000","last_undegraded":"2024-03-30T15:08:42.811785+0000","last_fullsized":"2024-03-30T15:08:42.811785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:07.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:07.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:08:45.156725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":325,"num_pg":32,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":838,"num_read_kb":833,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":322,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005230,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76672,"kb_used_data":5232,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668480,"statfs":{"total":95995035648,"available":95916523520,"internally_reserved":0,"allocated":5357568,"data_stored":2572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299571,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76300,"kb_used_data":4864,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668852,"statfs":{"total":95995035648,"available":95916904448,"internally_reserved":0,"allocated":4980736,"data_stored":2237499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575054,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72216,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93672936,"statfs":{"total":95995035648,"available":95921086464,"internally_reserved":0,"allocated":5054464,"data_stored":2264400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54132,"kb_used_data":4904,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93691020,"statfs":{"total":95995035648,"available":95939604480,"internally_reserved":0,"allocated":5021696,"data_stored":2240299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94640,"kb_used_data":5152,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650512,"statfs":{"total":95995035648,"available":95898124288,"internally_reserved":0,"allocated":5275648,"data_stored":2518024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117187,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95028,"kb_used_data":5540,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650124,"statfs":{"total":95995035648,"available":95897726976,"internally_reserved":0,"allocated":5672960,"data_stored":2868031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":91617735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953661,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94404,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650748,"statfs":{"total":95995035648,"available":95898365952,"internally_reserved":0,"allocated":5033984,"data_stored":2251908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659320,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76912,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668240,"statfs":{"total":95995035648,"available":95916277760,"internally_reserved":0,"allocated":5607424,"data_stored":2837856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":325,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:08:48.362 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-30T15:08:48.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:08:48.363 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:08:48.363 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T15:08:48.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T15:08:48.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:48.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:48.531 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:48 smithi028 systemd[1]: Stopping Ceph osd.0 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:08:48.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: from='client.25401 v1:172.21.15.28:0/3693915463' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[29876]: pgmap v989: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:08:48.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:08:48.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: from='client.25401 v1:172.21.15.28:0/3693915463' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:48.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:08:48.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:08:48.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:48 smithi028 ceph-mon[41614]: pgmap v989: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:08:48.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:48.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:08:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:08:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: from='client.25401 v1:172.21.15.28:0/3693915463' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:08:48.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:08:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:08:48.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:48 smithi134 ceph-mon[33714]: pgmap v989: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:08:48.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:48.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:48.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:48.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:48.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:48.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:48.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:49.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:08:49.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:49.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:49.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:49.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:49.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:49.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:49.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:49.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:49.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:49.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:49.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T15:08:49.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T15:08:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T15:08:49.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T15:08:49.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:49.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:49.773 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:49.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:49.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:49.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:49 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T15:08:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:49 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T15:08:49.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:50.008 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T15:08:49.911+0000 7f2e6df72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:08:50.008 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T15:08:49.911+0000 7f2e6df72700 -1 osd.0 668 *** Got signal Terminated *** 2024-03-30T15:08:50.008 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:49 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[46002]: 2024-03-30T15:08:49.911+0000 7f2e6df72700 -1 osd.0 668 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:08:50.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:50.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:50.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:50.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:50.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:50.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.389+0000 7fa4c7690700 1 -- 172.21.15.28:0/2176780885 >> v1:172.21.15.28:6789/0 conn(0x7fa4c010eb50 legacy=0x7fa4c0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.389+0000 7fa4c7690700 1 -- 172.21.15.28:0/2176780885 shutdown_connections 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.389+0000 7fa4c7690700 1 -- 172.21.15.28:0/2176780885 >> 172.21.15.28:0/2176780885 conn(0x7fa4c00fc9e0 msgr2=0x7fa4c00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.389+0000 7fa4c7690700 1 -- 172.21.15.28:0/2176780885 shutdown_connections 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.389+0000 7fa4c7690700 1 -- 172.21.15.28:0/2176780885 wait complete. 2024-03-30T15:08:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 librados: starting msgr at 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 librados: starting objecter 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 Processor -- start 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 -- start start 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 librados: setting wanted keys 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 librados: calling monclient init 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa4c0102990 con 0x7fa4c0103230 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa4c0102ba0 con 0x7fa4c010eb50 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c7690700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa4c0102dd0 con 0x7fa4c010b040 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c542c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa4c0103230 0x7fa4c0100e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39988/0 (socket says 172.21.15.28:39988) 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4c542c700 1 -- 172.21.15.28:0/765499335 learned_addr learned my addr 172.21.15.28:0/765499335 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4b67fc700 10 client.?.objecter ms_handle_connect 0x7fa4c010b040 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4b67fc700 10 client.?.objecter ms_handle_connect 0x7fa4c0103230 2024-03-30T15:08:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.390+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054186715 0 0) 0x7fa4c0102dd0 con 0x7fa4c010b040 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ac0031f0 con 0x7fa4c010b040 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 10 client.?.objecter ms_handle_connect 0x7fa4c010eb50 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564320562 0 0) 0x7fa4c0102990 con 0x7fa4c0103230 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c0102dd0 con 0x7fa4c0103230 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76442208 0 0) 0x7fa4c0102ba0 con 0x7fa4c010eb50 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c0102990 con 0x7fa4c010eb50 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2215368049 0 0) 0x7fa4ac0031f0 con 0x7fa4c010b040 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4c0102ba0 con 0x7fa4c010b040 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa4b0001830 con 0x7fa4c010b040 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 831645318 0 0) 0x7fa4c0102dd0 con 0x7fa4c0103230 2024-03-30T15:08:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4ac0031f0 con 0x7fa4c0103230 2024-03-30T15:08:50.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa4bc001a30 con 0x7fa4c0103230 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 958680816 0 0) 0x7fa4c0102990 con 0x7fa4c010eb50 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4c0102dd0 con 0x7fa4c010eb50 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa4b8001b80 con 0x7fa4c010eb50 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2177001826 0 0) 0x7fa4c0102ba0 con 0x7fa4c010b040 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 >> v1:172.21.15.134:6789/0 conn(0x7fa4c010eb50 legacy=0x7fa4c01a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 >> v1:172.21.15.28:6789/0 conn(0x7fa4c0103230 legacy=0x7fa4c0100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c01aca80 con 0x7fa4c010b040 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c01a9a70 con 0x7fa4c010b040 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4c7690700 10 client.58607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4c01a9f30 con 0x7fa4c010b040 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4c7690700 1 librados: init done 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa4b0001ae0 con 0x7fa4c010b040 2024-03-30T15:08:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.391+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa4b0004af0 con 0x7fa4c010b040 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.392+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a4005120 con 0x7fa4c010b040 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa4b001b330 con 0x7fa4c010b040 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(668..668 src has 442..668) ==== 9222+0+0 (unknown 2111780134 0 0) 0x7fa4b0088aa0 con 0x7fa4c010b040 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 10 client.58607.objecter ms_dispatch 0x7fa4c0001550 osd_map(668..668 src has 442..668) 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 3 client.58607.objecter handle_osd_map got epochs [668,668] > 0 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 3 client.58607.objecter handle_osd_map decoding full epoch 668 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 20 client.58607.objecter dump_active .. 0 homeless 2024-03-30T15:08:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.393+0000 7fa4b67fc700 10 client.58607.objecter ms_handle_connect 0x7fa4ac070480 2024-03-30T15:08:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:50.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.398+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa4b0055890 con 0x7fa4c010b040 2024-03-30T15:08:50.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:50.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:50.559 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:08:50.559 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:08:50.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T15:08:50.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: osd.0 marked itself down and dead 2024-03-30T15:08:50.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[41614]: pgmap v991: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: osd.0 marked itself down and dead 2024-03-30T15:08:50.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:50 smithi028 ceph-mon[29876]: pgmap v991: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:50.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:50.630+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fa4a4005410 con 0x7fa4c010b040 2024-03-30T15:08:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:50.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:50.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:50.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:50.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:08:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:08:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:08:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: osdmap e668: 8 total, 8 up, 8 in 2024-03-30T15:08:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:08:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: osd.0 marked itself down and dead 2024-03-30T15:08:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:50 smithi134 ceph-mon[33714]: pgmap v991: 246 pgs: 246 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:08:51.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:51.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:51.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:51.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:51.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:51.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:51.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:51.451 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:51 smithi028 podman[259585]: 2024-03-30 15:08:51.140597734 +0000 UTC m=+1.894430286 container died aecb81ef48477dcbef7d03c756da6be8d56287cbd47430c90d459f10940521d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.467+0000 7fa4b67fc700 1 -- 172.21.15.28:0/765499335 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v670) ==== 101+0+0 (unknown 2051896335 0 0) 0x7fa4b00592e0 con 0x7fa4c010b040 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 10 librados: watch_flush enter 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 10 librados: watch_flush exit 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 20 client.58607.objecter shutdown clearing up homeless session... 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 10 client.58607.objecter successfully canceled tick 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa4ac070480 legacy=0x7fa4ac072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 >> v1:172.21.15.28:6790/0 conn(0x7fa4c010b040 legacy=0x7fa4c01015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 shutdown_connections 2024-03-30T15:08:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.471+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 >> 172.21.15.28:0/765499335 conn(0x7fa4c00fc9e0 msgr2=0x7fa4c00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:51.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.472+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 shutdown_connections 2024-03-30T15:08:51.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.472+0000 7fa4c7690700 1 -- 172.21.15.28:0/765499335 wait complete. 2024-03-30T15:08:51.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:51.472+0000 7fa4c7690700 1 librados: shutdown 2024-03-30T15:08:51.478 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:08:51.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:51.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:51.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:51.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:51.723 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:51 smithi028 podman[259585]: 2024-03-30 15:08:51.650075076 +0000 UTC m=+2.403907628 container cleanup aecb81ef48477dcbef7d03c756da6be8d56287cbd47430c90d459f10940521d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-03-30T15:08:51.723 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:51 smithi028 bash[259585]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T15:08:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 42.2 scrub starts 2024-03-30T15:08:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 42.2 scrub ok 2024-03-30T15:08:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:08:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: osdmap e669: 8 total, 7 up, 8 in 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/765499335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.58607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:08:51.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 42.2 scrub starts 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 42.2 scrub ok 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: osdmap e669: 8 total, 7 up, 8 in 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/765499335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.58607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:08:51.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:51 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:08:51.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:51.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:51.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:51.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:51.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 42.2 scrub starts 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 42.2 scrub ok 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:08:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: osdmap e669: 8 total, 7 up, 8 in 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]: dispatch 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/765499335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.58607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:08:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:51 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:08:52.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:52.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:52.115 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:51 smithi028 podman[259816]: 2024-03-30 15:08:51.859905307 +0000 UTC m=+0.846851568 container remove aecb81ef48477dcbef7d03c756da6be8d56287cbd47430c90d459f10940521d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True) 2024-03-30T15:08:52.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:52.171 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:52.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:52.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:52.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:52.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:52.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:52.526 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:08:52.526 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:08:52.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:52.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:52.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:08:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:08:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: from='client.58607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: osdmap e670: 8 total, 7 up, 8 in 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: pgmap v994: 246 pgs: 34 stale+active+clean, 212 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:52.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[29876]: osdmap e671: 8 total, 7 up, 8 in 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:52.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: from='client.58607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:52.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: osdmap e670: 8 total, 7 up, 8 in 2024-03-30T15:08:52.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: pgmap v994: 246 pgs: 34 stale+active+clean, 212 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:08:52.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:52.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:52.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:52 smithi028 ceph-mon[41614]: osdmap e671: 8 total, 7 up, 8 in 2024-03-30T15:08:52.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:52.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:52.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:52.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-116"}]': finished 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: from='client.58607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: osdmap e670: 8 total, 7 up, 8 in 2024-03-30T15:08:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: pgmap v994: 246 pgs: 34 stale+active+clean, 212 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:08:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:08:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:08:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:52 smithi134 ceph-mon[33714]: osdmap e671: 8 total, 7 up, 8 in 2024-03-30T15:08:53.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:53.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:53.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:53.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:53.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:53.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:53.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:53.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:53.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:53.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:53.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:53.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:53.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:53.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:53.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 42.1b scrub starts 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 42.1b scrub ok 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 3.1a deep-scrub starts 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 3.1a deep-scrub ok 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T15:08:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T15:08:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: osdmap e672: 8 total, 7 up, 8 in 2024-03-30T15:08:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:53 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:53.938 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:53.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 42.1b scrub starts 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 42.1b scrub ok 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 3.1a deep-scrub starts 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 3.1a deep-scrub ok 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: osdmap e672: 8 total, 7 up, 8 in 2024-03-30T15:08:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 42.1b scrub starts 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 42.1b scrub ok 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 3.1a deep-scrub starts 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 3.1a deep-scrub ok 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: osdmap e672: 8 total, 7 up, 8 in 2024-03-30T15:08:53.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:53.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:53 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:08:54.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:54.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:54.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:54.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:54.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.244+0000 7fa11d33a700 1 -- 172.21.15.28:0/584858945 >> v1:172.21.15.28:6789/0 conn(0x7fa11810eab0 legacy=0x7fa118110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:54.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.244+0000 7fa11d33a700 1 -- 172.21.15.28:0/584858945 shutdown_connections 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.244+0000 7fa11d33a700 1 -- 172.21.15.28:0/584858945 >> 172.21.15.28:0/584858945 conn(0x7fa1180fc9e0 msgr2=0x7fa1180fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.244+0000 7fa11d33a700 1 -- 172.21.15.28:0/584858945 shutdown_connections 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.244+0000 7fa11d33a700 1 -- 172.21.15.28:0/584858945 wait complete. 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 librados: starting msgr at 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 librados: starting objecter 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 Processor -- start 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 -- start start 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 librados: setting wanted keys 2024-03-30T15:08:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 librados: calling monclient init 2024-03-30T15:08:54.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1180fec80 con 0x7fa11810afa0 2024-03-30T15:08:54.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa11810acf0 con 0x7fa118103190 2024-03-30T15:08:54.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.245+0000 7fa11d33a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa1181a9640 con 0x7fa11810eab0 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa1177fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fa11810eab0 0x7fa1181a5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37960/0 (socket says 172.21.15.28:37960) 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa1177fe700 1 -- 172.21.15.28:0/2140821533 learned_addr learned my addr 172.21.15.28:0/2140821533 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa1167fc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa11810afa0 0x7fa118108210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40008/0 (socket says 172.21.15.28:40008) 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter ms_handle_connect 0x7fa11810afa0 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter ms_handle_connect 0x7fa11810eab0 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter ms_handle_connect 0x7fa118103190 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995892874 0 0) 0x7fa1180fec80 con 0x7fa11810afa0 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1040031f0 con 0x7fa11810afa0 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804938623 0 0) 0x7fa11810acf0 con 0x7fa118103190 2024-03-30T15:08:54.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1180fec80 con 0x7fa118103190 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682079369 0 0) 0x7fa1181a9640 con 0x7fa11810eab0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa11810acf0 con 0x7fa11810eab0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1761541160 0 0) 0x7fa1040031f0 con 0x7fa11810afa0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1181a9640 con 0x7fa11810afa0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa100001960 con 0x7fa11810afa0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 993411418 0 0) 0x7fa1180fec80 con 0x7fa118103190 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1040031f0 con 0x7fa118103190 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3176052156 0 0) 0x7fa11810acf0 con 0x7fa11810eab0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1180fec80 con 0x7fa11810eab0 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa108001b30 con 0x7fa118103190 2024-03-30T15:08:54.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa110004550 con 0x7fa11810eab0 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1992659934 0 0) 0x7fa1040031f0 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 >> v1:172.21.15.28:6790/0 conn(0x7fa11810eab0 legacy=0x7fa1181a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 >> v1:172.21.15.28:6789/0 conn(0x7fa11810afa0 legacy=0x7fa118108210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.246+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1181aa820 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1181a9870 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa11d33a700 10 client.50617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1181a9d30 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa11d33a700 1 librados: init done 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa1080020a0 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa108004d20 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.247+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa0f8005100 con 0x7fa118103190 2024-03-30T15:08:54.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.248+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa108003a00 con 0x7fa118103190 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(672..672 src has 442..672) ==== 9158+0+0 (unknown 2298806009 0 0) 0x7fa108088ac0 con 0x7fa118103190 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 osd_map(672..672 src has 442..672) 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 3 client.50617.objecter handle_osd_map got epochs [672,672] > 0 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 3 client.50617.objecter handle_osd_map decoding full epoch 672 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 20 client.50617.objecter dump_active .. 0 homeless 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v672) ==== 54+0+16 (unknown 3537277599 0 2420000192) 0x7fa108089060 con 0x7fa118103190 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa104070520 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 10 client.50617.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 1 2024-03-30T15:08:54.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 20 client.50617.objecter _get_session osd=-1 returning homeless 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 10 client.50617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=673}) -- 0x7fa0f80053e0 con 0x7fa118103190 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa0f8000ed0 con 0x7fa118103190 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=672) ==== 24+0+0 (unknown 3977774111 0 0) 0x7fa108054fa0 con 0x7fa118103190 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa1157fa700 10 client.50617.objecter _check_command_map_dne tid 1 current 672 map_dne_bound 672 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa1157fa700 10 client.50617.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.249+0000 7fa1157fa700 15 client.50617.objecter _session_command_op_remove -1 1 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 10 client.50617.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 2 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 2 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 1 2 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 4 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 10 client.50617.objecter _send_command 2 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8004460 con 0x7fa0f8003910 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.250+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f8003910 2024-03-30T15:08:54.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.251+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8004460 con 0x7fa0f8003910 2024-03-30T15:08:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.251+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 2: 0 ) 2024-03-30T15:08:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.251+0000 7fa10ffff700 10 client.50617.objecter _finish_command 2 = system:0 2024-03-30T15:08:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.251+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 4 2024-03-30T15:08:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.251+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 1 2 2024-03-30T15:08:54.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 10 client.50617.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f80011c0 osd=1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 1 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 4 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 10 client.50617.objecter _send_command 3 2024-03-30T15:08:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.268+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8004e10 con 0x7fa0f8003910 2024-03-30T15:08:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8004e10 con 0x7fa0f8003910 2024-03-30T15:08:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 10 client.50617.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f80011c0 osd=1 4 2024-03-30T15:08:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 1 3 2024-03-30T15:08:54.272 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:08:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 10 client.50617.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 4 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 4 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 2 4 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 4 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 10 client.50617.objecter _send_command 4 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8010e30 con 0x7fa0f800d320 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.270+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f800d320 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.271+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8010e30 con 0x7fa0f800d320 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.271+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 4: 0 ) 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.271+0000 7fa10ffff700 10 client.50617.objecter _finish_command 4 = system:0 2024-03-30T15:08:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.271+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 4 2024-03-30T15:08:54.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.271+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 2 4 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 10 client.50617.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 5 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 5 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8001ad0 osd=2 3 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 2 5 2024-03-30T15:08:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 4 2024-03-30T15:08:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 10 client.50617.objecter _send_command 5 2024-03-30T15:08:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.288+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8004950 con 0x7fa0f800d320 2024-03-30T15:08:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8004950 con 0x7fa0f800d320 2024-03-30T15:08:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa10ffff700 10 client.50617.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8001ad0 osd=2 4 2024-03-30T15:08:54.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 2 5 2024-03-30T15:08:54.292 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 10 client.50617.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 6 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 6 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 3 6 2024-03-30T15:08:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 4 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 10 client.50617.objecter _send_command 6 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.290+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8003610 con 0x7fa0f80031b0 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f80031b0 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8003610 con 0x7fa0f80031b0 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 6: 0 ) 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 10 client.50617.objecter _finish_command 6 = system:0 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 4 2024-03-30T15:08:54.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.291+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 3 6 2024-03-30T15:08:54.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 10 client.50617.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 7 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 7 2024-03-30T15:08:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8004bb0 osd=3 3 2024-03-30T15:08:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 3 7 2024-03-30T15:08:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 4 2024-03-30T15:08:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 10 client.50617.objecter _send_command 7 2024-03-30T15:08:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.309+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8002c90 con 0x7fa0f80031b0 2024-03-30T15:08:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.310+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8002c90 con 0x7fa0f80031b0 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.310+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.310+0000 7fa10ffff700 10 client.50617.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.310+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8004bb0 osd=3 4 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.310+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 3 7 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:08:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 10 client.50617.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 8 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 8 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 4 8 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 4 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 10 client.50617.objecter _send_command 8 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8011f90 con 0x7fa0f80170c0 2024-03-30T15:08:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.311+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f80170c0 2024-03-30T15:08:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.312+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8011f90 con 0x7fa0f80170c0 2024-03-30T15:08:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.312+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 8: 0 ) 2024-03-30T15:08:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.312+0000 7fa10ffff700 10 client.50617.objecter _finish_command 8 = system:0 2024-03-30T15:08:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.312+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 4 2024-03-30T15:08:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.312+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 4 8 2024-03-30T15:08:54.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 10 client.50617.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 9 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 9 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f80117b0 osd=4 3 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 4 9 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 4 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 10 client.50617.objecter _send_command 9 2024-03-30T15:08:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.329+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8011350 con 0x7fa0f80170c0 2024-03-30T15:08:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8011350 con 0x7fa0f80170c0 2024-03-30T15:08:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa10ffff700 10 client.50617.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f80117b0 osd=4 4 2024-03-30T15:08:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 4 9 2024-03-30T15:08:54.333 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 10 client.50617.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 10 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 10 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 5 10 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 4 2024-03-30T15:08:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 10 client.50617.objecter _send_command 10 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.331+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fa0f801f5b0 con 0x7fa0f8016c20 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.332+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f8016c20 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.333+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f801f5b0 con 0x7fa0f8016c20 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.333+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 10: 0 ) 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.333+0000 7fa10ffff700 10 client.50617.objecter _finish_command 10 = system:0 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.333+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 4 2024-03-30T15:08:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.333+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 5 10 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 10 client.50617.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 11 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 11 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8011100 osd=5 3 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 5 11 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 4 2024-03-30T15:08:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 10 client.50617.objecter _send_command 11 2024-03-30T15:08:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.350+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f801f5b0 con 0x7fa0f8016c20 2024-03-30T15:08:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.351+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f801f5b0 con 0x7fa0f8016c20 2024-03-30T15:08:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.351+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.351+0000 7fa10ffff700 10 client.50617.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.351+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8011100 osd=5 4 2024-03-30T15:08:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.351+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 5 11 2024-03-30T15:08:54.353 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 10 client.50617.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 12 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 12 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 6 12 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 4 2024-03-30T15:08:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 10 client.50617.objecter _send_command 12 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8023910 con 0x7fa0f801fc30 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.352+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f801fc30 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.353+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8023910 con 0x7fa0f801fc30 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.353+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 12: 0 ) 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.353+0000 7fa10ffff700 10 client.50617.objecter _finish_command 12 = system:0 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.353+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 4 2024-03-30T15:08:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.353+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 6 12 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.369+0000 7fa11d33a700 10 client.50617.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 13 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 13 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f801f5b0 osd=6 3 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 6 13 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 4 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 10 client.50617.objecter _send_command 13 2024-03-30T15:08:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.370+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8023af0 con 0x7fa0f801fc30 2024-03-30T15:08:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.371+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8023af0 con 0x7fa0f801fc30 2024-03-30T15:08:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.371+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.371+0000 7fa10ffff700 10 client.50617.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.371+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f801f5b0 osd=6 4 2024-03-30T15:08:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.371+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 6 13 2024-03-30T15:08:54.373 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 10 client.50617.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 14 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 14 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 7 14 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 4 2024-03-30T15:08:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 10 client.50617.objecter _send_command 14 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fa0f8024e10 con 0x7fa0f8024210 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.372+0000 7fa10ffff700 10 client.50617.objecter ms_handle_connect 0x7fa0f8024210 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.373+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa0f8024e10 con 0x7fa0f8024210 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.373+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 14: 0 ) 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.373+0000 7fa10ffff700 10 client.50617.objecter _finish_command 14 = system:0 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.373+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 4 2024-03-30T15:08:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.373+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 7 14 2024-03-30T15:08:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.375+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== mon.1 v1:172.21.15.134:6789/0 11 ==== osd_map(673..673 src has 442..673) ==== 676+0+0 (unknown 2701704189 0 0) 0x7fa108086320 con 0x7fa118103190 2024-03-30T15:08:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.375+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 osd_map(673..673 src has 442..673) 2024-03-30T15:08:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.375+0000 7fa10ffff700 3 client.50617.objecter handle_osd_map got epochs [673,673] > 672 2024-03-30T15:08:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.375+0000 7fa10ffff700 3 client.50617.objecter handle_osd_map decoding incremental epoch 673 2024-03-30T15:08:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.375+0000 7fa10ffff700 20 client.50617.objecter dump_active .. 0 homeless 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 10 client.50617.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign -1 15 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 20 client.50617.objecter _get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_remove -1 15 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 20 client.50617.objecter get_session s=0x7fa0f8023af0 osd=7 3 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 15 client.50617.objecter _session_command_op_assign 7 15 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 4 2024-03-30T15:08:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 10 client.50617.objecter _send_command 15 2024-03-30T15:08:54.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.402+0000 7fa11d33a700 1 -- 172.21.15.28:0/2140821533 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa0f8024e10 con 0x7fa0f8024210 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.403+0000 7fa10ffff700 1 -- 172.21.15.28:0/2140821533 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa0f8024e10 con 0x7fa0f8024210 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.403+0000 7fa10ffff700 10 client.50617.objecter ms_dispatch 0x7fa118001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.403+0000 7fa10ffff700 10 client.50617.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.403+0000 7fa10ffff700 20 client.50617.objecter put_session s=0x7fa0f8023af0 osd=7 4 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:08:54.403+0000 7fa10ffff700 15 client.50617.objecter _session_command_op_remove 7 15 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:08:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:08:54.406 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:08:54.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:54.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:54.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:54.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:54.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:54.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:54.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:54.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 1.7 scrub starts 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 1.7 scrub ok 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 42.1a scrub starts 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: 42.1a scrub ok 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: pgmap v997: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2140821533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:54 smithi134 ceph-mon[33714]: osdmap e673: 8 total, 7 up, 8 in 2024-03-30T15:08:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:54.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:54.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 1.7 scrub starts 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 1.7 scrub ok 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 42.1a scrub starts 2024-03-30T15:08:54.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: 42.1a scrub ok 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: pgmap v997: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2140821533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[41614]: osdmap e673: 8 total, 7 up, 8 in 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 1.7 scrub starts 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 1.7 scrub ok 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 42.1a scrub starts 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: 42.1a scrub ok 2024-03-30T15:08:54.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: pgmap v997: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:54.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2140821533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:08:54.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:08:54.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:54 smithi028 ceph-mon[29876]: osdmap e673: 8 total, 7 up, 8 in 2024-03-30T15:08:54.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:08:55.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:55.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:55.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:55.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:55.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:55.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:55.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:55.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:55.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:08:55.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:55.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:08:55.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:08:55.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:55.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T15:08:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T15:08:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: Health check failed: Degraded data redundancy: 60/612 objects degraded (9.804%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:08:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: osdmap e674: 8 total, 7 up, 8 in 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:55 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: Health check failed: Degraded data redundancy: 60/612 objects degraded (9.804%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: osdmap e674: 8 total, 7 up, 8 in 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: Health check failed: Degraded data redundancy: 60/612 objects degraded (9.804%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: osdmap e674: 8 total, 7 up, 8 in 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:55 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:08:55.935 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:55 smithi028 podman[260291]: 2024-03-30 15:08:55.775125411 +0000 UTC m=+0.748228292 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:08:55.936 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:55 smithi028 podman[260291]: 2024-03-30 15:08:55.933441242 +0000 UTC m=+0.906544107 container create a24fc34f4d57019cc474bda3833bf7d1f7da7ab761f0ae6223554e8a330e3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-30T15:08:55.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:55.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:56.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:56.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:56.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:56.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:56.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:56.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:56.470 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:56 smithi028 podman[260291]: 2024-03-30 15:08:56.194169135 +0000 UTC m=+1.167272004 container init a24fc34f4d57019cc474bda3833bf7d1f7da7ab761f0ae6223554e8a330e3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T15:08:56.470 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:56 smithi028 podman[260291]: 2024-03-30 15:08:56.199889793 +0000 UTC m=+1.172992663 container start a24fc34f4d57019cc474bda3833bf7d1f7da7ab761f0ae6223554e8a330e3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-30T15:08:56.470 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:56 smithi028 podman[260291]: 2024-03-30 15:08:56.294033851 +0000 UTC m=+1.267136735 container attach a24fc34f4d57019cc474bda3833bf7d1f7da7ab761f0ae6223554e8a330e3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5) 2024-03-30T15:08:56.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:56.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:56.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:56.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:56.722 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:08:56 smithi028 podman[260291]: 2024-03-30 15:08:56.594703741 +0000 UTC m=+1.567806621 container died a24fc34f4d57019cc474bda3833bf7d1f7da7ab761f0ae6223554e8a330e3a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: pgmap v1000: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: osdmap e675: 8 total, 7 up, 8 in 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T15:08:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: pgmap v1000: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:56.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:56.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: osdmap e675: 8 total, 7 up, 8 in 2024-03-30T15:08:56.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:56.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:56.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T15:08:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T15:08:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: pgmap v1000: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:08:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: osdmap e675: 8 total, 7 up, 8 in 2024-03-30T15:08:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:08:56.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:56.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:57.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:57.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:57.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:57.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:57.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:57.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:57.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:57.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:57.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:57.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:57.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:57.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: osdmap e676: 8 total, 7 up, 8 in 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.770 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:08:57.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:08:57.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:08:57.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: osdmap e676: 8 total, 7 up, 8 in 2024-03-30T15:08:57.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:57.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:57.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:57.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:57.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:08:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:08:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:08:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: osdmap e676: 8 total, 7 up, 8 in 2024-03-30T15:08:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]: dispatch 2024-03-30T15:08:57.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T15:08:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:57.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:08:58.025 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:08:58.025 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T15:08:58.025 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-03-30T15:08:58.025 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T15:08:58.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-03-30T15:08:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:08:58.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:58.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:58.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:58.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:58.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:58.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:58.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:08:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:08:58.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:58.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:08:58.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:58.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:58.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:58.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:58.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[29876]: pgmap v1003: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:58.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:08:58.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:58.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]': finished 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[29876]: osdmap e677: 8 total, 7 up, 8 in 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[41614]: pgmap v1003: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]': finished 2024-03-30T15:08:58.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:08:58 smithi028 ceph-mon[41614]: osdmap e677: 8 total, 7 up, 8 in 2024-03-30T15:08:58.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:08:58.900 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:08:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:58.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:58.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:08:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:58 smithi134 ceph-mon[33714]: pgmap v1003: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 60/612 objects degraded (9.804%) 2024-03-30T15:08:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:58 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:08:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:58 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:08:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:58 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-118", "mode": "writeback"}]': finished 2024-03-30T15:08:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:08:58 smithi134 ceph-mon[33714]: osdmap e677: 8 total, 7 up, 8 in 2024-03-30T15:08:59.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:59.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:59.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:59.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:59.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:08:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:59.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:08:59.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:59.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:08:59.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:08:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:59.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:08:59.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:08:59.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:08:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:08:59.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:08:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:59.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:08:59.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:08:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:08:59.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:08:59.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:00.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:00.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:00.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:00.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:00.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:00.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: osdmap e678: 8 total, 7 up, 8 in 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 42.c scrub starts 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: 42.c scrub ok 2024-03-30T15:09:00.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[29876]: pgmap v1006: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: osdmap e678: 8 total, 7 up, 8 in 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 42.c scrub starts 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: 42.c scrub ok 2024-03-30T15:09:00.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:00 smithi028 ceph-mon[41614]: pgmap v1006: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:00.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:00.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: osdmap e678: 8 total, 7 up, 8 in 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:09:00.646 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:09:00.647 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 42.c scrub starts 2024-03-30T15:09:00.647 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: 42.c scrub ok 2024-03-30T15:09:00.647 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-mon[33714]: pgmap v1006: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:00.657 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:09:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:09:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:09:00.657 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:09:00.657 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:09:00.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:00.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:00.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:00.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:00.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:00.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:00.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:00.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:01.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:01.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:01.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:01.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:01.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:01.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 60/618 objects degraded (9.709%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: osdmap e679: 8 total, 7 up, 8 in 2024-03-30T15:09:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:09:01.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 60/618 objects degraded (9.709%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: osdmap e679: 8 total, 7 up, 8 in 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:09:01.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 60/618 objects degraded (9.709%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: osdmap e679: 8 total, 7 up, 8 in 2024-03-30T15:09:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]: dispatch 2024-03-30T15:09:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:01.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:01.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:01.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:01.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:01.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:01.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:02.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:02.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:02.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:02.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:02.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:02.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:02.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:02.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:02.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:02.609 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:02.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:02.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:02.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:02.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:09:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:09:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:09:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:09:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: osdmap e680: 8 total, 7 up, 8 in 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 1.5 scrub starts 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 1.5 scrub ok 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: pgmap v1009: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:02.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:02 smithi134 ceph-mon[33714]: osdmap e681: 8 total, 7 up, 8 in 2024-03-30T15:09:02.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:02.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: osdmap e680: 8 total, 7 up, 8 in 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 1.5 scrub starts 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 1.5 scrub ok 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: pgmap v1009: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[41614]: osdmap e681: 8 total, 7 up, 8 in 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-118"}]': finished 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: osdmap e680: 8 total, 7 up, 8 in 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 1.5 scrub starts 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 1.5 scrub ok 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: pgmap v1009: 246 pgs: 17 active+undersized+degraded, 68 active+undersized, 161 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 60/618 objects degraded (9.709%) 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:02.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:02 smithi028 ceph-mon[29876]: osdmap e681: 8 total, 7 up, 8 in 2024-03-30T15:09:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:02.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:03.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:03.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:03.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:03.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 -- 172.21.15.28:0/57208203 >> v1:172.21.15.28:6789/0 conn(0x7fdf90109e10 legacy=0x7fdf9010c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 -- 172.21.15.28:0/57208203 shutdown_connections 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 -- 172.21.15.28:0/57208203 >> 172.21.15.28:0/57208203 conn(0x7fdf900fca30 msgr2=0x7fdf900fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 -- 172.21.15.28:0/57208203 shutdown_connections 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 -- 172.21.15.28:0/57208203 wait complete. 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.462+0000 7fdf972ec700 1 librados: starting msgr at 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 librados: starting objecter 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 Processor -- start 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 -- start start 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 librados: setting wanted keys 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 librados: calling monclient init 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf901a7160 con 0x7fdf90101180 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf901a8380 con 0x7fdf9010d9b0 2024-03-30T15:09:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf972ec700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf901a95c0 con 0x7fdf90109e10 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf95088700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf90101180 0x7fdf90106530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35236/0 (socket says 172.21.15.28:35236) 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf95088700 1 -- 172.21.15.28:0/223253797 learned_addr learned my addr 172.21.15.28:0/223253797 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter ms_handle_connect 0x7fdf90101180 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter ms_handle_connect 0x7fdf90109e10 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674536078 0 0) 0x7fdf901a7160 con 0x7fdf90101180 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf7c0031f0 con 0x7fdf90101180 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter ms_handle_connect 0x7fdf9010d9b0 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344527078 0 0) 0x7fdf901a95c0 con 0x7fdf90109e10 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf901a7160 con 0x7fdf90109e10 2024-03-30T15:09:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690960902 0 0) 0x7fdf901a8380 con 0x7fdf9010d9b0 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.463+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf901a95c0 con 0x7fdf9010d9b0 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3111456690 0 0) 0x7fdf7c0031f0 con 0x7fdf90101180 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf901a8380 con 0x7fdf90101180 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf84001940 con 0x7fdf90101180 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3544343509 0 0) 0x7fdf901a95c0 con 0x7fdf9010d9b0 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf7c0031f0 con 0x7fdf9010d9b0 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3509497976 0 0) 0x7fdf901a7160 con 0x7fdf90109e10 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf901a95c0 con 0x7fdf90109e10 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf88001bb0 con 0x7fdf9010d9b0 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf8c0044b0 con 0x7fdf90109e10 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1477341925 0 0) 0x7fdf901a8380 con 0x7fdf90101180 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 >> v1:172.21.15.28:6790/0 conn(0x7fdf90109e10 legacy=0x7fdf901a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 >> v1:172.21.15.134:6789/0 conn(0x7fdf9010d9b0 legacy=0x7fdf901a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf901aa800 con 0x7fdf90101180 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf901a97f0 con 0x7fdf90101180 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf972ec700 10 client.25425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf901a9cb0 con 0x7fdf90101180 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf972ec700 1 librados: init done 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdf84003b20 con 0x7fdf90101180 2024-03-30T15:09:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.464+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf84005cd0 con 0x7fdf90101180 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.465+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf74005120 con 0x7fdf90101180 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdf84003680 con 0x7fdf90101180 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(682..682 src has 442..682) ==== 9186+0+0 (unknown 3143242757 0 0) 0x7fdf84089960 con 0x7fdf90101180 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 10 client.25425.objecter ms_dispatch 0x7fdf90001550 osd_map(682..682 src has 442..682) 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 3 client.25425.objecter handle_osd_map got epochs [682,682] > 0 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 3 client.25425.objecter handle_osd_map decoding full epoch 682 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.466+0000 7fdf827fc700 20 client.25425.objecter dump_active .. 0 homeless 2024-03-30T15:09:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.467+0000 7fdf827fc700 10 client.25425.objecter ms_handle_connect 0x7fdf7c070360 2024-03-30T15:09:03.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.471+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdf84056770 con 0x7fdf90101180 2024-03-30T15:09:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.357+0000 7f03a7276700 1 -- 172.21.15.28:0/1201156266 >> v1:172.21.15.28:6789/0 conn(0x7f03a01031f0 legacy=0x7f03a01035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.357+0000 7f03a7276700 1 -- 172.21.15.28:0/1201156266 shutdown_connections 2024-03-30T15:09:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.357+0000 7f03a7276700 1 -- 172.21.15.28:0/1201156266 >> 172.21.15.28:0/1201156266 conn(0x7f03a00fc9e0 msgr2=0x7f03a00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.357+0000 7f03a7276700 1 -- 172.21.15.28:0/1201156266 shutdown_connections 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.357+0000 7f03a7276700 1 -- 172.21.15.28:0/1201156266 wait complete. 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 librados: starting msgr at 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 librados: starting objecter 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 Processor -- start 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 -- start start 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 librados: setting wanted keys 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 librados: calling monclient init 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03a01a70d0 con 0x7f03a010b000 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03a01a82f0 con 0x7f03a01031f0 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a7276700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03a01a9530 con 0x7f03a010eb10 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a5813700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f03a010eb10 0x7f03a01a5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57408/0 (socket says 172.21.15.28:57408) 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a5813700 1 -- 172.21.15.28:0/3319211125 learned_addr learned my addr 172.21.15.28:0/3319211125 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03a4811700 1 --1- 172.21.15.28:0/3319211125 >> v1:172.21.15.28:6789/0 conn(0x7f03a010b000 0x7f03a01a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35216/0 (socket says 172.21.15.28:35216) 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03967fc700 10 client.?.objecter ms_handle_connect 0x7f03a010eb10 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03967fc700 10 client.?.objecter ms_handle_connect 0x7f03a010b000 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.358+0000 7f03967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 10 client.?.objecter ms_handle_connect 0x7f03a01031f0 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051849212 0 0) 0x7f03a01a9530 con 0x7f03a010eb10 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f038c0039c0 con 0x7f03a010eb10 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 545182835 0 0) 0x7f03a01a70d0 con 0x7f03a010b000 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03a01a9530 con 0x7f03a010b000 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68130431 0 0) 0x7f03a01a82f0 con 0x7f03a01031f0 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03a01a70d0 con 0x7f03a01031f0 2024-03-30T15:09:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3282097600 0 0) 0x7f038c0039c0 con 0x7f03a010eb10 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03a01a82f0 con 0x7f03a010eb10 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0398001c50 con 0x7f03a010eb10 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3337572747 0 0) 0x7f03a01a9530 con 0x7f03a010b000 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f038c0039c0 con 0x7f03a010b000 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03900013d0 con 0x7f03a010b000 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1284950764 0 0) 0x7f03a01a70d0 con 0x7f03a01031f0 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03a01a9530 con 0x7f03a01031f0 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f039c001860 con 0x7f03a01031f0 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3494107031 0 0) 0x7f03a01a82f0 con 0x7f03a010eb10 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 >> v1:172.21.15.134:6789/0 conn(0x7f03a01031f0 legacy=0x7f03a010a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 >> v1:172.21.15.28:6789/0 conn(0x7f03a010b000 legacy=0x7f03a01a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a01aa770 con 0x7f03a010eb10 2024-03-30T15:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03a01a8520 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03a7276700 10 client.58619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f03a01a89e0 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03a7276700 1 librados: init done 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0398003ec0 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.359+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0398006090 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.360+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0384005100 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f039801c8d0 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(681..681 src has 442..681) ==== 8813+0+0 (unknown 369599606 0 0) 0x7f0398089e80 con 0x7f03a010eb10 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 osd_map(681..681 src has 442..681) 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 3 client.58619.objecter handle_osd_map got epochs [681,681] > 0 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 3 client.58619.objecter handle_osd_map decoding full epoch 681 2024-03-30T15:09:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 20 client.58619.objecter dump_active .. 0 homeless 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.361+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v681) ==== 54+0+16 (unknown 2404815230 0 2420000192) 0x7f039808c2e0 con 0x7f03a010eb10 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f038c0704d0 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 10 client.58619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 1 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 10 client.58619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=682}) -- 0x7f03840053e0 con 0x7f03a010eb10 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0384000e30 con 0x7f03a010eb10 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=681) ==== 24+0+0 (unknown 2532386502 0 0) 0x7f03980564d0 con 0x7f03a010eb10 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03977fe700 10 client.58619.objecter _check_command_map_dne tid 1 current 681 map_dne_bound 681 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03977fe700 10 client.58619.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03977fe700 15 client.58619.objecter _session_command_op_remove -1 1 2024-03-30T15:09:03.580 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:03.580 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 10 client.58619.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 2 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 2 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 1 2 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 4 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 10 client.58619.objecter _send_command 2 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f03840022f0 con 0x7f0384002790 2024-03-30T15:09:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.362+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f0384002790 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.363+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f03840022f0 con 0x7f0384002790 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.363+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.363+0000 7f03967fc700 10 client.58619.objecter _finish_command 2 = system:0 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.363+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 4 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.363+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 1 2 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.370+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== mon.2 v1:172.21.15.28:6790/0 11 ==== osd_map(682..682 src has 442..682) ==== 711+0+0 (unknown 3678697759 0 0) 0x7f0398087850 con 0x7f03a010eb10 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.370+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 osd_map(682..682 src has 442..682) 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.370+0000 7f03967fc700 3 client.58619.objecter handle_osd_map got epochs [682,682] > 681 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.370+0000 7f03967fc700 3 client.58619.objecter handle_osd_map decoding incremental epoch 682 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.370+0000 7f03967fc700 20 client.58619.objecter dump_active .. 0 homeless 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 10 client.58619.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384001bb0 osd=1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 1 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 4 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 10 client.58619.objecter _send_command 3 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.389+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0384004c30 con 0x7f0384002790 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0384004c30 con 0x7f0384002790 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03967fc700 10 client.58619.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384001bb0 osd=1 4 2024-03-30T15:09:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 1 3 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 10 client.58619.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 4 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 4 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 2 4 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 4 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 10 client.58619.objecter _send_command 4 2024-03-30T15:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.391+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f0384010f40 con 0x7f038400d430 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f038400d430 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0384010f40 con 0x7f038400d430 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 10 client.58619.objecter _finish_command 4 = system:0 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 4 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.392+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 2 4 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 10 client.58619.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 5 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 5 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384004ec0 osd=2 3 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 2 5 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 4 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 10 client.58619.objecter _send_command 5 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.409+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0384003e20 con 0x7f038400d430 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0384003e20 con 0x7f038400d430 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03967fc700 10 client.58619.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384004ec0 osd=2 4 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 2 5 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 10 client.58619.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 6 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 6 2024-03-30T15:09:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 3 6 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 4 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 10 client.58619.objecter _send_command 6 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.411+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f0384015e10 con 0x7f0384012270 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f0384012270 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0384015e10 con 0x7f0384012270 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 10 client.58619.objecter _finish_command 6 = system:0 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 4 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.412+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 3 6 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 10 client.58619.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 7 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 7 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384004880 osd=3 3 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 3 7 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 4 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 10 client.58619.objecter _send_command 7 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.429+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f03840043c0 con 0x7f0384012270 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f03840043c0 con 0x7f0384012270 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03967fc700 10 client.58619.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384004880 osd=3 4 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 3 7 2024-03-30T15:09:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 10 client.58619.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 8 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 8 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 4 8 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 4 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 10 client.58619.objecter _send_command 8 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.431+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f0384017ce0 con 0x7f0384017140 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.432+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f0384017140 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.433+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0384017ce0 con 0x7f0384017140 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.433+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.433+0000 7f03967fc700 10 client.58619.objecter _finish_command 8 = system:0 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.433+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 4 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.433+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 4 8 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 10 client.58619.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 9 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 9 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f03840118c0 osd=4 3 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 4 9 2024-03-30T15:09:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 4 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 10 client.58619.objecter _send_command 9 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.450+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f03840184b0 con 0x7f0384017140 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.451+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f03840184b0 con 0x7f0384017140 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.451+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.451+0000 7f03967fc700 10 client.58619.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.451+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f03840118c0 osd=4 4 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.451+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 4 9 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 10 client.58619.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 10 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 10 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 5 10 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 4 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 10 client.58619.objecter _send_command 10 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f038401f5b0 con 0x7f03840184b0 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.452+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f03840184b0 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.453+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f038401f5b0 con 0x7f03840184b0 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.453+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.453+0000 7f03967fc700 10 client.58619.objecter _finish_command 10 = system:0 2024-03-30T15:09:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.453+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 4 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.453+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 5 10 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 10 client.58619.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 11 2024-03-30T15:09:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 11 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384011590 osd=5 3 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 5 11 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 4 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 10 client.58619.objecter _send_command 11 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.472+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0384017f30 con 0x7f03840184b0 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.473+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0384017f30 con 0x7f03840184b0 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.473+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.473+0000 7f03967fc700 10 client.58619.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.473+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384011590 osd=5 4 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.473+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 5 11 2024-03-30T15:09:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 10 client.58619.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 12 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 12 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 6 12 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 4 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 10 client.58619.objecter _send_command 12 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f0384023970 con 0x7f038401fc30 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.474+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f038401fc30 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.475+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0384023970 con 0x7f038401fc30 2024-03-30T15:09:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.475+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.475+0000 7f03967fc700 10 client.58619.objecter _finish_command 12 = system:0 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.475+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 4 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.475+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 6 12 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 10 client.58619.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 13 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 13 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f038401f5b0 osd=6 3 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 6 13 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 4 2024-03-30T15:09:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 10 client.58619.objecter _send_command 13 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.492+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0384025320 con 0x7f038401fc30 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0384025320 con 0x7f038401fc30 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03967fc700 10 client.58619.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f038401f5b0 osd=6 4 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 6 13 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 10 client.58619.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 14 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 14 2024-03-30T15:09:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 7 14 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 4 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 10 client.58619.objecter _send_command 14 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.494+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f0384025f60 con 0x7f0384025a40 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.495+0000 7f03967fc700 10 client.58619.objecter ms_handle_connect 0x7f0384025a40 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.496+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0384025f60 con 0x7f0384025a40 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.496+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.496+0000 7f03967fc700 10 client.58619.objecter _finish_command 14 = system:0 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.496+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 4 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.496+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 7 14 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 10 client.58619.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign -1 15 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 20 client.58619.objecter _get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 15 client.58619.objecter _session_command_op_remove -1 15 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 20 client.58619.objecter get_session s=0x7f0384025320 osd=7 3 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 15 client.58619.objecter _session_command_op_assign 7 15 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 4 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 10 client.58619.objecter _send_command 15 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.513+0000 7f03a7276700 1 -- 172.21.15.28:0/3319211125 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0384025f60 con 0x7f0384025a40 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.514+0000 7f03967fc700 1 -- 172.21.15.28:0/3319211125 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0384025f60 con 0x7f0384025a40 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.514+0000 7f03967fc700 10 client.58619.objecter ms_dispatch 0x7f03a0001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.514+0000 7f03967fc700 10 client.58619.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.514+0000 7f03967fc700 20 client.58619.objecter put_session s=0x7f0384025320 osd=7 4 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.514+0000 7f03967fc700 15 client.58619.objecter _session_command_op_remove 7 15 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:03.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:03.692+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fdf74005410 con 0x7fdf90101180 2024-03-30T15:09:03.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:03.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:03.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:03.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:03.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3319211125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: osdmap e682: 8 total, 7 up, 8 in 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:03.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:03.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3319211125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:03.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: osdmap e682: 8 total, 7 up, 8 in 2024-03-30T15:09:03.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:03.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:03.962 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:03.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:04.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:04.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:04.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3319211125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: osdmap e682: 8 total, 7 up, 8 in 2024-03-30T15:09:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:04.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:04.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:04.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:04.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.371+0000 7fdf827fc700 1 -- 172.21.15.28:0/223253797 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v683) ==== 87+0+0 (unknown 3154435188 0 0) 0x7fdf8405a1c0 con 0x7fdf90101180 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 10 librados: watch_flush enter 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 10 librados: watch_flush exit 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 20 client.25425.objecter shutdown clearing up homeless session... 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 10 client.25425.objecter successfully canceled tick 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdf7c070360 legacy=0x7fdf7c072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.375+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 >> v1:172.21.15.28:6789/0 conn(0x7fdf90101180 legacy=0x7fdf90106530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.376+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 shutdown_connections 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.376+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 >> 172.21.15.28:0/223253797 conn(0x7fdf900fca30 msgr2=0x7fdf9006a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:04.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.380+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 shutdown_connections 2024-03-30T15:09:04.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.380+0000 7fdf972ec700 1 -- 172.21.15.28:0/223253797 wait complete. 2024-03-30T15:09:04.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:04.380+0000 7fdf972ec700 1 librados: shutdown 2024-03-30T15:09:04.385 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:09:04.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:04.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:04.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:04.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:04.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:04.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:09:04.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:04.883 INFO:teuthology.orchestra.run.smithi028.stdout:6.d 2024-03-30T15:09:04.883 INFO:teuthology.orchestra.run.smithi028.stdout:6.a 2024-03-30T15:09:04.883 INFO:teuthology.orchestra.run.smithi028.stdout:6.9 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:6.18 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:325.17 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:325.10 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:12.c 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.7 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:4.1a 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.12 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.b 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.6 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.1b 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.1f 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:3.17 2024-03-30T15:09:04.884 INFO:teuthology.orchestra.run.smithi028.stdout:1.0 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:1.2 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:3.1 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:2.1 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:1.6 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:4.19 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:42.11 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:5.8 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:12.13 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:12.1d 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:4.3 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:3.10 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:3.f 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:12.8 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:5.7 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:325.18 2024-03-30T15:09:04.885 INFO:teuthology.orchestra.run.smithi028.stdout:325.12 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:1.1 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:322.5s1 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:3.18 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:12.10 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:5.1d 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:12.a 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:4.7 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:12.3 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:12.6 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:2.0 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:325.1d 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:322.3s0 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:325.2 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:5.2 2024-03-30T15:09:04.886 INFO:teuthology.orchestra.run.smithi028.stdout:5.15 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:12.19 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:5.1e 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:4.0 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:325.7 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:42.8 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:5.e 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:325.f 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:325.14 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:4.16 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:4.1e 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:4.6 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:12.1c 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:5.c 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:42.10 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:42.e 2024-03-30T15:09:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:325.6 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:42.16 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:42.1c 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:325.4 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:42.2 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:42.3 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:5.13 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:6.0 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:325.1c 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:4.b 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:5.1b 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:5.3 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:5.5 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:12.e 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:5.b 2024-03-30T15:09:04.888 INFO:teuthology.orchestra.run.smithi028.stdout:3.3 2024-03-30T15:09:04.889 INFO:teuthology.orchestra.run.smithi028.stdout:6.10 2024-03-30T15:09:04.889 INFO:teuthology.orchestra.run.smithi028.stdout:6.11 2024-03-30T15:09:04.889 INFO:teuthology.orchestra.run.smithi028.stdout:6.13 2024-03-30T15:09:04.889 INFO:teuthology.orchestra.run.smithi028.stdout:6.16 2024-03-30T15:09:04.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:04.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:04.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:05.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:05.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:05.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 6.7 deep-scrub starts 2024-03-30T15:09:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 6.7 deep-scrub ok 2024-03-30T15:09:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: pgmap v1012: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 1.3 scrub starts 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 1.3 scrub ok 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: osdmap e683: 8 total, 7 up, 8 in 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:04 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 6.7 deep-scrub starts 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 6.7 deep-scrub ok 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: pgmap v1012: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 1.3 scrub starts 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 1.3 scrub ok 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T15:09:05.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: osdmap e683: 8 total, 7 up, 8 in 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 6.7 deep-scrub starts 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 6.7 deep-scrub ok 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: pgmap v1012: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 1.3 scrub starts 2024-03-30T15:09:05.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 1.3 scrub ok 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/223253797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: osdmap e683: 8 total, 7 up, 8 in 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:04 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:05.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:05.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:05.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:05.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:05.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:05.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:05.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:05.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:05.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:05.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:05.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:05.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:06.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:06.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:06.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:06.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:06.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:06.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:06.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:06.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:06.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:06.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:06 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:06 smithi134 ceph-mon[33714]: osdmap e684: 8 total, 7 up, 8 in 2024-03-30T15:09:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:06 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:06 smithi134 ceph-mon[33714]: pgmap v1015: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[29876]: osdmap e684: 8 total, 7 up, 8 in 2024-03-30T15:09:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.712 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[29876]: pgmap v1015: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:06.713 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:06.713 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[41614]: osdmap e684: 8 total, 7 up, 8 in 2024-03-30T15:09:06.713 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.713 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:06.713 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:06 smithi028 ceph-mon[41614]: pgmap v1015: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/615 objects degraded (9.756%) 2024-03-30T15:09:06.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.0 --file /var/log/ceph/exp.2.0.0 2024-03-30T15:09:06.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:06.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:06.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:06.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:06.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:09:07.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:07.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:07.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:07.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:07.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:07.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:07.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:09:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:07.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:07.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:07.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:07.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:07 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:07 smithi134 ceph-mon[33714]: osdmap e685: 8 total, 7 up, 8 in 2024-03-30T15:09:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:07 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:07 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 60/615 objects degraded (9.756%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:07.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:07.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[41614]: osdmap e685: 8 total, 7 up, 8 in 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 60/615 objects degraded (9.756%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[29876]: osdmap e685: 8 total, 7 up, 8 in 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]: dispatch 2024-03-30T15:09:07.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:07 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 60/615 objects degraded (9.756%), 17 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:07.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:07.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:07.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:07.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:08.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:08.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:08.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:08.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:08.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:08.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:09:08.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:08.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]': finished 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: osdmap e686: 8 total, 7 up, 8 in 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[29876]: pgmap v1018: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 60/615 objects degraded (9.756%) 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]': finished 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: osdmap e686: 8 total, 7 up, 8 in 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:08 smithi028 ceph-mon[41614]: pgmap v1018: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 60/615 objects degraded (9.756%) 2024-03-30T15:09:08.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:08.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-120", "mode": "writeback"}]': finished 2024-03-30T15:09:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: osdmap e686: 8 total, 7 up, 8 in 2024-03-30T15:09:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:08.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:08 smithi134 ceph-mon[33714]: pgmap v1018: 246 pgs: 32 creating+peering, 17 active+undersized+degraded, 53 active+undersized, 144 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 60/615 objects degraded (9.756%) 2024-03-30T15:09:08.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:08.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:08.730 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:09:08.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:08.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:08.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:08.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:09.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:09.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:09.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:09.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:09.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:09.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:09.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:09.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: osdmap e687: 8 total, 7 up, 8 in 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: osdmap e687: 8 total, 7 up, 8 in 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:09 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: osdmap e687: 8 total, 7 up, 8 in 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:09 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]: dispatch 2024-03-30T15:09:09.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:09.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:09.728 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:09.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:09.849 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:09.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:09.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:09.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:10.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:10.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:10.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:10.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:10.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:10.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:10.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: osdmap e688: 8 total, 7 up, 8 in 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[29876]: pgmap v1021: 246 pgs: 18 active+undersized+degraded, 67 active+undersized, 161 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: osdmap e688: 8 total, 7 up, 8 in 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:10.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:10 smithi028 ceph-mon[41614]: pgmap v1021: 246 pgs: 18 active+undersized+degraded, 67 active+undersized, 161 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:10.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:10.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:10.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:Exporting 2.0 info 2.0( v 478'6 (0'0,478'6] local-lis/les=67/68 n=1 ec=67/67 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-03-30T15:09:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:Read #2:353ec8f9:::gateway.conf:head# 2024-03-30T15:09:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:Export successful 2024-03-30T15:09:10.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-120"}]': finished 2024-03-30T15:09:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: osdmap e688: 8 total, 7 up, 8 in 2024-03-30T15:09:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:10.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:10 smithi134 ceph-mon[33714]: pgmap v1021: 246 pgs: 18 active+undersized+degraded, 67 active+undersized, 161 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:10.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:10.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:10.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:10.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:10.931 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:09:10.931 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:09:10.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:10.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:10.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:11.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.111 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:11.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:11.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:11.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.488 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:11.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[41614]: osdmap e689: 8 total, 7 up, 8 in 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[29876]: osdmap e689: 8 total, 7 up, 8 in 2024-03-30T15:09:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:11 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:11 smithi134 ceph-mon[33714]: osdmap e689: 8 total, 7 up, 8 in 2024-03-30T15:09:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:11.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:11.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:11.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:11.864 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:11.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:11.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:11.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:12.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.0 2024-03-30T15:09:12.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:12.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:12.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:12.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:12.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:12.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:09:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: osdmap e690: 8 total, 7 up, 8 in 2024-03-30T15:09:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: pgmap v1024: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 61/618 objects degraded (9.871%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: osdmap e691: 8 total, 7 up, 8 in 2024-03-30T15:09:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:12 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: osdmap e690: 8 total, 7 up, 8 in 2024-03-30T15:09:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: pgmap v1024: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 61/618 objects degraded (9.871%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: osdmap e691: 8 total, 7 up, 8 in 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: osdmap e690: 8 total, 7 up, 8 in 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: pgmap v1024: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 61/618 objects degraded (9.871%) 2024-03-30T15:09:12.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 61/618 objects degraded (9.871%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:12.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:12.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: osdmap e691: 8 total, 7 up, 8 in 2024-03-30T15:09:12.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:12 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:12.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:12.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:12.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:13.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:13.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:13.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:13.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:13.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:13.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:13.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:13.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:13.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:13.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:13.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: osdmap e692: 8 total, 7 up, 8 in 2024-03-30T15:09:13.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:13 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: osdmap e692: 8 total, 7 up, 8 in 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: osdmap e692: 8 total, 7 up, 8 in 2024-03-30T15:09:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:13 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:13.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:13.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:13.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:13.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:13.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:14.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:14.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:14.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:14.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:14.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:09:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:14.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:14.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:14.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:14.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:14.613 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:14.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:14.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:14 smithi134 ceph-mon[33714]: pgmap v1027: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:14 smithi134 ceph-mon[33714]: osdmap e693: 8 total, 7 up, 8 in 2024-03-30T15:09:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[41614]: pgmap v1027: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[41614]: osdmap e693: 8 total, 7 up, 8 in 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[29876]: pgmap v1027: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[29876]: osdmap e693: 8 total, 7 up, 8 in 2024-03-30T15:09:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]: dispatch 2024-03-30T15:09:14.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:14.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:09:14.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:14.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:14.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 -- 172.21.15.28:0/3748736321 >> v1:172.21.15.28:6790/0 conn(0x7fc5a41022a0 legacy=0x7fc5a4102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 -- 172.21.15.28:0/3748736321 shutdown_connections 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 -- 172.21.15.28:0/3748736321 >> 172.21.15.28:0/3748736321 conn(0x7fc5a40fdb50 msgr2=0x7fc5a40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 -- 172.21.15.28:0/3748736321 shutdown_connections 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 -- 172.21.15.28:0/3748736321 wait complete. 2024-03-30T15:09:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 librados: starting msgr at 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.994+0000 7fc5ab458700 1 librados: starting objecter 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 Processor -- start 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 -- start start 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 librados: setting wanted keys 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 librados: calling monclient init 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a4107b50 con 0x7fc5a410ead0 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a41a86a0 con 0x7fc5a410af30 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5ab458700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc5a41a98c0 con 0x7fc5a41022a0 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5a91f4700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc5a41022a0 0x7fc5a41049a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34050/0 (socket says 172.21.15.28:34050) 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc5a91f4700 1 -- 172.21.15.28:0/847740342 learned_addr learned my addr 172.21.15.28:0/847740342 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter ms_handle_connect 0x7fc5a41022a0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter ms_handle_connect 0x7fc5a410ead0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter ms_handle_connect 0x7fc5a410af30 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 363817731 0 0) 0x7fc5a41a98c0 con 0x7fc5a41022a0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5900031f0 con 0x7fc5a41022a0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396187234 0 0) 0x7fc5a4107b50 con 0x7fc5a410ead0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a41a98c0 con 0x7fc5a410ead0 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3880590113 0 0) 0x7fc5a41a86a0 con 0x7fc5a410af30 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a4107b50 con 0x7fc5a410af30 2024-03-30T15:09:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1222834979 0 0) 0x7fc5900031f0 con 0x7fc5a41022a0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a41a86a0 con 0x7fc5a41022a0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5a0003990 con 0x7fc5a41022a0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4211865719 0 0) 0x7fc5a41a98c0 con 0x7fc5a410ead0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5900031f0 con 0x7fc5a410ead0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc59c001af0 con 0x7fc5a410ead0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.995+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 870266715 0 0) 0x7fc5a4107b50 con 0x7fc5a410af30 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a41a98c0 con 0x7fc5a410af30 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc594001310 con 0x7fc5a410af30 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3821720914 0 0) 0x7fc5a41a86a0 con 0x7fc5a41022a0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 >> v1:172.21.15.134:6789/0 conn(0x7fc5a410af30 legacy=0x7fc5a41050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 >> v1:172.21.15.28:6789/0 conn(0x7fc5a410ead0 legacy=0x7fc5a41a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a41aab20 con 0x7fc5a41022a0 2024-03-30T15:09:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a41a76f0 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc5ab458700 10 client.58631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5a41a7bb0 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc5ab458700 1 librados: init done 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc5a0003ba0 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.996+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc5a0004dc0 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.997+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc588005100 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.997+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc5a0003d50 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(693..693 src has 442..693) ==== 9194+0+0 (unknown 2958840111 0 0) 0x7fc5a0089c10 con 0x7fc5a41022a0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 osd_map(693..693 src has 442..693) 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 3 client.58631.objecter handle_osd_map got epochs [693,693] > 0 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 3 client.58631.objecter handle_osd_map decoding full epoch 693 2024-03-30T15:09:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 20 client.58631.objecter dump_active .. 0 homeless 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v693) ==== 54+0+16 (unknown 1565955074 0 2420000192) 0x7fc5a0089fd0 con 0x7fc5a41022a0 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc590070430 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 10 client.58631.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 1 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 10 client.58631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=694}) -- 0x7fc5880053e0 con 0x7fc5a41022a0 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc588000e30 con 0x7fc5a41022a0 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=693) ==== 24+0+0 (unknown 3476458933 0 0) 0x7fc5a00560e0 con 0x7fc5a41022a0 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59b7fe700 10 client.58631.objecter _check_command_map_dne tid 1 current 693 map_dne_bound 693 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59b7fe700 10 client.58631.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc59b7fe700 15 client.58631.objecter _session_command_op_remove -1 1 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 10 client.58631.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 2 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 2 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 1 2 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 4 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 10 client.58631.objecter _send_command 2 2024-03-30T15:09:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.998+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fc588004ee0 con 0x7fc588002790 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc588002790 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc588004ee0 con 0x7fc588002790 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 2 = system:0 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 4 2024-03-30T15:09:15.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:14.999+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 1 2 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 10 client.58631.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588004730 osd=1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 1 3 2024-03-30T15:09:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 4 2024-03-30T15:09:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 10 client.58631.objecter _send_command 3 2024-03-30T15:09:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.023+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc588004520 con 0x7fc588002790 2024-03-30T15:09:15.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc588004520 con 0x7fc588002790 2024-03-30T15:09:15.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588004730 osd=1 4 2024-03-30T15:09:15.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 1 3 2024-03-30T15:09:15.028 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 10 client.58631.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 4 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.026+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 4 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 2 4 2024-03-30T15:09:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 4 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc5ab458700 10 client.58631.objecter _send_command 4 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fc58800cb30 con 0x7fc588009020 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.027+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc588009020 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.028+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc58800cb30 con 0x7fc588009020 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.028+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.028+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 4 = system:0 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.028+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 4 2024-03-30T15:09:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.028+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 2 4 2024-03-30T15:09:15.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 10 client.58631.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 5 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 5 2024-03-30T15:09:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc5880021c0 osd=2 3 2024-03-30T15:09:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 2 5 2024-03-30T15:09:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 4 2024-03-30T15:09:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 10 client.58631.objecter _send_command 5 2024-03-30T15:09:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.045+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc588001a70 con 0x7fc588009020 2024-03-30T15:09:15.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.047+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc588001a70 con 0x7fc588009020 2024-03-30T15:09:15.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.047+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.047+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.047+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc5880021c0 osd=2 4 2024-03-30T15:09:15.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.047+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 2 5 2024-03-30T15:09:15.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 10 client.58631.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 6 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 6 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 3 6 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 4 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 10 client.58631.objecter _send_command 6 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fc58800ea00 con 0x7fc58800de60 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.048+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc58800de60 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.049+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc58800ea00 con 0x7fc58800de60 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.049+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.049+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 6 = system:0 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.049+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 4 2024-03-30T15:09:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.049+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 3 6 2024-03-30T15:09:15.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 10 client.58631.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 7 2024-03-30T15:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 7 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588003b30 osd=3 3 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 3 7 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 4 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 10 client.58631.objecter _send_command 7 2024-03-30T15:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.067+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc58800f3b0 con 0x7fc58800de60 2024-03-30T15:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc58800f3b0 con 0x7fc58800de60 2024-03-30T15:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588003b30 osd=3 4 2024-03-30T15:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 3 7 2024-03-30T15:09:15.071 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:15.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 10 client.58631.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 8 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 8 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 4 8 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 4 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 10 client.58631.objecter _send_command 8 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.069+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fc588016720 con 0x7fc588012be0 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc588012be0 2024-03-30T15:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc588016720 con 0x7fc588012be0 2024-03-30T15:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 8 = system:0 2024-03-30T15:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 4 2024-03-30T15:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.070+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 4 8 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 10 client.58631.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 9 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 9 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588003860 osd=4 3 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 4 9 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 4 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 10 client.58631.objecter _send_command 9 2024-03-30T15:09:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.088+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc58800ee30 con 0x7fc588012be0 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.090+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc58800ee30 con 0x7fc588012be0 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.090+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.090+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.090+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588003860 osd=4 4 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.090+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 4 9 2024-03-30T15:09:15.092 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 10 client.58631.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 10 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 10 2024-03-30T15:09:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 5 10 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 4 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 10 client.58631.objecter _send_command 10 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fc58800ed50 con 0x7fc58800d9c0 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.091+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc58800d9c0 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.092+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc58800ed50 con 0x7fc58800d9c0 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.092+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.092+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 10 = system:0 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.092+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 4 2024-03-30T15:09:15.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.092+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 5 10 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 10 client.58631.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 11 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 11 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc58800ebe0 osd=5 3 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 5 11 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 4 2024-03-30T15:09:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 10 client.58631.objecter _send_command 11 2024-03-30T15:09:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.109+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc58801f6c0 con 0x7fc58800d9c0 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc58801f6c0 con 0x7fc58800d9c0 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc58800ebe0 osd=5 4 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 5 11 2024-03-30T15:09:15.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 10 client.58631.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 12 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 12 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 6 12 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 4 2024-03-30T15:09:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 10 client.58631.objecter _send_command 12 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fc5880237d0 con 0x7fc58801fbd0 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.111+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc58801fbd0 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.112+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc5880237d0 con 0x7fc58801fbd0 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.112+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.112+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 12 = system:0 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.112+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 4 2024-03-30T15:09:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.112+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 6 12 2024-03-30T15:09:15.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 10 client.58631.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 13 2024-03-30T15:09:15.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 13 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc58800f000 osd=6 3 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 6 13 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 4 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 10 client.58631.objecter _send_command 13 2024-03-30T15:09:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.129+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc588025180 con 0x7fc58801fbd0 2024-03-30T15:09:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc588025180 con 0x7fc58801fbd0 2024-03-30T15:09:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc58800f000 osd=6 4 2024-03-30T15:09:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 6 13 2024-03-30T15:09:15.133 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 10 client.58631.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 14 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 14 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 7 14 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 4 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 10 client.58631.objecter _send_command 14 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.131+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fc588025dc0 con 0x7fc5880258a0 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.132+0000 7fc59a7fc700 10 client.58631.objecter ms_handle_connect 0x7fc5880258a0 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.132+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc588025dc0 con 0x7fc5880258a0 2024-03-30T15:09:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.132+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.133+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 14 = system:0 2024-03-30T15:09:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.133+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 4 2024-03-30T15:09:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.133+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 7 14 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 10 client.58631.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign -1 15 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 20 client.58631.objecter _get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_remove -1 15 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 20 client.58631.objecter get_session s=0x7fc588025180 osd=7 3 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 15 client.58631.objecter _session_command_op_assign 7 15 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 4 2024-03-30T15:09:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 10 client.58631.objecter _send_command 15 2024-03-30T15:09:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.150+0000 7fc5ab458700 1 -- 172.21.15.28:0/847740342 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc588025dc0 con 0x7fc5880258a0 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.151+0000 7fc59a7fc700 1 -- 172.21.15.28:0/847740342 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc588025dc0 con 0x7fc5880258a0 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.151+0000 7fc59a7fc700 10 client.58631.objecter ms_dispatch 0x7fc5a4001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.151+0000 7fc59a7fc700 10 client.58631.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.151+0000 7fc59a7fc700 20 client.58631.objecter put_session s=0x7fc588025180 osd=7 4 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:15.151+0000 7fc59a7fc700 15 client.58631.objecter _session_command_op_remove 7 15 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:15.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:15.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:15.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:15.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:15.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:15.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:15.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:15.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:15.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/847740342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:15.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:15.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]': finished 2024-03-30T15:09:15.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[29876]: osdmap e694: 8 total, 7 up, 8 in 2024-03-30T15:09:15.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/847740342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:15.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:15.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]': finished 2024-03-30T15:09:15.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:15 smithi028 ceph-mon[41614]: osdmap e694: 8 total, 7 up, 8 in 2024-03-30T15:09:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/847740342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:15 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-122", "mode": "writeback"}]': finished 2024-03-30T15:09:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:15 smithi134 ceph-mon[33714]: osdmap e694: 8 total, 7 up, 8 in 2024-03-30T15:09:15.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:15.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:15.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:15.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:15.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: marking collection for removal 2024-03-30T15:09:15.985 INFO:teuthology.orchestra.run.smithi028.stdout:setting '_remove' omap key 2024-03-30T15:09:15.985 INFO:teuthology.orchestra.run.smithi028.stdout:finish_remove_pgs 2.0_head removing 2.0 2024-03-30T15:09:15.986 INFO:teuthology.orchestra.run.smithi028.stdout:Remove successful 2024-03-30T15:09:16.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:16.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:16.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:16.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:16.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:16.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:16.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:16.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:16.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:16.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[41614]: pgmap v1030: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:16.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:16 smithi028 ceph-mon[29876]: pgmap v1030: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:16.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:16.570 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:16.570 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:09:16.570 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:09:16.570 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:16.571 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.571 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:16 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:16.571 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:16 smithi134 ceph-mon[33714]: pgmap v1030: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 61/621 objects degraded (9.823%) 2024-03-30T15:09:16.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:16.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:16.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:16.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:16.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:16.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:16.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:17.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:17.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:17.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:17.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:17.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:09:17.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:17.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:17.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:17.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:17.511 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:17.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:17.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:17.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:17.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: osdmap e695: 8 total, 7 up, 8 in 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 61/621 objects degraded (9.823%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: osdmap e695: 8 total, 7 up, 8 in 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 61/621 objects degraded (9.823%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:17.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:17 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:17.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:17.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: osdmap e695: 8 total, 7 up, 8 in 2024-03-30T15:09:17.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]: dispatch 2024-03-30T15:09:17.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 61/621 objects degraded (9.823%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:17.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:17 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:17.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:17.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:17.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:17.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:18.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:18.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:18.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:18.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:18.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.0.0 2024-03-30T15:09:18.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:09:18.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:18.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:09:18.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[41614]: osdmap e696: 8 total, 7 up, 8 in 2024-03-30T15:09:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[41614]: pgmap v1033: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 61/621 objects degraded (9.823%) 2024-03-30T15:09:18.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[29876]: osdmap e696: 8 total, 7 up, 8 in 2024-03-30T15:09:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:18 smithi028 ceph-mon[29876]: pgmap v1033: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 61/621 objects degraded (9.823%) 2024-03-30T15:09:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:18.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:18.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:18 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:18.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:18 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-122"}]': finished 2024-03-30T15:09:18.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:18 smithi134 ceph-mon[33714]: osdmap e696: 8 total, 7 up, 8 in 2024-03-30T15:09:18.551 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:18 smithi134 ceph-mon[33714]: pgmap v1033: 246 pgs: 18 active+undersized+degraded, 57 active+undersized, 171 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 61/621 objects degraded (9.823%) 2024-03-30T15:09:18.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:18.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:18.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:18.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:18.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:18.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:18.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:18.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:19.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:19.197 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:19.198 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:09:19.198 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:09:19.198 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:09:19.198 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:19.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:19.236 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:19.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:19.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:19.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:19.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:19.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:19.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:19.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:19 smithi028 ceph-mon[29876]: osdmap e697: 8 total, 7 up, 8 in 2024-03-30T15:09:19.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:19 smithi028 ceph-mon[41614]: osdmap e697: 8 total, 7 up, 8 in 2024-03-30T15:09:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:19.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.855+0000 7f9414ad4700 1 -- 172.21.15.28:0/987371741 >> v1:172.21.15.28:6789/0 conn(0x7f9410102240 legacy=0x7f9410102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.855+0000 7f9414ad4700 1 -- 172.21.15.28:0/987371741 shutdown_connections 2024-03-30T15:09:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.855+0000 7f9414ad4700 1 -- 172.21.15.28:0/987371741 >> 172.21.15.28:0/987371741 conn(0x7f94100fdaf0 msgr2=0x7f94100fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.855+0000 7f9414ad4700 1 -- 172.21.15.28:0/987371741 shutdown_connections 2024-03-30T15:09:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.855+0000 7f9414ad4700 1 -- 172.21.15.28:0/987371741 wait complete. 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 librados: starting msgr at 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 librados: starting objecter 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 Processor -- start 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 -- start start 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 librados: setting wanted keys 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 librados: calling monclient init 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94101a93f0 con 0x7f941010ea70 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94101aa610 con 0x7f9410102240 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f9414ad4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f94101ab850 con 0x7f941010aed0 2024-03-30T15:09:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f940dd9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f941010aed0 0x7f94101a42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48810/0 (socket says 172.21.15.28:48810) 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f940dd9b700 1 -- 172.21.15.28:0/2054086848 learned_addr learned my addr 172.21.15.28:0/2054086848 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f940ed9d700 1 --1- 172.21.15.28:0/2054086848 >> v1:172.21.15.28:6789/0 conn(0x7f941010ea70 0x7f94101a7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37688/0 (socket says 172.21.15.28:37688) 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter ms_handle_connect 0x7f941010ea70 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter ms_handle_connect 0x7f941010aed0 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter ms_handle_connect 0x7f9410102240 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.856+0000 7f94077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927542656 0 0) 0x7f94101a93f0 con 0x7f941010ea70 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93fc0031f0 con 0x7f941010ea70 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243318170 0 0) 0x7f94101ab850 con 0x7f941010aed0 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94101a93f0 con 0x7f941010aed0 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3945285749 0 0) 0x7f94101aa610 con 0x7f9410102240 2024-03-30T15:09:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94101ab850 con 0x7f9410102240 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2896655438 0 0) 0x7f93fc0031f0 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94101aa610 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9408001b60 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1643061570 0 0) 0x7f94101a93f0 con 0x7f941010aed0 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93fc0031f0 con 0x7f941010aed0 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f93f8001230 con 0x7f941010aed0 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1599392871 0 0) 0x7f94101ab850 con 0x7f9410102240 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94101a93f0 con 0x7f9410102240 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f94000044a0 con 0x7f9410102240 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 594625601 0 0) 0x7f94101aa610 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 >> v1:172.21.15.28:6790/0 conn(0x7f941010aed0 legacy=0x7f94101a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 >> v1:172.21.15.134:6789/0 conn(0x7f9410102240 legacy=0x7f9410107620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94101aca90 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94101aa7e0 con 0x7f941010ea70 2024-03-30T15:09:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f9414ad4700 10 client.25437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f94101aaca0 con 0x7f941010ea70 2024-03-30T15:09:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f9414ad4700 1 librados: init done 2024-03-30T15:09:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9408003ec0 con 0x7f941010ea70 2024-03-30T15:09:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.857+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9408005f40 con 0x7f941010ea70 2024-03-30T15:09:19.862 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:19 smithi134 ceph-mon[33714]: osdmap e697: 8 total, 7 up, 8 in 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.858+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f0005120 con 0x7f941010ea70 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9408003a20 con 0x7f941010ea70 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(698..698 src has 442..698) ==== 9186+0+0 (unknown 1060683135 0 0) 0x7f9408089dd0 con 0x7f941010ea70 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 10 client.25437.objecter ms_dispatch 0x7f9410001550 osd_map(698..698 src has 442..698) 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 3 client.25437.objecter handle_osd_map got epochs [698,698] > 0 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 3 client.25437.objecter handle_osd_map decoding full epoch 698 2024-03-30T15:09:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.859+0000 7f94077fe700 20 client.25437.objecter dump_active .. 0 homeless 2024-03-30T15:09:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.860+0000 7f94077fe700 10 client.25437.objecter ms_handle_connect 0x7f93fc070530 2024-03-30T15:09:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.864 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:19.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:19.864+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9408056be0 con 0x7f941010ea70 2024-03-30T15:09:19.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:09:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:19.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:19.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:19.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.086+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f93f0005410 con 0x7f941010ea70 2024-03-30T15:09:20.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:20.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:20.168 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:20.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:20.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:20.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:20.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.457+0000 7f94077fe700 1 -- 172.21.15.28:0/2054086848 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v699) ==== 97+0+0 (unknown 2922990973 0 0) 0x7f940805a630 con 0x7f941010ea70 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.461+0000 7f9414ad4700 10 librados: watch_flush enter 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.462+0000 7f9414ad4700 10 librados: watch_flush exit 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.462+0000 7f9414ad4700 20 client.25437.objecter shutdown clearing up homeless session... 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.462+0000 7f9414ad4700 10 client.25437.objecter successfully canceled tick 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.462+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 >> v1:172.21.15.28:6800/3011431093 conn(0x7f93fc070530 legacy=0x7f93fc0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.463+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 >> v1:172.21.15.28:6789/0 conn(0x7f941010ea70 legacy=0x7f94101a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.463+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 shutdown_connections 2024-03-30T15:09:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.463+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 >> 172.21.15.28:0/2054086848 conn(0x7f94100fdaf0 msgr2=0x7f9410111c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.464+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 shutdown_connections 2024-03-30T15:09:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.464+0000 7f9414ad4700 1 -- 172.21.15.28:0/2054086848 wait complete. 2024-03-30T15:09:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:20.464+0000 7f9414ad4700 1 librados: shutdown 2024-03-30T15:09:20.467 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:09:20.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:20.498 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:09:20.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:20.579 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-mon[33714]: osdmap e698: 8 total, 7 up, 8 in 2024-03-30T15:09:20.579 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.580 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.580 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-mon[33714]: pgmap v1036: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:20.580 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:20.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:20.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[41614]: osdmap e698: 8 total, 7 up, 8 in 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[41614]: pgmap v1036: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:20.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[29876]: osdmap e698: 8 total, 7 up, 8 in 2024-03-30T15:09:20.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:20.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[29876]: pgmap v1036: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:20.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:20.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:20.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:20.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:20.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:20.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:21.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:21.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:21.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:21.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:21.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:21.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:21.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:21.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:21 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:21.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:21.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:21.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:21 smithi134 ceph-mon[33714]: osdmap e699: 8 total, 7 up, 8 in 2024-03-30T15:09:21.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[41614]: osdmap e699: 8 total, 7 up, 8 in 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2054086848' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[29876]: osdmap e699: 8 total, 7 up, 8 in 2024-03-30T15:09:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:21.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:21.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57281.0:1 221.6s0 221:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:09:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 578.55343922199995, 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1519453040000001, 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57281", 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4286010033", 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208241+0000", 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208243+0000", 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1550000000000001e-06 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208253+0000", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3139999999999993e-06 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208264+0000", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1592e-05 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.208277+0000", 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2703e-05 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209635+0000", 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001357638 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:43.209639+0000", 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6949999999999999e-06 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358223+0000", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1485833839999999 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358319+0000", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6644999999999994e-05 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.358620+0000", 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030071200000000002 2024-03-30T15:09:21.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.359145+0000", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052535199999999998 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360113+0000", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096798800000000003 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:44.360187+0000", 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3126000000000004e-05 2024-03-30T15:09:21.771 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49354.0:1 224.6s1 224:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 571.55742541799998, 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025966369, 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49354", 2024-03-30T15:09:21.772 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1904812330", 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204255+0000", 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204256+0000", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.39e-06 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204263+0000", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0669999999999997e-06 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204271+0000", 2024-03-30T15:09:21.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1880000000000002e-06 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.204279+0000", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5639999999999997e-06 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228587+0000", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.024308065 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:50.228592+0000", 2024-03-30T15:09:21.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222766+0000", 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99417422799999999 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.222903+0000", 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013736600000000001 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.223366+0000", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000463187 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.226776+0000", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003409287 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230143+0000", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033669149999999998 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T14:59:51.230221+0000", 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8844000000000006e-05 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57626.0:1 262.5s0 262:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 497.52611026699998, 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0934634089999999, 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57626", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1398925", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235570+0000", 2024-03-30T15:09:21.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235572+0000", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9889999999999999e-06 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235581+0000", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8370000000000003e-06 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235591+0000", 2024-03-30T15:09:21.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0292000000000001e-05 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.235602+0000", 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0976e-05 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237121+0000", 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015190379999999999 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:04.237126+0000", 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1679999999999997e-06 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326125+0000", 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0889992150000001 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326273+0000", 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014821499999999999 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.326919+0000", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064574200000000004 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.327411+0000", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00049205899999999999 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.328990+0000", 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001578861 2024-03-30T15:09:21.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:05.329034+0000", 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4017000000000002e-05 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49657.0:1 263.6s0 263:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 494.53323177200002, 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.017319423, 2024-03-30T15:09:21.784 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49657", 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2923162696", 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228449+0000", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228451+0000", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4700000000000001e-06 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228459+0000", 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8490000000000006e-06 2024-03-30T15:09:21.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228469+0000", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5219999999999996e-06 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.228478+0000", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2150000000000007e-06 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238328+0000", 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098497410000000004 2024-03-30T15:09:21.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:07.238331+0000", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9009999999999996e-06 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242049+0000", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0037178710000001 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242184+0000", 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0001351 2024-03-30T15:09:21.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.242588+0000", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040368400000000002 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.244103+0000", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015145740000000001 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245701+0000", 2024-03-30T15:09:21.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015980180000000001 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:08.245768+0000", 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7477999999999993e-05 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24537.0:1 270.6s0 270:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:09:21.790 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 473.504825907, 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0134851680000001, 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24537", 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2126246388", 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256855+0000", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256858+0000", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6389999999999998e-06 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256868+0000", 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0231000000000001e-05 2024-03-30T15:09:21.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256886+0000", 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7988000000000001e-05 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.256901+0000", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4817999999999999e-05 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261660+0000", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004759012 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:28.261665+0000", 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0649999999999998e-06 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267449+0000", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0057839420000001 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.267596+0000", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014701699999999999 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.268682+0000", 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001085325 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.269079+0000", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039733100000000001 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270298+0000", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001219354 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:29.270340+0000", 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1446000000000003e-05 2024-03-30T15:09:21.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57869.0:1 286.5s0 286:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 414.48718576300001, 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0109918019999999, 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.798 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57869", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/714138788", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274498+0000", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9220000000000001e-06 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274509+0000", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1678999999999999e-05 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274525+0000", 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5781e-05 2024-03-30T15:09:21.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274540+0000", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4808e-05 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277431+0000", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028910849999999998 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277436+0000", 2024-03-30T15:09:21.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4589999999999997e-06 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.282966+0000", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0055296760000001 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283097+0000", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013078699999999999 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283531+0000", 2024-03-30T15:09:21.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043440599999999997 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283992+0000", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046019300000000002 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285413+0000", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014217050000000001 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285487+0000", 2024-03-30T15:09:21.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3300999999999995e-05 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57923.0:1 290.7s0 290:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 393.501285376, 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.011345699, 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57923", 2024-03-30T15:09:21.804 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3909045240", 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260397+0000", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.733e-06 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260404+0000", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.221e-06 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260415+0000", 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0659000000000001e-05 2024-03-30T15:09:21.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260425+0000", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0703999999999999e-05 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260972+0000", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054698500000000005 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260976+0000", 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1020000000000001e-06 2024-03-30T15:09:21.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269376+0000", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.008400816 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269503+0000", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126236 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269916+0000", 2024-03-30T15:09:21.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000413743 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271218+0000", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013020810000000001 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271674+0000", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00045557200000000001 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271741+0000", 2024-03-30T15:09:21.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6847000000000006e-05 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.6.52:31 101.d 101:b5bd8322:test-rados-api-smithi028-93253-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e520)", 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 208.440721572, 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050572109999999998, 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.810 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320961+0000", 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7549999999999999e-06 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320966+0000", 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8039999999999998e-06 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320979+0000", 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2858e-05 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320987+0000", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1110000000000008e-06 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321177+0000", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018962499999999999 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321306+0000", 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012909299999999999 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321547+0000", 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024123600000000001 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322529+0000", 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00098205800000000006 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322707+0000", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000177629 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325919+0000", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032115210000000002 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325973+0000", 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4014999999999997e-05 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.326016+0000", 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3506e-05 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(42.b,QREQUEST,e:589,from:4,reservation_nonce:7)", 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 121.94745270200001, 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005843053, 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.816 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814230+0000", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0260000000000002e-06 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814236+0000", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.913e-06 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814251+0000", 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5138e-05 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814259+0000", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2509999999999994e-06 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814321+0000", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1585000000000001e-05 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814327+0000", 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3740000000000003e-06 2024-03-30T15:09:21.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.819987+0000", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005660389 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820005+0000", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7144000000000001e-05 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820071+0000", 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6233000000000001e-05 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.820 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:2969 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94326471854208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:12.657744+0000", 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 69.103936934000004, 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091109899999999996, 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2969 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.821 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657744+0000", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657744+0000", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657749+0000", 2024-03-30T15:09:21.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5960000000000003e-06 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657767+0000", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8003e-05 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657815+0000", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8041999999999998e-05 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657843+0000", 2024-03-30T15:09:21.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7800000000000001e-05 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658290+0000", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000447429 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658467+0000", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000176916 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658655+0000", 2024-03-30T15:09:21.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018731299999999999 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:2970 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94326471858816 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:12.657905+0000", 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 69.103775782, 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00082916100000000003, 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:09:21.825 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2970 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657905+0000", 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657905+0000", 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657907+0000", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2670000000000001e-06 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657916+0000", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4599999999999992e-06 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657968+0000", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1181999999999999e-05 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.657989+0000", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1092999999999999e-05 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658550+0000", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00056108199999999997 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658650+0000", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9926999999999994e-05 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:12.658734+0000", 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4149999999999999e-05 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:14 323.1c 323:38000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 64.375726405999998, 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035226210000000001, 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.829 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.830 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385956+0000", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3300000000000001e-06 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385963+0000", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2639999999999997e-06 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385971+0000", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7199999999999995e-06 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385979+0000", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8450000000000003e-06 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386035+0000", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5769000000000001e-05 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386039+0000", 2024-03-30T15:09:21.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3150000000000002e-06 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389352+0000", 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033128849999999998 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389388+0000", 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6260999999999997e-05 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389477+0000", 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8232000000000005e-05 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:48 323.1c 323:38000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 62.375645220999999, 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033340269999999998, 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 48 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:09:21.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386038+0000", 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7089999999999998e-06 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386052+0000", 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3910000000000001e-05 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386066+0000", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3679000000000001e-05 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386075+0000", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9979999999999992e-06 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386131+0000", 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6069000000000001e-05 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386135+0000", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6399999999999996e-06 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389257+0000", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031219440000000002 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389283+0000", 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5605e-05 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389369+0000", 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6472999999999997e-05 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(4.1,QREQUEST,e:674,from:4,reservation_nonce:12)", 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:55.420365+0000", 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 26.341315774999998, 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024725340000000002, 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.839 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420365+0000", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420365+0000", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420367+0000", 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.03e-06 2024-03-30T15:09:21.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420374+0000", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9829999999999999e-06 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420386+0000", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2272999999999999e-05 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420393+0000", 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1840000000000002e-06 2024-03-30T15:09:21.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420456+0000", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2862999999999998e-05 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.420462+0000", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5720000000000002e-06 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.422758+0000", 2024-03-30T15:09:21.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002296841 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.422780+0000", 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1177000000000001e-05 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.422837+0000", 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7611000000000001e-05 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:374 326.1 326:86d02a7e:test-rados-api-smithi028-93253-119::foo:3 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.390447559999998, 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098792810000000002, 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 374 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371235+0000", 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3199999999999998e-06 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371242+0000", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.72e-06 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371254+0000", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2452999999999999e-05 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371267+0000", 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2567e-05 2024-03-30T15:09:21.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371317+0000", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9808999999999997e-05 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371321+0000", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.279e-06 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373335+0000", 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002014264 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373568+0000", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023216 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379648+0000", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060801650000000002 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379884+0000", 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023607599999999999 2024-03-30T15:09:21.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381008+0000", 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011244440000000001 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381112+0000", 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010402400000000001 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:409 326.1 326:86d02a7e:test-rados-api-smithi028-93253-119::foo:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:59.402099+0000", 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.359581379000002, 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00084387199999999996, 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 409 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402099+0000", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402099+0000", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402100+0000", 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1800000000000005e-07 2024-03-30T15:09:21.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402103+0000", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2009999999999999e-06 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402110+0000", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5220000000000004e-06 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402115+0000", 2024-03-30T15:09:21.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.096e-06 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402152+0000", 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7469999999999999e-05 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402216+0000", 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4035000000000006e-05 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4", 2024-03-30T15:09:21.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402293+0000", 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6699000000000005e-05 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402603+0000", 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031048300000000002 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402850+0000", 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024625599999999998 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402883+0000", 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2990000000000001e-05 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.402943+0000", 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0302e-05 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:414 326.4 326:20000000::::head [pgnls start_epoch 680] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e680)", 2024-03-30T15:09:21.855 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:01.390153+0000", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 20.371527477000001, 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026049390000000001, 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 414 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390153+0000", 2024-03-30T15:09:21.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390153+0000", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390156+0000", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1049999999999999e-06 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390165+0000", 2024-03-30T15:09:21.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5990000000000002e-06 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390180+0000", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5034999999999999e-05 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390188+0000", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3149999999999992e-06 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390251+0000", 2024-03-30T15:09:21.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2516999999999998e-05 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.390257+0000", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2650000000000002e-06 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392646+0000", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023891759999999998 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392680+0000", 2024-03-30T15:09:21.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3581999999999998e-05 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392758+0000", 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8344999999999996e-05 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:739 328.6 328:60000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.2931103019999997, 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003540091, 2024-03-30T15:09:21.860 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 739 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468573+0000", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1130000000000001e-06 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468584+0000", 2024-03-30T15:09:21.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0752e-05 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468605+0000", 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0743000000000001e-05 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468617+0000", 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1994000000000001e-05 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468677+0000", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9876000000000001e-05 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468681+0000", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0949999999999998e-06 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.471982+0000", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033012670000000001 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472017+0000", 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4706999999999998e-05 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472110+0000", 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3543999999999997e-05 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:796 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:20.487521+0000", 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 1.2741596479999999, 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013394259999999999, 2024-03-30T15:09:21.865 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 796 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487521+0000", 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487521+0000", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487523+0000", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0329999999999998e-06 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487530+0000", 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.657e-06 2024-03-30T15:09:21.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487540+0000", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0246e-05 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487549+0000", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8589999999999994e-06 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487598+0000", 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9026000000000001e-05 2024-03-30T15:09:21.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.487693+0000", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5081000000000007e-05 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.488144+0000", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00045124899999999998 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.488497+0000", 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00035297200000000001 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.488785+0000", 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028836999999999998 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:20.488860+0000", 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4932999999999996e-05 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.870 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:801 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:head [writefull 0~5] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:21.480807+0000", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.28087390499999998, 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019369560000000001, 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 801 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:21.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480807+0000", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480807+0000", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480810+0000", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3680000000000002e-06 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480820+0000", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8849999999999994e-06 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480841+0000", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1270000000000001e-05 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480852+0000", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1206000000000001e-05 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:21.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.480919+0000", 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6217000000000007e-05 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.481033+0000", 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114182 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.481727+0000", 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00069433600000000004 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.482324+0000", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00059670600000000002 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.482665+0000", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034141500000000002 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:21.482744+0000", 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8370999999999999e-05 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.875 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:21.876 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:21.876 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:21.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:21.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:21.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:22.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:22.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:22.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:22.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:22.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:22.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:22.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:22.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:22.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:09:22.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:22.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:22.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: osdmap e700: 8 total, 7 up, 8 in 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: pgmap v1039: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/624 objects degraded (9.936%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: osdmap e701: 8 total, 7 up, 8 in 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:22.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: osdmap e700: 8 total, 7 up, 8 in 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: pgmap v1039: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/624 objects degraded (9.936%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: osdmap e701: 8 total, 7 up, 8 in 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:22.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:22.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:22.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:22.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: osdmap e700: 8 total, 7 up, 8 in 2024-03-30T15:09:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: pgmap v1039: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/624 objects degraded (9.936%) 2024-03-30T15:09:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/624 objects degraded (9.936%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: osdmap e701: 8 total, 7 up, 8 in 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:23.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:get_pg_num_history pg_num_history pg_num_history(e668 pg_nums {1={20=1,559=11},2={67=3},3={68=32},4={70=32},5={72=32},6={74=32},12={110=32},42={113=32},101={130=32},296={430=32},297={448=32},298={458=32},299={462=32},300={465=32},301={478=32},302={484=32},303={489=32},304={493=32},305={503=32},306={513=32},307={522=32},308={531=32},309={533=32},310={540=32},311={546=32},312={551=32},313={568=32},314={575=32},315={587=32},316={599=32},317={608=8},318={610=32},319={612=32},320={623=8},321={624=32},322={644=8},323={645=32},324={652=32},325={660=32}} deleted_pools 447,296,457,297,461,298,464,299,477,300,483,301,488,302,492,303,502,304,512,305,521,306,530,307,538,309,539,308,545,310,550,311,567,312,574,313,586,314,598,315,604,316,605,101,619,319,620,318,635,321,636,320,639,317,651,323,659,324) 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:Importing pgid 2.0 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:Write #2:353ec8f9:::gateway.conf:head# 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:write_pg epoch 645 info 2.0( v 478'6 (0'0,478'6] local-lis/les=67/68 n=1 ec=67/67 lis/c=67/67 les/c/f=68/68/0 sis=67) 2024-03-30T15:09:23.093 INFO:teuthology.orchestra.run.smithi028.stdout:Import successful 2024-03-30T15:09:23.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:09:23.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:23.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:23.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:23.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:23.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:23.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:23.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:23.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:23.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: osdmap e702: 8 total, 7 up, 8 in 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: osdmap e702: 8 total, 7 up, 8 in 2024-03-30T15:09:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:23.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:23.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:23.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:23.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: osdmap e702: 8 total, 7 up, 8 in 2024-03-30T15:09:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]: dispatch 2024-03-30T15:09:24.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:24.038 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:24.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:24.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:24.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:24.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:24.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:24.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:24.528 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:24.528 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:09:24.528 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:09:24.528 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: pgmap v1042: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]': finished 2024-03-30T15:09:24.529 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:24 smithi134 ceph-mon[33714]: osdmap e703: 8 total, 7 up, 8 in 2024-03-30T15:09:24.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:24.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: pgmap v1042: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:24.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]': finished 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[29876]: osdmap e703: 8 total, 7 up, 8 in 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: pgmap v1042: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-124", "mode": "writeback"}]': finished 2024-03-30T15:09:24.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:24 smithi028 ceph-mon[41614]: osdmap e703: 8 total, 7 up, 8 in 2024-03-30T15:09:24.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:24.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:24.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.807+0000 7f3c67b5a700 1 -- 172.21.15.28:0/3769600049 >> v1:172.21.15.28:6790/0 conn(0x7f3c60101140 legacy=0x7f3c60101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- 172.21.15.28:0/3769600049 shutdown_connections 2024-03-30T15:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- 172.21.15.28:0/3769600049 >> 172.21.15.28:0/3769600049 conn(0x7f3c600fc9f0 msgr2=0x7f3c600fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- 172.21.15.28:0/3769600049 shutdown_connections 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- 172.21.15.28:0/3769600049 wait complete. 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 librados: starting msgr at 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 librados: starting objecter 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 Processor -- start 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- start start 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 librados: setting wanted keys 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 librados: calling monclient init 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c601a9420 con 0x7f3c6010d970 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c601aa640 con 0x7f3c60101140 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c67b5a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3c601ab880 con 0x7f3c60109dd0 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c650f5700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3c60109dd0 0x7f3c601a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48834/0 (socket says 172.21.15.28:48834) 2024-03-30T15:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c660f7700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3c6010d970 0x7f3c601a7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37722/0 (socket says 172.21.15.28:37722) 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.808+0000 7f3c650f5700 1 -- 172.21.15.28:0/605205541 learned_addr learned my addr 172.21.15.28:0/605205541 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter ms_handle_connect 0x7f3c6010d970 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter ms_handle_connect 0x7f3c60109dd0 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter ms_handle_connect 0x7f3c60101140 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772500372 0 0) 0x7f3c601a9420 con 0x7f3c6010d970 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c48003aa0 con 0x7f3c6010d970 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395895294 0 0) 0x7f3c601ab880 con 0x7f3c60109dd0 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c601a9420 con 0x7f3c60109dd0 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192479392 0 0) 0x7f3c601aa640 con 0x7f3c60101140 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c601ab880 con 0x7f3c60101140 2024-03-30T15:09:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2104026928 0 0) 0x7f3c48003aa0 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c601aa640 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4052419318 0 0) 0x7f3c601a9420 con 0x7f3c60109dd0 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c48003aa0 con 0x7f3c60109dd0 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3c50001ce0 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3c540018b0 con 0x7f3c60109dd0 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1443986659 0 0) 0x7f3c601ab880 con 0x7f3c60101140 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c601a9420 con 0x7f3c60101140 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3c58001510 con 0x7f3c60101140 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 725904677 0 0) 0x7f3c601aa640 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 >> v1:172.21.15.28:6790/0 conn(0x7f3c60109dd0 legacy=0x7f3c601a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 >> v1:172.21.15.134:6789/0 conn(0x7f3c60101140 legacy=0x7f3c60106440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c601acac0 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c601abab0 con 0x7f3c6010d970 2024-03-30T15:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c67b5a700 10 client.25446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c601abf70 con 0x7f3c6010d970 2024-03-30T15:09:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c67b5a700 1 librados: init done 2024-03-30T15:09:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3c50003ea0 con 0x7f3c6010d970 2024-03-30T15:09:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.809+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3c50005f70 con 0x7f3c6010d970 2024-03-30T15:09:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.810+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3c4c005100 con 0x7f3c6010d970 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.811+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3c50003a00 con 0x7f3c6010d970 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(703..703 src has 442..703) ==== 9194+0+0 (unknown 1067621499 0 0) 0x7f3c50089b50 con 0x7f3c6010d970 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 osd_map(703..703 src has 442..703) 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 3 client.25446.objecter handle_osd_map got epochs [703,703] > 0 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 3 client.25446.objecter handle_osd_map decoding full epoch 703 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 20 client.25446.objecter dump_active .. 0 homeless 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v703) ==== 54+0+16 (unknown 534793240 0 2420000192) 0x7f3c5008a060 con 0x7f3c6010d970 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c480703a0 2024-03-30T15:09:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 1 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 20 client.25446.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 10 client.25446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=704}) -- 0x7f3c4c0053e0 con 0x7f3c6010d970 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3c4c000ed0 con 0x7f3c6010d970 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 2804404127 0 0) 0x7f3c50056020 con 0x7f3c6010d970 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5ffff700 10 client.25446.objecter _check_command_map_dne tid 1 current 703 map_dne_bound 703 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5ffff700 10 client.25446.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c5ffff700 15 client.25446.objecter _session_command_op_remove -1 1 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 2 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.812+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 2 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 1 2 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 4 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c67b5a700 10 client.25446.objecter _send_command 2 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c0022f0 con 0x7f3c4c003910 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c003910 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c0022f0 con 0x7f3c4c003910 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 10 client.25446.objecter _finish_command 2 = system:0 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 4 2024-03-30T15:09:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.813+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 1 2 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c001bb0 osd=1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 1 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 4 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 10 client.25446.objecter _send_command 3 2024-03-30T15:09:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.830+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c004c80 con 0x7f3c4c003910 2024-03-30T15:09:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c004c80 con 0x7f3c4c003910 2024-03-30T15:09:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 10 client.25446.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c001bb0 osd=1 4 2024-03-30T15:09:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 1 3 2024-03-30T15:09:24.834 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 4 2024-03-30T15:09:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 4 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 2 4 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 4 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 10 client.25446.objecter _send_command 4 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c00ca20 con 0x7f3c4c008f10 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.832+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c008f10 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.833+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c00ca20 con 0x7f3c4c008f10 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.833+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.833+0000 7f3c5effd700 10 client.25446.objecter _finish_command 4 = system:0 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.833+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 4 2024-03-30T15:09:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.833+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 2 4 2024-03-30T15:09:24.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 5 2024-03-30T15:09:24.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 5 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c004ed0 osd=2 3 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 2 5 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 4 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 10 client.25446.objecter _send_command 5 2024-03-30T15:09:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.850+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c0047c0 con 0x7f3c4c008f10 2024-03-30T15:09:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c0047c0 con 0x7f3c4c008f10 2024-03-30T15:09:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 10 client.25446.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c004ed0 osd=2 4 2024-03-30T15:09:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 2 5 2024-03-30T15:09:24.854 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 6 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 6 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 3 6 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 4 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 10 client.25446.objecter _send_command 6 2024-03-30T15:09:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c011860 con 0x7f3c4c00dd50 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.852+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c00dd50 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.853+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c011860 con 0x7f3c4c00dd50 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.853+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.853+0000 7f3c5effd700 10 client.25446.objecter _finish_command 6 = system:0 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.853+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 4 2024-03-30T15:09:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.853+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 3 6 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 7 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 7 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c004a50 osd=3 3 2024-03-30T15:09:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 3 7 2024-03-30T15:09:24.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 4 2024-03-30T15:09:24.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 10 client.25446.objecter _send_command 7 2024-03-30T15:09:24.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.870+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c002d50 con 0x7f3c4c00dd50 2024-03-30T15:09:24.872 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c002d50 con 0x7f3c4c00dd50 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c5effd700 10 client.25446.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c004a50 osd=3 4 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 3 7 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 8 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 8 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 4 8 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 4 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 10 client.25446.objecter _send_command 8 2024-03-30T15:09:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.871+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c013700 con 0x7f3c4c012b90 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.872+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c012b90 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.873+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c013700 con 0x7f3c4c012b90 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.873+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.873+0000 7f3c5effd700 10 client.25446.objecter _finish_command 8 = system:0 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.873+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 4 2024-03-30T15:09:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.873+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 4 8 2024-03-30T15:09:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 9 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 9 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c002fa0 osd=4 3 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 4 9 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 4 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 10 client.25446.objecter _send_command 9 2024-03-30T15:09:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.890+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c0140b0 con 0x7f3c4c012b90 2024-03-30T15:09:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c0140b0 con 0x7f3c4c012b90 2024-03-30T15:09:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c5effd700 10 client.25446.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c002fa0 osd=4 4 2024-03-30T15:09:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 4 9 2024-03-30T15:09:24.894 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 10 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 10 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 5 10 2024-03-30T15:09:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 4 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 10 client.25446.objecter _send_command 10 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.892+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c01af80 con 0x7f3c4c0173c0 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.893+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c0173c0 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.894+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c01af80 con 0x7f3c4c0173c0 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.894+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.894+0000 7f3c5effd700 10 client.25446.objecter _finish_command 10 = system:0 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.894+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 4 2024-03-30T15:09:24.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.894+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 5 10 2024-03-30T15:09:24.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 11 2024-03-30T15:09:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 11 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c014080 osd=5 3 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 5 11 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 4 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 10 client.25446.objecter _send_command 11 2024-03-30T15:09:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.910+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c01b160 con 0x7f3c4c0173c0 2024-03-30T15:09:24.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c01b160 con 0x7f3c4c0173c0 2024-03-30T15:09:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c5effd700 10 client.25446.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c014080 osd=5 4 2024-03-30T15:09:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 5 11 2024-03-30T15:09:24.914 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 12 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.912+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 12 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 6 12 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 4 2024-03-30T15:09:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 10 client.25446.objecter _send_command 12 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c01c4f0 con 0x7f3c4c01b880 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.913+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c01b880 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.914+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c01c4f0 con 0x7f3c4c01b880 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.914+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.914+0000 7f3c5effd700 10 client.25446.objecter _finish_command 12 = system:0 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.914+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 4 2024-03-30T15:09:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.914+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 6 12 2024-03-30T15:09:24.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:24.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 13 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 13 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c01b160 osd=6 3 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 6 13 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 4 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 10 client.25446.objecter _send_command 13 2024-03-30T15:09:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.930+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c01c4f0 con 0x7f3c4c01b880 2024-03-30T15:09:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c01c4f0 con 0x7f3c4c01b880 2024-03-30T15:09:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c5effd700 10 client.25446.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c01b160 osd=6 4 2024-03-30T15:09:24.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 6 13 2024-03-30T15:09:24.934 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:24.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 14 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 14 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 7 14 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 4 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 10 client.25446.objecter _send_command 14 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.932+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f3c4c0239a0 con 0x7f3c4c01fe60 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.933+0000 7f3c5effd700 10 client.25446.objecter ms_handle_connect 0x7f3c4c01fe60 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.934+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3c4c0239a0 con 0x7f3c4c01fe60 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.934+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:24.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.934+0000 7f3c5effd700 10 client.25446.objecter _finish_command 14 = system:0 2024-03-30T15:09:24.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.934+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 4 2024-03-30T15:09:24.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.934+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 7 14 2024-03-30T15:09:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 10 client.25446.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign -1 15 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 20 client.25446.objecter _get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_remove -1 15 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 20 client.25446.objecter get_session s=0x7f3c4c01c4f0 osd=7 3 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 15 client.25446.objecter _session_command_op_assign 7 15 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 4 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 10 client.25446.objecter _send_command 15 2024-03-30T15:09:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.951+0000 7f3c67b5a700 1 -- 172.21.15.28:0/605205541 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3c4c013b00 con 0x7f3c4c01fe60 2024-03-30T15:09:24.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.953+0000 7f3c5effd700 1 -- 172.21.15.28:0/605205541 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3c4c013b00 con 0x7f3c4c01fe60 2024-03-30T15:09:24.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.953+0000 7f3c5effd700 10 client.25446.objecter ms_dispatch 0x7f3c60001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:24.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.953+0000 7f3c5effd700 10 client.25446.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.953+0000 7f3c5effd700 20 client.25446.objecter put_session s=0x7f3c4c01c4f0 osd=7 4 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:24.953+0000 7f3c5effd700 15 client.25446.objecter _session_command_op_remove 7 15 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:25.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:25.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:25.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:25.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:09:25.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:25.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:25.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:25.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:25.468 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:25.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:25.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:25.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:25.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:25.745 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/605205541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:25.745 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:25 smithi134 ceph-mon[33714]: osdmap e704: 8 total, 7 up, 8 in 2024-03-30T15:09:25.745 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.745 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:25.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:25.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/605205541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:25.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[41614]: osdmap e704: 8 total, 7 up, 8 in 2024-03-30T15:09:25.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/605205541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:25.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[29876]: osdmap e704: 8 total, 7 up, 8 in 2024-03-30T15:09:25.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:25.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:25.934 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:25.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:26.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.060 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:26.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:26.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:26.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:26.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:26.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:26.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:26.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:26.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.2.0.0 2024-03-30T15:09:26.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:26.603 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:26 smithi134 ceph-mon[33714]: pgmap v1045: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:26.603 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:26.603 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:26 smithi134 ceph-mon[33714]: osdmap e705: 8 total, 7 up, 8 in 2024-03-30T15:09:26.603 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.603 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:26.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:26.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[29876]: pgmap v1045: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[29876]: osdmap e705: 8 total, 7 up, 8 in 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[41614]: pgmap v1045: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[41614]: osdmap e705: 8 total, 7 up, 8 in 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]: dispatch 2024-03-30T15:09:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:26.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:26.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:26.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:26.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:26.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:27.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:27.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:27.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:27.114 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:09:27.114 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:09:27.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:27.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:27.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:27.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:27.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:27.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.569+0000 7fa616272700 1 -- 172.21.15.28:0/362786404 >> v1:172.21.15.134:6789/0 conn(0x7fa610101120 legacy=0x7fa610101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.569+0000 7fa616272700 1 -- 172.21.15.28:0/362786404 shutdown_connections 2024-03-30T15:09:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.569+0000 7fa616272700 1 -- 172.21.15.28:0/362786404 >> 172.21.15.28:0/362786404 conn(0x7fa6100fc9f0 msgr2=0x7fa6100fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.569+0000 7fa616272700 1 -- 172.21.15.28:0/362786404 shutdown_connections 2024-03-30T15:09:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.569+0000 7fa616272700 1 -- 172.21.15.28:0/362786404 wait complete. 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[41614]: osdmap e706: 8 total, 7 up, 8 in 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:27.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:27.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:27.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:27 smithi028 ceph-mon[29876]: osdmap e706: 8 total, 7 up, 8 in 2024-03-30T15:09:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 librados: starting msgr at 2024-03-30T15:09:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 librados: starting objecter 2024-03-30T15:09:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 Processor -- start 2024-03-30T15:09:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 -- start start 2024-03-30T15:09:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 librados: setting wanted keys 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 librados: calling monclient init 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6101a7520 con 0x7fa610101120 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6101a8700 con 0x7fa61010d950 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.570+0000 7fa616272700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa6101a98e0 con 0x7fa610109db0 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa610101120 0x7fa6101048d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37736/0 (socket says 172.21.15.28:37736) 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60ffff700 1 -- 172.21.15.28:0/2466955265 learned_addr learned my addr 172.21.15.28:0/2466955265 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter ms_handle_connect 0x7fa610101120 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter ms_handle_connect 0x7fa610109db0 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter ms_handle_connect 0x7fa61010d950 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103664502 0 0) 0x7fa6101a7520 con 0x7fa610101120 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f80035b0 con 0x7fa610101120 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370198531 0 0) 0x7fa6101a8700 con 0x7fa61010d950 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6101a7520 con 0x7fa61010d950 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2810671160 0 0) 0x7fa6101a98e0 con 0x7fa610109db0 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6101a8700 con 0x7fa610109db0 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2804220514 0 0) 0x7fa5f80035b0 con 0x7fa610101120 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6101a98e0 con 0x7fa610101120 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1646772556 0 0) 0x7fa6101a7520 con 0x7fa61010d950 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f80035b0 con 0x7fa61010d950 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa6000015d0 con 0x7fa610101120 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa608001bb0 con 0x7fa61010d950 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2233273525 0 0) 0x7fa6101a8700 con 0x7fa610109db0 2024-03-30T15:09:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6101a7520 con 0x7fa610109db0 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa6040018b0 con 0x7fa610109db0 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 648970261 0 0) 0x7fa5f80035b0 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 >> v1:172.21.15.28:6790/0 conn(0x7fa610109db0 legacy=0x7fa610104fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 >> v1:172.21.15.28:6789/0 conn(0x7fa610101120 legacy=0x7fa6101048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.571+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6101aaac0 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6101a9b10 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa616272700 10 client.50653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6101a9fd0 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa616272700 1 librados: init done 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa608003ec0 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.572+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa608004d20 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.573+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5fc005120 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.573+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa60801b3a0 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(706..706 src has 442..706) ==== 9214+0+0 (unknown 2987341860 0 0) 0x7fa608088a50 con 0x7fa61010d950 2024-03-30T15:09:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 10 client.50653.objecter ms_dispatch 0x7fa610001550 osd_map(706..706 src has 442..706) 2024-03-30T15:09:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 3 client.50653.objecter handle_osd_map got epochs [706,706] > 0 2024-03-30T15:09:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 3 client.50653.objecter handle_osd_map decoding full epoch 706 2024-03-30T15:09:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 20 client.50653.objecter dump_active .. 0 homeless 2024-03-30T15:09:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.574+0000 7fa60d7fa700 10 client.50653.objecter ms_handle_connect 0x7fa5f8074900 2024-03-30T15:09:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.578+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa608055820 con 0x7fa61010d950 2024-03-30T15:09:27.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:27.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:27 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:27 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:27 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-124"}]': finished 2024-03-30T15:09:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:27 smithi134 ceph-mon[33714]: osdmap e706: 8 total, 7 up, 8 in 2024-03-30T15:09:27.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:27.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:27.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:09:27.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:27.803+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fa5fc005410 con 0x7fa61010d950 2024-03-30T15:09:27.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:27.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:27.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:27.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:27.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:27.999 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:28.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:28.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:28.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:28.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:28.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:28.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:28.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.384+0000 7fa60d7fa700 1 -- 172.21.15.28:0/2466955265 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v707) ==== 91+0+0 (unknown 1500397465 0 0) 0x7fa608059270 con 0x7fa61010d950 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 10 librados: watch_flush enter 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 10 librados: watch_flush exit 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 20 client.50653.objecter shutdown clearing up homeless session... 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 10 client.50653.objecter successfully canceled tick 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa5f8074900 legacy=0x7fa5f8076dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 >> v1:172.21.15.134:6789/0 conn(0x7fa61010d950 legacy=0x7fa6101a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 shutdown_connections 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 >> 172.21.15.28:0/2466955265 conn(0x7fa6100fc9f0 msgr2=0x7fa610110b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 shutdown_connections 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.388+0000 7fa616272700 1 -- 172.21.15.28:0/2466955265 wait complete. 2024-03-30T15:09:28.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:28.389+0000 7fa616272700 1 librados: shutdown 2024-03-30T15:09:28.391 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:09:28.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:28.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:28.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:28.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:28.649 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:28.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:28.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:28.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2466955265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: from='client.50653 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: pgmap v1048: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:09:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: from='client.50653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:09:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:28 smithi134 ceph-mon[33714]: osdmap e707: 8 total, 7 up, 8 in 2024-03-30T15:09:28.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2466955265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: from='client.50653 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: pgmap v1048: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: from='client.50653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:09:28.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[29876]: osdmap e707: 8 total, 7 up, 8 in 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2466955265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: from='client.50653 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: pgmap v1048: 246 pgs: 18 active+undersized+degraded, 66 active+undersized, 162 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: from='client.50653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:09:28.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:28 smithi028 ceph-mon[41614]: osdmap e707: 8 total, 7 up, 8 in 2024-03-30T15:09:28.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:28.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:29.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:09:29.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:29.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:29.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:29.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:29.323 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:29.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:29.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:29.528 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:09:29.529 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:09:29.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:29.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:29.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:29.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:29.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:29.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:29.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:29.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:29.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:30.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.060 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:30.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:30.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:09:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:30.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:30.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:30.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: osdmap e708: 8 total, 7 up, 8 in 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-mon[33714]: pgmap v1051: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:30.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:30.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: osdmap e708: 8 total, 7 up, 8 in 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[41614]: pgmap v1051: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: osdmap e708: 8 total, 7 up, 8 in 2024-03-30T15:09:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:30.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:30 smithi028 ceph-mon[29876]: pgmap v1051: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:30.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:30.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:30.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:30.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:30.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:30.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:30.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:30.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:31.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:31.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:31.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:09:31.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.139+0000 7fd8a169d700 1 -- 172.21.15.28:0/1160310279 >> v1:172.21.15.28:6790/0 conn(0x7fd89c10ea80 legacy=0x7fd89c110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.139+0000 7fd8a169d700 1 -- 172.21.15.28:0/1160310279 shutdown_connections 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.139+0000 7fd8a169d700 1 -- 172.21.15.28:0/1160310279 >> 172.21.15.28:0/1160310279 conn(0x7fd89c0fdb20 msgr2=0x7fd89c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.139+0000 7fd8a169d700 1 -- 172.21.15.28:0/1160310279 shutdown_connections 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.139+0000 7fd8a169d700 1 -- 172.21.15.28:0/1160310279 wait complete. 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 librados: starting msgr at 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 librados: starting objecter 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 Processor -- start 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 -- start start 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 librados: setting wanted keys 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 librados: calling monclient init 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c1a70b0 con 0x7fd89c10ea80 2024-03-30T15:09:31.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c1a82d0 con 0x7fd89c10aee0 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd8a169d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c1a9510 con 0x7fd89c102250 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd89affd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd89c102250 0x7fd89c107550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53190/0 (socket says 172.21.15.28:53190) 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd89b7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd89c10ea80 0x7fd89c1a5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34664/0 (socket says 172.21.15.28:34664) 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.140+0000 7fd89b7fe700 1 -- 172.21.15.28:0/1404348389 learned_addr learned my addr 172.21.15.28:0/1404348389 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter ms_handle_connect 0x7fd89c10ea80 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter ms_handle_connect 0x7fd89c102250 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter ms_handle_connect 0x7fd89c10aee0 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127134710 0 0) 0x7fd89c1a9510 con 0x7fd89c102250 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8840031f0 con 0x7fd89c102250 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001952444 0 0) 0x7fd89c1a82d0 con 0x7fd89c10aee0 2024-03-30T15:09:31.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c1a9510 con 0x7fd89c10aee0 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3376644586 0 0) 0x7fd89c1a70b0 con 0x7fd89c10ea80 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c1a82d0 con 0x7fd89c10ea80 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2354935187 0 0) 0x7fd89c1a9510 con 0x7fd89c10aee0 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c1a70b0 con 0x7fd89c10aee0 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd890001720 con 0x7fd89c10aee0 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 740060836 0 0) 0x7fd8840031f0 con 0x7fd89c102250 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c1a9510 con 0x7fd89c102250 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd88c0017b0 con 0x7fd89c102250 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3757344008 0 0) 0x7fd89c1a82d0 con 0x7fd89c10ea80 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8840031f0 con 0x7fd89c10ea80 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd894004a60 con 0x7fd89c10ea80 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2290298540 0 0) 0x7fd89c1a70b0 con 0x7fd89c10aee0 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 >> v1:172.21.15.28:6790/0 conn(0x7fd89c102250 legacy=0x7fd89c107550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 >> v1:172.21.15.28:6789/0 conn(0x7fd89c10ea80 legacy=0x7fd89c1a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c1aa750 con 0x7fd89c10aee0 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c1a9740 con 0x7fd89c10aee0 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd8a169d700 10 client.50659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd89c1a9c00 con 0x7fd89c10aee0 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.141+0000 7fd8a169d700 1 librados: init done 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.142+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd890003b20 con 0x7fd89c10aee0 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.142+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd890004980 con 0x7fd89c10aee0 2024-03-30T15:09:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.142+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd888005100 con 0x7fd89c10aee0 2024-03-30T15:09:31.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.143+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd89001b020 con 0x7fd89c10aee0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(709..709 src has 442..709) ==== 9227+0+0 (unknown 568927836 0 0) 0x7fd890088730 con 0x7fd89c10aee0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 osd_map(709..709 src has 442..709) 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 3 client.50659.objecter handle_osd_map got epochs [709,709] > 0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 3 client.50659.objecter handle_osd_map decoding full epoch 709 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 20 client.50659.objecter dump_active .. 0 homeless 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v709) ==== 54+0+16 (unknown 459414241 0 2420000192) 0x7fd89008ab60 con 0x7fd89c10aee0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd8840749a0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 10 client.50659.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 1 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 10 client.50659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=710}) -- 0x7fd8880053e0 con 0x7fd89c10aee0 2024-03-30T15:09:31.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd888000e30 con 0x7fd89c10aee0 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=709) ==== 24+0+0 (unknown 2915325576 0 0) 0x7fd890054bc0 con 0x7fd89c10aee0 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8997fa700 10 client.50659.objecter _check_command_map_dne tid 1 current 709 map_dne_bound 709 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8997fa700 10 client.50659.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8997fa700 15 client.50659.objecter _session_command_op_remove -1 1 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 10 client.50659.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 2 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 2 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 1 2 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 4 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 10 client.50659.objecter _send_command 2 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.144+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fd8880022f0 con 0x7fd888002790 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd888002790 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8880022f0 con 0x7fd888002790 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 10 client.50659.objecter _finish_command 2 = system:0 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 4 2024-03-30T15:09:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.145+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 1 2 2024-03-30T15:09:31.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 10 client.50659.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888001bb0 osd=1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 1 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 4 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 10 client.50659.objecter _send_command 3 2024-03-30T15:09:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.182+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd888004c70 con 0x7fd888002790 2024-03-30T15:09:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.184+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd888004c70 con 0x7fd888002790 2024-03-30T15:09:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.184+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.184+0000 7fd883fff700 10 client.50659.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.184+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888001bb0 osd=1 4 2024-03-30T15:09:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.184+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 1 3 2024-03-30T15:09:31.186 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 10 client.50659.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 4 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 4 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 2 4 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 4 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 10 client.50659.objecter _send_command 4 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fd88800cb30 con 0x7fd888009020 2024-03-30T15:09:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.185+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd888009020 2024-03-30T15:09:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.186+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd88800cb30 con 0x7fd888009020 2024-03-30T15:09:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.186+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.186+0000 7fd883fff700 10 client.50659.objecter _finish_command 4 = system:0 2024-03-30T15:09:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.186+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 4 2024-03-30T15:09:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.186+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 2 4 2024-03-30T15:09:31.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 10 client.50659.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 5 2024-03-30T15:09:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 5 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004f00 osd=2 3 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 2 5 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 4 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 10 client.50659.objecter _send_command 5 2024-03-30T15:09:31.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.204+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd888003e30 con 0x7fd888009020 2024-03-30T15:09:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd888003e30 con 0x7fd888009020 2024-03-30T15:09:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 10 client.50659.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004f00 osd=2 4 2024-03-30T15:09:31.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 2 5 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 10 client.50659.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 6 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 6 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 3 6 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 4 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 10 client.50659.objecter _send_command 6 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fd888011970 con 0x7fd88800de60 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.206+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd88800de60 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.207+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd888011970 con 0x7fd88800de60 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.207+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.207+0000 7fd883fff700 10 client.50659.objecter _finish_command 6 = system:0 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.207+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 4 2024-03-30T15:09:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.207+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 3 6 2024-03-30T15:09:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 10 client.50659.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 7 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 7 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004890 osd=3 3 2024-03-30T15:09:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 3 7 2024-03-30T15:09:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 4 2024-03-30T15:09:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 10 client.50659.objecter _send_command 7 2024-03-30T15:09:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.224+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd888004430 con 0x7fd88800de60 2024-03-30T15:09:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd888004430 con 0x7fd88800de60 2024-03-30T15:09:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd883fff700 10 client.50659.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004890 osd=3 4 2024-03-30T15:09:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 3 7 2024-03-30T15:09:31.229 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 10 client.50659.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 8 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 8 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 4 8 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 4 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 10 client.50659.objecter _send_command 8 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.227+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fd88800dcb0 con 0x7fd888011b20 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd888011b20 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd88800dcb0 con 0x7fd888011b20 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:31.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 10 client.50659.objecter _finish_command 8 = system:0 2024-03-30T15:09:31.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 4 2024-03-30T15:09:31.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.228+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 4 8 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 10 client.50659.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 9 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 9 2024-03-30T15:09:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888004640 osd=4 3 2024-03-30T15:09:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 4 9 2024-03-30T15:09:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 4 2024-03-30T15:09:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 10 client.50659.objecter _send_command 9 2024-03-30T15:09:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.245+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd888012f50 con 0x7fd888011b20 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd888012f50 con 0x7fd888011b20 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd883fff700 10 client.50659.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888004640 osd=4 4 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 4 9 2024-03-30T15:09:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 10 client.50659.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 10 2024-03-30T15:09:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 10 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 5 10 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 4 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 10 client.50659.objecter _send_command 10 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.247+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fd88801b020 con 0x7fd8880174e0 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.248+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd8880174e0 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.249+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd88801b020 con 0x7fd8880174e0 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.249+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.249+0000 7fd883fff700 10 client.50659.objecter _finish_command 10 = system:0 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.249+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 4 2024-03-30T15:09:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.249+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 5 10 2024-03-30T15:09:31.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 10 client.50659.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 11 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 11 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd888014090 osd=5 3 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 5 11 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 4 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 10 client.50659.objecter _send_command 11 2024-03-30T15:09:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.267+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd88801c380 con 0x7fd8880174e0 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd88801c380 con 0x7fd8880174e0 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 10 client.50659.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd888014090 osd=5 4 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 5 11 2024-03-30T15:09:31.270 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 10 client.50659.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 12 2024-03-30T15:09:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 12 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 6 12 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 4 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 10 client.50659.objecter _send_command 12 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fd88801d770 con 0x7fd88801c590 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.269+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd88801c590 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.270+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd88801d770 con 0x7fd88801c590 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.270+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.270+0000 7fd883fff700 10 client.50659.objecter _finish_command 12 = system:0 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.270+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 4 2024-03-30T15:09:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.270+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 6 12 2024-03-30T15:09:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 10 client.50659.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 13 2024-03-30T15:09:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 13 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd88801c380 osd=6 3 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 6 13 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 4 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 10 client.50659.objecter _send_command 13 2024-03-30T15:09:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.287+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd88801d770 con 0x7fd88801c590 2024-03-30T15:09:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd88801d770 con 0x7fd88801c590 2024-03-30T15:09:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd883fff700 10 client.50659.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd88801c380 osd=6 4 2024-03-30T15:09:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 6 13 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 10 client.50659.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 14 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 14 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 7 14 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 4 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 10 client.50659.objecter _send_command 14 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.289+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fd88801d990 con 0x7fd8880200c0 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 10 client.50659.objecter ms_handle_connect 0x7fd8880200c0 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd894010ae0 con 0x7fd8880200c0 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 10 client.50659.objecter _finish_command 14 = system:0 2024-03-30T15:09:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 4 2024-03-30T15:09:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.290+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 7 14 2024-03-30T15:09:31.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 10 client.50659.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign -1 15 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 20 client.50659.objecter _get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_remove -1 15 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 20 client.50659.objecter get_session s=0x7fd88801d770 osd=7 3 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 15 client.50659.objecter _session_command_op_assign 7 15 2024-03-30T15:09:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 4 2024-03-30T15:09:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 10 client.50659.objecter _send_command 15 2024-03-30T15:09:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.308+0000 7fd8a169d700 1 -- 172.21.15.28:0/1404348389 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd88801cbf0 con 0x7fd8880200c0 2024-03-30T15:09:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.309+0000 7fd883fff700 1 -- 172.21.15.28:0/1404348389 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd88801cbf0 con 0x7fd8880200c0 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.309+0000 7fd883fff700 10 client.50659.objecter ms_dispatch 0x7fd89c001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.309+0000 7fd883fff700 10 client.50659.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.309+0000 7fd883fff700 20 client.50659.objecter put_session s=0x7fd88801d770 osd=7 4 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:31.309+0000 7fd883fff700 15 client.50659.objecter _session_command_op_remove 7 15 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:31.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:31.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:31.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:31.541 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:09:31.542 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:09:31.542 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T15:09:31.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:09:31.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 42.18 scrub starts 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 42.18 scrub ok 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: osdmap e709: 8 total, 7 up, 8 in 2024-03-30T15:09:31.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1404348389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 42.18 scrub starts 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 42.18 scrub ok 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:31.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: osdmap e709: 8 total, 7 up, 8 in 2024-03-30T15:09:31.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:31.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1404348389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:31.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:09:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:09:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 42.18 scrub starts 2024-03-30T15:09:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 42.18 scrub ok 2024-03-30T15:09:31.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: osdmap e709: 8 total, 7 up, 8 in 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:31.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1404348389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:31.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:31.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:31.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:31.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:31.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:31.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:31.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:32.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:32.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:32.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:32.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:32.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:32.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:32.349 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:09:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:09:32.349 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:32.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:32.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:32.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:32.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:09:32.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: osdmap e710: 8 total, 7 up, 8 in 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: pgmap v1054: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/633 objects degraded (9.795%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:32.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: osdmap e711: 8 total, 7 up, 8 in 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: osdmap e710: 8 total, 7 up, 8 in 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.516 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: pgmap v1054: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/633 objects degraded (9.795%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: osdmap e711: 8 total, 7 up, 8 in 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.517 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:32.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: osdmap e710: 8 total, 7 up, 8 in 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:09:32.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: pgmap v1054: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/633 objects degraded (9.795%) 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/633 objects degraded (9.795%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: osdmap e711: 8 total, 7 up, 8 in 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.656 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]: dispatch 2024-03-30T15:09:32.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:32.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:32.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:32.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:32.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:09:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:32.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:32.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:33.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:33.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:33.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:33.217 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:33.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:33.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:33.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:33.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:33.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T15:09:33.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: 42.15 scrub starts 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: 42.15 scrub ok 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]': finished 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[41614]: osdmap e712: 8 total, 7 up, 8 in 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: 42.15 scrub starts 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: 42.15 scrub ok 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]': finished 2024-03-30T15:09:33.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:33 smithi028 ceph-mon[29876]: osdmap e712: 8 total, 7 up, 8 in 2024-03-30T15:09:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:33.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:33.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: 42.15 scrub starts 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: 42.15 scrub ok 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-126", "mode": "writeback"}]': finished 2024-03-30T15:09:33.619 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:33 smithi134 ceph-mon[33714]: osdmap e712: 8 total, 7 up, 8 in 2024-03-30T15:09:33.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:33.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:33.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:33.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:33.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:33.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:34.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:09:34.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:34.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:34.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:34.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:34.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:34.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:34.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[29876]: pgmap v1057: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:09:34.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:09:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T15:09:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T15:09:34.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:34 smithi028 ceph-mon[41614]: pgmap v1057: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:34.488 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:34.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.604+0000 7f2ffa888700 1 -- 172.21.15.28:0/2304539650 >> v1:172.21.15.28:6789/0 conn(0x7f2ff4101140 legacy=0x7f2ff4101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.604+0000 7f2ffa888700 1 -- 172.21.15.28:0/2304539650 shutdown_connections 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.604+0000 7f2ffa888700 1 -- 172.21.15.28:0/2304539650 >> 172.21.15.28:0/2304539650 conn(0x7f2ff40fc9d0 msgr2=0x7f2ff40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.604+0000 7f2ffa888700 1 -- 172.21.15.28:0/2304539650 shutdown_connections 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.604+0000 7f2ffa888700 1 -- 172.21.15.28:0/2304539650 wait complete. 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 librados: starting msgr at 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 librados: starting objecter 2024-03-30T15:09:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 Processor -- start 2024-03-30T15:09:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 -- start start 2024-03-30T15:09:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 librados: setting wanted keys 2024-03-30T15:09:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.605+0000 7f2ffa888700 1 librados: calling monclient init 2024-03-30T15:09:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ffa888700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff41a7080 con 0x7f2ff4109dd0 2024-03-30T15:09:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ffa888700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff41a82a0 con 0x7f2ff4101140 2024-03-30T15:09:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ffa888700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2ff41a94e0 con 0x7f2ff410d970 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff37fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2ff4109dd0 0x7f2ff41a1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34674/0 (socket says 172.21.15.28:34674) 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff3fff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2ff4101140 0x7f2ff406aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:55072/0 (socket says 172.21.15.28:55072) 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff37fe700 1 -- 172.21.15.28:0/2659161285 learned_addr learned my addr 172.21.15.28:0/2659161285 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter ms_handle_connect 0x7f2ff4101140 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter ms_handle_connect 0x7f2ff410d970 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter ms_handle_connect 0x7f2ff4109dd0 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454897501 0 0) 0x7f2ff41a82a0 con 0x7f2ff4101140 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fdc0031f0 con 0x7f2ff4101140 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316061323 0 0) 0x7f2ff41a94e0 con 0x7f2ff410d970 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ff41a82a0 con 0x7f2ff410d970 2024-03-30T15:09:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862796164 0 0) 0x7f2ff41a7080 con 0x7f2ff4109dd0 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ff41a94e0 con 0x7f2ff4109dd0 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2823608041 0 0) 0x7f2fdc0031f0 con 0x7f2ff4101140 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ff41a7080 con 0x7f2ff4101140 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2fe40046e0 con 0x7f2ff4101140 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3765101031 0 0) 0x7f2ff41a82a0 con 0x7f2ff410d970 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fdc0031f0 con 0x7f2ff410d970 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.606+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2fec001470 con 0x7f2ff410d970 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801983180 0 0) 0x7f2ff41a94e0 con 0x7f2ff4109dd0 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ff41a82a0 con 0x7f2ff4109dd0 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2fe80018b0 con 0x7f2ff4109dd0 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3324988497 0 0) 0x7f2ff41a7080 con 0x7f2ff4101140 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 >> v1:172.21.15.28:6790/0 conn(0x7f2ff410d970 legacy=0x7f2ff41a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 >> v1:172.21.15.28:6789/0 conn(0x7f2ff4109dd0 legacy=0x7f2ff41a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff41aa720 con 0x7f2ff4101140 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff41a72b0 con 0x7f2ff4101140 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ffa888700 10 client.50665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ff41a7770 con 0x7f2ff4101140 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ffa888700 1 librados: init done 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2fe4001b70 con 0x7f2ff4101140 2024-03-30T15:09:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.607+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2fe4004d70 con 0x7f2ff4101140 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.608+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe0005120 con 0x7f2ff4101140 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2fe4005000 con 0x7f2ff4101140 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(713..713 src has 442..713) ==== 9222+0+0 (unknown 2779953988 0 0) 0x7f2fe4089b60 con 0x7f2ff4101140 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 10 client.50665.objecter ms_dispatch 0x7f2ff4001550 osd_map(713..713 src has 442..713) 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 3 client.50665.objecter handle_osd_map got epochs [713,713] > 0 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 3 client.50665.objecter handle_osd_map decoding full epoch 713 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.609+0000 7f2ff17fa700 20 client.50665.objecter dump_active .. 0 homeless 2024-03-30T15:09:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.610+0000 7f2ff17fa700 10 client.50665.objecter ms_handle_connect 0x7f2fdc070520 2024-03-30T15:09:34.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:34.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.614+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2fe4056930 con 0x7f2ff4101140 2024-03-30T15:09:34.642 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:09:34.642 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:09:34.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:34.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T15:09:34.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T15:09:34.643 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:34 smithi134 ceph-mon[33714]: pgmap v1057: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:34.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:34.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.824+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f2fe0005410 con 0x7f2ff4101140 2024-03-30T15:09:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.827+0000 7f2ff17fa700 1 -- 172.21.15.28:0/2659161285 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v713) ==== 81+0+57750 (unknown 654254404 0 2689007345) 0x7f2fe4004d50 con 0x7f2ff4101140 2024-03-30T15:09:34.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 713, 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:09:34.402114+0000", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T15:08:50.456069+0000", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:09:34.829 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 117, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 330, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:09:34.830 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.831 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.832 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "562", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "559", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.833 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.834 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.9000000953674316, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.9000000953674316, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.5499999523162842, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.5499999523162842, 2024-03-30T15:09:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.837 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.838 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:09:34.839 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.840 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.841 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6700000762939453, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6700000762939453, 2024-03-30T15:09:34.842 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.75, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.844 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.845 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.845 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:34.846 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.846 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.846 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.846 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:09:34.847 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.847 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.847 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.847 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.847 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.849 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.850 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:34.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.852 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.853 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:09:34.854 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.855 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.856 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.857 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:09:34.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.859 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.860 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.861 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.863 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.4900000095367432, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.4900000095367432, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.309999942779541, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.309999942779541, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.866 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.867 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.869 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:09:34.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.871 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.872 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.873 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.874 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.878 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.879 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.880 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:34.881 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-111", 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:08:12.612746+0000", 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:09:34.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 3, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 1, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-03-30T15:09:34.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "713", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "713", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "713", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "713", 2024-03-30T15:09:34.884 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 4, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 710, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [ 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: 330 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:34.885 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi028-93253-111", 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.886 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 8192, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {} 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 330, 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-126", 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:09:28.644011+0000", 2024-03-30T15:09:34.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8201, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:34.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "713", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "713", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "713", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "713", 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:34.889 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 4, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 710, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": 322, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "writeback", 2024-03-30T15:09:34.890 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:34.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:09:34.892 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 0, 2024-03-30T15:09:34.893 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 660, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 669, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:09:34.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 10 librados: watch_flush enter 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 10 librados: watch_flush exit 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 20 client.50665.objecter shutdown clearing up homeless session... 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 10 client.50665.objecter successfully canceled tick 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2fdc070520 legacy=0x7f2fdc0729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 >> v1:172.21.15.134:6789/0 conn(0x7f2ff4101140 legacy=0x7f2ff406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 shutdown_connections 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 >> 172.21.15.28:0/2659161285 conn(0x7f2ff40fc9d0 msgr2=0x7f2ff4110b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 shutdown_connections 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 -- 172.21.15.28:0/2659161285 wait complete. 2024-03-30T15:09:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:34.829+0000 7f2ffa888700 1 librados: shutdown 2024-03-30T15:09:34.897 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:34.897 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.897 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.897 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "exists" 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:09:34.899 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.901 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:09:34.902 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:09:34.903 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:09:34.904 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:09:34.906 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.907 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 698, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.912 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:34.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.915 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.916 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:34.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.920 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:34.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 708, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:34.924 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:09:34.925 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:34.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:27.042268+0000", 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 668 2024-03-30T15:09:34.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:21.783823+0000", 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:03.355730+0000", 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:12.781159+0000", 2024-03-30T15:09:34.930 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:50.311343+0000", 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:09:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:07.684740+0000", 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.932 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:09.629768+0000", 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:10.707566+0000", 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.933 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.17", 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "42.c", 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:34.934 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.936 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:09:34.937 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.938 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:34.939 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "testprofile-test-rados-api-smithi028-93253-111": { 2024-03-30T15:09:34.940 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-device-class": "", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-num-failure-domains": "0", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-osds-per-failure-domain": "0", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-root": "default", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van", 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "w": "8" 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:09:34.941 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:34.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.946 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.948 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:09:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.951 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:09:34.953 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.955 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 325, 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 326, 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.957 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 329, 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:34.959 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:09:34.960 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:34.960 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:09:34.960 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:09:34.960 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:09:34.960 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:34.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:34.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:35.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:35.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:35.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:35.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:35.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:35.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:35.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: osdmap e713: 8 total, 7 up, 8 in 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2659161285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T15:09:35.483 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:35 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T15:09:35.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:35.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:09:35.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:35.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:35.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: osdmap e713: 8 total, 7 up, 8 in 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2659161285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T15:09:35.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: osdmap e713: 8 total, 7 up, 8 in 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]: dispatch 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2659161285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T15:09:35.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:35 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T15:09:35.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:35.788 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:09:35.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:35.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:35.887 INFO:tasks.thrashosds.thrasher:j is {'epoch': 713, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:09:34.402114+0000', 'last_up_change': '2024-03-30T15:08:50.456069+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 117, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 330, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '562', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '559', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9000000953674316, 'score_stable': 2.9000000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 322, 'pool_name': 'test-rados-api-smithi028-93253-111', 'create_time': '2024-03-30T15:08:12.612746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '713', 'last_force_op_resend': '713', 'last_force_op_resend_prenautilus': '713', 'last_force_op_resend_preluminous': '713', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 710, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [330], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi028-93253-111', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 330, 'pool_name': 'test-rados-api-smithi028-93253-126', 'create_time': '2024-03-30T15:09:28.644011+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '713', 'last_force_op_resend': '713', 'last_force_op_resend_prenautilus': '713', 'last_force_op_resend_preluminous': '713', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 710, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 322, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 660, 'down_at': 669, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 698, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 708, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:27.042268+0000', 'dead_epoch': 668}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:21.783823+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:03.355730+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:12.781159+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:50.311343+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:07.684740+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:09.629768+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:10.707566+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.17', 'osds': [6, 7, 1]}, {'pgid': '42.c', 'osds': [3, 7, 1]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}, {'pgid': '5.13', 'mappings': [{'from': 3, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi028-93253-111': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 322, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 325, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 326, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 329, 'snaps': [{'begin': 4, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:09:35.887 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.13 2024-03-30T15:09:35.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 5.13 2024-03-30T15:09:35.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:35.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:36.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:36.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:36.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:36.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:36.417 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9001 ms) 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6010 ms) 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4994 ms) 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.418 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26022 ms) 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14063 ms) 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11937 ms) 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 00000012 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12011 ms) 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-30T15:09:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: require_osd_release = squid 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5984 ms) 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (559267 ms total) 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1005 ms) 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11017 ms) 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17977 ms) 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-30T15:09:36.420 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30002 ms total) 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7011 ms) 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8021 ms) 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: waiting for scrub... 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: done waiting 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23978 ms) 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-30T15:09:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10005 ms) 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8042 ms) 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8042 ms) 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9933 ms) 2024-03-30T15:09:36.422 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-30T15:09:36.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:36.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:36.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:36.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: osdmap e714: 8 total, 7 up, 8 in 2024-03-30T15:09:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:09:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:09:36.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T15:09:36.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T15:09:36.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:36 smithi134 ceph-mon[33714]: pgmap v1060: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:36.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:36.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:36.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: osdmap e714: 8 total, 7 up, 8 in 2024-03-30T15:09:36.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:09:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:09:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T15:09:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T15:09:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[29876]: pgmap v1060: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:36.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-126"}]': finished 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: osdmap e714: 8 total, 7 up, 8 in 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T15:09:36.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:36 smithi028 ceph-mon[41614]: pgmap v1060: 246 pgs: 18 active+undersized+degraded, 63 active+undersized, 165 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:36.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:36.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:36.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:36.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:36.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:36.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:37.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:37.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:37.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:37.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:37.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:37.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:37.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:37.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:37.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: osdmap e715: 8 total, 7 up, 8 in 2024-03-30T15:09:37.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:09:37.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:09:37.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: 1.4 scrub starts 2024-03-30T15:09:37.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: 1.4 scrub ok 2024-03-30T15:09:37.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:37.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:37.564 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:37 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:37.565 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:09:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:09:37.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57476.0:1 245.6s0 245:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 552.32904328500001, 2024-03-30T15:09:37.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0238222990000001, 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57476", 2024-03-30T15:09:37.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4051625719", 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232007+0000", 2024-03-30T15:09:37.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232010+0000", 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.604e-06 2024-03-30T15:09:37.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232018+0000", 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3520000000000007e-06 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232031+0000", 2024-03-30T15:09:37.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2765999999999999e-05 2024-03-30T15:09:37.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232043+0000", 2024-03-30T15:09:37.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1732e-05 2024-03-30T15:09:37.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232111+0000", 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8403000000000002e-05 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:25.232115+0000", 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7220000000000001e-06 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252496+0000", 2024-03-30T15:09:37.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0203810069999999 2024-03-30T15:09:37.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.252648+0000", 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000152017 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.254117+0000", 2024-03-30T15:09:37.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014689320000000001 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255025+0000", 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090758899999999999 2024-03-30T15:09:37.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:37.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255742+0000", 2024-03-30T15:09:37.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071763700000000003 2024-03-30T15:09:37.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:00:26.255830+0000", 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7538000000000001e-05 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49756.0:1 273.6s0 273:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:09:37.581 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 476.24715316599998, 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93441624899999998, 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.582 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49756", 2024-03-30T15:09:37.583 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/1885953997", 2024-03-30T15:09:37.583 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:37.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.583 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313898+0000", 2024-03-30T15:09:37.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313900+0000", 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.785e-06 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313911+0000", 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0925e-05 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313947+0000", 2024-03-30T15:09:37.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6205000000000003e-05 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.313960+0000", 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2768999999999999e-05 2024-03-30T15:09:37.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525411+0000", 2024-03-30T15:09:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.21145103200000001 2024-03-30T15:09:37.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:41.525415+0000", 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.014e-06 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.246985+0000", 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.72157016200000001 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247116+0000", 2024-03-30T15:09:37.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013093499999999999 2024-03-30T15:09:37.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247514+0000", 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039808699999999999 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:37.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.247951+0000", 2024-03-30T15:09:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043615700000000001 2024-03-30T15:09:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248270+0000", 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031954200000000002 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.248314+0000", 2024-03-30T15:09:37.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3636e-05 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.596 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.49768.0:1 274.5s0 274:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e338)", 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 475.328831557, 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0270852340000001, 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.597 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.49768", 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3544512235", 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.598 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:09:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232219+0000", 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232221+0000", 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3120000000000001e-06 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232228+0000", 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1680000000000001e-06 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232240+0000", 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2248e-05 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.232251+0000", 2024-03-30T15:09:37.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1559000000000001e-05 2024-03-30T15:09:37.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234151+0000", 2024-03-30T15:09:37.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001899277 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:42.234167+0000", 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5872e-05 2024-03-30T15:09:37.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259198+0000", 2024-03-30T15:09:37.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.025031091 2024-03-30T15:09:37.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:01:43.259304+0000", 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010670699999999999 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.607 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDECSubOpWrite(322.1s2 646/644 ECSubWrite(tid=1490, reqid=client.58535.0:2, at_version=646'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:09:37.608 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 82.190142578000007, 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038653149999999998, 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370909+0000", 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9200000000000002e-07 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370954+0000", 2024-03-30T15:09:37.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5198999999999999e-05 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370966+0000", 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.239e-05 2024-03-30T15:09:37.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370977+0000", 2024-03-30T15:09:37.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.128e-05 2024-03-30T15:09:37.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.371502+0000", 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052424500000000001 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.371516+0000", 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4e-05 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.374743+0000", 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032274980000000001 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.374773+0000", 2024-03-30T15:09:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0311000000000002e-05 2024-03-30T15:09:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDECSubOpWrite(322.1s2 646/644 ECSubWrite(tid=1491, reqid=client.58535.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=646'1))", 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:15.378532+0000", 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 82.182518274000003, 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017169920000000001, 2024-03-30T15:09:37.619 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.378532+0000", 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.378532+0000", 2024-03-30T15:09:37.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.378534+0000", 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3850000000000001e-06 2024-03-30T15:09:37.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.378550+0000", 2024-03-30T15:09:37.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5999999999999999e-05 2024-03-30T15:09:37.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.379388+0000", 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00083796599999999999 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.379400+0000", 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2396e-05 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.379517+0000", 2024-03-30T15:09:37.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011684300000000001 2024-03-30T15:09:37.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.379527+0000", 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0336000000000001e-05 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380212+0000", 2024-03-30T15:09:37.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00068510200000000004 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380249+0000", 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6964e-05 2024-03-30T15:09:37.628 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.58535.0:3 323.9 e646/645 323:96df10bd:test-rados-api-smithi028-93253-113::foo:head v 646'1, mlcod=0'0)", 2024-03-30T15:09:37.629 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:15.380030+0000", 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 82.181020446000005, 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010923949999999999, 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:09:37.630 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380030+0000", 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380030+0000", 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380031+0000", 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4099999999999998e-07 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380046+0000", 2024-03-30T15:09:37.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5392000000000001e-05 2024-03-30T15:09:37.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380053+0000", 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8539999999999999e-06 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380061+0000", 2024-03-30T15:09:37.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.259e-06 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380110+0000", 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8922999999999999e-05 2024-03-30T15:09:37.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.380130+0000", 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9488e-05 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:09:37.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.381080+0000", 2024-03-30T15:09:37.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00095005999999999997 2024-03-30T15:09:37.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.381123+0000", 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2778000000000003e-05 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.639 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.640 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:44 323.18 323:18000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:09:37.640 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:09:37.640 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 78.184453839, 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00320857, 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.641 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 44 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376601+0000", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0690000000000003e-06 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376615+0000", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.449e-05 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376646+0000", 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0706e-05 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376668+0000", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1396e-05 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376823+0000", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015560699999999999 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376828+0000", 2024-03-30T15:09:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.186e-06 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379596+0000", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027672209999999998 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379661+0000", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5042999999999994e-05 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379805+0000", 2024-03-30T15:09:37.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000144852 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:255 325.e 325:70ec8fab:test-rados-api-smithi028-93253-117::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e665)", 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 63.193086067999999, 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084883900000000002, 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.646 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 255 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:09:37.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367968+0000", 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.269e-06 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367978+0000", 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0389e-05 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367999+0000", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0820999999999999e-05 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368013+0000", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4369000000000001e-05 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368106+0000", 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2734999999999999e-05 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368116+0000", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9930000000000007e-06 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.371856+0000", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003739357 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.372060+0000", 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020420599999999999 2024-03-30T15:09:37.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.373720+0000", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016599029999999999 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.373823+0000", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103148 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.375420+0000", 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001597464 2024-03-30T15:09:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: osdmap e715: 8 total, 7 up, 8 in 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: 1.4 scrub starts 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: 1.4 scrub ok 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: osdmap e715: 8 total, 7 up, 8 in 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: 1.4 scrub starts 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: 1.4 scrub ok 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:37 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:37.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.375485+0000", 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4987000000000006e-05 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.376453+0000", 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096774899999999995 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.656 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:412 326.8 326:10000000::::head [pgnls start_epoch 680] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e680)", 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:01.384995+0000", 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 36.176055863000002, 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029667399999999998, 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 412 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.384995+0000", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.384995+0000", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.384997+0000", 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4329999999999998e-06 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.385006+0000", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0480000000000004e-06 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.385020+0000", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.401e-05 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.385030+0000", 2024-03-30T15:09:37.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3910000000000004e-06 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.385082+0000", 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2537000000000002e-05 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.385087+0000", 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3420000000000001e-06 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.387621+0000", 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002534016 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.387656+0000", 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5199e-05 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.387962+0000", 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030576399999999998 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.661 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:413 326.18 326:18000000::::head [pgnls start_epoch 680] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e680)", 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:01.388047+0000", 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 36.173003281, 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019035370000000001, 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 413 2024-03-30T15:09:37.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388047+0000", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388047+0000", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388049+0000", 2024-03-30T15:09:37.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6160000000000001e-06 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388055+0000", 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1199999999999999e-06 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388065+0000", 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4630000000000007e-06 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.388073+0000", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6060000000000003e-06 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.389835+0000", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017615700000000001 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.389874+0000", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8859000000000002e-05 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.389951+0000", 2024-03-30T15:09:37.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7303000000000002e-05 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:650 328.b 328:d7101345:test-rados-api-smithi028-93253-123::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e694)", 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.176082766, 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0061439140000000003, 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.666 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 650 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384970+0000", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384976+0000", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3720000000000001e-06 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384992+0000", 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5999e-05 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385002+0000", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5100000000000004e-06 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385059+0000", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6533000000000002e-05 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385065+0000", 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1789999999999996e-06 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.388747+0000", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036823210000000001 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.389663+0000", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091559200000000001 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.390602+0000", 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00093893999999999998 2024-03-30T15:09:37.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.390652+0000", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0096999999999998e-05 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.391052+0000", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000400374 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.391112+0000", 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9858000000000002e-05 2024-03-30T15:09:37.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:725 328.b 328:d7101345:test-rados-api-smithi028-93253-123::bar:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e694)", 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:15.424873+0000", 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.136177273000001, 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002946864, 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.672 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 725 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424873+0000", 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424873+0000", 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424875+0000", 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0720000000000001e-06 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424879+0000", 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3370000000000001e-06 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424888+0000", 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.901e-06 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424893+0000", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5929999999999996e-06 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.424942+0000", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8752999999999999e-05 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.425019+0000", 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6969000000000001e-05 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.427684+0000", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002664671 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.427748+0000", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3800999999999994e-05 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.427820+0000", 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2767000000000003e-05 2024-03-30T15:09:37.676 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:729 328.8 328:10000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 20.109518729000001, 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076206399999999997, 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.677 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 729 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451536+0000", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.873e-06 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451553+0000", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7091e-05 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451582+0000", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9487000000000001e-05 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451598+0000", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5882000000000001e-05 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451681+0000", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2874999999999995e-05 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451687+0000", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5450000000000003e-06 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459041+0000", 2024-03-30T15:09:37.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073537719999999997 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459066+0000", 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5463999999999999e-05 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459153+0000", 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6650999999999994e-05 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3121 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94326471856512 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:22.670234+0000", 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.890816600000001, 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001075718, 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3121 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.682 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670234+0000", 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670234+0000", 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670242+0000", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6879999999999999e-06 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670251+0000", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8349999999999993e-06 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670335+0000", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3936000000000004e-05 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.670403+0000", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8867000000000003e-05 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.671106+0000", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00070249699999999995 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.671196+0000", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0242999999999996e-05 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:22.671310+0000", 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113652 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:810 329.c 329:346a2a8b:test-rados-api-smithi028-93253-125::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 13.159858726, 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0077308790000000004, 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.686 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 810 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401195+0000", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3079999999999999e-06 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401207+0000", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1544e-05 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401226+0000", 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9400000000000001e-05 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401239+0000", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3115000000000001e-05 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401304+0000", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4243000000000001e-05 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401494+0000", 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019060099999999999 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407467+0000", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059725730000000001 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407561+0000", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4106999999999999e-05 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407894+0000", 2024-03-30T15:09:37.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033300799999999998 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407956+0000", 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2300999999999999e-05 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.408923+0000", 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096667900000000002 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.691 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:811 329.c 329:346a2a8b:test-rados-api-smithi028-93253-125::bar:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.409150+0000", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 13.151900685999999, 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030058680000000001, 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 811 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409150+0000", 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409150+0000", 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409151+0000", 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2440000000000001e-06 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409156+0000", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2610000000000004e-06 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409164+0000", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7190000000000007e-06 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409171+0000", 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2389999999999996e-06 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409215+0000", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3927000000000001e-05 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.409368+0000", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000152411 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.410576+0000", 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001208139 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.410687+0000", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110611 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.411034+0000", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034692300000000001 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.411130+0000", 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6834000000000002e-05 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.412156+0000", 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00102556 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24466.0:881 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e710)", 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:32.546430+0000", 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.0146202940000002, 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001698806, 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.697 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24466", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.134:0/2615461998", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 881 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546430+0000", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546430+0000", 2024-03-30T15:09:37.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546433+0000", 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1019999999999999e-06 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546441+0000", 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4959999999999996e-06 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546453+0000", 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.163e-05 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.546463+0000", 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0132999999999999e-05 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.547357+0000", 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00089380700000000004 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.547991+0000", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00063408099999999999 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:32.548129+0000", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013855700000000001 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:921 330.18 330:18000000:test-rados-api-smithi028-93253-127:::head [pgnls start_epoch 712] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e712)", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:33.385654+0000", 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.1753969949999998, 2024-03-30T15:09:37.701 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030457790000000002, 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 921 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385654+0000", 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385654+0000", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385657+0000", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8270000000000002e-06 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385667+0000", 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0169e-05 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385688+0000", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1044999999999999e-05 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385698+0000", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0545e-05 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385765+0000", 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7111000000000002e-05 2024-03-30T15:09:37.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.385771+0000", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7819999999999999e-06 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.388560+0000", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027889550000000001 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.388598+0000", 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7592000000000002e-05 2024-03-30T15:09:37.705 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:33.388699+0000", 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101753 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1040 330.18 330:18000000::::head [pgnls start_epoch 714] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e714)", 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 2.1413960759999999, 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032484010000000002, 2024-03-30T15:09:37.706 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1040 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419658+0000", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1540000000000002e-06 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419668+0000", 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0057e-05 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.708 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419690+0000", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.181e-05 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419702+0000", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2830000000000001e-05 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419782+0000", 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9348e-05 2024-03-30T15:09:37.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419788+0000", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3629999999999999e-06 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422758+0000", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029700490000000002 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422798+0000", 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9792999999999997e-05 2024-03-30T15:09:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422903+0000", 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104997 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(1.4,QREQUEST,e:715,from:5,reservation_nonce:5)", 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.87444922899999999, 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034844170000000001, 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:09:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:09:37.712 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686603+0000", 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0099999999999998e-06 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686609+0000", 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6250000000000004e-06 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686622+0000", 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.242e-05 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.713 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686630+0000", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8689999999999993e-06 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686682+0000", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1782e-05 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686687+0000", 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2870000000000004e-06 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690023+0000", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033354050000000001 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690039+0000", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6402999999999999e-05 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690086+0000", 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6616000000000002e-05 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.715 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.716 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.716 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:09:37.716 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:09:37.716 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.596+0000 7fed52838700 1 -- 172.21.15.28:0/1245680189 >> v1:172.21.15.28:6789/0 conn(0x7fed4c1152e0 legacy=0x7fed4c1176d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.596+0000 7fed52838700 1 -- 172.21.15.28:0/1245680189 shutdown_connections 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.596+0000 7fed52838700 1 -- 172.21.15.28:0/1245680189 >> 172.21.15.28:0/1245680189 conn(0x7fed4c0fca30 msgr2=0x7fed4c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.596+0000 7fed52838700 1 -- 172.21.15.28:0/1245680189 shutdown_connections 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.596+0000 7fed52838700 1 -- 172.21.15.28:0/1245680189 wait complete. 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 librados: starting msgr at 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 librados: starting objecter 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 Processor -- start 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 -- start start 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 librados: setting wanted keys 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 librados: calling monclient init 2024-03-30T15:09:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed4c1ad590 con 0x7fed4c100fd0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed4c1ae7b0 con 0x7fed4c106dd0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed52838700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fed4c1af9d0 con 0x7fed4c1152e0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed4bfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fed4c100fd0 0x7fed4c114960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34698/0 (socket says 172.21.15.28:34698) 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed4bfff700 1 -- 172.21.15.28:0/283809761 learned_addr learned my addr 172.21.15.28:0/283809761 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed50dd5700 1 --1- 172.21.15.28:0/283809761 >> v1:172.21.15.28:6790/0 conn(0x7fed4c1152e0 0x7fed4c1abc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53238/0 (socket says 172.21.15.28:53238) 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter ms_handle_connect 0x7fed4c1152e0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter ms_handle_connect 0x7fed4c100fd0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter ms_handle_connect 0x7fed4c106dd0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886534774 0 0) 0x7fed4c1af9d0 con 0x7fed4c1152e0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed380031f0 con 0x7fed4c1152e0 2024-03-30T15:09:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062520194 0 0) 0x7fed4c1ad590 con 0x7fed4c100fd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.597+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed4c1af9d0 con 0x7fed4c100fd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767343278 0 0) 0x7fed4c1ae7b0 con 0x7fed4c106dd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed4c1ad590 con 0x7fed4c106dd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3596179247 0 0) 0x7fed380031f0 con 0x7fed4c1152e0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed4c1ae7b0 con 0x7fed4c1152e0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fed44004910 con 0x7fed4c1152e0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4233847821 0 0) 0x7fed4c1af9d0 con 0x7fed4c100fd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed380031f0 con 0x7fed4c100fd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fed3c001230 con 0x7fed4c100fd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3364082330 0 0) 0x7fed4c1ad590 con 0x7fed4c106dd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed4c1af9d0 con 0x7fed4c106dd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fed34001450 con 0x7fed4c106dd0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4001265595 0 0) 0x7fed4c1ae7b0 con 0x7fed4c1152e0 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 >> v1:172.21.15.134:6789/0 conn(0x7fed4c106dd0 legacy=0x7fed4c1a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 >> v1:172.21.15.28:6789/0 conn(0x7fed4c100fd0 legacy=0x7fed4c114960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c1b0c10 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c1ad7c0 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed52838700 10 client.58670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fed4c1adc80 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed52838700 1 librados: init done 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fed44001450 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.598+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fed44005040 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.599+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c005120 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.600+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fed44001f00 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(716..716 src has 442..716) ==== 9214+0+0 (unknown 2445622305 0 0) 0x7fed44089df0 con 0x7fed4c1152e0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 10 client.58670.objecter ms_dispatch 0x7fed4c001550 osd_map(716..716 src has 442..716) 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 3 client.58670.objecter handle_osd_map got epochs [716,716] > 0 2024-03-30T15:09:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 3 client.58670.objecter handle_osd_map decoding full epoch 716 2024-03-30T15:09:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 20 client.58670.objecter dump_active .. 0 homeless 2024-03-30T15:09:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.601+0000 7fed497fa700 10 client.58670.objecter ms_handle_connect 0x7fed38070580 2024-03-30T15:09:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.607+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fed44056be0 con 0x7fed4c1152e0 2024-03-30T15:09:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:37.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:37.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:37.833+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fed2c005410 con 0x7fed4c1152e0 2024-03-30T15:09:37.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:37.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:37.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:37.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:38.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:38.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:38.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:38.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:09:38.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:38.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:38.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.434+0000 7fed497fa700 1 -- 172.21.15.28:0/283809761 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v717) ==== 101+0+0 (unknown 2433056693 0 0) 0x7fed4405a630 con 0x7fed4c1152e0 2024-03-30T15:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 10 librados: watch_flush enter 2024-03-30T15:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 10 librados: watch_flush exit 2024-03-30T15:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 20 client.58670.objecter shutdown clearing up homeless session... 2024-03-30T15:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 10 client.58670.objecter successfully canceled tick 2024-03-30T15:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 >> v1:172.21.15.28:6800/3011431093 conn(0x7fed38070580 legacy=0x7fed38072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 >> v1:172.21.15.28:6790/0 conn(0x7fed4c1152e0 legacy=0x7fed4c1abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 shutdown_connections 2024-03-30T15:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.438+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 >> 172.21.15.28:0/283809761 conn(0x7fed4c0fca30 msgr2=0x7fed4c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.440+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 shutdown_connections 2024-03-30T15:09:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.443+0000 7fed52838700 1 -- 172.21.15.28:0/283809761 wait complete. 2024-03-30T15:09:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.444+0000 7fed52838700 1 librados: shutdown 2024-03-30T15:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:09:38.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:38.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: osdmap e716: 8 total, 7 up, 8 in 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/283809761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: from='client.58670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:09:38.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[29876]: pgmap v1063: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: osdmap e716: 8 total, 7 up, 8 in 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/283809761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: from='client.58670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:09:38.568 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:38 smithi028 ceph-mon[41614]: pgmap v1063: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:38.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:38.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 -- 172.21.15.28:0/3497230228 >> v1:172.21.15.28:6790/0 conn(0x7f5200109dd0 legacy=0x7f520010c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 -- 172.21.15.28:0/3497230228 shutdown_connections 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 -- 172.21.15.28:0/3497230228 >> 172.21.15.28:0/3497230228 conn(0x7f52000fc9d0 msgr2=0x7f52000fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 -- 172.21.15.28:0/3497230228 shutdown_connections 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 -- 172.21.15.28:0/3497230228 wait complete. 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 librados: starting msgr at 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 librados: starting objecter 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.670+0000 7f5206205700 1 Processor -- start 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 -- start start 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 librados: setting wanted keys 2024-03-30T15:09:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 librados: calling monclient init 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5200106900 con 0x7f5200101140 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52001a8550 con 0x7f5200109dd0 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f5206205700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52001a9730 con 0x7f520010d970 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51ff7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5200101140 0x7f52001046d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60718/0 (socket says 172.21.15.28:60718) 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51ff7fe700 1 -- 172.21.15.28:0/1469990316 learned_addr learned my addr 172.21.15.28:0/1469990316 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter ms_handle_connect 0x7f5200101140 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter ms_handle_connect 0x7f520010d970 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431870549 0 0) 0x7f52001a9730 con 0x7f520010d970 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51e80031f0 con 0x7f520010d970 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3472102864 0 0) 0x7f5200106900 con 0x7f5200101140 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52001a9730 con 0x7f5200101140 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter ms_handle_connect 0x7f5200109dd0 2024-03-30T15:09:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.671+0000 7f51fcff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600183390 0 0) 0x7f52001a8550 con 0x7f5200109dd0 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5200106900 con 0x7f5200109dd0 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3316010154 0 0) 0x7f51e80031f0 con 0x7f520010d970 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52001a8550 con 0x7f520010d970 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f51f8001b70 con 0x7f520010d970 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1002664541 0 0) 0x7f52001a9730 con 0x7f5200101140 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51e80031f0 con 0x7f5200101140 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f51f0001890 con 0x7f5200101140 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 151446719 0 0) 0x7f5200106900 con 0x7f5200109dd0 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52001a9730 con 0x7f5200109dd0 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f51f4001370 con 0x7f5200109dd0 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3776381725 0 0) 0x7f52001a8550 con 0x7f520010d970 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 >> v1:172.21.15.134:6789/0 conn(0x7f5200109dd0 legacy=0x7f5200104de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 >> v1:172.21.15.28:6789/0 conn(0x7f5200101140 legacy=0x7f52001046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52001aa910 con 0x7f520010d970 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52001a75a0 con 0x7f520010d970 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f5206205700 10 client.58676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f52001a7a60 con 0x7f520010d970 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f5206205700 1 librados: init done 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f51f8003ec0 con 0x7f520010d970 2024-03-30T15:09:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.672+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f51f8004db0 con 0x7f520010d970 2024-03-30T15:09:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.673+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f51ec005100 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f51f8003a20 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(717..717 src has 442..717) ==== 9227+0+0 (unknown 3874373339 0 0) 0x7f51f8088b30 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 osd_map(717..717 src has 442..717) 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 3 client.58676.objecter handle_osd_map got epochs [717,717] > 0 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 3 client.58676.objecter handle_osd_map decoding full epoch 717 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 20 client.58676.objecter dump_active .. 0 homeless 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51e8074810 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v717) ==== 54+0+16 (unknown 1276096425 0 2420000192) 0x7f51f8088f70 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 10 client.58676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 1 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 20 client.58676.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 10 client.58676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=718}) -- 0x7f51ec0053e0 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.674+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f51ec000e30 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=717) ==== 24+0+0 (unknown 606173615 0 0) 0x7f51f8054fe0 con 0x7f520010d970 2024-03-30T15:09:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f51fdffb700 10 client.58676.objecter _check_command_map_dne tid 1 current 717 map_dne_bound 717 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f51fdffb700 10 client.58676.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f51fdffb700 15 client.58676.objecter _session_command_op_remove -1 1 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 10 client.58676.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 2 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 2 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 1 2 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 4 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 10 client.58676.objecter _send_command 2 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f51ec0022f0 con 0x7f51ec002750 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.675+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec002750 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.676+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec0022f0 con 0x7f51ec002750 2024-03-30T15:09:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.676+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.676+0000 7f51fcff9700 10 client.58676.objecter _finish_command 2 = system:0 2024-03-30T15:09:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.676+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 4 2024-03-30T15:09:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.676+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 1 2 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: osdmap e716: 8 total, 7 up, 8 in 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/283809761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: from='client.58670 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:09:38.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:09:38.683 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:38 smithi134 ceph-mon[33714]: pgmap v1063: 246 pgs: 32 unknown, 18 active+undersized+degraded, 52 active+undersized, 144 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 10 client.58676.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec001bb0 osd=1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 1 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 4 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 10 client.58676.objecter _send_command 3 2024-03-30T15:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.693+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec003aa0 con 0x7f51ec002750 2024-03-30T15:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec003aa0 con 0x7f51ec002750 2024-03-30T15:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f51fcff9700 10 client.58676.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec001bb0 osd=1 4 2024-03-30T15:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 1 3 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 10 client.58676.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 4 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 4 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 2 4 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 4 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 10 client.58676.objecter _send_command 4 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.695+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f51ec00cb30 con 0x7f51ec009020 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec009020 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec00cb30 con 0x7f51ec009020 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 10 client.58676.objecter _finish_command 4 = system:0 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 4 2024-03-30T15:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.696+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 2 4 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 10 client.58676.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 5 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 5 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec004eb0 osd=2 3 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 2 5 2024-03-30T15:09:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 4 2024-03-30T15:09:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 10 client.58676.objecter _send_command 5 2024-03-30T15:09:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.713+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec003640 con 0x7f51ec009020 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec003640 con 0x7f51ec009020 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 10 client.58676.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec004eb0 osd=2 4 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 2 5 2024-03-30T15:09:38.717 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 10 client.58676.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 6 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 6 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 3 6 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 4 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 10 client.58676.objecter _send_command 6 2024-03-30T15:09:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f51ec011a00 con 0x7f51ec00de60 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.715+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec00de60 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.716+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec011a00 con 0x7f51ec00de60 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.716+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.716+0000 7f51fcff9700 10 client.58676.objecter _finish_command 6 = system:0 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.716+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 4 2024-03-30T15:09:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.716+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 3 6 2024-03-30T15:09:38.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 10 client.58676.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 7 2024-03-30T15:09:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 7 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec0038d0 osd=3 3 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 3 7 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 4 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 10 client.58676.objecter _send_command 7 2024-03-30T15:09:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.732+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec004290 con 0x7f51ec00de60 2024-03-30T15:09:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec004290 con 0x7f51ec00de60 2024-03-30T15:09:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 10 client.58676.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec0038d0 osd=3 4 2024-03-30T15:09:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 3 7 2024-03-30T15:09:38.737 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 10 client.58676.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 8 2024-03-30T15:09:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 8 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 4 8 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 4 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 10 client.58676.objecter _send_command 8 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f51ec016840 con 0x7f51ec012d30 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.734+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec012d30 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.735+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec016840 con 0x7f51ec012d30 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.735+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.735+0000 7f51fcff9700 10 client.58676.objecter _finish_command 8 = system:0 2024-03-30T15:09:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.735+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 4 2024-03-30T15:09:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.735+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 4 8 2024-03-30T15:09:38.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 10 client.58676.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 9 2024-03-30T15:09:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 9 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00d4b0 osd=4 3 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 4 9 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 4 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 10 client.58676.objecter _send_command 9 2024-03-30T15:09:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.752+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec00cf90 con 0x7f51ec012d30 2024-03-30T15:09:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec00cf90 con 0x7f51ec012d30 2024-03-30T15:09:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f51fcff9700 10 client.58676.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00d4b0 osd=4 4 2024-03-30T15:09:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 4 9 2024-03-30T15:09:38.757 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 10 client.58676.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 10 2024-03-30T15:09:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 10 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 5 10 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 4 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 10 client.58676.objecter _send_command 10 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.754+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f51ec018340 con 0x7f51ec012890 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec012890 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec018340 con 0x7f51ec012890 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 10 client.58676.objecter _finish_command 10 = system:0 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 4 2024-03-30T15:09:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.755+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 5 10 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 10 client.58676.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 11 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 11 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00cf90 osd=5 3 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 5 11 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 4 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 10 client.58676.objecter _send_command 11 2024-03-30T15:09:38.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.772+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec018340 con 0x7f51ec012890 2024-03-30T15:09:38.775 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec018340 con 0x7f51ec012890 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f51fcff9700 10 client.58676.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00cf90 osd=5 4 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 5 11 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 10 client.58676.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 12 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 12 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 6 12 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 4 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 10 client.58676.objecter _send_command 12 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.773+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f51ec019040 con 0x7f51ec018a30 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.774+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec018a30 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.775+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec019040 con 0x7f51ec018a30 2024-03-30T15:09:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.775+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.775+0000 7f51fcff9700 10 client.58676.objecter _finish_command 12 = system:0 2024-03-30T15:09:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.775+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 4 2024-03-30T15:09:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.775+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 6 12 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 10 client.58676.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 13 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 13 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec018340 osd=6 3 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 6 13 2024-03-30T15:09:38.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 4 2024-03-30T15:09:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 10 client.58676.objecter _send_command 13 2024-03-30T15:09:38.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.792+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec019060 con 0x7f51ec018a30 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51f4003d10 con 0x7f51ec018a30 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 10 client.58676.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec018340 osd=6 4 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 6 13 2024-03-30T15:09:38.796 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 10 client.58676.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 14 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 14 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 7 14 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 4 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 10 client.58676.objecter _send_command 14 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f51ec017620 con 0x7f51ec0171c0 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.794+0000 7f51fcff9700 10 client.58676.objecter ms_handle_connect 0x7f51ec0171c0 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.795+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f51ec017620 con 0x7f51ec0171c0 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.795+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.795+0000 7f51fcff9700 10 client.58676.objecter _finish_command 14 = system:0 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.795+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 4 2024-03-30T15:09:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.795+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 7 14 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 10 client.58676.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign -1 15 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 20 client.58676.objecter _get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 15 client.58676.objecter _session_command_op_remove -1 15 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 20 client.58676.objecter get_session s=0x7f51ec00d1f0 osd=7 3 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 15 client.58676.objecter _session_command_op_assign 7 15 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 4 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 10 client.58676.objecter _send_command 15 2024-03-30T15:09:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.812+0000 7f5206205700 1 -- 172.21.15.28:0/1469990316 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f51ec017620 con 0x7f51ec0171c0 2024-03-30T15:09:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.814+0000 7f51fcff9700 1 -- 172.21.15.28:0/1469990316 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f51ec017620 con 0x7f51ec0171c0 2024-03-30T15:09:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.814+0000 7f51fcff9700 10 client.58676.objecter ms_dispatch 0x7f5200001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.814+0000 7f51fcff9700 10 client.58676.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.814+0000 7f51fcff9700 20 client.58676.objecter put_session s=0x7f51ec00d1f0 osd=7 4 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:38.814+0000 7f51fcff9700 15 client.58676.objecter _session_command_op_remove 7 15 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:38.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:38.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:38.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:38.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:38.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:39.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:39.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:39.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:39.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:39.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:39.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:39.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:39.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:39.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:39.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:39.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:09:39.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:39.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: from='client.58670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: osdmap e717: 8 total, 7 up, 8 in 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1469990316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: from='client.58670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: osdmap e717: 8 total, 7 up, 8 in 2024-03-30T15:09:39.658 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:09:39.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:09:39.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1469990316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: from='client.58670 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: osdmap e717: 8 total, 7 up, 8 in 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:09:39.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1469990316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:39.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:39.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:39.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:39.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:39.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:40.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:40.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:40.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:40.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:40.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:40.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:40.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:40.669 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: osdmap e718: 8 total, 7 up, 8 in 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T15:09:40.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-mon[33714]: pgmap v1066: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:40.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: osdmap e718: 8 total, 7 up, 8 in 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[29876]: pgmap v1066: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: osdmap e718: 8 total, 7 up, 8 in 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T15:09:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T15:09:40.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:40 smithi028 ceph-mon[41614]: pgmap v1066: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:40.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:40.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:40.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:40.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:40.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:40.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:41.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:41.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:41.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:41.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:41.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:41.288 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:41.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:41.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:41.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: osdmap e719: 8 total, 7 up, 8 in 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 42.4 scrub starts 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 42.4 scrub ok 2024-03-30T15:09:41.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 1.a deep-scrub starts 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: 1.a deep-scrub ok 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]': finished 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: osdmap e719: 8 total, 7 up, 8 in 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 42.4 scrub starts 2024-03-30T15:09:41.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 42.4 scrub ok 2024-03-30T15:09:41.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:41.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 1.a deep-scrub starts 2024-03-30T15:09:41.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: 1.a deep-scrub ok 2024-03-30T15:09:41.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:41.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:41 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]': finished 2024-03-30T15:09:41.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:41.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:41.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:41.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:41.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:41.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: osdmap e719: 8 total, 7 up, 8 in 2024-03-30T15:09:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]: dispatch 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 42.4 scrub starts 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 42.4 scrub ok 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 1.a deep-scrub starts 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: 1.a deep-scrub ok 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:41.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:41 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-128", "mode": "writeback"}]': finished 2024-03-30T15:09:41.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:42.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:42.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:42.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:42.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:42.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:42.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.520+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/3896624960 >> v1:172.21.15.28:6789/0 conn(0x7f5c7410d970 legacy=0x7f5c7410fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.520+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/3896624960 shutdown_connections 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.520+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/3896624960 >> 172.21.15.28:0/3896624960 conn(0x7f5c740fc9d0 msgr2=0x7f5c740fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.520+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/3896624960 shutdown_connections 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.520+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/3896624960 wait complete. 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 librados: starting msgr at 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 librados: starting objecter 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 Processor -- start 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 -- start start 2024-03-30T15:09:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 librados: setting wanted keys 2024-03-30T15:09:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 librados: calling monclient init 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5c741a92f0 con 0x7f5c74101140 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5c741aa510 con 0x7f5c7410d970 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c7a4e1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5c741ab750 con 0x7f5c74109dd0 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c73fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c74101140 0x7f5c74106310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60726/0 (socket says 172.21.15.28:60726) 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c737fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5c74109dd0 0x7f5c741a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47010/0 (socket says 172.21.15.28:47010) 2024-03-30T15:09:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.521+0000 7f5c73fff700 1 -- 172.21.15.28:0/2281379076 learned_addr learned my addr 172.21.15.28:0/2281379076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter ms_handle_connect 0x7f5c74101140 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter ms_handle_connect 0x7f5c74109dd0 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter ms_handle_connect 0x7f5c7410d970 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4275410447 0 0) 0x7f5c741ab750 con 0x7f5c74109dd0 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c5c0039c0 con 0x7f5c74109dd0 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353692257 0 0) 0x7f5c741a92f0 con 0x7f5c74101140 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c741ab750 con 0x7f5c74101140 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907055007 0 0) 0x7f5c741aa510 con 0x7f5c7410d970 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c741a92f0 con 0x7f5c7410d970 2024-03-30T15:09:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4220360499 0 0) 0x7f5c5c0039c0 con 0x7f5c74109dd0 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c741aa510 con 0x7f5c74109dd0 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5c680018b0 con 0x7f5c74109dd0 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2444910574 0 0) 0x7f5c741a92f0 con 0x7f5c7410d970 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c5c0039c0 con 0x7f5c7410d970 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1974107863 0 0) 0x7f5c741ab750 con 0x7f5c74101140 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c741a92f0 con 0x7f5c74101140 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5c6c001be0 con 0x7f5c7410d970 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5c64001230 con 0x7f5c74101140 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2100199079 0 0) 0x7f5c741aa510 con 0x7f5c74109dd0 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 >> v1:172.21.15.134:6789/0 conn(0x7f5c7410d970 legacy=0x7f5c741a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 >> v1:172.21.15.28:6789/0 conn(0x7f5c74101140 legacy=0x7f5c74106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c741ac990 con 0x7f5c74109dd0 2024-03-30T15:09:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c741ab980 con 0x7f5c74109dd0 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c7a4e1700 10 client.58682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c741abe40 con 0x7f5c74109dd0 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c7a4e1700 1 librados: init done 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.522+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5c68003b40 con 0x7f5c74109dd0 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.523+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5c68005bc0 con 0x7f5c74109dd0 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.523+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c60005120 con 0x7f5c74109dd0 2024-03-30T15:09:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.524+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5c680036a0 con 0x7f5c74109dd0 2024-03-30T15:09:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.524+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(721..721 src has 442..721) ==== 9222+0+0 (unknown 3947049133 0 0) 0x7f5c68089be0 con 0x7f5c74109dd0 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.524+0000 7f5c717fa700 10 client.58682.objecter ms_dispatch 0x7f5c74001550 osd_map(721..721 src has 442..721) 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.524+0000 7f5c717fa700 3 client.58682.objecter handle_osd_map got epochs [721,721] > 0 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.524+0000 7f5c717fa700 3 client.58682.objecter handle_osd_map decoding full epoch 721 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.525+0000 7f5c717fa700 20 client.58682.objecter dump_active .. 0 homeless 2024-03-30T15:09:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.525+0000 7f5c717fa700 10 client.58682.objecter ms_handle_connect 0x7f5c5c070490 2024-03-30T15:09:42.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:42.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.529+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5c680569d0 con 0x7f5c74109dd0 2024-03-30T15:09:42.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:42.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:42.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: osdmap e720: 8 total, 7 up, 8 in 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: 42.b scrub starts 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: 42.b scrub ok 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: pgmap v1069: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[41614]: osdmap e721: 8 total, 7 up, 8 in 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: osdmap e720: 8 total, 7 up, 8 in 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: 42.b scrub starts 2024-03-30T15:09:42.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: 42.b scrub ok 2024-03-30T15:09:42.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:09:42.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:09:42.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: pgmap v1069: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:42.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:42 smithi028 ceph-mon[29876]: osdmap e721: 8 total, 7 up, 8 in 2024-03-30T15:09:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:42.752+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"} v 0) -- 0x7f5c60005410 con 0x7f5c74109dd0 2024-03-30T15:09:42.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:42.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:09:42.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:42.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:42.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: osdmap e720: 8 total, 7 up, 8 in 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: 42.b scrub starts 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: 42.b scrub ok 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: pgmap v1069: 246 pgs: 18 active+undersized+degraded, 61 active+undersized, 167 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:42 smithi134 ceph-mon[33714]: osdmap e721: 8 total, 7 up, 8 in 2024-03-30T15:09:43.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:43.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:43.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:43.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:43.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:09:43.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.458+0000 7f5c717fa700 1 -- 172.21.15.28:0/2281379076 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]=0 clear 5.13 pg_upmap_items mapping v722) ==== 118+0+0 (unknown 357650744 0 0) 0x7f5c6805a420 con 0x7f5c74109dd0 2024-03-30T15:09:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 10 librados: watch_flush enter 2024-03-30T15:09:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 10 librados: watch_flush exit 2024-03-30T15:09:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 20 client.58682.objecter shutdown clearing up homeless session... 2024-03-30T15:09:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 10 client.58682.objecter successfully canceled tick 2024-03-30T15:09:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5c5c070490 legacy=0x7f5c5c072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 >> v1:172.21.15.28:6790/0 conn(0x7f5c74109dd0 legacy=0x7f5c741a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 shutdown_connections 2024-03-30T15:09:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.462+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 >> 172.21.15.28:0/2281379076 conn(0x7f5c740fc9d0 msgr2=0x7f5c74110e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.463+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 shutdown_connections 2024-03-30T15:09:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.463+0000 7f5c7a4e1700 1 -- 172.21.15.28:0/2281379076 wait complete. 2024-03-30T15:09:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:43.463+0000 7f5c7a4e1700 1 librados: shutdown 2024-03-30T15:09:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:clear 5.13 pg_upmap_items mapping 2024-03-30T15:09:43.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:43.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:43.566 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:09:43.566 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:09:43.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:43.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2281379076' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: from='client.58682 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 6.15 deep-scrub starts 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[29876]: 6.15 deep-scrub ok 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:09:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2281379076' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: from='client.58682 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 6.15 deep-scrub starts 2024-03-30T15:09:43.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:43 smithi028 ceph-mon[41614]: 6.15 deep-scrub ok 2024-03-30T15:09:43.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:43.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:43.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:43.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:43.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2281379076' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: from='client.58682 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]: dispatch 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 6.15 deep-scrub starts 2024-03-30T15:09:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:43 smithi134 ceph-mon[33714]: 6.15 deep-scrub ok 2024-03-30T15:09:44.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:44.052 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:44.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:44.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:44.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:44.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:44.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:44.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:44.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:44.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:44.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: from='client.58682 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]': finished 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: osdmap e722: 8 total, 7 up, 8 in 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 1.8 scrub starts 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 1.8 scrub ok 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T15:09:44.714 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[41614]: pgmap v1072: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: from='client.58682 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]': finished 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: osdmap e722: 8 total, 7 up, 8 in 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 1.8 scrub starts 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 1.8 scrub ok 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T15:09:44.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:44 smithi028 ceph-mon[29876]: pgmap v1072: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:44.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:44.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:44.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: from='client.58682 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.13"}]': finished 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: osdmap e722: 8 total, 7 up, 8 in 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 1.8 scrub starts 2024-03-30T15:09:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 1.8 scrub ok 2024-03-30T15:09:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T15:09:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T15:09:44.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:44 smithi134 ceph-mon[33714]: pgmap v1072: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:44.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:44.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:45.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:45.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:45.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:45.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:45.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:45.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:45.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/639 objects degraded (9.703%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: osdmap e723: 8 total, 7 up, 8 in 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 1.9 scrub starts 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 1.9 scrub ok 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 42.14 scrub starts 2024-03-30T15:09:45.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[29876]: 42.14 scrub ok 2024-03-30T15:09:45.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:09:45.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:45.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/639 objects degraded (9.703%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: osdmap e723: 8 total, 7 up, 8 in 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 1.9 scrub starts 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 1.9 scrub ok 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 42.14 scrub starts 2024-03-30T15:09:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:45 smithi028 ceph-mon[41614]: 42.14 scrub ok 2024-03-30T15:09:45.725 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:45.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:45.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:09:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:09:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/639 objects degraded (9.703%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: osdmap e723: 8 total, 7 up, 8 in 2024-03-30T15:09:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 1.9 scrub starts 2024-03-30T15:09:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 1.9 scrub ok 2024-03-30T15:09:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 42.14 scrub starts 2024-03-30T15:09:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:45 smithi134 ceph-mon[33714]: 42.14 scrub ok 2024-03-30T15:09:45.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:45.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:45.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:46.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:46.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:46.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:46.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:46.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:46.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:46.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:46.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:46.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:09:46.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:09:46.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[41614]: osdmap e724: 8 total, 7 up, 8 in 2024-03-30T15:09:46.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[41614]: pgmap v1075: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:46.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:09:46.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:09:46.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[29876]: osdmap e724: 8 total, 7 up, 8 in 2024-03-30T15:09:46.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:46 smithi028 ceph-mon[29876]: pgmap v1075: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:46.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.695+0000 7f86a3d18700 1 -- 172.21.15.28:0/3889052767 >> v1:172.21.15.28:6789/0 conn(0x7f869c10eab0 legacy=0x7f869c110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.695+0000 7f86a3d18700 1 -- 172.21.15.28:0/3889052767 shutdown_connections 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.695+0000 7f86a3d18700 1 -- 172.21.15.28:0/3889052767 >> 172.21.15.28:0/3889052767 conn(0x7f869c0fc9a0 msgr2=0x7f869c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.695+0000 7f86a3d18700 1 -- 172.21.15.28:0/3889052767 shutdown_connections 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.695+0000 7f86a3d18700 1 -- 172.21.15.28:0/3889052767 wait complete. 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 librados: starting msgr at 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 librados: starting objecter 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 Processor -- start 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 -- start start 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 librados: setting wanted keys 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 librados: calling monclient init 2024-03-30T15:09:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f869c0fec40 con 0x7f869c10eab0 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f869c10ad10 con 0x7f869c10afa0 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a3d18700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f869c1a9640 con 0x7f869c103190 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a1ab4700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f869c103190 0x7f869c107b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47038/0 (socket says 172.21.15.28:47038) 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f86a1ab4700 1 -- 172.21.15.28:0/645247607 learned_addr learned my addr 172.21.15.28:0/645247607 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f869affd700 10 client.?.objecter ms_handle_connect 0x7f869c103190 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f869affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f869affd700 10 client.?.objecter ms_handle_connect 0x7f869c10eab0 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.696+0000 7f869affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 10 client.?.objecter ms_handle_connect 0x7f869c10afa0 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404424815 0 0) 0x7f869c1a9640 con 0x7f869c103190 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8684003aa0 con 0x7f869c103190 2024-03-30T15:09:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843951086 0 0) 0x7f869c0fec40 con 0x7f869c10eab0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f869c1a9640 con 0x7f869c10eab0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183911379 0 0) 0x7f869c10ad10 con 0x7f869c10afa0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f869c0fec40 con 0x7f869c10afa0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4028835286 0 0) 0x7f8684003aa0 con 0x7f869c103190 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f869c10ad10 con 0x7f869c103190 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f868c001cb0 con 0x7f869c103190 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3646824664 0 0) 0x7f869c1a9640 con 0x7f869c10eab0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8684003aa0 con 0x7f869c10eab0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8694004550 con 0x7f869c10eab0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2590870491 0 0) 0x7f869c0fec40 con 0x7f869c10afa0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f869c1a9640 con 0x7f869c10afa0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8690001960 con 0x7f869c10afa0 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2352151332 0 0) 0x7f869c10ad10 con 0x7f869c103190 2024-03-30T15:09:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 >> v1:172.21.15.134:6789/0 conn(0x7f869c10afa0 legacy=0x7f869c108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 >> v1:172.21.15.28:6789/0 conn(0x7f869c10eab0 legacy=0x7f869c1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869c1aa820 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f869c1a9870 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f86a3d18700 10 client.58688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f869c1a9d30 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f86a3d18700 1 librados: init done 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f868c003ec0 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.697+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f868c006030 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.698+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8688005100 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f868c003a20 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(724..724 src has 442..724) ==== 9226+0+0 (unknown 2977677546 0 0) 0x7f868c089dd0 con 0x7f869c103190 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 osd_map(724..724 src has 442..724) 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 3 client.58688.objecter handle_osd_map got epochs [724,724] > 0 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 3 client.58688.objecter handle_osd_map decoding full epoch 724 2024-03-30T15:09:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 20 client.58688.objecter dump_active .. 0 homeless 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v724) ==== 54+0+16 (unknown 3210004184 0 2420000192) 0x7f868c08a200 con 0x7f869c103190 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.699+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f8684070420 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 10 client.58688.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 1 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter _get_session osd=-1 returning homeless 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 10 client.58688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=725}) -- 0x7f86880053e0 con 0x7f869c103190 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8688000ed0 con 0x7f869c103190 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=724) ==== 24+0+0 (unknown 1240363977 0 0) 0x7f868c056280 con 0x7f869c103190 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f869bfff700 10 client.58688.objecter _check_command_map_dne tid 1 current 724 map_dne_bound 724 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f869bfff700 10 client.58688.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f869bfff700 15 client.58688.objecter _session_command_op_remove -1 1 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 10 client.58688.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 2 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 2 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 1 2 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 4 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 10 client.58688.objecter _send_command 2 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f8688004460 con 0x7f8688003910 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.700+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f8688003910 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.701+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8688004460 con 0x7f8688003910 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.701+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 2: 0 ) 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.701+0000 7f869affd700 10 client.58688.objecter _finish_command 2 = system:0 2024-03-30T15:09:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.701+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 4 2024-03-30T15:09:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.701+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 1 2 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 10 client.58688.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 3 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 3 2024-03-30T15:09:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f86880011c0 osd=1 3 2024-03-30T15:09:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 1 3 2024-03-30T15:09:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 4 2024-03-30T15:09:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 10 client.58688.objecter _send_command 3 2024-03-30T15:09:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.719+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8688004e10 con 0x7f8688003910 2024-03-30T15:09:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.721+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8688004e10 con 0x7f8688003910 2024-03-30T15:09:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.721+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.721+0000 7f869affd700 10 client.58688.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.721+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f86880011c0 osd=1 4 2024-03-30T15:09:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.721+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 1 3 2024-03-30T15:09:46.723 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 10 client.58688.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 4 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 4 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 2 4 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 4 2024-03-30T15:09:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 10 client.58688.objecter _send_command 4 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f868800ca20 con 0x7f8688008f10 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.722+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f8688008f10 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.723+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f868800ca20 con 0x7f8688008f10 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.723+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 4: 0 ) 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.723+0000 7f869affd700 10 client.58688.objecter _finish_command 4 = system:0 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.723+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 4 2024-03-30T15:09:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.723+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 2 4 2024-03-30T15:09:46.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 10 client.58688.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 5 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 5 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688001ad0 osd=2 3 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 2 5 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 4 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 10 client.58688.objecter _send_command 5 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.741+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8688004950 con 0x7f8688008f10 2024-03-30T15:09:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.742+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8688004950 con 0x7f8688008f10 2024-03-30T15:09:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.742+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.742+0000 7f869affd700 10 client.58688.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.742+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688001ad0 osd=2 4 2024-03-30T15:09:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.742+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 2 5 2024-03-30T15:09:46.744 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 10 client.58688.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 6 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 6 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 3 6 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 4 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 10 client.58688.objecter _send_command 6 2024-03-30T15:09:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f8688003710 con 0x7f868800cbd0 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.743+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f868800cbd0 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.744+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8688003710 con 0x7f868800cbd0 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.744+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 6: 0 ) 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.744+0000 7f869affd700 10 client.58688.objecter _finish_command 6 = system:0 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.744+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 4 2024-03-30T15:09:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.744+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 3 6 2024-03-30T15:09:46.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 10 client.58688.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 7 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 7 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688002f60 osd=3 3 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 3 7 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 4 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 10 client.58688.objecter _send_command 7 2024-03-30T15:09:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.761+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f868800df90 con 0x7f868800cbd0 2024-03-30T15:09:46.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f868800df90 con 0x7f868800cbd0 2024-03-30T15:09:46.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f869affd700 10 client.58688.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688002f60 osd=3 4 2024-03-30T15:09:46.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 3 7 2024-03-30T15:09:46.764 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:09:46.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 10 client.58688.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 8 2024-03-30T15:09:46.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 8 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 4 8 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 4 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 10 client.58688.objecter _send_command 8 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.762+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f868801aae0 con 0x7f8688016f30 2024-03-30T15:09:46.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.763+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f8688016f30 2024-03-30T15:09:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.764+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f868801aae0 con 0x7f8688016f30 2024-03-30T15:09:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.764+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 8: 0 ) 2024-03-30T15:09:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.764+0000 7f869affd700 10 client.58688.objecter _finish_command 8 = system:0 2024-03-30T15:09:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.764+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 4 2024-03-30T15:09:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.764+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 4 8 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 10 client.58688.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 9 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 9 2024-03-30T15:09:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f8688002d50 osd=4 3 2024-03-30T15:09:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 4 9 2024-03-30T15:09:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 4 2024-03-30T15:09:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 10 client.58688.objecter _send_command 9 2024-03-30T15:09:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.781+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f868800daa0 con 0x7f8688016f30 2024-03-30T15:09:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f868800daa0 con 0x7f8688016f30 2024-03-30T15:09:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f869affd700 10 client.58688.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f8688002d50 osd=4 4 2024-03-30T15:09:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 4 9 2024-03-30T15:09:46.785 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:09:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 10 client.58688.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 10 2024-03-30T15:09:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 10 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 5 10 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 4 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 10 client.58688.objecter _send_command 10 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.783+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f868800da80 con 0x7f868800f0b0 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f868800f0b0 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f868800da80 con 0x7f868800f0b0 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 10: 0 ) 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 10 client.58688.objecter _finish_command 10 = system:0 2024-03-30T15:09:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 4 2024-03-30T15:09:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.784+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 5 10 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 10 client.58688.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 11 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 11 2024-03-30T15:09:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868800d850 osd=5 3 2024-03-30T15:09:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 5 11 2024-03-30T15:09:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 4 2024-03-30T15:09:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 10 client.58688.objecter _send_command 11 2024-03-30T15:09:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.802+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f868801c460 con 0x7f868800f0b0 2024-03-30T15:09:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f868801c460 con 0x7f868800f0b0 2024-03-30T15:09:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f869affd700 10 client.58688.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868800d850 osd=5 4 2024-03-30T15:09:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 5 11 2024-03-30T15:09:46.806 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:09:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 10 client.58688.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 12 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 12 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 6 12 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 4 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 10 client.58688.objecter _send_command 12 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.804+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f868801d010 con 0x7f868801cb80 2024-03-30T15:09:46.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.805+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f868801cb80 2024-03-30T15:09:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.806+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f868801d010 con 0x7f868801cb80 2024-03-30T15:09:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.806+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 12: 0 ) 2024-03-30T15:09:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.806+0000 7f869affd700 10 client.58688.objecter _finish_command 12 = system:0 2024-03-30T15:09:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.806+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 4 2024-03-30T15:09:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.806+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 6 12 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 10 client.58688.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 13 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 13 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868800dae0 osd=6 3 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 6 13 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 4 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 10 client.58688.objecter _send_command 13 2024-03-30T15:09:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.825+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f868801d010 con 0x7f868801cb80 2024-03-30T15:09:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f868801d010 con 0x7f868801cb80 2024-03-30T15:09:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f869affd700 10 client.58688.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868800dae0 osd=6 4 2024-03-30T15:09:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 6 13 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 10 client.58688.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 14 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 14 2024-03-30T15:09:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 7 14 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 4 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 10 client.58688.objecter _send_command 14 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.827+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f8688028090 con 0x7f8688024580 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 10 client.58688.objecter ms_handle_connect 0x7f8688024580 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8688028090 con 0x7f8688024580 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 14: 0 ) 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 10 client.58688.objecter _finish_command 14 = system:0 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 4 2024-03-30T15:09:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.828+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 7 14 2024-03-30T15:09:46.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 10 client.58688.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign -1 15 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 20 client.58688.objecter _get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_remove -1 15 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 20 client.58688.objecter get_session s=0x7f868801d010 osd=7 3 2024-03-30T15:09:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 15 client.58688.objecter _session_command_op_assign 7 15 2024-03-30T15:09:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 4 2024-03-30T15:09:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 10 client.58688.objecter _send_command 15 2024-03-30T15:09:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.860+0000 7f86a3d18700 1 -- 172.21.15.28:0/645247607 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8688028090 con 0x7f8688024580 2024-03-30T15:09:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.862+0000 7f869affd700 1 -- 172.21.15.28:0/645247607 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8688028090 con 0x7f8688024580 2024-03-30T15:09:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.862+0000 7f869affd700 10 client.58688.objecter ms_dispatch 0x7f869c001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:09:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.862+0000 7f869affd700 10 client.58688.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:09:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.862+0000 7f869affd700 20 client.58688.objecter put_session s=0x7f868801d010 osd=7 4 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:46.862+0000 7f869affd700 15 client.58688.objecter _session_command_op_remove 7 15 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:09:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:09:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:46.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:46 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:09:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:46 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:09:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:46 smithi134 ceph-mon[33714]: osdmap e724: 8 total, 7 up, 8 in 2024-03-30T15:09:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:46 smithi134 ceph-mon[33714]: pgmap v1075: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:46.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:47.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:47.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:47.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:47.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:47.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:47.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:47.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:47.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:47.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:47.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: 6.1 deep-scrub starts 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: 6.1 deep-scrub ok 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/645247607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: osdmap e725: 8 total, 7 up, 8 in 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: 6.1 deep-scrub starts 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: 6.1 deep-scrub ok 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/645247607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: osdmap e725: 8 total, 7 up, 8 in 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:47 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:47.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:47.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:47.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:47.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:47.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: 6.1 deep-scrub starts 2024-03-30T15:09:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: 6.1 deep-scrub ok 2024-03-30T15:09:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/645247607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: osdmap e725: 8 total, 7 up, 8 in 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:47 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]: dispatch 2024-03-30T15:09:48.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:48.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:48.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:48.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:48.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:48.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:48.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:48.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:48.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:48.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 -- 172.21.15.28:0/1832956460 >> v1:172.21.15.28:6790/0 conn(0x7f70b4109dd0 legacy=0x7f70b410c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 -- 172.21.15.28:0/1832956460 shutdown_connections 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 -- 172.21.15.28:0/1832956460 >> 172.21.15.28:0/1832956460 conn(0x7f70b40fc9d0 msgr2=0x7f70b40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 -- 172.21.15.28:0/1832956460 shutdown_connections 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 -- 172.21.15.28:0/1832956460 wait complete. 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 librados: starting msgr at 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 librados: starting objecter 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.639+0000 7f70b98f8700 1 Processor -- start 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 -- start start 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 librados: setting wanted keys 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 librados: calling monclient init 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b41a9320 con 0x7f70b4109dd0 2024-03-30T15:09:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b41aa540 con 0x7f70b410d970 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b98f8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b41ab780 con 0x7f70b4101140 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b2ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f70b4101140 0x7f70b4106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37454/0 (socket says 172.21.15.28:37454) 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b2ffd700 1 -- 172.21.15.28:0/2707872873 learned_addr learned my addr 172.21.15.28:0/2707872873 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b27fc700 1 --1- 172.21.15.28:0/2707872873 >> v1:172.21.15.28:6789/0 conn(0x7f70b4109dd0 0x7f70b41a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42172/0 (socket says 172.21.15.28:42172) 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter ms_handle_connect 0x7f70b4101140 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter ms_handle_connect 0x7f70b410d970 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter ms_handle_connect 0x7f70b4109dd0 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847407848 0 0) 0x7f70b41ab780 con 0x7f70b4101140 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f709c0031f0 con 0x7f70b4101140 2024-03-30T15:09:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269471604 0 0) 0x7f70b41aa540 con 0x7f70b410d970 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b41ab780 con 0x7f70b410d970 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 927423387 0 0) 0x7f70b41a9320 con 0x7f70b4109dd0 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b41aa540 con 0x7f70b4109dd0 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1591304960 0 0) 0x7f709c0031f0 con 0x7f70b4101140 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b41a9320 con 0x7f70b4101140 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.640+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a40019b0 con 0x7f70b4101140 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3190507526 0 0) 0x7f70b41ab780 con 0x7f70b410d970 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f709c0031f0 con 0x7f70b410d970 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70ac001420 con 0x7f70b410d970 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3631925815 0 0) 0x7f70b41aa540 con 0x7f70b4109dd0 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b41ab780 con 0x7f70b4109dd0 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4035200006 0 0) 0x7f70b41a9320 con 0x7f70b4101140 2024-03-30T15:09:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 >> v1:172.21.15.134:6789/0 conn(0x7f70b410d970 legacy=0x7f70b41a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 >> v1:172.21.15.28:6789/0 conn(0x7f70b4109dd0 legacy=0x7f70b41a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b41ac9c0 con 0x7f70b4101140 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70b41aa770 con 0x7f70b4101140 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b98f8700 10 client.58694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70b41aac30 con 0x7f70b4101140 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b98f8700 1 librados: init done 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f70a4003e70 con 0x7f70b4101140 2024-03-30T15:09:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.641+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a4005e60 con 0x7f70b4101140 2024-03-30T15:09:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.642+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a0005120 con 0x7f70b4101140 2024-03-30T15:09:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.642+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f70a40039d0 con 0x7f70b4101140 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(726..726 src has 442..726) ==== 9218+0+0 (unknown 2504245267 0 0) 0x7f70a4089e00 con 0x7f70b4101140 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 10 client.58694.objecter ms_dispatch 0x7f70b4001550 osd_map(726..726 src has 442..726) 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 3 client.58694.objecter handle_osd_map got epochs [726,726] > 0 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 3 client.58694.objecter handle_osd_map decoding full epoch 726 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 20 client.58694.objecter dump_active .. 0 homeless 2024-03-30T15:09:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.643+0000 7f70b88f6700 10 client.58694.objecter ms_handle_connect 0x7f709c070410 2024-03-30T15:09:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.648+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f70a4056bd0 con 0x7f70b4101140 2024-03-30T15:09:48.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:48.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T15:09:48.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T15:09:48.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: pgmap v1077: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[41614]: osdmap e726: 8 total, 7 up, 8 in 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T15:09:48.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: pgmap v1077: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:48.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:48 smithi028 ceph-mon[29876]: osdmap e726: 8 total, 7 up, 8 in 2024-03-30T15:09:48.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:48.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:48.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:48.871+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f70a0005410 con 0x7f70b4101140 2024-03-30T15:09:48.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:48.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:09:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: pgmap v1077: 246 pgs: 1 unknown, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 1 op/s; 62/639 objects degraded (9.703%) 2024-03-30T15:09:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-128"}]': finished 2024-03-30T15:09:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:48 smithi134 ceph-mon[33714]: osdmap e726: 8 total, 7 up, 8 in 2024-03-30T15:09:49.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:49.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:49.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:49.181 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:09:49.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:09:49.181 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T15:09:49.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:09:49.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:49.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:09:49.315 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:09:49.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.384+0000 7f70b88f6700 1 -- 172.21.15.28:0/2707872873 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v727) ==== 87+0+0 (unknown 3149877794 0 0) 0x7f70a405a620 con 0x7f70b4101140 2024-03-30T15:09:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 10 librados: watch_flush enter 2024-03-30T15:09:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 10 librados: watch_flush exit 2024-03-30T15:09:49.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 20 client.58694.objecter shutdown clearing up homeless session... 2024-03-30T15:09:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 10 client.58694.objecter successfully canceled tick 2024-03-30T15:09:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 >> v1:172.21.15.28:6800/3011431093 conn(0x7f709c070410 legacy=0x7f709c0728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 >> v1:172.21.15.28:6790/0 conn(0x7f70b4101140 legacy=0x7f70b4106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:49.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 shutdown_connections 2024-03-30T15:09:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.387+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 >> 172.21.15.28:0/2707872873 conn(0x7f70b40fc9d0 msgr2=0x7f70b4106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.388+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 shutdown_connections 2024-03-30T15:09:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.388+0000 7f70b98f8700 1 -- 172.21.15.28:0/2707872873 wait complete. 2024-03-30T15:09:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:49.388+0000 7f70b98f8700 1 librados: shutdown 2024-03-30T15:09:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:09:49.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:49.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:49.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:49.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:49.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:49.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:49.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:49.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:49.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2707872873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: from='client.58694 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: from='client.58694 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[29876]: osdmap e727: 8 total, 7 up, 8 in 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:09:49.927 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2707872873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: from='client.58694 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: from='client.58694 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:49.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:49 smithi028 ceph-mon[41614]: osdmap e727: 8 total, 7 up, 8 in 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2707872873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: from='client.58694 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: from='client.58694 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:09:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:49 smithi134 ceph-mon[33714]: osdmap e727: 8 total, 7 up, 8 in 2024-03-30T15:09:50.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:50.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:50.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:50.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:50.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:50.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:50.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:50.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:50.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:50.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:50.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:50.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:50.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:50.802 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:50.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:50.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:09:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-mon[33714]: 42.7 scrub starts 2024-03-30T15:09:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-mon[33714]: pgmap v1080: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-mon[33714]: osdmap e728: 8 total, 7 up, 8 in 2024-03-30T15:09:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:50 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:50.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[29876]: 42.7 scrub starts 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[29876]: pgmap v1080: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[29876]: osdmap e728: 8 total, 7 up, 8 in 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[41614]: 42.7 scrub starts 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[41614]: pgmap v1080: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[41614]: osdmap e728: 8 total, 7 up, 8 in 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:50.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:50 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:51.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:51.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:51.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:51.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:51.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:51.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:51.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:09:51.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:51.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:51.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:51.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:51.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:51.676 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:09:51.676 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:09:51.676 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:09:51.676 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:09:51.676 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:09:51.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:51.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:51.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:51.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: osdmap e729: 8 total, 7 up, 8 in 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:51 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:51.938 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:51.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: osdmap e729: 8 total, 7 up, 8 in 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded (PG_DEGRADED) 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: osdmap e729: 8 total, 7 up, 8 in 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:51 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:52.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:52.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:52.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:52.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:52.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:52.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:52.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:52.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:52.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:52.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:52.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: pgmap v1083: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: osdmap e730: 8 total, 7 up, 8 in 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:52.853 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: pgmap v1083: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:52.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:52.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: osdmap e730: 8 total, 7 up, 8 in 2024-03-30T15:09:52.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.854 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:52.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: pgmap v1083: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: osdmap e730: 8 total, 7 up, 8 in 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:09:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:52.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:53.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.048+0000 7f7eb8428700 1 -- 172.21.15.28:0/2715158810 >> v1:172.21.15.28:6789/0 conn(0x7f7eb0101bd0 legacy=0x7f7eb0101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.048+0000 7f7eb8428700 1 -- 172.21.15.28:0/2715158810 shutdown_connections 2024-03-30T15:09:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.048+0000 7f7eb8428700 1 -- 172.21.15.28:0/2715158810 >> 172.21.15.28:0/2715158810 conn(0x7f7eb00761e0 msgr2=0x7f7eb00765f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.048+0000 7f7eb8428700 1 -- 172.21.15.28:0/2715158810 shutdown_connections 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.048+0000 7f7eb8428700 1 -- 172.21.15.28:0/2715158810 wait complete. 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 librados: starting msgr at 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 librados: starting objecter 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 Processor -- start 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 -- start start 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 librados: setting wanted keys 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 librados: calling monclient init 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb01a7140 con 0x7f7eb0101bd0 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb01a8360 con 0x7f7eb0109e30 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb8428700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7eb01a95a0 con 0x7f7eb010d9d0 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb61c4700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7eb0101bd0 0x7f7eb0106530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42190/0 (socket says 172.21.15.28:42190) 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb69c5700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7eb010d9d0 0x7f7eb01a5800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:37476/0 (socket says 172.21.15.28:37476) 2024-03-30T15:09:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.049+0000 7f7eb61c4700 1 -- 172.21.15.28:0/2997651873 learned_addr learned my addr 172.21.15.28:0/2997651873 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f7eb0101bd0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f7eb010d9d0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f7eb0109e30 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601783359 0 0) 0x7f7eb01a7140 con 0x7f7eb0101bd0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e980031f0 con 0x7f7eb0101bd0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199640965 0 0) 0x7f7eb01a95a0 con 0x7f7eb010d9d0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb01a7140 con 0x7f7eb010d9d0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016248282 0 0) 0x7f7eb01a8360 con 0x7f7eb0109e30 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb01a95a0 con 0x7f7eb0109e30 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1376774510 0 0) 0x7f7e980031f0 con 0x7f7eb0101bd0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb01a8360 con 0x7f7eb0101bd0 2024-03-30T15:09:53.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3567024441 0 0) 0x7f7eb01a95a0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e980031f0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4134201268 0 0) 0x7f7eb01a7140 con 0x7f7eb010d9d0 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb01a95a0 con 0x7f7eb010d9d0 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ea8004590 con 0x7f7eb0101bd0 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ea40017f0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ea00014b0 con 0x7f7eb010d9d0 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 146170554 0 0) 0x7f7e980031f0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 >> v1:172.21.15.28:6790/0 conn(0x7f7eb010d9d0 legacy=0x7f7eb01a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 >> v1:172.21.15.28:6789/0 conn(0x7f7eb0101bd0 legacy=0x7f7eb0106530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb01aa7e0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb01a97d0 con 0x7f7eb0109e30 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eb8428700 10 client.50698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7eb01a9c90 con 0x7f7eb0109e30 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.050+0000 7f7eb8428700 1 librados: init done 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.051+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7ea4003b40 con 0x7f7eb0109e30 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.051+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7ea4004930 con 0x7f7eb0109e30 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e9c005120 con 0x7f7eb0109e30 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7ea4004c40 con 0x7f7eb0109e30 2024-03-30T15:09:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(730..730 src has 442..730) ==== 9226+0+0 (unknown 2412080902 0 0) 0x7f7ea4088750 con 0x7f7eb0109e30 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eaf7fe700 10 client.50698.objecter ms_dispatch 0x7f7eb0001550 osd_map(730..730 src has 442..730) 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eaf7fe700 3 client.50698.objecter handle_osd_map got epochs [730,730] > 0 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.052+0000 7f7eaf7fe700 3 client.50698.objecter handle_osd_map decoding full epoch 730 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.053+0000 7f7eaf7fe700 20 client.50698.objecter dump_active .. 0 homeless 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.053+0000 7f7eaf7fe700 10 client.50698.objecter ms_handle_connect 0x7f7e98074900 2024-03-30T15:09:53.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.057+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7ea4055510 con 0x7f7eb0109e30 2024-03-30T15:09:53.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:53.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:09:53.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:53.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:53.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.265+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f7e9c005410 con 0x7f7eb0109e30 2024-03-30T15:09:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.268+0000 7f7eaf7fe700 1 -- 172.21.15.28:0/2997651873 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v730) ==== 81+0+57674 (unknown 311439731 0 1372024989) 0x7f7ea4058f60 con 0x7f7eb0109e30 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 730, 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:09:52.383464+0000", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T15:08:50.456069+0000", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-03-30T15:09:53.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 117, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 332, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:09:53.271 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-03-30T15:09:53.273 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "562", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "559", 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.274 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.275 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.276 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.277 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.9000000953674316, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.9000000953674316, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.5499999523162842, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.5499999523162842, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:09:53.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:09:53.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.280 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.281 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:09:53.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6700000762939453, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6700000762939453, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.75, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.284 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.285 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.286 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.287 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.288 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:53.290 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.293 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.294 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.295 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.296 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.297 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:09:53.298 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.299 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.300 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.301 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.302 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:09:53.303 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.303 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.303 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.303 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.303 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.304 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.304 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.304 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.304 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.304 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.305 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.307 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.308 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:53.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.310 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.310 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.310 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-03-30T15:09:53.310 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-03-30T15:09:53.310 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:09:53.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.313 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.317 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.319 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.322 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:09:53.323 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:09:53.324 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.326 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:09:53.326 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:09:53.326 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.327 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.329 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.330 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.332 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.333 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:53.334 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:09:53.335 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.337 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.338 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.340 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.341 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:09:53.342 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-111", 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:08:12.612746+0000", 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 3, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 1, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.344 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "730", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "725", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "725", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "725", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 4, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 730, 2024-03-30T15:09:53.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [ 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: 332 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.346 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi028-93253-111", 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.347 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 8192, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {} 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 332, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-130", 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:09:49.642479+0000", 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:09:53.348 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:09:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "730", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:09:53.350 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 4, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 730, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": 322, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:09:53.351 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:09:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:09:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 0, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 660, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 669, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.355 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 10 librados: watch_flush enter 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 10 librados: watch_flush exit 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 20 client.50698.objecter shutdown clearing up homeless session... 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 10 client.50698.objecter successfully canceled tick 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7e98074900 legacy=0x7f7e98076dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 >> v1:172.21.15.134:6789/0 conn(0x7f7eb0109e30 legacy=0x7f7eb01a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 shutdown_connections 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 >> 172.21.15.28:0/2997651873 conn(0x7f7eb00761e0 msgr2=0x7f7eb0111190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 shutdown_connections 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 -- 172.21.15.28:0/2997651873 wait complete. 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:53.270+0000 7f7eb8428700 1 librados: shutdown 2024-03-30T15:09:53.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.358 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: "exists" 2024-03-30T15:09:53.359 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.360 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:09:53.361 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:09:53.362 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:09:53.363 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:09:53.366 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:09:53.367 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.368 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 716, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:09:53.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.372 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:53.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:09:53.375 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.377 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:09:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:09:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:09:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:09:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.378 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:09:53.379 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:09:53.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.381 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.383 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 728, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:09:53.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.393 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:27.042268+0000", 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 668 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.394 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:21.783823+0000", 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.395 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:03.355730+0000", 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.396 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:12.781159+0000", 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.398 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:50.311343+0000", 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:09:53.399 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:07.684740+0000", 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.401 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:09.629768+0000", 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:09:53.403 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:36.725884+0000", 2024-03-30T15:09:53.404 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:09:53.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.404 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.404 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-03-30T15:09:53.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.17", 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:09:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "42.c", 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-03-30T15:09:53.406 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:09:53.407 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:09:53.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.407 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.407 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:09:53.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:53.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.410 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:53.410 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:09:53.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:09:53.411 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:09:53.412 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.413 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.414 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [ 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-03-30T15:09:53.415 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:09:53.416 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:53.417 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:53.418 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:53.418 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.418 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:53.418 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:53.418 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:09:53.419 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:09:53.420 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: "testprofile-test-rados-api-smithi028-93253-111": { 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-device-class": "", 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-num-failure-domains": "0", 2024-03-30T15:09:53.421 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-osds-per-failure-domain": "0", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-root": "default", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:09:53.422 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van", 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: "w": "8" 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:09:53.425 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:09:53.433 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.433 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:09:53.433 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:09:53.434 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.438 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:09:53.440 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:09:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 325, 2024-03-30T15:09:53.446 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 326, 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 329, 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:09:53.448 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:09:53.449 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:09:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:53.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:53.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:53.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:53.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:53.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:53.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2997651873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:53 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:53 smithi134 ceph-mon[33714]: osdmap e731: 8 total, 7 up, 8 in 2024-03-30T15:09:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:53.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:53 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:53.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:53.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:53.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:53.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2997651873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[29876]: osdmap e731: 8 total, 7 up, 8 in 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2997651873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[41614]: osdmap e731: 8 total, 7 up, 8 in 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:53 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]: dispatch 2024-03-30T15:09:54.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:54.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:54.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:54.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:54.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:54.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:09:54.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:54.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:54.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:54.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:54.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:54.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:54.817 INFO:tasks.thrashosds.thrasher:j is {'epoch': 730, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:09:52.383464+0000', 'last_up_change': '2024-03-30T15:08:50.456069+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 117, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 332, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '562', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '559', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9000000953674316, 'score_stable': 2.9000000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 322, 'pool_name': 'test-rados-api-smithi028-93253-111', 'create_time': '2024-03-30T15:08:12.612746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '730', 'last_force_op_resend': '725', 'last_force_op_resend_prenautilus': '725', 'last_force_op_resend_preluminous': '725', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 730, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [332], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi028-93253-111', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 332, 'pool_name': 'test-rados-api-smithi028-93253-130', 'create_time': '2024-03-30T15:09:49.642479+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '730', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 730, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 322, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 660, 'down_at': 669, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 716, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:27.042268+0000', 'dead_epoch': 668}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:21.783823+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:03.355730+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:12.781159+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:50.311343+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:07.684740+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:09.629768+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:36.725884+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.17', 'osds': [6, 7, 1]}, {'pgid': '42.c', 'osds': [3, 7, 1]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.13', 'osds': [1, 3, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi028-93253-111': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 322, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 325, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 326, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 329, 'snaps': [{'begin': 4, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:09:54.817 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 42.c 2024-03-30T15:09:54.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap 42.c 2024-03-30T15:09:54.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:54.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:54.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: pgmap v1086: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]': finished 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: osdmap e732: 8 total, 7 up, 8 in 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: pgmap v1086: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]': finished 2024-03-30T15:09:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: osdmap e732: 8 total, 7 up, 8 in 2024-03-30T15:09:54.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.913 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:54 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: pgmap v1086: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-130", "mode": "writeback"}]': finished 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: osdmap e732: 8 total, 7 up, 8 in 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:54 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:09:54.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:55.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:55.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:55.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:55.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:55.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:55.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:55.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:55.682 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:55.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:55.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:55.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:55.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:55.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:56.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:56.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:56.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:56.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:56.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:56.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:56.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:56.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:56 smithi134 ceph-mon[33714]: osdmap e733: 8 total, 7 up, 8 in 2024-03-30T15:09:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:56 smithi134 ceph-mon[33714]: pgmap v1089: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:56.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[29876]: osdmap e733: 8 total, 7 up, 8 in 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[29876]: pgmap v1089: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[41614]: osdmap e733: 8 total, 7 up, 8 in 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]: dispatch 2024-03-30T15:09:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:56 smithi028 ceph-mon[41614]: pgmap v1089: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 63 active+undersized, 164 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:09:56.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:56.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:56.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:56.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:56.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:57.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:57.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:57.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:57.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:57.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:57.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:57.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:57.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:57.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:57.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:57.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:57.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:57.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: osdmap e734: 8 total, 7 up, 8 in 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[29876]: osdmap e735: 8 total, 7 up, 8 in 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:09:57.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:09:57.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:57.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:57.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: osdmap e734: 8 total, 7 up, 8 in 2024-03-30T15:09:57.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:09:57.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:57 smithi028 ceph-mon[41614]: osdmap e735: 8 total, 7 up, 8 in 2024-03-30T15:09:57.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:57.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:57.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-130"}]': finished 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: osdmap e734: 8 total, 7 up, 8 in 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:09:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:57 smithi134 ceph-mon[33714]: osdmap e735: 8 total, 7 up, 8 in 2024-03-30T15:09:58.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:58.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:58.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:58.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:58.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:58.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:58.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:58.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:58.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:58.662 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:09:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:09:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:58.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:58.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:58.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:58 smithi134 ceph-mon[33714]: pgmap v1092: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:58 smithi134 ceph-mon[33714]: osdmap e736: 8 total, 7 up, 8 in 2024-03-30T15:09:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:58 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:58.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[41614]: pgmap v1092: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:58.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[41614]: osdmap e736: 8 total, 7 up, 8 in 2024-03-30T15:09:58.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[29876]: pgmap v1092: 214 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 62/636 objects degraded (9.748%) 2024-03-30T15:09:58.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[29876]: osdmap e736: 8 total, 7 up, 8 in 2024-03-30T15:09:58.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:58 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:09:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:58.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:59.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:09:59.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:09:59.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:09:59.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.940+0000 7fb656764700 1 -- 172.21.15.28:0/1956477796 >> v1:172.21.15.28:6790/0 conn(0x7fb650101180 legacy=0x7fb650101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.940+0000 7fb656764700 1 -- 172.21.15.28:0/1956477796 shutdown_connections 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.940+0000 7fb656764700 1 -- 172.21.15.28:0/1956477796 >> 172.21.15.28:0/1956477796 conn(0x7fb6500fca30 msgr2=0x7fb6500fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.940+0000 7fb656764700 1 -- 172.21.15.28:0/1956477796 shutdown_connections 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.940+0000 7fb656764700 1 -- 172.21.15.28:0/1956477796 wait complete. 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 librados: starting msgr at 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 librados: starting objecter 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 Processor -- start 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 -- start start 2024-03-30T15:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 librados: setting wanted keys 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 librados: calling monclient init 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6501a7150 con 0x7fb650109e10 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6501a8370 con 0x7fb650101180 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb656764700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6501a95b0 con 0x7fb65010d9b0 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb654d01700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb65010d9b0 0x7fb6501a5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55694/0 (socket says 172.21.15.28:55694) 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb64f7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb650109e10 0x7fb6501a1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44998/0 (socket says 172.21.15.28:44998) 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.941+0000 7fb654d01700 1 -- 172.21.15.28:0/2319129139 learned_addr learned my addr 172.21.15.28:0/2319129139 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter ms_handle_connect 0x7fb650109e10 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter ms_handle_connect 0x7fb65010d9b0 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter ms_handle_connect 0x7fb650101180 2024-03-30T15:09:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071680278 0 0) 0x7fb6501a7150 con 0x7fb650109e10 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6380039c0 con 0x7fb650109e10 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336530016 0 0) 0x7fb6501a95b0 con 0x7fb65010d9b0 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6501a7150 con 0x7fb65010d9b0 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997723151 0 0) 0x7fb6501a8370 con 0x7fb650101180 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6501a95b0 con 0x7fb650101180 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2420267597 0 0) 0x7fb6380039c0 con 0x7fb650109e10 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6501a8370 con 0x7fb650109e10 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6440018b0 con 0x7fb650109e10 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1048495152 0 0) 0x7fb6501a7150 con 0x7fb65010d9b0 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6380039c0 con 0x7fb65010d9b0 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 959440495 0 0) 0x7fb6501a95b0 con 0x7fb650101180 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6501a7150 con 0x7fb650101180 2024-03-30T15:09:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb648001c10 con 0x7fb65010d9b0 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb640004380 con 0x7fb650101180 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2384351964 0 0) 0x7fb6501a8370 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 >> v1:172.21.15.28:6790/0 conn(0x7fb65010d9b0 legacy=0x7fb6501a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 >> v1:172.21.15.134:6789/0 conn(0x7fb650101180 legacy=0x7fb650106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6501aa7f0 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6501a97e0 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb656764700 10 client.25506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6501a9ca0 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb656764700 1 librados: init done 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb644001c60 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.942+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb644005d80 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.943+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63c005120 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.944+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb64401c640 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.944+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(736..736 src has 442..736) ==== 9218+0+0 (unknown 1245778938 0 0) 0x7fb644089b60 con 0x7fb650109e10 2024-03-30T15:09:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.944+0000 7fb64d7fa700 10 client.25506.objecter ms_dispatch 0x7fb650001550 osd_map(736..736 src has 442..736) 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.944+0000 7fb64d7fa700 3 client.25506.objecter handle_osd_map got epochs [736,736] > 0 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.944+0000 7fb64d7fa700 3 client.25506.objecter handle_osd_map decoding full epoch 736 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.945+0000 7fb64d7fa700 20 client.25506.objecter dump_active .. 0 homeless 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.945+0000 7fb64d7fa700 10 client.25506.objecter ms_handle_connect 0x7fb6380703f0 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:58.949+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb64405ec40 con 0x7fb650109e10 2024-03-30T15:09:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.161+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb63c005410 con 0x7fb650109e10 2024-03-30T15:09:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:09:59.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:59.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.390+0000 7fb64d7fa700 1 -- 172.21.15.28:0/2319129139 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v737) ==== 97+0+0 (unknown 4078982371 0 0) 0x7fb6440546f0 con 0x7fb650109e10 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 10 librados: watch_flush enter 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 10 librados: watch_flush exit 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 20 client.25506.objecter shutdown clearing up homeless session... 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 10 client.25506.objecter successfully canceled tick 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb6380703f0 legacy=0x7fb6380728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.394+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 >> v1:172.21.15.28:6789/0 conn(0x7fb650109e10 legacy=0x7fb6501a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.395+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 shutdown_connections 2024-03-30T15:09:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.395+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 >> 172.21.15.28:0/2319129139 conn(0x7fb6500fca30 msgr2=0x7fb65010de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:09:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.397+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 shutdown_connections 2024-03-30T15:09:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.397+0000 7fb656764700 1 -- 172.21.15.28:0/2319129139 wait complete. 2024-03-30T15:09:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:09:59.397+0000 7fb656764700 1 librados: shutdown 2024-03-30T15:09:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:09:59.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:09:59.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:09:59.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:09:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:09:59.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:09:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:09:59.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:09:59.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:09:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:09:59.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: osdmap e737: 8 total, 7 up, 8 in 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:09:59 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:09:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:09:59.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: osdmap e737: 8 total, 7 up, 8 in 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:59.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2319129139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: osdmap e737: 8 total, 7 up, 8 in 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:09:59.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:09:59 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:00.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:00.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:00.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:00.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:00.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:00.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:00.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:00.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:00.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:00.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:00.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:00.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:00.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized; 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T15:10:00.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: osd.0 (root=default,host=smithi028) is down 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: [WRN] PG_DEGRADED: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 1.0 is stuck undersized for 66s, current state active+undersized, last acting [7,6] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 1.1 is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 1.2 is stuck undersized for 66s, current state active+undersized, last acting [7,4] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 1.6 is stuck undersized for 66s, current state active+undersized, last acting [6,3] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 2.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,1] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 2.1 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.1 is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.3 is stuck undersized for 66s, current state active+undersized, last acting [4,6] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.b is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.f is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,4] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.10 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 3.12 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [5,7] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,2] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,5] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.b is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 4.16 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.2 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [6,5] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.5 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.7 is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized; 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: osd.0 (root=default,host=smithi028) is down 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: [WRN] PG_DEGRADED: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 1.0 is stuck undersized for 66s, current state active+undersized, last acting [7,6] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 1.1 is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 1.2 is stuck undersized for 66s, current state active+undersized, last acting [7,4] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 1.6 is stuck undersized for 66s, current state active+undersized, last acting [6,3] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 2.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,1] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 2.1 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.819 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.1 is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.3 is stuck undersized for 66s, current state active+undersized, last acting [4,6] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.b is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.f is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,4] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.10 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 3.12 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [5,7] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,2] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,5] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.b is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 4.16 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.2 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.820 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [6,5] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.5 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.7 is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.8 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.b is stuck undersized for 66s, current state active+undersized, last acting [2,5] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.c is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.e is stuck undersized for 66s, current state active+undersized+degraded, last acting [4,5] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 6.0 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 6.9 is stuck undersized for 66s, current state active+undersized, last acting [7,2] 2024-03-30T15:10:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 6.a is stuck undersized for 66s, current state active+undersized, last acting [5,6] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 6.d is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.3 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.6 is stuck undersized for 66s, current state active+undersized, last acting [1,7] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.8 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.a is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.c is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.e is stuck undersized for 66s, current state active+undersized, last acting [4,7] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.1c is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 12.1d is stuck undersized for 66s, current state active+undersized, last acting [2,6] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.2 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.3 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.8 is stuck undersized for 66s, current state active+undersized, last acting [7,5] 2024-03-30T15:10:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.e is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.10 is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.11 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.16 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 42.1c is stuck undersized for 66s, current state active+undersized, last acting [1,6] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 322.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [NONE,1,6] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pg 322.5 is stuck undersized for 66s, current state active+undersized, last acting [3,NONE,4] 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:10:00.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: application not enabled on pool 'test-rados-api-smithi028-93253-111' 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.8 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.b is stuck undersized for 66s, current state active+undersized, last acting [2,5] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.c is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.e is stuck undersized for 66s, current state active+undersized+degraded, last acting [4,5] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 6.0 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 6.9 is stuck undersized for 66s, current state active+undersized, last acting [7,2] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 6.a is stuck undersized for 66s, current state active+undersized, last acting [5,6] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 6.d is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.3 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.6 is stuck undersized for 66s, current state active+undersized, last acting [1,7] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.8 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.a is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.c is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.e is stuck undersized for 66s, current state active+undersized, last acting [4,7] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.1c is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 12.1d is stuck undersized for 66s, current state active+undersized, last acting [2,6] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.2 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.3 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.8 is stuck undersized for 66s, current state active+undersized, last acting [7,5] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.e is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.10 is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.11 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.16 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 42.1c is stuck undersized for 66s, current state active+undersized, last acting [1,6] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 322.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [NONE,1,6] 2024-03-30T15:10:00.825 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pg 322.5 is stuck undersized for 66s, current state active+undersized, last acting [3,NONE,4] 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: application not enabled on pool 'test-rados-api-smithi028-93253-111' 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: pgmap v1095: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: osdmap e738: 8 total, 7 up, 8 in 2024-03-30T15:10:00.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: pgmap v1095: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: osdmap e738: 8 total, 7 up, 8 in 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:00.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized; 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: [WRN] OSD_DOWN: 1 osds down 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: osd.0 (root=default,host=smithi028) is down 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: [WRN] PG_DEGRADED: Degraded data redundancy: 62/636 objects degraded (9.748%), 18 pgs degraded, 69 pgs undersized 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 1.0 is stuck undersized for 66s, current state active+undersized, last acting [7,6] 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 1.1 is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 1.2 is stuck undersized for 66s, current state active+undersized, last acting [7,4] 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 1.6 is stuck undersized for 66s, current state active+undersized, last acting [6,3] 2024-03-30T15:10:00.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 2.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,1] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 2.1 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.1 is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.3 is stuck undersized for 66s, current state active+undersized, last acting [4,6] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.b is stuck undersized for 66s, current state active+undersized, last acting [4,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.f is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,4] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.10 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 3.12 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.0 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [5,7] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.6 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,2] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.7 is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,5] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.b is stuck undersized for 66s, current state active+undersized+degraded, last acting [1,4] 2024-03-30T15:10:00.932 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 4.16 is stuck undersized for 66s, current state active+undersized+degraded, last acting [7,3] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.2 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [6,5] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.5 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.7 is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.8 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.b is stuck undersized for 66s, current state active+undersized, last acting [2,5] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.c is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.e is stuck undersized for 66s, current state active+undersized+degraded, last acting [4,5] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 5.15 is active+undersized+degraded, acting [5,1] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 6.0 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 6.9 is stuck undersized for 66s, current state active+undersized, last acting [7,2] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 6.a is stuck undersized for 66s, current state active+undersized, last acting [5,6] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 6.d is stuck undersized for 66s, current state active+undersized, last acting [5,1] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.3 is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.6 is stuck undersized for 66s, current state active+undersized, last acting [1,7] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.8 is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.a is stuck undersized for 66s, current state active+undersized, last acting [2,3] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.c is stuck undersized for 66s, current state active+undersized, last acting [5,2] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.e is stuck undersized for 66s, current state active+undersized, last acting [4,7] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.1c is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 12.1d is stuck undersized for 66s, current state active+undersized, last acting [2,6] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.2 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.3 is stuck undersized for 66s, current state active+undersized, last acting [6,5] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.8 is stuck undersized for 66s, current state active+undersized, last acting [7,5] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.e is stuck undersized for 66s, current state active+undersized, last acting [7,3] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.10 is stuck undersized for 66s, current state active+undersized, last acting [2,4] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.11 is stuck undersized for 66s, current state active+undersized, last acting [2,1] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.16 is stuck undersized for 66s, current state active+undersized, last acting [1,4] 2024-03-30T15:10:00.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 42.1c is stuck undersized for 66s, current state active+undersized, last acting [1,6] 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 322.3 is stuck undersized for 66s, current state active+undersized+degraded, last acting [NONE,1,6] 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pg 322.5 is stuck undersized for 66s, current state active+undersized, last acting [3,NONE,4] 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: application not enabled on pool 'ceph_test_rados_delete_pools_parallel.smithi028-93953' 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: application not enabled on pool 'test-rados-api-smithi028-93253-111' 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: pgmap v1095: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: osdmap e738: 8 total, 7 up, 8 in 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.935 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:00.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:00.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- 172.21.15.28:0/4018120250 >> v1:172.21.15.28:6789/0 conn(0x7f646010b030 legacy=0x7f646010d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- 172.21.15.28:0/4018120250 shutdown_connections 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- 172.21.15.28:0/4018120250 >> 172.21.15.28:0/4018120250 conn(0x7f64600fc9f0 msgr2=0x7f64600fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- 172.21.15.28:0/4018120250 shutdown_connections 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- 172.21.15.28:0/4018120250 wait complete. 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 librados: starting msgr at 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 librados: starting objecter 2024-03-30T15:10:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 Processor -- start 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 -- start start 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 librados: setting wanted keys 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.921+0000 7f646762f700 1 librados: calling monclient init 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f646762f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f646010ad80 con 0x7f6460103220 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f646762f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64601a8640 con 0x7f646010b030 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f646762f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64601a9820 con 0x7f646010eb40 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64653cb700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6460103220 0x7f6460107c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45006/0 (socket says 172.21.15.28:45006) 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64653cb700 1 -- 172.21.15.28:0/2308104447 learned_addr learned my addr 172.21.15.28:0/2308104447 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter ms_handle_connect 0x7f6460103220 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter ms_handle_connect 0x7f646010eb40 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter ms_handle_connect 0x7f646010b030 2024-03-30T15:10:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716377899 0 0) 0x7f646010ad80 con 0x7f6460103220 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f644c0031f0 con 0x7f6460103220 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223688207 0 0) 0x7f64601a9820 con 0x7f646010eb40 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f646010ad80 con 0x7f646010eb40 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106106847 0 0) 0x7f64601a8640 con 0x7f646010b030 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64601a9820 con 0x7f646010b030 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 575138350 0 0) 0x7f644c0031f0 con 0x7f6460103220 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64601a8640 con 0x7f6460103220 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f645c001930 con 0x7f6460103220 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2243340457 0 0) 0x7f646010ad80 con 0x7f646010eb40 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f644c0031f0 con 0x7f646010eb40 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6458001bd0 con 0x7f646010eb40 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4242294465 0 0) 0x7f64601a9820 con 0x7f646010b030 2024-03-30T15:10:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f646010ad80 con 0x7f646010b030 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6450001430 con 0x7f646010b030 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2592809334 0 0) 0x7f64601a8640 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 >> v1:172.21.15.28:6790/0 conn(0x7f646010eb40 legacy=0x7f64601a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 >> v1:172.21.15.134:6789/0 conn(0x7f646010b030 legacy=0x7f6460108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64601aaa40 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.922+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64601a9a50 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f646762f700 10 client.25512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64601a9f10 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f646762f700 1 librados: init done 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f645c001d00 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f645c005d90 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.923+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6444005120 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.924+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f645c003950 con 0x7f6460103220 2024-03-30T15:10:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(738..738 src has 442..738) ==== 9226+0+0 (unknown 1967116470 0 0) 0x7f645c089a50 con 0x7f6460103220 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 10 client.25512.objecter ms_dispatch 0x7f6460001550 osd_map(738..738 src has 442..738) 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 3 client.25512.objecter handle_osd_map got epochs [738,738] > 0 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 3 client.25512.objecter handle_osd_map decoding full epoch 738 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 20 client.25512.objecter dump_active .. 0 homeless 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.925+0000 7f64567fc700 10 client.25512.objecter ms_handle_connect 0x7f644c070480 2024-03-30T15:10:01.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:00.932+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f645c056830 con 0x7f6460103220 2024-03-30T15:10:01.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:01.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.155+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "42.c"} v 0) -- 0x7f6444005410 con 0x7f6460103220 2024-03-30T15:10:01.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:01.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:01.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.393+0000 7f64567fc700 1 -- 172.21.15.28:0/2308104447 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]=0 clear 42.c pg_upmap mapping v739) ==== 106+0+0 (unknown 757457930 0 0) 0x7f645c05a280 con 0x7f6460103220 2024-03-30T15:10:01.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 10 librados: watch_flush enter 2024-03-30T15:10:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 10 librados: watch_flush exit 2024-03-30T15:10:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 20 client.25512.objecter shutdown clearing up homeless session... 2024-03-30T15:10:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 10 client.25512.objecter successfully canceled tick 2024-03-30T15:10:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 >> v1:172.21.15.28:6800/3011431093 conn(0x7f644c070480 legacy=0x7f644c072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.397+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 >> v1:172.21.15.28:6789/0 conn(0x7f6460103220 legacy=0x7f6460107c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.399+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 shutdown_connections 2024-03-30T15:10:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.399+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 >> 172.21.15.28:0/2308104447 conn(0x7f64600fc9f0 msgr2=0x7f64600fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.400+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 shutdown_connections 2024-03-30T15:10:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.400+0000 7f646762f700 1 -- 172.21.15.28:0/2308104447 wait complete. 2024-03-30T15:10:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:01.400+0000 7f646762f700 1 librados: shutdown 2024-03-30T15:10:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:clear 42.c pg_upmap mapping 2024-03-30T15:10:01.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:01.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:01.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:01.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:01.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:01.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:01.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:01.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]: dispatch 2024-03-30T15:10:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]': finished 2024-03-30T15:10:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: osdmap e739: 8 total, 7 up, 8 in 2024-03-30T15:10:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]: dispatch 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]': finished 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: osdmap e739: 8 total, 7 up, 8 in 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]: dispatch 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2308104447' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "42.c"}]': finished 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: osdmap e739: 8 total, 7 up, 8 in 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]: dispatch 2024-03-30T15:10:01.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:01.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:01.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:02.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:02.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:02.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:02.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:02.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:10:02.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:02.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:02.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:02.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:02.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:02.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:02.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:02.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:02.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[29876]: pgmap v1098: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:02.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:02.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]': finished 2024-03-30T15:10:02.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[29876]: osdmap e740: 8 total, 7 up, 8 in 2024-03-30T15:10:02.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[41614]: pgmap v1098: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:02.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:02.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]': finished 2024-03-30T15:10:02.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:02 smithi028 ceph-mon[41614]: osdmap e740: 8 total, 7 up, 8 in 2024-03-30T15:10:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 -- 172.21.15.28:0/4002543885 >> v1:172.21.15.28:6790/0 conn(0x7f8228100d80 legacy=0x7f8228101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 -- 172.21.15.28:0/4002543885 shutdown_connections 2024-03-30T15:10:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 -- 172.21.15.28:0/4002543885 >> 172.21.15.28:0/4002543885 conn(0x7f82280fc9e0 msgr2=0x7f82280fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 -- 172.21.15.28:0/4002543885 shutdown_connections 2024-03-30T15:10:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 -- 172.21.15.28:0/4002543885 wait complete. 2024-03-30T15:10:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 librados: starting msgr at 2024-03-30T15:10:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.896+0000 7f822cedd700 1 librados: starting objecter 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 Processor -- start 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 -- start start 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 librados: setting wanted keys 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 librados: calling monclient init 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f822810f760 con 0x7f8228101960 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f82281aa840 con 0x7f8228110c80 2024-03-30T15:10:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822cedd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f82281aba20 con 0x7f8228100d80 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822659c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8228100d80 0x7f822810c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55710/0 (socket says 172.21.15.28:55710) 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f822659c700 1 -- 172.21.15.28:0/1080292685 learned_addr learned my addr 172.21.15.28:0/1080292685 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter ms_handle_connect 0x7f8228100d80 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter ms_handle_connect 0x7f8228101960 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter ms_handle_connect 0x7f8228110c80 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408499913 0 0) 0x7f822810f760 con 0x7f8228101960 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82100031f0 con 0x7f8228101960 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812827990 0 0) 0x7f82281aba20 con 0x7f8228100d80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f822810f760 con 0x7f8228100d80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588273130 0 0) 0x7f82281aa840 con 0x7f8228110c80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.897+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82281aba20 con 0x7f8228110c80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3094174841 0 0) 0x7f82100031f0 con 0x7f8228101960 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82281aa840 con 0x7f8228101960 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f821c001830 con 0x7f8228101960 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1677287588 0 0) 0x7f82281aba20 con 0x7f8228110c80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82100031f0 con 0x7f8228110c80 2024-03-30T15:10:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3099982287 0 0) 0x7f822810f760 con 0x7f8228100d80 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82281aba20 con 0x7f8228100d80 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f82200013f0 con 0x7f8228110c80 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8218004500 con 0x7f8228100d80 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1681069882 0 0) 0x7f82281aa840 con 0x7f8228101960 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 >> v1:172.21.15.28:6790/0 conn(0x7f8228100d80 legacy=0x7f822810c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 >> v1:172.21.15.134:6789/0 conn(0x7f8228110c80 legacy=0x7f82281a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82281acc00 con 0x7f8228101960 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f82281a9890 con 0x7f8228101960 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f822cedd700 10 client.25518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f82281a9d50 con 0x7f8228101960 2024-03-30T15:10:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f822cedd700 1 librados: init done 2024-03-30T15:10:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.898+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f821c003b20 con 0x7f8228101960 2024-03-30T15:10:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.899+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f821c004af0 con 0x7f8228101960 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.899+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8208005100 con 0x7f8228101960 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f821c005e10 con 0x7f8228101960 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(740..740 src has 442..740) ==== 9193+0+0 (unknown 1312814423 0 0) 0x7f821c088ab0 con 0x7f8228101960 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 osd_map(740..740 src has 442..740) 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 3 client.25518.objecter handle_osd_map got epochs [740,740] > 0 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 3 client.25518.objecter handle_osd_map decoding full epoch 740 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 20 client.25518.objecter dump_active .. 0 homeless 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v740) ==== 54+0+16 (unknown 1238699929 0 2420000192) 0x7f821c088ea0 con 0x7f8228101960 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.900+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f8210070490 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 10 client.25518.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 1 2024-03-30T15:10:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 10 client.25518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=741}) -- 0x7f82080053e0 con 0x7f8228101960 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8208000e30 con 0x7f8228101960 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=740) ==== 24+0+0 (unknown 1938033160 0 0) 0x7f821c054f60 con 0x7f8228101960 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f8224d99700 10 client.25518.objecter _check_command_map_dne tid 1 current 740 map_dne_bound 740 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f8224d99700 10 client.25518.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f8224d99700 15 client.25518.objecter _session_command_op_remove -1 1 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 10 client.25518.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 2 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 2 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 1 2 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 4 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 10 client.25518.objecter _send_command 2 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f8208002360 con 0x7f8208002790 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.901+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f8208002790 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.902+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8208002360 con 0x7f8208002790 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.902+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.902+0000 7f82177fe700 10 client.25518.objecter _finish_command 2 = system:0 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.902+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 4 2024-03-30T15:10:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.902+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 1 2 2024-03-30T15:10:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 10 client.25518.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 3 2024-03-30T15:10:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 3 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208001bb0 osd=1 3 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 1 3 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 4 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 10 client.25518.objecter _send_command 3 2024-03-30T15:10:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.920+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8208004cf0 con 0x7f8208002790 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8208004cf0 con 0x7f8208002790 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 10 client.25518.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208001bb0 osd=1 4 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 1 3 2024-03-30T15:10:02.923 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 10 client.25518.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 4 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 4 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 2 4 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 4 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 10 client.25518.objecter _send_command 4 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f820800cb30 con 0x7f8208009020 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.922+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f8208009020 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.923+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f820800cb30 con 0x7f8208009020 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.923+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 4: 0 ) 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.923+0000 7f82177fe700 10 client.25518.objecter _finish_command 4 = system:0 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.923+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 4 2024-03-30T15:10:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.923+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 2 4 2024-03-30T15:10:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:02 smithi134 ceph-mon[33714]: pgmap v1098: 246 pgs: 32 creating+peering, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 62/636 objects degraded (9.748%) 2024-03-30T15:10:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:02 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:02 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-132", "mode": "writeback"}]': finished 2024-03-30T15:10:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:02 smithi134 ceph-mon[33714]: osdmap e740: 8 total, 7 up, 8 in 2024-03-30T15:10:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 10 client.25518.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 5 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 5 2024-03-30T15:10:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208001970 osd=2 3 2024-03-30T15:10:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 2 5 2024-03-30T15:10:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 4 2024-03-30T15:10:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 10 client.25518.objecter _send_command 5 2024-03-30T15:10:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.944+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820800de90 con 0x7f8208009020 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f820800de90 con 0x7f8208009020 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 10 client.25518.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208001970 osd=2 4 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 2 5 2024-03-30T15:10:02.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 10 client.25518.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 6 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 6 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 3 6 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 4 2024-03-30T15:10:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 10 client.25518.objecter _send_command 6 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f8208003f20 con 0x7f8208003360 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.946+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f8208003360 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.947+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8208003f20 con 0x7f8208003360 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.947+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 6: 0 ) 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.947+0000 7f82177fe700 10 client.25518.objecter _finish_command 6 = system:0 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.947+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 4 2024-03-30T15:10:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.947+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 3 6 2024-03-30T15:10:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 10 client.25518.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 7 2024-03-30T15:10:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 7 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800e290 osd=3 3 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 3 7 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 4 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 10 client.25518.objecter _send_command 7 2024-03-30T15:10:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.965+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8208004920 con 0x7f8208003360 2024-03-30T15:10:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8208004920 con 0x7f8208003360 2024-03-30T15:10:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f82177fe700 10 client.25518.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800e290 osd=3 4 2024-03-30T15:10:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 3 7 2024-03-30T15:10:02.969 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 10 client.25518.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 8 2024-03-30T15:10:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 8 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 4 8 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 4 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 10 client.25518.objecter _send_command 8 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.967+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f8208016780 con 0x7f8208012c00 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.968+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f8208012c00 2024-03-30T15:10:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.969+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8208016780 con 0x7f8208012c00 2024-03-30T15:10:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.969+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 8: 0 ) 2024-03-30T15:10:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.969+0000 7f82177fe700 10 client.25518.objecter _finish_command 8 = system:0 2024-03-30T15:10:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.969+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 4 2024-03-30T15:10:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.969+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 4 8 2024-03-30T15:10:02.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 10 client.25518.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 9 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 9 2024-03-30T15:10:02.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800ee10 osd=4 3 2024-03-30T15:10:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 4 9 2024-03-30T15:10:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 4 2024-03-30T15:10:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 10 client.25518.objecter _send_command 9 2024-03-30T15:10:02.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.986+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f82080043a0 con 0x7f8208012c00 2024-03-30T15:10:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f82080043a0 con 0x7f8208012c00 2024-03-30T15:10:02.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f82177fe700 10 client.25518.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800ee10 osd=4 4 2024-03-30T15:10:02.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 4 9 2024-03-30T15:10:02.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 10 client.25518.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 10 2024-03-30T15:10:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 10 2024-03-30T15:10:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 5 10 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 4 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 10 client.25518.objecter _send_command 10 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.988+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f82080042c0 con 0x7f820800d9f0 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.989+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f820800d9f0 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.989+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f82080042c0 con 0x7f820800d9f0 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.989+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 10: 0 ) 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.989+0000 7f82177fe700 10 client.25518.objecter _finish_command 10 = system:0 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.990+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 4 2024-03-30T15:10:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:02.990+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 5 10 2024-03-30T15:10:03.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 10 client.25518.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 11 2024-03-30T15:10:03.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 11 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f8208004150 osd=5 3 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 5 11 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 4 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 10 client.25518.objecter _send_command 11 2024-03-30T15:10:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.006+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f82080042c0 con 0x7f820800d9f0 2024-03-30T15:10:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f82080042c0 con 0x7f820800d9f0 2024-03-30T15:10:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f82177fe700 10 client.25518.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f8208004150 osd=5 4 2024-03-30T15:10:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 5 11 2024-03-30T15:10:03.010 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 10 client.25518.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 12 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 12 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 6 12 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 4 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 10 client.25518.objecter _send_command 12 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.008+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f820801f500 con 0x7f820801b840 2024-03-30T15:10:03.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.009+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f820801b840 2024-03-30T15:10:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.010+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f820801f500 con 0x7f820801b840 2024-03-30T15:10:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.010+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 12: 0 ) 2024-03-30T15:10:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.010+0000 7f82177fe700 10 client.25518.objecter _finish_command 12 = system:0 2024-03-30T15:10:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.010+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 4 2024-03-30T15:10:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.010+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 6 12 2024-03-30T15:10:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 10 client.25518.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 13 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 13 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f82080042c0 osd=6 3 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 6 13 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 4 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 10 client.25518.objecter _send_command 13 2024-03-30T15:10:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.026+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820800cf60 con 0x7f820801b840 2024-03-30T15:10:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f820800cf60 con 0x7f820801b840 2024-03-30T15:10:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f82177fe700 10 client.25518.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f82080042c0 osd=6 4 2024-03-30T15:10:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 6 13 2024-03-30T15:10:03.030 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 10 client.25518.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 14 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 14 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 7 14 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 4 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 10 client.25518.objecter _send_command 14 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.028+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f820800ce80 con 0x7f82080188d0 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.029+0000 7f82177fe700 10 client.25518.objecter ms_handle_connect 0x7f82080188d0 2024-03-30T15:10:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.030+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f820800ce80 con 0x7f82080188d0 2024-03-30T15:10:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.030+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 14: 0 ) 2024-03-30T15:10:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.030+0000 7f82177fe700 10 client.25518.objecter _finish_command 14 = system:0 2024-03-30T15:10:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.030+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 4 2024-03-30T15:10:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.030+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 7 14 2024-03-30T15:10:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 10 client.25518.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign -1 15 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 20 client.25518.objecter _get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 15 client.25518.objecter _session_command_op_remove -1 15 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 20 client.25518.objecter get_session s=0x7f820800cd10 osd=7 3 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 15 client.25518.objecter _session_command_op_assign 7 15 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 4 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 10 client.25518.objecter _send_command 15 2024-03-30T15:10:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.046+0000 7f822cedd700 1 -- 172.21.15.28:0/1080292685 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820800ce80 con 0x7f82080188d0 2024-03-30T15:10:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.048+0000 7f82177fe700 1 -- 172.21.15.28:0/1080292685 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f820800ce80 con 0x7f82080188d0 2024-03-30T15:10:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.048+0000 7f82177fe700 10 client.25518.objecter ms_dispatch 0x7f8228001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.048+0000 7f82177fe700 10 client.25518.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.048+0000 7f82177fe700 20 client.25518.objecter put_session s=0x7f820800cd10 osd=7 4 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:03.048+0000 7f82177fe700 15 client.25518.objecter _session_command_op_remove 7 15 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:10:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:03.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:03.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:03.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:03.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:03.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:03.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:03.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:03.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:03.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:03.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:03.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:03.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:03.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:03.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1080292685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:03 smithi134 ceph-mon[33714]: osdmap e741: 8 total, 7 up, 8 in 2024-03-30T15:10:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1080292685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:03.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:03 smithi028 ceph-mon[29876]: osdmap e741: 8 total, 7 up, 8 in 2024-03-30T15:10:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1080292685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:03.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:03 smithi028 ceph-mon[41614]: osdmap e741: 8 total, 7 up, 8 in 2024-03-30T15:10:03.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:04.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:04.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:04.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:04.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:04.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:04.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:04.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:04.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:04.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:04.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:04.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:04.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:04.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:04.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[41614]: pgmap v1101: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[41614]: osdmap e742: 8 total, 7 up, 8 in 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[29876]: pgmap v1101: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[29876]: osdmap e742: 8 total, 7 up, 8 in 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:04.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:04 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:04.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:05.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:05.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:05.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:04 smithi134 ceph-mon[33714]: pgmap v1101: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:04 smithi134 ceph-mon[33714]: osdmap e742: 8 total, 7 up, 8 in 2024-03-30T15:10:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:04 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:05.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:05.290 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:05.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:05.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:05.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:05.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:05.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 -- 172.21.15.28:0/863709994 >> v1:172.21.15.28:6790/0 conn(0x7fbd9010af10 legacy=0x7fbd9010d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 -- 172.21.15.28:0/863709994 shutdown_connections 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 -- 172.21.15.28:0/863709994 >> 172.21.15.28:0/863709994 conn(0x7fbd900761c0 msgr2=0x7fbd900765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 -- 172.21.15.28:0/863709994 shutdown_connections 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 -- 172.21.15.28:0/863709994 wait complete. 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 librados: starting msgr at 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 librados: starting objecter 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.676+0000 7fbd95cc4700 1 Processor -- start 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 -- start start 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 librados: setting wanted keys 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 librados: calling monclient init 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd901a6f10 con 0x7fbd9010ea80 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd901a8130 con 0x7fbd9010af10 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd95cc4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd901a9370 con 0x7fbd90100a90 2024-03-30T15:10:05.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8f7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbd90100a90 0x7fbd90107450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55734/0 (socket says 172.21.15.28:55734) 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8f7fe700 1 -- 172.21.15.28:0/3267210847 learned_addr learned my addr 172.21.15.28:0/3267210847 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter ms_handle_connect 0x7fbd90100a90 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter ms_handle_connect 0x7fbd9010ea80 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter ms_handle_connect 0x7fbd9010af10 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142389021 0 0) 0x7fbd901a6f10 con 0x7fbd9010ea80 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd780031f0 con 0x7fbd9010ea80 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2964766198 0 0) 0x7fbd901a9370 con 0x7fbd90100a90 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd901a6f10 con 0x7fbd90100a90 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018648215 0 0) 0x7fbd901a8130 con 0x7fbd9010af10 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd901a9370 con 0x7fbd9010af10 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2760706030 0 0) 0x7fbd780031f0 con 0x7fbd9010ea80 2024-03-30T15:10:05.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd901a8130 con 0x7fbd9010ea80 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.677+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbd88001b70 con 0x7fbd9010ea80 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 552797344 0 0) 0x7fbd901a6f10 con 0x7fbd90100a90 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd780031f0 con 0x7fbd90100a90 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbd800019b0 con 0x7fbd90100a90 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4017930791 0 0) 0x7fbd901a9370 con 0x7fbd9010af10 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd901a6f10 con 0x7fbd9010af10 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbd84004540 con 0x7fbd9010af10 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2217680440 0 0) 0x7fbd901a8130 con 0x7fbd9010ea80 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 >> v1:172.21.15.28:6790/0 conn(0x7fbd90100a90 legacy=0x7fbd90107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 >> v1:172.21.15.134:6789/0 conn(0x7fbd9010af10 legacy=0x7fbd901a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901aa5b0 con 0x7fbd9010ea80 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901a70e0 con 0x7fbd9010ea80 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd95cc4700 10 client.25524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:05.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd901a7620 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd95cc4700 1 librados: init done 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbd88003ea0 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.678+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbd88004d50 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.679+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd7c005120 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.679+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbd88003a00 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(743..743 src has 442..743) ==== 9193+0+0 (unknown 1519380292 0 0) 0x7fbd88088b90 con 0x7fbd9010ea80 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 10 client.25524.objecter ms_dispatch 0x7fbd90001550 osd_map(743..743 src has 442..743) 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 3 client.25524.objecter handle_osd_map got epochs [743,743] > 0 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 3 client.25524.objecter handle_osd_map decoding full epoch 743 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 20 client.25524.objecter dump_active .. 0 homeless 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.680+0000 7fbd8cff9700 10 client.25524.objecter ms_handle_connect 0x7fbd780704e0 2024-03-30T15:10:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.684+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbd88055970 con 0x7fbd9010ea80 2024-03-30T15:10:05.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 64/642 objects degraded (9.969%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[29876]: osdmap e743: 8 total, 7 up, 8 in 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 64/642 objects degraded (9.969%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[41614]: osdmap e743: 8 total, 7 up, 8 in 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:05.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:05 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:05.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:05.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:05.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:05.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:05.904+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fbd7c005410 con 0x7fbd9010ea80 2024-03-30T15:10:05.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:05.967 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:05.967 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:06.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:06.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:06.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:10:06.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:06.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:05 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 64/642 objects degraded (9.969%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:05 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:05 smithi134 ceph-mon[33714]: osdmap e743: 8 total, 7 up, 8 in 2024-03-30T15:10:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:05 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:06.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:06.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.400+0000 7fbd8cff9700 1 -- 172.21.15.28:0/3267210847 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v744) ==== 91+0+0 (unknown 3282128464 0 0) 0x7fbd880593c0 con 0x7fbd9010ea80 2024-03-30T15:10:06.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 10 librados: watch_flush enter 2024-03-30T15:10:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 10 librados: watch_flush exit 2024-03-30T15:10:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 20 client.25524.objecter shutdown clearing up homeless session... 2024-03-30T15:10:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 10 client.25524.objecter successfully canceled tick 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbd780704e0 legacy=0x7fbd780729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.404+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 >> v1:172.21.15.28:6789/0 conn(0x7fbd9010ea80 legacy=0x7fbd901a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.405+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 shutdown_connections 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.405+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 >> 172.21.15.28:0/3267210847 conn(0x7fbd900761c0 msgr2=0x7fbd90111fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.405+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 shutdown_connections 2024-03-30T15:10:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.405+0000 7fbd95cc4700 1 -- 172.21.15.28:0/3267210847 wait complete. 2024-03-30T15:10:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:06.405+0000 7fbd95cc4700 1 librados: shutdown 2024-03-30T15:10:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:10:06.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:06.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:06.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:06.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:06.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:06.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:10:06.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:06.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:06.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:06.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:06.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: pgmap v1104: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[41614]: osdmap e744: 8 total, 7 up, 8 in 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: pgmap v1104: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:06.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:06 smithi028 ceph-mon[29876]: osdmap e744: 8 total, 7 up, 8 in 2024-03-30T15:10:07.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:07.079 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: pgmap v1104: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 64/642 objects degraded (9.969%) 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3267210847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:06 smithi134 ceph-mon[33714]: osdmap e744: 8 total, 7 up, 8 in 2024-03-30T15:10:07.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:07.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:07.334 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:07.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:07.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:07.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:07.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:07.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:10:07.653 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:10:07.653 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-30T15:10:07.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:10:07.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:07.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: osdmap e745: 8 total, 7 up, 8 in 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: osdmap e745: 8 total, 7 up, 8 in 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:07.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:07.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:07.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:07.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:07.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:07.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:07.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:08.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:10:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:10:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: osdmap e745: 8 total, 7 up, 8 in 2024-03-30T15:10:08.081 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:08.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:08.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:08.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:08.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:08.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:08.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:08.416 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:08.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:08.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:08.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:08.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.728 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:08.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:08.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:08.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 42.7 scrub starts 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 42.7 scrub ok 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: pgmap v1107: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 64/642 objects degraded (9.969%) 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: osdmap e746: 8 total, 7 up, 8 in 2024-03-30T15:10:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 42.7 scrub starts 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 42.7 scrub ok 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: pgmap v1107: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 64/642 objects degraded (9.969%) 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:08.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: osdmap e746: 8 total, 7 up, 8 in 2024-03-30T15:10:08.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:08.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:08 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:08.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:09.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:09.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 42.7 scrub starts 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 42.7 scrub ok 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: pgmap v1107: 246 pgs: 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 161 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 64/642 objects degraded (9.969%) 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:10:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:10:09.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:09.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: osdmap e746: 8 total, 7 up, 8 in 2024-03-30T15:10:09.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:09.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:08 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]: dispatch 2024-03-30T15:10:09.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:09.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:09.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:09.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:09.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:09.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:10:09.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:09.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:09.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:09.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:09.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[29876]: osdmap e747: 8 total, 7 up, 8 in 2024-03-30T15:10:09.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:09.955 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:09 smithi028 ceph-mon[41614]: osdmap e747: 8 total, 7 up, 8 in 2024-03-30T15:10:09.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:09.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:10.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:10.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-132"}]': finished 2024-03-30T15:10:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:09 smithi134 ceph-mon[33714]: osdmap e747: 8 total, 7 up, 8 in 2024-03-30T15:10:10.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:10.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:10.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:10.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:10.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:10.556 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:10.556 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:10.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:10.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:10.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:10 smithi028 ceph-mon[41614]: pgmap v1110: 246 pgs: 1 active+clean+snaptrim, 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 160 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s; 65/651 objects degraded (9.985%) 2024-03-30T15:10:10.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:10 smithi028 ceph-mon[41614]: osdmap e748: 8 total, 7 up, 8 in 2024-03-30T15:10:10.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:10 smithi028 ceph-mon[29876]: pgmap v1110: 246 pgs: 1 active+clean+snaptrim, 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 160 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s; 65/651 objects degraded (9.985%) 2024-03-30T15:10:10.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:10 smithi028 ceph-mon[29876]: osdmap e748: 8 total, 7 up, 8 in 2024-03-30T15:10:10.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:10.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:11.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:11.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:10 smithi134 ceph-mon[33714]: pgmap v1110: 246 pgs: 1 active+clean+snaptrim, 1 active+clean+remapped, 19 active+undersized+degraded, 65 active+undersized, 160 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s; 65/651 objects degraded (9.985%) 2024-03-30T15:10:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:10 smithi134 ceph-mon[33714]: osdmap e748: 8 total, 7 up, 8 in 2024-03-30T15:10:11.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:11.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:10:11.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:11.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.535+0000 7f6adba8d700 1 -- 172.21.15.28:0/1670204971 >> v1:172.21.15.28:6789/0 conn(0x7f6ad410eb40 legacy=0x7f6ad4110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 -- 172.21.15.28:0/1670204971 shutdown_connections 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 -- 172.21.15.28:0/1670204971 >> 172.21.15.28:0/1670204971 conn(0x7f6ad40fc9d0 msgr2=0x7f6ad40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 -- 172.21.15.28:0/1670204971 shutdown_connections 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 -- 172.21.15.28:0/1670204971 wait complete. 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 librados: starting msgr at 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 librados: starting objecter 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 Processor -- start 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 -- start start 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 librados: setting wanted keys 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.536+0000 7f6adba8d700 1 librados: calling monclient init 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6adba8d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ad41a7100 con 0x7f6ad410b030 2024-03-30T15:10:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6adba8d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ad41a8320 con 0x7f6ad4103220 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6adba8d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6ad41a9560 con 0x7f6ad410eb40 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6ad9028700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6ad410b030 0x7f6ad41a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52926/0 (socket says 172.21.15.28:52926) 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6ad9028700 1 -- 172.21.15.28:0/3721568727 learned_addr learned my addr 172.21.15.28:0/3721568727 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6ad9829700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f6ad4103220 0x7f6ad410a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:49866/0 (socket says 172.21.15.28:49866) 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter ms_handle_connect 0x7f6ad410b030 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter ms_handle_connect 0x7f6ad4103220 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter ms_handle_connect 0x7f6ad410eb40 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036388012 0 0) 0x7f6ad41a7100 con 0x7f6ad410b030 2024-03-30T15:10:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac0003aa0 con 0x7f6ad410b030 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217935427 0 0) 0x7f6ad41a9560 con 0x7f6ad410eb40 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ad41a7100 con 0x7f6ad410eb40 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974522406 0 0) 0x7f6ad41a8320 con 0x7f6ad4103220 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ad41a9560 con 0x7f6ad4103220 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3438872841 0 0) 0x7f6ad41a7100 con 0x7f6ad410eb40 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ad41a8320 con 0x7f6ad410eb40 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 876100257 0 0) 0x7f6ac0003aa0 con 0x7f6ad410b030 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.537+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ad41a7100 con 0x7f6ad410b030 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6acc004730 con 0x7f6ad410eb40 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999054829 0 0) 0x7f6ad41a9560 con 0x7f6ad4103220 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ac0003aa0 con 0x7f6ad4103220 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6ac4001230 con 0x7f6ad410b030 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6ad0001830 con 0x7f6ad4103220 2024-03-30T15:10:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 292138204 0 0) 0x7f6ad41a8320 con 0x7f6ad410eb40 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 >> v1:172.21.15.134:6789/0 conn(0x7f6ad4103220 legacy=0x7f6ad410a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 >> v1:172.21.15.28:6789/0 conn(0x7f6ad410b030 legacy=0x7f6ad41a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad41aa7a0 con 0x7f6ad410eb40 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad41a9790 con 0x7f6ad410eb40 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6adba8d700 10 client.58730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ad41a9c50 con 0x7f6ad410eb40 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6adba8d700 1 librados: init done 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6acc001e30 con 0x7f6ad410eb40 2024-03-30T15:10:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.538+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6acc004ff0 con 0x7f6ad410eb40 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.539+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6ab8005130 con 0x7f6ad410eb40 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6acc01b830 con 0x7f6ad410eb40 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(749..749 src has 442..749) ==== 9229+0+0 (unknown 317432794 0 0) 0x7f6acc088f40 con 0x7f6ad410eb40 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 osd_map(749..749 src has 442..749) 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 3 client.58730.objecter handle_osd_map got epochs [749,749] > 0 2024-03-30T15:10:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 3 client.58730.objecter handle_osd_map decoding full epoch 749 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.540+0000 7f6acaffd700 20 client.58730.objecter dump_active .. 0 homeless 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v749) ==== 54+0+16 (unknown 341118072 0 2420000192) 0x7f6acc08b3a0 con 0x7f6ad410eb40 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ac0078cd0 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 10 client.58730.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 1 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 10 client.58730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=750}) -- 0x7f6ab8005410 con 0x7f6ad410eb40 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6ab8000e60 con 0x7f6ad410eb40 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=749) ==== 24+0+0 (unknown 141371601 0 0) 0x7f6acc0553f0 con 0x7f6ad410eb40 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acbfff700 10 client.58730.objecter _check_command_map_dne tid 1 current 749 map_dne_bound 749 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acbfff700 10 client.58730.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acbfff700 15 client.58730.objecter _session_command_op_remove -1 1 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 10 client.58730.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 2 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 2 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 1 2 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 4 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 10 client.58730.objecter _send_command 2 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f6ab8002320 con 0x7f6ab80027c0 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.541+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab80027c0 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.542+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab8002320 con 0x7f6ab80027c0 2024-03-30T15:10:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.542+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.542+0000 7f6acaffd700 10 client.58730.objecter _finish_command 2 = system:0 2024-03-30T15:10:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.542+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 4 2024-03-30T15:10:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.542+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 1 2 2024-03-30T15:10:11.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 10 client.58730.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8001be0 osd=1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 1 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 4 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 10 client.58730.objecter _send_command 3 2024-03-30T15:10:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.565+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8004ca0 con 0x7f6ab80027c0 2024-03-30T15:10:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8004ca0 con 0x7f6ab80027c0 2024-03-30T15:10:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6acaffd700 10 client.58730.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8001be0 osd=1 4 2024-03-30T15:10:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 1 3 2024-03-30T15:10:11.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 10 client.58730.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 4 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 4 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 2 4 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 4 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 10 client.58730.objecter _send_command 4 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.567+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f6ab800cb60 con 0x7f6ab8009050 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab8009050 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab800cb60 con 0x7f6ab8009050 2024-03-30T15:10:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 4: 0 ) 2024-03-30T15:10:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 10 client.58730.objecter _finish_command 4 = system:0 2024-03-30T15:10:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 4 2024-03-30T15:10:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.568+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 2 4 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 10 client.58730.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 5 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 5 2024-03-30T15:10:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8004f30 osd=2 3 2024-03-30T15:10:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 2 5 2024-03-30T15:10:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 4 2024-03-30T15:10:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 10 client.58730.objecter _send_command 5 2024-03-30T15:10:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.586+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8003e60 con 0x7f6ab8009050 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8003e60 con 0x7f6ab8009050 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 10 client.58730.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8004f30 osd=2 4 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 2 5 2024-03-30T15:10:11.589 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 10 client.58730.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 6 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 6 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 3 6 2024-03-30T15:10:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 4 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 10 client.58730.objecter _send_command 6 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f6ab80119a0 con 0x7f6ab800de90 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.588+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab800de90 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.589+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab80119a0 con 0x7f6ab800de90 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.589+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 6: 0 ) 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.589+0000 7f6acaffd700 10 client.58730.objecter _finish_command 6 = system:0 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.589+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 4 2024-03-30T15:10:11.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.589+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 3 6 2024-03-30T15:10:11.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 10 client.58730.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 7 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 7 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab80048c0 osd=3 3 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 3 7 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 4 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 10 client.58730.objecter _send_command 7 2024-03-30T15:10:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.611+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8004460 con 0x7f6ab800de90 2024-03-30T15:10:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.614+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8004460 con 0x7f6ab800de90 2024-03-30T15:10:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.614+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.614+0000 7f6acaffd700 10 client.58730.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.614+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab80048c0 osd=3 4 2024-03-30T15:10:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.614+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 3 7 2024-03-30T15:10:11.616 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 10 client.58730.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 8 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 8 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 4 8 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 4 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 10 client.58730.objecter _send_command 8 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f6ab80126a0 con 0x7f6ab8011b50 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.615+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab8011b50 2024-03-30T15:10:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.616+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab80126a0 con 0x7f6ab8011b50 2024-03-30T15:10:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.616+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 8: 0 ) 2024-03-30T15:10:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.616+0000 7f6acaffd700 10 client.58730.objecter _finish_command 8 = system:0 2024-03-30T15:10:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.616+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 4 2024-03-30T15:10:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.616+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 4 8 2024-03-30T15:10:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 10 client.58730.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 9 2024-03-30T15:10:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 9 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8004700 osd=4 3 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 4 9 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 4 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 10 client.58730.objecter _send_command 9 2024-03-30T15:10:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.639+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8012e70 con 0x7f6ab8011b50 2024-03-30T15:10:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8012e70 con 0x7f6ab8011b50 2024-03-30T15:10:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6acaffd700 10 client.58730.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8004700 osd=4 4 2024-03-30T15:10:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 4 9 2024-03-30T15:10:11.643 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:11.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 10 client.58730.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 10 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 10 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 5 10 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 4 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 10 client.58730.objecter _send_command 10 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.641+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f6ab801b150 con 0x7f6ab800d270 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.642+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab800d270 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.643+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab801b150 con 0x7f6ab800d270 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.643+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 10: 0 ) 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.643+0000 7f6acaffd700 10 client.58730.objecter _finish_command 10 = system:0 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.643+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 4 2024-03-30T15:10:11.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.643+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 5 10 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 10 client.58730.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 11 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 11 2024-03-30T15:10:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8013ff0 osd=5 3 2024-03-30T15:10:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 5 11 2024-03-30T15:10:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 4 2024-03-30T15:10:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 10 client.58730.objecter _send_command 11 2024-03-30T15:10:11.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.661+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab801b150 con 0x7f6ab800d270 2024-03-30T15:10:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab801b150 con 0x7f6ab800d270 2024-03-30T15:10:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6acaffd700 10 client.58730.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8013ff0 osd=5 4 2024-03-30T15:10:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 5 11 2024-03-30T15:10:11.665 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 10 client.58730.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 12 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 12 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 6 12 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 4 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 10 client.58730.objecter _send_command 12 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.663+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f6ab801f520 con 0x7f6ab801b7d0 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.664+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab801b7d0 2024-03-30T15:10:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.665+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab801f520 con 0x7f6ab801b7d0 2024-03-30T15:10:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.665+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 12: 0 ) 2024-03-30T15:10:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.665+0000 7f6acaffd700 10 client.58730.objecter _finish_command 12 = system:0 2024-03-30T15:10:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.665+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 4 2024-03-30T15:10:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.665+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 6 12 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 10 client.58730.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 13 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 13 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab801b150 osd=6 3 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 6 13 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 4 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 10 client.58730.objecter _send_command 13 2024-03-30T15:10:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.681+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8020880 con 0x7f6ab801b7d0 2024-03-30T15:10:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.683+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8020880 con 0x7f6ab801b7d0 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.683+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.683+0000 7f6acaffd700 10 client.58730.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.683+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab801b150 osd=6 4 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.683+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 6 13 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 10 client.58730.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 14 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 14 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 7 14 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 4 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 10 client.58730.objecter _send_command 14 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f6ab8013e30 con 0x7f6ab8020a90 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.684+0000 7f6acaffd700 10 client.58730.objecter ms_handle_connect 0x7f6ab8020a90 2024-03-30T15:10:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.685+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ab8000e60 con 0x7f6ab8020a90 2024-03-30T15:10:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.685+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 14: 0 ) 2024-03-30T15:10:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.685+0000 7f6acaffd700 10 client.58730.objecter _finish_command 14 = system:0 2024-03-30T15:10:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.685+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 4 2024-03-30T15:10:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.685+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 7 14 2024-03-30T15:10:11.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 10 client.58730.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign -1 15 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 20 client.58730.objecter _get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_remove -1 15 2024-03-30T15:10:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 20 client.58730.objecter get_session s=0x7f6ab8020880 osd=7 3 2024-03-30T15:10:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 15 client.58730.objecter _session_command_op_assign 7 15 2024-03-30T15:10:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 4 2024-03-30T15:10:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 10 client.58730.objecter _send_command 15 2024-03-30T15:10:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.702+0000 7f6adba8d700 1 -- 172.21.15.28:0/3721568727 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6ab8000e60 con 0x7f6ab8020a90 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.704+0000 7f6acaffd700 1 -- 172.21.15.28:0/3721568727 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6ab8000e60 con 0x7f6ab8020a90 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.704+0000 7f6acaffd700 10 client.58730.objecter ms_dispatch 0x7f6ad4001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.704+0000 7f6acaffd700 10 client.58730.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.704+0000 7f6acaffd700 20 client.58730.objecter put_session s=0x7f6ab8020880 osd=7 4 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:11.704+0000 7f6acaffd700 15 client.58730.objecter _session_command_op_remove 7 15 2024-03-30T15:10:11.705 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:11.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:11.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:11.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 65/651 objects degraded (9.985%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:11.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:11.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:11.837 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 42.12 scrub starts 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 42.12 scrub ok 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 42.1b scrub starts 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 42.1b scrub ok 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: osdmap e749: 8 total, 7 up, 8 in 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:11.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3721568727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 65/651 objects degraded (9.985%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 42.12 scrub starts 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 42.12 scrub ok 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 42.1b scrub starts 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 42.1b scrub ok 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: osdmap e749: 8 total, 7 up, 8 in 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:11.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:11.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3721568727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:11.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:11.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:11.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:11.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:12.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 65/651 objects degraded (9.985%), 19 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 42.12 scrub starts 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 42.12 scrub ok 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 42.1b scrub starts 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 42.1b scrub ok 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: osdmap e749: 8 total, 7 up, 8 in 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3721568727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:12.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:12.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:12.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:12.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:12.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:12.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:12.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:12.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:12.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:12.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:12.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: 42.0 scrub starts 2024-03-30T15:10:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: 42.0 scrub ok 2024-03-30T15:10:12.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: pgmap v1113: 246 pgs: 1 active+clean+snaptrim, 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 143 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: osdmap e750: 8 total, 7 up, 8 in 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: 42.0 scrub starts 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: 42.0 scrub ok 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: pgmap v1113: 246 pgs: 1 active+clean+snaptrim, 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 143 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:10:12.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:10:12.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:12.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: osdmap e750: 8 total, 7 up, 8 in 2024-03-30T15:10:12.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:12.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:12 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:12.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:12.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:12.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:13.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: 42.0 scrub starts 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: 42.0 scrub ok 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: pgmap v1113: 246 pgs: 1 active+clean+snaptrim, 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 143 active+clean; 582 KiB data, 711 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:10:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: osdmap e750: 8 total, 7 up, 8 in 2024-03-30T15:10:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:12 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:13.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:13.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:13.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:13.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:13.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:13.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:13.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:13.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:13.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:13.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:13.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:13.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:14.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:14.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:14.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:14.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:14.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:14.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:14.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:14.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:14.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: osdmap e751: 8 total, 7 up, 8 in 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: 42.1d scrub starts 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: 42.1d scrub ok 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:10:14.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:14 smithi134 ceph-mon[33714]: pgmap v1116: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: osdmap e751: 8 total, 7 up, 8 in 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: 42.1d scrub starts 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: 42.1d scrub ok 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[29876]: pgmap v1116: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: osdmap e751: 8 total, 7 up, 8 in 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: 42.1d scrub starts 2024-03-30T15:10:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: 42.1d scrub ok 2024-03-30T15:10:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:10:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:10:14.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:14 smithi028 ceph-mon[41614]: pgmap v1116: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:14.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:14.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:14.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:14.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:15.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:15.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:15.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:15.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:15.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:15.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:15.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:15.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:15.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:15 smithi134 ceph-mon[33714]: osdmap e752: 8 total, 7 up, 8 in 2024-03-30T15:10:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[29876]: osdmap e752: 8 total, 7 up, 8 in 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[41614]: osdmap e752: 8 total, 7 up, 8 in 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]: dispatch 2024-03-30T15:10:15.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:15.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:15.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:15.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:16.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:16.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:16.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:16.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:16.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:16.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:16.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.399+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/2721929302 >> v1:172.21.15.28:6789/0 conn(0x7fdd34102260 legacy=0x7fdd34102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/2721929302 shutdown_connections 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/2721929302 >> 172.21.15.28:0/2721929302 conn(0x7fdd340fdb10 msgr2=0x7fdd340fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/2721929302 shutdown_connections 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/2721929302 wait complete. 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 librados: starting msgr at 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 librados: starting objecter 2024-03-30T15:10:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 Processor -- start 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- start start 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 librados: setting wanted keys 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 librados: calling monclient init 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.400+0000 7fdd3c4c4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd341a92c0 con 0x7fdd3410aef0 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd3c4c4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd341aa4e0 con 0x7fdd3410ea90 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd3c4c4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd341ab720 con 0x7fdd34102260 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd3a260700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdd34102260 0x7fdd34107530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47178/0 (socket says 172.21.15.28:47178) 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd3a260700 1 -- 172.21.15.28:0/4119202976 learned_addr learned my addr 172.21.15.28:0/4119202976 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd34102260 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd3410aef0 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter ms_handle_connect 0x7fdd3410ea90 2024-03-30T15:10:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923246635 0 0) 0x7fdd341ab720 con 0x7fdd34102260 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd200031f0 con 0x7fdd34102260 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116247929 0 0) 0x7fdd341aa4e0 con 0x7fdd3410ea90 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd341ab720 con 0x7fdd3410ea90 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261788512 0 0) 0x7fdd341a92c0 con 0x7fdd3410aef0 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd341aa4e0 con 0x7fdd3410aef0 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1012389520 0 0) 0x7fdd200031f0 con 0x7fdd34102260 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd341a92c0 con 0x7fdd34102260 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd30004620 con 0x7fdd34102260 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 55235779 0 0) 0x7fdd341ab720 con 0x7fdd3410ea90 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd200031f0 con 0x7fdd3410ea90 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd2c001c30 con 0x7fdd3410ea90 2024-03-30T15:10:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3564753391 0 0) 0x7fdd341aa4e0 con 0x7fdd3410aef0 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd341ab720 con 0x7fdd3410aef0 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd24001940 con 0x7fdd3410aef0 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3192068001 0 0) 0x7fdd341a92c0 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 >> v1:172.21.15.134:6789/0 conn(0x7fdd3410ea90 legacy=0x7fdd341a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 >> v1:172.21.15.28:6789/0 conn(0x7fdd3410aef0 legacy=0x7fdd341a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.401+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd341ac960 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd341aa710 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd3c4c4700 10 client.58736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd341aabd0 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd3c4c4700 1 librados: init done 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdd30001bd0 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.402+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdd30005220 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.403+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd18005120 con 0x7fdd34102260 2024-03-30T15:10:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.403+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdd30005460 con 0x7fdd34102260 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(753..753 src has 442..753) ==== 9237+0+0 (unknown 1334915118 0 0) 0x7fdd30089df0 con 0x7fdd34102260 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 10 client.58736.objecter ms_dispatch 0x7fdd34001550 osd_map(753..753 src has 442..753) 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 3 client.58736.objecter handle_osd_map got epochs [753,753] > 0 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 3 client.58736.objecter handle_osd_map decoding full epoch 753 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 20 client.58736.objecter dump_active .. 0 homeless 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.404+0000 7fdd2b7fe700 10 client.58736.objecter ms_handle_connect 0x7fdd20070380 2024-03-30T15:10:16.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.409+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdd30005200 con 0x7fdd34102260 2024-03-30T15:10:16.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:16.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:16.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.625+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fdd18005410 con 0x7fdd34102260 2024-03-30T15:10:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.628+0000 7fdd2b7fe700 1 -- 172.21.15.28:0/4119202976 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v754) ==== 81+0+57856 (unknown 4041607107 0 3867188925) 0x7fdd30056bd0 con 0x7fdd34102260 2024-03-30T15:10:16.630 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 754, 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:10:16.410026+0000", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T15:08:50.456069+0000", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:10:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 117, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 334, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:10:16.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.633 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "562", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "559", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.635 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.636 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:10:16.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.9000000953674316, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.9000000953674316, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.5499999523162842, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.5499999523162842, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.640 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:10:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.644 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6700000762939453, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6700000762939453, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.75, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:10:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.648 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.648 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.648 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.649 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.651 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.652 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.654 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.655 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.656 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.658 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:10:16.659 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.661 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.664 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.666 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.668 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.669 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.2400000095367432, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.2400000095367432, 2024-03-30T15:10:16.671 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.672 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.673 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.674 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.675 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:10:16.677 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 42, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_delete_pools_parallel.smithi028-93953", 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:53.915795+0000", 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.679 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.680 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.681 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.682 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-03-30T15:10:16.683 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-111", 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:08:12.612746+0000", 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 3, 2024-03-30T15:10:16.684 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 1, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 8, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 8, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 8, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 8, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 8, 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "754", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "754", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "754", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "754", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 6, 2024-03-30T15:10:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 751, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [ 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: 334 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.687 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi028-93253-111", 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 8192, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {} 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 334, 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "test-rados-api-smithi028-93253-134", 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T15:10:10.660313+0000", 2024-03-30T15:10:16.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8201, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:10:16.690 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "754", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "754", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "754", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "754", 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:10:16.691 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 6, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 751, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": 322, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "writeback", 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:10:16.692 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:10:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-03-30T15:10:16.694 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.695 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 0, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 660, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 669, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.696 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 10 librados: watch_flush enter 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 10 librados: watch_flush exit 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 20 client.58736.objecter shutdown clearing up homeless session... 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 10 client.58736.objecter successfully canceled tick 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdd20070380 legacy=0x7fdd20072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 >> v1:172.21.15.28:6790/0 conn(0x7fdd34102260 legacy=0x7fdd34107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 shutdown_connections 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 >> 172.21.15.28:0/4119202976 conn(0x7fdd340fdb10 msgr2=0x7fdd34107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 shutdown_connections 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 -- 172.21.15.28:0/4119202976 wait complete. 2024-03-30T15:10:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:16.630+0000 7fdd3c4c4700 1 librados: shutdown 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3394323575 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.700 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/3394323575", 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/3394323575", 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/3394323575", 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/3394323575", 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "exists" 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.701 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:10:16.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:10:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.706 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3451325043 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/3451325043", 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/3451325043", 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/3451325043", 2024-03-30T15:10:16.710 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/3451325043", 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.711 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 736, 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 32, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3063775514 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/3063775514", 2024-03-30T15:10:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/3063775514", 2024-03-30T15:10:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/3063775514", 2024-03-30T15:10:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/3063775514", 2024-03-30T15:10:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]': finished 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: osdmap e753: 8 total, 7 up, 8 in 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[29876]: pgmap v1119: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]': finished 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: osdmap e753: 8 total, 7 up, 8 in 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:10:16.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:10:16.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:16 smithi028 ceph-mon[41614]: pgmap v1119: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:10:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:10:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:10:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 749, 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:10:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:08:27.042268+0000", 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 668 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:21.783823+0000", 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:10:12.646266+0000", 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:12.781159+0000", 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:56.203636+0000", 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:10:16.187614+0000", 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:10:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:10:13.140466+0000", 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:10:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:09:36.725884+0000", 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.17", 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:10:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:10:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [ 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.13", 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-03-30T15:10:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:10:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "testprofile-test-rados-api-smithi028-93253-111": { 2024-03-30T15:10:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-device-class": "", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-num-failure-domains": "0", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-osds-per-failure-domain": "0", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-root": "default", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "jerasure-per-chunk-alignment": "false", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van", 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "w": "8" 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:10:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 322, 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 6, 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 325, 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 326, 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 329, 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 333, 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 6, 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:16.769 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:10:16.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:16.770 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:10:16.770 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:10:16.770 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:16.770 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-134", "mode": "writeback"}]': finished 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: osdmap e753: 8 total, 7 up, 8 in 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:10:16.771 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:16 smithi134 ceph-mon[33714]: pgmap v1119: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/642 objects degraded (9.657%) 2024-03-30T15:10:16.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:16.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:16.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:16.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:17.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:10:17.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:17.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:17.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:17.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- 172.21.15.28:0/2002314690 >> v1:172.21.15.28:6790/0 conn(0x7fe6fc10ead0 legacy=0x7fe6fc110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- 172.21.15.28:0/2002314690 shutdown_connections 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- 172.21.15.28:0/2002314690 >> 172.21.15.28:0/2002314690 conn(0x7fe6fc0fdb50 msgr2=0x7fe6fc0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- 172.21.15.28:0/2002314690 shutdown_connections 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- 172.21.15.28:0/2002314690 wait complete. 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 librados: starting msgr at 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 librados: starting objecter 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 Processor -- start 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 -- start start 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 librados: setting wanted keys 2024-03-30T15:10:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.238+0000 7fe70175e700 1 librados: calling monclient init 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe70175e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6fc1a7690 con 0x7fe6fc10af30 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe70175e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6fc1a8870 con 0x7fe6fc10ead0 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe70175e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6fc1a9a50 con 0x7fe6fc1022a0 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6faffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe6fc1022a0 0x7fe6fc104a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47198/0 (socket says 172.21.15.28:47198) 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6faffd700 1 -- 172.21.15.28:0/443416962 learned_addr learned my addr 172.21.15.28:0/443416962 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter ms_handle_connect 0x7fe6fc10af30 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter ms_handle_connect 0x7fe6fc1022a0 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter ms_handle_connect 0x7fe6fc10ead0 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968371356 0 0) 0x7fe6fc1a7690 con 0x7fe6fc10af30 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6e80031f0 con 0x7fe6fc10af30 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855381968 0 0) 0x7fe6fc1a9a50 con 0x7fe6fc1022a0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6fc1a7690 con 0x7fe6fc1022a0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975795985 0 0) 0x7fe6fc1a8870 con 0x7fe6fc10ead0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6fc1a9a50 con 0x7fe6fc10ead0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1502758693 0 0) 0x7fe6e80031f0 con 0x7fe6fc10af30 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6fc1a8870 con 0x7fe6fc10af30 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe6ec001830 con 0x7fe6fc10af30 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1351677604 0 0) 0x7fe6fc1a7690 con 0x7fe6fc1022a0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6e80031f0 con 0x7fe6fc1022a0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe6e40019b0 con 0x7fe6fc1022a0 2024-03-30T15:10:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.239+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4137691115 0 0) 0x7fe6fc1a9a50 con 0x7fe6fc10ead0 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6fc1a7690 con 0x7fe6fc10ead0 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe6f4004990 con 0x7fe6fc10ead0 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1984652715 0 0) 0x7fe6fc1a8870 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 >> v1:172.21.15.28:6790/0 conn(0x7fe6fc1022a0 legacy=0x7fe6fc104a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 >> v1:172.21.15.134:6789/0 conn(0x7fe6fc10ead0 legacy=0x7fe6fc1a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc1aac30 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6fc1a9c80 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe70175e700 10 client.25542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6fc1aa140 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe70175e700 1 librados: init done 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe6ec003b20 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.240+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe6ec004a70 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.241+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6dc005120 con 0x7fe6fc10af30 2024-03-30T15:10:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.241+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe6ec01b390 con 0x7fe6fc10af30 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(754..754 src has 442..754) ==== 9237+0+0 (unknown 4186228165 0 0) 0x7fe6ec0889e0 con 0x7fe6fc10af30 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 10 client.25542.objecter ms_dispatch 0x7fe6fc001550 osd_map(754..754 src has 442..754) 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 3 client.25542.objecter handle_osd_map got epochs [754,754] > 0 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 3 client.25542.objecter handle_osd_map decoding full epoch 754 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 20 client.25542.objecter dump_active .. 0 homeless 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.242+0000 7fe6f3fff700 10 client.25542.objecter ms_handle_connect 0x7fe6e80704e0 2024-03-30T15:10:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.247+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe6ec05dad0 con 0x7fe6fc10af30 2024-03-30T15:10:17.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:17.440 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:17.482+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fe6dc005410 con 0x7fe6fc10af30 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: osdmap e754: 8 total, 7 up, 8 in 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4119202976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 42.9 scrub starts 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 42.9 scrub ok 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/642 objects degraded (9.657%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[29876]: osdmap e755: 8 total, 7 up, 8 in 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:10:17.531 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: osdmap e754: 8 total, 7 up, 8 in 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4119202976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 42.9 scrub starts 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 42.9 scrub ok 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/642 objects degraded (9.657%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:17.532 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:17 smithi028 ceph-mon[41614]: osdmap e755: 8 total, 7 up, 8 in 2024-03-30T15:10:17.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:17.545 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:10:17.545 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:10:17.545 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:10:17.545 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:10:17.545 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:10:17.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:17.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T15:10:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: osdmap e754: 8 total, 7 up, 8 in 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]: dispatch 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4119202976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 42.9 scrub starts 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 42.9 scrub ok 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/642 objects degraded (9.657%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:17.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:17.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-134"}]': finished 2024-03-30T15:10:17.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:17 smithi134 ceph-mon[33714]: osdmap e755: 8 total, 7 up, 8 in 2024-03-30T15:10:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:17.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:17.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:17.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:17.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:17.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:18.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:18.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:18.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:18.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:18.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:18.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.421+0000 7fe6f3fff700 1 -- 172.21.15.28:0/443416962 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v756) ==== 101+0+0 (unknown 627822146 0 0) 0x7fe6ec086210 con 0x7fe6fc10af30 2024-03-30T15:10:18.423 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 10 librados: watch_flush enter 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 10 librados: watch_flush exit 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 20 client.25542.objecter shutdown clearing up homeless session... 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 10 client.25542.objecter successfully canceled tick 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe6e80704e0 legacy=0x7fe6e80729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 >> v1:172.21.15.28:6789/0 conn(0x7fe6fc10af30 legacy=0x7fe6fc105130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 shutdown_connections 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 >> 172.21.15.28:0/443416962 conn(0x7fe6fc0fdb50 msgr2=0x7fe6fc111f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 shutdown_connections 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 -- 172.21.15.28:0/443416962 wait complete. 2024-03-30T15:10:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:18.425+0000 7fe70175e700 1 librados: shutdown 2024-03-30T15:10:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:10:18.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:18.563 INFO:tasks.thrashosds.thrasher:j is {'epoch': 754, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:10:16.410026+0000', 'last_up_change': '2024-03-30T15:08:50.456069+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 117, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 334, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '562', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '559', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9000000953674316, 'score_stable': 2.9000000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 42, 'pool_name': 'ceph_test_rados_delete_pools_parallel.smithi028-93953', 'create_time': '2024-03-30T14:57:53.915795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 322, 'pool_name': 'test-rados-api-smithi028-93253-111', 'create_time': '2024-03-30T15:08:12.612746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '754', 'last_force_op_resend': '754', 'last_force_op_resend_prenautilus': '754', 'last_force_op_resend_preluminous': '754', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 751, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [334], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi028-93253-111', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 334, 'pool_name': 'test-rados-api-smithi028-93253-134', 'create_time': '2024-03-30T15:10:10.660313+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '754', 'last_force_op_resend': '754', 'last_force_op_resend_prenautilus': '754', 'last_force_op_resend_preluminous': '754', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 751, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 322, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 660, 'down_at': 669, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 3394323575}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 3394323575}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 3394323575}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 3394323575}]}, 'public_addr': '172.21.15.28:6801/3394323575', 'cluster_addr': '172.21.15.28:6802/3394323575', 'heartbeat_back_addr': '172.21.15.28:6804/3394323575', 'heartbeat_front_addr': '172.21.15.28:6803/3394323575', 'state': ['exists']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 3451325043}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 3451325043}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 3451325043}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 3451325043}]}, 'public_addr': '172.21.15.28:6809/3451325043', 'cluster_addr': '172.21.15.28:6810/3451325043', 'heartbeat_back_addr': '172.21.15.28:6812/3451325043', 'heartbeat_front_addr': '172.21.15.28:6811/3451325043', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 736, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 3063775514}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 3063775514}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 3063775514}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 3063775514}]}, 'public_addr': '172.21.15.134:6800/3063775514', 'cluster_addr': '172.21.15.134:6801/3063775514', 'heartbeat_back_addr': '172.21.15.134:6803/3063775514', 'heartbeat_front_addr': '172.21.15.134:6802/3063775514', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:08:27.042268+0000', 'dead_epoch': 668}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:21.783823+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:10:12.646266+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:12.781159+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:56.203636+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:10:16.187614+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:10:13.140466+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:09:36.725884+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.17', 'osds': [6, 7, 1]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.13', 'osds': [1, 3, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi028-93253-111': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 322, 'snaps': [{'begin': 3, 'length': 2}, {'begin': 6, 'length': 1}]}, {'pool': 325, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 326, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 329, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 333, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:10:18.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:10:18.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:18.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:10:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:10:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:10:18.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:18 smithi134 ceph-mon[33714]: pgmap v1122: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 62/642 objects degraded (9.657%) 2024-03-30T15:10:18.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:18.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:18.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:10:18.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:10:18.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:18.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[29876]: pgmap v1122: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 62/642 objects degraded (9.657%) 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:10:18.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:10:18.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:10:18.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:18 smithi028 ceph-mon[41614]: pgmap v1122: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 60 active+undersized, 167 active+clean; 582 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 62/642 objects degraded (9.657%) 2024-03-30T15:10:18.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:18.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:18.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:18.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:18.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:19.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:19.102 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:19.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:19.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:19.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:19.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:19.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:19.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:19 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:10:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:19 smithi134 ceph-mon[33714]: osdmap e756: 8 total, 7 up, 8 in 2024-03-30T15:10:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:19 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:10:19.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:19 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:10:19.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:10:19.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[41614]: osdmap e756: 8 total, 7 up, 8 in 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/443416962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[29876]: osdmap e756: 8 total, 7 up, 8 in 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:10:19.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:19 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:10:19.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:19.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:10:19.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:19.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:19.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:19.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:19.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:20.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:20.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:20.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:20.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:20.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:20.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:20.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:20.547 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: osdmap e757: 8 total, 7 up, 8 in 2024-03-30T15:10:20.547 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: 5.10 deep-scrub starts 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: 5.10 deep-scrub ok 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[29876]: pgmap v1125: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: osdmap e757: 8 total, 7 up, 8 in 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:10:20.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: 5.10 deep-scrub starts 2024-03-30T15:10:20.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: 5.10 deep-scrub ok 2024-03-30T15:10:20.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:20 smithi028 ceph-mon[41614]: pgmap v1125: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:20.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:20.707 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: osdmap e757: 8 total, 7 up, 8 in 2024-03-30T15:10:20.707 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.707 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:20.707 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:10:20.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:10:20.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: 5.10 deep-scrub starts 2024-03-30T15:10:20.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: 5.10 deep-scrub ok 2024-03-30T15:10:20.708 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-mon[33714]: pgmap v1125: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:20.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:20.734 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:20.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:20.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:20.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:21.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:21.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:21.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:10:21.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:21.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:21.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:21.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:21.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:21.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:21.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:21.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:21.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:21.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: osdmap e758: 8 total, 7 up, 8 in 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: osdmap e758: 8 total, 7 up, 8 in 2024-03-30T15:10:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:10:21.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:21 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:10:21.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:21.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:21.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: osdmap e758: 8 total, 7 up, 8 in 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:10:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:21 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:10:21.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:21.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:22.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:22.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:22.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:22.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:22.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:22.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:22.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:22.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:22.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 42.1e scrub starts 2024-03-30T15:10:22.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 42.1e scrub ok 2024-03-30T15:10:22.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:10:22.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: osdmap e759: 8 total, 7 up, 8 in 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: pgmap v1128: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/645 objects degraded (9.612%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:22.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 42.1e scrub starts 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 42.1e scrub ok 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: osdmap e759: 8 total, 7 up, 8 in 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:10:22.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:10:22.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:10:22.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: pgmap v1128: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:22.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/645 objects degraded (9.612%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:22.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:22 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:22.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:22.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:22.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:22.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:22.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 42.1e scrub starts 2024-03-30T15:10:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 42.1e scrub ok 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: osdmap e759: 8 total, 7 up, 8 in 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: pgmap v1128: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 582 KiB data, 748 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 62/645 objects degraded (9.612%) 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/645 objects degraded (9.612%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:22.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:22 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:22.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:23.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:23.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:23.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:23.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:23.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:23.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:23.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:23.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: osdmap e760: 8 total, 7 up, 8 in 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:23.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]': finished 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: osdmap e760: 8 total, 7 up, 8 in 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T15:10:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T15:10:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]': finished 2024-03-30T15:10:23.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:23.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:23.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: osdmap e760: 8 total, 7 up, 8 in 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]: dispatch 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T15:10:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T15:10:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:10:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-136", "mode": "writeback"}]': finished 2024-03-30T15:10:23.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:24.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 -- 172.21.15.28:0/3745545853 >> v1:172.21.15.28:6789/0 conn(0x7f6308109df0 legacy=0x7f630810c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 -- 172.21.15.28:0/3745545853 shutdown_connections 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 -- 172.21.15.28:0/3745545853 >> 172.21.15.28:0/3745545853 conn(0x7f63080fca30 msgr2=0x7f63080fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 -- 172.21.15.28:0/3745545853 shutdown_connections 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 -- 172.21.15.28:0/3745545853 wait complete. 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 librados: starting msgr at 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.073+0000 7f630fba1700 1 librados: starting objecter 2024-03-30T15:10:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 Processor -- start 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 -- start start 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 librados: setting wanted keys 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 librados: calling monclient init 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f630810fa30 con 0x7f6308101160 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f630810fc40 con 0x7f630810d990 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630fba1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f630810fe50 con 0x7f6308109df0 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630d93d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6308101160 0x7f630810def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48054/0 (socket says 172.21.15.28:48054) 2024-03-30T15:10:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f630d93d700 1 -- 172.21.15.28:0/1590591792 learned_addr learned my addr 172.21.15.28:0/1590591792 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter ms_handle_connect 0x7f6308101160 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter ms_handle_connect 0x7f6308109df0 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter ms_handle_connect 0x7f630810d990 2024-03-30T15:10:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89801918 0 0) 0x7f630810fa30 con 0x7f6308101160 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62f00031f0 con 0x7f6308101160 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870533505 0 0) 0x7f6308114e10 con 0x7f6308109df0 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f630810fa30 con 0x7f6308109df0 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509329651 0 0) 0x7f630810fc40 con 0x7f630810d990 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6308114e10 con 0x7f630810d990 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3649941152 0 0) 0x7f62f00031f0 con 0x7f6308101160 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.074+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f630810fc40 con 0x7f6308101160 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62f8001c70 con 0x7f6308101160 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 714847427 0 0) 0x7f630810fa30 con 0x7f6308109df0 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62f00031f0 con 0x7f6308109df0 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3880888756 0 0) 0x7f6308114e10 con 0x7f630810d990 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f630810fa30 con 0x7f630810d990 2024-03-30T15:10:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62fc004650 con 0x7f6308109df0 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6300001830 con 0x7f630810d990 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2655522123 0 0) 0x7f630810fc40 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 >> v1:172.21.15.28:6790/0 conn(0x7f6308109df0 legacy=0x7f630810e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 >> v1:172.21.15.134:6789/0 conn(0x7f630810d990 legacy=0x7f63081a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63081ac9e0 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63081a99b0 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f630fba1700 10 client.25548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63081a9e70 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f630fba1700 1 librados: init done 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f62f8002080 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.075+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f62f8006030 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.076+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4005120 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.076+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f62f8003cd0 con 0x7f6308101160 2024-03-30T15:10:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(761..761 src has 442..761) ==== 9237+0+0 (unknown 3533380698 0 0) 0x7f62f8089de0 con 0x7f6308101160 2024-03-30T15:10:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 10 client.25548.objecter ms_dispatch 0x7f6308001550 osd_map(761..761 src has 442..761) 2024-03-30T15:10:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 3 client.25548.objecter handle_osd_map got epochs [761,761] > 0 2024-03-30T15:10:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 3 client.25548.objecter handle_osd_map decoding full epoch 761 2024-03-30T15:10:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 20 client.25548.objecter dump_active .. 0 homeless 2024-03-30T15:10:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.077+0000 7f6306ffd700 10 client.25548.objecter ms_handle_connect 0x7f62f0070450 2024-03-30T15:10:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.082+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f62f8056bc0 con 0x7f6308101160 2024-03-30T15:10:24.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:24.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:24.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:24.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.287+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62f4000b90 con 0x7f62f0070450 2024-03-30T15:10:24.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.321+0000 7f6306ffd700 1 -- 172.21.15.28:0/1590591792 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+635121 (unknown 2965378022 0 121236991) 0x7f62f4000b90 con 0x7f62f0070450 2024-03-30T15:10:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:10:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.326+0000 7f630fba1700 10 librados: watch_flush enter 2024-03-30T15:10:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 10 librados: watch_flush exit 2024-03-30T15:10:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 20 client.25548.objecter shutdown clearing up homeless session... 2024-03-30T15:10:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 10 client.25548.objecter successfully canceled tick 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 >> v1:172.21.15.28:6800/3011431093 conn(0x7f62f0070450 legacy=0x7f62f0072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 >> v1:172.21.15.28:6789/0 conn(0x7f6308101160 legacy=0x7f630810def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 shutdown_connections 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 >> 172.21.15.28:0/1590591792 conn(0x7f63080fca30 msgr2=0x7f630806ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 shutdown_connections 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 -- 172.21.15.28:0/1590591792 wait complete. 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:24.327+0000 7f630fba1700 1 librados: shutdown 2024-03-30T15:10:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:10:24.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:24.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:24.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:24.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:24.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: 42.1a deep-scrub starts 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: 42.1a deep-scrub ok 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: osdmap e761: 8 total, 7 up, 8 in 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:10:24.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: pgmap v1131: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[29876]: from='client.25548 v1:172.21.15.28:0/1590591792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: 42.1a deep-scrub starts 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: 42.1a deep-scrub ok 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: osdmap e761: 8 total, 7 up, 8 in 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: pgmap v1131: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:10:24.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:24 smithi028 ceph-mon[41614]: from='client.25548 v1:172.21.15.28:0/1590591792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:10:24.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:24.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:24.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:24.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:24.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: 42.1a deep-scrub starts 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: 42.1a deep-scrub ok 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: osdmap e761: 8 total, 7 up, 8 in 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: pgmap v1131: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:10:24.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:24 smithi134 ceph-mon[33714]: from='client.25548 v1:172.21.15.28:0/1590591792' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:10:24.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.069+0000 7f354e1b7700 1 -- 172.21.15.28:0/2087214492 >> v1:172.21.15.28:6789/0 conn(0x7f354810b000 legacy=0x7f354810d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- 172.21.15.28:0/2087214492 shutdown_connections 2024-03-30T15:10:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- 172.21.15.28:0/2087214492 >> 172.21.15.28:0/2087214492 conn(0x7f35480fc9e0 msgr2=0x7f35480fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- 172.21.15.28:0/2087214492 shutdown_connections 2024-03-30T15:10:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- 172.21.15.28:0/2087214492 wait complete. 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 librados: starting msgr at 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 librados: starting objecter 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 Processor -- start 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- start start 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 librados: setting wanted keys 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 librados: calling monclient init 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35481a6f40 con 0x7f354810b000 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35481a8160 con 0x7f354810eb10 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.070+0000 7f354e1b7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35481a93a0 con 0x7f35481031f0 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f35477fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f35481031f0 0x7f354810a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44770/0 (socket says 172.21.15.28:44770) 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f35477fe700 1 -- 172.21.15.28:0/3019671386 learned_addr learned my addr 172.21.15.28:0/3019671386 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter ms_handle_connect 0x7f354810b000 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter ms_handle_connect 0x7f35481031f0 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter ms_handle_connect 0x7f354810eb10 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975280171 0 0) 0x7f35481a6f40 con 0x7f354810b000 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3534003aa0 con 0x7f354810b000 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280618265 0 0) 0x7f35481a93a0 con 0x7f35481031f0 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35481a6f40 con 0x7f35481031f0 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995049804 0 0) 0x7f35481a8160 con 0x7f354810eb10 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35481a93a0 con 0x7f354810eb10 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2017610339 0 0) 0x7f3534003aa0 con 0x7f354810b000 2024-03-30T15:10:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35481a8160 con 0x7f354810b000 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3530004550 con 0x7f354810b000 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2938578542 0 0) 0x7f35481a6f40 con 0x7f35481031f0 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3534003aa0 con 0x7f35481031f0 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2437081256 0 0) 0x7f35481a93a0 con 0x7f354810eb10 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35481a6f40 con 0x7f354810eb10 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35380015b0 con 0x7f35481031f0 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35400017b0 con 0x7f354810eb10 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 554407230 0 0) 0x7f35481a8160 con 0x7f354810b000 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 >> v1:172.21.15.28:6790/0 conn(0x7f35481031f0 legacy=0x7f354810a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 >> v1:172.21.15.134:6789/0 conn(0x7f354810eb10 legacy=0x7f35481a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35481aa5e0 con 0x7f354810b000 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35481a7170 con 0x7f354810b000 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f354e1b7700 10 client.25554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f35481a7630 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.071+0000 7f354e1b7700 1 librados: init done 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.072+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3530001180 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.072+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3530004c30 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.072+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3528005100 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.073+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3530001b40 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.073+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(762..762 src has 442..762) ==== 9237+0+0 (unknown 1687087537 0 0) 0x7f35300889a0 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.073+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 osd_map(762..762 src has 442..762) 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.073+0000 7f3544ff9700 3 client.25554.objecter handle_osd_map got epochs [762,762] > 0 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.073+0000 7f3544ff9700 3 client.25554.objecter handle_osd_map decoding full epoch 762 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3544ff9700 20 client.25554.objecter dump_active .. 0 homeless 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v762) ==== 54+0+16 (unknown 2383308727 0 2420000192) 0x7f3530089060 con 0x7f354810b000 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f3534070540 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 10 client.25554.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 1 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 10 client.25554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=763}) -- 0x7f35280053e0 con 0x7f354810b000 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3528000ed0 con 0x7f354810b000 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=762) ==== 24+0+0 (unknown 3417820278 0 0) 0x7f3530054e50 con 0x7f354810b000 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3545ffb700 10 client.25554.objecter _check_command_map_dne tid 1 current 762 map_dne_bound 762 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3545ffb700 10 client.25554.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3545ffb700 15 client.25554.objecter _session_command_op_remove -1 1 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 10 client.25554.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 2 2024-03-30T15:10:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 2 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 1 2 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 4 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 10 client.25554.objecter _send_command 2 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f3528002330 con 0x7f3528003910 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.074+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f3528003910 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.075+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3528002330 con 0x7f3528003910 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.075+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.075+0000 7f3544ff9700 10 client.25554.objecter _finish_command 2 = system:0 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.075+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 4 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.075+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 1 2 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 10 client.25554.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528001bb0 osd=1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 4 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 10 client.25554.objecter _send_command 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.092+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3528004c80 con 0x7f3528003910 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3528004c80 con 0x7f3528003910 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 10 client.25554.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528001bb0 osd=1 4 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 1 3 2024-03-30T15:10:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 10 client.25554.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 4 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 4 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 2 4 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 4 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 10 client.25554.objecter _send_command 4 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f352800ca20 con 0x7f3528008f10 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.094+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f3528008f10 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.095+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f352800ca20 con 0x7f3528008f10 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.095+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 4: 0 ) 2024-03-30T15:10:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.095+0000 7f3544ff9700 10 client.25554.objecter _finish_command 4 = system:0 2024-03-30T15:10:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.095+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 4 2024-03-30T15:10:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.095+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 2 4 2024-03-30T15:10:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 10 client.25554.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 5 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 5 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528004f10 osd=2 3 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 2 5 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 4 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 10 client.25554.objecter _send_command 5 2024-03-30T15:10:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.112+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3528004820 con 0x7f3528008f10 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.114+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3528004820 con 0x7f3528008f10 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.114+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.114+0000 7f3544ff9700 10 client.25554.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.114+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528004f10 osd=2 4 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.114+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 2 5 2024-03-30T15:10:25.116 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 10 client.25554.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 6 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 6 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 3 6 2024-03-30T15:10:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 4 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 10 client.25554.objecter _send_command 6 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f352800e8f0 con 0x7f352800dd50 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.115+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f352800dd50 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.116+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f352800e8f0 con 0x7f352800dd50 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.116+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 6: 0 ) 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.116+0000 7f3544ff9700 10 client.25554.objecter _finish_command 6 = system:0 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.116+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 4 2024-03-30T15:10:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.116+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 3 6 2024-03-30T15:10:25.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 10 client.25554.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 7 2024-03-30T15:10:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 7 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528004ab0 osd=3 3 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 3 7 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 4 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 10 client.25554.objecter _send_command 7 2024-03-30T15:10:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.134+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f352800f270 con 0x7f352800dd50 2024-03-30T15:10:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f352800f270 con 0x7f352800dd50 2024-03-30T15:10:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f3544ff9700 10 client.25554.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528004ab0 osd=3 4 2024-03-30T15:10:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 3 7 2024-03-30T15:10:25.138 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 10 client.25554.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 8 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 8 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 4 8 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 4 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 10 client.25554.objecter _send_command 8 2024-03-30T15:10:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.136+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f3528016640 con 0x7f3528012a90 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.137+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f3528012a90 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.138+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3528016640 con 0x7f3528012a90 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.138+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 8: 0 ) 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.138+0000 7f3544ff9700 10 client.25554.objecter _finish_command 8 = system:0 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.138+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 4 2024-03-30T15:10:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.138+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 4 8 2024-03-30T15:10:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 10 client.25554.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 9 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 9 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f3528002fa0 osd=4 3 2024-03-30T15:10:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 4 9 2024-03-30T15:10:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 4 2024-03-30T15:10:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 10 client.25554.objecter _send_command 9 2024-03-30T15:10:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.155+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f352800ee10 con 0x7f3528012a90 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f352800ee10 con 0x7f3528012a90 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 10 client.25554.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f3528002fa0 osd=4 4 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 4 9 2024-03-30T15:10:25.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 10 client.25554.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 10 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 10 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 5 10 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 4 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 10 client.25554.objecter _send_command 10 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f352801b1d0 con 0x7f352800d8b0 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.157+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f352800d8b0 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.158+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f352801b1d0 con 0x7f352800d8b0 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.158+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 10: 0 ) 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.158+0000 7f3544ff9700 10 client.25554.objecter _finish_command 10 = system:0 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.158+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 4 2024-03-30T15:10:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.158+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 5 10 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 10 client.25554.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 11 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 11 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352800ee10 osd=5 3 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 5 11 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 4 2024-03-30T15:10:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 10 client.25554.objecter _send_command 11 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.175+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f352801b1d0 con 0x7f352800d8b0 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f352801b1d0 con 0x7f352800d8b0 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 10 client.25554.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352800ee10 osd=5 4 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 5 11 2024-03-30T15:10:25.179 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 10 client.25554.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 12 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 12 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 6 12 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 4 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 10 client.25554.objecter _send_command 12 2024-03-30T15:10:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f352801f570 con 0x7f352801b850 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.177+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f352801b850 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.178+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f352801f570 con 0x7f352801b850 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.178+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 12: 0 ) 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.178+0000 7f3544ff9700 10 client.25554.objecter _finish_command 12 = system:0 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.178+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 4 2024-03-30T15:10:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.178+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 6 12 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 10 client.25554.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 13 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 13 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352801b1d0 osd=6 3 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 6 13 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 4 2024-03-30T15:10:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 10 client.25554.objecter _send_command 13 2024-03-30T15:10:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.195+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f352800ce20 con 0x7f352801b850 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f352800ce20 con 0x7f352801b850 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 10 client.25554.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352801b1d0 osd=6 4 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 6 13 2024-03-30T15:10:25.201 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 10 client.25554.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 14 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 14 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 7 14 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 4 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 10 client.25554.objecter _send_command 14 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f3528020bc0 con 0x7f352800cbd0 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.200+0000 7f3544ff9700 10 client.25554.objecter ms_handle_connect 0x7f352800cbd0 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.201+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3528020bc0 con 0x7f352800cbd0 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.201+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 14: 0 ) 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.201+0000 7f3544ff9700 10 client.25554.objecter _finish_command 14 = system:0 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.201+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 4 2024-03-30T15:10:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.201+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 7 14 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 10 client.25554.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign -1 15 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 20 client.25554.objecter _get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_remove -1 15 2024-03-30T15:10:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 20 client.25554.objecter get_session s=0x7f352800ec80 osd=7 3 2024-03-30T15:10:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 15 client.25554.objecter _session_command_op_assign 7 15 2024-03-30T15:10:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 4 2024-03-30T15:10:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 10 client.25554.objecter _send_command 15 2024-03-30T15:10:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.218+0000 7f354e1b7700 1 -- 172.21.15.28:0/3019671386 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3528020bc0 con 0x7f352800cbd0 2024-03-30T15:10:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.220+0000 7f3544ff9700 1 -- 172.21.15.28:0/3019671386 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f3528020bc0 con 0x7f352800cbd0 2024-03-30T15:10:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.220+0000 7f3544ff9700 10 client.25554.objecter ms_dispatch 0x7f3548001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.220+0000 7f3544ff9700 10 client.25554.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.220+0000 7f3544ff9700 20 client.25554.objecter put_session s=0x7f352800ec80 osd=7 4 2024-03-30T15:10:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:25.220+0000 7f3544ff9700 15 client.25554.objecter _session_command_op_remove 7 15 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:25.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:25.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:25.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: osdmap e762: 8 total, 7 up, 8 in 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: 42.6 scrub starts 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: 42.6 scrub ok 2024-03-30T15:10:25.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3019671386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: osdmap e762: 8 total, 7 up, 8 in 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: 42.6 scrub starts 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: 42.6 scrub ok 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3019671386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:25.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:25.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:25.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:25.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:25.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:25.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:25.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:25.888 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:25.888 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:25.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: osdmap e762: 8 total, 7 up, 8 in 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]: dispatch 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: 42.6 scrub starts 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: 42.6 scrub ok 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:25.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3019671386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:26.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:26.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:26.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:26.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:26.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:26.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:26.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:26.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:26.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:26.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:26.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: osdmap e763: 8 total, 7 up, 8 in 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: 6.2 deep-scrub starts 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: 6.2 deep-scrub ok 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[29876]: pgmap v1134: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: osdmap e763: 8 total, 7 up, 8 in 2024-03-30T15:10:26.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: 6.2 deep-scrub starts 2024-03-30T15:10:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: 6.2 deep-scrub ok 2024-03-30T15:10:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:10:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:10:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:26 smithi028 ceph-mon[41614]: pgmap v1134: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:26.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:26.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1131,"stamp":"2024-03-30T15:10:24.107661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596412,"num_objects":216,"num_object_clones":1,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":62,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":2102,"num_read_kb":1842,"num_write":768,"num_write_kb":7898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":1,"num_flush_kb":3907,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2328,"ondisk_log_size":2328,"up":656,"acting":657,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":727,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":768248,"kb_used_data":53768,"kb_used_omap":12,"kb_used_meta":714355,"kb_avail":749192968,"statfs":{"total":767960285184,"available":767173599232,"internally_reserved":0,"allocated":55058432,"data_stored":28360541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":731500080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4000008,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3,"num_write_kb":7814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":1,"num_flush_kb":3907,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001464"},"pg_stats":[{"pgid":"335.1f","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.453061+0000","last_change":"2024-03-30T15:10:20.453638+0000","last_active":"2024-03-30T15:10:23.453061+0000","last_peered":"2024-03-30T15:10:23.453061+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.453638+0000","last_became_peered":"2024-03-30T15:10:20.453638+0000","last_unstale":"2024-03-30T15:10:23.453061+0000","last_undegraded":"2024-03-30T15:10:23.453061+0000","last_fullsized":"2024-03-30T15:10:20.439414+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:25.665856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.1e","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452874+0000","last_change":"2024-03-30T15:10:20.443889+0000","last_active":"2024-03-30T15:10:23.452874+0000","last_peered":"2024-03-30T15:10:23.452874+0000","last_clean":"2024-03-30T15:10:23.452874+0000","last_became_active":"2024-03-30T15:10:20.443518+0000","last_became_peered":"2024-03-30T15:10:20.443518+0000","last_unstale":"2024-03-30T15:10:23.452874+0000","last_undegraded":"2024-03-30T15:10:23.452874+0000","last_fullsized":"2024-03-30T15:10:23.452874+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:45.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.1d","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.491555+0000","last_change":"2024-03-30T15:10:20.453100+0000","last_active":"2024-03-30T15:10:23.491555+0000","last_peered":"2024-03-30T15:10:23.491555+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.453100+0000","last_became_peered":"2024-03-30T15:10:20.453100+0000","last_unstale":"2024-03-30T15:10:23.491555+0000","last_undegraded":"2024-03-30T15:10:23.491555+0000","last_fullsized":"2024-03-30T15:10:20.440481+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:34.603400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"335.1c","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.458052+0000","last_change":"2024-03-30T15:10:20.441524+0000","last_active":"2024-03-30T15:10:23.458052+0000","last_peered":"2024-03-30T15:10:23.458052+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.441524+0000","last_became_peered":"2024-03-30T15:10:20.441524+0000","last_unstale":"2024-03-30T15:10:23.458052+0000","last_undegraded":"2024-03-30T15:10:23.458052+0000","last_fullsized":"2024-03-30T15:10:20.433705+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:48.084814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"335.1b","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.452041+0000","last_change":"2024-03-30T15:10:20.450243+0000","last_active":"2024-03-30T15:10:23.452041+0000","last_peered":"2024-03-30T15:10:23.452041+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.450243+0000","last_became_peered":"2024-03-30T15:10:20.450243+0000","last_unstale":"2024-03-30T15:10:23.452041+0000","last_undegraded":"2024-03-30T15:10:23.452041+0000","last_fullsized":"2024-03-30T15:10:20.437641+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:32.270333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.1a","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442691+0000","last_change":"2024-03-30T15:10:20.453038+0000","last_active":"2024-03-30T15:10:23.442691+0000","last_peered":"2024-03-30T15:10:23.442691+0000","last_clean":"2024-03-30T15:10:23.442691+0000","last_became_active":"2024-03-30T15:10:20.452649+0000","last_became_peered":"2024-03-30T15:10:20.452649+0000","last_unstale":"2024-03-30T15:10:23.442691+0000","last_undegraded":"2024-03-30T15:10:23.442691+0000","last_fullsized":"2024-03-30T15:10:23.442691+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:43.009484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"335.19","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457251+0000","last_change":"2024-03-30T15:10:20.445631+0000","last_active":"2024-03-30T15:10:23.457251+0000","last_peered":"2024-03-30T15:10:23.457251+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.445631+0000","last_became_peered":"2024-03-30T15:10:20.445631+0000","last_unstale":"2024-03-30T15:10:23.457251+0000","last_undegraded":"2024-03-30T15:10:23.457251+0000","last_fullsized":"2024-03-30T15:10:20.433647+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:26.685093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"335.18","version":"761'4","reported_seq":26,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.502704+0000","last_change":"2024-03-30T15:10:20.449800+0000","last_active":"2024-03-30T15:10:23.502704+0000","last_peered":"2024-03-30T15:10:23.502704+0000","last_clean":"2024-03-30T15:10:23.502704+0000","last_became_active":"2024-03-30T15:10:20.449504+0000","last_became_peered":"2024-03-30T15:10:20.449504+0000","last_unstale":"2024-03-30T15:10:23.502704+0000","last_undegraded":"2024-03-30T15:10:23.502704+0000","last_fullsized":"2024-03-30T15:10:23.502704+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:31.383725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":1,"num_flush_kb":3907,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.17","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453097+0000","last_change":"2024-03-30T15:10:20.445862+0000","last_active":"2024-03-30T15:10:23.453097+0000","last_peered":"2024-03-30T15:10:23.453097+0000","last_clean":"2024-03-30T15:10:23.453097+0000","last_became_active":"2024-03-30T15:10:20.445499+0000","last_became_peered":"2024-03-30T15:10:20.445499+0000","last_unstale":"2024-03-30T15:10:23.453097+0000","last_undegraded":"2024-03-30T15:10:23.453097+0000","last_fullsized":"2024-03-30T15:10:23.453097+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:24.789195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.16","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.453229+0000","last_change":"2024-03-30T15:10:20.451698+0000","last_active":"2024-03-30T15:10:23.453229+0000","last_peered":"2024-03-30T15:10:23.453229+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.451698+0000","last_became_peered":"2024-03-30T15:10:20.451698+0000","last_unstale":"2024-03-30T15:10:23.453229+0000","last_undegraded":"2024-03-30T15:10:23.453229+0000","last_fullsized":"2024-03-30T15:10:20.439296+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:19.631969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.15","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457758+0000","last_change":"2024-03-30T15:10:20.443457+0000","last_active":"2024-03-30T15:10:23.457758+0000","last_peered":"2024-03-30T15:10:23.457758+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.443457+0000","last_became_peered":"2024-03-30T15:10:20.443457+0000","last_unstale":"2024-03-30T15:10:23.457758+0000","last_undegraded":"2024-03-30T15:10:23.457758+0000","last_fullsized":"2024-03-30T15:10:20.438004+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:23.400723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"335.14","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.459521+0000","last_change":"2024-03-30T15:10:20.440345+0000","last_active":"2024-03-30T15:10:23.459521+0000","last_peered":"2024-03-30T15:10:23.459521+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.440345+0000","last_became_peered":"2024-03-30T15:10:20.440345+0000","last_unstale":"2024-03-30T15:10:23.459521+0000","last_undegraded":"2024-03-30T15:10:23.459521+0000","last_fullsized":"2024-03-30T15:10:20.433673+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:33.411241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"335.13","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454086+0000","last_change":"2024-03-30T15:10:20.441514+0000","last_active":"2024-03-30T15:10:23.454086+0000","last_peered":"2024-03-30T15:10:23.454086+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.441514+0000","last_became_peered":"2024-03-30T15:10:20.441514+0000","last_unstale":"2024-03-30T15:10:23.454086+0000","last_undegraded":"2024-03-30T15:10:23.454086+0000","last_fullsized":"2024-03-30T15:10:20.434668+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:40.374083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"335.12","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452036+0000","last_change":"2024-03-30T15:10:20.444358+0000","last_active":"2024-03-30T15:10:23.452036+0000","last_peered":"2024-03-30T15:10:23.452036+0000","last_clean":"2024-03-30T15:10:23.452036+0000","last_became_active":"2024-03-30T15:10:20.444021+0000","last_became_peered":"2024-03-30T15:10:20.444021+0000","last_unstale":"2024-03-30T15:10:23.452036+0000","last_undegraded":"2024-03-30T15:10:23.452036+0000","last_fullsized":"2024-03-30T15:10:23.452036+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:39.877602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.11","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.489080+0000","last_change":"2024-03-30T15:10:20.449544+0000","last_active":"2024-03-30T15:10:23.489080+0000","last_peered":"2024-03-30T15:10:23.489080+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.449544+0000","last_became_peered":"2024-03-30T15:10:20.449544+0000","last_unstale":"2024-03-30T15:10:23.489080+0000","last_undegraded":"2024-03-30T15:10:23.489080+0000","last_fullsized":"2024-03-30T15:10:20.438559+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.539993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"335.10","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.488922+0000","last_change":"2024-03-30T15:10:20.452510+0000","last_active":"2024-03-30T15:10:23.488922+0000","last_peered":"2024-03-30T15:10:23.488922+0000","last_clean":"2024-03-30T15:10:23.488922+0000","last_became_active":"2024-03-30T15:10:20.452155+0000","last_became_peered":"2024-03-30T15:10:20.452155+0000","last_unstale":"2024-03-30T15:10:23.488922+0000","last_undegraded":"2024-03-30T15:10:23.488922+0000","last_fullsized":"2024-03-30T15:10:23.488922+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:39.121935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"335.7","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453264+0000","last_change":"2024-03-30T15:10:20.449404+0000","last_active":"2024-03-30T15:10:23.453264+0000","last_peered":"2024-03-30T15:10:23.453264+0000","last_clean":"2024-03-30T15:10:23.453264+0000","last_became_active":"2024-03-30T15:10:20.448000+0000","last_became_peered":"2024-03-30T15:10:20.448000+0000","last_unstale":"2024-03-30T15:10:23.453264+0000","last_undegraded":"2024-03-30T15:10:23.453264+0000","last_fullsized":"2024-03-30T15:10:23.453264+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:22.067676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"335.6","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458599+0000","last_change":"2024-03-30T15:10:20.441135+0000","last_active":"2024-03-30T15:10:23.458599+0000","last_peered":"2024-03-30T15:10:23.458599+0000","last_clean":"2024-03-30T15:10:23.458599+0000","last_became_active":"2024-03-30T15:10:20.440826+0000","last_became_peered":"2024-03-30T15:10:20.440826+0000","last_unstale":"2024-03-30T15:10:23.458599+0000","last_undegraded":"2024-03-30T15:10:23.458599+0000","last_fullsized":"2024-03-30T15:10:23.458599+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:45.945420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"335.5","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454824+0000","last_change":"2024-03-30T15:10:20.444906+0000","last_active":"2024-03-30T15:10:23.454824+0000","last_peered":"2024-03-30T15:10:23.454824+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.444906+0000","last_became_peered":"2024-03-30T15:10:20.444906+0000","last_unstale":"2024-03-30T15:10:23.454824+0000","last_undegraded":"2024-03-30T15:10:23.454824+0000","last_fullsized":"2024-03-30T15:10:20.434814+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:36.143423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"335.4","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442361+0000","last_change":"2024-03-30T15:10:20.455945+0000","last_active":"2024-03-30T15:10:23.442361+0000","last_peered":"2024-03-30T15:10:23.442361+0000","last_clean":"2024-03-30T15:10:23.442361+0000","last_became_active":"2024-03-30T15:10:20.455609+0000","last_became_peered":"2024-03-30T15:10:20.455609+0000","last_unstale":"2024-03-30T15:10:23.442361+0000","last_undegraded":"2024-03-30T15:10:23.442361+0000","last_fullsized":"2024-03-30T15:10:23.442361+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:26.107430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"335.3","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457470+0000","last_change":"2024-03-30T15:10:20.454905+0000","last_active":"2024-03-30T15:10:23.457470+0000","last_peered":"2024-03-30T15:10:23.457470+0000","last_clean":"2024-03-30T15:10:23.457470+0000","last_became_active":"2024-03-30T15:10:20.454602+0000","last_became_peered":"2024-03-30T15:10:20.454602+0000","last_unstale":"2024-03-30T15:10:23.457470+0000","last_undegraded":"2024-03-30T15:10:23.457470+0000","last_fullsized":"2024-03-30T15:10:23.457470+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:27.991542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"335.2","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453834+0000","last_change":"2024-03-30T15:10:20.448772+0000","last_active":"2024-03-30T15:10:23.453834+0000","last_peered":"2024-03-30T15:10:23.453834+0000","last_clean":"2024-03-30T15:10:23.453834+0000","last_became_active":"2024-03-30T15:10:20.448480+0000","last_became_peered":"2024-03-30T15:10:20.448480+0000","last_unstale":"2024-03-30T15:10:23.453834+0000","last_undegraded":"2024-03-30T15:10:23.453834+0000","last_fullsized":"2024-03-30T15:10:23.453834+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:49.165848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"335.1","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442320+0000","last_change":"2024-03-30T15:10:20.451747+0000","last_active":"2024-03-30T15:10:23.442320+0000","last_peered":"2024-03-30T15:10:23.442320+0000","last_clean":"2024-03-30T15:10:23.442320+0000","last_became_active":"2024-03-30T15:10:20.451353+0000","last_became_peered":"2024-03-30T15:10:20.451353+0000","last_unstale":"2024-03-30T15:10:23.442320+0000","last_undegraded":"2024-03-30T15:10:23.442320+0000","last_fullsized":"2024-03-30T15:10:23.442320+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:20.618163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"335.0","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457153+0000","last_change":"2024-03-30T15:10:20.448123+0000","last_active":"2024-03-30T15:10:23.457153+0000","last_peered":"2024-03-30T15:10:23.457153+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.448123+0000","last_became_peered":"2024-03-30T15:10:20.448123+0000","last_unstale":"2024-03-30T15:10:23.457153+0000","last_undegraded":"2024-03-30T15:10:23.457153+0000","last_fullsized":"2024-03-30T15:10:20.437971+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.851992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"335.a","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457515+0000","last_change":"2024-03-30T15:10:20.442030+0000","last_active":"2024-03-30T15:10:23.457515+0000","last_peered":"2024-03-30T15:10:23.457515+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.442030+0000","last_became_peered":"2024-03-30T15:10:20.442030+0000","last_unstale":"2024-03-30T15:10:23.457515+0000","last_undegraded":"2024-03-30T15:10:23.457515+0000","last_fullsized":"2024-03-30T15:10:20.433639+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:48.235129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"322.7","version":"745'13","reported_seq":332,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459984+0000","last_change":"2024-03-30T15:10:09.408883+0000","last_active":"2024-03-30T15:10:23.459984+0000","last_peered":"2024-03-30T15:10:23.459984+0000","last_clean":"2024-03-30T15:10:23.459984+0000","last_became_active":"2024-03-30T15:08:14.380692+0000","last_became_peered":"2024-03-30T15:08:14.380692+0000","last_unstale":"2024-03-30T15:10:23.459984+0000","last_undegraded":"2024-03-30T15:10:23.459984+0000","last_fullsized":"2024-03-30T15:10:23.459984+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050114210000000003,"stat_sum":{"num_bytes":7,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"335.b","version":"0'0","reported_seq":16,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442684+0000","last_change":"2024-03-30T15:10:20.454917+0000","last_active":"2024-03-30T15:10:23.442684+0000","last_peered":"2024-03-30T15:10:23.442684+0000","last_clean":"2024-03-30T15:10:19.422492+0000","last_became_active":"2024-03-30T15:10:20.454917+0000","last_became_peered":"2024-03-30T15:10:20.454917+0000","last_unstale":"2024-03-30T15:10:23.442684+0000","last_undegraded":"2024-03-30T15:10:23.442684+0000","last_fullsized":"2024-03-30T15:10:20.438032+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:47.514426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"322.6","version":"668'5","reported_seq":324,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454445+0000","last_change":"2024-03-30T15:10:07.408967+0000","last_active":"2024-03-30T15:10:23.454445+0000","last_peered":"2024-03-30T15:10:23.454445+0000","last_clean":"2024-03-30T15:10:23.454445+0000","last_became_active":"2024-03-30T15:08:14.373744+0000","last_became_peered":"2024-03-30T15:08:14.373744+0000","last_unstale":"2024-03-30T15:10:23.454445+0000","last_undegraded":"2024-03-30T15:10:23.454445+0000","last_fullsized":"2024-03-30T15:10:23.454445+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060905400000000004,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"335.8","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442315+0000","last_change":"2024-03-30T15:10:20.450081+0000","last_active":"2024-03-30T15:10:23.442315+0000","last_peered":"2024-03-30T15:10:23.442315+0000","last_clean":"2024-03-30T15:10:23.442315+0000","last_became_active":"2024-03-30T15:10:20.449367+0000","last_became_peered":"2024-03-30T15:10:20.449367+0000","last_unstale":"2024-03-30T15:10:23.442315+0000","last_undegraded":"2024-03-30T15:10:23.442315+0000","last_fullsized":"2024-03-30T15:10:23.442315+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:27.505484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"322.5","version":"0'0","reported_seq":233,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.489547+0000","last_change":"2024-03-30T15:08:51.474906+0000","last_active":"2024-03-30T15:10:23.489547+0000","last_peered":"2024-03-30T15:10:23.489547+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"2024-03-30T15:08:51.474906+0000","last_became_peered":"2024-03-30T15:08:51.474906+0000","last_unstale":"2024-03-30T15:10:23.489547+0000","last_undegraded":"2024-03-30T15:10:23.489547+0000","last_fullsized":"2024-03-30T15:08:51.469091+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,4],"acting":[3,2147483647,4],"avail_no_missing":["4(2)","3(0)"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"335.9","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457470+0000","last_change":"2024-03-30T15:10:20.447583+0000","last_active":"2024-03-30T15:10:23.457470+0000","last_peered":"2024-03-30T15:10:23.457470+0000","last_clean":"2024-03-30T15:10:23.457470+0000","last_became_active":"2024-03-30T15:10:20.446854+0000","last_became_peered":"2024-03-30T15:10:20.446854+0000","last_unstale":"2024-03-30T15:10:23.457470+0000","last_undegraded":"2024-03-30T15:10:23.457470+0000","last_fullsized":"2024-03-30T15:10:23.457470+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:30.790084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"322.4","version":"704'10","reported_seq":328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489282+0000","last_change":"2024-03-30T15:10:07.398470+0000","last_active":"2024-03-30T15:10:23.489282+0000","last_peered":"2024-03-30T15:10:23.489282+0000","last_clean":"2024-03-30T15:10:23.489282+0000","last_became_active":"2024-03-30T15:08:14.374287+0000","last_became_peered":"2024-03-30T15:08:14.374287+0000","last_unstale":"2024-03-30T15:10:23.489282+0000","last_undegraded":"2024-03-30T15:10:23.489282+0000","last_fullsized":"2024-03-30T15:10:23.489282+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028193229999999999,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"335.e","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442298+0000","last_change":"2024-03-30T15:10:20.450441+0000","last_active":"2024-03-30T15:10:23.442298+0000","last_peered":"2024-03-30T15:10:23.442298+0000","last_clean":"2024-03-30T15:10:23.442298+0000","last_became_active":"2024-03-30T15:10:20.450172+0000","last_became_peered":"2024-03-30T15:10:20.450172+0000","last_unstale":"2024-03-30T15:10:23.442298+0000","last_undegraded":"2024-03-30T15:10:23.442298+0000","last_fullsized":"2024-03-30T15:10:23.442298+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.178075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"322.3","version":"724'4","reported_seq":241,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.454217+0000","last_change":"2024-03-30T15:09:05.377271+0000","last_active":"2024-03-30T15:10:23.454217+0000","last_peered":"2024-03-30T15:10:23.454217+0000","last_clean":"2024-03-30T15:08:13.347849+0000","last_became_active":"2024-03-30T15:08:51.492326+0000","last_became_peered":"2024-03-30T15:08:51.492326+0000","last_unstale":"2024-03-30T15:10:23.454217+0000","last_undegraded":"2024-03-30T15:09:04.393704+0000","last_fullsized":"2024-03-30T15:08:51.482966+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,1,6],"acting":[2147483647,1,6],"avail_no_missing":["1(1)","6(2)"],"object_location_counts":[{"shards":"1(1),6(2)","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"335.f","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443581+0000","last_change":"2024-03-30T15:10:20.455065+0000","last_active":"2024-03-30T15:10:23.443581+0000","last_peered":"2024-03-30T15:10:23.443581+0000","last_clean":"2024-03-30T15:10:23.443581+0000","last_became_active":"2024-03-30T15:10:20.454625+0000","last_became_peered":"2024-03-30T15:10:20.454625+0000","last_unstale":"2024-03-30T15:10:23.443581+0000","last_undegraded":"2024-03-30T15:10:23.443581+0000","last_fullsized":"2024-03-30T15:10:23.443581+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:46.710168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"322.2","version":"732'4","reported_seq":323,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.488978+0000","last_change":"2024-03-30T15:10:07.397471+0000","last_active":"2024-03-30T15:10:23.488978+0000","last_peered":"2024-03-30T15:10:23.488978+0000","last_clean":"2024-03-30T15:10:23.488978+0000","last_became_active":"2024-03-30T15:08:14.377722+0000","last_became_peered":"2024-03-30T15:08:14.377722+0000","last_unstale":"2024-03-30T15:10:23.488978+0000","last_undegraded":"2024-03-30T15:10:23.488978+0000","last_fullsized":"2024-03-30T15:10:23.488978+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030371529999999999,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"335.c","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489376+0000","last_change":"2024-03-30T15:10:20.453209+0000","last_active":"2024-03-30T15:10:23.489376+0000","last_peered":"2024-03-30T15:10:23.489376+0000","last_clean":"2024-03-30T15:10:23.489376+0000","last_became_active":"2024-03-30T15:10:20.452903+0000","last_became_peered":"2024-03-30T15:10:20.452903+0000","last_unstale":"2024-03-30T15:10:23.489376+0000","last_undegraded":"2024-03-30T15:10:23.489376+0000","last_fullsized":"2024-03-30T15:10:23.489376+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:43.613791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"322.1","version":"753'11","reported_seq":328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.488859+0000","last_change":"2024-03-30T15:10:07.395753+0000","last_active":"2024-03-30T15:10:23.488859+0000","last_peered":"2024-03-30T15:10:23.488859+0000","last_clean":"2024-03-30T15:10:23.488859+0000","last_became_active":"2024-03-30T15:08:14.375417+0000","last_became_peered":"2024-03-30T15:08:14.375417+0000","last_unstale":"2024-03-30T15:10:23.488859+0000","last_undegraded":"2024-03-30T15:10:23.488859+0000","last_fullsized":"2024-03-30T15:10:23.488859+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002065215,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"335.d","version":"0'0","reported_seq":19,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452366+0000","last_change":"2024-03-30T15:10:20.454821+0000","last_active":"2024-03-30T15:10:23.452366+0000","last_peered":"2024-03-30T15:10:23.452366+0000","last_clean":"2024-03-30T15:10:23.452366+0000","last_became_active":"2024-03-30T15:10:20.454534+0000","last_became_peered":"2024-03-30T15:10:20.454534+0000","last_unstale":"2024-03-30T15:10:23.452366+0000","last_undegraded":"2024-03-30T15:10:23.452366+0000","last_fullsized":"2024-03-30T15:10:23.452366+0000","mapping_epoch":757,"log_start":"0'0","ondisk_log_start":"0'0","created":757,"last_epoch_clean":758,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.422492+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.422492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:28.687192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"322.0","version":"761'23","reported_seq":344,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453648+0000","last_change":"2024-03-30T15:10:07.405836+0000","last_active":"2024-03-30T15:10:23.453648+0000","last_peered":"2024-03-30T15:10:23.453648+0000","last_clean":"2024-03-30T15:10:23.453648+0000","last_became_active":"2024-03-30T15:08:14.374433+0000","last_became_peered":"2024-03-30T15:08:14.374433+0000","last_unstale":"2024-03-30T15:10:23.453648+0000","last_undegraded":"2024-03-30T15:10:23.453648+0000","last_fullsized":"2024-03-30T15:10:23.453648+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":644,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:13.347849+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.347849+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000929969,"stat_sum":{"num_bytes":4000034,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":11,"num_write_kb":3915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"},{"start":"6","length":"1"}]},{"pgid":"42.15","version":"0'0","reported_seq":1328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442906+0000","last_change":"2024-03-30T15:09:32.851311+0000","last_active":"2024-03-30T15:10:23.442906+0000","last_peered":"2024-03-30T15:10:23.442906+0000","last_clean":"2024-03-30T15:10:23.442906+0000","last_became_active":"2024-03-30T14:57:56.252375+0000","last_became_peered":"2024-03-30T14:57:56.252375+0000","last_unstale":"2024-03-30T15:10:23.442906+0000","last_undegraded":"2024-03-30T15:10:23.442906+0000","last_fullsized":"2024-03-30T15:10:23.442906+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:32.851231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:09:32.851231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:54.516161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.14","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452319+0000","last_change":"2024-03-30T15:09:44.882024+0000","last_active":"2024-03-30T15:10:23.452319+0000","last_peered":"2024-03-30T15:10:23.452319+0000","last_clean":"2024-03-30T15:10:23.452319+0000","last_became_active":"2024-03-30T14:57:56.254162+0000","last_became_peered":"2024-03-30T14:57:56.254162+0000","last_unstale":"2024-03-30T15:10:23.452319+0000","last_undegraded":"2024-03-30T15:10:23.452319+0000","last_fullsized":"2024-03-30T15:10:23.452319+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:44.881866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:09:44.881866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:56.652482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.17","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.460040+0000","last_change":"2024-03-30T15:08:00.188376+0000","last_active":"2024-03-30T15:10:23.460040+0000","last_peered":"2024-03-30T15:10:23.460040+0000","last_clean":"2024-03-30T15:10:23.460040+0000","last_became_active":"2024-03-30T14:57:56.251710+0000","last_became_peered":"2024-03-30T14:57:56.251710+0000","last_unstale":"2024-03-30T15:10:23.460040+0000","last_undegraded":"2024-03-30T15:10:23.460040+0000","last_fullsized":"2024-03-30T15:10:23.460040+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.188301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.188301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.16","version":"0'0","reported_seq":1327,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454644+0000","last_change":"2024-03-30T15:08:51.488241+0000","last_active":"2024-03-30T15:10:23.454644+0000","last_peered":"2024-03-30T15:10:23.454644+0000","last_clean":"2024-03-30T15:08:49.464990+0000","last_became_active":"2024-03-30T15:08:51.488241+0000","last_became_peered":"2024-03-30T15:08:51.488241+0000","last_unstale":"2024-03-30T15:10:23.454644+0000","last_undegraded":"2024-03-30T15:10:23.454644+0000","last_fullsized":"2024-03-30T15:08:51.476762+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.895505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.895505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.412545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.11","version":"0'0","reported_seq":188,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441655+0000","last_change":"2024-03-30T15:08:51.477990+0000","last_active":"2024-03-30T15:10:23.441655+0000","last_peered":"2024-03-30T15:10:23.441655+0000","last_clean":"2024-03-30T14:57:53.916755+0000","last_became_active":"2024-03-30T15:08:51.477990+0000","last_became_peered":"2024-03-30T15:08:51.477990+0000","last_unstale":"2024-03-30T15:10:23.441655+0000","last_undegraded":"2024-03-30T15:10:23.441655+0000","last_fullsized":"2024-03-30T15:08:51.467608+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:27.558982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:05:27.558982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.10","version":"0'0","reported_seq":1341,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441561+0000","last_change":"2024-03-30T15:08:51.473776+0000","last_active":"2024-03-30T15:10:23.441561+0000","last_peered":"2024-03-30T15:10:23.441561+0000","last_clean":"2024-03-30T15:08:49.451114+0000","last_became_active":"2024-03-30T15:08:51.473776+0000","last_became_peered":"2024-03-30T15:08:51.473776+0000","last_unstale":"2024-03-30T15:10:23.441561+0000","last_undegraded":"2024-03-30T15:10:23.441561+0000","last_fullsized":"2024-03-30T15:08:51.467292+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.201691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.005887+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.201691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.13","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458153+0000","last_change":"2024-03-30T15:08:13.183528+0000","last_active":"2024-03-30T15:10:23.458153+0000","last_peered":"2024-03-30T15:10:23.458153+0000","last_clean":"2024-03-30T15:10:23.458153+0000","last_became_active":"2024-03-30T14:57:56.243845+0000","last_became_peered":"2024-03-30T14:57:56.243845+0000","last_unstale":"2024-03-30T15:10:23.458153+0000","last_undegraded":"2024-03-30T15:10:23.458153+0000","last_fullsized":"2024-03-30T15:10:23.458153+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:13.183373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:13.183373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.12","version":"0'0","reported_seq":1328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442505+0000","last_change":"2024-03-30T15:10:10.905455+0000","last_active":"2024-03-30T15:10:23.442505+0000","last_peered":"2024-03-30T15:10:23.442505+0000","last_clean":"2024-03-30T15:10:23.442505+0000","last_became_active":"2024-03-30T14:57:56.238861+0000","last_became_peered":"2024-03-30T14:57:56.238861+0000","last_unstale":"2024-03-30T15:10:23.442505+0000","last_undegraded":"2024-03-30T15:10:23.442505+0000","last_fullsized":"2024-03-30T15:10:23.442505+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:10.905287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:10.905287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:28.953743+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.1d","version":"0'0","reported_seq":1327,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490373+0000","last_change":"2024-03-30T15:10:13.500564+0000","last_active":"2024-03-30T15:10:23.490373+0000","last_peered":"2024-03-30T15:10:23.490373+0000","last_clean":"2024-03-30T15:10:23.490373+0000","last_became_active":"2024-03-30T14:57:56.239451+0000","last_became_peered":"2024-03-30T14:57:56.239451+0000","last_unstale":"2024-03-30T15:10:23.490373+0000","last_undegraded":"2024-03-30T15:10:23.490373+0000","last_fullsized":"2024-03-30T15:10:23.490373+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:13.500476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:13.500476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:22.401899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.1c","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454309+0000","last_change":"2024-03-30T15:08:51.492278+0000","last_active":"2024-03-30T15:10:23.454309+0000","last_peered":"2024-03-30T15:10:23.454309+0000","last_clean":"2024-03-30T15:08:49.463289+0000","last_became_active":"2024-03-30T15:08:51.492278+0000","last_became_peered":"2024-03-30T15:08:51.492278+0000","last_unstale":"2024-03-30T15:10:23.454309+0000","last_undegraded":"2024-03-30T15:10:23.454309+0000","last_fullsized":"2024-03-30T15:08:51.482896+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:42.144953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:43.330250+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.144953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1f","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.460035+0000","last_change":"2024-03-30T15:08:10.138813+0000","last_active":"2024-03-30T15:10:23.460035+0000","last_peered":"2024-03-30T15:10:23.460035+0000","last_clean":"2024-03-30T15:10:23.460035+0000","last_became_active":"2024-03-30T14:57:56.252622+0000","last_became_peered":"2024-03-30T14:57:56.252622+0000","last_unstale":"2024-03-30T15:10:23.460035+0000","last_undegraded":"2024-03-30T15:10:23.460035+0000","last_fullsized":"2024-03-30T15:10:23.460035+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.138657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.138657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1e","version":"0'0","reported_seq":1352,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442258+0000","last_change":"2024-03-30T15:10:21.137663+0000","last_active":"2024-03-30T15:10:23.442258+0000","last_peered":"2024-03-30T15:10:23.442258+0000","last_clean":"2024-03-30T15:10:23.442258+0000","last_became_active":"2024-03-30T14:57:56.244910+0000","last_became_peered":"2024-03-30T14:57:56.244910+0000","last_unstale":"2024-03-30T15:10:23.442258+0000","last_undegraded":"2024-03-30T15:10:23.442258+0000","last_fullsized":"2024-03-30T15:10:23.442258+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:21.137532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:21.137532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.538649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.19","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457540+0000","last_change":"2024-03-30T15:08:15.202644+0000","last_active":"2024-03-30T15:10:23.457540+0000","last_peered":"2024-03-30T15:10:23.457540+0000","last_clean":"2024-03-30T15:10:23.457540+0000","last_became_active":"2024-03-30T14:57:56.252045+0000","last_became_peered":"2024-03-30T14:57:56.252045+0000","last_unstale":"2024-03-30T15:10:23.457540+0000","last_undegraded":"2024-03-30T15:10:23.457540+0000","last_fullsized":"2024-03-30T15:10:23.457540+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:15.202555+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.202555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.18","version":"0'0","reported_seq":1332,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453592+0000","last_change":"2024-03-30T15:09:30.149342+0000","last_active":"2024-03-30T15:10:23.453592+0000","last_peered":"2024-03-30T15:10:23.453592+0000","last_clean":"2024-03-30T15:10:23.453592+0000","last_became_active":"2024-03-30T14:57:56.226822+0000","last_became_peered":"2024-03-30T14:57:56.226822+0000","last_unstale":"2024-03-30T15:10:23.453592+0000","last_undegraded":"2024-03-30T15:10:23.453592+0000","last_fullsized":"2024-03-30T15:10:23.453592+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:30.149261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:09:30.149261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:35.291462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.1b","version":"0'0","reported_seq":1350,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.441800+0000","last_change":"2024-03-30T15:10:11.168927+0000","last_active":"2024-03-30T15:10:23.441800+0000","last_peered":"2024-03-30T15:10:23.441800+0000","last_clean":"2024-03-30T15:10:23.441800+0000","last_became_active":"2024-03-30T14:57:56.236041+0000","last_became_peered":"2024-03-30T14:57:56.236041+0000","last_unstale":"2024-03-30T15:10:23.441800+0000","last_undegraded":"2024-03-30T15:10:23.441800+0000","last_fullsized":"2024-03-30T15:10:23.441800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:11.168853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:00.582709+0000","last_clean_scrub_stamp":"2024-03-30T15:10:11.168853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:38.053649+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.1a","version":"0'0","reported_seq":1350,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442991+0000","last_change":"2024-03-30T15:10:23.220004+0000","last_active":"2024-03-30T15:10:23.442991+0000","last_peered":"2024-03-30T15:10:23.442991+0000","last_clean":"2024-03-30T15:10:23.442991+0000","last_became_active":"2024-03-30T14:57:56.290707+0000","last_became_peered":"2024-03-30T14:57:56.290707+0000","last_unstale":"2024-03-30T15:10:23.442991+0000","last_undegraded":"2024-03-30T15:10:23.442991+0000","last_fullsized":"2024-03-30T15:10:23.442991+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:23.219920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:23.219920+0000","last_clean_scrub_stamp":"2024-03-30T15:10:23.219920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.104800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.5","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459107+0000","last_change":"2024-03-30T15:07:59.150468+0000","last_active":"2024-03-30T15:10:23.459107+0000","last_peered":"2024-03-30T15:10:23.459107+0000","last_clean":"2024-03-30T15:10:23.459107+0000","last_became_active":"2024-03-30T14:57:56.248136+0000","last_became_peered":"2024-03-30T14:57:56.248136+0000","last_unstale":"2024-03-30T15:10:23.459107+0000","last_undegraded":"2024-03-30T15:10:23.459107+0000","last_fullsized":"2024-03-30T15:10:23.459107+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:59.150361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:59.150361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.4","version":"0'0","reported_seq":1328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490053+0000","last_change":"2024-03-30T15:09:40.498296+0000","last_active":"2024-03-30T15:10:23.490053+0000","last_peered":"2024-03-30T15:10:23.490053+0000","last_clean":"2024-03-30T15:10:23.490053+0000","last_became_active":"2024-03-30T14:57:56.219327+0000","last_became_peered":"2024-03-30T14:57:56.219327+0000","last_unstale":"2024-03-30T15:10:23.490053+0000","last_undegraded":"2024-03-30T15:10:23.490053+0000","last_fullsized":"2024-03-30T15:10:23.490053+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:40.497821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:09:40.497821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:59.718864+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.7","version":"0'0","reported_seq":1331,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443497+0000","last_change":"2024-03-30T15:10:07.826982+0000","last_active":"2024-03-30T15:10:23.443497+0000","last_peered":"2024-03-30T15:10:23.443497+0000","last_clean":"2024-03-30T15:10:23.443497+0000","last_became_active":"2024-03-30T14:57:56.273787+0000","last_became_peered":"2024-03-30T14:57:56.273787+0000","last_unstale":"2024-03-30T15:10:23.443497+0000","last_undegraded":"2024-03-30T15:10:23.443497+0000","last_fullsized":"2024-03-30T15:10:23.443497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:07.826848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:07.826848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:29.647166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.6","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459569+0000","last_change":"2024-03-30T15:07:35.172101+0000","last_active":"2024-03-30T15:10:23.459569+0000","last_peered":"2024-03-30T15:10:23.459569+0000","last_clean":"2024-03-30T15:10:23.459569+0000","last_became_active":"2024-03-30T14:57:56.248849+0000","last_became_peered":"2024-03-30T14:57:56.248849+0000","last_unstale":"2024-03-30T15:10:23.459569+0000","last_undegraded":"2024-03-30T15:10:23.459569+0000","last_fullsized":"2024-03-30T15:10:23.459569+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.171990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:50.298282+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.171990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.1","version":"0'0","reported_seq":1332,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453620+0000","last_change":"2024-03-30T15:08:09.224705+0000","last_active":"2024-03-30T15:10:23.453620+0000","last_peered":"2024-03-30T15:10:23.453620+0000","last_clean":"2024-03-30T15:10:23.453620+0000","last_became_active":"2024-03-30T14:57:56.255267+0000","last_became_peered":"2024-03-30T14:57:56.255267+0000","last_unstale":"2024-03-30T15:10:23.453620+0000","last_undegraded":"2024-03-30T15:10:23.453620+0000","last_fullsized":"2024-03-30T15:10:23.453620+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.224499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:12.137432+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.224499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.0","version":"0'0","reported_seq":1331,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.456977+0000","last_change":"2024-03-30T15:10:11.852776+0000","last_active":"2024-03-30T15:10:23.456977+0000","last_peered":"2024-03-30T15:10:23.456977+0000","last_clean":"2024-03-30T15:10:23.456977+0000","last_became_active":"2024-03-30T14:57:56.242148+0000","last_became_peered":"2024-03-30T14:57:56.242148+0000","last_unstale":"2024-03-30T15:10:23.456977+0000","last_undegraded":"2024-03-30T15:10:23.456977+0000","last_fullsized":"2024-03-30T15:10:23.456977+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:11.852671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:11.852671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.851097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.3","version":"0'0","reported_seq":1329,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.452241+0000","last_change":"2024-03-30T15:08:51.489853+0000","last_active":"2024-03-30T15:10:23.452241+0000","last_peered":"2024-03-30T15:10:23.452241+0000","last_clean":"2024-03-30T15:08:49.468298+0000","last_became_active":"2024-03-30T15:08:51.489853+0000","last_became_peered":"2024-03-30T15:08:51.489853+0000","last_unstale":"2024-03-30T15:10:23.452241+0000","last_undegraded":"2024-03-30T15:10:23.452241+0000","last_fullsized":"2024-03-30T15:08:51.480895+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.551547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.551547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.2","version":"0'0","reported_seq":1346,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441732+0000","last_change":"2024-03-30T15:08:51.478314+0000","last_active":"2024-03-30T15:10:23.441732+0000","last_peered":"2024-03-30T15:10:23.441732+0000","last_clean":"2024-03-30T15:08:50.045058+0000","last_became_active":"2024-03-30T15:08:51.478314+0000","last_became_peered":"2024-03-30T15:08:51.478314+0000","last_unstale":"2024-03-30T15:10:23.441732+0000","last_undegraded":"2024-03-30T15:10:23.441732+0000","last_fullsized":"2024-03-30T15:08:51.467758+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:50.044945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:01:21.414672+0000","last_clean_scrub_stamp":"2024-03-30T15:08:50.044945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.d","version":"0'0","reported_seq":1326,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452452+0000","last_change":"2024-03-30T15:08:09.548782+0000","last_active":"2024-03-30T15:10:23.452452+0000","last_peered":"2024-03-30T15:10:23.452452+0000","last_clean":"2024-03-30T15:10:23.452452+0000","last_became_active":"2024-03-30T14:57:56.219425+0000","last_became_peered":"2024-03-30T14:57:56.219425+0000","last_unstale":"2024-03-30T15:10:23.452452+0000","last_undegraded":"2024-03-30T15:10:23.452452+0000","last_fullsized":"2024-03-30T15:10:23.452452+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:09.548687+0000","last_clean_scrub_stamp":"2024-03-30T15:08:09.548687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.c","version":"0'0","reported_seq":943,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454149+0000","last_change":"2024-03-30T15:10:03.139094+0000","last_active":"2024-03-30T15:10:23.454149+0000","last_peered":"2024-03-30T15:10:23.454149+0000","last_clean":"2024-03-30T15:10:23.454149+0000","last_became_active":"2024-03-30T15:10:03.138752+0000","last_became_peered":"2024-03-30T15:10:03.138752+0000","last_unstale":"2024-03-30T15:10:23.454149+0000","last_undegraded":"2024-03-30T15:10:23.454149+0000","last_fullsized":"2024-03-30T15:10:23.454149+0000","mapping_epoch":739,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":740,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:59.704217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:43.564123+0000","last_clean_scrub_stamp":"2024-03-30T15:08:59.704217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.f","version":"0'0","reported_seq":1320,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443868+0000","last_change":"2024-03-30T15:08:02.711973+0000","last_active":"2024-03-30T15:10:23.443868+0000","last_peered":"2024-03-30T15:10:23.443868+0000","last_clean":"2024-03-30T15:10:23.443868+0000","last_became_active":"2024-03-30T14:57:56.260098+0000","last_became_peered":"2024-03-30T14:57:56.260098+0000","last_unstale":"2024-03-30T15:10:23.443868+0000","last_undegraded":"2024-03-30T15:10:23.443868+0000","last_fullsized":"2024-03-30T15:10:23.443868+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.711881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.711881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.e","version":"0'0","reported_seq":1323,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.443245+0000","last_change":"2024-03-30T15:08:51.489126+0000","last_active":"2024-03-30T15:10:23.443245+0000","last_peered":"2024-03-30T15:10:23.443245+0000","last_clean":"2024-03-30T15:08:49.471647+0000","last_became_active":"2024-03-30T15:08:51.489126+0000","last_became_peered":"2024-03-30T15:08:51.489126+0000","last_unstale":"2024-03-30T15:10:23.443245+0000","last_undegraded":"2024-03-30T15:10:23.443245+0000","last_fullsized":"2024-03-30T15:08:51.479928+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:26.321030+0000","last_clean_scrub_stamp":"2024-03-30T15:07:26.321030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.9","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.451534+0000","last_change":"2024-03-30T15:10:16.899062+0000","last_active":"2024-03-30T15:10:23.451534+0000","last_peered":"2024-03-30T15:10:23.451534+0000","last_clean":"2024-03-30T15:10:23.451534+0000","last_became_active":"2024-03-30T14:57:56.255540+0000","last_became_peered":"2024-03-30T14:57:56.255540+0000","last_unstale":"2024-03-30T15:10:23.451534+0000","last_undegraded":"2024-03-30T15:10:23.451534+0000","last_fullsized":"2024-03-30T15:10:23.451534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:16.898897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:10:16.898897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:32.585262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.8","version":"0'0","reported_seq":1323,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441665+0000","last_change":"2024-03-30T15:08:51.477692+0000","last_active":"2024-03-30T15:10:23.441665+0000","last_peered":"2024-03-30T15:10:23.441665+0000","last_clean":"2024-03-30T15:08:49.468553+0000","last_became_active":"2024-03-30T15:08:51.477692+0000","last_became_peered":"2024-03-30T15:08:51.477692+0000","last_unstale":"2024-03-30T15:10:23.441665+0000","last_undegraded":"2024-03-30T15:10:23.441665+0000","last_fullsized":"2024-03-30T15:08:51.473244+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:56.238398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:06:56.238398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"42.b","version":"0'0","reported_seq":1328,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489253+0000","last_change":"2024-03-30T15:09:41.494972+0000","last_active":"2024-03-30T15:10:23.489253+0000","last_peered":"2024-03-30T15:10:23.489253+0000","last_clean":"2024-03-30T15:10:23.489253+0000","last_became_active":"2024-03-30T14:57:56.199048+0000","last_became_peered":"2024-03-30T14:57:56.199048+0000","last_unstale":"2024-03-30T15:10:23.489253+0000","last_undegraded":"2024-03-30T15:10:23.489253+0000","last_fullsized":"2024-03-30T15:10:23.489253+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:41.494887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:09:41.494887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:46.430445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"42.a","version":"0'0","reported_seq":1322,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457705+0000","last_change":"2024-03-30T15:07:58.173279+0000","last_active":"2024-03-30T15:10:23.457705+0000","last_peered":"2024-03-30T15:10:23.457705+0000","last_clean":"2024-03-30T15:10:23.457705+0000","last_became_active":"2024-03-30T14:57:56.227670+0000","last_became_peered":"2024-03-30T14:57:56.227670+0000","last_unstale":"2024-03-30T15:10:23.457705+0000","last_undegraded":"2024-03-30T15:10:23.457705+0000","last_fullsized":"2024-03-30T15:10:23.457705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.173193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:53.916755+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.173193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"86'128","reported_seq":1715,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.441700+0000","last_change":"2024-03-30T15:09:44.858207+0000","last_active":"2024-03-30T15:10:23.441700+0000","last_peered":"2024-03-30T15:10:23.441700+0000","last_clean":"2024-03-30T15:10:23.441700+0000","last_became_active":"2024-03-30T15:06:48.338292+0000","last_became_peered":"2024-03-30T15:06:48.338292+0000","last_unstale":"2024-03-30T15:10:23.441700+0000","last_undegraded":"2024-03-30T15:10:23.441700+0000","last_fullsized":"2024-03-30T15:10:23.441700+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:44.858115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:09:44.858115+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:48.215651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1340,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452326+0000","last_change":"2024-03-30T15:09:32.976473+0000","last_active":"2024-03-30T15:10:23.452326+0000","last_peered":"2024-03-30T15:10:23.452326+0000","last_clean":"2024-03-30T15:10:23.452326+0000","last_became_active":"2024-03-30T14:57:55.129332+0000","last_became_peered":"2024-03-30T14:57:55.129332+0000","last_unstale":"2024-03-30T15:10:23.452326+0000","last_undegraded":"2024-03-30T15:10:23.452326+0000","last_fullsized":"2024-03-30T15:10:23.452326+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:32.976369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:32.976369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:42.310010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1455,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489980+0000","last_change":"2024-03-30T15:08:06.185130+0000","last_active":"2024-03-30T15:10:23.489980+0000","last_peered":"2024-03-30T15:10:23.489980+0000","last_clean":"2024-03-30T15:10:23.489980+0000","last_became_active":"2024-03-30T14:54:39.329627+0000","last_became_peered":"2024-03-30T14:54:39.329627+0000","last_unstale":"2024-03-30T15:10:23.489980+0000","last_undegraded":"2024-03-30T15:10:23.489980+0000","last_fullsized":"2024-03-30T15:10:23.489980+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:08:06.185037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.185037+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":191,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.489488+0000","last_change":"2024-03-30T15:08:51.475344+0000","last_active":"2024-03-30T15:10:23.489488+0000","last_peered":"2024-03-30T15:10:23.489488+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.475344+0000","last_became_peered":"2024-03-30T15:08:51.475344+0000","last_unstale":"2024-03-30T15:10:23.489488+0000","last_undegraded":"2024-03-30T15:10:23.489488+0000","last_fullsized":"2024-03-30T15:08:51.468917+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:06:11.536298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.573846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"488'9","reported_seq":637,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452452+0000","last_change":"2024-03-30T15:09:29.992979+0000","last_active":"2024-03-30T15:10:23.452452+0000","last_peered":"2024-03-30T15:10:23.452452+0000","last_clean":"2024-03-30T15:10:23.452452+0000","last_became_active":"2024-03-30T15:05:06.356660+0000","last_became_peered":"2024-03-30T15:05:06.356660+0000","last_unstale":"2024-03-30T15:10:23.452452+0000","last_undegraded":"2024-03-30T15:10:23.452452+0000","last_fullsized":"2024-03-30T15:10:23.452452+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'9","last_scrub_stamp":"2024-03-30T15:09:29.992874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:09:29.992874+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:56.178958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1416,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.491291+0000","last_change":"2024-03-30T15:09:32.466309+0000","last_active":"2024-03-30T15:10:23.491291+0000","last_peered":"2024-03-30T15:10:23.491291+0000","last_clean":"2024-03-30T15:10:23.491291+0000","last_became_active":"2024-03-30T14:54:43.352166+0000","last_became_peered":"2024-03-30T14:54:43.352166+0000","last_unstale":"2024-03-30T15:10:23.491291+0000","last_undegraded":"2024-03-30T15:10:23.491291+0000","last_fullsized":"2024-03-30T15:10:23.491291+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:32.466204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:32.466204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:32.935461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":716,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.456834+0000","last_change":"2024-03-30T15:09:43.710661+0000","last_active":"2024-03-30T15:10:23.456834+0000","last_peered":"2024-03-30T15:10:23.456834+0000","last_clean":"2024-03-30T15:10:23.456834+0000","last_became_active":"2024-03-30T15:06:48.340482+0000","last_became_peered":"2024-03-30T15:06:48.340482+0000","last_unstale":"2024-03-30T15:10:23.456834+0000","last_undegraded":"2024-03-30T15:10:23.456834+0000","last_fullsized":"2024-03-30T15:10:23.456834+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:43.710551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:09:43.710551+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:10.203201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1340,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452578+0000","last_change":"2024-03-30T15:09:33.957898+0000","last_active":"2024-03-30T15:10:23.452578+0000","last_peered":"2024-03-30T15:10:23.452578+0000","last_clean":"2024-03-30T15:10:23.452578+0000","last_became_active":"2024-03-30T14:57:55.230661+0000","last_became_peered":"2024-03-30T14:57:55.230661+0000","last_unstale":"2024-03-30T15:10:23.452578+0000","last_undegraded":"2024-03-30T15:10:23.452578+0000","last_fullsized":"2024-03-30T15:10:23.452578+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:33.957802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:33.957802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:55.684374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":1482,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.491319+0000","last_change":"2024-03-30T15:09:38.505347+0000","last_active":"2024-03-30T15:10:23.491319+0000","last_peered":"2024-03-30T15:10:23.491319+0000","last_clean":"2024-03-30T15:10:23.491319+0000","last_became_active":"2024-03-30T14:54:39.334889+0000","last_became_peered":"2024-03-30T14:54:39.334889+0000","last_unstale":"2024-03-30T15:10:23.491319+0000","last_undegraded":"2024-03-30T15:10:23.491319+0000","last_fullsized":"2024-03-30T15:10:23.491319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:09:38.505260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:38.505260+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:47.122204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1442,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.451951+0000","last_change":"2024-03-30T15:09:38.003739+0000","last_active":"2024-03-30T15:10:23.451951+0000","last_peered":"2024-03-30T15:10:23.451951+0000","last_clean":"2024-03-30T15:10:23.451951+0000","last_became_active":"2024-03-30T14:54:37.325702+0000","last_became_peered":"2024-03-30T14:54:37.325702+0000","last_unstale":"2024-03-30T15:10:23.451951+0000","last_undegraded":"2024-03-30T15:10:23.451951+0000","last_fullsized":"2024-03-30T15:10:23.451951+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:38.003660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:09:38.003660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:42.757103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1435,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.455097+0000","last_change":"2024-03-30T15:08:51.487893+0000","last_active":"2024-03-30T15:10:23.455097+0000","last_peered":"2024-03-30T15:10:23.455097+0000","last_clean":"2024-03-30T15:08:49.463805+0000","last_became_active":"2024-03-30T15:08:51.487893+0000","last_became_peered":"2024-03-30T15:08:51.487893+0000","last_unstale":"2024-03-30T15:10:23.455097+0000","last_undegraded":"2024-03-30T15:10:23.455097+0000","last_fullsized":"2024-03-30T15:08:51.475182+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:19.878011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:07:19.878011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1445,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443706+0000","last_change":"2024-03-30T15:09:29.036457+0000","last_active":"2024-03-30T15:10:23.443706+0000","last_peered":"2024-03-30T15:10:23.443706+0000","last_clean":"2024-03-30T15:10:23.443706+0000","last_became_active":"2024-03-30T14:54:43.352334+0000","last_became_peered":"2024-03-30T14:54:43.352334+0000","last_unstale":"2024-03-30T15:10:23.443706+0000","last_undegraded":"2024-03-30T15:10:23.443706+0000","last_fullsized":"2024-03-30T15:10:23.443706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:29.036353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:29.036353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:33.171918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1341,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.455545+0000","last_change":"2024-03-30T15:08:51.495423+0000","last_active":"2024-03-30T15:10:23.455545+0000","last_peered":"2024-03-30T15:10:23.455545+0000","last_clean":"2024-03-30T15:08:49.464985+0000","last_became_active":"2024-03-30T15:08:51.495423+0000","last_became_peered":"2024-03-30T15:08:51.495423+0000","last_unstale":"2024-03-30T15:10:23.455545+0000","last_undegraded":"2024-03-30T15:10:23.455545+0000","last_fullsized":"2024-03-30T15:08:51.486644+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:05.112284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:05.112284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1468,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490534+0000","last_change":"2024-03-30T15:10:22.590481+0000","last_active":"2024-03-30T15:10:23.490534+0000","last_peered":"2024-03-30T15:10:23.490534+0000","last_clean":"2024-03-30T15:10:23.490534+0000","last_became_active":"2024-03-30T14:54:39.334792+0000","last_became_peered":"2024-03-30T14:54:39.334792+0000","last_unstale":"2024-03-30T15:10:23.490534+0000","last_undegraded":"2024-03-30T15:10:23.490534+0000","last_fullsized":"2024-03-30T15:10:23.490534+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:10:22.590358+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:10:22.590358+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:33.301130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1437,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489619+0000","last_change":"2024-03-30T15:10:17.607979+0000","last_active":"2024-03-30T15:10:23.489619+0000","last_peered":"2024-03-30T15:10:23.489619+0000","last_clean":"2024-03-30T15:10:23.489619+0000","last_became_active":"2024-03-30T14:54:37.474275+0000","last_became_peered":"2024-03-30T14:54:37.474275+0000","last_unstale":"2024-03-30T15:10:23.489619+0000","last_undegraded":"2024-03-30T15:10:23.489619+0000","last_fullsized":"2024-03-30T15:10:23.489619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:17.607877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:10:17.607877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:45.270561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1421,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.460639+0000","last_change":"2024-03-30T15:08:21.357777+0000","last_active":"2024-03-30T15:10:23.460639+0000","last_peered":"2024-03-30T15:10:23.460639+0000","last_clean":"2024-03-30T15:10:23.460639+0000","last_became_active":"2024-03-30T14:54:41.351632+0000","last_became_peered":"2024-03-30T14:54:41.351632+0000","last_unstale":"2024-03-30T15:10:23.460639+0000","last_undegraded":"2024-03-30T15:10:23.460639+0000","last_fullsized":"2024-03-30T15:10:23.460639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.357646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.357646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1335,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452443+0000","last_change":"2024-03-30T15:08:10.524135+0000","last_active":"2024-03-30T15:10:23.452443+0000","last_peered":"2024-03-30T15:10:23.452443+0000","last_clean":"2024-03-30T15:10:23.452443+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:10:23.452443+0000","last_undegraded":"2024-03-30T15:10:23.452443+0000","last_fullsized":"2024-03-30T15:10:23.452443+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.524046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.524046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":716,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457699+0000","last_change":"2024-03-30T15:09:40.697474+0000","last_active":"2024-03-30T15:10:23.457699+0000","last_peered":"2024-03-30T15:10:23.457699+0000","last_clean":"2024-03-30T15:10:23.457699+0000","last_became_active":"2024-03-30T15:06:48.342510+0000","last_became_peered":"2024-03-30T15:06:48.342510+0000","last_unstale":"2024-03-30T15:10:23.457699+0000","last_undegraded":"2024-03-30T15:10:23.457699+0000","last_fullsized":"2024-03-30T15:10:23.457699+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_clean_scrub_stamp":"2024-03-30T15:09:40.697341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:45.611751+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1338,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.456294+0000","last_change":"2024-03-30T15:08:06.129401+0000","last_active":"2024-03-30T15:10:23.456294+0000","last_peered":"2024-03-30T15:10:23.456294+0000","last_clean":"2024-03-30T15:10:23.456294+0000","last_became_active":"2024-03-30T14:57:54.084008+0000","last_became_peered":"2024-03-30T14:57:54.084008+0000","last_unstale":"2024-03-30T15:10:23.456294+0000","last_undegraded":"2024-03-30T15:10:23.456294+0000","last_fullsized":"2024-03-30T15:10:23.456294+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:06.129319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:08:06.129319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1478,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.456401+0000","last_change":"2024-03-30T15:07:28.992483+0000","last_active":"2024-03-30T15:10:23.456401+0000","last_peered":"2024-03-30T15:10:23.456401+0000","last_clean":"2024-03-30T15:10:23.456401+0000","last_became_active":"2024-03-30T14:54:39.338047+0000","last_became_peered":"2024-03-30T14:54:39.338047+0000","last_unstale":"2024-03-30T15:10:23.456401+0000","last_undegraded":"2024-03-30T15:10:23.456401+0000","last_fullsized":"2024-03-30T15:10:23.456401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:28.992348+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:05:26.068493+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.992348+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454738+0000","last_change":"2024-03-30T15:07:22.928759+0000","last_active":"2024-03-30T15:10:23.454738+0000","last_peered":"2024-03-30T15:10:23.454738+0000","last_clean":"2024-03-30T15:10:23.454738+0000","last_became_active":"2024-03-30T14:57:56.285906+0000","last_became_peered":"2024-03-30T14:57:56.285906+0000","last_unstale":"2024-03-30T15:10:23.454738+0000","last_undegraded":"2024-03-30T15:10:23.454738+0000","last_fullsized":"2024-03-30T15:10:23.454738+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:22.928677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:07:22.928677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"670'9","reported_seq":1622,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.490589+0000","last_change":"2024-03-30T15:08:51.491105+0000","last_active":"2024-03-30T15:10:23.490589+0000","last_peered":"2024-03-30T15:10:23.490589+0000","last_clean":"2024-03-30T15:08:49.462545+0000","last_became_active":"2024-03-30T15:08:51.491105+0000","last_became_peered":"2024-03-30T15:08:51.491105+0000","last_unstale":"2024-03-30T15:10:23.490589+0000","last_undegraded":"2024-03-30T15:08:51.469897+0000","last_fullsized":"2024-03-30T15:08:51.469742+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:15.143526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.143526+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1412,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.458441+0000","last_change":"2024-03-30T15:08:51.483889+0000","last_active":"2024-03-30T15:10:23.458441+0000","last_peered":"2024-03-30T15:10:23.458441+0000","last_clean":"2024-03-30T15:08:49.476400+0000","last_became_active":"2024-03-30T15:08:51.483889+0000","last_became_peered":"2024-03-30T15:08:51.483889+0000","last_unstale":"2024-03-30T15:10:23.458441+0000","last_undegraded":"2024-03-30T15:10:23.458441+0000","last_fullsized":"2024-03-30T15:08:51.480434+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:57.951208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:06:57.951208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":716,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490028+0000","last_change":"2024-03-30T15:09:01.410926+0000","last_active":"2024-03-30T15:10:23.490028+0000","last_peered":"2024-03-30T15:10:23.490028+0000","last_clean":"2024-03-30T15:10:23.490028+0000","last_became_active":"2024-03-30T15:06:48.341420+0000","last_became_peered":"2024-03-30T15:06:48.341420+0000","last_unstale":"2024-03-30T15:10:23.490028+0000","last_undegraded":"2024-03-30T15:10:23.490028+0000","last_fullsized":"2024-03-30T15:10:23.490028+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:01.410849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:09:01.410849+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.307157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1338,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442626+0000","last_change":"2024-03-30T15:08:51.479419+0000","last_active":"2024-03-30T15:10:23.442626+0000","last_peered":"2024-03-30T15:10:23.442626+0000","last_clean":"2024-03-30T15:08:49.451389+0000","last_became_active":"2024-03-30T15:08:51.479419+0000","last_became_peered":"2024-03-30T15:08:51.479419+0000","last_unstale":"2024-03-30T15:10:23.442626+0000","last_undegraded":"2024-03-30T15:10:23.442626+0000","last_fullsized":"2024-03-30T15:08:51.467835+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:07:36.165947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1421,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489246+0000","last_change":"2024-03-30T15:07:57.209970+0000","last_active":"2024-03-30T15:10:23.489246+0000","last_peered":"2024-03-30T15:10:23.489246+0000","last_clean":"2024-03-30T15:10:23.489246+0000","last_became_active":"2024-03-30T14:54:41.354550+0000","last_became_peered":"2024-03-30T14:54:41.354550+0000","last_unstale":"2024-03-30T15:10:23.489246+0000","last_undegraded":"2024-03-30T15:10:23.489246+0000","last_fullsized":"2024-03-30T15:10:23.489246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.209884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.209884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1441,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.442389+0000","last_change":"2024-03-30T15:08:51.484538+0000","last_active":"2024-03-30T15:10:23.442389+0000","last_peered":"2024-03-30T15:10:23.442389+0000","last_clean":"2024-03-30T15:08:49.469863+0000","last_became_active":"2024-03-30T15:08:51.484538+0000","last_became_peered":"2024-03-30T15:08:51.484538+0000","last_unstale":"2024-03-30T15:10:23.442389+0000","last_undegraded":"2024-03-30T15:08:51.474193+0000","last_fullsized":"2024-03-30T15:08:51.474077+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:08:10.679491+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.679491+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1406,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.443601+0000","last_change":"2024-03-30T15:08:51.489487+0000","last_active":"2024-03-30T15:10:23.443601+0000","last_peered":"2024-03-30T15:10:23.443601+0000","last_clean":"2024-03-30T15:08:49.470027+0000","last_became_active":"2024-03-30T15:08:51.489487+0000","last_became_peered":"2024-03-30T15:08:51.489487+0000","last_unstale":"2024-03-30T15:10:23.443601+0000","last_undegraded":"2024-03-30T15:08:51.480978+0000","last_fullsized":"2024-03-30T15:08:51.480859+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:08:15.656821+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:08:15.656821+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1417,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489333+0000","last_change":"2024-03-30T15:08:10.175409+0000","last_active":"2024-03-30T15:10:23.489333+0000","last_peered":"2024-03-30T15:10:23.489333+0000","last_clean":"2024-03-30T15:10:23.489333+0000","last_became_active":"2024-03-30T14:54:43.352422+0000","last_became_peered":"2024-03-30T14:54:43.352422+0000","last_unstale":"2024-03-30T15:10:23.489333+0000","last_undegraded":"2024-03-30T15:10:23.489333+0000","last_fullsized":"2024-03-30T15:10:23.489333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.175319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":716,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458345+0000","last_change":"2024-03-30T15:09:36.693004+0000","last_active":"2024-03-30T15:10:23.458345+0000","last_peered":"2024-03-30T15:10:23.458345+0000","last_clean":"2024-03-30T15:10:23.458345+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:10:23.458345+0000","last_undegraded":"2024-03-30T15:10:23.458345+0000","last_fullsized":"2024-03-30T15:10:23.458345+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:36.692860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:09:36.692860+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:04.843335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1361,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.441942+0000","last_change":"2024-03-30T15:10:08.126978+0000","last_active":"2024-03-30T15:10:23.441942+0000","last_peered":"2024-03-30T15:10:23.441942+0000","last_clean":"2024-03-30T15:10:23.441942+0000","last_became_active":"2024-03-30T14:57:50.293085+0000","last_became_peered":"2024-03-30T14:57:50.293085+0000","last_unstale":"2024-03-30T15:10:23.441942+0000","last_undegraded":"2024-03-30T15:10:23.441942+0000","last_fullsized":"2024-03-30T15:10:23.441942+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:08.126898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:10:08.126898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:15.452160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"116'9","reported_seq":1509,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454058+0000","last_change":"2024-03-30T15:08:55.121752+0000","last_active":"2024-03-30T15:10:23.454058+0000","last_peered":"2024-03-30T15:10:23.454058+0000","last_clean":"2024-03-30T15:10:23.454058+0000","last_became_active":"2024-03-30T14:57:56.260304+0000","last_became_peered":"2024-03-30T14:57:56.260304+0000","last_unstale":"2024-03-30T15:10:23.454058+0000","last_undegraded":"2024-03-30T15:10:23.454058+0000","last_fullsized":"2024-03-30T15:10:23.454058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:08:55.121636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:55.121636+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"475'28","reported_seq":1475,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489968+0000","last_change":"2024-03-30T15:08:55.428333+0000","last_active":"2024-03-30T15:10:23.489968+0000","last_peered":"2024-03-30T15:10:23.489968+0000","last_clean":"2024-03-30T15:10:23.489968+0000","last_became_active":"2024-03-30T14:54:39.326329+0000","last_became_peered":"2024-03-30T14:54:39.326329+0000","last_unstale":"2024-03-30T15:10:23.489968+0000","last_undegraded":"2024-03-30T15:10:23.489968+0000","last_fullsized":"2024-03-30T15:10:23.489968+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:08:55.428215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:55.428215+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":356,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.455081+0000","last_change":"2024-03-30T15:08:51.488286+0000","last_active":"2024-03-30T15:10:23.455081+0000","last_peered":"2024-03-30T15:10:23.455081+0000","last_clean":"2024-03-30T14:58:25.167329+0000","last_became_active":"2024-03-30T15:08:51.488286+0000","last_became_peered":"2024-03-30T15:08:51.488286+0000","last_unstale":"2024-03-30T15:10:23.455081+0000","last_undegraded":"2024-03-30T15:08:51.476424+0000","last_fullsized":"2024-03-30T15:08:51.476303+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1425,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442732+0000","last_change":"2024-03-30T15:09:35.837457+0000","last_active":"2024-03-30T15:10:23.442732+0000","last_peered":"2024-03-30T15:10:23.442732+0000","last_clean":"2024-03-30T15:10:23.442732+0000","last_became_active":"2024-03-30T14:54:43.350677+0000","last_became_peered":"2024-03-30T14:54:43.350677+0000","last_unstale":"2024-03-30T15:10:23.442732+0000","last_undegraded":"2024-03-30T15:10:23.442732+0000","last_fullsized":"2024-03-30T15:10:23.442732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:35.837355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:35.837355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:42.195267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":719,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453170+0000","last_change":"2024-03-30T15:08:53.885921+0000","last_active":"2024-03-30T15:10:23.453170+0000","last_peered":"2024-03-30T15:10:23.453170+0000","last_clean":"2024-03-30T15:10:23.453170+0000","last_became_active":"2024-03-30T15:06:48.342151+0000","last_became_peered":"2024-03-30T15:06:48.342151+0000","last_unstale":"2024-03-30T15:10:23.453170+0000","last_undegraded":"2024-03-30T15:10:23.453170+0000","last_fullsized":"2024-03-30T15:10:23.453170+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:08:53.885806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:08:53.885806+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1345,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441740+0000","last_change":"2024-03-30T15:08:51.471981+0000","last_active":"2024-03-30T15:10:23.441740+0000","last_peered":"2024-03-30T15:10:23.441740+0000","last_clean":"2024-03-30T15:08:49.451859+0000","last_became_active":"2024-03-30T15:08:51.471981+0000","last_became_peered":"2024-03-30T15:08:51.471981+0000","last_unstale":"2024-03-30T15:10:23.441740+0000","last_undegraded":"2024-03-30T15:10:23.441740+0000","last_fullsized":"2024-03-30T15:08:51.468068+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:25.015600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.015600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"670'9","reported_seq":230,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.452524+0000","last_change":"2024-03-30T15:08:51.490203+0000","last_active":"2024-03-30T15:10:23.452524+0000","last_peered":"2024-03-30T15:10:23.452524+0000","last_clean":"2024-03-30T14:54:44.512143+0000","last_became_active":"2024-03-30T15:08:51.490203+0000","last_became_peered":"2024-03-30T15:08:51.490203+0000","last_unstale":"2024-03-30T15:10:23.452524+0000","last_undegraded":"2024-03-30T15:08:51.485526+0000","last_fullsized":"2024-03-30T15:08:51.485385+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:05:16.605990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:05:16.605990+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1472,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454279+0000","last_change":"2024-03-30T15:08:01.102530+0000","last_active":"2024-03-30T15:10:23.454279+0000","last_peered":"2024-03-30T15:10:23.454279+0000","last_clean":"2024-03-30T15:10:23.454279+0000","last_became_active":"2024-03-30T14:54:39.336081+0000","last_became_peered":"2024-03-30T14:54:39.336081+0000","last_unstale":"2024-03-30T15:10:23.454279+0000","last_undegraded":"2024-03-30T15:10:23.454279+0000","last_fullsized":"2024-03-30T15:10:23.454279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:08:01.102431+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.102431+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1438,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459279+0000","last_change":"2024-03-30T15:10:08.782008+0000","last_active":"2024-03-30T15:10:23.459279+0000","last_peered":"2024-03-30T15:10:23.459279+0000","last_clean":"2024-03-30T15:10:23.459279+0000","last_became_active":"2024-03-30T14:54:37.474394+0000","last_became_peered":"2024-03-30T14:54:37.474394+0000","last_unstale":"2024-03-30T15:10:23.459279+0000","last_undegraded":"2024-03-30T15:10:23.459279+0000","last_fullsized":"2024-03-30T15:10:23.459279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:08.781830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:10:08.781830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:37.266019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":188,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442264+0000","last_change":"2024-03-30T15:08:51.479067+0000","last_active":"2024-03-30T15:10:23.442264+0000","last_peered":"2024-03-30T15:10:23.442264+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.479067+0000","last_became_peered":"2024-03-30T15:08:51.479067+0000","last_unstale":"2024-03-30T15:10:23.442264+0000","last_undegraded":"2024-03-30T15:10:23.442264+0000","last_fullsized":"2024-03-30T15:08:51.467645+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:10.553985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:10.553985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":714,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.452605+0000","last_change":"2024-03-30T15:08:51.488541+0000","last_active":"2024-03-30T15:10:23.452605+0000","last_peered":"2024-03-30T15:10:23.452605+0000","last_clean":"2024-03-30T15:08:49.470182+0000","last_became_active":"2024-03-30T15:08:51.488541+0000","last_became_peered":"2024-03-30T15:08:51.488541+0000","last_unstale":"2024-03-30T15:10:23.452605+0000","last_undegraded":"2024-03-30T15:10:23.452605+0000","last_fullsized":"2024-03-30T15:08:51.476551+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.409455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.409455+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1351,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442564+0000","last_change":"2024-03-30T15:10:12.136154+0000","last_active":"2024-03-30T15:10:23.442564+0000","last_peered":"2024-03-30T15:10:23.442564+0000","last_clean":"2024-03-30T15:10:23.442564+0000","last_became_active":"2024-03-30T14:57:56.290391+0000","last_became_peered":"2024-03-30T14:57:56.290391+0000","last_unstale":"2024-03-30T15:10:23.442564+0000","last_undegraded":"2024-03-30T15:10:23.442564+0000","last_fullsized":"2024-03-30T15:10:23.442564+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:12.136071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:10:12.136071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:19.819810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.452259+0000","last_change":"2024-03-30T15:08:51.489751+0000","last_active":"2024-03-30T15:10:23.452259+0000","last_peered":"2024-03-30T15:10:23.452259+0000","last_clean":"2024-03-30T15:08:49.467487+0000","last_became_active":"2024-03-30T15:08:51.489751+0000","last_became_peered":"2024-03-30T15:08:51.489751+0000","last_unstale":"2024-03-30T15:10:23.452259+0000","last_undegraded":"2024-03-30T15:10:23.452259+0000","last_fullsized":"2024-03-30T15:08:51.479930+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:06:54.318005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1092,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.457866+0000","last_change":"2024-03-30T15:08:51.491420+0000","last_active":"2024-03-30T15:10:23.457866+0000","last_peered":"2024-03-30T15:10:23.457866+0000","last_clean":"2024-03-30T15:04:44.987512+0000","last_became_active":"2024-03-30T15:08:51.491420+0000","last_became_peered":"2024-03-30T15:08:51.491420+0000","last_unstale":"2024-03-30T15:10:23.457866+0000","last_undegraded":"2024-03-30T15:08:51.478157+0000","last_fullsized":"2024-03-30T15:08:51.478034+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1449,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454942+0000","last_change":"2024-03-30T15:10:11.361840+0000","last_active":"2024-03-30T15:10:23.454942+0000","last_peered":"2024-03-30T15:10:23.454942+0000","last_clean":"2024-03-30T15:10:23.454942+0000","last_became_active":"2024-03-30T14:54:37.474042+0000","last_became_peered":"2024-03-30T14:54:37.474042+0000","last_unstale":"2024-03-30T15:10:23.454942+0000","last_undegraded":"2024-03-30T15:10:23.454942+0000","last_fullsized":"2024-03-30T15:10:23.454942+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:11.361763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:10:11.361763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:32.059928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1437,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453810+0000","last_change":"2024-03-30T15:09:46.390181+0000","last_active":"2024-03-30T15:10:23.453810+0000","last_peered":"2024-03-30T15:10:23.453810+0000","last_clean":"2024-03-30T15:10:23.453810+0000","last_became_active":"2024-03-30T14:54:43.352777+0000","last_became_peered":"2024-03-30T14:54:43.352777+0000","last_unstale":"2024-03-30T15:10:23.453810+0000","last_undegraded":"2024-03-30T15:10:23.453810+0000","last_fullsized":"2024-03-30T15:10:23.453810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_clean_scrub_stamp":"2024-03-30T15:09:46.390100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:10.859158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":718,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454281+0000","last_change":"2024-03-30T15:09:04.195801+0000","last_active":"2024-03-30T15:10:23.454281+0000","last_peered":"2024-03-30T15:10:23.454281+0000","last_clean":"2024-03-30T15:10:23.454281+0000","last_became_active":"2024-03-30T15:06:48.339258+0000","last_became_peered":"2024-03-30T15:06:48.339258+0000","last_unstale":"2024-03-30T15:10:23.454281+0000","last_undegraded":"2024-03-30T15:10:23.454281+0000","last_fullsized":"2024-03-30T15:10:23.454281+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:09:04.195717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:09:04.195717+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":191,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.491393+0000","last_change":"2024-03-30T15:08:51.490697+0000","last_active":"2024-03-30T15:10:23.491393+0000","last_peered":"2024-03-30T15:10:23.491393+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.490697+0000","last_became_peered":"2024-03-30T15:08:51.490697+0000","last_unstale":"2024-03-30T15:10:23.491393+0000","last_undegraded":"2024-03-30T15:10:23.491393+0000","last_fullsized":"2024-03-30T15:08:51.469974+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:17.574991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:17.574991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.573838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1425,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.460678+0000","last_change":"2024-03-30T15:08:51.490874+0000","last_active":"2024-03-30T15:10:23.460678+0000","last_peered":"2024-03-30T15:10:23.460678+0000","last_clean":"2024-03-30T15:08:49.478492+0000","last_became_active":"2024-03-30T15:08:51.490874+0000","last_became_peered":"2024-03-30T15:08:51.490874+0000","last_unstale":"2024-03-30T15:10:23.460678+0000","last_undegraded":"2024-03-30T15:10:23.460678+0000","last_fullsized":"2024-03-30T15:08:51.485831+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:52.973770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:52.973770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1077,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.455610+0000","last_change":"2024-03-30T15:08:51.482555+0000","last_active":"2024-03-30T15:10:23.455610+0000","last_peered":"2024-03-30T15:10:23.455610+0000","last_clean":"2024-03-30T15:04:44.947244+0000","last_became_active":"2024-03-30T15:08:51.482555+0000","last_became_peered":"2024-03-30T15:08:51.482555+0000","last_unstale":"2024-03-30T15:10:23.455610+0000","last_undegraded":"2024-03-30T15:08:51.477512+0000","last_fullsized":"2024-03-30T15:08:51.477400+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2383,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.442039+0000","last_change":"2024-03-30T15:08:51.485260+0000","last_active":"2024-03-30T15:10:23.442039+0000","last_peered":"2024-03-30T15:10:23.442039+0000","last_clean":"2024-03-30T15:08:50.154417+0000","last_became_active":"2024-03-30T15:08:51.485260+0000","last_became_peered":"2024-03-30T15:08:51.485260+0000","last_unstale":"2024-03-30T15:10:23.442039+0000","last_undegraded":"2024-03-30T15:08:51.473685+0000","last_fullsized":"2024-03-30T15:08:51.473574+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:07:34.264815+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.264815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":937,"num_read_kb":932,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.489358+0000","last_change":"2024-03-30T15:08:51.483102+0000","last_active":"2024-03-30T15:10:23.489358+0000","last_peered":"2024-03-30T15:10:23.489358+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.483102+0000","last_became_peered":"2024-03-30T15:08:51.483102+0000","last_unstale":"2024-03-30T15:10:23.489358+0000","last_undegraded":"2024-03-30T15:10:23.489358+0000","last_fullsized":"2024-03-30T15:08:51.468057+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:29.592039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.592039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.587746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454746+0000","last_change":"2024-03-30T15:09:01.200193+0000","last_active":"2024-03-30T15:10:23.454746+0000","last_peered":"2024-03-30T15:10:23.454746+0000","last_clean":"2024-03-30T15:10:23.454746+0000","last_became_active":"2024-03-30T14:54:43.357415+0000","last_became_peered":"2024-03-30T14:54:43.357415+0000","last_unstale":"2024-03-30T15:10:23.454746+0000","last_undegraded":"2024-03-30T15:10:23.454746+0000","last_fullsized":"2024-03-30T15:10:23.454746+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:01.200051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:09:01.200051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1333,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443273+0000","last_change":"2024-03-30T15:10:17.979712+0000","last_active":"2024-03-30T15:10:23.443273+0000","last_peered":"2024-03-30T15:10:23.443273+0000","last_clean":"2024-03-30T15:10:23.443273+0000","last_became_active":"2024-03-30T14:57:55.128983+0000","last_became_peered":"2024-03-30T14:57:55.128983+0000","last_unstale":"2024-03-30T15:10:23.443273+0000","last_undegraded":"2024-03-30T15:10:23.443273+0000","last_fullsized":"2024-03-30T15:10:23.443273+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:17.979617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:10:17.979617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:35.510265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443180+0000","last_change":"2024-03-30T15:10:22.024472+0000","last_active":"2024-03-30T15:10:23.443180+0000","last_peered":"2024-03-30T15:10:23.443180+0000","last_clean":"2024-03-30T15:10:23.443180+0000","last_became_active":"2024-03-30T14:54:41.350025+0000","last_became_peered":"2024-03-30T14:54:41.350025+0000","last_unstale":"2024-03-30T15:10:23.443180+0000","last_undegraded":"2024-03-30T15:10:23.443180+0000","last_fullsized":"2024-03-30T15:10:23.443180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:22.024395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:10:22.024395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:44.190564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"475'28","reported_seq":1668,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453026+0000","last_change":"2024-03-30T15:08:42.905186+0000","last_active":"2024-03-30T15:10:23.453026+0000","last_peered":"2024-03-30T15:10:23.453026+0000","last_clean":"2024-03-30T15:10:23.453026+0000","last_became_active":"2024-03-30T14:54:39.642877+0000","last_became_peered":"2024-03-30T14:54:39.642877+0000","last_unstale":"2024-03-30T15:10:23.453026+0000","last_undegraded":"2024-03-30T15:10:23.453026+0000","last_fullsized":"2024-03-30T15:10:23.453026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'28","last_scrub_stamp":"2024-03-30T15:08:42.905060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.905060+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1332,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458061+0000","last_change":"2024-03-30T15:10:07.759400+0000","last_active":"2024-03-30T15:10:23.458061+0000","last_peered":"2024-03-30T15:10:23.458061+0000","last_clean":"2024-03-30T15:10:23.458061+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:10:23.458061+0000","last_undegraded":"2024-03-30T15:10:23.458061+0000","last_fullsized":"2024-03-30T15:10:23.458061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:07.759283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:10:07.759283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:30.726554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":1701,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442013+0000","last_change":"2024-03-30T15:08:51.489179+0000","last_active":"2024-03-30T15:10:23.442013+0000","last_peered":"2024-03-30T15:10:23.442013+0000","last_clean":"2024-03-30T15:08:49.468687+0000","last_became_active":"2024-03-30T15:08:51.489179+0000","last_became_peered":"2024-03-30T15:08:51.489179+0000","last_unstale":"2024-03-30T15:10:23.442013+0000","last_undegraded":"2024-03-30T15:10:23.442013+0000","last_fullsized":"2024-03-30T15:08:51.473612+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:45.099372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:45.099372+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1418,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.460164+0000","last_change":"2024-03-30T15:09:03.414486+0000","last_active":"2024-03-30T15:10:23.460164+0000","last_peered":"2024-03-30T15:10:23.460164+0000","last_clean":"2024-03-30T15:10:23.460164+0000","last_became_active":"2024-03-30T14:54:43.356410+0000","last_became_peered":"2024-03-30T14:54:43.356410+0000","last_unstale":"2024-03-30T15:10:23.460164+0000","last_undegraded":"2024-03-30T15:10:23.460164+0000","last_fullsized":"2024-03-30T15:10:23.460164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:03.414376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:03.414376+0000","last_clean_scrub_stamp":"2024-03-30T15:09:03.414376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":1718,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441993+0000","last_change":"2024-03-30T15:08:51.487084+0000","last_active":"2024-03-30T15:10:23.441993+0000","last_peered":"2024-03-30T15:10:23.441993+0000","last_clean":"2024-03-30T15:08:49.467404+0000","last_became_active":"2024-03-30T15:08:51.487084+0000","last_became_peered":"2024-03-30T15:08:51.487084+0000","last_unstale":"2024-03-30T15:10:23.441993+0000","last_undegraded":"2024-03-30T15:10:23.441993+0000","last_fullsized":"2024-03-30T15:08:51.469963+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:07:56.659850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:07:56.659850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1339,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453367+0000","last_change":"2024-03-30T15:08:50.920253+0000","last_active":"2024-03-30T15:10:23.453367+0000","last_peered":"2024-03-30T15:10:23.453367+0000","last_clean":"2024-03-30T15:10:23.453367+0000","last_became_active":"2024-03-30T14:57:55.239330+0000","last_became_peered":"2024-03-30T14:57:55.239330+0000","last_unstale":"2024-03-30T15:10:23.453367+0000","last_undegraded":"2024-03-30T15:10:23.453367+0000","last_fullsized":"2024-03-30T15:10:23.453367+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:50.920160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:50.920160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1450,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443036+0000","last_change":"2024-03-30T15:09:40.140919+0000","last_active":"2024-03-30T15:10:23.443036+0000","last_peered":"2024-03-30T15:10:23.443036+0000","last_clean":"2024-03-30T15:10:23.443036+0000","last_became_active":"2024-03-30T14:54:41.342752+0000","last_became_peered":"2024-03-30T14:54:41.342752+0000","last_unstale":"2024-03-30T15:10:23.443036+0000","last_undegraded":"2024-03-30T15:10:23.443036+0000","last_fullsized":"2024-03-30T15:10:23.443036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:40.140796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:09:40.140796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:05.024698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1478,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.456298+0000","last_change":"2024-03-30T15:08:51.494613+0000","last_active":"2024-03-30T15:10:23.456298+0000","last_peered":"2024-03-30T15:10:23.456298+0000","last_clean":"2024-03-30T15:08:49.465077+0000","last_became_active":"2024-03-30T15:08:51.494613+0000","last_became_peered":"2024-03-30T15:08:51.494613+0000","last_unstale":"2024-03-30T15:10:23.456298+0000","last_undegraded":"2024-03-30T15:08:51.486659+0000","last_fullsized":"2024-03-30T15:08:51.486535+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:06:44.835068+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.835068+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.412530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1462,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442113+0000","last_change":"2024-03-30T15:08:51.471872+0000","last_active":"2024-03-30T15:10:23.442113+0000","last_peered":"2024-03-30T15:10:23.442113+0000","last_clean":"2024-03-30T15:08:49.452028+0000","last_became_active":"2024-03-30T15:08:51.471872+0000","last_became_peered":"2024-03-30T15:08:51.471872+0000","last_unstale":"2024-03-30T15:10:23.442113+0000","last_undegraded":"2024-03-30T15:10:23.442113+0000","last_fullsized":"2024-03-30T15:08:51.468133+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:45.129264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:08:45.129264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1449,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454200+0000","last_change":"2024-03-30T15:09:47.416652+0000","last_active":"2024-03-30T15:10:23.454200+0000","last_peered":"2024-03-30T15:10:23.454200+0000","last_clean":"2024-03-30T15:10:23.454200+0000","last_became_active":"2024-03-30T14:54:37.474384+0000","last_became_peered":"2024-03-30T14:54:37.474384+0000","last_unstale":"2024-03-30T15:10:23.454200+0000","last_undegraded":"2024-03-30T15:10:23.454200+0000","last_fullsized":"2024-03-30T15:10:23.454200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:47.416580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:09:47.416580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:56.976950+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1433,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443437+0000","last_change":"2024-03-30T15:09:31.891613+0000","last_active":"2024-03-30T15:10:23.443437+0000","last_peered":"2024-03-30T15:10:23.443437+0000","last_clean":"2024-03-30T15:10:23.443437+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:10:23.443437+0000","last_undegraded":"2024-03-30T15:10:23.443437+0000","last_fullsized":"2024-03-30T15:10:23.443437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:31.891500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:31.891500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:42.647544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":711,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457351+0000","last_change":"2024-03-30T15:08:51.479202+0000","last_active":"2024-03-30T15:10:23.457351+0000","last_peered":"2024-03-30T15:10:23.457351+0000","last_clean":"2024-03-30T15:08:49.475152+0000","last_became_active":"2024-03-30T15:08:51.479202+0000","last_became_peered":"2024-03-30T15:08:51.479202+0000","last_unstale":"2024-03-30T15:10:23.457351+0000","last_undegraded":"2024-03-30T15:10:23.457351+0000","last_fullsized":"2024-03-30T15:08:51.471169+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:06:48.962124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.962124+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":192,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.458543+0000","last_change":"2024-03-30T15:08:51.483455+0000","last_active":"2024-03-30T15:10:23.458543+0000","last_peered":"2024-03-30T15:10:23.458543+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.483455+0000","last_became_peered":"2024-03-30T15:08:51.483455+0000","last_unstale":"2024-03-30T15:10:23.458543+0000","last_undegraded":"2024-03-30T15:10:23.458543+0000","last_fullsized":"2024-03-30T15:08:51.471934+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:20.548676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:20.548676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.455163+0000","last_change":"2024-03-30T15:08:51.488378+0000","last_active":"2024-03-30T15:10:23.455163+0000","last_peered":"2024-03-30T15:10:23.455163+0000","last_clean":"2024-03-30T14:54:40.325599+0000","last_became_active":"2024-03-30T15:08:51.488378+0000","last_became_peered":"2024-03-30T15:08:51.488378+0000","last_unstale":"2024-03-30T15:10:23.455163+0000","last_undegraded":"2024-03-30T15:10:23.455163+0000","last_fullsized":"2024-03-30T15:08:51.480993+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:42.814674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.814674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"475'41","reported_seq":1637,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459089+0000","last_change":"2024-03-30T15:08:42.423012+0000","last_active":"2024-03-30T15:10:23.459089+0000","last_peered":"2024-03-30T15:10:23.459089+0000","last_clean":"2024-03-30T15:10:23.459089+0000","last_became_active":"2024-03-30T14:54:58.400430+0000","last_became_peered":"2024-03-30T14:54:58.400430+0000","last_unstale":"2024-03-30T15:10:23.459089+0000","last_undegraded":"2024-03-30T15:10:23.459089+0000","last_fullsized":"2024-03-30T15:10:23.459089+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'41","last_scrub_stamp":"2024-03-30T15:08:42.422879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:42.422879+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1432,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.489546+0000","last_change":"2024-03-30T15:08:51.493922+0000","last_active":"2024-03-30T15:10:23.489546+0000","last_peered":"2024-03-30T15:10:23.489546+0000","last_clean":"2024-03-30T15:08:49.460923+0000","last_became_active":"2024-03-30T15:08:51.493922+0000","last_became_peered":"2024-03-30T15:08:51.493922+0000","last_unstale":"2024-03-30T15:10:23.489546+0000","last_undegraded":"2024-03-30T15:10:23.489546+0000","last_fullsized":"2024-03-30T15:08:51.469678+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:07:31.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.587756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1441,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458159+0000","last_change":"2024-03-30T15:09:48.745509+0000","last_active":"2024-03-30T15:10:23.458159+0000","last_peered":"2024-03-30T15:10:23.458159+0000","last_clean":"2024-03-30T15:10:23.458159+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:10:23.458159+0000","last_undegraded":"2024-03-30T15:10:23.458159+0000","last_fullsized":"2024-03-30T15:10:23.458159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:09:48.745406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:09:48.745406+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:11.072424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1329,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490055+0000","last_change":"2024-03-30T15:08:00.231743+0000","last_active":"2024-03-30T15:10:23.490055+0000","last_peered":"2024-03-30T15:10:23.490055+0000","last_clean":"2024-03-30T15:10:23.490055+0000","last_became_active":"2024-03-30T14:57:56.211474+0000","last_became_peered":"2024-03-30T14:57:56.211474+0000","last_unstale":"2024-03-30T15:10:23.490055+0000","last_undegraded":"2024-03-30T15:10:23.490055+0000","last_fullsized":"2024-03-30T15:10:23.490055+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.231663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.231663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442838+0000","last_change":"2024-03-30T15:08:51.488420+0000","last_active":"2024-03-30T15:10:23.442838+0000","last_peered":"2024-03-30T15:10:23.442838+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.488420+0000","last_became_peered":"2024-03-30T15:08:51.488420+0000","last_unstale":"2024-03-30T15:10:23.442838+0000","last_undegraded":"2024-03-30T15:10:23.442838+0000","last_fullsized":"2024-03-30T15:08:51.477860+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:19.578344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:19.578344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1058,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.455268+0000","last_change":"2024-03-30T15:08:51.488101+0000","last_active":"2024-03-30T15:10:23.455268+0000","last_peered":"2024-03-30T15:10:23.455268+0000","last_clean":"2024-03-30T15:04:44.983256+0000","last_became_active":"2024-03-30T15:08:51.488101+0000","last_became_peered":"2024-03-30T15:08:51.488101+0000","last_unstale":"2024-03-30T15:10:23.455268+0000","last_undegraded":"2024-03-30T15:08:51.474496+0000","last_fullsized":"2024-03-30T15:08:51.474371+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1438,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459764+0000","last_change":"2024-03-30T15:10:15.868071+0000","last_active":"2024-03-30T15:10:23.459764+0000","last_peered":"2024-03-30T15:10:23.459764+0000","last_clean":"2024-03-30T15:10:23.459764+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:10:23.459764+0000","last_undegraded":"2024-03-30T15:10:23.459764+0000","last_fullsized":"2024-03-30T15:10:23.459764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:15.867967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:37.491750+0000","last_clean_scrub_stamp":"2024-03-30T15:10:15.867967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:43.027878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1450,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442035+0000","last_change":"2024-03-30T15:09:44.047335+0000","last_active":"2024-03-30T15:10:23.442035+0000","last_peered":"2024-03-30T15:10:23.442035+0000","last_clean":"2024-03-30T15:10:23.442035+0000","last_became_active":"2024-03-30T14:54:41.345860+0000","last_became_peered":"2024-03-30T14:54:41.345860+0000","last_unstale":"2024-03-30T15:10:23.442035+0000","last_undegraded":"2024-03-30T15:10:23.442035+0000","last_fullsized":"2024-03-30T15:10:23.442035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:44.047252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:09:44.047252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:01.311426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":186,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441904+0000","last_change":"2024-03-30T15:08:51.477473+0000","last_active":"2024-03-30T15:10:23.441904+0000","last_peered":"2024-03-30T15:10:23.441904+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.477473+0000","last_became_peered":"2024-03-30T15:08:51.477473+0000","last_unstale":"2024-03-30T15:10:23.441904+0000","last_undegraded":"2024-03-30T15:10:23.441904+0000","last_fullsized":"2024-03-30T15:08:51.470096+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:21.573522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.573522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490377+0000","last_change":"2024-03-30T15:09:43.536263+0000","last_active":"2024-03-30T15:10:23.490377+0000","last_peered":"2024-03-30T15:10:23.490377+0000","last_clean":"2024-03-30T15:10:23.490377+0000","last_became_active":"2024-03-30T14:57:55.073702+0000","last_became_peered":"2024-03-30T14:57:55.073702+0000","last_unstale":"2024-03-30T15:10:23.490377+0000","last_undegraded":"2024-03-30T15:10:23.490377+0000","last_fullsized":"2024-03-30T15:10:23.490377+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:43.536157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:43.536157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:54.184213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1519,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452162+0000","last_change":"2024-03-30T15:08:59.995404+0000","last_active":"2024-03-30T15:10:23.452162+0000","last_peered":"2024-03-30T15:10:23.452162+0000","last_clean":"2024-03-30T15:10:23.452162+0000","last_became_active":"2024-03-30T14:54:58.400590+0000","last_became_peered":"2024-03-30T14:54:58.400590+0000","last_unstale":"2024-03-30T15:10:23.452162+0000","last_undegraded":"2024-03-30T15:10:23.452162+0000","last_fullsized":"2024-03-30T15:10:23.452162+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:08:59.995234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:59.995234+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1437,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443385+0000","last_change":"2024-03-30T15:10:15.958051+0000","last_active":"2024-03-30T15:10:23.443385+0000","last_peered":"2024-03-30T15:10:23.443385+0000","last_clean":"2024-03-30T15:10:23.443385+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:10:23.443385+0000","last_undegraded":"2024-03-30T15:10:23.443385+0000","last_fullsized":"2024-03-30T15:10:23.443385+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:15.957957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:10:15.957957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:34.926678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1437,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442472+0000","last_change":"2024-03-30T15:08:51.476897+0000","last_active":"2024-03-30T15:10:23.442472+0000","last_peered":"2024-03-30T15:10:23.442472+0000","last_clean":"2024-03-30T15:08:49.451677+0000","last_became_active":"2024-03-30T15:08:51.476897+0000","last_became_peered":"2024-03-30T15:08:51.476897+0000","last_unstale":"2024-03-30T15:10:23.442472+0000","last_undegraded":"2024-03-30T15:10:23.442472+0000","last_fullsized":"2024-03-30T15:08:51.468166+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:48.713441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:48.713441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1417,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.441871+0000","last_change":"2024-03-30T15:08:11.701750+0000","last_active":"2024-03-30T15:10:23.441871+0000","last_peered":"2024-03-30T15:10:23.441871+0000","last_clean":"2024-03-30T15:10:23.441871+0000","last_became_active":"2024-03-30T14:54:43.355795+0000","last_became_peered":"2024-03-30T14:54:43.355795+0000","last_unstale":"2024-03-30T15:10:23.441871+0000","last_undegraded":"2024-03-30T15:10:23.441871+0000","last_fullsized":"2024-03-30T15:10:23.441871+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.701619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.701619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1353,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.451574+0000","last_change":"2024-03-30T15:09:28.978095+0000","last_active":"2024-03-30T15:10:23.451574+0000","last_peered":"2024-03-30T15:10:23.451574+0000","last_clean":"2024-03-30T15:10:23.451574+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:10:23.451574+0000","last_undegraded":"2024-03-30T15:10:23.451574+0000","last_fullsized":"2024-03-30T15:10:23.451574+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:09:28.978002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:28.978002+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:47.834448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1437,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.441682+0000","last_change":"2024-03-30T15:08:51.477521+0000","last_active":"2024-03-30T15:10:23.441682+0000","last_peered":"2024-03-30T15:10:23.441682+0000","last_clean":"2024-03-30T15:08:49.451179+0000","last_became_active":"2024-03-30T15:08:51.477521+0000","last_became_peered":"2024-03-30T15:08:51.477521+0000","last_unstale":"2024-03-30T15:10:23.441682+0000","last_undegraded":"2024-03-30T15:10:23.441682+0000","last_fullsized":"2024-03-30T15:08:51.467398+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:58.835804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:06:58.835804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1465,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490130+0000","last_change":"2024-03-30T15:08:08.213331+0000","last_active":"2024-03-30T15:10:23.490130+0000","last_peered":"2024-03-30T15:10:23.490130+0000","last_clean":"2024-03-30T15:10:23.490130+0000","last_became_active":"2024-03-30T14:54:39.334553+0000","last_became_peered":"2024-03-30T14:54:39.334553+0000","last_unstale":"2024-03-30T15:10:23.490130+0000","last_undegraded":"2024-03-30T15:10:23.490130+0000","last_fullsized":"2024-03-30T15:10:23.490130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:08:08.213219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.213219+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443547+0000","last_change":"2024-03-30T15:08:04.783312+0000","last_active":"2024-03-30T15:10:23.443547+0000","last_peered":"2024-03-30T15:10:23.443547+0000","last_clean":"2024-03-30T15:10:23.443547+0000","last_became_active":"2024-03-30T14:54:37.328347+0000","last_became_peered":"2024-03-30T14:54:37.328347+0000","last_unstale":"2024-03-30T15:10:23.443547+0000","last_undegraded":"2024-03-30T15:10:23.443547+0000","last_fullsized":"2024-03-30T15:10:23.443547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:04.783237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.783237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1329,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442791+0000","last_change":"2024-03-30T15:09:41.889489+0000","last_active":"2024-03-30T15:10:23.442791+0000","last_peered":"2024-03-30T15:10:23.442791+0000","last_clean":"2024-03-30T15:10:23.442791+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:10:23.442791+0000","last_undegraded":"2024-03-30T15:10:23.442791+0000","last_fullsized":"2024-03-30T15:10:23.442791+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:41.889404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:09:41.889404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:44.389580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.488964+0000","last_change":"2024-03-30T15:09:47.457711+0000","last_active":"2024-03-30T15:10:23.488964+0000","last_peered":"2024-03-30T15:10:23.488964+0000","last_clean":"2024-03-30T15:10:23.488964+0000","last_became_active":"2024-03-30T14:57:55.085084+0000","last_became_peered":"2024-03-30T14:57:55.085084+0000","last_unstale":"2024-03-30T15:10:23.488964+0000","last_undegraded":"2024-03-30T15:10:23.488964+0000","last_fullsized":"2024-03-30T15:10:23.488964+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:47.457437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:47.457437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:03.161345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"76'8","reported_seq":1619,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442599+0000","last_change":"2024-03-30T15:09:33.841789+0000","last_active":"2024-03-30T15:10:23.442599+0000","last_peered":"2024-03-30T15:10:23.442599+0000","last_clean":"2024-03-30T15:10:23.442599+0000","last_became_active":"2024-03-30T14:54:41.352590+0000","last_became_peered":"2024-03-30T14:54:41.352590+0000","last_unstale":"2024-03-30T15:10:23.442599+0000","last_undegraded":"2024-03-30T15:10:23.442599+0000","last_fullsized":"2024-03-30T15:10:23.442599+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:09:33.841706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:09:33.841706+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:03.166566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1469,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.456988+0000","last_change":"2024-03-30T15:08:51.260729+0000","last_active":"2024-03-30T15:10:23.456988+0000","last_peered":"2024-03-30T15:10:23.456988+0000","last_clean":"2024-03-30T15:10:23.456988+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:10:23.456988+0000","last_undegraded":"2024-03-30T15:10:23.456988+0000","last_fullsized":"2024-03-30T15:10:23.456988+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:08:51.260584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:51.260584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1444,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.443717+0000","last_change":"2024-03-30T15:08:51.487647+0000","last_active":"2024-03-30T15:10:23.443717+0000","last_peered":"2024-03-30T15:10:23.443717+0000","last_clean":"2024-03-30T15:08:49.470208+0000","last_became_active":"2024-03-30T15:08:51.487647+0000","last_became_peered":"2024-03-30T15:08:51.487647+0000","last_unstale":"2024-03-30T15:10:23.443717+0000","last_undegraded":"2024-03-30T15:08:51.470586+0000","last_fullsized":"2024-03-30T15:08:51.470466+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:07:32.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.333139+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1421,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.458443+0000","last_change":"2024-03-30T15:08:51.490030+0000","last_active":"2024-03-30T15:10:23.458443+0000","last_peered":"2024-03-30T15:10:23.458443+0000","last_clean":"2024-03-30T15:08:49.476019+0000","last_became_active":"2024-03-30T15:08:51.490030+0000","last_became_peered":"2024-03-30T15:08:51.490030+0000","last_unstale":"2024-03-30T15:10:23.458443+0000","last_undegraded":"2024-03-30T15:10:23.458443+0000","last_fullsized":"2024-03-30T15:08:51.477055+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:49.296863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:08:49.296863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1340,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453311+0000","last_change":"2024-03-30T15:09:47.870909+0000","last_active":"2024-03-30T15:10:23.453311+0000","last_peered":"2024-03-30T15:10:23.453311+0000","last_clean":"2024-03-30T15:10:23.453311+0000","last_became_active":"2024-03-30T14:57:55.235971+0000","last_became_peered":"2024-03-30T14:57:55.235971+0000","last_unstale":"2024-03-30T15:10:23.453311+0000","last_undegraded":"2024-03-30T15:10:23.453311+0000","last_fullsized":"2024-03-30T15:10:23.453311+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:47.870818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:47.870818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:51.896438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":1036,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459140+0000","last_change":"2024-03-30T15:09:42.557089+0000","last_active":"2024-03-30T15:10:23.459140+0000","last_peered":"2024-03-30T15:10:23.459140+0000","last_clean":"2024-03-30T15:10:23.459140+0000","last_became_active":"2024-03-30T15:01:40.313659+0000","last_became_peered":"2024-03-30T15:01:40.313659+0000","last_unstale":"2024-03-30T15:10:23.459140+0000","last_undegraded":"2024-03-30T15:10:23.459140+0000","last_fullsized":"2024-03-30T15:10:23.459140+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:09:42.556992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:42.556992+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:05.602501+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1444,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.451570+0000","last_change":"2024-03-30T15:08:51.486022+0000","last_active":"2024-03-30T15:10:23.451570+0000","last_peered":"2024-03-30T15:10:23.451570+0000","last_clean":"2024-03-30T15:08:49.466505+0000","last_became_active":"2024-03-30T15:08:51.486022+0000","last_became_peered":"2024-03-30T15:08:51.486022+0000","last_unstale":"2024-03-30T15:10:23.451570+0000","last_undegraded":"2024-03-30T15:10:23.451570+0000","last_fullsized":"2024-03-30T15:08:51.474841+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:58.498678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:07:58.498678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1421,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459665+0000","last_change":"2024-03-30T15:08:59.295446+0000","last_active":"2024-03-30T15:10:23.459665+0000","last_peered":"2024-03-30T15:10:23.459665+0000","last_clean":"2024-03-30T15:10:23.459665+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:10:23.459665+0000","last_undegraded":"2024-03-30T15:10:23.459665+0000","last_fullsized":"2024-03-30T15:10:23.459665+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:59.295336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:59.295336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1416,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442990+0000","last_change":"2024-03-30T15:09:42.843146+0000","last_active":"2024-03-30T15:10:23.442990+0000","last_peered":"2024-03-30T15:10:23.442990+0000","last_clean":"2024-03-30T15:10:23.442990+0000","last_became_active":"2024-03-30T14:54:43.357570+0000","last_became_peered":"2024-03-30T14:54:43.357570+0000","last_unstale":"2024-03-30T15:10:23.442990+0000","last_undegraded":"2024-03-30T15:10:23.442990+0000","last_fullsized":"2024-03-30T15:10:23.442990+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_clean_scrub_stamp":"2024-03-30T15:09:42.843021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:09.148023+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1334,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489930+0000","last_change":"2024-03-30T15:09:42.516763+0000","last_active":"2024-03-30T15:10:23.489930+0000","last_peered":"2024-03-30T15:10:23.489930+0000","last_clean":"2024-03-30T15:10:23.489930+0000","last_became_active":"2024-03-30T14:57:55.122628+0000","last_became_peered":"2024-03-30T14:57:55.122628+0000","last_unstale":"2024-03-30T15:10:23.489930+0000","last_undegraded":"2024-03-30T15:10:23.489930+0000","last_fullsized":"2024-03-30T15:10:23.489930+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:42.516632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:42.516632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:53.530437+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1055,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.443713+0000","last_change":"2024-03-30T15:08:51.490192+0000","last_active":"2024-03-30T15:10:23.443713+0000","last_peered":"2024-03-30T15:10:23.443713+0000","last_clean":"2024-03-30T15:04:44.840494+0000","last_became_active":"2024-03-30T15:08:51.490192+0000","last_became_peered":"2024-03-30T15:08:51.490192+0000","last_unstale":"2024-03-30T15:10:23.443713+0000","last_undegraded":"2024-03-30T15:08:51.480539+0000","last_fullsized":"2024-03-30T15:08:51.480426+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1436,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442228+0000","last_change":"2024-03-30T15:10:13.973254+0000","last_active":"2024-03-30T15:10:23.442228+0000","last_peered":"2024-03-30T15:10:23.442228+0000","last_clean":"2024-03-30T15:10:23.442228+0000","last_became_active":"2024-03-30T14:54:37.327445+0000","last_became_peered":"2024-03-30T14:54:37.327445+0000","last_unstale":"2024-03-30T15:10:23.442228+0000","last_undegraded":"2024-03-30T15:10:23.442228+0000","last_fullsized":"2024-03-30T15:10:23.442228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:13.973165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:10:13.973165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:27.771724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":521,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453276+0000","last_change":"2024-03-30T15:10:18.917370+0000","last_active":"2024-03-30T15:10:23.453276+0000","last_peered":"2024-03-30T15:10:23.453276+0000","last_clean":"2024-03-30T15:10:23.453276+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:10:23.453276+0000","last_undegraded":"2024-03-30T15:10:23.453276+0000","last_fullsized":"2024-03-30T15:10:23.453276+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:18.917285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:10:18.917285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:32.309736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1445,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443536+0000","last_change":"2024-03-30T15:09:31.094175+0000","last_active":"2024-03-30T15:10:23.443536+0000","last_peered":"2024-03-30T15:10:23.443536+0000","last_clean":"2024-03-30T15:10:23.443536+0000","last_became_active":"2024-03-30T14:54:43.351166+0000","last_became_peered":"2024-03-30T14:54:43.351166+0000","last_unstale":"2024-03-30T15:10:23.443536+0000","last_undegraded":"2024-03-30T15:10:23.443536+0000","last_fullsized":"2024-03-30T15:10:23.443536+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:31.094064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:31.094064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:42.333925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":191,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.443051+0000","last_change":"2024-03-30T15:08:51.487477+0000","last_active":"2024-03-30T15:10:23.443051+0000","last_peered":"2024-03-30T15:10:23.443051+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.487477+0000","last_became_peered":"2024-03-30T15:08:51.487477+0000","last_unstale":"2024-03-30T15:10:23.443051+0000","last_undegraded":"2024-03-30T15:10:23.443051+0000","last_fullsized":"2024-03-30T15:08:51.471058+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:14.547919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:14.547919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1450,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459443+0000","last_change":"2024-03-30T15:09:00.314183+0000","last_active":"2024-03-30T15:10:23.459443+0000","last_peered":"2024-03-30T15:10:23.459443+0000","last_clean":"2024-03-30T15:10:23.459443+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:10:23.459443+0000","last_undegraded":"2024-03-30T15:10:23.459443+0000","last_fullsized":"2024-03-30T15:10:23.459443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:09:00.314085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:00.314085+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442085+0000","last_change":"2024-03-30T15:08:51.481789+0000","last_active":"2024-03-30T15:10:23.442085+0000","last_peered":"2024-03-30T15:10:23.442085+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.481789+0000","last_became_peered":"2024-03-30T15:08:51.481789+0000","last_unstale":"2024-03-30T15:10:23.442085+0000","last_undegraded":"2024-03-30T15:10:23.442085+0000","last_fullsized":"2024-03-30T15:08:51.472587+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:36.536916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:36.536916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"116'9","reported_seq":1504,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443035+0000","last_change":"2024-03-30T15:09:34.856280+0000","last_active":"2024-03-30T15:10:23.443035+0000","last_peered":"2024-03-30T15:10:23.443035+0000","last_clean":"2024-03-30T15:10:23.443035+0000","last_became_active":"2024-03-30T14:57:56.272818+0000","last_became_peered":"2024-03-30T14:57:56.272818+0000","last_unstale":"2024-03-30T15:10:23.443035+0000","last_undegraded":"2024-03-30T15:10:23.443035+0000","last_fullsized":"2024-03-30T15:10:23.443035+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-03-30T15:09:34.856155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:09:34.856155+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:36.822903+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1416,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490147+0000","last_change":"2024-03-30T15:09:34.473464+0000","last_active":"2024-03-30T15:10:23.490147+0000","last_peered":"2024-03-30T15:10:23.490147+0000","last_clean":"2024-03-30T15:10:23.490147+0000","last_became_active":"2024-03-30T14:54:43.349516+0000","last_became_peered":"2024-03-30T14:54:43.349516+0000","last_unstale":"2024-03-30T15:10:23.490147+0000","last_undegraded":"2024-03-30T15:10:23.490147+0000","last_fullsized":"2024-03-30T15:10:23.490147+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:34.473349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:09:34.473349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:49.492105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":191,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.443635+0000","last_change":"2024-03-30T15:08:51.479190+0000","last_active":"2024-03-30T15:10:23.443635+0000","last_peered":"2024-03-30T15:10:23.443635+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.479190+0000","last_became_peered":"2024-03-30T15:08:51.479190+0000","last_unstale":"2024-03-30T15:10:23.443635+0000","last_undegraded":"2024-03-30T15:10:23.443635+0000","last_fullsized":"2024-03-30T15:08:51.470151+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:18.568685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:18.568685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"475'20","reported_seq":1390,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.455096+0000","last_change":"2024-03-30T15:09:31.179875+0000","last_active":"2024-03-30T15:10:23.455096+0000","last_peered":"2024-03-30T15:10:23.455096+0000","last_clean":"2024-03-30T15:10:23.455096+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:10:23.455096+0000","last_undegraded":"2024-03-30T15:10:23.455096+0000","last_fullsized":"2024-03-30T15:10:23.455096+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:09:31.179731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:31.179731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:51.482251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1428,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442579+0000","last_change":"2024-03-30T15:08:17.645966+0000","last_active":"2024-03-30T15:10:23.442579+0000","last_peered":"2024-03-30T15:10:23.442579+0000","last_clean":"2024-03-30T15:10:23.442579+0000","last_became_active":"2024-03-30T14:54:37.474409+0000","last_became_peered":"2024-03-30T14:54:37.474409+0000","last_unstale":"2024-03-30T15:10:23.442579+0000","last_undegraded":"2024-03-30T15:10:23.442579+0000","last_fullsized":"2024-03-30T15:10:23.442579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.645871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.645871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"670'9","reported_seq":1618,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.459212+0000","last_change":"2024-03-30T15:08:51.484157+0000","last_active":"2024-03-30T15:10:23.459212+0000","last_peered":"2024-03-30T15:10:23.459212+0000","last_clean":"2024-03-30T15:08:49.476510+0000","last_became_active":"2024-03-30T15:08:51.484157+0000","last_became_peered":"2024-03-30T15:08:51.484157+0000","last_unstale":"2024-03-30T15:10:23.459212+0000","last_undegraded":"2024-03-30T15:08:51.480994+0000","last_fullsized":"2024-03-30T15:08:51.480866+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:06:44.020829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:06:44.020829+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":242,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.443326+0000","last_change":"2024-03-30T15:08:51.492044+0000","last_active":"2024-03-30T15:10:23.443326+0000","last_peered":"2024-03-30T15:10:23.443326+0000","last_clean":"2024-03-30T14:56:35.330088+0000","last_became_active":"2024-03-30T15:08:51.492044+0000","last_became_peered":"2024-03-30T15:08:51.492044+0000","last_unstale":"2024-03-30T15:10:23.443326+0000","last_undegraded":"2024-03-30T15:08:51.481375+0000","last_fullsized":"2024-03-30T15:08:51.481263+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:07:30.551096+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.551096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1333,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.491390+0000","last_change":"2024-03-30T15:09:48.479621+0000","last_active":"2024-03-30T15:10:23.491390+0000","last_peered":"2024-03-30T15:10:23.491390+0000","last_clean":"2024-03-30T15:10:23.491390+0000","last_became_active":"2024-03-30T14:57:55.175751+0000","last_became_peered":"2024-03-30T14:57:55.175751+0000","last_unstale":"2024-03-30T15:10:23.491390+0000","last_undegraded":"2024-03-30T15:10:23.491390+0000","last_fullsized":"2024-03-30T15:10:23.491390+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:48.479537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:09:48.479537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:04.738356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1460,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.491399+0000","last_change":"2024-03-30T15:09:30.377329+0000","last_active":"2024-03-30T15:10:23.491399+0000","last_peered":"2024-03-30T15:10:23.491399+0000","last_clean":"2024-03-30T15:10:23.491399+0000","last_became_active":"2024-03-30T14:54:39.334163+0000","last_became_peered":"2024-03-30T14:54:39.334163+0000","last_unstale":"2024-03-30T15:10:23.491399+0000","last_undegraded":"2024-03-30T15:10:23.491399+0000","last_fullsized":"2024-03-30T15:10:23.491399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:09:30.377230+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:09:30.377230+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:44.451822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1436,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489630+0000","last_change":"2024-03-30T15:09:35.483222+0000","last_active":"2024-03-30T15:10:23.489630+0000","last_peered":"2024-03-30T15:10:23.489630+0000","last_clean":"2024-03-30T15:10:23.489630+0000","last_became_active":"2024-03-30T14:54:37.329753+0000","last_became_peered":"2024-03-30T14:54:37.329753+0000","last_unstale":"2024-03-30T15:10:23.489630+0000","last_undegraded":"2024-03-30T15:10:23.489630+0000","last_fullsized":"2024-03-30T15:10:23.489630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:35.482998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:09:35.482998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:59.765087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1432,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454404+0000","last_change":"2024-03-30T15:07:35.184388+0000","last_active":"2024-03-30T15:10:23.454404+0000","last_peered":"2024-03-30T15:10:23.454404+0000","last_clean":"2024-03-30T15:10:23.454404+0000","last_became_active":"2024-03-30T14:54:41.348358+0000","last_became_peered":"2024-03-30T14:54:41.348358+0000","last_unstale":"2024-03-30T15:10:23.454404+0000","last_undegraded":"2024-03-30T15:10:23.454404+0000","last_fullsized":"2024-03-30T15:10:23.454404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.184314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.184314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":191,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457772+0000","last_change":"2024-03-30T15:08:51.478956+0000","last_active":"2024-03-30T15:10:23.457772+0000","last_peered":"2024-03-30T15:10:23.457772+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.478956+0000","last_became_peered":"2024-03-30T15:08:51.478956+0000","last_unstale":"2024-03-30T15:10:23.457772+0000","last_undegraded":"2024-03-30T15:10:23.457772+0000","last_fullsized":"2024-03-30T15:08:51.471508+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1333,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489328+0000","last_change":"2024-03-30T15:10:21.578617+0000","last_active":"2024-03-30T15:10:23.489328+0000","last_peered":"2024-03-30T15:10:23.489328+0000","last_clean":"2024-03-30T15:10:23.489328+0000","last_became_active":"2024-03-30T14:57:55.123053+0000","last_became_peered":"2024-03-30T14:57:55.123053+0000","last_unstale":"2024-03-30T15:10:23.489328+0000","last_undegraded":"2024-03-30T15:10:23.489328+0000","last_fullsized":"2024-03-30T15:10:23.489328+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:21.578525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:10:21.578525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:30.419834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1461,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454640+0000","last_change":"2024-03-30T15:10:21.415184+0000","last_active":"2024-03-30T15:10:23.454640+0000","last_peered":"2024-03-30T15:10:23.454640+0000","last_clean":"2024-03-30T15:10:23.454640+0000","last_became_active":"2024-03-30T14:54:39.339328+0000","last_became_peered":"2024-03-30T14:54:39.339328+0000","last_unstale":"2024-03-30T15:10:23.454640+0000","last_undegraded":"2024-03-30T15:10:23.454640+0000","last_fullsized":"2024-03-30T15:10:23.454640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:10:21.415088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:10:21.415088+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:34.838513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1451,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443165+0000","last_change":"2024-03-30T15:10:16.977971+0000","last_active":"2024-03-30T15:10:23.443165+0000","last_peered":"2024-03-30T15:10:23.443165+0000","last_clean":"2024-03-30T15:10:23.443165+0000","last_became_active":"2024-03-30T14:54:37.329731+0000","last_became_peered":"2024-03-30T14:54:37.329731+0000","last_unstale":"2024-03-30T15:10:23.443165+0000","last_undegraded":"2024-03-30T15:10:23.443165+0000","last_fullsized":"2024-03-30T15:10:23.443165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:10:16.977784+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:10:16.977784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:21.256074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":499,"reported_epoch":761,"state":"active+clean+remapped","last_fresh":"2024-03-30T15:10:23.455264+0000","last_change":"2024-03-30T15:09:45.488879+0000","last_active":"2024-03-30T15:10:23.455264+0000","last_peered":"2024-03-30T15:10:23.455264+0000","last_clean":"2024-03-30T15:10:23.455264+0000","last_became_active":"2024-03-30T15:09:45.488464+0000","last_became_peered":"2024-03-30T15:09:45.488464+0000","last_unstale":"2024-03-30T15:10:23.455264+0000","last_undegraded":"2024-03-30T15:10:23.455264+0000","last_fullsized":"2024-03-30T15:10:23.455264+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:47.219469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:47.219469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457094+0000","last_change":"2024-03-30T15:08:51.481393+0000","last_active":"2024-03-30T15:10:23.457094+0000","last_peered":"2024-03-30T15:10:23.457094+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.481393+0000","last_became_peered":"2024-03-30T15:08:51.481393+0000","last_unstale":"2024-03-30T15:10:23.457094+0000","last_undegraded":"2024-03-30T15:10:23.457094+0000","last_fullsized":"2024-03-30T15:08:51.474347+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:06:53.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:06:53.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":192,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457825+0000","last_change":"2024-03-30T15:08:51.489377+0000","last_active":"2024-03-30T15:10:23.457825+0000","last_peered":"2024-03-30T15:10:23.457825+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.489377+0000","last_became_peered":"2024-03-30T15:08:51.489377+0000","last_unstale":"2024-03-30T15:10:23.457825+0000","last_undegraded":"2024-03-30T15:10:23.457825+0000","last_fullsized":"2024-03-30T15:08:51.475600+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:49.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:04:49.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:25.764404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1379,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442371+0000","last_change":"2024-03-30T15:08:44.633095+0000","last_active":"2024-03-30T15:10:23.442371+0000","last_peered":"2024-03-30T15:10:23.442371+0000","last_clean":"2024-03-30T15:10:23.442371+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:10:23.442371+0000","last_undegraded":"2024-03-30T15:10:23.442371+0000","last_fullsized":"2024-03-30T15:10:23.442371+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:08:44.632984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:44.632984+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459733+0000","last_change":"2024-03-30T15:08:43.367882+0000","last_active":"2024-03-30T15:10:23.459733+0000","last_peered":"2024-03-30T15:10:23.459733+0000","last_clean":"2024-03-30T15:10:23.459733+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:10:23.459733+0000","last_undegraded":"2024-03-30T15:10:23.459733+0000","last_fullsized":"2024-03-30T15:10:23.459733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:43.367786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:43.367786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1428,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442384+0000","last_change":"2024-03-30T15:10:19.990058+0000","last_active":"2024-03-30T15:10:23.442384+0000","last_peered":"2024-03-30T15:10:23.442384+0000","last_clean":"2024-03-30T15:10:23.442384+0000","last_became_active":"2024-03-30T14:54:41.353838+0000","last_became_peered":"2024-03-30T14:54:41.353838+0000","last_unstale":"2024-03-30T15:10:23.442384+0000","last_undegraded":"2024-03-30T15:10:23.442384+0000","last_fullsized":"2024-03-30T15:10:23.442384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.989982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:45.755668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.452398+0000","last_change":"2024-03-30T15:08:51.489512+0000","last_active":"2024-03-30T15:10:23.452398+0000","last_peered":"2024-03-30T15:10:23.452398+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.489512+0000","last_became_peered":"2024-03-30T15:08:51.489512+0000","last_unstale":"2024-03-30T15:10:23.452398+0000","last_undegraded":"2024-03-30T15:10:23.452398+0000","last_fullsized":"2024-03-30T15:08:51.485037+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:55.573865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:05:55.573865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1338,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454410+0000","last_change":"2024-03-30T15:08:08.177938+0000","last_active":"2024-03-30T15:10:23.454410+0000","last_peered":"2024-03-30T15:10:23.454410+0000","last_clean":"2024-03-30T15:10:23.454410+0000","last_became_active":"2024-03-30T14:57:54.067995+0000","last_became_peered":"2024-03-30T14:57:54.067995+0000","last_unstale":"2024-03-30T15:10:23.454410+0000","last_undegraded":"2024-03-30T15:10:23.454410+0000","last_fullsized":"2024-03-30T15:10:23.454410+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:08.177858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:08.177858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":1022,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457945+0000","last_change":"2024-03-30T15:10:06.755351+0000","last_active":"2024-03-30T15:10:23.457945+0000","last_peered":"2024-03-30T15:10:23.457945+0000","last_clean":"2024-03-30T15:10:23.457945+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:10:23.457945+0000","last_undegraded":"2024-03-30T15:10:23.457945+0000","last_fullsized":"2024-03-30T15:10:23.457945+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:10:06.755229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:10:06.755229+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:10.747080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.460322+0000","last_change":"2024-03-30T15:08:51.486467+0000","last_active":"2024-03-30T15:10:23.460322+0000","last_peered":"2024-03-30T15:10:23.460322+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.486467+0000","last_became_peered":"2024-03-30T15:08:51.486467+0000","last_unstale":"2024-03-30T15:10:23.460322+0000","last_undegraded":"2024-03-30T15:10:23.460322+0000","last_fullsized":"2024-03-30T15:08:51.480621+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1433,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.451901+0000","last_change":"2024-03-30T15:08:44.876411+0000","last_active":"2024-03-30T15:10:23.451901+0000","last_peered":"2024-03-30T15:10:23.451901+0000","last_clean":"2024-03-30T15:10:23.451901+0000","last_became_active":"2024-03-30T14:54:41.347082+0000","last_became_peered":"2024-03-30T14:54:41.347082+0000","last_unstale":"2024-03-30T15:10:23.451901+0000","last_undegraded":"2024-03-30T15:10:23.451901+0000","last_fullsized":"2024-03-30T15:10:23.451901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:44.876314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:44.876314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1416,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442444+0000","last_change":"2024-03-30T15:09:01.637001+0000","last_active":"2024-03-30T15:10:23.442444+0000","last_peered":"2024-03-30T15:10:23.442444+0000","last_clean":"2024-03-30T15:10:23.442444+0000","last_became_active":"2024-03-30T14:54:43.352275+0000","last_became_peered":"2024-03-30T14:54:43.352275+0000","last_unstale":"2024-03-30T15:10:23.442444+0000","last_undegraded":"2024-03-30T15:10:23.442444+0000","last_fullsized":"2024-03-30T15:10:23.442444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:01.636912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:09:01.636912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1331,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453414+0000","last_change":"2024-03-30T15:07:33.284903+0000","last_active":"2024-03-30T15:10:23.453414+0000","last_peered":"2024-03-30T15:10:23.453414+0000","last_clean":"2024-03-30T15:10:23.453414+0000","last_became_active":"2024-03-30T14:57:55.238531+0000","last_became_peered":"2024-03-30T14:57:55.238531+0000","last_unstale":"2024-03-30T15:10:23.453414+0000","last_undegraded":"2024-03-30T15:10:23.453414+0000","last_fullsized":"2024-03-30T15:10:23.453414+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:33.284814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:33.284814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1466,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453471+0000","last_change":"2024-03-30T15:08:16.603568+0000","last_active":"2024-03-30T15:10:23.453471+0000","last_peered":"2024-03-30T15:10:23.453471+0000","last_clean":"2024-03-30T15:10:23.453471+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:10:23.453471+0000","last_undegraded":"2024-03-30T15:10:23.453471+0000","last_fullsized":"2024-03-30T15:10:23.453471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:16.603468+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:08:16.603468+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":192,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454118+0000","last_change":"2024-03-30T15:08:51.479322+0000","last_active":"2024-03-30T15:10:23.454118+0000","last_peered":"2024-03-30T15:10:23.454118+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.479322+0000","last_became_peered":"2024-03-30T15:08:51.479322+0000","last_unstale":"2024-03-30T15:10:23.454118+0000","last_undegraded":"2024-03-30T15:10:23.454118+0000","last_fullsized":"2024-03-30T15:08:51.475124+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":186,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.443553+0000","last_change":"2024-03-30T15:08:51.482533+0000","last_active":"2024-03-30T15:10:23.443553+0000","last_peered":"2024-03-30T15:10:23.443553+0000","last_clean":"2024-03-30T14:54:40.325599+0000","last_became_active":"2024-03-30T15:08:51.482533+0000","last_became_peered":"2024-03-30T15:08:51.482533+0000","last_unstale":"2024-03-30T15:10:23.443553+0000","last_undegraded":"2024-03-30T15:10:23.443553+0000","last_fullsized":"2024-03-30T15:08:51.477003+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:28.601815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:07:28.601815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:27.022412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1428,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.455291+0000","last_change":"2024-03-30T15:08:02.083375+0000","last_active":"2024-03-30T15:10:23.455291+0000","last_peered":"2024-03-30T15:10:23.455291+0000","last_clean":"2024-03-30T15:10:23.455291+0000","last_became_active":"2024-03-30T14:54:43.356516+0000","last_became_peered":"2024-03-30T14:54:43.356516+0000","last_unstale":"2024-03-30T15:10:23.455291+0000","last_undegraded":"2024-03-30T15:10:23.455291+0000","last_fullsized":"2024-03-30T15:10:23.455291+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:02.083294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.083294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1339,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453102+0000","last_change":"2024-03-30T15:08:41.886052+0000","last_active":"2024-03-30T15:10:23.453102+0000","last_peered":"2024-03-30T15:10:23.453102+0000","last_clean":"2024-03-30T15:10:23.453102+0000","last_became_active":"2024-03-30T14:57:55.226633+0000","last_became_peered":"2024-03-30T14:57:55.226633+0000","last_unstale":"2024-03-30T15:10:23.453102+0000","last_undegraded":"2024-03-30T15:10:23.453102+0000","last_fullsized":"2024-03-30T15:10:23.453102+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:41.885921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:08:41.885921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1066,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.443585+0000","last_change":"2024-03-30T15:08:51.491167+0000","last_active":"2024-03-30T15:10:23.443585+0000","last_peered":"2024-03-30T15:10:23.443585+0000","last_clean":"2024-03-30T15:04:45.005298+0000","last_became_active":"2024-03-30T15:08:51.491167+0000","last_became_peered":"2024-03-30T15:08:51.491167+0000","last_unstale":"2024-03-30T15:10:23.443585+0000","last_undegraded":"2024-03-30T15:08:51.480589+0000","last_fullsized":"2024-03-30T15:08:51.480476+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:07:25.577074+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:07:25.577074+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1434,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453938+0000","last_change":"2024-03-30T15:07:30.031279+0000","last_active":"2024-03-30T15:10:23.453938+0000","last_peered":"2024-03-30T15:10:23.453938+0000","last_clean":"2024-03-30T15:10:23.453938+0000","last_became_active":"2024-03-30T14:54:37.324435+0000","last_became_peered":"2024-03-30T14:54:37.324435+0000","last_unstale":"2024-03-30T15:10:23.453938+0000","last_undegraded":"2024-03-30T15:10:23.453938+0000","last_fullsized":"2024-03-30T15:10:23.453938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:07:30.031154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"76'8","reported_seq":1635,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453364+0000","last_change":"2024-03-30T15:08:04.561718+0000","last_active":"2024-03-30T15:10:23.453364+0000","last_peered":"2024-03-30T15:10:23.453364+0000","last_clean":"2024-03-30T15:10:23.453364+0000","last_became_active":"2024-03-30T14:54:41.349832+0000","last_became_peered":"2024-03-30T14:54:41.349832+0000","last_unstale":"2024-03-30T15:10:23.453364+0000","last_undegraded":"2024-03-30T15:10:23.453364+0000","last_fullsized":"2024-03-30T15:10:23.453364+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:08:04.561610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:04.561610+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1426,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.443069+0000","last_change":"2024-03-30T15:09:43.824103+0000","last_active":"2024-03-30T15:10:23.443069+0000","last_peered":"2024-03-30T15:10:23.443069+0000","last_clean":"2024-03-30T15:10:23.443069+0000","last_became_active":"2024-03-30T14:54:43.354580+0000","last_became_peered":"2024-03-30T14:54:43.354580+0000","last_unstale":"2024-03-30T15:10:23.443069+0000","last_undegraded":"2024-03-30T15:10:23.443069+0000","last_fullsized":"2024-03-30T15:10:23.443069+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:09:43.824004+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T14:57:29.995691+0000","last_clean_scrub_stamp":"2024-03-30T15:09:43.824004+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:56.635860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1339,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.452577+0000","last_change":"2024-03-30T15:07:57.453748+0000","last_active":"2024-03-30T15:10:23.452577+0000","last_peered":"2024-03-30T15:10:23.452577+0000","last_clean":"2024-03-30T15:10:23.452577+0000","last_became_active":"2024-03-30T14:57:55.228908+0000","last_became_peered":"2024-03-30T14:57:55.228908+0000","last_unstale":"2024-03-30T15:10:23.452577+0000","last_undegraded":"2024-03-30T15:10:23.452577+0000","last_fullsized":"2024-03-30T15:10:23.452577+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:57.453629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:07:57.453629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1383,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.454730+0000","last_change":"2024-03-30T15:07:21.894935+0000","last_active":"2024-03-30T15:10:23.454730+0000","last_peered":"2024-03-30T15:10:23.454730+0000","last_clean":"2024-03-30T15:10:23.454730+0000","last_became_active":"2024-03-30T14:57:56.270437+0000","last_became_peered":"2024-03-30T14:57:56.270437+0000","last_unstale":"2024-03-30T15:10:23.454730+0000","last_undegraded":"2024-03-30T15:10:23.454730+0000","last_fullsized":"2024-03-30T15:10:23.454730+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:07:21.894855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:21.894855+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1428,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489550+0000","last_change":"2024-03-30T15:08:52.362024+0000","last_active":"2024-03-30T15:10:23.489550+0000","last_peered":"2024-03-30T15:10:23.489550+0000","last_clean":"2024-03-30T15:10:23.489550+0000","last_became_active":"2024-03-30T14:54:37.474186+0000","last_became_peered":"2024-03-30T14:54:37.474186+0000","last_unstale":"2024-03-30T15:10:23.489550+0000","last_undegraded":"2024-03-30T15:10:23.489550+0000","last_fullsized":"2024-03-30T15:10:23.489550+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_clean_scrub_stamp":"2024-03-30T15:08:52.361880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1420,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490105+0000","last_change":"2024-03-30T15:08:53.335367+0000","last_active":"2024-03-30T15:10:23.490105+0000","last_peered":"2024-03-30T15:10:23.490105+0000","last_clean":"2024-03-30T15:10:23.490105+0000","last_became_active":"2024-03-30T14:54:41.347412+0000","last_became_peered":"2024-03-30T14:54:41.347412+0000","last_unstale":"2024-03-30T15:10:23.490105+0000","last_undegraded":"2024-03-30T15:10:23.490105+0000","last_fullsized":"2024-03-30T15:10:23.490105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:53.335289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:53.335289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1343,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453499+0000","last_change":"2024-03-30T15:10:19.878575+0000","last_active":"2024-03-30T15:10:23.453499+0000","last_peered":"2024-03-30T15:10:23.453499+0000","last_clean":"2024-03-30T15:10:23.453499+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:10:23.453499+0000","last_undegraded":"2024-03-30T15:10:23.453499+0000","last_fullsized":"2024-03-30T15:10:23.453499+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:19.878469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:10:19.878469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:22.252945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1335,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459770+0000","last_change":"2024-03-30T15:10:21.740271+0000","last_active":"2024-03-30T15:10:23.459770+0000","last_peered":"2024-03-30T15:10:23.459770+0000","last_clean":"2024-03-30T15:10:23.459770+0000","last_became_active":"2024-03-30T14:57:55.102648+0000","last_became_peered":"2024-03-30T14:57:55.102648+0000","last_unstale":"2024-03-30T15:10:23.459770+0000","last_undegraded":"2024-03-30T15:10:23.459770+0000","last_fullsized":"2024-03-30T15:10:23.459770+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:21.740192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:10:21.740192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:41.012304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":1503,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442655+0000","last_change":"2024-03-30T15:09:45.082349+0000","last_active":"2024-03-30T15:10:23.442655+0000","last_peered":"2024-03-30T15:10:23.442655+0000","last_clean":"2024-03-30T15:10:23.442655+0000","last_became_active":"2024-03-30T14:54:39.334350+0000","last_became_peered":"2024-03-30T14:54:39.334350+0000","last_unstale":"2024-03-30T15:10:23.442655+0000","last_undegraded":"2024-03-30T15:10:23.442655+0000","last_fullsized":"2024-03-30T15:10:23.442655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:09:45.082243+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:09:45.082243+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:01.819836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":188,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.490187+0000","last_change":"2024-03-30T15:08:51.488676+0000","last_active":"2024-03-30T15:10:23.490187+0000","last_peered":"2024-03-30T15:10:23.490187+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.488676+0000","last_became_peered":"2024-03-30T15:08:51.488676+0000","last_unstale":"2024-03-30T15:10:23.490187+0000","last_undegraded":"2024-03-30T15:10:23.490187+0000","last_fullsized":"2024-03-30T15:08:51.469799+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:42.537779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:04:42.537779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.573829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1435,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.455035+0000","last_change":"2024-03-30T15:08:51.488002+0000","last_active":"2024-03-30T15:10:23.455035+0000","last_peered":"2024-03-30T15:10:23.455035+0000","last_clean":"2024-03-30T15:08:49.463795+0000","last_became_active":"2024-03-30T15:08:51.488002+0000","last_became_peered":"2024-03-30T15:08:51.488002+0000","last_unstale":"2024-03-30T15:10:23.455035+0000","last_undegraded":"2024-03-30T15:10:23.455035+0000","last_fullsized":"2024-03-30T15:08:51.481407+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.130991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.130991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1417,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490387+0000","last_change":"2024-03-30T15:08:50.270822+0000","last_active":"2024-03-30T15:10:23.490387+0000","last_peered":"2024-03-30T15:10:23.490387+0000","last_clean":"2024-03-30T15:10:23.490387+0000","last_became_active":"2024-03-30T14:54:43.348739+0000","last_became_peered":"2024-03-30T14:54:43.348739+0000","last_unstale":"2024-03-30T15:10:23.490387+0000","last_undegraded":"2024-03-30T15:10:23.490387+0000","last_fullsized":"2024-03-30T15:10:23.490387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:50.270727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:50.270727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1341,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.454812+0000","last_change":"2024-03-30T15:08:51.494658+0000","last_active":"2024-03-30T15:10:23.454812+0000","last_peered":"2024-03-30T15:10:23.454812+0000","last_clean":"2024-03-30T15:08:49.463249+0000","last_became_active":"2024-03-30T15:08:51.494658+0000","last_became_peered":"2024-03-30T15:08:51.494658+0000","last_unstale":"2024-03-30T15:10:23.454812+0000","last_undegraded":"2024-03-30T15:10:23.454812+0000","last_fullsized":"2024-03-30T15:08:51.483885+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:08:21.207705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1461,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.490451+0000","last_change":"2024-03-30T15:09:36.491629+0000","last_active":"2024-03-30T15:10:23.490451+0000","last_peered":"2024-03-30T15:10:23.490451+0000","last_clean":"2024-03-30T15:10:23.490451+0000","last_became_active":"2024-03-30T15:02:02.490285+0000","last_became_peered":"2024-03-30T15:02:02.490285+0000","last_unstale":"2024-03-30T15:10:23.490451+0000","last_undegraded":"2024-03-30T15:10:23.490451+0000","last_fullsized":"2024-03-30T15:10:23.490451+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:09:36.491535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:36.491535+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:43.900915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459561+0000","last_change":"2024-03-30T15:08:52.272982+0000","last_active":"2024-03-30T15:10:23.459561+0000","last_peered":"2024-03-30T15:10:23.459561+0000","last_clean":"2024-03-30T15:10:23.459561+0000","last_became_active":"2024-03-30T14:54:37.331082+0000","last_became_peered":"2024-03-30T14:54:37.331082+0000","last_unstale":"2024-03-30T15:10:23.459561+0000","last_undegraded":"2024-03-30T15:10:23.459561+0000","last_fullsized":"2024-03-30T15:10:23.459561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:52.272886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:52.272886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1420,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.442387+0000","last_change":"2024-03-30T15:08:18.689134+0000","last_active":"2024-03-30T15:10:23.442387+0000","last_peered":"2024-03-30T15:10:23.442387+0000","last_clean":"2024-03-30T15:10:23.442387+0000","last_became_active":"2024-03-30T14:54:41.351809+0000","last_became_peered":"2024-03-30T14:54:41.351809+0000","last_unstale":"2024-03-30T15:10:23.442387+0000","last_undegraded":"2024-03-30T15:10:23.442387+0000","last_fullsized":"2024-03-30T15:10:23.442387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:18.688995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:18.688995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1425,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457715+0000","last_change":"2024-03-30T15:10:14.915374+0000","last_active":"2024-03-30T15:10:23.457715+0000","last_peered":"2024-03-30T15:10:23.457715+0000","last_clean":"2024-03-30T15:10:23.457715+0000","last_became_active":"2024-03-30T14:54:43.356599+0000","last_became_peered":"2024-03-30T14:54:43.356599+0000","last_unstale":"2024-03-30T15:10:23.457715+0000","last_undegraded":"2024-03-30T15:10:23.457715+0000","last_fullsized":"2024-03-30T15:10:23.457715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:14.915282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:10:14.915282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:18.903658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1332,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453898+0000","last_change":"2024-03-30T15:08:49.112865+0000","last_active":"2024-03-30T15:10:23.453898+0000","last_peered":"2024-03-30T15:10:23.453898+0000","last_clean":"2024-03-30T15:10:23.453898+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:10:23.453898+0000","last_undegraded":"2024-03-30T15:10:23.453898+0000","last_fullsized":"2024-03-30T15:10:23.453898+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:49.112779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:08:49.112779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":1444,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.489481+0000","last_change":"2024-03-30T15:08:51.482796+0000","last_active":"2024-03-30T15:10:23.489481+0000","last_peered":"2024-03-30T15:10:23.489481+0000","last_clean":"2024-03-30T15:08:49.461597+0000","last_became_active":"2024-03-30T15:08:51.482796+0000","last_became_peered":"2024-03-30T15:08:51.482796+0000","last_unstale":"2024-03-30T15:10:23.489481+0000","last_undegraded":"2024-03-30T15:08:51.469602+0000","last_fullsized":"2024-03-30T15:08:51.468739+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:06:42.502138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:06:42.502138+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.573854+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1429,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.459087+0000","last_change":"2024-03-30T15:09:04.368252+0000","last_active":"2024-03-30T15:10:23.459087+0000","last_peered":"2024-03-30T15:10:23.459087+0000","last_clean":"2024-03-30T15:10:23.459087+0000","last_became_active":"2024-03-30T14:54:37.330879+0000","last_became_peered":"2024-03-30T14:54:37.330879+0000","last_unstale":"2024-03-30T15:10:23.459087+0000","last_undegraded":"2024-03-30T15:10:23.459087+0000","last_fullsized":"2024-03-30T15:10:23.459087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:09:04.368135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:09:04.368135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1424,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.457799+0000","last_change":"2024-03-30T15:08:51.491047+0000","last_active":"2024-03-30T15:10:23.457799+0000","last_peered":"2024-03-30T15:10:23.457799+0000","last_clean":"2024-03-30T15:08:49.475950+0000","last_became_active":"2024-03-30T15:08:51.491047+0000","last_became_peered":"2024-03-30T15:08:51.491047+0000","last_unstale":"2024-03-30T15:10:23.457799+0000","last_undegraded":"2024-03-30T15:10:23.457799+0000","last_fullsized":"2024-03-30T15:08:51.477687+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:20.313498+0000","last_clean_scrub_stamp":"2024-03-30T15:08:20.313498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.453905+0000","last_change":"2024-03-30T15:08:51.479487+0000","last_active":"2024-03-30T15:10:23.453905+0000","last_peered":"2024-03-30T15:10:23.453905+0000","last_clean":"2024-03-30T14:54:42.332090+0000","last_became_active":"2024-03-30T15:08:51.479487+0000","last_became_peered":"2024-03-30T15:08:51.479487+0000","last_unstale":"2024-03-30T15:10:23.453905+0000","last_undegraded":"2024-03-30T15:10:23.453905+0000","last_fullsized":"2024-03-30T15:08:51.475248+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:04:47.569001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:04:47.569001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.412506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1335,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.457605+0000","last_change":"2024-03-30T15:10:20.766537+0000","last_active":"2024-03-30T15:10:23.457605+0000","last_peered":"2024-03-30T15:10:23.457605+0000","last_clean":"2024-03-30T15:10:23.457605+0000","last_became_active":"2024-03-30T14:57:55.104653+0000","last_became_peered":"2024-03-30T14:57:55.104653+0000","last_unstale":"2024-03-30T15:10:23.457605+0000","last_undegraded":"2024-03-30T15:10:23.457605+0000","last_fullsized":"2024-03-30T15:10:23.457605+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:20.766455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:10:20.766455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:41.687809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1448,"reported_epoch":761,"state":"active+undersized+degraded","last_fresh":"2024-03-30T15:10:23.442550+0000","last_change":"2024-03-30T15:08:51.480330+0000","last_active":"2024-03-30T15:10:23.442550+0000","last_peered":"2024-03-30T15:10:23.442550+0000","last_clean":"2024-03-30T15:08:49.451959+0000","last_became_active":"2024-03-30T15:08:51.480330+0000","last_became_peered":"2024-03-30T15:08:51.480330+0000","last_unstale":"2024-03-30T15:10:23.442550+0000","last_undegraded":"2024-03-30T15:08:51.468832+0000","last_fullsized":"2024-03-30T15:08:51.468722+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:07:29.102268+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:07:29.102268+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:26.135266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1337,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453037+0000","last_change":"2024-03-30T15:08:17.607910+0000","last_active":"2024-03-30T15:10:23.453037+0000","last_peered":"2024-03-30T15:10:23.453037+0000","last_clean":"2024-03-30T15:10:23.453037+0000","last_became_active":"2024-03-30T14:57:56.274216+0000","last_became_peered":"2024-03-30T14:57:56.274216+0000","last_unstale":"2024-03-30T15:10:23.453037+0000","last_undegraded":"2024-03-30T15:10:23.453037+0000","last_fullsized":"2024-03-30T15:10:23.453037+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:17.607819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:08:17.607819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1423,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.488849+0000","last_change":"2024-03-30T15:08:56.393797+0000","last_active":"2024-03-30T15:10:23.488849+0000","last_peered":"2024-03-30T15:10:23.488849+0000","last_clean":"2024-03-30T15:10:23.488849+0000","last_became_active":"2024-03-30T14:54:41.350472+0000","last_became_peered":"2024-03-30T14:54:41.350472+0000","last_unstale":"2024-03-30T15:10:23.488849+0000","last_undegraded":"2024-03-30T15:10:23.488849+0000","last_fullsized":"2024-03-30T15:10:23.488849+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:56.393609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:08:56.393609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1010,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.458062+0000","last_change":"2024-03-30T15:10:08.695389+0000","last_active":"2024-03-30T15:10:23.458062+0000","last_peered":"2024-03-30T15:10:23.458062+0000","last_clean":"2024-03-30T15:10:23.458062+0000","last_became_active":"2024-03-30T15:01:40.313517+0000","last_became_peered":"2024-03-30T15:01:40.313517+0000","last_unstale":"2024-03-30T15:10:23.458062+0000","last_undegraded":"2024-03-30T15:10:23.458062+0000","last_fullsized":"2024-03-30T15:10:23.458062+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:08.695264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:10:08.695264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:11:15.656684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.442494+0000","last_change":"2024-03-30T15:08:51.485711+0000","last_active":"2024-03-30T15:10:23.442494+0000","last_peered":"2024-03-30T15:10:23.442494+0000","last_clean":"2024-03-30T14:57:46.318935+0000","last_became_active":"2024-03-30T15:08:51.485711+0000","last_became_peered":"2024-03-30T15:08:51.485711+0000","last_unstale":"2024-03-30T15:10:23.442494+0000","last_undegraded":"2024-03-30T15:10:23.442494+0000","last_fullsized":"2024-03-30T15:08:51.474434+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:05:15.581095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:05:15.581095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:24.988371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1473,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489000+0000","last_change":"2024-03-30T15:09:39.480441+0000","last_active":"2024-03-30T15:10:23.489000+0000","last_peered":"2024-03-30T15:10:23.489000+0000","last_clean":"2024-03-30T15:10:23.489000+0000","last_became_active":"2024-03-30T15:02:19.650995+0000","last_became_peered":"2024-03-30T15:02:19.650995+0000","last_unstale":"2024-03-30T15:10:23.489000+0000","last_undegraded":"2024-03-30T15:10:23.489000+0000","last_fullsized":"2024-03-30T15:10:23.489000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:09:39.480305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:09:39.480305+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:10:44.501110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":189,"reported_epoch":761,"state":"active+undersized","last_fresh":"2024-03-30T15:10:23.460231+0000","last_change":"2024-03-30T15:08:51.487625+0000","last_active":"2024-03-30T15:10:23.460231+0000","last_peered":"2024-03-30T15:10:23.460231+0000","last_clean":"2024-03-30T14:54:36.304294+0000","last_became_active":"2024-03-30T15:08:51.487625+0000","last_became_peered":"2024-03-30T15:08:51.487625+0000","last_unstale":"2024-03-30T15:10:23.460231+0000","last_undegraded":"2024-03-30T15:10:23.460231+0000","last_fullsized":"2024-03-30T15:08:51.480237+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1424,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.453731+0000","last_change":"2024-03-30T15:07:32.100519+0000","last_active":"2024-03-30T15:10:23.453731+0000","last_peered":"2024-03-30T15:10:23.453731+0000","last_clean":"2024-03-30T15:10:23.453731+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:10:23.453731+0000","last_undegraded":"2024-03-30T15:10:23.453731+0000","last_fullsized":"2024-03-30T15:10:23.453731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:32.100436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:07:32.100436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1416,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T15:10:23.489234+0000","last_change":"2024-03-30T15:08:54.376225+0000","last_active":"2024-03-30T15:10:23.489234+0000","last_peered":"2024-03-30T15:10:23.489234+0000","last_clean":"2024-03-30T15:10:23.489234+0000","last_became_active":"2024-03-30T14:54:43.351243+0000","last_became_peered":"2024-03-30T14:54:43.351243+0000","last_unstale":"2024-03-30T15:10:23.489234+0000","last_undegraded":"2024-03-30T15:10:23.489234+0000","last_fullsized":"2024-03-30T15:10:23.489234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:54.376084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:08:54.376084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":335,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":1,"num_flush_kb":3907,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":82,"acting":82,"num_store_stats":7},{"poolid":42,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":88,"acting":88,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":937,"num_read_kb":932,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":29,"acting":29,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":747,"ondisk_log_size":747,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":83,"acting":84,"num_store_stats":8},{"poolid":322,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6189056,"data_stored":6189056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":87,"acting":87,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005250,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96268,"kb_used_data":6780,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93648884,"statfs":{"total":95995035648,"available":95896457216,"internally_reserved":0,"allocated":6942720,"data_stored":3521364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299591,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100056,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93645096,"statfs":{"total":95995035648,"available":95892578304,"internally_reserved":0,"allocated":6565888,"data_stored":3174962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575074,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95956,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93649196,"statfs":{"total":95995035648,"available":95896776704,"internally_reserved":0,"allocated":6627328,"data_stored":3200740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":54128,"kb_used_data":4904,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93691024,"statfs":{"total":95995035648,"available":95939608576,"internally_reserved":0,"allocated":5021696,"data_stored":2240299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411547,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118376,"kb_used_data":6676,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93626776,"statfs":{"total":95995035648,"available":95873818624,"internally_reserved":0,"allocated":6836224,"data_stored":3442076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117207,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102688,"kb_used_data":9040,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93642464,"statfs":{"total":95995035648,"available":95889883136,"internally_reserved":0,"allocated":9256960,"data_stored":5808887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95877575},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953681,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100112,"kb_used_data":6468,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93645040,"statfs":{"total":95995035648,"available":95892520960,"internally_reserved":0,"allocated":6623232,"data_stored":3198012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659340,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100664,"kb_used_data":7016,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93644488,"statfs":{"total":95995035648,"available":95891955712,"internally_reserved":0,"allocated":7184384,"data_stored":3774201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":42,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2039808,"data_stored":2039808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2019328,"data_stored":2019328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2043904,"data_stored":2043904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":322,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":335,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:10:26.794 INFO:tasks.thrashosds.thrasher:Setting 6.4 to [1, 4, 3] 2024-03-30T15:10:26.794 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.4', '1', '4', '3'] 2024-03-30T15:10:26.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pg-upmap 6.4 1 4 3 2024-03-30T15:10:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:26.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:26.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:26.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:26.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:10:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-136"}]': finished 2024-03-30T15:10:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: osdmap e763: 8 total, 7 up, 8 in 2024-03-30T15:10:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: 6.2 deep-scrub starts 2024-03-30T15:10:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: 6.2 deep-scrub ok 2024-03-30T15:10:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:10:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:10:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:26 smithi134 ceph-mon[33714]: pgmap v1134: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 65 active+undersized, 162 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:27.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:27.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:27.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:27.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:27.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:27.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:27.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:27.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:27.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:27.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: osdmap e764: 8 total, 7 up, 8 in 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 42.1f scrub starts 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 42.1f scrub ok 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 42.f scrub starts 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: 42.f scrub ok 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: osdmap e765: 8 total, 7 up, 8 in 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: osdmap e764: 8 total, 7 up, 8 in 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 42.1f scrub starts 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 42.1f scrub ok 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 42.f scrub starts 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: 42.f scrub ok 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:27.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: osdmap e765: 8 total, 7 up, 8 in 2024-03-30T15:10:27.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:10:27.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:27.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:27.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:27.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: osdmap e764: 8 total, 7 up, 8 in 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 42.1f scrub starts 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 42.1f scrub ok 2024-03-30T15:10:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 42.f scrub starts 2024-03-30T15:10:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: 42.f scrub ok 2024-03-30T15:10:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 62/648 objects degraded (9.568%), 18 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: osdmap e765: 8 total, 7 up, 8 in 2024-03-30T15:10:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:27.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:27.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:28.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:28.097 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:28.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:28.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:28.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:28.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:28.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:28.468 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:28.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:10:28.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:10:28.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 42.a scrub starts 2024-03-30T15:10:28.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 42.a scrub ok 2024-03-30T15:10:28.469 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: pgmap v1137: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: osdmap e766: 8 total, 7 up, 8 in 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:28.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:28.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 42.a scrub starts 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 42.a scrub ok 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: pgmap v1137: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: osdmap e766: 8 total, 7 up, 8 in 2024-03-30T15:10:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:28.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:28.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:10:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:10:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 42.a scrub starts 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 42.a scrub ok 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: pgmap v1137: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: osdmap e766: 8 total, 7 up, 8 in 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:28.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:28.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:29.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:29.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:29.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:29.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:10:29.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:29.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:29.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: 42.d scrub starts 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: 42.d scrub ok 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: osdmap e767: 8 total, 7 up, 8 in 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: 42.d scrub starts 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: 42.d scrub ok 2024-03-30T15:10:29.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:10:29.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:10:29.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:29.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: osdmap e767: 8 total, 7 up, 8 in 2024-03-30T15:10:29.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:29 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:29.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:29.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:29.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:29.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:29.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:29.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: 42.d scrub starts 2024-03-30T15:10:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: 42.d scrub ok 2024-03-30T15:10:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:10:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:10:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: osdmap e767: 8 total, 7 up, 8 in 2024-03-30T15:10:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:29 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:10:29.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:30.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:30.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:30.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:30.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:30.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:30.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:30.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[41614]: pgmap v1140: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[41614]: osdmap e768: 8 total, 7 up, 8 in 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[29876]: pgmap v1140: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:30.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:10:30.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[29876]: osdmap e768: 8 total, 7 up, 8 in 2024-03-30T15:10:30.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:30.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:30.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:30.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:30.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:30.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:30.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:30.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:30.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:30.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-mon[33714]: pgmap v1140: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:10:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-mon[33714]: osdmap e768: 8 total, 7 up, 8 in 2024-03-30T15:10:30.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:10:31.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 -- 172.21.15.28:0/2783509714 >> v1:172.21.15.28:6790/0 conn(0x7f063410aee0 legacy=0x7f063410d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 -- 172.21.15.28:0/2783509714 shutdown_connections 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 -- 172.21.15.28:0/2783509714 >> 172.21.15.28:0/2783509714 conn(0x7f06340fdb20 msgr2=0x7f06340fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 -- 172.21.15.28:0/2783509714 shutdown_connections 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 -- 172.21.15.28:0/2783509714 wait complete. 2024-03-30T15:10:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 librados: starting msgr at 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.083+0000 7f063c4cc700 1 librados: starting objecter 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 Processor -- start 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 -- start start 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 librados: setting wanted keys 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 librados: calling monclient init 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06341078c0 con 0x7f063410aee0 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0634107ad0 con 0x7f0634102250 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063c4cc700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f06341a9660 con 0x7f063410ea80 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063aa69700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f063410ea80 0x7f06341a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36604/0 (socket says 172.21.15.28:36604) 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f063aa69700 1 -- 172.21.15.28:0/4226926958 learned_addr learned my addr 172.21.15.28:0/4226926958 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f0639a67700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f063410aee0 0x7f0634104e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47596/0 (socket says 172.21.15.28:47596) 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter ms_handle_connect 0x7f063410ea80 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter ms_handle_connect 0x7f063410aee0 2024-03-30T15:10:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter ms_handle_connect 0x7f0634102250 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1180078392 0 0) 0x7f06341a9660 con 0x7f063410ea80 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f061c0031f0 con 0x7f063410ea80 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263272656 0 0) 0x7f06341078c0 con 0x7f063410aee0 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06341a9660 con 0x7f063410aee0 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52124430 0 0) 0x7f0634107ad0 con 0x7f0634102250 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06341078c0 con 0x7f0634102250 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4120832006 0 0) 0x7f061c0031f0 con 0x7f063410ea80 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0634107ad0 con 0x7f063410ea80 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.084+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f06240018b0 con 0x7f063410ea80 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 683830621 0 0) 0x7f06341a9660 con 0x7f063410aee0 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f061c0031f0 con 0x7f063410aee0 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0628004440 con 0x7f063410aee0 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4272705939 0 0) 0x7f06341078c0 con 0x7f0634102250 2024-03-30T15:10:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06341a9660 con 0x7f0634102250 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f062c001c10 con 0x7f0634102250 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 463332575 0 0) 0x7f0634107ad0 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 >> v1:172.21.15.134:6789/0 conn(0x7f0634102250 legacy=0x7f0634104710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 >> v1:172.21.15.28:6789/0 conn(0x7f063410aee0 legacy=0x7f0634104e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06341aa880 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06341a9890 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f063c4cc700 10 client.58760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f06341a9d50 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f063c4cc700 1 librados: init done 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0624003b40 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.085+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0624005c20 con 0x7f063410ea80 2024-03-30T15:10:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.086+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0620005120 con 0x7f063410ea80 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.086+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f062401c2c0 con 0x7f063410ea80 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(768..768 src has 442..768) ==== 9237+0+0 (unknown 1790310445 0 0) 0x7f0624089960 con 0x7f063410ea80 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 10 client.58760.objecter ms_dispatch 0x7f0634001550 osd_map(768..768 src has 442..768) 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 3 client.58760.objecter handle_osd_map got epochs [768,768] > 0 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 3 client.58760.objecter handle_osd_map decoding full epoch 768 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 20 client.58760.objecter dump_active .. 0 homeless 2024-03-30T15:10:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.087+0000 7f06337fe700 10 client.58760.objecter ms_handle_connect 0x7f061c070430 2024-03-30T15:10:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.092+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0624056740 con 0x7f063410ea80 2024-03-30T15:10:31.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:31.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:31.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.309+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f0620005410 con 0x7f063410ea80 2024-03-30T15:10:31.336 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:31.336 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.398+0000 7f06337fe700 1 -- 172.21.15.28:0/4226926958 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v769) ==== 87+0+0 (unknown 6623129 0 0) 0x7f062405a190 con 0x7f063410ea80 2024-03-30T15:10:31.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 10 librados: watch_flush enter 2024-03-30T15:10:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 10 librados: watch_flush exit 2024-03-30T15:10:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 20 client.58760.objecter shutdown clearing up homeless session... 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 10 client.58760.objecter successfully canceled tick 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 >> v1:172.21.15.28:6800/3011431093 conn(0x7f061c070430 legacy=0x7f061c0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 >> v1:172.21.15.28:6790/0 conn(0x7f063410ea80 legacy=0x7f06341a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 shutdown_connections 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 >> 172.21.15.28:0/4226926958 conn(0x7f06340fdb20 msgr2=0x7f0634111c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 shutdown_connections 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 -- 172.21.15.28:0/4226926958 wait complete. 2024-03-30T15:10:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:31.402+0000 7f063c4cc700 1 librados: shutdown 2024-03-30T15:10:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:10:31.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:31.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:31.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:31.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:31.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: 322.2s0 scrub starts 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: 322.2 scrub ok 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4226926958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.58760 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.58760 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:10:31.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: osdmap e769: 8 total, 7 up, 8 in 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: 322.2s0 scrub starts 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: 322.2 scrub ok 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4226926958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.58760 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:31.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:10:31.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.58760 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:10:31.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: osdmap e769: 8 total, 7 up, 8 in 2024-03-30T15:10:31.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:31.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:31.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: 322.2s0 scrub starts 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: 322.2 scrub ok 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:10:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4226926958' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.58760 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.58760 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: osdmap e769: 8 total, 7 up, 8 in 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-30T15:10:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:31.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:32.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:32.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:32.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:32.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:32.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:32.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:32.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:32.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:32.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:32.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:32.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:32.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[29876]: pgmap v1143: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:10:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[29876]: osdmap e770: 8 total, 7 up, 8 in 2024-03-30T15:10:32.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[41614]: pgmap v1143: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:32.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:10:32.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:32 smithi028 ceph-mon[41614]: osdmap e770: 8 total, 7 up, 8 in 2024-03-30T15:10:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:32 smithi134 ceph-mon[33714]: pgmap v1143: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-30T15:10:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:32 smithi134 ceph-mon[33714]: osdmap e770: 8 total, 7 up, 8 in 2024-03-30T15:10:32.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:32.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:32.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:33.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:33.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:33.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- 172.21.15.28:0/2054198063 >> v1:172.21.15.28:6790/0 conn(0x7f3114101130 legacy=0x7f3114101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- 172.21.15.28:0/2054198063 shutdown_connections 2024-03-30T15:10:33.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- 172.21.15.28:0/2054198063 >> 172.21.15.28:0/2054198063 conn(0x7f31140fc9e0 msgr2=0x7f31140fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:33.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- 172.21.15.28:0/2054198063 shutdown_connections 2024-03-30T15:10:33.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- 172.21.15.28:0/2054198063 wait complete. 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 librados: starting msgr at 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 librados: starting objecter 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 Processor -- start 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 -- start start 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 librados: setting wanted keys 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.291+0000 7f311ac79700 1 librados: calling monclient init 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f311ac79700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31141a5fe0 con 0x7f3114109dc0 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f311ac79700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31141a7200 con 0x7f311410d960 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f311ac79700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31141a8440 con 0x7f3114101130 2024-03-30T15:10:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3118a15700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3114101130 0x7f3114106480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36628/0 (socket says 172.21.15.28:36628) 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3113fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3114109dc0 0x7f31141a0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47606/0 (socket says 172.21.15.28:47606) 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3118a15700 1 -- 172.21.15.28:0/1480636172 learned_addr learned my addr 172.21.15.28:0/1480636172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter ms_handle_connect 0x7f3114101130 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter ms_handle_connect 0x7f3114109dc0 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter ms_handle_connect 0x7f311410d960 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964226917 0 0) 0x7f31141a8440 con 0x7f3114101130 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3100003aa0 con 0x7f3114101130 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1800160828 0 0) 0x7f31141a5fe0 con 0x7f3114109dc0 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31141a8440 con 0x7f3114109dc0 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458406961 0 0) 0x7f31141a7200 con 0x7f311410d960 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31141a5fe0 con 0x7f311410d960 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4020454115 0 0) 0x7f3100003aa0 con 0x7f3114101130 2024-03-30T15:10:33.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31141a7200 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f310c001990 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.292+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3499144498 0 0) 0x7f31141a8440 con 0x7f3114109dc0 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3100003aa0 con 0x7f3114109dc0 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 993299683 0 0) 0x7f31141a5fe0 con 0x7f311410d960 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31141a8440 con 0x7f311410d960 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3104001830 con 0x7f3114109dc0 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30fc0017c0 con 0x7f311410d960 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 578149180 0 0) 0x7f31141a7200 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 >> v1:172.21.15.134:6789/0 conn(0x7f311410d960 legacy=0x7f31141a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 >> v1:172.21.15.28:6789/0 conn(0x7f3114109dc0 legacy=0x7f31141a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31141a9680 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31141a6210 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f310c003ec0 con 0x7f3114101130 2024-03-30T15:10:33.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f310c005520 con 0x7f3114101130 2024-03-30T15:10:33.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f311ac79700 10 client.58766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:33.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f31141a66d0 con 0x7f3114101130 2024-03-30T15:10:33.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.293+0000 7f311ac79700 1 librados: init done 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.294+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3114102610 con 0x7f3114101130 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f310c005500 con 0x7f3114101130 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(770..770 src has 442..770) ==== 9237+0+0 (unknown 1308062512 0 0) 0x7f310c08a0a0 con 0x7f3114101130 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 10 client.58766.objecter ms_dispatch 0x7f3114001550 osd_map(770..770 src has 442..770) 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 3 client.58766.objecter handle_osd_map got epochs [770,770] > 0 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 3 client.58766.objecter handle_osd_map decoding full epoch 770 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.295+0000 7f3111ffb700 20 client.58766.objecter dump_active .. 0 homeless 2024-03-30T15:10:33.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.296+0000 7f3111ffb700 10 client.58766.objecter ms_handle_connect 0x7f3100074990 2024-03-30T15:10:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.304+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f310c056e60 con 0x7f3114101130 2024-03-30T15:10:33.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:33.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:33.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:33.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:33.538+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]} v 0) -- 0x7f3114068b30 con 0x7f3114101130 2024-03-30T15:10:33.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:33.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:33.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:33.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:33.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:33.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.864 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:33.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:10:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]: dispatch 2024-03-30T15:10:33.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:33.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:33.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:34.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:34.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:34.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:10:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:34.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:34.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:34.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:34.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.494+0000 7f3111ffb700 1 -- 172.21.15.28:0/1480636172 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]=0 set 6.4 pg_upmap mapping to [1,4,3] v772) ==== 127+0+0 (unknown 2840745531 0 0) 0x7f310c001490 con 0x7f3114101130 2024-03-30T15:10:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.498+0000 7f311ac79700 10 librados: watch_flush enter 2024-03-30T15:10:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.498+0000 7f311ac79700 10 librados: watch_flush exit 2024-03-30T15:10:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.498+0000 7f311ac79700 20 client.58766.objecter shutdown clearing up homeless session... 2024-03-30T15:10:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.498+0000 7f311ac79700 10 client.58766.objecter successfully canceled tick 2024-03-30T15:10:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:34.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.498+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3100074990 legacy=0x7f3100076e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.499+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 >> v1:172.21.15.28:6790/0 conn(0x7f3114101130 legacy=0x7f3114106480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:34.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.502+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 shutdown_connections 2024-03-30T15:10:34.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.502+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 >> 172.21.15.28:0/1480636172 conn(0x7f31140fc9e0 msgr2=0x7f3114110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.504+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 shutdown_connections 2024-03-30T15:10:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.505+0000 7f311ac79700 1 -- 172.21.15.28:0/1480636172 wait complete. 2024-03-30T15:10:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:34.505+0000 7f311ac79700 1 librados: shutdown 2024-03-30T15:10:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:set 6.4 pg_upmap mapping to [1,4,3] 2024-03-30T15:10:34.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:34.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:34.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:34.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]': finished 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[29876]: osdmap e771: 8 total, 7 up, 8 in 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1480636172' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[29876]: from='client.58766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[29876]: pgmap v1146: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]': finished 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[41614]: osdmap e771: 8 total, 7 up, 8 in 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1480636172' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[41614]: from='client.58766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:34 smithi028 ceph-mon[41614]: pgmap v1146: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:34.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:34.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:34.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-138"}]': finished 2024-03-30T15:10:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:34 smithi134 ceph-mon[33714]: osdmap e771: 8 total, 7 up, 8 in 2024-03-30T15:10:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1480636172' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:34 smithi134 ceph-mon[33714]: from='client.58766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]: dispatch 2024-03-30T15:10:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:34 smithi134 ceph-mon[33714]: pgmap v1146: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 61 active+undersized, 166 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:34.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:35.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:35.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:35.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:35.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:35.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:35.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:35.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:35.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:35.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:35.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:35.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:35 smithi028 ceph-mon[41614]: from='client.58766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]': finished 2024-03-30T15:10:35.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:35 smithi028 ceph-mon[41614]: osdmap e772: 8 total, 7 up, 8 in 2024-03-30T15:10:35.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:35 smithi028 ceph-mon[29876]: from='client.58766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]': finished 2024-03-30T15:10:35.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:35 smithi028 ceph-mon[29876]: osdmap e772: 8 total, 7 up, 8 in 2024-03-30T15:10:35.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:35.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:35.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:10:35.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:35 smithi134 ceph-mon[33714]: from='client.58766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.4", "id": [1, 4, 3]}]': finished 2024-03-30T15:10:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:35 smithi134 ceph-mon[33714]: osdmap e772: 8 total, 7 up, 8 in 2024-03-30T15:10:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:35.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:36.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:36.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:36.182 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:36.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:36.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:36.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:36.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:36.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.643+0000 7fc7e328e700 1 -- 172.21.15.28:0/1525742472 >> v1:172.21.15.28:6790/0 conn(0x7fc7dc101130 legacy=0x7fc7dc101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:36.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.643+0000 7fc7e328e700 1 -- 172.21.15.28:0/1525742472 shutdown_connections 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.643+0000 7fc7e328e700 1 -- 172.21.15.28:0/1525742472 >> 172.21.15.28:0/1525742472 conn(0x7fc7dc0fc9e0 msgr2=0x7fc7dc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- 172.21.15.28:0/1525742472 shutdown_connections 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- 172.21.15.28:0/1525742472 wait complete. 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 librados: starting msgr at 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 librados: starting objecter 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 Processor -- start 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- start start 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 librados: setting wanted keys 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 librados: calling monclient init 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7dc10fa30 con 0x7fc7dc10d960 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7dc10fc40 con 0x7fc7dc101130 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.644+0000 7fc7e328e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7dc1ab940 con 0x7fc7dc109dc0 2024-03-30T15:10:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7e0829700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc7dc109dc0 0x7fc7dc10e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36632/0 (socket says 172.21.15.28:36632) 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7e0829700 1 -- 172.21.15.28:0/3882922384 learned_addr learned my addr 172.21.15.28:0/3882922384 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter ms_handle_connect 0x7fc7dc10d960 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter ms_handle_connect 0x7fc7dc109dc0 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter ms_handle_connect 0x7fc7dc101130 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075409076 0 0) 0x7fc7dc1ab940 con 0x7fc7dc109dc0 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c80031f0 con 0x7fc7dc109dc0 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868506633 0 0) 0x7fc7dc10fa30 con 0x7fc7dc10d960 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc1ab940 con 0x7fc7dc10d960 2024-03-30T15:10:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463544112 0 0) 0x7fc7dc10fc40 con 0x7fc7dc101130 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc10fa30 con 0x7fc7dc101130 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3595070647 0 0) 0x7fc7c80031f0 con 0x7fc7dc109dc0 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc10fc40 con 0x7fc7dc109dc0 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7cc001230 con 0x7fc7dc109dc0 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 689332110 0 0) 0x7fc7dc1ab940 con 0x7fc7dc10d960 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7c80031f0 con 0x7fc7dc10d960 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7d4001af0 con 0x7fc7dc10d960 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3995060689 0 0) 0x7fc7dc10fa30 con 0x7fc7dc101130 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc1ab940 con 0x7fc7dc101130 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7d8001800 con 0x7fc7dc101130 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2482085569 0 0) 0x7fc7c80031f0 con 0x7fc7dc10d960 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 >> v1:172.21.15.28:6790/0 conn(0x7fc7dc109dc0 legacy=0x7fc7dc10e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 >> v1:172.21.15.134:6789/0 conn(0x7fc7dc101130 legacy=0x7fc7dc10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc1acb20 con 0x7fc7dc10d960 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.645+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc1abb10 con 0x7fc7dc10d960 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.646+0000 7fc7e328e700 10 client.25572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.646+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7dc1abfd0 con 0x7fc7dc10d960 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.646+0000 7fc7e328e700 1 librados: init done 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.646+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc7d4002060 con 0x7fc7dc10d960 2024-03-30T15:10:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.646+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc7d4004d40 con 0x7fc7dc10d960 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.647+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc7c0005100 con 0x7fc7dc10d960 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.647+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc7d4006010 con 0x7fc7dc10d960 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(774..774 src has 442..774) ==== 9275+0+0 (unknown 1377239932 0 0) 0x7fc7d4088c80 con 0x7fc7dc10d960 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 osd_map(774..774 src has 442..774) 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 3 client.25572.objecter handle_osd_map got epochs [774,774] > 0 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 3 client.25572.objecter handle_osd_map decoding full epoch 774 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 20 client.25572.objecter dump_active .. 0 homeless 2024-03-30T15:10:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v774) ==== 54+0+16 (unknown 426413159 0 2420000192) 0x7fc7d408b0b0 con 0x7fc7dc10d960 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c8070430 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 10 client.25572.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 1 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 10 client.25572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=775}) -- 0x7fc7c00053e0 con 0x7fc7dc10d960 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc7c0000e30 con 0x7fc7dc10d960 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=774) ==== 24+0+0 (unknown 807087230 0 0) 0x7fc7d40550e0 con 0x7fc7dc10d960 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d37fe700 10 client.25572.objecter _check_command_map_dne tid 1 current 774 map_dne_bound 774 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d37fe700 10 client.25572.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7d37fe700 15 client.25572.objecter _session_command_op_remove -1 1 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 10 client.25572.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 2 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 2 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 1 2 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 4 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 10 client.25572.objecter _send_command 2 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.648+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fc7c0002360 con 0x7fc7c0002790 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c0002790 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c0002360 con 0x7fc7c0002790 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 2 = system:0 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 4 2024-03-30T15:10:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.649+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 1 2 2024-03-30T15:10:36.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 10 client.25572.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0001bb0 osd=1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 1 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 4 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 10 client.25572.objecter _send_command 3 2024-03-30T15:10:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.666+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c0004cf0 con 0x7fc7c0002790 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c0004cf0 con 0x7fc7c0002790 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0001bb0 osd=1 4 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 1 3 2024-03-30T15:10:36.669 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 10 client.25572.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 4 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 4 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 2 4 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 4 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 10 client.25572.objecter _send_command 4 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6809/3451325043 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fc7c000cb30 con 0x7fc7c0009020 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.668+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c0009020 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.669+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.2 v1:172.21.15.28:6809/3451325043 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c000cb30 con 0x7fc7c0009020 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.669+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 4: 0 ) 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.669+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 4 = system:0 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.669+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 4 2024-03-30T15:10:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.669+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 2 4 2024-03-30T15:10:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:36.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 10 client.25572.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 5 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 5 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0001970 osd=2 3 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 2 5 2024-03-30T15:10:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 4 2024-03-30T15:10:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 10 client.25572.objecter _send_command 5 2024-03-30T15:10:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.686+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6809/3451325043 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c000de90 con 0x7fc7c0009020 2024-03-30T15:10:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.2 v1:172.21.15.28:6809/3451325043 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c000de90 con 0x7fc7c0009020 2024-03-30T15:10:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0001970 osd=2 4 2024-03-30T15:10:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 2 5 2024-03-30T15:10:36.690 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 10 client.25572.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 6 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 6 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 3 6 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 4 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 10 client.25572.objecter _send_command 6 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.687+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fc7c0003f20 con 0x7fc7c0003360 2024-03-30T15:10:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c0003360 2024-03-30T15:10:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c0003f20 con 0x7fc7c0003360 2024-03-30T15:10:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 6: 0 ) 2024-03-30T15:10:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 6 = system:0 2024-03-30T15:10:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 4 2024-03-30T15:10:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.688+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 3 6 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 10 client.25572.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 7 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 7 2024-03-30T15:10:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000e290 osd=3 3 2024-03-30T15:10:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 3 7 2024-03-30T15:10:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 4 2024-03-30T15:10:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 10 client.25572.objecter _send_command 7 2024-03-30T15:10:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.705+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c0004920 con 0x7fc7c0003360 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.706+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c0004920 con 0x7fc7c0003360 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.706+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.706+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.706+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000e290 osd=3 4 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.706+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 3 7 2024-03-30T15:10:36.708 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 10 client.25572.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 8 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 8 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 4 8 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 4 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 10 client.25572.objecter _send_command 8 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fc7c00167b0 con 0x7fc7c0012c00 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.707+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c0012c00 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.708+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c00167b0 con 0x7fc7c0012c00 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.708+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 8: 0 ) 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.708+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 8 = system:0 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.708+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 4 2024-03-30T15:10:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.708+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 4 8 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 10 client.25572.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 9 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 9 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000edb0 osd=4 3 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 4 9 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 4 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 10 client.25572.objecter _send_command 9 2024-03-30T15:10:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.727+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6800/3063775514 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c00044c0 con 0x7fc7c0012c00 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.728+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c00044c0 con 0x7fc7c0012c00 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.728+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.728+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.728+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000edb0 osd=4 4 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.728+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 4 9 2024-03-30T15:10:36.730 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 10 client.25572.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 10 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 10 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 5 10 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 4 2024-03-30T15:10:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 10 client.25572.objecter _send_command 10 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fc7c000db00 con 0x7fc7c0004270 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.729+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c0004270 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.730+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c000db00 con 0x7fc7c0004270 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.730+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 10: 0 ) 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.730+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 10 = system:0 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.730+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 4 2024-03-30T15:10:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.730+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 5 10 2024-03-30T15:10:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 10 client.25572.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 11 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 11 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0017fa0 osd=5 3 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 5 11 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 4 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 10 client.25572.objecter _send_command 11 2024-03-30T15:10:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.746+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c000db00 con 0x7fc7c0004270 2024-03-30T15:10:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c000db00 con 0x7fc7c0004270 2024-03-30T15:10:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0017fa0 osd=5 4 2024-03-30T15:10:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 5 11 2024-03-30T15:10:36.750 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 10 client.25572.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 12 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 12 2024-03-30T15:10:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 6 12 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 4 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 10 client.25572.objecter _send_command 12 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.748+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fc7c001f520 con 0x7fc7c001b870 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.749+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c001b870 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.750+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c001f520 con 0x7fc7c001b870 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.750+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 12: 0 ) 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.750+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 12 = system:0 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.750+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 4 2024-03-30T15:10:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.750+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 6 12 2024-03-30T15:10:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 10 client.25572.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 13 2024-03-30T15:10:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 13 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c000db00 osd=6 3 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 6 13 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 4 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 10 client.25572.objecter _send_command 13 2024-03-30T15:10:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.766+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c0020ed0 con 0x7fc7c001b870 2024-03-30T15:10:36.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: osdmap e773: 8 total, 7 up, 8 in 2024-03-30T15:10:36.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.768 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: pgmap v1149: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[41614]: osdmap e774: 8 total, 7 up, 8 in 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: osdmap e773: 8 total, 7 up, 8 in 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: pgmap v1149: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:36.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:36 smithi028 ceph-mon[29876]: osdmap e774: 8 total, 7 up, 8 in 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c0020ed0 con 0x7fc7c001b870 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c000db00 osd=6 4 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 6 13 2024-03-30T15:10:36.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 10 client.25572.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 14 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 14 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 7 14 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 4 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 10 client.25572.objecter _send_command 14 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fc7c0021a70 con 0x7fc7c00215f0 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.768+0000 7fc7d27fc700 10 client.25572.objecter ms_handle_connect 0x7fc7c00215f0 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.769+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc7c0021a70 con 0x7fc7c00215f0 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.769+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 14: 0 ) 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.769+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 14 = system:0 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.769+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 4 2024-03-30T15:10:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.769+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 7 14 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 10 client.25572.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign -1 15 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 20 client.25572.objecter _get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_remove -1 15 2024-03-30T15:10:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 20 client.25572.objecter get_session s=0x7fc7c0020ed0 osd=7 3 2024-03-30T15:10:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 15 client.25572.objecter _session_command_op_assign 7 15 2024-03-30T15:10:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 4 2024-03-30T15:10:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 10 client.25572.objecter _send_command 15 2024-03-30T15:10:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.786+0000 7fc7e328e700 1 -- 172.21.15.28:0/3882922384 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc7c0021a70 con 0x7fc7c00215f0 2024-03-30T15:10:36.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.788+0000 7fc7d27fc700 1 -- 172.21.15.28:0/3882922384 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc7c0021a70 con 0x7fc7c00215f0 2024-03-30T15:10:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.788+0000 7fc7d27fc700 10 client.25572.objecter ms_dispatch 0x7fc7dc001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.788+0000 7fc7d27fc700 10 client.25572.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.788+0000 7fc7d27fc700 20 client.25572.objecter put_session s=0x7fc7c0020ed0 osd=7 4 2024-03-30T15:10:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:36.788+0000 7fc7d27fc700 15 client.25572.objecter _session_command_op_remove 7 15 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:36.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:36.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: osdmap e773: 8 total, 7 up, 8 in 2024-03-30T15:10:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:10:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: pgmap v1149: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:10:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:36 smithi134 ceph-mon[33714]: osdmap e774: 8 total, 7 up, 8 in 2024-03-30T15:10:36.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:36.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:37.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:37.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:37.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:37.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:37.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:37.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:37.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3882922384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: osdmap e775: 8 total, 7 up, 8 in 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:37.533 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3882922384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: osdmap e775: 8 total, 7 up, 8 in 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:37.534 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:37.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:37.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:37.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:37.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:37.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:37.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:37.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:10:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3882922384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:10:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: osdmap e775: 8 total, 7 up, 8 in 2024-03-30T15:10:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-30T15:10:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:37.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:38.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:38.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:38.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:38.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:38.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:38.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:38.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:38.517 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:38.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:38.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:38.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:38.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:38.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[29876]: pgmap v1152: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:38.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:10:38.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[29876]: osdmap e776: 8 total, 7 up, 8 in 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[41614]: pgmap v1152: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[41614]: osdmap e776: 8 total, 7 up, 8 in 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.821 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:38 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:38 smithi134 ceph-mon[33714]: pgmap v1152: 246 pgs: 32 unknown, 1 active+clean+remapped, 18 active+undersized+degraded, 51 active+undersized, 144 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 62/648 objects degraded (9.568%) 2024-03-30T15:10:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:38 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_count","val": "3"}]': finished 2024-03-30T15:10:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:38 smithi134 ceph-mon[33714]: osdmap e776: 8 total, 7 up, 8 in 2024-03-30T15:10:38.831 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:38 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-30T15:10:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:38.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:38.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:38.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:38.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:39.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:39.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:39.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:39.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:39.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:39.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:39.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:39.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:39.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:39.713 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:39.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:39.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:39.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:39.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:39.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:39.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:40.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:40.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:40.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:40.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:40.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:10:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.413+0000 7f9d54996700 1 -- 172.21.15.28:0/3642860169 >> v1:172.21.15.134:6789/0 conn(0x7f9d50109d30 legacy=0x7f9d5010f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.413+0000 7f9d54996700 1 -- 172.21.15.28:0/3642860169 shutdown_connections 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.413+0000 7f9d54996700 1 -- 172.21.15.28:0/3642860169 >> 172.21.15.28:0/3642860169 conn(0x7f9d5006d550 msgr2=0x7f9d5006d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 -- 172.21.15.28:0/3642860169 shutdown_connections 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 -- 172.21.15.28:0/3642860169 wait complete. 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 librados: starting msgr at 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 librados: starting objecter 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 Processor -- start 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 -- start start 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 librados: setting wanted keys 2024-03-30T15:10:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 librados: calling monclient init 2024-03-30T15:10:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.414+0000 7f9d54996700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d501b1a50 con 0x7f9d50110100 2024-03-30T15:10:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d54996700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d501b2c70 con 0x7f9d50109d30 2024-03-30T15:10:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d54996700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d501b3eb0 con 0x7f9d50109240 2024-03-30T15:10:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d4e59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9d50109240 0x7f9d5011ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45318/0 (socket says 172.21.15.28:45318) 2024-03-30T15:10:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d4e59c700 1 -- 172.21.15.28:0/852636156 learned_addr learned my addr 172.21.15.28:0/852636156 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter ms_handle_connect 0x7f9d50109240 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter ms_handle_connect 0x7f9d50110100 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter ms_handle_connect 0x7f9d50109d30 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372579458 0 0) 0x7f9d501b1a50 con 0x7f9d50110100 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d3c0031f0 con 0x7f9d50110100 2024-03-30T15:10:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622521925 0 0) 0x7f9d501b3eb0 con 0x7f9d50109240 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d501b1a50 con 0x7f9d50109240 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417629560 0 0) 0x7f9d501b2c70 con 0x7f9d50109d30 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d501b3eb0 con 0x7f9d50109d30 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 450672790 0 0) 0x7f9d3c0031f0 con 0x7f9d50110100 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d501b2c70 con 0x7f9d50110100 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d48001bd0 con 0x7f9d50110100 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 971673901 0 0) 0x7f9d501b1a50 con 0x7f9d50109240 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d3c0031f0 con 0x7f9d50109240 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 520493443 0 0) 0x7f9d501b3eb0 con 0x7f9d50109d30 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d501b1a50 con 0x7f9d50109d30 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d40001760 con 0x7f9d50109240 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d38001350 con 0x7f9d50109d30 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3791476760 0 0) 0x7f9d501b2c70 con 0x7f9d50110100 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 >> v1:172.21.15.28:6790/0 conn(0x7f9d50109240 legacy=0x7f9d5011ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 >> v1:172.21.15.134:6789/0 conn(0x7f9d50109d30 legacy=0x7f9d501ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d501b50f0 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.415+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d501b1c80 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.416+0000 7f9d54996700 10 client.25575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.416+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d501b2140 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.416+0000 7f9d54996700 1 librados: init done 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.416+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9d48003ec0 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.416+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d48004d00 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d30005120 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9d4801b620 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(778..778 src has 442..778) ==== 9296+0+0 (unknown 401515393 0 0) 0x7f9d48088b90 con 0x7f9d50110100 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d477fe700 10 client.25575.objecter ms_dispatch 0x7f9d50001550 osd_map(778..778 src has 442..778) 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d477fe700 3 client.25575.objecter handle_osd_map got epochs [778,778] > 0 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.417+0000 7f9d477fe700 3 client.25575.objecter handle_osd_map decoding full epoch 778 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.418+0000 7f9d477fe700 20 client.25575.objecter dump_active .. 0 homeless 2024-03-30T15:10:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.418+0000 7f9d477fe700 10 client.25575.objecter ms_handle_connect 0x7f9d3c0706b0 2024-03-30T15:10:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.422+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9d48055910 con 0x7f9d50110100 2024-03-30T15:10:40.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[29876]: osdmap e777: 8 total, 7 up, 8 in 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[29876]: pgmap v1155: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[41614]: osdmap e777: 8 total, 7 up, 8 in 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:40 smithi028 ceph-mon[41614]: pgmap v1155: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:40.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:40.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:40.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:40.642+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f9d30005410 con 0x7f9d50110100 2024-03-30T15:10:40.661 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:10:40.661 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:10:40.661 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:10:40.661 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:10:40.661 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:10:40.661 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:10:40.661 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:10:40.662 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 2, 3, 4, 5, 6, 7] 2024-03-30T15:10:40.662 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T15:10:40.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T15:10:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_period","val": "3"}]': finished 2024-03-30T15:10:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-mon[33714]: osdmap e777: 8 total, 7 up, 8 in 2024-03-30T15:10:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:10:40.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-mon[33714]: pgmap v1155: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:40.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:40.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:40.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:40.972 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:40 smithi028 systemd[1]: Stopping Ceph osd.2 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:10:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:40.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:41.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:41.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:41.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:41.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:41.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:41.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.407+0000 7f9d477fe700 1 -- 172.21.15.28:0/852636156 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v779) ==== 97+0+0 (unknown 208502748 0 0) 0x7f9d48059360 con 0x7f9d50110100 2024-03-30T15:10:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.410+0000 7f9d54996700 10 librados: watch_flush enter 2024-03-30T15:10:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.411+0000 7f9d54996700 10 librados: watch_flush exit 2024-03-30T15:10:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.411+0000 7f9d54996700 20 client.25575.objecter shutdown clearing up homeless session... 2024-03-30T15:10:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.411+0000 7f9d54996700 10 client.25575.objecter successfully canceled tick 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.412+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9d3c0706b0 legacy=0x7f9d3c072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.412+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 >> v1:172.21.15.28:6789/0 conn(0x7f9d50110100 legacy=0x7f9d501b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.412+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 shutdown_connections 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.412+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 >> 172.21.15.28:0/852636156 conn(0x7f9d5006d550 msgr2=0x7f9d5010a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.413+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 shutdown_connections 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.413+0000 7f9d54996700 1 -- 172.21.15.28:0/852636156 wait complete. 2024-03-30T15:10:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:41.413+0000 7f9d54996700 1 librados: shutdown 2024-03-30T15:10:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:10:41.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:41.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: osdmap e778: 8 total, 7 up, 8 in 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:41.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:41.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:10:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:41.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:10:41.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: osdmap e778: 8 total, 7 up, 8 in 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: osdmap e778: 8 total, 7 up, 8 in 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:10:41.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:41.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:10:41.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:41.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:41.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:41.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:42.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:42.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:42.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:42.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:42.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: osdmap e779: 8 total, 7 up, 8 in 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: pgmap v1158: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:10:42.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:42.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: osdmap e779: 8 total, 7 up, 8 in 2024-03-30T15:10:42.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: pgmap v1158: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:42.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:42.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:42.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/852636156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: osdmap e779: 8 total, 7 up, 8 in 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: pgmap v1158: 246 pgs: 1 active+clean+remapped, 18 active+undersized+degraded, 62 active+undersized, 165 active+clean; 4.4 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/648 objects degraded (9.568%) 2024-03-30T15:10:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:42 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:10:42.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:42.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:42.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:42.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:42.871 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[58365]: 2024-03-30T15:10:42.783+0000 7f51bcff7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:10:42.871 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[58365]: 2024-03-30T15:10:42.783+0000 7f51bcff7700 -1 osd.2 779 *** Got signal Terminated *** 2024-03-30T15:10:42.871 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:42 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[58365]: 2024-03-30T15:10:42.783+0000 7f51bcff7700 -1 osd.2 779 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:10:42.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:42.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:42.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:43.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:43.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:43.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:43.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:43.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:43.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:43.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:43.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:43.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:43.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:10:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:10:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[41614]: osd.2 marked itself down and dead 2024-03-30T15:10:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:10:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:10:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:43 smithi028 ceph-mon[29876]: osd.2 marked itself down and dead 2024-03-30T15:10:43.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:43.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:43.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:43.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:43 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:10:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:43 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:10:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:43 smithi134 ceph-mon[33714]: osd.2 marked itself down and dead 2024-03-30T15:10:43.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:44.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:44.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:44.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:44.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:44.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:44.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:44.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:44.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:44.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:44.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:44.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:44.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:10:44.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[41614]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[41614]: osdmap e780: 8 total, 6 up, 8 in 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[41614]: pgmap v1160: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 5.1 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[41614]: osdmap e781: 8 total, 6 up, 8 in 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[29876]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[29876]: osdmap e780: 8 total, 6 up, 8 in 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[29876]: pgmap v1160: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 5.1 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:44 smithi028 ceph-mon[29876]: osdmap e781: 8 total, 6 up, 8 in 2024-03-30T15:10:44.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:44.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:44.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:44.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:44.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:44 smithi134 ceph-mon[33714]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:10:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:44 smithi134 ceph-mon[33714]: osdmap e780: 8 total, 6 up, 8 in 2024-03-30T15:10:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:44 smithi134 ceph-mon[33714]: pgmap v1160: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 5.1 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:44 smithi134 ceph-mon[33714]: osdmap e781: 8 total, 6 up, 8 in 2024-03-30T15:10:44.972 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:44 smithi028 podman[277241]: 2024-03-30 15:10:44.706446985 +0000 UTC m=+2.053964166 container died 2a9758b8317074d91d2097d07e77ed361f6e0c9e6cf2843208cca38de9ef8583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:10:44.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:45.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:45.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:45.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:45.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:45.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:45.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:45.406 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:45 smithi028 podman[277241]: 2024-03-30 15:10:45.187535384 +0000 UTC m=+2.535052545 container cleanup 2a9758b8317074d91d2097d07e77ed361f6e0c9e6cf2843208cca38de9ef8583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-30T15:10:45.409 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:45 smithi028 bash[277241]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T15:10:45.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:45.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:45.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:45.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:45 smithi028 ceph-mon[41614]: Health check failed: Reduced data availability: 20 pgs inactive, 22 pgs peering (PG_AVAILABILITY) 2024-03-30T15:10:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:45 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 49/651 objects degraded (7.527%), 16 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:45.698 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:45 smithi028 podman[277379]: 2024-03-30 15:10:45.405736669 +0000 UTC m=+1.431143679 container remove 2a9758b8317074d91d2097d07e77ed361f6e0c9e6cf2843208cca38de9ef8583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0) 2024-03-30T15:10:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:45 smithi028 ceph-mon[29876]: Health check failed: Reduced data availability: 20 pgs inactive, 22 pgs peering (PG_AVAILABILITY) 2024-03-30T15:10:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:45 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 49/651 objects degraded (7.527%), 16 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(5.1,QREQUEST,e:627,from:4,reservation_nonce:10)", 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:07:57.206020+0000", 2024-03-30T15:10:45.699 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 168.48976860100001, 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019584609999999999, 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206020+0000", 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206020+0000", 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206023+0000", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.021e-06 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206033+0000", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5370000000000003e-06 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206052+0000", 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9585e-05 2024-03-30T15:10:45.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206065+0000", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2537e-05 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206148+0000", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2799999999999993e-05 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.206156+0000", 2024-03-30T15:10:45.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4109999999999996e-06 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.207826+0000", 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016698279999999999 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.207852+0000", 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6261000000000001e-05 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:07:57.207979+0000", 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012648100000000001 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.1s1 646/644 ECSubWrite(tid=1490, reqid=client.58535.0:2, at_version=646'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:15.372932+0000", 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 150.322856363, 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001196573, 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.704 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.372932+0000", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.372932+0000", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.372933+0000", 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8700000000000005e-07 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.372979+0000", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5606999999999999e-05 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.372990+0000", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1189000000000001e-05 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.373001+0000", 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0831e-05 2024-03-30T15:10:45.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.373101+0000", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100807 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.373112+0000", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0362e-05 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.374098+0000", 2024-03-30T15:10:45.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098614300000000009 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.374129+0000", 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0947e-05 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.1s1 646/644 ECSubWrite(tid=1491, reqid=client.58535.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=646'1))", 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:15.378495+0000", 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 150.317293957, 2024-03-30T15:10:45.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00077191300000000005, 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378495+0000", 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378495+0000", 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378495+0000", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.06e-07 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378509+0000", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3617e-05 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378518+0000", 2024-03-30T15:10:45.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8710000000000003e-06 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378527+0000", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2990000000000002e-06 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378595+0000", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7495000000000003e-05 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.378607+0000", 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1841999999999999e-05 2024-03-30T15:10:45.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379235+0000", 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062851900000000002 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:15.379267+0000", 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1563999999999998e-05 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpRead(322.1s1 648/644 ECSubRead(tid=1492, to_read={322:96df10bd:test-rados-api-smithi028-93253-113::foo:head=0,4096,0}, subchunks={322:96df10bd:test-rados-api-smithi028-93253-113::foo:head=[0,1]}, attrs_to_read=))", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.369995+0000", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 148.32579360400001, 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002339481, 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.369995+0000", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.369995+0000", 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.369995+0000", 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3500000000000002e-07 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.370003+0000", 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6009999999999997e-06 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.370015+0000", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2483999999999999e-05 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.370027+0000", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1618e-05 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.370094+0000", 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6637999999999995e-05 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.372335+0000", 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022408049999999998 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(4.4,QREQUEST,e:667,from:5,reservation_nonce:11)", 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:42.417695+0000", 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 123.27809403099999, 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00275362, 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417695+0000", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417695+0000", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417697+0000", 2024-03-30T15:10:45.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1720000000000001e-06 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417703+0000", 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1179999999999997e-06 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417718+0000", 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5133000000000001e-05 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417726+0000", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8369999999999997e-06 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417785+0000", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9463000000000003e-05 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.417790+0000", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1379999999999998e-06 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.420386+0000", 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002595712 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.420403+0000", 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7102999999999999e-05 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:42.420448+0000", 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4944000000000002e-05 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.1s1 678/644 ECSubWrite(tid=1508, reqid=osd.4.0:1508, at_version=678'6, trim_to=0'0, roll_forward_to=674'4))", 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:59.379304+0000", 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 106.316485011, 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001139861, 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379304+0000", 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379304+0000", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379305+0000", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.164e-06 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379335+0000", 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9709999999999998e-05 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379345+0000", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0467e-05 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379354+0000", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0319999999999992e-06 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379786+0000", 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00043180600000000002 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.379795+0000", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4809999999999995e-06 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.380407+0000", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00061167100000000005 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.380444+0000", 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6529999999999998e-05 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.724 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:662 328.12 328:48000000:test-rados-api-smithi028-93253-123:::head [pgnls start_epoch 694] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e694)", 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:15.397941+0000", 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 90.297847959999999, 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023251909999999999, 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 662 2024-03-30T15:10:45.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397941+0000", 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397941+0000", 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397942+0000", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.765e-06 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397949+0000", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4470000000000003e-06 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397961+0000", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1855000000000001e-05 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.397968+0000", 2024-03-30T15:10:45.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3239999999999997e-06 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.398021+0000", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2686999999999999e-05 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.398025+0000", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1520000000000002e-06 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.400140+0000", 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002115479 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.400174+0000", 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4091999999999998e-05 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.400266+0000", 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1390000000000004e-05 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.0s1 699/644 ECSubWrite(tid=1189, reqid=client.58535.0:793, at_version=0'0, trim_to=0'0, roll_forward_to=699'12))", 2024-03-30T15:10:45.730 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:20.481185+0000", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 85.214603240000002, 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012499399999999999, 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481185+0000", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481185+0000", 2024-03-30T15:10:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481187+0000", 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.097e-06 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481198+0000", 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1114000000000001e-05 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481208+0000", 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0009e-05 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481217+0000", 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0459999999999994e-06 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481295+0000", 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8763000000000005e-05 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481306+0000", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0509999999999999e-05 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.482399+0000", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001093236 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.482435+0000", 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6164999999999997e-05 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.0s1 700/644 ECSubWrite(tid=1192, reqid=client.58535.0:798, at_version=700'15, trim_to=0'0, roll_forward_to=699'13))", 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:21.470969+0000", 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 84.224819647999993, 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001007226, 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470969+0000", 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470969+0000", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470969+0000", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.41e-07 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471007+0000", 2024-03-30T15:10:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.786e-05 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471018+0000", 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1209000000000001e-05 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471031+0000", 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2625e-05 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471105+0000", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3584999999999994e-05 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471116+0000", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1392999999999999e-05 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471936+0000", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081986199999999998 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471976+0000", 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0351e-05 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.0s1 712/644 ECSubWrite(tid=1206, reqid=osd.1.13:95, at_version=0'0, trim_to=0'0, roll_forward_to=712'22))", 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:33.409394+0000", 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 72.286394943000005, 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010901179999999999, 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.739 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409394+0000", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409394+0000", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409395+0000", 2024-03-30T15:10:45.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.28e-06 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409405+0000", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0268000000000001e-05 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409416+0000", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0532e-05 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409424+0000", 2024-03-30T15:10:45.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5690000000000005e-06 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409679+0000", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000254644 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409688+0000", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1840000000000005e-06 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.410451+0000", 2024-03-30T15:10:45.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00076265100000000004 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.410484+0000", 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2990000000000001e-05 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:1255 331.13 331:c8000000::::head [pgnls start_epoch 726] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e726)", 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:48.405104+0000", 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 57.290684681000002, 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023931389999999999, 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1255 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405104+0000", 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405104+0000", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405107+0000", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6189999999999998e-06 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405115+0000", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6610000000000006e-06 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405131+0000", 2024-03-30T15:10:45.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5361000000000001e-05 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405141+0000", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0757999999999999e-05 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405207+0000", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5979e-05 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.405212+0000", 2024-03-30T15:10:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4909999999999999e-06 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.407370+0000", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021584249999999998 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.407407+0000", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6569000000000002e-05 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:48.407497+0000", 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0276000000000001e-05 2024-03-30T15:10:45.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(42.7,RELEASE,e:727,from:7,reservation_nonce:0)", 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:49.825418+0000", 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 55.870370962999999, 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003522467, 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825418+0000", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825418+0000", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825420+0000", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8419999999999999e-06 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825425+0000", 2024-03-30T15:10:45.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0340000000000004e-06 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825435+0000", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0574000000000001e-05 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825442+0000", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1709999999999996e-06 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825495+0000", 2024-03-30T15:10:45.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2904e-05 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.825501+0000", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4999999999999999e-06 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.828899+0000", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033987510000000002 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.828918+0000", 2024-03-30T15:10:45.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8323000000000001e-05 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:49.828940+0000", 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2368e-05 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.1s1 753/644 ECSubWrite(tid=1539, reqid=osd.2.18:227, at_version=0'0, trim_to=0'0, roll_forward_to=753'7))", 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:15.423028+0000", 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 30.272760737999999, 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000946213, 2024-03-30T15:10:45.752 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423028+0000", 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423028+0000", 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423029+0000", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.362e-06 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423044+0000", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4579000000000001e-05 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423054+0000", 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.747e-06 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423062+0000", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5629999999999992e-06 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423114+0000", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2318999999999998e-05 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423124+0000", 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3049999999999998e-06 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423724+0000", 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00060033299999999999 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.423974+0000", 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025000499999999998 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.756 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:2085 335.18 e761/757 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head v 761'1, mlcod=0'0)", 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.455768+0000", 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.240020952999998, 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063351800000000002, 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455768+0000", 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455768+0000", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455768+0000", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5400000000000002e-07 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455777+0000", 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6610000000000006e-06 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455790+0000", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3132999999999999e-05 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455801+0000", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0893999999999999e-05 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455888+0000", 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6909000000000004e-05 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455914+0000", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6461e-05 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.456313+0000", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000398343 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.456401+0000", 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8663000000000001e-05 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.58535.0:1796 335.18 e761/757 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head v 761'2, mlcod=0'0)", 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.456198+0000", 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.239590979999999, 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081498309999999997, 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.456198+0000", 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.456198+0000", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.456199+0000", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.449e-06 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.461283+0000", 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050842179999999997 2024-03-30T15:10:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.461296+0000", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2976000000000001e-05 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.461307+0000", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0251000000000001e-05 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.461363+0000", 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6140999999999997e-05 2024-03-30T15:10:45.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.461386+0000", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3036000000000001e-05 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.464296+0000", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002909895 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.464348+0000", 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1864999999999999e-05 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDECSubOpWrite(322.0s1 761/644 ECSubWrite(tid=1218, reqid=osd.6.52:49, at_version=761'23, trim_to=0'0, roll_forward_to=712'22))", 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.480214+0000", 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.215574512, 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049187520000000002, 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.480214+0000", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.480214+0000", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.480215+0000", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7899999999999997e-07 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.481199+0000", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098364099999999994 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.481216+0000", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7728e-05 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.481226+0000", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5750000000000007e-06 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.481320+0000", 2024-03-30T15:10:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3870999999999996e-05 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.481334+0000", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4243999999999999e-05 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.485091+0000", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037573749999999999 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.485133+0000", 2024-03-30T15:10:45.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1439000000000001e-05 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:1798 335.0 335:00000000:%01:::head [pgnls start_epoch 761] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.486401+0000", 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.209387473, 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00095627899999999998, 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:10:45.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1798 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486401+0000", 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486401+0000", 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486403+0000", 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9300000000000002e-06 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486410+0000", 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9229999999999996e-06 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486420+0000", 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0302000000000001e-05 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486427+0000", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0049999999999998e-06 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486483+0000", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6101999999999999e-05 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.486512+0000", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8824999999999999e-05 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.487357+0000", 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084519200000000004 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.58535.0:1831 335.18 e761/757 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head v 761'4, mlcod=761'2)", 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.501855+0000", 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 22.193933207000001, 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00068608199999999997, 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501855+0000", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501855+0000", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501857+0000", 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5290000000000001e-06 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501866+0000", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5070000000000006e-06 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501878+0000", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1941e-05 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501890+0000", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1779e-05 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501958+0000", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7298000000000005e-05 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.501987+0000", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9882e-05 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.502487+0000", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049914700000000002 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.502542+0000", 2024-03-30T15:10:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4999e-05 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.4.0:1551 337.6 e780/773 337:60000000:.ceph-internal::hit_set_337.6_archive_2024-03-30 15%3a10%3a41.416579Z_2024-03-30 15%3a10%3a44.424687Z:head v 780'4, has_updated_hit_set_history, mlcod=779'2)", 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:44.425012+0000", 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.2707767089999999, 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00098136100000000004, 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425012+0000", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425012+0000", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425014+0000", 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.64e-06 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425023+0000", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7310000000000005e-06 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425037+0000", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4032999999999999e-05 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425048+0000", 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.045e-05 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425120+0000", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2181000000000006e-05 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425146+0000", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6176000000000001e-05 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425862+0000", 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071600199999999998 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425993+0000", 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013114800000000001 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.58535.0:2006 337.6 e780/773 337:602f83fe:::foo:head v 780'5, mlcod=779'2)", 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:44.425432+0000", 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.27035619, 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063301999999999998, 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:10:45.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425432+0000", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425432+0000", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425434+0000", 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1659999999999999e-06 2024-03-30T15:10:45.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425458+0000", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4178000000000001e-05 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425466+0000", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6759999999999996e-06 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425478+0000", 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1015999999999999e-05 2024-03-30T15:10:45.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425535+0000", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7895e-05 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425556+0000", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0662e-05 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.425997+0000", 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044113300000000002 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:10:45.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.426065+0000", 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8293999999999994e-05 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:10:45.785 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:10:45.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:45.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:45.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:45 smithi134 ceph-mon[33714]: Health check failed: Reduced data availability: 20 pgs inactive, 22 pgs peering (PG_AVAILABILITY) 2024-03-30T15:10:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:45 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 49/651 objects degraded (7.527%), 16 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:45.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:45.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:46.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:46.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:46.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:46.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:46.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:46.321 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:46.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:46.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:46.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:46.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:46.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:46 smithi028 ceph-mon[41614]: pgmap v1162: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 4.2 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:46.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:46 smithi028 ceph-mon[29876]: pgmap v1162: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 4.2 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:46.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:46.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:46.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:46.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:46 smithi134 ceph-mon[33714]: pgmap v1162: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 4.2 KiB/s wr, 1 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:46.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:46.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:46.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:47.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:10:47.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:47.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:47.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:47.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:47.270 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:10:47.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:47.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:47.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:47.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.587+0000 7f9cf9f37700 1 -- 172.21.15.28:0/4070146463 >> v1:172.21.15.28:6789/0 conn(0x7f9cf4109dc0 legacy=0x7f9cf410c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.587+0000 7f9cf9f37700 1 -- 172.21.15.28:0/4070146463 shutdown_connections 2024-03-30T15:10:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.587+0000 7f9cf9f37700 1 -- 172.21.15.28:0/4070146463 >> 172.21.15.28:0/4070146463 conn(0x7f9cf40fc9e0 msgr2=0x7f9cf40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.588+0000 7f9cf9f37700 1 -- 172.21.15.28:0/4070146463 shutdown_connections 2024-03-30T15:10:47.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.588+0000 7f9cf9f37700 1 -- 172.21.15.28:0/4070146463 wait complete. 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 librados: starting msgr at 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 librados: starting objecter 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 Processor -- start 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 -- start start 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 librados: setting wanted keys 2024-03-30T15:10:47.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.589+0000 7f9cf9f37700 1 librados: calling monclient init 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9cf410d960 0x7f9cf41a5820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54670/0 (socket says 172.21.15.28:54670) 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf3fff700 1 -- 172.21.15.28:0/2147562389 learned_addr learned my addr 172.21.15.28:0/2147562389 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9cf41a7120 con 0x7f9cf410d960 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter ms_handle_connect 0x7f9cf410d960 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9cf41a8340 con 0x7f9cf4101130 2024-03-30T15:10:47.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9cf41a7330 con 0x7f9cf4109dc0 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter ms_handle_connect 0x7f9cf4109dc0 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735825670 0 0) 0x7f9cf41a7120 con 0x7f9cf410d960 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cdc0039c0 con 0x7f9cf410d960 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter ms_handle_connect 0x7f9cf4101130 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289681158 0 0) 0x7f9cf41a7330 con 0x7f9cf4109dc0 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cf41a7120 con 0x7f9cf4109dc0 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 858367839 0 0) 0x7f9cf41a8340 con 0x7f9cf4101130 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.590+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cf41a7330 con 0x7f9cf4101130 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1420670848 0 0) 0x7f9cf41a7120 con 0x7f9cf4109dc0 2024-03-30T15:10:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cf41a8340 con 0x7f9cf4109dc0 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2281428441 0 0) 0x7f9cdc0039c0 con 0x7f9cf410d960 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cf41a7120 con 0x7f9cf410d960 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9ce8004610 con 0x7f9cf4109dc0 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9cec001ce0 con 0x7f9cf410d960 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 749347826 0 0) 0x7f9cf41a7330 con 0x7f9cf4101130 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cdc0039c0 con 0x7f9cf4101130 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9ce4001870 con 0x7f9cf4101130 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2791015967 0 0) 0x7f9cf41a8340 con 0x7f9cf4109dc0 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 >> v1:172.21.15.134:6789/0 conn(0x7f9cf4101130 legacy=0x7f9cf406a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 >> v1:172.21.15.28:6789/0 conn(0x7f9cf410d960 legacy=0x7f9cf41a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf41a7560 con 0x7f9cf4109dc0 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf41a7850 con 0x7f9cf4109dc0 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf9f37700 10 client.58784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:47.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.591+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9cf41a7d10 con 0x7f9cf4109dc0 2024-03-30T15:10:47.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.592+0000 7f9cf9f37700 1 librados: init done 2024-03-30T15:10:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.592+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9ce8001ac0 con 0x7f9cf4109dc0 2024-03-30T15:10:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.592+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9ce8004f00 con 0x7f9cf4109dc0 2024-03-30T15:10:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.593+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9ce801b800 con 0x7f9cf4109dc0 2024-03-30T15:10:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.593+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9cf4102610 con 0x7f9cf4109dc0 2024-03-30T15:10:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(781..781 src has 442..781) ==== 9296+0+0 (unknown 3827038833 0 0) 0x7f9ce8088af0 con 0x7f9cf4109dc0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 osd_map(781..781 src has 442..781) 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 3 client.58784.objecter handle_osd_map got epochs [781,781] > 0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 3 client.58784.objecter handle_osd_map decoding full epoch 781 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 20 client.58784.objecter dump_active .. 0 homeless 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v781) ==== 54+0+16 (unknown 1362223828 0 2420000192) 0x7f9ce8056670 con 0x7f9cf4109dc0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 1 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 20 client.58784.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 10 client.58784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=782}) -- 0x7f9cf4069d00 con 0x7f9cf4109dc0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9cf4113710 con 0x7f9cf4109dc0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.594+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cdc070470 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=781) ==== 24+0+0 (unknown 2852545450 0 0) 0x7f9ce8055d40 con 0x7f9cf4109dc0 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf1ffb700 10 client.58784.objecter _check_command_map_dne tid 1 current 781 map_dne_bound 781 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf1ffb700 10 client.58784.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf1ffb700 15 client.58784.objecter _session_command_op_remove -1 1 2024-03-30T15:10:47.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 2 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 2 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 1 2 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 4 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 10 client.58784.objecter _send_command 2 2024-03-30T15:10:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.595+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f9cf4113710 con 0x7f9cf4062e50 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf4062e50 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9cf4113710 con 0x7f9cf4062e50 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 2 = system:0 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 4 2024-03-30T15:10:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.596+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 1 2 2024-03-30T15:10:47.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 3 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 3 2024-03-30T15:10:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf4068b30 osd=1 3 2024-03-30T15:10:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 1 3 2024-03-30T15:10:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 4 2024-03-30T15:10:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 10 client.58784.objecter _send_command 3 2024-03-30T15:10:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.615+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf4063480 con 0x7f9cf4062e50 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.616+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf4063480 con 0x7f9cf4062e50 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.616+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.616+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.616+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf4068b30 osd=1 4 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.616+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 1 3 2024-03-30T15:10:47.618 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 4 2024-03-30T15:10:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 10 client.58784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f9cf4063630 con 0x7f9cf4109dc0 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=781) ==== 24+0+0 (unknown 1631212233 0 0) 0x7f9ce80540b0 con 0x7f9cf4109dc0 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf17fa700 10 client.58784.objecter _check_command_map_dne tid 4 current 781 map_dne_bound 781 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf17fa700 10 client.58784.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf17fa700 15 client.58784.objecter _session_command_op_remove -1 4 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 5 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 5 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 3 5 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 4 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 10 client.58784.objecter _send_command 5 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.617+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9cf4062c90 con 0x7f9cf410ea90 2024-03-30T15:10:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf410ea90 2024-03-30T15:10:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9ce400aa10 con 0x7f9cf410ea90 2024-03-30T15:10:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 5: 0 ) 2024-03-30T15:10:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 5 = system:0 2024-03-30T15:10:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 4 2024-03-30T15:10:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.618+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 3 5 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 6 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 6 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf4062b20 osd=3 3 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 3 6 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 4 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 10 client.58784.objecter _send_command 6 2024-03-30T15:10:47.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.637+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf410e000 con 0x7f9cf410ea90 2024-03-30T15:10:47.641 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.639+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf410e000 con 0x7f9cf410ea90 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.639+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.639+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.639+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf4062b20 osd=3 4 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.639+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 3 6 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 7 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 7 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 4 7 2024-03-30T15:10:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 4 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 10 client.58784.objecter _send_command 7 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9cf41b37b0 con 0x7f9cf41ab200 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.640+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf41ab200 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.641+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9cf41b37b0 con 0x7f9cf41ab200 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.641+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 7: 0 ) 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.641+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 7 = system:0 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.641+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 4 2024-03-30T15:10:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.641+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 4 7 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 8 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 8 2024-03-30T15:10:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf410e400 osd=4 3 2024-03-30T15:10:47.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 4 8 2024-03-30T15:10:47.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 4 2024-03-30T15:10:47.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 10 client.58784.objecter _send_command 8 2024-03-30T15:10:47.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.659+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf41aa830 con 0x7f9cf41ab200 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf41aa830 con 0x7f9cf41ab200 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf410e400 osd=4 4 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 4 8 2024-03-30T15:10:47.662 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 9 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 9 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 5 9 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 4 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 10 client.58784.objecter _send_command 9 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9cf41afd40 con 0x7f9cf41b3960 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.661+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf41b3960 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.662+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9cf41afd40 con 0x7f9cf41b3960 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.662+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 9: 0 ) 2024-03-30T15:10:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.662+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 9 = system:0 2024-03-30T15:10:47.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.662+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 4 2024-03-30T15:10:47.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.662+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 5 9 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 10 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 10 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41af5a0 osd=5 3 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 5 10 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 4 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 10 client.58784.objecter _send_command 10 2024-03-30T15:10:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.679+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf41b5e70 con 0x7f9cf41b3960 2024-03-30T15:10:47.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf41b5e70 con 0x7f9cf41b3960 2024-03-30T15:10:47.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41af5a0 osd=5 4 2024-03-30T15:10:47.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 5 10 2024-03-30T15:10:47.683 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 11 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 11 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 6 11 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 4 2024-03-30T15:10:47.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 10 client.58784.objecter _send_command 11 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.681+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9cf41b60c0 con 0x7f9cf41b9320 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf41b9320 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9cf41b60c0 con 0x7f9cf41b9320 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 11: 0 ) 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 11 = system:0 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 4 2024-03-30T15:10:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.682+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 6 11 2024-03-30T15:10:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 12 2024-03-30T15:10:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 12 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41b5e70 osd=6 3 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 6 12 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 4 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 10 client.58784.objecter _send_command 12 2024-03-30T15:10:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.699+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf41b56a0 con 0x7f9cf41b9320 2024-03-30T15:10:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf41b56a0 con 0x7f9cf41b9320 2024-03-30T15:10:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41b5e70 osd=6 4 2024-03-30T15:10:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 6 12 2024-03-30T15:10:47.703 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 13 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 13 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 7 13 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 4 2024-03-30T15:10:47.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 10 client.58784.objecter _send_command 13 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.701+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9cf4105490 con 0x7f9cf41b5a20 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 10 client.58784.objecter ms_handle_connect 0x7f9cf41b5a20 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9cf4105490 con 0x7f9cf41b5a20 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 13: 0 ) 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 13 = system:0 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 4 2024-03-30T15:10:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.702+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 7 13 2024-03-30T15:10:47.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 10 client.58784.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign -1 14 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 20 client.58784.objecter _get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_remove -1 14 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 20 client.58784.objecter get_session s=0x7f9cf41b58b0 osd=7 3 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 15 client.58784.objecter _session_command_op_assign 7 14 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 4 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 10 client.58784.objecter _send_command 14 2024-03-30T15:10:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.719+0000 7f9cf9f37700 1 -- 172.21.15.28:0/2147562389 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9cf4105490 con 0x7f9cf41b5a20 2024-03-30T15:10:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.721+0000 7f9cf0ff9700 1 -- 172.21.15.28:0/2147562389 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9cf4105490 con 0x7f9cf41b5a20 2024-03-30T15:10:47.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.721+0000 7f9cf0ff9700 10 client.58784.objecter ms_dispatch 0x7f9cf4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.721+0000 7f9cf0ff9700 10 client.58784.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.721+0000 7f9cf0ff9700 20 client.58784.objecter put_session s=0x7f9cf41b58b0 osd=7 4 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:47.721+0000 7f9cf0ff9700 15 client.58784.objecter _session_command_op_remove 7 14 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:47.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:47.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:47.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:47.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:47.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:47.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:48.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:48.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:48.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:48.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:48.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:48.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:48.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:48.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:48.660 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:48.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2147562389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:48.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:48 smithi028 ceph-mon[41614]: pgmap v1163: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 457 B/s rd, 3.6 KiB/s wr, 0 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:48.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2147562389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:48.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:48 smithi028 ceph-mon[29876]: pgmap v1163: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 457 B/s rd, 3.6 KiB/s wr, 0 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:48.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2147562389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:48.667 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:48 smithi134 ceph-mon[33714]: pgmap v1163: 246 pgs: 1 active+clean+remapped, 81 peering, 106 active+clean, 42 active+undersized, 16 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 457 B/s rd, 3.6 KiB/s wr, 0 op/s; 49/651 objects degraded (7.527%) 2024-03-30T15:10:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:48.686 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:48.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:48.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:48.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:48.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:48.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:48.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:49.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:49.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:49.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:49.222 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:48 smithi028 podman[277937]: 2024-03-30 15:10:48.84514915 +0000 UTC m=+0.650399629 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:10:49.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:49.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:49.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:49.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:49.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:49.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:49.617 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:49 smithi028 podman[277937]: 2024-03-30 15:10:49.288661457 +0000 UTC m=+1.093911923 container create 9eb21894d9d586be3224c0de40060cf24679ad047e45a4e11ba7acdb8336b0a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-4e50bbc, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:10:49.617 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:49 smithi028 podman[277937]: 2024-03-30 15:10:49.525645497 +0000 UTC m=+1.330895972 container init 9eb21894d9d586be3224c0de40060cf24679ad047e45a4e11ba7acdb8336b0a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-30T15:10:49.617 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:49 smithi028 podman[277937]: 2024-03-30 15:10:49.531371649 +0000 UTC m=+1.336622116 container start 9eb21894d9d586be3224c0de40060cf24679ad047e45a4e11ba7acdb8336b0a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-30T15:10:49.617 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:49 smithi028 podman[277937]: 2024-03-30 15:10:49.616189178 +0000 UTC m=+1.421439655 container attach 9eb21894d9d586be3224c0de40060cf24679ad047e45a4e11ba7acdb8336b0a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:10:49.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:49.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:10:49.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:49.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:49.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:49.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:49.974 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:10:49.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:49.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:50.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:50.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:50.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:50.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:50.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:50 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:10:50.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:50 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 127/675 objects degraded (18.815%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:50.323 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:10:50 smithi028 podman[277937]: 2024-03-30 15:10:50.015564563 +0000 UTC m=+1.820815024 container died 9eb21894d9d586be3224c0de40060cf24679ad047e45a4e11ba7acdb8336b0a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T15:10:50.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:50 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:10:50.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:50 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 127/675 objects degraded (18.815%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:50.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:50.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:50.410 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:50 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:10:50.410 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:50 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 127/675 objects degraded (18.815%), 28 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:50.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:50.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:50.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:50.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:10:50.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T15:10:50.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:50.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:10:50.800 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T15:10:50.800 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-30T15:10:50.800 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T15:10:50.800 INFO:tasks.thrashosds.thrasher:remote for exp osd 0 is ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T15:10:50.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-03-30T15:10:50.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:50.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:50.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:50.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:10:51.081 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:10:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:10:51.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:51.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:51.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:51 smithi028 ceph-mon[29876]: pgmap v1164: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 9.0 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:51.290 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:51 smithi028 ceph-mon[41614]: pgmap v1164: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 9.0 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:51.349 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:51 smithi134 ceph-mon[33714]: pgmap v1164: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 9.0 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:51.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:51.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:51.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:51.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.519+0000 7f53c795d700 1 -- 172.21.15.28:0/4018297513 >> v1:172.21.15.28:6790/0 conn(0x7f53c0101980 legacy=0x7f53c010f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.519+0000 7f53c795d700 1 -- 172.21.15.28:0/4018297513 shutdown_connections 2024-03-30T15:10:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.519+0000 7f53c795d700 1 -- 172.21.15.28:0/4018297513 >> 172.21.15.28:0/4018297513 conn(0x7f53c00fc9e0 msgr2=0x7f53c00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.519+0000 7f53c795d700 1 -- 172.21.15.28:0/4018297513 shutdown_connections 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.519+0000 7f53c795d700 1 -- 172.21.15.28:0/4018297513 wait complete. 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 librados: starting msgr at 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 librados: starting objecter 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 Processor -- start 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 -- start start 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 librados: setting wanted keys 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 librados: calling monclient init 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53c01a9360 con 0x7f53c0110c70 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53c01aa580 con 0x7f53c0100da0 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c795d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53c01ab7c0 con 0x7f53c0101980 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c4ef8700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f53c0101980 0x7f53c01a40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50676/0 (socket says 172.21.15.28:50676) 2024-03-30T15:10:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c5efa700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f53c0110c70 0x7f53c01a7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49614/0 (socket says 172.21.15.28:49614) 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53c4ef8700 1 -- 172.21.15.28:0/2945981370 learned_addr learned my addr 172.21.15.28:0/2945981370 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter ms_handle_connect 0x7f53c0101980 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter ms_handle_connect 0x7f53c0110c70 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter ms_handle_connect 0x7f53c0100da0 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884786959 0 0) 0x7f53c01ab7c0 con 0x7f53c0101980 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a80031f0 con 0x7f53c0101980 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848732974 0 0) 0x7f53c01aa580 con 0x7f53c0100da0 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c01ab7c0 con 0x7f53c0100da0 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444982680 0 0) 0x7f53c01a9360 con 0x7f53c0110c70 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.520+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53c01aa580 con 0x7f53c0110c70 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3189860835 0 0) 0x7f53c01ab7c0 con 0x7f53c0100da0 2024-03-30T15:10:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53c01a9360 con 0x7f53c0100da0 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3854347348 0 0) 0x7f53a80031f0 con 0x7f53c0101980 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53c01ab7c0 con 0x7f53c0101980 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53b8001bf0 con 0x7f53c0100da0 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53b4001510 con 0x7f53c0101980 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4165206366 0 0) 0x7f53c01aa580 con 0x7f53c0110c70 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53a80031f0 con 0x7f53c0110c70 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53b00017f0 con 0x7f53c0110c70 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2715696385 0 0) 0x7f53c01a9360 con 0x7f53c0100da0 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 >> v1:172.21.15.28:6790/0 conn(0x7f53c0101980 legacy=0x7f53c01a40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 >> v1:172.21.15.28:6789/0 conn(0x7f53c0110c70 legacy=0x7f53c01a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c01aca00 con 0x7f53c0100da0 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53c01a9590 con 0x7f53c0100da0 2024-03-30T15:10:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53c795d700 10 client.50788.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53c01a9a50 con 0x7f53c0100da0 2024-03-30T15:10:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53c795d700 1 librados: init done 2024-03-30T15:10:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f53b8003ec0 con 0x7f53c0100da0 2024-03-30T15:10:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.521+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f53b8004e40 con 0x7f53c0100da0 2024-03-30T15:10:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.522+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ac005120 con 0x7f53c0100da0 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.522+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f53b8003a20 con 0x7f53c0100da0 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(781..781 src has 442..781) ==== 9296+0+0 (unknown 3827038833 0 0) 0x7f53b8088d00 con 0x7f53c0100da0 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 10 client.50788.objecter ms_dispatch 0x7f53c0001550 osd_map(781..781 src has 442..781) 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 3 client.50788.objecter handle_osd_map got epochs [781,781] > 0 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 3 client.50788.objecter handle_osd_map decoding full epoch 781 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 20 client.50788.objecter dump_active .. 0 homeless 2024-03-30T15:10:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.523+0000 7f53be7fc700 10 client.50788.objecter ms_handle_connect 0x7f53a8070460 2024-03-30T15:10:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.528+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f53b8055a80 con 0x7f53c0100da0 2024-03-30T15:10:51.589 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:51.589 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:51.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:51.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:51.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:51.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:51.745+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f53ac005410 con 0x7f53c0100da0 2024-03-30T15:10:51.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:51.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:51.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:51.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:51.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:52.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:52.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:10:52.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.232+0000 7f53be7fc700 1 -- 172.21.15.28:0/2945981370 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v782) ==== 91+0+0 (unknown 3691475477 0 0) 0x7f53b80594d0 con 0x7f53c0100da0 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 10 librados: watch_flush enter 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 10 librados: watch_flush exit 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 20 client.50788.objecter shutdown clearing up homeless session... 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 10 client.50788.objecter successfully canceled tick 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 >> v1:172.21.15.28:6800/3011431093 conn(0x7f53a8070460 legacy=0x7f53a8072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 >> v1:172.21.15.134:6789/0 conn(0x7f53c0100da0 legacy=0x7f53c010f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 shutdown_connections 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.235+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 >> 172.21.15.28:0/2945981370 conn(0x7f53c00fc9e0 msgr2=0x7f53c00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.236+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 shutdown_connections 2024-03-30T15:10:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.236+0000 7f53c795d700 1 -- 172.21.15.28:0/2945981370 wait complete. 2024-03-30T15:10:52.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:52.236+0000 7f53c795d700 1 librados: shutdown 2024-03-30T15:10:52.239 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:10:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:52.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:52.418 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:10:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:52 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2945981370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:52 smithi134 ceph-mon[33714]: from='client.50788 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2945981370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[29876]: from='client.50788 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2945981370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:52 smithi028 ceph-mon[41614]: from='client.50788 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:10:52.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:52.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:52.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:52.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:52.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:52.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:52.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:52.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:52.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:53.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:10:53.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:53.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:53.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:53.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:53.324 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: pgmap v1165: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 8.3 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: from='client.50788 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: osdmap e782: 8 total, 6 up, 8 in 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: osdmap e783: 8 total, 6 up, 8 in 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: pgmap v1165: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 8.3 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: from='client.50788 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: osdmap e782: 8 total, 6 up, 8 in 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: osdmap e783: 8 total, 6 up, 8 in 2024-03-30T15:10:53.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:53.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:53.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:53.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:53.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:53.562 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: pgmap v1165: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 91 active+undersized, 2 undersized+degraded+peered, 26 active+undersized+degraded; 4.4 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 941 B/s rd, 8.3 KiB/s wr, 2 op/s; 127/675 objects degraded (18.815%) 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: from='client.50788 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: osdmap e782: 8 total, 6 up, 8 in 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: osdmap e783: 8 total, 6 up, 8 in 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:10:53.563 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:10:53.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:53.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:53.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:53.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:53.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:53.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:53.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:54.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:54.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:54.161 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:10:54.161 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:10:54.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:54.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:54.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:54.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:54.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:54.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:54.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:54.633 INFO:teuthology.orchestra.run.smithi028.stdout:6.d 2024-03-30T15:10:54.633 INFO:teuthology.orchestra.run.smithi028.stdout:6.a 2024-03-30T15:10:54.633 INFO:teuthology.orchestra.run.smithi028.stdout:6.9 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:6.18 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:325.17 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:325.10 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:12.c 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.7 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:4.1a 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.12 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.b 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.6 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.1b 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.1f 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:3.17 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:1.0 2024-03-30T15:10:54.634 INFO:teuthology.orchestra.run.smithi028.stdout:1.2 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:3.1 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:2.1 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:1.6 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:4.19 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:42.11 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:5.8 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:12.13 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:12.1d 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:4.3 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:3.10 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:3.f 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:12.8 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:5.7 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:325.18 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:325.12 2024-03-30T15:10:54.635 INFO:teuthology.orchestra.run.smithi028.stdout:1.1 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:322.5s1 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:3.18 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:12.10 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:5.1d 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:12.a 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:4.7 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:12.3 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:12.6 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:2.0 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:325.1d 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:322.3s0 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:325.2 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:5.2 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:5.15 2024-03-30T15:10:54.636 INFO:teuthology.orchestra.run.smithi028.stdout:12.19 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:5.1e 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:4.0 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:325.7 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:42.8 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:5.e 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:325.f 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:325.14 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:4.16 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:4.1e 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:4.6 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:12.1c 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:5.c 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:42.10 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:42.e 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:325.6 2024-03-30T15:10:54.637 INFO:teuthology.orchestra.run.smithi028.stdout:42.16 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:42.1c 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:325.4 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:42.2 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:42.3 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:5.13 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:6.0 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:325.1c 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:4.b 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:5.1b 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:5.3 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:5.5 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:12.e 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:5.b 2024-03-30T15:10:54.638 INFO:teuthology.orchestra.run.smithi028.stdout:3.3 2024-03-30T15:10:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:6.10 2024-03-30T15:10:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:6.11 2024-03-30T15:10:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:6.13 2024-03-30T15:10:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:6.16 2024-03-30T15:10:54.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:54.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:54.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:54.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:10:54.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.872+0000 7f9d1021d700 1 -- 172.21.15.28:0/2175069003 >> v1:172.21.15.28:6789/0 conn(0x7f9d08101130 legacy=0x7f9d08101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.872+0000 7f9d1021d700 1 -- 172.21.15.28:0/2175069003 shutdown_connections 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.872+0000 7f9d1021d700 1 -- 172.21.15.28:0/2175069003 >> 172.21.15.28:0/2175069003 conn(0x7f9d080fc9e0 msgr2=0x7f9d080fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.872+0000 7f9d1021d700 1 -- 172.21.15.28:0/2175069003 shutdown_connections 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.872+0000 7f9d1021d700 1 -- 172.21.15.28:0/2175069003 wait complete. 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 librados: starting msgr at 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 librados: starting objecter 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 Processor -- start 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 -- start start 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 librados: setting wanted keys 2024-03-30T15:10:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 librados: calling monclient init 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d081a9340 con 0x7f9d08109dc0 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d081aa520 con 0x7f9d0810d960 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d1021d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d081ab760 con 0x7f9d08101130 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d0dfb9700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9d08101130 0x7f9d080fe870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50700/0 (socket says 172.21.15.28:50700) 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d0d7b8700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9d08109dc0 0x7f9d081a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49628/0 (socket says 172.21.15.28:49628) 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d0dfb9700 1 -- 172.21.15.28:0/1162709350 learned_addr learned my addr 172.21.15.28:0/1162709350 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter ms_handle_connect 0x7f9d08101130 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter ms_handle_connect 0x7f9d08109dc0 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter ms_handle_connect 0x7f9d0810d960 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.873+0000 7f9d06ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:10:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451093981 0 0) 0x7f9d081ab760 con 0x7f9d08101130 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cf00039c0 con 0x7f9d08101130 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227714960 0 0) 0x7f9d081a9340 con 0x7f9d08109dc0 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d081ab760 con 0x7f9d08109dc0 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006986747 0 0) 0x7f9d081aa520 con 0x7f9d0810d960 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d081a9340 con 0x7f9d0810d960 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 281461322 0 0) 0x7f9cf00039c0 con 0x7f9d08101130 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d081aa520 con 0x7f9d08101130 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d00001b10 con 0x7f9d08101130 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3164312266 0 0) 0x7f9d081a9340 con 0x7f9d0810d960 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cf00039c0 con 0x7f9d0810d960 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9cf8001070 con 0x7f9d0810d960 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2212974126 0 0) 0x7f9d081ab760 con 0x7f9d08109dc0 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d081a9340 con 0x7f9d08109dc0 2024-03-30T15:10:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9cfc001830 con 0x7f9d08109dc0 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1447191591 0 0) 0x7f9d081aa520 con 0x7f9d08101130 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 >> v1:172.21.15.134:6789/0 conn(0x7f9d0810d960 legacy=0x7f9d081a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 >> v1:172.21.15.28:6789/0 conn(0x7f9d08109dc0 legacy=0x7f9d081a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d081ac9a0 con 0x7f9d08101130 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d081a9570 con 0x7f9d08101130 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d1021d700 10 client.58796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d081a9a30 con 0x7f9d08101130 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d1021d700 1 librados: init done 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9d00001d70 con 0x7f9d08101130 2024-03-30T15:10:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.874+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d00005320 con 0x7f9d08101130 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.875+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9d08102610 con 0x7f9d08101130 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9d00005300 con 0x7f9d08101130 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(783..783 src has 442..783) ==== 9296+0+0 (unknown 3301037594 0 0) 0x7f9d0008a0b0 con 0x7f9d08101130 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 osd_map(783..783 src has 442..783) 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 3 client.58796.objecter handle_osd_map got epochs [783,783] > 0 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 3 client.58796.objecter handle_osd_map decoding full epoch 783 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 20 client.58796.objecter dump_active .. 0 homeless 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v783) ==== 54+0+16 (unknown 1156222854 0 2420000192) 0x7f9d0008c4e0 con 0x7f9d08101130 2024-03-30T15:10:54.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.876+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9cf00705c0 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 10 client.58796.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 1 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 10 client.58796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=784}) -- 0x7f9d080fe330 con 0x7f9d08101130 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9d08068b30 con 0x7f9d08101130 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=783) ==== 24+0+0 (unknown 1274154663 0 0) 0x7f9d00056500 con 0x7f9d08101130 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d07fff700 10 client.58796.objecter _check_command_map_dne tid 1 current 783 map_dne_bound 783 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d07fff700 10 client.58796.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d07fff700 15 client.58796.objecter _session_command_op_remove -1 1 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 10 client.58796.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 2 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 2 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 1 2 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 4 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 10 client.58796.objecter _send_command 2 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f9d081ab310 con 0x7f9d0806a690 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.877+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d0806a690 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.878+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d081ab310 con 0x7f9d0806a690 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.878+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 2: 0 ) 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.878+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 2 = system:0 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.878+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 4 2024-03-30T15:10:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.878+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 1 2 2024-03-30T15:10:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 10 client.58796.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 3 2024-03-30T15:10:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d0806a520 osd=1 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 1 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 4 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 10 client.58796.objecter _send_command 3 2024-03-30T15:10:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.896+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d081ab4c0 con 0x7f9d0806a690 2024-03-30T15:10:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:54.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d081ab4c0 con 0x7f9d0806a690 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d0806a520 osd=1 4 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 1 3 2024-03-30T15:10:54.900 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:10:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d1021d700 10 client.58796.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 4 2024-03-30T15:10:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d1021d700 20 client.58796.objecter _get_session osd=-1 returning homeless 2024-03-30T15:10:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d1021d700 10 client.58796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f9d081aa1c0 con 0x7f9d08101130 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=783) ==== 24+0+0 (unknown 2161000388 0 0) 0x7f9d00054870 con 0x7f9d08101130 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d077fe700 10 client.58796.objecter _check_command_map_dne tid 4 current 783 map_dne_bound 783 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d077fe700 10 client.58796.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.898+0000 7f9d077fe700 15 client.58796.objecter _session_command_op_remove -1 4 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 10 client.58796.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 5 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 5 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 3 5 2024-03-30T15:10:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 4 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 10 client.58796.objecter _send_command 5 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9d080635b0 con 0x7f9d0802c9b0 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.899+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d0802c9b0 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.900+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d080635b0 con 0x7f9d0802c9b0 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.900+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 5: 0 ) 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.900+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 5 = system:0 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.900+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 4 2024-03-30T15:10:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.900+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 3 5 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 10 client.58796.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 6 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 6 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081aa370 osd=3 3 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 3 6 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 4 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 10 client.58796.objecter _send_command 6 2024-03-30T15:10:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.917+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d0802dcd0 con 0x7f9d0802c9b0 2024-03-30T15:10:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d0802dcd0 con 0x7f9d0802c9b0 2024-03-30T15:10:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081aa370 osd=3 4 2024-03-30T15:10:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 3 6 2024-03-30T15:10:54.921 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:10:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 10 client.58796.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 7 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 7 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 4 7 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 4 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 10 client.58796.objecter _send_command 7 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.919+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9d08062eb0 con 0x7f9d081b2170 2024-03-30T15:10:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.920+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d081b2170 2024-03-30T15:10:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.921+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d08062eb0 con 0x7f9d081b2170 2024-03-30T15:10:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.921+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 7: 0 ) 2024-03-30T15:10:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.921+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 7 = system:0 2024-03-30T15:10:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.921+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 4 2024-03-30T15:10:54.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.921+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 4 7 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 10 client.58796.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 8 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 8 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d08062b80 osd=4 3 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 4 8 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 4 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 10 client.58796.objecter _send_command 8 2024-03-30T15:10:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.938+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d0802d870 con 0x7f9d081b2170 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.939+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d0802d870 con 0x7f9d081b2170 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.939+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.939+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.939+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d08062b80 osd=4 4 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.939+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 4 8 2024-03-30T15:10:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:10:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 10 client.58796.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 9 2024-03-30T15:10:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 9 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 5 9 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 4 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 10 client.58796.objecter _send_command 9 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9d081b67f0 con 0x7f9d081b5c80 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.940+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d081b5c80 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.941+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d081b67f0 con 0x7f9d081b5c80 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.941+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 9: 0 ) 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.941+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 9 = system:0 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.941+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 4 2024-03-30T15:10:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.941+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 5 9 2024-03-30T15:10:54.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 10 client.58796.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 10 2024-03-30T15:10:54.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 10 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d0802db00 osd=5 3 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 5 10 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 4 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 10 client.58796.objecter _send_command 10 2024-03-30T15:10:54.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.958+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d081b71a0 con 0x7f9d081b5c80 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d081b71a0 con 0x7f9d081b5c80 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d0802db00 osd=5 4 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 5 10 2024-03-30T15:10:54.961 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 10 client.58796.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 11 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 11 2024-03-30T15:10:54.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 6 11 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 4 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 10 client.58796.objecter _send_command 11 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9d081bf250 con 0x7f9d081bb660 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.960+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d081bb660 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.961+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d081bf250 con 0x7f9d081bb660 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.961+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 11: 0 ) 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.961+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 11 = system:0 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.961+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 4 2024-03-30T15:10:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.961+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 6 11 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 10 client.58796.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 12 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 12 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081b8320 osd=6 3 2024-03-30T15:10:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 6 12 2024-03-30T15:10:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 4 2024-03-30T15:10:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 10 client.58796.objecter _send_command 12 2024-03-30T15:10:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.978+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d081bf210 con 0x7f9d081bb660 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d081bf210 con 0x7f9d081bb660 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081b8320 osd=6 4 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 6 12 2024-03-30T15:10:54.981 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 10 client.58796.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 13 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 13 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 7 13 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 4 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 10 client.58796.objecter _send_command 13 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9d081c0a20 con 0x7f9d081bfe40 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.980+0000 7f9d06ffd700 10 client.58796.objecter ms_handle_connect 0x7f9d081bfe40 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.981+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9d081c0a20 con 0x7f9d081bfe40 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.981+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 13: 0 ) 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.981+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 13 = system:0 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.981+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 4 2024-03-30T15:10:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.981+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 7 13 2024-03-30T15:10:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 10 client.58796.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:10:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign -1 14 2024-03-30T15:10:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 20 client.58796.objecter _get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_remove -1 14 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 20 client.58796.objecter get_session s=0x7f9d081bf210 osd=7 3 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 15 client.58796.objecter _session_command_op_assign 7 14 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 4 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 10 client.58796.objecter _send_command 14 2024-03-30T15:10:55.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.998+0000 7f9d1021d700 1 -- 172.21.15.28:0/1162709350 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9d081b8110 con 0x7f9d081bfe40 2024-03-30T15:10:55.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.999+0000 7f9d06ffd700 1 -- 172.21.15.28:0/1162709350 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9d081b8110 con 0x7f9d081bfe40 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.999+0000 7f9d06ffd700 10 client.58796.objecter ms_dispatch 0x7f9d08001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.999+0000 7f9d06ffd700 10 client.58796.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.999+0000 7f9d06ffd700 20 client.58796.objecter put_session s=0x7f9d081bf210 osd=7 4 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:10:54.999+0000 7f9d06ffd700 15 client.58796.objecter _session_command_op_remove 7 14 2024-03-30T15:10:55.001 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:10:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:10:55.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:55.095 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:10:55.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:55.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:55.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:55.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:55.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:55.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:55.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: pgmap v1168: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1162709350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: pgmap v1168: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:10:55.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1162709350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:55.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:55.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:55.557 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:10:55.557 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:10:55.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: pgmap v1168: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:55.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:10:55.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:10:55.558 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1162709350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:10:55.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:55.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:55.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:55.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:55.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:56.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:56.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:56.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 5.15 --file /var/log/ceph/exp.5.15.0 2024-03-30T15:10:56.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:56.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:56.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:56.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:56.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:56.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:56.413 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:56 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:10:56.413 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:56 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:10:56.413 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:56 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:10:56.414 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:56 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:10:56.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:10:56.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:56 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:10:56.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:56.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:56.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:56.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:10:56.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:10:56.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:10:56.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:56.790 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:10:56.790 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:10:56.790 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:10:56.790 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:10:56.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:56.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:57.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:57.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:57.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:57.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:57.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:57.369 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:57 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:10:57.369 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:57 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:10:57.369 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:57 smithi134 ceph-mon[33714]: pgmap v1169: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:57.369 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:57 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:10:57.369 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:57 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:10:57.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:10:57.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:10:57.380 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[41614]: pgmap v1169: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[29876]: pgmap v1169: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 3 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:10:57.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:57 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:10:57.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:57.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:10:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:57.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:57.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:57.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:57.654 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:10:57.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:57.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:10:57.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:57.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:57.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:57.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:58.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:58.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:58.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:58.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:10:58.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:58.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:58.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:58.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:58 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 127/678 objects degraded (18.732%), 30 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:58.461 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:58 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 127/678 objects degraded (18.732%), 30 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:58.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:58 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 127/678 objects degraded (18.732%), 30 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-03-30T15:10:58.461 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:10:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:10:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:10:58.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:58.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:58.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:10:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:58.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:58.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:58.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:Exporting 5.15 info 5.15( v 76'8 (0'0,76'8] local-lis/les=72/73 n=1 ec=72/72 lis/c=72/72 les/c/f=73/73/0 sis=72) 2024-03-30T15:10:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:Read #5:a93a5511:::notify.2:head# 2024-03-30T15:10:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:Export successful 2024-03-30T15:10:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:58.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:59.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:10:59.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:10:59.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:10:59.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:10:59.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 322.7s0 scrub starts 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 322.7 scrub ok 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:10:59.289 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[41614]: pgmap v1170: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 322.7s0 scrub starts 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 322.7 scrub ok 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:10:59.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:10:59 smithi028 ceph-mon[29876]: pgmap v1170: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:10:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:10:59.318 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:10:59.318 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:10:59.318 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:10:59.318 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:10:59.318 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 322.7s0 scrub starts 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 322.7 scrub ok 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:10:59.319 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:10:59 smithi134 ceph-mon[33714]: pgmap v1170: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:10:59.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:59.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:10:59.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:10:59.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:10:59.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:10:59.743 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:10:59.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:59.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:10:59.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:10:59.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:10:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:10:59.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:00.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:00.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:00.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:11:00.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:00.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:00.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:00.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:00.351 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:11:00.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:00.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[29876]: 42.13 scrub starts 2024-03-30T15:11:00.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[29876]: 42.13 scrub ok 2024-03-30T15:11:00.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:11:00.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:11:00.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[41614]: 42.13 scrub starts 2024-03-30T15:11:00.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[41614]: 42.13 scrub ok 2024-03-30T15:11:00.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:11:00.429 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:00 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:11:00.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:00 smithi134 ceph-mon[33714]: 42.13 scrub starts 2024-03-30T15:11:00.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:00 smithi134 ceph-mon[33714]: 42.13 scrub ok 2024-03-30T15:11:00.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:00 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:11:00.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:00 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:11:00.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:00.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:00.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:00.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:00.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:00.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:00.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:00.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:00.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:00.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.0 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 5.15 2024-03-30T15:11:01.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:01.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:01.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:01.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:01.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.0/config 2024-03-30T15:11:01.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:01.312 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:01.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[41614]: pgmap v1171: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 2 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[29876]: pgmap v1171: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 2 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:11:01.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:11:01.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:01 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:11:01.492 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:01 smithi134 ceph-mon[33714]: pgmap v1171: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 2 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:01.492 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:01 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:11:01.493 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:01 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:11:01.493 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:01 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:11:01.493 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:01 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:11:01.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:01.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:01.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:01.690 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:01.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:01.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:01.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:01.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:02.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:02.059 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:11:02.059 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:29 323.d 323:b0000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:17.395272+0000", 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 164.66230650099999, 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035830129999999999, 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.060 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 29 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395272+0000", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395272+0000", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395274+0000", 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0899999999999999e-06 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395280+0000", 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3500000000000002e-06 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395289+0000", 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4679999999999998e-06 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395299+0000", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1640000000000007e-06 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395353+0000", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4007e-05 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.395357+0000", 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4329999999999997e-06 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.398700+0000", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033427259999999999 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.398738+0000", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7910000000000001e-05 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.398855+0000", 2024-03-30T15:11:02.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116865 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:63 323.d 323:b0000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.394575+0000", 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 162.66300336200001, 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031624999999999999, 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 63 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394575+0000", 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394575+0000", 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394577+0000", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7999999999999999e-06 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394587+0000", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3530000000000001e-06 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394602+0000", 2024-03-30T15:11:02.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4829e-05 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394610+0000", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0269999999999995e-06 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394669+0000", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9725999999999997e-05 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.394673+0000", 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.946e-06 2024-03-30T15:11:02.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.397639+0000", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002965685 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.397661+0000", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2084000000000001e-05 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.397737+0000", 2024-03-30T15:11:02.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6050000000000005e-05 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(4.1,QREQUEST,e:674,from:4,reservation_nonce:13)", 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:55.423272+0000", 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.63430614399999, 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0021256320000000001, 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.070 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423272+0000", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423272+0000", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423274+0000", 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2989999999999999e-06 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423282+0000", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0690000000000002e-06 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423298+0000", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5483999999999999e-05 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423308+0000", 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8269999999999993e-06 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423385+0000", 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7812999999999995e-05 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.423394+0000", 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1780000000000003e-06 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.425297+0000", 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001903063 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.425316+0000", 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9324e-05 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:55.425397+0000", 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1575000000000004e-05 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:381 326.14 326:28000000:%01:::head [pgnls start_epoch 678] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:59.383833+0000", 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 122.673744899, 2024-03-30T15:11:02.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048835170000000004, 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 381 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383833+0000", 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383833+0000", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383835+0000", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.53e-06 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383841+0000", 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.105e-06 2024-03-30T15:11:02.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383850+0000", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7350000000000008e-06 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383857+0000", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1550000000000004e-06 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383905+0000", 2024-03-30T15:11:02.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8137999999999999e-05 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.383909+0000", 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2880000000000003e-06 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.388598+0000", 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046885820000000002 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.388629+0000", 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1695000000000001e-05 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.388717+0000", 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7288999999999997e-05 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:415 326.14 326:28000000::::head [pgnls start_epoch 680] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e680)", 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:01.392978+0000", 2024-03-30T15:11:02.080 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 120.664600265, 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036966809999999998, 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 415 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392978+0000", 2024-03-30T15:11:02.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392978+0000", 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392980+0000", 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5849999999999998e-06 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.392989+0000", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4379999999999995e-06 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.393005+0000", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6569e-05 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.393013+0000", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8930000000000005e-06 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.393070+0000", 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6986e-05 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.393074+0000", 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1609999999999996e-06 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.396546+0000", 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003471731 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.396581+0000", 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5119e-05 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:01.396674+0000", 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3199e-05 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:733 328.c 328:30000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:11:02.085 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.460664+0000", 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 104.596914368, 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027507600000000001, 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 733 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460664+0000", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460664+0000", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460667+0000", 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9749999999999999e-06 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460675+0000", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0209999999999999e-06 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460689+0000", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.471e-05 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460697+0000", 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7549999999999994e-06 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460770+0000", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3108999999999994e-05 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.460778+0000", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6329999999999996e-06 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.463135+0000", 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023574920000000001 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.463274+0000", 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013844700000000001 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.463414+0000", 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014061799999999999 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.090 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(3.0,QREQUEST,e:725,from:1,reservation_nonce:15)", 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:47.412307+0000", 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 74.645271463, 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022838619999999998, 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412307+0000", 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412307+0000", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412309+0000", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0229999999999999e-06 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412315+0000", 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.426e-06 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412327+0000", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.242e-05 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412336+0000", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8810000000000002e-06 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412425+0000", 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8318000000000004e-05 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.412430+0000", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5210000000000002e-06 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.414513+0000", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020828499999999998 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.414532+0000", 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9040000000000001e-05 2024-03-30T15:11:02.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:47.414590+0000", 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8383e-05 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1300 332.18 332:18000000:test-rados-api-smithi028-93253-131:::head [pgnls start_epoch 732] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e732)", 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:54.403408+0000", 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 67.654170390999994, 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023657610000000001, 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1300 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403408+0000", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403408+0000", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403411+0000", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3569999999999998e-06 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403422+0000", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0614e-05 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403444+0000", 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2150999999999998e-05 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403457+0000", 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3083e-05 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403543+0000", 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6206999999999996e-05 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.403549+0000", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.755e-06 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.405573+0000", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020240509999999998 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.405610+0000", 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7599999999999999e-05 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.405773+0000", 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000162943 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1463 332.18 332:18000000::::head [pgnls start_epoch 734] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e734)", 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:56.407513+0000", 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 65.650064693000004, 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023675390000000001, 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.100 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1463 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407513+0000", 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407513+0000", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407517+0000", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5209999999999998e-06 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407528+0000", 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1547e-05 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407546+0000", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7175000000000001e-05 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407557+0000", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1569000000000001e-05 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407625+0000", 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7606000000000003e-05 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.407630+0000", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5520000000000004e-06 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.409720+0000", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020901499999999998 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.409763+0000", 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2438999999999998e-05 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.409881+0000", 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011798000000000001 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3220 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94326471851904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:12.679439+0000", 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 49.378138976999999, 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000913312, 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.105 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3220 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679439+0000", 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679439+0000", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679444+0000", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4229999999999996e-06 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679459+0000", 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4783000000000001e-05 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679502+0000", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2709000000000001e-05 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.679526+0000", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3796999999999999e-05 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.680186+0000", 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00066004499999999995 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.680259+0000", 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3127999999999995e-05 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:12.680352+0000", 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3426999999999998e-05 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1698 334.8 334:10000000:%01:::head [pgnls start_epoch 753] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e753)", 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:15.436607+0000", 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 46.620970862, 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020358659999999999, 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1698 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436607+0000", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436607+0000", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436610+0000", 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8080000000000001e-06 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436619+0000", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0410000000000003e-06 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436636+0000", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6631000000000001e-05 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436645+0000", 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3470000000000005e-06 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436714+0000", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9488999999999998e-05 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.436721+0000", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3099999999999997e-06 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.438483+0000", 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017623750000000001 2024-03-30T15:11:02.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.438535+0000", 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1808000000000003e-05 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:15.438643+0000", 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108057 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.115 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1732 334.8 334:10000000::::head [pgnls start_epoch 755] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e755)", 2024-03-30T15:11:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:02.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:17.404266+0000", 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 44.653311893000001, 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031147459999999998, 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1732 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.117 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404266+0000", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404266+0000", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404270+0000", 2024-03-30T15:11:02.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7869999999999999e-06 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404283+0000", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3053e-05 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404310+0000", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7097000000000002e-05 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404324+0000", 2024-03-30T15:11:02.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3961999999999999e-05 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404405+0000", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1038000000000006e-05 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.404411+0000", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3709999999999996e-06 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.407190+0000", 2024-03-30T15:11:02.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002778993 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.407231+0000", 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1019000000000001e-05 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.407381+0000", 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000149426 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3240 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94326471851904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:22.680626+0000", 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 39.376951568999999, 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00089794899999999999, 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3240 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680626+0000", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680626+0000", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680633+0000", 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2389999999999999e-06 2024-03-30T15:11:02.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680640+0000", 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6860000000000006e-06 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680659+0000", 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9045000000000001e-05 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.680670+0000", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0549e-05 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.681403+0000", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00073293399999999997 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.681458+0000", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4611999999999998e-05 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:22.681524+0000", 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6884000000000005e-05 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1796 335.18 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.446112+0000", 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.611465590999998, 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.020362603, 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.126 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1796 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.446112+0000", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.446112+0000", 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.446116+0000", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6349999999999999e-06 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.450009+0000", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038932849999999998 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.450029+0000", 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9358e-05 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.450040+0000", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1473e-05 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.450114+0000", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3362999999999995e-05 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.450121+0000", 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0140000000000001e-06 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.451946+0000", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018250549999999999 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.452169+0000", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00022325700000000001 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.455957+0000", 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037881669999999998 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456023+0000", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5860999999999998e-05 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456119+0000", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6448999999999998e-05 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.464533+0000", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084139790000000002 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:11:02.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466324+0000", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017908830000000001 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466366+0000", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1658e-05 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466429+0000", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3276999999999997e-05 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466475+0000", 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5889000000000001e-05 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(osd.6.0:2085 335.18 e760/757 ondisk, result = 0)", 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.456544+0000", 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.601033655000002, 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035176830000000002, 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.133 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456544+0000", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456544+0000", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456546+0000", 2024-03-30T15:11:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.607e-06 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456552+0000", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5940000000000001e-06 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456565+0000", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3869999999999999e-05 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456575+0000", 2024-03-30T15:11:02.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4490000000000005e-06 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460047+0000", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034719019999999998 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460053+0000", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0220000000000001e-06 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460062+0000", 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2390000000000008e-06 2024-03-30T15:11:02.136 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(osd.6.0:2085 335.18 e760/757 ondisk, result = 0)", 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.458764+0000", 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.598813862999997, 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001360258, 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458764+0000", 2024-03-30T15:11:02.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458764+0000", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458769+0000", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3900000000000003e-06 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458780+0000", 2024-03-30T15:11:02.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1437000000000001e-05 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458803+0000", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2626e-05 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458814+0000", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0932e-05 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460079+0000", 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001265551 2024-03-30T15:11:02.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460083+0000", 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9310000000000001e-06 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.460124+0000", 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1390999999999997e-05 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1797 335.18 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:11:02.140 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.466947+0000", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.590631195999997, 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.019184199999999998, 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1797 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466947+0000", 2024-03-30T15:11:02.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466947+0000", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466949+0000", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2479999999999999e-06 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466954+0000", 2024-03-30T15:11:02.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8570000000000003e-06 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466965+0000", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0645999999999999e-05 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466971+0000", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3180000000000003e-06 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467010+0000", 2024-03-30T15:11:02.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9087e-05 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467089+0000", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8685999999999998e-05 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.486015+0000", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018926691999999998 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.486078+0000", 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2700000000000006e-05 2024-03-30T15:11:02.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.486131+0000", 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2966000000000001e-05 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.1.13:104 335.18 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+flush+map_snap_clone+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.467596+0000", 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.589981882000004, 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003014423, 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.145 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467596+0000", 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467596+0000", 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467598+0000", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5600000000000001e-06 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467603+0000", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5289999999999999e-06 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467614+0000", 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0378e-05 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467621+0000", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1940000000000001e-06 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467669+0000", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7757999999999999e-05 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.467777+0000", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108677 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.470611+0000", 2024-03-30T15:11:02.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028333270000000001 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1831 335.18 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.501624+0000", 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 38.555953701999996, 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011032800000000001, 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:11:02.149 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1831 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501624+0000", 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501624+0000", 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501625+0000", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6300000000000005e-07 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501627+0000", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4619999999999999e-06 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501633+0000", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2750000000000003e-06 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501637+0000", 2024-03-30T15:11:02.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.33e-06 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501671+0000", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3711999999999998e-05 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501722+0000", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1144999999999999e-05 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.501789+0000", 2024-03-30T15:11:02.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7361999999999996e-05 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.502180+0000", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039087200000000001 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.502410+0000", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000229664 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.502674+0000", 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00026417500000000001 2024-03-30T15:11:02.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.502703+0000", 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9536999999999999e-05 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.502728+0000", 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4182999999999999e-05 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.154 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25087.0:6 42.d 42:b3c717c4:::4.obj:head [write 0~81] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.457077+0000", 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 17.600501237, 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090047100000000002, 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25087", 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4227304124", 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 6 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457077+0000", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457077+0000", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457078+0000", 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5519999999999999e-06 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457085+0000", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8009999999999998e-06 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457094+0000", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5499999999999995e-06 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457103+0000", 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9269999999999994e-06 2024-03-30T15:11:02.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457150+0000", 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7581999999999999e-05 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457226+0000", 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5827000000000004e-05 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,5", 2024-03-30T15:11:02.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457308+0000", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2181000000000005e-05 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457688+0000", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037997600000000002 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457881+0000", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019327799999999999 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457914+0000", 2024-03-30T15:11:02.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2931e-05 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457951+0000", 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6668999999999997e-05 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.457977+0000", 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6197e-05 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:11:02.160 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:11:02.161 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 -- 172.21.15.28:0/307799029 >> v1:172.21.15.28:6790/0 conn(0x7ff1b8101130 legacy=0x7ff1b8101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 -- 172.21.15.28:0/307799029 shutdown_connections 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 -- 172.21.15.28:0/307799029 >> 172.21.15.28:0/307799029 conn(0x7ff1b80fc9e0 msgr2=0x7ff1b80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 -- 172.21.15.28:0/307799029 shutdown_connections 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 -- 172.21.15.28:0/307799029 wait complete. 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 librados: starting msgr at 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.203+0000 7ff1bf343700 1 librados: starting objecter 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 Processor -- start 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 -- start start 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 librados: setting wanted keys 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 librados: calling monclient init 2024-03-30T15:11:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1b8106950 con 0x7ff1b810d960 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1b81a8540 con 0x7ff1b8101130 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bf343700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff1b81a9720 con 0x7ff1b8109dc0 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bc8de700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff1b8109dc0 0x7ff1b8104e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:33482/0 (socket says 172.21.15.28:33482) 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1bc8de700 1 -- 172.21.15.28:0/2506553291 learned_addr learned my addr 172.21.15.28:0/2506553291 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter ms_handle_connect 0x7ff1b810d960 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter ms_handle_connect 0x7ff1b8109dc0 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter ms_handle_connect 0x7ff1b8101130 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782526774 0 0) 0x7ff1b8106950 con 0x7ff1b810d960 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a00031f0 con 0x7ff1b810d960 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61258631 0 0) 0x7ff1b81a8540 con 0x7ff1b8101130 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b8106950 con 0x7ff1b8101130 2024-03-30T15:11:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2793240851 0 0) 0x7ff1b81a9720 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b81a8540 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.204+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3497968462 0 0) 0x7ff1b81a8540 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1b81a9720 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1ac001a30 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 650707123 0 0) 0x7ff1a00031f0 con 0x7ff1b810d960 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1b81a8540 con 0x7ff1b810d960 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2095654854 0 0) 0x7ff1b8106950 con 0x7ff1b8101130 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1a00031f0 con 0x7ff1b8101130 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1a8001af0 con 0x7ff1b810d960 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1b0001800 con 0x7ff1b8101130 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3773268088 0 0) 0x7ff1b81a9720 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 >> v1:172.21.15.134:6789/0 conn(0x7ff1b8101130 legacy=0x7ff1b8104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 >> v1:172.21.15.28:6789/0 conn(0x7ff1b810d960 legacy=0x7ff1b81a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b81aa900 con 0x7ff1b8109dc0 2024-03-30T15:11:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b81a8770 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1bf343700 10 client.58802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1b81a8c30 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1bf343700 1 librados: init done 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff1ac000f80 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.205+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff1ac005ce0 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.206+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a4005120 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.206+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff1ac003930 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(783..783 src has 442..783) ==== 9296+0+0 (unknown 3301037594 0 0) 0x7ff1ac089d00 con 0x7ff1b8109dc0 2024-03-30T15:11:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 10 client.58802.objecter ms_dispatch 0x7ff1b8001550 osd_map(783..783 src has 442..783) 2024-03-30T15:11:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 3 client.58802.objecter handle_osd_map got epochs [783,783] > 0 2024-03-30T15:11:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 3 client.58802.objecter handle_osd_map decoding full epoch 783 2024-03-30T15:11:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 20 client.58802.objecter dump_active .. 0 homeless 2024-03-30T15:11:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.207+0000 7ff1b67fc700 10 client.58802.objecter ms_handle_connect 0x7ff1a00704e0 2024-03-30T15:11:02.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.211+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff1ac056aa0 con 0x7ff1b8109dc0 2024-03-30T15:11:02.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:02.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:02.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:02.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:02.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:02 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:02.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:02 smithi134 ceph-mon[33714]: 322.1s0 scrub starts 2024-03-30T15:11:02.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:02 smithi134 ceph-mon[33714]: 322.1 scrub ok 2024-03-30T15:11:02.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:02.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[41614]: 322.1s0 scrub starts 2024-03-30T15:11:02.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[41614]: 322.1 scrub ok 2024-03-30T15:11:02.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:02.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[29876]: 322.1s0 scrub starts 2024-03-30T15:11:02.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:02 smithi028 ceph-mon[29876]: 322.1 scrub ok 2024-03-30T15:11:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:02.442+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff1a4005410 con 0x7ff1b8109dc0 2024-03-30T15:11:02.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:02.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:02.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:02.616 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:02.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:02.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:02.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:11:02.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:02.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:02.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:02.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:03.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:03.068 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:11:03.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:03.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:03.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:03.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:03.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.405+0000 7ff1b67fc700 1 -- 172.21.15.28:0/2506553291 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v784) ==== 101+0+0 (unknown 908492273 0 0) 0x7ff1ac05a4f0 con 0x7ff1b8109dc0 2024-03-30T15:11:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 10 librados: watch_flush enter 2024-03-30T15:11:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 10 librados: watch_flush exit 2024-03-30T15:11:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 20 client.58802.objecter shutdown clearing up homeless session... 2024-03-30T15:11:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 10 client.58802.objecter successfully canceled tick 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff1a00704e0 legacy=0x7ff1a00729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 >> v1:172.21.15.28:6790/0 conn(0x7ff1b8109dc0 legacy=0x7ff1b8104e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 shutdown_connections 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 >> 172.21.15.28:0/2506553291 conn(0x7ff1b80fc9e0 msgr2=0x7ff1b81105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 shutdown_connections 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 -- 172.21.15.28:0/2506553291 wait complete. 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.408+0000 7ff1bf343700 1 librados: shutdown 2024-03-30T15:11:03.410 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[29876]: pgmap v1172: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2506553291' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[29876]: from='client.58802 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[41614]: pgmap v1172: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2506553291' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:03 smithi028 ceph-mon[41614]: from='client.58802 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.488 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:03 smithi134 ceph-mon[33714]: pgmap v1172: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.2 KiB/s wr, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:03.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2506553291' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:03 smithi134 ceph-mon[33714]: from='client.58802 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:03.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:03.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:03.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:03.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:03.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:03.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:04.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:04.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:04.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:04.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:04.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:04.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:04.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:04.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:04.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:04.525 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: 42.15 scrub starts 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: 42.15 scrub ok 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: from='client.58802 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: osdmap e784: 8 total, 6 up, 8 in 2024-03-30T15:11:04.526 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2192117813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:04.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:04.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: 42.15 scrub starts 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: 42.15 scrub ok 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: from='client.58802 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: osdmap e784: 8 total, 6 up, 8 in 2024-03-30T15:11:04.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2192117813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: 42.15 scrub starts 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: 42.15 scrub ok 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: from='client.58802 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: osdmap e784: 8 total, 6 up, 8 in 2024-03-30T15:11:04.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2192117813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:04.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:11:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:04.863 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:04.863 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.929+0000 7fd370788700 1 -- 172.21.15.28:0/2638443043 >> v1:172.21.15.28:6790/0 conn(0x7fd36810aee0 legacy=0x7fd36810d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.929+0000 7fd370788700 1 -- 172.21.15.28:0/2638443043 shutdown_connections 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.929+0000 7fd370788700 1 -- 172.21.15.28:0/2638443043 >> 172.21.15.28:0/2638443043 conn(0x7fd3680fdb20 msgr2=0x7fd3680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.929+0000 7fd370788700 1 -- 172.21.15.28:0/2638443043 shutdown_connections 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.929+0000 7fd370788700 1 -- 172.21.15.28:0/2638443043 wait complete. 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 librados: starting msgr at 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 librados: starting objecter 2024-03-30T15:11:04.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 Processor -- start 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 -- start start 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 librados: setting wanted keys 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 librados: calling monclient init 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3681a7190 con 0x7fd368102250 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3681a83b0 con 0x7fd36810aee0 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd370788700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd3681a95f0 con 0x7fd36810ea80 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd36e524700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd368102250 0x7fd368107620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41038/0 (socket says 172.21.15.28:41038) 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd36e524700 1 -- 172.21.15.28:0/2192117813 learned_addr learned my addr 172.21.15.28:0/2192117813 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd3677fe700 10 client.?.objecter ms_handle_connect 0x7fd368102250 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd3677fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd3677fe700 10 client.?.objecter ms_handle_connect 0x7fd36810ea80 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd3677fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.930+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638516870 0 0) 0x7fd3681a95f0 con 0x7fd36810ea80 2024-03-30T15:11:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3500031f0 con 0x7fd36810ea80 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 10 client.?.objecter ms_handle_connect 0x7fd36810aee0 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103923743 0 0) 0x7fd3681a7190 con 0x7fd368102250 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3681a95f0 con 0x7fd368102250 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 735657992 0 0) 0x7fd3681a83b0 con 0x7fd36810aee0 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3681a7190 con 0x7fd36810aee0 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3404523850 0 0) 0x7fd3500031f0 con 0x7fd36810ea80 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3681a83b0 con 0x7fd36810ea80 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd3600018b0 con 0x7fd36810ea80 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1705467582 0 0) 0x7fd3681a95f0 con 0x7fd368102250 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3500031f0 con 0x7fd368102250 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd358001650 con 0x7fd368102250 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3291354037 0 0) 0x7fd3681a7190 con 0x7fd36810aee0 2024-03-30T15:11:04.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3681a95f0 con 0x7fd36810aee0 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd35c001500 con 0x7fd36810aee0 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2101264499 0 0) 0x7fd3681a83b0 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 >> v1:172.21.15.134:6789/0 conn(0x7fd36810aee0 legacy=0x7fd3681a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 >> v1:172.21.15.28:6789/0 conn(0x7fd368102250 legacy=0x7fd368107620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3681aa830 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3681a9820 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd370788700 10 client.58808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3681a9ce0 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd370788700 1 librados: init done 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd360003b40 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.931+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd360004a30 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.932+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd354005100 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd3600036a0 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(784..784 src has 442..784) ==== 9296+0+0 (unknown 3076598767 0 0) 0x7fd360088780 con 0x7fd36810ea80 2024-03-30T15:11:04.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 osd_map(784..784 src has 442..784) 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 3 client.58808.objecter handle_osd_map got epochs [784,784] > 0 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 3 client.58808.objecter handle_osd_map decoding full epoch 784 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 20 client.58808.objecter dump_active .. 0 homeless 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v784) ==== 54+0+16 (unknown 2312917249 0 2420000192) 0x7fd36008abe0 con 0x7fd36810ea80 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd3500703b0 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 10 client.58808.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 1 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 20 client.58808.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 10 client.58808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=785}) -- 0x7fd3540053e0 con 0x7fd36810ea80 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.933+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd354000e30 con 0x7fd36810ea80 2024-03-30T15:11:04.870 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=784) ==== 24+0+0 (unknown 30262567 0 0) 0x7fd360054bf0 con 0x7fd36810ea80 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd36cd21700 10 client.58808.objecter _check_command_map_dne tid 1 current 784 map_dne_bound 784 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd36cd21700 10 client.58808.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd36cd21700 15 client.58808.objecter _session_command_op_remove -1 1 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 10 client.58808.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 2 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 2 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 1 2 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 4 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 10 client.58808.objecter _send_command 2 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fd3540022f0 con 0x7fd354002790 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.934+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd354002790 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.935+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3540022f0 con 0x7fd354002790 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.935+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.935+0000 7fd3677fe700 10 client.58808.objecter _finish_command 2 = system:0 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.935+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 4 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.935+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 1 2 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 10 client.58808.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 3 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd354001bb0 osd=1 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 1 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 4 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 10 client.58808.objecter _send_command 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.952+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd354004c70 con 0x7fd354002790 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd354004c70 con 0x7fd354002790 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 10 client.58808.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd354001bb0 osd=1 4 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 1 3 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 10 client.58808.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 4 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 10 client.58808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fd354004f00 con 0x7fd36810ea80 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=784) ==== 24+0+0 (unknown 3404854340 0 0) 0x7fd360052f60 con 0x7fd36810ea80 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd367fff700 10 client.58808.objecter _check_command_map_dne tid 4 current 784 map_dne_bound 784 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd367fff700 10 client.58808.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd367fff700 15 client.58808.objecter _session_command_op_remove -1 4 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 10 client.58808.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 5 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 5 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 3 5 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 4 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 10 client.58808.objecter _send_command 5 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd3540048a0 con 0x7fd354004240 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.954+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd354004240 2024-03-30T15:11:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.955+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd3540048a0 con 0x7fd354004240 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.955+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.955+0000 7fd3677fe700 10 client.58808.objecter _finish_command 5 = system:0 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.955+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 4 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.955+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 3 5 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 10 client.58808.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 6 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 6 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd354001a00 osd=3 3 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 3 6 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 4 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 10 client.58808.objecter _send_command 6 2024-03-30T15:11:04.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.972+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd354003780 con 0x7fd354004240 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd354003780 con 0x7fd354004240 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd3677fe700 10 client.58808.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd354001a00 osd=3 4 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 3 6 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 10 client.58808.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 7 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 7 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 4 7 2024-03-30T15:11:04.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 4 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 10 client.58808.objecter _send_command 7 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.974+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd354012050 con 0x7fd35400e540 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd35400e540 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd354012050 con 0x7fd35400e540 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 10 client.58808.objecter _finish_command 7 = system:0 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 4 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.975+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 4 7 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 10 client.58808.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 8 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 8 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd3540092e0 osd=4 3 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 4 8 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 4 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 10 client.58808.objecter _send_command 8 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.992+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd354008dc0 con 0x7fd35400e540 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd354008dc0 con 0x7fd35400e540 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 10 client.58808.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd3540092e0 osd=4 4 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 4 8 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 10 client.58808.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 9 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 9 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 5 9 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 4 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 10 client.58808.objecter _send_command 9 2024-03-30T15:11:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd354016e90 con 0x7fd354013380 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.994+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd354013380 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.995+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd354016e90 con 0x7fd354013380 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.995+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.995+0000 7fd3677fe700 10 client.58808.objecter _finish_command 9 = system:0 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.995+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 4 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:03.995+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 5 9 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 10 client.58808.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 10 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 10 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400db90 osd=5 3 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 5 10 2024-03-30T15:11:04.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 4 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 10 client.58808.objecter _send_command 10 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.012+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd35400d610 con 0x7fd354013380 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd35400d610 con 0x7fd354013380 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 10 client.58808.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400db90 osd=5 4 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 5 10 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 10 client.58808.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 11 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 11 2024-03-30T15:11:04.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 6 11 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 4 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 10 client.58808.objecter _send_command 11 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd35400d530 con 0x7fd354012ee0 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.014+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd354012ee0 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.015+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd35400d530 con 0x7fd354012ee0 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.015+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.015+0000 7fd3677fe700 10 client.58808.objecter _finish_command 11 = system:0 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.015+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 4 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.015+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 6 11 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 10 client.58808.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 12 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 12 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400d3c0 osd=6 3 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 6 12 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 4 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 10 client.58808.objecter _send_command 12 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.032+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd35400d530 con 0x7fd354012ee0 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd35400d530 con 0x7fd354012ee0 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 10 client.58808.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400d3c0 osd=6 4 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 6 12 2024-03-30T15:11:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 10 client.58808.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 13 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 13 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 7 13 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 4 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 10 client.58808.objecter _send_command 13 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd354023fa0 con 0x7fd3540202f0 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.034+0000 7fd3677fe700 10 client.58808.objecter ms_handle_connect 0x7fd3540202f0 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.035+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd354023fa0 con 0x7fd3540202f0 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.035+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.035+0000 7fd3677fe700 10 client.58808.objecter _finish_command 13 = system:0 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.035+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 4 2024-03-30T15:11:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.035+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 7 13 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 10 client.58808.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign -1 14 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 20 client.58808.objecter _get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 15 client.58808.objecter _session_command_op_remove -1 14 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 20 client.58808.objecter get_session s=0x7fd35400d530 osd=7 3 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 15 client.58808.objecter _session_command_op_assign 7 14 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 4 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 10 client.58808.objecter _send_command 14 2024-03-30T15:11:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.052+0000 7fd370788700 1 -- 172.21.15.28:0/2192117813 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd354025950 con 0x7fd3540202f0 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.054+0000 7fd3677fe700 1 -- 172.21.15.28:0/2192117813 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd354025950 con 0x7fd3540202f0 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.054+0000 7fd3677fe700 10 client.58808.objecter ms_dispatch 0x7fd368001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.054+0000 7fd3677fe700 10 client.58808.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.054+0000 7fd3677fe700 20 client.58808.objecter put_session s=0x7fd35400d530 osd=7 4 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:04.054+0000 7fd3677fe700 15 client.58808.objecter _session_command_op_remove 7 14 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:04.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:04.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:05.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:05.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:05.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:05.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:05.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:05.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:05.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:11:05.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[41614]: pgmap v1174: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:05.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:11:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:11:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[29876]: pgmap v1174: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:11:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:05 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:11:05.485 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:05 smithi134 ceph-mon[33714]: pgmap v1174: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:05.485 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:05 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:11:05.485 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:05 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:11:05.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:05.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:05.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:05.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:05.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:05.702 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:11:05.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:05.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:05.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:05.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:06.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:06.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:06.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:06.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:06.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:06.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:06.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:11:06.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: marking collection for removal 2024-03-30T15:11:06.411 INFO:teuthology.orchestra.run.smithi028.stdout:setting '_remove' omap key 2024-03-30T15:11:06.411 INFO:teuthology.orchestra.run.smithi028.stdout:finish_remove_pgs 5.15_head removing 5.15 2024-03-30T15:11:06.411 INFO:teuthology.orchestra.run.smithi028.stdout:Remove successful 2024-03-30T15:11:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:06.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:06 smithi134 ceph-mon[33714]: 42.17 scrub starts 2024-03-30T15:11:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:06 smithi134 ceph-mon[33714]: 42.17 scrub ok 2024-03-30T15:11:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:06 smithi134 ceph-mon[33714]: 322.4s0 scrub starts 2024-03-30T15:11:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:06 smithi134 ceph-mon[33714]: 322.4 scrub ok 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[29876]: 42.17 scrub starts 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[29876]: 42.17 scrub ok 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[29876]: 322.4s0 scrub starts 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[29876]: 322.4 scrub ok 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[41614]: 42.17 scrub starts 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[41614]: 42.17 scrub ok 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[41614]: 322.4s0 scrub starts 2024-03-30T15:11:06.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:06 smithi028 ceph-mon[41614]: 322.4 scrub ok 2024-03-30T15:11:06.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:06.542 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:06.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:06.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:06.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:06.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:06.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:06.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:06.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:07.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:07.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:07.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:07.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:07.420 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:07.420 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:07.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:07.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:07.599 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: 42.19 scrub starts 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: 42.19 scrub ok 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: pgmap v1175: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: osdmap e785: 8 total, 6 up, 8 in 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: 42.19 scrub starts 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: 42.19 scrub ok 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: pgmap v1175: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: osdmap e785: 8 total, 6 up, 8 in 2024-03-30T15:11:07.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:11:07.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:07 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:11:07.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:07.649 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:07.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: 42.19 scrub starts 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: 42.19 scrub ok 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: pgmap v1175: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: osdmap e785: 8 total, 6 up, 8 in 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:11:07.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:07 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:11:07.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:07.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:07.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:08.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:08.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:08.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:08.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:08.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:11:08.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:08.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:08.395 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:11:08.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:11:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:11:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:11:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:08 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:08.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:11:08.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:08 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:11:08.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:08.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-30T15:11:08.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:08.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:08.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:08.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:08.817 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T15:11:08.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:08.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:08.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:09.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:09.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:09.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:09.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:09.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:09.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:09.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:11:09.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:11:09.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[41614]: pgmap v1177: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:09.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:11:09.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:11:09.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:09 smithi028 ceph-mon[29876]: pgmap v1177: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 -- 172.21.15.28:0/4093718100 >> v1:172.21.15.28:6790/0 conn(0x7f5ae0109dc0 legacy=0x7f5ae010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 -- 172.21.15.28:0/4093718100 shutdown_connections 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 -- 172.21.15.28:0/4093718100 >> 172.21.15.28:0/4093718100 conn(0x7f5ae00fc9e0 msgr2=0x7f5ae00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 -- 172.21.15.28:0/4093718100 shutdown_connections 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 -- 172.21.15.28:0/4093718100 wait complete. 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 librados: starting msgr at 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 librados: starting objecter 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.344+0000 7f5ae6e65700 1 Processor -- start 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 -- start start 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 librados: setting wanted keys 2024-03-30T15:11:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 librados: calling monclient init 2024-03-30T15:11:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01a92a0 con 0x7f5ae0101130 2024-03-30T15:11:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01aa4c0 con 0x7f5ae0109dc0 2024-03-30T15:11:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae6e65700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5ae01ab700 con 0x7f5ae010d960 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae4c01700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ae0101130 0x7f5ae0106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33774/0 (socket says 172.21.15.28:33774) 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5ae4c01700 1 -- 172.21.15.28:0/3991904500 learned_addr learned my addr 172.21.15.28:0/3991904500 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter ms_handle_connect 0x7f5ae010d960 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter ms_handle_connect 0x7f5ae0101130 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter ms_handle_connect 0x7f5ae0109dc0 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525999709 0 0) 0x7f5ae01aa4c0 con 0x7f5ae0109dc0 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad00031f0 con 0x7f5ae0109dc0 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241860851 0 0) 0x7f5ae01a92a0 con 0x7f5ae0101130 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae01aa4c0 con 0x7f5ae0101130 2024-03-30T15:11:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543233081 0 0) 0x7f5ae01ab700 con 0x7f5ae010d960 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae01a92a0 con 0x7f5ae010d960 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 566672661 0 0) 0x7f5ad00031f0 con 0x7f5ae0109dc0 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ae01ab700 con 0x7f5ae0109dc0 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.345+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5acc004540 con 0x7f5ae0109dc0 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1678704699 0 0) 0x7f5ae01a92a0 con 0x7f5ae010d960 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ad00031f0 con 0x7f5ae010d960 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1335894885 0 0) 0x7f5ae01aa4c0 con 0x7f5ae0101130 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ae01a92a0 con 0x7f5ae0101130 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5ad8001bb0 con 0x7f5ae010d960 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5ad4001130 con 0x7f5ae0101130 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3576552475 0 0) 0x7f5ae01ab700 con 0x7f5ae0109dc0 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 >> v1:172.21.15.28:6790/0 conn(0x7f5ae010d960 legacy=0x7f5ae01a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 >> v1:172.21.15.28:6789/0 conn(0x7f5ae0101130 legacy=0x7f5ae0106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:09.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae01ac940 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae01ab930 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5acc001b70 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5acc004f70 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5ae6e65700 10 client.50812.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ae01abd30 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.346+0000 7f5ae6e65700 1 librados: init done 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.347+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5acc01b940 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.347+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae0102610 con 0x7f5ae0109dc0 2024-03-30T15:11:09.351 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(785..785 src has 442..785) ==== 9296+0+0 (unknown 637142690 0 0) 0x7f5acc088ca0 con 0x7f5ae0109dc0 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 10 client.50812.objecter ms_dispatch 0x7f5ae0001550 osd_map(785..785 src has 442..785) 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 3 client.50812.objecter handle_osd_map got epochs [785,785] > 0 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 3 client.50812.objecter handle_osd_map decoding full epoch 785 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 20 client.50812.objecter dump_active .. 0 homeless 2024-03-30T15:11:09.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.348+0000 7f5addffb700 10 client.50812.objecter ms_handle_connect 0x7f5ad0070420 2024-03-30T15:11:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.352+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5acc055a20 con 0x7f5ae0109dc0 2024-03-30T15:11:09.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:09.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:09.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:09.572+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5ae0068b10 con 0x7f5ae0109dc0 2024-03-30T15:11:09.589 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:09 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:11:09.590 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:09 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:11:09.590 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:09 smithi134 ceph-mon[33714]: pgmap v1177: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:09.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:09.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:09.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:09.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:09.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:09.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:09.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:10.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:10.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:10.121 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:10.121 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:10.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:10.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.262+0000 7f5addffb700 1 -- 172.21.15.28:0/3991904500 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v786) ==== 87+0+0 (unknown 3490252432 0 0) 0x7f5acc059470 con 0x7f5ae0109dc0 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 10 librados: watch_flush enter 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 10 librados: watch_flush exit 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 20 client.50812.objecter shutdown clearing up homeless session... 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 10 client.50812.objecter successfully canceled tick 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5ad0070420 legacy=0x7f5ad00728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 >> v1:172.21.15.134:6789/0 conn(0x7f5ae0109dc0 legacy=0x7f5ae01a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 shutdown_connections 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 >> 172.21.15.28:0/3991904500 conn(0x7f5ae00fc9e0 msgr2=0x7f5ae0106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 shutdown_connections 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.265+0000 7f5ae6e65700 1 -- 172.21.15.28:0/3991904500 wait complete. 2024-03-30T15:11:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:10.266+0000 7f5ae6e65700 1 librados: shutdown 2024-03-30T15:11:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:11:10.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:10.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:10.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3991904500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:10 smithi134 ceph-mon[33714]: from='client.50812 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3991904500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:10 smithi028 ceph-mon[29876]: from='client.50812 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3991904500' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:10 smithi028 ceph-mon[41614]: from='client.50812 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:10.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:10.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:10.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:10.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:10.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:10.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:10.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:11.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:11.015 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:11.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:11.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:11:11.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:11.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:11.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:11.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:11.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: pgmap v1178: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: from='client.50812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[29876]: osdmap e786: 8 total, 6 up, 8 in 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: pgmap v1178: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: from='client.50812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:11.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:11 smithi028 ceph-mon[41614]: osdmap e786: 8 total, 6 up, 8 in 2024-03-30T15:11:11.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:11.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:11:11.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:11.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: pgmap v1178: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: from='client.50812 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:11.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:11 smithi134 ceph-mon[33714]: osdmap e786: 8 total, 6 up, 8 in 2024-03-30T15:11:11.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:11.789 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:11.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:11.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:12.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:12.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:12.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:12.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:12.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:12.277 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:12 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:12.277 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:12 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:12.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:12.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:12.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:12.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:12.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:12.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:12.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:12 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:12.680 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:11:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=sqlstore.transactions t=2024-03-30T15:11:12.452698447Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-30T15:11:12.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:12.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:12.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:12.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:12.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:13.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:13.024 INFO:teuthology.orchestra.run.smithi028.stdout:6.f 2024-03-30T15:11:13.024 INFO:teuthology.orchestra.run.smithi028.stdout:6.e 2024-03-30T15:11:13.024 INFO:teuthology.orchestra.run.smithi028.stdout:337.15 2024-03-30T15:11:13.024 INFO:teuthology.orchestra.run.smithi028.stdout:5.14 2024-03-30T15:11:13.024 INFO:teuthology.orchestra.run.smithi028.stdout:3.1e 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:5.1c 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:5.b 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:3.0 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:3.4 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.1d 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:337.7 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:322.0s2 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:42.1b 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.16 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.c 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:42.5 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.b 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.d 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.9 2024-03-30T15:11:13.025 INFO:teuthology.orchestra.run.smithi028.stdout:12.a 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:12.8 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:42.7 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:5.a 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:1.5 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:337.1a 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:6.1 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:12.0 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:6.0 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:3.5 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:3.9 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:1.a 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:3.1f 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:1.9 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:6.8 2024-03-30T15:11:13.026 INFO:teuthology.orchestra.run.smithi028.stdout:1.8 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:1.1 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:12.17 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:5.1e 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:6.9 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:12.1f 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.10 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.c 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.11 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:6.12 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:12.f 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:12.5 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:12.7 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:3.1a 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.5 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.1d 2024-03-30T15:11:13.027 INFO:teuthology.orchestra.run.smithi028.stdout:337.8 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:4.4 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:6.1c 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:6.3 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:337.b 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:3.13 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:6.2 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:337.3 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:4.18 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:5.4 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:6.14 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:42.1e 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:2.1 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:4.19 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:5.8 2024-03-30T15:11:13.028 INFO:teuthology.orchestra.run.smithi028.stdout:4.1c 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:4.12 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:4.6 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:12.15 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:4.a 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:4.d 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:12.1c 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.10 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.1a 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.11 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.1d 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.2 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.c 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:5.6 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:42.1 2024-03-30T15:11:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:6.17 2024-03-30T15:11:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:13.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:13.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:11:13.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:13.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.193+0000 7f156da76700 1 -- 172.21.15.28:0/1142381282 >> v1:172.21.15.28:6790/0 conn(0x7f156810aed0 legacy=0x7f156810d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.193+0000 7f156da76700 1 -- 172.21.15.28:0/1142381282 shutdown_connections 2024-03-30T15:11:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.193+0000 7f156da76700 1 -- 172.21.15.28:0/1142381282 >> 172.21.15.28:0/1142381282 conn(0x7f15680fdaf0 msgr2=0x7f15680fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.193+0000 7f156da76700 1 -- 172.21.15.28:0/1142381282 shutdown_connections 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.193+0000 7f156da76700 1 -- 172.21.15.28:0/1142381282 wait complete. 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 librados: starting msgr at 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 librados: starting objecter 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 Processor -- start 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 -- start start 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 librados: setting wanted keys 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 librados: calling monclient init 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15681a7120 con 0x7f156810ea70 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.194+0000 7f156da76700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15681a8340 con 0x7f156810aed0 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f15677fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1568102240 0x7f15681075b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:35724/0 (socket says 172.21.15.28:35724) 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f15677fe700 1 -- 172.21.15.28:0/3983017837 learned_addr learned my addr 172.21.15.28:0/3983017837 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15681a9580 con 0x7f1568102240 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter ms_handle_connect 0x7f156810aed0 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter ms_handle_connect 0x7f1568102240 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter ms_handle_connect 0x7f156810ea70 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628200745 0 0) 0x7f15681a8340 con 0x7f156810aed0 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15540031f0 con 0x7f156810aed0 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474397389 0 0) 0x7f15681a9580 con 0x7f1568102240 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15681a8340 con 0x7f1568102240 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1180599748 0 0) 0x7f15681a7120 con 0x7f156810ea70 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15681a9580 con 0x7f156810ea70 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3982739980 0 0) 0x7f15540031f0 con 0x7f156810aed0 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15681a7120 con 0x7f156810aed0 2024-03-30T15:11:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f15500044b0 con 0x7f156810aed0 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1904523633 0 0) 0x7f15681a8340 con 0x7f1568102240 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15540031f0 con 0x7f1568102240 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1558001830 con 0x7f1568102240 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3418325017 0 0) 0x7f15681a9580 con 0x7f156810ea70 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15681a8340 con 0x7f156810ea70 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1560001470 con 0x7f156810ea70 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1781768307 0 0) 0x7f15681a7120 con 0x7f156810aed0 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 >> v1:172.21.15.28:6790/0 conn(0x7f1568102240 legacy=0x7f15681075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 >> v1:172.21.15.28:6789/0 conn(0x7f156810ea70 legacy=0x7f15681a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.195+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15681aa7c0 con 0x7f156810aed0 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15681a97b0 con 0x7f156810aed0 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f156da76700 10 client.50818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15681a9c70 con 0x7f156810aed0 2024-03-30T15:11:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f156da76700 1 librados: init done 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1550004750 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.196+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1550004fd0 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.197+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1548005100 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.197+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1550005280 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(787..787 src has 442..787) ==== 9296+0+0 (unknown 85994185 0 0) 0x7f15500889e0 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 osd_map(787..787 src has 442..787) 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 3 client.50818.objecter handle_osd_map got epochs [787,787] > 0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 3 client.50818.objecter handle_osd_map decoding full epoch 787 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 20 client.50818.objecter dump_active .. 0 homeless 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v787) ==== 54+0+16 (unknown 2531953402 0 2420000192) 0x7f1550088dd0 con 0x7f156810aed0 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f1554070480 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 10 client.50818.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 1 2024-03-30T15:11:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 10 client.50818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=788}) -- 0x7f15480053e0 con 0x7f156810aed0 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1548000e30 con 0x7f156810aed0 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=787) ==== 24+0+0 (unknown 305505748 0 0) 0x7f1550054f10 con 0x7f156810aed0 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1565ffb700 10 client.50818.objecter _check_command_map_dne tid 1 current 787 map_dne_bound 787 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1565ffb700 10 client.50818.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f1565ffb700 15 client.50818.objecter _session_command_op_remove -1 1 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 10 client.50818.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 2 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 2 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 1 2 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 4 2024-03-30T15:11:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 10 client.50818.objecter _send_command 2 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.198+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f1548002330 con 0x7f1548002790 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.199+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f1548002790 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.200+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1548002330 con 0x7f1548002790 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.200+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.200+0000 7f1564ff9700 10 client.50818.objecter _finish_command 2 = system:0 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.200+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 4 2024-03-30T15:11:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.200+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 1 2 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 10 client.50818.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f1548001bb0 osd=1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 1 3 2024-03-30T15:11:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 4 2024-03-30T15:11:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 10 client.50818.objecter _send_command 3 2024-03-30T15:11:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.217+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1548004cc0 con 0x7f1548002790 2024-03-30T15:11:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.218+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1548004cc0 con 0x7f1548002790 2024-03-30T15:11:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.218+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.218+0000 7f1564ff9700 10 client.50818.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.218+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f1548001bb0 osd=1 4 2024-03-30T15:11:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.218+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 1 3 2024-03-30T15:11:13.220 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 10 client.50818.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 4 2024-03-30T15:11:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 10 client.50818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f1548001970 con 0x7f156810aed0 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=787) ==== 24+0+0 (unknown 3641283767 0 0) 0x7f1550004fd0 con 0x7f156810aed0 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f15657fa700 10 client.50818.objecter _check_command_map_dne tid 4 current 787 map_dne_bound 787 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f15657fa700 10 client.50818.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f15657fa700 15 client.50818.objecter _session_command_op_remove -1 4 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 10 client.50818.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 5 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 5 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 3 5 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 4 2024-03-30T15:11:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 10 client.50818.objecter _send_command 5 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1548004920 con 0x7f15480043d0 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.219+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f15480043d0 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.220+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1548004920 con 0x7f15480043d0 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.220+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.220+0000 7f1564ff9700 10 client.50818.objecter _finish_command 5 = system:0 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.220+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 4 2024-03-30T15:11:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.220+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 3 5 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 10 client.50818.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 6 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 6 2024-03-30T15:11:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f1548003d50 osd=3 3 2024-03-30T15:11:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 3 6 2024-03-30T15:11:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 4 2024-03-30T15:11:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 10 client.50818.objecter _send_command 6 2024-03-30T15:11:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.257+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1548003830 con 0x7f15480043d0 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1548003830 con 0x7f15480043d0 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 10 client.50818.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f1548003d50 osd=3 4 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 3 6 2024-03-30T15:11:13.260 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 10 client.50818.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 7 2024-03-30T15:11:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 7 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 4 7 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 4 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 10 client.50818.objecter _send_command 7 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1548016460 con 0x7f1548012950 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.259+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f1548012950 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.260+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1548016460 con 0x7f1548012950 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.260+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.260+0000 7f1564ff9700 10 client.50818.objecter _finish_command 7 = system:0 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.260+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 4 2024-03-30T15:11:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.260+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 4 7 2024-03-30T15:11:13.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 10 client.50818.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 8 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 8 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f15480092e0 osd=4 3 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 4 8 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 4 2024-03-30T15:11:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 10 client.50818.objecter _send_command 8 2024-03-30T15:11:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.281+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1548008e20 con 0x7f1548012950 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1548008e20 con 0x7f1548012950 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 10 client.50818.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f15480092e0 osd=4 4 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 4 8 2024-03-30T15:11:13.284 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 10 client.50818.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 9 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 9 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 5 9 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 4 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 10 client.50818.objecter _send_command 9 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f154800e2d0 con 0x7f1548017790 2024-03-30T15:11:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.283+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f1548017790 2024-03-30T15:11:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.284+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f154800e2d0 con 0x7f1548017790 2024-03-30T15:11:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.284+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.284+0000 7f1564ff9700 10 client.50818.objecter _finish_command 9 = system:0 2024-03-30T15:11:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.284+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 4 2024-03-30T15:11:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.284+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 5 9 2024-03-30T15:11:13.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:13.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 10 client.50818.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 10 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 10 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154800db90 osd=5 3 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 5 10 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 4 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 10 client.50818.objecter _send_command 10 2024-03-30T15:11:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.305+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1548018ad0 con 0x7f1548017790 2024-03-30T15:11:13.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1548018ad0 con 0x7f1548017790 2024-03-30T15:11:13.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f1564ff9700 10 client.50818.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154800db90 osd=5 4 2024-03-30T15:11:13.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 5 10 2024-03-30T15:11:13.309 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:13.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 10 client.50818.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 11 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 11 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 6 11 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 4 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 10 client.50818.objecter _send_command 11 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.307+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f154801f8e0 con 0x7f1548018ad0 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f1548018ad0 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f154801f8e0 con 0x7f1548018ad0 2024-03-30T15:11:13.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 10 client.50818.objecter _finish_command 11 = system:0 2024-03-30T15:11:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 4 2024-03-30T15:11:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.308+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 6 11 2024-03-30T15:11:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 10 client.50818.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 12 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 12 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154800e330 osd=6 3 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 6 12 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 4 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 10 client.50818.objecter _send_command 12 2024-03-30T15:11:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.327+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f154801fac0 con 0x7f1548018ad0 2024-03-30T15:11:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.329+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f154801fac0 con 0x7f1548018ad0 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.329+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.329+0000 7f1564ff9700 10 client.50818.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.329+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154800e330 osd=6 4 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.329+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 6 12 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 10 client.50818.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 13 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 13 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 7 13 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 4 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 10 client.50818.objecter _send_command 13 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1548020d90 con 0x7f15480201e0 2024-03-30T15:11:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.330+0000 7f1564ff9700 10 client.50818.objecter ms_handle_connect 0x7f15480201e0 2024-03-30T15:11:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.331+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1548020d90 con 0x7f15480201e0 2024-03-30T15:11:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.331+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.331+0000 7f1564ff9700 10 client.50818.objecter _finish_command 13 = system:0 2024-03-30T15:11:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.331+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 4 2024-03-30T15:11:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.331+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 7 13 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 10 client.50818.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign -1 14 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 20 client.50818.objecter _get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 15 client.50818.objecter _session_command_op_remove -1 14 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 20 client.50818.objecter get_session s=0x7f154801f900 osd=7 3 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 15 client.50818.objecter _session_command_op_assign 7 14 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 4 2024-03-30T15:11:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 10 client.50818.objecter _send_command 14 2024-03-30T15:11:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.354+0000 7f156da76700 1 -- 172.21.15.28:0/3983017837 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f15480242b0 con 0x7f15480201e0 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.356+0000 7f1564ff9700 1 -- 172.21.15.28:0/3983017837 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f15480242b0 con 0x7f15480201e0 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.356+0000 7f1564ff9700 10 client.50818.objecter ms_dispatch 0x7f1568001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.356+0000 7f1564ff9700 10 client.50818.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.356+0000 7f1564ff9700 20 client.50818.objecter put_session s=0x7f154801f900 osd=7 4 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:13.356+0000 7f1564ff9700 15 client.50818.objecter _session_command_op_remove 7 14 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:13.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:13.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:13.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[41614]: pgmap v1180: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:13.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[41614]: osdmap e787: 8 total, 6 up, 8 in 2024-03-30T15:11:13.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3983017837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:13.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[29876]: pgmap v1180: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:13.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[29876]: osdmap e787: 8 total, 6 up, 8 in 2024-03-30T15:11:13.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3983017837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:13.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:13.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:13.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:13 smithi134 ceph-mon[33714]: pgmap v1180: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 804 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:13 smithi134 ceph-mon[33714]: osdmap e787: 8 total, 6 up, 8 in 2024-03-30T15:11:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3983017837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:13.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:13.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:13.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:13.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:14.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:14.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:14.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:14.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:14.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:14.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:14.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:14.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:14.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:14.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:14.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:14.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:14.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:14.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:14.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:14.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:15.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:15.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:15.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:15.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:15.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:15.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:15.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:15 smithi028 ceph-mon[41614]: pgmap v1182: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:15 smithi028 ceph-mon[29876]: pgmap v1182: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:15.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:15.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:15.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:15.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:15.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:15 smithi134 ceph-mon[33714]: pgmap v1182: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:15.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:15.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:15.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:15.937 INFO:tasks.thrashosds.thrasher:Moving pg 5.15 from osd.0 to osd.2 2024-03-30T15:11:15.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.15.0 2024-03-30T15:11:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:15.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:16.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:16.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:16.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:16.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:16.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:16.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:16.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:16.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:16 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:16.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:16 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:16.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:16 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:16.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:16 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:16.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:16.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:16.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:16.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:16 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:16.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:16 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:16.702 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:16.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:16.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:16.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:16.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:16.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:17.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:11:17.074 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:11:17.074 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:11:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:11:17.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:17.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:17.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:17.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:17.352 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T15:11:17.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:17.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:17 smithi028 ceph-mon[29876]: pgmap v1183: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:17.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:17 smithi028 ceph-mon[29876]: osdmap e788: 8 total, 6 up, 8 in 2024-03-30T15:11:17.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:17 smithi028 ceph-mon[41614]: pgmap v1183: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:17.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:17 smithi028 ceph-mon[41614]: osdmap e788: 8 total, 6 up, 8 in 2024-03-30T15:11:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:17.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:17.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:11:17.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:17 smithi134 ceph-mon[33714]: pgmap v1183: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:17.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:17 smithi134 ceph-mon[33714]: osdmap e788: 8 total, 6 up, 8 in 2024-03-30T15:11:17.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:17.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:17.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:17.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:11:17.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:17.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:17.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:18.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:18.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:18.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:18.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:18.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:18.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:18.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:18 smithi028 ceph-mon[29876]: osdmap e789: 8 total, 6 up, 8 in 2024-03-30T15:11:18.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:18 smithi028 ceph-mon[29876]: pgmap v1186: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:18.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:18 smithi028 ceph-mon[41614]: osdmap e789: 8 total, 6 up, 8 in 2024-03-30T15:11:18.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:18 smithi028 ceph-mon[41614]: pgmap v1186: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:18.525 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:18.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:18 smithi134 ceph-mon[33714]: osdmap e789: 8 total, 6 up, 8 in 2024-03-30T15:11:18.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:18 smithi134 ceph-mon[33714]: pgmap v1186: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:18.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:18.734 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:18.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:18.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:18.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:18.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:18.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:19.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:19.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:19.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:19.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:19.349 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:19.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:19.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.571+0000 7fd2f52ca700 1 -- 172.21.15.28:0/4175296648 >> v1:172.21.15.28:6790/0 conn(0x7fd2f010ea80 legacy=0x7fd2f0110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.571+0000 7fd2f52ca700 1 -- 172.21.15.28:0/4175296648 shutdown_connections 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.571+0000 7fd2f52ca700 1 -- 172.21.15.28:0/4175296648 >> 172.21.15.28:0/4175296648 conn(0x7fd2f00fdb20 msgr2=0x7fd2f00fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.571+0000 7fd2f52ca700 1 -- 172.21.15.28:0/4175296648 shutdown_connections 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.571+0000 7fd2f52ca700 1 -- 172.21.15.28:0/4175296648 wait complete. 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 librados: starting msgr at 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 librados: starting objecter 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 Processor -- start 2024-03-30T15:11:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 -- start start 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 librados: setting wanted keys 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 librados: calling monclient init 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2f0107aa0 con 0x7fd2f010ea80 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2f01a8650 con 0x7fd2f0102250 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2f52ca700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2f01a9830 con 0x7fd2f010aee0 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2ef7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd2f010ea80 0x7fd2f01a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51388/0 (socket says 172.21.15.28:51388) 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2ee7fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd2f010aee0 0x7fd2f0105000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52664/0 (socket says 172.21.15.28:52664) 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2ef7fe700 1 -- 172.21.15.28:0/2992320315 learned_addr learned my addr 172.21.15.28:0/2992320315 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter ms_handle_connect 0x7fd2f010ea80 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter ms_handle_connect 0x7fd2f010aee0 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter ms_handle_connect 0x7fd2f0102250 2024-03-30T15:11:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.572+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003962431 0 0) 0x7fd2f0107aa0 con 0x7fd2f010ea80 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2d80031f0 con 0x7fd2f010ea80 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260457561 0 0) 0x7fd2f01a9830 con 0x7fd2f010aee0 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2f0107aa0 con 0x7fd2f010aee0 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122929088 0 0) 0x7fd2f01a8650 con 0x7fd2f0102250 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2f01a9830 con 0x7fd2f0102250 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3416879949 0 0) 0x7fd2d80031f0 con 0x7fd2f010ea80 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2f01a8650 con 0x7fd2f010ea80 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd2e80047c0 con 0x7fd2f010ea80 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2740093907 0 0) 0x7fd2f0107aa0 con 0x7fd2f010aee0 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2d80031f0 con 0x7fd2f010aee0 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd2e4001850 con 0x7fd2f010aee0 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3033240281 0 0) 0x7fd2f01a9830 con 0x7fd2f0102250 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2f0107aa0 con 0x7fd2f0102250 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd2e0001830 con 0x7fd2f0102250 2024-03-30T15:11:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3845438425 0 0) 0x7fd2f01a8650 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 >> v1:172.21.15.28:6790/0 conn(0x7fd2f010aee0 legacy=0x7fd2f0105000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 >> v1:172.21.15.134:6789/0 conn(0x7fd2f0102250 legacy=0x7fd2f01048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f01aaa50 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01a9a60 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2f52ca700 10 client.25623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2f01a9f20 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2f52ca700 1 librados: init done 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd2e80013f0 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.573+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd2e80050c0 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.574+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2dc005150 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd2e8001de0 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(789..789 src has 442..789) ==== 9296+0+0 (unknown 1682857588 0 0) 0x7fd2e8089f50 con 0x7fd2f010ea80 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 10 client.25623.objecter ms_dispatch 0x7fd2f0001550 osd_map(789..789 src has 442..789) 2024-03-30T15:11:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 3 client.25623.objecter handle_osd_map got epochs [789,789] > 0 2024-03-30T15:11:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 3 client.25623.objecter handle_osd_map decoding full epoch 789 2024-03-30T15:11:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 20 client.25623.objecter dump_active .. 0 homeless 2024-03-30T15:11:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.575+0000 7fd2d7fff700 10 client.25623.objecter ms_handle_connect 0x7fd2d8070390 2024-03-30T15:11:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.580+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd2e8056cf0 con 0x7fd2f010ea80 2024-03-30T15:11:19.602 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:19.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:19.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:19.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.799+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd2dc005440 con 0x7fd2f010ea80 2024-03-30T15:11:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.851+0000 7fd2d7fff700 1 -- 172.21.15.28:0/2992320315 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v790) ==== 97+0+0 (unknown 2761101854 0 0) 0x7fd2e805a740 con 0x7fd2f010ea80 2024-03-30T15:11:19.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 10 librados: watch_flush enter 2024-03-30T15:11:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 10 librados: watch_flush exit 2024-03-30T15:11:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 20 client.25623.objecter shutdown clearing up homeless session... 2024-03-30T15:11:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 10 client.25623.objecter successfully canceled tick 2024-03-30T15:11:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd2d8070390 legacy=0x7fd2d8072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.854+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 >> v1:172.21.15.28:6789/0 conn(0x7fd2f010ea80 legacy=0x7fd2f01a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.856+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 shutdown_connections 2024-03-30T15:11:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.856+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 >> 172.21.15.28:0/2992320315 conn(0x7fd2f00fdb20 msgr2=0x7fd2f01116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.857+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 shutdown_connections 2024-03-30T15:11:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.858+0000 7fd2f52ca700 1 -- 172.21.15.28:0/2992320315 wait complete. 2024-03-30T15:11:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:19.858+0000 7fd2f52ca700 1 librados: shutdown 2024-03-30T15:11:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:11:19.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:19.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:19.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:20.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:20.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:20.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:11:20.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:20.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:20.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:20.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:20.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:20.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:20.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:20.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:20.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:20.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:20.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:20.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[29876]: osdmap e790: 8 total, 6 up, 8 in 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[29876]: pgmap v1188: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[41614]: osdmap e790: 8 total, 6 up, 8 in 2024-03-30T15:11:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:20 smithi028 ceph-mon[41614]: pgmap v1188: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:21.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:21.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:21.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:21.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:20 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2992320315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:20 smithi134 ceph-mon[33714]: osdmap e790: 8 total, 6 up, 8 in 2024-03-30T15:11:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:20 smithi134 ceph-mon[33714]: pgmap v1188: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:get_pg_num_history pg_num_history pg_num_history(e779 pg_nums {1={20=1,559=11},2={67=3},3={68=32},4={70=32},5={72=32},6={74=32},12={110=32},42={113=32},101={130=32},296={430=32},297={448=32},298={458=32},299={462=32},300={465=32},301={478=32},302={484=32},303={489=32},304={493=32},305={503=32},306={513=32},307={522=32},308={531=32},309={533=32},310={540=32},311={546=32},312={551=32},313={568=32},314={575=32},315={587=32},316={599=32},317={608=8},318={610=32},319={612=32},320={623=8},321={624=32},322={644=8},323={645=32},324={652=32},325={660=32},326={672=32},327={682=32},328={690=32},329={698=32},330={708=32},331={716=32},332={728=32},333={736=32},334={749=32},335={757=32},336={765=32},337={773=32}} deleted_pools 447,296,457,297,461,298,464,299,477,300,483,301,488,302,492,303,502,304,512,305,521,306,530,307,538,309,539,308,545,310,550,311,567,312,574,313,586,314,598,315,604,316,605,101,619,319,620,318,635,321,636,320,639,317,651,323,659,324,671,325,681,326,689,327,697,328,707,329,715,330,727,331,735,332,748,333,756,334,764,335,772,336) 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:Importing pgid 5.15 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:Write #5:a93a5511:::notify.2:head# 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:snapset 0=[]:{} 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:write_pg epoch 643 info 5.15( v 76'8 (0'0,76'8] local-lis/les=72/73 n=1 ec=72/72 lis/c=72/72 les/c/f=73/73/0 sis=72) 2024-03-30T15:11:21.209 INFO:teuthology.orchestra.run.smithi028.stdout:Import successful 2024-03-30T15:11:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:21.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:21.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:21.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:21.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:21.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:21.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:21.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:21.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:21.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:21.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:22.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:22.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:22.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:22.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:22.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:22.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:22.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:22.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:22.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:22.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:22.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:22.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:22.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:22.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:22.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:22.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:22.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:22.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:23.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:23.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: pgmap v1189: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: osdmap e791: 8 total, 6 up, 8 in 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: pgmap v1189: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: osdmap e791: 8 total, 6 up, 8 in 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:23.161 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:23.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: pgmap v1189: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: osdmap e791: 8 total, 6 up, 8 in 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:23.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:23.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:23.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:23.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:23.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:23.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:23.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.5.15.0 2024-03-30T15:11:23.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:23.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:23.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:23.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:23.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:23.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:23.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:23.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:23.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:24.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:24.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:24.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:24.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:24.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.337+0000 7fb227b1f700 1 -- 172.21.15.28:0/729357832 >> v1:172.21.15.28:6789/0 conn(0x7fb220109e80 legacy=0x7fb22010c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.337+0000 7fb227b1f700 1 -- 172.21.15.28:0/729357832 shutdown_connections 2024-03-30T15:11:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.337+0000 7fb227b1f700 1 -- 172.21.15.28:0/729357832 >> 172.21.15.28:0/729357832 conn(0x7fb2200fcaa0 msgr2=0x7fb2200feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.337+0000 7fb227b1f700 1 -- 172.21.15.28:0/729357832 shutdown_connections 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.337+0000 7fb227b1f700 1 -- 172.21.15.28:0/729357832 wait complete. 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 librados: starting msgr at 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 librados: starting objecter 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 Processor -- start 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 -- start start 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 librados: setting wanted keys 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.338+0000 7fb227b1f700 1 librados: calling monclient init 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb227b1f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb220106950 con 0x7fb220109e80 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb2258bb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb2201011f0 0x7fb220106170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52700/0 (socket says 172.21.15.28:52700) 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb2258bb700 1 -- 172.21.15.28:0/713334743 learned_addr learned my addr 172.21.15.28:0/713334743 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb2201ae8d0 con 0x7fb22010da20 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb2201afab0 con 0x7fb2201011f0 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter ms_handle_connect 0x7fb2201011f0 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter ms_handle_connect 0x7fb220109e80 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter ms_handle_connect 0x7fb22010da20 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667311777 0 0) 0x7fb220106950 con 0x7fb220109e80 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb208003aa0 con 0x7fb220109e80 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977276891 0 0) 0x7fb2201afab0 con 0x7fb2201011f0 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb220106950 con 0x7fb2201011f0 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360373310 0 0) 0x7fb2201ae8d0 con 0x7fb22010da20 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2201afab0 con 0x7fb22010da20 2024-03-30T15:11:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3528891296 0 0) 0x7fb208003aa0 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2201ae8d0 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb214001980 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2087651416 0 0) 0x7fb2201afab0 con 0x7fb22010da20 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb208003aa0 con 0x7fb22010da20 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3387808297 0 0) 0x7fb220106950 con 0x7fb2201011f0 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2201afab0 con 0x7fb2201011f0 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb210001ce0 con 0x7fb22010da20 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb218001940 con 0x7fb2201011f0 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2846860673 0 0) 0x7fb2201ae8d0 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 >> v1:172.21.15.28:6790/0 conn(0x7fb2201011f0 legacy=0x7fb220106170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.339+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 >> v1:172.21.15.134:6789/0 conn(0x7fb22010da20 legacy=0x7fb2201ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2201b0c90 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2201ad920 con 0x7fb220109e80 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb227b1f700 10 client.25632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2201adde0 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb227b1f700 1 librados: init done 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb214001c80 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.340+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb214004c50 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.341+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb20c005100 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.341+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb21401b2f0 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(791..791 src has 442..791) ==== 9296+0+0 (unknown 1150579231 0 0) 0x7fb214088750 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 osd_map(791..791 src has 442..791) 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 3 client.25632.objecter handle_osd_map got epochs [791,791] > 0 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 3 client.25632.objecter handle_osd_map decoding full epoch 791 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 20 client.25632.objecter dump_active .. 0 homeless 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v791) ==== 54+0+16 (unknown 3176943710 0 2420000192) 0x7fb21408abb0 con 0x7fb220109e80 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb2080703e0 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 10 client.25632.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 1 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 10 client.25632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=792}) -- 0x7fb20c0053e0 con 0x7fb220109e80 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fb20c000e30 con 0x7fb220109e80 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=791) ==== 24+0+0 (unknown 3559983423 0 0) 0x7fb214054bc0 con 0x7fb220109e80 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21ffff700 10 client.25632.objecter _check_command_map_dne tid 1 current 791 map_dne_bound 791 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21ffff700 10 client.25632.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb21ffff700 15 client.25632.objecter _session_command_op_remove -1 1 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 10 client.25632.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 2 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 2 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 1 2 2024-03-30T15:11:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 4 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 10 client.25632.objecter _send_command 2 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.342+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fb20c002330 con 0x7fb20c002790 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.343+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c002790 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.344+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c002330 con 0x7fb20c002790 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.344+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.344+0000 7fb21effd700 10 client.25632.objecter _finish_command 2 = system:0 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.344+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 4 2024-03-30T15:11:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.344+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 1 2 2024-03-30T15:11:24.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 10 client.25632.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 3 2024-03-30T15:11:24.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c001bb0 osd=1 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 1 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 4 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 10 client.25632.objecter _send_command 3 2024-03-30T15:11:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.362+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c004cc0 con 0x7fb20c002790 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c004cc0 con 0x7fb20c002790 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 10 client.25632.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c001bb0 osd=1 4 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 1 3 2024-03-30T15:11:24.366 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 10 client.25632.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 4 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 10 client.25632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fb20c001970 con 0x7fb220109e80 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mon_get_version_reply(handle=2 version=791) ==== 24+0+0 (unknown 520960092 0 0) 0x7fb214052f30 con 0x7fb220109e80 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21f7fe700 10 client.25632.objecter _check_command_map_dne tid 4 current 791 map_dne_bound 791 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21f7fe700 10 client.25632.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb21f7fe700 15 client.25632.objecter _session_command_op_remove -1 4 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 10 client.25632.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 5 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 5 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 3 5 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 4 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 10 client.25632.objecter _send_command 5 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.364+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb20c004920 con 0x7fb20c0043d0 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c0043d0 2024-03-30T15:11:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c004920 con 0x7fb20c0043d0 2024-03-30T15:11:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 10 client.25632.objecter _finish_command 5 = system:0 2024-03-30T15:11:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 4 2024-03-30T15:11:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.365+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 3 5 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 10 client.25632.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 6 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 6 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c003d50 osd=3 3 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 3 6 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 4 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 10 client.25632.objecter _send_command 6 2024-03-30T15:11:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.382+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c003830 con 0x7fb20c0043d0 2024-03-30T15:11:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c003830 con 0x7fb20c0043d0 2024-03-30T15:11:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb21effd700 10 client.25632.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c003d50 osd=3 4 2024-03-30T15:11:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 3 6 2024-03-30T15:11:24.387 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 10 client.25632.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 7 2024-03-30T15:11:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 7 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 4 7 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 4 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 10 client.25632.objecter _send_command 7 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.384+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb20c012050 con 0x7fb20c00e540 2024-03-30T15:11:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.385+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c00e540 2024-03-30T15:11:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.386+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c012050 con 0x7fb20c00e540 2024-03-30T15:11:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.386+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.386+0000 7fb21effd700 10 client.25632.objecter _finish_command 7 = system:0 2024-03-30T15:11:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.386+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 4 2024-03-30T15:11:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.386+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 4 7 2024-03-30T15:11:24.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 10 client.25632.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 8 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 8 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c0092e0 osd=4 3 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 4 8 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 4 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 10 client.25632.objecter _send_command 8 2024-03-30T15:11:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.402+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c008e20 con 0x7fb20c00e540 2024-03-30T15:11:24.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.404+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c008e20 con 0x7fb20c00e540 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.404+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.404+0000 7fb21effd700 10 client.25632.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.404+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c0092e0 osd=4 4 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.404+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 4 8 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:24.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 10 client.25632.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 9 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 9 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 5 9 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 4 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 10 client.25632.objecter _send_command 9 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb20c00e300 con 0x7fb20c013380 2024-03-30T15:11:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.405+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c013380 2024-03-30T15:11:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.406+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c00e300 con 0x7fb20c013380 2024-03-30T15:11:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.406+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.406+0000 7fb21effd700 10 client.25632.objecter _finish_command 9 = system:0 2024-03-30T15:11:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.406+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 4 2024-03-30T15:11:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.406+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 5 9 2024-03-30T15:11:24.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 10 client.25632.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 10 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 10 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c00db90 osd=5 3 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 5 10 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 4 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 10 client.25632.objecter _send_command 10 2024-03-30T15:11:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.423+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c00d650 con 0x7fb20c013380 2024-03-30T15:11:24.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c00d650 con 0x7fb20c013380 2024-03-30T15:11:24.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb21effd700 10 client.25632.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c00db90 osd=5 4 2024-03-30T15:11:24.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 5 10 2024-03-30T15:11:24.428 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:24.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 10 client.25632.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 11 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 11 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 6 11 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 4 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 10 client.25632.objecter _send_command 11 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.425+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb20c00d9a0 con 0x7fb20c012ee0 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c012ee0 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c00d9a0 con 0x7fb20c012ee0 2024-03-30T15:11:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 10 client.25632.objecter _finish_command 11 = system:0 2024-03-30T15:11:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 4 2024-03-30T15:11:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.426+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 6 11 2024-03-30T15:11:24.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 10 client.25632.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 12 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 12 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c00d680 osd=6 3 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 6 12 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 4 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 10 client.25632.objecter _send_command 12 2024-03-30T15:11:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.444+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c01cfe0 con 0x7fb20c012ee0 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.446+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c01cfe0 con 0x7fb20c012ee0 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.446+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.446+0000 7fb21effd700 10 client.25632.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.446+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c00d680 osd=6 4 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.446+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 6 12 2024-03-30T15:11:24.449 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 10 client.25632.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 13 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 13 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 7 13 2024-03-30T15:11:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 4 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 10 client.25632.objecter _send_command 13 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb20c01db60 con 0x7fb20c01d700 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.447+0000 7fb21effd700 10 client.25632.objecter ms_handle_connect 0x7fb20c01d700 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.448+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fb20c01db60 con 0x7fb20c01d700 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.448+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.448+0000 7fb21effd700 10 client.25632.objecter _finish_command 13 = system:0 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.448+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 4 2024-03-30T15:11:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.448+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 7 13 2024-03-30T15:11:24.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 10 client.25632.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign -1 14 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 20 client.25632.objecter _get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_remove -1 14 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 20 client.25632.objecter get_session s=0x7fb20c01cfe0 osd=7 3 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 15 client.25632.objecter _session_command_op_assign 7 14 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 4 2024-03-30T15:11:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 10 client.25632.objecter _send_command 14 2024-03-30T15:11:24.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.465+0000 7fb227b1f700 1 -- 172.21.15.28:0/713334743 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb20c01db60 con 0x7fb20c01d700 2024-03-30T15:11:24.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.466+0000 7fb21effd700 1 -- 172.21.15.28:0/713334743 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb20c01db60 con 0x7fb20c01d700 2024-03-30T15:11:24.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.466+0000 7fb21effd700 10 client.25632.objecter ms_dispatch 0x7fb220001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:24.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.466+0000 7fb21effd700 10 client.25632.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:24.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.467+0000 7fb21effd700 20 client.25632.objecter put_session s=0x7fb20c01cfe0 osd=7 4 2024-03-30T15:11:24.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:24.467+0000 7fb21effd700 15 client.25632.objecter _session_command_op_remove 7 14 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:24.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:24.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:24.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:24.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:24.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:24.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:24.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:24.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:25.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.085 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:25.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:25.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:25.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:25.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[41614]: pgmap v1191: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:25.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/713334743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:25.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:11:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[29876]: pgmap v1191: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/713334743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:25.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:25 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:11:25.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:25.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:25.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:25.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:25 smithi134 ceph-mon[33714]: pgmap v1191: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/713334743' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:25.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:25 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:11:25.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:25.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:25.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:25.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:25.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:25.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:25.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:26.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:26.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:26.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:26.317 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:26.317 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:26.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:26.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:26.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:26.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:26.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:26.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: pgmap v1192: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:11:26.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: pgmap v1192: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:11:26.614 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:26 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:26.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:26.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:26.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:26.910 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.910 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.910 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.910 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.910 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: pgmap v1192: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:26.911 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:11:26.911 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:11:26.911 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:26 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:26.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:11:26.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:26.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:26.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:27.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:27.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:27.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:27.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:27.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:27.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:27.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:27.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:27.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[41614]: osdmap e792: 8 total, 6 up, 8 in 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[41614]: osdmap e793: 8 total, 6 up, 8 in 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:27.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[29876]: osdmap e792: 8 total, 6 up, 8 in 2024-03-30T15:11:27.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:27 smithi028 ceph-mon[29876]: osdmap e793: 8 total, 6 up, 8 in 2024-03-30T15:11:27.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:27.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:27.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:27 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:27 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:27 smithi134 ceph-mon[33714]: osdmap e792: 8 total, 6 up, 8 in 2024-03-30T15:11:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:27 smithi134 ceph-mon[33714]: osdmap e793: 8 total, 6 up, 8 in 2024-03-30T15:11:27.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:27.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:28.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:28.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:28.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:28.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:28.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:28.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:28.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:28.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:28.500 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:28.542 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 2] live_osds: [1, 3, 4, 5, 6, 7] 2024-03-30T15:11:28.542 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:11:28.543 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-30T15:11:28.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 110 3.0 2024-03-30T15:11:28.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.589+0000 7fa8c6af3700 1 -- 172.21.15.28:0/2420205140 >> v1:172.21.15.28:6790/0 conn(0x7fa8c0105f00 legacy=0x7fa8c01062e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- 172.21.15.28:0/2420205140 shutdown_connections 2024-03-30T15:11:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- 172.21.15.28:0/2420205140 >> 172.21.15.28:0/2420205140 conn(0x7fa8c0077370 msgr2=0x7fa8c00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- 172.21.15.28:0/2420205140 shutdown_connections 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- 172.21.15.28:0/2420205140 wait complete. 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 librados: starting msgr at 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 librados: starting objecter 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 Processor -- start 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- start start 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 librados: setting wanted keys 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 librados: calling monclient init 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8c01a70c0 con 0x7fa8c0109d80 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8c01a82e0 con 0x7fa8c0105f00 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8c6af3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa8c01a9520 con 0x7fa8c010d950 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.590+0000 7fa8bffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa8c0109d80 0x7fa8c01a1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59756/0 (socket says 172.21.15.28:59756) 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bffff700 1 -- 172.21.15.28:0/164421778 learned_addr learned my addr 172.21.15.28:0/164421778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8c5090700 1 --1- 172.21.15.28:0/164421778 >> v1:172.21.15.28:6790/0 conn(0x7fa8c010d950 0x7fa8c01a5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:32870/0 (socket says 172.21.15.28:32870) 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter ms_handle_connect 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter ms_handle_connect 0x7fa8c010d950 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter ms_handle_connect 0x7fa8c0105f00 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163798669 0 0) 0x7fa8c01a70c0 con 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b00031f0 con 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837061774 0 0) 0x7fa8c01a9520 con 0x7fa8c010d950 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c01a70c0 con 0x7fa8c010d950 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030275259 0 0) 0x7fa8c01a82e0 con 0x7fa8c0105f00 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c01a9520 con 0x7fa8c0105f00 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1515344650 0 0) 0x7fa8b00031f0 con 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8c01a82e0 con 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa8ac001930 con 0x7fa8c0109d80 2024-03-30T15:11:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3304814942 0 0) 0x7fa8c01a70c0 con 0x7fa8c010d950 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8b00031f0 con 0x7fa8c010d950 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1424617064 0 0) 0x7fa8c01a9520 con 0x7fa8c0105f00 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8c01a70c0 con 0x7fa8c0105f00 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa8b8001b70 con 0x7fa8c010d950 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa8b4004610 con 0x7fa8c0105f00 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3194867284 0 0) 0x7fa8c01a82e0 con 0x7fa8c0109d80 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 >> v1:172.21.15.28:6790/0 conn(0x7fa8c010d950 legacy=0x7fa8c01a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 >> v1:172.21.15.134:6789/0 conn(0x7fa8c0105f00 legacy=0x7fa8c010d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c01aa760 con 0x7fa8c0109d80 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.591+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c01a9750 con 0x7fa8c0109d80 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.592+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa8ac003b40 con 0x7fa8c0109d80 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.592+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa8ac005d90 con 0x7fa8c0109d80 2024-03-30T15:11:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.592+0000 7fa8c6af3700 10 client.25638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.592+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8c01a9c10 con 0x7fa8c0109d80 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.592+0000 7fa8c6af3700 1 librados: init done 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c0102810 con 0x7fa8c0109d80 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa8ac0036a0 con 0x7fa8c0109d80 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(793..793 src has 442..793) ==== 9296+0+0 (unknown 2794632974 0 0) 0x7fa8ac089d40 con 0x7fa8c0109d80 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8bdffb700 10 client.25638.objecter ms_dispatch 0x7fa8c0001550 osd_map(793..793 src has 442..793) 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8bdffb700 3 client.25638.objecter handle_osd_map got epochs [793,793] > 0 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.593+0000 7fa8bdffb700 3 client.25638.objecter handle_osd_map decoding full epoch 793 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.594+0000 7fa8bdffb700 20 client.25638.objecter dump_active .. 0 homeless 2024-03-30T15:11:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.594+0000 7fa8bdffb700 10 client.25638.objecter ms_handle_connect 0x7fa8b0070500 2024-03-30T15:11:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.598+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa8ac056ae0 con 0x7fa8c0109d80 2024-03-30T15:11:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:28.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:28.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:28.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:28.820+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fa8c010c7f0 con 0x7fa8c0109d80 2024-03-30T15:11:28.823 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:28 smithi028 ceph-mon[41614]: pgmap v1195: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:28.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:28 smithi028 ceph-mon[29876]: pgmap v1195: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:28.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:28.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:28 smithi134 ceph-mon[33714]: pgmap v1195: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:28.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:28.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:28.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:29.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:29.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:29.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:29.329 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:29.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:29.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:29.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:11:29.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.503+0000 7fa8bdffb700 1 -- 172.21.15.28:0/164421778 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v794) ==== 91+0+0 (unknown 1561144003 0 0) 0x7fa8ac05a530 con 0x7fa8c0109d80 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 10 librados: watch_flush enter 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 10 librados: watch_flush exit 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 20 client.25638.objecter shutdown clearing up homeless session... 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 10 client.25638.objecter successfully canceled tick 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa8b0070500 legacy=0x7fa8b00729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 >> v1:172.21.15.28:6789/0 conn(0x7fa8c0109d80 legacy=0x7fa8c01a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 shutdown_connections 2024-03-30T15:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.507+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 >> 172.21.15.28:0/164421778 conn(0x7fa8c0077370 msgr2=0x7fa8c0106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.508+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 shutdown_connections 2024-03-30T15:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.508+0000 7fa8c6af3700 1 -- 172.21.15.28:0/164421778 wait complete. 2024-03-30T15:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:29.508+0000 7fa8c6af3700 1 librados: shutdown 2024-03-30T15:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:11:29.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:29.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:29.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:29.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:11:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:11:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:29.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:11:29.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:29.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:29.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:30.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:30.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:30.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:30.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:30.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:30.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:30.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:30.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[29876]: osdmap e794: 8 total, 6 up, 8 in 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[29876]: pgmap v1197: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[41614]: osdmap e794: 8 total, 6 up, 8 in 2024-03-30T15:11:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:30 smithi028 ceph-mon[41614]: pgmap v1197: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:30.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:30.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:30 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:30.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/164421778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:11:30.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:30 smithi134 ceph-mon[33714]: osdmap e794: 8 total, 6 up, 8 in 2024-03-30T15:11:30.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:30 smithi134 ceph-mon[33714]: pgmap v1197: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:30.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:30.884 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:30.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:30.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:31.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:31.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:31.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:31.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:31.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:31.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:31.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:31.398 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:31.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:31.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:31.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:31.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:31.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:31.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:11:31.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:31 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:11:31.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:31.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:31.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:31 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:31 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:31 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:11:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:31 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:11:31.936 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:32.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:32.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:32.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:32.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:32.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:32.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:32.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:11:32.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:32.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:32.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.405+0000 7f792fe68700 1 -- 172.21.15.28:0/700250909 >> v1:172.21.15.28:6789/0 conn(0x7f792810d9b0 legacy=0x7f792810fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.405+0000 7f792fe68700 1 -- 172.21.15.28:0/700250909 shutdown_connections 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.405+0000 7f792fe68700 1 -- 172.21.15.28:0/700250909 >> 172.21.15.28:0/700250909 conn(0x7f79280fca30 msgr2=0x7f79280fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.405+0000 7f792fe68700 1 -- 172.21.15.28:0/700250909 shutdown_connections 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.405+0000 7f792fe68700 1 -- 172.21.15.28:0/700250909 wait complete. 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 librados: starting msgr at 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 librados: starting objecter 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 Processor -- start 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 -- start start 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 librados: setting wanted keys 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 librados: calling monclient init 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79281a9350 con 0x7f7928109e10 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79281aa570 con 0x7f7928101180 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792fe68700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79281ab7b0 con 0x7f792810d9b0 2024-03-30T15:11:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792d403700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7928109e10 0x7f79281a4220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59774/0 (socket says 172.21.15.28:59774) 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.406+0000 7f792d403700 1 -- 172.21.15.28:0/1712197072 learned_addr learned my addr 172.21.15.28:0/1712197072 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter ms_handle_connect 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter ms_handle_connect 0x7f7928101180 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter ms_handle_connect 0x7f7928109e10 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630321172 0 0) 0x7f79281ab7b0 con 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79100035b0 con 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 883218033 0 0) 0x7f79281aa570 con 0x7f7928101180 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79281ab7b0 con 0x7f7928101180 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4018609648 0 0) 0x7f79100035b0 con 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79281aa570 con 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f79180048c0 con 0x7f792810d9b0 2024-03-30T15:11:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 446484779 0 0) 0x7f79281ab7b0 con 0x7f7928101180 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79100035b0 con 0x7f7928101180 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7920001870 con 0x7f7928101180 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048365056 0 0) 0x7f79281a9350 con 0x7f7928109e10 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79281ab7b0 con 0x7f7928109e10 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1012382989 0 0) 0x7f79281aa570 con 0x7f792810d9b0 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 >> v1:172.21.15.134:6789/0 conn(0x7f7928101180 legacy=0x7f79281063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 >> v1:172.21.15.28:6789/0 conn(0x7f7928109e10 legacy=0x7f79281a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79281ac9f0 con 0x7f792810d9b0 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79281aa7a0 con 0x7f792810d9b0 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.407+0000 7f792fe68700 10 client.58847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.408+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f79281aac60 con 0x7f792810d9b0 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.408+0000 7f792fe68700 1 librados: init done 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.408+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7918001ef0 con 0x7f792810d9b0 2024-03-30T15:11:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.408+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f79180050d0 con 0x7f792810d9b0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.408+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7914005120 con 0x7f792810d9b0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.409+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7918003a00 con 0x7f792810d9b0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(795..795 src has 442..795) ==== 9296+0+0 (unknown 2252905317 0 0) 0x7f7918088f80 con 0x7f792810d9b0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 10 client.58847.objecter ms_dispatch 0x7f7928001550 osd_map(795..795 src has 442..795) 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 3 client.58847.objecter handle_osd_map got epochs [795,795] > 0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 3 client.58847.objecter handle_osd_map decoding full epoch 795 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 20 client.58847.objecter dump_active .. 0 homeless 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.410+0000 7f7926ffd700 10 client.58847.objecter ms_handle_connect 0x7f79100704f0 2024-03-30T15:11:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.414+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7918055d20 con 0x7f792810d9b0 2024-03-30T15:11:32.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:32.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[41614]: osdmap e795: 8 total, 6 up, 8 in 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[41614]: 42.6 scrub starts 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[41614]: 42.6 scrub ok 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[41614]: pgmap v1199: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[29876]: osdmap e795: 8 total, 6 up, 8 in 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[29876]: 42.6 scrub starts 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[29876]: 42.6 scrub ok 2024-03-30T15:11:32.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:32 smithi028 ceph-mon[29876]: pgmap v1199: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:32.640+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) -- 0x7f7914000b90 con 0x7f79100704f0 2024-03-30T15:11:32.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:32.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:32.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:32.870 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:32.870 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:32.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:32 smithi134 ceph-mon[33714]: osdmap e795: 8 total, 6 up, 8 in 2024-03-30T15:11:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:32 smithi134 ceph-mon[33714]: 42.6 scrub starts 2024-03-30T15:11:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:32 smithi134 ceph-mon[33714]: 42.6 scrub ok 2024-03-30T15:11:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:32 smithi134 ceph-mon[33714]: pgmap v1199: 246 pgs: 1 active+clean+remapped, 106 active+clean, 20 undersized+peered, 89 active+undersized, 2 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 127/678 objects degraded (18.732%) 2024-03-30T15:11:32.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:32.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:33.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:33.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:33.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:33.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:33.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:33.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:33.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[41614]: from='client.58847 v1:172.21.15.28:0/1712197072' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:33.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[29876]: from='client.58847 v1:172.21.15.28:0/1712197072' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:11:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:33 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.528+0000 7f7926ffd700 1 -- 172.21.15.28:0/1712197072 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+392 (unknown 0 0 4106373266) 0x7f7914000b90 con 0x7f79100704f0 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:moved 35 / 738 (4.74255%) 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:avg 123 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 27.8927 -> 27.8927 (expected baseline 10.1242) 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.6 with 83 -> 96 pgs (0.674797 -> 0.780488 * mean) 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 104 -> 91 pgs (0.845528 -> 0.739837 * mean) 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:11:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:oload 110 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 3 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0011 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0012 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 weight 1.0000 -> 0.8383 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 weight 1.0000 -> 0.8491 2024-03-30T15:11:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 weight 1.0000 -> 0.8548 2024-03-30T15:11:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 10 librados: watch_flush enter 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 10 librados: watch_flush exit 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 20 client.58847.objecter shutdown clearing up homeless session... 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 10 client.58847.objecter successfully canceled tick 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 >> v1:172.21.15.28:6800/3011431093 conn(0x7f79100704f0 legacy=0x7f79100729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 >> v1:172.21.15.28:6790/0 conn(0x7f792810d9b0 legacy=0x7f79281a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 shutdown_connections 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 >> 172.21.15.28:0/1712197072 conn(0x7f79280fca30 msgr2=0x7f7928110b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 shutdown_connections 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.533+0000 7f792fe68700 1 -- 172.21.15.28:0/1712197072 wait complete. 2024-03-30T15:11:33.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:33.534+0000 7f792fe68700 1 librados: shutdown 2024-03-30T15:11:33.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:33.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:33.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:11:33.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:33.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:33.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:33 smithi134 ceph-mon[33714]: from='client.58847 v1:172.21.15.28:0/1712197072' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:33 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:33 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]: dispatch 2024-03-30T15:11:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:33 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:11:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:33 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:11:33.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:33.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:33.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:34.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:34.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:34.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:34.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:34.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:34.390 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-30T15:11:34.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 110 0.05 2024-03-30T15:11:34.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:34.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:34.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:34.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]': finished 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: osdmap e796: 8 total, 6 up, 8 in 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[41614]: pgmap v1201: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]': finished 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: osdmap e796: 8 total, 6 up, 8 in 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:11:34.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:11:34.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:34 smithi028 ceph-mon[29876]: pgmap v1201: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:34.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:34.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:34.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:34.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:34.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'56022','3':'55645','4':'54937'}"}]': finished 2024-03-30T15:11:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: osdmap e796: 8 total, 6 up, 8 in 2024-03-30T15:11:34.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:11:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:11:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:11:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:11:34.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:34 smithi134 ceph-mon[33714]: pgmap v1201: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:34.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:34.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:34.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:35.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:35.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:35.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:35.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:35.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:35.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:35.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:35.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:35.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:35.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:35.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:11:35.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 13 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[41614]: osdmap e797: 8 total, 6 up, 8 in 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 13 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:35.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:35 smithi028 ceph-mon[29876]: osdmap e797: 8 total, 6 up, 8 in 2024-03-30T15:11:35.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:35.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:35.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:35.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:35 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:11:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:35 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:11:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:35 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 13 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:35 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:35 smithi134 ceph-mon[33714]: osdmap e797: 8 total, 6 up, 8 in 2024-03-30T15:11:35.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:35.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:36.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 -- 172.21.15.28:0/1067403343 >> v1:172.21.15.28:6789/0 conn(0x7f868410ea70 legacy=0x7f8684110f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 -- 172.21.15.28:0/1067403343 shutdown_connections 2024-03-30T15:11:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 -- 172.21.15.28:0/1067403343 >> 172.21.15.28:0/1067403343 conn(0x7f86840fdaf0 msgr2=0x7f86840fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 -- 172.21.15.28:0/1067403343 shutdown_connections 2024-03-30T15:11:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 -- 172.21.15.28:0/1067403343 wait complete. 2024-03-30T15:11:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 librados: starting msgr at 2024-03-30T15:11:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.108+0000 7f868b71f700 1 librados: starting objecter 2024-03-30T15:11:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 Processor -- start 2024-03-30T15:11:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 -- start start 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 librados: setting wanted keys 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 librados: calling monclient init 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86841a70f0 con 0x7f8684102240 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86841a8310 con 0x7f868410aed0 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f868b71f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f86841a9550 con 0x7f868410ea70 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f8689cbc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f868410ea70 0x7f86841a57b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:32898/0 (socket says 172.21.15.28:32898) 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f8689cbc700 1 -- 172.21.15.28:0/2553550517 learned_addr learned my addr 172.21.15.28:0/2553550517 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter ms_handle_connect 0x7f868410ea70 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter ms_handle_connect 0x7f8684102240 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106485557 0 0) 0x7f86841a9550 con 0x7f868410ea70 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86700031f0 con 0x7f868410ea70 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter ms_handle_connect 0x7f868410aed0 2024-03-30T15:11:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783687114 0 0) 0x7f86841a70f0 con 0x7f8684102240 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86841a9550 con 0x7f8684102240 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269167367 0 0) 0x7f86841a8310 con 0x7f868410aed0 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.109+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86841a70f0 con 0x7f868410aed0 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2840321516 0 0) 0x7f86700031f0 con 0x7f868410ea70 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86841a8310 con 0x7f868410ea70 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f867c0048d0 con 0x7f868410ea70 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1402920052 0 0) 0x7f86841a9550 con 0x7f8684102240 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86700031f0 con 0x7f8684102240 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f86800017f0 con 0x7f8684102240 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3244880511 0 0) 0x7f86841a8310 con 0x7f868410ea70 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 >> v1:172.21.15.134:6789/0 conn(0x7f868410aed0 legacy=0x7f86841a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 >> v1:172.21.15.28:6789/0 conn(0x7f8684102240 legacy=0x7f8684107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86841aa790 con 0x7f868410ea70 2024-03-30T15:11:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86841a7320 con 0x7f868410ea70 2024-03-30T15:11:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f868b71f700 10 client.58853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f86841a77e0 con 0x7f868410ea70 2024-03-30T15:11:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f868b71f700 1 librados: init done 2024-03-30T15:11:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.110+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f867c001e60 con 0x7f868410ea70 2024-03-30T15:11:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.111+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f867c0051f0 con 0x7f868410ea70 2024-03-30T15:11:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.111+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8668005100 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.112+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f867c005430 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(798..798 src has 442..798) ==== 10167+0+0 (unknown 4178561925 0 0) 0x7f867c08a4e0 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 osd_map(798..798 src has 442..798) 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 3 client.58853.objecter handle_osd_map got epochs [798,798] > 0 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 3 client.58853.objecter handle_osd_map decoding full epoch 798 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 20 client.58853.objecter dump_active .. 0 homeless 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v798) ==== 54+0+16 (unknown 3772535743 0 2420000192) 0x7f867c08c910 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 10 client.58853.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 1 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 20 client.58853.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 10 client.58853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=799}) -- 0x7f86680053e0 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8668000e30 con 0x7f868410ea70 2024-03-30T15:11:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8670070330 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=798) ==== 24+0+0 (unknown 2950179814 0 0) 0x7f867c0565d0 con 0x7f868410ea70 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867b7fe700 10 client.58853.objecter _check_command_map_dne tid 1 current 798 map_dne_bound 798 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867b7fe700 10 client.58853.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f867b7fe700 15 client.58853.objecter _session_command_op_remove -1 1 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.113+0000 7f868b71f700 10 client.58853.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 2 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 2 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 1 2 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 4 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 10 client.58853.objecter _send_command 2 2024-03-30T15:11:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f8668002360 con 0x7f8668002790 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.114+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8668002790 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.115+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8668002360 con 0x7f8668002790 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.115+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.115+0000 7f867a7fc700 10 client.58853.objecter _finish_command 2 = system:0 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.115+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 4 2024-03-30T15:11:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.115+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 1 2 2024-03-30T15:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 10 client.58853.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 3 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 3 2024-03-30T15:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668001bb0 osd=1 3 2024-03-30T15:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 1 3 2024-03-30T15:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 4 2024-03-30T15:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 10 client.58853.objecter _send_command 3 2024-03-30T15:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.138+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8668004cf0 con 0x7f8668002790 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8668004cf0 con 0x7f8668002790 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 10 client.58853.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668001bb0 osd=1 4 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 1 3 2024-03-30T15:11:36.141 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 10 client.58853.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 4 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 10 client.58853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f8668001970 con 0x7f868410ea70 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=798) ==== 24+0+0 (unknown 1692670597 0 0) 0x7f867c054940 con 0x7f868410ea70 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867affd700 10 client.58853.objecter _check_command_map_dne tid 4 current 798 map_dne_bound 798 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867affd700 10 client.58853.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f867affd700 15 client.58853.objecter _session_command_op_remove -1 4 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 10 client.58853.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 5 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 5 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 3 5 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 4 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 10 client.58853.objecter _send_command 5 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.140+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f86680049a0 con 0x7f8668004400 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8668004400 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f86680049a0 con 0x7f8668004400 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 10 client.58853.objecter _finish_command 5 = system:0 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 4 2024-03-30T15:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.141+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 3 5 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 10 client.58853.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 6 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 6 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668003d80 osd=3 3 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 3 6 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 4 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 10 client.58853.objecter _send_command 6 2024-03-30T15:11:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.158+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86680038c0 con 0x7f8668004400 2024-03-30T15:11:36.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f86680038c0 con 0x7f8668004400 2024-03-30T15:11:36.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f867a7fc700 10 client.58853.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668003d80 osd=3 4 2024-03-30T15:11:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 3 6 2024-03-30T15:11:36.162 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:36.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 10 client.58853.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 7 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 7 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 4 7 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 4 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 10 client.58853.objecter _send_command 7 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.160+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8668012050 con 0x7f866800e540 2024-03-30T15:11:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.161+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f866800e540 2024-03-30T15:11:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.162+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8668012050 con 0x7f866800e540 2024-03-30T15:11:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.162+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.162+0000 7f867a7fc700 10 client.58853.objecter _finish_command 7 = system:0 2024-03-30T15:11:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.162+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 4 2024-03-30T15:11:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.162+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 4 7 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 10 client.58853.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 8 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 8 2024-03-30T15:11:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668003b50 osd=4 3 2024-03-30T15:11:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 4 8 2024-03-30T15:11:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 4 2024-03-30T15:11:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 10 client.58853.objecter _send_command 8 2024-03-30T15:11:36.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.179+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8668009070 con 0x7f866800e540 2024-03-30T15:11:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8668009070 con 0x7f866800e540 2024-03-30T15:11:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f867a7fc700 10 client.58853.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668003b50 osd=4 4 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 4 8 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 10 client.58853.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 9 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 9 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 5 9 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 4 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 10 client.58853.objecter _send_command 9 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.181+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f866801b2a0 con 0x7f8668017790 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.182+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8668017790 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.183+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f866801b2a0 con 0x7f8668017790 2024-03-30T15:11:36.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.183+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.183+0000 7f867a7fc700 10 client.58853.objecter _finish_command 9 = system:0 2024-03-30T15:11:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.183+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 4 2024-03-30T15:11:36.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.183+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 5 9 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.199+0000 7f868b71f700 10 client.58853.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.199+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 10 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.199+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 10 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f8668009310 osd=5 3 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 5 10 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 4 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 10 client.58853.objecter _send_command 10 2024-03-30T15:11:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.200+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f866800d8c0 con 0x7f8668017790 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.201+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f866800d8c0 con 0x7f8668017790 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.201+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.201+0000 7f867a7fc700 10 client.58853.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.201+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f8668009310 osd=5 4 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.201+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 5 10 2024-03-30T15:11:36.203 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 10 client.58853.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 11 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 11 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 6 11 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 4 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 10 client.58853.objecter _send_command 11 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f866800db50 con 0x7f8668012ee0 2024-03-30T15:11:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.202+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8668012ee0 2024-03-30T15:11:36.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.203+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f866800db50 con 0x7f8668012ee0 2024-03-30T15:11:36.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.203+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:36.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.203+0000 7f867a7fc700 10 client.58853.objecter _finish_command 11 = system:0 2024-03-30T15:11:36.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.203+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 4 2024-03-30T15:11:36.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.203+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 6 11 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 10 client.58853.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 12 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 12 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f866800d8c0 osd=6 3 2024-03-30T15:11:36.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 6 12 2024-03-30T15:11:36.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 4 2024-03-30T15:11:36.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 10 client.58853.objecter _send_command 12 2024-03-30T15:11:36.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.220+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f866801fd00 con 0x7f8668012ee0 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f866801fd00 con 0x7f8668012ee0 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 10 client.58853.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f866800d8c0 osd=6 4 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 6 12 2024-03-30T15:11:36.223 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 10 client.58853.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 13 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 13 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 7 13 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 4 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 10 client.58853.objecter _send_command 13 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8668023fd0 con 0x7f8668020240 2024-03-30T15:11:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.222+0000 7f867a7fc700 10 client.58853.objecter ms_handle_connect 0x7f8668020240 2024-03-30T15:11:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.223+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8668023fd0 con 0x7f8668020240 2024-03-30T15:11:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.223+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.223+0000 7f867a7fc700 10 client.58853.objecter _finish_command 13 = system:0 2024-03-30T15:11:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.223+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 4 2024-03-30T15:11:36.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.223+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 7 13 2024-03-30T15:11:36.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 10 client.58853.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign -1 14 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 20 client.58853.objecter _get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 15 client.58853.objecter _session_command_op_remove -1 14 2024-03-30T15:11:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 20 client.58853.objecter get_session s=0x7f866800db70 osd=7 3 2024-03-30T15:11:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 15 client.58853.objecter _session_command_op_assign 7 14 2024-03-30T15:11:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 4 2024-03-30T15:11:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 10 client.58853.objecter _send_command 14 2024-03-30T15:11:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.240+0000 7f868b71f700 1 -- 172.21.15.28:0/2553550517 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86680241b0 con 0x7f8668020240 2024-03-30T15:11:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.242+0000 7f867a7fc700 1 -- 172.21.15.28:0/2553550517 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f86680241b0 con 0x7f8668020240 2024-03-30T15:11:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.242+0000 7f867a7fc700 10 client.58853.objecter ms_dispatch 0x7f8684001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.242+0000 7f867a7fc700 10 client.58853.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.242+0000 7f867a7fc700 20 client.58853.objecter put_session s=0x7f866800db70 osd=7 4 2024-03-30T15:11:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:36.242+0000 7f867a7fc700 15 client.58853.objecter _session_command_op_remove 7 14 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:36.255 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:36.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:36.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:36.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:36.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:36.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:36 smithi134 ceph-mon[33714]: osdmap e798: 8 total, 6 up, 8 in 2024-03-30T15:11:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:36 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T15:11:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:36 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T15:11:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2553550517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:36.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:36 smithi134 ceph-mon[33714]: pgmap v1204: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 442 B/s rd, 0 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:36.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:36.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[41614]: osdmap e798: 8 total, 6 up, 8 in 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2553550517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[41614]: pgmap v1204: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 442 B/s rd, 0 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[29876]: osdmap e798: 8 total, 6 up, 8 in 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2553550517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:36.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:36 smithi028 ceph-mon[29876]: pgmap v1204: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 442 B/s rd, 0 op/s; 106/678 objects degraded (15.634%) 2024-03-30T15:11:36.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:36.888 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:36.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:36.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:37.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:37.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:37.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:37.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:37.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:37.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:37.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:37.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:37.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:37.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:37.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.767 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:37.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:37.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[41614]: osdmap e799: 8 total, 6 up, 8 in 2024-03-30T15:11:37.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T15:11:37.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T15:11:37.816 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:37.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[29876]: osdmap e799: 8 total, 6 up, 8 in 2024-03-30T15:11:37.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T15:11:37.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T15:11:37.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:37.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:37.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:37.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:37.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:37 smithi134 ceph-mon[33714]: osdmap e799: 8 total, 6 up, 8 in 2024-03-30T15:11:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:37 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T15:11:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:37 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T15:11:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:37.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:38.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:38.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:38.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:38.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:38.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:38.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:38.387 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:38.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:38.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:38.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:38.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:11:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:11:38.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: osdmap e800: 8 total, 6 up, 8 in 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 12.3 scrub starts 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: 12.3 scrub ok 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[29876]: pgmap v1207: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 106/678 objects degraded (15.634%) 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: osdmap e800: 8 total, 6 up, 8 in 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 12.3 scrub starts 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: 12.3 scrub ok 2024-03-30T15:11:38.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:38 smithi028 ceph-mon[41614]: pgmap v1207: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 106/678 objects degraded (15.634%) 2024-03-30T15:11:38.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:38.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:38.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:38.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:38.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:11:38.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:11:38.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:11:38.933 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:11:38.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: osdmap e800: 8 total, 6 up, 8 in 2024-03-30T15:11:38.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 12.3 scrub starts 2024-03-30T15:11:38.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: 12.3 scrub ok 2024-03-30T15:11:38.934 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:38 smithi134 ceph-mon[33714]: pgmap v1207: 246 pgs: 1 active+clean+remapped, 11 unknown, 31 peering, 91 active+clean, 15 undersized+peered, 74 active+undersized, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 106/678 objects degraded (15.634%) 2024-03-30T15:11:38.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:38.995 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:11:38.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:11:38.995 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:11:38.995 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:11:39.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:39.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:39.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:39.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:39.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:39.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:39.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:39.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:39.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:39.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:39.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 4.1 scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 4.1 scrub ok 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 42.e scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[41614]: 42.e scrub ok 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 4.1 scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 4.1 scrub ok 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 42.e scrub starts 2024-03-30T15:11:39.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:39 smithi028 ceph-mon[29876]: 42.e scrub ok 2024-03-30T15:11:39.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:11:39.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:39.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:11:39.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:39.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:39.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 4.1 scrub starts 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 4.1 scrub ok 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 42.e scrub starts 2024-03-30T15:11:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:39 smithi134 ceph-mon[33714]: 42.e scrub ok 2024-03-30T15:11:40.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:40.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:40.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:40.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:40.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:40.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:40.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:40.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:40.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:40.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:40.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:11:40.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:11:40.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[29876]: pgmap v1208: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 87 B/s, 4 objects/s recovering 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T15:11:40.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:40 smithi028 ceph-mon[41614]: pgmap v1208: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 87 B/s, 4 objects/s recovering 2024-03-30T15:11:40.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:40.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:40.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:40.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:40.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:40.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:11:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:11:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:11:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:11:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T15:11:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T15:11:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:40 smithi134 ceph-mon[33714]: pgmap v1208: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 87 B/s, 4 objects/s recovering 2024-03-30T15:11:41.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:41.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:41.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:41.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:41.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- 172.21.15.28:0/3369706359 >> v1:172.21.15.28:6789/0 conn(0x7f2b4810afa0 legacy=0x7f2b4810d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- 172.21.15.28:0/3369706359 shutdown_connections 2024-03-30T15:11:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- 172.21.15.28:0/3369706359 >> 172.21.15.28:0/3369706359 conn(0x7f2b480fc9e0 msgr2=0x7f2b480fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- 172.21.15.28:0/3369706359 shutdown_connections 2024-03-30T15:11:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- 172.21.15.28:0/3369706359 wait complete. 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 librados: starting msgr at 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 librados: starting objecter 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 Processor -- start 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 -- start start 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 librados: setting wanted keys 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.327+0000 7f2b50189700 1 librados: calling monclient init 2024-03-30T15:11:41.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b50189700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b481a9390 con 0x7f2b4810afa0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b50189700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b481aa5b0 con 0x7f2b4810eab0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b50189700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b481ab7f0 con 0x7f2b48103190 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b4d724700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2b4810afa0 0x7f2b481a42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37424/0 (socket says 172.21.15.28:37424) 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b4d724700 1 -- 172.21.15.28:0/11085643 learned_addr learned my addr 172.21.15.28:0/11085643 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter ms_handle_connect 0x7f2b48103190 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter ms_handle_connect 0x7f2b4810eab0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter ms_handle_connect 0x7f2b4810afa0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270185598 0 0) 0x7f2b481aa5b0 con 0x7f2b4810eab0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b34003aa0 con 0x7f2b4810eab0 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 432014643 0 0) 0x7f2b481ab7f0 con 0x7f2b48103190 2024-03-30T15:11:41.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b34003f40 con 0x7f2b48103190 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78724281 0 0) 0x7f2b481a9390 con 0x7f2b4810afa0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b481ab7f0 con 0x7f2b4810afa0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3867930910 0 0) 0x7f2b34003aa0 con 0x7f2b4810eab0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b481a9390 con 0x7f2b4810eab0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b40001b70 con 0x7f2b4810eab0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3170183690 0 0) 0x7f2b34003f40 con 0x7f2b48103190 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b34003aa0 con 0x7f2b48103190 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b440013d0 con 0x7f2b48103190 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2578456787 0 0) 0x7f2b481ab7f0 con 0x7f2b4810afa0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.328+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b34003f40 con 0x7f2b4810afa0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b380044e0 con 0x7f2b4810afa0 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3007052394 0 0) 0x7f2b34003aa0 con 0x7f2b48103190 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 >> v1:172.21.15.134:6789/0 conn(0x7f2b4810eab0 legacy=0x7f2b481a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 >> v1:172.21.15.28:6789/0 conn(0x7f2b4810afa0 legacy=0x7f2b481a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:41.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b481aca30 con 0x7f2b48103190 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b481a95c0 con 0x7f2b48103190 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b50189700 10 client.58859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b481a9a80 con 0x7f2b48103190 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b50189700 1 librados: init done 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2b44001d30 con 0x7f2b48103190 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.329+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2b44004970 con 0x7f2b48103190 2024-03-30T15:11:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.330+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4806ad30 con 0x7f2b48103190 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.330+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2b44001570 con 0x7f2b48103190 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.331+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(800..800 src has 442..800) ==== 10138+0+0 (unknown 2815062243 0 0) 0x7f2b44088cb0 con 0x7f2b48103190 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.331+0000 7f2b3effd700 10 client.58859.objecter ms_dispatch 0x7f2b48001550 osd_map(800..800 src has 442..800) 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.331+0000 7f2b3effd700 3 client.58859.objecter handle_osd_map got epochs [800,800] > 0 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.331+0000 7f2b3effd700 3 client.58859.objecter handle_osd_map decoding full epoch 800 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.332+0000 7f2b3effd700 20 client.58859.objecter dump_active .. 0 homeless 2024-03-30T15:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.332+0000 7f2b3effd700 10 client.58859.objecter ms_handle_connect 0x7f2b340707a0 2024-03-30T15:11:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.337+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2b44055700 con 0x7f2b48103190 2024-03-30T15:11:41.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:41.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:41.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.552+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f2b48102320 con 0x7f2b340707a0 2024-03-30T15:11:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.571+0000 7f2b3effd700 1 -- 172.21.15.28:0/11085643 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+366 (unknown 0 0 1308603553) 0x7f2b48102320 con 0x7f2b340707a0 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:moved 6 / 738 (0.813008%) 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:avg 123 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 27.8927 -> 28.0595 (expected baseline 10.1242) 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.4 with 87 -> 88 pgs (0.707317 -> 0.715447 * mean) 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.7 with 104 -> 106 pgs (0.845528 -> 0.861789 * mean) 2024-03-30T15:11:41.574 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:oload 110 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0011 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0012 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 weight 0.8491 -> 0.7991 2024-03-30T15:11:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 weight 0.8548 -> 0.8048 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 10 librados: watch_flush enter 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 10 librados: watch_flush exit 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 20 client.58859.objecter shutdown clearing up homeless session... 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 10 client.58859.objecter successfully canceled tick 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2b340707a0 legacy=0x7f2b34072c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 >> v1:172.21.15.28:6790/0 conn(0x7f2b48103190 legacy=0x7f2b4810a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 shutdown_connections 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 >> 172.21.15.28:0/11085643 conn(0x7f2b480fc9e0 msgr2=0x7f2b480fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 shutdown_connections 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 -- 172.21.15.28:0/11085643 wait complete. 2024-03-30T15:11:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:41.575+0000 7f2b50189700 1 librados: shutdown 2024-03-30T15:11:41.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:41.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:41.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:41.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:41.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:41 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:41.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:41.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:41.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:41 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:41 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:41 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]: dispatch 2024-03-30T15:11:42.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:42.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:42.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:42.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:42.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:42.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:42.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:42.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:42.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:42.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:42.754 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:42.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:42.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:42.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:42.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: from='client.58859 v1:172.21.15.28:0/11085643' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]': finished 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: osdmap e801: 8 total, 6 up, 8 in 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: pgmap v1210: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 81 B/s, 3 objects/s recovering 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:42 smithi134 ceph-mon[33714]: osdmap e802: 8 total, 6 up, 8 in 2024-03-30T15:11:42.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: from='client.58859 v1:172.21.15.28:0/11085643' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]': finished 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: osdmap e801: 8 total, 6 up, 8 in 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: pgmap v1210: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 81 B/s, 3 objects/s recovering 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[29876]: osdmap e802: 8 total, 6 up, 8 in 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: from='client.58859 v1:172.21.15.28:0/11085643' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'52746','3':'52369'}"}]': finished 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: osdmap e801: 8 total, 6 up, 8 in 2024-03-30T15:11:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:11:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:11:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: pgmap v1210: 246 pgs: 23 active+clean+remapped, 103 active+clean, 15 undersized+peered, 78 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 21 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 106/678 objects degraded (15.634%); 29/678 objects misplaced (4.277%); 81 B/s, 3 objects/s recovering 2024-03-30T15:11:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 106/678 objects degraded (15.634%), 23 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:42 smithi028 ceph-mon[41614]: osdmap e802: 8 total, 6 up, 8 in 2024-03-30T15:11:43.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.102+0000 7fe58bbe1700 1 -- 172.21.15.28:0/2424282266 >> v1:172.21.15.28:6790/0 conn(0x7fe5841011f0 legacy=0x7fe5841015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.102+0000 7fe58bbe1700 1 -- 172.21.15.28:0/2424282266 shutdown_connections 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.102+0000 7fe58bbe1700 1 -- 172.21.15.28:0/2424282266 >> 172.21.15.28:0/2424282266 conn(0x7fe5840fcaa0 msgr2=0x7fe5840feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.102+0000 7fe58bbe1700 1 -- 172.21.15.28:0/2424282266 shutdown_connections 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.102+0000 7fe58bbe1700 1 -- 172.21.15.28:0/2424282266 wait complete. 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 librados: starting msgr at 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 librados: starting objecter 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 Processor -- start 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 -- start start 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 librados: setting wanted keys 2024-03-30T15:11:43.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 librados: calling monclient init 2024-03-30T15:11:43.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5841a9490 con 0x7fe584109e80 2024-03-30T15:11:43.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5841aa6b0 con 0x7fe58410da20 2024-03-30T15:11:43.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58bbe1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5841ab8f0 con 0x7fe5841011f0 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58997d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe5841011f0 0x7fe58410f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41804/0 (socket says 172.21.15.28:41804) 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe58997d700 1 -- 172.21.15.28:0/226328135 learned_addr learned my addr 172.21.15.28:0/226328135 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter ms_handle_connect 0x7fe5841011f0 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter ms_handle_connect 0x7fe584109e80 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter ms_handle_connect 0x7fe58410da20 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3521025720 0 0) 0x7fe5841a9490 con 0x7fe584109e80 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe56c0031f0 con 0x7fe584109e80 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988642828 0 0) 0x7fe5841ab8f0 con 0x7fe5841011f0 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5841a9490 con 0x7fe5841011f0 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2888224321 0 0) 0x7fe5841aa6b0 con 0x7fe58410da20 2024-03-30T15:11:43.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.103+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5841ab8f0 con 0x7fe58410da20 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2447483544 0 0) 0x7fe56c0031f0 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5841aa6b0 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe5780017b0 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3591289690 0 0) 0x7fe5841a9490 con 0x7fe5841011f0 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe56c0031f0 con 0x7fe5841011f0 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 361456945 0 0) 0x7fe5841ab8f0 con 0x7fe58410da20 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5841a9490 con 0x7fe58410da20 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe57c0017b0 con 0x7fe5841011f0 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe574001850 con 0x7fe58410da20 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2879913703 0 0) 0x7fe5841aa6b0 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 >> v1:172.21.15.28:6790/0 conn(0x7fe5841011f0 legacy=0x7fe58410f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 >> v1:172.21.15.134:6789/0 conn(0x7fe58410da20 legacy=0x7fe5841a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5841acb30 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841a96c0 con 0x7fe584109e80 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe58bbe1700 10 client.25662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5841a9b80 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe58bbe1700 1 librados: init done 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe578003b40 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.104+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe578004aa0 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.105+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe570005120 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe5780036a0 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(802..802 src has 442..802) ==== 10266+0+0 (unknown 1328701820 0 0) 0x7fe578088bb0 con 0x7fe584109e80 2024-03-30T15:11:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 10 client.25662.objecter ms_dispatch 0x7fe584001550 osd_map(802..802 src has 442..802) 2024-03-30T15:11:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 3 client.25662.objecter handle_osd_map got epochs [802,802] > 0 2024-03-30T15:11:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 3 client.25662.objecter handle_osd_map decoding full epoch 802 2024-03-30T15:11:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 20 client.25662.objecter dump_active .. 0 homeless 2024-03-30T15:11:43.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.106+0000 7fe582ffd700 10 client.25662.objecter ms_handle_connect 0x7fe56c070420 2024-03-30T15:11:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.111+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe578055580 con 0x7fe584109e80 2024-03-30T15:11:43.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:43.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:43.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.332+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fe570005410 con 0x7fe584109e80 2024-03-30T15:11:43.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:43.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:43.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.412+0000 7fe582ffd700 1 -- 172.21.15.28:0/226328135 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v803) ==== 101+0+0 (unknown 355519235 0 0) 0x7fe578058fd0 con 0x7fe584109e80 2024-03-30T15:11:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.415+0000 7fe58bbe1700 10 librados: watch_flush enter 2024-03-30T15:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 10 librados: watch_flush exit 2024-03-30T15:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 20 client.25662.objecter shutdown clearing up homeless session... 2024-03-30T15:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 10 client.25662.objecter successfully canceled tick 2024-03-30T15:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe56c070420 legacy=0x7fe56c0728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 >> v1:172.21.15.28:6789/0 conn(0x7fe584109e80 legacy=0x7fe5841a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.416+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 shutdown_connections 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.417+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 >> 172.21.15.28:0/226328135 conn(0x7fe5840fcaa0 msgr2=0x7fe58406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.417+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 shutdown_connections 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.417+0000 7fe58bbe1700 1 -- 172.21.15.28:0/226328135 wait complete. 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:43.417+0000 7fe58bbe1700 1 librados: shutdown 2024-03-30T15:11:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:11:43.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:43.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:43.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:43.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:43.721 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-30T15:11:43.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 110 1.0 2024-03-30T15:11:43.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:43.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:43.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:43.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:43.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T15:11:43.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T15:11:43.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:43.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:43.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[29876]: osdmap e803: 8 total, 6 up, 8 in 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:43.920 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:43 smithi028 ceph-mon[41614]: osdmap e803: 8 total, 6 up, 8 in 2024-03-30T15:11:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T15:11:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T15:11:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:11:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:11:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/226328135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:11:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:43 smithi134 ceph-mon[33714]: osdmap e803: 8 total, 6 up, 8 in 2024-03-30T15:11:44.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:44.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:44.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:44.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:44.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:44.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:44.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:44.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:44.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:44.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:44.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:44.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:44 smithi134 ceph-mon[33714]: 5.16 scrub starts 2024-03-30T15:11:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:44 smithi134 ceph-mon[33714]: 5.16 scrub ok 2024-03-30T15:11:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:44 smithi134 ceph-mon[33714]: pgmap v1213: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 121 B/s, 5 objects/s recovering 2024-03-30T15:11:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:44 smithi134 ceph-mon[33714]: osdmap e804: 8 total, 6 up, 8 in 2024-03-30T15:11:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:44.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:44.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[29876]: 5.16 scrub starts 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[29876]: 5.16 scrub ok 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[29876]: pgmap v1213: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 121 B/s, 5 objects/s recovering 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[29876]: osdmap e804: 8 total, 6 up, 8 in 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[41614]: 5.16 scrub starts 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[41614]: 5.16 scrub ok 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[41614]: pgmap v1213: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 121 B/s, 5 objects/s recovering 2024-03-30T15:11:44.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:44 smithi028 ceph-mon[41614]: osdmap e804: 8 total, 6 up, 8 in 2024-03-30T15:11:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:45.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:45.060 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:45.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:45.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:45.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:45.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:45.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:45.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:45.575 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:45.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:45.702 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:45.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:45.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[29876]: osdmap e805: 8 total, 6 up, 8 in 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T15:11:45.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T15:11:45.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T15:11:45.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T15:11:45.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:45 smithi028 ceph-mon[41614]: osdmap e805: 8 total, 6 up, 8 in 2024-03-30T15:11:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:45 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T15:11:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:45 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T15:11:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:45 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T15:11:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:45 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T15:11:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:45 smithi134 ceph-mon[33714]: osdmap e805: 8 total, 6 up, 8 in 2024-03-30T15:11:45.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:45.957 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:45.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:46.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:46.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:46.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:46.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:46.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:46.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:46.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:46.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:46.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:46.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[29876]: 322.6s0 deep-scrub starts 2024-03-30T15:11:46.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[29876]: 322.6 deep-scrub ok 2024-03-30T15:11:46.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:11:46.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:11:46.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[29876]: pgmap v1216: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 59 B/s, 3 objects/s recovering 2024-03-30T15:11:46.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[41614]: 322.6s0 deep-scrub starts 2024-03-30T15:11:46.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[41614]: 322.6 deep-scrub ok 2024-03-30T15:11:46.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:11:46.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:11:46.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:46 smithi028 ceph-mon[41614]: pgmap v1216: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 59 B/s, 3 objects/s recovering 2024-03-30T15:11:46.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:46.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:46.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:46.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:46 smithi134 ceph-mon[33714]: 322.6s0 deep-scrub starts 2024-03-30T15:11:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:46 smithi134 ceph-mon[33714]: 322.6 deep-scrub ok 2024-03-30T15:11:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:46 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:11:46.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:46 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:11:46.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:46 smithi134 ceph-mon[33714]: pgmap v1216: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 59 B/s, 3 objects/s recovering 2024-03-30T15:11:46.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:46.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:46.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:47.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:11:47.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:47.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:47.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:47.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.355+0000 7f890005d700 1 -- 172.21.15.28:0/3194612301 >> v1:172.21.15.28:6790/0 conn(0x7f88f8102240 legacy=0x7f88f8102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- 172.21.15.28:0/3194612301 shutdown_connections 2024-03-30T15:11:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- 172.21.15.28:0/3194612301 >> 172.21.15.28:0/3194612301 conn(0x7f88f80fdaf0 msgr2=0x7f88f80fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- 172.21.15.28:0/3194612301 shutdown_connections 2024-03-30T15:11:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- 172.21.15.28:0/3194612301 wait complete. 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 librados: starting msgr at 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 librados: starting objecter 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 Processor -- start 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- start start 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 librados: setting wanted keys 2024-03-30T15:11:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 librados: calling monclient init 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88f8076040 con 0x7f88f810aed0 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88f8076250 con 0x7f88f8102240 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.356+0000 7f890005d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f88f8076460 con 0x7f88f810ea70 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88fd5f8700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f88f810aed0 0x7f88f8074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37460/0 (socket says 172.21.15.28:37460) 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88fd5f8700 1 -- 172.21.15.28:0/3690462475 learned_addr learned my addr 172.21.15.28:0/3690462475 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter ms_handle_connect 0x7f88f810ea70 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter ms_handle_connect 0x7f88f810aed0 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter ms_handle_connect 0x7f88f8102240 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260116552 0 0) 0x7f88f8076460 con 0x7f88f810ea70 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88e4003aa0 con 0x7f88f810ea70 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47007841 0 0) 0x7f88f8076040 con 0x7f88f810aed0 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f8076460 con 0x7f88f810aed0 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213663906 0 0) 0x7f88f8076250 con 0x7f88f8102240 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f8076040 con 0x7f88f8102240 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1147799273 0 0) 0x7f88e4003aa0 con 0x7f88f810ea70 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f8076250 con 0x7f88f810ea70 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88f0001590 con 0x7f88f810ea70 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1898361139 0 0) 0x7f88f8076460 con 0x7f88f810aed0 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88e4003aa0 con 0x7f88f810aed0 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2173429776 0 0) 0x7f88f8076040 con 0x7f88f8102240 2024-03-30T15:11:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f8076460 con 0x7f88f8102240 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88e80017b0 con 0x7f88f810aed0 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88f4001430 con 0x7f88f8102240 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4053348427 0 0) 0x7f88f8076250 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 >> v1:172.21.15.134:6789/0 conn(0x7f88f8102240 legacy=0x7f88f8074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 >> v1:172.21.15.28:6789/0 conn(0x7f88f810aed0 legacy=0x7f88f8074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f81aca90 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.357+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88f81a9a80 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f890005d700 10 client.58871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88f81a9f40 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f890005d700 1 librados: init done 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f88f0003ea0 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f88f0004d40 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.358+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f88dc005100 con 0x7f88f810ea70 2024-03-30T15:11:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.359+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f88f0003a00 con 0x7f88f810ea70 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.359+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(805..805 src has 442..805) ==== 10237+0+0 (unknown 214304967 0 0) 0x7f88f0088f50 con 0x7f88f810ea70 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.359+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 osd_map(805..805 src has 442..805) 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.359+0000 7f88eeffd700 3 client.58871.objecter handle_osd_map got epochs [805,805] > 0 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.359+0000 7f88eeffd700 3 client.58871.objecter handle_osd_map decoding full epoch 805 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88eeffd700 20 client.58871.objecter dump_active .. 0 homeless 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v805) ==== 54+0+16 (unknown 1577514061 0 2420000192) 0x7f88f008b3b0 con 0x7f88f810ea70 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88e4070520 2024-03-30T15:11:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 10 client.58871.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 1 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 20 client.58871.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 10 client.58871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=806}) -- 0x7f88dc0053e0 con 0x7f88f810ea70 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f88dc000e30 con 0x7f88f810ea70 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=805) ==== 24+0+0 (unknown 263134707 0 0) 0x7f88f0055010 con 0x7f88f810ea70 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88effff700 10 client.58871.objecter _check_command_map_dne tid 1 current 805 map_dne_bound 805 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88effff700 10 client.58871.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f88effff700 15 client.58871.objecter _session_command_op_remove -1 1 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 10 client.58871.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.360+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 2 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 2 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 1 2 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 4 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 10 client.58871.objecter _send_command 2 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f88dc0022f0 con 0x7f88dc002790 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.361+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc002790 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.362+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc0022f0 con 0x7f88dc002790 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.362+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.362+0000 7f88eeffd700 10 client.58871.objecter _finish_command 2 = system:0 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.362+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 4 2024-03-30T15:11:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.362+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 1 2 2024-03-30T15:11:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 10 client.58871.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc001bb0 osd=1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 1 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 4 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 10 client.58871.objecter _send_command 3 2024-03-30T15:11:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.379+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc004c70 con 0x7f88dc002790 2024-03-30T15:11:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc004c70 con 0x7f88dc002790 2024-03-30T15:11:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 10 client.58871.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc001bb0 osd=1 4 2024-03-30T15:11:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 1 3 2024-03-30T15:11:47.383 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f890005d700 10 client.58871.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 4 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f890005d700 20 client.58871.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f890005d700 10 client.58871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f88dc004f00 con 0x7f88f810ea70 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=805) ==== 24+0+0 (unknown 3297979536 0 0) 0x7f88f0053380 con 0x7f88f810ea70 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88ef7fe700 10 client.58871.objecter _check_command_map_dne tid 4 current 805 map_dne_bound 805 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88ef7fe700 10 client.58871.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.381+0000 7f88ef7fe700 15 client.58871.objecter _session_command_op_remove -1 4 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 10 client.58871.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 5 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 5 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 3 5 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 4 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 10 client.58871.objecter _send_command 5 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f88dc0048a0 con 0x7f88dc004240 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc004240 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc0048a0 con 0x7f88dc004240 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 10 client.58871.objecter _finish_command 5 = system:0 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 4 2024-03-30T15:11:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.382+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 3 5 2024-03-30T15:11:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 10 client.58871.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 6 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 6 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc001a00 osd=3 3 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 3 6 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 4 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 10 client.58871.objecter _send_command 6 2024-03-30T15:11:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.400+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc003780 con 0x7f88dc004240 2024-03-30T15:11:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc003780 con 0x7f88dc004240 2024-03-30T15:11:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f88eeffd700 10 client.58871.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc001a00 osd=3 4 2024-03-30T15:11:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 3 6 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 10 client.58871.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 7 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 7 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 4 7 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 4 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 10 client.58871.objecter _send_command 7 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.402+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f88dc009aa0 con 0x7f88dc00e540 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.403+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc00e540 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.404+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc009aa0 con 0x7f88dc00e540 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.404+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.404+0000 7f88eeffd700 10 client.58871.objecter _finish_command 7 = system:0 2024-03-30T15:11:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.404+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 4 2024-03-30T15:11:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.404+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 4 7 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 10 client.58871.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 8 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 8 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0092e0 osd=4 3 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 4 8 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 4 2024-03-30T15:11:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 10 client.58871.objecter _send_command 8 2024-03-30T15:11:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.422+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc008e80 con 0x7f88dc00e540 2024-03-30T15:11:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc008e80 con 0x7f88dc00e540 2024-03-30T15:11:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 10 client.58871.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0092e0 osd=4 4 2024-03-30T15:11:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 4 8 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 10 client.58871.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 9 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 9 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 5 9 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 4 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 10 client.58871.objecter _send_command 9 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f88dc012ba0 con 0x7f88dc012050 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.425+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc012050 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.426+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc012ba0 con 0x7f88dc012050 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.426+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.426+0000 7f88eeffd700 10 client.58871.objecter _finish_command 9 = system:0 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.426+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 4 2024-03-30T15:11:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.426+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 5 9 2024-03-30T15:11:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 10 client.58871.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 10 2024-03-30T15:11:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 10 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0090e0 osd=5 3 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 5 10 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 4 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 10 client.58871.objecter _send_command 10 2024-03-30T15:11:47.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.444+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc013520 con 0x7f88dc012050 2024-03-30T15:11:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc013520 con 0x7f88dc012050 2024-03-30T15:11:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f88eeffd700 10 client.58871.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0090e0 osd=5 4 2024-03-30T15:11:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 5 10 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 10 client.58871.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 11 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 11 2024-03-30T15:11:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 6 11 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 4 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 10 client.58871.objecter _send_command 11 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.447+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f88dc01b540 con 0x7f88dc017a30 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc017a30 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc01b540 con 0x7f88dc017a30 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 10 client.58871.objecter _finish_command 11 = system:0 2024-03-30T15:11:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 4 2024-03-30T15:11:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.448+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 6 11 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 10 client.58871.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 12 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 12 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc0146a0 osd=6 3 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 6 12 2024-03-30T15:11:47.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 4 2024-03-30T15:11:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 10 client.58871.objecter _send_command 12 2024-03-30T15:11:47.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.466+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc01d8e0 con 0x7f88dc017a30 2024-03-30T15:11:47.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc01d8e0 con 0x7f88dc017a30 2024-03-30T15:11:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 10 client.58871.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc0146a0 osd=6 4 2024-03-30T15:11:47.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 6 12 2024-03-30T15:11:47.470 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 10 client.58871.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 13 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 13 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 7 13 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 4 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 10 client.58871.objecter _send_command 13 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f88dc01cc60 con 0x7f88dc01d690 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.468+0000 7f88eeffd700 10 client.58871.objecter ms_handle_connect 0x7f88dc01d690 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:11:47.471 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.469+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f88dc01cc60 con 0x7f88dc01d690 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.469+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.469+0000 7f88eeffd700 10 client.58871.objecter _finish_command 13 = system:0 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.469+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 4 2024-03-30T15:11:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.469+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 7 13 2024-03-30T15:11:47.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 10 client.58871.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign -1 14 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 20 client.58871.objecter _get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 15 client.58871.objecter _session_command_op_remove -1 14 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 20 client.58871.objecter get_session s=0x7f88dc01dd40 osd=7 3 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 15 client.58871.objecter _session_command_op_assign 7 14 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 4 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 10 client.58871.objecter _send_command 14 2024-03-30T15:11:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.486+0000 7f890005d700 1 -- 172.21.15.28:0/3690462475 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f88dc01cc60 con 0x7f88dc01d690 2024-03-30T15:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.489+0000 7f88eeffd700 1 -- 172.21.15.28:0/3690462475 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f88dc01cc60 con 0x7f88dc01d690 2024-03-30T15:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.489+0000 7f88eeffd700 10 client.58871.objecter ms_dispatch 0x7f88f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.489+0000 7f88eeffd700 10 client.58871.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.489+0000 7f88eeffd700 20 client.58871.objecter put_session s=0x7f88dc01dd40 osd=7 4 2024-03-30T15:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:47.489+0000 7f88eeffd700 15 client.58871.objecter _session_command_op_remove 7 14 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:47.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:47.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:47.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:47.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:11:47.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3690462475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 17 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 98/678 objects degraded (14.454%), 22 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:11:47.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3690462475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 17 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:47.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:47 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 98/678 objects degraded (14.454%), 22 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:47.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:47.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:47.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:11:47.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:11:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T15:11:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T15:11:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3690462475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 17 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:47.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:47 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 98/678 objects degraded (14.454%), 22 pgs degraded, 107 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:47.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:48.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:48.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:48.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:48.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:48.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:11:48.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:48.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:48.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:48.590 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:48.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:48.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:48.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:48.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:48.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:48.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:48.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:48 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:11:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:48 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:11:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:48 smithi134 ceph-mon[33714]: 6.7 deep-scrub starts 2024-03-30T15:11:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:48 smithi134 ceph-mon[33714]: 6.7 deep-scrub ok 2024-03-30T15:11:48.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:48 smithi134 ceph-mon[33714]: pgmap v1217: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 893 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 41 B/s, 2 objects/s recovering 2024-03-30T15:11:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[29876]: 6.7 deep-scrub starts 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[29876]: 6.7 deep-scrub ok 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[29876]: pgmap v1217: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 893 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 41 B/s, 2 objects/s recovering 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[41614]: 6.7 deep-scrub starts 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[41614]: 6.7 deep-scrub ok 2024-03-30T15:11:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:48 smithi028 ceph-mon[41614]: pgmap v1217: 246 pgs: 26 active+clean+remapped, 104 active+clean, 15 undersized+peered, 74 active+undersized, 5 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 893 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 41 B/s, 2 objects/s recovering 2024-03-30T15:11:48.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:49.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:49.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:49.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:49.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:49.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:49.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:49.508 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:49.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:49.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:49.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:49.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:49.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[41614]: 6.2 deep-scrub starts 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[41614]: 6.2 deep-scrub ok 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[41614]: 1.5 scrub starts 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[41614]: 1.5 scrub ok 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[29876]: 6.2 deep-scrub starts 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[29876]: 6.2 deep-scrub ok 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[29876]: 1.5 scrub starts 2024-03-30T15:11:49.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:49 smithi028 ceph-mon[29876]: 1.5 scrub ok 2024-03-30T15:11:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:49 smithi134 ceph-mon[33714]: 6.2 deep-scrub starts 2024-03-30T15:11:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:49 smithi134 ceph-mon[33714]: 6.2 deep-scrub ok 2024-03-30T15:11:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:49 smithi134 ceph-mon[33714]: 1.5 scrub starts 2024-03-30T15:11:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:49 smithi134 ceph-mon[33714]: 1.5 scrub ok 2024-03-30T15:11:49.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:50.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:50.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:50.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:50.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 -- 172.21.15.28:0/2453665969 >> v1:172.21.15.28:6790/0 conn(0x7fb75410d960 legacy=0x7fb75410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 -- 172.21.15.28:0/2453665969 shutdown_connections 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 -- 172.21.15.28:0/2453665969 >> 172.21.15.28:0/2453665969 conn(0x7fb7540fc9a0 msgr2=0x7fb7540fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 -- 172.21.15.28:0/2453665969 shutdown_connections 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 -- 172.21.15.28:0/2453665969 wait complete. 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 librados: starting msgr at 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 librados: starting objecter 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.363+0000 7fb758b1e700 1 Processor -- start 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 -- start start 2024-03-30T15:11:50.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 librados: setting wanted keys 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 librados: calling monclient init 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb754106780 con 0x7fb754109dc0 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb754106990 con 0x7fb754101130 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb758b1e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7541a8540 con 0x7fb75410d960 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb752d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb75410d960 0x7fb7541a4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49824/0 (socket says 172.21.15.28:49824) 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb751d9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb754109dc0 0x7fb754103ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48602/0 (socket says 172.21.15.28:48602) 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb752d9d700 1 -- 172.21.15.28:0/3521400361 learned_addr learned my addr 172.21.15.28:0/3521400361 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter ms_handle_connect 0x7fb754109dc0 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter ms_handle_connect 0x7fb75410d960 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter ms_handle_connect 0x7fb754101130 2024-03-30T15:11:50.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175764109 0 0) 0x7fb754106780 con 0x7fb754109dc0 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb73c0031f0 con 0x7fb754109dc0 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1301131407 0 0) 0x7fb7541a8540 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb754106780 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399733899 0 0) 0x7fb754106990 con 0x7fb754101130 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7541a8540 con 0x7fb754101130 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2925791624 0 0) 0x7fb754106780 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb754106990 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.364+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb74c004830 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3633231140 0 0) 0x7fb73c0031f0 con 0x7fb754109dc0 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb754106780 con 0x7fb754109dc0 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb748001830 con 0x7fb754109dc0 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2939967786 0 0) 0x7fb754106990 con 0x7fb75410d960 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 >> v1:172.21.15.134:6789/0 conn(0x7fb754101130 legacy=0x7fb7541035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:50.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 >> v1:172.21.15.28:6789/0 conn(0x7fb754109dc0 legacy=0x7fb754103ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7541a9720 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7541a8770 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb758b1e700 10 client.58877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7541a8c30 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb758b1e700 1 librados: init done 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb74c001e70 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.365+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb74c0050f0 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.366+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb734005120 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.366+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb74c0052b0 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(805..805 src has 442..805) ==== 10237+0+0 (unknown 214304967 0 0) 0x7fb74c08a330 con 0x7fb75410d960 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 10 client.58877.objecter ms_dispatch 0x7fb754001550 osd_map(805..805 src has 442..805) 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 3 client.58877.objecter handle_osd_map got epochs [805,805] > 0 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 3 client.58877.objecter handle_osd_map decoding full epoch 805 2024-03-30T15:11:50.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 20 client.58877.objecter dump_active .. 0 homeless 2024-03-30T15:11:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.367+0000 7fb7437fe700 10 client.58877.objecter ms_handle_connect 0x7fb73c0704d0 2024-03-30T15:11:50.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:50.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.371+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb74c056d20 con 0x7fb75410d960 2024-03-30T15:11:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:50.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:50.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:50.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:50.590+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) -- 0x7fb734000b90 con 0x7fb73c0704d0 2024-03-30T15:11:50.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:50.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:50.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:50.780 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:50.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:50.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:50.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:11:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T15:11:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T15:11:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-mon[33714]: 6.e deep-scrub starts 2024-03-30T15:11:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-mon[33714]: 6.e deep-scrub ok 2024-03-30T15:11:50.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:50 smithi134 ceph-mon[33714]: pgmap v1218: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 35 B/s, 1 objects/s recovering 2024-03-30T15:11:50.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T15:11:50.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T15:11:50.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[29876]: 6.e deep-scrub starts 2024-03-30T15:11:50.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[29876]: 6.e deep-scrub ok 2024-03-30T15:11:50.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[29876]: pgmap v1218: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 35 B/s, 1 objects/s recovering 2024-03-30T15:11:50.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T15:11:50.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T15:11:50.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[41614]: 6.e deep-scrub starts 2024-03-30T15:11:50.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[41614]: 6.e deep-scrub ok 2024-03-30T15:11:50.969 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:50 smithi028 ceph-mon[41614]: pgmap v1218: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%); 35 B/s, 1 objects/s recovering 2024-03-30T15:11:51.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:51.035 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:51.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:51.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:11:51.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:51.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:51.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:51.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:51.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:51.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.598+0000 7fb7437fe700 1 -- 172.21.15.28:0/3521400361 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+362 (unknown 0 0 3660571064) 0x7fb734000b90 con 0x7fb73c0704d0 2024-03-30T15:11:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:moved 9 / 738 (1.21951%) 2024-03-30T15:11:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:avg 123 2024-03-30T15:11:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 28.0595 -> 27.3038 (expected baseline 10.1242) 2024-03-30T15:11:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 85 -> 78 pgs (0.691057 -> 0.634146 * mean) 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.7 with 106 -> 112 pgs (0.861789 -> 0.910569 * mean) 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:oload 110 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 1 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0011 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0012 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 weight 0.7991 -> 0.6649 2024-03-30T15:11:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 weight 0.8048 -> 0.6751 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.601+0000 7fb758b1e700 10 librados: watch_flush enter 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.601+0000 7fb758b1e700 10 librados: watch_flush exit 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.601+0000 7fb758b1e700 20 client.58877.objecter shutdown clearing up homeless session... 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.601+0000 7fb758b1e700 10 client.58877.objecter successfully canceled tick 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.602+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb73c0704d0 legacy=0x7fb73c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.602+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 >> v1:172.21.15.28:6790/0 conn(0x7fb75410d960 legacy=0x7fb7541a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.602+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 shutdown_connections 2024-03-30T15:11:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.602+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 >> 172.21.15.28:0/3521400361 conn(0x7fb7540fc9a0 msgr2=0x7fb754110e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.603+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 shutdown_connections 2024-03-30T15:11:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.603+0000 7fb758b1e700 1 -- 172.21.15.28:0/3521400361 wait complete. 2024-03-30T15:11:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:51.603+0000 7fb758b1e700 1 librados: shutdown 2024-03-30T15:11:51.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: 12.12 deep-scrub starts 2024-03-30T15:11:51.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: 12.12 deep-scrub ok 2024-03-30T15:11:51.618 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: from='client.58877 v1:172.21.15.28:0/3521400361' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]': finished 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[41614]: osdmap e806: 8 total, 6 up, 8 in 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: 12.12 deep-scrub starts 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: 12.12 deep-scrub ok 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: from='client.58877 v1:172.21.15.28:0/3521400361' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]': finished 2024-03-30T15:11:51.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:51 smithi028 ceph-mon[29876]: osdmap e806: 8 total, 6 up, 8 in 2024-03-30T15:11:51.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:51.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:51.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:51.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:51.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:51.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:51.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: 12.12 deep-scrub starts 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: 12.12 deep-scrub ok 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: from='client.58877 v1:172.21.15.28:0/3521400361' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]: dispatch 2024-03-30T15:11:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:11:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'44246','3':'43576'}"}]': finished 2024-03-30T15:11:51.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:51 smithi134 ceph-mon[33714]: osdmap e806: 8 total, 6 up, 8 in 2024-03-30T15:11:52.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:52.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:52.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:52.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:52.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:52.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:52.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:52.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:52.539 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:52.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:52.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:52.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:52.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:52.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: 42.b scrub starts 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: 42.b scrub ok 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: 1.4 scrub starts 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: 1.4 scrub ok 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: pgmap v1220: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 662 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%) 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: osdmap e807: 8 total, 6 up, 8 in 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:52 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:52.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: 42.b scrub starts 2024-03-30T15:11:52.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: 42.b scrub ok 2024-03-30T15:11:52.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: 1.4 scrub starts 2024-03-30T15:11:52.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: 1.4 scrub ok 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: pgmap v1220: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 662 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%) 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: osdmap e807: 8 total, 6 up, 8 in 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: 42.b scrub starts 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: 42.b scrub ok 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: 1.4 scrub starts 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: 1.4 scrub ok 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: pgmap v1220: 246 pgs: 26 active+clean+remapped, 105 active+clean, 15 undersized+peered, 74 active+undersized, 4 active+undersized+remapped, 2 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 662 B/s rd, 0 op/s; 98/678 objects degraded (14.454%); 37/678 objects misplaced (5.457%) 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: osdmap e807: 8 total, 6 up, 8 in 2024-03-30T15:11:52.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:11:52.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:52 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:11:52.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:52.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:53.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:53.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:53.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:53.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:53.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:53.350 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-30T15:11:53.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 1000 0.05 2024-03-30T15:11:53.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:53.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:53.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:53.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:53.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:53.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:53.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:11:53.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:11:53.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[29876]: osdmap e808: 8 total, 6 up, 8 in 2024-03-30T15:11:53.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:11:53.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:11:53.887 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:53 smithi028 ceph-mon[41614]: osdmap e808: 8 total, 6 up, 8 in 2024-03-30T15:11:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:53 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:11:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:53 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:11:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:53 smithi134 ceph-mon[33714]: osdmap e808: 8 total, 6 up, 8 in 2024-03-30T15:11:53.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:53.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:53.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:54.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.990+0000 7f6d2719b700 1 -- 172.21.15.28:0/1240346912 >> v1:172.21.15.28:6789/0 conn(0x7f6d20102250 legacy=0x7f6d20102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.990+0000 7f6d2719b700 1 -- 172.21.15.28:0/1240346912 shutdown_connections 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.990+0000 7f6d2719b700 1 -- 172.21.15.28:0/1240346912 >> 172.21.15.28:0/1240346912 conn(0x7f6d200fdb20 msgr2=0x7f6d200fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 -- 172.21.15.28:0/1240346912 shutdown_connections 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 -- 172.21.15.28:0/1240346912 wait complete. 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 librados: starting msgr at 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 librados: starting objecter 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 Processor -- start 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 -- start start 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 librados: setting wanted keys 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 librados: calling monclient init 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d20107940 con 0x7f6d20102250 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.991+0000 7f6d2719b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d20107b50 con 0x7f6d2010ea80 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d2719b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6d201a96a0 con 0x7f6d2010aee0 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6d2010aee0 0x7f6d20104ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:49848/0 (socket says 172.21.15.28:49848) 2024-03-30T15:11:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1ffff700 1 -- 172.21.15.28:0/1211532722 learned_addr learned my addr 172.21.15.28:0/1211532722 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter ms_handle_connect 0x7f6d2010aee0 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter ms_handle_connect 0x7f6d20102250 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter ms_handle_connect 0x7f6d2010ea80 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426644037 0 0) 0x7f6d201a96a0 con 0x7f6d2010aee0 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d100031f0 con 0x7f6d2010aee0 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269495949 0 0) 0x7f6d20107940 con 0x7f6d20102250 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d201a96a0 con 0x7f6d20102250 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940500537 0 0) 0x7f6d20107b50 con 0x7f6d2010ea80 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d20107940 con 0x7f6d2010ea80 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1265595187 0 0) 0x7f6d100031f0 con 0x7f6d2010aee0 2024-03-30T15:11:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d20107b50 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d0c001890 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3683851666 0 0) 0x7f6d201a96a0 con 0x7f6d20102250 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d100031f0 con 0x7f6d20102250 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d14004540 con 0x7f6d20102250 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2631277856 0 0) 0x7f6d20107940 con 0x7f6d2010ea80 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d201a96a0 con 0x7f6d2010ea80 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d18001af0 con 0x7f6d2010ea80 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 214677611 0 0) 0x7f6d20107b50 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 >> v1:172.21.15.134:6789/0 conn(0x7f6d2010ea80 legacy=0x7f6d201a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 >> v1:172.21.15.28:6789/0 conn(0x7f6d20102250 legacy=0x7f6d20104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d201aa8e0 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.992+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d201a98d0 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6d0c001c60 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6d0c005d10 con 0x7f6d2010aee0 2024-03-30T15:11:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d2719b700 10 client.58883.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d201a9d90 con 0x7f6d2010aee0 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d2719b700 1 librados: init done 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.993+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d20103730 con 0x7f6d2010aee0 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.994+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6d0c01c5d0 con 0x7f6d2010aee0 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.994+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(808..808 src has 442..808) ==== 10303+0+0 (unknown 3660501291 0 0) 0x7f6d0c089f50 con 0x7f6d2010aee0 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.994+0000 7f6d1dffb700 10 client.58883.objecter ms_dispatch 0x7f6d20001550 osd_map(808..808 src has 442..808) 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.994+0000 7f6d1dffb700 3 client.58883.objecter handle_osd_map got epochs [808,808] > 0 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.994+0000 7f6d1dffb700 3 client.58883.objecter handle_osd_map decoding full epoch 808 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.995+0000 7f6d1dffb700 20 client.58883.objecter dump_active .. 0 homeless 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.995+0000 7f6d1dffb700 10 client.58883.objecter ms_handle_connect 0x7f6d10070470 2024-03-30T15:11:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:53.999+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6d0c0568e0 con 0x7f6d2010aee0 2024-03-30T15:11:54.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:54.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.217+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f6d20068b30 con 0x7f6d2010aee0 2024-03-30T15:11:54.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:54.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:54.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.420+0000 7f6d1dffb700 1 -- 172.21.15.28:0/1211532722 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v809) ==== 87+0+0 (unknown 4004748592 0 0) 0x7f6d0c05a330 con 0x7f6d2010aee0 2024-03-30T15:11:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 10 librados: watch_flush enter 2024-03-30T15:11:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 10 librados: watch_flush exit 2024-03-30T15:11:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 20 client.58883.objecter shutdown clearing up homeless session... 2024-03-30T15:11:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 10 client.58883.objecter successfully canceled tick 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6d10070470 legacy=0x7f6d10072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.423+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 >> v1:172.21.15.28:6790/0 conn(0x7f6d2010aee0 legacy=0x7f6d20104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.424+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 shutdown_connections 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.424+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 >> 172.21.15.28:0/1211532722 conn(0x7f6d200fdb20 msgr2=0x7f6d20109fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.424+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 shutdown_connections 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.424+0000 7f6d2719b700 1 -- 172.21.15.28:0/1211532722 wait complete. 2024-03-30T15:11:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:54.424+0000 7f6d2719b700 1 librados: shutdown 2024-03-30T15:11:54.426 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:11:54.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:54.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:54.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:54.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:54.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:54.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: pgmap v1223: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 4 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 54 B/s, 1 objects/s recovering 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1211532722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: from='client.58883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: from='client.58883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[29876]: osdmap e809: 8 total, 6 up, 8 in 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: pgmap v1223: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 4 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 54 B/s, 1 objects/s recovering 2024-03-30T15:11:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1211532722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: from='client.58883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: from='client.58883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:54.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:54 smithi028 ceph-mon[41614]: osdmap e809: 8 total, 6 up, 8 in 2024-03-30T15:11:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:54.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:54.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: pgmap v1223: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 3.2 KiB/s wr, 4 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 54 B/s, 1 objects/s recovering 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1211532722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: from='client.58883 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:11:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:11:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: from='client.58883 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:11:54.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:54 smithi134 ceph-mon[33714]: osdmap e809: 8 total, 6 up, 8 in 2024-03-30T15:11:54.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:54.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:55.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:55.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:55.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:55.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:55.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:55.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:55.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:55.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:55.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:55.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:55.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:55.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:55.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:55.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:55 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:55 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 26 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:55.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:55 smithi134 ceph-mon[33714]: osdmap e810: 8 total, 6 up, 8 in 2024-03-30T15:11:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 26 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:55.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[41614]: osdmap e810: 8 total, 6 up, 8 in 2024-03-30T15:11:55.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:11:55.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 26 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:11:55.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:55 smithi028 ceph-mon[29876]: osdmap e810: 8 total, 6 up, 8 in 2024-03-30T15:11:55.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:56.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:56.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:56.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:56.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:56.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:56.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:56.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:56.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.543+0000 7f7217de9700 1 -- 172.21.15.28:0/1977951302 >> v1:172.21.15.28:6790/0 conn(0x7f721010d960 legacy=0x7f721010fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- 172.21.15.28:0/1977951302 shutdown_connections 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- 172.21.15.28:0/1977951302 >> 172.21.15.28:0/1977951302 conn(0x7f72100fc9e0 msgr2=0x7f72100fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- 172.21.15.28:0/1977951302 shutdown_connections 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- 172.21.15.28:0/1977951302 wait complete. 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 librados: starting msgr at 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 librados: starting objecter 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 Processor -- start 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- start start 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 librados: setting wanted keys 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 librados: calling monclient init 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72101a7060 con 0x7f721010d960 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.544+0000 7f7217de9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72101a8280 con 0x7f7210101130 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7217de9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72101a94c0 con 0x7f7210109dc0 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7216386700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f721010d960 0x7f72101a5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48642/0 (socket says 172.21.15.28:48642) 2024-03-30T15:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7216386700 1 -- 172.21.15.28:0/1169429870 learned_addr learned my addr 172.21.15.28:0/1169429870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter ms_handle_connect 0x7f721010d960 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter ms_handle_connect 0x7f7210101130 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter ms_handle_connect 0x7f7210109dc0 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919833721 0 0) 0x7f72101a8280 con 0x7f7210101130 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71fc003aa0 con 0x7f7210101130 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423282439 0 0) 0x7f72101a7060 con 0x7f721010d960 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72101a8280 con 0x7f721010d960 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527425731 0 0) 0x7f72101a94c0 con 0x7f7210109dc0 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72101a7060 con 0x7f7210109dc0 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3684382741 0 0) 0x7f72101a8280 con 0x7f721010d960 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72101a94c0 con 0x7f721010d960 2024-03-30T15:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3525198376 0 0) 0x7f71fc003aa0 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72101a8280 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1500876386 0 0) 0x7f72101a7060 con 0x7f7210109dc0 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71fc003aa0 con 0x7f7210109dc0 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72080047b0 con 0x7f721010d960 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72040017f0 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f720c001050 con 0x7f7210109dc0 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2236754077 0 0) 0x7f72101a8280 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 >> v1:172.21.15.28:6790/0 conn(0x7f7210109dc0 legacy=0x7f72101a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 >> v1:172.21.15.28:6789/0 conn(0x7f721010d960 legacy=0x7f72101a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.545+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72101aa700 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72101a84b0 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7217de9700 10 client.50884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72101a8970 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7217de9700 1 librados: init done 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7204003b20 con 0x7f7210101130 2024-03-30T15:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7204004a10 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.546+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f71f4005100 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.547+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7204003680 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(810..810 src has 442..810) ==== 10278+0+0 (unknown 2138029277 0 0) 0x7f7204088b30 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 osd_map(810..810 src has 442..810) 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 3 client.50884.objecter handle_osd_map got epochs [810,810] > 0 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 3 client.50884.objecter handle_osd_map decoding full epoch 810 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 20 client.50884.objecter dump_active .. 0 homeless 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71fc0703e0 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v810) ==== 54+0+16 (unknown 1038827610 0 2420000192) 0x7f7204089060 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 10 client.50884.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 1 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 20 client.50884.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 10 client.50884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=811}) -- 0x7f71f40053e0 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.548+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f71f4000e70 con 0x7f7210101130 2024-03-30T15:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=810) ==== 24+0+0 (unknown 1404501708 0 0) 0x7f7204054bb0 con 0x7f7210101130 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7203fff700 10 client.50884.objecter _check_command_map_dne tid 1 current 810 map_dne_bound 810 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7203fff700 10 client.50884.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7203fff700 15 client.50884.objecter _session_command_op_remove -1 1 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 10 client.50884.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 2 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 2 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 1 2 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 4 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 10 client.50884.objecter _send_command 2 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f71f4002380 con 0x7f71f40038d0 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.549+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f40038d0 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.550+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f4002380 con 0x7f71f40038d0 2024-03-30T15:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.550+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 2: 0 ) 2024-03-30T15:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.550+0000 7f7202ffd700 10 client.50884.objecter _finish_command 2 = system:0 2024-03-30T15:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.550+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 4 2024-03-30T15:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.550+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 1 2 2024-03-30T15:11:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 10 client.50884.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4001bb0 osd=1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 1 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 4 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 10 client.50884.objecter _send_command 3 2024-03-30T15:11:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.568+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f4001750 con 0x7f71f40038d0 2024-03-30T15:11:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71f4001750 con 0x7f71f40038d0 2024-03-30T15:11:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7202ffd700 10 client.50884.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4001bb0 osd=1 4 2024-03-30T15:11:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 1 3 2024-03-30T15:11:56.572 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7217de9700 10 client.50884.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 4 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7217de9700 20 client.50884.objecter _get_session osd=-1 returning homeless 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7217de9700 10 client.50884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.570+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f71f40019e0 con 0x7f7210101130 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=810) ==== 24+0+0 (unknown 2559233967 0 0) 0x7f7204052f20 con 0x7f7210101130 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f72037fe700 10 client.50884.objecter _check_command_map_dne tid 4 current 810 map_dne_bound 810 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f72037fe700 10 client.50884.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f72037fe700 15 client.50884.objecter _session_command_op_remove -1 4 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 10 client.50884.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 5 2024-03-30T15:11:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 5 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 3 5 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 4 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 10 client.50884.objecter _send_command 5 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f71f4003550 con 0x7f71f4002f50 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.571+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f4002f50 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.572+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f4003550 con 0x7f71f4002f50 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.572+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 5: 0 ) 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.572+0000 7f7202ffd700 10 client.50884.objecter _finish_command 5 = system:0 2024-03-30T15:11:56.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.572+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 4 2024-03-30T15:11:56.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.572+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 3 5 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 10 client.50884.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 6 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 6 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4004a20 osd=3 3 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 3 6 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 4 2024-03-30T15:11:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 10 client.50884.objecter _send_command 6 2024-03-30T15:11:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.589+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f40029a0 con 0x7f71f4002f50 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.590+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71f40029a0 con 0x7f71f4002f50 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.590+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.590+0000 7f7202ffd700 10 client.50884.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.590+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4004a20 osd=3 4 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.590+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 3 6 2024-03-30T15:11:56.592 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 10 client.50884.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 7 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 7 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 4 7 2024-03-30T15:11:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 4 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 10 client.50884.objecter _send_command 7 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f71f40098e0 con 0x7f71f400e430 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.591+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f400e430 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.592+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f40098e0 con 0x7f71f400e430 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.592+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 7: 0 ) 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.592+0000 7f7202ffd700 10 client.50884.objecter _finish_command 7 = system:0 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.592+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 4 2024-03-30T15:11:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.592+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 4 7 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 10 client.50884.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 8 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 8 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f40091d0 osd=4 3 2024-03-30T15:11:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 4 8 2024-03-30T15:11:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 4 2024-03-30T15:11:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 10 client.50884.objecter _send_command 8 2024-03-30T15:11:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.609+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f4010b90 con 0x7f71f400e430 2024-03-30T15:11:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f720c00a860 con 0x7f71f400e430 2024-03-30T15:11:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7202ffd700 10 client.50884.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f40091d0 osd=4 4 2024-03-30T15:11:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 4 8 2024-03-30T15:11:56.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:11:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 10 client.50884.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 9 2024-03-30T15:11:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 9 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 5 9 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 4 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 10 client.50884.objecter _send_command 9 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.611+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f71f400e140 con 0x7f71f4008dc0 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f4008dc0 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f400e140 con 0x7f71f4008dc0 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 9: 0 ) 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 10 client.50884.objecter _finish_command 9 = system:0 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 4 2024-03-30T15:11:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.612+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 5 9 2024-03-30T15:11:56.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 10 client.50884.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 10 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 10 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f4008c50 osd=5 3 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 5 10 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 4 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 10 client.50884.objecter _send_command 10 2024-03-30T15:11:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.631+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f400d500 con 0x7f71f4008dc0 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71f400d500 con 0x7f71f4008dc0 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 10 client.50884.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f4008c50 osd=5 4 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 5 10 2024-03-30T15:11:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 10 client.50884.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 11 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 11 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 6 11 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 4 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 10 client.50884.objecter _send_command 11 2024-03-30T15:11:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f71f400d880 con 0x7f71f4017560 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.634+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f4017560 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.635+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f400d880 con 0x7f71f4017560 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.635+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 11: 0 ) 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.635+0000 7f7202ffd700 10 client.50884.objecter _finish_command 11 = system:0 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.635+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 4 2024-03-30T15:11:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.635+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 6 11 2024-03-30T15:11:56.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 10 client.50884.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 12 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 12 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f400d500 osd=6 3 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 6 12 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 4 2024-03-30T15:11:56.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 10 client.50884.objecter _send_command 12 2024-03-30T15:11:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.652+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f401fb10 con 0x7f71f4017560 2024-03-30T15:11:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.654+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71f401fb10 con 0x7f71f4017560 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.654+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.654+0000 7f7202ffd700 10 client.50884.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.654+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f400d500 osd=6 4 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.654+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 6 12 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:11:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 10 client.50884.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 13 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 13 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 7 13 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 4 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 10 client.50884.objecter _send_command 13 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f71f4020db0 con 0x7f71f4020230 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.655+0000 7f7202ffd700 10 client.50884.objecter ms_handle_connect 0x7f71f4020230 2024-03-30T15:11:56.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.656+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f71f4020db0 con 0x7f71f4020230 2024-03-30T15:11:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.656+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 13: 0 ) 2024-03-30T15:11:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.656+0000 7f7202ffd700 10 client.50884.objecter _finish_command 13 = system:0 2024-03-30T15:11:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.656+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 4 2024-03-30T15:11:56.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.656+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 7 13 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 10 client.50884.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign -1 14 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 20 client.50884.objecter _get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 15 client.50884.objecter _session_command_op_remove -1 14 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 20 client.50884.objecter get_session s=0x7f71f400d900 osd=7 3 2024-03-30T15:11:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 15 client.50884.objecter _session_command_op_assign 7 14 2024-03-30T15:11:56.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 4 2024-03-30T15:11:56.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 10 client.50884.objecter _send_command 14 2024-03-30T15:11:56.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.673+0000 7f7217de9700 1 -- 172.21.15.28:0/1169429870 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f71f4021030 con 0x7f71f4020230 2024-03-30T15:11:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.675+0000 7f7202ffd700 1 -- 172.21.15.28:0/1169429870 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71f4021030 con 0x7f71f4020230 2024-03-30T15:11:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.675+0000 7f7202ffd700 10 client.50884.objecter ms_dispatch 0x7f7210001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:11:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.675+0000 7f7202ffd700 10 client.50884.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:11:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.675+0000 7f7202ffd700 20 client.50884.objecter put_session s=0x7f71f400d900 osd=7 4 2024-03-30T15:11:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:11:56.675+0000 7f7202ffd700 15 client.50884.objecter _session_command_op_remove 7 14 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:11:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:11:56.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[29876]: pgmap v1226: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 81 B/s, 2 objects/s recovering 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1169429870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[41614]: pgmap v1226: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 81 B/s, 2 objects/s recovering 2024-03-30T15:11:56.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1169429870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:56.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:56.765 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:11:56.766 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:11:56.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:56.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:56.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:56 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:11:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:56 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:11:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:56 smithi134 ceph-mon[33714]: pgmap v1226: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 81 B/s, 2 objects/s recovering 2024-03-30T15:11:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1169429870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:11:56.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:57.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:57.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:57.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:57.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:57.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:57.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:57.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:11:57.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:57.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:57.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:11:57.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:57.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:11:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:57.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:57.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:57.862 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:11:57.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:57.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:57.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:58.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:58.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:58.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:58.183 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:58.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:58.309 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:58.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:11:58.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:11:58.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:58.539 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:58.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:58.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:58 smithi028 ceph-mon[29876]: osdmap e811: 8 total, 6 up, 8 in 2024-03-30T15:11:58.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:11:58 smithi028 ceph-mon[29876]: pgmap v1228: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 69 B/s, 2 objects/s recovering 2024-03-30T15:11:58.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:58 smithi028 ceph-mon[41614]: osdmap e811: 8 total, 6 up, 8 in 2024-03-30T15:11:58.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:11:58 smithi028 ceph-mon[41614]: pgmap v1228: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 69 B/s, 2 objects/s recovering 2024-03-30T15:11:58.609 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:11:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:11:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:11:58.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:58 smithi134 ceph-mon[33714]: osdmap e811: 8 total, 6 up, 8 in 2024-03-30T15:11:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:11:58 smithi134 ceph-mon[33714]: pgmap v1228: 246 pgs: 1 remapped+peering, 26 active+clean+remapped, 111 active+clean, 11 undersized+peered, 67 active+undersized, 1 undersized+degraded+peered, 4 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.0 KiB/s wr, 5 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%); 69 B/s, 2 objects/s recovering 2024-03-30T15:11:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:58.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:58.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:11:58.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:58.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:11:59.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:59.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:59.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:11:59.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:11:59.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:11:59.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:59.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:59.533 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:11:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:11:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:11:59.533 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:11:59.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:11:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:11:59.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:11:59.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:11:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:11:59.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:11:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:11:59.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:11:59.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:11:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:11:59.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:00.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:00.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:00.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:00.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:00.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:12:00.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:00.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:00.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:12:00.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:00 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:00.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:00 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:00.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:00.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:00.542 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:00.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:00.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:00.663 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:00 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 94 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:00.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:00.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:00.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:00.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:00.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:00.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:00.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:00.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:01.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:01.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:01.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:01.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:01.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:01.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:01.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:01.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:01 smithi028 ceph-mon[41614]: pgmap v1229: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:01.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:01 smithi028 ceph-mon[29876]: pgmap v1229: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.577+0000 7f0c98e76700 1 -- 172.21.15.28:0/2959664839 >> v1:172.21.15.28:6790/0 conn(0x7f0c94103190 legacy=0x7f0c94103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 -- 172.21.15.28:0/2959664839 shutdown_connections 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 -- 172.21.15.28:0/2959664839 >> 172.21.15.28:0/2959664839 conn(0x7f0c940fc9e0 msgr2=0x7f0c940fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 -- 172.21.15.28:0/2959664839 shutdown_connections 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 -- 172.21.15.28:0/2959664839 wait complete. 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 librados: starting msgr at 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.578+0000 7f0c98e76700 1 librados: starting objecter 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 Processor -- start 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 -- start start 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 librados: setting wanted keys 2024-03-30T15:12:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 librados: calling monclient init 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c940fec80 con 0x7f0c94103190 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c94102ac0 con 0x7f0c9410eab0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c98e76700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0c94102cf0 con 0x7f0c9410afa0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c91d9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f0c9410afa0 0x7f0c94101500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58018/0 (socket says 172.21.15.28:58018) 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c91d9b700 1 -- 172.21.15.28:0/1612106317 learned_addr learned my addr 172.21.15.28:0/1612106317 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter ms_handle_connect 0x7f0c9410afa0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter ms_handle_connect 0x7f0c94103190 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter ms_handle_connect 0x7f0c9410eab0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046333399 0 0) 0x7f0c94102cf0 con 0x7f0c9410afa0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c7c0031f0 con 0x7f0c9410afa0 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871252160 0 0) 0x7f0c940fec80 con 0x7f0c94103190 2024-03-30T15:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c94102cf0 con 0x7f0c94103190 2024-03-30T15:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.579+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156429172 0 0) 0x7f0c94102ac0 con 0x7f0c9410eab0 2024-03-30T15:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c940fec80 con 0x7f0c9410eab0 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4028434572 0 0) 0x7f0c7c0031f0 con 0x7f0c9410afa0 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c94102ac0 con 0x7f0c9410afa0 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c88001970 con 0x7f0c9410afa0 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1576754456 0 0) 0x7f0c94102cf0 con 0x7f0c94103190 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c7c0031f0 con 0x7f0c94103190 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3034977166 0 0) 0x7f0c940fec80 con 0x7f0c9410eab0 2024-03-30T15:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c94102cf0 con 0x7f0c9410eab0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c84004610 con 0x7f0c94103190 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c8c001b90 con 0x7f0c9410eab0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2381933966 0 0) 0x7f0c94102ac0 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 >> v1:172.21.15.134:6789/0 conn(0x7f0c9410eab0 legacy=0x7f0c941a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 >> v1:172.21.15.28:6789/0 conn(0x7f0c94103190 legacy=0x7f0c94100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c941aca40 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c941a9a70 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c98e76700 10 client.58895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c941a9f30 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c98e76700 1 librados: init done 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0c88001c70 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.580+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0c88005da0 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.581+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c74005120 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.582+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0c8801c420 con 0x7f0c9410afa0 2024-03-30T15:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(811..811 src has 442..811) ==== 10278+0+0 (unknown 3160130499 0 0) 0x7f0c88089ff0 con 0x7f0c9410afa0 2024-03-30T15:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 10 client.58895.objecter ms_dispatch 0x7f0c94001550 osd_map(811..811 src has 442..811) 2024-03-30T15:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 3 client.58895.objecter handle_osd_map got epochs [811,811] > 0 2024-03-30T15:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 3 client.58895.objecter handle_osd_map decoding full epoch 811 2024-03-30T15:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 20 client.58895.objecter dump_active .. 0 homeless 2024-03-30T15:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.583+0000 7f0c837fe700 10 client.58895.objecter ms_handle_connect 0x7f0c7c0704e0 2024-03-30T15:12:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.588+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0c880569c0 con 0x7f0c9410afa0 2024-03-30T15:12:01.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:01.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:01.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:01 smithi134 ceph-mon[33714]: pgmap v1229: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:01.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:01.811+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f0c74005410 con 0x7f0c9410afa0 2024-03-30T15:12:01.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:01.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:01.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:01.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:01.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:02.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.114+0000 7f08ea763700 1 -- 172.21.15.28:0/1980234415 >> v1:172.21.15.28:6789/0 conn(0x7f08e410d960 legacy=0x7f08e410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.114+0000 7f08ea763700 1 -- 172.21.15.28:0/1980234415 shutdown_connections 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.114+0000 7f08ea763700 1 -- 172.21.15.28:0/1980234415 >> 172.21.15.28:0/1980234415 conn(0x7f08e40fc9a0 msgr2=0x7f08e40fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.114+0000 7f08ea763700 1 -- 172.21.15.28:0/1980234415 shutdown_connections 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.114+0000 7f08ea763700 1 -- 172.21.15.28:0/1980234415 wait complete. 2024-03-30T15:12:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 librados: starting msgr at 2024-03-30T15:12:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 librados: starting objecter 2024-03-30T15:12:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 Processor -- start 2024-03-30T15:12:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 -- start start 2024-03-30T15:12:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 librados: setting wanted keys 2024-03-30T15:12:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.115+0000 7f08ea763700 1 librados: calling monclient init 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08ea763700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a7150 con 0x7f08e410d960 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08ea763700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a8370 con 0x7f08e4109dc0 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08ea763700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f08e41a95b0 con 0x7f08e4101130 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e3fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f08e4101130 0x7f08e4106500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58022/0 (socket says 172.21.15.28:58022) 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e3fff700 1 -- 172.21.15.28:0/1790332010 learned_addr learned my addr 172.21.15.28:0/1790332010 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter ms_handle_connect 0x7f08e4101130 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter ms_handle_connect 0x7f08e410d960 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13988791 0 0) 0x7f08e41a7150 con 0x7f08e410d960 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08cc0031f0 con 0x7f08e410d960 2024-03-30T15:12:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032468641 0 0) 0x7f08e41a95b0 con 0x7f08e4101130 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e41a7150 con 0x7f08e4101130 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter ms_handle_connect 0x7f08e4109dc0 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2998429888 0 0) 0x7f08cc0031f0 con 0x7f08e410d960 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e41a95b0 con 0x7f08e410d960 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.116+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f08dc0047f0 con 0x7f08e410d960 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 498628479 0 0) 0x7f08e41a8370 con 0x7f08e4109dc0 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08cc0031f0 con 0x7f08e4109dc0 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3124636698 0 0) 0x7f08e41a7150 con 0x7f08e4101130 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08e41a8370 con 0x7f08e4101130 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f08d4001a10 con 0x7f08e4101130 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 56753875 0 0) 0x7f08e41a95b0 con 0x7f08e410d960 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 >> v1:172.21.15.28:6790/0 conn(0x7f08e4101130 legacy=0x7f08e4106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 >> v1:172.21.15.134:6789/0 conn(0x7f08e4109dc0 legacy=0x7f08e41a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e41aa7f0 con 0x7f08e410d960 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41a7380 con 0x7f08e410d960 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08ea763700 10 client.25698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f08e41a7840 con 0x7f08e410d960 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08ea763700 1 librados: init done 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f08dc001e60 con 0x7f08e410d960 2024-03-30T15:12:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.117+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f08dc005050 con 0x7f08e410d960 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.118+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d0005120 con 0x7f08e410d960 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.118+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f08dc0051f0 con 0x7f08e410d960 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(811..811 src has 442..811) ==== 10278+0+0 (unknown 3160130499 0 0) 0x7f08dc0891a0 con 0x7f08e410d960 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 10 client.25698.objecter ms_dispatch 0x7f08e4001550 osd_map(811..811 src has 442..811) 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 3 client.25698.objecter handle_osd_map got epochs [811,811] > 0 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 3 client.25698.objecter handle_osd_map decoding full epoch 811 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 20 client.25698.objecter dump_active .. 0 homeless 2024-03-30T15:12:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.119+0000 7f08e17fa700 10 client.25698.objecter ms_handle_connect 0x7f08cc070590 2024-03-30T15:12:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.124+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f08dc056af0 con 0x7f08e410d960 2024-03-30T15:12:02.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:02.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:02.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.265+0000 7f0c837fe700 1 -- 172.21.15.28:0/1612106317 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v812) ==== 97+0+0 (unknown 4035166059 0 0) 0x7f0c8805a410 con 0x7f0c9410afa0 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 10 librados: watch_flush enter 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 10 librados: watch_flush exit 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 20 client.58895.objecter shutdown clearing up homeless session... 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 10 client.58895.objecter successfully canceled tick 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0c7c0704e0 legacy=0x7f0c7c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.268+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 >> v1:172.21.15.28:6790/0 conn(0x7f0c9410afa0 legacy=0x7f0c94101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.269+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 shutdown_connections 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.269+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 >> 172.21.15.28:0/1612106317 conn(0x7f0c940fc9e0 msgr2=0x7f0c94110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.269+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 shutdown_connections 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.269+0000 7f0c98e76700 1 -- 172.21.15.28:0/1612106317 wait complete. 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.269+0000 7f0c98e76700 1 librados: shutdown 2024-03-30T15:12:02.272 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:12:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.338+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f08d0000b90 con 0x7f08cc070590 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.339+0000 7f08e17fa700 1 -- 172.21.15.28:0/1790332010 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+300 (unknown 125648544 0 2422649331) 0x7f08d0000b90 con 0x7f08cc070590 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 738 (0%) 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:avg 123 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 27.3038 -> 27.3038 (expected baseline 10.1242) 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 78 -> 78 pgs (0.634146 -> 0.634146 * mean) 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.7 with 112 -> 112 pgs (0.910569 -> 0.910569 * mean) 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-03-30T15:12:02.342 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-03-30T15:12:02.343 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0011 2024-03-30T15:12:02.343 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0111 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.341+0000 7f08ea763700 10 librados: watch_flush enter 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.341+0000 7f08ea763700 10 librados: watch_flush exit 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.341+0000 7f08ea763700 20 client.25698.objecter shutdown clearing up homeless session... 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.341+0000 7f08ea763700 10 client.25698.objecter successfully canceled tick 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.341+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 >> v1:172.21.15.28:6800/3011431093 conn(0x7f08cc070590 legacy=0x7f08cc072a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 >> v1:172.21.15.28:6789/0 conn(0x7f08e410d960 legacy=0x7f08e41a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 shutdown_connections 2024-03-30T15:12:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 >> 172.21.15.28:0/1790332010 conn(0x7f08e40fc9a0 msgr2=0x7f08e4110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 shutdown_connections 2024-03-30T15:12:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 -- 172.21.15.28:0/1790332010 wait complete. 2024-03-30T15:12:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:02.342+0000 7f08ea763700 1 librados: shutdown 2024-03-30T15:12:02.345 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:02 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:02.345 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1612106317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.345 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:02 smithi134 ceph-mon[33714]: from='client.58895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-03-30T15:12:02.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:02.363 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:12:02.363 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:12:02.363 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:12:02.363 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:12:02.363 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:12:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:02.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1612106317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[41614]: from='client.58895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1612106317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:02 smithi028 ceph-mon[29876]: from='client.58895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:02.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:02.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:02.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:02.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:02.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:02.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:02.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:02.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:03.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:03.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:12:03.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:03.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:03.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:03.256 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:03.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:03.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:03.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:03.515 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:03 smithi134 ceph-mon[33714]: pgmap v1230: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:03.515 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:03 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:03.515 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:03 smithi134 ceph-mon[33714]: from='client.58895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:03.515 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:03 smithi134 ceph-mon[33714]: osdmap e812: 8 total, 6 up, 8 in 2024-03-30T15:12:03.515 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:03 smithi134 ceph-mon[33714]: osdmap e813: 8 total, 6 up, 8 in 2024-03-30T15:12:03.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:03.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:03.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[41614]: pgmap v1230: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:03.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:03.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[41614]: from='client.58895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:03.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[41614]: osdmap e812: 8 total, 6 up, 8 in 2024-03-30T15:12:03.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[41614]: osdmap e813: 8 total, 6 up, 8 in 2024-03-30T15:12:03.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[29876]: pgmap v1230: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 661 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:03.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:03.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[29876]: from='client.58895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:03.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[29876]: osdmap e812: 8 total, 6 up, 8 in 2024-03-30T15:12:03.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:03 smithi028 ceph-mon[29876]: osdmap e813: 8 total, 6 up, 8 in 2024-03-30T15:12:03.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:03.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:03.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:03.885 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:03.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:03.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:04.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:04.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:04.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:04.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:04.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:04.422 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:04 smithi134 ceph-mon[33714]: from='client.25698 v1:172.21.15.28:0/1790332010' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.460+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2211369451 >> v1:172.21.15.28:6790/0 conn(0x7fbcb0101130 legacy=0x7fbcb0101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.460+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2211369451 shutdown_connections 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.460+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2211369451 >> 172.21.15.28:0/2211369451 conn(0x7fbcb00fc9e0 msgr2=0x7fbcb00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.460+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2211369451 shutdown_connections 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.460+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2211369451 wait complete. 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 librados: starting msgr at 2024-03-30T15:12:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 librados: starting objecter 2024-03-30T15:12:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 Processor -- start 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 -- start start 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 librados: setting wanted keys 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 librados: calling monclient init 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcb010f9c0 con 0x7fbcb0101130 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcb010fbd0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.461+0000 7fbcb56ab700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbcb010fde0 con 0x7fbcb010d960 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcaf7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbcb010d960 0x7fbcb01a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58042/0 (socket says 172.21.15.28:58042) 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcaeffd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbcb0101130 0x7fbcb010dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54690/0 (socket says 172.21.15.28:54690) 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcaf7fe700 1 -- 172.21.15.28:0/2584389635 learned_addr learned my addr 172.21.15.28:0/2584389635 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter ms_handle_connect 0x7fbcb010d960 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter ms_handle_connect 0x7fbcb0101130 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter ms_handle_connect 0x7fbcb0109dc0 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691547779 0 0) 0x7fbcb010fbd0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc980031f0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377401319 0 0) 0x7fbcb010fde0 con 0x7fbcb010d960 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb010fbd0 con 0x7fbcb010d960 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382313080 0 0) 0x7fbcb010f9c0 con 0x7fbcb0101130 2024-03-30T15:12:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb010fde0 con 0x7fbcb0101130 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1074830976 0 0) 0x7fbc980031f0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcb010f9c0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbca4001870 con 0x7fbcb0109dc0 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2520597417 0 0) 0x7fbcb010fbd0 con 0x7fbcb010d960 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc980031f0 con 0x7fbcb010d960 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 951975300 0 0) 0x7fbcb010fde0 con 0x7fbcb0101130 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcb010fbd0 con 0x7fbcb0101130 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbca8001bf0 con 0x7fbcb010d960 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbca0001370 con 0x7fbcb0101130 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2783387184 0 0) 0x7fbcb010f9c0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 >> v1:172.21.15.28:6790/0 conn(0x7fbcb010d960 legacy=0x7fbcb01a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 >> v1:172.21.15.28:6789/0 conn(0x7fbcb0101130 legacy=0x7fbcb010dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb01ac990 con 0x7fbcb0109dc0 2024-03-30T15:12:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb01a9980 con 0x7fbcb0109dc0 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcb56ab700 10 client.50902.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbcb01a9e40 con 0x7fbcb0109dc0 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.462+0000 7fbcb56ab700 1 librados: init done 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.463+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbca4003b20 con 0x7fbcb0109dc0 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.463+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbca4004a30 con 0x7fbcb0109dc0 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.463+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbc9c005100 con 0x7fbcb0109dc0 2024-03-30T15:12:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.464+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbca4003680 con 0x7fbcb0109dc0 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.464+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(813..813 src has 442..813) ==== 10278+0+0 (unknown 1027222629 0 0) 0x7fbca4088b60 con 0x7fbcb0109dc0 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.464+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 osd_map(813..813 src has 442..813) 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.464+0000 7fbc97fff700 3 client.50902.objecter handle_osd_map got epochs [813,813] > 0 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.464+0000 7fbc97fff700 3 client.50902.objecter handle_osd_map decoding full epoch 813 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbc97fff700 20 client.50902.objecter dump_active .. 0 homeless 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v813) ==== 54+0+16 (unknown 158023941 0 2420000192) 0x7fbca4089060 con 0x7fbcb0109dc0 2024-03-30T15:12:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc98070540 2024-03-30T15:12:04.470 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:04 smithi028 ceph-mon[41614]: from='client.25698 v1:172.21.15.28:0/1790332010' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:04.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:04 smithi028 ceph-mon[29876]: from='client.25698 v1:172.21.15.28:0/1790332010' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 1 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 10 client.50902.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=814}) -- 0x7fbc9c0053e0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbc9c000ed0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=813) ==== 24+0+0 (unknown 2253009620 0 0) 0x7fbca4054be0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcad7fa700 10 client.50902.objecter _check_command_map_dne tid 1 current 813 map_dne_bound 813 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcad7fa700 10 client.50902.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcad7fa700 15 client.50902.objecter _session_command_op_remove -1 1 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 2 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 2 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 1 2 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 4 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 10 client.50902.objecter _send_command 2 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.465+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c004460 con 0x7fbc9c003910 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.466+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c003910 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.467+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c004460 con 0x7fbc9c003910 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.467+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.467+0000 7fbc97fff700 10 client.50902.objecter _finish_command 2 = system:0 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.467+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 4 2024-03-30T15:12:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.467+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 1 2 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c0011c0 osd=1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 1 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 4 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 10 client.50902.objecter _send_command 3 2024-03-30T15:12:04.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.483+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c004e10 con 0x7fbc9c003910 2024-03-30T15:12:04.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c004e10 con 0x7fbc9c003910 2024-03-30T15:12:04.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbc97fff700 10 client.50902.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c0011c0 osd=1 4 2024-03-30T15:12:04.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 1 3 2024-03-30T15:12:04.488 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 4 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbcb56ab700 20 client.50902.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbcb56ab700 10 client.50902.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.485+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fbc9c001ad0 con 0x7fbcb0109dc0 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=813) ==== 24+0+0 (unknown 1299579831 0 0) 0x7fbca4052f50 con 0x7fbcb0109dc0 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcacff9700 10 client.50902.objecter _check_command_map_dne tid 4 current 813 map_dne_bound 813 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcacff9700 10 client.50902.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcacff9700 15 client.50902.objecter _session_command_op_remove -1 4 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 5 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 5 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 3 5 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 4 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 10 client.50902.objecter _send_command 5 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c0035c0 con 0x7fbc9c002f60 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.486+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c002f60 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.487+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c0035c0 con 0x7fbc9c002f60 2024-03-30T15:12:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.487+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.487+0000 7fbc97fff700 10 client.50902.objecter _finish_command 5 = system:0 2024-03-30T15:12:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.487+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 4 2024-03-30T15:12:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.487+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 3 5 2024-03-30T15:12:04.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 6 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 6 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c001c80 osd=3 3 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 3 6 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 4 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 10 client.50902.objecter _send_command 6 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.504+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c0029e0 con 0x7fbc9c002f60 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.505+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c0029e0 con 0x7fbc9c002f60 2024-03-30T15:12:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.505+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.505+0000 7fbc97fff700 10 client.50902.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.505+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c001c80 osd=3 4 2024-03-30T15:12:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.505+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 3 6 2024-03-30T15:12:04.508 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 7 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 7 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 4 7 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 4 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 10 client.50902.objecter _send_command 7 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c011f40 con 0x7fbc9c00e430 2024-03-30T15:12:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.506+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c00e430 2024-03-30T15:12:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.507+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c011f40 con 0x7fbc9c00e430 2024-03-30T15:12:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.507+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 7: 0 ) 2024-03-30T15:12:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.507+0000 7fbc97fff700 10 client.50902.objecter _finish_command 7 = system:0 2024-03-30T15:12:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.507+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 4 2024-03-30T15:12:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.507+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 4 7 2024-03-30T15:12:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:04.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 8 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 8 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c0091d0 osd=4 3 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 4 8 2024-03-30T15:12:04.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 4 2024-03-30T15:12:04.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 10 client.50902.objecter _send_command 8 2024-03-30T15:12:04.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.524+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c008d10 con 0x7fbc9c00e430 2024-03-30T15:12:04.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c008d10 con 0x7fbc9c00e430 2024-03-30T15:12:04.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 10 client.50902.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c0091d0 osd=4 4 2024-03-30T15:12:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 4 8 2024-03-30T15:12:04.529 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 9 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 9 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 5 9 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 4 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 10 client.50902.objecter _send_command 9 2024-03-30T15:12:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c00e260 con 0x7fbc9c013270 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c013270 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c00e260 con 0x7fbc9c013270 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 9: 0 ) 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 10 client.50902.objecter _finish_command 9 = system:0 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 4 2024-03-30T15:12:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.527+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 5 9 2024-03-30T15:12:04.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 10 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 10 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c00da80 osd=5 3 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 5 10 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 4 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 10 client.50902.objecter _send_command 10 2024-03-30T15:12:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.544+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c00d500 con 0x7fbc9c013270 2024-03-30T15:12:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c00d500 con 0x7fbc9c013270 2024-03-30T15:12:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 10 client.50902.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c00da80 osd=5 4 2024-03-30T15:12:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 5 10 2024-03-30T15:12:04.549 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 11 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 11 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 6 11 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 4 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 10 client.50902.objecter _send_command 11 2024-03-30T15:12:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c01b760 con 0x7fbc9c012dd0 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c012dd0 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c01b760 con 0x7fbc9c012dd0 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 11: 0 ) 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 10 client.50902.objecter _finish_command 11 = system:0 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 4 2024-03-30T15:12:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.547+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 6 11 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 12 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 12 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c00d500 osd=6 3 2024-03-30T15:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 6 12 2024-03-30T15:12:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 4 2024-03-30T15:12:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 10 client.50902.objecter _send_command 12 2024-03-30T15:12:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.565+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c01b760 con 0x7fbc9c012dd0 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c01b760 con 0x7fbc9c012dd0 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 10 client.50902.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c00d500 osd=6 4 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 6 12 2024-03-30T15:12:04.568 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 13 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 13 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 7 13 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 4 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 10 client.50902.objecter _send_command 13 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbc9c01fae0 con 0x7fbc9c01bde0 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 10 client.50902.objecter ms_handle_connect 0x7fbc9c01bde0 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbc9c01fae0 con 0x7fbc9c01bde0 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 13: 0 ) 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 10 client.50902.objecter _finish_command 13 = system:0 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 4 2024-03-30T15:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.567+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 7 13 2024-03-30T15:12:04.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 10 client.50902.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign -1 14 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 20 client.50902.objecter _get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_remove -1 14 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 20 client.50902.objecter get_session s=0x7fbc9c01b760 osd=7 3 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 15 client.50902.objecter _session_command_op_assign 7 14 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 4 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 10 client.50902.objecter _send_command 14 2024-03-30T15:12:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.583+0000 7fbcb56ab700 1 -- 172.21.15.28:0/2584389635 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc9c00d710 con 0x7fbc9c01bde0 2024-03-30T15:12:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.586+0000 7fbc97fff700 1 -- 172.21.15.28:0/2584389635 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbc9c00d710 con 0x7fbc9c01bde0 2024-03-30T15:12:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.586+0000 7fbc97fff700 10 client.50902.objecter ms_dispatch 0x7fbcb0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.586+0000 7fbc97fff700 10 client.50902.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.586+0000 7fbc97fff700 20 client.50902.objecter put_session s=0x7fbc9c01b760 osd=7 4 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:04.586+0000 7fbc97fff700 15 client.50902.objecter _session_command_op_remove 7 14 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:04.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:04.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:04.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:04.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:04.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:04.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:04.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:04.977 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:04.977 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:04.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:05.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:05.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:05.213 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-30T15:12:05.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight-by-utilization 1000 0.05 2024-03-30T15:12:05.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:05.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:05.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:05.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:05 smithi028 ceph-mon[29876]: pgmap v1233: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:05.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2584389635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:05 smithi028 ceph-mon[41614]: pgmap v1233: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:05.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2584389635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:05.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:05.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:05.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:05.675 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:05 smithi134 ceph-mon[33714]: pgmap v1233: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:05.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2584389635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:05.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:12:05.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:05.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:05.788 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:05.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:05.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:05.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:12:05.997 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:06.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.038 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:06.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:06.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:06.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:06.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:06.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:06.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:06.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:06.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:06.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:06.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:06.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:06.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:06.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:06.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:07.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:07.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:07.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:07.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:07.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:07.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:07.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:07 smithi028 ceph-mon[29876]: pgmap v1234: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:07.299 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:07 smithi028 ceph-mon[41614]: pgmap v1234: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:07.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:07.417 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:07 smithi134 ceph-mon[33714]: pgmap v1234: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:07.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:07.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.560+0000 7fe8a217a700 1 -- 172.21.15.28:0/910542542 >> v1:172.21.15.28:6789/0 conn(0x7fe89c10d960 legacy=0x7fe89c10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.560+0000 7fe8a217a700 1 -- 172.21.15.28:0/910542542 shutdown_connections 2024-03-30T15:12:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.560+0000 7fe8a217a700 1 -- 172.21.15.28:0/910542542 >> 172.21.15.28:0/910542542 conn(0x7fe89c0fc9e0 msgr2=0x7fe89c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.560+0000 7fe8a217a700 1 -- 172.21.15.28:0/910542542 shutdown_connections 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.560+0000 7fe8a217a700 1 -- 172.21.15.28:0/910542542 wait complete. 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 librados: starting msgr at 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 librados: starting objecter 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 Processor -- start 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 -- start start 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 librados: setting wanted keys 2024-03-30T15:12:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 librados: calling monclient init 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe89c0fec00 con 0x7fe89c10d960 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe89c1aa930 con 0x7fe89c109dc0 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8a217a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe89c1abb10 con 0x7fe89c101130 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe89b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe89c101130 0x7fe89c0fce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58062/0 (socket says 172.21.15.28:58062) 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe89b7fe700 1 -- 172.21.15.28:0/663537943 learned_addr learned my addr 172.21.15.28:0/663537943 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8997fa700 10 client.?.objecter ms_handle_connect 0x7fe89c101130 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.561+0000 7fe8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 10 client.?.objecter ms_handle_connect 0x7fe89c10d960 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 10 client.?.objecter ms_handle_connect 0x7fe89c109dc0 2024-03-30T15:12:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4116846791 0 0) 0x7fe89c0fec00 con 0x7fe89c10d960 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8880031f0 con 0x7fe89c10d960 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785932620 0 0) 0x7fe89c1abb10 con 0x7fe89c101130 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe89c0fec00 con 0x7fe89c101130 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532639230 0 0) 0x7fe89c1aa930 con 0x7fe89c109dc0 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.562+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe89c1abb10 con 0x7fe89c109dc0 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1434750643 0 0) 0x7fe89c0fec00 con 0x7fe89c101130 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe89c1aa930 con 0x7fe89c101130 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe88c0017e0 con 0x7fe89c101130 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4207837215 0 0) 0x7fe8880031f0 con 0x7fe89c10d960 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe89c0fec00 con 0x7fe89c10d960 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2012784466 0 0) 0x7fe89c1abb10 con 0x7fe89c109dc0 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8880031f0 con 0x7fe89c109dc0 2024-03-30T15:12:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe8940017b0 con 0x7fe89c10d960 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe8840017f0 con 0x7fe89c109dc0 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3822888510 0 0) 0x7fe89c1aa930 con 0x7fe89c101130 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 >> v1:172.21.15.134:6789/0 conn(0x7fe89c109dc0 legacy=0x7fe89c0fd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 >> v1:172.21.15.28:6789/0 conn(0x7fe89c10d960 legacy=0x7fe89c1a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c1accf0 con 0x7fe89c101130 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe89c1aab00 con 0x7fe89c101130 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8a217a700 10 client.58913.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe89c1aafc0 con 0x7fe89c101130 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8a217a700 1 librados: init done 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe88c003b40 con 0x7fe89c101130 2024-03-30T15:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.563+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe88c005c30 con 0x7fe89c101130 2024-03-30T15:12:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.564+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c005120 con 0x7fe89c101130 2024-03-30T15:12:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe88c0036a0 con 0x7fe89c101130 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(813..813 src has 442..813) ==== 10278+0+0 (unknown 1027222629 0 0) 0x7fe88c089d50 con 0x7fe89c101130 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 10 client.58913.objecter ms_dispatch 0x7fe89c001550 osd_map(813..813 src has 442..813) 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 3 client.58913.objecter handle_osd_map got epochs [813,813] > 0 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 3 client.58913.objecter handle_osd_map decoding full epoch 813 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 20 client.58913.objecter dump_active .. 0 homeless 2024-03-30T15:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.565+0000 7fe8997fa700 10 client.58913.objecter ms_handle_connect 0x7fe8880704d0 2024-03-30T15:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.571+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe88c0ba270 con 0x7fe89c101130 2024-03-30T15:12:07.644 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:07.645 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:07.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:07.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:07.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:12:07.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.789+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7fe87c000b90 con 0x7fe8880704d0 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.791+0000 7fe8997fa700 1 -- 172.21.15.28:0/663537943 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+300 (unknown 125648544 0 2422649331) 0x7fe87c000b90 con 0x7fe8880704d0 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 738 (0%) 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:avg 123 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 27.3038 -> 27.3038 (expected baseline 10.1242) 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.3 with 78 -> 78 pgs (0.634146 -> 0.634146 * mean) 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.7 with 112 -> 112 pgs (0.910569 -> 0.910569 * mean) 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:12:07.792 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-03-30T15:12:07.793 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-03-30T15:12:07.793 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-03-30T15:12:07.793 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0011 2024-03-30T15:12:07.793 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0111 2024-03-30T15:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 10 librados: watch_flush enter 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 10 librados: watch_flush exit 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 20 client.58913.objecter shutdown clearing up homeless session... 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 10 client.58913.objecter successfully canceled tick 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe8880704d0 legacy=0x7fe888072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 >> v1:172.21.15.28:6790/0 conn(0x7fe89c101130 legacy=0x7fe89c0fce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 shutdown_connections 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 >> 172.21.15.28:0/663537943 conn(0x7fe89c0fc9e0 msgr2=0x7fe89c06a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 shutdown_connections 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 -- 172.21.15.28:0/663537943 wait complete. 2024-03-30T15:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:07.794+0000 7fe8a217a700 1 librados: shutdown 2024-03-30T15:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-03-30T15:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:07.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:07.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:07.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:08.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:08.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:08.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:08.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:08.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:08.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:12:08.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:08.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:08.472 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:08.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:08.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:08.619 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:08.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:08 smithi134 ceph-mon[33714]: from='client.58913 v1:172.21.15.28:0/663537943' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:08.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:08 smithi134 ceph-mon[33714]: pgmap v1235: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:08.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:08.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:08.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:08.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:08.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[29876]: from='client.58913 v1:172.21.15.28:0/663537943' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:08.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[29876]: pgmap v1235: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:08.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:08.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:08.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[41614]: from='client.58913 v1:172.21.15.28:0/663537943' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T15:12:08.919 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:08 smithi028 ceph-mon[41614]: pgmap v1235: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:08.923 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:08.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:08.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:08.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:09.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:09.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:09.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:09.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:09.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:09.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:09.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:09.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:09.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:09.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:09.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:09.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:09.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:10.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:10.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:10.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:10.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:10.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:10.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:10.348 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:10.348 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:10.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:10.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:10.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:10.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:10.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:10.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 -- 172.21.15.28:0/1844903288 >> v1:172.21.15.28:6789/0 conn(0x7f44c0109dc0 legacy=0x7f44c010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 -- 172.21.15.28:0/1844903288 shutdown_connections 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 -- 172.21.15.28:0/1844903288 >> 172.21.15.28:0/1844903288 conn(0x7f44c00fc9e0 msgr2=0x7f44c00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 -- 172.21.15.28:0/1844903288 shutdown_connections 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 -- 172.21.15.28:0/1844903288 wait complete. 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 librados: starting msgr at 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 librados: starting objecter 2024-03-30T15:12:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.719+0000 7f44c5734700 1 Processor -- start 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 -- start start 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 librados: setting wanted keys 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 librados: calling monclient init 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c01a9440 con 0x7f44c010d960 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c01aa660 con 0x7f44c0101130 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44c5734700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f44c01ab8a0 con 0x7f44c0109dc0 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44be7fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f44c0109dc0 0x7f44c01a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51240/0 (socket says 172.21.15.28:51240) 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44be7fc700 1 -- 172.21.15.28:0/1700512760 learned_addr learned my addr 172.21.15.28:0/1700512760 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44bf7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f44c010d960 0x7f44c01a7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57616/0 (socket says 172.21.15.28:57616) 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter ms_handle_connect 0x7f44c010d960 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter ms_handle_connect 0x7f44c0109dc0 2024-03-30T15:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter ms_handle_connect 0x7f44c0101130 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955662016 0 0) 0x7f44c01ab8a0 con 0x7f44c0109dc0 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44ac0031f0 con 0x7f44c0109dc0 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614122472 0 0) 0x7f44c01a9440 con 0x7f44c010d960 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c01ab8a0 con 0x7f44c010d960 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369953311 0 0) 0x7f44c01aa660 con 0x7f44c0101130 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.720+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c01a9440 con 0x7f44c0101130 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3302536371 0 0) 0x7f44ac0031f0 con 0x7f44c0109dc0 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c01aa660 con 0x7f44c0109dc0 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f44a8004540 con 0x7f44c0109dc0 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 167645383 0 0) 0x7f44c01ab8a0 con 0x7f44c010d960 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44ac0031f0 con 0x7f44c010d960 2024-03-30T15:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f44b0001af0 con 0x7f44c010d960 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3724745567 0 0) 0x7f44c01a9440 con 0x7f44c0101130 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c01ab8a0 con 0x7f44c0101130 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f44b8001890 con 0x7f44c0101130 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1095359098 0 0) 0x7f44c01aa660 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 >> v1:172.21.15.134:6789/0 conn(0x7f44c0101130 legacy=0x7f44c010f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 >> v1:172.21.15.28:6789/0 conn(0x7f44c010d960 legacy=0x7f44c01a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c01acae0 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44c01a9670 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44c5734700 10 client.58919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f44c01a9b30 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44c5734700 1 librados: init done 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f44a80019e0 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.721+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f44a8004ca0 con 0x7f44c0109dc0 2024-03-30T15:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.722+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a0005120 con 0x7f44c0109dc0 2024-03-30T15:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f44a8004f00 con 0x7f44c0109dc0 2024-03-30T15:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(813..813 src has 442..813) ==== 10278+0+0 (unknown 1027222629 0 0) 0x7f44a8089f30 con 0x7f44c0109dc0 2024-03-30T15:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 10 client.58919.objecter ms_dispatch 0x7f44c0001550 osd_map(813..813 src has 442..813) 2024-03-30T15:12:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 3 client.58919.objecter handle_osd_map got epochs [813,813] > 0 2024-03-30T15:12:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 3 client.58919.objecter handle_osd_map decoding full epoch 813 2024-03-30T15:12:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 20 client.58919.objecter dump_active .. 0 homeless 2024-03-30T15:12:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.723+0000 7f44b7fff700 10 client.58919.objecter ms_handle_connect 0x7f44ac070500 2024-03-30T15:12:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.728+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f44a8004ca0 con 0x7f44c0109dc0 2024-03-30T15:12:10.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:10.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:10.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:10.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:10.942+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f44a0005410 con 0x7f44c0109dc0 2024-03-30T15:12:10.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:12:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:10.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:11.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:11.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:11.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:11.199 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:11.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.215+0000 7f44b7fff700 1 -- 172.21.15.28:0/1700512760 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v814) ==== 91+0+0 (unknown 2382655380 0 0) 0x7f44a8056900 con 0x7f44c0109dc0 2024-03-30T15:12:11.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 10 librados: watch_flush enter 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 10 librados: watch_flush exit 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 20 client.58919.objecter shutdown clearing up homeless session... 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 10 client.58919.objecter successfully canceled tick 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 >> v1:172.21.15.28:6800/3011431093 conn(0x7f44ac070500 legacy=0x7f44ac0729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 >> v1:172.21.15.28:6790/0 conn(0x7f44c0109dc0 legacy=0x7f44c01a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:11.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 shutdown_connections 2024-03-30T15:12:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 >> 172.21.15.28:0/1700512760 conn(0x7f44c00fc9e0 msgr2=0x7f44c0069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 shutdown_connections 2024-03-30T15:12:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.219+0000 7f44c5734700 1 -- 172.21.15.28:0/1700512760 wait complete. 2024-03-30T15:12:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:11.220+0000 7f44c5734700 1 librados: shutdown 2024-03-30T15:12:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:12:11.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:11.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:11.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[41614]: pgmap v1236: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1700512760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[41614]: from='client.58919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[29876]: pgmap v1236: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1700512760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:11 smithi028 ceph-mon[29876]: from='client.58919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.480 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:11 smithi134 ceph-mon[33714]: pgmap v1236: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:11.480 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1700512760' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.480 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:11 smithi134 ceph-mon[33714]: from='client.58919 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:11.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:11.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:11.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:11.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:11.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:11.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:11.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:11.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:12.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:12.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:12.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:12.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:12.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:12.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:12 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:12.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:12 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:12.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:12 smithi134 ceph-mon[33714]: from='client.58919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:12.375 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:12 smithi134 ceph-mon[33714]: osdmap e814: 8 total, 6 up, 8 in 2024-03-30T15:12:12.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:12.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[29876]: from='client.58919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[29876]: osdmap e814: 8 total, 6 up, 8 in 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:12.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:12.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[41614]: from='client.58919 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:12.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:12 smithi028 ceph-mon[41614]: osdmap e814: 8 total, 6 up, 8 in 2024-03-30T15:12:12.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.544+0000 7f0e223a3700 1 -- 172.21.15.28:0/2025703450 >> v1:172.21.15.28:6790/0 conn(0x7f0e1c109dd0 legacy=0x7f0e1c10c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.544+0000 7f0e223a3700 1 -- 172.21.15.28:0/2025703450 shutdown_connections 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.544+0000 7f0e223a3700 1 -- 172.21.15.28:0/2025703450 >> 172.21.15.28:0/2025703450 conn(0x7f0e1c0fc9d0 msgr2=0x7f0e1c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.544+0000 7f0e223a3700 1 -- 172.21.15.28:0/2025703450 shutdown_connections 2024-03-30T15:12:12.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.544+0000 7f0e223a3700 1 -- 172.21.15.28:0/2025703450 wait complete. 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 librados: starting msgr at 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 librados: starting objecter 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 Processor -- start 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 -- start start 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 librados: setting wanted keys 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.545+0000 7f0e223a3700 1 librados: calling monclient init 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e223a3700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e1c1a70c0 con 0x7f0e1c109dd0 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e223a3700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e1c1a82e0 con 0x7f0e1c101140 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e223a3700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0e1c1a9520 con 0x7f0e1c10d970 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e1bfff700 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f0e1c101140 0x7f0e1c106480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:38526/0 (socket says 172.21.15.28:38526) 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e1bfff700 1 -- 172.21.15.28:0/3167986022 learned_addr learned my addr 172.21.15.28:0/3167986022 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e20940700 1 --1- 172.21.15.28:0/3167986022 >> v1:172.21.15.28:6790/0 conn(0x7f0e1c10d970 0x7f0e1c1a5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51268/0 (socket says 172.21.15.28:51268) 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter ms_handle_connect 0x7f0e1c109dd0 2024-03-30T15:12:12.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter ms_handle_connect 0x7f0e1c101140 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter ms_handle_connect 0x7f0e1c10d970 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479071939 0 0) 0x7f0e1c1a82e0 con 0x7f0e1c101140 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e040039c0 con 0x7f0e1c101140 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266485412 0 0) 0x7f0e1c1a70c0 con 0x7f0e1c109dd0 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1c1a82e0 con 0x7f0e1c109dd0 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867211467 0 0) 0x7f0e1c1a9520 con 0x7f0e1c10d970 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1c1a70c0 con 0x7f0e1c10d970 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 169324850 0 0) 0x7f0e040039c0 con 0x7f0e1c101140 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e1c1a9520 con 0x7f0e1c101140 2024-03-30T15:12:12.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e0c001930 con 0x7f0e1c101140 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3172198981 0 0) 0x7f0e1c1a82e0 con 0x7f0e1c109dd0 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e040039c0 con 0x7f0e1c109dd0 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1902073638 0 0) 0x7f0e1c1a70c0 con 0x7f0e1c10d970 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e1c1a82e0 con 0x7f0e1c10d970 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e10004450 con 0x7f0e1c109dd0 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e140015b0 con 0x7f0e1c10d970 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.546+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2786031232 0 0) 0x7f0e1c1a9520 con 0x7f0e1c101140 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 >> v1:172.21.15.28:6790/0 conn(0x7f0e1c10d970 legacy=0x7f0e1c1a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 >> v1:172.21.15.28:6789/0 conn(0x7f0e1c109dd0 legacy=0x7f0e1c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1c1aa760 con 0x7f0e1c101140 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1c1a8510 con 0x7f0e1c101140 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e223a3700 10 client.50920.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e1c1a89d0 con 0x7f0e1c101140 2024-03-30T15:12:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e223a3700 1 librados: init done 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0e0c003b40 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0e0c005d30 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.547+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0e08005100 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.548+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0e0c0036a0 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(815..815 src has 442..815) ==== 10278+0+0 (unknown 3200165032 0 0) 0x7f0e0c089eb0 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 osd_map(815..815 src has 442..815) 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 3 client.50920.objecter handle_osd_map got epochs [815,815] > 0 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 3 client.50920.objecter handle_osd_map decoding full epoch 815 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 20 client.50920.objecter dump_active .. 0 homeless 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v815) ==== 54+0+16 (unknown 481307735 0 2420000192) 0x7f0e0c08a2c0 con 0x7f0e1c101140 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e04070520 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 10 client.50920.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 1 2024-03-30T15:12:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 10 client.50920.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=816}) -- 0x7f0e080053e0 con 0x7f0e1c101140 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0e08000ed0 con 0x7f0e1c101140 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=815) ==== 24+0+0 (unknown 1740529625 0 0) 0x7f0e0c055f30 con 0x7f0e1c101140 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e1a7fc700 10 client.50920.objecter _check_command_map_dne tid 1 current 815 map_dne_bound 815 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e1a7fc700 10 client.50920.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e1a7fc700 15 client.50920.objecter _session_command_op_remove -1 1 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 10 client.50920.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 2 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 2 2024-03-30T15:12:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 1 2 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 4 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 10 client.50920.objecter _send_command 2 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.549+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f0e08004460 con 0x7f0e08003910 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e08003910 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e08004460 con 0x7f0e08003910 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 10 client.50920.objecter _finish_command 2 = system:0 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 4 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.550+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 1 2 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.567+0000 7f0e223a3700 10 client.50920.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 3 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e080011c0 osd=1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 4 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 10 client.50920.objecter _send_command 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.568+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e08004e10 con 0x7f0e08003910 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e08004e10 con 0x7f0e08003910 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e197fa700 10 client.50920.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e080011c0 osd=1 4 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 1 3 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e223a3700 10 client.50920.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 4 2024-03-30T15:12:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e223a3700 20 client.50920.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e223a3700 10 client.50920.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.569+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f0e08001ad0 con 0x7f0e1c101140 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=815) ==== 24+0+0 (unknown 2894229178 0 0) 0x7f0e0c0542a0 con 0x7f0e1c101140 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e19ffb700 10 client.50920.objecter _check_command_map_dne tid 4 current 815 map_dne_bound 815 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e19ffb700 10 client.50920.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e19ffb700 15 client.50920.objecter _session_command_op_remove -1 4 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 10 client.50920.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 5 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 5 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 3 5 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 4 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 10 client.50920.objecter _send_command 5 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0e080035c0 con 0x7f0e08002f60 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.570+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e08002f60 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.571+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e0c034e50 con 0x7f0e08002f60 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.571+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.571+0000 7f0e197fa700 10 client.50920.objecter _finish_command 5 = system:0 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.571+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 4 2024-03-30T15:12:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.571+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 3 5 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 10 client.50920.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 6 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 6 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08001c80 osd=3 3 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 3 6 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 4 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 10 client.50920.objecter _send_command 6 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.588+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e080029e0 con 0x7f0e08002f60 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e080029e0 con 0x7f0e08002f60 2024-03-30T15:12:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 10 client.50920.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08001c80 osd=3 4 2024-03-30T15:12:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 3 6 2024-03-30T15:12:12.594 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 10 client.50920.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 7 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 7 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 4 7 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 4 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 10 client.50920.objecter _send_command 7 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6800/3063775514 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0e08009980 con 0x7f0e0800e430 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.590+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e0800e430 2024-03-30T15:12:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.591+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.4 v1:172.21.15.134:6800/3063775514 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e08009980 con 0x7f0e0800e430 2024-03-30T15:12:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.591+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 7: 0 ) 2024-03-30T15:12:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.591+0000 7f0e197fa700 10 client.50920.objecter _finish_command 7 = system:0 2024-03-30T15:12:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.591+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 4 2024-03-30T15:12:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.591+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 4 7 2024-03-30T15:12:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:12.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 10 client.50920.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 8 2024-03-30T15:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 8 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e080091d0 osd=4 3 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 4 8 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 4 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 10 client.50920.objecter _send_command 8 2024-03-30T15:12:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.608+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6800/3063775514 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e08016880 con 0x7f0e0800e430 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.4 v1:172.21.15.134:6800/3063775514 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e08016880 con 0x7f0e0800e430 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 10 client.50920.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e080091d0 osd=4 4 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 4 8 2024-03-30T15:12:12.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 10 client.50920.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 9 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 9 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 5 9 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 4 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 10 client.50920.objecter _send_command 9 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0e0800e160 con 0x7f0e08008dc0 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.610+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e08008dc0 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.611+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e0800e160 con 0x7f0e08008dc0 2024-03-30T15:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.611+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 9: 0 ) 2024-03-30T15:12:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.611+0000 7f0e197fa700 10 client.50920.objecter _finish_command 9 = system:0 2024-03-30T15:12:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.611+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 4 2024-03-30T15:12:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.611+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 5 9 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 10 client.50920.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 10 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 10 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08008c50 osd=5 3 2024-03-30T15:12:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 5 10 2024-03-30T15:12:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 4 2024-03-30T15:12:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 10 client.50920.objecter _send_command 10 2024-03-30T15:12:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.628+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e0800d500 con 0x7f0e08008dc0 2024-03-30T15:12:12.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e0800d500 con 0x7f0e08008dc0 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e197fa700 10 client.50920.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08008c50 osd=5 4 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 5 10 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 10 client.50920.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 11 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 11 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 6 11 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 4 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 10 client.50920.objecter _send_command 11 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.629+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0e080178e0 con 0x7f0e0800d670 2024-03-30T15:12:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.630+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e0800d670 2024-03-30T15:12:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.631+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e080178e0 con 0x7f0e0800d670 2024-03-30T15:12:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.631+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 11: 0 ) 2024-03-30T15:12:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.631+0000 7f0e197fa700 10 client.50920.objecter _finish_command 11 = system:0 2024-03-30T15:12:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.631+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 4 2024-03-30T15:12:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.631+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 6 11 2024-03-30T15:12:12.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 10 client.50920.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 12 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 12 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e0800d500 osd=6 3 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 6 12 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 4 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 10 client.50920.objecter _send_command 12 2024-03-30T15:12:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.647+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e0801fb90 con 0x7f0e0800d670 2024-03-30T15:12:12.650 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:12.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e0801fb90 con 0x7f0e0800d670 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e197fa700 10 client.50920.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e0800d500 osd=6 4 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 6 12 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 10 client.50920.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 13 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 13 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 7 13 2024-03-30T15:12:12.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 4 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 10 client.50920.objecter _send_command 13 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.649+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0e08023da0 con 0x7f0e080200a0 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 10 client.50920.objecter ms_handle_connect 0x7f0e080200a0 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0e08023da0 con 0x7f0e080200a0 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 13: 0 ) 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 10 client.50920.objecter _finish_command 13 = system:0 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 4 2024-03-30T15:12:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.650+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 7 13 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 10 client.50920.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign -1 14 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 20 client.50920.objecter _get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_remove -1 14 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 20 client.50920.objecter get_session s=0x7f0e08017940 osd=7 3 2024-03-30T15:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 15 client.50920.objecter _session_command_op_assign 7 14 2024-03-30T15:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 4 2024-03-30T15:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 10 client.50920.objecter _send_command 14 2024-03-30T15:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.667+0000 7f0e223a3700 1 -- 172.21.15.28:0/3167986022 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0e08016c80 con 0x7f0e080200a0 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.668+0000 7f0e197fa700 1 -- 172.21.15.28:0/3167986022 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0e08016c80 con 0x7f0e080200a0 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.668+0000 7f0e197fa700 10 client.50920.objecter ms_dispatch 0x7f0e1c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.668+0000 7f0e197fa700 10 client.50920.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.668+0000 7f0e197fa700 20 client.50920.objecter put_session s=0x7f0e08017940 osd=7 4 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:12.668+0000 7f0e197fa700 15 client.50920.objecter _session_command_op_remove 7 14 2024-03-30T15:12:12.670 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:12.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:12.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:12.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:12.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:12.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:12.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57869.0:1 286.5s0 286:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 585.68574192999995, 2024-03-30T15:12:12.963 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0109918019999999, 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57869", 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/714138788", 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:12:12.964 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274495+0000", 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274498+0000", 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9220000000000001e-06 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.965 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274509+0000", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1678999999999999e-05 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274525+0000", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5781e-05 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.274540+0000", 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4808e-05 2024-03-30T15:12:12.966 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277431+0000", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028910849999999998 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:27.277436+0000", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4589999999999997e-06 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.282966+0000", 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0055296760000001 2024-03-30T15:12:12.967 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283097+0000", 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013078699999999999 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283531+0000", 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043440599999999997 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:12:12.968 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.283992+0000", 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046019300000000002 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285413+0000", 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014217050000000001 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:28.285487+0000", 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3300999999999995e-05 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.969 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.57923.0:1 290.7s0 290:fbfde494:::oid_MultiWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e404)", 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 564.69984154300005, 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.011345699, 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.57923", 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3909045240", 2024-03-30T15:12:12.970 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260395+0000", 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.971 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260397+0000", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.733e-06 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260404+0000", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.221e-06 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260415+0000", 2024-03-30T15:12:12.972 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0659000000000001e-05 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260425+0000", 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0703999999999999e-05 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260972+0000", 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054698500000000005 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:12.973 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:48.260976+0000", 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1020000000000001e-06 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269376+0000", 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.008400816 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269503+0000", 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126236 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.974 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.269916+0000", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000413743 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271218+0000", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013020810000000001 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271674+0000", 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00045557200000000001 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.975 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:02:49.271741+0000", 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6847000000000006e-05 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.6.52:31 101.d 101:b5bd8322:test-rados-api-smithi028-93253-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e520)", 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 379.63927773900002, 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050572109999999998, 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.976 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:12.977 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.977 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.977 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.977 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.978 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320961+0000", 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7549999999999999e-06 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320966+0000", 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8039999999999998e-06 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320979+0000", 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2858e-05 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.979 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320987+0000", 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1110000000000008e-06 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321177+0000", 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018962499999999999 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.980 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321306+0000", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012909299999999999 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321547+0000", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024123600000000001 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322529+0000", 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00098205800000000006 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322707+0000", 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000177629 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325919+0000", 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032115210000000002 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325973+0000", 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4014999999999997e-05 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.326016+0000", 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3506e-05 2024-03-30T15:12:12.983 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(42.b,QREQUEST,e:589,from:4,reservation_nonce:7)", 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 293.14600886900001, 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005843053, 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.984 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814230+0000", 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0260000000000002e-06 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.986 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814236+0000", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.913e-06 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814251+0000", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5138e-05 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814259+0000", 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2509999999999994e-06 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814321+0000", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1585000000000001e-05 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814327+0000", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3740000000000003e-06 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.819987+0000", 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005660389 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.988 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820005+0000", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7144000000000001e-05 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820071+0000", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6233000000000001e-05 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:14 323.1c 323:38000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 235.574282573, 2024-03-30T15:12:12.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035226210000000001, 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385956+0000", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3300000000000001e-06 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385963+0000", 2024-03-30T15:12:12.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2639999999999997e-06 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385971+0000", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7199999999999995e-06 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385979+0000", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8450000000000003e-06 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386035+0000", 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5769000000000001e-05 2024-03-30T15:12:12.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386039+0000", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3150000000000002e-06 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389352+0000", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033128849999999998 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389388+0000", 2024-03-30T15:12:12.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6260999999999997e-05 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389477+0000", 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8232000000000005e-05 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:48 323.1c 323:38000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:12:12.994 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 233.57420138800001, 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033340269999999998, 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 48 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:12:12.995 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386038+0000", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7089999999999998e-06 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386052+0000", 2024-03-30T15:12:12.996 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3910000000000001e-05 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386066+0000", 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3679000000000001e-05 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386075+0000", 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9979999999999992e-06 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386131+0000", 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6069000000000001e-05 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386135+0000", 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6399999999999996e-06 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389257+0000", 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031219440000000002 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389283+0000", 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5605e-05 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389369+0000", 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6472999999999997e-05 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:12.999 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:374 326.1 326:86d02a7e:test-rados-api-smithi028-93253-119::foo:3 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.58900372700001, 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098792810000000002, 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 374 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.000 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371235+0000", 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3199999999999998e-06 2024-03-30T15:12:13.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371242+0000", 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.72e-06 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371254+0000", 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2452999999999999e-05 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.002 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371267+0000", 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2567e-05 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371317+0000", 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9808999999999997e-05 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371321+0000", 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.279e-06 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.003 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373335+0000", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002014264 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373568+0000", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023216 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379648+0000", 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060801650000000002 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379884+0000", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023607599999999999 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381008+0000", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011244440000000001 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381112+0000", 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010402400000000001 2024-03-30T15:12:13.005 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:739 328.6 328:60000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 175.49166646899999, 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003540091, 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:12:13.006 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 739 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468573+0000", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1130000000000001e-06 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468584+0000", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0752e-05 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468605+0000", 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0743000000000001e-05 2024-03-30T15:12:13.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468617+0000", 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1994000000000001e-05 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.009 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468677+0000", 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9876000000000001e-05 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468681+0000", 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0949999999999998e-06 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.471982+0000", 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033012670000000001 2024-03-30T15:12:13.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472017+0000", 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4706999999999998e-05 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472110+0000", 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3543999999999997e-05 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.2.18:218 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.571706495, 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038503399999999998, 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388540+0000", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0299000000000001e-05 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388555+0000", 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4694e-05 2024-03-30T15:12:13.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388588+0000", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3259000000000001e-05 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388615+0000", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6579000000000001e-05 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388773+0000", 2024-03-30T15:12:13.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015847999999999999 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388786+0000", 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2490000000000001e-05 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392259+0000", 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034730429999999999 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.015 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392324+0000", 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5017000000000004e-05 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392381+0000", 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6478999999999999e-05 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.2.18:219 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:12:13.016 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.57158051799999, 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047187009999999996, 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:12:13.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388658+0000", 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.844e-06 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388670+0000", 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1576000000000001e-05 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388690+0000", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0233000000000001e-05 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388708+0000", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8253e-05 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388886+0000", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017824900000000001 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting_for_map not empty", 2024-03-30T15:12:13.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388923+0000", 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6492999999999998e-05 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392406+0000", 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003483112 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392456+0000", 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0176000000000002e-05 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.393375+0000", 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091876499999999999 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.1.13:98 322.7s0 322:e28744e7:test-rados-api-smithi028-93253-133::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e740)", 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 130.55855076700001, 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043215160000000001, 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.021 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401689+0000", 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3390000000000001e-06 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401704+0000", 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4541e-05 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401725+0000", 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0951e-05 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.023 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401740+0000", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5288e-05 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401813+0000", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3269999999999995e-05 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401819+0000", 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8479999999999997e-06 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.405904+0000", 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040851610000000003 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406007+0000", 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103118 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.1.13:99 322.7s0 322:e28744e7:test-rados-api-smithi028-93253-133::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e740)", 2024-03-30T15:12:13.025 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 130.55847309999999, 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043348759999999997, 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401765+0000", 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5300000000000002e-07 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401772+0000", 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3599999999999998e-06 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.027 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401779+0000", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401786+0000", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8889999999999996e-06 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401869+0000", 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3461999999999995e-05 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting_for_map not empty", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401874+0000", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1030000000000001e-06 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406025+0000", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041505989999999996 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406098+0000", 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3430000000000007e-05 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.029 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.58535.0:1796 335.18 e761/757 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head v 761'2, mlcod=0'0)", 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 109.50402504500001, 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010047844, 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:12:13.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456213+0000", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2669999999999999e-06 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458432+0000", 2024-03-30T15:12:13.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022190819999999998 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458445+0000", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3278000000000001e-05 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458458+0000", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2884000000000001e-05 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458514+0000", 2024-03-30T15:12:13.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6298000000000002e-05 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458539+0000", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4734000000000001e-05 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466218+0000", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076791109999999997 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466259+0000", 2024-03-30T15:12:13.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1189999999999997e-05 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3287 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94326471858816 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e780)", 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 88.991938610000005, 2024-03-30T15:12:13.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.48388409100000002, 2024-03-30T15:12:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.035 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:12:13.036 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3287 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968302+0000", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4709999999999999e-06 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968313+0000", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1367e-05 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968326+0000", 2024-03-30T15:12:13.038 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3147e-05 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968335+0000", 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9339999999999995e-06 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968415+0000", 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0068999999999995e-05 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:13.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968425+0000", 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6760000000000002e-06 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.449263+0000", 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.48083808 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.449643+0000", 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00038021300000000001 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452182+0000", 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025391340000000002 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.58535.0:2006 337.6 e780/773 337:602f83fe:::foo:head v 780'5, mlcod=779'2)", 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 88.534779955000005, 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062043699999999999, 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.041 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425458+0000", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2720000000000001e-06 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425485+0000", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6529e-05 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425496+0000", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1866e-05 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425504+0000", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4030000000000001e-06 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425562+0000", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7983999999999999e-05 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425583+0000", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1489999999999999e-05 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:13.044 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.431493+0000", 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059094819999999998 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.431661+0000", 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016834400000000001 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25087.0:3 42.f e781/114 42:f4b50b61:::1.obj:head v 781'1, mlcod=0'0)", 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 88.511720069000006, 2024-03-30T15:12:13.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005080513, 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448518+0000", 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0470000000000001e-06 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448524+0000", 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.754e-06 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448532+0000", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8199999999999997e-06 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448540+0000", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8690000000000004e-06 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452558+0000", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040181899999999996 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452577+0000", 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8814e-05 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453487+0000", 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091028099999999998 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453597+0000", 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109738 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34511.0:36 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e789)", 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 55.180334502999997, 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003981933, 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34511", 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3837265254", 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 36 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.050 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779905+0000", 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6759999999999999e-06 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779914+0000", 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3349999999999996e-06 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779931+0000", 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6429000000000001e-05 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779954+0000", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3750000000000001e-05 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.780027+0000", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2199000000000004e-05 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.780033+0000", 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2009999999999996e-06 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.783773+0000", 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037399569999999999 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.783884+0000", 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011138599999999999 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.054 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3408 5.e 5:7759931f:::notify.3:head [watch ping cookie 94326471855360 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 30.264870806000001, 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73455413700000005, 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3408 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695367+0000", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9999999999999996e-07 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695370+0000", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.427e-06 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695378+0000", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3620000000000006e-06 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695385+0000", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0809999999999999e-06 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695489+0000", 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104722 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695502+0000", 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2381000000000001e-05 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.427572+0000", 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73207046600000003 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.058 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.427901+0000", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032879799999999999 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429748+0000", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018470139999999999 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429832+0000", 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3953000000000001e-05 2024-03-30T15:12:13.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429920+0000", 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8133000000000005e-05 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3416 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94326471855360 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 30.264485039, 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73630579799999996, 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.060 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3416 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:12:13.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695753+0000", 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5239999999999999e-06 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695758+0000", 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2059999999999998e-06 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695769+0000", 2024-03-30T15:12:13.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0219e-05 2024-03-30T15:12:13.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695777+0000", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0919999999999998e-06 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695832+0000", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5680999999999998e-05 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695842+0000", 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0159999999999999e-05 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428004+0000", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73216151900000004 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428057+0000", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2581999999999999e-05 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429961+0000", 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001904642 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430042+0000", 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0393000000000001e-05 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.432057+0000", 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00201578 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3417 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94326471855360 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-03-30T15:12:13.068 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 30.264439861, 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73634551999999998, 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3417 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695798+0000", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.09e-07 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695801+0000", 2024-03-30T15:12:13.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2760000000000001e-06 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695818+0000", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7108999999999998e-05 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695825+0000", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6170000000000003e-06 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695876+0000", 2024-03-30T15:12:13.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1863000000000001e-05 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695887+0000", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0618999999999999e-05 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428149+0000", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73226165499999996 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428204+0000", 2024-03-30T15:12:13.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5049000000000001e-05 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430518+0000", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023138199999999999 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430600+0000", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2034999999999996e-05 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.432142+0000", 2024-03-30T15:12:13.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015426680000000001 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:12:13.074 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:12:13.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:13.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:13.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:13.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:13.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[29876]: pgmap v1238: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[29876]: osdmap e815: 8 total, 6 up, 8 in 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3167986022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[29876]: 337.f scrub starts 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[29876]: 337.f scrub ok 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[41614]: pgmap v1238: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[41614]: osdmap e815: 8 total, 6 up, 8 in 2024-03-30T15:12:13.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3167986022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:13.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[41614]: 337.f scrub starts 2024-03-30T15:12:13.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:13 smithi028 ceph-mon[41614]: 337.f scrub ok 2024-03-30T15:12:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:13.474 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:13.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:13.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:13.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:13.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:13 smithi134 ceph-mon[33714]: pgmap v1238: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:13 smithi134 ceph-mon[33714]: osdmap e815: 8 total, 6 up, 8 in 2024-03-30T15:12:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3167986022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:13 smithi134 ceph-mon[33714]: 337.f scrub starts 2024-03-30T15:12:13.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:13 smithi134 ceph-mon[33714]: 337.f scrub ok 2024-03-30T15:12:13.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:13.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:13.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:12:13.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:13.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:13.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:13.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:12:14.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:14.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:14.093 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:14.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:14.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:14.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:14.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 337.0 scrub starts 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 337.0 scrub ok 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 1.3 scrub starts 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 1.3 scrub ok 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 337.0 scrub starts 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 337.0 scrub ok 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 1.3 scrub starts 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 1.3 scrub ok 2024-03-30T15:12:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:12:14.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:14 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:12:14.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 337.0 scrub starts 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 337.0 scrub ok 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 1.3 scrub starts 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 1.3 scrub ok 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:12:14.462 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:14 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:12:14.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:14.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:14.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:14.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:14.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:14.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:14.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:15.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:15.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:15.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:15.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:15.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:15.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:12:15.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[41614]: 1.6 scrub starts 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[41614]: 1.6 scrub ok 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[41614]: pgmap v1240: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[41614]: 42.4 scrub starts 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[41614]: 42.4 scrub ok 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[29876]: 1.6 scrub starts 2024-03-30T15:12:15.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[29876]: 1.6 scrub ok 2024-03-30T15:12:15.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[29876]: pgmap v1240: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:15.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[29876]: 42.4 scrub starts 2024-03-30T15:12:15.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:15 smithi028 ceph-mon[29876]: 42.4 scrub ok 2024-03-30T15:12:15.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 2] live_osds: [1, 3, 4, 5, 6, 7] 2024-03-30T15:12:15.458 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:12:15.458 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [1, 3, 4, 5, 6, 7] 2024-03-30T15:12:15.459 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T15:12:15.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T15:12:15.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:12:15.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:15 smithi134 ceph-mon[33714]: 1.6 scrub starts 2024-03-30T15:12:15.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:15 smithi134 ceph-mon[33714]: 1.6 scrub ok 2024-03-30T15:12:15.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:15 smithi134 ceph-mon[33714]: pgmap v1240: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:15.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:15 smithi134 ceph-mon[33714]: 42.4 scrub starts 2024-03-30T15:12:15.489 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:15 smithi134 ceph-mon[33714]: 42.4 scrub ok 2024-03-30T15:12:15.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:15.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:15.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:15.795 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:15 smithi134 systemd[1]: Stopping Ceph osd.4 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:12:15.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:15.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:15.922 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:12:15.922 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:12:15.922 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:12:15.922 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:12:15.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:15.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.003+0000 7f99c0754700 1 -- 172.21.15.28:0/668416200 >> v1:172.21.15.28:6789/0 conn(0x7f99b810aed0 legacy=0x7f99b810d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.003+0000 7f99c0754700 1 -- 172.21.15.28:0/668416200 shutdown_connections 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.003+0000 7f99c0754700 1 -- 172.21.15.28:0/668416200 >> 172.21.15.28:0/668416200 conn(0x7f99b80fdaf0 msgr2=0x7f99b80fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.003+0000 7f99c0754700 1 -- 172.21.15.28:0/668416200 shutdown_connections 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.003+0000 7f99c0754700 1 -- 172.21.15.28:0/668416200 wait complete. 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 librados: starting msgr at 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 librados: starting objecter 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 Processor -- start 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 -- start start 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 librados: setting wanted keys 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 librados: calling monclient init 2024-03-30T15:12:16.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99b81a7180 con 0x7f99b8102240 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99b81a83a0 con 0x7f99b810ea70 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99c0754700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99b81a95e0 con 0x7f99b810aed0 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99be4f0700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99b8102240 0x7f99b8107610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57658/0 (socket says 172.21.15.28:57658) 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99be4f0700 1 -- 172.21.15.28:0/3399263160 learned_addr learned my addr 172.21.15.28:0/3399263160 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99bdcef700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f99b810aed0 0x7f99b81a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51286/0 (socket says 172.21.15.28:51286) 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99b77fe700 10 client.?.objecter ms_handle_connect 0x7f99b8102240 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99b77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99b77fe700 10 client.?.objecter ms_handle_connect 0x7f99b810aed0 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.004+0000 7f99b77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 10 client.?.objecter ms_handle_connect 0x7f99b810ea70 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55839144 0 0) 0x7f99b81a7180 con 0x7f99b8102240 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99a00039c0 con 0x7f99b8102240 2024-03-30T15:12:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 401133058 0 0) 0x7f99b81a95e0 con 0x7f99b810aed0 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b81a7180 con 0x7f99b810aed0 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625792046 0 0) 0x7f99b81a83a0 con 0x7f99b810ea70 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b81a95e0 con 0x7f99b810ea70 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2935943955 0 0) 0x7f99a00039c0 con 0x7f99b8102240 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99b81a83a0 con 0x7f99b8102240 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f99b00013d0 con 0x7f99b8102240 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 49315816 0 0) 0x7f99b81a7180 con 0x7f99b810aed0 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99a00039c0 con 0x7f99b810aed0 2024-03-30T15:12:16.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f99ac004480 con 0x7f99b810aed0 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3697448694 0 0) 0x7f99b81a95e0 con 0x7f99b810ea70 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99b81a7180 con 0x7f99b810ea70 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f99a8001c30 con 0x7f99b810ea70 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1129487012 0 0) 0x7f99b81a83a0 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 >> v1:172.21.15.28:6790/0 conn(0x7f99b810aed0 legacy=0x7f99b81a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 >> v1:172.21.15.134:6789/0 conn(0x7f99b810ea70 legacy=0x7f99b81a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b81aa820 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99b81a9810 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99c0754700 10 client.25728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99b81a9cd0 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99c0754700 1 librados: init done 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f99b0001680 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.005+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f99b0005d30 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.006+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a4005120 con 0x7f99b8102240 2024-03-30T15:12:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f99b001c650 con 0x7f99b8102240 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(815..815 src has 442..815) ==== 10278+0+0 (unknown 3200165032 0 0) 0x7f99b0089f90 con 0x7f99b8102240 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 10 client.25728.objecter ms_dispatch 0x7f99b8001550 osd_map(815..815 src has 442..815) 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 3 client.25728.objecter handle_osd_map got epochs [815,815] > 0 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 3 client.25728.objecter handle_osd_map decoding full epoch 815 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 20 client.25728.objecter dump_active .. 0 homeless 2024-03-30T15:12:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.007+0000 7f99b77fe700 10 client.25728.objecter ms_handle_connect 0x7f99a0070420 2024-03-30T15:12:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.012+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f99b0056960 con 0x7f99b8102240 2024-03-30T15:12:16.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:16.045 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:16.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:16.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:16.242+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f99a4005410 con 0x7f99b8102240 2024-03-30T15:12:16.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:16.309 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:16.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:16.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:16.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:12:16.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:16 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:12:16.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:16.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:12:16.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:12:16.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:12:16.607 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:12:16.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:12:16.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:16.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:16.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:16.881 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:16.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:16.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T15:12:16.812+0000 7f99c819c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:12:16.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T15:12:16.812+0000 7f99c819c700 -1 osd.4 815 *** Got signal Terminated *** 2024-03-30T15:12:16.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[39142]: 2024-03-30T15:12:16.812+0000 7f99c819c700 -1 osd.4 815 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:12:16.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:16.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:17.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:17.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:17.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.239+0000 7f99b77fe700 1 -- 172.21.15.28:0/3399263160 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v816) ==== 101+0+0 (unknown 2370321254 0 0) 0x7f99b005a3b0 con 0x7f99b8102240 2024-03-30T15:12:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.242+0000 7f99c0754700 10 librados: watch_flush enter 2024-03-30T15:12:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 10 librados: watch_flush exit 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 20 client.25728.objecter shutdown clearing up homeless session... 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 10 client.25728.objecter successfully canceled tick 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 >> v1:172.21.15.28:6800/3011431093 conn(0x7f99a0070420 legacy=0x7f99a00728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 >> v1:172.21.15.28:6789/0 conn(0x7f99b8102240 legacy=0x7f99b8107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 shutdown_connections 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 >> 172.21.15.28:0/3399263160 conn(0x7f99b80fdaf0 msgr2=0x7f99b8111f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 shutdown_connections 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.243+0000 7f99c0754700 1 -- 172.21.15.28:0/3399263160 wait complete. 2024-03-30T15:12:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:17.244+0000 7f99c0754700 1 librados: shutdown 2024-03-30T15:12:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:12:17.295 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:17 smithi134 ceph-mon[33714]: pgmap v1241: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:17.295 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:17.295 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:17 smithi134 ceph-mon[33714]: osd.4 marked itself down and dead 2024-03-30T15:12:17.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:17.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:17.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:17.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:17.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:17.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:17.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[41614]: pgmap v1241: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:17.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:17.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[41614]: osd.4 marked itself down and dead 2024-03-30T15:12:17.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[29876]: pgmap v1241: 246 pgs: 28 active+clean+remapped, 112 active+clean, 11 undersized+peered, 64 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:17.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:17.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:17 smithi028 ceph-mon[29876]: osd.4 marked itself down and dead 2024-03-30T15:12:17.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:17.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:17.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:17.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:17 smithi134 podman[153678]: 2024-03-30 15:12:17.655835305 +0000 UTC m=+0.928334979 container died bf53347284fcbcb853c2d1264ba507661d46e879df7f034023ed245d576c7767 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-30T15:12:17.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:17.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:17.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:18.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:18.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:18.238 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:18 smithi134 podman[153678]: 2024-03-30 15:12:18.131113247 +0000 UTC m=+1.403612965 container cleanup bf53347284fcbcb853c2d1264ba507661d46e879df7f034023ed245d576c7767 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2) 2024-03-30T15:12:18.239 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:18 smithi134 bash[153678]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T15:12:18.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:18.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:18.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:18.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: Health check update: 3 osds down (OSD_DOWN) 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: osdmap e816: 8 total, 5 up, 8 in 2024-03-30T15:12:18.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[29876]: osdmap e817: 8 total, 5 up, 8 in 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: Health check update: 3 osds down (OSD_DOWN) 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: osdmap e816: 8 total, 5 up, 8 in 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-mon[41614]: osdmap e817: 8 total, 5 up, 8 in 2024-03-30T15:12:18.527 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:18.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.524+0000 7f9f200f4700 1 -- 172.21.15.28:0/1465639327 >> v1:172.21.15.28:6790/0 conn(0x7f9f1810b030 legacy=0x7f9f1810d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:18.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.524+0000 7f9f200f4700 1 -- 172.21.15.28:0/1465639327 shutdown_connections 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.524+0000 7f9f200f4700 1 -- 172.21.15.28:0/1465639327 >> 172.21.15.28:0/1465639327 conn(0x7f9f180fc9d0 msgr2=0x7f9f180fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 -- 172.21.15.28:0/1465639327 shutdown_connections 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 -- 172.21.15.28:0/1465639327 wait complete. 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 librados: starting msgr at 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 librados: starting objecter 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 Processor -- start 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 -- start start 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 librados: setting wanted keys 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.525+0000 7f9f200f4700 1 librados: calling monclient init 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f200f4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9f181a7120 con 0x7f9f18103220 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f200f4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9f181a8340 con 0x7f9f1810eb40 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f200f4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9f181a9580 con 0x7f9f1810b030 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f1de90700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f18103220 0x7f9f1810a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57666/0 (socket says 172.21.15.28:57666) 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f1de90700 1 -- 172.21.15.28:0/2456595330 learned_addr learned my addr 172.21.15.28:0/2456595330 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f1d68f700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9f1810b030 0x7f9f181a1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51294/0 (socket says 172.21.15.28:51294) 2024-03-30T15:12:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter ms_handle_connect 0x7f9f1810b030 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter ms_handle_connect 0x7f9f18103220 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter ms_handle_connect 0x7f9f1810eb40 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848675336 0 0) 0x7f9f181a9580 con 0x7f9f1810b030 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f040031f0 con 0x7f9f1810b030 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504159838 0 0) 0x7f9f181a7120 con 0x7f9f18103220 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f181a9580 con 0x7f9f18103220 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980705682 0 0) 0x7f9f181a8340 con 0x7f9f1810eb40 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f181a7120 con 0x7f9f1810eb40 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 175632986 0 0) 0x7f9f040031f0 con 0x7f9f1810b030 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f181a8340 con 0x7f9f1810b030 2024-03-30T15:12:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9f08004540 con 0x7f9f1810b030 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3573812757 0 0) 0x7f9f181a9580 con 0x7f9f18103220 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f040031f0 con 0x7f9f18103220 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 34352499 0 0) 0x7f9f181a7120 con 0x7f9f1810eb40 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f181a9580 con 0x7f9f1810eb40 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9f14001870 con 0x7f9f18103220 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9f10001c10 con 0x7f9f1810eb40 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 260641678 0 0) 0x7f9f181a8340 con 0x7f9f1810b030 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 >> v1:172.21.15.134:6789/0 conn(0x7f9f1810eb40 legacy=0x7f9f181a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 >> v1:172.21.15.28:6789/0 conn(0x7f9f18103220 legacy=0x7f9f1810a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f181aa7c0 con 0x7f9f1810b030 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.526+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f181a7350 con 0x7f9f1810b030 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f200f4700 10 client.58937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f181a7810 con 0x7f9f1810b030 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f200f4700 1 librados: init done 2024-03-30T15:12:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9f080019e0 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9f08004cb0 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.527+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9efc005130 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.528+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9f08004f30 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(817..817 src has 442..817) ==== 9787+0+0 (unknown 2405136405 0 0) 0x7f9f08089e30 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 osd_map(817..817 src has 442..817) 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 3 client.58937.objecter handle_osd_map got epochs [817,817] > 0 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 3 client.58937.objecter handle_osd_map decoding full epoch 817 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 20 client.58937.objecter dump_active .. 0 homeless 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v817) ==== 54+0+16 (unknown 3681240185 0 2420000192) 0x7f9f0808a210 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9f04070530 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 10 client.58937.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 1 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 10 client.58937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=818}) -- 0x7f9efc005410 con 0x7f9f1810b030 2024-03-30T15:12:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9efc000e60 con 0x7f9f1810b030 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=817) ==== 24+0+0 (unknown 3750599079 0 0) 0x7f9f08056090 con 0x7f9f1810b030 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0ffff700 10 client.58937.objecter _check_command_map_dne tid 1 current 817 map_dne_bound 817 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0ffff700 10 client.58937.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0ffff700 15 client.58937.objecter _session_command_op_remove -1 1 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 10 client.58937.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 2 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 2 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 1 2 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 4 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 10 client.58937.objecter _send_command 2 2024-03-30T15:12:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f9efc002300 con 0x7f9efc0027c0 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.529+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9efc0027c0 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.530+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9efc002300 con 0x7f9efc0027c0 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.530+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.530+0000 7f9f0effd700 10 client.58937.objecter _finish_command 2 = system:0 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.530+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 4 2024-03-30T15:12:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.530+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 1 2 2024-03-30T15:12:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 10 client.58937.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc001be0 osd=1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 1 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 4 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 10 client.58937.objecter _send_command 3 2024-03-30T15:12:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.547+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9efc004c80 con 0x7f9efc0027c0 2024-03-30T15:12:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9efc004c80 con 0x7f9efc0027c0 2024-03-30T15:12:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f0effd700 10 client.58937.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc001be0 osd=1 4 2024-03-30T15:12:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 1 3 2024-03-30T15:12:18.551 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f200f4700 10 client.58937.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 4 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f200f4700 20 client.58937.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f200f4700 10 client.58937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.549+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f9efc004f10 con 0x7f9f1810b030 2024-03-30T15:12:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=817) ==== 24+0+0 (unknown 347187396 0 0) 0x7f9f08054400 con 0x7f9f1810b030 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f0f7fe700 10 client.58937.objecter _check_command_map_dne tid 4 current 817 map_dne_bound 817 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f0f7fe700 10 client.58937.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f0f7fe700 15 client.58937.objecter _session_command_op_remove -1 4 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 10 client.58937.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 5 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 5 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 3 5 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 4 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 10 client.58937.objecter _send_command 5 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9efc004970 con 0x7f9efc0043d0 2024-03-30T15:12:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.550+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9efc0043d0 2024-03-30T15:12:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.551+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9efc004970 con 0x7f9efc0043d0 2024-03-30T15:12:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.551+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.551+0000 7f9f0effd700 10 client.58937.objecter _finish_command 5 = system:0 2024-03-30T15:12:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.551+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 4 2024-03-30T15:12:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.551+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 3 5 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: Health check update: 3 osds down (OSD_DOWN) 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3399263160' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:18.567 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: osdmap e816: 8 total, 5 up, 8 in 2024-03-30T15:12:18.568 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:18 smithi134 ceph-mon[33714]: osdmap e817: 8 total, 5 up, 8 in 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 10 client.58937.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 6 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 6 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc003d50 osd=3 3 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 3 6 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 4 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 10 client.58937.objecter _send_command 6 2024-03-30T15:12:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.567+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9efc0037d0 con 0x7f9efc0043d0 2024-03-30T15:12:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9efc0037d0 con 0x7f9efc0043d0 2024-03-30T15:12:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 10 client.58937.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc003d50 osd=3 4 2024-03-30T15:12:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 3 6 2024-03-30T15:12:18.571 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 10 client.58937.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 7 2024-03-30T15:12:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 10 client.58937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=3) -- 0x7f9efc009310 con 0x7f9f1810b030 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== mon.2 v1:172.21.15.28:6790/0 12 ==== mon_get_version_reply(handle=3 version=817) ==== 24+0+0 (unknown 2935908938 0 0) 0x7f9f08055c20 con 0x7f9f1810b030 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0ffff700 10 client.58937.objecter _check_command_map_dne tid 7 current 817 map_dne_bound 817 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0ffff700 10 client.58937.objecter _finish_command 7 = osd:6 osd down 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f0ffff700 15 client.58937.objecter _session_command_op_remove -1 7 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 10 client.58937.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 8 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 8 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 5 8 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 4 2024-03-30T15:12:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 10 client.58937.objecter _send_command 8 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.569+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6804/3668151179 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f9efc012410 con 0x7f9efc008f00 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.570+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9efc008f00 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.571+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9efc012410 con 0x7f9efc008f00 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.571+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 8: 0 ) 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.571+0000 7f9f0effd700 10 client.58937.objecter _finish_command 8 = system:0 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.571+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 4 2024-03-30T15:12:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.571+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 5 8 2024-03-30T15:12:18.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 10 client.58937.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 9 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 9 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc008d90 osd=5 3 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 5 9 2024-03-30T15:12:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 4 2024-03-30T15:12:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 10 client.58937.objecter _send_command 9 2024-03-30T15:12:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.587+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9efc009830 con 0x7f9efc008f00 2024-03-30T15:12:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9efc009830 con 0x7f9efc008f00 2024-03-30T15:12:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f0effd700 10 client.58937.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc008d90 osd=5 4 2024-03-30T15:12:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 5 9 2024-03-30T15:12:18.591 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f200f4700 10 client.58937.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.589+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 10 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 10 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 6 10 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 4 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 10 client.58937.objecter _send_command 10 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6808/2135224194 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f9efc00e3c0 con 0x7f9efc0125c0 2024-03-30T15:12:18.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.590+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9efc0125c0 2024-03-30T15:12:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.591+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9efc00e3c0 con 0x7f9efc0125c0 2024-03-30T15:12:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.591+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 10: 0 ) 2024-03-30T15:12:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.591+0000 7f9f0effd700 10 client.58937.objecter _finish_command 10 = system:0 2024-03-30T15:12:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.591+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 4 2024-03-30T15:12:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.591+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 6 10 2024-03-30T15:12:18.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 10 client.58937.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 11 2024-03-30T15:12:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 11 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc009a40 osd=6 3 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 6 11 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 4 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 10 client.58937.objecter _send_command 11 2024-03-30T15:12:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.608+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9efc0139b0 con 0x7f9efc0125c0 2024-03-30T15:12:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9efc0139b0 con 0x7f9efc0125c0 2024-03-30T15:12:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f0effd700 10 client.58937.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc009a40 osd=6 4 2024-03-30T15:12:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 6 11 2024-03-30T15:12:18.612 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 10 client.58937.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 12 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 12 2024-03-30T15:12:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 7 12 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 4 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 10 client.58937.objecter _send_command 12 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.610+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6812/3758190298 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f9efc020020 con 0x7f9efc00d950 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.611+0000 7f9f0effd700 10 client.58937.objecter ms_handle_connect 0x7f9efc00d950 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.612+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9efc020020 con 0x7f9efc00d950 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.612+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 12: 0 ) 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.612+0000 7f9f0effd700 10 client.58937.objecter _finish_command 12 = system:0 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.612+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 4 2024-03-30T15:12:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.612+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 7 12 2024-03-30T15:12:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 10 client.58937.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign -1 13 2024-03-30T15:12:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 20 client.58937.objecter _get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_remove -1 13 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 20 client.58937.objecter get_session s=0x7f9efc0139b0 osd=7 3 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 15 client.58937.objecter _session_command_op_assign 7 13 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 4 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 10 client.58937.objecter _send_command 13 2024-03-30T15:12:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.629+0000 7f9f200f4700 1 -- 172.21.15.28:0/2456595330 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9efc00d760 con 0x7f9efc00d950 2024-03-30T15:12:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.631+0000 7f9f0effd700 1 -- 172.21.15.28:0/2456595330 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9efc00d760 con 0x7f9efc00d950 2024-03-30T15:12:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.631+0000 7f9f0effd700 10 client.58937.objecter ms_dispatch 0x7f9f18001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.631+0000 7f9f0effd700 10 client.58937.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.631+0000 7f9f0effd700 20 client.58937.objecter put_session s=0x7f9efc0139b0 osd=7 4 2024-03-30T15:12:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:18.631+0000 7f9f0effd700 15 client.58937.objecter _session_command_op_remove 7 13 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:18.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:18.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:18.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:18.915 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:18 smithi134 podman[153847]: 2024-03-30 15:12:18.80187874 +0000 UTC m=+1.145563429 container remove bf53347284fcbcb853c2d1264ba507661d46e879df7f034023ed245d576c7767 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T15:12:18.916 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:18 smithi134 bash[153960]: Error: no container with name or ID "ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4" found: no such container 2024-03-30T15:12:18.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:18.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:19.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:19.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:19.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:19.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:19.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:19.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:19.458 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:19.459 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: pgmap v1244: 246 pgs: 3 stale+active+undersized+degraded, 1 stale+undersized+peered, 11 stale+active+undersized, 25 stale+active+clean, 28 active+clean+remapped, 87 active+clean, 10 undersized+peered, 53 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:19.459 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2456595330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:19.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:19.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:19.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:19.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:19.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: pgmap v1244: 246 pgs: 3 stale+active+undersized+degraded, 1 stale+undersized+peered, 11 stale+active+undersized, 25 stale+active+clean, 28 active+clean+remapped, 87 active+clean, 10 undersized+peered, 53 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2456595330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:19.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:12:19.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:12:19.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:12:19.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:12:19.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: pgmap v1244: 246 pgs: 3 stale+active+undersized+degraded, 1 stale+undersized+peered, 11 stale+active+undersized, 25 stale+active+clean, 28 active+clean+remapped, 87 active+clean, 10 undersized+peered, 53 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 25 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 27/735 objects misplaced (3.673%) 2024-03-30T15:12:19.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2456595330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:19.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:19.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:19.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:19.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:19.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:12:20.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:20.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:12:20.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:20.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:20.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 1.7 scrub starts 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 1.7 scrub ok 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 1.9 scrub starts 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 1.9 scrub ok 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 42.18 deep-scrub starts 2024-03-30T15:12:20.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[29876]: 42.18 deep-scrub ok 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 1.7 scrub starts 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 1.7 scrub ok 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 1.9 scrub starts 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 1.9 scrub ok 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 42.18 deep-scrub starts 2024-03-30T15:12:20.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:20 smithi028 ceph-mon[41614]: 42.18 deep-scrub ok 2024-03-30T15:12:20.377 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:20 smithi134 podman[154198]: 2024-03-30 15:12:20.133906162 +0000 UTC m=+0.449544136 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:12:20.377 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:20 smithi134 podman[154198]: 2024-03-30 15:12:20.251409921 +0000 UTC m=+0.567047897 container create 851fad7ed0ad04a77158dff2c002c1e2c7310c15ad2b28f65684b020c09482ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-30T15:12:20.377 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 1.7 scrub starts 2024-03-30T15:12:20.377 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 1.7 scrub ok 2024-03-30T15:12:20.377 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 1.9 scrub starts 2024-03-30T15:12:20.377 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 1.9 scrub ok 2024-03-30T15:12:20.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 42.18 deep-scrub starts 2024-03-30T15:12:20.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-mon[33714]: 42.18 deep-scrub ok 2024-03-30T15:12:20.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:20.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:20.630 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:20 smithi134 podman[154198]: 2024-03-30 15:12:20.529411981 +0000 UTC m=+0.845049951 container init 851fad7ed0ad04a77158dff2c002c1e2c7310c15ad2b28f65684b020c09482ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-30T15:12:20.630 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:20 smithi134 podman[154198]: 2024-03-30 15:12:20.534795315 +0000 UTC m=+0.850433293 container start 851fad7ed0ad04a77158dff2c002c1e2c7310c15ad2b28f65684b020c09482ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T15:12:20.630 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:20 smithi134 podman[154198]: 2024-03-30 15:12:20.629184695 +0000 UTC m=+0.944822688 container attach 851fad7ed0ad04a77158dff2c002c1e2c7310c15ad2b28f65684b020c09482ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-30T15:12:20.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:20.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:20.761 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:20.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:20.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:20.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:20.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:20.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:21.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:21.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:21.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:21.180 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:21 smithi134 podman[154198]: 2024-03-30 15:12:21.011103232 +0000 UTC m=+1.326741196 container died 851fad7ed0ad04a77158dff2c002c1e2c7310c15ad2b28f65684b020c09482ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:12:21.197 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:21.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:21.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:21.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:12:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:12:21.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: pgmap v1245: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.9 KiB/s rd, 28 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 23 pgs inactive, 1 pg down (PG_AVAILABILITY) 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:12:21.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: pgmap v1245: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.9 KiB/s rd, 28 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 23 pgs inactive, 1 pg down (PG_AVAILABILITY) 2024-03-30T15:12:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:12:21.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:21 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:12:21.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:21.575 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: pgmap v1245: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.9 KiB/s rd, 28 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 23 pgs inactive, 1 pg down (PG_AVAILABILITY) 2024-03-30T15:12:21.670 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 72 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:21.671 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:12:21.671 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:21 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:12:21.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:21.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T15:12:21.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:21.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:12:21.770 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T15:12:21.770 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-03-30T15:12:21.770 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi134.front.sepia.ceph.com 2024-03-30T15:12:21.770 INFO:tasks.thrashosds.thrasher:remote for exp osd 2 is ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T15:12:21.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-30T15:12:21.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:21.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:21.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T15:12:22.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- 172.21.15.28:0/4205018044 >> v1:172.21.15.28:6789/0 conn(0x7f139c10ea80 legacy=0x7f139c110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- 172.21.15.28:0/4205018044 shutdown_connections 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- 172.21.15.28:0/4205018044 >> 172.21.15.28:0/4205018044 conn(0x7f139c0fdb20 msgr2=0x7f139c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- 172.21.15.28:0/4205018044 shutdown_connections 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- 172.21.15.28:0/4205018044 wait complete. 2024-03-30T15:12:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 librados: starting msgr at 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 librados: starting objecter 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 Processor -- start 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 -- start start 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 librados: setting wanted keys 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.142+0000 7f13a25b8700 1 librados: calling monclient init 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13a25b8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f139c107a20 con 0x7f139c102250 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13a25b8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f139c1a8650 con 0x7f139c10ea80 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13a25b8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f139c1a9830 con 0x7f139c10aee0 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f139b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f139c10aee0 0x7f139c104f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57516/0 (socket says 172.21.15.28:57516) 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f139b7fe700 1 -- 172.21.15.28:0/1515548883 learned_addr learned my addr 172.21.15.28:0/1515548883 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter ms_handle_connect 0x7f139c10aee0 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter ms_handle_connect 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter ms_handle_connect 0x7f139c10ea80 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 180704691 0 0) 0x7f139c107a20 con 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13840031f0 con 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667041853 0 0) 0x7f139c1a9830 con 0x7f139c10aee0 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f139c107a20 con 0x7f139c10aee0 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102035057 0 0) 0x7f139c1a8650 con 0x7f139c10ea80 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.143+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f139c1a9830 con 0x7f139c10ea80 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1364297847 0 0) 0x7f13840031f0 con 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f139c1a8650 con 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f138c0018b0 con 0x7f139c102250 2024-03-30T15:12:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1597975025 0 0) 0x7f139c107a20 con 0x7f139c10aee0 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13840031f0 con 0x7f139c10aee0 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3626790286 0 0) 0x7f139c1a9830 con 0x7f139c10ea80 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f139c107a20 con 0x7f139c10ea80 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1390001760 con 0x7f139c10aee0 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f13940017b0 con 0x7f139c10ea80 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 517760036 0 0) 0x7f139c1a8650 con 0x7f139c102250 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 >> v1:172.21.15.28:6790/0 conn(0x7f139c10aee0 legacy=0x7f139c104f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 >> v1:172.21.15.134:6789/0 conn(0x7f139c10ea80 legacy=0x7f139c1a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c1aaa10 con 0x7f139c102250 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f139c1a9a60 con 0x7f139c102250 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13a25b8700 10 client.25740.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f139c1a9f20 con 0x7f139c102250 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13a25b8700 1 librados: init done 2024-03-30T15:12:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f138c003b40 con 0x7f139c102250 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.144+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f138c004a00 con 0x7f139c102250 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.145+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1388005120 con 0x7f139c102250 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.145+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f138c01b320 con 0x7f139c102250 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(817..817 src has 442..817) ==== 9787+0+0 (unknown 2405136405 0 0) 0x7f138c088a70 con 0x7f139c102250 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 10 client.25740.objecter ms_dispatch 0x7f139c001550 osd_map(817..817 src has 442..817) 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 3 client.25740.objecter handle_osd_map got epochs [817,817] > 0 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 3 client.25740.objecter handle_osd_map decoding full epoch 817 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 20 client.25740.objecter dump_active .. 0 homeless 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.146+0000 7f13997fa700 10 client.25740.objecter ms_handle_connect 0x7f13840704b0 2024-03-30T15:12:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.150+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f138c055620 con 0x7f139c102250 2024-03-30T15:12:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:22.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:22.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:22.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:22.368+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1388005410 con 0x7f139c102250 2024-03-30T15:12:22.369 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:22.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[41614]: 3.2 scrub starts 2024-03-30T15:12:22.370 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[41614]: 3.2 scrub ok 2024-03-30T15:12:22.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:22.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[29876]: 3.2 scrub starts 2024-03-30T15:12:22.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:22 smithi028 ceph-mon[29876]: 3.2 scrub ok 2024-03-30T15:12:22.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:22.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:22.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:22.649 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:22 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:22.649 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:22 smithi134 ceph-mon[33714]: 3.2 scrub starts 2024-03-30T15:12:22.649 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:22 smithi134 ceph-mon[33714]: 3.2 scrub ok 2024-03-30T15:12:22.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:22.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:22.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:22.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:22.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:22.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:22.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:23.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:23.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:23.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:23.220 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:23.220 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.252+0000 7f13997fa700 1 -- 172.21.15.28:0/1515548883 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v818) ==== 87+0+0 (unknown 1552353921 0 0) 0x7f138c059070 con 0x7f139c102250 2024-03-30T15:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.255+0000 7f13a25b8700 10 librados: watch_flush enter 2024-03-30T15:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 10 librados: watch_flush exit 2024-03-30T15:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 20 client.25740.objecter shutdown clearing up homeless session... 2024-03-30T15:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 10 client.25740.objecter successfully canceled tick 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 >> v1:172.21.15.28:6800/3011431093 conn(0x7f13840704b0 legacy=0x7f1384072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 >> v1:172.21.15.28:6789/0 conn(0x7f139c102250 legacy=0x7f139c104870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 shutdown_connections 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 >> 172.21.15.28:0/1515548883 conn(0x7f139c0fdb20 msgr2=0x7f139c1116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 shutdown_connections 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.256+0000 7f13a25b8700 1 -- 172.21.15.28:0/1515548883 wait complete. 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:23.257+0000 7f13a25b8700 1 librados: shutdown 2024-03-30T15:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:12:23.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:23.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:23.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: 42.0 deep-scrub starts 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: 42.0 deep-scrub ok 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: pgmap v1246: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 3 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:12:23.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: 42.0 deep-scrub starts 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: 42.0 deep-scrub ok 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: pgmap v1246: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 3 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:23.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:23.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:23.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:23.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: 42.0 deep-scrub starts 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: 42.0 deep-scrub ok 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: pgmap v1246: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 3 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 20 KiB/s, 0 objects/s recovering 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:12:23.665 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:12:23.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:23.666 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:23.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:23.756 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:12:23.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:23.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:23.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:12:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:23.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:23.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:24.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:24.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:24.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:24.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:24.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:24.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:24.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:24.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:24.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:24.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:24.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:12:24.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:12:24.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:24.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[41614]: osdmap e818: 8 total, 5 up, 8 in 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:12:24.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:24 smithi028 ceph-mon[29876]: osdmap e818: 8 total, 5 up, 8 in 2024-03-30T15:12:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:24 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:12:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:24 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:12:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:24 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1515548883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:12:24.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:24 smithi134 ceph-mon[33714]: osdmap e818: 8 total, 5 up, 8 in 2024-03-30T15:12:24.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:24.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:24.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:24.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:25.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:25.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:25.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:25.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:25.357 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:25 smithi028 ceph-mon[41614]: pgmap v1248: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 32 KiB/s wr, 5 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 23 KiB/s, 0 objects/s recovering 2024-03-30T15:12:25.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:25 smithi028 ceph-mon[29876]: pgmap v1248: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 32 KiB/s wr, 5 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 23 KiB/s, 0 objects/s recovering 2024-03-30T15:12:25.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:25.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:25.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:25.582 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:25 smithi134 ceph-mon[33714]: pgmap v1248: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 32 KiB/s wr, 5 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 23 KiB/s, 0 objects/s recovering 2024-03-30T15:12:25.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:25.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:25.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:25.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:25.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:25.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:25.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:26.018 INFO:teuthology.orchestra.run.smithi028.stdout:6.f 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:6.e 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:6.8 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:337.15 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:5.14 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:5.1c 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:5.b 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:3.0 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:3.4 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:12.1d 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:337.7 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:322.0s2 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:42.1b 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:12.16 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:12.c 2024-03-30T15:12:26.019 INFO:teuthology.orchestra.run.smithi028.stdout:42.5 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.b 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.d 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.9 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.a 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.8 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:42.7 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:5.a 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:1.5 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:337.1a 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:6.1 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:12.0 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:6.0 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:3.5 2024-03-30T15:12:26.020 INFO:teuthology.orchestra.run.smithi028.stdout:3.9 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:1.a 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:3.1f 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:1.9 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:1.8 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:1.1 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:12.17 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:3.1e 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:5.15 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:5.1e 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:6.9 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:12.1f 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:337.10 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:337.c 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:337.11 2024-03-30T15:12:26.021 INFO:teuthology.orchestra.run.smithi028.stdout:6.12 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:12.f 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:12.5 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:12.7 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:3.1a 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:337.5 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:337.1d 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:337.8 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:4.4 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:6.1c 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:6.3 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:337.b 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:3.13 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:6.2 2024-03-30T15:12:26.022 INFO:teuthology.orchestra.run.smithi028.stdout:337.3 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.18 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:5.4 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:6.14 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:42.1e 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:2.1 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.19 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:5.8 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.1c 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.12 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.6 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:12.15 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.a 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:4.d 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:12.1c 2024-03-30T15:12:26.023 INFO:teuthology.orchestra.run.smithi028.stdout:42.10 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.1a 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.11 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.1d 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.2 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.c 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:5.6 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:42.1 2024-03-30T15:12:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:6.17 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- 172.21.15.28:0/2231594022 >> v1:172.21.15.28:6789/0 conn(0x7fefe010ea60 legacy=0x7fefe0110f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- 172.21.15.28:0/2231594022 shutdown_connections 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- 172.21.15.28:0/2231594022 >> 172.21.15.28:0/2231594022 conn(0x7fefe0077370 msgr2=0x7fefe00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- 172.21.15.28:0/2231594022 shutdown_connections 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- 172.21.15.28:0/2231594022 wait complete. 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 librados: starting msgr at 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 librados: starting objecter 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 Processor -- start 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 -- start start 2024-03-30T15:12:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 librados: setting wanted keys 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.043+0000 7fefe8738700 1 librados: calling monclient init 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefe8738700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fefe01078d0 con 0x7fefe010aec0 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefe8738700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fefe0107ae0 con 0x7fefe010ea60 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefe8738700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fefe01a9640 con 0x7fefe0100740 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefe64d4700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fefe0100740 0x7fefe0104770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57542/0 (socket says 172.21.15.28:57542) 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefe64d4700 1 -- 172.21.15.28:0/3056611310 learned_addr learned my addr 172.21.15.28:0/3056611310 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter ms_handle_connect 0x7fefe0100740 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter ms_handle_connect 0x7fefe010aec0 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter ms_handle_connect 0x7fefe010ea60 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427130115 0 0) 0x7fefe01078d0 con 0x7fefe010aec0 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc80031f0 con 0x7fefe010aec0 2024-03-30T15:12:26.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114143037 0 0) 0x7fefe01a9640 con 0x7fefe0100740 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe01078d0 con 0x7fefe0100740 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945088750 0 0) 0x7fefe0107ae0 con 0x7fefe010ea60 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe01a9640 con 0x7fefe010ea60 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2168329891 0 0) 0x7fefc80031f0 con 0x7fefe010aec0 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefe0107ae0 con 0x7fefe010aec0 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2508509668 0 0) 0x7fefe01078d0 con 0x7fefe0100740 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefc80031f0 con 0x7fefe0100740 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fefd4001830 con 0x7fefe010aec0 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 876899579 0 0) 0x7fefe01a9640 con 0x7fefe010ea60 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefe01078d0 con 0x7fefe010ea60 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fefd0001d30 con 0x7fefe0100740 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.044+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fefd80019a0 con 0x7fefe010ea60 2024-03-30T15:12:26.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2693905027 0 0) 0x7fefe0107ae0 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 >> v1:172.21.15.28:6790/0 conn(0x7fefe0100740 legacy=0x7fefe0104770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 >> v1:172.21.15.134:6789/0 conn(0x7fefe010ea60 legacy=0x7fefe01a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe01aa880 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fefe01a9870 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefe8738700 10 client.25746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fefe01a9d70 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefe8738700 1 librados: init done 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fefd4003b20 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fefd4004aa0 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.045+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fefcc005100 con 0x7fefe010aec0 2024-03-30T15:12:26.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.046+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fefd4003680 con 0x7fefe010aec0 2024-03-30T15:12:26.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(818..818 src has 442..818) ==== 9787+0+0 (unknown 1214060060 0 0) 0x7fefd4088bc0 con 0x7fefe010aec0 2024-03-30T15:12:26.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 osd_map(818..818 src has 442..818) 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 3 client.25746.objecter handle_osd_map got epochs [818,818] > 0 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 3 client.25746.objecter handle_osd_map decoding full epoch 818 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 20 client.25746.objecter dump_active .. 0 homeless 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefc8070470 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v818) ==== 54+0+16 (unknown 3294470018 0 2420000192) 0x7fefd4089060 con 0x7fefe010aec0 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.047+0000 7fefe8738700 10 client.25746.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 1 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 10 client.25746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=819}) -- 0x7fefcc0053e0 con 0x7fefe010aec0 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fefcc000e70 con 0x7fefe010aec0 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=818) ==== 24+0+0 (unknown 3430283604 0 0) 0x7fefd4054e40 con 0x7fefe010aec0 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe4cd1700 10 client.25746.objecter _check_command_map_dne tid 1 current 818 map_dne_bound 818 2024-03-30T15:12:26.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe4cd1700 10 client.25746.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe4cd1700 15 client.25746.objecter _session_command_op_remove -1 1 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 10 client.25746.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 2 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 2 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 1 2 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 4 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 10 client.25746.objecter _send_command 2 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fefcc002380 con 0x7fefcc0038d0 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.048+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefcc0038d0 2024-03-30T15:12:26.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.049+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fefcc002380 con 0x7fefcc0038d0 2024-03-30T15:12:26.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.049+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:26.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.049+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 2 = system:0 2024-03-30T15:12:26.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.049+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 4 2024-03-30T15:12:26.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.049+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 1 2 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 10 client.25746.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 3 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 3 2024-03-30T15:12:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc001bb0 osd=1 3 2024-03-30T15:12:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 1 3 2024-03-30T15:12:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 4 2024-03-30T15:12:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 10 client.25746.objecter _send_command 3 2024-03-30T15:12:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.066+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fefcc001750 con 0x7fefcc0038d0 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.067+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fefcc001750 con 0x7fefcc0038d0 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.067+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.067+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.067+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc001bb0 osd=1 4 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.067+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 1 3 2024-03-30T15:12:26.069 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 10 client.25746.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 4 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 10 client.25746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fefcc0019e0 con 0x7fefe010aec0 2024-03-30T15:12:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mon_get_version_reply(handle=2 version=818) ==== 24+0+0 (unknown 122275895 0 0) 0x7fefd40531b0 con 0x7fefe010aec0 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefdffff700 10 client.25746.objecter _check_command_map_dne tid 4 current 818 map_dne_bound 818 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefdffff700 10 client.25746.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefdffff700 15 client.25746.objecter _session_command_op_remove -1 4 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 10 client.25746.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 5 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 5 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 3 5 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 4 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 10 client.25746.objecter _send_command 5 2024-03-30T15:12:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fefcc003500 con 0x7fefcc002f20 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.068+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefcc002f20 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.069+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fefcc003500 con 0x7fefcc002f20 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.069+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.069+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 5 = system:0 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.069+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 4 2024-03-30T15:12:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.069+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 3 5 2024-03-30T15:12:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:26.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:26.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 10 client.25746.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:26.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 6 2024-03-30T15:12:26.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 6 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc004f70 osd=3 3 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 3 6 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 4 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 10 client.25746.objecter _send_command 6 2024-03-30T15:12:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.096+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fefcc0029a0 con 0x7fefcc002f20 2024-03-30T15:12:26.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fefcc0029a0 con 0x7fefcc002f20 2024-03-30T15:12:26.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:26.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:26.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc004f70 osd=3 4 2024-03-30T15:12:26.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 3 6 2024-03-30T15:12:26.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefe8738700 10 client.25746.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 7 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefe8738700 20 client.25746.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefe8738700 10 client.25746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.099+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=3) -- 0x7fefcc0091d0 con 0x7fefe010aec0 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== mon.0 v1:172.21.15.28:6789/0 12 ==== mon_get_version_reply(handle=3 version=818) ==== 24+0+0 (unknown 3171289785 0 0) 0x7fefd40549d0 con 0x7fefe010aec0 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe4cd1700 10 client.25746.objecter _check_command_map_dne tid 7 current 818 map_dne_bound 818 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe4cd1700 10 client.25746.objecter _finish_command 7 = osd:6 osd down 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe4cd1700 15 client.25746.objecter _session_command_op_remove -1 7 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 10 client.25746.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 8 2024-03-30T15:12:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 8 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 5 8 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 4 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 10 client.25746.objecter _send_command 8 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6804/3668151179 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fefcc012290 con 0x7fefcc008dc0 2024-03-30T15:12:26.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.100+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefcc008dc0 2024-03-30T15:12:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.101+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fefcc012290 con 0x7fefcc008dc0 2024-03-30T15:12:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.101+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 8: 0 ) 2024-03-30T15:12:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.101+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 8 = system:0 2024-03-30T15:12:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.101+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 4 2024-03-30T15:12:26.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.101+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 5 8 2024-03-30T15:12:26.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 10 client.25746.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 9 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 9 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc008c50 osd=5 3 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 5 9 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 4 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 10 client.25746.objecter _send_command 9 2024-03-30T15:12:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.119+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fefcc0096f0 con 0x7fefcc008dc0 2024-03-30T15:12:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fefcc0096f0 con 0x7fefcc008dc0 2024-03-30T15:12:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc008c50 osd=5 4 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 5 9 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 10 client.25746.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 10 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 10 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 6 10 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 4 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 10 client.25746.objecter _send_command 10 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.121+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6808/2135224194 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fefcc00e280 con 0x7fefcc012440 2024-03-30T15:12:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.122+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefcc012440 2024-03-30T15:12:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.123+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fefcc00e280 con 0x7fefcc012440 2024-03-30T15:12:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.123+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 10: 0 ) 2024-03-30T15:12:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.123+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 10 = system:0 2024-03-30T15:12:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.123+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 4 2024-03-30T15:12:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.123+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 6 10 2024-03-30T15:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 10 client.25746.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 11 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 11 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc009900 osd=6 3 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 6 11 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 4 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 10 client.25746.objecter _send_command 11 2024-03-30T15:12:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.140+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fefcc013830 con 0x7fefcc012440 2024-03-30T15:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fefcc013830 con 0x7fefcc012440 2024-03-30T15:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc009900 osd=6 4 2024-03-30T15:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 6 11 2024-03-30T15:12:26.144 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:26.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 10 client.25746.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:26.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 12 2024-03-30T15:12:26.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 12 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 7 12 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 4 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 10 client.25746.objecter _send_command 12 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.142+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6812/3758190298 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fefcc01fd70 con 0x7fefcc01c260 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 10 client.25746.objecter ms_handle_connect 0x7fefcc01c260 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fefcc01fd70 con 0x7fefcc01c260 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 12: 0 ) 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 12 = system:0 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 4 2024-03-30T15:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.143+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 7 12 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 10 client.25746.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign -1 13 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 20 client.25746.objecter _get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 15 client.25746.objecter _session_command_op_remove -1 13 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 20 client.25746.objecter get_session s=0x7fefcc013830 osd=7 3 2024-03-30T15:12:26.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 15 client.25746.objecter _session_command_op_assign 7 13 2024-03-30T15:12:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 4 2024-03-30T15:12:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 10 client.25746.objecter _send_command 13 2024-03-30T15:12:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.160+0000 7fefe8738700 1 -- 172.21.15.28:0/3056611310 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fefcc01fd70 con 0x7fefcc01c260 2024-03-30T15:12:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.161+0000 7fefdf7fe700 1 -- 172.21.15.28:0/3056611310 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fefcc01fd70 con 0x7fefcc01c260 2024-03-30T15:12:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.161+0000 7fefdf7fe700 10 client.25746.objecter ms_dispatch 0x7fefe0001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.161+0000 7fefdf7fe700 10 client.25746.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.161+0000 7fefdf7fe700 20 client.25746.objecter put_session s=0x7fefcc013830 osd=7 4 2024-03-30T15:12:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:26.161+0000 7fefdf7fe700 15 client.25746.objecter _session_command_op_remove 7 13 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:26.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:26.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:26.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:26.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:26.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3056611310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:26.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3056611310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:26.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:26.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:26.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:26.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:12:26.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3056611310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:26.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:26.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:26.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:26.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:26.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:26.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:26.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:27.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:27.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:27.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:27.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:27.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:27.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:27.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:27.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:27.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:27.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:27 smithi028 ceph-mon[41614]: pgmap v1249: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 25 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 18 KiB/s, 0 objects/s recovering 2024-03-30T15:12:27.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:27 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:12:27.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:27 smithi028 ceph-mon[29876]: pgmap v1249: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 25 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 18 KiB/s, 0 objects/s recovering 2024-03-30T15:12:27.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:27 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:12:27.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:27 smithi134 ceph-mon[33714]: pgmap v1249: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 25 KiB/s wr, 4 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%); 18 KiB/s, 0 objects/s recovering 2024-03-30T15:12:27.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:27 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:12:27.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:27.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:27.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:27.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:27.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:28.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:28.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:28.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:28.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:28.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:28.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:28.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:28.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:28.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:28.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:28.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:28 smithi028 ceph-mon[41614]: osdmap e819: 8 total, 5 up, 8 in 2024-03-30T15:12:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:28 smithi028 ceph-mon[41614]: pgmap v1251: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:28 smithi028 ceph-mon[29876]: osdmap e819: 8 total, 5 up, 8 in 2024-03-30T15:12:28.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:28 smithi028 ceph-mon[29876]: pgmap v1251: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:28.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:28.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:28.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 12.1d --file /var/log/ceph/exp.12.1d.2 2024-03-30T15:12:28.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:28.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:28 smithi134 ceph-mon[33714]: osdmap e819: 8 total, 5 up, 8 in 2024-03-30T15:12:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:28 smithi134 ceph-mon[33714]: pgmap v1251: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:28.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:29.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T15:12:29.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:29.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:29.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:29.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:29.341 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:12:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:12:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:12:29.341 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:12:29.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:29.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:12:29.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:29.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:29.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:29.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:29.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:12:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:29 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:29.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:29.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:29.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:29.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:29 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:12:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:29 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:12:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:29 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:29.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:30.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:30.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:30.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:30.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:30.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:30.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:12:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:30.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:30.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:30.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:30.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:30 smithi028 ceph-mon[41614]: pgmap v1252: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:30.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:30 smithi028 ceph-mon[29876]: pgmap v1252: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:30.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:30.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:30.832 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:30 smithi134 ceph-mon[33714]: pgmap v1252: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:30.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:30.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:30.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:31.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:31.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:31.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:31.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:31.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:31.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:12:31.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:31.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:31.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:31.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:31.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:31.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:31.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:31.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:31.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:31.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:31.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:31.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:32.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:Exporting 12.1d info 12.1d( empty local-lis/les=669/670 n=0 ec=110/110 lis/c=669/110 les/c/f=670/113/0 sis=669) 2024-03-30T15:12:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:Export successful 2024-03-30T15:12:32.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:32.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:32.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:32.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:32.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:32.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:32.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:32.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:32.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:32.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:32 smithi028 ceph-mon[41614]: pgmap v1253: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:32.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:32 smithi028 ceph-mon[29876]: pgmap v1253: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:32.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:32.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:32.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:32.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:32 smithi134 ceph-mon[33714]: pgmap v1253: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:33.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:33.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:33.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:33.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:33.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:33.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:33.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:33.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:33.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:33.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:33.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:33.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:33.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:33.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:33.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:34.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:34.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:34.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.2 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 12.1d 2024-03-30T15:12:34.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:34.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:34.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:34.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:34.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:34.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:34.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:34.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:34.657 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.2/config 2024-03-30T15:12:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:34.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:34.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:34.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:34.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:34.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:34.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:34.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:12:34.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: osdmap e820: 8 total, 5 up, 8 in 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[29876]: pgmap v1255: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: osdmap e820: 8 total, 5 up, 8 in 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:12:34.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:34 smithi028 ceph-mon[41614]: pgmap v1255: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:35.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:35.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:35.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:35.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: osdmap e820: 8 total, 5 up, 8 in 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:12:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:34 smithi134 ceph-mon[33714]: pgmap v1255: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:35.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:35.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:35.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:35.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:35.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:35.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:35.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:35.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:35.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:35.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:35.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:35 smithi028 ceph-mon[29876]: osdmap e821: 8 total, 5 up, 8 in 2024-03-30T15:12:35.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:35 smithi028 ceph-mon[41614]: osdmap e821: 8 total, 5 up, 8 in 2024-03-30T15:12:35.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:35.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:35.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:36.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:36.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:36.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:35 smithi134 ceph-mon[33714]: osdmap e821: 8 total, 5 up, 8 in 2024-03-30T15:12:36.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:36.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:36.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:36.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 -- 172.21.15.28:0/1847619185 >> v1:172.21.15.28:6790/0 conn(0x7f868c10aed0 legacy=0x7f868c10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 -- 172.21.15.28:0/1847619185 shutdown_connections 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 -- 172.21.15.28:0/1847619185 >> 172.21.15.28:0/1847619185 conn(0x7f868c0fdaf0 msgr2=0x7f868c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 -- 172.21.15.28:0/1847619185 shutdown_connections 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 -- 172.21.15.28:0/1847619185 wait complete. 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 librados: starting msgr at 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 librados: starting objecter 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.543+0000 7f8692e97700 1 Processor -- start 2024-03-30T15:12:36.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 -- start start 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 librados: setting wanted keys 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 librados: calling monclient init 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f868c076130 con 0x7f868c10aed0 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f868c076340 con 0x7f868c10ea70 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8692e97700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f868c1aba40 con 0x7f868c102240 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8690c33700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f868c102240 0x7f868c074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58696/0 (socket says 172.21.15.28:58696) 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8690c33700 1 -- 172.21.15.28:0/2268171207 learned_addr learned my addr 172.21.15.28:0/2268171207 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f868bfff700 1 --1- 172.21.15.28:0/2268171207 >> v1:172.21.15.28:6789/0 conn(0x7f868c10aed0 0x7f868c074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44276/0 (socket says 172.21.15.28:44276) 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter ms_handle_connect 0x7f868c10aed0 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter ms_handle_connect 0x7f868c102240 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:36.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter ms_handle_connect 0x7f868c10ea70 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825627455 0 0) 0x7f868c076130 con 0x7f868c10aed0 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86740035b0 con 0x7f868c10aed0 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693507816 0 0) 0x7f868c1aba40 con 0x7f868c102240 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f868c076130 con 0x7f868c102240 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174450003 0 0) 0x7f868c076340 con 0x7f868c10ea70 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.544+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f868c1aba40 con 0x7f868c10ea70 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 744319487 0 0) 0x7f868c076130 con 0x7f868c102240 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f868c076340 con 0x7f868c102240 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8684001bb0 con 0x7f868c102240 2024-03-30T15:12:36.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 780111929 0 0) 0x7f86740035b0 con 0x7f868c10aed0 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f868c076130 con 0x7f868c10aed0 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f86800015d0 con 0x7f868c10aed0 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1661194946 0 0) 0x7f868c1aba40 con 0x7f868c10ea70 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86740035b0 con 0x7f868c10ea70 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f867c001830 con 0x7f868c10ea70 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1018394978 0 0) 0x7f868c076340 con 0x7f868c102240 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 >> v1:172.21.15.134:6789/0 conn(0x7f868c10ea70 legacy=0x7f868c1a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 >> v1:172.21.15.28:6789/0 conn(0x7f868c10aed0 legacy=0x7f868c074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c1acc20 con 0x7f868c102240 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f868c1abc70 con 0x7f868c102240 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8692e97700 10 client.58955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f868c1ac130 con 0x7f868c102240 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8692e97700 1 librados: init done 2024-03-30T15:12:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.545+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8684003ec0 con 0x7f868c102240 2024-03-30T15:12:36.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.546+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8684005ff0 con 0x7f868c102240 2024-03-30T15:12:36.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8684006250 con 0x7f868c102240 2024-03-30T15:12:36.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8678005100 con 0x7f868c102240 2024-03-30T15:12:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(822..822 src has 442..822) ==== 9787+0+0 (unknown 3135382878 0 0) 0x7f8684089ef0 con 0x7f868c102240 2024-03-30T15:12:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 osd_map(822..822 src has 442..822) 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 3 client.58955.objecter handle_osd_map got epochs [822,822] > 0 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 3 client.58955.objecter handle_osd_map decoding full epoch 822 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 20 client.58955.objecter dump_active .. 0 homeless 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v822) ==== 54+0+16 (unknown 4025173286 0 2420000192) 0x7f868405edb0 con 0x7f868c102240 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.547+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f8674070480 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 10 client.58955.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 1 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 10 client.58955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=823}) -- 0x7f86780053e0 con 0x7f868c102240 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8678000e30 con 0x7f868c102240 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=822) ==== 24+0+0 (unknown 175199679 0 0) 0x7f8684056170 con 0x7f868c102240 2024-03-30T15:12:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f868affd700 10 client.58955.objecter _check_command_map_dne tid 1 current 822 map_dne_bound 822 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f868affd700 10 client.58955.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f868affd700 15 client.58955.objecter _session_command_op_remove -1 1 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 10 client.58955.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 2 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 2 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 1 2 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 4 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 10 client.58955.objecter _send_command 2 2024-03-30T15:12:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f8678002360 con 0x7f8678002790 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.548+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f8678002790 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.549+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8678002360 con 0x7f8678002790 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.549+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.549+0000 7f8689ffb700 10 client.58955.objecter _finish_command 2 = system:0 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.549+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 4 2024-03-30T15:12:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.549+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 1 2 2024-03-30T15:12:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 10 client.58955.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678001bb0 osd=1 3 2024-03-30T15:12:36.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 1 3 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 4 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 10 client.58955.objecter _send_command 3 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.567+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8678004cf0 con 0x7f8678002790 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.568+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8678004cf0 con 0x7f8678002790 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.568+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:36.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.568+0000 7f8689ffb700 10 client.58955.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.568+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678001bb0 osd=1 4 2024-03-30T15:12:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.568+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 1 3 2024-03-30T15:12:36.571 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 10 client.58955.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 4 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 10 client.58955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f8678001970 con 0x7f868c102240 2024-03-30T15:12:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=822) ==== 24+0+0 (unknown 3243074780 0 0) 0x7f86840544e0 con 0x7f868c102240 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f868a7fc700 10 client.58955.objecter _check_command_map_dne tid 4 current 822 map_dne_bound 822 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f868a7fc700 10 client.58955.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f868a7fc700 15 client.58955.objecter _session_command_op_remove -1 4 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 10 client.58955.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 5 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 5 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 3 5 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 4 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 10 client.58955.objecter _send_command 5 2024-03-30T15:12:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f86780049a0 con 0x7f8678004400 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.569+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f8678004400 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.570+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f86780049a0 con 0x7f8678004400 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.570+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.570+0000 7f8689ffb700 10 client.58955.objecter _finish_command 5 = system:0 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.570+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 4 2024-03-30T15:12:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.570+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 3 5 2024-03-30T15:12:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 10 client.58955.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 6 2024-03-30T15:12:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 6 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678003d80 osd=3 3 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 3 6 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 4 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 10 client.58955.objecter _send_command 6 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.587+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8678003800 con 0x7f8678004400 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8678003800 con 0x7f8678004400 2024-03-30T15:12:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8689ffb700 10 client.58955.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678003d80 osd=3 4 2024-03-30T15:12:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 3 6 2024-03-30T15:12:36.591 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8692e97700 10 client.58955.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 7 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8692e97700 20 client.58955.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8692e97700 10 client.58955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.588+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=3) -- 0x7f86780092e0 con 0x7f868c102240 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== mon.2 v1:172.21.15.28:6790/0 12 ==== mon_get_version_reply(handle=3 version=822) ==== 24+0+0 (unknown 2063771218 0 0) 0x7f8684055d00 con 0x7f868c102240 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f868affd700 10 client.58955.objecter _check_command_map_dne tid 7 current 822 map_dne_bound 822 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f868affd700 10 client.58955.objecter _finish_command 7 = osd:6 osd down 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f868affd700 15 client.58955.objecter _session_command_op_remove -1 7 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 10 client.58955.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 8 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 8 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 5 8 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 4 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 10 client.58955.objecter _send_command 8 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6804/3668151179 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f8678012320 con 0x7f8678008ed0 2024-03-30T15:12:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.589+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f8678008ed0 2024-03-30T15:12:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.590+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8678012320 con 0x7f8678008ed0 2024-03-30T15:12:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.590+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 8: 0 ) 2024-03-30T15:12:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.590+0000 7f8689ffb700 10 client.58955.objecter _finish_command 8 = system:0 2024-03-30T15:12:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.590+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 4 2024-03-30T15:12:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.590+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 5 8 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 10 client.58955.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 9 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 9 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678008d60 osd=5 3 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 5 9 2024-03-30T15:12:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 4 2024-03-30T15:12:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 10 client.58955.objecter _send_command 9 2024-03-30T15:12:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.607+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8678009800 con 0x7f8678008ed0 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8678009800 con 0x7f8678008ed0 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 10 client.58955.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678008d60 osd=5 4 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 5 9 2024-03-30T15:12:36.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 10 client.58955.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 10 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 10 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 6 10 2024-03-30T15:12:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 4 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 10 client.58955.objecter _send_command 10 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6808/2135224194 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f8678013040 con 0x7f86780124d0 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.609+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f86780124d0 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.610+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8678013040 con 0x7f86780124d0 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.610+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 10: 0 ) 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.610+0000 7f8689ffb700 10 client.58955.objecter _finish_command 10 = system:0 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.610+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 4 2024-03-30T15:12:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.610+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 6 10 2024-03-30T15:12:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 10 client.58955.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 11 2024-03-30T15:12:36.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 11 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678009a60 osd=6 3 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 6 11 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 4 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 10 client.58955.objecter _send_command 11 2024-03-30T15:12:36.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.627+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f86780139f0 con 0x7f86780124d0 2024-03-30T15:12:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f86780139f0 con 0x7f86780124d0 2024-03-30T15:12:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8689ffb700 10 client.58955.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678009a60 osd=6 4 2024-03-30T15:12:36.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 6 11 2024-03-30T15:12:36.632 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 10 client.58955.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 12 2024-03-30T15:12:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 12 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 7 12 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 4 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 10 client.58955.objecter _send_command 12 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.629+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6812/3758190298 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f867801bb00 con 0x7f8678017ee0 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.630+0000 7f8689ffb700 10 client.58955.objecter ms_handle_connect 0x7f8678017ee0 2024-03-30T15:12:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.631+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f867801bb00 con 0x7f8678017ee0 2024-03-30T15:12:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.631+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 12: 0 ) 2024-03-30T15:12:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.631+0000 7f8689ffb700 10 client.58955.objecter _finish_command 12 = system:0 2024-03-30T15:12:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.631+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 4 2024-03-30T15:12:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.631+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 7 12 2024-03-30T15:12:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 10 client.58955.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign -1 13 2024-03-30T15:12:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 20 client.58955.objecter _get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 15 client.58955.objecter _session_command_op_remove -1 13 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 20 client.58955.objecter get_session s=0x7f8678014b70 osd=7 3 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 15 client.58955.objecter _session_command_op_assign 7 13 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 4 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 10 client.58955.objecter _send_command 13 2024-03-30T15:12:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.647+0000 7f8692e97700 1 -- 172.21.15.28:0/2268171207 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f867801bac0 con 0x7f8678017ee0 2024-03-30T15:12:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.649+0000 7f8689ffb700 1 -- 172.21.15.28:0/2268171207 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f867801bac0 con 0x7f8678017ee0 2024-03-30T15:12:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.649+0000 7f8689ffb700 10 client.58955.objecter ms_dispatch 0x7f868c001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.649+0000 7f8689ffb700 10 client.58955.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.649+0000 7f8689ffb700 20 client.58955.objecter put_session s=0x7f8678014b70 osd=7 4 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:36.649+0000 7f8689ffb700 15 client.58955.objecter _session_command_op_remove 7 13 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-30T15:12:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:36.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:36.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:36.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:12:36.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:12:36.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[41614]: osdmap e822: 8 total, 5 up, 8 in 2024-03-30T15:12:36.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[41614]: pgmap v1258: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2268171207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[29876]: osdmap e822: 8 total, 5 up, 8 in 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[29876]: pgmap v1258: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:36.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2268171207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:36.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:36.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:36.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:36.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:37.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:37.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:37.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:36 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:12:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:36 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:12:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:36 smithi134 ceph-mon[33714]: osdmap e822: 8 total, 5 up, 8 in 2024-03-30T15:12:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:36 smithi134 ceph-mon[33714]: pgmap v1258: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2268171207' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:37.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:37.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.291+0000 7f156b0ce700 1 -- 172.21.15.28:0/44030123 >> v1:172.21.15.28:6790/0 conn(0x7f1564102240 legacy=0x7f1564102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.291+0000 7f156b0ce700 1 -- 172.21.15.28:0/44030123 shutdown_connections 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.291+0000 7f156b0ce700 1 -- 172.21.15.28:0/44030123 >> 172.21.15.28:0/44030123 conn(0x7f15640fdaf0 msgr2=0x7f15640fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.291+0000 7f156b0ce700 1 -- 172.21.15.28:0/44030123 shutdown_connections 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.291+0000 7f156b0ce700 1 -- 172.21.15.28:0/44030123 wait complete. 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 librados: starting msgr at 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 librados: starting objecter 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 Processor -- start 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 -- start start 2024-03-30T15:12:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 librados: setting wanted keys 2024-03-30T15:12:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 librados: calling monclient init 2024-03-30T15:12:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15641a70c0 con 0x7f1564102240 2024-03-30T15:12:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15641a82e0 con 0x7f156410aed0 2024-03-30T15:12:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f156b0ce700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15641a9520 con 0x7f156410ea70 2024-03-30T15:12:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1568e6a700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1564102240 0x7f1564107550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44288/0 (socket says 172.21.15.28:44288) 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1568e6a700 1 -- 172.21.15.28:0/989690992 learned_addr learned my addr 172.21.15.28:0/989690992 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter ms_handle_connect 0x7f156410ea70 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter ms_handle_connect 0x7f1564102240 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter ms_handle_connect 0x7f156410aed0 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188637301 0 0) 0x7f15641a9520 con 0x7f156410ea70 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15540039c0 con 0x7f156410ea70 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 691493194 0 0) 0x7f15641a70c0 con 0x7f1564102240 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15641a9520 con 0x7f1564102240 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677263036 0 0) 0x7f15641a82e0 con 0x7f156410aed0 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.292+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15641a70c0 con 0x7f156410aed0 2024-03-30T15:12:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1934075036 0 0) 0x7f15540039c0 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15641a82e0 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f155c001590 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2158445866 0 0) 0x7f15641a9520 con 0x7f1564102240 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15540039c0 con 0x7f1564102240 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753882207 0 0) 0x7f15641a70c0 con 0x7f156410aed0 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15641a9520 con 0x7f156410aed0 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f15580043e0 con 0x7f1564102240 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f15500017f0 con 0x7f156410aed0 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 136705590 0 0) 0x7f15641a82e0 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 >> v1:172.21.15.134:6789/0 conn(0x7f156410aed0 legacy=0x7f15641a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 >> v1:172.21.15.28:6789/0 conn(0x7f1564102240 legacy=0x7f1564107550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15641aa760 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15641a72f0 con 0x7f156410ea70 2024-03-30T15:12:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f155c003ea0 con 0x7f156410ea70 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f155c004e30 con 0x7f156410ea70 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f156b0ce700 10 client.58961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f15641a77b0 con 0x7f156410ea70 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.293+0000 7f156b0ce700 1 librados: init done 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.294+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1548005120 con 0x7f156410ea70 2024-03-30T15:12:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.294+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f155c004e30 con 0x7f156410ea70 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(822..822 src has 442..822) ==== 9787+0+0 (unknown 3135382878 0 0) 0x7f155c088cd0 con 0x7f156410ea70 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 10 client.58961.objecter ms_dispatch 0x7f1564001550 osd_map(822..822 src has 442..822) 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 3 client.58961.objecter handle_osd_map got epochs [822,822] > 0 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 3 client.58961.objecter handle_osd_map decoding full epoch 822 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 20 client.58961.objecter dump_active .. 0 homeless 2024-03-30T15:12:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.295+0000 7f1561ffb700 10 client.58961.objecter ms_handle_connect 0x7f15540704c0 2024-03-30T15:12:37.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.299+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f155c05db90 con 0x7f156410ea70 2024-03-30T15:12:37.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:37.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:37.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:37.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:37.524+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1548005410 con 0x7f156410ea70 2024-03-30T15:12:37.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:37.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:37.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:37.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:37.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 76 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:37.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: osdmap e823: 8 total, 5 up, 8 in 2024-03-30T15:12:37.789 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/989690992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: from='client.58961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 76 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: osdmap e823: 8 total, 5 up, 8 in 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/989690992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: from='client.58961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:37.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:37.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:37.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:37.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:37.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:37.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:37.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:38.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:38.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:38.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 76 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: osdmap e823: 8 total, 5 up, 8 in 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/989690992' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: from='client.58961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:12:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:38.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:38.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:38.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.437+0000 7f1561ffb700 1 -- 172.21.15.28:0/989690992 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v824) ==== 97+0+0 (unknown 1370181994 0 0) 0x7f155c053640 con 0x7f156410ea70 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.440+0000 7f156b0ce700 10 librados: watch_flush enter 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.440+0000 7f156b0ce700 10 librados: watch_flush exit 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.440+0000 7f156b0ce700 20 client.58961.objecter shutdown clearing up homeless session... 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.440+0000 7f156b0ce700 10 client.58961.objecter successfully canceled tick 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.441+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 >> v1:172.21.15.28:6800/3011431093 conn(0x7f15540704c0 legacy=0x7f1554072980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.441+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 >> v1:172.21.15.28:6790/0 conn(0x7f156410ea70 legacy=0x7f15641a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.441+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 shutdown_connections 2024-03-30T15:12:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.441+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 >> 172.21.15.28:0/989690992 conn(0x7f15640fdaf0 msgr2=0x7f156410eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.442+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 shutdown_connections 2024-03-30T15:12:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.442+0000 7f156b0ce700 1 -- 172.21.15.28:0/989690992 wait complete. 2024-03-30T15:12:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:38.442+0000 7f156b0ce700 1 librados: shutdown 2024-03-30T15:12:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:12:38.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:38.457 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:38.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:38.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:38.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:38.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:38.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:38.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:38.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:38.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:39.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:39.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:39.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[41614]: pgmap v1260: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:39.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:39.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[41614]: from='client.58961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:39.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[41614]: osdmap e824: 8 total, 5 up, 8 in 2024-03-30T15:12:39.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[29876]: pgmap v1260: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:39.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:39.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[29876]: from='client.58961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:39.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:38 smithi028 ceph-mon[29876]: osdmap e824: 8 total, 5 up, 8 in 2024-03-30T15:12:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:38 smithi134 ceph-mon[33714]: pgmap v1260: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:38 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:38 smithi134 ceph-mon[33714]: from='client.58961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:38 smithi134 ceph-mon[33714]: osdmap e824: 8 total, 5 up, 8 in 2024-03-30T15:12:39.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:39.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:39.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:39.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:39.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:39.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:39.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:39.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:12:39.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:39.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:39.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:39.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:12:39.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:39.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:39.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:39.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:40.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:40.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:40.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:40.199 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:40.199 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:40.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:40.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:40.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:40.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:40.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:40.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:40.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:40.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:40.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:40.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:40.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:12:41.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:41.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:41.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:12:41.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:41.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:41.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:41.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:41.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:41.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:41 smithi028 ceph-mon[29876]: pgmap v1262: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 943 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:41.519 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:41 smithi028 ceph-mon[41614]: pgmap v1262: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 943 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:41.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:41.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:41 smithi134 ceph-mon[33714]: pgmap v1262: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 943 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:41.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:41.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: marking collection for removal 2024-03-30T15:12:41.831 INFO:teuthology.orchestra.run.smithi028.stdout:setting '_remove' omap key 2024-03-30T15:12:41.831 INFO:teuthology.orchestra.run.smithi028.stdout:finish_remove_pgs 12.1d_head removing 12.1d 2024-03-30T15:12:41.832 INFO:teuthology.orchestra.run.smithi028.stdout:Remove successful 2024-03-30T15:12:41.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:41.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:41.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:41.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:42.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:42.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:42.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:42.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:42.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:42 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:42.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:42 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:42.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:42.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:42.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:42.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:42.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:42.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:42 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:42.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:42.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:42.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:42.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:43.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:43.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:43.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:43.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.4 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-03-30T15:12:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:43.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:43.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:43.331 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.4/config 2024-03-30T15:12:43.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:43.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:43 smithi134 ceph-mon[33714]: pgmap v1263: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:43.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:43 smithi134 ceph-mon[33714]: osdmap e825: 8 total, 5 up, 8 in 2024-03-30T15:12:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:43 smithi028 ceph-mon[29876]: pgmap v1263: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:43.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:43 smithi028 ceph-mon[29876]: osdmap e825: 8 total, 5 up, 8 in 2024-03-30T15:12:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:43 smithi028 ceph-mon[41614]: pgmap v1263: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 797 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:43.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:43 smithi028 ceph-mon[41614]: osdmap e825: 8 total, 5 up, 8 in 2024-03-30T15:12:43.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:43.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:43.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:43.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:43.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:43.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:43.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:44.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:44.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:44.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:44.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:44.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:44.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:44.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:44.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:44.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:44.603 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:12:44.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:44.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:44.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:44.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:44.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:45.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:45.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:45.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:45.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:45.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 -- 172.21.15.28:0/2090665955 >> v1:172.21.15.28:6790/0 conn(0x7fe5dc10eab0 legacy=0x7fe5dc110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 -- 172.21.15.28:0/2090665955 shutdown_connections 2024-03-30T15:12:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 -- 172.21.15.28:0/2090665955 >> 172.21.15.28:0/2090665955 conn(0x7fe5dc0fc9e0 msgr2=0x7fe5dc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 -- 172.21.15.28:0/2090665955 shutdown_connections 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 -- 172.21.15.28:0/2090665955 wait complete. 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 librados: starting msgr at 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.379+0000 7fe5e0e31700 1 librados: starting objecter 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 Processor -- start 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 -- start start 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 librados: setting wanted keys 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 librados: calling monclient init 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5dc10ab10 con 0x7fe5dc10afa0 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5dc10ad20 con 0x7fe5dc103190 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5e0e31700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe5dc1a9640 con 0x7fe5dc10eab0 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d9d9b700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe5dc10afa0 0x7fe5dc108070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53088/0 (socket says 172.21.15.28:53088) 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d9d9b700 1 -- 172.21.15.28:0/4190787127 learned_addr learned my addr 172.21.15.28:0/4190787127 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter ms_handle_connect 0x7fe5dc10afa0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter ms_handle_connect 0x7fe5dc10eab0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter ms_handle_connect 0x7fe5dc103190 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195123790 0 0) 0x7fe5dc10ab10 con 0x7fe5dc10afa0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5c80031f0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123130574 0 0) 0x7fe5dc1a9640 con 0x7fe5dc10eab0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.380+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5dc10ab10 con 0x7fe5dc10eab0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547302818 0 0) 0x7fe5dc10ad20 con 0x7fe5dc103190 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5dc1a9640 con 0x7fe5dc103190 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1639190110 0 0) 0x7fe5c80031f0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5dc10ad20 con 0x7fe5dc10afa0 2024-03-30T15:12:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe5c40018b0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 244101191 0 0) 0x7fe5dc10ab10 con 0x7fe5dc10eab0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5c80031f0 con 0x7fe5dc10eab0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe5d4004400 con 0x7fe5dc10eab0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3356711775 0 0) 0x7fe5dc1a9640 con 0x7fe5dc103190 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5dc10ab10 con 0x7fe5dc103190 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe5cc001bf0 con 0x7fe5dc103190 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1492112546 0 0) 0x7fe5dc10ad20 con 0x7fe5dc10afa0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 >> v1:172.21.15.28:6790/0 conn(0x7fe5dc10eab0 legacy=0x7fe5dc1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 >> v1:172.21.15.134:6789/0 conn(0x7fe5dc103190 legacy=0x7fe5dc107960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc1aa860 con 0x7fe5dc10afa0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc1a9870 con 0x7fe5dc10afa0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5e0e31700 10 client.25764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5dc1a9d30 con 0x7fe5dc10afa0 2024-03-30T15:12:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5e0e31700 1 librados: init done 2024-03-30T15:12:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe5c4003b40 con 0x7fe5dc10afa0 2024-03-30T15:12:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.381+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe5c4005ca0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.382+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe5bc005100 con 0x7fe5dc10afa0 2024-03-30T15:12:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe5c401c340 con 0x7fe5dc10afa0 2024-03-30T15:12:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(826..826 src has 442..826) ==== 9787+0+0 (unknown 2832139881 0 0) 0x7fe5c4089c30 con 0x7fe5dc10afa0 2024-03-30T15:12:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 osd_map(826..826 src has 442..826) 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 3 client.25764.objecter handle_osd_map got epochs [826,826] > 0 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 3 client.25764.objecter handle_osd_map decoding full epoch 826 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 20 client.25764.objecter dump_active .. 0 homeless 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v826) ==== 54+0+16 (unknown 2469407434 0 2420000192) 0x7fe5c4089ff0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5c8070370 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 1 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 20 client.25764.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 10 client.25764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=827}) -- 0x7fe5bc0053e0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.383+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe5bc000e30 con 0x7fe5dc10afa0 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_get_version_reply(handle=1 version=826) ==== 24+0+0 (unknown 1167119987 0 0) 0x7fe5c4055eb0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5d8d99700 10 client.25764.objecter _check_command_map_dne tid 1 current 826 map_dne_bound 826 2024-03-30T15:12:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5d8d99700 10 client.25764.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5d8d99700 15 client.25764.objecter _session_command_op_remove -1 1 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 2 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 2 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 1 2 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 4 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 10 client.25764.objecter _send_command 2 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fe5bc0022f0 con 0x7fe5bc002790 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.384+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5bc002790 2024-03-30T15:12:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.385+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5bc0022f0 con 0x7fe5bc002790 2024-03-30T15:12:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.385+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.385+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 2 = system:0 2024-03-30T15:12:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.385+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 4 2024-03-30T15:12:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.385+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 1 2 2024-03-30T15:12:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 3 2024-03-30T15:12:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 3 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc001bb0 osd=1 3 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 1 3 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 4 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 10 client.25764.objecter _send_command 3 2024-03-30T15:12:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.402+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe5bc004c70 con 0x7fe5bc002790 2024-03-30T15:12:45.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.403+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5bc004c70 con 0x7fe5bc002790 2024-03-30T15:12:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.403+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.403+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.403+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc001bb0 osd=1 4 2024-03-30T15:12:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.403+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 1 3 2024-03-30T15:12:45.406 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 4 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 10 client.25764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fe5bc004f00 con 0x7fe5dc10afa0 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 11 ==== mon_get_version_reply(handle=2 version=826) ==== 24+0+0 (unknown 2393696016 0 0) 0x7fe5c4054220 con 0x7fe5dc10afa0 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5d3fff700 10 client.25764.objecter _check_command_map_dne tid 4 current 826 map_dne_bound 826 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5d3fff700 10 client.25764.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5d3fff700 15 client.25764.objecter _session_command_op_remove -1 4 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 5 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 5 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 3 5 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 4 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 10 client.25764.objecter _send_command 5 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe5bc0048a0 con 0x7fe5bc004240 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.404+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5bc004240 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.405+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5bc0048a0 con 0x7fe5bc004240 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.405+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.405+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 5 = system:0 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.405+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 4 2024-03-30T15:12:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.405+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 3 5 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 6 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 6 2024-03-30T15:12:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc001a00 osd=3 3 2024-03-30T15:12:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 3 6 2024-03-30T15:12:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 4 2024-03-30T15:12:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 10 client.25764.objecter _send_command 6 2024-03-30T15:12:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.422+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe5bc003780 con 0x7fe5bc004240 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.423+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5bc003780 con 0x7fe5bc004240 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.423+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.423+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.423+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc001a00 osd=3 4 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.423+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 3 6 2024-03-30T15:12:45.425 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 7 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 10 client.25764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=3) -- 0x7fe5bc0092e0 con 0x7fe5dc10afa0 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== mon.0 v1:172.21.15.28:6789/0 12 ==== mon_get_version_reply(handle=3 version=826) ==== 24+0+0 (unknown 887301534 0 0) 0x7fe5c4055a40 con 0x7fe5dc10afa0 2024-03-30T15:12:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5d8d99700 10 client.25764.objecter _check_command_map_dne tid 7 current 826 map_dne_bound 826 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5d8d99700 10 client.25764.objecter _finish_command 7 = osd:6 osd down 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5d8d99700 15 client.25764.objecter _session_command_op_remove -1 7 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 8 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 8 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 5 8 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 4 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 10 client.25764.objecter _send_command 8 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.424+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6804/3668151179 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fe5bc0122e0 con 0x7fe5bc008ed0 2024-03-30T15:12:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.425+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5bc008ed0 2024-03-30T15:12:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.426+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5bc0122e0 con 0x7fe5bc008ed0 2024-03-30T15:12:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.426+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 8: 0 ) 2024-03-30T15:12:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.426+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 8 = system:0 2024-03-30T15:12:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.426+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 4 2024-03-30T15:12:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.426+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 5 8 2024-03-30T15:12:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:12:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:12:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:45 smithi134 ceph-mon[33714]: pgmap v1265: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:45.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:45 smithi134 ceph-mon[33714]: osdmap e826: 8 total, 5 up, 8 in 2024-03-30T15:12:45.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 9 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 9 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc008d60 osd=5 3 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 5 9 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 4 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 10 client.25764.objecter _send_command 9 2024-03-30T15:12:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.442+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe5bc009800 con 0x7fe5bc008ed0 2024-03-30T15:12:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5bc009800 con 0x7fe5bc008ed0 2024-03-30T15:12:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc008d60 osd=5 4 2024-03-30T15:12:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 5 9 2024-03-30T15:12:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 10 2024-03-30T15:12:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 10 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 6 10 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 4 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 10 client.25764.objecter _send_command 10 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.445+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6808/2135224194 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fe5bc013000 con 0x7fe5bc012490 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.446+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5bc012490 2024-03-30T15:12:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.447+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5bc013000 con 0x7fe5bc012490 2024-03-30T15:12:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.447+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 10: 0 ) 2024-03-30T15:12:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.447+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 10 = system:0 2024-03-30T15:12:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.447+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 4 2024-03-30T15:12:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.447+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 6 10 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 11 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 11 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc009a10 osd=6 3 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 6 11 2024-03-30T15:12:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 4 2024-03-30T15:12:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 10 client.25764.objecter _send_command 11 2024-03-30T15:12:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.463+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe5bc0139b0 con 0x7fe5bc012490 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5bc0139b0 con 0x7fe5bc012490 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc009a10 osd=6 4 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 6 11 2024-03-30T15:12:45.467 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 12 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.465+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 12 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 7 12 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 4 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 10 client.25764.objecter _send_command 12 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6812/3758190298 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fe5bc01b9b0 con 0x7fe5bc017ea0 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.466+0000 7fe5d37fe700 10 client.25764.objecter ms_handle_connect 0x7fe5bc017ea0 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.467+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5bc01b9b0 con 0x7fe5bc017ea0 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.467+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 12: 0 ) 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.467+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 12 = system:0 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.467+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 4 2024-03-30T15:12:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.467+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 7 12 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 10 client.25764.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign -1 13 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 20 client.25764.objecter _get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_remove -1 13 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 20 client.25764.objecter get_session s=0x7fe5bc0139b0 osd=7 3 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 15 client.25764.objecter _session_command_op_assign 7 13 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 4 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 10 client.25764.objecter _send_command 13 2024-03-30T15:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.483+0000 7fe5e0e31700 1 -- 172.21.15.28:0/4190787127 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe5bc013430 con 0x7fe5bc017ea0 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.485+0000 7fe5d37fe700 1 -- 172.21.15.28:0/4190787127 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe5bc013430 con 0x7fe5bc017ea0 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.485+0000 7fe5d37fe700 10 client.25764.objecter ms_dispatch 0x7fe5dc001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.485+0000 7fe5d37fe700 10 client.25764.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.485+0000 7fe5d37fe700 20 client.25764.objecter put_session s=0x7fe5bc0139b0 osd=7 4 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:45.485+0000 7fe5d37fe700 15 client.25764.objecter _session_command_op_remove 7 13 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[41614]: pgmap v1265: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[41614]: osdmap e826: 8 total, 5 up, 8 in 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:12:45.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:12:45.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[29876]: pgmap v1265: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:45.490 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:45 smithi028 ceph-mon[29876]: osdmap e826: 8 total, 5 up, 8 in 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.f 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.e 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.6 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.1e 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.1d 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.1a 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.17 2024-03-30T15:12:45.521 INFO:teuthology.orchestra.run.smithi134.stdout:6.16 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:6.15 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.f 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.d 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.a 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.9 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.6 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:6.7 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:6.12 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.1f 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.1c 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.1a 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.1 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.18 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:5.14 2024-03-30T15:12:45.522 INFO:teuthology.orchestra.run.smithi134.stdout:322.1s0 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:12.11 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.a 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.3 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:5.0 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.1d 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.b 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.6 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.1b 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:42.1f 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:4.1a 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.1a 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:12.2 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.11 2024-03-30T15:12:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:3.8 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:6.14 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:3.14 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:3.1c 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:12.f 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:1.2 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:4.18 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:1.5 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:12.0 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:337.a 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:337.13 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:5.13 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:3.9 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:4.1 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:322.5s2 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:4.b 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:4.e 2024-03-30T15:12:45.524 INFO:teuthology.orchestra.run.smithi134.stdout:322.2s0 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.1f 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:5.1d 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.16 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.18 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.1a 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:3.f 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.1b 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.1e 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:5.c 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.1c 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:42.10 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:3.e 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:12.e 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:337.1f 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:3.15 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:337.1e 2024-03-30T15:12:45.525 INFO:teuthology.orchestra.run.smithi134.stdout:337.6 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.d 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:5.5 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:337.1d 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:337.8 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:337.d 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.13 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.1b 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:42.1a 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.c 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.1d 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:5.10 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.f 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:3.13 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:42.14 2024-03-30T15:12:45.526 INFO:teuthology.orchestra.run.smithi134.stdout:4.5 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.15 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.16 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.17 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:4.9 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:1.3 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.1b 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.1d 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:1.6 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.1e 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:6.4 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.9 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.4 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.6 2024-03-30T15:12:45.527 INFO:teuthology.orchestra.run.smithi134.stdout:42.b 2024-03-30T15:12:45.528 INFO:teuthology.orchestra.run.smithi134.stdout:42.c 2024-03-30T15:12:45.528 INFO:teuthology.orchestra.run.smithi134.stdout:42.f 2024-03-30T15:12:45.528 INFO:teuthology.orchestra.run.smithi134.stdout:322.4s0 2024-03-30T15:12:45.528 INFO:teuthology.orchestra.run.smithi134.stdout:4.2 2024-03-30T15:12:45.528 INFO:teuthology.orchestra.run.smithi134.stdout:5.11 2024-03-30T15:12:45.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:45.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:45.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:45.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:45.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:45.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:45.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:45.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:46.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:46.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:46.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:46.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:46.331 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:46 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4190787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:46.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:46.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:46.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:46 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4190787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:46.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:46 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4190787127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:46.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:46.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:46.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:46.686 INFO:tasks.thrashosds.thrasher:Moving pg 12.1d from osd.2 to osd.4 2024-03-30T15:12:46.686 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi028.front.sepia.ceph.com to ubuntu@smithi134.front.sepia.ceph.com 2024-03-30T15:12:46.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-03-30T15:12:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:46.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-03-30T15:12:46.762 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-03-30T15:12:46.776 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.Ln0qRBwH1W 2024-03-30T15:12:46.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.12.1d.2 /tmp/tmp.Ln0qRBwH1W 2024-03-30T15:12:46.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:46.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:46.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.Ln0qRBwH1W 2024-03-30T15:12:46.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:46.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:46.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:46.981 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.Ln0qRBwH1W is 11KB 2024-03-30T15:12:47.030 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.Ln0qRBwH1W 2024-03-30T15:12:47.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:47.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv /var/log/ceph/exp.12.1d.2 /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.12.1d.2 2024-03-30T15:12:47.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell -n osd.4 --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.12.1d.2 2024-03-30T15:12:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:47.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:47.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:47.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:47.313 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/osd.4/config 2024-03-30T15:12:47.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:47.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:47.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:47 smithi134 ceph-mon[33714]: pgmap v1267: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.387+0000 7fd8366b8700 1 -- 172.21.15.28:0/229370406 >> v1:172.21.15.28:6790/0 conn(0x7fd830102250 legacy=0x7fd830102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.387+0000 7fd8366b8700 1 -- 172.21.15.28:0/229370406 shutdown_connections 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.387+0000 7fd8366b8700 1 -- 172.21.15.28:0/229370406 >> 172.21.15.28:0/229370406 conn(0x7fd8300fdae0 msgr2=0x7fd8300fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.387+0000 7fd8366b8700 1 -- 172.21.15.28:0/229370406 shutdown_connections 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.387+0000 7fd8366b8700 1 -- 172.21.15.28:0/229370406 wait complete. 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 librados: starting msgr at 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 librados: starting objecter 2024-03-30T15:12:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 Processor -- start 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 -- start start 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 librados: setting wanted keys 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 librados: calling monclient init 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd830107940 con 0x7fd83010ea80 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd830107b50 con 0x7fd83010aee0 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd8366b8700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8301a96a0 con 0x7fd830102250 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.388+0000 7fd82ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd830102250 0x7fd830104790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:44798/0 (socket says 172.21.15.28:44798) 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82ffff700 1 -- 172.21.15.28:0/1609773023 learned_addr learned my addr 172.21.15.28:0/1609773023 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd834c55700 1 --1- 172.21.15.28:0/1609773023 >> v1:172.21.15.28:6789/0 conn(0x7fd83010ea80 0x7fd8301a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53116/0 (socket says 172.21.15.28:53116) 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter ms_handle_connect 0x7fd83010ea80 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter ms_handle_connect 0x7fd830102250 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter ms_handle_connect 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3752518694 0 0) 0x7fd830107940 con 0x7fd83010ea80 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8180031f0 con 0x7fd83010ea80 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142348764 0 0) 0x7fd830107b50 con 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd830107940 con 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758498629 0 0) 0x7fd8301a96a0 con 0x7fd830102250 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd830107b50 con 0x7fd830102250 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1278128267 0 0) 0x7fd830107940 con 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8301a96a0 con 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd824001180 con 0x7fd83010aee0 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1350190148 0 0) 0x7fd8180031f0 con 0x7fd83010ea80 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd830107940 con 0x7fd83010ea80 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3124484025 0 0) 0x7fd830107b50 con 0x7fd830102250 2024-03-30T15:12:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8180031f0 con 0x7fd830102250 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd828001b30 con 0x7fd83010ea80 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd820004470 con 0x7fd830102250 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1961010969 0 0) 0x7fd8301a96a0 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 >> v1:172.21.15.28:6790/0 conn(0x7fd830102250 legacy=0x7fd830104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 >> v1:172.21.15.28:6789/0 conn(0x7fd83010ea80 legacy=0x7fd8301a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8301aa8e0 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301a98d0 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd8366b8700 10 client.50968.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8301a9d90 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd8366b8700 1 librados: init done 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd824003b20 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.389+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8240049a0 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.390+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81c005120 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.391+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd824003680 con 0x7fd83010aee0 2024-03-30T15:12:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(826..826 src has 442..826) ==== 9787+0+0 (unknown 2832139881 0 0) 0x7fd824088bf0 con 0x7fd83010aee0 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 10 client.50968.objecter ms_dispatch 0x7fd830001550 osd_map(826..826 src has 442..826) 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 3 client.50968.objecter handle_osd_map got epochs [826,826] > 0 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 3 client.50968.objecter handle_osd_map decoding full epoch 826 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 20 client.50968.objecter dump_active .. 0 homeless 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.392+0000 7fd82d7fa700 10 client.50968.objecter ms_handle_connect 0x7fd818070570 2024-03-30T15:12:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.396+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd824055780 con 0x7fd83010aee0 2024-03-30T15:12:47.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:47 smithi028 ceph-mon[29876]: pgmap v1267: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:47.462 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:47 smithi028 ceph-mon[41614]: pgmap v1267: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:47.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:47.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:47.559 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:12:47.559 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.35600.0:6785 101.c e463/130 101:33efc025:test-rados-api-smithi028-93253-70::foo:head v 463'57, mlcod=424'55)", 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 505.19203629200001, 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010387468, 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.560 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305190+0000", 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3799999999999998e-07 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.305197+0000", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4060000000000002e-06 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309914+0000", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047169780000000001 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309931+0000", 2024-03-30T15:12:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7027000000000001e-05 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.309985+0000", 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4101999999999998e-05 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.310010+0000", 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4870000000000001e-05 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315540+0000", 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.005530559 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:04:22.315577+0000", 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7187999999999997e-05 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.35600.0:7982 310.1f 310:fc57b2b2:test-rados-api-smithi028-93253-91::602c57ffb51af99d6f3b54c0ee9587bb110fb990:head [writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:12:47.564 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 392.17179416800002, 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063536169999999998, 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.35600", 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3141701961", 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7982 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:12:47.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325432+0000", 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325435+0000", 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.982e-06 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325446+0000", 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1218e-05 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325469+0000", 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2940000000000001e-05 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325481+0000", 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1826e-05 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325563+0000", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1933000000000003e-05 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.325569+0000", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2530000000000001e-06 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330393+0000", 2024-03-30T15:12:47.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048238509999999997 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330531+0000", 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013782599999999999 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.330675+0000", 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144227 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331230+0000", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055465600000000005 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331394+0000", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016427999999999999 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331696+0000", 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030202799999999998 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331745+0000", 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8900999999999998e-05 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:15.331786+0000", 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0695999999999998e-05 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.571 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDScrubReserve(5.18,RELEASE,e:545,from:4,reservation_nonce:0)", 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 389.05092477099998, 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037887667999999999, 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.572 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446301+0000", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446302+0000", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1600000000000001e-07 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446305+0000", 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.779e-06 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446312+0000", 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5499999999999997e-06 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446317+0000", 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1740000000000002e-06 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446358+0000", 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1288000000000002e-05 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.446363+0000", 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.361e-06 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484147+0000", 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037783543000000003 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484166+0000", 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9171999999999999e-05 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:06:18.484189+0000", 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3085e-05 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.576 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:7 323.0 323:00000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 270.12391191799998, 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010003161, 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 7 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373314+0000", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373316+0000", 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6390000000000001e-06 2024-03-30T15:12:47.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373322+0000", 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9900000000000002e-06 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373331+0000", 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8210000000000008e-06 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373341+0000", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0428000000000001e-05 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373391+0000", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9799999999999998e-05 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.373395+0000", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.228e-06 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.381593+0000", 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081977409999999997 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.381643+0000", 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0346999999999998e-05 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:17.383317+0000", 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016741670000000001 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.581 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:41 323.0 323:00000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:19.367282+0000", 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 268.12994376500001, 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052386860000000002, 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 41 2024-03-30T15:12:47.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367282+0000", 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367282+0000", 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367286+0000", 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8130000000000002e-06 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367300+0000", 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4187e-05 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367326+0000", 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6118e-05 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367341+0000", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4063000000000001e-05 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367414+0000", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3171999999999997e-05 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.367419+0000", 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6060000000000002e-06 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372346+0000", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049269049999999997 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372385+0000", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8949999999999998e-05 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:19.372521+0000", 2024-03-30T15:12:47.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013587199999999999 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:387 326.1a 326:58000000:%01:::head [pgnls start_epoch 678] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:08:59.390339+0000", 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 228.106887156, 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029639330000000002, 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.587 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 387 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390339+0000", 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390339+0000", 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390341+0000", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.717e-06 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390347+0000", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5910000000000001e-06 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390357+0000", 2024-03-30T15:12:47.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9520000000000006e-06 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390365+0000", 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9729999999999997e-06 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390414+0000", 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9168000000000001e-05 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.390419+0000", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4880000000000001e-06 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.393159+0000", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002740073 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.393199+0000", 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9798000000000001e-05 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:08:59.393303+0000", 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010417299999999999 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:649 328.2 328:40b10774:test-rados-api-smithi028-93253-123::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e694)", 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:15.376882+0000", 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 212.12034414300001, 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077751979999999997, 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 649 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376882+0000", 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376882+0000", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376885+0000", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2219999999999998e-06 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376895+0000", 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9170000000000001e-06 2024-03-30T15:12:47.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376905+0000", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0237e-05 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376914+0000", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4039999999999995e-06 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.376974+0000", 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0288000000000003e-05 2024-03-30T15:12:47.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for blocked object", 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.377208+0000", 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023446099999999999 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.383803+0000", 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0065942120000000003 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", 2024-03-30T15:12:47.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.383917+0000", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114771 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.384473+0000", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055572300000000005 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.384542+0000", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9089000000000002e-05 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:15.384657+0000", 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114874 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:793 322.0s0 322:0b783e73:test-rados-api-smithi028-93253-125::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e699)", 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:20.468669+0000", 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 207.02855666400001, 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012361197000000001, 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 793 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468669+0000", 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468669+0000", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468671+0000", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.979e-06 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468680+0000", 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9639999999999992e-06 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468694+0000", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3147e-05 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.468705+0000", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1701e-05 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.472043+0000", 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033372889999999998 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.472216+0000", 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000173976 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.473069+0000", 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00085296600000000003 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:12:47.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.480146+0000", 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.007076489 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.480225+0000", 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9370999999999996e-05 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:20.481031+0000", 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080531500000000002 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.603 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:798 322.0s0 322:0b783e73:test-rados-api-smithi028-93253-125::foo:head [writefull 0~5] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:21.467671+0000", 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 206.029555141, 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049495709999999998, 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.604 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 798 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467671+0000", 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467671+0000", 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467674+0000", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3110000000000001e-06 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467687+0000", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3069e-05 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467705+0000", 2024-03-30T15:12:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8145999999999999e-05 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467715+0000", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6860000000000001e-06 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467787+0000", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1575000000000005e-05 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.467794+0000", 2024-03-30T15:12:47.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4569999999999999e-06 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470071+0000", 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002277065 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470166+0000", 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4594999999999998e-05 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:12:47.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.470762+0000", 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059596200000000001 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.471951+0000", 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011891930000000001 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.472535+0000", 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000583808 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:21.472621+0000", 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5704000000000005e-05 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:962 330.10 330:0a686296:test-rados-api-smithi028-93253-127::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e712)", 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:33.407041+0000", 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 194.09018537599999, 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002933415, 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 962 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407041+0000", 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407041+0000", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407042+0000", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9099999999999991e-07 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407046+0000", 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9389999999999999e-06 2024-03-30T15:12:47.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407054+0000", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7620000000000001e-06 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407062+0000", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3010000000000001e-06 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407098+0000", 2024-03-30T15:12:47.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6532000000000002e-05 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.407162+0000", 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3632000000000003e-05 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409739+0000", 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025767619999999998 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409828+0000", 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9129000000000007e-05 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:33.409974+0000", 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000146367 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.615 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:1038 330.10 330:08000000::::head [pgnls start_epoch 714] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e714)", 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:09:35.414556+0000", 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 192.082670108, 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034157580000000001, 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1038 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414556+0000", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414556+0000", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414558+0000", 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6950000000000001e-06 2024-03-30T15:12:47.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414564+0000", 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.759e-06 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414574+0000", 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0044000000000001e-05 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414584+0000", 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5619999999999993e-06 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414632+0000", 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8412000000000003e-05 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.414638+0000", 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1599999999999997e-06 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.417067+0000", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002429161 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.417104+0000", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7336000000000003e-05 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:09:35.417972+0000", 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00086762899999999997 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.620 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:1526 333.7 333:e28744e7:test-rados-api-smithi028-93253-133::foo:head [writefull 0~1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:02.401289+0000", 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 165.095936835, 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062742930000000002, 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1526 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401289+0000", 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401289+0000", 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401291+0000", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1040000000000001e-06 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401301+0000", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8479999999999996e-06 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401313+0000", 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1630999999999999e-05 2024-03-30T15:12:47.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401325+0000", 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1897e-05 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401380+0000", 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5649000000000002e-05 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for blocked object", 2024-03-30T15:12:47.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.401608+0000", 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000227452 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.406485+0000", 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048771980000000001 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.406592+0000", 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106582 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.406745+0000", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015322099999999999 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.407166+0000", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042099200000000002 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.407460+0000", 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000293825 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.407492+0000", 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2141999999999998e-05 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:02.407564+0000", 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1752e-05 2024-03-30T15:12:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.58535.0:1701 334.14 334:28000000:%01:::head [pgnls start_epoch 753] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e753)", 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:15.439925+0000", 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 152.057300712, 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033941449999999999, 2024-03-30T15:12:47.628 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.58535", 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 1701 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439925+0000", 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439925+0000", 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439928+0000", 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3549999999999999e-06 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439935+0000", 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6690000000000006e-06 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439950+0000", 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4814000000000001e-05 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.439958+0000", 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7970000000000001e-06 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.440018+0000", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9621999999999999e-05 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.440022+0000", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0350000000000003e-06 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.443181+0000", 2024-03-30T15:12:47.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031589999999999999 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.443216+0000", 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5348000000000001e-05 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:15.443320+0000", 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010350500000000001 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(osd.6.52:47 322.0s0 322:1e0d35cd:test-rados-api-smithi028-93253-137::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.452180+0000", 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 144.04504641899999, 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003007895, 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452180+0000", 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452180+0000", 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452184+0000", 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8800000000000001e-06 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452196+0000", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2418999999999999e-05 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452218+0000", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1988999999999999e-05 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452229+0000", 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1511e-05 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452311+0000", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1521999999999996e-05 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452317+0000", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5659999999999997e-06 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.454715+0000", 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023984340000000001 2024-03-30T15:12:47.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455188+0000", 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047257400000000002 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(osd.6.52:48 322.0s0 322:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.452251+0000", 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 144.04497476, 2024-03-30T15:12:47.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032067179999999999, 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452251+0000", 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452251+0000", 2024-03-30T15:12:47.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452252+0000", 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0699999999999997e-07 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452257+0000", 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1780000000000002e-06 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452264+0000", 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2119999999999997e-06 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452269+0000", 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.062e-06 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452379+0000", 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109398 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting_for_map not empty", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.452386+0000", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7959999999999996e-06 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455353+0000", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029662109999999998 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.455458+0000", 2024-03-30T15:12:47.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010535399999999999 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(osd.6.52:49 322.0s0 322:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e761)", 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:23.467194+0000", 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 144.03003228399999, 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.018280177000000002, 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467194+0000", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467194+0000", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467197+0000", 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.055e-06 2024-03-30T15:12:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467207+0000", 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0443e-05 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467224+0000", 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6543000000000001e-05 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467236+0000", 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.187e-05 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467299+0000", 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3541999999999996e-05 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.467410+0000", 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000110888 2024-03-30T15:12:47.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.480147+0000", 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012737307 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_committed", 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.485115+0000", 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049679110000000002 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.485408+0000", 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029295900000000003 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:23.485474+0000", 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5659000000000003e-05 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.25087.0:2 42.1a 42:5c6776db:::0.obj:head [write 0~103] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e780)", 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:43.968994+0000", 2024-03-30T15:12:47.648 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 123.528232167, 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.46392696, 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.25087", 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/4227304124", 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 2 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.968994+0000", 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.968994+0000", 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.968996+0000", 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5500000000000001e-06 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.969005+0000", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.816e-06 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.969016+0000", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1014e-05 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.969028+0000", 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1793999999999999e-05 2024-03-30T15:12:47.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.969107+0000", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9113e-05 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:43.969119+0000", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1623999999999999e-05 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.428595+0000", 2024-03-30T15:12:47.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.45947580500000001 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.428727+0000", 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013216599999999999 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3", 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.428876+0000", 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148956 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:12:47.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.432386+0000", 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003510469 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.432697+0000", 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000311112 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.432747+0000", 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9461e-05 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:44.432921+0000", 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017407999999999999 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:3310 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94326471853056 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e783)", 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:10:57.686711+0000", 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 109.81051508199999, 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000415826, 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.655 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3310 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686711+0000", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686711+0000", 2024-03-30T15:12:47.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686717+0000", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3770000000000001e-06 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686738+0000", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0744999999999999e-05 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686787+0000", 2024-03-30T15:12:47.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8396000000000002e-05 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686807+0000", 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0818999999999998e-05 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.686965+0000", 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158145 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.687037+0000", 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1946000000000005e-05 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:10:57.687127+0000", 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9398e-05 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:3439 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94326471853056 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e809)", 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:11:57.698182+0000", 2024-03-30T15:12:47.659 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 49.799044412999997, 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044198799999999998, 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3439 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698182+0000", 2024-03-30T15:12:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698182+0000", 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698187+0000", 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8450000000000002e-06 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698199+0000", 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2629000000000001e-05 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698221+0000", 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2166000000000001e-05 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698232+0000", 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1127999999999999e-05 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698438+0000", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020507899999999999 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698524+0000", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6874999999999997e-05 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:11:57.698624+0000", 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9265999999999995e-05 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.24394.0:3479 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94326471853056 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e817)", 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-03-30T15:12:17.649175+0000", 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 29.848051386000002, 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001013393, 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "continuous": false, 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.24394", 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 3479 2024-03-30T15:12:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649175+0000", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649175+0000", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649178+0000", 2024-03-30T15:12:47.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2509999999999999e-06 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649188+0000", 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0311e-05 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649206+0000", 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8227e-05 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-03-30T15:12:47.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649217+0000", 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0869999999999999e-05 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649280+0000", 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2630000000000002e-05 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.649397+0000", 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011711599999999999 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:12:47.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-03-30T15:12:17.650188+0000", 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00079098799999999996 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:12:47.668 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:12:47.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:47.633+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd81c005410 con 0x7fd83010aee0 2024-03-30T15:12:47.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:47.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:47.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:47.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:47.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:47.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:48.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:48.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:48.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:48.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:48.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:48.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.440+0000 7fd82d7fa700 1 -- 172.21.15.28:0/1609773023 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v828) ==== 91+0+0 (unknown 814558363 0 0) 0x7fd8240591d0 con 0x7fd83010aee0 2024-03-30T15:12:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.443+0000 7fd8366b8700 10 librados: watch_flush enter 2024-03-30T15:12:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.443+0000 7fd8366b8700 10 librados: watch_flush exit 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.443+0000 7fd8366b8700 20 client.50968.objecter shutdown clearing up homeless session... 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.443+0000 7fd8366b8700 10 client.50968.objecter successfully canceled tick 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.443+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd818070570 legacy=0x7fd818072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 >> v1:172.21.15.134:6789/0 conn(0x7fd83010aee0 legacy=0x7fd830104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 shutdown_connections 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 >> 172.21.15.28:0/1609773023 conn(0x7fd8300fdae0 msgr2=0x7fd830109fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 shutdown_connections 2024-03-30T15:12:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 -- 172.21.15.28:0/1609773023 wait complete. 2024-03-30T15:12:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:48.444+0000 7fd8366b8700 1 librados: shutdown 2024-03-30T15:12:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:12:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:48.485 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-30T15:12:48.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:48.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:48.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[29876]: osdmap e827: 8 total, 5 up, 8 in 2024-03-30T15:12:48.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1609773023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[29876]: from='client.50968 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[29876]: pgmap v1269: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[41614]: osdmap e827: 8 total, 5 up, 8 in 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1609773023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[41614]: from='client.50968 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:48 smithi028 ceph-mon[41614]: pgmap v1269: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:48.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:48.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:48.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:48.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:48.839 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:48 smithi134 ceph-mon[33714]: osdmap e827: 8 total, 5 up, 8 in 2024-03-30T15:12:48.839 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1609773023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.839 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:48 smithi134 ceph-mon[33714]: from='client.50968 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:12:48.839 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:48 smithi134 ceph-mon[33714]: pgmap v1269: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:48.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:48.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:48.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:49.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:49.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:49.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:49.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:49.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:49.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:49.256 INFO:teuthology.orchestra.run.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e815 pg_nums {1={20=1,559=11},2={67=3},3={68=32},4={70=32},5={72=32},6={74=32},12={110=32},42={113=32},101={130=32},296={430=32},297={448=32},298={458=32},299={462=32},300={465=32},301={478=32},302={484=32},303={489=32},304={493=32},305={503=32},306={513=32},307={522=32},308={531=32},309={533=32},310={540=32},311={546=32},312={551=32},313={568=32},314={575=32},315={587=32},316={599=32},317={608=8},318={610=32},319={612=32},320={623=8},321={624=32},322={644=8},323={645=32},324={652=32},325={660=32},326={672=32},327={682=32},328={690=32},329={698=32},330={708=32},331={716=32},332={728=32},333={736=32},334={749=32},335={757=32},336={765=32},337={773=32}} deleted_pools 447,296,457,297,461,298,464,299,477,300,483,301,488,302,492,303,502,304,512,305,521,306,530,307,538,309,539,308,545,310,550,311,567,312,574,313,586,314,598,315,604,316,605,101,619,319,620,318,635,321,636,320,639,317,651,323,659,324,671,325,681,326,689,327,697,328,707,329,715,330,727,331,735,332,748,333,756,334,764,335,772,336) 2024-03-30T15:12:49.256 INFO:teuthology.orchestra.run.smithi134.stdout:Importing pgid 12.1d 2024-03-30T15:12:49.256 INFO:teuthology.orchestra.run.smithi134.stdout:write_pg epoch 752 info 12.1d( empty local-lis/les=669/670 n=0 ec=110/110 lis/c=669/110 les/c/f=670/113/0 sis=669) 2024-03-30T15:12:49.257 INFO:teuthology.orchestra.run.smithi134.stdout:Import successful 2024-03-30T15:12:49.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:12:49.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:49.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:49.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:49.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:50.193 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:49 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:50.193 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:49 smithi134 ceph-mon[33714]: from='client.50968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:50.194 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:49 smithi134 ceph-mon[33714]: osdmap e828: 8 total, 5 up, 8 in 2024-03-30T15:12:50.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:50.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[29876]: from='client.50968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:50.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[29876]: osdmap e828: 8 total, 5 up, 8 in 2024-03-30T15:12:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:12:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[41614]: from='client.50968 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:12:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:49 smithi028 ceph-mon[41614]: osdmap e828: 8 total, 5 up, 8 in 2024-03-30T15:12:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:50.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:50.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.12.1d.2 2024-03-30T15:12:50.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:50.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/exp.12.1d.2 2024-03-30T15:12:50.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:50.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:50.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:50.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:12:50.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:50.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[29876]: 337.11 scrub starts 2024-03-30T15:12:50.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[29876]: 337.11 scrub ok 2024-03-30T15:12:50.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[29876]: pgmap v1271: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[41614]: 337.11 scrub starts 2024-03-30T15:12:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[41614]: 337.11 scrub ok 2024-03-30T15:12:50.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:50 smithi028 ceph-mon[41614]: pgmap v1271: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:50 smithi134 ceph-mon[33714]: 337.11 scrub starts 2024-03-30T15:12:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:50 smithi134 ceph-mon[33714]: 337.11 scrub ok 2024-03-30T15:12:50.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:50 smithi134 ceph-mon[33714]: pgmap v1271: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:50.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:50.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:50.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:50.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:50.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:51.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:51.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:51.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:51.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:51.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:12:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:12:51.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:51.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:51.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:51.553 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:12:51.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:12:51.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:51.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:12:51.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:12:51.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:51.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:51.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:51.836 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:12:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:12:51.836 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:12:51.836 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:12:51.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:51.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:51 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:12:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:51 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:12:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:12:52.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:52.051 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:52.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:52.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:52.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:12:52.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:52.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:52.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:52.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:52.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:52.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:52.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:12:52.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:52.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:52.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:52 smithi028 ceph-mon[41614]: pgmap v1272: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:52.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:52 smithi028 ceph-mon[41614]: osdmap e829: 8 total, 5 up, 8 in 2024-03-30T15:12:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:52 smithi028 ceph-mon[29876]: pgmap v1272: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:52 smithi028 ceph-mon[29876]: osdmap e829: 8 total, 5 up, 8 in 2024-03-30T15:12:52.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:52.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:52.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:52 smithi134 ceph-mon[33714]: pgmap v1272: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 644 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:52.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:52 smithi134 ceph-mon[33714]: osdmap e829: 8 total, 5 up, 8 in 2024-03-30T15:12:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:52.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:53.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:53.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:53.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:53.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:53.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:53.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:53.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:53.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:53.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:53.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:53.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:53.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:53.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[29876]: 337.e scrub starts 2024-03-30T15:12:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[29876]: 337.e scrub ok 2024-03-30T15:12:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[41614]: 337.e scrub starts 2024-03-30T15:12:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[41614]: 337.e scrub ok 2024-03-30T15:12:53.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:53.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:53.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:53.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:53 smithi134 ceph-mon[33714]: 337.e scrub starts 2024-03-30T15:12:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:53 smithi134 ceph-mon[33714]: 337.e scrub ok 2024-03-30T15:12:53.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:12:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:53.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:54.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:54.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:54.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:54.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:54.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:54.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:54.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:54.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:54.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:54.681 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:12:54.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:54.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[29876]: 42.12 scrub starts 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[29876]: 42.12 scrub ok 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[29876]: pgmap v1274: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[41614]: 42.12 scrub starts 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[41614]: 42.12 scrub ok 2024-03-30T15:12:54.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:54 smithi028 ceph-mon[41614]: pgmap v1274: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:54.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:54.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:54 smithi134 ceph-mon[33714]: 42.12 scrub starts 2024-03-30T15:12:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:54 smithi134 ceph-mon[33714]: 42.12 scrub ok 2024-03-30T15:12:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:54 smithi134 ceph-mon[33714]: pgmap v1274: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:54.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:54.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:54.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:55.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:55.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:55.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:55.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:55.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:55.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:55.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:55.461 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0, 2, 4] live_osds: [1, 3, 5, 6, 7] 2024-03-30T15:12:55.461 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T15:12:55.461 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-03-30T15:12:55.462 DEBUG:teuthology.orchestra.run.smithi134:osd.4> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T15:12:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:55.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:55.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl start ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T15:12:55.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:55.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[41614]: 337.4 scrub starts 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[41614]: 337.4 scrub ok 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[41614]: 337.1a scrub starts 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[41614]: 337.1a scrub ok 2024-03-30T15:12:55.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[29876]: 337.4 scrub starts 2024-03-30T15:12:55.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[29876]: 337.4 scrub ok 2024-03-30T15:12:55.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:55.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[29876]: 337.1a scrub starts 2024-03-30T15:12:55.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:55 smithi028 ceph-mon[29876]: 337.1a scrub ok 2024-03-30T15:12:55.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:55.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:55.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:55.785 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:55 smithi134 ceph-mon[33714]: 337.4 scrub starts 2024-03-30T15:12:55.785 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:55 smithi134 ceph-mon[33714]: 337.4 scrub ok 2024-03-30T15:12:55.785 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:55 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 206/735 objects degraded (28.027%), 55 pgs degraded, 81 pgs undersized (PG_DEGRADED) 2024-03-30T15:12:55.785 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:55 smithi134 ceph-mon[33714]: 337.1a scrub starts 2024-03-30T15:12:55.785 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:55 smithi134 ceph-mon[33714]: 337.1a scrub ok 2024-03-30T15:12:55.785 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:55 smithi134 systemd[1]: Starting Ceph osd.4 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:12:55.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:55.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:55.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:55.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:55.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:56.060 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:55 smithi134 podman[158674]: 2024-03-30 15:12:55.687065827 +0000 UTC m=+0.018314888 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:12:56.060 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:55 smithi134 podman[158674]: 2024-03-30 15:12:55.800678634 +0000 UTC m=+0.131927677 container create 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-03-30T15:12:56.060 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:55 smithi134 podman[158674]: 2024-03-30 15:12:55.998483543 +0000 UTC m=+0.329732588 container init 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True) 2024-03-30T15:12:56.060 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 podman[158674]: 2024-03-30 15:12:56.00368981 +0000 UTC m=+0.334938861 container start 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-30T15:12:56.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 -- 172.21.15.28:0/2332133848 >> v1:172.21.15.28:6789/0 conn(0x7f5030103210 legacy=0x7f50301035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 -- 172.21.15.28:0/2332133848 shutdown_connections 2024-03-30T15:12:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 -- 172.21.15.28:0/2332133848 >> 172.21.15.28:0/2332133848 conn(0x7f50300fc9e0 msgr2=0x7f50300fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 -- 172.21.15.28:0/2332133848 shutdown_connections 2024-03-30T15:12:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 -- 172.21.15.28:0/2332133848 wait complete. 2024-03-30T15:12:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 librados: starting msgr at 2024-03-30T15:12:56.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.098+0000 7f5034e2a700 1 librados: starting objecter 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 Processor -- start 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 -- start start 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 librados: setting wanted keys 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 librados: calling monclient init 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50301a92e0 con 0x7f503010eb30 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50301aa500 con 0x7f5030103210 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f5034e2a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f50301ab740 con 0x7f503010b020 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f502ed9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f503010eb30 0x7f50301a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36376/0 (socket says 172.21.15.28:36376) 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f502dd9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f503010b020 0x7f50301a42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38550/0 (socket says 172.21.15.28:38550) 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f502ed9d700 1 -- 172.21.15.28:0/233586102 learned_addr learned my addr 172.21.15.28:0/233586102 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:56.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f502e59c700 1 --1- 172.21.15.28:0/233586102 >> v1:172.21.15.134:6789/0 conn(0x7f5030103210 0x7f503010a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.28:33270/0 (socket says 172.21.15.28:33270) 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter ms_handle_connect 0x7f503010b020 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter ms_handle_connect 0x7f5030103210 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter ms_handle_connect 0x7f503010eb30 2024-03-30T15:12:56.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.099+0000 7f501f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972954737 0 0) 0x7f50301aa500 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50180039c0 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622805598 0 0) 0x7f50301ab740 con 0x7f503010b020 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50301aa500 con 0x7f503010b020 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1673266400 0 0) 0x7f50301a92e0 con 0x7f503010eb30 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50301ab740 con 0x7f503010eb30 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1088260776 0 0) 0x7f50180039c0 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50301a92e0 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5020004400 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2059288264 0 0) 0x7f50301aa500 con 0x7f503010b020 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50180039c0 con 0x7f503010b020 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f50240013d0 con 0x7f503010b020 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 401164414 0 0) 0x7f50301a92e0 con 0x7f5030103210 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 >> v1:172.21.15.28:6790/0 conn(0x7f503010b020 legacy=0x7f50301a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:56.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 >> v1:172.21.15.28:6789/0 conn(0x7f503010eb30 legacy=0x7f50301a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50301ac980 con 0x7f5030103210 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50301aa730 con 0x7f5030103210 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f5034e2a700 10 client.50977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50301aabf0 con 0x7f5030103210 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f5034e2a700 1 librados: init done 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5020001b10 con 0x7f5030103210 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.100+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5020004cc0 con 0x7f5030103210 2024-03-30T15:12:56.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.101+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5010005100 con 0x7f5030103210 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.102+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5020003780 con 0x7f5030103210 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.103+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(829..829 src has 442..829) ==== 9787+0+0 (unknown 2641779490 0 0) 0x7f5020089d00 con 0x7f5030103210 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.103+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 osd_map(829..829 src has 442..829) 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.103+0000 7f501f7fe700 3 client.50977.objecter handle_osd_map got epochs [829,829] > 0 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.103+0000 7f501f7fe700 3 client.50977.objecter handle_osd_map decoding full epoch 829 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f501f7fe700 20 client.50977.objecter dump_active .. 0 homeless 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f50180704b0 2024-03-30T15:12:56.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v829) ==== 54+0+16 (unknown 2813340565 0 2420000192) 0x7f502008a100 con 0x7f5030103210 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 10 client.50977.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 1 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 10 client.50977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=830}) -- 0x7f50100053e0 con 0x7f5030103210 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5010000e70 con 0x7f5030103210 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=829) ==== 24+0+0 (unknown 2423003755 0 0) 0x7f5020055f60 con 0x7f5030103210 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f502cd99700 10 client.50977.objecter _check_command_map_dne tid 1 current 829 map_dne_bound 829 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f502cd99700 10 client.50977.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f502cd99700 15 client.50977.objecter _session_command_op_remove -1 1 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 10 client.50977.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 2 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 2 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 1 2 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 4 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 10 client.50977.objecter _send_command 2 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.104+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f5010004420 con 0x7f50100038d0 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f50100038d0 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5010004420 con 0x7f50100038d0 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 2: 0 ) 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 10 client.50977.objecter _finish_command 2 = system:0 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 4 2024-03-30T15:12:56.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.105+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 1 2 2024-03-30T15:12:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 10 client.50977.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010001bb0 osd=1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 1 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 4 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 10 client.50977.objecter _send_command 3 2024-03-30T15:12:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.123+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5010004dd0 con 0x7f50100038d0 2024-03-30T15:12:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5010004dd0 con 0x7f50100038d0 2024-03-30T15:12:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f501f7fe700 10 client.50977.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010001bb0 osd=1 4 2024-03-30T15:12:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 1 3 2024-03-30T15:12:56.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:12:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f5034e2a700 10 client.50977.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 4 2024-03-30T15:12:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f5034e2a700 20 client.50977.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f5034e2a700 10 client.50977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.124+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f50100018e0 con 0x7f5030103210 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=829) ==== 24+0+0 (unknown 1531980552 0 0) 0x7f50200542d0 con 0x7f5030103210 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f501ffff700 10 client.50977.objecter _check_command_map_dne tid 4 current 829 map_dne_bound 829 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f501ffff700 10 client.50977.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f501ffff700 15 client.50977.objecter _session_command_op_remove -1 4 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 10 client.50977.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 5 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 5 2024-03-30T15:12:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 3 5 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 4 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 10 client.50977.objecter _send_command 5 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5010003530 con 0x7f50100049c0 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.125+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f50100049c0 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.126+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5010003530 con 0x7f50100049c0 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.126+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 5: 0 ) 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.126+0000 7f501f7fe700 10 client.50977.objecter _finish_command 5 = system:0 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.126+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 4 2024-03-30T15:12:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.126+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 3 5 2024-03-30T15:12:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 10 client.50977.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 6 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 6 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010004850 osd=3 3 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 3 6 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 4 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 10 client.50977.objecter _send_command 6 2024-03-30T15:12:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.143+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50100029a0 con 0x7f50100049c0 2024-03-30T15:12:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f50100029a0 con 0x7f50100049c0 2024-03-30T15:12:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 10 client.50977.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010004850 osd=3 4 2024-03-30T15:12:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 3 6 2024-03-30T15:12:56.147 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 10 client.50977.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 7 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter _get_session osd=-1 returning homeless 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 10 client.50977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=3) -- 0x7f50100091d0 con 0x7f5030103210 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== mon.1 v1:172.21.15.134:6789/0 12 ==== mon_get_version_reply(handle=3 version=829) ==== 24+0+0 (unknown 3776964998 0 0) 0x7f5020055af0 con 0x7f5030103210 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f502cd99700 10 client.50977.objecter _check_command_map_dne tid 7 current 829 map_dne_bound 829 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f502cd99700 10 client.50977.objecter _finish_command 7 = osd:6 osd down 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f502cd99700 15 client.50977.objecter _session_command_op_remove -1 7 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 10 client.50977.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 8 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 8 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 5 8 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 4 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 10 client.50977.objecter _send_command 8 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.145+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6804/3668151179 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7f5010012180 con 0x7f501000e430 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.146+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f501000e430 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.147+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5010012180 con 0x7f501000e430 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.147+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 8: 0 ) 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.147+0000 7f501f7fe700 10 client.50977.objecter _finish_command 8 = system:0 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.147+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 4 2024-03-30T15:12:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.147+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 5 8 2024-03-30T15:12:56.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 10 client.50977.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 9 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 9 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010002da0 osd=5 3 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 5 9 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 4 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 10 client.50977.objecter _send_command 9 2024-03-30T15:12:56.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.163+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f50100096f0 con 0x7f501000e430 2024-03-30T15:12:56.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f50100096f0 con 0x7f501000e430 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f501f7fe700 10 client.50977.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010002da0 osd=5 4 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 5 9 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 10 client.50977.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 10 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 10 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 6 10 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 4 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 10 client.50977.objecter _send_command 10 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.165+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6808/2135224194 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7f5010012ed0 con 0x7f5010012360 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.166+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f5010012360 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.167+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5010012ed0 con 0x7f5010012360 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.167+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 10: 0 ) 2024-03-30T15:12:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.167+0000 7f501f7fe700 10 client.50977.objecter _finish_command 10 = system:0 2024-03-30T15:12:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.167+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 4 2024-03-30T15:12:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.167+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 6 10 2024-03-30T15:12:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 10 client.50977.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 11 2024-03-30T15:12:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 11 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f50100099e0 osd=6 3 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 6 11 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 4 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 10 client.50977.objecter _send_command 11 2024-03-30T15:12:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.184+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5010013880 con 0x7f5010012360 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5010013880 con 0x7f5010012360 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 10 client.50977.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f50100099e0 osd=6 4 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 6 11 2024-03-30T15:12:56.188 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 10 client.50977.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 12 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 12 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 7 12 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 4 2024-03-30T15:12:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 10 client.50977.objecter _send_command 12 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6812/3758190298 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7f501001b9a0 con 0x7f5010017d40 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.187+0000 7f501f7fe700 10 client.50977.objecter ms_handle_connect 0x7f5010017d40 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.188+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f501001b9a0 con 0x7f5010017d40 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.188+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 12: 0 ) 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.188+0000 7f501f7fe700 10 client.50977.objecter _finish_command 12 = system:0 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.188+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 4 2024-03-30T15:12:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.188+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 7 12 2024-03-30T15:12:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 10 client.50977.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:12:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign -1 13 2024-03-30T15:12:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 20 client.50977.objecter _get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_remove -1 13 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 20 client.50977.objecter get_session s=0x7f5010013880 osd=7 3 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 15 client.50977.objecter _session_command_op_assign 7 13 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 4 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 10 client.50977.objecter _send_command 13 2024-03-30T15:12:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.205+0000 7f5034e2a700 1 -- 172.21.15.28:0/233586102 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5010013300 con 0x7f5010017d40 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.207+0000 7f501f7fe700 1 -- 172.21.15.28:0/233586102 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5010013300 con 0x7f5010017d40 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.207+0000 7f501f7fe700 10 client.50977.objecter ms_dispatch 0x7f5030001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.207+0000 7f501f7fe700 10 client.50977.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.207+0000 7f501f7fe700 20 client.50977.objecter put_session s=0x7f5010013880 osd=7 4 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.207+0000 7f501f7fe700 15 client.50977.objecter _session_command_op_remove 7 13 2024-03-30T15:12:56.209 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:12:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:12:56.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:56.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:56.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:56.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:56.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:56.430 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 podman[158674]: 2024-03-30 15:12:56.115327248 +0000 UTC m=+0.446576294 container attach 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T15:12:56.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/233586102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:56.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:56 smithi028 ceph-mon[29876]: pgmap v1275: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:56.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/233586102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:56.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:56 smithi028 ceph-mon[41614]: pgmap v1275: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:56.602 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:56.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:56.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:56.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:56.836 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/233586102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:12:56.836 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-mon[33714]: pgmap v1275: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:56.836 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T15:12:56.836 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T15:12:56.836 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T15:12:56.836 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 bash[158674]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate[158724]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-30T15:12:56.837 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:56 smithi134 podman[158674]: 2024-03-30 15:12:56.497020891 +0000 UTC m=+0.828269936 container died 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-30T15:12:56.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:56.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:56.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:56.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:57.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:57.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:57.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:57.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 -- 172.21.15.28:0/3107606751 >> v1:172.21.15.28:6789/0 conn(0x7f5f5010d960 legacy=0x7f5f5010fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:57.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 -- 172.21.15.28:0/3107606751 shutdown_connections 2024-03-30T15:12:57.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 -- 172.21.15.28:0/3107606751 >> 172.21.15.28:0/3107606751 conn(0x7f5f500fc9e0 msgr2=0x7f5f500fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:57.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 -- 172.21.15.28:0/3107606751 shutdown_connections 2024-03-30T15:12:57.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 -- 172.21.15.28:0/3107606751 wait complete. 2024-03-30T15:12:57.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 librados: starting msgr at 2024-03-30T15:12:57.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.489+0000 7f5f54b15700 1 librados: starting objecter 2024-03-30T15:12:57.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 Processor -- start 2024-03-30T15:12:57.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 -- start start 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 librados: setting wanted keys 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 librados: calling monclient init 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5f501a6f00 con 0x7f5f5010d960 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5f501a8120 con 0x7f5f50101130 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f54b15700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5f501a9360 con 0x7f5f50109dc0 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f4dd9b700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5f50109dc0 0x7f5f501a1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38556/0 (socket says 172.21.15.28:38556) 2024-03-30T15:12:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f4ed9d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f5010d960 0x7f5f501a55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36404/0 (socket says 172.21.15.28:36404) 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f4dd9b700 1 -- 172.21.15.28:0/2170538103 learned_addr learned my addr 172.21.15.28:0/2170538103 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter ms_handle_connect 0x7f5f5010d960 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter ms_handle_connect 0x7f5f50109dc0 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter ms_handle_connect 0x7f5f50101130 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:12:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450518059 0 0) 0x7f5f501a6f00 con 0x7f5f5010d960 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f38003aa0 con 0x7f5f5010d960 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479696191 0 0) 0x7f5f501a9360 con 0x7f5f50109dc0 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f501a6f00 con 0x7f5f50109dc0 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269505984 0 0) 0x7f5f501a8120 con 0x7f5f50101130 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f501a9360 con 0x7f5f50101130 2024-03-30T15:12:57.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3656750400 0 0) 0x7f5f38003aa0 con 0x7f5f5010d960 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f501a8120 con 0x7f5f5010d960 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.490+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5f48001d80 con 0x7f5f5010d960 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1850786904 0 0) 0x7f5f501a6f00 con 0x7f5f50109dc0 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f38003aa0 con 0x7f5f50109dc0 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5f44001890 con 0x7f5f50109dc0 2024-03-30T15:12:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3031372862 0 0) 0x7f5f501a9360 con 0x7f5f50101130 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f501a6f00 con 0x7f5f50101130 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3696120241 0 0) 0x7f5f501a8120 con 0x7f5f5010d960 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 >> v1:172.21.15.28:6790/0 conn(0x7f5f50109dc0 legacy=0x7f5f501a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 >> v1:172.21.15.134:6789/0 conn(0x7f5f50101130 legacy=0x7f5f5006a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f501aa5a0 con 0x7f5f5010d960 2024-03-30T15:12:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f501a9590 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f54b15700 10 client.25785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f501a9a50 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f54b15700 1 librados: init done 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5f48002080 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.491+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5f48005060 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.492+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f30005120 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.492+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5f48005280 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(829..829 src has 442..829) ==== 9787+0+0 (unknown 2641779490 0 0) 0x7f5f48089f80 con 0x7f5f5010d960 2024-03-30T15:12:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 10 client.25785.objecter ms_dispatch 0x7f5f50001550 osd_map(829..829 src has 442..829) 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 3 client.25785.objecter handle_osd_map got epochs [829,829] > 0 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 3 client.25785.objecter handle_osd_map decoding full epoch 829 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 20 client.25785.objecter dump_active .. 0 homeless 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.493+0000 7f5f3f7fe700 10 client.25785.objecter ms_handle_connect 0x7f5f38070390 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.497+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5f48056b30 con 0x7f5f5010d960 2024-03-30T15:12:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:56.712+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f5f30005410 con 0x7f5f5010d960 2024-03-30T15:12:57.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:57.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:57.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:57.362 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 podman[158905]: 2024-03-30 15:12:57.110540674 +0000 UTC m=+0.604565186 container remove 790ba279f978765b79667a1e7c917ffd31f5f243d87d1186b7208f79fea8a0be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-30T15:12:57.362 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 podman[159025]: 2024-03-30 15:12:57.226102241 +0000 UTC m=+0.017993845 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:12:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:57.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:57.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.473+0000 7f5f3f7fe700 1 -- 172.21.15.28:0/2170538103 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v830) ==== 101+0+0 (unknown 4000144555 0 0) 0x7f5f4805a580 con 0x7f5f5010d960 2024-03-30T15:12:57.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.476+0000 7f5f54b15700 10 librados: watch_flush enter 2024-03-30T15:12:57.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.476+0000 7f5f54b15700 10 librados: watch_flush exit 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.476+0000 7f5f54b15700 20 client.25785.objecter shutdown clearing up homeless session... 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.476+0000 7f5f54b15700 10 client.25785.objecter successfully canceled tick 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.478+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5f38070390 legacy=0x7f5f38072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.478+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 >> v1:172.21.15.28:6789/0 conn(0x7f5f5010d960 legacy=0x7f5f501a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.479+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 shutdown_connections 2024-03-30T15:12:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.479+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 >> 172.21.15.28:0/2170538103 conn(0x7f5f500fc9e0 msgr2=0x7f5f500fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:12:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.479+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 shutdown_connections 2024-03-30T15:12:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.479+0000 7f5f54b15700 1 -- 172.21.15.28:0/2170538103 wait complete. 2024-03-30T15:12:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:12:57.479+0000 7f5f54b15700 1 librados: shutdown 2024-03-30T15:12:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:12:57.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:57.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:57.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:57.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:57.664 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:57.665 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 podman[159025]: 2024-03-30 15:12:57.361447189 +0000 UTC m=+0.153338787 container create 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc) 2024-03-30T15:12:57.665 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 podman[159025]: 2024-03-30 15:12:57.563732381 +0000 UTC m=+0.355623976 container init 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-30T15:12:57.665 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 podman[159025]: 2024-03-30 15:12:57.569385254 +0000 UTC m=+0.361276852 container start 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-30T15:12:57.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:12:57.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:57.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:12:57.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:57.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:57.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:57.868 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:12:57.868 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:12:57.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:57.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:57.915 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:12:57.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 bash[159025]: 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 2024-03-30T15:12:57.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:57 smithi134 systemd[1]: Started Ceph osd.4 for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:12:57.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:58.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:58.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:58.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:58.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:58.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:58.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:58.383 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:12:58.104+0000 7fa97166d7c0 -1 Falling back to public interface 2024-03-30T15:12:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:12:58.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:12:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:58.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:58.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:58.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:58.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:12:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: osdmap e830: 8 total, 5 up, 8 in 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 337.9 scrub starts 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 337.9 scrub ok 2024-03-30T15:12:58.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 42.a scrub starts 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: 42.a scrub ok 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[29876]: pgmap v1277: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: osdmap e830: 8 total, 5 up, 8 in 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 337.9 scrub starts 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 337.9 scrub ok 2024-03-30T15:12:58.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 42.a scrub starts 2024-03-30T15:12:58.625 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: 42.a scrub ok 2024-03-30T15:12:58.625 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:58 smithi028 ceph-mon[41614]: pgmap v1277: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:12:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:12:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:12:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:12:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2170538103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:12:58.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: osdmap e830: 8 total, 5 up, 8 in 2024-03-30T15:12:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 337.9 scrub starts 2024-03-30T15:12:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 337.9 scrub ok 2024-03-30T15:12:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 42.a scrub starts 2024-03-30T15:12:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: 42.a scrub ok 2024-03-30T15:12:58.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:58 smithi134 ceph-mon[33714]: pgmap v1277: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:12:58.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:58.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:58.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:58.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:58.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:59.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:59.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:59.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:12:59.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:12:59.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:59.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:59.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:59.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:59.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:12:59.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:12:59.654 INFO:teuthology.orchestra.run.smithi134.stderr:no valid command found; 10 closest matches: 2024-03-30T15:12:59.654 INFO:teuthology.orchestra.run.smithi134.stderr:0 2024-03-30T15:12:59.654 INFO:teuthology.orchestra.run.smithi134.stderr:1 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:2 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:abort 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:assert 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:config diff 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:config diff get 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:config get 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:config help [] 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:config set ... 2024-03-30T15:12:59.655 INFO:teuthology.orchestra.run.smithi134.stderr:admin_socket: invalid command 2024-03-30T15:12:59.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:59 smithi134 ceph-mon[33714]: 3.c deep-scrub starts 2024-03-30T15:12:59.655 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:12:59 smithi134 ceph-mon[33714]: 3.c deep-scrub ok 2024-03-30T15:12:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:12:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:12:59.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:59 smithi028 ceph-mon[41614]: 3.c deep-scrub starts 2024-03-30T15:12:59.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:12:59 smithi028 ceph-mon[41614]: 3.c deep-scrub ok 2024-03-30T15:12:59.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:59 smithi028 ceph-mon[29876]: 3.c deep-scrub starts 2024-03-30T15:12:59.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:12:59 smithi028 ceph-mon[29876]: 3.c deep-scrub ok 2024-03-30T15:12:59.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:12:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:12:59.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:12:59.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:12:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:12:59.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:12:59.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:00.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:00.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:00.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:00.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:00.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:00.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T15:13:00.329 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T15:13:00.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:13:00.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:00.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:00.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:00.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:00.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 337.1b scrub starts 2024-03-30T15:13:00.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: 337.1b scrub ok 2024-03-30T15:13:00.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-mon[33714]: pgmap v1278: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 337.1b scrub starts 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: 337.1b scrub ok 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[29876]: pgmap v1278: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:00.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 337.1b scrub starts 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: 337.1b scrub ok 2024-03-30T15:13:00.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:00 smithi028 ceph-mon[41614]: pgmap v1278: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:00.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:00.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:00.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:00.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:00.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:01.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:01.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:01.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:01.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:01.180 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:13:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:13:00.857+0000 7fa97166d7c0 -1 osd.4 815 log_to_monitors true 2024-03-30T15:13:01.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:01.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:13:01.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:01.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:01.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:01.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:01.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:01.612 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:01.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:01.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[29876]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[41614]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:13:01.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:01 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:13:01.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:01.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:01.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:01.930 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:13:01.545+0000 7fa962ac5700 -1 osd.4 831 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T15:13:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-mon[33714]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T15:13:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:13:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:01 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:13:01.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:01.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:02.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:02.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:02.163 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:02.163 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:02.183 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:02.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:02.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:02.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:02.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:02.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:02.561 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:02.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:02.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:02.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:02.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:02.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:02.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T15:13:02.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[29876]: osdmap e831: 8 total, 5 up, 8 in 2024-03-30T15:13:02.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[29876]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[29876]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[29876]: pgmap v1280: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[41614]: osdmap e831: 8 total, 5 up, 8 in 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[41614]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[41614]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.929 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:02 smithi028 ceph-mon[41614]: pgmap v1280: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:02 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T15:13:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:02 smithi134 ceph-mon[33714]: osdmap e831: 8 total, 5 up, 8 in 2024-03-30T15:13:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:02 smithi134 ceph-mon[33714]: from='osd.4 v1:172.21.15.134:6800/4077237787' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:02 smithi134 ceph-mon[33714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-03-30T15:13:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:02 smithi134 ceph-mon[33714]: pgmap v1280: 246 pgs: 1 down, 1 stale+undersized+peered, 12 active+clean+remapped, 49 active+clean, 26 undersized+peered, 98 active+undersized, 9 undersized+degraded+peered, 4 active+undersized+remapped, 46 active+undersized+degraded; 4.5 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 206/735 objects degraded (28.027%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:02.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:02.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:02.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:13:03.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:03.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:03.220 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:03.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:03.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:03.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:03.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:03.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:03.649 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: osd.4 v1:172.21.15.134:6800/4077237787 boot 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: osdmap e832: 8 total, 6 up, 8 in 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:13:03.650 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:03 smithi134 ceph-mon[33714]: osdmap e833: 8 total, 6 up, 8 in 2024-03-30T15:13:03.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: osd.4 v1:172.21.15.134:6800/4077237787 boot 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: osdmap e832: 8 total, 6 up, 8 in 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[29876]: osdmap e833: 8 total, 6 up, 8 in 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: Health check update: 2 osds down (OSD_DOWN) 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: osd.4 v1:172.21.15.134:6800/4077237787 boot 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: osdmap e832: 8 total, 6 up, 8 in 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:13:03.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:13:03.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:03 smithi028 ceph-mon[41614]: osdmap e833: 8 total, 6 up, 8 in 2024-03-30T15:13:03.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:03.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:03.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:03.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:04.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:04.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:04.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:04.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:04.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:04.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:04.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:04.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:04.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:04.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: pgmap v1283: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[29876]: osdmap e834: 8 total, 6 up, 8 in 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:13:04.689 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: pgmap v1283: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:04.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:04 smithi028 ceph-mon[41614]: osdmap e834: 8 total, 6 up, 8 in 2024-03-30T15:13:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:04.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:04.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:04.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:04.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:13:04.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:13:04.772 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: pgmap v1283: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:04.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:04 smithi134 ceph-mon[33714]: osdmap e834: 8 total, 6 up, 8 in 2024-03-30T15:13:04.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:04.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:04.901 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:04.901 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:04.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:04.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:04.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:05.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:05.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:05.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:05.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:05.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:05.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:13:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:05.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.430+0000 7f829100b700 1 -- 172.21.15.28:0/3326740899 >> v1:172.21.15.28:6789/0 conn(0x7f828c101130 legacy=0x7f828c101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.430+0000 7f829100b700 1 -- 172.21.15.28:0/3326740899 shutdown_connections 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.430+0000 7f829100b700 1 -- 172.21.15.28:0/3326740899 >> 172.21.15.28:0/3326740899 conn(0x7f828c0fc9e0 msgr2=0x7f828c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.431+0000 7f829100b700 1 -- 172.21.15.28:0/3326740899 shutdown_connections 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.431+0000 7f829100b700 1 -- 172.21.15.28:0/3326740899 wait complete. 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.431+0000 7f829100b700 1 librados: starting msgr at 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.431+0000 7f829100b700 1 librados: starting objecter 2024-03-30T15:13:05.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.431+0000 7f829100b700 1 Processor -- start 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 -- start start 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 librados: setting wanted keys 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 librados: calling monclient init 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f828c1a6040 con 0x7f828c109dc0 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f828c1a7260 con 0x7f828c10d960 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f829100b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f828c1a84a0 con 0x7f828c101130 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f828ad9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f828c101130 0x7f828c1064c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50210/0 (socket says 172.21.15.28:50210) 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f828ad9d700 1 -- 172.21.15.28:0/627572624 learned_addr learned my addr 172.21.15.28:0/627572624 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f828a59c700 1 --1- 172.21.15.28:0/627572624 >> v1:172.21.15.28:6789/0 conn(0x7f828c109dc0 0x7f828c1a0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45652/0 (socket says 172.21.15.28:45652) 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter ms_handle_connect 0x7f828c109dc0 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter ms_handle_connect 0x7f828c101130 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter ms_handle_connect 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289898939 0 0) 0x7f828c1a6040 con 0x7f828c109dc0 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82740031f0 con 0x7f828c109dc0 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281690581 0 0) 0x7f828c1a7260 con 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f828c1a6040 con 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352063982 0 0) 0x7f828c1a84a0 con 0x7f828c101130 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.432+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f828c1a7260 con 0x7f828c101130 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 303529160 0 0) 0x7f828c1a6040 con 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f828c1a84a0 con 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8284001bf0 con 0x7f828c10d960 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1991953683 0 0) 0x7f828c1a7260 con 0x7f828c101130 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f828c1a6040 con 0x7f828c101130 2024-03-30T15:13:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1386784619 0 0) 0x7f82740031f0 con 0x7f828c109dc0 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f828c1a7260 con 0x7f828c109dc0 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f827c001510 con 0x7f828c101130 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8280001830 con 0x7f828c109dc0 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 335892137 0 0) 0x7f828c1a84a0 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 >> v1:172.21.15.28:6790/0 conn(0x7f828c101130 legacy=0x7f828c1064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 >> v1:172.21.15.28:6789/0 conn(0x7f828c109dc0 legacy=0x7f828c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c1a96e0 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f828c1a86d0 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f829100b700 10 client.50995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f828c1a8b90 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f829100b700 1 librados: init done 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8284003ea0 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.433+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8284004dc0 con 0x7f828c10d960 2024-03-30T15:13:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.434+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8278005100 con 0x7f828c10d960 2024-03-30T15:13:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.434+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8284003a00 con 0x7f828c10d960 2024-03-30T15:13:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(834..834 src has 442..834) ==== 10311+0+0 (unknown 171624173 0 0) 0x7f8284088ff0 con 0x7f828c10d960 2024-03-30T15:13:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 osd_map(834..834 src has 442..834) 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 3 client.50995.objecter handle_osd_map got epochs [834,834] > 0 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 3 client.50995.objecter handle_osd_map decoding full epoch 834 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 20 client.50995.objecter dump_active .. 0 homeless 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v834) ==== 54+0+16 (unknown 2187923809 0 2420000192) 0x7f828408b450 con 0x7f828c10d960 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f82740704f0 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 10 client.50995.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 1 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 20 client.50995.objecter _get_session osd=-1 returning homeless 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 10 client.50995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=835}) -- 0x7f82780053e0 con 0x7f828c10d960 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.435+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8278000e30 con 0x7f828c10d960 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_get_version_reply(handle=1 version=834) ==== 24+0+0 (unknown 2928091753 0 0) 0x7f8284055070 con 0x7f828c10d960 2024-03-30T15:13:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f828959a700 10 client.50995.objecter _check_command_map_dne tid 1 current 834 map_dne_bound 834 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f828959a700 10 client.50995.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f828959a700 15 client.50995.objecter _session_command_op_remove -1 1 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 10 client.50995.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 2 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 2 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 1 2 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 4 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 10 client.50995.objecter _send_command 2 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f8278002360 con 0x7f8278002790 2024-03-30T15:13:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.436+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f8278002790 2024-03-30T15:13:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.437+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8278002360 con 0x7f8278002790 2024-03-30T15:13:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.437+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 2: 0 ) 2024-03-30T15:13:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.437+0000 7f8273fff700 10 client.50995.objecter _finish_command 2 = system:0 2024-03-30T15:13:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.437+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 4 2024-03-30T15:13:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.437+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 1 2 2024-03-30T15:13:05.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:05.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 10 client.50995.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278001bb0 osd=1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 1 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 4 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 10 client.50995.objecter _send_command 3 2024-03-30T15:13:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.455+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8278004cf0 con 0x7f8278002790 2024-03-30T15:13:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8278004cf0 con 0x7f8278002790 2024-03-30T15:13:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 10 client.50995.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278001bb0 osd=1 4 2024-03-30T15:13:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 1 3 2024-03-30T15:13:05.459 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f829100b700 10 client.50995.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 4 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f829100b700 20 client.50995.objecter _get_session osd=-1 returning homeless 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f829100b700 10 client.50995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=2) -- 0x7f8278001970 con 0x7f828c10d960 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== mon.1 v1:172.21.15.134:6789/0 11 ==== mon_get_version_reply(handle=2 version=834) ==== 24+0+0 (unknown 1706758922 0 0) 0x7f82840533e0 con 0x7f828c10d960 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8288d99700 10 client.50995.objecter _check_command_map_dne tid 4 current 834 map_dne_bound 834 2024-03-30T15:13:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8288d99700 10 client.50995.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.457+0000 7f8288d99700 15 client.50995.objecter _session_command_op_remove -1 4 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 10 client.50995.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 5 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 5 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 3 5 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 4 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 10 client.50995.objecter _send_command 5 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f82780049a0 con 0x7f8278004400 2024-03-30T15:13:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.458+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f8278004400 2024-03-30T15:13:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.459+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f82780049a0 con 0x7f8278004400 2024-03-30T15:13:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.459+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 5: 0 ) 2024-03-30T15:13:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.459+0000 7f8273fff700 10 client.50995.objecter _finish_command 5 = system:0 2024-03-30T15:13:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.459+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 4 2024-03-30T15:13:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.459+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 3 5 2024-03-30T15:13:05.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 10 client.50995.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 6 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 6 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278003d80 osd=3 3 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 3 6 2024-03-30T15:13:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 4 2024-03-30T15:13:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 10 client.50995.objecter _send_command 6 2024-03-30T15:13:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.476+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f82780038c0 con 0x7f8278004400 2024-03-30T15:13:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f82780038c0 con 0x7f8278004400 2024-03-30T15:13:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f8273fff700 10 client.50995.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278003d80 osd=3 4 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 3 6 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 10 client.50995.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 7 2024-03-30T15:13:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 7 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 4 7 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 4 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 10 client.50995.objecter _send_command 7 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.477+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6800/4077237787 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8278014260 con 0x7f8278010750 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.478+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f8278010750 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.479+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8278014260 con 0x7f8278010750 2024-03-30T15:13:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.479+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 7: 0 ) 2024-03-30T15:13:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.479+0000 7f8273fff700 10 client.50995.objecter _finish_command 7 = system:0 2024-03-30T15:13:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.479+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 4 2024-03-30T15:13:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.479+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 4 7 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 10 client.50995.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 8 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 8 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f8278003b50 osd=4 3 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 4 8 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 4 2024-03-30T15:13:05.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 10 client.50995.objecter _send_command 8 2024-03-30T15:13:05.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.495+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6800/4077237787 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f827800b280 con 0x7f8278010750 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.497+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f827800b280 con 0x7f8278010750 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.497+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.497+0000 7f8273fff700 10 client.50995.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f8278003b50 osd=4 4 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 4 8 2024-03-30T15:13:05.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 10 client.50995.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 9 2024-03-30T15:13:05.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 9 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 5 9 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 4 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 10 client.50995.objecter _send_command 9 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f827800a650 con 0x7f8278008b10 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.498+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f8278008b10 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.499+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f827800a650 con 0x7f8278008b10 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.499+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 9: 0 ) 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.499+0000 7f8273fff700 10 client.50995.objecter _finish_command 9 = system:0 2024-03-30T15:13:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.499+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 4 2024-03-30T15:13:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.499+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 5 9 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 10 client.50995.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 10 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 10 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827800b4e0 osd=5 3 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 5 10 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 4 2024-03-30T15:13:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 10 client.50995.objecter _send_command 10 2024-03-30T15:13:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.517+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f827800a8a0 con 0x7f8278008b10 2024-03-30T15:13:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.518+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f827800a8a0 con 0x7f8278008b10 2024-03-30T15:13:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.518+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.518+0000 7f8273fff700 10 client.50995.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.518+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827800b4e0 osd=5 4 2024-03-30T15:13:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.518+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 5 10 2024-03-30T15:13:05.520 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 10 client.50995.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 11 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 11 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 6 11 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 4 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 10 client.50995.objecter _send_command 11 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f827801f9d0 con 0x7f827801bec0 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.519+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f827801bec0 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.520+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f827801f9d0 con 0x7f827801bec0 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.520+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 11: 0 ) 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.520+0000 7f8273fff700 10 client.50995.objecter _finish_command 11 = system:0 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.520+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 4 2024-03-30T15:13:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.520+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 6 11 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 10 client.50995.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 12 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 12 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827800a8a0 osd=6 3 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 6 12 2024-03-30T15:13:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 4 2024-03-30T15:13:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 10 client.50995.objecter _send_command 12 2024-03-30T15:13:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.536+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f827801f9d0 con 0x7f827801bec0 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.538+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f827801f9d0 con 0x7f827801bec0 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.538+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.538+0000 7f8273fff700 10 client.50995.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.538+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827800a8a0 osd=6 4 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.538+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 6 12 2024-03-30T15:13:05.540 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 10 client.50995.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 13 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 13 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 7 13 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 4 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 10 client.50995.objecter _send_command 13 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8278023dc0 con 0x7f82780200f0 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.539+0000 7f8273fff700 10 client.50995.objecter ms_handle_connect 0x7f82780200f0 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.540+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8278023dc0 con 0x7f82780200f0 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.540+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 13: 0 ) 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.540+0000 7f8273fff700 10 client.50995.objecter _finish_command 13 = system:0 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.540+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 4 2024-03-30T15:13:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.540+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 7 13 2024-03-30T15:13:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 10 client.50995.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign -1 14 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 20 client.50995.objecter _get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 15 client.50995.objecter _session_command_op_remove -1 14 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 20 client.50995.objecter get_session s=0x7f827801f9d0 osd=7 3 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 15 client.50995.objecter _session_command_op_assign 7 14 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 4 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 10 client.50995.objecter _send_command 14 2024-03-30T15:13:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.556+0000 7f829100b700 1 -- 172.21.15.28:0/627572624 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8278025770 con 0x7f82780200f0 2024-03-30T15:13:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.558+0000 7f8273fff700 1 -- 172.21.15.28:0/627572624 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8278025770 con 0x7f82780200f0 2024-03-30T15:13:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.558+0000 7f8273fff700 10 client.50995.objecter ms_dispatch 0x7f828c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.558+0000 7f8273fff700 10 client.50995.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.558+0000 7f8273fff700 20 client.50995.objecter put_session s=0x7f827801f9d0 osd=7 4 2024-03-30T15:13:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:05.558+0000 7f8273fff700 15 client.50995.objecter _session_command_op_remove 7 14 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:05.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:05.585 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:05.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:05.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 147/735 objects degraded (20.000%), 40 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:13:05.709 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/627572624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:05.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 147/735 objects degraded (20.000%), 40 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:13:05.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/627572624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 147/735 objects degraded (20.000%), 40 pgs degraded, 80 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:13:05.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/627572624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:05.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:05.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:05.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:05.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:05.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:06.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:06.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.226 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:06.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:06.288 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:06.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:06.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:06.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:06.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:06.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:06.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:06 smithi134 ceph-mon[33714]: 337.d scrub starts 2024-03-30T15:13:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:06 smithi134 ceph-mon[33714]: 337.d scrub ok 2024-03-30T15:13:06.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:06 smithi134 ceph-mon[33714]: pgmap v1285: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:06.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[29876]: 337.d scrub starts 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[29876]: 337.d scrub ok 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[29876]: pgmap v1285: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[41614]: 337.d scrub starts 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[41614]: 337.d scrub ok 2024-03-30T15:13:06.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:06 smithi028 ceph-mon[41614]: pgmap v1285: 246 pgs: 19 remapped+peering, 1 stale+undersized+peered, 12 active+clean+remapped, 86 active+clean, 21 undersized+peered, 66 active+undersized, 4 undersized+degraded+peered, 1 active+undersized+remapped, 36 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 147/735 objects degraded (20.000%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:06.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:06.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:13:06.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:13:06.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:06.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:06.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:07.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:07.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:07.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:07.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:07.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.345+0000 7fb0dc214700 1 -- 172.21.15.28:0/890689868 >> v1:172.21.15.28:6789/0 conn(0x7fb0d410ea60 legacy=0x7fb0d4110f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.345+0000 7fb0dc214700 1 -- 172.21.15.28:0/890689868 shutdown_connections 2024-03-30T15:13:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.345+0000 7fb0dc214700 1 -- 172.21.15.28:0/890689868 >> 172.21.15.28:0/890689868 conn(0x7fb0d4077370 msgr2=0x7fb0d40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.345+0000 7fb0dc214700 1 -- 172.21.15.28:0/890689868 shutdown_connections 2024-03-30T15:13:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.345+0000 7fb0dc214700 1 -- 172.21.15.28:0/890689868 wait complete. 2024-03-30T15:13:07.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 librados: starting msgr at 2024-03-30T15:13:07.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 librados: starting objecter 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 Processor -- start 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 -- start start 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 librados: setting wanted keys 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 librados: calling monclient init 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0d41a7140 con 0x7fb0d410ea60 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0d41a8360 con 0x7fb0d4100740 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0dc214700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0d41a95a0 con 0x7fb0d410aec0 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0d97af700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb0d410aec0 0x7fb0d41a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:50230/0 (socket says 172.21.15.28:50230) 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.346+0000 7fb0d97af700 1 -- 172.21.15.28:0/4097323662 learned_addr learned my addr 172.21.15.28:0/4097323662 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter ms_handle_connect 0x7fb0d410ea60 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter ms_handle_connect 0x7fb0d410aec0 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:07.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter ms_handle_connect 0x7fb0d4100740 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229511154 0 0) 0x7fb0d41a7140 con 0x7fb0d410ea60 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c00039c0 con 0x7fb0d410ea60 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707840957 0 0) 0x7fb0d41a95a0 con 0x7fb0d410aec0 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d41a7140 con 0x7fb0d410aec0 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589017441 0 0) 0x7fb0d41a8360 con 0x7fb0d4100740 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d41a95a0 con 0x7fb0d4100740 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2956730248 0 0) 0x7fb0c00039c0 con 0x7fb0d410ea60 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0d41a8360 con 0x7fb0d410ea60 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0cc0046a0 con 0x7fb0d410ea60 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3872475377 0 0) 0x7fb0d41a7140 con 0x7fb0d410aec0 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0c00039c0 con 0x7fb0d410aec0 2024-03-30T15:13:07.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0c4001890 con 0x7fb0d410aec0 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 12653566 0 0) 0x7fb0d41a95a0 con 0x7fb0d4100740 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0d41a7140 con 0x7fb0d4100740 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0d00017f0 con 0x7fb0d4100740 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1666797739 0 0) 0x7fb0d41a8360 con 0x7fb0d410ea60 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 >> v1:172.21.15.28:6790/0 conn(0x7fb0d410aec0 legacy=0x7fb0d41a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 >> v1:172.21.15.134:6789/0 conn(0x7fb0d4100740 legacy=0x7fb0d41075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.347+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d41aa7e0 con 0x7fb0d410ea60 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d41a97d0 con 0x7fb0d410ea60 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0dc214700 10 client.25803.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb0cc003ea0 con 0x7fb0d410ea60 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0d41a9c90 con 0x7fb0d410ea60 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0dc214700 1 librados: init done 2024-03-30T15:13:07.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.348+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb0cc0053e0 con 0x7fb0d410ea60 2024-03-30T15:13:07.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.349+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d4103800 con 0x7fb0d410ea60 2024-03-30T15:13:07.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.349+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb0cc003a00 con 0x7fb0d410ea60 2024-03-30T15:13:07.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(834..834 src has 442..834) ==== 10311+0+0 (unknown 171624173 0 0) 0x7fb0cc089150 con 0x7fb0d410ea60 2024-03-30T15:13:07.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 10 client.25803.objecter ms_dispatch 0x7fb0d4001550 osd_map(834..834 src has 442..834) 2024-03-30T15:13:07.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 3 client.25803.objecter handle_osd_map got epochs [834,834] > 0 2024-03-30T15:13:07.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 3 client.25803.objecter handle_osd_map decoding full epoch 834 2024-03-30T15:13:07.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 20 client.25803.objecter dump_active .. 0 homeless 2024-03-30T15:13:07.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.350+0000 7fb0caffd700 10 client.25803.objecter ms_handle_connect 0x7fb0c00704f0 2024-03-30T15:13:07.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.355+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb0cc055b00 con 0x7fb0d410ea60 2024-03-30T15:13:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:07.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:07.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:07.579+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fb0d4105ba0 con 0x7fb0d410ea60 2024-03-30T15:13:07.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:07.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:07.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:07.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:07.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:13:07.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:13:07.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:13:07.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:13:07.788 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T15:13:07.789 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T15:13:07.789 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:07 smithi134 ceph-mon[33714]: osdmap e835: 8 total, 6 up, 8 in 2024-03-30T15:13:07.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T15:13:07.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[41614]: osdmap e835: 8 total, 6 up, 8 in 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T15:13:07.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:07 smithi028 ceph-mon[29876]: osdmap e835: 8 total, 6 up, 8 in 2024-03-30T15:13:07.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:07.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:08.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:08.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:08.161 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:08.161 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:13:08.161 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:13:08.161 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:08.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:08.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:08.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:08.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:08.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:08.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:08.515 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.519+0000 7fb0caffd700 1 -- 172.21.15.28:0/4097323662 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v836) ==== 87+0+0 (unknown 569523649 0 0) 0x7fb0cc059550 con 0x7fb0d410ea60 2024-03-30T15:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 10 librados: watch_flush enter 2024-03-30T15:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 10 librados: watch_flush exit 2024-03-30T15:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 20 client.25803.objecter shutdown clearing up homeless session... 2024-03-30T15:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 10 client.25803.objecter successfully canceled tick 2024-03-30T15:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb0c00704f0 legacy=0x7fb0c00729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.522+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 >> v1:172.21.15.28:6789/0 conn(0x7fb0d410ea60 legacy=0x7fb0d41a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.523+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 shutdown_connections 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.523+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 >> 172.21.15.28:0/4097323662 conn(0x7fb0d4077370 msgr2=0x7fb0d410f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.523+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 shutdown_connections 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.523+0000 7fb0dc214700 1 -- 172.21.15.28:0/4097323662 wait complete. 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:08.523+0000 7fb0dc214700 1 librados: shutdown 2024-03-30T15:13:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:13:08.602 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:08.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:08.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:08.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:08.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:08.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:08.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: 4.f deep-scrub starts 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: 4.f deep-scrub ok 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:08.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[29876]: pgmap v1287: 246 pgs: 19 remapped+peering, 12 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 1 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: 4.f deep-scrub starts 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: 4.f deep-scrub ok 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:08.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:08 smithi028 ceph-mon[41614]: pgmap v1287: 246 pgs: 19 remapped+peering, 12 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 1 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:08.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: 4.f deep-scrub starts 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: 4.f deep-scrub ok 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:08.895 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:08 smithi134 ceph-mon[33714]: pgmap v1287: 246 pgs: 19 remapped+peering, 12 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 1 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 17/735 objects misplaced (2.313%) 2024-03-30T15:13:08.920 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-30T15:13:08.921 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-03-30T15:13:08.921 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T15:13:08.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl start ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T15:13:08.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:09.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:09.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:09.223 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:09 smithi028 systemd[1]: Starting Ceph osd.2 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:13:09.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:09.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:09.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:09.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:09.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[41614]: 42.1a scrub starts 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[41614]: 42.1a scrub ok 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[41614]: osdmap e836: 8 total, 6 up, 8 in 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[29876]: 42.1a scrub starts 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[29876]: 42.1a scrub ok 2024-03-30T15:13:09.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:09.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:09.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:09 smithi028 ceph-mon[29876]: osdmap e836: 8 total, 6 up, 8 in 2024-03-30T15:13:09.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:09.669 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:09.767 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:09 smithi134 ceph-mon[33714]: 42.1a scrub starts 2024-03-30T15:13:09.767 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:09 smithi134 ceph-mon[33714]: 42.1a scrub ok 2024-03-30T15:13:09.767 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:09 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:09.767 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4097323662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:09.768 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:09 smithi134 ceph-mon[33714]: osdmap e836: 8 total, 6 up, 8 in 2024-03-30T15:13:09.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:09.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:09.809 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:13:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:13:09.810 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:13:09.811 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:09.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:09.897 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:09 smithi028 podman[299605]: 2024-03-30 15:13:09.785985806 +0000 UTC m=+0.097561195 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:13:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:09.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:09.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:10.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:10.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:10.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:10.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:10.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:13:10.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:10.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:10.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:10.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:10.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:10 smithi028 ceph-mon[29876]: pgmap v1289: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:10.575 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:10 smithi028 ceph-mon[41614]: pgmap v1289: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:10.576 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:10 smithi028 podman[299605]: 2024-03-30 15:13:10.323274207 +0000 UTC m=+0.634849624 container create 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-30T15:13:10.576 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:10 smithi028 podman[299605]: 2024-03-30 15:13:10.56768571 +0000 UTC m=+0.879261090 container init 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:13:10.576 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:10 smithi028 podman[299605]: 2024-03-30 15:13:10.574461815 +0000 UTC m=+0.886037190 container start 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-30T15:13:10.618 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:10.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:10.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:10 smithi134 ceph-mon[33714]: pgmap v1289: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:10.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:10.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:10.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:10.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:10.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:10.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:10.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:10 smithi028 podman[299605]: 2024-03-30 15:13:10.675798178 +0000 UTC m=+0.987373558 container attach 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-30T15:13:11.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:11.035 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:11.058 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:11.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:11.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:11.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T15:13:11.314 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 bash[299605]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate[299655]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-30T15:13:11.315 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:11 smithi028 podman[299605]: 2024-03-30 15:13:11.044465602 +0000 UTC m=+1.356040961 container died 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-30T15:13:11.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:11.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:11.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:11.582 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:11.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:11.594 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:11.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:11.689 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:11.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:11.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:11.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:11.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:11.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:12.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:12.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:12.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:12.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:12.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.363+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/2048051812 >> v1:172.21.15.134:6789/0 conn(0x7ffbb4100780 legacy=0x7ffbb4100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/2048051812 shutdown_connections 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/2048051812 >> 172.21.15.28:0/2048051812 conn(0x7ffbb4077370 msgr2=0x7ffbb40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/2048051812 shutdown_connections 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/2048051812 wait complete. 2024-03-30T15:13:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 librados: starting msgr at 2024-03-30T15:13:12.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.364+0000 7ffbbbd7a700 1 librados: starting objecter 2024-03-30T15:13:12.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 Processor -- start 2024-03-30T15:13:12.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 -- start start 2024-03-30T15:13:12.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 librados: setting wanted keys 2024-03-30T15:13:12.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 librados: calling monclient init 2024-03-30T15:13:12.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a6fd0 con 0x7ffbb410af00 2024-03-30T15:13:12.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a81f0 con 0x7ffbb410ea70 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbbbd7a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffbb41a9430 con 0x7ffbb4100780 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbb9b16700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ffbb4100780 0x7ffbb4107450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57790/0 (socket says 172.21.15.28:57790) 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbb9b16700 1 -- 172.21.15.28:0/1503039075 learned_addr learned my addr 172.21.15.28:0/1503039075 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbaaffd700 10 client.?.objecter ms_handle_connect 0x7ffbb410af00 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.365+0000 7ffbaaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 10 client.?.objecter ms_handle_connect 0x7ffbb4100780 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738004376 0 0) 0x7ffbb41a9430 con 0x7ffbb4100780 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba00039c0 con 0x7ffbb4100780 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 10 client.?.objecter ms_handle_connect 0x7ffbb410ea70 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:12.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819000884 0 0) 0x7ffbb41a6fd0 con 0x7ffbb410af00 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb41a9430 con 0x7ffbb410af00 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1568093868 0 0) 0x7ffba00039c0 con 0x7ffbb4100780 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbb41a6fd0 con 0x7ffbb4100780 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffbb00018e0 con 0x7ffbb4100780 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3810769827 0 0) 0x7ffbb41a9430 con 0x7ffbb410af00 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffba00039c0 con 0x7ffbb410af00 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffba4001a30 con 0x7ffbb410af00 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773630620 0 0) 0x7ffbb41a81f0 con 0x7ffbb410ea70 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb41a9430 con 0x7ffbb410ea70 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 372947404 0 0) 0x7ffbb41a6fd0 con 0x7ffbb4100780 2024-03-30T15:13:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 >> v1:172.21.15.134:6789/0 conn(0x7ffbb410ea70 legacy=0x7ffbb41a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 >> v1:172.21.15.28:6789/0 conn(0x7ffbb410af00 legacy=0x7ffbb41a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.366+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb41aa670 con 0x7ffbb4100780 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb41a7200 con 0x7ffbb4100780 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbbbd7a700 10 client.59012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbb41a76c0 con 0x7ffbb4100780 2024-03-30T15:13:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbbbd7a700 1 librados: init done 2024-03-30T15:13:12.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ffbb0003680 con 0x7ffbb4100780 2024-03-30T15:13:12.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.367+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ffbb0004f00 con 0x7ffbb4100780 2024-03-30T15:13:12.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.368+0000 7ffbbbd7a700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ffb98005100 con 0x7ffbb4100780 2024-03-30T15:13:12.373 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:12 smithi028 podman[299814]: 2024-03-30 15:13:12.119505081 +0000 UTC m=+1.066609644 container remove 15084b8962e4921e4171f392b9567801715c897eee35d52608023171500c07f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T15:13:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.369+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ffbb00050a0 con 0x7ffbb4100780 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(836..836 src has 442..836) ==== 10311+0+0 (unknown 3861442089 0 0) 0x7ffbb008a240 con 0x7ffbb4100780 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 osd_map(836..836 src has 442..836) 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 3 client.59012.objecter handle_osd_map got epochs [836,836] > 0 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 3 client.59012.objecter handle_osd_map decoding full epoch 836 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 20 client.59012.objecter dump_active .. 0 homeless 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v836) ==== 54+0+16 (unknown 3154421399 0 2420000192) 0x7ffbb008b610 con 0x7ffbb4100780 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffba0078c10 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 1 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 20 client.59012.objecter _get_session osd=-1 returning homeless 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 10 client.59012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=837}) -- 0x7ffb9c0052b0 con 0x7ffbb4100780 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.370+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ffb9c0056d0 con 0x7ffbb4100780 2024-03-30T15:13:12.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=836) ==== 24+0+0 (unknown 2306335631 0 0) 0x7ffbb00562a0 con 0x7ffbb4100780 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffbabfff700 10 client.59012.objecter _check_command_map_dne tid 1 current 836 map_dne_bound 836 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffbabfff700 10 client.59012.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffbabfff700 15 client.59012.objecter _session_command_op_remove -1 1 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 2 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 2 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 1 2 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 4 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 10 client.59012.objecter _send_command 2 2024-03-30T15:13:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.371+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c001450 con 0x7ffb9c004df0 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c004df0 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c001450 con 0x7ffb9c004df0 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 2: 0 ) 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 2 = system:0 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 4 2024-03-30T15:13:12.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.372+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 1 2 2024-03-30T15:13:12.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:12.392 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:12.392 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.6.52:31 101.d 101:b5bd8322:test-rados-api-smithi028-93253-84::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e520)", 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 439.06951362699999, 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050572109999999998, 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.393 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320959+0000", 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.394 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320961+0000", 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7549999999999999e-06 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320966+0000", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8039999999999998e-06 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320979+0000", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2858e-05 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.320987+0000", 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1110000000000008e-06 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321177+0000", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018962499999999999 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321306+0000", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012909299999999999 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.321547+0000", 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024123600000000001 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.397 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322529+0000", 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00098205800000000006 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.322707+0000", 2024-03-30T15:13:12.398 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000177629 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325919+0000", 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032115210000000002 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.325973+0000", 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4014999999999997e-05 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.399 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:05:53.326016+0000", 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3506e-05 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(42.b,QREQUEST,e:589,from:4,reservation_nonce:7)", 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 352.57624475699998, 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005843053, 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.400 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.401 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814228+0000", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814230+0000", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0260000000000002e-06 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814236+0000", 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.913e-06 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814251+0000", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5138e-05 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814259+0000", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2509999999999994e-06 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814321+0000", 2024-03-30T15:13:12.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1585000000000001e-05 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.814327+0000", 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3740000000000003e-06 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.819987+0000", 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005660389 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820005+0000", 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7144000000000001e-05 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:07:19.820071+0000", 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6233000000000001e-05 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:14 323.1c 323:38000000:%01:::head [pgnls start_epoch 648] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e648)", 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 295.00451846099998, 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035226210000000001, 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 14 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.406 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385954+0000", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385956+0000", 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3300000000000001e-06 2024-03-30T15:13:12.407 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385963+0000", 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2639999999999997e-06 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385971+0000", 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7199999999999995e-06 2024-03-30T15:13:12.408 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.410 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.385979+0000", 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8450000000000003e-06 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386035+0000", 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5769000000000001e-05 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.386039+0000", 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3150000000000002e-06 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389352+0000", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033128849999999998 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389388+0000", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6260999999999997e-05 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:17.389477+0000", 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8232000000000005e-05 2024-03-30T15:13:12.412 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:48 323.1c 323:38000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 293.00443727599998, 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033340269999999998, 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:12.413 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 48 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386035+0000", 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.414 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386038+0000", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7089999999999998e-06 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386052+0000", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3910000000000001e-05 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386066+0000", 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3679000000000001e-05 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386075+0000", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9979999999999992e-06 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386131+0000", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6069000000000001e-05 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.386135+0000", 2024-03-30T15:13:12.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6399999999999996e-06 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389257+0000", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031219440000000002 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389283+0000", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5605e-05 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.389369+0000", 2024-03-30T15:13:12.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6472999999999997e-05 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:374 326.1 326:86d02a7e:test-rados-api-smithi028-93253-119::foo:3 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e678)", 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 253.019239615, 2024-03-30T15:13:12.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098792810000000002, 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 374 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371233+0000", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371235+0000", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3199999999999998e-06 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371242+0000", 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.72e-06 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371254+0000", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2452999999999999e-05 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371267+0000", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2567e-05 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371317+0000", 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9808999999999997e-05 2024-03-30T15:13:12.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.371321+0000", 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.279e-06 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373335+0000", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002014264 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.373568+0000", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023216 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379648+0000", 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060801650000000002 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.379884+0000", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023607599999999999 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381008+0000", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011244440000000001 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:59.381112+0000", 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010402400000000001 2024-03-30T15:13:12.424 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:739 328.6 328:60000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 234.92190235699999, 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003540091, 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.425 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 739 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468570+0000", 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468573+0000", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1130000000000001e-06 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468584+0000", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0752e-05 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468605+0000", 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0743000000000001e-05 2024-03-30T15:13:12.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468617+0000", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1994000000000001e-05 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468677+0000", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9876000000000001e-05 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.468681+0000", 2024-03-30T15:13:12.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0949999999999998e-06 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.471982+0000", 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033012670000000001 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472017+0000", 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4706999999999998e-05 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.472110+0000", 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3543999999999997e-05 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.2.18:218 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 228.001942383, 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038503399999999998, 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.430 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388530+0000", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388540+0000", 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0299000000000001e-05 2024-03-30T15:13:12.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388555+0000", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4694e-05 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388588+0000", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3259000000000001e-05 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388615+0000", 2024-03-30T15:13:12.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6579000000000001e-05 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388773+0000", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015847999999999999 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388786+0000", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2490000000000001e-05 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392259+0000", 2024-03-30T15:13:12.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034730429999999999 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392324+0000", 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5017000000000004e-05 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392381+0000", 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6478999999999999e-05 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.434 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.2.18:219 322.7s0 322:f8676c2a:test-rados-api-smithi028-93253-125::bam:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 228.00181640599999, 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047187009999999996, 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:13:12.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388656+0000", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388658+0000", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.844e-06 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388670+0000", 2024-03-30T15:13:12.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1576000000000001e-05 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388690+0000", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0233000000000001e-05 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388708+0000", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8253e-05 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388886+0000", 2024-03-30T15:13:12.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017824900000000001 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting_for_map not empty", 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.388923+0000", 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6492999999999998e-05 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392406+0000", 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003483112 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.392456+0000", 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0176000000000002e-05 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.393375+0000", 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091876499999999999 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.1.13:98 322.7s0 322:e28744e7:test-rados-api-smithi028-93253-133::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e740)", 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:13:12.439 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 189.98878665500001, 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043215160000000001, 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401686+0000", 2024-03-30T15:13:12.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401689+0000", 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3390000000000001e-06 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401704+0000", 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4541e-05 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401725+0000", 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0951e-05 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401740+0000", 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5288e-05 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401813+0000", 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3269999999999995e-05 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401819+0000", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8479999999999997e-06 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.405904+0000", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040851610000000003 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406007+0000", 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103118 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.443 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(osd.1.13:99 322.7s0 322:e28744e7:test-rados-api-smithi028-93253-133::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e740)", 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 189.98870898800001, 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043348759999999997, 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401764+0000", 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401765+0000", 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5300000000000002e-07 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401772+0000", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3599999999999998e-06 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401779+0000", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401786+0000", 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8889999999999996e-06 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401869+0000", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3461999999999995e-05 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting_for_map not empty", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.401874+0000", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1030000000000001e-06 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406025+0000", 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041505989999999996 2024-03-30T15:13:12.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:02.406098+0000", 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3430000000000007e-05 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.58535.0:1796 335.18 e761/757 335:1e0d35cd:test-rados-api-smithi028-93253-137::foo:head v 761'2, mlcod=0'0)", 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.93426093299999, 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010047844, 2024-03-30T15:13:12.448 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456212+0000", 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.456213+0000", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2669999999999999e-06 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458432+0000", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022190819999999998 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458445+0000", 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3278000000000001e-05 2024-03-30T15:13:12.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458458+0000", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2884000000000001e-05 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458514+0000", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6298000000000002e-05 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.458539+0000", 2024-03-30T15:13:12.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4734000000000001e-05 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466218+0000", 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076791109999999997 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:23.466259+0000", 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1189999999999997e-05 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3287 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94326471858816 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e780)", 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 148.422174498, 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.48388409100000002, 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3287 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.453 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968298+0000", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968302+0000", 2024-03-30T15:13:12.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4709999999999999e-06 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968313+0000", 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1367e-05 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968326+0000", 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3147e-05 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968335+0000", 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9339999999999995e-06 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968415+0000", 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0068999999999995e-05 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968425+0000", 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6760000000000002e-06 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.449263+0000", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.48083808 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.449643+0000", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00038021300000000001 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452182+0000", 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025391340000000002 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.58535.0:2006 337.6 e780/773 337:602f83fe:::foo:head v 780'5, mlcod=779'2)", 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 147.965015843, 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062043699999999999, 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:13:12.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425457+0000", 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425458+0000", 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2720000000000001e-06 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425485+0000", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6529e-05 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425496+0000", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1866e-05 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425504+0000", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4030000000000001e-06 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425562+0000", 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7983999999999999e-05 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.425583+0000", 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1489999999999999e-05 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:13:12.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.431493+0000", 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059094819999999998 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.431661+0000", 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016834400000000001 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25087.0:3 42.f e781/114 42:f4b50b61:::1.obj:head v 781'1, mlcod=0'0)", 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:13:12.462 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 147.941955957, 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005080513, 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448517+0000", 2024-03-30T15:13:12.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448518+0000", 2024-03-30T15:13:12.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0470000000000001e-06 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448524+0000", 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.754e-06 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448532+0000", 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8199999999999997e-06 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.448540+0000", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8690000000000004e-06 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452558+0000", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040181899999999996 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.452577+0000", 2024-03-30T15:13:12.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8814e-05 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 3 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 3 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c0011b0 osd=1 3 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 1 3 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 4 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 10 client.59012.objecter _send_command 3 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.400+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c004490 con 0x7ffb9c004df0 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.402+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c004490 con 0x7ffb9c004df0 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.402+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:12.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.402+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.402+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c0011b0 osd=1 4 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.402+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 1 3 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 4 2024-03-30T15:13:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter _get_session osd=-1 returning homeless 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 10 client.59012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7ffb9c0020d0 con 0x7ffbb4100780 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 11 ==== mon_get_version_reply(handle=2 version=836) ==== 24+0+0 (unknown 1112265452 0 0) 0x7ffbb0054610 con 0x7ffbb4100780 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffbab7fe700 10 client.59012.objecter _check_command_map_dne tid 4 current 836 map_dne_bound 836 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffbab7fe700 10 client.59012.objecter _finish_command 4 = osd:6 osd down 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffbab7fe700 15 client.59012.objecter _session_command_op_remove -1 4 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 5 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 5 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 3 5 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 4 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.403+0000 7ffba8ff9700 10 client.59012.objecter _send_command 5 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.404+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6813/1506157250 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c00cb10 con 0x7ffb9c008fd0 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.404+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c008fd0 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.405+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c00cb10 con 0x7ffb9c008fd0 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.405+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 5: 0 ) 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.405+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 5 = system:0 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.405+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 4 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.405+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 3 5 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 6 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 6 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c0024d0 osd=3 3 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 3 6 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 4 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 10 client.59012.objecter _send_command 6 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.424+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c001bb0 con 0x7ffb9c008fd0 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.425+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c001bb0 con 0x7ffb9c008fd0 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.425+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.425+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.425+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c0024d0 osd=3 4 2024-03-30T15:13:12.476 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:12.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:13:12.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453487+0000", 2024-03-30T15:13:12.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091028099999999998 2024-03-30T15:13:12.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453597+0000", 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109738 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.34511.0:36 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e789)", 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 114.610570391, 2024-03-30T15:13:12.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003981933, 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.34511", 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/3837265254", 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 36 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779902+0000", 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779905+0000", 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6759999999999999e-06 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779914+0000", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3349999999999996e-06 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779931+0000", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6429000000000001e-05 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.779954+0000", 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3750000000000001e-05 2024-03-30T15:13:12.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.780027+0000", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2199000000000004e-05 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.780033+0000", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2009999999999996e-06 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.783773+0000", 2024-03-30T15:13:12.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037399569999999999 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:17.783884+0000", 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011138599999999999 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3408 5.e 5:7759931f:::notify.3:head [watch ping cookie 94326471855360 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 89.695106694000003, 2024-03-30T15:13:12.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73455413700000005, 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3408 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695366+0000", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695367+0000", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9999999999999996e-07 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695370+0000", 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.427e-06 2024-03-30T15:13:12.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695378+0000", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3620000000000006e-06 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695385+0000", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0809999999999999e-06 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695489+0000", 2024-03-30T15:13:12.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104722 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695502+0000", 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2381000000000001e-05 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.427572+0000", 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73207046600000003 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:13:12.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.427901+0000", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032879799999999999 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429748+0000", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018470139999999999 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429832+0000", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3953000000000001e-05 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429920+0000", 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8133000000000005e-05 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3416 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94326471855360 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 89.694720927000006, 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73630579799999996, 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.488 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3416 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695752+0000", 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695753+0000", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5239999999999999e-06 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695758+0000", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2059999999999998e-06 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695769+0000", 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0219e-05 2024-03-30T15:13:12.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695777+0000", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0919999999999998e-06 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695832+0000", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5680999999999998e-05 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695842+0000", 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0159999999999999e-05 2024-03-30T15:13:12.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428004+0000", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73216151900000004 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428057+0000", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2581999999999999e-05 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.429961+0000", 2024-03-30T15:13:12.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001904642 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430042+0000", 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0393000000000001e-05 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.432057+0000", 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00201578 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3417 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94326471855360 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 89.694675748999998, 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73634551999999998, 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3417 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695797+0000", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695798+0000", 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.09e-07 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695801+0000", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2760000000000001e-06 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695818+0000", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7108999999999998e-05 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695825+0000", 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6170000000000003e-06 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695876+0000", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1863000000000001e-05 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:42.695887+0000", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0618999999999999e-05 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428149+0000", 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73226165499999996 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for degraded object", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.428204+0000", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5049000000000001e-05 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430518+0000", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023138199999999999 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.430600+0000", 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2034999999999996e-05 2024-03-30T15:13:12.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:43.432142+0000", 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015426680000000001 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3548 5.e 5:7759931f:::notify.3:head [watch ping cookie 94326471855360 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e825)", 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:12:47.706313+0000", 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.684159365999999, 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090398500000000005, 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.499 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3548 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706313+0000", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706313+0000", 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706314+0000", 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1200000000000002e-07 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706334+0000", 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0554e-05 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706341+0000", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4500000000000001e-06 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706347+0000", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1519999999999997e-06 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.706400+0000", 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2815000000000002e-05 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.707021+0000", 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00062109999999999997 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:47.707217+0000", 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000196202 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3557 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94326471854208 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e827)", 2024-03-30T15:13:12.503 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:12:52.707174+0000", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.68329859, 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00056559199999999996, 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3557 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707174+0000", 2024-03-30T15:13:12.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707174+0000", 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707177+0000", 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4060000000000001e-06 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:12.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707186+0000", 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4859999999999997e-06 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707203+0000", 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7010000000000001e-05 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707214+0000", 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1212e-05 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707377+0000", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016274400000000001 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707463+0000", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5921999999999995e-05 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:12:52.707740+0000", 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027681199999999998 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:12.508 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:12.508 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.425+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 3 6 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 7 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 7 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 4 7 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 4 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 10 client.59012.objecter _send_command 7 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6800/4077237787 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c008910 con 0x7ffb9c00de40 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.426+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c00de40 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.427+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c008910 con 0x7ffb9c00de40 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.427+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 7: 0 ) 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.427+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 7 = system:0 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.427+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 4 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.427+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 4 7 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.438+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== mon.2 v1:172.21.15.28:6790/0 12 ==== osd_map(837..837 src has 442..837) ==== 288+0+0 (unknown 1991328574 0 0) 0x7ffbb0001100 con 0x7ffbb4100780 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.438+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 osd_map(837..837 src has 442..837) 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.438+0000 7ffbaaffd700 3 client.59012.objecter handle_osd_map got epochs [837,837] > 836 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.438+0000 7ffbaaffd700 3 client.59012.objecter handle_osd_map decoding incremental epoch 837 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.438+0000 7ffbaaffd700 20 client.59012.objecter dump_active .. 0 homeless 2024-03-30T15:13:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 8 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 8 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c008110 osd=4 3 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 4 8 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 4 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 10 client.59012.objecter _send_command 8 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.447+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6800/4077237787 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c00f260 con 0x7ffb9c00de40 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c00f260 con 0x7ffb9c00de40 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c008110 osd=4 4 2024-03-30T15:13:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 4 8 2024-03-30T15:13:12.513 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 9 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 9 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 5 9 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 4 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 10 client.59012.objecter _send_command 9 2024-03-30T15:13:12.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.450+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6804/3668151179 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c016700 con 0x7ffb9c012bc0 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c012bc0 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c016700 con 0x7ffb9c012bc0 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 9: 0 ) 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 9 = system:0 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 4 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.451+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 5 9 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 10 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 10 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00f4c0 osd=5 3 2024-03-30T15:13:12.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 5 10 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 4 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 10 client.59012.objecter _send_command 10 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.469+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c00ed40 con 0x7ffb9c012bc0 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c00ed40 con 0x7ffb9c012bc0 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00f4c0 osd=5 4 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 5 10 2024-03-30T15:13:12.517 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 11 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 11 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 6 11 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 4 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 10 client.59012.objecter _send_command 11 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.471+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6808/2135224194 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c00ed20 con 0x7ffb9c00d9a0 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c00d9a0 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c00ed20 con 0x7ffb9c00d9a0 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 11: 0 ) 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 11 = system:0 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 4 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.472+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 6 11 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 12 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 12 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00eaf0 osd=6 3 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 6 12 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 4 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 10 client.59012.objecter _send_command 12 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.490+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c01b2b0 con 0x7ffb9c00d9a0 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.491+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c01b2b0 con 0x7ffb9c00d9a0 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.491+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.491+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.491+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00eaf0 osd=6 4 2024-03-30T15:13:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.491+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 6 12 2024-03-30T15:13:12.521 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 13 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 13 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 7 13 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 4 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 10 client.59012.objecter _send_command 13 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6812/3758190298 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ffb9c01d550 con 0x7ffb9c01b9d0 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.492+0000 7ffbaaffd700 10 client.59012.objecter ms_handle_connect 0x7ffb9c01b9d0 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.493+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffb9c01d550 con 0x7ffb9c01b9d0 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.493+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 13: 0 ) 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.493+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 13 = system:0 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.493+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 4 2024-03-30T15:13:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.493+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 7 13 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 10 client.59012.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign -1 14 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 20 client.59012.objecter _get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_remove -1 14 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 20 client.59012.objecter get_session s=0x7ffb9c00ed80 osd=7 3 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 15 client.59012.objecter _session_command_op_assign 7 14 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 4 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 10 client.59012.objecter _send_command 14 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.510+0000 7ffba8ff9700 1 -- 172.21.15.28:0/1503039075 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffb9c01d7d0 con 0x7ffb9c01b9d0 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.512+0000 7ffbaaffd700 1 -- 172.21.15.28:0/1503039075 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ffb9c01d7d0 con 0x7ffb9c01b9d0 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.512+0000 7ffbaaffd700 10 client.59012.objecter ms_dispatch 0x7ffbb4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.512+0000 7ffbaaffd700 10 client.59012.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.512+0000 7ffbaaffd700 20 client.59012.objecter put_session s=0x7ffb9c00ed80 osd=7 4 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:12.512+0000 7ffbaaffd700 15 client.59012.objecter _session_command_op_remove 7 14 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-30T15:13:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:12.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:12.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:12.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[41614]: pgmap v1290: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1503039075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[41614]: osdmap e837: 8 total, 6 up, 8 in 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[29876]: pgmap v1290: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1503039075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:12.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:12 smithi028 ceph-mon[29876]: osdmap e837: 8 total, 6 up, 8 in 2024-03-30T15:13:12.647 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:12 smithi028 podman[300085]: 2024-03-30 15:13:12.427255551 +0000 UTC m=+0.021229841 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:13:12.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:12.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:12.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:12.918 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:12.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:12 smithi134 ceph-mon[33714]: pgmap v1290: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 2 active+undersized+remapped, 1 undersized+degraded+peered, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1503039075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:12 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 12 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:12 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 101/735 objects degraded (13.741%), 29 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:12.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:12 smithi134 ceph-mon[33714]: osdmap e837: 8 total, 6 up, 8 in 2024-03-30T15:13:13.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:13.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:13.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:13.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:13.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:13.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:13.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:13:13.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:13.440 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:13.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:13.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:13.613 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:13.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:13.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:13.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:13.806 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:13 smithi028 podman[300085]: 2024-03-30 15:13:13.497362932 +0000 UTC m=+1.091337193 container create dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc) 2024-03-30T15:13:13.806 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:13 smithi028 podman[300085]: 2024-03-30 15:13:13.713924317 +0000 UTC m=+1.307898596 container init dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-4e50bbc, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:13:13.807 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:13 smithi028 podman[300085]: 2024-03-30 15:13:13.719936824 +0000 UTC m=+1.313911086 container start dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, RELEASE=squid-4e50bbc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:13:13.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:13:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:13.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:13.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:13.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:14.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:14.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:14.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:14.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:14.222 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:13 smithi028 bash[300085]: dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 2024-03-30T15:13:14.223 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:13 smithi028 systemd[1]: Started Ceph osd.2 for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:13:14.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:14.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:14.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:14.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:14.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:14.679 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:14 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:13:14.259+0000 7fa9f23e27c0 -1 Falling back to public interface 2024-03-30T15:13:14.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:14.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:13:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:13:14.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:14.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:14.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:14.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:14.957 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:14.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:15.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:15.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:15.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- 172.21.15.28:0/444191478 >> v1:172.21.15.28:6789/0 conn(0x7ff24c10aee0 legacy=0x7ff24c10d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- 172.21.15.28:0/444191478 shutdown_connections 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- 172.21.15.28:0/444191478 >> 172.21.15.28:0/444191478 conn(0x7ff24c0fdb20 msgr2=0x7ff24c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- 172.21.15.28:0/444191478 shutdown_connections 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- 172.21.15.28:0/444191478 wait complete. 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 librados: starting msgr at 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 librados: starting objecter 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 Processor -- start 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 -- start start 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 librados: setting wanted keys 2024-03-30T15:13:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.186+0000 7ff251cfb700 1 librados: calling monclient init 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff251cfb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1a7170 con 0x7ff24c102250 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff251cfb700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1a8390 con 0x7ff24c10ea80 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff251cfb700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff24c1a95d0 con 0x7ff24c10aee0 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff24b7fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff24c102250 0x7ff24c107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41844/0 (socket says 172.21.15.28:41844) 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff24affd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff24c10aee0 0x7ff24c1a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:57828/0 (socket says 172.21.15.28:57828) 2024-03-30T15:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff24b7fe700 1 -- 172.21.15.28:0/728571133 learned_addr learned my addr 172.21.15.28:0/728571133 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter ms_handle_connect 0x7ff24c102250 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter ms_handle_connect 0x7ff24c10aee0 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter ms_handle_connect 0x7ff24c10ea80 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955553161 0 0) 0x7ff24c1a7170 con 0x7ff24c102250 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2340039c0 con 0x7ff24c102250 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969239328 0 0) 0x7ff24c1a8390 con 0x7ff24c10ea80 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff24c1a7170 con 0x7ff24c10ea80 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672430508 0 0) 0x7ff24c1a95d0 con 0x7ff24c10aee0 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff24c1a8390 con 0x7ff24c10aee0 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1838577723 0 0) 0x7ff2340039c0 con 0x7ff24c102250 2024-03-30T15:13:15.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff24c1a95d0 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff23c001960 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 602875687 0 0) 0x7ff24c1a7170 con 0x7ff24c10ea80 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2340039c0 con 0x7ff24c10ea80 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.187+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1398234278 0 0) 0x7ff24c1a8390 con 0x7ff24c10aee0 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff24c1a7170 con 0x7ff24c10aee0 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff244001c30 con 0x7ff24c10ea80 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4068498963 0 0) 0x7ff24c1a95d0 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 >> v1:172.21.15.28:6790/0 conn(0x7ff24c10aee0 legacy=0x7ff24c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 >> v1:172.21.15.134:6789/0 conn(0x7ff24c10ea80 legacy=0x7ff24c1a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c1aa810 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c1a85c0 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff251cfb700 10 client.25818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff24c1a8a80 con 0x7ff24c102250 2024-03-30T15:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff251cfb700 1 librados: init done 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff23c003b20 con 0x7ff24c102250 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.188+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff23c005c50 con 0x7ff24c102250 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.189+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff238005120 con 0x7ff24c102250 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.189+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff23c003680 con 0x7ff24c102250 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(837..837 src has 442..837) ==== 10311+0+0 (unknown 930361432 0 0) 0x7ff23c089e50 con 0x7ff24c102250 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 10 client.25818.objecter ms_dispatch 0x7ff24c001550 osd_map(837..837 src has 442..837) 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 3 client.25818.objecter handle_osd_map got epochs [837,837] > 0 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 3 client.25818.objecter handle_osd_map decoding full epoch 837 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 20 client.25818.objecter dump_active .. 0 homeless 2024-03-30T15:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.190+0000 7ff248ff9700 10 client.25818.objecter ms_handle_connect 0x7ff2340703e0 2024-03-30T15:13:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.194+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff23c056800 con 0x7ff24c102250 2024-03-30T15:13:15.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:15.237 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:15.305 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:15.305 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:15.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:15.420+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff238005410 con 0x7ff24c102250 2024-03-30T15:13:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:15 smithi134 ceph-mon[33714]: pgmap v1292: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:15.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:15.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:15.499 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:15 smithi028 ceph-mon[41614]: pgmap v1292: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:15.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:15 smithi028 ceph-mon[29876]: pgmap v1292: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:15.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:15.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:15.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:15.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:15.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:13:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:15.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:15.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:16.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:16.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:16.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:16.204 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:16.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:16.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.246+0000 7ff248ff9700 1 -- 172.21.15.28:0/728571133 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v838) ==== 97+0+0 (unknown 736643064 0 0) 0x7ff23c05a250 con 0x7ff24c102250 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 10 librados: watch_flush enter 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 10 librados: watch_flush exit 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 20 client.25818.objecter shutdown clearing up homeless session... 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 10 client.25818.objecter successfully canceled tick 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff2340703e0 legacy=0x7ff2340728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 >> v1:172.21.15.28:6789/0 conn(0x7ff24c102250 legacy=0x7ff24c107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 shutdown_connections 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.248+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 >> 172.21.15.28:0/728571133 conn(0x7ff24c0fdb20 msgr2=0x7ff24c107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.249+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 shutdown_connections 2024-03-30T15:13:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.249+0000 7ff251cfb700 1 -- 172.21.15.28:0/728571133 wait complete. 2024-03-30T15:13:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:16.249+0000 7ff251cfb700 1 librados: shutdown 2024-03-30T15:13:16.251 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:13:16.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:16.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:16.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:16.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:16.472 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:16.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:16.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:16.624 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:16.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:16.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:13:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:16.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:16.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:16.847 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:16.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:16.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:16.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:17.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:17.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:17.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:17.204 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:16 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:13:16.992+0000 7fa9f23e27c0 -1 osd.2 779 log_to_monitors true 2024-03-30T15:13:17.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:17.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:17.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:17.377 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:13:17.377 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:13:17.377 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:13:17.377 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: pgmap v1293: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: osdmap e838: 8 total, 6 up, 8 in 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.378 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:17 smithi134 ceph-mon[33714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:17.473 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:13:17.382+0000 7fa9e383a700 -1 osd.2 839 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: pgmap v1293: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: osdmap e838: 8 total, 6 up, 8 in 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[41614]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: pgmap v1293: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 751 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/728571133' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: osdmap e838: 8 total, 6 up, 8 in 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:17 smithi028 ceph-mon[29876]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T15:13:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:17.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:17.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:17.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:17.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:17.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:17.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:17.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:17.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:17.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:17.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:18.058 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:18.058 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:18.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:18.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:18.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:18.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[41614]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T15:13:18.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[41614]: osdmap e839: 8 total, 6 up, 8 in 2024-03-30T15:13:18.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[41614]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.328 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[41614]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[29876]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T15:13:18.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[29876]: osdmap e839: 8 total, 6 up, 8 in 2024-03-30T15:13:18.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[29876]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-mon[29876]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.329 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:18.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:18.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.361 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-03-30T15:13:18.361 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-03-30T15:13:18.361 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T15:13:18.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:18.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl start ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T15:13:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:18.368 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:18 smithi134 ceph-mon[33714]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T15:13:18.368 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:18 smithi134 ceph-mon[33714]: osdmap e839: 8 total, 6 up, 8 in 2024-03-30T15:13:18.368 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:18 smithi134 ceph-mon[33714]: from='osd.2 v1:172.21.15.28:6801/2294873818' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.368 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:18 smithi134 ceph-mon[33714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:18.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:18.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:18.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:18.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:18.629 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:18 smithi028 systemd[1]: Starting Ceph osd.0 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:13:18.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:13:18.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:18.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:18.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:18.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:18.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:18.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:18.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:19.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:19.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:19.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:19.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: pgmap v1296: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: Health check update: 1 osds down (OSD_DOWN) 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: osd.2 v1:172.21.15.28:6801/2294873818 boot 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: osdmap e840: 8 total, 7 up, 8 in 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:19.386 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:19 smithi134 ceph-mon[33714]: osdmap e841: 8 total, 7 up, 8 in 2024-03-30T15:13:19.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:19.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: pgmap v1296: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: Health check update: 1 osds down (OSD_DOWN) 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: osd.2 v1:172.21.15.28:6801/2294873818 boot 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: osdmap e840: 8 total, 7 up, 8 in 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:19.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[41614]: osdmap e841: 8 total, 7 up, 8 in 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: pgmap v1296: 246 pgs: 30 active+clean+remapped, 111 active+clean, 11 undersized+peered, 63 active+undersized, 1 undersized+degraded+peered, 2 active+undersized+remapped, 28 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 101/735 objects degraded (13.741%); 31/735 objects misplaced (4.218%) 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: Health check update: 1 osds down (OSD_DOWN) 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: osd.2 v1:172.21.15.28:6801/2294873818 boot 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: osdmap e840: 8 total, 7 up, 8 in 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:19.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:19 smithi028 ceph-mon[29876]: osdmap e841: 8 total, 7 up, 8 in 2024-03-30T15:13:19.640 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:19 smithi028 podman[301416]: 2024-03-30 15:13:19.266065353 +0000 UTC m=+0.027572401 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:13:19.641 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:19 smithi028 podman[301416]: 2024-03-30 15:13:19.393163718 +0000 UTC m=+0.154670746 container create 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, ceph=True, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-30T15:13:19.641 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:19 smithi028 podman[301416]: 2024-03-30 15:13:19.638031718 +0000 UTC m=+0.399538762 container init 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-30T15:13:19.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:19.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:13:19.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:19.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:19.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:19.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:19.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:19 smithi028 podman[301416]: 2024-03-30 15:13:19.644272786 +0000 UTC m=+0.405779816 container start 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-30T15:13:19.916 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:19 smithi028 podman[301416]: 2024-03-30 15:13:19.746007118 +0000 UTC m=+0.507514150 container attach 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-30T15:13:19.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:20.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:20.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:20.223 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T15:13:20.242 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T15:13:20.243 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T15:13:20.243 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-30T15:13:20.243 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T15:13:20.244 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T15:13:20.244 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-30T15:13:20.244 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T15:13:20.244 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 bash[301416]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-30T15:13:20.244 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-30T15:13:20.245 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-30T15:13:20.245 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-30T15:13:20.245 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T15:13:20.245 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate[301433]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-30T15:13:20.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:20.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:20.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:20.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:20.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:20.514 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:20 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:20.514 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:20 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:20.514 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:20 smithi134 ceph-mon[33714]: osdmap e842: 8 total, 7 up, 8 in 2024-03-30T15:13:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:20.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[29876]: osdmap e842: 8 total, 7 up, 8 in 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:20.642 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:20 smithi028 ceph-mon[41614]: osdmap e842: 8 total, 7 up, 8 in 2024-03-30T15:13:20.643 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:20 smithi028 podman[301416]: 2024-03-30 15:13:20.354264097 +0000 UTC m=+1.115771128 container died 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-30T15:13:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:20.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:20.763 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:20.763 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:20.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:20.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:20.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:21.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:21.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3765624408 >> v1:172.21.15.28:6790/0 conn(0x7fcc5010ae70 legacy=0x7fcc5010d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3765624408 shutdown_connections 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3765624408 >> 172.21.15.28:0/3765624408 conn(0x7fcc50077370 msgr2=0x7fcc500fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3765624408 shutdown_connections 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3765624408 wait complete. 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 librados: starting msgr at 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.075+0000 7fcc57f8c700 1 librados: starting objecter 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 Processor -- start 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 -- start start 2024-03-30T15:13:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 librados: setting wanted keys 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 librados: calling monclient init 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc501a7180 con 0x7fcc5010ae70 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc501a83a0 con 0x7fcc5010eb70 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc57f8c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcc501a95e0 con 0x7fcc50106fd0 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc55d28700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fcc50106fd0 0x7fcc500ff5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41826/0 (socket says 172.21.15.28:41826) 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc55d28700 1 -- 172.21.15.28:0/3732845046 learned_addr learned my addr 172.21.15.28:0/3732845046 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter ms_handle_connect 0x7fcc50106fd0 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter ms_handle_connect 0x7fcc5010ae70 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter ms_handle_connect 0x7fcc5010eb70 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993786669 0 0) 0x7fcc501a95e0 con 0x7fcc50106fd0 2024-03-30T15:13:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc380031f0 con 0x7fcc50106fd0 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230175164 0 0) 0x7fcc501a7180 con 0x7fcc5010ae70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc501a95e0 con 0x7fcc5010ae70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286929049 0 0) 0x7fcc501a83a0 con 0x7fcc5010eb70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.076+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc501a7180 con 0x7fcc5010eb70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3807433562 0 0) 0x7fcc501a95e0 con 0x7fcc5010ae70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc501a83a0 con 0x7fcc5010ae70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2344352236 0 0) 0x7fcc380031f0 con 0x7fcc50106fd0 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc501a95e0 con 0x7fcc50106fd0 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcc44004500 con 0x7fcc5010ae70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcc48001a30 con 0x7fcc50106fd0 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1532714120 0 0) 0x7fcc501a7180 con 0x7fcc5010eb70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc380031f0 con 0x7fcc5010eb70 2024-03-30T15:13:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcc40001af0 con 0x7fcc5010eb70 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1767693572 0 0) 0x7fcc501a95e0 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 >> v1:172.21.15.134:6789/0 conn(0x7fcc5010eb70 legacy=0x7fcc501a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 >> v1:172.21.15.28:6789/0 conn(0x7fcc5010ae70 legacy=0x7fcc501a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc501aa820 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc501a73b0 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc57f8c700 10 client.59030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc501a7870 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc57f8c700 1 librados: init done 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fcc48001c10 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.077+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fcc48005c50 con 0x7fcc50106fd0 2024-03-30T15:13:21.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.078+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fcc3c005100 con 0x7fcc50106fd0 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fcc48003930 con 0x7fcc50106fd0 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(842..842 src has 442..842) ==== 10253+0+0 (unknown 1776959638 0 0) 0x7fcc48089e80 con 0x7fcc50106fd0 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 osd_map(842..842 src has 442..842) 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 3 client.59030.objecter handle_osd_map got epochs [842,842] > 0 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 3 client.59030.objecter handle_osd_map decoding full epoch 842 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 20 client.59030.objecter dump_active .. 0 homeless 2024-03-30T15:13:21.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v842) ==== 54+0+16 (unknown 3573851177 0 2420000192) 0x7fcc4808a2b0 con 0x7fcc50106fd0 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc38070440 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 1 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 20 client.59030.objecter _get_session osd=-1 returning homeless 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 10 client.59030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=843}) -- 0x7fcc3c0053e0 con 0x7fcc50106fd0 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.079+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcc3c000ed0 con 0x7fcc50106fd0 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_get_version_reply(handle=1 version=842) ==== 24+0+0 (unknown 660737358 0 0) 0x7fcc48055f30 con 0x7fcc50106fd0 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc4ffff700 10 client.59030.objecter _check_command_map_dne tid 1 current 842 map_dne_bound 842 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc4ffff700 10 client.59030.objecter _finish_command 1 = osd:6 osd down 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc4ffff700 15 client.59030.objecter _session_command_op_remove -1 1 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 2 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 2 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 1 2 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 4 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 10 client.59030.objecter _send_command 2 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c004460 con 0x7fcc3c003910 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.080+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c003910 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.081+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 2: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c004460 con 0x7fcc3c003910 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.081+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 2: 0 ) 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.081+0000 7fcc4effd700 10 client.59030.objecter _finish_command 2 = system:0 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.081+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 4 2024-03-30T15:13:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.081+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 1 2 2024-03-30T15:13:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c0011c0 osd=1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 1 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 4 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 10 client.59030.objecter _send_command 3 2024-03-30T15:13:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.099+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c004e10 con 0x7fcc3c003910 2024-03-30T15:13:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c004e10 con 0x7fcc3c003910 2024-03-30T15:13:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 3: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 10 client.59030.objecter _finish_command 3 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c0011c0 osd=1 4 2024-03-30T15:13:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 1 3 2024-03-30T15:13:21.103 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 4 2024-03-30T15:13:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 4 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 2 4 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 4 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 10 client.59030.objecter _send_command 4 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6801/2294873818 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c00ca20 con 0x7fcc3c008f10 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.101+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c008f10 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.102+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 4: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc48034e50 con 0x7fcc3c008f10 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.102+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 4: 0 ) 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.102+0000 7fcc4effd700 10 client.59030.objecter _finish_command 4 = system:0 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.102+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 4 2024-03-30T15:13:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.102+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 2 4 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 5 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 5 2024-03-30T15:13:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c001ad0 osd=2 3 2024-03-30T15:13:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 2 5 2024-03-30T15:13:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 4 2024-03-30T15:13:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 10 client.59030.objecter _send_command 5 2024-03-30T15:13:21.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.119+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c0048f0 con 0x7fcc3c008f10 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c0048f0 con 0x7fcc3c008f10 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 10 client.59030.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c001ad0 osd=2 4 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 2 5 2024-03-30T15:13:21.122 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:13:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 6 2024-03-30T15:13:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 6 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 3 6 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 4 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 10 client.59030.objecter _send_command 6 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6813/1506157250 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c0036b0 con 0x7fcc3c00cbd0 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.121+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c00cbd0 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.122+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 6: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c0036b0 con 0x7fcc3c00cbd0 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.122+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 6: 0 ) 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.122+0000 7fcc4effd700 10 client.59030.objecter _finish_command 6 = system:0 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.122+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 4 2024-03-30T15:13:21.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.122+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 3 6 2024-03-30T15:13:21.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:21.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 7 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 7 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c002f60 osd=3 3 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 3 7 2024-03-30T15:13:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 4 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 10 client.59030.objecter _send_command 7 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.138+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c00df60 con 0x7fcc3c00cbd0 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c00df60 con 0x7fcc3c00cbd0 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 10 client.59030.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c002f60 osd=3 4 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 3 7 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 8 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 8 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 4 8 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 4 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 10 client.59030.objecter _send_command 8 2024-03-30T15:13:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6800/4077237787 -- command(tid 8: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c01aa40 con 0x7fcc3c016f30 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.140+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c016f30 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.141+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 8: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c01aa40 con 0x7fcc3c016f30 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.141+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 8: 0 ) 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.141+0000 7fcc4effd700 10 client.59030.objecter _finish_command 8 = system:0 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.141+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 4 2024-03-30T15:13:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.141+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 4 8 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 9 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 9 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00e1c0 osd=4 3 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 4 9 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 4 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 10 client.59030.objecter _send_command 9 2024-03-30T15:13:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.158+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c00d9e0 con 0x7fcc3c016f30 2024-03-30T15:13:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.160+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c00d9e0 con 0x7fcc3c016f30 2024-03-30T15:13:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.160+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.160+0000 7fcc4effd700 10 client.59030.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.160+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00e1c0 osd=4 4 2024-03-30T15:13:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.160+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 4 9 2024-03-30T15:13:21.163 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 10 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 10 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 5 10 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 4 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 10 client.59030.objecter _send_command 10 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6804/3668151179 -- command(tid 10: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c00d900 con 0x7fcc3c00f0b0 2024-03-30T15:13:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.161+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c00f0b0 2024-03-30T15:13:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.162+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 10: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c00d900 con 0x7fcc3c00f0b0 2024-03-30T15:13:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.162+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 10: 0 ) 2024-03-30T15:13:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.162+0000 7fcc4effd700 10 client.59030.objecter _finish_command 10 = system:0 2024-03-30T15:13:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.162+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 4 2024-03-30T15:13:21.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.162+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 5 10 2024-03-30T15:13:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:21.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 11 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 11 2024-03-30T15:13:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00d790 osd=5 3 2024-03-30T15:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 5 11 2024-03-30T15:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 4 2024-03-30T15:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 10 client.59030.objecter _send_command 11 2024-03-30T15:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.179+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c00d900 con 0x7fcc3c00f0b0 2024-03-30T15:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.180+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c00d900 con 0x7fcc3c00f0b0 2024-03-30T15:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.180+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.180+0000 7fcc4effd700 10 client.59030.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.180+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00d790 osd=5 4 2024-03-30T15:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.180+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 5 11 2024-03-30T15:13:21.182 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 12 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 12 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 6 12 2024-03-30T15:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 4 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 10 client.59030.objecter _send_command 12 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6808/2135224194 -- command(tid 12: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c0236e0 con 0x7fcc3c01fb00 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.181+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c01fb00 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.182+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 12: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c0236e0 con 0x7fcc3c01fb00 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.182+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 12: 0 ) 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.182+0000 7fcc4effd700 10 client.59030.objecter _finish_command 12 = system:0 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.182+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 4 2024-03-30T15:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.182+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 6 12 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 13 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 13 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c00db50 osd=6 3 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 6 13 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 4 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 10 client.59030.objecter _send_command 13 2024-03-30T15:13:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.198+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c025090 con 0x7fcc3c01fb00 2024-03-30T15:13:21.201 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c025090 con 0x7fcc3c01fb00 2024-03-30T15:13:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc4effd700 10 client.59030.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c00db50 osd=6 4 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 6 13 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 14 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 14 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 7 14 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 4 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 10 client.59030.objecter _send_command 14 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.200+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6812/3758190298 -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c025c60 con 0x7fcc3c0257b0 2024-03-30T15:13:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 10 client.59030.objecter ms_handle_connect 0x7fcc3c0257b0 2024-03-30T15:13:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcc3c025c60 con 0x7fcc3c0257b0 2024-03-30T15:13:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 14: 0 ) 2024-03-30T15:13:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 10 client.59030.objecter _finish_command 14 = system:0 2024-03-30T15:13:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 4 2024-03-30T15:13:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.201+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 7 14 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 10 client.59030.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign -1 15 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 20 client.59030.objecter _get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_remove -1 15 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 20 client.59030.objecter get_session s=0x7fcc3c025090 osd=7 3 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 15 client.59030.objecter _session_command_op_assign 7 15 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 4 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 10 client.59030.objecter _send_command 15 2024-03-30T15:13:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.218+0000 7fcc57f8c700 1 -- 172.21.15.28:0/3732845046 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcc3c025c60 con 0x7fcc3c0257b0 2024-03-30T15:13:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.220+0000 7fcc4effd700 1 -- 172.21.15.28:0/3732845046 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fcc3c025c60 con 0x7fcc3c0257b0 2024-03-30T15:13:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.220+0000 7fcc4effd700 10 client.59030.objecter ms_dispatch 0x7fcc50001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.220+0000 7fcc4effd700 10 client.59030.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.220+0000 7fcc4effd700 20 client.59030.objecter put_session s=0x7fcc3c025090 osd=7 4 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:21.220+0000 7fcc4effd700 15 client.59030.objecter _session_command_op_remove 7 15 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:21.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:21.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:21.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:21.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: pgmap v1299: 246 pgs: 15 remapped+peering, 26 peering, 17 active+clean+remapped, 111 active+clean, 8 undersized+peered, 48 active+undersized, 1 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/735 objects degraded (9.388%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 69/735 objects degraded (9.388%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3732845046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:21.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:13:21.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:21.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:21.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:21.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:21.699 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: pgmap v1299: 246 pgs: 15 remapped+peering, 26 peering, 17 active+clean+remapped, 111 active+clean, 8 undersized+peered, 48 active+undersized, 1 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/735 objects degraded (9.388%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 69/735 objects degraded (9.388%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3732845046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: pgmap v1299: 246 pgs: 15 remapped+peering, 26 peering, 17 active+clean+remapped, 111 active+clean, 8 undersized+peered, 48 active+undersized, 1 undersized+degraded+peered, 20 active+undersized+degraded; 4.5 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/735 objects degraded (9.388%); 19/735 objects misplaced (2.585%) 2024-03-30T15:13:21.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 69/735 objects degraded (9.388%), 21 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:21.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:21.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]: dispatch 2024-03-30T15:13:21.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3732845046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:21.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:21.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:21.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:21.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:21.973 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:21 smithi028 podman[301661]: 2024-03-30 15:13:21.718994208 +0000 UTC m=+1.587921930 container remove 87b84a2ec6dbc465e1ce6020f484c105d1b0abdc90002d69a71900cc9ba2d76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-30T15:13:22.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:22.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:22.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:22.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:22.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:22.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:22.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:22.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:22 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:22.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]': finished 2024-03-30T15:13:22.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:22 smithi134 ceph-mon[33714]: osdmap e843: 8 total, 7 up, 8 in 2024-03-30T15:13:22.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]': finished 2024-03-30T15:13:22.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[29876]: osdmap e843: 8 total, 7 up, 8 in 2024-03-30T15:13:22.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:22.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-140"}]': finished 2024-03-30T15:13:22.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:22 smithi028 ceph-mon[41614]: osdmap e843: 8 total, 7 up, 8 in 2024-03-30T15:13:22.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:22.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-30T15:13:22.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-30T15:13:22.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:22.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:22.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:22.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:22.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:22.912 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:22 smithi028 podman[302065]: 2024-03-30 15:13:22.649709492 +0000 UTC m=+0.020271005 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:13:22.912 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:22 smithi028 podman[302065]: 2024-03-30 15:13:22.774684772 +0000 UTC m=+0.145246282 container create 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-30T15:13:22.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:22.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:23.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:23.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:13:23.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:23.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:23.262 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:23 smithi028 podman[302065]: 2024-03-30 15:13:23.011463778 +0000 UTC m=+0.382025255 container init 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-30T15:13:23.262 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:23 smithi028 podman[302065]: 2024-03-30 15:13:23.017103466 +0000 UTC m=+0.387664930 container start 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:13:23.262 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:23 smithi028 bash[302065]: 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c 2024-03-30T15:13:23.262 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:23 smithi028 systemd[1]: Started Ceph osd.0 for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:13:23.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:23.404 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:13:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-03-30T15:13:23.405 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: pgmap v1302: 214 pgs: 14 remapped+peering, 22 peering, 16 active+clean+remapped, 99 active+clean, 7 undersized+peered, 42 active+undersized, 1 undersized+degraded+peered, 13 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/648 objects degraded (8.796%); 19/648 objects misplaced (2.932%) 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: osdmap e844: 8 total, 7 up, 8 in 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: osdmap e845: 8 total, 7 up, 8 in 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:23.405 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:23.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:23.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: pgmap v1302: 214 pgs: 14 remapped+peering, 22 peering, 16 active+clean+remapped, 99 active+clean, 7 undersized+peered, 42 active+undersized, 1 undersized+degraded+peered, 13 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/648 objects degraded (8.796%); 19/648 objects misplaced (2.932%) 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: osdmap e844: 8 total, 7 up, 8 in 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: osdmap e845: 8 total, 7 up, 8 in 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:23.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: pgmap v1302: 214 pgs: 14 remapped+peering, 22 peering, 16 active+clean+remapped, 99 active+clean, 7 undersized+peered, 42 active+undersized, 1 undersized+degraded+peered, 13 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/648 objects degraded (8.796%); 19/648 objects misplaced (2.932%) 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: Health check update: Reduced data availability: 11 pgs inactive (PG_AVAILABILITY) 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: osdmap e844: 8 total, 7 up, 8 in 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: osdmap e845: 8 total, 7 up, 8 in 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:23.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:23.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:23.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:23.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:23.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:23.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:23.870 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:23.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:23.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:23.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:23.972 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:13:23.564+0000 7f1a82ee87c0 -1 Falling back to public interface 2024-03-30T15:13:24.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:24.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:24.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:24.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:24.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:24.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:24.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:13:24.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:24.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] delete_pools_parallel: process_1_[94057]: starting. 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: creating pool ceph_test_rados_delete_pools_parallel.smithi028-93953 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: created object 0... 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: created object 25... 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: created object 49... 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: finishing. 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_1_[94057]: shutting down. 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_2_[94062]: starting. 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_2_[94062]: deleting pool ceph_test_rados_delete_pools_parallel.smithi028-93953 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_2_[94062]: shutting down. 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******************************* 2024-03-30T15:13:24.587 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: starting. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: creating pool ceph_test_rados_delete_pools_parallel.smithi028-93953 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: created object 0... 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: created object 25... 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: created object 49... 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: finishing. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_3_[301859]: shutting down. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******************************* 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: starting. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: listing objects. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: listed object 0... 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: listed object 25... 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: saw 50 objects 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_5_[301861]: shutting down. 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******************************* 2024-03-30T15:13:24.588 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_4_[301860]: starting. 2024-03-30T15:13:24.589 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_4_[301860]: deleting pool ceph_test_rados_delete_pools_parallel.smithi028-93953 2024-03-30T15:13:24.589 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: process_4_[301860]: shutting down. 2024-03-30T15:13:24.589 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******************************* 2024-03-30T15:13:24.589 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******************************* 2024-03-30T15:13:24.589 INFO:tasks.workunit.client.0.smithi028.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-30T15:13:24.624 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:24.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:13:24.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:24.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:24.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[29876]: osdmap e846: 8 total, 7 up, 8 in 2024-03-30T15:13:24.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[29876]: pgmap v1306: 246 pgs: 7 creating+peering, 16 unknown, 25 active+clean+remapped, 142 active+clean, 38 active+undersized, 18 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 48/786 objects degraded (6.107%); 33/786 objects misplaced (4.198%); 0 B/s, 0 objects/s recovering 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[41614]: osdmap e846: 8 total, 7 up, 8 in 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:24 smithi028 ceph-mon[41614]: pgmap v1306: 246 pgs: 7 creating+peering, 16 unknown, 25 active+clean+remapped, 142 active+clean, 38 active+undersized, 18 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 48/786 objects degraded (6.107%); 33/786 objects misplaced (4.198%); 0 B/s, 0 objects/s recovering 2024-03-30T15:13:24.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:24.867 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:24.867 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:24 smithi134 ceph-mon[33714]: osdmap e846: 8 total, 7 up, 8 in 2024-03-30T15:13:24.867 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.867 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:24.867 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:24 smithi134 ceph-mon[33714]: pgmap v1306: 246 pgs: 7 creating+peering, 16 unknown, 25 active+clean+remapped, 142 active+clean, 38 active+undersized, 18 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 48/786 objects degraded (6.107%); 33/786 objects misplaced (4.198%); 0 B/s, 0 objects/s recovering 2024-03-30T15:13:24.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:25.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:25.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:25.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:25.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.241+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1187254296 >> v1:172.21.15.28:6790/0 conn(0x7f01cc10e930 legacy=0x7f01cc110d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.241+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1187254296 shutdown_connections 2024-03-30T15:13:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.241+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1187254296 >> 172.21.15.28:0/1187254296 conn(0x7f01cc0fc7e0 msgr2=0x7f01cc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1187254296 shutdown_connections 2024-03-30T15:13:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1187254296 wait complete. 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 librados: starting msgr at 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 librados: starting objecter 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 Processor -- start 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- start start 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 librados: setting wanted keys 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 librados: calling monclient init 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01cc1a9270 con 0x7f01cc10ae20 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01cc1aa450 con 0x7f01cc103010 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01d1ba1700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f01cc1ab690 con 0x7f01cc10e930 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01cbfff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01cc10ae20 0x7f01cc1a4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35348/0 (socket says 172.21.15.28:35348) 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.242+0000 7f01cbfff700 1 -- 172.21.15.28:0/1266374080 learned_addr learned my addr 172.21.15.28:0/1266374080 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter ms_handle_connect 0x7f01cc10e930 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter ms_handle_connect 0x7f01cc10ae20 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter ms_handle_connect 0x7f01cc103010 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156047888 0 0) 0x7f01cc1ab690 con 0x7f01cc10e930 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01bc0039c0 con 0x7f01cc10e930 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791711041 0 0) 0x7f01cc1aa450 con 0x7f01cc103010 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01cc1ab690 con 0x7f01cc103010 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068968990 0 0) 0x7f01cc1a9270 con 0x7f01cc10ae20 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01cc1aa450 con 0x7f01cc10ae20 2024-03-30T15:13:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1081148310 0 0) 0x7f01bc0039c0 con 0x7f01cc10e930 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01cc1a9270 con 0x7f01cc10e930 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4241029560 0 0) 0x7f01cc1ab690 con 0x7f01cc103010 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01bc0039c0 con 0x7f01cc103010 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3777553780 0 0) 0x7f01cc1aa450 con 0x7f01cc10ae20 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01cc1ab690 con 0x7f01cc10ae20 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01c4004810 con 0x7f01cc10e930 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01c0001870 con 0x7f01cc103010 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01b8001890 con 0x7f01cc10ae20 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1162244869 0 0) 0x7f01cc1a9270 con 0x7f01cc10e930 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 >> v1:172.21.15.134:6789/0 conn(0x7f01cc103010 legacy=0x7f01cc10a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 >> v1:172.21.15.28:6789/0 conn(0x7f01cc10ae20 legacy=0x7f01cc1a4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc1ac8d0 con 0x7f01cc10e930 2024-03-30T15:13:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01cc1a94a0 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f01c4003ea0 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f01c4005350 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01d1ba1700 10 client.59057.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f01cc1a9960 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.243+0000 7f01d1ba1700 1 librados: init done 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01cc06a440 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f01c401bc50 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(847..847 src has 442..847) ==== 9670+0+0 (unknown 1512828554 0 0) 0x7f01c4088f30 con 0x7f01cc10e930 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 10 client.59057.objecter ms_dispatch 0x7f01cc001550 osd_map(847..847 src has 442..847) 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 3 client.59057.objecter handle_osd_map got epochs [847,847] > 0 2024-03-30T15:13:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 3 client.59057.objecter handle_osd_map decoding full epoch 847 2024-03-30T15:13:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 20 client.59057.objecter dump_active .. 0 homeless 2024-03-30T15:13:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.245+0000 7f01c9ffb700 10 client.59057.objecter ms_handle_connect 0x7f01bc070450 2024-03-30T15:13:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.250+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f01c4055b60 con 0x7f01cc10e930 2024-03-30T15:13:25.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:25.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:25.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:25.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.465+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f01cc1021a0 con 0x7f01cc10e930 2024-03-30T15:13:25.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:25.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.582+0000 7f01c9ffb700 1 -- 172.21.15.28:0/1266374080 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v848) ==== 91+0+0 (unknown 3113106858 0 0) 0x7f01c40595b0 con 0x7f01cc10e930 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.585+0000 7f01d1ba1700 10 librados: watch_flush enter 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.585+0000 7f01d1ba1700 10 librados: watch_flush exit 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.585+0000 7f01d1ba1700 20 client.59057.objecter shutdown clearing up homeless session... 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.585+0000 7f01d1ba1700 10 client.59057.objecter successfully canceled tick 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.586+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 >> v1:172.21.15.28:6800/3011431093 conn(0x7f01bc070450 legacy=0x7f01bc072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.586+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 >> v1:172.21.15.28:6790/0 conn(0x7f01cc10e930 legacy=0x7f01cc1a7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.586+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 shutdown_connections 2024-03-30T15:13:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.586+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 >> 172.21.15.28:0/1266374080 conn(0x7f01cc0fc7e0 msgr2=0x7f01cc110d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.586+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 shutdown_connections 2024-03-30T15:13:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.587+0000 7f01d1ba1700 1 -- 172.21.15.28:0/1266374080 wait complete. 2024-03-30T15:13:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:25.587+0000 7f01d1ba1700 1 librados: shutdown 2024-03-30T15:13:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:13:25.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:25.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive) 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: osdmap e847: 8 total, 7 up, 8 in 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1266374080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.722 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:25 smithi134 ceph-mon[33714]: from='client.59057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive) 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: osdmap e847: 8 total, 7 up, 8 in 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1266374080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[41614]: from='client.59057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive) 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: osdmap e847: 8 total, 7 up, 8 in 2024-03-30T15:13:25.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:25.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1266374080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:25 smithi028 ceph-mon[29876]: from='client.59057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:13:25.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:25.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:25.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:25.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:26.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:26.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:26.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:26.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:26.279 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:26.279 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-03-30T15:13:26.279 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-03-30T15:13:26.279 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-03-30T15:13:26.279 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:26.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:26.363 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:13:26.037+0000 7f1a82ee87c0 -1 osd.0 668 log_to_monitors true 2024-03-30T15:13:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:26.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:26.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:26.488 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:26.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:26.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:26.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:26.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='client.59057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: osdmap e848: 8 total, 7 up, 8 in 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: 12.1d scrub starts 2024-03-30T15:13:26.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: 12.1d scrub ok 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[29876]: pgmap v1309: 214 pgs: 13 unknown, 25 active+clean+remapped, 128 active+clean, 37 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 KiB/s wr, 4 op/s; 39/684 objects degraded (5.702%); 33/684 objects misplaced (4.825%); 0 B/s, 1 objects/s recovering 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='client.59057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: osdmap e848: 8 total, 7 up, 8 in 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: 12.1d scrub starts 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: 12.1d scrub ok 2024-03-30T15:13:26.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:26 smithi028 ceph-mon[41614]: pgmap v1309: 214 pgs: 13 unknown, 25 active+clean+remapped, 128 active+clean, 37 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 KiB/s wr, 4 op/s; 39/684 objects degraded (5.702%); 33/684 objects misplaced (4.825%); 0 B/s, 1 objects/s recovering 2024-03-30T15:13:26.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:26.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:26.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:26.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='client.59057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: osdmap e848: 8 total, 7 up, 8 in 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]: dispatch 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: 12.1d scrub starts 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: 12.1d scrub ok 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T15:13:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:26 smithi134 ceph-mon[33714]: pgmap v1309: 214 pgs: 13 unknown, 25 active+clean+remapped, 128 active+clean, 37 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.1 KiB/s wr, 4 op/s; 39/684 objects degraded (5.702%); 33/684 objects misplaced (4.825%); 0 B/s, 1 objects/s recovering 2024-03-30T15:13:26.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:13:26.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:27.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:27.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:27.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:27.208 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:27.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:27.376 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:13:27.080+0000 7f1a74340700 -1 osd.0 849 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T15:13:27.383 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:27.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:27.414 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:13:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:13:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:13:27.414 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:13:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:27.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:27.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 322.0s0 scrub starts 2024-03-30T15:13:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 322.0 scrub ok 2024-03-30T15:13:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 39/684 objects degraded (5.702%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:13:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]': finished 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: osdmap e849: 8 total, 7 up, 8 in 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:13:27.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 322.0s0 scrub starts 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 322.0 scrub ok 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 39/684 objects degraded (5.702%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]': finished 2024-03-30T15:13:27.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: osdmap e849: 8 total, 7 up, 8 in 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:13:27.726 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:13:27.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:27 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:13:27.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 322.0s0 scrub starts 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 322.0 scrub ok 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 39/684 objects degraded (5.702%), 11 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-142", "mode": "writeback"}]': finished 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: osdmap e849: 8 total, 7 up, 8 in 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='osd.0 v1:172.21.15.28:6809/2927075629' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.800 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:13:27.801 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:27 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:13:27.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:27.922 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:27.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:28.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:28.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:28.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:28.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-03-30T15:13:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:28.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:28.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:28.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:28.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:28.422 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:28.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:28.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:28.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:13:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:13:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T15:13:28.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: osd.0 v1:172.21.15.28:6809/2927075629 boot 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: osdmap e850: 8 total, 8 up, 8 in 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: pgmap v1312: 214 pgs: 25 active+clean+remapped, 135 active+clean, 43 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 39/708 objects degraded (5.508%); 33/708 objects misplaced (4.661%) 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:13:28.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: osd.0 v1:172.21.15.28:6809/2927075629 boot 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: osdmap e850: 8 total, 8 up, 8 in 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: pgmap v1312: 214 pgs: 25 active+clean+remapped, 135 active+clean, 43 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 39/708 objects degraded (5.508%); 33/708 objects misplaced (4.661%) 2024-03-30T15:13:28.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:28 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:13:28.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:28.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:28.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: osd.0 v1:172.21.15.28:6809/2927075629 boot 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: osdmap e850: 8 total, 8 up, 8 in 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T15:13:28.886 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.887 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:13:28.887 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:13:28.887 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:13:28.887 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: pgmap v1312: 214 pgs: 25 active+clean+remapped, 135 active+clean, 43 active+undersized, 11 active+undersized+degraded; 4.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 39/708 objects degraded (5.508%); 33/708 objects misplaced (4.661%) 2024-03-30T15:13:28.887 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:28 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:13:28.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:28.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:28.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:29.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:29.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:29.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:29.283 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-03-30T15:13:29.283 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-03-30T15:13:29.283 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-03-30T15:13:29.283 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-03-30T15:13:29.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDECSubOpWrite(322.1s2 646/644 ECSubWrite(tid=1490, reqid=client.58535.0:2, at_version=646'1, trim_to=0'0, roll_forward_to=0'0))", 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 313.81622510199998, 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038653149999999998, 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370908+0000", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370909+0000", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9200000000000002e-07 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370954+0000", 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5198999999999999e-05 2024-03-30T15:13:29.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370966+0000", 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.239e-05 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.370977+0000", 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.128e-05 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.286 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.371502+0000", 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052424500000000001 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.371516+0000", 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4e-05 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.374743+0000", 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032274980000000001 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.287 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:15.374773+0000", 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0311000000000002e-05 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:44 323.18 323:18000000::::head [pgnls start_epoch 650] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e650)", 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 309.81053636299998, 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00320857, 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.288 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 44 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.289 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376597+0000", 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376601+0000", 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0690000000000003e-06 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376615+0000", 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.449e-05 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.290 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376646+0000", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0706e-05 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376668+0000", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1396e-05 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376823+0000", 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015560699999999999 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.376828+0000", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.186e-06 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379596+0000", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027672209999999998 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379661+0000", 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5042999999999994e-05 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:19.379805+0000", 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000144852 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:255 325.e 325:70ec8fab:test-rados-api-smithi028-93253-117::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e665)", 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 294.81916859199998, 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084883900000000002, 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.293 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 255 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.294 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367965+0000", 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367968+0000", 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.269e-06 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367978+0000", 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0389e-05 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.295 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.367999+0000", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0820999999999999e-05 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368013+0000", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4369000000000001e-05 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368106+0000", 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2734999999999999e-05 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.296 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.368116+0000", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9930000000000007e-06 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.371856+0000", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003739357 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.372060+0000", 2024-03-30T15:13:29.297 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020420599999999999 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.373720+0000", 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016599029999999999 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.298 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.373823+0000", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103148 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.375420+0000", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001597464 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.375485+0000", 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4987000000000006e-05 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:08:34.376453+0000", 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096774899999999995 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:650 328.b 328:d7101345:test-rados-api-smithi028-93253-123::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e694)", 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 253.80216529, 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0061439140000000003, 2024-03-30T15:13:29.300 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 650 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384968+0000", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384970+0000", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384976+0000", 2024-03-30T15:13:29.302 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3720000000000001e-06 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.384992+0000", 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5999e-05 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385002+0000", 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5100000000000004e-06 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.303 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385059+0000", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6533000000000002e-05 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.385065+0000", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1789999999999996e-06 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.388747+0000", 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036823210000000001 2024-03-30T15:13:29.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.389663+0000", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091559200000000001 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.390602+0000", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00093893999999999998 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.390652+0000", 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0096999999999998e-05 2024-03-30T15:13:29.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.391052+0000", 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000400374 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:15.391112+0000", 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9858000000000002e-05 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:729 328.8 328:10000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 251.735601253, 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076206399999999997, 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 729 2024-03-30T15:13:29.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451532+0000", 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451536+0000", 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.873e-06 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451553+0000", 2024-03-30T15:13:29.309 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7091e-05 2024-03-30T15:13:29.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.310 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451582+0000", 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9487000000000001e-05 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451598+0000", 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5882000000000001e-05 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451681+0000", 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2874999999999995e-05 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.451687+0000", 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5450000000000003e-06 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.312 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459041+0000", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073537719999999997 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459066+0000", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5463999999999999e-05 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:17.459153+0000", 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6650999999999994e-05 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.313 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:810 329.c 329:346a2a8b:test-rados-api-smithi028-93253-125::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e703)", 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 244.78594125000001, 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0077308790000000004, 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.314 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 810 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401192+0000", 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401195+0000", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3079999999999999e-06 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401207+0000", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1544e-05 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401226+0000", 2024-03-30T15:13:29.316 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9400000000000001e-05 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401239+0000", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3115000000000001e-05 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401304+0000", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4243000000000001e-05 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for blocked object", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.401494+0000", 2024-03-30T15:13:29.317 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019060099999999999 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407467+0000", 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059725730000000001 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407561+0000", 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4106999999999999e-05 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.318 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407894+0000", 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033300799999999998 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.407956+0000", 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2300999999999999e-05 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:24.408923+0000", 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00096667900000000002 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1040 330.18 330:18000000::::head [pgnls start_epoch 714] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e714)", 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 233.7674786, 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032484010000000002, 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1040 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419655+0000", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419658+0000", 2024-03-30T15:13:29.321 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1540000000000002e-06 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419668+0000", 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0057e-05 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419690+0000", 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.181e-05 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.322 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419702+0000", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2830000000000001e-05 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419782+0000", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9348e-05 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.419788+0000", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3629999999999999e-06 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.323 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422758+0000", 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029700490000000002 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422798+0000", 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9792999999999997e-05 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:35.422903+0000", 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104997 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.324 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(1.4,QREQUEST,e:715,from:5,reservation_nonce:5)", 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 232.50053175299999, 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034844170000000001, 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686601+0000", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686603+0000", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0099999999999998e-06 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686609+0000", 2024-03-30T15:13:29.326 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6250000000000004e-06 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686622+0000", 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.242e-05 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686630+0000", 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8689999999999993e-06 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686682+0000", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1782e-05 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.686687+0000", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2870000000000004e-06 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690023+0000", 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033354050000000001 2024-03-30T15:13:29.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690039+0000", 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6402999999999999e-05 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:36.690086+0000", 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6616000000000002e-05 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.329 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3185 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e730)", 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:53.566057+0000", 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 215.621076273, 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056883380000000002, 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3185 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.330 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566057+0000", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566057+0000", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566060+0000", 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.926e-06 2024-03-30T15:13:29.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566070+0000", 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0008e-05 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566087+0000", 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7462e-05 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.332 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566113+0000", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5948e-05 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566169+0000", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5290000000000001e-05 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.566174+0000", 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3970000000000002e-06 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.333 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.571676+0000", 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055022489999999999 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:53.571745+0000", 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9058000000000002e-05 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.334 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1310 332.16 332:68000000:test-rados-api-smithi028-93253-131:::head [pgnls start_epoch 732] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e732)", 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:54.412164+0000", 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 214.774968833, 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042482220000000003, 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1310 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412164+0000", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412164+0000", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412168+0000", 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3000000000000002e-06 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412178+0000", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0232e-05 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412196+0000", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8309999999999999e-05 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412206+0000", 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9080000000000007e-06 2024-03-30T15:13:29.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412279+0000", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2379999999999995e-05 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.412283+0000", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8300000000000003e-06 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.416313+0000", 2024-03-30T15:13:29.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040294989999999998 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.416341+0000", 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7832000000000001e-05 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:54.416413+0000", 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1931e-05 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.339 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1473 332.16 332:68000000::::head [pgnls start_epoch 734] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e734)", 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:09:56.413949+0000", 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 212.773184241, 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041127660000000003, 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1473 2024-03-30T15:13:29.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413949+0000", 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413949+0000", 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413952+0000", 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2210000000000002e-06 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413962+0000", 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7990000000000005e-06 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413980+0000", 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8087000000000001e-05 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.413989+0000", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1009999999999998e-06 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.414057+0000", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7766000000000001e-05 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.414061+0000", 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.403e-06 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.417927+0000", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038658500000000001 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.417965+0000", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7910999999999997e-05 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:09:56.418062+0000", 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6627999999999997e-05 2024-03-30T15:13:29.344 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:1734 334.4 334:20000000::::head [pgnls start_epoch 755] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e755)", 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:17.411810+0000", 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 191.77532281399999, 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060126700000000003, 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1734 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411810+0000", 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411810+0000", 2024-03-30T15:13:29.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411814+0000", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6710000000000001e-06 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411829+0000", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4592e-05 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411852+0000", 2024-03-30T15:13:29.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3547999999999999e-05 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411867+0000", 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5225000000000001e-05 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411982+0000", 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011501 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.348 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.411989+0000", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6560000000000003e-06 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.417561+0000", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055722940000000002 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.417611+0000", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9855000000000003e-05 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.349 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:17.417823+0000", 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000211819 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24394.0:3288 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94326471856512 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e780)", 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:43.968310+0000", 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 165.21882311300001, 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.47710206700000002, 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.350 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24394", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2248569927", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3288 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968310+0000", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968310+0000", 2024-03-30T15:13:29.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968313+0000", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5210000000000001e-06 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968320+0000", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8820000000000003e-06 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968331+0000", 2024-03-30T15:13:29.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1695999999999999e-05 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968341+0000", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7380000000000006e-06 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968421+0000", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0113999999999999e-05 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:43.968433+0000", 2024-03-30T15:13:29.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1453e-05 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.437875+0000", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.469442945 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.438055+0000", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017939799999999999 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.445412+0000", 2024-03-30T15:13:29.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073573199999999997 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25087.0:3 42.f 42:f4b50b61:::1.obj:head [write 0~198] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:10:44.435907+0000", 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 164.75122621899999, 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017938780000000001, 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.355 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25087", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/4227304124", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435907+0000", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435907+0000", 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435909+0000", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.012e-06 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435919+0000", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6260000000000007e-06 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435928+0000", 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0999999999999993e-06 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.435950+0000", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2269e-05 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.436109+0000", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015929799999999999 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.436116+0000", 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5690000000000001e-06 2024-03-30T15:13:29.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.446564+0000", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010448106 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.446695+0000", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013146499999999999 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,5", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.447477+0000", 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00078155499999999997 2024-03-30T15:13:29.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.450780+0000", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033031179999999999 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453423+0000", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026427769999999998 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453735+0000", 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031233900000000002 2024-03-30T15:13:29.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453798+0000", 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2928000000000005e-05 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:10:44.453846+0000", 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7617999999999997e-05 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(12.16,QREQUEST,e:805,from:6,reservation_nonce:3)", 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:11:45.719987+0000", 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 103.46714614299999, 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0044291060000000004, 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.719987+0000", 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.362 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.719987+0000", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.719990+0000", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6429999999999999e-06 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.719996+0000", 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3330000000000002e-06 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.720009+0000", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3226e-05 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.720017+0000", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8990000000000001e-06 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.720072+0000", 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4410000000000003e-05 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.720076+0000", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8300000000000003e-06 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.723848+0000", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037710780000000002 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.723859+0000", 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1946e-05 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:11:45.724416+0000", 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055674100000000003 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.58535.0:2073 337.16 337:68000000::::head [pgnls start_epoch 843] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e843)", 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:13:21.323101+0000", 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 7.8640324220000002, 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081201380000000007, 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.58535", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.28:0/2751046486", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 2073 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323101+0000", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.367 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323101+0000", 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323104+0000", 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8059999999999999e-06 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323112+0000", 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7329999999999992e-06 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.368 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323131+0000", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8499999999999999e-05 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323142+0000", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.136e-05 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323227+0000", 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4998000000000002e-05 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.369 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.323234+0000", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6039999999999998e-06 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.331017+0000", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007783168 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.331056+0000", 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9283000000000003e-05 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:21.331221+0000", 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016468599999999999 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDECSubOpWrite(322.1s2 846/832 ECSubWrite(tid=7, reqid=client.58535.0:2127, at_version=846'12, trim_to=0'0, roll_forward_to=0'0))", 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:13:23.581153+0000", 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.6059802640000003, 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003892946, 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.371 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581153+0000", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581153+0000", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.372 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581154+0000", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4799999999999998e-07 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581179+0000", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5763999999999999e-05 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581188+0000", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2160000000000006e-06 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581198+0000", 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0053e-05 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581267+0000", 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9585000000000005e-05 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.581278+0000", 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1e-05 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.374 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.585013+0000", 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037344969999999998 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.585046+0000", 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3283e-05 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDECSubOpWrite(322.1s2 846/832 ECSubWrite(tid=8, reqid=client.58535.0:2127, at_version=0'0, trim_to=0'0, roll_forward_to=846'12))", 2024-03-30T15:13:29.375 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:13:23.591193+0000", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.5959404399999997, 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0039173159999999997, 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591193+0000", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591193+0000", 2024-03-30T15:13:29.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591194+0000", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0240000000000001e-06 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591207+0000", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2883e-05 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591217+0000", 2024-03-30T15:13:29.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0176999999999999e-05 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591226+0000", 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5140000000000007e-06 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591532+0000", 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000305896 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_started", 2024-03-30T15:13:29.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.591544+0000", 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1236000000000001e-05 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_committed", 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.595073+0000", 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035294160000000001 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:23.595110+0000", 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7169999999999998e-05 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDScrubReserve(12.1d,QREQUEST,e:848,from:6,reservation_nonce:3)", 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:13:25.714547+0000", 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.472585917, 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032609140000000002, 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714547+0000", 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714547+0000", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714549+0000", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714555+0000", 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8420000000000001e-06 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714568+0000", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2486999999999999e-05 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714575+0000", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9290000000000001e-06 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714623+0000", 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8405999999999997e-05 2024-03-30T15:13:29.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op must wait for map", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.714627+0000", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4739999999999999e-06 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.717728+0000", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031006129999999999 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.717746+0000", 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7714e-05 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:25.717808+0000", 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2430999999999999e-05 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24466.0:1116 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e849)", 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-03-30T15:13:27.903128+0000", 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 1.284004817, 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73099361100000004, 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "continuous": false, 2024-03-30T15:13:29.384 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24466", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "v1:172.21.15.134:0/2615461998", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1116 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903128+0000", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903128+0000", 2024-03-30T15:13:29.385 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903132+0000", 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4819999999999999e-06 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903144+0000", 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2089000000000001e-05 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-03-30T15:13:29.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903162+0000", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7869000000000001e-05 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903172+0000", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9799999999999993e-06 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903280+0000", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010860499999999999 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-03-30T15:13:29.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:27.903293+0000", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.235e-05 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:28.633905+0000", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.73061268899999998 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:28.634063+0000", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000157181 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-03-30T15:13:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-03-30T15:13:28.634122+0000", 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9366e-05 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-03-30T15:13:29.389 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-03-30T15:13:29.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:29.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:29.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:29.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:29.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: osdmap e851: 8 total, 8 up, 8 in 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:13:29.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: osdmap e851: 8 total, 8 up, 8 in 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:13:29.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:29 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:13:29.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:29.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:29.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:29.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: osdmap e851: 8 total, 8 up, 8 in 2024-03-30T15:13:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:13:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T15:13:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T15:13:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:13:29.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:29 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:13:30.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:13:30.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:30.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:30.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:30.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:30.246 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:30.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:30.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 -- 172.21.15.28:0/4189906090 >> v1:172.21.15.28:6790/0 conn(0x7fe7e410d960 legacy=0x7fe7e410fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 -- 172.21.15.28:0/4189906090 shutdown_connections 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 -- 172.21.15.28:0/4189906090 >> 172.21.15.28:0/4189906090 conn(0x7fe7e40fc9e0 msgr2=0x7fe7e40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 -- 172.21.15.28:0/4189906090 shutdown_connections 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 -- 172.21.15.28:0/4189906090 wait complete. 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 librados: starting msgr at 2024-03-30T15:13:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.339+0000 7fe7e98eb700 1 librados: starting objecter 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 Processor -- start 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 -- start start 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 librados: setting wanted keys 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 librados: calling monclient init 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7e41a5e80 con 0x7fe7e4109dc0 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7e41a70a0 con 0x7fe7e410d960 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e98eb700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe7e41a82e0 con 0x7fe7e4101130 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e2ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe7e4101130 0x7fe7e4106310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51340/0 (socket says 172.21.15.28:51340) 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e2ffd700 1 -- 172.21.15.28:0/3939700323 learned_addr learned my addr 172.21.15.28:0/3939700323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e88e9700 10 client.?.objecter ms_handle_connect 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e88e9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e88e9700 10 client.?.objecter ms_handle_connect 0x7fe7e4109dc0 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.340+0000 7fe7e88e9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133411615 0 0) 0x7fe7e41a82e0 con 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d0003aa0 con 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 10 client.?.objecter ms_handle_connect 0x7fe7e410d960 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082722 0 0) 0x7fe7e41a5e80 con 0x7fe7e4109dc0 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7e41a82e0 con 0x7fe7e4109dc0 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012559929 0 0) 0x7fe7e41a70a0 con 0x7fe7e410d960 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7e41a5e80 con 0x7fe7e410d960 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2950528596 0 0) 0x7fe7d0003aa0 con 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7e41a70a0 con 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe7d4001d40 con 0x7fe7e4101130 2024-03-30T15:13:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4228356123 0 0) 0x7fe7e41a82e0 con 0x7fe7e4109dc0 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7d0003aa0 con 0x7fe7e4109dc0 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe7cc001a30 con 0x7fe7e4109dc0 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4228679837 0 0) 0x7fe7e41a5e80 con 0x7fe7e410d960 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7e41a82e0 con 0x7fe7e410d960 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe7dc001850 con 0x7fe7e410d960 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 815887689 0 0) 0x7fe7e41a70a0 con 0x7fe7e4101130 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.134:6789/0 conn(0x7fe7e410d960 legacy=0x7fe7e41a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6789/0 conn(0x7fe7e4109dc0 legacy=0x7fe7e41a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e41a9520 con 0x7fe7e4101130 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e41a60b0 con 0x7fe7e4101130 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe7d4002020 con 0x7fe7e4101130 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe7d4006170 con 0x7fe7e4101130 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e98eb700 10 client.59066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7e41a6570 con 0x7fe7e4101130 2024-03-30T15:13:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.341+0000 7fe7e98eb700 1 librados: init done 2024-03-30T15:13:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.342+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe7e4102610 con 0x7fe7e4101130 2024-03-30T15:13:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe7d4003cd0 con 0x7fe7e4101130 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(852..852 src has 442..852) ==== 8871+0+0 (unknown 964412262 0 0) 0x7fe7d4089e90 con 0x7fe7e4101130 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 osd_map(852..852 src has 442..852) 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 3 client.59066.objecter handle_osd_map got epochs [852,852] > 0 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 3 client.59066.objecter handle_osd_map decoding full epoch 852 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 20 client.59066.objecter dump_active .. 0 homeless 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.343+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v852) ==== 54+0+16 (unknown 316656647 0 2420000192) 0x7fe7d408c2f0 con 0x7fe7e4101130 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7d00704f0 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 1 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 1 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 0 1 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 4 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 10 client.59066.objecter _send_command 1 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41048f0 con 0x7fe7e41a7270 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.344+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e41a7270 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.345+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41048f0 con 0x7fe7e41a7270 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.345+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 1: 0 ) 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.345+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 1 = system:0 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.345+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 4 2024-03-30T15:13:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.345+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 0 1 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 2 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 2 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e4113d70 osd=0 3 2024-03-30T15:13:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 0 2 2024-03-30T15:13:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 4 2024-03-30T15:13:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 10 client.59066.objecter _send_command 2 2024-03-30T15:13:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.362+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e4068b30 con 0x7fe7e41a7270 2024-03-30T15:13:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.363+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e4068b30 con 0x7fe7e41a7270 2024-03-30T15:13:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.363+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.363+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.363+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 4 2024-03-30T15:13:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.363+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 0 2 2024-03-30T15:13:30.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 1 3 2024-03-30T15:13:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 4 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 10 client.59066.objecter _send_command 3 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe7e402d9c0 con 0x7fe7e410e2c0 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.364+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e410e2c0 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.365+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e402d9c0 con 0x7fe7e410e2c0 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.365+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 3: 0 ) 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.365+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 3 = system:0 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.365+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 4 2024-03-30T15:13:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.365+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 1 3 2024-03-30T15:13:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 4 2024-03-30T15:13:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 4 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41a7fb0 osd=1 3 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 1 4 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 4 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 10 client.59066.objecter _send_command 4 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.382+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e402ddc0 con 0x7fe7e410e2c0 2024-03-30T15:13:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e402ddc0 con 0x7fe7e410e2c0 2024-03-30T15:13:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 4 2024-03-30T15:13:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 1 4 2024-03-30T15:13:30.385 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.383+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 5 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 5 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 2 5 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 4 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 10 client.59066.objecter _send_command 5 2024-03-30T15:13:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41afef0 con 0x7fe7e406ac20 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e406ac20 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41afef0 con 0x7fe7e406ac20 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 5: 0 ) 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 5 = system:0 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 4 2024-03-30T15:13:30.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.384+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 2 5 2024-03-30T15:13:30.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 6 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 6 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e410edb0 osd=2 3 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 2 6 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 4 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 10 client.59066.objecter _send_command 6 2024-03-30T15:13:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.401+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e406b290 con 0x7fe7e406ac20 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e406b290 con 0x7fe7e406ac20 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 4 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 2 6 2024-03-30T15:13:30.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:13:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 7 2024-03-30T15:13:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 7 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 3 7 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 4 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 10 client.59066.objecter _send_command 7 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe7e4063390 con 0x7fe7e406a810 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.403+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e406a810 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.404+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e4063390 con 0x7fe7e406a810 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.404+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 7: 0 ) 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.404+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 7 = system:0 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.404+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 4 2024-03-30T15:13:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.404+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 3 7 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 8 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 8 2024-03-30T15:13:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e406a6a0 osd=3 3 2024-03-30T15:13:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 3 8 2024-03-30T15:13:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 4 2024-03-30T15:13:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 10 client.59066.objecter _send_command 8 2024-03-30T15:13:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.421+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e40628d0 con 0x7fe7e406a810 2024-03-30T15:13:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:30.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.422+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e40628d0 con 0x7fe7e406a810 2024-03-30T15:13:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.422+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.422+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.422+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 4 2024-03-30T15:13:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.422+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 3 8 2024-03-30T15:13:30.425 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 9 2024-03-30T15:13:30.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 9 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 4 9 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 4 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 10 client.59066.objecter _send_command 9 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41b4120 con 0x7fe7e4062a40 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.423+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e4062a40 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.424+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41b4120 con 0x7fe7e4062a40 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.424+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 9: 0 ) 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.424+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 9 = system:0 2024-03-30T15:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.424+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 4 2024-03-30T15:13:30.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.424+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 4 9 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 10 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 10 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e40628d0 osd=4 3 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 4 10 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 4 2024-03-30T15:13:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 10 client.59066.objecter _send_command 10 2024-03-30T15:13:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.440+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e41b4120 con 0x7fe7e4062a40 2024-03-30T15:13:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e41b4120 con 0x7fe7e4062a40 2024-03-30T15:13:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 4 2024-03-30T15:13:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 4 10 2024-03-30T15:13:30.444 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 11 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 11 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 5 11 2024-03-30T15:13:30.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 4 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 10 client.59066.objecter _send_command 11 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.442+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41c0560 con 0x7fe7e41bc980 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e41bc980 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41c0560 con 0x7fe7e41bc980 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 11: 0 ) 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 11 = system:0 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 4 2024-03-30T15:13:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.443+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 5 11 2024-03-30T15:13:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 12 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 12 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41b4120 osd=5 3 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 5 12 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 4 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 10 client.59066.objecter _send_command 12 2024-03-30T15:13:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.460+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e41c1f40 con 0x7fe7e41bc980 2024-03-30T15:13:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e41c1f40 con 0x7fe7e41bc980 2024-03-30T15:13:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 4 2024-03-30T15:13:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 5 12 2024-03-30T15:13:30.464 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 13 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 13 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 6 13 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 4 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 10 client.59066.objecter _send_command 13 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41c2ac0 con 0x7fe7e41c2660 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.462+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e41c2660 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.463+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41c2ac0 con 0x7fe7e41c2660 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.463+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 13: 0 ) 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.463+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 13 = system:0 2024-03-30T15:13:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.463+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 4 2024-03-30T15:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.463+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 6 13 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 14 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 14 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41c1f40 osd=6 3 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 6 14 2024-03-30T15:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 4 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 10 client.59066.objecter _send_command 14 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.480+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e41c2ac0 con 0x7fe7e41c2660 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e41c2ac0 con 0x7fe7e41c2660 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 4 2024-03-30T15:13:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 6 14 2024-03-30T15:13:30.487 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 15 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 15 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 7 15 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 4 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 10 client.59066.objecter _send_command 15 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.482+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe7e41c2e10 con 0x7fe7e41b0600 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 10 client.59066.objecter ms_handle_connect 0x7fe7e41b0600 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe7e41c2e10 con 0x7fe7e41b0600 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 15: 0 ) 2024-03-30T15:13:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 15 = system:0 2024-03-30T15:13:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 4 2024-03-30T15:13:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.483+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 7 15 2024-03-30T15:13:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 10 client.59066.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign -1 16 2024-03-30T15:13:30.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 20 client.59066.objecter _get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_remove -1 16 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 20 client.59066.objecter get_session s=0x7fe7e41c2ac0 osd=7 3 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 15 client.59066.objecter _session_command_op_assign 7 16 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 4 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 10 client.59066.objecter _send_command 16 2024-03-30T15:13:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.500+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7e41c2e10 con 0x7fe7e41b0600 2024-03-30T15:13:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e88e9700 1 -- 172.21.15.28:0/3939700323 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7e41c2e10 con 0x7fe7e41b0600 2024-03-30T15:13:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e88e9700 10 client.59066.objecter ms_dispatch 0x7fe7e4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e88e9700 10 client.59066.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e88e9700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 4 2024-03-30T15:13:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e88e9700 15 client.59066.objecter _session_command_op_remove 7 16 2024-03-30T15:13:30.504 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 librados: watch_flush enter 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 librados: watch_flush exit 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.0 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6809/2927075629 conn(0x7fe7e41a7270 legacy=0x7fe7e41a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e4113d70 osd=0 1 2024-03-30T15:13:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.1 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6805/1850313547 conn(0x7fe7e410e2c0 legacy=0x7fe7e410e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41a7fb0 osd=1 1 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.2 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6801/2294873818 conn(0x7fe7e406ac20 legacy=0x7fe7e41af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e410edb0 osd=2 1 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.3 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6813/1506157250 conn(0x7fe7e406a810 legacy=0x7fe7e41b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e406a6a0 osd=3 1 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.4 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.134:6800/4077237787 conn(0x7fe7e4062a40 legacy=0x7fe7e41bad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e40628d0 osd=4 1 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.5 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.134:6804/3668151179 conn(0x7fe7e41bc980 legacy=0x7fe7e41bed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41b4120 osd=5 1 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.6 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.134:6808/2135224194 conn(0x7fe7e41c2660 legacy=0x7fe7e41c0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c1f40 osd=6 1 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter close_session for osd.7 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.134:6812/3758190298 conn(0x7fe7e41b0600 legacy=0x7fe7e41b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter put_session s=0x7fe7e41c2ac0 osd=7 1 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 20 client.59066.objecter shutdown clearing up homeless session... 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 10 client.59066.objecter successfully canceled tick 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e37fe700 1 -- 172.21.15.28:0/3939700323 reap_dead start 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e37fe700 1 -- 172.21.15.28:0/3939700323 reap_dead start 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe7d00704f0 legacy=0x7fe7d00729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e37fe700 1 -- 172.21.15.28:0/3939700323 reap_dead start 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e37fe700 1 -- 172.21.15.28:0/3939700323 reap_dead start 2024-03-30T15:13:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e37fe700 1 -- 172.21.15.28:0/3939700323 reap_dead start 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> v1:172.21.15.28:6790/0 conn(0x7fe7e4101130 legacy=0x7fe7e4106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 shutdown_connections 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.502+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 >> 172.21.15.28:0/3939700323 conn(0x7fe7e40fc9e0 msgr2=0x7fe7e40fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.503+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 shutdown_connections 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.503+0000 7fe7e98eb700 1 -- 172.21.15.28:0/3939700323 wait complete. 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:30.503+0000 7fe7e98eb700 1 librados: shutdown 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:30.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:30.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:30.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: osdmap e852: 8 total, 8 up, 8 in 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: pgmap v1315: 214 pgs: 9 peering, 16 active+clean+remapped, 156 active+clean, 26 active+undersized, 7 active+undersized+degraded; 4.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 2 op/s; 29/708 objects degraded (4.096%); 16/708 objects misplaced (2.260%); 1 B/s, 0 objects/s recovering 2024-03-30T15:13:30.651 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3939700323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:30.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:30.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:30.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:30.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:30.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:30.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:13:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:13:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:13:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: osdmap e852: 8 total, 8 up, 8 in 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: pgmap v1315: 214 pgs: 9 peering, 16 active+clean+remapped, 156 active+clean, 26 active+undersized, 7 active+undersized+degraded; 4.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 2 op/s; 29/708 objects degraded (4.096%); 16/708 objects misplaced (2.260%); 1 B/s, 0 objects/s recovering 2024-03-30T15:13:30.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3939700323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: osdmap e852: 8 total, 8 up, 8 in 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:13:30.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:13:30.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:30.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: pgmap v1315: 214 pgs: 9 peering, 16 active+clean+remapped, 156 active+clean, 26 active+undersized, 7 active+undersized+degraded; 4.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 2 op/s; 29/708 objects degraded (4.096%); 16/708 objects misplaced (2.260%); 1 B/s, 0 objects/s recovering 2024-03-30T15:13:30.976 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3939700323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:31.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:31.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:31.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:31.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:31.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:31.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:31.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:31.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:31.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:31.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:31.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:31.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:31.917 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:13:31.917 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:13:31.917 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: osdmap e853: 8 total, 8 up, 8 in 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T15:13:31.918 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:31.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:31.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:31.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:13:31.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:13:31.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: osdmap e853: 8 total, 8 up, 8 in 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:13:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: osdmap e853: 8 total, 8 up, 8 in 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T15:13:31.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:31.952 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:31.953 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:32.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:32.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:32.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:32.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:32.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:32.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:32.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:13:32.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:32.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:32.830 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:32.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 12.19 scrub starts 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 12.19 scrub ok 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: osdmap e854: 8 total, 8 up, 8 in 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: pgmap v1318: 214 pgs: 9 peering, 200 active+clean, 4 active+undersized, 1 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/708 objects degraded (0.989%) 2024-03-30T15:13:32.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: Health check update: Degraded data redundancy: 29/708 objects degraded (4.096%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: osdmap e855: 8 total, 8 up, 8 in 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 12.19 scrub starts 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 12.19 scrub ok 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: osdmap e854: 8 total, 8 up, 8 in 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: pgmap v1318: 214 pgs: 9 peering, 200 active+clean, 4 active+undersized, 1 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/708 objects degraded (0.989%) 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: Health check update: Degraded data redundancy: 29/708 objects degraded (4.096%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: osdmap e855: 8 total, 8 up, 8 in 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 12.19 scrub starts 2024-03-30T15:13:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 12.19 scrub ok 2024-03-30T15:13:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:13:32.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: osdmap e854: 8 total, 8 up, 8 in 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: pgmap v1318: 214 pgs: 9 peering, 200 active+clean, 4 active+undersized, 1 active+undersized+degraded; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/708 objects degraded (0.989%) 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: Health check update: Degraded data redundancy: 29/708 objects degraded (4.096%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: osdmap e855: 8 total, 8 up, 8 in 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:32.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:33.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:33.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:33.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:33.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:33.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:33.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:33.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:33.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:33.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:33.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:33.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- 172.21.15.28:0/3367984721 >> v1:172.21.15.28:6790/0 conn(0x7f4da410ae80 legacy=0x7f4da410d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- 172.21.15.28:0/3367984721 shutdown_connections 2024-03-30T15:13:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- 172.21.15.28:0/3367984721 >> 172.21.15.28:0/3367984721 conn(0x7f4da4077370 msgr2=0x7f4da40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- 172.21.15.28:0/3367984721 shutdown_connections 2024-03-30T15:13:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- 172.21.15.28:0/3367984721 wait complete. 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 librados: starting msgr at 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 librados: starting objecter 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 Processor -- start 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 -- start start 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 librados: setting wanted keys 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.936+0000 7f4dac59a700 1 librados: calling monclient init 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4dac59a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4da41a9340 con 0x7f4da410ae80 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4dac59a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4da41aa560 con 0x7f4da4107000 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4dac59a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4da41ab7a0 con 0x7f4da410eb50 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4daab37700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f4da410eb50 0x7f4da41a7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51368/0 (socket says 172.21.15.28:51368) 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4daab37700 1 -- 172.21.15.28:0/2655863476 learned_addr learned my addr 172.21.15.28:0/2655863476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4da9b35700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4da410ae80 0x7f4da41a43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58888/0 (socket says 172.21.15.28:58888) 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter ms_handle_connect 0x7f4da410ae80 2024-03-30T15:13:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter ms_handle_connect 0x7f4da4107000 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter ms_handle_connect 0x7f4da410eb50 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1290162693 0 0) 0x7f4da41a9340 con 0x7f4da410ae80 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d900031f0 con 0x7f4da410ae80 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975437389 0 0) 0x7f4da41aa560 con 0x7f4da4107000 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da41a9340 con 0x7f4da4107000 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640039910 0 0) 0x7f4da41ab7a0 con 0x7f4da410eb50 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da41aa560 con 0x7f4da410eb50 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 46778988 0 0) 0x7f4da41a9340 con 0x7f4da4107000 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4da41ab7a0 con 0x7f4da4107000 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3976007366 0 0) 0x7f4d900031f0 con 0x7f4da410ae80 2024-03-30T15:13:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4da41a9340 con 0x7f4da410ae80 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4da0001830 con 0x7f4da4107000 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4d94004400 con 0x7f4da410ae80 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 356604749 0 0) 0x7f4da41aa560 con 0x7f4da410eb50 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d900031f0 con 0x7f4da410eb50 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.937+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4d9c001ce0 con 0x7f4da410eb50 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 532784331 0 0) 0x7f4da41ab7a0 con 0x7f4da4107000 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 >> v1:172.21.15.28:6790/0 conn(0x7f4da410eb50 legacy=0x7f4da41a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 >> v1:172.21.15.28:6789/0 conn(0x7f4da410ae80 legacy=0x7f4da41a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da41ac9e0 con 0x7f4da4107000 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4da41aa790 con 0x7f4da4107000 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4dac59a700 10 client.51061.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4da41aac50 con 0x7f4da4107000 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4dac59a700 1 librados: init done 2024-03-30T15:13:33.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4da0003b40 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.938+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4da0004ab0 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.939+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d88005120 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f4da00036a0 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(855..855 src has 442..855) ==== 8871+0+0 (unknown 4026927969 0 0) 0x7f4da0088720 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 10 client.51061.objecter ms_dispatch 0x7f4da4001550 osd_map(855..855 src has 442..855) 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 3 client.51061.objecter handle_osd_map got epochs [855,855] > 0 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 3 client.51061.objecter handle_osd_map decoding full epoch 855 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 20 client.51061.objecter dump_active .. 0 homeless 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.940+0000 7f4d9b7fe700 10 client.51061.objecter ms_handle_connect 0x7f4d90070470 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:32.945+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4da0055650 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.162+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f4d88005410 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.573+0000 7f4d9b7fe700 1 -- 172.21.15.28:0/2655863476 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v856) ==== 113+0+0 (unknown 1634917011 0 0) 0x7f4da00590a0 con 0x7f4da4107000 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 10 librados: watch_flush enter 2024-03-30T15:13:33.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 10 librados: watch_flush exit 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 20 client.51061.objecter shutdown clearing up homeless session... 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 10 client.51061.objecter successfully canceled tick 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4d90070470 legacy=0x7f4d90072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 >> v1:172.21.15.134:6789/0 conn(0x7f4da4107000 legacy=0x7f4da4102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 shutdown_connections 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 >> 172.21.15.28:0/2655863476 conn(0x7f4da4077370 msgr2=0x7f4da41117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 shutdown_connections 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 -- 172.21.15.28:0/2655863476 wait complete. 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:33.577+0000 7f4dac59a700 1 librados: shutdown 2024-03-30T15:13:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T15:13:33.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:33.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:33.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:13:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:13:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 322.5s0 scrub starts 2024-03-30T15:13:33.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 322.5 scrub ok 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 1.1 scrub starts 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: 1.1 scrub ok 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2655863476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.51061 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.51061 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: osdmap e856: 8 total, 8 up, 8 in 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:33.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:33 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:33.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:13:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:13:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 322.5s0 scrub starts 2024-03-30T15:13:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 322.5 scrub ok 2024-03-30T15:13:33.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 1.1 scrub starts 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: 1.1 scrub ok 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2655863476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.51061 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.51061 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: osdmap e856: 8 total, 8 up, 8 in 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:13:33.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 322.5s0 scrub starts 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 322.5 scrub ok 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 1.1 scrub starts 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: 1.1 scrub ok 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2655863476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.51061 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.51061 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: osdmap e856: 8 total, 8 up, 8 in 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:33.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:33 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:34.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:34.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:34.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:34.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:13:34.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:34.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-03-30T15:13:34.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:34.565 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:34.565 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: pgmap v1321: 214 pgs: 214 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: osdmap e857: 8 total, 8 up, 8 in 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.651 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: pgmap v1321: 214 pgs: 214 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: osdmap e857: 8 total, 8 up, 8 in 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:34 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:34.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:34.883 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: pgmap v1321: 214 pgs: 214 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]': finished 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: osdmap e857: 8 total, 8 up, 8 in 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.884 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:34 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-142"}]: dispatch 2024-03-30T15:13:34.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:34.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- 172.21.15.28:0/3950061950 >> v1:172.21.15.28:6789/0 conn(0x7fd168102250 legacy=0x7fd168102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- 172.21.15.28:0/3950061950 shutdown_connections 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- 172.21.15.28:0/3950061950 >> 172.21.15.28:0/3950061950 conn(0x7fd1680fdb20 msgr2=0x7fd1680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- 172.21.15.28:0/3950061950 shutdown_connections 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- 172.21.15.28:0/3950061950 wait complete. 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 librados: starting msgr at 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 librados: starting objecter 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 Processor -- start 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 -- start start 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 librados: setting wanted keys 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.020+0000 7fd16da1b700 1 librados: calling monclient init 2024-03-30T15:13:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16da1b700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1681a9200 con 0x7fd16810aee0 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16da1b700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1681aa420 con 0x7fd16810ea80 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16da1b700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd1681ab640 con 0x7fd168102250 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd166ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd168102250 0x7fd168107450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:51378/0 (socket says 172.21.15.28:51378) 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd166ffd700 1 -- 172.21.15.28:0/3001473352 learned_addr learned my addr 172.21.15.28:0/3001473352 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter ms_handle_connect 0x7fd168102250 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter ms_handle_connect 0x7fd16810aee0 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter ms_handle_connect 0x7fd16810ea80 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815142846 0 0) 0x7fd1681ab640 con 0x7fd168102250 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1540031f0 con 0x7fd168102250 2024-03-30T15:13:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240721256 0 0) 0x7fd1681a9200 con 0x7fd16810aee0 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1681ab640 con 0x7fd16810aee0 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896837385 0 0) 0x7fd1681aa420 con 0x7fd16810ea80 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1681a9200 con 0x7fd16810ea80 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3189151075 0 0) 0x7fd1540031f0 con 0x7fd168102250 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1681aa420 con 0x7fd168102250 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd160004510 con 0x7fd168102250 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2667300517 0 0) 0x7fd1681ab640 con 0x7fd16810aee0 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1540031f0 con 0x7fd16810aee0 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd150001810 con 0x7fd16810aee0 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.021+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 843445649 0 0) 0x7fd1681a9200 con 0x7fd16810ea80 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1681ab640 con 0x7fd16810ea80 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd158001b30 con 0x7fd16810ea80 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4243483722 0 0) 0x7fd1681aa420 con 0x7fd168102250 2024-03-30T15:13:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 >> v1:172.21.15.134:6789/0 conn(0x7fd16810ea80 legacy=0x7fd1681a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 >> v1:172.21.15.28:6789/0 conn(0x7fd16810aee0 legacy=0x7fd1681a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1681ac880 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1681a93d0 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16da1b700 10 client.59078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1681a9890 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16da1b700 1 librados: init done 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd160001b90 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.022+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd160004ea0 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.023+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd148005120 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.023+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd160003680 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(857..857 src has 442..857) ==== 8837+0+0 (unknown 4192810352 0 0) 0x7fd160089a20 con 0x7fd168102250 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 10 client.59078.objecter ms_dispatch 0x7fd168001550 osd_map(857..857 src has 442..857) 2024-03-30T15:13:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 3 client.59078.objecter handle_osd_map got epochs [857,857] > 0 2024-03-30T15:13:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 3 client.59078.objecter handle_osd_map decoding full epoch 857 2024-03-30T15:13:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 20 client.59078.objecter dump_active .. 0 homeless 2024-03-30T15:13:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.024+0000 7fd16ca19700 10 client.59078.objecter ms_handle_connect 0x7fd154070430 2024-03-30T15:13:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.028+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd160056990 con 0x7fd168102250 2024-03-30T15:13:35.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:35.095 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:35.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:35.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:13:35.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:35.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.261+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fd148005410 con 0x7fd168102250 2024-03-30T15:13:35.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:35.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:35.426 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:13:35.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:35.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:35.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:35.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.580+0000 7fd16ca19700 1 -- 172.21.15.28:0/3001473352 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v858) ==== 101+0+0 (unknown 3530457387 0 0) 0x7fd16005a3e0 con 0x7fd168102250 2024-03-30T15:13:35.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.583+0000 7fd16da1b700 10 librados: watch_flush enter 2024-03-30T15:13:35.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.583+0000 7fd16da1b700 10 librados: watch_flush exit 2024-03-30T15:13:35.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.583+0000 7fd16da1b700 20 client.59078.objecter shutdown clearing up homeless session... 2024-03-30T15:13:35.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.583+0000 7fd16da1b700 10 client.59078.objecter successfully canceled tick 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.584+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd154070430 legacy=0x7fd1540728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.584+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 >> v1:172.21.15.28:6790/0 conn(0x7fd168102250 legacy=0x7fd168107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.585+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 shutdown_connections 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.585+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 >> 172.21.15.28:0/3001473352 conn(0x7fd1680fdb20 msgr2=0x7fd16810ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.585+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 shutdown_connections 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.585+0000 7fd16da1b700 1 -- 172.21.15.28:0/3001473352 wait complete. 2024-03-30T15:13:35.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:35.586+0000 7fd16da1b700 1 librados: shutdown 2024-03-30T15:13:35.590 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:13:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:35.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.734 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:35.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:35.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:35.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:35.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:35.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:36.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:36.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T15:13:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/708 objects degraded (0.989%), 1 pg degraded) 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3001473352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: from='client.59078 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: from='client.59078 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:35 smithi134 ceph-mon[33714]: osdmap e858: 8 total, 8 up, 8 in 2024-03-30T15:13:36.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/708 objects degraded (0.989%), 1 pg degraded) 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3001473352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: from='client.59078 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: from='client.59078 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[29876]: osdmap e858: 8 total, 8 up, 8 in 2024-03-30T15:13:36.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/708 objects degraded (0.989%), 1 pg degraded) 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3001473352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: from='client.59078 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: from='client.59078 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:36.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:35 smithi028 ceph-mon[41614]: osdmap e858: 8 total, 8 up, 8 in 2024-03-30T15:13:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:36.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:36.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:36.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:36.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:36.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:36.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:36.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: pgmap v1324: 182 pgs: 182 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: osdmap e859: 8 total, 8 up, 8 in 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:36.950 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:36 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:36.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:36.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:37.069 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:37.069 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:37.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:37.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:13:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:13:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:13:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: pgmap v1324: 182 pgs: 182 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: osdmap e859: 8 total, 8 up, 8 in 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: pgmap v1324: 182 pgs: 182 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-30T15:13:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T15:13:37.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T15:13:37.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: osdmap e859: 8 total, 8 up, 8 in 2024-03-30T15:13:37.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:37.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:36 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:37.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:37.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:37.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:37.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:37.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:37.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:37.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:37.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:13:37.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:37.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:37.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:37.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:38.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:38.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: osdmap e860: 8 total, 8 up, 8 in 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:38.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: osdmap e860: 8 total, 8 up, 8 in 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:38.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:38.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:38.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:13:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:13:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T15:13:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: osdmap e860: 8 total, 8 up, 8 in 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:38.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:38.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:38.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:38.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:38.404 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:38.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:38.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:38.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:38.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 -- 172.21.15.28:0/3850646027 >> v1:172.21.15.28:6789/0 conn(0x7f2be0107000 legacy=0x7f2be01073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 -- 172.21.15.28:0/3850646027 shutdown_connections 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 -- 172.21.15.28:0/3850646027 >> 172.21.15.28:0/3850646027 conn(0x7f2be0077370 msgr2=0x7f2be00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 -- 172.21.15.28:0/3850646027 shutdown_connections 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 -- 172.21.15.28:0/3850646027 wait complete. 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 librados: starting msgr at 2024-03-30T15:13:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.659+0000 7f2be4a83700 1 librados: starting objecter 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 Processor -- start 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 -- start start 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 librados: setting wanted keys 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 librados: calling monclient init 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2be01a7110 con 0x7f2be010ae80 2024-03-30T15:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2be01a8350 con 0x7f2be0107000 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2be4a83700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2be01a9590 con 0x7f2be010eb50 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2bded9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f2be010eb50 0x7f2be01a5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45374/0 (socket says 172.21.15.28:45374) 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2bded9d700 1 -- 172.21.15.28:0/3390234749 learned_addr learned my addr 172.21.15.28:0/3390234749 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.660+0000 7f2bd77fe700 10 client.?.objecter ms_handle_connect 0x7f2be010ae80 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 10 client.?.objecter ms_handle_connect 0x7f2be010eb50 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1782542786 0 0) 0x7f2be01a7110 con 0x7f2be010ae80 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bcc0031f0 con 0x7f2be010ae80 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 10 client.?.objecter ms_handle_connect 0x7f2be0107000 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560851980 0 0) 0x7f2be01a9590 con 0x7f2be010eb50 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be01a7110 con 0x7f2be010eb50 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1146227621 0 0) 0x7f2be01a8350 con 0x7f2be0107000 2024-03-30T15:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be01a9590 con 0x7f2be0107000 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2473480130 0 0) 0x7f2bcc0031f0 con 0x7f2be010ae80 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be01a8350 con 0x7f2be010ae80 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1536893362 0 0) 0x7f2be01a7110 con 0x7f2be010eb50 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bcc0031f0 con 0x7f2be010eb50 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2bc80018b0 con 0x7f2be010ae80 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2bd8001b70 con 0x7f2be010eb50 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3635420469 0 0) 0x7f2be01a9590 con 0x7f2be0107000 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be01a7110 con 0x7f2be0107000 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2bd0001510 con 0x7f2be0107000 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1740482330 0 0) 0x7f2bcc0031f0 con 0x7f2be010eb50 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 >> v1:172.21.15.134:6789/0 conn(0x7f2be0107000 legacy=0x7f2be00ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 >> v1:172.21.15.28:6789/0 conn(0x7f2be010ae80 legacy=0x7f2be01a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be01aa7d0 con 0x7f2be010eb50 2024-03-30T15:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2be01a97c0 con 0x7f2be010eb50 2024-03-30T15:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2be4a83700 10 client.59084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.661+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2be01a9c80 con 0x7f2be010eb50 2024-03-30T15:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.662+0000 7f2be4a83700 1 librados: init done 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.662+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f2bd8003ea0 con 0x7f2be010eb50 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.662+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f2bd8005e90 con 0x7f2be010eb50 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.663+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f2bd8006140 con 0x7f2be010eb50 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc0005120 con 0x7f2be010eb50 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(861..861 src has 442..861) ==== 8801+0+0 (unknown 2684825233 0 0) 0x7f2bd808a970 con 0x7f2be010eb50 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2bd77fe700 10 client.59084.objecter ms_dispatch 0x7f2be0001550 osd_map(861..861 src has 442..861) 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2bd77fe700 3 client.59084.objecter handle_osd_map got epochs [861,861] > 0 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2bd77fe700 3 client.59084.objecter handle_osd_map decoding full epoch 861 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.664+0000 7f2bd77fe700 20 client.59084.objecter dump_active .. 0 homeless 2024-03-30T15:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.665+0000 7f2bd77fe700 10 client.59084.objecter ms_handle_connect 0x7f2bcc070520 2024-03-30T15:13:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.670+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2bd8057900 con 0x7f2be010eb50 2024-03-30T15:13:38.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:38.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:38.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:38.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:38.896+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f2bc0005410 con 0x7f2be010eb50 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: pgmap v1327: 214 pgs: 13 creating+peering, 19 unknown, 182 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:38.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: osdmap e861: 8 total, 8 up, 8 in 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: pgmap v1327: 214 pgs: 13 creating+peering, 19 unknown, 182 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:13:38.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:38.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: osdmap e861: 8 total, 8 up, 8 in 2024-03-30T15:13:38.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:38.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:38 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:38.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:39.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:39.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:39.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: pgmap v1327: 214 pgs: 13 creating+peering, 19 unknown, 182 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: osdmap e861: 8 total, 8 up, 8 in 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:39.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:38 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:39.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:39.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:39.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.350+0000 7fdbc021f700 1 -- 172.21.15.28:0/2749653077 >> v1:172.21.15.28:6789/0 conn(0x7fdbb810d950 legacy=0x7fdbb810fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.350+0000 7fdbc021f700 1 -- 172.21.15.28:0/2749653077 shutdown_connections 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.350+0000 7fdbc021f700 1 -- 172.21.15.28:0/2749653077 >> 172.21.15.28:0/2749653077 conn(0x7fdbb80fc9f0 msgr2=0x7fdbb80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.350+0000 7fdbc021f700 1 -- 172.21.15.28:0/2749653077 shutdown_connections 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.350+0000 7fdbc021f700 1 -- 172.21.15.28:0/2749653077 wait complete. 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 librados: starting msgr at 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 librados: starting objecter 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 Processor -- start 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 -- start start 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 librados: setting wanted keys 2024-03-30T15:13:39.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.351+0000 7fdbc021f700 1 librados: calling monclient init 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbc021f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbb806b450 con 0x7fdbb810d950 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbc021f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbb81a8530 con 0x7fdbb8109db0 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbc021f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdbb81a9710 con 0x7fdbb8101120 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbbe7bc700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdbb810d950 0x7fdbb81a5e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37478/0 (socket says 172.21.15.28:37478) 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbbdfbb700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdbb8101120 0x7fdbb8105780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45382/0 (socket says 172.21.15.28:45382) 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbbe7bc700 1 -- 172.21.15.28:0/3516542223 learned_addr learned my addr 172.21.15.28:0/3516542223 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter ms_handle_connect 0x7fdbb810d950 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter ms_handle_connect 0x7fdbb8109db0 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter ms_handle_connect 0x7fdbb8101120 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2644990325 0 0) 0x7fdbb81a8530 con 0x7fdbb8109db0 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba4003aa0 con 0x7fdbb8109db0 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766175694 0 0) 0x7fdbb806b450 con 0x7fdbb810d950 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb81a8530 con 0x7fdbb810d950 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306834780 0 0) 0x7fdbb81a9710 con 0x7fdbb8101120 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb806b450 con 0x7fdbb8101120 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3539420239 0 0) 0x7fdba4003aa0 con 0x7fdbb8109db0 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.352+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbb81a9710 con 0x7fdbb8109db0 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdba8001a30 con 0x7fdbb8109db0 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 114856066 0 0) 0x7fdbb81a8530 con 0x7fdbb810d950 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba4003aa0 con 0x7fdbb810d950 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3560473136 0 0) 0x7fdbb806b450 con 0x7fdbb8101120 2024-03-30T15:13:39.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbb81a8530 con 0x7fdbb8101120 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdbb00049c0 con 0x7fdbb810d950 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdbb40017b0 con 0x7fdbb8101120 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2441762815 0 0) 0x7fdbb81a9710 con 0x7fdbb8109db0 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6790/0 conn(0x7fdbb8101120 legacy=0x7fdbb8105780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6789/0 conn(0x7fdbb810d950 legacy=0x7fdbb81a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb81aa8f0 con 0x7fdbb8109db0 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb81a7580 con 0x7fdbb8109db0 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbc021f700 10 client.51079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbb81a7a40 con 0x7fdbb8109db0 2024-03-30T15:13:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbc021f700 1 librados: init done 2024-03-30T15:13:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdba8001c90 con 0x7fdbb8109db0 2024-03-30T15:13:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.353+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdba8004940 con 0x7fdbb8109db0 2024-03-30T15:13:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.354+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdb9c005130 con 0x7fdbb8109db0 2024-03-30T15:13:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdba8003930 con 0x7fdbb8109db0 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(861..861 src has 442..861) ==== 8801+0+0 (unknown 2684825233 0 0) 0x7fdba80885d0 con 0x7fdbb8109db0 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 osd_map(861..861 src has 442..861) 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 3 client.51079.objecter handle_osd_map got epochs [861,861] > 0 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 3 client.51079.objecter handle_osd_map decoding full epoch 861 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 20 client.51079.objecter dump_active .. 0 homeless 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v861) ==== 54+0+16 (unknown 1331679206 0 2420000192) 0x7fdba8055540 con 0x7fdbb8109db0 2024-03-30T15:13:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.355+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdba4074980 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 10 client.51079.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 1 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 1 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 0 1 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 4 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 10 client.51079.objecter _send_command 1 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c004b80 con 0x7fdb9c004760 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c004760 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c004b80 con 0x7fdb9c004760 2024-03-30T15:13:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 1: 0 ) 2024-03-30T15:13:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 1 = system:0 2024-03-30T15:13:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 4 2024-03-30T15:13:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.356+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 0 1 2024-03-30T15:13:39.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 10 client.51079.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 2 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 2 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c005410 osd=0 3 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 0 2 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 4 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 10 client.51079.objecter _send_command 2 2024-03-30T15:13:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.373+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c007b30 con 0x7fdb9c004760 2024-03-30T15:13:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c007b30 con 0x7fdb9c004760 2024-03-30T15:13:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 4 2024-03-30T15:13:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 0 2 2024-03-30T15:13:39.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:13:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 10 client.51079.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 1 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 4 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 10 client.51079.objecter _send_command 3 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.375+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c002190 con 0x7fdb9c0042d0 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c0042d0 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c002190 con 0x7fdb9c0042d0 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 3: 0 ) 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 3 = system:0 2024-03-30T15:13:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 4 2024-03-30T15:13:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.376+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 1 3 2024-03-30T15:13:39.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 10 client.51079.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 4 2024-03-30T15:13:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 4 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c004160 osd=1 3 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 1 4 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 4 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 10 client.51079.objecter _send_command 4 2024-03-30T15:13:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.393+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c001630 con 0x7fdb9c0042d0 2024-03-30T15:13:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c001630 con 0x7fdb9c0042d0 2024-03-30T15:13:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 4 2024-03-30T15:13:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 1 4 2024-03-30T15:13:39.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:39.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 10 client.51079.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 5 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 5 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 2 5 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 4 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 10 client.51079.objecter _send_command 5 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c011040 con 0x7fdb9c0089c0 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.395+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c0089c0 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.396+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c011040 con 0x7fdb9c0089c0 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.396+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 5: 0 ) 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.396+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 5 = system:0 2024-03-30T15:13:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.396+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 4 2024-03-30T15:13:39.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.396+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 2 5 2024-03-30T15:13:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 10 client.51079.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 6 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 6 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c001a30 osd=2 3 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 2 6 2024-03-30T15:13:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 4 2024-03-30T15:13:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 10 client.51079.objecter _send_command 6 2024-03-30T15:13:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.413+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c008050 con 0x7fdb9c0089c0 2024-03-30T15:13:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c008050 con 0x7fdb9c0089c0 2024-03-30T15:13:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 4 2024-03-30T15:13:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 2 6 2024-03-30T15:13:39.418 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:13:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 10 client.51079.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 7 2024-03-30T15:13:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 7 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 3 7 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 4 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 10 client.51079.objecter _send_command 7 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.414+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c011d60 con 0x7fdb9c0111f0 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c0111f0 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c011d60 con 0x7fdb9c0111f0 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 7: 0 ) 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 7 = system:0 2024-03-30T15:13:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 4 2024-03-30T15:13:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.415+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 3 7 2024-03-30T15:13:39.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 10 client.51079.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 8 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 8 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c00cd60 osd=3 3 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 3 8 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 4 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 10 client.51079.objecter _send_command 8 2024-03-30T15:13:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.432+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c012710 con 0x7fdb9c0111f0 2024-03-30T15:13:39.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c012710 con 0x7fdb9c0111f0 2024-03-30T15:13:39.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 4 2024-03-30T15:13:39.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 3 8 2024-03-30T15:13:39.436 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 10 client.51079.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 9 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 9 2024-03-30T15:13:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 4 9 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 4 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 10 client.51079.objecter _send_command 9 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c01a860 con 0x7fdb9c016c00 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.434+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c016c00 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.435+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c01a860 con 0x7fdb9c016c00 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.435+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 9: 0 ) 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.435+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 9 = system:0 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.435+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 4 2024-03-30T15:13:39.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.435+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 4 9 2024-03-30T15:13:39.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 10 client.51079.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 10 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 10 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c012710 osd=4 3 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 4 10 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 4 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 10 client.51079.objecter _send_command 10 2024-03-30T15:13:39.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.452+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c01aa40 con 0x7fdb9c016c00 2024-03-30T15:13:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c01aa40 con 0x7fdb9c016c00 2024-03-30T15:13:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 4 2024-03-30T15:13:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 4 10 2024-03-30T15:13:39.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 10 client.51079.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 11 2024-03-30T15:13:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 11 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 5 11 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 4 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 10 client.51079.objecter _send_command 11 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.454+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c01bdd0 con 0x7fdb9c01b160 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c01b160 2024-03-30T15:13:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c01bdd0 con 0x7fdb9c01b160 2024-03-30T15:13:39.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 11: 0 ) 2024-03-30T15:13:39.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 11 = system:0 2024-03-30T15:13:39.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 4 2024-03-30T15:13:39.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.455+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 5 11 2024-03-30T15:13:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 10 client.51079.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 12 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 12 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c01aa40 osd=5 3 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 5 12 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 4 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 10 client.51079.objecter _send_command 12 2024-03-30T15:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.472+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c01c050 con 0x7fdb9c01b160 2024-03-30T15:13:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdba8058f90 con 0x7fdb9c01b160 2024-03-30T15:13:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 4 2024-03-30T15:13:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 5 12 2024-03-30T15:13:39.476 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 10 client.51079.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 13 2024-03-30T15:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 13 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 6 13 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 4 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 10 client.51079.objecter _send_command 13 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c023480 con 0x7fdb9c01f740 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.474+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c01f740 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.475+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c023480 con 0x7fdb9c01f740 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.475+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 13: 0 ) 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.475+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 13 = system:0 2024-03-30T15:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.475+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 4 2024-03-30T15:13:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.475+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 6 13 2024-03-30T15:13:39.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 10 client.51079.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 14 2024-03-30T15:13:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 14 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c01bdd0 osd=6 3 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 6 14 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 4 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 10 client.51079.objecter _send_command 14 2024-03-30T15:13:39.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.493+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c023660 con 0x7fdb9c01f740 2024-03-30T15:13:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c023660 con 0x7fdb9c01f740 2024-03-30T15:13:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 4 2024-03-30T15:13:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 6 14 2024-03-30T15:13:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 10 client.51079.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 15 2024-03-30T15:13:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 15 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 7 15 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 4 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 10 client.51079.objecter _send_command 15 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.495+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdb9c024920 con 0x7fdb9c023d80 2024-03-30T15:13:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.496+0000 7fdbaeffd700 10 client.51079.objecter ms_handle_connect 0x7fdb9c023d80 2024-03-30T15:13:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.497+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb9c024920 con 0x7fdb9c023d80 2024-03-30T15:13:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.497+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 15: 0 ) 2024-03-30T15:13:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.497+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 15 = system:0 2024-03-30T15:13:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.497+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 4 2024-03-30T15:13:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.497+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 7 15 2024-03-30T15:13:39.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 10 client.51079.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign -1 16 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 20 client.51079.objecter _get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_remove -1 16 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 20 client.51079.objecter get_session s=0x7fdb9c023660 osd=7 3 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 15 client.51079.objecter _session_command_op_assign 7 16 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 4 2024-03-30T15:13:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 10 client.51079.objecter _send_command 16 2024-03-30T15:13:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.514+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdb9c024920 con 0x7fdb9c023d80 2024-03-30T15:13:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.516+0000 7fdbaeffd700 1 -- 172.21.15.28:0/3516542223 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdb9c024920 con 0x7fdb9c023d80 2024-03-30T15:13:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.516+0000 7fdbaeffd700 10 client.51079.objecter ms_dispatch 0x7fdbb8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.516+0000 7fdbaeffd700 10 client.51079.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.516+0000 7fdbaeffd700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 4 2024-03-30T15:13:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.516+0000 7fdbaeffd700 15 client.51079.objecter _session_command_op_remove 7 16 2024-03-30T15:13:39.518 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 librados: watch_flush enter 2024-03-30T15:13:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 librados: watch_flush exit 2024-03-30T15:13:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.0 2024-03-30T15:13:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6809/2927075629 conn(0x7fdb9c004760 legacy=0x7fdb9c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c005410 osd=0 1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6805/1850313547 conn(0x7fdb9c0042d0 legacy=0x7fdb9c00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c004160 osd=1 1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.2 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6801/2294873818 conn(0x7fdb9c0089c0 legacy=0x7fdb9c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c001a30 osd=2 1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.3 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6813/1506157250 conn(0x7fdb9c0111f0 legacy=0x7fdb9c011650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c00cd60 osd=3 1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.4 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.134:6800/4077237787 conn(0x7fdb9c016c00 legacy=0x7fdb9c018ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c012710 osd=4 1 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.5 2024-03-30T15:13:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.134:6804/3668151179 conn(0x7fdb9c01b160 legacy=0x7fdb9c01b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01aa40 osd=5 1 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.6 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.134:6808/2135224194 conn(0x7fdb9c01f740 legacy=0x7fdb9c021b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbbe7bc700 1 -- 172.21.15.28:0/3516542223 reap_dead start 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c01bdd0 osd=6 1 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter close_session for osd.7 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.134:6812/3758190298 conn(0x7fdb9c023d80 legacy=0x7fdb9c0241e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbbe7bc700 1 -- 172.21.15.28:0/3516542223 reap_dead start 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter put_session s=0x7fdb9c023660 osd=7 1 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 20 client.51079.objecter shutdown clearing up homeless session... 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 10 client.51079.objecter successfully canceled tick 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbbe7bc700 1 -- 172.21.15.28:0/3516542223 reap_dead start 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbbe7bc700 1 -- 172.21.15.28:0/3516542223 reap_dead start 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdba4074980 legacy=0x7fdba4076e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> v1:172.21.15.134:6789/0 conn(0x7fdbb8109db0 legacy=0x7fdbb8105e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 shutdown_connections 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 >> 172.21.15.28:0/3516542223 conn(0x7fdbb80fc9f0 msgr2=0x7fdbb8110750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 shutdown_connections 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 -- 172.21.15.28:0/3516542223 wait complete. 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.517+0000 7fdbc021f700 1 librados: shutdown 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:39.522 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.589+0000 7f2bd77fe700 1 -- 172.21.15.28:0/3390234749 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v862) ==== 113+0+0 (unknown 2784024500 0 0) 0x7f2bd805b350 con 0x7f2be010eb50 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 10 librados: watch_flush enter 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 10 librados: watch_flush exit 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 20 client.59084.objecter shutdown clearing up homeless session... 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 10 client.59084.objecter successfully canceled tick 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 >> v1:172.21.15.28:6800/3011431093 conn(0x7f2bcc070520 legacy=0x7f2bcc0729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 >> v1:172.21.15.28:6790/0 conn(0x7f2be010eb50 legacy=0x7f2be01a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 shutdown_connections 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 >> 172.21.15.28:0/3390234749 conn(0x7f2be0077370 msgr2=0x7f2be010de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 shutdown_connections 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 -- 172.21.15.28:0/3390234749 wait complete. 2024-03-30T15:13:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:39.592+0000 7f2be4a83700 1 librados: shutdown 2024-03-30T15:13:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T15:13:39.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:39.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:39.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:13:39.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:39.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:39.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:39.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:39.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:40.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3390234749' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.59084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3516542223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.59084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: osdmap e862: 8 total, 8 up, 8 in 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:39 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3390234749' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.59084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3516542223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:13:40.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.59084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: osdmap e862: 8 total, 8 up, 8 in 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3390234749' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.59084 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3516542223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:13:40.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:13:40.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:40.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.59084 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:13:40.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: osdmap e862: 8 total, 8 up, 8 in 2024-03-30T15:13:40.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:39 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]: dispatch 2024-03-30T15:13:40.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:40.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:40.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:40.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:40.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:40.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:40.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:40.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:40.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:40.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:40.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:41.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.038 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:41.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:41.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:41.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 5.1b deep-scrub starts 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 5.1b deep-scrub ok 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: pgmap v1330: 214 pgs: 13 creating+peering, 2 unknown, 199 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]': finished 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[29876]: osdmap e863: 8 total, 8 up, 8 in 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:41.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 5.1b deep-scrub starts 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 5.1b deep-scrub ok 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: pgmap v1330: 214 pgs: 13 creating+peering, 2 unknown, 199 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]': finished 2024-03-30T15:13:41.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:40 smithi028 ceph-mon[41614]: osdmap e863: 8 total, 8 up, 8 in 2024-03-30T15:13:41.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:41.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:41.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 5.1b deep-scrub starts 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 5.1b deep-scrub ok 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: pgmap v1330: 214 pgs: 13 creating+peering, 2 unknown, 199 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:13:41.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-144", "mode": "readproxy"}]': finished 2024-03-30T15:13:41.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:40 smithi134 ceph-mon[33714]: osdmap e863: 8 total, 8 up, 8 in 2024-03-30T15:13:41.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:41.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:41.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:41.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:41.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:41.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:41.673 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:41.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:41.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:41.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:41.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:42.027 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:42.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:42.108 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:13:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:13:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:13:42.108 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:13:42.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[29876]: 6.18 deep-scrub starts 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[29876]: 6.18 deep-scrub ok 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[29876]: 6.1c deep-scrub starts 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[29876]: 6.1c deep-scrub ok 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[41614]: 6.18 deep-scrub starts 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[41614]: 6.18 deep-scrub ok 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[41614]: 6.1c deep-scrub starts 2024-03-30T15:13:42.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:41 smithi028 ceph-mon[41614]: 6.1c deep-scrub ok 2024-03-30T15:13:42.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:41 smithi134 ceph-mon[33714]: 6.18 deep-scrub starts 2024-03-30T15:13:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:41 smithi134 ceph-mon[33714]: 6.18 deep-scrub ok 2024-03-30T15:13:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:41 smithi134 ceph-mon[33714]: 6.1c deep-scrub starts 2024-03-30T15:13:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:41 smithi134 ceph-mon[33714]: 6.1c deep-scrub ok 2024-03-30T15:13:42.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:42.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:42.312 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:42.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:42.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:42.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:42.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:42.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-03-30T15:13:42.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:42.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:42.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:42.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:42.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:42.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 6.3 deep-scrub starts 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 6.3 deep-scrub ok 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: pgmap v1332: 214 pgs: 4 creating+peering, 210 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 221 B/s wr, 1 op/s 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:13:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 6.3 deep-scrub starts 2024-03-30T15:13:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 6.3 deep-scrub ok 2024-03-30T15:13:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: pgmap v1332: 214 pgs: 4 creating+peering, 210 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 221 B/s wr, 1 op/s 2024-03-30T15:13:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:13:42.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:42 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:13:43.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:43.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:13:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:13:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 6.3 deep-scrub starts 2024-03-30T15:13:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 6.3 deep-scrub ok 2024-03-30T15:13:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: pgmap v1332: 214 pgs: 4 creating+peering, 210 active+clean; 4.4 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 221 B/s wr, 1 op/s 2024-03-30T15:13:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:13:43.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:42 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:13:43.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:43.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:43.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:43.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:43.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:13:43.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:43.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:13:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:43.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:43.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:43.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:43.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:43.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:43.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:44.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:44.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 322.2s0 scrub starts 2024-03-30T15:13:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:43 smithi134 ceph-mon[33714]: 322.2 scrub ok 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 322.2s0 scrub starts 2024-03-30T15:13:44.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[41614]: 322.2 scrub ok 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 322.2s0 scrub starts 2024-03-30T15:13:44.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:43 smithi028 ceph-mon[29876]: 322.2 scrub ok 2024-03-30T15:13:44.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:44.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:44.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:44.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:44.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:44.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:44.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:44.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:44.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:44.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:45.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:45.122 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: pgmap v1333: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:13:45.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: pgmap v1333: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:44 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: pgmap v1333: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T15:13:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T15:13:45.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:13:45.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:44 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:13:45.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:45.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:45.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:45.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:45.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:45.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:45.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:45.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:45.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:45.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:45.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[41614]: osdmap e864: 8 total, 8 up, 8 in 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T15:13:45.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:13:45.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:13:45.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[29876]: osdmap e864: 8 total, 8 up, 8 in 2024-03-30T15:13:45.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T15:13:45.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:45 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T15:13:45.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:46.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:46.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:46.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:13:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:13:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:45 smithi134 ceph-mon[33714]: osdmap e864: 8 total, 8 up, 8 in 2024-03-30T15:13:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:45 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T15:13:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:45 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T15:13:46.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:46.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:46.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:46.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:46.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:46.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:46.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:46.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:46.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:46.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 -- 172.21.15.28:0/1367866300 >> v1:172.21.15.28:6790/0 conn(0x7ff27810d0f0 legacy=0x7ff27810f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 -- 172.21.15.28:0/1367866300 shutdown_connections 2024-03-30T15:13:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 -- 172.21.15.28:0/1367866300 >> 172.21.15.28:0/1367866300 conn(0x7ff2780ffcb0 msgr2=0x7ff2781020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 -- 172.21.15.28:0/1367866300 shutdown_connections 2024-03-30T15:13:46.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 -- 172.21.15.28:0/1367866300 wait complete. 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 librados: starting msgr at 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.855+0000 7ff27e022700 1 librados: starting objecter 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 Processor -- start 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 -- start start 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 librados: setting wanted keys 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 librados: calling monclient init 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2781a92b0 con 0x7ff278104460 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2781aa4d0 con 0x7ff278110c90 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff27e022700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff2781ab710 con 0x7ff27810d0f0 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff2777fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff278104460 0x7ff278109750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37496/0 (socket says 172.21.15.28:37496) 2024-03-30T15:13:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff2777fe700 1 -- 172.21.15.28:0/1760973819 learned_addr learned my addr 172.21.15.28:0/1760973819 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter ms_handle_connect 0x7ff27810d0f0 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter ms_handle_connect 0x7ff278104460 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781328824 0 0) 0x7ff2781ab710 con 0x7ff27810d0f0 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2600035b0 con 0x7ff27810d0f0 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter ms_handle_connect 0x7ff278110c90 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181374116 0 0) 0x7ff2781a92b0 con 0x7ff278104460 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.856+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2781ab710 con 0x7ff278104460 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983479586 0 0) 0x7ff2781aa4d0 con 0x7ff278110c90 2024-03-30T15:13:46.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2781a92b0 con 0x7ff278110c90 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2129287484 0 0) 0x7ff2600035b0 con 0x7ff27810d0f0 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2781aa4d0 con 0x7ff27810d0f0 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff26c004540 con 0x7ff27810d0f0 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 238666141 0 0) 0x7ff2781ab710 con 0x7ff278104460 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2600035b0 con 0x7ff278104460 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff268001870 con 0x7ff278104460 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1544830748 0 0) 0x7ff2781a92b0 con 0x7ff278110c90 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2781ab710 con 0x7ff278110c90 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff270001bb0 con 0x7ff278110c90 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4148562651 0 0) 0x7ff2781aa4d0 con 0x7ff27810d0f0 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 >> v1:172.21.15.134:6789/0 conn(0x7ff278110c90 legacy=0x7ff2781a7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 >> v1:172.21.15.28:6789/0 conn(0x7ff278104460 legacy=0x7ff278109750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2781ac950 con 0x7ff27810d0f0 2024-03-30T15:13:46.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2781a94e0 con 0x7ff27810d0f0 2024-03-30T15:13:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff27e022700 10 client.59096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2781a99a0 con 0x7ff27810d0f0 2024-03-30T15:13:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff27e022700 1 librados: init done 2024-03-30T15:13:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff26c001b70 con 0x7ff27810d0f0 2024-03-30T15:13:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.857+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff26c005070 con 0x7ff27810d0f0 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.858+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff264005120 con 0x7ff27810d0f0 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.859+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff26c003760 con 0x7ff27810d0f0 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(864..864 src has 442..864) ==== 8801+0+0 (unknown 3622034158 0 0) 0x7ff26c088a30 con 0x7ff27810d0f0 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 10 client.59096.objecter ms_dispatch 0x7ff278001550 osd_map(864..864 src has 442..864) 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 3 client.59096.objecter handle_osd_map got epochs [864,864] > 0 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 3 client.59096.objecter handle_osd_map decoding full epoch 864 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 20 client.59096.objecter dump_active .. 0 homeless 2024-03-30T15:13:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.860+0000 7ff274ff9700 10 client.59096.objecter ms_handle_connect 0x7ff2600703f0 2024-03-30T15:13:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:46.864+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff26c0559c0 con 0x7ff27810d0f0 2024-03-30T15:13:46.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:46.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:46.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.080+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff264005410 con 0x7ff27810d0f0 2024-03-30T15:13:47.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:47.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:46 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:13:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:46 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:13:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:46 smithi134 ceph-mon[33714]: pgmap v1335: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-30T15:13:47.197 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[41614]: pgmap v1335: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:13:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:46 smithi028 ceph-mon[29876]: pgmap v1335: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-03-30T15:13:47.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:47.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:47.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:47.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:47.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:47.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:47.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.575+0000 7ff274ff9700 1 -- 172.21.15.28:0/1760973819 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v865) ==== 87+0+0 (unknown 2417716571 0 0) 0x7ff26c059410 con 0x7ff27810d0f0 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 10 librados: watch_flush enter 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 10 librados: watch_flush exit 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 20 client.59096.objecter shutdown clearing up homeless session... 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 10 client.59096.objecter successfully canceled tick 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff2600703f0 legacy=0x7ff2600728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 >> v1:172.21.15.28:6790/0 conn(0x7ff27810d0f0 legacy=0x7ff2781a4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 shutdown_connections 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 >> 172.21.15.28:0/1760973819 conn(0x7ff2780ffcb0 msgr2=0x7ff278113e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 shutdown_connections 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.579+0000 7ff27e022700 1 -- 172.21.15.28:0/1760973819 wait complete. 2024-03-30T15:13:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:47.580+0000 7ff27e022700 1 librados: shutdown 2024-03-30T15:13:47.583 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:13:47.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:47.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:47.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:47.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:47.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:47.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:47.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:48.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:48.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 1.a scrub starts 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: 1.a scrub ok 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1760973819' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: from='client.59096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:48.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: from='client.59096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:48.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:47 smithi134 ceph-mon[33714]: osdmap e865: 8 total, 8 up, 8 in 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 1.a scrub starts 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: 1.a scrub ok 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1760973819' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: from='client.59096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: from='client.59096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:48.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[29876]: osdmap e865: 8 total, 8 up, 8 in 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 1.a scrub starts 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: 1.a scrub ok 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1760973819' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: from='client.59096 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: from='client.59096 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:13:48.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:47 smithi028 ceph-mon[41614]: osdmap e865: 8 total, 8 up, 8 in 2024-03-30T15:13:48.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:48.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:48.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:48.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:48.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:48.596 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:48.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:48.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:48.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:48.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:48.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:49.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:49.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:49.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:48 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:13:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:48 smithi134 ceph-mon[33714]: pgmap v1337: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-03-30T15:13:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:49.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:48 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:13:49.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:48 smithi028 ceph-mon[29876]: pgmap v1337: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-03-30T15:13:49.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:48 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:13:49.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:48 smithi028 ceph-mon[41614]: pgmap v1337: 214 pgs: 214 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 807 B/s rd, 0 op/s 2024-03-30T15:13:49.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:49.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:49.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:49.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:49.592 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:13:49.592 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:13:49.592 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:13:49.592 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:13:49.592 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:13:49.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:49.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:49.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:49.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:49.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:49.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:50.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:50.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:50.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:50.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:50.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:50.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:50.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:50.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-03-30T15:13:50.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:50.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:50.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:50.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:50.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:50.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:50.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:13:50.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:51.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:51.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:51.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:13:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:13:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.179+0000 7fc64206c700 1 -- 172.21.15.28:0/774419768 >> v1:172.21.15.28:6790/0 conn(0x7fc63c102240 legacy=0x7fc63c102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 -- 172.21.15.28:0/774419768 shutdown_connections 2024-03-30T15:13:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 -- 172.21.15.28:0/774419768 >> 172.21.15.28:0/774419768 conn(0x7fc63c0fdaf0 msgr2=0x7fc63c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 -- 172.21.15.28:0/774419768 shutdown_connections 2024-03-30T15:13:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 -- 172.21.15.28:0/774419768 wait complete. 2024-03-30T15:13:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 librados: starting msgr at 2024-03-30T15:13:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 librados: starting objecter 2024-03-30T15:13:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 Processor -- start 2024-03-30T15:13:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 -- start start 2024-03-30T15:13:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 librados: setting wanted keys 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.180+0000 7fc64206c700 1 librados: calling monclient init 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc64206c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc63c1a93a0 con 0x7fc63c10aed0 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc64206c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc63c1aa5c0 con 0x7fc63c10ea70 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc64206c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc63c1ab800 con 0x7fc63c102240 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc63b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fc63c102240 0x7fc63c1075c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54702/0 (socket says 172.21.15.28:54702) 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc63b7fe700 1 -- 172.21.15.28:0/1723639613 learned_addr learned my addr 172.21.15.28:0/1723639613 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter ms_handle_connect 0x7fc63c102240 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter ms_handle_connect 0x7fc63c10ea70 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter ms_handle_connect 0x7fc63c10aed0 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169715869 0 0) 0x7fc63c1ab800 con 0x7fc63c102240 2024-03-30T15:13:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6280039c0 con 0x7fc63c102240 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324905149 0 0) 0x7fc63c1aa5c0 con 0x7fc63c10ea70 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63c1ab800 con 0x7fc63c10ea70 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562464395 0 0) 0x7fc63c1a93a0 con 0x7fc63c10aed0 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63c1aa5c0 con 0x7fc63c10aed0 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1422114648 0 0) 0x7fc6280039c0 con 0x7fc63c102240 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc63c1a93a0 con 0x7fc63c102240 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.181+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc62c004860 con 0x7fc63c102240 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3533594479 0 0) 0x7fc63c1ab800 con 0x7fc63c10ea70 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6280039c0 con 0x7fc63c10ea70 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc634001210 con 0x7fc63c10ea70 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2285567090 0 0) 0x7fc63c1a93a0 con 0x7fc63c102240 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 >> v1:172.21.15.134:6789/0 conn(0x7fc63c10ea70 legacy=0x7fc63c1a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 >> v1:172.21.15.28:6789/0 conn(0x7fc63c10aed0 legacy=0x7fc63c1a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c1aca40 con 0x7fc63c102240 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c1aa7f0 con 0x7fc63c102240 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc64206c700 10 client.59102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc63c1aacf0 con 0x7fc63c102240 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc64206c700 1 librados: init done 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fc62c004a80 con 0x7fc63c102240 2024-03-30T15:13:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.182+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fc62c004ff0 con 0x7fc63c102240 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.183+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c005120 con 0x7fc63c102240 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.183+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fc62c005190 con 0x7fc63c102240 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(865..865 src has 442..865) ==== 8801+0+0 (unknown 535741733 0 0) 0x7fc62c089d50 con 0x7fc63c102240 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 10 client.59102.objecter ms_dispatch 0x7fc63c001550 osd_map(865..865 src has 442..865) 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 3 client.59102.objecter handle_osd_map got epochs [865,865] > 0 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 3 client.59102.objecter handle_osd_map decoding full epoch 865 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 20 client.59102.objecter dump_active .. 0 homeless 2024-03-30T15:13:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.184+0000 7fc638ff9700 10 client.59102.objecter ms_handle_connect 0x7fc628070290 2024-03-30T15:13:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.192+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc62c0ba270 con 0x7fc63c102240 2024-03-30T15:13:51.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:51.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:51.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:51.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:51.417+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7fc61c005410 con 0x7fc63c102240 2024-03-30T15:13:51.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:51.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:51.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.617 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:51.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:51 smithi134 ceph-mon[33714]: pgmap v1338: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:13:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:51 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:51 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:51.722 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[41614]: pgmap v1338: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:13:51.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[29876]: pgmap v1338: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:13:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:51 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:51.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:51.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:51.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:51.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:51.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:52.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:52.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:52.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:52.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:52.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:52.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.259+0000 7fc638ff9700 1 -- 172.21.15.28:0/1723639613 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v867) ==== 113+0+0 (unknown 2014377770 0 0) 0x7fc62c056ce0 con 0x7fc63c102240 2024-03-30T15:13:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 10 librados: watch_flush enter 2024-03-30T15:13:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 10 librados: watch_flush exit 2024-03-30T15:13:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 20 client.59102.objecter shutdown clearing up homeless session... 2024-03-30T15:13:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 10 client.59102.objecter successfully canceled tick 2024-03-30T15:13:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 >> v1:172.21.15.28:6800/3011431093 conn(0x7fc628070290 legacy=0x7fc628072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.263+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 >> v1:172.21.15.28:6790/0 conn(0x7fc63c102240 legacy=0x7fc63c1075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.267+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 shutdown_connections 2024-03-30T15:13:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.267+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 >> 172.21.15.28:0/1723639613 conn(0x7fc63c0fdaf0 msgr2=0x7fc63c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.267+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 shutdown_connections 2024-03-30T15:13:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.267+0000 7fc64206c700 1 -- 172.21.15.28:0/1723639613 wait complete. 2024-03-30T15:13:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.268+0000 7fc64206c700 1 librados: shutdown 2024-03-30T15:13:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T15:13:52.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:52.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:52.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: osdmap e866: 8 total, 8 up, 8 in 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1723639613' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.59102 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: osdmap e866: 8 total, 8 up, 8 in 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1723639613' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.59102 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:52.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:52 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.346+0000 7f5de83a9700 1 -- 172.21.15.28:0/3260452181 >> v1:172.21.15.28:6789/0 conn(0x7f5de0102250 legacy=0x7f5de0102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.346+0000 7f5de83a9700 1 -- 172.21.15.28:0/3260452181 shutdown_connections 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.346+0000 7f5de83a9700 1 -- 172.21.15.28:0/3260452181 >> 172.21.15.28:0/3260452181 conn(0x7f5de00fdb20 msgr2=0x7f5de00fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.346+0000 7f5de83a9700 1 -- 172.21.15.28:0/3260452181 shutdown_connections 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.346+0000 7f5de83a9700 1 -- 172.21.15.28:0/3260452181 wait complete. 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 librados: starting msgr at 2024-03-30T15:13:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 librados: starting objecter 2024-03-30T15:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 Processor -- start 2024-03-30T15:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 -- start start 2024-03-30T15:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 librados: setting wanted keys 2024-03-30T15:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 librados: calling monclient init 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de01a7130 con 0x7f5de0102250 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de01a8350 con 0x7f5de010aee0 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de83a9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5de01a9590 con 0x7f5de010ea80 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de6946700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5de010ea80 0x7f5de01a57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54728/0 (socket says 172.21.15.28:54728) 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 learned_addr learned my addr 172.21.15.28:0/3085432650 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5de6145700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5de0102250 0x7f5de01075c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33374/0 (socket says 172.21.15.28:33374) 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter ms_handle_connect 0x7f5de0102250 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter ms_handle_connect 0x7f5de010ea80 2024-03-30T15:13:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter ms_handle_connect 0x7f5de010aee0 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.347+0000 7f5dd77fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950252803 0 0) 0x7f5de01a7130 con 0x7f5de0102250 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dcc0031f0 con 0x7f5de0102250 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876354133 0 0) 0x7f5de01a9590 con 0x7f5de010ea80 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5de01a7130 con 0x7f5de010ea80 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067671618 0 0) 0x7f5de01a8350 con 0x7f5de010aee0 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5de01a9590 con 0x7f5de010aee0 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 33017446 0 0) 0x7f5dcc0031f0 con 0x7f5de0102250 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5de01a8350 con 0x7f5de0102250 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5ddc004550 con 0x7f5de0102250 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2865259243 0 0) 0x7f5de01a7130 con 0x7f5de010ea80 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5dcc0031f0 con 0x7f5de010ea80 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2955672080 0 0) 0x7f5de01a9590 con 0x7f5de010aee0 2024-03-30T15:13:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5de01a7130 con 0x7f5de010aee0 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5dd8001bb0 con 0x7f5de010ea80 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5dd0001000 con 0x7f5de010aee0 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2008515025 0 0) 0x7f5de01a8350 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6790/0 conn(0x7f5de010ea80 legacy=0x7f5de01a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.134:6789/0 conn(0x7f5de010aee0 legacy=0x7f5de01a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de01aa7d0 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5de01a97c0 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5de83a9700 10 client.25905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5de01a9c80 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5de83a9700 1 librados: init done 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5ddc003bb0 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.348+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5ddc004ef0 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.349+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5dc4005100 con 0x7f5de0102250 2024-03-30T15:13:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5ddc0049a0 con 0x7f5de0102250 2024-03-30T15:13:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(867..867 src has 442..867) ==== 8793+0+0 (unknown 1135492240 0 0) 0x7f5ddc089890 con 0x7f5de0102250 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 osd_map(867..867 src has 442..867) 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 3 client.25905.objecter handle_osd_map got epochs [867,867] > 0 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 3 client.25905.objecter handle_osd_map decoding full epoch 867 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 20 client.25905.objecter dump_active .. 0 homeless 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.350+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v867) ==== 54+0+16 (unknown 3504210969 0 2420000192) 0x7f5ddc089cd0 con 0x7f5de0102250 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dcc070550 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 10 client.25905.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 1 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 1 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 0 1 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 4 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 10 client.25905.objecter _send_command 1 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5dc4004b50 con 0x7f5dc4004730 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.351+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc4004730 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.352+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc4004b50 con 0x7f5dc4004730 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.352+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 1: 0 ) 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.352+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 1 = system:0 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.352+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 4 2024-03-30T15:13:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.352+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 0 1 2024-03-30T15:13:52.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:52.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 10 client.25905.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 2 2024-03-30T15:13:52.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 2 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc40053e0 osd=0 3 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 0 2 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 4 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 10 client.25905.objecter _send_command 2 2024-03-30T15:13:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.369+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc4007b00 con 0x7f5dc4004730 2024-03-30T15:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.370+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc4007b00 con 0x7f5dc4004730 2024-03-30T15:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.370+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.370+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.370+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 4 2024-03-30T15:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.370+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 0 2 2024-03-30T15:13:52.373 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 10 client.25905.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 1 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 4 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 10 client.25905.objecter _send_command 3 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5dc4002160 con 0x7f5dc40042a0 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.371+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc40042a0 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.372+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc4002160 con 0x7f5dc40042a0 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.372+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 3: 0 ) 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.372+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 3 = system:0 2024-03-30T15:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.372+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 4 2024-03-30T15:13:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.372+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 1 3 2024-03-30T15:13:52.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 10 client.25905.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 4 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 4 2024-03-30T15:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4004130 osd=1 3 2024-03-30T15:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 1 4 2024-03-30T15:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 4 2024-03-30T15:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 10 client.25905.objecter _send_command 4 2024-03-30T15:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.389+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc4001600 con 0x7f5dc40042a0 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc4001600 con 0x7f5dc40042a0 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 4 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 1 4 2024-03-30T15:13:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 10 client.25905.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 5 2024-03-30T15:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 5 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 2 5 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 4 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 10 client.25905.objecter _send_command 5 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5dc4010fb0 con 0x7f5dc4008990 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.391+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc4008990 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.392+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc4010fb0 con 0x7f5dc4008990 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.392+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 5: 0 ) 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.392+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 5 = system:0 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.392+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 4 2024-03-30T15:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.392+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 2 5 2024-03-30T15:13:52.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 10 client.25905.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 6 2024-03-30T15:13:52.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 6 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4001a00 osd=2 3 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 2 6 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 4 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 10 client.25905.objecter _send_command 6 2024-03-30T15:13:52.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.408+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc4008020 con 0x7f5dc4008990 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc4008020 con 0x7f5dc4008990 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 4 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 2 6 2024-03-30T15:13:52.412 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:13:52.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 10 client.25905.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 7 2024-03-30T15:13:52.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 7 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 3 7 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 4 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 10 client.25905.objecter _send_command 7 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5dc4011ce0 con 0x7f5dc4011190 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.410+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc4011190 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.411+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc4011ce0 con 0x7f5dc4011190 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.411+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 7: 0 ) 2024-03-30T15:13:52.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.411+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 7 = system:0 2024-03-30T15:13:52.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.411+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 4 2024-03-30T15:13:52.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.411+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 3 7 2024-03-30T15:13:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 10 client.25905.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 8 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 8 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc40082f0 osd=3 3 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 3 8 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 4 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 10 client.25905.objecter _send_command 8 2024-03-30T15:13:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.428+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc4012690 con 0x7f5dc4011190 2024-03-30T15:13:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc4012690 con 0x7f5dc4011190 2024-03-30T15:13:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.432 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 4 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 3 8 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 10 client.25905.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 9 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 9 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 4 9 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 4 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 10 client.25905.objecter _send_command 9 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.430+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5dc401a790 con 0x7f5dc4016b70 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc4016b70 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc401a790 con 0x7f5dc4016b70 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 9: 0 ) 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 9 = system:0 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 4 2024-03-30T15:13:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.431+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 4 9 2024-03-30T15:13:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 10 client.25905.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 10 2024-03-30T15:13:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 10 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc4013810 osd=4 3 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 4 10 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 4 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 10 client.25905.objecter _send_command 10 2024-03-30T15:13:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.448+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc400c870 con 0x7f5dc4016b70 2024-03-30T15:13:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc400c870 con 0x7f5dc4016b70 2024-03-30T15:13:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 4 2024-03-30T15:13:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 4 10 2024-03-30T15:13:52.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:13:52.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 10 client.25905.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 11 2024-03-30T15:13:52.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 11 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 5 11 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 4 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 10 client.25905.objecter _send_command 11 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.450+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5dc400c8a0 con 0x7f5dc401ce10 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc401ce10 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dd000c8c0 con 0x7f5dc401ce10 2024-03-30T15:13:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 11: 0 ) 2024-03-30T15:13:52.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 11 = system:0 2024-03-30T15:13:52.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 4 2024-03-30T15:13:52.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.451+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 5 11 2024-03-30T15:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 10 client.25905.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 12 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 12 2024-03-30T15:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401bf80 osd=5 3 2024-03-30T15:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 5 12 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 4 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 10 client.25905.objecter _send_command 12 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.468+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc401c910 con 0x7f5dc401ce10 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc401c910 con 0x7f5dc401ce10 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 4 2024-03-30T15:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 5 12 2024-03-30T15:13:52.475 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 10 client.25905.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 13 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 13 2024-03-30T15:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 6 13 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 4 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 10 client.25905.objecter _send_command 13 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.470+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5dc4023360 con 0x7f5dc401f850 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.471+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc401f850 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.472+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc4023360 con 0x7f5dc401f850 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.472+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 13: 0 ) 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.472+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 13 = system:0 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.472+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 4 2024-03-30T15:13:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.472+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 6 13 2024-03-30T15:13:52.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:52.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 10 client.25905.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 14 2024-03-30T15:13:52.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 14 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401cb30 osd=6 3 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 6 14 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 4 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 10 client.25905.objecter _send_command 14 2024-03-30T15:13:52.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.489+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc401c420 con 0x7f5dc401f850 2024-03-30T15:13:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.490+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc401c420 con 0x7f5dc401f850 2024-03-30T15:13:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 4 2024-03-30T15:13:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 6 14 2024-03-30T15:13:52.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 10 client.25905.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 15 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 15 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 7 15 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 4 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 10 client.25905.objecter _send_command 15 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5dc401c630 con 0x7f5dc401b470 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.491+0000 7f5dd77fe700 10 client.25905.objecter ms_handle_connect 0x7f5dc401b470 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.492+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5dc401c630 con 0x7f5dc401b470 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.492+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 15: 0 ) 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.492+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 15 = system:0 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.492+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 4 2024-03-30T15:13:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.492+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 7 15 2024-03-30T15:13:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 10 client.25905.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign -1 16 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 20 client.25905.objecter _get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_remove -1 16 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 20 client.25905.objecter get_session s=0x7f5dc401c420 osd=7 3 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 15 client.25905.objecter _session_command_op_assign 7 16 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 4 2024-03-30T15:13:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 10 client.25905.objecter _send_command 16 2024-03-30T15:13:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.509+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5dc401c630 con 0x7f5dc401b470 2024-03-30T15:13:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5dd77fe700 1 -- 172.21.15.28:0/3085432650 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5dc401c630 con 0x7f5dc401b470 2024-03-30T15:13:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5dd77fe700 10 client.25905.objecter ms_dispatch 0x7f5de0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:13:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5dd77fe700 10 client.25905.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:13:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5dd77fe700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 4 2024-03-30T15:13:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5dd77fe700 15 client.25905.objecter _session_command_op_remove 7 16 2024-03-30T15:13:52.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:13:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 librados: watch_flush enter 2024-03-30T15:13:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 librados: watch_flush exit 2024-03-30T15:13:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.0 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6809/2927075629 conn(0x7f5dc4004730 legacy=0x7f5dc4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40053e0 osd=0 1 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.1 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6805/1850313547 conn(0x7f5dc40042a0 legacy=0x7f5dc400ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4004130 osd=1 1 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.2 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6801/2294873818 conn(0x7f5dc4008990 legacy=0x7f5dc400f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4001a00 osd=2 1 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.3 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6813/1506157250 conn(0x7f5dc4011190 legacy=0x7f5dc40115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc40082f0 osd=3 1 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.4 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.134:6800/4077237787 conn(0x7f5dc4016b70 legacy=0x7f5dc4018f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc4013810 osd=4 1 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.5 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.134:6804/3668151179 conn(0x7f5dc401ce10 legacy=0x7f5dc4011ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401bf80 osd=5 1 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.6 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.134:6808/2135224194 conn(0x7f5dc401f850 legacy=0x7f5dc4021c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401cb30 osd=6 1 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter close_session for osd.7 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.134:6812/3758190298 conn(0x7f5dc401b470 legacy=0x7f5dc4024ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter put_session s=0x7f5dc401c420 osd=7 1 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 20 client.25905.objecter shutdown clearing up homeless session... 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 10 client.25905.objecter successfully canceled tick 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 reap_dead start 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 reap_dead start 2024-03-30T15:13:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 reap_dead start 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 reap_dead start 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de6946700 1 -- 172.21.15.28:0/3085432650 reap_dead start 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5dcc070550 legacy=0x7f5dcc072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> v1:172.21.15.28:6789/0 conn(0x7f5de0102250 legacy=0x7f5de01075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.511+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 shutdown_connections 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.512+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 >> 172.21.15.28:0/3085432650 conn(0x7f5de00fdb20 msgr2=0x7f5de0111c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.512+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 shutdown_connections 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.512+0000 7f5de83a9700 1 -- 172.21.15.28:0/3085432650 wait complete. 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:52.512+0000 7f5de83a9700 1 librados: shutdown 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:13:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:13:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:13:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:13:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:13:52.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:52.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:52.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: osdmap e866: 8 total, 8 up, 8 in 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1723639613' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.59102 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:13:52.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:52 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]': finished 2024-03-30T15:13:52.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:52.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:52.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:52.892 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:52.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:52.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:53.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:53.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:53.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:53.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:53.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:53.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:53.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:53.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: pgmap v1340: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.59102 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: osdmap e867: 8 total, 8 up, 8 in 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3085432650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: osdmap e868: 8 total, 8 up, 8 in 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: pgmap v1340: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.59102 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: osdmap e867: 8 total, 8 up, 8 in 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3085432650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: osdmap e868: 8 total, 8 up, 8 in 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:53.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:53.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-03-30T15:13:53.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: pgmap v1340: 214 pgs: 1 active+clean+scrubbing, 213 active+clean; 4.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-30T15:13:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.59102 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:13:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: osdmap e867: 8 total, 8 up, 8 in 2024-03-30T15:13:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3085432650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:13:53.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: osdmap e868: 8 total, 8 up, 8 in 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-144"}]: dispatch 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:13:53.681 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:13:53.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:53.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:53.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:53.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:54.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:54.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:54.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:54.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:54.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:54.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:54.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:54.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:54.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:54.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:54.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:54.735 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:54 smithi028 ceph-mon[41614]: osdmap e869: 8 total, 8 up, 8 in 2024-03-30T15:13:54.736 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:54 smithi028 ceph-mon[41614]: pgmap v1344: 182 pgs: 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:13:54.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:54 smithi028 ceph-mon[29876]: osdmap e869: 8 total, 8 up, 8 in 2024-03-30T15:13:54.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:54 smithi028 ceph-mon[29876]: pgmap v1344: 182 pgs: 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:13:54.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:54.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:54 smithi134 ceph-mon[33714]: osdmap e869: 8 total, 8 up, 8 in 2024-03-30T15:13:54.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:54 smithi134 ceph-mon[33714]: pgmap v1344: 182 pgs: 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:13:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:54.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:55.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:55.058 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:55.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:55.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:13:55.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:55.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:55.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:55.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:55.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:55.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:55.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:55.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:55.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:55.834 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:55 smithi134 ceph-mon[33714]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T15:13:55.834 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:55 smithi134 ceph-mon[33714]: osdmap e870: 8 total, 8 up, 8 in 2024-03-30T15:13:55.834 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:55 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.834 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:55 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[29876]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T15:13:55.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[29876]: osdmap e870: 8 total, 8 up, 8 in 2024-03-30T15:13:55.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[41614]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-30T15:13:55.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[41614]: osdmap e870: 8 total, 8 up, 8 in 2024-03-30T15:13:55.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:55 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:13:55.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:55.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:55.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:56.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:56.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:13:56.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:56.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:56.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:56.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:56.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:56.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:56.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:56.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:56.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:56.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:56.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:56.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:56 smithi134 ceph-mon[33714]: osdmap e871: 8 total, 8 up, 8 in 2024-03-30T15:13:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:56 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:56 smithi134 ceph-mon[33714]: pgmap v1347: 214 pgs: 32 unknown, 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:13:56.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:56.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:56.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[29876]: osdmap e871: 8 total, 8 up, 8 in 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[29876]: pgmap v1347: 214 pgs: 32 unknown, 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[41614]: osdmap e871: 8 total, 8 up, 8 in 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:13:56.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:56 smithi028 ceph-mon[41614]: pgmap v1347: 214 pgs: 32 unknown, 2 activating+undersized+remapped, 4 peering, 176 active+clean; 4.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:13:57.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:57.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:57.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:57.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:57.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:57.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:57.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:13:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:57.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:13:57.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:57.658 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:13:57.658 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:13:57.667 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: osdmap e872: 8 total, 8 up, 8 in 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: osdmap e873: 8 total, 8 up, 8 in 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: osdmap e872: 8 total, 8 up, 8 in 2024-03-30T15:13:57.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:13:57.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: osdmap e873: 8 total, 8 up, 8 in 2024-03-30T15:13:57.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:57 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:57.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:57.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:57.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- 172.21.15.28:0/678302838 >> v1:172.21.15.28:6790/0 conn(0x7f155810ea80 legacy=0x7f1558110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- 172.21.15.28:0/678302838 shutdown_connections 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- 172.21.15.28:0/678302838 >> 172.21.15.28:0/678302838 conn(0x7f15580fdb20 msgr2=0x7f15580fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- 172.21.15.28:0/678302838 shutdown_connections 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- 172.21.15.28:0/678302838 wait complete. 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 librados: starting msgr at 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 librados: starting objecter 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 Processor -- start 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 -- start start 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 librados: setting wanted keys 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.831+0000 7f155e6dd700 1 librados: calling monclient init 2024-03-30T15:13:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f155e6dd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15581a9370 con 0x7f155810ea80 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f155e6dd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15581aa590 con 0x7f1558102250 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f155e6dd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f15581ab7d0 con 0x7f155810aee0 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15577fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f155810aee0 0x7f15581a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54748/0 (socket says 172.21.15.28:54748) 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15577fe700 1 -- 172.21.15.28:0/254939804 learned_addr learned my addr 172.21.15.28:0/254939804 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter ms_handle_connect 0x7f155810ea80 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter ms_handle_connect 0x7f155810aee0 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter ms_handle_connect 0x7f1558102250 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673355198 0 0) 0x7f15581ab7d0 con 0x7f155810aee0 2024-03-30T15:13:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15400039c0 con 0x7f155810aee0 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953611140 0 0) 0x7f15581a9370 con 0x7f155810ea80 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15581ab7d0 con 0x7f155810ea80 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649890329 0 0) 0x7f15581aa590 con 0x7f1558102250 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15581a9370 con 0x7f1558102250 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1314515072 0 0) 0x7f15400039c0 con 0x7f155810aee0 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15581aa590 con 0x7f155810aee0 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f154c001930 con 0x7f155810aee0 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2067246307 0 0) 0x7f15581ab7d0 con 0x7f155810ea80 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15400039c0 con 0x7f155810ea80 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 987495648 0 0) 0x7f15581a9370 con 0x7f1558102250 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15581ab7d0 con 0x7f1558102250 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.832+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1550001930 con 0x7f155810ea80 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1548001890 con 0x7f1558102250 2024-03-30T15:13:57.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3681784667 0 0) 0x7f15581aa590 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 >> v1:172.21.15.134:6789/0 conn(0x7f1558102250 legacy=0x7f15581075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 >> v1:172.21.15.28:6789/0 conn(0x7f155810ea80 legacy=0x7f15581a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15581aca10 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15581a95a0 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f155e6dd700 10 client.59114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f15581a9a60 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f155e6dd700 1 librados: init done 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f154c003b40 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.833+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f154c005d10 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.834+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1544005120 con 0x7f155810aee0 2024-03-30T15:13:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.834+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f154c01c5d0 con 0x7f155810aee0 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(873..873 src has 442..873) ==== 8801+0+0 (unknown 1139739995 0 0) 0x7f154c089840 con 0x7f155810aee0 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 10 client.59114.objecter ms_dispatch 0x7f1558001550 osd_map(873..873 src has 442..873) 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 3 client.59114.objecter handle_osd_map got epochs [873,873] > 0 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 3 client.59114.objecter handle_osd_map decoding full epoch 873 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 20 client.59114.objecter dump_active .. 0 homeless 2024-03-30T15:13:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.835+0000 7f15557fa700 10 client.59114.objecter ms_handle_connect 0x7f1540070440 2024-03-30T15:13:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:57.840+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f154c0567d0 con 0x7f155810aee0 2024-03-30T15:13:57.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:13:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: osdmap e872: 8 total, 8 up, 8 in 2024-03-30T15:13:57.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:13:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi028-93253-111", "overlaypool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:13:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: osdmap e873: 8 total, 8 up, 8 in 2024-03-30T15:13:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:57 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]: dispatch 2024-03-30T15:13:57.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:57.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:57.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.056+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1544005410 con 0x7f155810aee0 2024-03-30T15:13:58.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:58.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:13:58.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:58.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:58.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:58.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:58.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:58.523 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:13:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:13:58.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:58.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.588+0000 7f15557fa700 1 -- 172.21.15.28:0/254939804 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v874) ==== 97+0+0 (unknown 2964104177 0 0) 0x7f154c05a220 con 0x7f155810aee0 2024-03-30T15:13:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 10 librados: watch_flush enter 2024-03-30T15:13:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 10 librados: watch_flush exit 2024-03-30T15:13:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 20 client.59114.objecter shutdown clearing up homeless session... 2024-03-30T15:13:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 10 client.59114.objecter successfully canceled tick 2024-03-30T15:13:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1540070440 legacy=0x7f1540072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 >> v1:172.21.15.28:6790/0 conn(0x7f155810aee0 legacy=0x7f15581a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 shutdown_connections 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.592+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 >> 172.21.15.28:0/254939804 conn(0x7f15580fdb20 msgr2=0x7f1558111bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.593+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 shutdown_connections 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.593+0000 7f155e6dd700 1 -- 172.21.15.28:0/254939804 wait complete. 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:58.593+0000 7f155e6dd700 1 librados: shutdown 2024-03-30T15:13:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-03-30T15:13:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:58.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:13:58.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:13:58.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:13:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:58.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/254939804' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: from='client.59114 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: pgmap v1350: 214 pgs: 14 unknown, 2 activating+undersized+remapped, 198 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:58.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]': finished 2024-03-30T15:13:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: from='client.59114 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:58.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:58 smithi134 ceph-mon[33714]: osdmap e874: 8 total, 8 up, 8 in 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/254939804' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: from='client.59114 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: pgmap v1350: 214 pgs: 14 unknown, 2 activating+undersized+remapped, 198 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]': finished 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: from='client.59114 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:58.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[29876]: osdmap e874: 8 total, 8 up, 8 in 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/254939804' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: from='client.59114 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: pgmap v1350: 214 pgs: 14 unknown, 2 activating+undersized+remapped, 198 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi028-93253-146", "mode": "writeback"}]': finished 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: from='client.59114 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-30T15:13:58.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:58 smithi028 ceph-mon[41614]: osdmap e874: 8 total, 8 up, 8 in 2024-03-30T15:13:58.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:58.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:58.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.043+0000 7fea1ca8d700 1 -- 172.21.15.28:0/3524886224 >> v1:172.21.15.28:6789/0 conn(0x7fea1810ae80 legacy=0x7fea1810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.043+0000 7fea1ca8d700 1 -- 172.21.15.28:0/3524886224 shutdown_connections 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.043+0000 7fea1ca8d700 1 -- 172.21.15.28:0/3524886224 >> 172.21.15.28:0/3524886224 conn(0x7fea180761d0 msgr2=0x7fea180765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.043+0000 7fea1ca8d700 1 -- 172.21.15.28:0/3524886224 shutdown_connections 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.043+0000 7fea1ca8d700 1 -- 172.21.15.28:0/3524886224 wait complete. 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 librados: starting msgr at 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 librados: starting objecter 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 Processor -- start 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 -- start start 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 librados: setting wanted keys 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 librados: calling monclient init 2024-03-30T15:13:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fea18107960 con 0x7fea18108110 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fea181a8690 con 0x7fea1810ea80 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1ca8d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fea181a9870 con 0x7fea1810ae80 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1659c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fea18108110 0x7fea181047b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39660/0 (socket says 172.21.15.28:39660) 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea1659c700 1 -- 172.21.15.28:0/50083688 learned_addr learned my addr 172.21.15.28:0/50083688 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.044+0000 7fea15d9b700 1 --1- 172.21.15.28:0/50083688 >> v1:172.21.15.28:6790/0 conn(0x7fea1810ae80 0x7fea18104ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45904/0 (socket says 172.21.15.28:45904) 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter ms_handle_connect 0x7fea18108110 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter ms_handle_connect 0x7fea1810ae80 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter ms_handle_connect 0x7fea1810ea80 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343440097 0 0) 0x7fea18107960 con 0x7fea18108110 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea000035b0 con 0x7fea18108110 2024-03-30T15:13:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726700797 0 0) 0x7fea181a9870 con 0x7fea1810ae80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea18107960 con 0x7fea1810ae80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572715462 0 0) 0x7fea181a8690 con 0x7fea1810ea80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea181a9870 con 0x7fea1810ea80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1375036441 0 0) 0x7fea000035b0 con 0x7fea18108110 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea181a8690 con 0x7fea18108110 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 785523568 0 0) 0x7fea18107960 con 0x7fea1810ae80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea000035b0 con 0x7fea1810ae80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fea08001930 con 0x7fea18108110 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fea0c001430 con 0x7fea1810ae80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3940921184 0 0) 0x7fea181a9870 con 0x7fea1810ea80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea18107960 con 0x7fea1810ea80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fea10001b00 con 0x7fea1810ea80 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3484833495 0 0) 0x7fea181a8690 con 0x7fea18108110 2024-03-30T15:13:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 >> v1:172.21.15.28:6790/0 conn(0x7fea1810ae80 legacy=0x7fea18104ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 >> v1:172.21.15.134:6789/0 conn(0x7fea1810ea80 legacy=0x7fea181a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea181aaa50 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea181a9aa0 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea1ca8d700 10 client.25917.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fea181a9f60 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea1ca8d700 1 librados: init done 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fea08001cb0 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.045+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fea08005d30 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.046+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f8005120 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.047+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fea080040c0 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.047+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(874..874 src has 442..874) ==== 8801+0+0 (unknown 504650743 0 0) 0x7fea080899c0 con 0x7fea18108110 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.048+0000 7fea077fe700 10 client.25917.objecter ms_dispatch 0x7fea18001550 osd_map(874..874 src has 442..874) 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.048+0000 7fea077fe700 3 client.25917.objecter handle_osd_map got epochs [874,874] > 0 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.048+0000 7fea077fe700 3 client.25917.objecter handle_osd_map decoding full epoch 874 2024-03-30T15:13:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.048+0000 7fea077fe700 20 client.25917.objecter dump_active .. 0 homeless 2024-03-30T15:13:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.048+0000 7fea077fe700 10 client.25917.objecter ms_handle_connect 0x7fea00070440 2024-03-30T15:13:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.052+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fea08056930 con 0x7fea18108110 2024-03-30T15:13:59.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:13:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:59.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:13:59.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:13:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:59.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:13:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.274+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7fe9f8005410 con 0x7fea18108110 2024-03-30T15:13:59.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:59.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:59.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:59.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.609+0000 7fea077fe700 1 -- 172.21.15.28:0/50083688 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v875) ==== 113+0+0 (unknown 2642897530 0 0) 0x7fea0805a380 con 0x7fea18108110 2024-03-30T15:13:59.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.612+0000 7fea1ca8d700 10 librados: watch_flush enter 2024-03-30T15:13:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.613+0000 7fea1ca8d700 10 librados: watch_flush exit 2024-03-30T15:13:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.613+0000 7fea1ca8d700 20 client.25917.objecter shutdown clearing up homeless session... 2024-03-30T15:13:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.613+0000 7fea1ca8d700 10 client.25917.objecter successfully canceled tick 2024-03-30T15:13:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.617+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 >> v1:172.21.15.28:6800/3011431093 conn(0x7fea00070440 legacy=0x7fea00072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.617+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 >> v1:172.21.15.28:6789/0 conn(0x7fea18108110 legacy=0x7fea181047b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:13:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.617+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 shutdown_connections 2024-03-30T15:13:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.617+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 >> 172.21.15.28:0/50083688 conn(0x7fea180761d0 msgr2=0x7fea18110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:13:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.619+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 shutdown_connections 2024-03-30T15:13:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.619+0000 7fea1ca8d700 1 -- 172.21.15.28:0/50083688 wait complete. 2024-03-30T15:13:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:13:59.619+0000 7fea1ca8d700 1 librados: shutdown 2024-03-30T15:13:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T15:13:59.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:13:59.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:13:59.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:13:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:13:59.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:13:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:59 smithi134 ceph-mon[33714]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T15:13:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:59 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:13:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[41614]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[29876]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-30T15:13:59.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-30T15:13:59.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:13:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:13:59.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:13:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:13:59.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:13:59.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:00.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:00.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:00.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:00.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:00.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:14:00.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:00.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:00.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:00.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: osdmap e875: 8 total, 8 up, 8 in 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: pgmap v1353: 214 pgs: 214 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:14:00.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:14:00.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: osdmap e876: 8 total, 8 up, 8 in 2024-03-30T15:14:00.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:00.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:00.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:00.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:00.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:00.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:00.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:00.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:00.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: osdmap e875: 8 total, 8 up, 8 in 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: pgmap v1353: 214 pgs: 214 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: osdmap e876: 8 total, 8 up, 8 in 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:00.921 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_count","val": "2"}]': finished 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/50083688' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: osdmap e875: 8 total, 8 up, 8 in 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: pgmap v1353: 214 pgs: 214 active+clean; 4.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s; 54 B/s, 4 objects/s recovering 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_period","val": "600"}]': finished 2024-03-30T15:14:00.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: osdmap e876: 8 total, 8 up, 8 in 2024-03-30T15:14:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:00 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-30T15:14:01.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:01.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-03-30T15:14:01.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:01.082 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:01.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:01.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:01.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:01.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:01.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:01.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:01.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:01.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:14:01.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:01.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:01.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:01.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:01.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:01.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:01.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:14:01.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:14:01.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: osdmap e877: 8 total, 8 up, 8 in 2024-03-30T15:14:01.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: osdmap e877: 8 total, 8 up, 8 in 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.838 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:01 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-30T15:14:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-30T15:14:01.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: osdmap e877: 8 total, 8 up, 8 in 2024-03-30T15:14:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:01 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-30T15:14:01.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:01.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:02.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:02.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:02.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:02.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:02.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:02.321 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:02.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:02.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:02.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:02.576 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:02.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:02.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:02.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:02.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[41614]: pgmap v1356: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 3 op/s; 68 B/s, 4 objects/s recovering 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[41614]: osdmap e878: 8 total, 8 up, 8 in 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[29876]: pgmap v1356: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 3 op/s; 68 B/s, 4 objects/s recovering 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[29876]: osdmap e878: 8 total, 8 up, 8 in 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:02 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:02 smithi134 ceph-mon[33714]: pgmap v1356: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 3 op/s; 68 B/s, 4 objects/s recovering 2024-03-30T15:14:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:02 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-30T15:14:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:02 smithi134 ceph-mon[33714]: osdmap e878: 8 total, 8 up, 8 in 2024-03-30T15:14:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:02 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-30T15:14:02.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:02.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:02.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:03.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:03.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:03.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:03.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:03.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:03.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:03.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:03.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:03.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:03.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:03.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:03.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:03.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:03.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:03.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:03.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:03.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:14:03.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[41614]: osdmap e879: 8 total, 8 up, 8 in 2024-03-30T15:14:03.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:03.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:03.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:14:03.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:03 smithi028 ceph-mon[29876]: osdmap e879: 8 total, 8 up, 8 in 2024-03-30T15:14:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:03 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:03 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:03 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-146","var": "target_max_objects","val": "1"}]': finished 2024-03-30T15:14:03.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:03 smithi134 ceph-mon[33714]: osdmap e879: 8 total, 8 up, 8 in 2024-03-30T15:14:03.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:03.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:04.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:04.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:04.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:04.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:04.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:04.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:04.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- 172.21.15.28:0/3390196815 >> v1:172.21.15.28:6789/0 conn(0x7f544010aec0 legacy=0x7f544010d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- 172.21.15.28:0/3390196815 shutdown_connections 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- 172.21.15.28:0/3390196815 >> 172.21.15.28:0/3390196815 conn(0x7f54400fdb00 msgr2=0x7f54400fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- 172.21.15.28:0/3390196815 shutdown_connections 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- 172.21.15.28:0/3390196815 wait complete. 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 librados: starting msgr at 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 librados: starting objecter 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 Processor -- start 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 -- start start 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 librados: setting wanted keys 2024-03-30T15:14:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.465+0000 7f54463d6700 1 librados: calling monclient init 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f54463d6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54401a70f0 con 0x7f544010aec0 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f54463d6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54401a8310 con 0x7f544010ea60 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f54463d6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f54401a9550 con 0x7f5440102230 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5440102230 0x7f5440107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45912/0 (socket says 172.21.15.28:45912) 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543ffff700 1 -- 172.21.15.28:0/901033409 learned_addr learned my addr 172.21.15.28:0/901033409 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543f7fe700 1 --1- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6789/0 conn(0x7f544010aec0 0x7f54401a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39688/0 (socket says 172.21.15.28:39688) 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter ms_handle_connect 0x7f5440102230 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter ms_handle_connect 0x7f544010aec0 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter ms_handle_connect 0x7f544010ea60 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926648484 0 0) 0x7f54401a9550 con 0x7f5440102230 2024-03-30T15:14:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54280031f0 con 0x7f5440102230 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345587690 0 0) 0x7f54401a70f0 con 0x7f544010aec0 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54401a9550 con 0x7f544010aec0 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047981825 0 0) 0x7f54401a8310 con 0x7f544010ea60 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54401a70f0 con 0x7f544010ea60 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.466+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 891053600 0 0) 0x7f54280031f0 con 0x7f5440102230 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54401a8310 con 0x7f5440102230 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5430001a00 con 0x7f5440102230 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3683891744 0 0) 0x7f54401a9550 con 0x7f544010aec0 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54280031f0 con 0x7f544010aec0 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2411502270 0 0) 0x7f54401a70f0 con 0x7f544010ea60 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54401a9550 con 0x7f544010ea60 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5434001510 con 0x7f544010aec0 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5438001bb0 con 0x7f544010ea60 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3187109695 0 0) 0x7f54401a8310 con 0x7f5440102230 2024-03-30T15:14:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.134:6789/0 conn(0x7f544010ea60 legacy=0x7f54401a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6789/0 conn(0x7f544010aec0 legacy=0x7f54401a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54401aa790 con 0x7f5440102230 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54401a7320 con 0x7f5440102230 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f54463d6700 10 client.59126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f54401a77e0 con 0x7f5440102230 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f54463d6700 1 librados: init done 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5430003dd0 con 0x7f5440102230 2024-03-30T15:14:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.467+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5430005df0 con 0x7f5440102230 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.468+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f542c005100 con 0x7f5440102230 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5430003930 con 0x7f5440102230 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(879..879 src has 442..879) ==== 8827+0+0 (unknown 429327278 0 0) 0x7f5430089a70 con 0x7f5440102230 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 osd_map(879..879 src has 442..879) 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 3 client.59126.objecter handle_osd_map got epochs [879,879] > 0 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 3 client.59126.objecter handle_osd_map decoding full epoch 879 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 20 client.59126.objecter dump_active .. 0 homeless 2024-03-30T15:14:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v879) ==== 54+0+16 (unknown 2886036469 0 2420000192) 0x7f5430089eb0 con 0x7f5440102230 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.469+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f5428070390 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 10 client.59126.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 1 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 1 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 0 1 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 4 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 10 client.59126.objecter _send_command 1 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f542c004b50 con 0x7f542c004730 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c004730 2024-03-30T15:14:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c004b50 con 0x7f542c004730 2024-03-30T15:14:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 1: 0 ) 2024-03-30T15:14:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 10 client.59126.objecter _finish_command 1 = system:0 2024-03-30T15:14:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 4 2024-03-30T15:14:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.470+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 0 1 2024-03-30T15:14:04.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 10 client.59126.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 2 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 2 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0053e0 osd=0 3 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 0 2 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 4 2024-03-30T15:14:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 10 client.59126.objecter _send_command 2 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.488+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c007b00 con 0x7f542c004730 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.489+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c007b00 con 0x7f542c004730 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.489+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.489+0000 7f543d7fa700 10 client.59126.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.489+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 4 2024-03-30T15:14:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.489+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 0 2 2024-03-30T15:14:04.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 10 client.59126.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 1 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 4 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 10 client.59126.objecter _send_command 3 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f542c002160 con 0x7f542c0042a0 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.490+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c0042a0 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.491+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c002160 con 0x7f542c0042a0 2024-03-30T15:14:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.491+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 3: 0 ) 2024-03-30T15:14:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.491+0000 7f543d7fa700 10 client.59126.objecter _finish_command 3 = system:0 2024-03-30T15:14:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.491+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 4 2024-03-30T15:14:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.491+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 1 3 2024-03-30T15:14:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 10 client.59126.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 4 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 4 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c004130 osd=1 3 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 1 4 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 4 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 10 client.59126.objecter _send_command 4 2024-03-30T15:14:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.508+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c001600 con 0x7f542c0042a0 2024-03-30T15:14:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c001600 con 0x7f542c0042a0 2024-03-30T15:14:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 10 client.59126.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 4 2024-03-30T15:14:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 1 4 2024-03-30T15:14:04.512 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 10 client.59126.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 5 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 5 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 2 5 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 4 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 10 client.59126.objecter _send_command 5 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f542c008c30 con 0x7f542c011af0 2024-03-30T15:14:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.510+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c011af0 2024-03-30T15:14:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.511+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c008c30 con 0x7f542c011af0 2024-03-30T15:14:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.511+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 5: 0 ) 2024-03-30T15:14:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.511+0000 7f543d7fa700 10 client.59126.objecter _finish_command 5 = system:0 2024-03-30T15:14:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.511+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 4 2024-03-30T15:14:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.511+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 2 5 2024-03-30T15:14:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 10 client.59126.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 6 2024-03-30T15:14:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 6 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c008480 osd=2 3 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 2 6 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 4 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 10 client.59126.objecter _send_command 6 2024-03-30T15:14:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.528+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c007fc0 con 0x7f542c011af0 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.529+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c007fc0 con 0x7f542c011af0 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.529+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.529+0000 7f543d7fa700 10 client.59126.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.529+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 4 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.529+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 2 6 2024-03-30T15:14:04.531 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:14:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 10 client.59126.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 7 2024-03-30T15:14:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 7 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 3 7 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 4 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 10 client.59126.objecter _send_command 7 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f542c016170 con 0x7f542c015600 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.530+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c015600 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.531+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c016170 con 0x7f542c015600 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.531+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 7: 0 ) 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.531+0000 7f543d7fa700 10 client.59126.objecter _finish_command 7 = system:0 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.531+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 4 2024-03-30T15:14:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.531+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 3 7 2024-03-30T15:14:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 10 client.59126.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 8 2024-03-30T15:14:04.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 8 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c008250 osd=3 3 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 3 8 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 4 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 10 client.59126.objecter _send_command 8 2024-03-30T15:14:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.548+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c017ca0 con 0x7f542c015600 2024-03-30T15:14:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.549+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c017ca0 con 0x7f542c015600 2024-03-30T15:14:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.549+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.549+0000 7f543d7fa700 10 client.59126.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.551 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.549+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 4 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.549+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 3 8 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 10 client.59126.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 9 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 9 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 4 9 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 4 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 10 client.59126.objecter _send_command 9 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f542c01ebc0 con 0x7f542c01afc0 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.550+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c01afc0 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.551+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c01ebc0 con 0x7f542c01afc0 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.551+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 9: 0 ) 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.551+0000 7f543d7fa700 10 client.59126.objecter _finish_command 9 = system:0 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.551+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 4 2024-03-30T15:14:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.551+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 4 9 2024-03-30T15:14:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 10 client.59126.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 10 2024-03-30T15:14:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 10 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c017ca0 osd=4 3 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 4 10 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 4 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 10 client.59126.objecter _send_command 10 2024-03-30T15:14:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.568+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c020f90 con 0x7f542c01afc0 2024-03-30T15:14:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c020f90 con 0x7f542c01afc0 2024-03-30T15:14:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f543d7fa700 10 client.59126.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 4 2024-03-30T15:14:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 4 10 2024-03-30T15:14:04.572 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:14:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 10 client.59126.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 11 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 11 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 5 11 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 4 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 10 client.59126.objecter _send_command 11 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.570+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f542c0202e0 con 0x7f542c020d40 2024-03-30T15:14:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.571+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c020d40 2024-03-30T15:14:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.572+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c0202e0 con 0x7f542c020d40 2024-03-30T15:14:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.572+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 11: 0 ) 2024-03-30T15:14:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.572+0000 7f543d7fa700 10 client.59126.objecter _finish_command 11 = system:0 2024-03-30T15:14:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.572+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 4 2024-03-30T15:14:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.572+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 5 11 2024-03-30T15:14:04.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 10 client.59126.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 12 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 12 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0213f0 osd=5 3 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 5 12 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 4 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 10 client.59126.objecter _send_command 12 2024-03-30T15:14:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.589+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c0202e0 con 0x7f542c020d40 2024-03-30T15:14:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c0202e0 con 0x7f542c020d40 2024-03-30T15:14:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f543d7fa700 10 client.59126.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 4 2024-03-30T15:14:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 5 12 2024-03-30T15:14:04.593 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:14:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 10 client.59126.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 13 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 13 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 6 13 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 4 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 10 client.59126.objecter _send_command 13 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.591+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f542c017ab0 con 0x7f542c023c80 2024-03-30T15:14:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.592+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c023c80 2024-03-30T15:14:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.593+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c017ab0 con 0x7f542c023c80 2024-03-30T15:14:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.593+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 13: 0 ) 2024-03-30T15:14:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.593+0000 7f543d7fa700 10 client.59126.objecter _finish_command 13 = system:0 2024-03-30T15:14:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.593+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 4 2024-03-30T15:14:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.593+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 6 13 2024-03-30T15:14:04.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 10 client.59126.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 14 2024-03-30T15:14:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 14 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c0202e0 osd=6 3 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 6 14 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 4 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 10 client.59126.objecter _send_command 14 2024-03-30T15:14:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.610+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c028f90 con 0x7f542c023c80 2024-03-30T15:14:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c028f90 con 0x7f542c023c80 2024-03-30T15:14:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f543d7fa700 10 client.59126.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 4 2024-03-30T15:14:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 6 14 2024-03-30T15:14:04.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:14:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 10 client.59126.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 15 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 15 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 7 15 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 4 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 10 client.59126.objecter _send_command 15 2024-03-30T15:14:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.612+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f542c029af0 con 0x7f542c0296b0 2024-03-30T15:14:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.613+0000 7f543d7fa700 10 client.59126.objecter ms_handle_connect 0x7f542c0296b0 2024-03-30T15:14:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.614+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f542c029af0 con 0x7f542c0296b0 2024-03-30T15:14:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.614+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 15: 0 ) 2024-03-30T15:14:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.614+0000 7f543d7fa700 10 client.59126.objecter _finish_command 15 = system:0 2024-03-30T15:14:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.614+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 4 2024-03-30T15:14:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.614+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 7 15 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 10 client.59126.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign -1 16 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 20 client.59126.objecter _get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 15 client.59126.objecter _session_command_op_remove -1 16 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 20 client.59126.objecter get_session s=0x7f542c028f90 osd=7 3 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 15 client.59126.objecter _session_command_op_assign 7 16 2024-03-30T15:14:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 4 2024-03-30T15:14:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 10 client.59126.objecter _send_command 16 2024-03-30T15:14:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.631+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f542c029af0 con 0x7f542c0296b0 2024-03-30T15:14:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.632+0000 7f543d7fa700 1 -- 172.21.15.28:0/901033409 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f542c029af0 con 0x7f542c0296b0 2024-03-30T15:14:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.632+0000 7f543d7fa700 10 client.59126.objecter ms_dispatch 0x7f5440001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.632+0000 7f543d7fa700 10 client.59126.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.632+0000 7f543d7fa700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 4 2024-03-30T15:14:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.632+0000 7f543d7fa700 15 client.59126.objecter _session_command_op_remove 7 16 2024-03-30T15:14:04.634 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 librados: watch_flush enter 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 librados: watch_flush exit 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.0 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6809/2927075629 conn(0x7f542c004730 legacy=0x7f542c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0053e0 osd=0 1 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.1 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6805/1850313547 conn(0x7f542c0042a0 legacy=0x7f542c00ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c004130 osd=1 1 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.2 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6801/2294873818 conn(0x7f542c011af0 legacy=0x7f542c013ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008480 osd=2 1 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.3 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6813/1506157250 conn(0x7f542c015600 legacy=0x7f542c015a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c008250 osd=3 1 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.4 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.134:6800/4077237787 conn(0x7f542c01afc0 legacy=0x7f542c01d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c017ca0 osd=4 1 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.5 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.134:6804/3668151179 conn(0x7f542c020d40 legacy=0x7f542c01ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0213f0 osd=5 1 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.6 2024-03-30T15:14:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.134:6808/2135224194 conn(0x7f542c023c80 legacy=0x7f542c026070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c0202e0 osd=6 1 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter close_session for osd.7 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.134:6812/3758190298 conn(0x7f542c0296b0 legacy=0x7f542c0277c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter put_session s=0x7f542c028f90 osd=7 1 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 20 client.59126.objecter shutdown clearing up homeless session... 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 10 client.59126.objecter successfully canceled tick 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f5444973700 1 -- 172.21.15.28:0/901033409 reap_dead start 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f5444973700 1 -- 172.21.15.28:0/901033409 reap_dead start 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f5444973700 1 -- 172.21.15.28:0/901033409 reap_dead start 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f5444973700 1 -- 172.21.15.28:0/901033409 reap_dead start 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f5444973700 1 -- 172.21.15.28:0/901033409 reap_dead start 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5428070390 legacy=0x7f5428072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.633+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> v1:172.21.15.28:6790/0 conn(0x7f5440102230 legacy=0x7f5440107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.634+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 shutdown_connections 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.634+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 >> 172.21.15.28:0/901033409 conn(0x7f54400fdb00 msgr2=0x7f5440111bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.634+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 shutdown_connections 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.634+0000 7f54463d6700 1 -- 172.21.15.28:0/901033409 wait complete. 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:04.634+0000 7f54463d6700 1 librados: shutdown 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:14:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:14:04.706 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:14:04.706 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:14:04.707 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:14:04.707 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:14:04.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:04 smithi028 ceph-mon[41614]: pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 123 B/s, 6 objects/s recovering 2024-03-30T15:14:04.707 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/901033409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:04.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:04 smithi028 ceph-mon[29876]: pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 123 B/s, 6 objects/s recovering 2024-03-30T15:14:04.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/901033409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:04.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:04.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:04.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:04 smithi134 ceph-mon[33714]: pgmap v1359: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 123 B/s, 6 objects/s recovering 2024-03-30T15:14:04.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/901033409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:04.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:04.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:05.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:05.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:05.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:05.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:05.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:05.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:05.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:05.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:05.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:05.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:05.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:05.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:05.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:05.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:05 smithi134 ceph-mon[33714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:14:05.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:05 smithi028 ceph-mon[41614]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:14:05.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:05 smithi028 ceph-mon[29876]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-30T15:14:05.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:06.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:06.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:06.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:06.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:06.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:06.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:06.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:06.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:06.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:06.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:06.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:06.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:06.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:14:06.792 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:06.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:06.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:06 smithi134 ceph-mon[33714]: pgmap v1360: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 88 B/s, 4 objects/s recovering 2024-03-30T15:14:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:06 smithi028 ceph-mon[29876]: pgmap v1360: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 88 B/s, 4 objects/s recovering 2024-03-30T15:14:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:06 smithi028 ceph-mon[41614]: pgmap v1360: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 88 B/s, 4 objects/s recovering 2024-03-30T15:14:07.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:07.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:07.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:07.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:07.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:07.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:07.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:07.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.790+0000 7f5d3e11a700 1 -- 172.21.15.28:0/2532654121 >> v1:172.21.15.28:6790/0 conn(0x7f5d38109df0 legacy=0x7f5d3810c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.790+0000 7f5d3e11a700 1 -- 172.21.15.28:0/2532654121 shutdown_connections 2024-03-30T15:14:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.790+0000 7f5d3e11a700 1 -- 172.21.15.28:0/2532654121 >> 172.21.15.28:0/2532654121 conn(0x7f5d380fca10 msgr2=0x7f5d380fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.790+0000 7f5d3e11a700 1 -- 172.21.15.28:0/2532654121 shutdown_connections 2024-03-30T15:14:07.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.790+0000 7f5d3e11a700 1 -- 172.21.15.28:0/2532654121 wait complete. 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 librados: starting msgr at 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 librados: starting objecter 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 Processor -- start 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 -- start start 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 librados: setting wanted keys 2024-03-30T15:14:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 librados: calling monclient init 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d381a7140 con 0x7f5d38109df0 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d381a8360 con 0x7f5d3810d990 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d3e11a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5d381a95a0 con 0x7f5d38101160 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d377fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5d38101160 0x7f5d38106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:45928/0 (socket says 172.21.15.28:45928) 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.791+0000 7f5d377fe700 1 -- 172.21.15.28:0/3426466992 learned_addr learned my addr 172.21.15.28:0/3426466992 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter ms_handle_connect 0x7f5d38101160 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter ms_handle_connect 0x7f5d38109df0 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter ms_handle_connect 0x7f5d3810d990 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364806894 0 0) 0x7f5d381a7140 con 0x7f5d38109df0 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d200031f0 con 0x7f5d38109df0 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298772163 0 0) 0x7f5d381a95a0 con 0x7f5d38101160 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d381a7140 con 0x7f5d38101160 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646111451 0 0) 0x7f5d381a8360 con 0x7f5d3810d990 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d381a95a0 con 0x7f5d3810d990 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3704590820 0 0) 0x7f5d200031f0 con 0x7f5d38109df0 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d381a8360 con 0x7f5d38109df0 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d2c001370 con 0x7f5d38109df0 2024-03-30T15:14:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3612562962 0 0) 0x7f5d381a7140 con 0x7f5d38101160 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d200031f0 con 0x7f5d38101160 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d28001a10 con 0x7f5d38101160 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2235993560 0 0) 0x7f5d381a95a0 con 0x7f5d3810d990 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d381a7140 con 0x7f5d3810d990 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d30001c30 con 0x7f5d3810d990 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2683339351 0 0) 0x7f5d381a8360 con 0x7f5d38109df0 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 >> v1:172.21.15.28:6790/0 conn(0x7f5d38101160 legacy=0x7f5d38106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 >> v1:172.21.15.134:6789/0 conn(0x7f5d3810d990 legacy=0x7f5d381a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d381aa7e0 con 0x7f5d38109df0 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d381a7370 con 0x7f5d38109df0 2024-03-30T15:14:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d3e11a700 10 client.25929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d381a7830 con 0x7f5d38109df0 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.792+0000 7f5d3e11a700 1 librados: init done 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.793+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5d2c001af0 con 0x7f5d38109df0 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.793+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5d2c004c40 con 0x7f5d38109df0 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.793+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d24005120 con 0x7f5d38109df0 2024-03-30T15:14:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.794+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5d2c003860 con 0x7f5d38109df0 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.794+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(879..879 src has 442..879) ==== 8827+0+0 (unknown 429327278 0 0) 0x7f5d2c088800 con 0x7f5d38109df0 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.794+0000 7f5d34ff9700 10 client.25929.objecter ms_dispatch 0x7f5d38001550 osd_map(879..879 src has 442..879) 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.794+0000 7f5d34ff9700 3 client.25929.objecter handle_osd_map got epochs [879,879] > 0 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.794+0000 7f5d34ff9700 3 client.25929.objecter handle_osd_map decoding full epoch 879 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.795+0000 7f5d34ff9700 20 client.25929.objecter dump_active .. 0 homeless 2024-03-30T15:14:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.795+0000 7f5d34ff9700 10 client.25929.objecter ms_handle_connect 0x7f5d20070320 2024-03-30T15:14:07.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:07.799+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5d2c004c40 con 0x7f5d38109df0 2024-03-30T15:14:07.809 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:07.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:07.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.020+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f5d24005410 con 0x7f5d38109df0 2024-03-30T15:14:08.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:08.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:08.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:08.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:08.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:08.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:08.392 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:08.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:08.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:08.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:08.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:08.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:08.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.699+0000 7f5d34ff9700 1 -- 172.21.15.28:0/3426466992 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v880) ==== 113+0+0 (unknown 2058930009 0 0) 0x7f5d2c055770 con 0x7f5d38109df0 2024-03-30T15:14:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 10 librados: watch_flush enter 2024-03-30T15:14:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 10 librados: watch_flush exit 2024-03-30T15:14:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 20 client.25929.objecter shutdown clearing up homeless session... 2024-03-30T15:14:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 10 client.25929.objecter successfully canceled tick 2024-03-30T15:14:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 >> v1:172.21.15.28:6800/3011431093 conn(0x7f5d20070320 legacy=0x7f5d200727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.703+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 >> v1:172.21.15.28:6789/0 conn(0x7f5d38109df0 legacy=0x7f5d381a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.704+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 shutdown_connections 2024-03-30T15:14:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.704+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 >> 172.21.15.28:0/3426466992 conn(0x7f5d380fca10 msgr2=0x7f5d3806ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:08.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.707+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 shutdown_connections 2024-03-30T15:14:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.707+0000 7f5d3e11a700 1 -- 172.21.15.28:0/3426466992 wait complete. 2024-03-30T15:14:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:08.707+0000 7f5d3e11a700 1 librados: shutdown 2024-03-30T15:14:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T15:14:08.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:08.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:08.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:08.965 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:08.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[41614]: pgmap v1361: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 B/s wr, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:14:08.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:08 smithi028 ceph-mon[29876]: pgmap v1361: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 B/s wr, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-30T15:14:09.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:09.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:08 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:14:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:08 smithi134 ceph-mon[33714]: pgmap v1361: 214 pgs: 214 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 B/s wr, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-30T15:14:09.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:14:09.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:09.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:09.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:09.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:09.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:09.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:09.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[41614]: osdmap e880: 8 total, 8 up, 8 in 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[29876]: osdmap e880: 8 total, 8 up, 8 in 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:09.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:09 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:09.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:10.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:10.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:10.140 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:09 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3426466992' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:14:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:09 smithi134 ceph-mon[33714]: osdmap e880: 8 total, 8 up, 8 in 2024-03-30T15:14:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:09 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:09 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:10.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:10.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:10.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.396 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:10.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:10.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:10.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:10.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 -- 172.21.15.28:0/2186380866 >> v1:172.21.15.28:6790/0 conn(0x7f43a010ea90 legacy=0x7f43a0110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 -- 172.21.15.28:0/2186380866 shutdown_connections 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 -- 172.21.15.28:0/2186380866 >> 172.21.15.28:0/2186380866 conn(0x7f43a00fdb30 msgr2=0x7f43a00fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 -- 172.21.15.28:0/2186380866 shutdown_connections 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 -- 172.21.15.28:0/2186380866 wait complete. 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 librados: starting msgr at 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 librados: starting objecter 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.655+0000 7f43a820f700 1 Processor -- start 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 -- start start 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 librados: setting wanted keys 2024-03-30T15:14:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 librados: calling monclient init 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43a0107760 con 0x7f43a010aef0 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43a0107970 con 0x7f43a0102260 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a820f700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43a0107b80 con 0x7f43a010ea90 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a67ac700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f43a010ea90 0x7f43a01a8320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38420/0 (socket says 172.21.15.28:38420) 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a57aa700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f43a010aef0 0x7f43a0074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34610/0 (socket says 172.21.15.28:34610) 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f43a67ac700 1 -- 172.21.15.28:0/2534622290 learned_addr learned my addr 172.21.15.28:0/2534622290 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter ms_handle_connect 0x7f43a010ea90 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter ms_handle_connect 0x7f43a010aef0 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter ms_handle_connect 0x7f43a0102260 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067099953 0 0) 0x7f43a0107760 con 0x7f43a010aef0 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43880031f0 con 0x7f43a010aef0 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 501787770 0 0) 0x7f43a0107b80 con 0x7f43a010ea90 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43a0107760 con 0x7f43a010ea90 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327060105 0 0) 0x7f43a0107970 con 0x7f43a0102260 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43a0107b80 con 0x7f43a0102260 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1488213945 0 0) 0x7f43880031f0 con 0x7f43a010aef0 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43a0107970 con 0x7f43a010aef0 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4394001830 con 0x7f43a010aef0 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3625259429 0 0) 0x7f43a0107760 con 0x7f43a010ea90 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.656+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43880031f0 con 0x7f43a010ea90 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2441574315 0 0) 0x7f43a0107b80 con 0x7f43a0102260 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43a0107760 con 0x7f43a0102260 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4390001d60 con 0x7f43a010ea90 2024-03-30T15:14:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4398001760 con 0x7f43a0102260 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 626984045 0 0) 0x7f43a0107970 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 >> v1:172.21.15.28:6790/0 conn(0x7f43a010ea90 legacy=0x7f43a01a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 >> v1:172.21.15.134:6789/0 conn(0x7f43a0102260 legacy=0x7f43a0074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a01acad0 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43a01a9aa0 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f43a820f700 10 client.25935.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f43a01a9f60 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f43a820f700 1 librados: init done 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4394003b40 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.657+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4394005c10 con 0x7f43a010aef0 2024-03-30T15:14:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.658+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f438c005120 con 0x7f43a010aef0 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.658+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f43940036a0 con 0x7f43a010aef0 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(881..881 src has 442..881) ==== 8856+0+0 (unknown 1118970032 0 0) 0x7f4394089890 con 0x7f43a010aef0 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 10 client.25935.objecter ms_dispatch 0x7f43a0001550 osd_map(881..881 src has 442..881) 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 3 client.25935.objecter handle_osd_map got epochs [881,881] > 0 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 3 client.25935.objecter handle_osd_map decoding full epoch 881 2024-03-30T15:14:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 20 client.25935.objecter dump_active .. 0 homeless 2024-03-30T15:14:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.659+0000 7f439effd700 10 client.25935.objecter ms_handle_connect 0x7f4388070500 2024-03-30T15:14:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.663+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f43940567f0 con 0x7f43a010aef0 2024-03-30T15:14:10.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:10.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:10.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:10.885+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f438c005410 con 0x7f43a010aef0 2024-03-30T15:14:10.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:10.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:10.930 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:10 smithi028 ceph-mon[41614]: osdmap e881: 8 total, 8 up, 8 in 2024-03-30T15:14:10.930 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:10 smithi028 ceph-mon[41614]: pgmap v1364: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s; 33 B/s, 1 objects/s recovering 2024-03-30T15:14:10.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:10 smithi028 ceph-mon[29876]: osdmap e881: 8 total, 8 up, 8 in 2024-03-30T15:14:10.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:10 smithi028 ceph-mon[29876]: pgmap v1364: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s; 33 B/s, 1 objects/s recovering 2024-03-30T15:14:11.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:11.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:11.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:11.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:10 smithi134 ceph-mon[33714]: osdmap e881: 8 total, 8 up, 8 in 2024-03-30T15:14:11.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:10 smithi134 ceph-mon[33714]: pgmap v1364: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s; 33 B/s, 1 objects/s recovering 2024-03-30T15:14:11.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:11.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:11.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:11.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-03-30T15:14:11.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:11.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:11.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:11.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:11.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.728+0000 7f439effd700 1 -- 172.21.15.28:0/2534622290 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v883) ==== 91+0+0 (unknown 1991939775 0 0) 0x7f439405a240 con 0x7f43a010aef0 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.731+0000 7f43a820f700 10 librados: watch_flush enter 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.731+0000 7f43a820f700 10 librados: watch_flush exit 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.731+0000 7f43a820f700 20 client.25935.objecter shutdown clearing up homeless session... 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.731+0000 7f43a820f700 10 client.25935.objecter successfully canceled tick 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.732+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4388070500 legacy=0x7f43880729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.732+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 >> v1:172.21.15.28:6789/0 conn(0x7f43a010aef0 legacy=0x7f43a0074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.732+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 shutdown_connections 2024-03-30T15:14:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.732+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 >> 172.21.15.28:0/2534622290 conn(0x7f43a00fdb30 msgr2=0x7f43a0111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.733+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 shutdown_connections 2024-03-30T15:14:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.733+0000 7f43a820f700 1 -- 172.21.15.28:0/2534622290 wait complete. 2024-03-30T15:14:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:11.734+0000 7f43a820f700 1 librados: shutdown 2024-03-30T15:14:11.736 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:14:11.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:11.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:11.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:11.960 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:11.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:12.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:11 smithi134 ceph-mon[33714]: osdmap e882: 8 total, 8 up, 8 in 2024-03-30T15:14:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:11 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:12.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:12.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[41614]: osdmap e882: 8 total, 8 up, 8 in 2024-03-30T15:14:12.212 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:12.213 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:12.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[29876]: osdmap e882: 8 total, 8 up, 8 in 2024-03-30T15:14:12.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:12.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:12.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:12.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:12.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:12.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:12.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:12.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:12.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:12.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:12.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:12.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: osdmap e883: 8 total, 8 up, 8 in 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: pgmap v1367: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:14:12.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: osdmap e883: 8 total, 8 up, 8 in 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: pgmap v1367: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:14:12.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:12 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:14:12.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:12.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:12.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:13.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:13.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:13.166 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2534622290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: osdmap e883: 8 total, 8 up, 8 in 2024-03-30T15:14:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:14:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:14:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: pgmap v1367: 214 pgs: 8 peering, 206 active+clean; 4.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-03-30T15:14:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:14:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:12 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:14:13.182 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:14:13.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:14:13.183 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-03-30T15:14:13.183 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-03-30T15:14:13.183 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:14:13.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:13.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:13.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:13.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:13.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:13.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:13.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:13.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:13.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:13.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:13.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:13.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[41614]: osdmap e884: 8 total, 8 up, 8 in 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[41614]: 1.8 scrub starts 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[41614]: 1.8 scrub ok 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[29876]: osdmap e884: 8 total, 8 up, 8 in 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[29876]: 1.8 scrub starts 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[29876]: 1.8 scrub ok 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:13.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:13 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:13.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:14:13.936 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:13.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:14.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:14.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:14.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:14.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:13 smithi134 ceph-mon[33714]: osdmap e884: 8 total, 8 up, 8 in 2024-03-30T15:14:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:13 smithi134 ceph-mon[33714]: 1.8 scrub starts 2024-03-30T15:14:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:13 smithi134 ceph-mon[33714]: 1.8 scrub ok 2024-03-30T15:14:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:13 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:13 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:14.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:14.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:14.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:14.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:14.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:14.523 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:14.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:14.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:14.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:14:14.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:14.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:14.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:14.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: 12.c scrub starts 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: 12.c scrub ok 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: osdmap e885: 8 total, 8 up, 8 in 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: pgmap v1370: 214 pgs: 214 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 66 B/s, 3 objects/s recovering 2024-03-30T15:14:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: 322.6s0 scrub starts 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[29876]: 322.6 scrub ok 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: 12.c scrub starts 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: 12.c scrub ok 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: osdmap e885: 8 total, 8 up, 8 in 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: pgmap v1370: 214 pgs: 214 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 66 B/s, 3 objects/s recovering 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: 322.6s0 scrub starts 2024-03-30T15:14:14.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:14 smithi028 ceph-mon[41614]: 322.6 scrub ok 2024-03-30T15:14:15.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:15.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:15.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:15.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: 12.c scrub starts 2024-03-30T15:14:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: 12.c scrub ok 2024-03-30T15:14:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: osdmap e885: 8 total, 8 up, 8 in 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: pgmap v1370: 214 pgs: 214 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 66 B/s, 3 objects/s recovering 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: 322.6s0 scrub starts 2024-03-30T15:14:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:14 smithi134 ceph-mon[33714]: 322.6 scrub ok 2024-03-30T15:14:15.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:15.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:15.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:15.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:15.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:15.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:15.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:15.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:15.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:15.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:15.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:16.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:16.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:16.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:16.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 12.e scrub starts 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 12.e scrub ok 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 4.e scrub starts 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 4.e scrub ok 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: osdmap e886: 8 total, 8 up, 8 in 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:14:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:15 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 12.e scrub starts 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 12.e scrub ok 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 4.e scrub starts 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 4.e scrub ok 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: osdmap e886: 8 total, 8 up, 8 in 2024-03-30T15:14:16.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 12.e scrub starts 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 12.e scrub ok 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 4.e scrub starts 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 4.e scrub ok 2024-03-30T15:14:16.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]': finished 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: osdmap e886: 8 total, 8 up, 8 in 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-146"}]: dispatch 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:14:16.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:15 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:14:16.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:16.300 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:16.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:16.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:16.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:16.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:16.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:16.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:16.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:16.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:16.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: osdmap e887: 8 total, 8 up, 8 in 2024-03-30T15:14:16.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[29876]: pgmap v1373: 182 pgs: 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 11 B/s, 0 objects/s recovering 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: osdmap e887: 8 total, 8 up, 8 in 2024-03-30T15:14:16.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:16 smithi028 ceph-mon[41614]: pgmap v1373: 182 pgs: 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 11 B/s, 0 objects/s recovering 2024-03-30T15:14:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:16.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:16.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:17.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:17.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:17.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:14:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:14:17.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:14:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:14:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: osdmap e887: 8 total, 8 up, 8 in 2024-03-30T15:14:17.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:16 smithi134 ceph-mon[33714]: pgmap v1373: 182 pgs: 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 11 B/s, 0 objects/s recovering 2024-03-30T15:14:17.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.250+0000 7f7187d22700 1 -- 172.21.15.28:0/1513709850 >> v1:172.21.15.28:6789/0 conn(0x7f7180103190 legacy=0x7f7180103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.250+0000 7f7187d22700 1 -- 172.21.15.28:0/1513709850 shutdown_connections 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.250+0000 7f7187d22700 1 -- 172.21.15.28:0/1513709850 >> 172.21.15.28:0/1513709850 conn(0x7f71800fc9e0 msgr2=0x7f71800fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.250+0000 7f7187d22700 1 -- 172.21.15.28:0/1513709850 shutdown_connections 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.250+0000 7f7187d22700 1 -- 172.21.15.28:0/1513709850 wait complete. 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 librados: starting msgr at 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 librados: starting objecter 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 Processor -- start 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 -- start start 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 librados: setting wanted keys 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 librados: calling monclient init 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71801a91a0 con 0x7f7180103190 2024-03-30T15:14:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71801aa3c0 con 0x7f718010afa0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7187d22700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f71801ab5e0 con 0x7f718010eab0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f71862bf700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f718010eab0 0x7f71801a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38442/0 (socket says 172.21.15.28:38442) 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f7185abe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7180103190 0x7f718010a570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34632/0 (socket says 172.21.15.28:34632) 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.251+0000 7f71862bf700 1 -- 172.21.15.28:0/3370148600 learned_addr learned my addr 172.21.15.28:0/3370148600 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter ms_handle_connect 0x7f718010eab0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter ms_handle_connect 0x7f7180103190 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter ms_handle_connect 0x7f718010afa0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465229415 0 0) 0x7f71801ab5e0 con 0x7f718010eab0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f716c003aa0 con 0x7f718010eab0 2024-03-30T15:14:17.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214762249 0 0) 0x7f71801a91a0 con 0x7f7180103190 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71801ab5e0 con 0x7f7180103190 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514525512 0 0) 0x7f71801aa3c0 con 0x7f718010afa0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71801a91a0 con 0x7f718010afa0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1540834756 0 0) 0x7f716c003aa0 con 0x7f718010eab0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71801aa3c0 con 0x7f718010eab0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7178001cb0 con 0x7f718010eab0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 68630287 0 0) 0x7f71801ab5e0 con 0x7f7180103190 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f716c003aa0 con 0x7f7180103190 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f717c0044e0 con 0x7f7180103190 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1823505673 0 0) 0x7f71801a91a0 con 0x7f718010afa0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71801ab5e0 con 0x7f718010afa0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7170001a30 con 0x7f718010afa0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3062651556 0 0) 0x7f71801aa3c0 con 0x7f718010eab0 2024-03-30T15:14:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 >> v1:172.21.15.134:6789/0 conn(0x7f718010afa0 legacy=0x7f71801a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 >> v1:172.21.15.28:6789/0 conn(0x7f7180103190 legacy=0x7f718010a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71801ac800 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.252+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71801a93d0 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.253+0000 7f7187d22700 10 client.59144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.253+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71801a9890 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.253+0000 7f7187d22700 1 librados: init done 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.253+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7178004170 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.253+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f71780061b0 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.254+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7164005120 con 0x7f718010eab0 2024-03-30T15:14:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.254+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f717801cad0 con 0x7f718010eab0 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(888..888 src has 442..888) ==== 8793+0+0 (unknown 2849724334 0 0) 0x7f7178089db0 con 0x7f718010eab0 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 10 client.59144.objecter ms_dispatch 0x7f7180001550 osd_map(888..888 src has 442..888) 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 3 client.59144.objecter handle_osd_map got epochs [888,888] > 0 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 3 client.59144.objecter handle_osd_map decoding full epoch 888 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 20 client.59144.objecter dump_active .. 0 homeless 2024-03-30T15:14:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.255+0000 7f7176ffd700 10 client.59144.objecter ms_handle_connect 0x7f716c070420 2024-03-30T15:14:17.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.259+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7178056d40 con 0x7f718010eab0 2024-03-30T15:14:17.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:17.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:17.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:17.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:17.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.486+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f7164005410 con 0x7f718010eab0 2024-03-30T15:14:17.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.588+0000 7f7176ffd700 1 -- 172.21.15.28:0/3370148600 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v889) ==== 113+0+0 (unknown 2951137956 0 0) 0x7f717805a790 con 0x7f718010eab0 2024-03-30T15:14:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 10 librados: watch_flush enter 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 10 librados: watch_flush exit 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 20 client.59144.objecter shutdown clearing up homeless session... 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 10 client.59144.objecter successfully canceled tick 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 >> v1:172.21.15.28:6800/3011431093 conn(0x7f716c070420 legacy=0x7f716c0728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 >> v1:172.21.15.28:6790/0 conn(0x7f718010eab0 legacy=0x7f71801a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 shutdown_connections 2024-03-30T15:14:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.591+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 >> 172.21.15.28:0/3370148600 conn(0x7f71800fc9e0 msgr2=0x7f71800fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.592+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 shutdown_connections 2024-03-30T15:14:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.592+0000 7f7187d22700 1 -- 172.21.15.28:0/3370148600 wait complete. 2024-03-30T15:14:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:17.592+0000 7f7187d22700 1 librados: shutdown 2024-03-30T15:14:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T15:14:17.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:17.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:17.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:17.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:17.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:17.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:18.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:18.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 12.10 scrub starts 2024-03-30T15:14:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 12.10 scrub ok 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: osdmap e888: 8 total, 8 up, 8 in 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3370148600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.59144 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.59144 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:14:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: osdmap e889: 8 total, 8 up, 8 in 2024-03-30T15:14:18.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:17 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:18.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 12.10 scrub starts 2024-03-30T15:14:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 12.10 scrub ok 2024-03-30T15:14:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:14:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:14:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: osdmap e888: 8 total, 8 up, 8 in 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3370148600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.59144 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.59144 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: osdmap e889: 8 total, 8 up, 8 in 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 12.10 scrub starts 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 12.10 scrub ok 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: osdmap e888: 8 total, 8 up, 8 in 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3370148600' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.59144 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:14:18.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:18.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.59144 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:14:18.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: osdmap e889: 8 total, 8 up, 8 in 2024-03-30T15:14:18.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:17 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-30T15:14:18.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:18.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 -- 172.21.15.28:0/4104193078 >> v1:172.21.15.28:6790/0 conn(0x7f72d8101130 legacy=0x7f72d8101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 -- 172.21.15.28:0/4104193078 shutdown_connections 2024-03-30T15:14:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 -- 172.21.15.28:0/4104193078 >> 172.21.15.28:0/4104193078 conn(0x7f72d80fc9e0 msgr2=0x7f72d80fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 -- 172.21.15.28:0/4104193078 shutdown_connections 2024-03-30T15:14:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 -- 172.21.15.28:0/4104193078 wait complete. 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.358+0000 7f72dd1a0700 1 librados: starting msgr at 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 librados: starting objecter 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 Processor -- start 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 -- start start 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 librados: setting wanted keys 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 librados: calling monclient init 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72d81a92d0 con 0x7f72d810d960 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72d81aa4f0 con 0x7f72d8101130 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72dd1a0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f72d81ab730 con 0x7f72d8109dc0 2024-03-30T15:14:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72d659c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f72d8109dc0 0x7f72d81a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38452/0 (socket says 172.21.15.28:38452) 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72d659c700 1 -- 172.21.15.28:0/1687920380 learned_addr learned my addr 172.21.15.28:0/1687920380 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72cffff700 10 client.?.objecter ms_handle_connect 0x7f72d8109dc0 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72cffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72cffff700 10 client.?.objecter ms_handle_connect 0x7f72d810d960 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.359+0000 7f72cffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 10 client.?.objecter ms_handle_connect 0x7f72d8101130 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550145175 0 0) 0x7f72d81ab730 con 0x7f72d8109dc0 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72c4003aa0 con 0x7f72d8109dc0 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484607175 0 0) 0x7f72d81a92d0 con 0x7f72d810d960 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d81ab730 con 0x7f72d810d960 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328673489 0 0) 0x7f72d81aa4f0 con 0x7f72d8101130 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d81a92d0 con 0x7f72d8101130 2024-03-30T15:14:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3572947478 0 0) 0x7f72c4003aa0 con 0x7f72d8109dc0 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72d81aa4f0 con 0x7f72d8109dc0 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4273626862 0 0) 0x7f72d81ab730 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72c4003aa0 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72c00013d0 con 0x7f72d8109dc0 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72d0001af0 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 623502911 0 0) 0x7f72d81a92d0 con 0x7f72d8101130 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72d81ab730 con 0x7f72d8101130 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72c8003820 con 0x7f72d8101130 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1912311337 0 0) 0x7f72c4003aa0 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6790/0 conn(0x7f72d8109dc0 legacy=0x7f72d81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.134:6789/0 conn(0x7f72d8101130 legacy=0x7f72d810f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d81ac970 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72d81a9500 con 0x7f72d810d960 2024-03-30T15:14:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72dd1a0700 10 client.25947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72d81a99c0 con 0x7f72d810d960 2024-03-30T15:14:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72dd1a0700 1 librados: init done 2024-03-30T15:14:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f72d0003ea0 con 0x7f72d810d960 2024-03-30T15:14:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.360+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f72d0004cb0 con 0x7f72d810d960 2024-03-30T15:14:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.361+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f72b8005100 con 0x7f72d810d960 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.362+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f72d0003a00 con 0x7f72d810d960 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.362+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(889..889 src has 442..889) ==== 8806+0+0 (unknown 1381385257 0 0) 0x7f72d0088750 con 0x7f72d810d960 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.362+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 osd_map(889..889 src has 442..889) 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.362+0000 7f72cffff700 3 client.25947.objecter handle_osd_map got epochs [889,889] > 0 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.362+0000 7f72cffff700 3 client.25947.objecter handle_osd_map decoding full epoch 889 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72cffff700 20 client.25947.objecter dump_active .. 0 homeless 2024-03-30T15:14:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v889) ==== 54+0+16 (unknown 1018410643 0 2420000192) 0x7f72d0088b80 con 0x7f72d810d960 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72c4070580 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 1 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 1 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 0 1 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 4 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 10 client.25947.objecter _send_command 1 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72b8004b50 con 0x7f72b8004730 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.363+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8004730 2024-03-30T15:14:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.364+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b8004b50 con 0x7f72b8004730 2024-03-30T15:14:18.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.364+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 1: 0 ) 2024-03-30T15:14:18.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.364+0000 7f72cffff700 10 client.25947.objecter _finish_command 1 = system:0 2024-03-30T15:14:18.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.364+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 4 2024-03-30T15:14:18.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.364+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 0 1 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 2 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 2 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80053e0 osd=0 3 2024-03-30T15:14:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 0 2 2024-03-30T15:14:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 4 2024-03-30T15:14:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 10 client.25947.objecter _send_command 2 2024-03-30T15:14:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.386+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b80079f0 con 0x7f72b8004730 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b80079f0 con 0x7f72b8004730 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72cffff700 10 client.25947.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 4 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 0 2 2024-03-30T15:14:18.389 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.387+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 3 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 3 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 1 3 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 4 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 10 client.25947.objecter _send_command 3 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f72b8002180 con 0x7f72b8004280 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8004280 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b8002180 con 0x7f72b8004280 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 3: 0 ) 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 10 client.25947.objecter _finish_command 3 = system:0 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 4 2024-03-30T15:14:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.388+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 1 3 2024-03-30T15:14:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 4 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 4 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8004110 osd=1 3 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 1 4 2024-03-30T15:14:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 4 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 10 client.25947.objecter _send_command 4 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.406+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b8001600 con 0x7f72b8004280 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b8001600 con 0x7f72b8004280 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72cffff700 10 client.25947.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 4 2024-03-30T15:14:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 1 4 2024-03-30T15:14:18.410 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:14:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 5 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 5 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 2 5 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 4 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 10 client.25947.objecter _send_command 5 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.407+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f72b80110e0 con 0x7f72b800d5d0 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b800d5d0 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b80110e0 con 0x7f72b800d5d0 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 5: 0 ) 2024-03-30T15:14:18.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 10 client.25947.objecter _finish_command 5 = system:0 2024-03-30T15:14:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 4 2024-03-30T15:14:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.408+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 2 5 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 6 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 6 2024-03-30T15:14:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8008370 osd=2 3 2024-03-30T15:14:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 2 6 2024-03-30T15:14:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 4 2024-03-30T15:14:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 10 client.25947.objecter _send_command 6 2024-03-30T15:14:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.425+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b8007df0 con 0x7f72b800d5d0 2024-03-30T15:14:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.426+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b8007df0 con 0x7f72b800d5d0 2024-03-30T15:14:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.426+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.426+0000 7f72cffff700 10 client.25947.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.426+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 4 2024-03-30T15:14:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.426+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 2 6 2024-03-30T15:14:18.429 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:14:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 7 2024-03-30T15:14:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 7 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 3 7 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 4 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 10 client.25947.objecter _send_command 7 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f72b8015d50 con 0x7f72b800d160 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.427+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b800d160 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.428+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b8015d50 con 0x7f72b800d160 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.428+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 7: 0 ) 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.428+0000 7f72cffff700 10 client.25947.objecter _finish_command 7 = system:0 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.428+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 4 2024-03-30T15:14:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.428+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 3 7 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 8 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 8 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80081f0 osd=3 3 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 3 8 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 4 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 10 client.25947.objecter _send_command 8 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.445+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b800c7c0 con 0x7f72b800d160 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b800c7c0 con 0x7f72b800d160 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72cffff700 10 client.25947.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 4 2024-03-30T15:14:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 3 8 2024-03-30T15:14:18.448 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:14:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 9 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 9 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 4 9 2024-03-30T15:14:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 4 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 10 client.25947.objecter _send_command 9 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.446+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f72b8011ec0 con 0x7f72b8011a60 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.447+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8011a60 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.448+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b8011ec0 con 0x7f72b8011a60 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.448+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 9: 0 ) 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.448+0000 7f72cffff700 10 client.25947.objecter _finish_command 9 = system:0 2024-03-30T15:14:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.448+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 4 2024-03-30T15:14:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.448+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 4 9 2024-03-30T15:14:18.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 10 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 10 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b800c7c0 osd=4 3 2024-03-30T15:14:18.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 4 10 2024-03-30T15:14:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 4 2024-03-30T15:14:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 10 client.25947.objecter _send_command 10 2024-03-30T15:14:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.465+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b8011ec0 con 0x7f72b8011a60 2024-03-30T15:14:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b8011ec0 con 0x7f72b8011a60 2024-03-30T15:14:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72cffff700 10 client.25947.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 4 2024-03-30T15:14:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 4 10 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 11 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 11 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 5 11 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 4 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 10 client.25947.objecter _send_command 11 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.467+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f72b801ec00 con 0x7f72b8012030 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8012030 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b801ec00 con 0x7f72b8012030 2024-03-30T15:14:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 11: 0 ) 2024-03-30T15:14:18.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 10 client.25947.objecter _finish_command 11 = system:0 2024-03-30T15:14:18.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 4 2024-03-30T15:14:18.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.468+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 5 11 2024-03-30T15:14:18.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 12 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 12 2024-03-30T15:14:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8011ec0 osd=5 3 2024-03-30T15:14:18.498 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 5 12 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 4 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 10 client.25947.objecter _send_command 12 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.496+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b80205b0 con 0x7f72b8012030 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.498+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b80205b0 con 0x7f72b8012030 2024-03-30T15:14:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.498+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.498+0000 7f72cffff700 10 client.25947.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.498+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 4 2024-03-30T15:14:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.498+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 5 12 2024-03-30T15:14:18.500 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 13 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 13 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 6 13 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 4 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 10 client.25947.objecter _send_command 13 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f72b8021160 con 0x7f72b8020cd0 2024-03-30T15:14:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.499+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8020cd0 2024-03-30T15:14:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.500+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b8021160 con 0x7f72b8020cd0 2024-03-30T15:14:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.500+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 13: 0 ) 2024-03-30T15:14:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.500+0000 7f72cffff700 10 client.25947.objecter _finish_command 13 = system:0 2024-03-30T15:14:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.500+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 4 2024-03-30T15:14:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.500+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 6 13 2024-03-30T15:14:18.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 14 2024-03-30T15:14:18.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 14 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b80205b0 osd=6 3 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 6 14 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 4 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 10 client.25947.objecter _send_command 14 2024-03-30T15:14:18.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.517+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b8021160 con 0x7f72b8020cd0 2024-03-30T15:14:18.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b8021160 con 0x7f72b8020cd0 2024-03-30T15:14:18.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72cffff700 10 client.25947.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 4 2024-03-30T15:14:18.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 6 14 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 15 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 15 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 7 15 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 4 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 10 client.25947.objecter _send_command 15 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.519+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f72b80277d0 con 0x7f72b8023cc0 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 10 client.25947.objecter ms_handle_connect 0x7f72b8023cc0 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f72b80277d0 con 0x7f72b8023cc0 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 15: 0 ) 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 10 client.25947.objecter _finish_command 15 = system:0 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 4 2024-03-30T15:14:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.520+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 7 15 2024-03-30T15:14:18.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 10 client.25947.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:18.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign -1 16 2024-03-30T15:14:18.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 20 client.25947.objecter _get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_remove -1 16 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 20 client.25947.objecter get_session s=0x7f72b8021160 osd=7 3 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 15 client.25947.objecter _session_command_op_assign 7 16 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 4 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 10 client.25947.objecter _send_command 16 2024-03-30T15:14:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.538+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f72b80277d0 con 0x7f72b8023cc0 2024-03-30T15:14:18.541 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.540+0000 7f72cffff700 1 -- 172.21.15.28:0/1687920380 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f72b80277d0 con 0x7f72b8023cc0 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.540+0000 7f72cffff700 10 client.25947.objecter ms_dispatch 0x7f72d8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.540+0000 7f72cffff700 10 client.25947.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.540+0000 7f72cffff700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 4 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.540+0000 7f72cffff700 15 client.25947.objecter _session_command_op_remove 7 16 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 librados: watch_flush enter 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 librados: watch_flush exit 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.0 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6809/2927075629 conn(0x7f72b8004730 legacy=0x7f72b8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80053e0 osd=0 1 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.1 2024-03-30T15:14:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6805/1850313547 conn(0x7f72b8004280 legacy=0x7f72b800ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8004110 osd=1 1 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.2 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6801/2294873818 conn(0x7f72b800d5d0 legacy=0x7f72b800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8008370 osd=2 1 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.3 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6813/1506157250 conn(0x7f72b800d160 legacy=0x7f72b8014460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80081f0 osd=3 1 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.4 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72d759e700 1 -- 172.21.15.28:0/1687920380 reap_dead start 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.134:6800/4077237787 conn(0x7f72b8011a60 legacy=0x7f72b8019710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72d759e700 1 -- 172.21.15.28:0/1687920380 reap_dead start 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b800c7c0 osd=4 1 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.5 2024-03-30T15:14:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.134:6804/3668151179 conn(0x7f72b8012030 legacy=0x7f72b801d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8011ec0 osd=5 1 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.6 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.134:6808/2135224194 conn(0x7f72b8020cd0 legacy=0x7f72b801ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b80205b0 osd=6 1 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter close_session for osd.7 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.134:6812/3758190298 conn(0x7f72b8023cc0 legacy=0x7f72b80260b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter put_session s=0x7f72b8021160 osd=7 1 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 20 client.25947.objecter shutdown clearing up homeless session... 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 10 client.25947.objecter successfully canceled tick 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6800/3011431093 conn(0x7f72c4070580 legacy=0x7f72c4072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> v1:172.21.15.28:6789/0 conn(0x7f72d810d960 legacy=0x7f72d81a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72d759e700 1 -- 172.21.15.28:0/1687920380 reap_dead start 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 shutdown_connections 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 >> 172.21.15.28:0/1687920380 conn(0x7f72d80fc9e0 msgr2=0x7f72d81105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 shutdown_connections 2024-03-30T15:14:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 -- 172.21.15.28:0/1687920380 wait complete. 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:18.541+0000 7f72dd1a0700 1 librados: shutdown 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:14:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:14:18.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:18.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:18.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:18.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:18.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:18.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:18.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-03-30T15:14:18.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:18.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:18.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:19.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 12.1c scrub starts 2024-03-30T15:14:19.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 12.1c scrub ok 2024-03-30T15:14:19.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T15:14:19.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T15:14:19.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:14:19.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: pgmap v1376: 214 pgs: 5 creating+peering, 27 unknown, 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1687920380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: osdmap e890: 8 total, 8 up, 8 in 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 12.1c scrub starts 2024-03-30T15:14:19.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 12.1c scrub ok 2024-03-30T15:14:19.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T15:14:19.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: pgmap v1376: 214 pgs: 5 creating+peering, 27 unknown, 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1687920380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:14:19.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: osdmap e890: 8 total, 8 up, 8 in 2024-03-30T15:14:19.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:18 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:19.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 12.1c scrub starts 2024-03-30T15:14:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 12.1c scrub ok 2024-03-30T15:14:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: pgmap v1376: 214 pgs: 5 creating+peering, 27 unknown, 182 active+clean; 4.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1687920380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: osdmap e890: 8 total, 8 up, 8 in 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:18 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:19.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:19.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:19.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:19.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:19.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:19.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:19.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:19.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:19.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:19.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:19.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]': finished 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: osdmap e891: 8 total, 8 up, 8 in 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.965 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]': finished 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: osdmap e891: 8 total, 8 up, 8 in 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:19 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:19.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:20.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:20.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:20.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:20.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]': finished 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: osdmap e891: 8 total, 8 up, 8 in 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:19 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-148"}]: dispatch 2024-03-30T15:14:20.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:20.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:20.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:20.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:20.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:20.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:20.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:20.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:20.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:20.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:20.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:20.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:20.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:21.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:21.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:21.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:21.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:21.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T15:14:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T15:14:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:14:21.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 322.7s0 scrub starts 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 322.7 scrub ok 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: pgmap v1379: 214 pgs: 5 creating+peering, 9 unknown, 200 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:14:21.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:20 smithi134 ceph-mon[33714]: osdmap e892: 8 total, 8 up, 8 in 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 322.7s0 scrub starts 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 322.7 scrub ok 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:14:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: pgmap v1379: 214 pgs: 5 creating+peering, 9 unknown, 200 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[29876]: osdmap e892: 8 total, 8 up, 8 in 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 322.7s0 scrub starts 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 322.7 scrub ok 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:14:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: pgmap v1379: 214 pgs: 5 creating+peering, 9 unknown, 200 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-30T15:14:21.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:20 smithi028 ceph-mon[41614]: osdmap e892: 8 total, 8 up, 8 in 2024-03-30T15:14:21.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:21.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:21.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:21.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:21.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:21.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:21.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:21.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:21.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:21.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 -- 172.21.15.28:0/3273048348 >> v1:172.21.15.28:6789/0 conn(0x7f9d6c10ea90 legacy=0x7f9d6c110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 -- 172.21.15.28:0/3273048348 shutdown_connections 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 -- 172.21.15.28:0/3273048348 >> 172.21.15.28:0/3273048348 conn(0x7f9d6c0fdb10 msgr2=0x7f9d6c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 -- 172.21.15.28:0/3273048348 shutdown_connections 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 -- 172.21.15.28:0/3273048348 wait complete. 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 librados: starting msgr at 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 librados: starting objecter 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.991+0000 7f9d720a0700 1 Processor -- start 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 -- start start 2024-03-30T15:14:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 librados: setting wanted keys 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 librados: calling monclient init 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d6c1a93e0 con 0x7f9d6c10ea90 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d6c1aa600 con 0x7f9d6c102260 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d720a0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9d6c1ab840 con 0x7f9d6c10aef0 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d6affd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9d6c10aef0 0x7f9d6c1a42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39508/0 (socket says 172.21.15.28:39508) 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d6affd700 1 -- 172.21.15.28:0/1858919945 learned_addr learned my addr 172.21.15.28:0/1858919945 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter ms_handle_connect 0x7f9d6c10aef0 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter ms_handle_connect 0x7f9d6c10ea90 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter ms_handle_connect 0x7f9d6c102260 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3960264901 0 0) 0x7f9d6c1ab840 con 0x7f9d6c10aef0 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d54003aa0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069580444 0 0) 0x7f9d6c1aa600 con 0x7f9d6c102260 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d6c1ab840 con 0x7f9d6c102260 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595087368 0 0) 0x7f9d6c1a93e0 con 0x7f9d6c10ea90 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.992+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d6c1aa600 con 0x7f9d6c10ea90 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3780223814 0 0) 0x7f9d54003aa0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d6c1a93e0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d5c001230 con 0x7f9d6c10aef0 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 739900216 0 0) 0x7f9d6c1ab840 con 0x7f9d6c102260 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d54003aa0 con 0x7f9d6c102260 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d60001830 con 0x7f9d6c102260 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3841138015 0 0) 0x7f9d6c1aa600 con 0x7f9d6c10ea90 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d6c1ab840 con 0x7f9d6c10ea90 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d64001ca0 con 0x7f9d6c10ea90 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3362962935 0 0) 0x7f9d6c1a93e0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 >> v1:172.21.15.134:6789/0 conn(0x7f9d6c102260 legacy=0x7f9d6c1075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 >> v1:172.21.15.28:6789/0 conn(0x7f9d6c10ea90 legacy=0x7f9d6c1a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c1aca80 con 0x7f9d6c10aef0 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c1a9610 con 0x7f9d6c10aef0 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9d5c001600 con 0x7f9d6c10aef0 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9d5c005cd0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d720a0700 10 client.59156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d6c1a9ad0 con 0x7f9d6c10aef0 2024-03-30T15:14:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.993+0000 7f9d720a0700 1 librados: init done 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.994+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c103740 con 0x7f9d6c10aef0 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9d5c003930 con 0x7f9d6c10aef0 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(893..893 src has 442..893) ==== 8793+0+0 (unknown 1051576225 0 0) 0x7f9d5c089b90 con 0x7f9d6c10aef0 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 10 client.59156.objecter ms_dispatch 0x7f9d6c001550 osd_map(893..893 src has 442..893) 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 3 client.59156.objecter handle_osd_map got epochs [893,893] > 0 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 3 client.59156.objecter handle_osd_map decoding full epoch 893 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 20 client.59156.objecter dump_active .. 0 homeless 2024-03-30T15:14:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:21.995+0000 7f9d68ff9700 10 client.59156.objecter ms_handle_connect 0x7f9d540703f0 2024-03-30T15:14:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.000+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9d5c056b00 con 0x7f9d6c10aef0 2024-03-30T15:14:22.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:22.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:22.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:22.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: osdmap e893: 8 total, 8 up, 8 in 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:14:22.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:14:22.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:22.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:22.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: osdmap e893: 8 total, 8 up, 8 in 2024-03-30T15:14:22.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:21 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:14:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:14:22.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: osdmap e893: 8 total, 8 up, 8 in 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:21 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.217+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f9d6c075a20 con 0x7f9d6c10aef0 2024-03-30T15:14:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:14:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:14:22.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:22.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:22.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.602+0000 7f9d68ff9700 1 -- 172.21.15.28:0/1858919945 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v894) ==== 101+0+0 (unknown 2640160628 0 0) 0x7f9d5c05a550 con 0x7f9d6c10aef0 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.605+0000 7f9d720a0700 10 librados: watch_flush enter 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.605+0000 7f9d720a0700 10 librados: watch_flush exit 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.605+0000 7f9d720a0700 20 client.59156.objecter shutdown clearing up homeless session... 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.605+0000 7f9d720a0700 10 client.59156.objecter successfully canceled tick 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9d540703f0 legacy=0x7f9d540728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:22.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 >> v1:172.21.15.28:6790/0 conn(0x7f9d6c10aef0 legacy=0x7f9d6c1a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 shutdown_connections 2024-03-30T15:14:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 >> 172.21.15.28:0/1858919945 conn(0x7f9d6c0fdb10 msgr2=0x7f9d6c111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 shutdown_connections 2024-03-30T15:14:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 -- 172.21.15.28:0/1858919945 wait complete. 2024-03-30T15:14:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:22.606+0000 7f9d720a0700 1 librados: shutdown 2024-03-30T15:14:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:14:22.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:22.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:22.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:22.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:23.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:23.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:14:23.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: pgmap v1382: 214 pgs: 9 creating+peering, 23 unknown, 182 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1858919945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.59156 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.59156 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: osdmap e894: 8 total, 8 up, 8 in 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:14:23.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:22 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:14:23.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:23.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:23.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:14:23.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:14:23.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: pgmap v1382: 214 pgs: 9 creating+peering, 23 unknown, 182 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1858919945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.59156 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:14:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.59156 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: osdmap e894: 8 total, 8 up, 8 in 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:14:23.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: pgmap v1382: 214 pgs: 9 creating+peering, 23 unknown, 182 active+clean; 4.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1858919945' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.59156 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:23.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.59156 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: osdmap e894: 8 total, 8 up, 8 in 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-150"}]: dispatch 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:14:23.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:22 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:14:23.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T15:14:23.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:23.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:23.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:23.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:23.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:23.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:23.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:23.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:23.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:23.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:23.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:23.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:24.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T15:14:24.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T15:14:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:14:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:14:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:14:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:14:24.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:23 smithi134 ceph-mon[33714]: osdmap e895: 8 total, 8 up, 8 in 2024-03-30T15:14:24.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:14:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[41614]: osdmap e895: 8 total, 8 up, 8 in 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:14:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:23 smithi028 ceph-mon[29876]: osdmap e895: 8 total, 8 up, 8 in 2024-03-30T15:14:24.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:24.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:24.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-30T15:14:24.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:24.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:24.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:24.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:24.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:24.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:24.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:24.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:24.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:24.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:25.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: pgmap v1385: 182 pgs: 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: osdmap e896: 8 total, 8 up, 8 in 2024-03-30T15:14:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 322.1s0 scrub starts 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[29876]: 322.1 scrub ok 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:14:25.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: pgmap v1385: 182 pgs: 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: osdmap e896: 8 total, 8 up, 8 in 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 322.1s0 scrub starts 2024-03-30T15:14:25.074 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:24 smithi028 ceph-mon[41614]: 322.1 scrub ok 2024-03-30T15:14:25.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:14:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:14:25.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: pgmap v1385: 182 pgs: 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: osdmap e896: 8 total, 8 up, 8 in 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 322.1s0 scrub starts 2024-03-30T15:14:25.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:24 smithi134 ceph-mon[33714]: 322.1 scrub ok 2024-03-30T15:14:25.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:25.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:25.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:25.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:25.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:25.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:25.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:25.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.782 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:25.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:25.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:25.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:25.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd set noscrub 2024-03-30T15:14:26.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:26.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:26.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:14:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:14:26.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: osdmap e897: 8 total, 8 up, 8 in 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4267092211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: from='client.59162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:14:26.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:25 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: osdmap e897: 8 total, 8 up, 8 in 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4267092211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: from='client.59162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: osdmap e897: 8 total, 8 up, 8 in 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-152"}]: dispatch 2024-03-30T15:14:26.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4267092211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: from='client.59162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:14:26.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:14:26.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:25 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:14:26.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:26.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:26.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:26.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:26.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.498+0000 7f0f82f83700 1 -- 172.21.15.28:0/2720246335 >> v1:172.21.15.28:6790/0 conn(0x7f0f7c109dc0 legacy=0x7f0f7c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.498+0000 7f0f82f83700 1 -- 172.21.15.28:0/2720246335 shutdown_connections 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.498+0000 7f0f82f83700 1 -- 172.21.15.28:0/2720246335 >> 172.21.15.28:0/2720246335 conn(0x7f0f7c0fc9e0 msgr2=0x7f0f7c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.498+0000 7f0f82f83700 1 -- 172.21.15.28:0/2720246335 shutdown_connections 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.498+0000 7f0f82f83700 1 -- 172.21.15.28:0/2720246335 wait complete. 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 librados: starting msgr at 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 librados: starting objecter 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 Processor -- start 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 -- start start 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 librados: setting wanted keys 2024-03-30T15:14:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.499+0000 7f0f82f83700 1 librados: calling monclient init 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f82f83700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0f7c1a6010 con 0x7f0f7c101130 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f82f83700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0f7c1a7230 con 0x7f0f7c10d960 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f82f83700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0f7c1a8470 con 0x7f0f7c109dc0 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f7bfff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f0f7c109dc0 0x7f0f7c1a0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39538/0 (socket says 172.21.15.28:39538) 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f7bfff700 1 -- 172.21.15.28:0/4267092211 learned_addr learned my addr 172.21.15.28:0/4267092211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter ms_handle_connect 0x7f0f7c109dc0 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter ms_handle_connect 0x7f0f7c101130 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter ms_handle_connect 0x7f0f7c10d960 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82184301 0 0) 0x7f0f7c1a7230 con 0x7f0f7c10d960 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f6c0031f0 con 0x7f0f7c10d960 2024-03-30T15:14:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231862885 0 0) 0x7f0f7c1a6010 con 0x7f0f7c101130 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f7c1a7230 con 0x7f0f7c101130 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243190391 0 0) 0x7f0f7c1a8470 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f7c1a6010 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1497559938 0 0) 0x7f0f7c1a6010 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f7c1a8470 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2274015296 0 0) 0x7f0f6c0031f0 con 0x7f0f7c10d960 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0f7c1a6010 con 0x7f0f7c10d960 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.500+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0f68004610 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0f74001470 con 0x7f0f7c10d960 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3548729470 0 0) 0x7f0f7c1a8470 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 >> v1:172.21.15.134:6789/0 conn(0x7f0f7c10d960 legacy=0x7f0f7c1a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 >> v1:172.21.15.28:6789/0 conn(0x7f0f7c101130 legacy=0x7f0f7c1064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c1a96b0 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c1a6240 con 0x7f0f7c109dc0 2024-03-30T15:14:26.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0f68001aa0 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0f680050d0 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f82f83700 10 client.59162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0f7c1a6700 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.501+0000 7f0f82f83700 1 librados: init done 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.502+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7c102610 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.503+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0f680053a0 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(896..896 src has 442..896) ==== 8793+0+0 (unknown 2427114774 0 0) 0x7f0f68089c20 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 10 client.59162.objecter ms_dispatch 0x7f0f7c001550 osd_map(896..896 src has 442..896) 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 3 client.59162.objecter handle_osd_map got epochs [896,896] > 0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 3 client.59162.objecter handle_osd_map decoding full epoch 896 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 20 client.59162.objecter dump_active .. 0 homeless 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.504+0000 7f0f79ffb700 10 client.59162.objecter ms_handle_connect 0x7f0f6c074810 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.510+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0f68056b90 con 0x7f0f7c109dc0 2024-03-30T15:14:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:25.759+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f0f7c104a80 con 0x7f0f7c109dc0 2024-03-30T15:14:26.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.606+0000 7f0f79ffb700 1 -- 172.21.15.28:0/4267092211 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v898) ==== 113+0+0 (unknown 3479781568 0 0) 0x7f0f6805a5e0 con 0x7f0f7c109dc0 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.609+0000 7f0f82f83700 10 librados: watch_flush enter 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.609+0000 7f0f82f83700 10 librados: watch_flush exit 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.609+0000 7f0f82f83700 20 client.59162.objecter shutdown clearing up homeless session... 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.610+0000 7f0f82f83700 10 client.59162.objecter successfully canceled tick 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.610+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0f6c074810 legacy=0x7f0f6c076cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.610+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 >> v1:172.21.15.28:6790/0 conn(0x7f0f7c109dc0 legacy=0x7f0f7c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.611+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 shutdown_connections 2024-03-30T15:14:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.611+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 >> 172.21.15.28:0/4267092211 conn(0x7f0f7c0fc9e0 msgr2=0x7f0f7c106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.612+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 shutdown_connections 2024-03-30T15:14:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.612+0000 7f0f82f83700 1 -- 172.21.15.28:0/4267092211 wait complete. 2024-03-30T15:14:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:26.612+0000 7f0f82f83700 1 librados: shutdown 2024-03-30T15:14:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T15:14:26.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:26.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:26.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:26.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:26.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:26.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: 12.17 deep-scrub starts 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: 12.17 deep-scrub ok 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: 12.12 deep-scrub starts 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: 12.12 deep-scrub ok 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: pgmap v1388: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: from='client.59162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[29876]: osdmap e898: 8 total, 8 up, 8 in 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: 12.17 deep-scrub starts 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: 12.17 deep-scrub ok 2024-03-30T15:14:26.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: 12.12 deep-scrub starts 2024-03-30T15:14:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: 12.12 deep-scrub ok 2024-03-30T15:14:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: pgmap v1388: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:14:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: from='client.59162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:14:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:26 smithi028 ceph-mon[41614]: osdmap e898: 8 total, 8 up, 8 in 2024-03-30T15:14:27.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:27.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:27.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:27.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: 12.17 deep-scrub starts 2024-03-30T15:14:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: 12.17 deep-scrub ok 2024-03-30T15:14:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: 12.12 deep-scrub starts 2024-03-30T15:14:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: 12.12 deep-scrub ok 2024-03-30T15:14:27.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: pgmap v1388: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-30T15:14:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: from='client.59162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:14:27.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:26 smithi134 ceph-mon[33714]: osdmap e898: 8 total, 8 up, 8 in 2024-03-30T15:14:27.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:27.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:27.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:27.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.478+0000 7f70b5b70700 1 -- 172.21.15.28:0/3336957507 >> v1:172.21.15.28:6790/0 conn(0x7f70b010eb10 legacy=0x7f70b0110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.478+0000 7f70b5b70700 1 -- 172.21.15.28:0/3336957507 shutdown_connections 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.478+0000 7f70b5b70700 1 -- 172.21.15.28:0/3336957507 >> 172.21.15.28:0/3336957507 conn(0x7f70b00fc9a0 msgr2=0x7f70b00fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.478+0000 7f70b5b70700 1 -- 172.21.15.28:0/3336957507 shutdown_connections 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.478+0000 7f70b5b70700 1 -- 172.21.15.28:0/3336957507 wait complete. 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 librados: starting msgr at 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 librados: starting objecter 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 Processor -- start 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 -- start start 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 librados: setting wanted keys 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 librados: calling monclient init 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b010aba0 con 0x7f70b010b000 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b010adb0 con 0x7f70b010eb10 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70b5b70700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70b01a9660 con 0x7f70b01031f0 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70af7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f70b01031f0 0x7f70b01079f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:39556/0 (socket says 172.21.15.28:39556) 2024-03-30T15:14:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.479+0000 7f70af7fe700 1 -- 172.21.15.28:0/441197422 learned_addr learned my addr 172.21.15.28:0/441197422 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter ms_handle_connect 0x7f70b010b000 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter ms_handle_connect 0x7f70b01031f0 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter ms_handle_connect 0x7f70b010eb10 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057677261 0 0) 0x7f70b010aba0 con 0x7f70b010b000 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70980031f0 con 0x7f70b010b000 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152284856 0 0) 0x7f70b01a9660 con 0x7f70b01031f0 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b010aba0 con 0x7f70b01031f0 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408369575 0 0) 0x7f70b010adb0 con 0x7f70b010eb10 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b01a9660 con 0x7f70b010eb10 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3932304206 0 0) 0x7f70b010aba0 con 0x7f70b01031f0 2024-03-30T15:14:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b010adb0 con 0x7f70b01031f0 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a00018f0 con 0x7f70b01031f0 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 256131385 0 0) 0x7f70980031f0 con 0x7f70b010b000 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b010aba0 con 0x7f70b010b000 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2475957019 0 0) 0x7f70b01a9660 con 0x7f70b010eb10 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70980031f0 con 0x7f70b010eb10 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a4001890 con 0x7f70b010b000 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a80017b0 con 0x7f70b010eb10 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3581852749 0 0) 0x7f70b010adb0 con 0x7f70b01031f0 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.134:6789/0 conn(0x7f70b010eb10 legacy=0x7f70b01a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6789/0 conn(0x7f70b010b000 legacy=0x7f70b0108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b01aa880 con 0x7f70b01031f0 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70b01a9890 con 0x7f70b01031f0 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70b5b70700 10 client.59168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70b01a9d50 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70b5b70700 1 librados: init done 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f70a0003b40 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.480+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f70a0005d30 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.481+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f709c005130 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f70a001c650 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(898..898 src has 442..898) ==== 8420+0+0 (unknown 2295625723 0 0) 0x7f70a0089820 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 osd_map(898..898 src has 442..898) 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 3 client.59168.objecter handle_osd_map got epochs [898,898] > 0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 3 client.59168.objecter handle_osd_map decoding full epoch 898 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 20 client.59168.objecter dump_active .. 0 homeless 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.482+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v898) ==== 54+0+16 (unknown 1448783734 0 2420000192) 0x7f70a0089c60 con 0x7f70b01031f0 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 10 client.59168.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 1 2024-03-30T15:14:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 1 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 0 1 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 4 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 10 client.59168.objecter _send_command 1 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f70980704e0 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f709c004b80 con 0x7f709c004760 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.483+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c004760 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.484+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c004b80 con 0x7f709c004760 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.484+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 1: 0 ) 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.484+0000 7f70acff9700 10 client.59168.objecter _finish_command 1 = system:0 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.484+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 4 2024-03-30T15:14:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.484+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 0 1 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 10 client.59168.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 2 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 2 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c005410 osd=0 3 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 0 2 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 4 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 10 client.59168.objecter _send_command 2 2024-03-30T15:14:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.501+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c007b30 con 0x7f709c004760 2024-03-30T15:14:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c007b30 con 0x7f709c004760 2024-03-30T15:14:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70acff9700 10 client.59168.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 4 2024-03-30T15:14:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 0 2 2024-03-30T15:14:27.511 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:14:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 10 client.59168.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 1 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 4 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 10 client.59168.objecter _send_command 3 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.503+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f709c002130 con 0x7f709c001bb0 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c001bb0 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c002130 con 0x7f709c001bb0 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 3: 0 ) 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 10 client.59168.objecter _finish_command 3 = system:0 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 4 2024-03-30T15:14:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.504+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 1 3 2024-03-30T15:14:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 10 client.59168.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 4 2024-03-30T15:14:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 4 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c004fc0 osd=1 3 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 1 4 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 4 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 10 client.59168.objecter _send_command 4 2024-03-30T15:14:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.526+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c001630 con 0x7f709c001bb0 2024-03-30T15:14:27.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c001630 con 0x7f709c001bb0 2024-03-30T15:14:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70acff9700 10 client.59168.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 4 2024-03-30T15:14:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 1 4 2024-03-30T15:14:27.530 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 10 client.59168.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 5 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 5 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 2 5 2024-03-30T15:14:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 4 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 10 client.59168.objecter _send_command 5 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.528+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f709c010f70 con 0x7f709c0089c0 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c0089c0 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c010f70 con 0x7f709c0089c0 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 5: 0 ) 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 10 client.59168.objecter _finish_command 5 = system:0 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 4 2024-03-30T15:14:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.529+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 2 5 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 10 client.59168.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 6 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 6 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c001a30 osd=2 3 2024-03-30T15:14:27.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 2 6 2024-03-30T15:14:27.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 4 2024-03-30T15:14:27.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 10 client.59168.objecter _send_command 6 2024-03-30T15:14:27.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.547+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c008030 con 0x7f709c0089c0 2024-03-30T15:14:27.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c008030 con 0x7f709c0089c0 2024-03-30T15:14:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70acff9700 10 client.59168.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 4 2024-03-30T15:14:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 2 6 2024-03-30T15:14:27.550 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 10 client.59168.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 7 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 7 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 3 7 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 4 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 10 client.59168.objecter _send_command 7 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.548+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f709c00d440 con 0x7f709c00cfb0 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c00cfb0 2024-03-30T15:14:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c00d440 con 0x7f709c00cfb0 2024-03-30T15:14:27.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 7: 0 ) 2024-03-30T15:14:27.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 10 client.59168.objecter _finish_command 7 = system:0 2024-03-30T15:14:27.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 4 2024-03-30T15:14:27.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.549+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 3 7 2024-03-30T15:14:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 10 client.59168.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 8 2024-03-30T15:14:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 8 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c0082b0 osd=3 3 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 3 8 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 4 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 10 client.59168.objecter _send_command 8 2024-03-30T15:14:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.567+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c00caf0 con 0x7f709c00cfb0 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.568+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c00caf0 con 0x7f709c00cfb0 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.568+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.568+0000 7f70acff9700 10 client.59168.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.568+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 4 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.568+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 3 8 2024-03-30T15:14:27.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 10 client.59168.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 9 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 9 2024-03-30T15:14:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 4 9 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 4 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 10 client.59168.objecter _send_command 9 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f709c00cad0 con 0x7f709c011e30 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.569+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c011e30 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.570+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c00cad0 con 0x7f709c011e30 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.570+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 9: 0 ) 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.570+0000 7f70acff9700 10 client.59168.objecter _finish_command 9 = system:0 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.570+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 4 2024-03-30T15:14:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.570+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 4 9 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 10 client.59168.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 10 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 10 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c00c8a0 osd=4 3 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 4 10 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 4 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 10 client.59168.objecter _send_command 10 2024-03-30T15:14:27.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.590+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c01a900 con 0x7f709c011e30 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.600+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c01a900 con 0x7f709c011e30 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.600+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.600+0000 7f70acff9700 10 client.59168.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.600+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 4 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.600+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 4 10 2024-03-30T15:14:27.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 10 client.59168.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 11 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 11 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 5 11 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 4 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 10 client.59168.objecter _send_command 11 2024-03-30T15:14:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f709c01cc10 con 0x7f709c01b020 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.601+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c01b020 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.602+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c01cc10 con 0x7f709c01b020 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.602+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 11: 0 ) 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.602+0000 7f70acff9700 10 client.59168.objecter _finish_command 11 = system:0 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.602+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 4 2024-03-30T15:14:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.602+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 5 11 2024-03-30T15:14:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 10 client.59168.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 12 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 12 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01a900 osd=5 3 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 5 12 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 4 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 10 client.59168.objecter _send_command 12 2024-03-30T15:14:27.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.620+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c01cc10 con 0x7f709c01b020 2024-03-30T15:14:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c01cc10 con 0x7f709c01b020 2024-03-30T15:14:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70acff9700 10 client.59168.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 4 2024-03-30T15:14:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 5 12 2024-03-30T15:14:27.624 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:14:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 10 client.59168.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 13 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 13 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 6 13 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 4 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 10 client.59168.objecter _send_command 13 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.622+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f709c0232a0 con 0x7f709c01f790 2024-03-30T15:14:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.623+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c01f790 2024-03-30T15:14:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.624+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c0232a0 con 0x7f709c01f790 2024-03-30T15:14:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.624+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 13: 0 ) 2024-03-30T15:14:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.624+0000 7f70acff9700 10 client.59168.objecter _finish_command 13 = system:0 2024-03-30T15:14:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.624+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 4 2024-03-30T15:14:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.624+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 6 13 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 10 client.59168.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 14 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 14 2024-03-30T15:14:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01cc10 osd=6 3 2024-03-30T15:14:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 6 14 2024-03-30T15:14:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 4 2024-03-30T15:14:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 10 client.59168.objecter _send_command 14 2024-03-30T15:14:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.641+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c0232a0 con 0x7f709c01f790 2024-03-30T15:14:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.643+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c0232a0 con 0x7f709c01f790 2024-03-30T15:14:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.643+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.643+0000 7f70acff9700 10 client.59168.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.643+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 4 2024-03-30T15:14:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.643+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 6 14 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 10 client.59168.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 15 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 15 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 7 15 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 4 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 10 client.59168.objecter _send_command 15 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f709c027550 con 0x7f709c0237e0 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.644+0000 7f70acff9700 10 client.59168.objecter ms_handle_connect 0x7f709c0237e0 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.645+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f709c027550 con 0x7f709c0237e0 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.645+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 15: 0 ) 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.645+0000 7f70acff9700 10 client.59168.objecter _finish_command 15 = system:0 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.645+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 4 2024-03-30T15:14:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.645+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 7 15 2024-03-30T15:14:27.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 10 client.59168.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign -1 16 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 20 client.59168.objecter _get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_remove -1 16 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 20 client.59168.objecter get_session s=0x7f709c01d0a0 osd=7 3 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 15 client.59168.objecter _session_command_op_assign 7 16 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 4 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 10 client.59168.objecter _send_command 16 2024-03-30T15:14:27.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.673+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f709c0113a0 con 0x7f709c0237e0 2024-03-30T15:14:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70acff9700 1 -- 172.21.15.28:0/441197422 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f709c0113a0 con 0x7f709c0237e0 2024-03-30T15:14:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70acff9700 10 client.59168.objecter ms_dispatch 0x7f70b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70acff9700 10 client.59168.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70acff9700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 4 2024-03-30T15:14:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70acff9700 15 client.59168.objecter _session_command_op_remove 7 16 2024-03-30T15:14:27.677 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:14:27.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 librados: watch_flush enter 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 librados: watch_flush exit 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.0 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6809/2927075629 conn(0x7f709c004760 legacy=0x7f709c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c005410 osd=0 1 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.1 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6805/1850313547 conn(0x7f709c001bb0 legacy=0x7f709c00ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c004fc0 osd=1 1 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.2 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6801/2294873818 conn(0x7f709c0089c0 legacy=0x7f709c00f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c001a30 osd=2 1 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.3 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6813/1506157250 conn(0x7f709c00cfb0 legacy=0x7f709c014930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c0082b0 osd=3 1 2024-03-30T15:14:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.4 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.134:6800/4077237787 conn(0x7f709c011e30 legacy=0x7f709c0191e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c00c8a0 osd=4 1 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.5 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.134:6804/3668151179 conn(0x7f709c01b020 legacy=0x7f709c01b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01a900 osd=5 1 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.6 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.134:6808/2135224194 conn(0x7f709c01f790 legacy=0x7f709c021b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01cc10 osd=6 1 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter close_session for osd.7 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.134:6812/3758190298 conn(0x7f709c0237e0 legacy=0x7f709c025ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter put_session s=0x7f709c01d0a0 osd=7 1 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 20 client.59168.objecter shutdown clearing up homeless session... 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 10 client.59168.objecter successfully canceled tick 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70affff700 1 -- 172.21.15.28:0/441197422 reap_dead start 2024-03-30T15:14:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70affff700 1 -- 172.21.15.28:0/441197422 reap_dead start 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70affff700 1 -- 172.21.15.28:0/441197422 reap_dead start 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70affff700 1 -- 172.21.15.28:0/441197422 reap_dead start 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70affff700 1 -- 172.21.15.28:0/441197422 reap_dead start 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.675+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6800/3011431093 conn(0x7f70980704e0 legacy=0x7f70980729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> v1:172.21.15.28:6790/0 conn(0x7f70b01031f0 legacy=0x7f70b01079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 shutdown_connections 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 >> 172.21.15.28:0/441197422 conn(0x7f70b00fc9a0 msgr2=0x7f70b00fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 shutdown_connections 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 -- 172.21.15.28:0/441197422 wait complete. 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:27.676+0000 7f70b5b70700 1 librados: shutdown 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:14:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:14:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:14:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:14:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:14:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:27.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:27.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:27.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:27.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:27.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:27.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:28.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:28.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:28.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:14:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:14:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/441197422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:28.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: osdmap e899: 8 total, 8 up, 8 in 2024-03-30T15:14:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: 322.4s0 scrub starts 2024-03-30T15:14:28.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:27 smithi134 ceph-mon[33714]: 322.4 scrub ok 2024-03-30T15:14:28.190 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/441197422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: osdmap e899: 8 total, 8 up, 8 in 2024-03-30T15:14:28.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: 322.4s0 scrub starts 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[41614]: 322.4 scrub ok 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/441197422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: osdmap e899: 8 total, 8 up, 8 in 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: 322.4s0 scrub starts 2024-03-30T15:14:28.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:27 smithi028 ceph-mon[29876]: 322.4 scrub ok 2024-03-30T15:14:28.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:28.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:28.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:28.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:28.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:28.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:28.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:28.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:28.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:28.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:28.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:28.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:28.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:29.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:29.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:29.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: 5.16 deep-scrub starts 2024-03-30T15:14:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: 5.16 deep-scrub ok 2024-03-30T15:14:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: 5.0 deep-scrub starts 2024-03-30T15:14:29.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: 5.0 deep-scrub ok 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: pgmap v1391: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: osdmap e900: 8 total, 8 up, 8 in 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:28 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: 5.16 deep-scrub starts 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: 5.16 deep-scrub ok 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: 5.0 deep-scrub starts 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: 5.0 deep-scrub ok 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: pgmap v1391: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: osdmap e900: 8 total, 8 up, 8 in 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: 5.16 deep-scrub starts 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: 5.16 deep-scrub ok 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: 5.0 deep-scrub starts 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: 5.0 deep-scrub ok 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: pgmap v1391: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi028-93253-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T15:14:29.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: osdmap e900: 8 total, 8 up, 8 in 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi028-93253-111","var": "dedup_tier","val": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:28 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi028-93253-111", "tierpool": "test-rados-api-smithi028-93253-154"}]: dispatch 2024-03-30T15:14:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:29.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:29.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:29.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:29.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:29.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:29.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:29.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:29.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:29.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:29.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:29.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:29.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:29.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:29.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T15:14:29.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 3.b scrub ok 2024-03-30T15:14:29.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:14:29.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:14:29.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: osdmap e901: 8 total, 8 up, 8 in 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 3.b scrub ok 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:14:29.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: osdmap e901: 8 total, 8 up, 8 in 2024-03-30T15:14:29.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:14:29.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:14:29.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:29 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:14:30.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.097 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:30.112 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:14:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-03-30T15:14:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-03-30T15:14:30.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:14:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T15:14:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 3.b scrub ok 2024-03-30T15:14:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:14:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:14:30.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:14:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:14:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: osdmap e901: 8 total, 8 up, 8 in 2024-03-30T15:14:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:14:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:14:30.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:29 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:14:30.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:30.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:30.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:30.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:30.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:30.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:30.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:30.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:30.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-03-30T15:14:30.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:30.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:30.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:30.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:31.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:31.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:31.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: pgmap v1394: 182 pgs: 182 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: osdmap e902: 8 total, 8 up, 8 in 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.144 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: pgmap v1394: 182 pgs: 182 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: osdmap e902: 8 total, 8 up, 8 in 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:30 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:31.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-03-30T15:14:31.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: pgmap v1394: 182 pgs: 182 active+clean; 4.4 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: osdmap e902: 8 total, 8 up, 8 in 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:30 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:31.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:31.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:31.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:31.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:31.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:31.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:31.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T15:14:31.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5 2024-03-30T15:14:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:31.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.5' 2024-03-30T15:14:31.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:31.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:31.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6 2024-03-30T15:14:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:32.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.6' 2024-03-30T15:14:32.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T15:14:32.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T15:14:32.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: osdmap e903: 8 total, 8 up, 8 in 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:14:32.071 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:32.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: osdmap e903: 8 total, 8 up, 8 in 2024-03-30T15:14:32.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:31 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:32.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:14:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: osdmap e903: 8 total, 8 up, 8 in 2024-03-30T15:14:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2751046486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:31 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]: dispatch 2024-03-30T15:14:32.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:32.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:32.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4 2024-03-30T15:14:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:32.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.4' 2024-03-30T15:14:32.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:32.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: LibRadosTwoPoolsECPP.TryFlush (8030 ms) 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12968 ms) 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8001 ms) 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13013 ms) 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-30T15:14:32.608 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8030 ms) 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8038 ms) 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: ok, hit_set contains 336:602f83fe:::foo:head 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8031 ms) 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,0 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: first is 1711811442 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,0 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,0 2024-03-30T15:14:32.609 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,1711811450,1711811451,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,1711811450,1711811451,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811442,1711811444,1711811445,1711811447,1711811448,1711811450,1711811451,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: got ls 1711811445,1711811447,1711811448,1711811450,1711811451,1711811453,1711811454,0 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: first now 1711811445, trimmed 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (167802 ms) 2024-03-30T15:14:32.610 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: foo0 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13288 ms) 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17994 ms) 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22177 ms) 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4843 ms) 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3005 ms) 2024-03-30T15:14:32.611 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3007 ms) 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2985 ms) 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (376244 ms total) 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1008130 ms total) 2024-03-30T15:14:32.612 INFO:tasks.workunit.client.0.smithi028.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-30T15:14:32.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92956 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92956 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93200 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93200 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93744 2024-03-30T15:14:32.620 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93744 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93401 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93401 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93273 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93273 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93038 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93038 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93318 2024-03-30T15:14:32.621 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93318 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94018 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94018 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94059 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94059 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93054 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93054 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93802 2024-03-30T15:14:32.622 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93802 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93650 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93650 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92930 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92930 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92996 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92996 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93692 2024-03-30T15:14:32.623 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93692 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92916 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92916 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92941 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92941 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94159 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94159 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93222 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93222 2024-03-30T15:14:32.624 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94219 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94219 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93302 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93302 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93907 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93907 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94246 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94246 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.625 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93239 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93239 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93727 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93727 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94125 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94125 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93853 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93853 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.626 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94185 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94185 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93636 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93636 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93152 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93152 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94232 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94232 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93439 2024-03-30T15:14:32.627 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93439 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92909 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92909 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=92975 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 92975 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93355 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93355 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93023 2024-03-30T15:14:32.628 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93023 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93555 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93555 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=93578 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 93578 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94090 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94090 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ for t in "${!pids[@]}" 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ pid=94268 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ wait 94268 2024-03-30T15:14:32.629 INFO:tasks.workunit.client.0.smithi028.stderr:+ exit 0 2024-03-30T15:14:32.630 INFO:tasks.workunit.client.0.smithi028.stderr:+ cleanup 2024-03-30T15:14:32.630 INFO:tasks.workunit.client.0.smithi028.stderr:+ pkill -P 92896 2024-03-30T15:14:32.630 INFO:tasks.workunit.client.0.smithi028.stderr:+ true 2024-03-30T15:14:32.630 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T15:14:32.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T15:14:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:32.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.3' 2024-03-30T15:14:32.673 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-03-30T15:14:32.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T15:14:32.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1 2024-03-30T15:14:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:32.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.1' 2024-03-30T15:14:32.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7 2024-03-30T15:14:32.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-03-30T15:14:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:32.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.7' 2024-03-30T15:14:32.988 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:33.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2 2024-03-30T15:14:33.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:33.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.2' 2024-03-30T15:14:33.041 DEBUG:teuthology.parallel:result is None 2024-03-30T15:14:33.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T15:14:33.109 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T15:14:33.110 DEBUG:teuthology.orchestra.run.smithi028:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T15:14:33.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0 2024-03-30T15:14:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-03-30T15:14:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:14:33.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd.0' 2024-03-30T15:14:33.177 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T15:14:33.178 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-30T15:14:33.188 INFO:tasks.thrashosds:joining thrashosds 2024-03-30T15:14:33.188 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 5.1e scrub starts 2024-03-30T15:14:33.188 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 5.1e scrub ok 2024-03-30T15:14:33.188 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:33.188 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 4.14 deep-scrub starts 2024-03-30T15:14:33.189 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 4.14 deep-scrub ok 2024-03-30T15:14:33.189 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:14:33.189 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:14:33.189 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:33.190 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:32 smithi134 ceph-mon[33714]: osdmap e904: 8 total, 8 up, 8 in 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 5.1e scrub starts 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 5.1e scrub ok 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 4.14 deep-scrub starts 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 4.14 deep-scrub ok 2024-03-30T15:14:33.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[41614]: osdmap e904: 8 total, 8 up, 8 in 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 5.1e scrub starts 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 5.1e scrub ok 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 4.14 deep-scrub starts 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 4.14 deep-scrub ok 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:14:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: from='client.58535 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi028-93253-111"}]': finished 2024-03-30T15:14:33.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:32 smithi028 ceph-mon[29876]: osdmap e904: 8 total, 8 up, 8 in 2024-03-30T15:14:34.059 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[29876]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[29876]: 1.3 scrub starts 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[29876]: 1.3 scrub ok 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[41614]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[41614]: 1.3 scrub starts 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[41614]: 1.3 scrub ok 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:14:34.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:33 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:14:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:33 smithi134 ceph-mon[33714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T15:14:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:33 smithi134 ceph-mon[33714]: 1.3 scrub starts 2024-03-30T15:14:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:33 smithi134 ceph-mon[33714]: 1.3 scrub ok 2024-03-30T15:14:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:33 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:14:34.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:33 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:14:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 1.4 scrub starts 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 1.4 scrub ok 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:14:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:34 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:14:35.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:14:35.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:14:35.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 1.4 scrub starts 2024-03-30T15:14:35.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 1.4 scrub ok 2024-03-30T15:14:35.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 1.4 scrub starts 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 1.4 scrub ok 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: pgmap v1399: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:14:35.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:14:35.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:14:35.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:14:35.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:34 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- 172.21.15.28:0/2522360659 >> v1:172.21.15.28:6790/0 conn(0x7f7a4010ea80 legacy=0x7f7a40110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- 172.21.15.28:0/2522360659 shutdown_connections 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- 172.21.15.28:0/2522360659 >> 172.21.15.28:0/2522360659 conn(0x7f7a400fdb20 msgr2=0x7f7a400fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- 172.21.15.28:0/2522360659 shutdown_connections 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- 172.21.15.28:0/2522360659 wait complete. 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 librados: starting msgr at 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 librados: starting objecter 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 Processor -- start 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 -- start start 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 librados: setting wanted keys 2024-03-30T15:14:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.573+0000 7f7a46860700 1 librados: calling monclient init 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a46860700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a401a9390 con 0x7f7a40102250 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a46860700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a401aa5d0 con 0x7f7a4010aee0 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a46860700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7a401ab810 con 0x7f7a4010ea80 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3ffff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7a40102250 0x7f7a401073f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56468/0 (socket says 172.21.15.28:56468) 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3ffff700 1 -- 172.21.15.28:0/4084813280 learned_addr learned my addr 172.21.15.28:0/4084813280 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a4010ea80 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a40102250 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter ms_handle_connect 0x7f7a4010aee0 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041730732 0 0) 0x7f7a401ab810 con 0x7f7a4010ea80 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a2c0031f0 con 0x7f7a4010ea80 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2016185816 0 0) 0x7f7a401a9390 con 0x7f7a40102250 2024-03-30T15:14:35.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a401ab810 con 0x7f7a40102250 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3108749249 0 0) 0x7f7a401aa5d0 con 0x7f7a4010aee0 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a401a9390 con 0x7f7a4010aee0 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.574+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1685898480 0 0) 0x7f7a2c0031f0 con 0x7f7a4010ea80 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a401aa5d0 con 0x7f7a4010ea80 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a38004880 con 0x7f7a4010ea80 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 904704013 0 0) 0x7f7a401ab810 con 0x7f7a40102250 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a2c0031f0 con 0x7f7a40102250 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a28001870 con 0x7f7a40102250 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1043721751 0 0) 0x7f7a401a9390 con 0x7f7a4010aee0 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a401ab810 con 0x7f7a4010aee0 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a300017c0 con 0x7f7a4010aee0 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4202705741 0 0) 0x7f7a401aa5d0 con 0x7f7a4010ea80 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 >> v1:172.21.15.134:6789/0 conn(0x7f7a4010aee0 legacy=0x7f7a401a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 >> v1:172.21.15.28:6789/0 conn(0x7f7a40102250 legacy=0x7f7a401073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a401aca50 con 0x7f7a4010ea80 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a401a95c0 con 0x7f7a4010ea80 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a46860700 10 client.59174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a401a9a80 con 0x7f7a4010ea80 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a46860700 1 librados: init done 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7a38004ae0 con 0x7f7a4010ea80 2024-03-30T15:14:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.575+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7a38005010 con 0x7f7a4010ea80 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.576+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a20005120 con 0x7f7a4010ea80 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7a38003be0 con 0x7f7a4010ea80 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(904..904 src has 550..904) ==== 7561+0+0 (unknown 2318270959 0 0) 0x7f7a380886d0 con 0x7f7a4010ea80 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 10 client.59174.objecter ms_dispatch 0x7f7a40001550 osd_map(904..904 src has 550..904) 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 3 client.59174.objecter handle_osd_map got epochs [904,904] > 0 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 3 client.59174.objecter handle_osd_map decoding full epoch 904 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 20 client.59174.objecter dump_active .. 0 homeless 2024-03-30T15:14:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.577+0000 7f7a3d7fa700 10 client.59174.objecter ms_handle_connect 0x7f7a2c070530 2024-03-30T15:14:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.582+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7a38055b20 con 0x7f7a4010ea80 2024-03-30T15:14:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.805+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f7a20005410 con 0x7f7a4010ea80 2024-03-30T15:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.918+0000 7f7a3d7fa700 1 -- 172.21.15.28:0/4084813280 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v905) ==== 113+0+0 (unknown 2074373223 0 0) 0x7f7a38059570 con 0x7f7a4010ea80 2024-03-30T15:14:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 10 librados: watch_flush enter 2024-03-30T15:14:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 10 librados: watch_flush exit 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 20 client.59174.objecter shutdown clearing up homeless session... 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 10 client.59174.objecter successfully canceled tick 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7a2c070530 legacy=0x7f7a2c0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 >> v1:172.21.15.28:6790/0 conn(0x7f7a4010ea80 legacy=0x7f7a401a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 shutdown_connections 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 >> 172.21.15.28:0/4084813280 conn(0x7f7a400fdb20 msgr2=0x7f7a4010a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 shutdown_connections 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 -- 172.21.15.28:0/4084813280 wait complete. 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.922+0000 7f7a46860700 1 librados: shutdown 2024-03-30T15:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T15:14:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:35 smithi134 ceph-mon[33714]: 4.1 deep-scrub starts 2024-03-30T15:14:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:35 smithi134 ceph-mon[33714]: 4.1 deep-scrub ok 2024-03-30T15:14:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:35 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4084813280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:35 smithi134 ceph-mon[33714]: from='client.59174 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[41614]: 4.1 deep-scrub starts 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[41614]: 4.1 deep-scrub ok 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4084813280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[41614]: from='client.59174 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[29876]: 4.1 deep-scrub starts 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[29876]: 4.1 deep-scrub ok 2024-03-30T15:14:36.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4084813280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:36.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:35 smithi028 ceph-mon[29876]: from='client.59174 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: from='client.59174 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: osdmap e905: 8 total, 8 up, 8 in 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3702286035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: from='client.59180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: 4.5 deep-scrub starts 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: 4.5 deep-scrub ok 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: 4.13 scrub starts 2024-03-30T15:14:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:36 smithi134 ceph-mon[33714]: 4.13 scrub ok 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: from='client.59174 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: osdmap e905: 8 total, 8 up, 8 in 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3702286035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: from='client.59180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: 4.5 deep-scrub starts 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: 4.5 deep-scrub ok 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: 4.13 scrub starts 2024-03-30T15:14:37.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[29876]: 4.13 scrub ok 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: from='client.59174 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: osdmap e905: 8 total, 8 up, 8 in 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3702286035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: from='client.59180 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: pgmap v1401: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: 4.5 deep-scrub starts 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: 4.5 deep-scrub ok 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: 4.13 scrub starts 2024-03-30T15:14:37.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:36 smithi028 ceph-mon[41614]: 4.13 scrub ok 2024-03-30T15:14:37.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.757+0000 7f1fe95d6700 1 -- 172.21.15.28:0/2014792618 >> v1:172.21.15.28:6790/0 conn(0x7f1fe4109de0 legacy=0x7f1fe410c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:37.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.757+0000 7f1fe95d6700 1 -- 172.21.15.28:0/2014792618 shutdown_connections 2024-03-30T15:14:37.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.757+0000 7f1fe95d6700 1 -- 172.21.15.28:0/2014792618 >> 172.21.15.28:0/2014792618 conn(0x7f1fe4076230 msgr2=0x7f1fe4076640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:37.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.757+0000 7f1fe95d6700 1 -- 172.21.15.28:0/2014792618 shutdown_connections 2024-03-30T15:14:37.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.757+0000 7f1fe95d6700 1 -- 172.21.15.28:0/2014792618 wait complete. 2024-03-30T15:14:37.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 librados: starting msgr at 2024-03-30T15:14:37.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 librados: starting objecter 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 Processor -- start 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 -- start start 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 librados: setting wanted keys 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 librados: calling monclient init 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fe41a9390 con 0x7f1fe410d980 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fe41aa5b0 con 0x7f1fe4109de0 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe95d6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1fe41ab7f0 con 0x7f1fe4101b20 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe2ffd700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f1fe4101b20 0x7f1fe4100fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34164/0 (socket says 172.21.15.28:34164) 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fe2ffd700 1 -- 172.21.15.28:0/3702286035 learned_addr learned my addr 172.21.15.28:0/3702286035 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fcbfff700 10 client.?.objecter ms_handle_connect 0x7f1fe4101b20 2024-03-30T15:14:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fcbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fcbfff700 10 client.?.objecter ms_handle_connect 0x7f1fe410d980 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fcbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.758+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764048464 0 0) 0x7f1fe41ab7f0 con 0x7f1fe4101b20 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fcc0039c0 con 0x7f1fe4101b20 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 10 client.?.objecter ms_handle_connect 0x7f1fe4109de0 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450238290 0 0) 0x7f1fe41a9390 con 0x7f1fe410d980 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe41ab7f0 con 0x7f1fe410d980 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692660653 0 0) 0x7f1fe41aa5b0 con 0x7f1fe4109de0 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe41a9390 con 0x7f1fe4109de0 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2643604231 0 0) 0x7f1fcc0039c0 con 0x7f1fe4101b20 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fe41aa5b0 con 0x7f1fe4101b20 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fd8001930 con 0x7f1fe4101b20 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2551831968 0 0) 0x7f1fe41ab7f0 con 0x7f1fe410d980 2024-03-30T15:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fcc0039c0 con 0x7f1fe410d980 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fdc001b40 con 0x7f1fe410d980 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2974996817 0 0) 0x7f1fe41a9390 con 0x7f1fe4109de0 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fe41ab7f0 con 0x7f1fe4109de0 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fd4001980 con 0x7f1fe4109de0 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2087309960 0 0) 0x7f1fe41aa5b0 con 0x7f1fe4101b20 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 >> v1:172.21.15.134:6789/0 conn(0x7f1fe4109de0 legacy=0x7f1fe41a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 >> v1:172.21.15.28:6789/0 conn(0x7f1fe410d980 legacy=0x7f1fe41a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe41aca30 con 0x7f1fe4101b20 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe41a95c0 con 0x7f1fe4101b20 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fe95d6700 10 client.59180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a9a80 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fe95d6700 1 librados: init done 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1fd8003b40 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.759+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1fd8005d10 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.760+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe406a390 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.761+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f1fd80036a0 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.761+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(904..904 src has 550..904) ==== 7561+0+0 (unknown 2318270959 0 0) 0x7f1fd8089410 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.761+0000 7f1fcbfff700 10 client.59180.objecter ms_dispatch 0x7f1fe4001550 osd_map(904..904 src has 550..904) 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.761+0000 7f1fcbfff700 3 client.59180.objecter handle_osd_map got epochs [904,904] > 0 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.761+0000 7f1fcbfff700 3 client.59180.objecter handle_osd_map decoding full epoch 904 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.762+0000 7f1fcbfff700 20 client.59180.objecter dump_active .. 0 homeless 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.762+0000 7f1fcbfff700 10 client.59180.objecter ms_handle_connect 0x7f1fcc0703d0 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.770+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1fd8005d10 con 0x7f1fe4101b20 2024-03-30T15:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:35.991+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1fe4068b30 con 0x7f1fe4101b20 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.931+0000 7f1fcbfff700 1 -- 172.21.15.28:0/3702286035 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v906) ==== 87+0+0 (unknown 1325431038 0 0) 0x7f1fd80569a0 con 0x7f1fe4101b20 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.933+0000 7f1fe95d6700 10 librados: watch_flush enter 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 10 librados: watch_flush exit 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 20 client.59180.objecter shutdown clearing up homeless session... 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 10 client.59180.objecter successfully canceled tick 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1fcc0703d0 legacy=0x7f1fcc072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 >> v1:172.21.15.28:6790/0 conn(0x7f1fe4101b20 legacy=0x7f1fe4100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 shutdown_connections 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 >> 172.21.15.28:0/3702286035 conn(0x7f1fe4076230 msgr2=0x7f1fe4111150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 shutdown_connections 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 -- 172.21.15.28:0/3702286035 wait complete. 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:36.934+0000 7f1fe95d6700 1 librados: shutdown 2024-03-30T15:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-03-30T15:14:38.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:14:38.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:14:38.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: from='client.59180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: osdmap e906: 8 total, 8 up, 8 in 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: osdmap e907: 8 total, 8 up, 8 in 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:14:38.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: from='client.59180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: osdmap e906: 8 total, 8 up, 8 in 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: osdmap e907: 8 total, 8 up, 8 in 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:38.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:37 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:14:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:14:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-30T15:14:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: from='client.59180 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: osdmap e906: 8 total, 8 up, 8 in 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: osdmap e907: 8 total, 8 up, 8 in 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:37 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- 172.21.15.28:0/3760275690 >> v1:172.21.15.28:6790/0 conn(0x7fd8b410eb50 legacy=0x7fd8b4110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- 172.21.15.28:0/3760275690 shutdown_connections 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- 172.21.15.28:0/3760275690 >> 172.21.15.28:0/3760275690 conn(0x7fd8b40fc9e0 msgr2=0x7fd8b40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- 172.21.15.28:0/3760275690 shutdown_connections 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- 172.21.15.28:0/3760275690 wait complete. 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 librados: starting msgr at 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 librados: starting objecter 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 Processor -- start 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 -- start start 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 librados: setting wanted keys 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.205+0000 7fd8bbbad700 1 librados: calling monclient init 2024-03-30T15:14:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8bbbad700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8b41a7150 con 0x7fd8b410eb50 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8bbbad700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8b41a8370 con 0x7fd8b410b040 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8bbbad700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd8b41a95b0 con 0x7fd8b4103230 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b9949700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd8b4103230 0x7fd8b410a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34180/0 (socket says 172.21.15.28:34180) 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b9949700 1 -- 172.21.15.28:0/466727703 learned_addr learned my addr 172.21.15.28:0/466727703 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter ms_handle_connect 0x7fd8b4103230 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter ms_handle_connect 0x7fd8b410eb50 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter ms_handle_connect 0x7fd8b410b040 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638505690 0 0) 0x7fd8b41a95b0 con 0x7fd8b4103230 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c0031f0 con 0x7fd8b4103230 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3960517851 0 0) 0x7fd8b41a7150 con 0x7fd8b410eb50 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b41a95b0 con 0x7fd8b410eb50 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758790349 0 0) 0x7fd8b41a8370 con 0x7fd8b410b040 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b41a7150 con 0x7fd8b410b040 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 321882053 0 0) 0x7fd89c0031f0 con 0x7fd8b4103230 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8b41a8370 con 0x7fd8b4103230 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8a4001ce0 con 0x7fd8b4103230 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3891698795 0 0) 0x7fd8b41a95b0 con 0x7fd8b410eb50 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c0031f0 con 0x7fd8b410eb50 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8ac004440 con 0x7fd8b410eb50 2024-03-30T15:14:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3113568013 0 0) 0x7fd8b41a7150 con 0x7fd8b410b040 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8b41a95b0 con 0x7fd8b410b040 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8a8001830 con 0x7fd8b410b040 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4293582624 0 0) 0x7fd8b41a8370 con 0x7fd8b4103230 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.134:6789/0 conn(0x7fd8b410b040 legacy=0x7fd8b41a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.206+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6789/0 conn(0x7fd8b410eb50 legacy=0x7fd8b41a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b41aa7f0 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b41a7380 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8bbbad700 10 client.59186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8b41a7840 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8bbbad700 1 librados: init done 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd8a4002020 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd8a4006120 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.207+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd8a0005100 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.208+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd8a4003cb0 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(907..907 src has 550..907) ==== 7561+0+0 (unknown 2607323480 0 0) 0x7fd8a40897c0 con 0x7fd8b4103230 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 osd_map(907..907 src has 550..907) 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 3 client.59186.objecter handle_osd_map got epochs [907,907] > 0 2024-03-30T15:14:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 3 client.59186.objecter handle_osd_map decoding full epoch 907 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 20 client.59186.objecter dump_active .. 0 homeless 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v907) ==== 54+0+16 (unknown 198880407 0 2420000192) 0x7fd8a4089c10 con 0x7fd8b4103230 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd89c0703e0 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 1 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 1 2024-03-30T15:14:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 0 1 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 4 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 10 client.59186.objecter _send_command 1 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8a0004b70 con 0x7fd8a0004730 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.209+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a0004730 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.210+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a0004b70 con 0x7fd8a0004730 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.210+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 1: 0 ) 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.210+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 1 = system:0 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.210+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 4 2024-03-30T15:14:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.210+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 0 1 2024-03-30T15:14:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 2 2024-03-30T15:14:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 2 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a00053e0 osd=0 3 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 0 2 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 4 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 10 client.59186.objecter _send_command 2 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.227+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a000be00 con 0x7fd8a0004730 2024-03-30T15:14:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a000be00 con 0x7fd8a0004730 2024-03-30T15:14:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 4 2024-03-30T15:14:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 0 2 2024-03-30T15:14:38.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 3 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 3 2024-03-30T15:14:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 1 3 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 4 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 10 client.59186.objecter _send_command 3 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6805/1850313547 -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd8a00022a0 con 0x7fd8a0004310 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.229+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a0004310 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.230+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a00022a0 con 0x7fd8a0004310 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.230+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 3: 0 ) 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.230+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 3 = system:0 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.230+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 4 2024-03-30T15:14:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.230+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 1 3 2024-03-30T15:14:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 4 2024-03-30T15:14:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 4 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0004130 osd=1 3 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 1 4 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 4 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 10 client.59186.objecter _send_command 4 2024-03-30T15:14:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.248+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6805/1850313547 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0001720 con 0x7fd8a0004310 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0001720 con 0x7fd8a0004310 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 4 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 1 4 2024-03-30T15:14:38.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-03-30T15:14:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 5 2024-03-30T15:14:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 5 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 2 5 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 4 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 10 client.59186.objecter _send_command 5 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6801/2294873818 -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd8a00154f0 con 0x7fd8a00119e0 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.250+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a00119e0 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.251+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a00154f0 con 0x7fd8a00119e0 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.251+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 5: 0 ) 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.251+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 5 = system:0 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.251+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 4 2024-03-30T15:14:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.251+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 2 5 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 6 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 6 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0001980 osd=2 3 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 2 6 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 4 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 10 client.59186.objecter _send_command 6 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.267+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6801/2294873818 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a000c510 con 0x7fd8a00119e0 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a000c510 con 0x7fd8a00119e0 2024-03-30T15:14:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 4 2024-03-30T15:14:38.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 2 6 2024-03-30T15:14:38.271 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 7 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 7 2024-03-30T15:14:38.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 3 7 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 4 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 10 client.59186.objecter _send_command 7 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.269+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6813/1506157250 -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd8a00173e0 con 0x7fd8a0016820 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a0016820 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a00173e0 con 0x7fd8a0016820 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 7: 0 ) 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 7 = system:0 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 4 2024-03-30T15:14:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.270+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 3 7 2024-03-30T15:14:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 8 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 8 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a000c7a0 osd=3 3 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 3 8 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 4 2024-03-30T15:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 10 client.59186.objecter _send_command 8 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.287+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.28:6813/1506157250 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0017d90 con 0x7fd8a0016820 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0017d90 con 0x7fd8a0016820 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 4 2024-03-30T15:14:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 3 8 2024-03-30T15:14:38.291 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-03-30T15:14:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 9 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 9 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 4 9 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 4 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 10 client.59186.objecter _send_command 9 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6800/4077237787 -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd8a001eca0 con 0x7fd8a001b050 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.289+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a001b050 2024-03-30T15:14:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.290+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a001eca0 con 0x7fd8a001b050 2024-03-30T15:14:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.290+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 9: 0 ) 2024-03-30T15:14:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.290+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 9 = system:0 2024-03-30T15:14:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.290+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 4 2024-03-30T15:14:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.290+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 4 9 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 10 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 10 2024-03-30T15:14:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0017d90 osd=4 3 2024-03-30T15:14:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 4 10 2024-03-30T15:14:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 4 2024-03-30T15:14:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 10 client.59186.objecter _send_command 10 2024-03-30T15:14:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.307+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6800/4077237787 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0010b10 con 0x7fd8a001b050 2024-03-30T15:14:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0010b10 con 0x7fd8a001b050 2024-03-30T15:14:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 4 2024-03-30T15:14:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 4 10 2024-03-30T15:14:38.311 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-03-30T15:14:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 11 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 11 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 5 11 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 4 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 10 client.59186.objecter _send_command 11 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.309+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6804/3668151179 -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd8a0020430 con 0x7fd8a0015e70 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a0015e70 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a0020430 con 0x7fd8a0015e70 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 11: 0 ) 2024-03-30T15:14:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 11 = system:0 2024-03-30T15:14:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 4 2024-03-30T15:14:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.310+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 5 11 2024-03-30T15:14:38.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 12 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 12 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0010b10 osd=5 3 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 5 12 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 4 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 10 client.59186.objecter _send_command 12 2024-03-30T15:14:38.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.327+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6804/3668151179 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0020430 con 0x7fd8a0015e70 2024-03-30T15:14:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0020430 con 0x7fd8a0015e70 2024-03-30T15:14:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 4 2024-03-30T15:14:38.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 5 12 2024-03-30T15:14:38.331 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 13 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 13 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 6 13 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 4 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 10 client.59186.objecter _send_command 13 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.329+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6808/2135224194 -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd8a0027620 con 0x7fd8a0016410 2024-03-30T15:14:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.330+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a0016410 2024-03-30T15:14:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.331+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a0027620 con 0x7fd8a0016410 2024-03-30T15:14:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.331+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 13: 0 ) 2024-03-30T15:14:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.331+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 13 = system:0 2024-03-30T15:14:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.331+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 4 2024-03-30T15:14:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.331+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 6 13 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 14 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 14 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0020430 osd=6 3 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 6 14 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 4 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 10 client.59186.objecter _send_command 14 2024-03-30T15:14:38.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.348+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6808/2135224194 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0028fd0 con 0x7fd8a0016410 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0028fd0 con 0x7fd8a0016410 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 4 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 6 14 2024-03-30T15:14:38.352 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-03-30T15:14:38.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:14:38.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 15 2024-03-30T15:14:38.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 15 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 7 15 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 4 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 10 client.59186.objecter _send_command 15 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.350+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6812/3758190298 -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd8a0029b50 con 0x7fd8a00296f0 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.351+0000 7fd8b2ffd700 10 client.59186.objecter ms_handle_connect 0x7fd8a00296f0 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.352+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd8a0029b50 con 0x7fd8a00296f0 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.352+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 15: 0 ) 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.352+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 15 = system:0 2024-03-30T15:14:38.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.352+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 4 2024-03-30T15:14:38.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.352+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 7 15 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 10 client.59186.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign -1 16 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 20 client.59186.objecter _get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_remove -1 16 2024-03-30T15:14:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 20 client.59186.objecter get_session s=0x7fd8a0028fd0 osd=7 3 2024-03-30T15:14:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 15 client.59186.objecter _session_command_op_assign 7 16 2024-03-30T15:14:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.368+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 4 2024-03-30T15:14:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.369+0000 7fd8bbbad700 10 client.59186.objecter _send_command 16 2024-03-30T15:14:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.369+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 --> v1:172.21.15.134:6812/3758190298 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fd8a0029b50 con 0x7fd8a00296f0 2024-03-30T15:14:38.371 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.370+0000 7fd8b2ffd700 1 -- 172.21.15.28:0/466727703 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd8a0029b50 con 0x7fd8a00296f0 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.370+0000 7fd8b2ffd700 10 client.59186.objecter ms_dispatch 0x7fd8b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.370+0000 7fd8b2ffd700 10 client.59186.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.370+0000 7fd8b2ffd700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 4 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.370+0000 7fd8b2ffd700 15 client.59186.objecter _session_command_op_remove 7 16 2024-03-30T15:14:38.372 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-03-30T15:14:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 librados: watch_flush enter 2024-03-30T15:14:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 librados: watch_flush exit 2024-03-30T15:14:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.0 2024-03-30T15:14:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6809/2927075629 conn(0x7fd8a0004730 legacy=0x7fd8a0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a00053e0 osd=0 1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6805/1850313547 conn(0x7fd8a0004310 legacy=0x7fd8a000f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0004130 osd=1 1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.2 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6801/2294873818 conn(0x7fd8a00119e0 legacy=0x7fd8a0013dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0001980 osd=2 1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.3 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6813/1506157250 conn(0x7fd8a0016820 legacy=0x7fd8a0016cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a000c7a0 osd=3 1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.4 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.134:6800/4077237787 conn(0x7fd8a001b050 legacy=0x7fd8a001d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0017d90 osd=4 1 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.5 2024-03-30T15:14:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8ba14a700 1 -- 172.21.15.28:0/466727703 reap_dead start 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.134:6804/3668151179 conn(0x7fd8a0015e70 legacy=0x7fd8a001eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8ba14a700 1 -- 172.21.15.28:0/466727703 reap_dead start 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0010b10 osd=5 1 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.6 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.134:6808/2135224194 conn(0x7fd8a0016410 legacy=0x7fd8a0025d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0020430 osd=6 1 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter close_session for osd.7 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.134:6812/3758190298 conn(0x7fd8a00296f0 legacy=0x7fd8a0027800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8ba14a700 1 -- 172.21.15.28:0/466727703 reap_dead start 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter put_session s=0x7fd8a0028fd0 osd=7 1 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 20 client.59186.objecter shutdown clearing up homeless session... 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 10 client.59186.objecter successfully canceled tick 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd89c0703e0 legacy=0x7fd89c0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> v1:172.21.15.28:6790/0 conn(0x7fd8b4103230 legacy=0x7fd8b410a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 shutdown_connections 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 >> 172.21.15.28:0/466727703 conn(0x7fd8b40fc9e0 msgr2=0x7fd8b40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 shutdown_connections 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 -- 172.21.15.28:0/466727703 wait complete. 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:38.371+0000 7fd8bbbad700 1 librados: shutdown 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T15:14:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T15:14:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:38 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/466727703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:38 smithi028 ceph-mon[29876]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-30T15:14:39.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:38 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/466727703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:39.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:38 smithi028 ceph-mon[41614]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-30T15:14:39.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:38 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/466727703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-30T15:14:39.433 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:38 smithi134 ceph-mon[33714]: pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-30T15:14:39.848 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T15:14:39.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:14:40.052 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:14:40.053 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:14:40.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-03-30T15:14:40.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset noscrub 2024-03-30T15:14:41.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:40 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:41.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:41.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:41 smithi028 ceph-mon[29876]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:14:41.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:41 smithi028 ceph-mon[41614]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:14:41.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:41 smithi134 ceph-mon[33714]: pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:14:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:42 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:42 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:42.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:42 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:42.717 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:42 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:43.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[29876]: pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-30T15:14:43.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[29876]: osdmap e908: 8 total, 8 up, 8 in 2024-03-30T15:14:43.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[29876]: osdmap e909: 8 total, 8 up, 8 in 2024-03-30T15:14:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:43 smithi134 ceph-mon[33714]: pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-30T15:14:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:43 smithi134 ceph-mon[33714]: osdmap e908: 8 total, 8 up, 8 in 2024-03-30T15:14:43.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:43 smithi134 ceph-mon[33714]: osdmap e909: 8 total, 8 up, 8 in 2024-03-30T15:14:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[41614]: pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-30T15:14:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[41614]: osdmap e908: 8 total, 8 up, 8 in 2024-03-30T15:14:43.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:43 smithi028 ceph-mon[41614]: osdmap e909: 8 total, 8 up, 8 in 2024-03-30T15:14:44.632 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:14:44.632 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:14:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.509+0000 7f35e8c62700 1 -- 172.21.15.28:0/498447072 >> v1:172.21.15.28:6789/0 conn(0x7f35e410afa0 legacy=0x7f35e410d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.509+0000 7f35e8c62700 1 -- 172.21.15.28:0/498447072 shutdown_connections 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.509+0000 7f35e8c62700 1 -- 172.21.15.28:0/498447072 >> 172.21.15.28:0/498447072 conn(0x7f35e40fc9e0 msgr2=0x7f35e40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.509+0000 7f35e8c62700 1 -- 172.21.15.28:0/498447072 shutdown_connections 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 -- 172.21.15.28:0/498447072 wait complete. 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 librados: starting msgr at 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 librados: starting objecter 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 Processor -- start 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 -- start start 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 librados: setting wanted keys 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.510+0000 7f35e8c62700 1 librados: calling monclient init 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35e8c62700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35e41a9350 con 0x7f35e4103190 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35e8c62700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35e41aa570 con 0x7f35e410afa0 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35e8c62700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35e41ab7b0 con 0x7f35e410eab0 2024-03-30T15:14:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35e259c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35e4103190 0x7f35e410a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46622/0 (socket says 172.21.15.28:46622) 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35e259c700 1 -- 172.21.15.28:0/2334064084 learned_addr learned my addr 172.21.15.28:0/2334064084 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter ms_handle_connect 0x7f35e4103190 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter ms_handle_connect 0x7f35e410afa0 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274663187 0 0) 0x7f35e41a9350 con 0x7f35e4103190 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35cc0039c0 con 0x7f35e4103190 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter ms_handle_connect 0x7f35e410eab0 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3278112667 0 0) 0x7f35e41aa570 con 0x7f35e410afa0 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e41a9350 con 0x7f35e410afa0 2024-03-30T15:14:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114314118 0 0) 0x7f35e41ab7b0 con 0x7f35e410eab0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e41aa570 con 0x7f35e410eab0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.511+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3896695194 0 0) 0x7f35cc0039c0 con 0x7f35e4103190 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e41ab7b0 con 0x7f35e4103190 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35d40013d0 con 0x7f35e4103190 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1126488374 0 0) 0x7f35e41a9350 con 0x7f35e410afa0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35cc0039c0 con 0x7f35e410afa0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35d80044c0 con 0x7f35e410afa0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3248472520 0 0) 0x7f35e41aa570 con 0x7f35e410eab0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e41a9350 con 0x7f35e410eab0 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 657493835 0 0) 0x7f35e41ab7b0 con 0x7f35e4103190 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 >> v1:172.21.15.28:6790/0 conn(0x7f35e410eab0 legacy=0x7f35e41a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 >> v1:172.21.15.134:6789/0 conn(0x7f35e410afa0 legacy=0x7f35e41a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e41ac9f0 con 0x7f35e4103190 2024-03-30T15:14:44.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35e41aa7a0 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35e8c62700 10 client.25989.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f35e41aac60 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35e8c62700 1 librados: init done 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.512+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f35d4001630 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.513+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35d4005c40 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.513+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c4005120 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.514+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f35d4004040 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(909..909 src has 550..909) ==== 7561+0+0 (unknown 3118835766 0 0) 0x7f35d40894d0 con 0x7f35e4103190 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 10 client.25989.objecter ms_dispatch 0x7f35e4001550 osd_map(909..909 src has 550..909) 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 3 client.25989.objecter handle_osd_map got epochs [909,909] > 0 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 3 client.25989.objecter handle_osd_map decoding full epoch 909 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 20 client.25989.objecter dump_active .. 0 homeless 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.515+0000 7f35d37fe700 10 client.25989.objecter ms_handle_connect 0x7f35cc070410 2024-03-30T15:14:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.520+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f35d4005c20 con 0x7f35e4103190 2024-03-30T15:14:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.730+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35c4000b90 con 0x7f35cc070410 2024-03-30T15:14:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.742+0000 7f35d37fe700 1 -- 172.21.15.28:0/2334064084 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+452531 (unknown 2965378022 0 2211923935) 0x7f35c4000b90 con 0x7f35cc070410 2024-03-30T15:14:44.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:14:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 10 librados: watch_flush enter 2024-03-30T15:14:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 10 librados: watch_flush exit 2024-03-30T15:14:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 20 client.25989.objecter shutdown clearing up homeless session... 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 10 client.25989.objecter successfully canceled tick 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 >> v1:172.21.15.28:6800/3011431093 conn(0x7f35cc070410 legacy=0x7f35cc0728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 >> v1:172.21.15.28:6789/0 conn(0x7f35e4103190 legacy=0x7f35e410a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 shutdown_connections 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.747+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 >> 172.21.15.28:0/2334064084 conn(0x7f35e40fc9e0 msgr2=0x7f35e40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.748+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 shutdown_connections 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.748+0000 7f35e8c62700 1 -- 172.21.15.28:0/2334064084 wait complete. 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:44.748+0000 7f35e8c62700 1 librados: shutdown 2024-03-30T15:14:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:14:45.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-30T15:14:45.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:45.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:45 smithi134 ceph-mon[33714]: pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:14:45.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:45 smithi028 ceph-mon[29876]: pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:14:45.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:45 smithi028 ceph-mon[41614]: pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-30T15:14:46.090 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1409,"stamp":"2024-03-30T15:14:44.214822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2333,"num_read_kb":2085,"num_write":729,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2310,"ondisk_log_size":2310,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":668348,"kb_used_data":63180,"kb_used_omap":108,"kb_used_meta":602131,"kb_avail":749292868,"statfs":{"total":767960285184,"available":767275896832,"internally_reserved":0,"allocated":64696320,"data_stored":33515190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111128,"internal_metadata":616582632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.623276"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":2034,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965474+0000","last_change":"2024-03-30T15:13:55.602346+0000","last_active":"2024-03-30T15:14:42.965474+0000","last_peered":"2024-03-30T15:14:42.965474+0000","last_clean":"2024-03-30T15:14:42.965474+0000","last_became_active":"2024-03-30T15:13:55.601989+0000","last_became_peered":"2024-03-30T15:13:55.601989+0000","last_unstale":"2024-03-30T15:14:42.965474+0000","last_undegraded":"2024-03-30T15:14:42.965474+0000","last_fullsized":"2024-03-30T15:14:42.965474+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:18.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:18.837242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967812+0000","last_change":"2024-03-30T15:14:00.634325+0000","last_active":"2024-03-30T15:14:42.967812+0000","last_peered":"2024-03-30T15:14:42.967812+0000","last_clean":"2024-03-30T15:14:42.967812+0000","last_became_active":"2024-03-30T15:14:00.633295+0000","last_became_peered":"2024-03-30T15:14:00.633295+0000","last_unstale":"2024-03-30T15:14:42.967812+0000","last_undegraded":"2024-03-30T15:14:42.967812+0000","last_fullsized":"2024-03-30T15:14:42.967812+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:19.524864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:19.524864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"475'20","reported_seq":1673,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877046+0000","last_change":"2024-03-30T15:14:20.756235+0000","last_active":"2024-03-30T15:14:37.877046+0000","last_peered":"2024-03-30T15:14:37.877046+0000","last_clean":"2024-03-30T15:14:37.877046+0000","last_became_active":"2024-03-30T15:13:03.535865+0000","last_became_peered":"2024-03-30T15:13:03.535865+0000","last_unstale":"2024-03-30T15:14:37.877046+0000","last_undegraded":"2024-03-30T15:14:37.877046+0000","last_fullsized":"2024-03-30T15:14:37.877046+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:14:20.756053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.756053+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:22.102152+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1260,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053238+0000","last_change":"2024-03-30T15:14:28.185811+0000","last_active":"2024-03-30T15:14:39.053238+0000","last_peered":"2024-03-30T15:14:39.053238+0000","last_clean":"2024-03-30T15:14:39.053238+0000","last_became_active":"2024-03-30T15:13:28.625255+0000","last_became_peered":"2024-03-30T15:13:28.625255+0000","last_unstale":"2024-03-30T15:14:39.053238+0000","last_undegraded":"2024-03-30T15:14:39.053238+0000","last_fullsized":"2024-03-30T15:14:39.053238+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:28.185698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:14:28.185698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"833'11","reported_seq":1001,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968805+0000","last_change":"2024-03-30T15:13:46.704186+0000","last_active":"2024-03-30T15:14:42.968805+0000","last_peered":"2024-03-30T15:14:42.968805+0000","last_clean":"2024-03-30T15:14:42.968805+0000","last_became_active":"2024-03-30T15:13:03.543934+0000","last_became_peered":"2024-03-30T15:13:03.543934+0000","last_unstale":"2024-03-30T15:14:42.968805+0000","last_undegraded":"2024-03-30T15:14:42.968805+0000","last_fullsized":"2024-03-30T15:14:42.968805+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'11","last_scrub_stamp":"2024-03-30T15:13:46.704092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.704092+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.405370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1685,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877680+0000","last_change":"2024-03-30T15:13:52.597484+0000","last_active":"2024-03-30T15:14:37.877680+0000","last_peered":"2024-03-30T15:14:37.877680+0000","last_clean":"2024-03-30T15:14:37.877680+0000","last_became_active":"2024-03-30T15:13:52.596681+0000","last_became_peered":"2024-03-30T15:13:52.596681+0000","last_unstale":"2024-03-30T15:14:37.877680+0000","last_undegraded":"2024-03-30T15:14:37.877680+0000","last_fullsized":"2024-03-30T15:14:37.877680+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:49.842503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:49.842503+0000","last_clean_scrub_stamp":"2024-03-30T15:11:49.842503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":1024,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595136+0000","last_change":"2024-03-30T15:14:12.850623+0000","last_active":"2024-03-30T15:14:42.595136+0000","last_peered":"2024-03-30T15:14:42.595136+0000","last_clean":"2024-03-30T15:14:42.595136+0000","last_became_active":"2024-03-30T15:13:19.287238+0000","last_became_peered":"2024-03-30T15:13:19.287238+0000","last_unstale":"2024-03-30T15:14:42.595136+0000","last_undegraded":"2024-03-30T15:14:42.595136+0000","last_fullsized":"2024-03-30T15:14:42.595136+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:12.850408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.850408+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:28.008340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1650,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968959+0000","last_change":"2024-03-30T15:14:15.617984+0000","last_active":"2024-03-30T15:14:42.968959+0000","last_peered":"2024-03-30T15:14:42.968959+0000","last_clean":"2024-03-30T15:14:42.968959+0000","last_became_active":"2024-03-30T15:13:19.284720+0000","last_became_peered":"2024-03-30T15:13:19.284720+0000","last_unstale":"2024-03-30T15:14:42.968959+0000","last_undegraded":"2024-03-30T15:14:42.968959+0000","last_fullsized":"2024-03-30T15:14:42.968959+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.617863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.617863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:26.436478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"475'31","reported_seq":80,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877317+0000","last_change":"2024-03-30T15:14:09.740147+0000","last_active":"2024-03-30T15:14:37.877317+0000","last_peered":"2024-03-30T15:14:37.877317+0000","last_clean":"2024-03-30T15:14:37.877317+0000","last_became_active":"2024-03-30T15:14:09.722049+0000","last_became_peered":"2024-03-30T15:14:09.722049+0000","last_unstale":"2024-03-30T15:14:37.877317+0000","last_undegraded":"2024-03-30T15:14:37.877317+0000","last_fullsized":"2024-03-30T15:14:37.877317+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:13:06.419015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.419015+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1752,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967588+0000","last_change":"2024-03-30T15:14:19.617578+0000","last_active":"2024-03-30T15:14:42.967588+0000","last_peered":"2024-03-30T15:14:42.967588+0000","last_clean":"2024-03-30T15:14:42.967588+0000","last_became_active":"2024-03-30T15:13:03.529657+0000","last_became_peered":"2024-03-30T15:13:03.529657+0000","last_unstale":"2024-03-30T15:14:42.967588+0000","last_undegraded":"2024-03-30T15:14:42.967588+0000","last_fullsized":"2024-03-30T15:14:42.967588+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:19.617498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.617498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:23.958547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1752,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596813+0000","last_change":"2024-03-30T15:13:30.478717+0000","last_active":"2024-03-30T15:14:42.596813+0000","last_peered":"2024-03-30T15:14:42.596813+0000","last_clean":"2024-03-30T15:14:42.596813+0000","last_became_active":"2024-03-30T15:13:28.634488+0000","last_became_peered":"2024-03-30T15:13:28.634488+0000","last_unstale":"2024-03-30T15:14:42.596813+0000","last_undegraded":"2024-03-30T15:14:42.596813+0000","last_fullsized":"2024-03-30T15:14:42.596813+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:30.478588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.478588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.374563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":97,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.597739+0000","last_change":"2024-03-30T15:14:33.036325+0000","last_active":"2024-03-30T15:14:37.597739+0000","last_peered":"2024-03-30T15:14:37.597739+0000","last_clean":"2024-03-30T15:14:37.597739+0000","last_became_active":"2024-03-30T15:13:52.592738+0000","last_became_peered":"2024-03-30T15:13:52.592738+0000","last_unstale":"2024-03-30T15:14:37.597739+0000","last_undegraded":"2024-03-30T15:14:37.597739+0000","last_fullsized":"2024-03-30T15:14:37.597739+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:33.036198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.036198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:36.457410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595673+0000","last_change":"2024-03-30T15:13:31.436323+0000","last_active":"2024-03-30T15:14:42.595673+0000","last_peered":"2024-03-30T15:14:42.595673+0000","last_clean":"2024-03-30T15:14:42.595673+0000","last_became_active":"2024-03-30T15:13:28.634783+0000","last_became_peered":"2024-03-30T15:13:28.634783+0000","last_unstale":"2024-03-30T15:14:42.595673+0000","last_undegraded":"2024-03-30T15:14:42.595673+0000","last_fullsized":"2024-03-30T15:14:42.595673+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.436194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.436194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"475'24","reported_seq":1716,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877813+0000","last_change":"2024-03-30T15:14:14.688704+0000","last_active":"2024-03-30T15:14:37.877813+0000","last_peered":"2024-03-30T15:14:37.877813+0000","last_clean":"2024-03-30T15:14:37.877813+0000","last_became_active":"2024-03-30T15:13:03.539403+0000","last_became_peered":"2024-03-30T15:13:03.539403+0000","last_unstale":"2024-03-30T15:14:37.877813+0000","last_undegraded":"2024-03-30T15:14:37.877813+0000","last_fullsized":"2024-03-30T15:14:37.877813+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:14.688571+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.688571+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:20.327879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1648,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876881+0000","last_change":"2024-03-30T15:13:41.719907+0000","last_active":"2024-03-30T15:14:37.876881+0000","last_peered":"2024-03-30T15:14:37.876881+0000","last_clean":"2024-03-30T15:14:37.876881+0000","last_became_active":"2024-03-30T15:13:19.284770+0000","last_became_peered":"2024-03-30T15:13:19.284770+0000","last_unstale":"2024-03-30T15:14:37.876881+0000","last_undegraded":"2024-03-30T15:14:37.876881+0000","last_fullsized":"2024-03-30T15:14:37.876881+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:41.719797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:13:41.719797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.319763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1737,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596893+0000","last_change":"2024-03-30T15:14:17.900734+0000","last_active":"2024-03-30T15:14:42.596893+0000","last_peered":"2024-03-30T15:14:42.596893+0000","last_clean":"2024-03-30T15:14:42.596893+0000","last_became_active":"2024-03-30T15:13:03.533222+0000","last_became_peered":"2024-03-30T15:13:03.533222+0000","last_unstale":"2024-03-30T15:14:42.596893+0000","last_undegraded":"2024-03-30T15:14:42.596893+0000","last_fullsized":"2024-03-30T15:14:42.596893+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.900644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.900644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:42.201720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1644,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968763+0000","last_change":"2024-03-30T15:14:32.445558+0000","last_active":"2024-03-30T15:14:42.968763+0000","last_peered":"2024-03-30T15:14:42.968763+0000","last_clean":"2024-03-30T15:14:42.968763+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:14:42.968763+0000","last_undegraded":"2024-03-30T15:14:42.968763+0000","last_fullsized":"2024-03-30T15:14:42.968763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:32.445474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:32.445474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:41.549467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":1024,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595337+0000","last_change":"2024-03-30T15:13:46.902799+0000","last_active":"2024-03-30T15:14:42.595337+0000","last_peered":"2024-03-30T15:14:42.595337+0000","last_clean":"2024-03-30T15:14:42.595337+0000","last_became_active":"2024-03-30T15:13:19.288713+0000","last_became_peered":"2024-03-30T15:13:19.288713+0000","last_unstale":"2024-03-30T15:14:42.595337+0000","last_undegraded":"2024-03-30T15:14:42.595337+0000","last_fullsized":"2024-03-30T15:14:42.595337+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:46.902697+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.902697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:10.399599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596800+0000","last_change":"2024-03-30T15:13:32.456654+0000","last_active":"2024-03-30T15:14:42.596800+0000","last_peered":"2024-03-30T15:14:42.596800+0000","last_clean":"2024-03-30T15:14:42.596800+0000","last_became_active":"2024-03-30T15:13:19.288815+0000","last_became_peered":"2024-03-30T15:13:19.288815+0000","last_unstale":"2024-03-30T15:14:42.596800+0000","last_undegraded":"2024-03-30T15:14:42.596800+0000","last_fullsized":"2024-03-30T15:14:42.596800+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:32.456571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:13:32.456571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"475'34","reported_seq":1801,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596908+0000","last_change":"2024-03-30T15:13:07.392582+0000","last_active":"2024-03-30T15:14:42.596908+0000","last_peered":"2024-03-30T15:14:42.596908+0000","last_clean":"2024-03-30T15:14:42.596908+0000","last_became_active":"2024-03-30T15:13:03.503223+0000","last_became_peered":"2024-03-30T15:13:03.503223+0000","last_unstale":"2024-03-30T15:14:42.596908+0000","last_undegraded":"2024-03-30T15:14:42.596908+0000","last_fullsized":"2024-03-30T15:14:42.596908+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_clean_scrub_stamp":"2024-03-30T15:13:07.392488+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1669,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595871+0000","last_change":"2024-03-30T15:14:00.624520+0000","last_active":"2024-03-30T15:14:42.595871+0000","last_peered":"2024-03-30T15:14:42.595871+0000","last_clean":"2024-03-30T15:14:42.595871+0000","last_became_active":"2024-03-30T15:14:00.624194+0000","last_became_peered":"2024-03-30T15:14:00.624194+0000","last_unstale":"2024-03-30T15:14:42.595871+0000","last_undegraded":"2024-03-30T15:14:42.595871+0000","last_fullsized":"2024-03-30T15:14:42.595871+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:40.486895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.486895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:07.691772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"883'17","reported_seq":72,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877520+0000","last_change":"2024-03-30T15:14:09.733902+0000","last_active":"2024-03-30T15:14:37.877520+0000","last_peered":"2024-03-30T15:14:37.877520+0000","last_clean":"2024-03-30T15:14:37.877520+0000","last_became_active":"2024-03-30T15:14:09.730624+0000","last_became_peered":"2024-03-30T15:14:09.730624+0000","last_unstale":"2024-03-30T15:14:37.877520+0000","last_undegraded":"2024-03-30T15:14:37.877520+0000","last_fullsized":"2024-03-30T15:14:37.877520+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'11","last_scrub_stamp":"2024-03-30T15:11:39.928695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:39.928695+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1717,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595892+0000","last_change":"2024-03-30T15:13:33.913218+0000","last_active":"2024-03-30T15:14:42.595892+0000","last_peered":"2024-03-30T15:14:42.595892+0000","last_clean":"2024-03-30T15:14:42.595892+0000","last_became_active":"2024-03-30T15:13:28.623165+0000","last_became_peered":"2024-03-30T15:13:28.623165+0000","last_unstale":"2024-03-30T15:14:42.595892+0000","last_undegraded":"2024-03-30T15:14:42.595892+0000","last_fullsized":"2024-03-30T15:14:42.595892+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:33.913138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:13:33.913138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.588570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":991,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877376+0000","last_change":"2024-03-30T15:13:55.607087+0000","last_active":"2024-03-30T15:14:37.877376+0000","last_peered":"2024-03-30T15:14:37.877376+0000","last_clean":"2024-03-30T15:14:37.877376+0000","last_became_active":"2024-03-30T15:13:55.606453+0000","last_became_peered":"2024-03-30T15:13:55.606453+0000","last_unstale":"2024-03-30T15:14:37.877376+0000","last_undegraded":"2024-03-30T15:14:37.877376+0000","last_fullsized":"2024-03-30T15:14:37.877376+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:11:48.884603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:11:48.884603+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1476,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595225+0000","last_change":"2024-03-30T15:13:46.195107+0000","last_active":"2024-03-30T15:14:37.595225+0000","last_peered":"2024-03-30T15:14:37.595225+0000","last_clean":"2024-03-30T15:14:37.595225+0000","last_became_active":"2024-03-30T15:13:28.646505+0000","last_became_peered":"2024-03-30T15:13:28.646505+0000","last_unstale":"2024-03-30T15:14:37.595225+0000","last_undegraded":"2024-03-30T15:14:37.595225+0000","last_fullsized":"2024-03-30T15:14:37.595225+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:46.195015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.195015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.641593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1639,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876537+0000","last_change":"2024-03-30T15:14:22.781690+0000","last_active":"2024-03-30T15:14:37.876537+0000","last_peered":"2024-03-30T15:14:37.876537+0000","last_clean":"2024-03-30T15:14:37.876537+0000","last_became_active":"2024-03-30T15:13:03.538990+0000","last_became_peered":"2024-03-30T15:13:03.538990+0000","last_unstale":"2024-03-30T15:14:37.876537+0000","last_undegraded":"2024-03-30T15:14:37.876537+0000","last_fullsized":"2024-03-30T15:14:37.876537+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.781602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.781602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.412116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"475'38","reported_seq":1791,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966839+0000","last_change":"2024-03-30T15:14:00.651598+0000","last_active":"2024-03-30T15:14:42.966839+0000","last_peered":"2024-03-30T15:14:42.966839+0000","last_clean":"2024-03-30T15:14:42.966839+0000","last_became_active":"2024-03-30T15:14:00.630774+0000","last_became_peered":"2024-03-30T15:14:00.630774+0000","last_unstale":"2024-03-30T15:14:42.966839+0000","last_undegraded":"2024-03-30T15:14:42.966839+0000","last_fullsized":"2024-03-30T15:14:42.966839+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:12:17.847789+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:12:17.847789+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1710,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968465+0000","last_change":"2024-03-30T15:13:40.071287+0000","last_active":"2024-03-30T15:14:42.968465+0000","last_peered":"2024-03-30T15:14:42.968465+0000","last_clean":"2024-03-30T15:14:42.968465+0000","last_became_active":"2024-03-30T15:13:28.632229+0000","last_became_peered":"2024-03-30T15:13:28.632229+0000","last_unstale":"2024-03-30T15:14:42.968465+0000","last_undegraded":"2024-03-30T15:14:42.968465+0000","last_fullsized":"2024-03-30T15:14:42.968465+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:13:40.071196+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.071196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:07.554377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":62,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876645+0000","last_change":"2024-03-30T15:14:09.723294+0000","last_active":"2024-03-30T15:14:37.876645+0000","last_peered":"2024-03-30T15:14:37.876645+0000","last_clean":"2024-03-30T15:14:37.876645+0000","last_became_active":"2024-03-30T15:14:09.722923+0000","last_became_peered":"2024-03-30T15:14:09.722923+0000","last_unstale":"2024-03-30T15:14:37.876645+0000","last_undegraded":"2024-03-30T15:14:37.876645+0000","last_fullsized":"2024-03-30T15:14:37.876645+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_clean_scrub_stamp":"2024-03-30T15:11:48.594615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1023,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595651+0000","last_change":"2024-03-30T15:14:33.930988+0000","last_active":"2024-03-30T15:14:42.595651+0000","last_peered":"2024-03-30T15:14:42.595651+0000","last_clean":"2024-03-30T15:14:42.595651+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:14:42.595651+0000","last_undegraded":"2024-03-30T15:14:42.595651+0000","last_fullsized":"2024-03-30T15:14:42.595651+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:33.930858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.930858+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:55.942461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1482,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595917+0000","last_change":"2024-03-30T15:14:23.185170+0000","last_active":"2024-03-30T15:14:37.595917+0000","last_peered":"2024-03-30T15:14:37.595917+0000","last_clean":"2024-03-30T15:14:37.595917+0000","last_became_active":"2024-03-30T15:13:19.290216+0000","last_became_peered":"2024-03-30T15:13:19.290216+0000","last_unstale":"2024-03-30T15:14:37.595917+0000","last_undegraded":"2024-03-30T15:14:37.595917+0000","last_fullsized":"2024-03-30T15:14:37.595917+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.185098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.185098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:23.258738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"899'12","reported_seq":1887,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594688+0000","last_change":"2024-03-30T15:14:28.186777+0000","last_active":"2024-03-30T15:14:42.594688+0000","last_peered":"2024-03-30T15:14:42.594688+0000","last_clean":"2024-03-30T15:14:42.594688+0000","last_became_active":"2024-03-30T15:13:03.504632+0000","last_became_peered":"2024-03-30T15:13:03.504632+0000","last_unstale":"2024-03-30T15:14:42.594688+0000","last_undegraded":"2024-03-30T15:14:42.594688+0000","last_fullsized":"2024-03-30T15:14:42.594688+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'12","last_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_deep_scrub":"899'12","last_deep_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_clean_scrub_stamp":"2024-03-30T15:14:28.186699+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:54.695167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"904'35","reported_seq":1734,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876615+0000","last_change":"2024-03-30T15:14:35.744220+0000","last_active":"2024-03-30T15:14:37.876615+0000","last_peered":"2024-03-30T15:14:37.876615+0000","last_clean":"2024-03-30T15:14:37.876615+0000","last_became_active":"2024-03-30T15:13:03.533274+0000","last_became_peered":"2024-03-30T15:13:03.533274+0000","last_unstale":"2024-03-30T15:14:37.876615+0000","last_undegraded":"2024-03-30T15:14:37.876615+0000","last_fullsized":"2024-03-30T15:14:37.876615+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'35","last_scrub_stamp":"2024-03-30T15:14:35.744005+0000","last_deep_scrub":"904'35","last_deep_scrub_stamp":"2024-03-30T15:14:35.744005+0000","last_clean_scrub_stamp":"2024-03-30T15:14:35.744005+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.293034+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1304,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053648+0000","last_change":"2024-03-30T15:13:28.647418+0000","last_active":"2024-03-30T15:14:39.053648+0000","last_peered":"2024-03-30T15:14:39.053648+0000","last_clean":"2024-03-30T15:14:39.053648+0000","last_became_active":"2024-03-30T15:13:28.645096+0000","last_became_peered":"2024-03-30T15:13:28.645096+0000","last_unstale":"2024-03-30T15:14:39.053648+0000","last_undegraded":"2024-03-30T15:14:39.053648+0000","last_fullsized":"2024-03-30T15:14:39.053648+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1732,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966613+0000","last_change":"2024-03-30T15:13:42.150953+0000","last_active":"2024-03-30T15:14:42.966613+0000","last_peered":"2024-03-30T15:14:42.966613+0000","last_clean":"2024-03-30T15:14:42.966613+0000","last_became_active":"2024-03-30T15:13:19.274607+0000","last_became_peered":"2024-03-30T15:13:19.274607+0000","last_unstale":"2024-03-30T15:14:42.966613+0000","last_undegraded":"2024-03-30T15:14:42.966613+0000","last_fullsized":"2024-03-30T15:14:42.966613+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.150834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.914073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1038,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968958+0000","last_change":"2024-03-30T15:14:03.599433+0000","last_active":"2024-03-30T15:14:42.968958+0000","last_peered":"2024-03-30T15:14:42.968958+0000","last_clean":"2024-03-30T15:14:42.968958+0000","last_became_active":"2024-03-30T15:14:03.598713+0000","last_became_peered":"2024-03-30T15:14:03.598713+0000","last_unstale":"2024-03-30T15:14:42.968958+0000","last_undegraded":"2024-03-30T15:14:42.968958+0000","last_fullsized":"2024-03-30T15:14:42.968958+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:18.560194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:18.560194+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1483,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595510+0000","last_change":"2024-03-30T15:13:37.186884+0000","last_active":"2024-03-30T15:14:37.595510+0000","last_peered":"2024-03-30T15:14:37.595510+0000","last_clean":"2024-03-30T15:14:37.595510+0000","last_became_active":"2024-03-30T15:13:28.643398+0000","last_became_peered":"2024-03-30T15:13:28.643398+0000","last_unstale":"2024-03-30T15:14:37.595510+0000","last_undegraded":"2024-03-30T15:14:37.595510+0000","last_fullsized":"2024-03-30T15:14:37.595510+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:37.186764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.186764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.763227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"851'10","reported_seq":1547,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053493+0000","last_change":"2024-03-30T15:14:19.204349+0000","last_active":"2024-03-30T15:14:39.053493+0000","last_peered":"2024-03-30T15:14:39.053493+0000","last_clean":"2024-03-30T15:14:39.053493+0000","last_became_active":"2024-03-30T15:13:28.632781+0000","last_became_peered":"2024-03-30T15:13:28.632781+0000","last_unstale":"2024-03-30T15:14:39.053493+0000","last_undegraded":"2024-03-30T15:14:39.053493+0000","last_fullsized":"2024-03-30T15:14:39.053493+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'10","last_scrub_stamp":"2024-03-30T15:14:19.204263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.204263+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.838172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"475'25","reported_seq":1795,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595129+0000","last_change":"2024-03-30T15:14:22.141020+0000","last_active":"2024-03-30T15:14:42.595129+0000","last_peered":"2024-03-30T15:14:42.595129+0000","last_clean":"2024-03-30T15:14:42.595129+0000","last_became_active":"2024-03-30T15:13:03.519156+0000","last_became_peered":"2024-03-30T15:13:03.519156+0000","last_unstale":"2024-03-30T15:14:42.595129+0000","last_undegraded":"2024-03-30T15:14:42.595129+0000","last_fullsized":"2024-03-30T15:14:42.595129+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:14:22.140887+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.140887+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.046893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1746,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596175+0000","last_change":"2024-03-30T15:14:16.872980+0000","last_active":"2024-03-30T15:14:42.596175+0000","last_peered":"2024-03-30T15:14:42.596175+0000","last_clean":"2024-03-30T15:14:42.596175+0000","last_became_active":"2024-03-30T15:13:19.286426+0000","last_became_peered":"2024-03-30T15:13:19.286426+0000","last_unstale":"2024-03-30T15:14:42.596175+0000","last_undegraded":"2024-03-30T15:14:42.596175+0000","last_fullsized":"2024-03-30T15:14:42.596175+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.872862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.872862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:43.944961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1348,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052661+0000","last_change":"2024-03-30T15:14:22.150928+0000","last_active":"2024-03-30T15:14:39.052661+0000","last_peered":"2024-03-30T15:14:39.052661+0000","last_clean":"2024-03-30T15:14:39.052661+0000","last_became_active":"2024-03-30T15:13:52.589257+0000","last_became_peered":"2024-03-30T15:13:52.589257+0000","last_unstale":"2024-03-30T15:14:39.052661+0000","last_undegraded":"2024-03-30T15:14:39.052661+0000","last_fullsized":"2024-03-30T15:14:39.052661+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.150808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.150808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:29.638924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1046,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968888+0000","last_change":"2024-03-30T15:14:03.596583+0000","last_active":"2024-03-30T15:14:42.968888+0000","last_peered":"2024-03-30T15:14:42.968888+0000","last_clean":"2024-03-30T15:14:42.968888+0000","last_became_active":"2024-03-30T15:14:03.596165+0000","last_became_peered":"2024-03-30T15:14:03.596165+0000","last_unstale":"2024-03-30T15:14:42.968888+0000","last_undegraded":"2024-03-30T15:14:42.968888+0000","last_fullsized":"2024-03-30T15:14:42.968888+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:13.604719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:13.604719+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1480,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595803+0000","last_change":"2024-03-30T15:14:20.304484+0000","last_active":"2024-03-30T15:14:37.595803+0000","last_peered":"2024-03-30T15:14:37.595803+0000","last_clean":"2024-03-30T15:14:37.595803+0000","last_became_active":"2024-03-30T15:13:19.281718+0000","last_became_peered":"2024-03-30T15:13:19.281718+0000","last_unstale":"2024-03-30T15:14:37.595803+0000","last_undegraded":"2024-03-30T15:14:37.595803+0000","last_fullsized":"2024-03-30T15:14:37.595803+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.304352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.304352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.098567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968444+0000","last_change":"2024-03-30T15:13:30.648769+0000","last_active":"2024-03-30T15:14:42.968444+0000","last_peered":"2024-03-30T15:14:42.968444+0000","last_clean":"2024-03-30T15:14:42.968444+0000","last_became_active":"2024-03-30T15:13:28.622538+0000","last_became_peered":"2024-03-30T15:13:28.622538+0000","last_unstale":"2024-03-30T15:14:42.968444+0000","last_undegraded":"2024-03-30T15:14:42.968444+0000","last_fullsized":"2024-03-30T15:14:42.968444+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:30.648694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.648694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.244373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"475'37","reported_seq":1349,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052926+0000","last_change":"2024-03-30T15:13:28.646691+0000","last_active":"2024-03-30T15:14:39.052926+0000","last_peered":"2024-03-30T15:14:39.052926+0000","last_clean":"2024-03-30T15:14:39.052926+0000","last_became_active":"2024-03-30T15:13:28.645038+0000","last_became_peered":"2024-03-30T15:13:28.645038+0000","last_unstale":"2024-03-30T15:14:39.052926+0000","last_undegraded":"2024-03-30T15:14:39.052926+0000","last_fullsized":"2024-03-30T15:14:39.052926+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1775,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596457+0000","last_change":"2024-03-30T15:13:52.591664+0000","last_active":"2024-03-30T15:14:42.596457+0000","last_peered":"2024-03-30T15:14:42.596457+0000","last_clean":"2024-03-30T15:14:42.596457+0000","last_became_active":"2024-03-30T15:13:52.591330+0000","last_became_peered":"2024-03-30T15:13:52.591330+0000","last_unstale":"2024-03-30T15:14:42.596457+0000","last_undegraded":"2024-03-30T15:14:42.596457+0000","last_fullsized":"2024-03-30T15:14:42.596457+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:39.476204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.476204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:03.771719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1766,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594862+0000","last_change":"2024-03-30T15:14:12.245431+0000","last_active":"2024-03-30T15:14:42.594862+0000","last_peered":"2024-03-30T15:14:42.594862+0000","last_clean":"2024-03-30T15:14:42.594862+0000","last_became_active":"2024-03-30T15:13:52.588705+0000","last_became_peered":"2024-03-30T15:13:52.588705+0000","last_unstale":"2024-03-30T15:14:42.594862+0000","last_undegraded":"2024-03-30T15:14:42.594862+0000","last_fullsized":"2024-03-30T15:14:42.594862+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:12.245351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.245351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:30.371930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":1041,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595051+0000","last_change":"2024-03-30T15:14:33.261448+0000","last_active":"2024-03-30T15:14:42.595051+0000","last_peered":"2024-03-30T15:14:42.595051+0000","last_clean":"2024-03-30T15:14:42.595051+0000","last_became_active":"2024-03-30T15:13:03.514491+0000","last_became_peered":"2024-03-30T15:13:03.514491+0000","last_unstale":"2024-03-30T15:14:42.595051+0000","last_undegraded":"2024-03-30T15:14:42.595051+0000","last_fullsized":"2024-03-30T15:14:42.595051+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:33.261365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.261365+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:54.793305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1254,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053324+0000","last_change":"2024-03-30T15:14:14.201276+0000","last_active":"2024-03-30T15:14:39.053324+0000","last_peered":"2024-03-30T15:14:39.053324+0000","last_clean":"2024-03-30T15:14:39.053324+0000","last_became_active":"2024-03-30T15:13:28.642223+0000","last_became_peered":"2024-03-30T15:13:28.642223+0000","last_unstale":"2024-03-30T15:14:39.053324+0000","last_undegraded":"2024-03-30T15:14:39.053324+0000","last_fullsized":"2024-03-30T15:14:39.053324+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:14.201134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.201134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:31.718490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1730,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596969+0000","last_change":"2024-03-30T15:13:31.905462+0000","last_active":"2024-03-30T15:14:42.596969+0000","last_peered":"2024-03-30T15:14:42.596969+0000","last_clean":"2024-03-30T15:14:42.596969+0000","last_became_active":"2024-03-30T15:13:28.622981+0000","last_became_peered":"2024-03-30T15:13:28.622981+0000","last_unstale":"2024-03-30T15:14:42.596969+0000","last_undegraded":"2024-03-30T15:14:42.596969+0000","last_fullsized":"2024-03-30T15:14:42.596969+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.905371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.905371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.331086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"475'24","reported_seq":1349,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053526+0000","last_change":"2024-03-30T15:13:52.603549+0000","last_active":"2024-03-30T15:14:39.053526+0000","last_peered":"2024-03-30T15:14:39.053526+0000","last_clean":"2024-03-30T15:14:39.053526+0000","last_became_active":"2024-03-30T15:13:52.591080+0000","last_became_peered":"2024-03-30T15:13:52.591080+0000","last_unstale":"2024-03-30T15:14:39.053526+0000","last_undegraded":"2024-03-30T15:14:39.053526+0000","last_fullsized":"2024-03-30T15:14:39.053526+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2946,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966323+0000","last_change":"2024-03-30T15:13:39.069148+0000","last_active":"2024-03-30T15:14:42.966323+0000","last_peered":"2024-03-30T15:14:42.966323+0000","last_clean":"2024-03-30T15:14:42.966323+0000","last_became_active":"2024-03-30T15:13:28.632404+0000","last_became_peered":"2024-03-30T15:13:28.632404+0000","last_unstale":"2024-03-30T15:14:42.966323+0000","last_undegraded":"2024-03-30T15:14:42.966323+0000","last_fullsized":"2024-03-30T15:14:42.966323+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:13:39.069031+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.069031+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.220616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1194,"num_read_kb":1189,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1320,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052872+0000","last_change":"2024-03-30T15:14:09.726829+0000","last_active":"2024-03-30T15:14:39.052872+0000","last_peered":"2024-03-30T15:14:39.052872+0000","last_clean":"2024-03-30T15:14:39.052872+0000","last_became_active":"2024-03-30T15:14:09.726541+0000","last_became_peered":"2024-03-30T15:14:09.726541+0000","last_unstale":"2024-03-30T15:14:39.052872+0000","last_undegraded":"2024-03-30T15:14:39.052872+0000","last_fullsized":"2024-03-30T15:14:39.052872+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:36.922218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:11:36.922218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1758,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596198+0000","last_change":"2024-03-30T15:14:29.204333+0000","last_active":"2024-03-30T15:14:42.596198+0000","last_peered":"2024-03-30T15:14:42.596198+0000","last_clean":"2024-03-30T15:14:42.596198+0000","last_became_active":"2024-03-30T15:13:03.514904+0000","last_became_peered":"2024-03-30T15:13:03.514904+0000","last_unstale":"2024-03-30T15:14:42.596198+0000","last_undegraded":"2024-03-30T15:14:42.596198+0000","last_fullsized":"2024-03-30T15:14:42.596198+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.204254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.204254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.733748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1640,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967236+0000","last_change":"2024-03-30T15:14:24.106210+0000","last_active":"2024-03-30T15:14:42.967236+0000","last_peered":"2024-03-30T15:14:42.967236+0000","last_clean":"2024-03-30T15:14:42.967236+0000","last_became_active":"2024-03-30T15:13:19.276949+0000","last_became_peered":"2024-03-30T15:13:19.276949+0000","last_unstale":"2024-03-30T15:14:42.967236+0000","last_undegraded":"2024-03-30T15:14:42.967236+0000","last_fullsized":"2024-03-30T15:14:42.967236+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:24.106039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:24.106039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.706646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1736,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967719+0000","last_change":"2024-03-30T15:14:12.151419+0000","last_active":"2024-03-30T15:14:42.967719+0000","last_peered":"2024-03-30T15:14:42.967719+0000","last_clean":"2024-03-30T15:14:42.967719+0000","last_became_active":"2024-03-30T15:13:19.271491+0000","last_became_peered":"2024-03-30T15:13:19.271491+0000","last_unstale":"2024-03-30T15:14:42.967719+0000","last_undegraded":"2024-03-30T15:14:42.967719+0000","last_fullsized":"2024-03-30T15:14:42.967719+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:12.151317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.151317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.046779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"905'38","reported_seq":2048,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969185+0000","last_change":"2024-03-30T15:14:36.416421+0000","last_active":"2024-03-30T15:14:42.969185+0000","last_peered":"2024-03-30T15:14:42.969185+0000","last_clean":"2024-03-30T15:14:42.969185+0000","last_became_active":"2024-03-30T15:13:03.543824+0000","last_became_peered":"2024-03-30T15:13:03.543824+0000","last_unstale":"2024-03-30T15:14:42.969185+0000","last_undegraded":"2024-03-30T15:14:42.969185+0000","last_fullsized":"2024-03-30T15:14:42.969185+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'38","last_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_deep_scrub":"905'38","last_deep_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.416317+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:42.691935+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1632,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595705+0000","last_change":"2024-03-30T15:12:20.989844+0000","last_active":"2024-03-30T15:14:42.595705+0000","last_peered":"2024-03-30T15:14:42.595705+0000","last_clean":"2024-03-30T15:14:42.595705+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:14:42.595705+0000","last_undegraded":"2024-03-30T15:14:42.595705+0000","last_fullsized":"2024-03-30T15:14:42.595705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:20.989740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:12:20.989740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":2006,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966461+0000","last_change":"2024-03-30T15:13:29.044762+0000","last_active":"2024-03-30T15:14:42.966461+0000","last_peered":"2024-03-30T15:14:42.966461+0000","last_clean":"2024-03-30T15:14:42.966461+0000","last_became_active":"2024-03-30T15:13:28.632600+0000","last_became_peered":"2024-03-30T15:13:28.632600+0000","last_unstale":"2024-03-30T15:14:42.966461+0000","last_undegraded":"2024-03-30T15:14:42.966461+0000","last_fullsized":"2024-03-30T15:14:42.966461+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:29.044665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:29.044665+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1726,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596534+0000","last_change":"2024-03-30T15:13:03.523181+0000","last_active":"2024-03-30T15:14:42.596534+0000","last_peered":"2024-03-30T15:14:42.596534+0000","last_clean":"2024-03-30T15:14:42.596534+0000","last_became_active":"2024-03-30T15:13:03.522675+0000","last_became_peered":"2024-03-30T15:13:03.522675+0000","last_unstale":"2024-03-30T15:14:42.596534+0000","last_undegraded":"2024-03-30T15:14:42.596534+0000","last_fullsized":"2024-03-30T15:14:42.596534+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.840893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":2068,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965576+0000","last_change":"2024-03-30T15:14:12.746871+0000","last_active":"2024-03-30T15:14:42.965576+0000","last_peered":"2024-03-30T15:14:42.965576+0000","last_clean":"2024-03-30T15:14:42.965576+0000","last_became_active":"2024-03-30T15:14:12.746444+0000","last_became_peered":"2024-03-30T15:14:12.746444+0000","last_unstale":"2024-03-30T15:14:42.965576+0000","last_undegraded":"2024-03-30T15:14:42.965576+0000","last_fullsized":"2024-03-30T15:14:42.965576+0000","mapping_epoch":883,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:27.092990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:27.092990+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.631034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1664,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969221+0000","last_change":"2024-03-30T15:13:52.590016+0000","last_active":"2024-03-30T15:14:42.969221+0000","last_peered":"2024-03-30T15:14:42.969221+0000","last_clean":"2024-03-30T15:14:42.969221+0000","last_became_active":"2024-03-30T15:13:52.589696+0000","last_became_peered":"2024-03-30T15:13:52.589696+0000","last_unstale":"2024-03-30T15:14:42.969221+0000","last_undegraded":"2024-03-30T15:14:42.969221+0000","last_fullsized":"2024-03-30T15:14:42.969221+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:46.745785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:11:46.745785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":97,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.597548+0000","last_change":"2024-03-30T15:14:36.036176+0000","last_active":"2024-03-30T15:14:37.597548+0000","last_peered":"2024-03-30T15:14:37.597548+0000","last_clean":"2024-03-30T15:14:37.597548+0000","last_became_active":"2024-03-30T15:13:52.594698+0000","last_became_peered":"2024-03-30T15:13:52.594698+0000","last_unstale":"2024-03-30T15:14:37.597548+0000","last_undegraded":"2024-03-30T15:14:37.597548+0000","last_fullsized":"2024-03-30T15:14:37.597548+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:36.036047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.036047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:57.199133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"475'22","reported_seq":1790,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596694+0000","last_change":"2024-03-30T15:13:29.494270+0000","last_active":"2024-03-30T15:14:42.596694+0000","last_peered":"2024-03-30T15:14:42.596694+0000","last_clean":"2024-03-30T15:14:42.596694+0000","last_became_active":"2024-03-30T15:13:28.631530+0000","last_became_peered":"2024-03-30T15:13:28.631530+0000","last_unstale":"2024-03-30T15:14:42.596694+0000","last_undegraded":"2024-03-30T15:14:42.596694+0000","last_fullsized":"2024-03-30T15:14:42.596694+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:13:29.494150+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:13:29.494150+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1600,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595647+0000","last_change":"2024-03-30T15:14:17.334654+0000","last_active":"2024-03-30T15:14:37.595647+0000","last_peered":"2024-03-30T15:14:37.595647+0000","last_clean":"2024-03-30T15:14:37.595647+0000","last_became_active":"2024-03-30T15:13:28.614801+0000","last_became_peered":"2024-03-30T15:13:28.614801+0000","last_unstale":"2024-03-30T15:14:37.595647+0000","last_undegraded":"2024-03-30T15:14:37.595647+0000","last_fullsized":"2024-03-30T15:14:37.595647+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.334571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.334571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:37.752428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1765,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594789+0000","last_change":"2024-03-30T15:13:36.535601+0000","last_active":"2024-03-30T15:14:42.594789+0000","last_peered":"2024-03-30T15:14:42.594789+0000","last_clean":"2024-03-30T15:14:42.594789+0000","last_became_active":"2024-03-30T15:13:19.290979+0000","last_became_peered":"2024-03-30T15:13:19.290979+0000","last_unstale":"2024-03-30T15:14:42.594789+0000","last_undegraded":"2024-03-30T15:14:42.594789+0000","last_fullsized":"2024-03-30T15:14:42.594789+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:36.535520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.535520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:45.609458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1739,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967535+0000","last_change":"2024-03-30T15:14:31.060531+0000","last_active":"2024-03-30T15:14:42.967535+0000","last_peered":"2024-03-30T15:14:42.967535+0000","last_clean":"2024-03-30T15:14:42.967535+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:14:42.967535+0000","last_undegraded":"2024-03-30T15:14:42.967535+0000","last_fullsized":"2024-03-30T15:14:42.967535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:31.060450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:31.060450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.835750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":1021,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595214+0000","last_change":"2024-03-30T15:13:32.864468+0000","last_active":"2024-03-30T15:14:42.595214+0000","last_peered":"2024-03-30T15:14:42.595214+0000","last_clean":"2024-03-30T15:14:42.595214+0000","last_became_active":"2024-03-30T15:13:28.623160+0000","last_became_peered":"2024-03-30T15:13:28.623160+0000","last_unstale":"2024-03-30T15:14:42.595214+0000","last_undegraded":"2024-03-30T15:14:42.595214+0000","last_fullsized":"2024-03-30T15:14:42.595214+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:32.864377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:32.864377+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1260,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053174+0000","last_change":"2024-03-30T15:14:13.185434+0000","last_active":"2024-03-30T15:14:39.053174+0000","last_peered":"2024-03-30T15:14:39.053174+0000","last_clean":"2024-03-30T15:14:39.053174+0000","last_became_active":"2024-03-30T15:13:28.639814+0000","last_became_peered":"2024-03-30T15:13:28.639814+0000","last_unstale":"2024-03-30T15:14:39.053174+0000","last_undegraded":"2024-03-30T15:14:39.053174+0000","last_fullsized":"2024-03-30T15:14:39.053174+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:13.185353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:13.185353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.512661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1364,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053357+0000","last_change":"2024-03-30T15:14:09.726789+0000","last_active":"2024-03-30T15:14:39.053357+0000","last_peered":"2024-03-30T15:14:39.053357+0000","last_clean":"2024-03-30T15:14:39.053357+0000","last_became_active":"2024-03-30T15:14:09.726470+0000","last_became_peered":"2024-03-30T15:14:09.726470+0000","last_unstale":"2024-03-30T15:14:39.053357+0000","last_undegraded":"2024-03-30T15:14:39.053357+0000","last_fullsized":"2024-03-30T15:14:39.053357+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:47.455580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.455580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"845'43","reported_seq":1999,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596598+0000","last_change":"2024-03-30T15:13:42.932913+0000","last_active":"2024-03-30T15:14:42.596598+0000","last_peered":"2024-03-30T15:14:42.596598+0000","last_clean":"2024-03-30T15:14:42.596598+0000","last_became_active":"2024-03-30T15:13:19.293582+0000","last_became_peered":"2024-03-30T15:13:19.293582+0000","last_unstale":"2024-03-30T15:14:42.596598+0000","last_undegraded":"2024-03-30T15:14:42.596598+0000","last_fullsized":"2024-03-30T15:14:42.596598+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'43","last_scrub_stamp":"2024-03-30T15:13:42.932748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.932748+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.764445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1665,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877178+0000","last_change":"2024-03-30T15:13:28.784472+0000","last_active":"2024-03-30T15:14:37.877178+0000","last_peered":"2024-03-30T15:14:37.877178+0000","last_clean":"2024-03-30T15:14:37.877178+0000","last_became_active":"2024-03-30T15:13:28.624999+0000","last_became_peered":"2024-03-30T15:13:28.624999+0000","last_unstale":"2024-03-30T15:14:37.877178+0000","last_undegraded":"2024-03-30T15:14:37.877178+0000","last_fullsized":"2024-03-30T15:14:37.877178+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.784351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.784351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1740,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595840+0000","last_change":"2024-03-30T15:12:16.942168+0000","last_active":"2024-03-30T15:14:42.595840+0000","last_peered":"2024-03-30T15:14:42.595840+0000","last_clean":"2024-03-30T15:14:42.595840+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:14:42.595840+0000","last_undegraded":"2024-03-30T15:14:42.595840+0000","last_fullsized":"2024-03-30T15:14:42.595840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:12:16.942087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:12:16.942087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1595,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877727+0000","last_change":"2024-03-30T15:14:16.682735+0000","last_active":"2024-03-30T15:14:37.877727+0000","last_peered":"2024-03-30T15:14:37.877727+0000","last_clean":"2024-03-30T15:14:37.877727+0000","last_became_active":"2024-03-30T15:13:03.545212+0000","last_became_peered":"2024-03-30T15:13:03.545212+0000","last_unstale":"2024-03-30T15:14:37.877727+0000","last_undegraded":"2024-03-30T15:14:37.877727+0000","last_fullsized":"2024-03-30T15:14:37.877727+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.682638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.682638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:40.967052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1256,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053123+0000","last_change":"2024-03-30T15:14:00.635645+0000","last_active":"2024-03-30T15:14:39.053123+0000","last_peered":"2024-03-30T15:14:39.053123+0000","last_clean":"2024-03-30T15:14:39.053123+0000","last_became_active":"2024-03-30T15:14:00.632424+0000","last_became_peered":"2024-03-30T15:14:00.632424+0000","last_unstale":"2024-03-30T15:14:39.053123+0000","last_undegraded":"2024-03-30T15:14:39.053123+0000","last_fullsized":"2024-03-30T15:14:39.053123+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:38.079399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:11:38.079399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"475'15","reported_seq":1342,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053232+0000","last_change":"2024-03-30T15:13:28.633451+0000","last_active":"2024-03-30T15:14:39.053232+0000","last_peered":"2024-03-30T15:14:39.053232+0000","last_clean":"2024-03-30T15:14:39.053232+0000","last_became_active":"2024-03-30T15:13:28.629693+0000","last_became_peered":"2024-03-30T15:13:28.629693+0000","last_unstale":"2024-03-30T15:14:39.053232+0000","last_undegraded":"2024-03-30T15:14:39.053232+0000","last_fullsized":"2024-03-30T15:14:39.053232+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596506+0000","last_change":"2024-03-30T15:12:59.144443+0000","last_active":"2024-03-30T15:14:42.596506+0000","last_peered":"2024-03-30T15:14:42.596506+0000","last_clean":"2024-03-30T15:14:42.596506+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:14:42.596506+0000","last_undegraded":"2024-03-30T15:14:42.596506+0000","last_fullsized":"2024-03-30T15:14:42.596506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_clean_scrub_stamp":"2024-03-30T15:12:59.144310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1608,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595502+0000","last_change":"2024-03-30T15:13:28.205972+0000","last_active":"2024-03-30T15:14:37.595502+0000","last_peered":"2024-03-30T15:14:37.595502+0000","last_clean":"2024-03-30T15:14:37.595502+0000","last_became_active":"2024-03-30T15:13:19.284117+0000","last_became_peered":"2024-03-30T15:13:19.284117+0000","last_unstale":"2024-03-30T15:14:37.595502+0000","last_undegraded":"2024-03-30T15:14:37.595502+0000","last_fullsized":"2024-03-30T15:14:37.595502+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.205876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.205876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:39.104664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1295,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052895+0000","last_change":"2024-03-30T15:14:30.194970+0000","last_active":"2024-03-30T15:14:39.052895+0000","last_peered":"2024-03-30T15:14:39.052895+0000","last_clean":"2024-03-30T15:14:39.052895+0000","last_became_active":"2024-03-30T15:13:52.588594+0000","last_became_peered":"2024-03-30T15:13:52.588594+0000","last_unstale":"2024-03-30T15:14:39.052895+0000","last_undegraded":"2024-03-30T15:14:39.052895+0000","last_fullsized":"2024-03-30T15:14:39.052895+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:30.194832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:30.194832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:32.945100+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1604,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877416+0000","last_change":"2024-03-30T15:14:00.635095+0000","last_active":"2024-03-30T15:14:37.877416+0000","last_peered":"2024-03-30T15:14:37.877416+0000","last_clean":"2024-03-30T15:14:37.877416+0000","last_became_active":"2024-03-30T15:14:00.634711+0000","last_became_peered":"2024-03-30T15:14:00.634711+0000","last_unstale":"2024-03-30T15:14:37.877416+0000","last_undegraded":"2024-03-30T15:14:37.877416+0000","last_fullsized":"2024-03-30T15:14:37.877416+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:37.755201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.755201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.272904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"475'35","reported_seq":1829,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968334+0000","last_change":"2024-03-30T15:13:37.663625+0000","last_active":"2024-03-30T15:14:42.968334+0000","last_peered":"2024-03-30T15:14:42.968334+0000","last_clean":"2024-03-30T15:14:42.968334+0000","last_became_active":"2024-03-30T15:13:19.285236+0000","last_became_peered":"2024-03-30T15:13:19.285236+0000","last_unstale":"2024-03-30T15:14:42.968334+0000","last_undegraded":"2024-03-30T15:14:42.968334+0000","last_fullsized":"2024-03-30T15:14:42.968334+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:13:37.663475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.663475+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.956850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1734,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967353+0000","last_change":"2024-03-30T15:12:19.812425+0000","last_active":"2024-03-30T15:14:42.967353+0000","last_peered":"2024-03-30T15:14:42.967353+0000","last_clean":"2024-03-30T15:14:42.967353+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:14:42.967353+0000","last_undegraded":"2024-03-30T15:14:42.967353+0000","last_fullsized":"2024-03-30T15:14:42.967353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:19.812343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:12:19.812343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":97,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595297+0000","last_change":"2024-03-30T15:14:14.317102+0000","last_active":"2024-03-30T15:14:37.595297+0000","last_peered":"2024-03-30T15:14:37.595297+0000","last_clean":"2024-03-30T15:14:37.595297+0000","last_became_active":"2024-03-30T15:13:52.587447+0000","last_became_peered":"2024-03-30T15:13:52.587447+0000","last_unstale":"2024-03-30T15:14:37.595297+0000","last_undegraded":"2024-03-30T15:14:37.595297+0000","last_fullsized":"2024-03-30T15:14:37.595297+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:14.317001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.317001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.560518+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1732,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965622+0000","last_change":"2024-03-30T15:14:15.099798+0000","last_active":"2024-03-30T15:14:42.965622+0000","last_peered":"2024-03-30T15:14:42.965622+0000","last_clean":"2024-03-30T15:14:42.965622+0000","last_became_active":"2024-03-30T15:13:19.271586+0000","last_became_peered":"2024-03-30T15:13:19.271586+0000","last_unstale":"2024-03-30T15:14:42.965622+0000","last_undegraded":"2024-03-30T15:14:42.965622+0000","last_fullsized":"2024-03-30T15:14:42.965622+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.099699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.099699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:18.098833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1654,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965500+0000","last_change":"2024-03-30T15:12:22.500804+0000","last_active":"2024-03-30T15:14:42.965500+0000","last_peered":"2024-03-30T15:14:42.965500+0000","last_clean":"2024-03-30T15:14:42.965500+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:14:42.965500+0000","last_undegraded":"2024-03-30T15:14:42.965500+0000","last_fullsized":"2024-03-30T15:14:42.965500+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:12:22.500717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:22.500717+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1575,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.594786+0000","last_change":"2024-03-30T15:13:36.165499+0000","last_active":"2024-03-30T15:14:37.594786+0000","last_peered":"2024-03-30T15:14:37.594786+0000","last_clean":"2024-03-30T15:14:37.594786+0000","last_became_active":"2024-03-30T15:13:28.646999+0000","last_became_peered":"2024-03-30T15:13:28.646999+0000","last_unstale":"2024-03-30T15:14:37.594786+0000","last_undegraded":"2024-03-30T15:14:37.594786+0000","last_fullsized":"2024-03-30T15:14:37.594786+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:36.165416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.165416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:05.472693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"475'18","reported_seq":1747,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876704+0000","last_change":"2024-03-30T15:14:25.778635+0000","last_active":"2024-03-30T15:14:37.876704+0000","last_peered":"2024-03-30T15:14:37.876704+0000","last_clean":"2024-03-30T15:14:37.876704+0000","last_became_active":"2024-03-30T15:14:00.632108+0000","last_became_peered":"2024-03-30T15:14:00.632108+0000","last_unstale":"2024-03-30T15:14:37.876704+0000","last_undegraded":"2024-03-30T15:14:37.876704+0000","last_fullsized":"2024-03-30T15:14:37.876704+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:14:25.778547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:25.778547+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:32.231962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1743,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966364+0000","last_change":"2024-03-30T15:14:22.044920+0000","last_active":"2024-03-30T15:14:42.966364+0000","last_peered":"2024-03-30T15:14:42.966364+0000","last_clean":"2024-03-30T15:14:42.966364+0000","last_became_active":"2024-03-30T15:13:03.550868+0000","last_became_peered":"2024-03-30T15:13:03.550868+0000","last_unstale":"2024-03-30T15:14:42.966364+0000","last_undegraded":"2024-03-30T15:14:42.966364+0000","last_fullsized":"2024-03-30T15:14:42.966364+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.044816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.044816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.372065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1627,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966782+0000","last_change":"2024-03-30T15:10:58.066365+0000","last_active":"2024-03-30T15:14:42.966782+0000","last_peered":"2024-03-30T15:14:42.966782+0000","last_clean":"2024-03-30T15:14:42.966782+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:14:42.966782+0000","last_undegraded":"2024-03-30T15:14:42.966782+0000","last_fullsized":"2024-03-30T15:14:42.966782+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:58.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:10:58.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":70,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876372+0000","last_change":"2024-03-30T15:14:15.728874+0000","last_active":"2024-03-30T15:14:37.876372+0000","last_peered":"2024-03-30T15:14:37.876372+0000","last_clean":"2024-03-30T15:14:37.876372+0000","last_became_active":"2024-03-30T15:14:09.725078+0000","last_became_peered":"2024-03-30T15:14:09.725078+0000","last_unstale":"2024-03-30T15:14:37.876372+0000","last_undegraded":"2024-03-30T15:14:37.876372+0000","last_fullsized":"2024-03-30T15:14:37.876372+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.728771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.728771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.523390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"833'10","reported_seq":1980,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966122+0000","last_change":"2024-03-30T15:13:04.947263+0000","last_active":"2024-03-30T15:14:42.966122+0000","last_peered":"2024-03-30T15:14:42.966122+0000","last_clean":"2024-03-30T15:14:42.966122+0000","last_became_active":"2024-03-30T15:13:03.549786+0000","last_became_peered":"2024-03-30T15:13:03.549786+0000","last_unstale":"2024-03-30T15:14:42.966122+0000","last_undegraded":"2024-03-30T15:14:42.966122+0000","last_fullsized":"2024-03-30T15:14:42.966122+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'10","last_scrub_stamp":"2024-03-30T15:13:04.947176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:04.947176+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"475'27","reported_seq":1776,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595140+0000","last_change":"2024-03-30T15:14:24.914466+0000","last_active":"2024-03-30T15:14:42.595140+0000","last_peered":"2024-03-30T15:14:42.595140+0000","last_clean":"2024-03-30T15:14:42.595140+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:14:42.595140+0000","last_undegraded":"2024-03-30T15:14:42.595140+0000","last_fullsized":"2024-03-30T15:14:42.595140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:14:24.914288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:24.914288+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:43.784084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1752,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968494+0000","last_change":"2024-03-30T15:13:36.062294+0000","last_active":"2024-03-30T15:14:42.968494+0000","last_peered":"2024-03-30T15:14:42.968494+0000","last_clean":"2024-03-30T15:14:42.968494+0000","last_became_active":"2024-03-30T15:13:28.632685+0000","last_became_peered":"2024-03-30T15:13:28.632685+0000","last_unstale":"2024-03-30T15:14:42.968494+0000","last_undegraded":"2024-03-30T15:14:42.968494+0000","last_fullsized":"2024-03-30T15:14:42.968494+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:13:36.062171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.062171+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.071532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1726,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596332+0000","last_change":"2024-03-30T15:13:44.911542+0000","last_active":"2024-03-30T15:14:42.596332+0000","last_peered":"2024-03-30T15:14:42.596332+0000","last_clean":"2024-03-30T15:14:42.596332+0000","last_became_active":"2024-03-30T15:13:28.625385+0000","last_became_peered":"2024-03-30T15:13:28.625385+0000","last_unstale":"2024-03-30T15:14:42.596332+0000","last_undegraded":"2024-03-30T15:14:42.596332+0000","last_fullsized":"2024-03-30T15:14:42.596332+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.911436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.911436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:47.418564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1665,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969180+0000","last_change":"2024-03-30T15:14:21.534146+0000","last_active":"2024-03-30T15:14:42.969180+0000","last_peered":"2024-03-30T15:14:42.969180+0000","last_clean":"2024-03-30T15:14:42.969180+0000","last_became_active":"2024-03-30T15:13:52.592711+0000","last_became_peered":"2024-03-30T15:13:52.592711+0000","last_unstale":"2024-03-30T15:14:42.969180+0000","last_undegraded":"2024-03-30T15:14:42.969180+0000","last_fullsized":"2024-03-30T15:14:42.969180+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:21.534007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.534007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.099017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"475'12","reported_seq":93,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.458491+0000","last_change":"2024-03-30T15:14:21.380980+0000","last_active":"2024-03-30T15:14:43.458491+0000","last_peered":"2024-03-30T15:14:43.458491+0000","last_clean":"2024-03-30T15:14:43.458491+0000","last_became_active":"2024-03-30T15:14:00.631854+0000","last_became_peered":"2024-03-30T15:14:00.631854+0000","last_unstale":"2024-03-30T15:14:43.458491+0000","last_undegraded":"2024-03-30T15:14:43.458491+0000","last_fullsized":"2024-03-30T15:14:43.458491+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:14:21.380891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.380891+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:26.416878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1751,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965579+0000","last_change":"2024-03-30T15:13:31.683113+0000","last_active":"2024-03-30T15:14:42.965579+0000","last_peered":"2024-03-30T15:14:42.965579+0000","last_clean":"2024-03-30T15:14:42.965579+0000","last_became_active":"2024-03-30T15:13:28.623365+0000","last_became_peered":"2024-03-30T15:13:28.623365+0000","last_unstale":"2024-03-30T15:14:42.965579+0000","last_undegraded":"2024-03-30T15:14:42.965579+0000","last_fullsized":"2024-03-30T15:14:42.965579+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.682971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.682971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1728,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595988+0000","last_change":"2024-03-30T15:14:27.928955+0000","last_active":"2024-03-30T15:14:42.595988+0000","last_peered":"2024-03-30T15:14:42.595988+0000","last_clean":"2024-03-30T15:14:42.595988+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:14:42.595988+0000","last_undegraded":"2024-03-30T15:14:42.595988+0000","last_fullsized":"2024-03-30T15:14:42.595988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:27.928851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:27.928851+0000","last_clean_scrub_stamp":"2024-03-30T15:14:27.928851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.764039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1723,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967001+0000","last_change":"2024-03-30T15:13:44.201709+0000","last_active":"2024-03-30T15:14:42.967001+0000","last_peered":"2024-03-30T15:14:42.967001+0000","last_clean":"2024-03-30T15:14:42.967001+0000","last_became_active":"2024-03-30T15:13:03.537074+0000","last_became_peered":"2024-03-30T15:13:03.537074+0000","last_unstale":"2024-03-30T15:14:42.967001+0000","last_undegraded":"2024-03-30T15:14:42.967001+0000","last_fullsized":"2024-03-30T15:14:42.967001+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.201602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.201602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.021081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1606,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877807+0000","last_change":"2024-03-30T15:13:03.530702+0000","last_active":"2024-03-30T15:14:37.877807+0000","last_peered":"2024-03-30T15:14:37.877807+0000","last_clean":"2024-03-30T15:14:37.877807+0000","last_became_active":"2024-03-30T15:13:03.529999+0000","last_became_peered":"2024-03-30T15:13:03.529999+0000","last_unstale":"2024-03-30T15:14:37.877807+0000","last_undegraded":"2024-03-30T15:14:37.877807+0000","last_fullsized":"2024-03-30T15:14:37.877807+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:13.914480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:13.914480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"475'15","reported_seq":1341,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053735+0000","last_change":"2024-03-30T15:13:28.646408+0000","last_active":"2024-03-30T15:14:39.053735+0000","last_peered":"2024-03-30T15:14:39.053735+0000","last_clean":"2024-03-30T15:14:39.053735+0000","last_became_active":"2024-03-30T15:13:28.642687+0000","last_became_peered":"2024-03-30T15:13:28.642687+0000","last_unstale":"2024-03-30T15:14:39.053735+0000","last_undegraded":"2024-03-30T15:14:39.053735+0000","last_fullsized":"2024-03-30T15:14:39.053735+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1743,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965832+0000","last_change":"2024-03-30T15:13:46.180286+0000","last_active":"2024-03-30T15:14:42.965832+0000","last_peered":"2024-03-30T15:14:42.965832+0000","last_clean":"2024-03-30T15:14:42.965832+0000","last_became_active":"2024-03-30T15:13:03.548886+0000","last_became_peered":"2024-03-30T15:13:03.548886+0000","last_unstale":"2024-03-30T15:14:42.965832+0000","last_undegraded":"2024-03-30T15:14:42.965832+0000","last_fullsized":"2024-03-30T15:14:42.965832+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:46.180140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.180140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.861407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":822,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969115+0000","last_change":"2024-03-30T15:14:23.521674+0000","last_active":"2024-03-30T15:14:42.969115+0000","last_peered":"2024-03-30T15:14:42.969115+0000","last_clean":"2024-03-30T15:14:42.969115+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:14:42.969115+0000","last_undegraded":"2024-03-30T15:14:42.969115+0000","last_fullsized":"2024-03-30T15:14:42.969115+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.521577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.521577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.497032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1578,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.596257+0000","last_change":"2024-03-30T15:13:27.166879+0000","last_active":"2024-03-30T15:14:37.596257+0000","last_peered":"2024-03-30T15:14:37.596257+0000","last_clean":"2024-03-30T15:14:37.596257+0000","last_became_active":"2024-03-30T15:13:19.279299+0000","last_became_peered":"2024-03-30T15:13:19.279299+0000","last_unstale":"2024-03-30T15:14:37.596257+0000","last_undegraded":"2024-03-30T15:14:37.596257+0000","last_fullsized":"2024-03-30T15:14:37.596257+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:27.166790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:27.166790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:52.798993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1247,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053302+0000","last_change":"2024-03-30T15:13:52.590156+0000","last_active":"2024-03-30T15:14:39.053302+0000","last_peered":"2024-03-30T15:14:39.053302+0000","last_clean":"2024-03-30T15:14:39.053302+0000","last_became_active":"2024-03-30T15:13:52.589809+0000","last_became_peered":"2024-03-30T15:13:52.589809+0000","last_unstale":"2024-03-30T15:14:39.053302+0000","last_undegraded":"2024-03-30T15:14:39.053302+0000","last_fullsized":"2024-03-30T15:14:39.053302+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:25.719723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:25.719723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:54.190231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"475'15","reported_seq":1757,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596416+0000","last_change":"2024-03-30T15:14:29.924425+0000","last_active":"2024-03-30T15:14:42.596416+0000","last_peered":"2024-03-30T15:14:42.596416+0000","last_clean":"2024-03-30T15:14:42.596416+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:14:42.596416+0000","last_undegraded":"2024-03-30T15:14:42.596416+0000","last_fullsized":"2024-03-30T15:14:42.596416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:14:29.924285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.924285+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:47.854739+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1304,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052942+0000","last_change":"2024-03-30T15:14:00.634621+0000","last_active":"2024-03-30T15:14:39.052942+0000","last_peered":"2024-03-30T15:14:39.052942+0000","last_clean":"2024-03-30T15:14:39.052942+0000","last_became_active":"2024-03-30T15:14:00.634218+0000","last_became_peered":"2024-03-30T15:14:00.634218+0000","last_unstale":"2024-03-30T15:14:39.052942+0000","last_undegraded":"2024-03-30T15:14:39.052942+0000","last_fullsized":"2024-03-30T15:14:39.052942+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:45.094026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:11:45.094026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"878'15","reported_seq":1900,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967495+0000","last_change":"2024-03-30T15:14:00.650824+0000","last_active":"2024-03-30T15:14:42.967495+0000","last_peered":"2024-03-30T15:14:42.967495+0000","last_clean":"2024-03-30T15:14:42.967495+0000","last_became_active":"2024-03-30T15:14:00.638157+0000","last_became_peered":"2024-03-30T15:14:00.638157+0000","last_unstale":"2024-03-30T15:14:42.967495+0000","last_undegraded":"2024-03-30T15:14:42.967495+0000","last_fullsized":"2024-03-30T15:14:42.967495+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'13","last_scrub_stamp":"2024-03-30T15:13:06.936305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.936305+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1627,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877693+0000","last_change":"2024-03-30T15:13:34.747694+0000","last_active":"2024-03-30T15:14:37.877693+0000","last_peered":"2024-03-30T15:14:37.877693+0000","last_clean":"2024-03-30T15:14:37.877693+0000","last_became_active":"2024-03-30T15:13:19.280923+0000","last_became_peered":"2024-03-30T15:13:19.280923+0000","last_unstale":"2024-03-30T15:14:37.877693+0000","last_undegraded":"2024-03-30T15:14:37.877693+0000","last_fullsized":"2024-03-30T15:14:37.877693+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:34.747586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:34.747586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.200164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1256,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053197+0000","last_change":"2024-03-30T15:14:17.194256+0000","last_active":"2024-03-30T15:14:39.053197+0000","last_peered":"2024-03-30T15:14:39.053197+0000","last_clean":"2024-03-30T15:14:39.053197+0000","last_became_active":"2024-03-30T15:13:28.637092+0000","last_became_peered":"2024-03-30T15:13:28.637092+0000","last_unstale":"2024-03-30T15:14:39.053197+0000","last_undegraded":"2024-03-30T15:14:39.053197+0000","last_fullsized":"2024-03-30T15:14:39.053197+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.194169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.194169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:19.028824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"903'25","reported_seq":1709,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596586+0000","last_change":"2024-03-30T15:14:32.245733+0000","last_active":"2024-03-30T15:14:42.596586+0000","last_peered":"2024-03-30T15:14:42.596586+0000","last_clean":"2024-03-30T15:14:42.596586+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:14:42.596586+0000","last_undegraded":"2024-03-30T15:14:42.596586+0000","last_fullsized":"2024-03-30T15:14:42.596586+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"903'25","last_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_deep_scrub":"903'25","last_deep_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_clean_scrub_stamp":"2024-03-30T15:14:32.245633+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.205989+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1748,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965762+0000","last_change":"2024-03-30T15:14:29.136087+0000","last_active":"2024-03-30T15:14:42.965762+0000","last_peered":"2024-03-30T15:14:42.965762+0000","last_clean":"2024-03-30T15:14:42.965762+0000","last_became_active":"2024-03-30T15:13:19.277421+0000","last_became_peered":"2024-03-30T15:13:19.277421+0000","last_unstale":"2024-03-30T15:14:42.965762+0000","last_undegraded":"2024-03-30T15:14:42.965762+0000","last_fullsized":"2024-03-30T15:14:42.965762+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.135910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.135910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:49.766429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"852'12","reported_seq":1986,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596223+0000","last_change":"2024-03-30T15:13:30.869874+0000","last_active":"2024-03-30T15:14:42.596223+0000","last_peered":"2024-03-30T15:14:42.596223+0000","last_clean":"2024-03-30T15:14:42.596223+0000","last_became_active":"2024-03-30T15:13:29.615517+0000","last_became_peered":"2024-03-30T15:13:29.615517+0000","last_unstale":"2024-03-30T15:14:42.596223+0000","last_undegraded":"2024-03-30T15:14:42.596223+0000","last_fullsized":"2024-03-30T15:14:42.596223+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"852'12","last_scrub_stamp":"2024-03-30T15:13:30.869737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.869737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.977931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1309,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053411+0000","last_change":"2024-03-30T15:14:00.649300+0000","last_active":"2024-03-30T15:14:39.053411+0000","last_peered":"2024-03-30T15:14:39.053411+0000","last_clean":"2024-03-30T15:14:39.053411+0000","last_became_active":"2024-03-30T15:14:00.638394+0000","last_became_peered":"2024-03-30T15:14:00.638394+0000","last_unstale":"2024-03-30T15:14:39.053411+0000","last_undegraded":"2024-03-30T15:14:39.053411+0000","last_fullsized":"2024-03-30T15:14:39.053411+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:11:40.102459+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:11:40.102459+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1594,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876987+0000","last_change":"2024-03-30T15:13:06.782014+0000","last_active":"2024-03-30T15:14:37.876987+0000","last_peered":"2024-03-30T15:14:37.876987+0000","last_clean":"2024-03-30T15:14:37.876987+0000","last_became_active":"2024-03-30T15:13:03.509862+0000","last_became_peered":"2024-03-30T15:13:03.509862+0000","last_unstale":"2024-03-30T15:14:37.876987+0000","last_undegraded":"2024-03-30T15:14:37.876987+0000","last_fullsized":"2024-03-30T15:14:37.876987+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:06.781928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.781928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"475'24","reported_seq":1720,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876835+0000","last_change":"2024-03-30T15:14:36.715235+0000","last_active":"2024-03-30T15:14:37.876835+0000","last_peered":"2024-03-30T15:14:37.876835+0000","last_clean":"2024-03-30T15:14:37.876835+0000","last_became_active":"2024-03-30T15:13:03.536364+0000","last_became_peered":"2024-03-30T15:13:03.536364+0000","last_unstale":"2024-03-30T15:14:37.876835+0000","last_undegraded":"2024-03-30T15:14:37.876835+0000","last_fullsized":"2024-03-30T15:14:37.876835+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:36.715131+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.715131+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:01.194138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1674,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877294+0000","last_change":"2024-03-30T15:13:04.847959+0000","last_active":"2024-03-30T15:14:37.877294+0000","last_peered":"2024-03-30T15:14:37.877294+0000","last_clean":"2024-03-30T15:14:37.877294+0000","last_became_active":"2024-03-30T15:13:03.539915+0000","last_became_peered":"2024-03-30T15:13:03.539915+0000","last_unstale":"2024-03-30T15:14:37.877294+0000","last_undegraded":"2024-03-30T15:14:37.877294+0000","last_fullsized":"2024-03-30T15:14:37.877294+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:04.847864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:04.847864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1758,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595017+0000","last_change":"2024-03-30T15:14:00.630183+0000","last_active":"2024-03-30T15:14:42.595017+0000","last_peered":"2024-03-30T15:14:42.595017+0000","last_clean":"2024-03-30T15:14:42.595017+0000","last_became_active":"2024-03-30T15:14:00.629833+0000","last_became_peered":"2024-03-30T15:14:00.629833+0000","last_unstale":"2024-03-30T15:14:42.595017+0000","last_undegraded":"2024-03-30T15:14:42.595017+0000","last_fullsized":"2024-03-30T15:14:42.595017+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:57.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:12:57.294119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1214,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052739+0000","last_change":"2024-03-30T15:14:00.635564+0000","last_active":"2024-03-30T15:14:39.052739+0000","last_peered":"2024-03-30T15:14:39.052739+0000","last_clean":"2024-03-30T15:14:39.052739+0000","last_became_active":"2024-03-30T15:14:00.635207+0000","last_became_peered":"2024-03-30T15:14:00.635207+0000","last_unstale":"2024-03-30T15:14:39.052739+0000","last_undegraded":"2024-03-30T15:14:39.052739+0000","last_fullsized":"2024-03-30T15:14:39.052739+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":70,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876592+0000","last_change":"2024-03-30T15:14:29.787668+0000","last_active":"2024-03-30T15:14:37.876592+0000","last_peered":"2024-03-30T15:14:37.876592+0000","last_clean":"2024-03-30T15:14:37.876592+0000","last_became_active":"2024-03-30T15:14:09.726417+0000","last_became_peered":"2024-03-30T15:14:09.726417+0000","last_unstale":"2024-03-30T15:14:37.876592+0000","last_undegraded":"2024-03-30T15:14:37.876592+0000","last_fullsized":"2024-03-30T15:14:37.876592+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.787527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.787527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:37.823723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"475'15","reported_seq":1776,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595247+0000","last_change":"2024-03-30T15:13:38.465288+0000","last_active":"2024-03-30T15:14:42.595247+0000","last_peered":"2024-03-30T15:14:42.595247+0000","last_clean":"2024-03-30T15:14:42.595247+0000","last_became_active":"2024-03-30T15:13:19.293272+0000","last_became_peered":"2024-03-30T15:13:19.293272+0000","last_unstale":"2024-03-30T15:14:42.595247+0000","last_undegraded":"2024-03-30T15:14:42.595247+0000","last_fullsized":"2024-03-30T15:14:42.595247+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:38.465198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:38.465198+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1758,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967122+0000","last_change":"2024-03-30T15:14:19.969427+0000","last_active":"2024-03-30T15:14:42.967122+0000","last_peered":"2024-03-30T15:14:42.967122+0000","last_clean":"2024-03-30T15:14:42.967122+0000","last_became_active":"2024-03-30T15:13:03.525235+0000","last_became_peered":"2024-03-30T15:13:03.525235+0000","last_unstale":"2024-03-30T15:14:42.967122+0000","last_undegraded":"2024-03-30T15:14:42.967122+0000","last_fullsized":"2024-03-30T15:14:42.967122+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:14:19.969342+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.969342+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.130177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":865,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053420+0000","last_change":"2024-03-30T15:13:29.620544+0000","last_active":"2024-03-30T15:14:39.053420+0000","last_peered":"2024-03-30T15:14:39.053420+0000","last_clean":"2024-03-30T15:14:39.053420+0000","last_became_active":"2024-03-30T15:13:29.619329+0000","last_became_peered":"2024-03-30T15:13:29.619329+0000","last_unstale":"2024-03-30T15:14:39.053420+0000","last_undegraded":"2024-03-30T15:14:39.053420+0000","last_fullsized":"2024-03-30T15:14:39.053420+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:46.479258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:46.479258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1328,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052652+0000","last_change":"2024-03-30T15:14:29.215562+0000","last_active":"2024-03-30T15:14:39.052652+0000","last_peered":"2024-03-30T15:14:39.052652+0000","last_clean":"2024-03-30T15:14:39.052652+0000","last_became_active":"2024-03-30T15:13:28.634012+0000","last_became_peered":"2024-03-30T15:13:28.634012+0000","last_unstale":"2024-03-30T15:14:39.052652+0000","last_undegraded":"2024-03-30T15:14:39.052652+0000","last_fullsized":"2024-03-30T15:14:39.052652+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.215490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.215490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.444348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1218,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052745+0000","last_change":"2024-03-30T15:13:31.137742+0000","last_active":"2024-03-30T15:14:39.052745+0000","last_peered":"2024-03-30T15:14:39.052745+0000","last_clean":"2024-03-30T15:14:39.052745+0000","last_became_active":"2024-03-30T15:13:28.626496+0000","last_became_peered":"2024-03-30T15:13:28.626496+0000","last_unstale":"2024-03-30T15:14:39.052745+0000","last_undegraded":"2024-03-30T15:14:39.052745+0000","last_fullsized":"2024-03-30T15:14:39.052745+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.137663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.137663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.939202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"475'19","reported_seq":1677,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966006+0000","last_change":"2024-03-30T15:10:55.115248+0000","last_active":"2024-03-30T15:14:42.966006+0000","last_peered":"2024-03-30T15:14:42.966006+0000","last_clean":"2024-03-30T15:14:42.966006+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:14:42.966006+0000","last_undegraded":"2024-03-30T15:14:42.966006+0000","last_fullsized":"2024-03-30T15:14:42.966006+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:10:55.115152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:10:55.115152+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596062+0000","last_change":"2024-03-30T15:14:20.859711+0000","last_active":"2024-03-30T15:14:42.596062+0000","last_peered":"2024-03-30T15:14:42.596062+0000","last_clean":"2024-03-30T15:14:42.596062+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:14:42.596062+0000","last_undegraded":"2024-03-30T15:14:42.596062+0000","last_fullsized":"2024-03-30T15:14:42.596062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.859619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.859619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:36.845652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1735,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966719+0000","last_change":"2024-03-30T15:14:27.114393+0000","last_active":"2024-03-30T15:14:42.966719+0000","last_peered":"2024-03-30T15:14:42.966719+0000","last_clean":"2024-03-30T15:14:42.966719+0000","last_became_active":"2024-03-30T15:13:03.540174+0000","last_became_peered":"2024-03-30T15:13:03.540174+0000","last_unstale":"2024-03-30T15:14:42.966719+0000","last_undegraded":"2024-03-30T15:14:42.966719+0000","last_fullsized":"2024-03-30T15:14:42.966719+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:27.114273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_clean_scrub_stamp":"2024-03-30T15:14:27.114273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:35.103581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1227,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053530+0000","last_change":"2024-03-30T15:14:20.165761+0000","last_active":"2024-03-30T15:14:39.053530+0000","last_peered":"2024-03-30T15:14:39.053530+0000","last_clean":"2024-03-30T15:14:39.053530+0000","last_became_active":"2024-03-30T15:13:28.624430+0000","last_became_peered":"2024-03-30T15:13:28.624430+0000","last_unstale":"2024-03-30T15:14:39.053530+0000","last_undegraded":"2024-03-30T15:14:39.053530+0000","last_fullsized":"2024-03-30T15:14:39.053530+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.165646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.165646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.792269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1660,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595673+0000","last_change":"2024-03-30T15:14:23.181759+0000","last_active":"2024-03-30T15:14:42.595673+0000","last_peered":"2024-03-30T15:14:42.595673+0000","last_clean":"2024-03-30T15:14:42.595673+0000","last_became_active":"2024-03-30T15:13:03.527402+0000","last_became_peered":"2024-03-30T15:13:03.527402+0000","last_unstale":"2024-03-30T15:14:42.595673+0000","last_undegraded":"2024-03-30T15:14:42.595673+0000","last_fullsized":"2024-03-30T15:14:42.595673+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.181637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.181637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:28.900759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"475'8","reported_seq":1342,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.457532+0000","last_change":"2024-03-30T15:14:23.386867+0000","last_active":"2024-03-30T15:14:43.457532+0000","last_peered":"2024-03-30T15:14:43.457532+0000","last_clean":"2024-03-30T15:14:43.457532+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:14:43.457532+0000","last_undegraded":"2024-03-30T15:14:43.457532+0000","last_fullsized":"2024-03-30T15:14:43.457532+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:14:23.386767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.386767+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:52.358728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1316,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053572+0000","last_change":"2024-03-30T15:13:28.626812+0000","last_active":"2024-03-30T15:14:39.053572+0000","last_peered":"2024-03-30T15:14:39.053572+0000","last_clean":"2024-03-30T15:14:39.053572+0000","last_became_active":"2024-03-30T15:13:28.626493+0000","last_became_peered":"2024-03-30T15:13:28.626493+0000","last_unstale":"2024-03-30T15:14:39.053572+0000","last_undegraded":"2024-03-30T15:14:39.053572+0000","last_fullsized":"2024-03-30T15:14:39.053572+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1767,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968299+0000","last_change":"2024-03-30T15:14:34.385293+0000","last_active":"2024-03-30T15:14:42.968299+0000","last_peered":"2024-03-30T15:14:42.968299+0000","last_clean":"2024-03-30T15:14:42.968299+0000","last_became_active":"2024-03-30T15:14:09.727046+0000","last_became_peered":"2024-03-30T15:14:09.727046+0000","last_unstale":"2024-03-30T15:14:42.968299+0000","last_undegraded":"2024-03-30T15:14:42.968299+0000","last_fullsized":"2024-03-30T15:14:42.968299+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:34.385192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:34.385192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:45.959150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1728,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.966087+0000","last_change":"2024-03-30T15:13:28.065073+0000","last_active":"2024-03-30T15:14:42.966087+0000","last_peered":"2024-03-30T15:14:42.966087+0000","last_clean":"2024-03-30T15:14:42.966087+0000","last_became_active":"2024-03-30T15:13:19.277971+0000","last_became_peered":"2024-03-30T15:13:19.277971+0000","last_unstale":"2024-03-30T15:14:42.966087+0000","last_undegraded":"2024-03-30T15:14:42.966087+0000","last_fullsized":"2024-03-30T15:14:42.966087+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.064921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.064921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.324547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1649,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969405+0000","last_change":"2024-03-30T15:14:25.522723+0000","last_active":"2024-03-30T15:14:42.969405+0000","last_peered":"2024-03-30T15:14:42.969405+0000","last_clean":"2024-03-30T15:14:42.969405+0000","last_became_active":"2024-03-30T15:13:19.281726+0000","last_became_peered":"2024-03-30T15:13:19.281726+0000","last_unstale":"2024-03-30T15:14:42.969405+0000","last_undegraded":"2024-03-30T15:14:42.969405+0000","last_fullsized":"2024-03-30T15:14:42.969405+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_clean_scrub_stamp":"2024-03-30T15:14:25.522636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:49.743677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"475'24","reported_seq":1775,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969388+0000","last_change":"2024-03-30T15:14:33.413179+0000","last_active":"2024-03-30T15:14:42.969388+0000","last_peered":"2024-03-30T15:14:42.969388+0000","last_clean":"2024-03-30T15:14:42.969388+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:14:42.969388+0000","last_undegraded":"2024-03-30T15:14:42.969388+0000","last_fullsized":"2024-03-30T15:14:42.969388+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:33.413037+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.413037+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.439782+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1191,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052771+0000","last_change":"2024-03-30T15:13:28.631279+0000","last_active":"2024-03-30T15:14:39.052771+0000","last_peered":"2024-03-30T15:14:39.052771+0000","last_clean":"2024-03-30T15:14:39.052771+0000","last_became_active":"2024-03-30T15:13:28.627893+0000","last_became_peered":"2024-03-30T15:13:28.627893+0000","last_unstale":"2024-03-30T15:14:39.052771+0000","last_undegraded":"2024-03-30T15:14:39.052771+0000","last_fullsized":"2024-03-30T15:14:39.052771+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1312,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053674+0000","last_change":"2024-03-30T15:14:31.203592+0000","last_active":"2024-03-30T15:14:39.053674+0000","last_peered":"2024-03-30T15:14:39.053674+0000","last_clean":"2024-03-30T15:14:39.053674+0000","last_became_active":"2024-03-30T15:13:52.594341+0000","last_became_peered":"2024-03-30T15:13:52.594341+0000","last_unstale":"2024-03-30T15:14:39.053674+0000","last_undegraded":"2024-03-30T15:14:39.053674+0000","last_fullsized":"2024-03-30T15:14:39.053674+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:31.203492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:31.203492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.217512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1751,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.597716+0000","last_change":"2024-03-30T15:13:44.520601+0000","last_active":"2024-03-30T15:14:42.597716+0000","last_peered":"2024-03-30T15:14:42.597716+0000","last_clean":"2024-03-30T15:14:42.597716+0000","last_became_active":"2024-03-30T15:13:03.517348+0000","last_became_peered":"2024-03-30T15:13:03.517348+0000","last_unstale":"2024-03-30T15:14:42.597716+0000","last_undegraded":"2024-03-30T15:14:42.597716+0000","last_fullsized":"2024-03-30T15:14:42.597716+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.520521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:59.701443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1668,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969141+0000","last_change":"2024-03-30T15:14:09.727891+0000","last_active":"2024-03-30T15:14:42.969141+0000","last_peered":"2024-03-30T15:14:42.969141+0000","last_clean":"2024-03-30T15:14:42.969141+0000","last_became_active":"2024-03-30T15:14:09.727015+0000","last_became_peered":"2024-03-30T15:14:09.727015+0000","last_unstale":"2024-03-30T15:14:42.969141+0000","last_undegraded":"2024-03-30T15:14:42.969141+0000","last_fullsized":"2024-03-30T15:14:42.969141+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:45.725848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:11:45.725848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.370971+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"475'25","reported_seq":1350,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053537+0000","last_change":"2024-03-30T15:14:00.656230+0000","last_active":"2024-03-30T15:14:39.053537+0000","last_peered":"2024-03-30T15:14:39.053537+0000","last_clean":"2024-03-30T15:14:39.053537+0000","last_became_active":"2024-03-30T15:14:00.638539+0000","last_became_peered":"2024-03-30T15:14:00.638539+0000","last_unstale":"2024-03-30T15:14:39.053537+0000","last_undegraded":"2024-03-30T15:14:39.053537+0000","last_fullsized":"2024-03-30T15:14:39.053537+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:11:47.028449+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.028449+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1776,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595052+0000","last_change":"2024-03-30T15:14:34.266129+0000","last_active":"2024-03-30T15:14:42.595052+0000","last_peered":"2024-03-30T15:14:42.595052+0000","last_clean":"2024-03-30T15:14:42.595052+0000","last_became_active":"2024-03-30T15:14:09.726865+0000","last_became_peered":"2024-03-30T15:14:09.726865+0000","last_unstale":"2024-03-30T15:14:42.595052+0000","last_undegraded":"2024-03-30T15:14:42.595052+0000","last_fullsized":"2024-03-30T15:14:42.595052+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:14:34.266038+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:14:34.266038+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.734977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"883'15","reported_seq":2026,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969283+0000","last_change":"2024-03-30T15:14:09.732603+0000","last_active":"2024-03-30T15:14:42.969283+0000","last_peered":"2024-03-30T15:14:42.969283+0000","last_clean":"2024-03-30T15:14:42.969283+0000","last_became_active":"2024-03-30T15:14:09.727307+0000","last_became_peered":"2024-03-30T15:14:09.727307+0000","last_unstale":"2024-03-30T15:14:42.969283+0000","last_undegraded":"2024-03-30T15:14:42.969283+0000","last_fullsized":"2024-03-30T15:14:42.969283+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:11:02.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:02.788609+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1733,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967654+0000","last_change":"2024-03-30T15:13:41.111641+0000","last_active":"2024-03-30T15:14:42.967654+0000","last_peered":"2024-03-30T15:14:42.967654+0000","last_clean":"2024-03-30T15:14:42.967654+0000","last_became_active":"2024-03-30T15:13:19.271296+0000","last_became_peered":"2024-03-30T15:13:19.271296+0000","last_unstale":"2024-03-30T15:14:42.967654+0000","last_undegraded":"2024-03-30T15:14:42.967654+0000","last_fullsized":"2024-03-30T15:14:42.967654+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:13:41.111529+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T15:13:41.111529+0000","last_clean_scrub_stamp":"2024-03-30T15:13:41.111529+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.789083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1663,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969001+0000","last_change":"2024-03-30T15:13:52.588733+0000","last_active":"2024-03-30T15:14:42.969001+0000","last_peered":"2024-03-30T15:14:42.969001+0000","last_clean":"2024-03-30T15:14:42.969001+0000","last_became_active":"2024-03-30T15:13:52.588042+0000","last_became_peered":"2024-03-30T15:13:52.588042+0000","last_unstale":"2024-03-30T15:14:42.969001+0000","last_undegraded":"2024-03-30T15:14:42.969001+0000","last_fullsized":"2024-03-30T15:14:42.969001+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:02.667523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:02.667523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.371086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"475'18","reported_seq":1706,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.597228+0000","last_change":"2024-03-30T15:13:42.509798+0000","last_active":"2024-03-30T15:14:42.597228+0000","last_peered":"2024-03-30T15:14:42.597228+0000","last_clean":"2024-03-30T15:14:42.597228+0000","last_became_active":"2024-03-30T15:13:03.516406+0000","last_became_peered":"2024-03-30T15:13:03.516406+0000","last_unstale":"2024-03-30T15:14:42.597228+0000","last_undegraded":"2024-03-30T15:14:42.597228+0000","last_fullsized":"2024-03-30T15:14:42.597228+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:13:42.509704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.509704+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.374132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1639,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877232+0000","last_change":"2024-03-30T15:13:33.729411+0000","last_active":"2024-03-30T15:14:37.877232+0000","last_peered":"2024-03-30T15:14:37.877232+0000","last_clean":"2024-03-30T15:14:37.877232+0000","last_became_active":"2024-03-30T15:13:19.285258+0000","last_became_peered":"2024-03-30T15:13:19.285258+0000","last_unstale":"2024-03-30T15:14:37.877232+0000","last_undegraded":"2024-03-30T15:14:37.877232+0000","last_fullsized":"2024-03-30T15:14:37.877232+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:33.729289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_clean_scrub_stamp":"2024-03-30T15:13:33.729289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.358577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1631,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877183+0000","last_change":"2024-03-30T15:13:26.857132+0000","last_active":"2024-03-30T15:14:37.877183+0000","last_peered":"2024-03-30T15:14:37.877183+0000","last_clean":"2024-03-30T15:14:37.877183+0000","last_became_active":"2024-03-30T15:13:19.291186+0000","last_became_peered":"2024-03-30T15:13:19.291186+0000","last_unstale":"2024-03-30T15:14:37.877183+0000","last_undegraded":"2024-03-30T15:14:37.877183+0000","last_fullsized":"2024-03-30T15:14:37.877183+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:26.857055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:26.857055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:42.881521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1645,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969496+0000","last_change":"2024-03-30T15:12:50.677660+0000","last_active":"2024-03-30T15:14:42.969496+0000","last_peered":"2024-03-30T15:14:42.969496+0000","last_clean":"2024-03-30T15:14:42.969496+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:14:42.969496+0000","last_undegraded":"2024-03-30T15:14:42.969496+0000","last_fullsized":"2024-03-30T15:14:42.969496+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:50.677579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:12:50.677579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1646,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596766+0000","last_change":"2024-03-30T15:14:00.638500+0000","last_active":"2024-03-30T15:14:42.596766+0000","last_peered":"2024-03-30T15:14:42.596766+0000","last_clean":"2024-03-30T15:14:42.596766+0000","last_became_active":"2024-03-30T15:14:00.638153+0000","last_became_peered":"2024-03-30T15:14:00.638153+0000","last_unstale":"2024-03-30T15:14:42.596766+0000","last_undegraded":"2024-03-30T15:14:42.596766+0000","last_fullsized":"2024-03-30T15:14:42.596766+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:01.111602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:01.111602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"475'34","reported_seq":105,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.596030+0000","last_change":"2024-03-30T15:13:52.602453+0000","last_active":"2024-03-30T15:14:37.596030+0000","last_peered":"2024-03-30T15:14:37.596030+0000","last_clean":"2024-03-30T15:14:37.596030+0000","last_became_active":"2024-03-30T15:13:52.587538+0000","last_became_peered":"2024-03-30T15:13:52.587538+0000","last_unstale":"2024-03-30T15:14:37.596030+0000","last_undegraded":"2024-03-30T15:14:37.596030+0000","last_fullsized":"2024-03-30T15:14:37.596030+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:37.497100+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.497100+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:03.512478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1313,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053122+0000","last_change":"2024-03-30T15:13:43.145133+0000","last_active":"2024-03-30T15:14:39.053122+0000","last_peered":"2024-03-30T15:14:39.053122+0000","last_clean":"2024-03-30T15:14:39.053122+0000","last_became_active":"2024-03-30T15:13:28.643964+0000","last_became_peered":"2024-03-30T15:13:28.643964+0000","last_unstale":"2024-03-30T15:14:39.053122+0000","last_undegraded":"2024-03-30T15:14:39.053122+0000","last_fullsized":"2024-03-30T15:14:39.053122+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:43.145050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:43.145050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.720473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1774,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.597583+0000","last_change":"2024-03-30T15:14:20.084320+0000","last_active":"2024-03-30T15:14:42.597583+0000","last_peered":"2024-03-30T15:14:42.597583+0000","last_clean":"2024-03-30T15:14:42.597583+0000","last_became_active":"2024-03-30T15:14:09.724748+0000","last_became_peered":"2024-03-30T15:14:09.724748+0000","last_unstale":"2024-03-30T15:14:42.597583+0000","last_undegraded":"2024-03-30T15:14:42.597583+0000","last_fullsized":"2024-03-30T15:14:42.597583+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.084242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.084242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:29.241716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1653,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877616+0000","last_change":"2024-03-30T15:13:03.545413+0000","last_active":"2024-03-30T15:14:37.877616+0000","last_peered":"2024-03-30T15:14:37.877616+0000","last_clean":"2024-03-30T15:14:37.877616+0000","last_became_active":"2024-03-30T15:13:03.536453+0000","last_became_peered":"2024-03-30T15:13:03.536453+0000","last_unstale":"2024-03-30T15:14:37.877616+0000","last_undegraded":"2024-03-30T15:14:37.877616+0000","last_fullsized":"2024-03-30T15:14:37.877616+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:33.910931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:11:33.910931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595246+0000","last_change":"2024-03-30T15:13:34.474161+0000","last_active":"2024-03-30T15:14:42.595246+0000","last_peered":"2024-03-30T15:14:42.595246+0000","last_clean":"2024-03-30T15:14:42.595246+0000","last_became_active":"2024-03-30T15:13:28.632549+0000","last_became_peered":"2024-03-30T15:13:28.632549+0000","last_unstale":"2024-03-30T15:14:42.595246+0000","last_undegraded":"2024-03-30T15:14:42.595246+0000","last_fullsized":"2024-03-30T15:14:42.595246+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:34.474067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:13:34.474067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.521823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"475'7","reported_seq":1706,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876824+0000","last_change":"2024-03-30T15:13:03.510230+0000","last_active":"2024-03-30T15:14:37.876824+0000","last_peered":"2024-03-30T15:14:37.876824+0000","last_clean":"2024-03-30T15:14:37.876824+0000","last_became_active":"2024-03-30T15:13:03.509107+0000","last_became_peered":"2024-03-30T15:13:03.509107+0000","last_unstale":"2024-03-30T15:14:37.876824+0000","last_undegraded":"2024-03-30T15:14:37.876824+0000","last_fullsized":"2024-03-30T15:14:37.876824+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:11:41.878570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:11:41.878570+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1745,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596340+0000","last_change":"2024-03-30T15:14:21.851335+0000","last_active":"2024-03-30T15:14:42.596340+0000","last_peered":"2024-03-30T15:14:42.596340+0000","last_clean":"2024-03-30T15:14:42.596340+0000","last_became_active":"2024-03-30T15:13:03.518001+0000","last_became_peered":"2024-03-30T15:13:03.518001+0000","last_unstale":"2024-03-30T15:14:42.596340+0000","last_undegraded":"2024-03-30T15:14:42.596340+0000","last_fullsized":"2024-03-30T15:14:42.596340+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:21.851223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.851223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:47.274601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1727,"reported_epoch":908,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965983+0000","last_change":"2024-03-30T15:13:03.529355+0000","last_active":"2024-03-30T15:14:42.965983+0000","last_peered":"2024-03-30T15:14:42.965983+0000","last_clean":"2024-03-30T15:14:42.965983+0000","last_became_active":"2024-03-30T15:13:03.526657+0000","last_became_peered":"2024-03-30T15:13:03.526657+0000","last_unstale":"2024-03-30T15:14:42.965983+0000","last_undegraded":"2024-03-30T15:14:42.965983+0000","last_fullsized":"2024-03-30T15:14:42.965983+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:54.102039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:10:54.102039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1740,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595522+0000","last_change":"2024-03-30T15:14:00.638502+0000","last_active":"2024-03-30T15:14:42.595522+0000","last_peered":"2024-03-30T15:14:42.595522+0000","last_clean":"2024-03-30T15:14:42.595522+0000","last_became_active":"2024-03-30T15:14:00.638063+0000","last_became_peered":"2024-03-30T15:14:00.638063+0000","last_unstale":"2024-03-30T15:14:42.595522+0000","last_undegraded":"2024-03-30T15:14:42.595522+0000","last_fullsized":"2024-03-30T15:14:42.595522+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:17.973865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:12:17.973865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1646,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594823+0000","last_change":"2024-03-30T15:14:26.138759+0000","last_active":"2024-03-30T15:14:42.594823+0000","last_peered":"2024-03-30T15:14:42.594823+0000","last_clean":"2024-03-30T15:14:42.594823+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:14:42.594823+0000","last_undegraded":"2024-03-30T15:14:42.594823+0000","last_fullsized":"2024-03-30T15:14:42.594823+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_clean_scrub_stamp":"2024-03-30T15:14:26.138647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.059280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"475'15","reported_seq":72,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876753+0000","last_change":"2024-03-30T15:14:09.736900+0000","last_active":"2024-03-30T15:14:37.876753+0000","last_peered":"2024-03-30T15:14:37.876753+0000","last_clean":"2024-03-30T15:14:37.876753+0000","last_became_active":"2024-03-30T15:14:09.725322+0000","last_became_peered":"2024-03-30T15:14:09.725322+0000","last_unstale":"2024-03-30T15:14:37.876753+0000","last_undegraded":"2024-03-30T15:14:37.876753+0000","last_fullsized":"2024-03-30T15:14:37.876753+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:26.727540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:26.727540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:40.304956+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1737,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596009+0000","last_change":"2024-03-30T15:13:03.533616+0000","last_active":"2024-03-30T15:14:42.596009+0000","last_peered":"2024-03-30T15:14:42.596009+0000","last_clean":"2024-03-30T15:14:42.596009+0000","last_became_active":"2024-03-30T15:13:03.532804+0000","last_became_peered":"2024-03-30T15:13:03.532804+0000","last_unstale":"2024-03-30T15:14:42.596009+0000","last_undegraded":"2024-03-30T15:14:42.596009+0000","last_fullsized":"2024-03-30T15:14:42.596009+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:42.869533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:11:42.869533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1729,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595506+0000","last_change":"2024-03-30T15:13:39.898458+0000","last_active":"2024-03-30T15:14:42.595506+0000","last_peered":"2024-03-30T15:14:42.595506+0000","last_clean":"2024-03-30T15:14:42.595506+0000","last_became_active":"2024-03-30T15:13:28.624284+0000","last_became_peered":"2024-03-30T15:13:28.624284+0000","last_unstale":"2024-03-30T15:14:42.595506+0000","last_undegraded":"2024-03-30T15:14:42.595506+0000","last_fullsized":"2024-03-30T15:14:42.595506+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.898372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.543464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1304,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053037+0000","last_change":"2024-03-30T15:13:40.224279+0000","last_active":"2024-03-30T15:14:39.053037+0000","last_peered":"2024-03-30T15:14:39.053037+0000","last_clean":"2024-03-30T15:14:39.053037+0000","last_became_active":"2024-03-30T15:13:28.631113+0000","last_became_peered":"2024-03-30T15:13:28.631113+0000","last_unstale":"2024-03-30T15:14:39.053037+0000","last_undegraded":"2024-03-30T15:14:39.053037+0000","last_fullsized":"2024-03-30T15:14:39.053037+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.224204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.715829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1659,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595250+0000","last_change":"2024-03-30T15:14:09.727337+0000","last_active":"2024-03-30T15:14:42.595250+0000","last_peered":"2024-03-30T15:14:42.595250+0000","last_clean":"2024-03-30T15:14:42.595250+0000","last_became_active":"2024-03-30T15:14:09.726973+0000","last_became_peered":"2024-03-30T15:14:09.726973+0000","last_unstale":"2024-03-30T15:14:42.595250+0000","last_undegraded":"2024-03-30T15:14:42.595250+0000","last_fullsized":"2024-03-30T15:14:42.595250+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:05.038649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:05.038649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"475'34","reported_seq":1586,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.595536+0000","last_change":"2024-03-30T15:13:42.244532+0000","last_active":"2024-03-30T15:14:37.595536+0000","last_peered":"2024-03-30T15:14:37.595536+0000","last_clean":"2024-03-30T15:14:37.595536+0000","last_became_active":"2024-03-30T15:13:28.614224+0000","last_became_peered":"2024-03-30T15:13:28.614224+0000","last_unstale":"2024-03-30T15:14:37.595536+0000","last_undegraded":"2024-03-30T15:14:37.595536+0000","last_fullsized":"2024-03-30T15:14:37.595536+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:42.244406+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.244406+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:52.631648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1647,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969010+0000","last_change":"2024-03-30T15:13:35.659341+0000","last_active":"2024-03-30T15:14:42.969010+0000","last_peered":"2024-03-30T15:14:42.969010+0000","last_clean":"2024-03-30T15:14:42.969010+0000","last_became_active":"2024-03-30T15:13:19.281642+0000","last_became_peered":"2024-03-30T15:13:19.281642+0000","last_unstale":"2024-03-30T15:14:42.969010+0000","last_undegraded":"2024-03-30T15:14:42.969010+0000","last_fullsized":"2024-03-30T15:14:42.969010+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:35.659115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:13:35.659115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:47.209280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":62,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876373+0000","last_change":"2024-03-30T15:14:09.725445+0000","last_active":"2024-03-30T15:14:37.876373+0000","last_peered":"2024-03-30T15:14:37.876373+0000","last_clean":"2024-03-30T15:14:37.876373+0000","last_became_active":"2024-03-30T15:14:09.725019+0000","last_became_peered":"2024-03-30T15:14:09.725019+0000","last_unstale":"2024-03-30T15:14:37.876373+0000","last_undegraded":"2024-03-30T15:14:37.876373+0000","last_fullsized":"2024-03-30T15:14:37.876373+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:43.743015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:43.743015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.009132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":85,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.457901+0000","last_change":"2024-03-30T15:14:22.358228+0000","last_active":"2024-03-30T15:14:43.457901+0000","last_peered":"2024-03-30T15:14:43.457901+0000","last_clean":"2024-03-30T15:14:43.457901+0000","last_became_active":"2024-03-30T15:14:00.631826+0000","last_became_peered":"2024-03-30T15:14:00.631826+0000","last_unstale":"2024-03-30T15:14:43.457901+0000","last_undegraded":"2024-03-30T15:14:43.457901+0000","last_fullsized":"2024-03-30T15:14:43.457901+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.358146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.358146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.743258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1254,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.052764+0000","last_change":"2024-03-30T15:14:16.147467+0000","last_active":"2024-03-30T15:14:39.052764+0000","last_peered":"2024-03-30T15:14:39.052764+0000","last_clean":"2024-03-30T15:14:39.052764+0000","last_became_active":"2024-03-30T15:13:28.645477+0000","last_became_peered":"2024-03-30T15:13:28.645477+0000","last_unstale":"2024-03-30T15:14:39.052764+0000","last_undegraded":"2024-03-30T15:14:39.052764+0000","last_fullsized":"2024-03-30T15:14:39.052764+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.147377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.147377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:16.240357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"475'15","reported_seq":1684,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.876631+0000","last_change":"2024-03-30T15:13:36.760503+0000","last_active":"2024-03-30T15:14:37.876631+0000","last_peered":"2024-03-30T15:14:37.876631+0000","last_clean":"2024-03-30T15:14:37.876631+0000","last_became_active":"2024-03-30T15:13:19.279835+0000","last_became_peered":"2024-03-30T15:13:19.279835+0000","last_unstale":"2024-03-30T15:14:37.876631+0000","last_undegraded":"2024-03-30T15:14:37.876631+0000","last_fullsized":"2024-03-30T15:14:37.876631+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:36.760405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.760405+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:40.249543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1334,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:39.053460+0000","last_change":"2024-03-30T15:13:52.592354+0000","last_active":"2024-03-30T15:14:39.053460+0000","last_peered":"2024-03-30T15:14:39.053460+0000","last_clean":"2024-03-30T15:14:39.053460+0000","last_became_active":"2024-03-30T15:13:52.592009+0000","last_became_peered":"2024-03-30T15:13:52.592009+0000","last_unstale":"2024-03-30T15:14:39.053460+0000","last_undegraded":"2024-03-30T15:14:39.053460+0000","last_fullsized":"2024-03-30T15:14:39.053460+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.090246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1738,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594724+0000","last_change":"2024-03-30T15:13:03.420409+0000","last_active":"2024-03-30T15:14:42.594724+0000","last_peered":"2024-03-30T15:14:42.594724+0000","last_clean":"2024-03-30T15:14:42.594724+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:14:42.594724+0000","last_undegraded":"2024-03-30T15:14:42.594724+0000","last_fullsized":"2024-03-30T15:14:42.594724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:03.420327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:13:03.420327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1655,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-03-30T15:14:37.877053+0000","last_change":"2024-03-30T15:13:03.529442+0000","last_active":"2024-03-30T15:14:37.877053+0000","last_peered":"2024-03-30T15:14:37.877053+0000","last_clean":"2024-03-30T15:14:37.877053+0000","last_became_active":"2024-03-30T15:13:03.529100+0000","last_became_peered":"2024-03-30T15:13:03.529100+0000","last_unstale":"2024-03-30T15:14:37.877053+0000","last_undegraded":"2024-03-30T15:14:37.877053+0000","last_fullsized":"2024-03-30T15:14:37.877053+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:35.859491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:11:35.859491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1194,"num_read_kb":1189,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":2008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005302,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107428,"kb_used_data":7980,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93637724,"statfs":{"total":95995035648,"available":95885029376,"internally_reserved":0,"allocated":8171520,"data_stored":4279862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299643,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107092,"kb_used_data":7400,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93638060,"statfs":{"total":95995035648,"available":95885373440,"internally_reserved":0,"allocated":7577600,"data_stored":3820247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107128,"kb_used_data":7436,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93638024,"statfs":{"total":95995035648,"available":95885336576,"internally_reserved":0,"allocated":7614464,"data_stored":3821449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":850,"seq":3650722201616,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40892,"kb_used_data":7880,"kb_used_omap":17,"kb_used_meta":32750,"kb_avail":93704260,"statfs":{"total":95995035648,"available":95953162240,"internally_reserved":0,"allocated":8069120,"data_stored":4080643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17527,"internal_metadata":33536905},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":840,"seq":3607772528658,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41336,"kb_used_data":8112,"kb_used_omap":46,"kb_used_meta":32465,"kb_avail":93703816,"statfs":{"total":95995035648,"available":95952707584,"internally_reserved":0,"allocated":8306688,"data_stored":4341984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47515,"internal_metadata":33244773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117259,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111864,"kb_used_data":8256,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633288,"statfs":{"total":95995035648,"available":95880486912,"internally_reserved":0,"allocated":8454144,"data_stored":4542389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":832,"seq":3573412790293,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40764,"kb_used_data":7880,"kb_used_omap":37,"kb_used_meta":32602,"kb_avail":93704388,"statfs":{"total":95995035648,"available":95953293312,"internally_reserved":0,"allocated":8069120,"data_stored":4079964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38119,"internal_metadata":33385241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659392,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111844,"kb_used_data":8236,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633308,"statfs":{"total":95995035648,"available":95880507392,"internally_reserved":0,"allocated":8433664,"data_stored":4548652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:14:46.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:14:46.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:46 smithi028 ceph-mon[41614]: from='client.25989 v1:172.21.15.28:0/2334064084' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:46.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:46 smithi028 ceph-mon[29876]: from='client.25989 v1:172.21.15.28:0/2334064084' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:46.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:46 smithi134 ceph-mon[33714]: from='client.25989 v1:172.21.15.28:0/2334064084' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:47.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:47 smithi028 ceph-mon[29876]: pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:14:47.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:47 smithi028 ceph-mon[41614]: pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:14:47.680 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:47 smithi134 ceph-mon[33714]: pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:14:48.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.737+0000 7f3f58e65700 1 -- 172.21.15.28:0/694033451 >> v1:172.21.15.28:6789/0 conn(0x7f3f5410ea80 legacy=0x7f3f54110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.737+0000 7f3f58e65700 1 -- 172.21.15.28:0/694033451 shutdown_connections 2024-03-30T15:14:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.737+0000 7f3f58e65700 1 -- 172.21.15.28:0/694033451 >> 172.21.15.28:0/694033451 conn(0x7f3f540761d0 msgr2=0x7f3f540765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.738+0000 7f3f58e65700 1 -- 172.21.15.28:0/694033451 shutdown_connections 2024-03-30T15:14:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.738+0000 7f3f58e65700 1 -- 172.21.15.28:0/694033451 wait complete. 2024-03-30T15:14:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.738+0000 7f3f58e65700 1 librados: starting msgr at 2024-03-30T15:14:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.738+0000 7f3f58e65700 1 librados: starting objecter 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.738+0000 7f3f58e65700 1 Processor -- start 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 -- start start 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 librados: setting wanted keys 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 librados: calling monclient init 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f541a93f0 con 0x7f3f54108110 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f541aa610 con 0x7f3f5410ae80 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f58e65700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f541ab850 con 0x7f3f5410ea80 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f5259c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f54108110 0x7f3f54107500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37470/0 (socket says 172.21.15.28:37470) 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f52d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f3f5410ea80 0x7f3f541a7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47238/0 (socket says 172.21.15.28:47238) 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f5259c700 1 -- 172.21.15.28:0/2151169666 learned_addr learned my addr 172.21.15.28:0/2151169666 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter ms_handle_connect 0x7f3f54108110 2024-03-30T15:14:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter ms_handle_connect 0x7f3f5410ea80 2024-03-30T15:14:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter ms_handle_connect 0x7f3f5410ae80 2024-03-30T15:14:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4188700292 0 0) 0x7f3f541ab850 con 0x7f3f5410ea80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f400039c0 con 0x7f3f5410ea80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1927884940 0 0) 0x7f3f541a93f0 con 0x7f3f54108110 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f541ab850 con 0x7f3f54108110 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137526575 0 0) 0x7f3f541aa610 con 0x7f3f5410ae80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.739+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f541a93f0 con 0x7f3f5410ae80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 393684438 0 0) 0x7f3f541ab850 con 0x7f3f54108110 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f541aa610 con 0x7f3f54108110 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f44001a30 con 0x7f3f54108110 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3077913563 0 0) 0x7f3f400039c0 con 0x7f3f5410ea80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f541ab850 con 0x7f3f5410ea80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3536158836 0 0) 0x7f3f541a93f0 con 0x7f3f5410ae80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f400039c0 con 0x7f3f5410ae80 2024-03-30T15:14:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f4c0047f0 con 0x7f3f5410ea80 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f3c0017f0 con 0x7f3f5410ae80 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 857950112 0 0) 0x7f3f541aa610 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 >> v1:172.21.15.28:6790/0 conn(0x7f3f5410ea80 legacy=0x7f3f541a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 >> v1:172.21.15.134:6789/0 conn(0x7f3f5410ae80 legacy=0x7f3f541a4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f541aca90 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f541aba80 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f58e65700 10 client.25995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f541abf40 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f58e65700 1 librados: init done 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f3f44001ce0 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.740+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3f44005cd0 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.741+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5406a1f0 con 0x7f3f54108110 2024-03-30T15:14:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f3f44003930 con 0x7f3f54108110 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(909..909 src has 654..909) ==== 7561+0+0 (unknown 3432648697 0 0) 0x7f3f44089550 con 0x7f3f54108110 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 10 client.25995.objecter ms_dispatch 0x7f3f54001550 osd_map(909..909 src has 654..909) 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 3 client.25995.objecter handle_osd_map got epochs [909,909] > 0 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 3 client.25995.objecter handle_osd_map decoding full epoch 909 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 20 client.25995.objecter dump_active .. 0 homeless 2024-03-30T15:14:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.742+0000 7f3f4bfff700 10 client.25995.objecter ms_handle_connect 0x7f3f40070450 2024-03-30T15:14:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.747+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3f440569a0 con 0x7f3f54108110 2024-03-30T15:14:48.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:48 smithi028 ceph-mon[29876]: pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-30T15:14:48.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:48 smithi028 ceph-mon[41614]: pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-30T15:14:48.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:48 smithi134 ceph-mon[33714]: pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 44 op/s 2024-03-30T15:14:48.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:48.969+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f3f54102f20 con 0x7f3f54108110 2024-03-30T15:14:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.608+0000 7f3f4bfff700 1 -- 172.21.15.28:0/2151169666 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v910) ==== 91+0+0 (unknown 2447864544 0 0) 0x7f3f4405a3f0 con 0x7f3f54108110 2024-03-30T15:14:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.612+0000 7f3f58e65700 10 librados: watch_flush enter 2024-03-30T15:14:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.612+0000 7f3f58e65700 10 librados: watch_flush exit 2024-03-30T15:14:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.612+0000 7f3f58e65700 20 client.25995.objecter shutdown clearing up homeless session... 2024-03-30T15:14:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.612+0000 7f3f58e65700 10 client.25995.objecter successfully canceled tick 2024-03-30T15:14:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.613+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 >> v1:172.21.15.28:6800/3011431093 conn(0x7f3f40070450 legacy=0x7f3f40072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.613+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 >> v1:172.21.15.28:6789/0 conn(0x7f3f54108110 legacy=0x7f3f54107500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.614+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 shutdown_connections 2024-03-30T15:14:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.614+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 >> 172.21.15.28:0/2151169666 conn(0x7f3f540761d0 msgr2=0x7f3f54109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.614+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 shutdown_connections 2024-03-30T15:14:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.614+0000 7f3f58e65700 1 -- 172.21.15.28:0/2151169666 wait complete. 2024-03-30T15:14:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:49.614+0000 7f3f58e65700 1 librados: shutdown 2024-03-30T15:14:49.617 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-03-30T15:14:49.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:49.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-30T15:14:50.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-03-30T15:14:50.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[41614]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-30T15:14:50.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:50.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[41614]: osdmap e910: 8 total, 8 up, 8 in 2024-03-30T15:14:50.637 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[41614]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-03-30T15:14:50.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[29876]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-30T15:14:50.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:50.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[29876]: osdmap e910: 8 total, 8 up, 8 in 2024-03-30T15:14:50.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:50 smithi028 ceph-mon[29876]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-03-30T15:14:50.904 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:14:50.905 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:14:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:50 smithi134 ceph-mon[33714]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-30T15:14:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:50 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2151169666' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-30T15:14:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:50 smithi134 ceph-mon[33714]: osdmap e910: 8 total, 8 up, 8 in 2024-03-30T15:14:50.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:50 smithi134 ceph-mon[33714]: pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-03-30T15:14:51.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:14:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:14:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:51 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:14:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:14:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:51 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:14:51.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:51 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:14:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:52.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-03-30T15:14:52.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: 3.1d deep-scrub starts 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.625+0000 7fb6b7271700 1 -- 172.21.15.28:0/1337091217 >> v1:172.21.15.28:6790/0 conn(0x7fb6b010eb10 legacy=0x7fb6b0110f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.625+0000 7fb6b7271700 1 -- 172.21.15.28:0/1337091217 shutdown_connections 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.625+0000 7fb6b7271700 1 -- 172.21.15.28:0/1337091217 >> 172.21.15.28:0/1337091217 conn(0x7fb6b00fc9e0 msgr2=0x7fb6b00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.625+0000 7fb6b7271700 1 -- 172.21.15.28:0/1337091217 shutdown_connections 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.625+0000 7fb6b7271700 1 -- 172.21.15.28:0/1337091217 wait complete. 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 librados: starting msgr at 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 librados: starting objecter 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 Processor -- start 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 -- start start 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 librados: setting wanted keys 2024-03-30T15:14:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 librados: calling monclient init 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6b0102950 con 0x7fb6b010eb10 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6b0102b60 con 0x7fb6b01031f0 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b7271700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6b0102d70 con 0x7fb6b010b000 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b480c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fb6b010b000 0x7fb6b0101560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47258/0 (socket says 172.21.15.28:47258) 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6b480c700 1 -- 172.21.15.28:0/3267381128 learned_addr learned my addr 172.21.15.28:0/3267381128 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b010b000 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b010eb10 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b01031f0 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.626+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765125167 0 0) 0x7fb6b0102d70 con 0x7fb6b010b000 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb69c0031f0 con 0x7fb6b010b000 2024-03-30T15:14:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686364746 0 0) 0x7fb6b0102950 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b0102d70 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 794097120 0 0) 0x7fb6b0102b60 con 0x7fb6b01031f0 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b0102950 con 0x7fb6b01031f0 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3526243204 0 0) 0x7fb6b0102d70 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6b0102b60 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 257202314 0 0) 0x7fb69c0031f0 con 0x7fb6b010b000 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6b0102d70 con 0x7fb6b010b000 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6a80016f0 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6a0001a10 con 0x7fb6b010b000 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4017960747 0 0) 0x7fb6b0102950 con 0x7fb6b01031f0 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb69c0031f0 con 0x7fb6b01031f0 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6ac001870 con 0x7fb6b01031f0 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4187111577 0 0) 0x7fb6b0102b60 con 0x7fb6b010eb10 2024-03-30T15:14:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 >> v1:172.21.15.28:6790/0 conn(0x7fb6b010b000 legacy=0x7fb6b0101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 >> v1:172.21.15.134:6789/0 conn(0x7fb6b01031f0 legacy=0x7fb6b0100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b01acaa0 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b01a9a90 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6b7271700 10 client.26001.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6b01a9f50 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6b7271700 1 librados: init done 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb6a8001960 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.627+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6a8004fb0 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.628+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb694005120 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.629+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb6a801b650 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(911..911 src has 654..911) ==== 7561+0+0 (unknown 3537968931 0 0) 0x7fb6a80883d0 con 0x7fb6b010eb10 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 10 client.26001.objecter ms_dispatch 0x7fb6b0001550 osd_map(911..911 src has 654..911) 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 3 client.26001.objecter handle_osd_map got epochs [911,911] > 0 2024-03-30T15:14:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 3 client.26001.objecter handle_osd_map decoding full epoch 911 2024-03-30T15:14:52.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 20 client.26001.objecter dump_active .. 0 homeless 2024-03-30T15:14:52.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.630+0000 7fb6a67fc700 10 client.26001.objecter ms_handle_connect 0x7fb69c070320 2024-03-30T15:14:52.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.634+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb6a8055960 con 0x7fb6b010eb10 2024-03-30T15:14:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.841+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb694000b90 con 0x7fb69c070320 2024-03-30T15:14:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.855+0000 7fb6a67fc700 1 -- 172.21.15.28:0/3267381128 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+452469 (unknown 2965378022 0 1393456055) 0x7fb694000b90 con 0x7fb69c070320 2024-03-30T15:14:52.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:14:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.858+0000 7fb6b7271700 10 librados: watch_flush enter 2024-03-30T15:14:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.858+0000 7fb6b7271700 10 librados: watch_flush exit 2024-03-30T15:14:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.858+0000 7fb6b7271700 20 client.26001.objecter shutdown clearing up homeless session... 2024-03-30T15:14:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.858+0000 7fb6b7271700 10 client.26001.objecter successfully canceled tick 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb69c070320 legacy=0x7fb69c0727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 >> v1:172.21.15.28:6789/0 conn(0x7fb6b010eb10 legacy=0x7fb6b01a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 shutdown_connections 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 >> 172.21.15.28:0/3267381128 conn(0x7fb6b00fc9e0 msgr2=0x7fb6b00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 shutdown_connections 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 -- 172.21.15.28:0/3267381128 wait complete. 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:52.859+0000 7fb6b7271700 1 librados: shutdown 2024-03-30T15:14:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: 3.1d deep-scrub starts 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: 3.1d deep-scrub ok 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: 5.1a deep-scrub starts 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: 5.1a deep-scrub ok 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[41614]: osdmap e911: 8 total, 8 up, 8 in 2024-03-30T15:14:52.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: 3.1d deep-scrub ok 2024-03-30T15:14:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: 5.1a deep-scrub starts 2024-03-30T15:14:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: 5.1a deep-scrub ok 2024-03-30T15:14:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-30T15:14:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:52 smithi028 ceph-mon[29876]: osdmap e911: 8 total, 8 up, 8 in 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: 3.1d deep-scrub starts 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: 3.1d deep-scrub ok 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: 5.1a deep-scrub starts 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: 5.1a deep-scrub ok 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: pgmap v1414: 174 pgs: 174 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-30T15:14:52.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:52 smithi134 ceph-mon[33714]: osdmap e911: 8 total, 8 up, 8 in 2024-03-30T15:14:53.606 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 3.6 scrub starts 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 3.6 scrub ok 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T15:14:54.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: from='client.26001 v1:172.21.15.28:0/3267381128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 5.12 deep-scrub starts 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: 5.12 deep-scrub ok 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 3.6 scrub starts 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 3.6 scrub ok 2024-03-30T15:14:54.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: from='client.26001 v1:172.21.15.28:0/3267381128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 5.12 deep-scrub starts 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: 5.12 deep-scrub ok 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:14:54.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:53 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:14:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 3.6 scrub starts 2024-03-30T15:14:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 3.6 scrub ok 2024-03-30T15:14:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: from='client.26001 v1:172.21.15.28:0/3267381128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 5.12 deep-scrub starts 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: 5.12 deep-scrub ok 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:14:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:53 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:14:54.585 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1415,"stamp":"2024-03-30T15:14:52.598688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2767,"num_read_kb":2519,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2596,"ondisk_log_size":2596,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":668360,"kb_used_data":60924,"kb_used_omap":108,"kb_used_meta":606291,"kb_avail":749292856,"statfs":{"total":767960285184,"available":767275884544,"internally_reserved":0,"allocated":62386176,"data_stored":32224070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111128,"internal_metadata":620842472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":434,"num_read_kb":434,"num_write":286,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.383865"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":2036,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.951887+0000","last_change":"2024-03-30T15:13:55.602346+0000","last_active":"2024-03-30T15:14:43.951887+0000","last_peered":"2024-03-30T15:14:43.951887+0000","last_clean":"2024-03-30T15:14:43.951887+0000","last_became_active":"2024-03-30T15:13:55.601989+0000","last_became_peered":"2024-03-30T15:13:55.601989+0000","last_unstale":"2024-03-30T15:14:43.951887+0000","last_undegraded":"2024-03-30T15:14:43.951887+0000","last_fullsized":"2024-03-30T15:14:43.951887+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:18.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:18.837242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967812+0000","last_change":"2024-03-30T15:14:00.634325+0000","last_active":"2024-03-30T15:14:42.967812+0000","last_peered":"2024-03-30T15:14:42.967812+0000","last_clean":"2024-03-30T15:14:42.967812+0000","last_became_active":"2024-03-30T15:14:00.633295+0000","last_became_peered":"2024-03-30T15:14:00.633295+0000","last_unstale":"2024-03-30T15:14:42.967812+0000","last_undegraded":"2024-03-30T15:14:42.967812+0000","last_fullsized":"2024-03-30T15:14:42.967812+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:19.524864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:19.524864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"909'30","reported_seq":1694,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841353+0000","last_change":"2024-03-30T15:14:20.756235+0000","last_active":"2024-03-30T15:14:49.841353+0000","last_peered":"2024-03-30T15:14:49.841353+0000","last_clean":"2024-03-30T15:14:49.841353+0000","last_became_active":"2024-03-30T15:13:03.535865+0000","last_became_peered":"2024-03-30T15:13:03.535865+0000","last_unstale":"2024-03-30T15:14:49.841353+0000","last_undegraded":"2024-03-30T15:14:49.841353+0000","last_fullsized":"2024-03-30T15:14:49.841353+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'20","last_scrub_stamp":"2024-03-30T15:14:20.756053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.756053+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:22.102152+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1266,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614978+0000","last_change":"2024-03-30T15:14:28.185811+0000","last_active":"2024-03-30T15:14:49.614978+0000","last_peered":"2024-03-30T15:14:49.614978+0000","last_clean":"2024-03-30T15:14:49.614978+0000","last_became_active":"2024-03-30T15:13:28.625255+0000","last_became_peered":"2024-03-30T15:13:28.625255+0000","last_unstale":"2024-03-30T15:14:49.614978+0000","last_undegraded":"2024-03-30T15:14:49.614978+0000","last_fullsized":"2024-03-30T15:14:49.614978+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:28.185698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:14:28.185698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"833'11","reported_seq":1002,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968866+0000","last_change":"2024-03-30T15:13:46.704186+0000","last_active":"2024-03-30T15:14:42.968866+0000","last_peered":"2024-03-30T15:14:42.968866+0000","last_clean":"2024-03-30T15:14:42.968866+0000","last_became_active":"2024-03-30T15:13:03.543934+0000","last_became_peered":"2024-03-30T15:13:03.543934+0000","last_unstale":"2024-03-30T15:14:42.968866+0000","last_undegraded":"2024-03-30T15:14:42.968866+0000","last_fullsized":"2024-03-30T15:14:42.968866+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'11","last_scrub_stamp":"2024-03-30T15:13:46.704092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.704092+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.405370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1691,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841841+0000","last_change":"2024-03-30T15:13:52.597484+0000","last_active":"2024-03-30T15:14:49.841841+0000","last_peered":"2024-03-30T15:14:49.841841+0000","last_clean":"2024-03-30T15:14:49.841841+0000","last_became_active":"2024-03-30T15:13:52.596681+0000","last_became_peered":"2024-03-30T15:13:52.596681+0000","last_unstale":"2024-03-30T15:14:49.841841+0000","last_undegraded":"2024-03-30T15:14:49.841841+0000","last_fullsized":"2024-03-30T15:14:49.841841+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:49.842503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:49.842503+0000","last_clean_scrub_stamp":"2024-03-30T15:11:49.842503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":1024,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595136+0000","last_change":"2024-03-30T15:14:12.850623+0000","last_active":"2024-03-30T15:14:42.595136+0000","last_peered":"2024-03-30T15:14:42.595136+0000","last_clean":"2024-03-30T15:14:42.595136+0000","last_became_active":"2024-03-30T15:13:19.287238+0000","last_became_peered":"2024-03-30T15:13:19.287238+0000","last_unstale":"2024-03-30T15:14:42.595136+0000","last_undegraded":"2024-03-30T15:14:42.595136+0000","last_fullsized":"2024-03-30T15:14:42.595136+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:12.850408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.850408+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:28.008340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1650,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968959+0000","last_change":"2024-03-30T15:14:15.617984+0000","last_active":"2024-03-30T15:14:42.968959+0000","last_peered":"2024-03-30T15:14:42.968959+0000","last_clean":"2024-03-30T15:14:42.968959+0000","last_became_active":"2024-03-30T15:13:19.284720+0000","last_became_peered":"2024-03-30T15:13:19.284720+0000","last_unstale":"2024-03-30T15:14:42.968959+0000","last_undegraded":"2024-03-30T15:14:42.968959+0000","last_fullsized":"2024-03-30T15:14:42.968959+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.617863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.617863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:26.436478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"909'45","reported_seq":107,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840437+0000","last_change":"2024-03-30T15:14:09.740147+0000","last_active":"2024-03-30T15:14:49.840437+0000","last_peered":"2024-03-30T15:14:49.840437+0000","last_clean":"2024-03-30T15:14:49.840437+0000","last_became_active":"2024-03-30T15:14:09.722049+0000","last_became_peered":"2024-03-30T15:14:09.722049+0000","last_unstale":"2024-03-30T15:14:49.840437+0000","last_undegraded":"2024-03-30T15:14:49.840437+0000","last_fullsized":"2024-03-30T15:14:49.840437+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'31","last_scrub_stamp":"2024-03-30T15:13:06.419015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.419015+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820358+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1752,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.967588+0000","last_change":"2024-03-30T15:14:19.617578+0000","last_active":"2024-03-30T15:14:42.967588+0000","last_peered":"2024-03-30T15:14:42.967588+0000","last_clean":"2024-03-30T15:14:42.967588+0000","last_became_active":"2024-03-30T15:13:03.529657+0000","last_became_peered":"2024-03-30T15:13:03.529657+0000","last_unstale":"2024-03-30T15:14:42.967588+0000","last_undegraded":"2024-03-30T15:14:42.967588+0000","last_fullsized":"2024-03-30T15:14:42.967588+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:19.617498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.617498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:23.958547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1752,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596813+0000","last_change":"2024-03-30T15:13:30.478717+0000","last_active":"2024-03-30T15:14:42.596813+0000","last_peered":"2024-03-30T15:14:42.596813+0000","last_clean":"2024-03-30T15:14:42.596813+0000","last_became_active":"2024-03-30T15:13:28.634488+0000","last_became_peered":"2024-03-30T15:13:28.634488+0000","last_unstale":"2024-03-30T15:14:42.596813+0000","last_undegraded":"2024-03-30T15:14:42.596813+0000","last_fullsized":"2024-03-30T15:14:42.596813+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:30.478588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.478588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.374563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.354458+0000","last_change":"2024-03-30T15:14:33.036325+0000","last_active":"2024-03-30T15:14:50.354458+0000","last_peered":"2024-03-30T15:14:50.354458+0000","last_clean":"2024-03-30T15:14:50.354458+0000","last_became_active":"2024-03-30T15:13:52.592738+0000","last_became_peered":"2024-03-30T15:13:52.592738+0000","last_unstale":"2024-03-30T15:14:50.354458+0000","last_undegraded":"2024-03-30T15:14:50.354458+0000","last_fullsized":"2024-03-30T15:14:50.354458+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:33.036198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.036198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:36.457410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595673+0000","last_change":"2024-03-30T15:13:31.436323+0000","last_active":"2024-03-30T15:14:42.595673+0000","last_peered":"2024-03-30T15:14:42.595673+0000","last_clean":"2024-03-30T15:14:42.595673+0000","last_became_active":"2024-03-30T15:13:28.634783+0000","last_became_peered":"2024-03-30T15:13:28.634783+0000","last_unstale":"2024-03-30T15:14:42.595673+0000","last_undegraded":"2024-03-30T15:14:42.595673+0000","last_fullsized":"2024-03-30T15:14:42.595673+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.436194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.436194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"909'32","reported_seq":1734,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841972+0000","last_change":"2024-03-30T15:14:14.688704+0000","last_active":"2024-03-30T15:14:49.841972+0000","last_peered":"2024-03-30T15:14:49.841972+0000","last_clean":"2024-03-30T15:14:49.841972+0000","last_became_active":"2024-03-30T15:13:03.539403+0000","last_became_peered":"2024-03-30T15:13:03.539403+0000","last_unstale":"2024-03-30T15:14:49.841972+0000","last_undegraded":"2024-03-30T15:14:49.841972+0000","last_fullsized":"2024-03-30T15:14:49.841972+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:14.688571+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:09.473429+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.688571+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:20.327879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1654,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841157+0000","last_change":"2024-03-30T15:13:41.719907+0000","last_active":"2024-03-30T15:14:49.841157+0000","last_peered":"2024-03-30T15:14:49.841157+0000","last_clean":"2024-03-30T15:14:49.841157+0000","last_became_active":"2024-03-30T15:13:19.284770+0000","last_became_peered":"2024-03-30T15:13:19.284770+0000","last_unstale":"2024-03-30T15:14:49.841157+0000","last_undegraded":"2024-03-30T15:14:49.841157+0000","last_fullsized":"2024-03-30T15:14:49.841157+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:41.719797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:13:41.719797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.319763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1737,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596893+0000","last_change":"2024-03-30T15:14:17.900734+0000","last_active":"2024-03-30T15:14:42.596893+0000","last_peered":"2024-03-30T15:14:42.596893+0000","last_clean":"2024-03-30T15:14:42.596893+0000","last_became_active":"2024-03-30T15:13:03.533222+0000","last_became_peered":"2024-03-30T15:13:03.533222+0000","last_unstale":"2024-03-30T15:14:42.596893+0000","last_undegraded":"2024-03-30T15:14:42.596893+0000","last_fullsized":"2024-03-30T15:14:42.596893+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.900644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.900644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:42.201720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1644,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968763+0000","last_change":"2024-03-30T15:14:32.445558+0000","last_active":"2024-03-30T15:14:42.968763+0000","last_peered":"2024-03-30T15:14:42.968763+0000","last_clean":"2024-03-30T15:14:42.968763+0000","last_became_active":"2024-03-30T14:57:56.262960+0000","last_became_peered":"2024-03-30T14:57:56.262960+0000","last_unstale":"2024-03-30T15:14:42.968763+0000","last_undegraded":"2024-03-30T15:14:42.968763+0000","last_fullsized":"2024-03-30T15:14:42.968763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:32.445474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:32.445474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:41.549467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":1024,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595337+0000","last_change":"2024-03-30T15:13:46.902799+0000","last_active":"2024-03-30T15:14:42.595337+0000","last_peered":"2024-03-30T15:14:42.595337+0000","last_clean":"2024-03-30T15:14:42.595337+0000","last_became_active":"2024-03-30T15:13:19.288713+0000","last_became_peered":"2024-03-30T15:13:19.288713+0000","last_unstale":"2024-03-30T15:14:42.595337+0000","last_undegraded":"2024-03-30T15:14:42.595337+0000","last_fullsized":"2024-03-30T15:14:42.595337+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:46.902697+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.902697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:10.399599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596800+0000","last_change":"2024-03-30T15:13:32.456654+0000","last_active":"2024-03-30T15:14:42.596800+0000","last_peered":"2024-03-30T15:14:42.596800+0000","last_clean":"2024-03-30T15:14:42.596800+0000","last_became_active":"2024-03-30T15:13:19.288815+0000","last_became_peered":"2024-03-30T15:13:19.288815+0000","last_unstale":"2024-03-30T15:14:42.596800+0000","last_undegraded":"2024-03-30T15:14:42.596800+0000","last_fullsized":"2024-03-30T15:14:42.596800+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:32.456571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:13:32.456571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"909'46","reported_seq":1819,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.318677+0000","last_change":"2024-03-30T15:13:07.392582+0000","last_active":"2024-03-30T15:14:45.318677+0000","last_peered":"2024-03-30T15:14:45.318677+0000","last_clean":"2024-03-30T15:14:45.318677+0000","last_became_active":"2024-03-30T15:13:03.503223+0000","last_became_peered":"2024-03-30T15:13:03.503223+0000","last_unstale":"2024-03-30T15:14:45.318677+0000","last_undegraded":"2024-03-30T15:14:45.318677+0000","last_fullsized":"2024-03-30T15:14:45.318677+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_clean_scrub_stamp":"2024-03-30T15:13:07.392488+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1669,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595871+0000","last_change":"2024-03-30T15:14:00.624520+0000","last_active":"2024-03-30T15:14:42.595871+0000","last_peered":"2024-03-30T15:14:42.595871+0000","last_clean":"2024-03-30T15:14:42.595871+0000","last_became_active":"2024-03-30T15:14:00.624194+0000","last_became_peered":"2024-03-30T15:14:00.624194+0000","last_unstale":"2024-03-30T15:14:42.595871+0000","last_undegraded":"2024-03-30T15:14:42.595871+0000","last_fullsized":"2024-03-30T15:14:42.595871+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:40.486895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:30.744962+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.486895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:07.691772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"883'17","reported_seq":80,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840473+0000","last_change":"2024-03-30T15:14:09.733902+0000","last_active":"2024-03-30T15:14:49.840473+0000","last_peered":"2024-03-30T15:14:49.840473+0000","last_clean":"2024-03-30T15:14:49.840473+0000","last_became_active":"2024-03-30T15:14:09.730624+0000","last_became_peered":"2024-03-30T15:14:09.730624+0000","last_unstale":"2024-03-30T15:14:49.840473+0000","last_undegraded":"2024-03-30T15:14:49.840473+0000","last_fullsized":"2024-03-30T15:14:49.840473+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'11","last_scrub_stamp":"2024-03-30T15:11:39.928695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:39.928695+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1717,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595892+0000","last_change":"2024-03-30T15:13:33.913218+0000","last_active":"2024-03-30T15:14:42.595892+0000","last_peered":"2024-03-30T15:14:42.595892+0000","last_clean":"2024-03-30T15:14:42.595892+0000","last_became_active":"2024-03-30T15:13:28.623165+0000","last_became_peered":"2024-03-30T15:13:28.623165+0000","last_unstale":"2024-03-30T15:14:42.595892+0000","last_undegraded":"2024-03-30T15:14:42.595892+0000","last_fullsized":"2024-03-30T15:14:42.595892+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:33.913138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:13:33.913138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.588570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":997,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841540+0000","last_change":"2024-03-30T15:13:55.607087+0000","last_active":"2024-03-30T15:14:49.841540+0000","last_peered":"2024-03-30T15:14:49.841540+0000","last_clean":"2024-03-30T15:14:49.841540+0000","last_became_active":"2024-03-30T15:13:55.606453+0000","last_became_peered":"2024-03-30T15:13:55.606453+0000","last_unstale":"2024-03-30T15:14:49.841540+0000","last_undegraded":"2024-03-30T15:14:49.841540+0000","last_fullsized":"2024-03-30T15:14:49.841540+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:11:48.884603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:11:48.884603+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1481,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353695+0000","last_change":"2024-03-30T15:13:46.195107+0000","last_active":"2024-03-30T15:14:50.353695+0000","last_peered":"2024-03-30T15:14:50.353695+0000","last_clean":"2024-03-30T15:14:50.353695+0000","last_became_active":"2024-03-30T15:13:28.646505+0000","last_became_peered":"2024-03-30T15:13:28.646505+0000","last_unstale":"2024-03-30T15:14:50.353695+0000","last_undegraded":"2024-03-30T15:14:50.353695+0000","last_fullsized":"2024-03-30T15:14:50.353695+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:46.195015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.195015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.641593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1645,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840843+0000","last_change":"2024-03-30T15:14:22.781690+0000","last_active":"2024-03-30T15:14:49.840843+0000","last_peered":"2024-03-30T15:14:49.840843+0000","last_clean":"2024-03-30T15:14:49.840843+0000","last_became_active":"2024-03-30T15:13:03.538990+0000","last_became_peered":"2024-03-30T15:13:03.538990+0000","last_unstale":"2024-03-30T15:14:49.840843+0000","last_undegraded":"2024-03-30T15:14:49.840843+0000","last_fullsized":"2024-03-30T15:14:49.840843+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.781602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.781602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.412116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"909'50","reported_seq":1811,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.269127+0000","last_change":"2024-03-30T15:14:00.651598+0000","last_active":"2024-03-30T15:14:45.269127+0000","last_peered":"2024-03-30T15:14:45.269127+0000","last_clean":"2024-03-30T15:14:45.269127+0000","last_became_active":"2024-03-30T15:14:00.630774+0000","last_became_peered":"2024-03-30T15:14:00.630774+0000","last_unstale":"2024-03-30T15:14:45.269127+0000","last_undegraded":"2024-03-30T15:14:45.269127+0000","last_fullsized":"2024-03-30T15:14:45.269127+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'38","last_scrub_stamp":"2024-03-30T15:12:17.847789+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:12:17.847789+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.127341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":1712,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953264+0000","last_change":"2024-03-30T15:13:40.071287+0000","last_active":"2024-03-30T15:14:43.953264+0000","last_peered":"2024-03-30T15:14:43.953264+0000","last_clean":"2024-03-30T15:14:43.953264+0000","last_became_active":"2024-03-30T15:13:28.632229+0000","last_became_peered":"2024-03-30T15:13:28.632229+0000","last_unstale":"2024-03-30T15:14:43.953264+0000","last_undegraded":"2024-03-30T15:14:43.953264+0000","last_fullsized":"2024-03-30T15:14:43.953264+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:13:40.071196+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.071196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:07.554377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":68,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840538+0000","last_change":"2024-03-30T15:14:09.723294+0000","last_active":"2024-03-30T15:14:49.840538+0000","last_peered":"2024-03-30T15:14:49.840538+0000","last_clean":"2024-03-30T15:14:49.840538+0000","last_became_active":"2024-03-30T15:14:09.722923+0000","last_became_peered":"2024-03-30T15:14:09.722923+0000","last_unstale":"2024-03-30T15:14:49.840538+0000","last_undegraded":"2024-03-30T15:14:49.840538+0000","last_fullsized":"2024-03-30T15:14:49.840538+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_clean_scrub_stamp":"2024-03-30T15:11:48.594615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"86'128","reported_seq":1023,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595651+0000","last_change":"2024-03-30T15:14:33.930988+0000","last_active":"2024-03-30T15:14:42.595651+0000","last_peered":"2024-03-30T15:14:42.595651+0000","last_clean":"2024-03-30T15:14:42.595651+0000","last_became_active":"2024-03-30T15:06:48.343507+0000","last_became_peered":"2024-03-30T15:06:48.343507+0000","last_unstale":"2024-03-30T15:14:42.595651+0000","last_undegraded":"2024-03-30T15:14:42.595651+0000","last_fullsized":"2024-03-30T15:14:42.595651+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:33.930858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.930858+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:55.942461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1487,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353939+0000","last_change":"2024-03-30T15:14:23.185170+0000","last_active":"2024-03-30T15:14:50.353939+0000","last_peered":"2024-03-30T15:14:50.353939+0000","last_clean":"2024-03-30T15:14:50.353939+0000","last_became_active":"2024-03-30T15:13:19.290216+0000","last_became_peered":"2024-03-30T15:13:19.290216+0000","last_unstale":"2024-03-30T15:14:50.353939+0000","last_undegraded":"2024-03-30T15:14:50.353939+0000","last_fullsized":"2024-03-30T15:14:50.353939+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.185098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.185098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:23.258738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"899'12","reported_seq":1888,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.726614+0000","last_change":"2024-03-30T15:14:28.186777+0000","last_active":"2024-03-30T15:14:42.726614+0000","last_peered":"2024-03-30T15:14:42.726614+0000","last_clean":"2024-03-30T15:14:42.726614+0000","last_became_active":"2024-03-30T15:13:03.504632+0000","last_became_peered":"2024-03-30T15:13:03.504632+0000","last_unstale":"2024-03-30T15:14:42.726614+0000","last_undegraded":"2024-03-30T15:14:42.726614+0000","last_fullsized":"2024-03-30T15:14:42.726614+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'12","last_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_deep_scrub":"899'12","last_deep_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_clean_scrub_stamp":"2024-03-30T15:14:28.186699+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:54.695167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"909'49","reported_seq":1761,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840917+0000","last_change":"2024-03-30T15:14:35.744220+0000","last_active":"2024-03-30T15:14:49.840917+0000","last_peered":"2024-03-30T15:14:49.840917+0000","last_clean":"2024-03-30T15:14:49.840917+0000","last_became_active":"2024-03-30T15:13:03.533274+0000","last_became_peered":"2024-03-30T15:13:03.533274+0000","last_unstale":"2024-03-30T15:14:49.840917+0000","last_undegraded":"2024-03-30T15:14:49.840917+0000","last_fullsized":"2024-03-30T15:14:49.840917+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'35","last_scrub_stamp":"2024-03-30T15:14:35.744005+0000","last_deep_scrub":"904'35","last_deep_scrub_stamp":"2024-03-30T15:14:35.744005+0000","last_clean_scrub_stamp":"2024-03-30T15:14:35.744005+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.293034+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1310,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615123+0000","last_change":"2024-03-30T15:13:28.647418+0000","last_active":"2024-03-30T15:14:49.615123+0000","last_peered":"2024-03-30T15:14:49.615123+0000","last_clean":"2024-03-30T15:14:49.615123+0000","last_became_active":"2024-03-30T15:13:28.645096+0000","last_became_peered":"2024-03-30T15:13:28.645096+0000","last_unstale":"2024-03-30T15:14:49.615123+0000","last_undegraded":"2024-03-30T15:14:49.615123+0000","last_fullsized":"2024-03-30T15:14:49.615123+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:07:24.562661+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:07:24.562661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1734,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952731+0000","last_change":"2024-03-30T15:13:42.150953+0000","last_active":"2024-03-30T15:14:43.952731+0000","last_peered":"2024-03-30T15:14:43.952731+0000","last_clean":"2024-03-30T15:14:43.952731+0000","last_became_active":"2024-03-30T15:13:19.274607+0000","last_became_peered":"2024-03-30T15:13:19.274607+0000","last_unstale":"2024-03-30T15:14:43.952731+0000","last_undegraded":"2024-03-30T15:14:43.952731+0000","last_fullsized":"2024-03-30T15:14:43.952731+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.150834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.914073+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1038,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968958+0000","last_change":"2024-03-30T15:14:03.599433+0000","last_active":"2024-03-30T15:14:42.968958+0000","last_peered":"2024-03-30T15:14:42.968958+0000","last_clean":"2024-03-30T15:14:42.968958+0000","last_became_active":"2024-03-30T15:14:03.598713+0000","last_became_peered":"2024-03-30T15:14:03.598713+0000","last_unstale":"2024-03-30T15:14:42.968958+0000","last_undegraded":"2024-03-30T15:14:42.968958+0000","last_fullsized":"2024-03-30T15:14:42.968958+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:18.560194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:18.560194+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1488,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.354038+0000","last_change":"2024-03-30T15:13:37.186884+0000","last_active":"2024-03-30T15:14:50.354038+0000","last_peered":"2024-03-30T15:14:50.354038+0000","last_clean":"2024-03-30T15:14:50.354038+0000","last_became_active":"2024-03-30T15:13:28.643398+0000","last_became_peered":"2024-03-30T15:13:28.643398+0000","last_unstale":"2024-03-30T15:14:50.354038+0000","last_undegraded":"2024-03-30T15:14:50.354038+0000","last_fullsized":"2024-03-30T15:14:50.354038+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:37.186764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.186764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.763227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"851'10","reported_seq":1555,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615192+0000","last_change":"2024-03-30T15:14:19.204349+0000","last_active":"2024-03-30T15:14:49.615192+0000","last_peered":"2024-03-30T15:14:49.615192+0000","last_clean":"2024-03-30T15:14:49.615192+0000","last_became_active":"2024-03-30T15:13:28.632781+0000","last_became_peered":"2024-03-30T15:13:28.632781+0000","last_unstale":"2024-03-30T15:14:49.615192+0000","last_undegraded":"2024-03-30T15:14:49.615192+0000","last_fullsized":"2024-03-30T15:14:49.615192+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'10","last_scrub_stamp":"2024-03-30T15:14:19.204263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.204263+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.838172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"909'35","reported_seq":1810,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.288047+0000","last_change":"2024-03-30T15:14:22.141020+0000","last_active":"2024-03-30T15:14:45.288047+0000","last_peered":"2024-03-30T15:14:45.288047+0000","last_clean":"2024-03-30T15:14:45.288047+0000","last_became_active":"2024-03-30T15:13:03.519156+0000","last_became_peered":"2024-03-30T15:13:03.519156+0000","last_unstale":"2024-03-30T15:14:45.288047+0000","last_undegraded":"2024-03-30T15:14:45.288047+0000","last_fullsized":"2024-03-30T15:14:45.288047+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:14:22.140887+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.283859+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.140887+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.046893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1746,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596175+0000","last_change":"2024-03-30T15:14:16.872980+0000","last_active":"2024-03-30T15:14:42.596175+0000","last_peered":"2024-03-30T15:14:42.596175+0000","last_clean":"2024-03-30T15:14:42.596175+0000","last_became_active":"2024-03-30T15:13:19.286426+0000","last_became_peered":"2024-03-30T15:13:19.286426+0000","last_unstale":"2024-03-30T15:14:42.596175+0000","last_undegraded":"2024-03-30T15:14:42.596175+0000","last_fullsized":"2024-03-30T15:14:42.596175+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.872862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.872862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:43.944961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1354,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613777+0000","last_change":"2024-03-30T15:14:22.150928+0000","last_active":"2024-03-30T15:14:49.613777+0000","last_peered":"2024-03-30T15:14:49.613777+0000","last_clean":"2024-03-30T15:14:49.613777+0000","last_became_active":"2024-03-30T15:13:52.589257+0000","last_became_peered":"2024-03-30T15:13:52.589257+0000","last_unstale":"2024-03-30T15:14:49.613777+0000","last_undegraded":"2024-03-30T15:14:49.613777+0000","last_fullsized":"2024-03-30T15:14:49.613777+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.150808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.150808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:29.638924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1046,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968888+0000","last_change":"2024-03-30T15:14:03.596583+0000","last_active":"2024-03-30T15:14:42.968888+0000","last_peered":"2024-03-30T15:14:42.968888+0000","last_clean":"2024-03-30T15:14:42.968888+0000","last_became_active":"2024-03-30T15:14:03.596165+0000","last_became_peered":"2024-03-30T15:14:03.596165+0000","last_unstale":"2024-03-30T15:14:42.968888+0000","last_undegraded":"2024-03-30T15:14:42.968888+0000","last_fullsized":"2024-03-30T15:14:42.968888+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:12:13.604719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:12:13.604719+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1485,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353722+0000","last_change":"2024-03-30T15:14:20.304484+0000","last_active":"2024-03-30T15:14:50.353722+0000","last_peered":"2024-03-30T15:14:50.353722+0000","last_clean":"2024-03-30T15:14:50.353722+0000","last_became_active":"2024-03-30T15:13:19.281718+0000","last_became_peered":"2024-03-30T15:13:19.281718+0000","last_unstale":"2024-03-30T15:14:50.353722+0000","last_undegraded":"2024-03-30T15:14:50.353722+0000","last_fullsized":"2024-03-30T15:14:50.353722+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.304352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.304352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.098567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968444+0000","last_change":"2024-03-30T15:13:30.648769+0000","last_active":"2024-03-30T15:14:42.968444+0000","last_peered":"2024-03-30T15:14:42.968444+0000","last_clean":"2024-03-30T15:14:42.968444+0000","last_became_active":"2024-03-30T15:13:28.622538+0000","last_became_peered":"2024-03-30T15:13:28.622538+0000","last_unstale":"2024-03-30T15:14:42.968444+0000","last_undegraded":"2024-03-30T15:14:42.968444+0000","last_fullsized":"2024-03-30T15:14:42.968444+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:30.648694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:54.318005+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.648694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.244373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"909'47","reported_seq":1370,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614768+0000","last_change":"2024-03-30T15:13:28.646691+0000","last_active":"2024-03-30T15:14:49.614768+0000","last_peered":"2024-03-30T15:14:49.614768+0000","last_clean":"2024-03-30T15:14:49.614768+0000","last_became_active":"2024-03-30T15:13:28.645038+0000","last_became_peered":"2024-03-30T15:13:28.645038+0000","last_unstale":"2024-03-30T15:14:49.614768+0000","last_undegraded":"2024-03-30T15:14:49.614768+0000","last_fullsized":"2024-03-30T15:14:49.614768+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'37","last_scrub_stamp":"2024-03-30T15:08:02.516392+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:08:02.516392+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1775,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596457+0000","last_change":"2024-03-30T15:13:52.591664+0000","last_active":"2024-03-30T15:14:42.596457+0000","last_peered":"2024-03-30T15:14:42.596457+0000","last_clean":"2024-03-30T15:14:42.596457+0000","last_became_active":"2024-03-30T15:13:52.591330+0000","last_became_peered":"2024-03-30T15:13:52.591330+0000","last_unstale":"2024-03-30T15:14:42.596457+0000","last_undegraded":"2024-03-30T15:14:42.596457+0000","last_fullsized":"2024-03-30T15:14:42.596457+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:39.476204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.476204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:03.771719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1766,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594862+0000","last_change":"2024-03-30T15:14:12.245431+0000","last_active":"2024-03-30T15:14:42.594862+0000","last_peered":"2024-03-30T15:14:42.594862+0000","last_clean":"2024-03-30T15:14:42.594862+0000","last_became_active":"2024-03-30T15:13:52.588705+0000","last_became_peered":"2024-03-30T15:13:52.588705+0000","last_unstale":"2024-03-30T15:14:42.594862+0000","last_undegraded":"2024-03-30T15:14:42.594862+0000","last_fullsized":"2024-03-30T15:14:42.594862+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:12.245351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.245351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:30.371930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":1041,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595051+0000","last_change":"2024-03-30T15:14:33.261448+0000","last_active":"2024-03-30T15:14:42.595051+0000","last_peered":"2024-03-30T15:14:42.595051+0000","last_clean":"2024-03-30T15:14:42.595051+0000","last_became_active":"2024-03-30T15:13:03.514491+0000","last_became_peered":"2024-03-30T15:13:03.514491+0000","last_unstale":"2024-03-30T15:14:42.595051+0000","last_undegraded":"2024-03-30T15:14:42.595051+0000","last_fullsized":"2024-03-30T15:14:42.595051+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:14:33.261365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.261365+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:54.793305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1260,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614797+0000","last_change":"2024-03-30T15:14:14.201276+0000","last_active":"2024-03-30T15:14:49.614797+0000","last_peered":"2024-03-30T15:14:49.614797+0000","last_clean":"2024-03-30T15:14:49.614797+0000","last_became_active":"2024-03-30T15:13:28.642223+0000","last_became_peered":"2024-03-30T15:13:28.642223+0000","last_unstale":"2024-03-30T15:14:49.614797+0000","last_undegraded":"2024-03-30T15:14:49.614797+0000","last_fullsized":"2024-03-30T15:14:49.614797+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:14.201134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.201134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:31.718490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1730,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596969+0000","last_change":"2024-03-30T15:13:31.905462+0000","last_active":"2024-03-30T15:14:42.596969+0000","last_peered":"2024-03-30T15:14:42.596969+0000","last_clean":"2024-03-30T15:14:42.596969+0000","last_became_active":"2024-03-30T15:13:28.622981+0000","last_became_peered":"2024-03-30T15:13:28.622981+0000","last_unstale":"2024-03-30T15:14:42.596969+0000","last_undegraded":"2024-03-30T15:14:42.596969+0000","last_fullsized":"2024-03-30T15:14:42.596969+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.905371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.905371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.331086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"909'36","reported_seq":1373,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614870+0000","last_change":"2024-03-30T15:13:52.603549+0000","last_active":"2024-03-30T15:14:49.614870+0000","last_peered":"2024-03-30T15:14:49.614870+0000","last_clean":"2024-03-30T15:14:49.614870+0000","last_became_active":"2024-03-30T15:13:52.591080+0000","last_became_peered":"2024-03-30T15:13:52.591080+0000","last_unstale":"2024-03-30T15:14:49.614870+0000","last_undegraded":"2024-03-30T15:14:49.614870+0000","last_fullsized":"2024-03-30T15:14:49.614870+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:08:01.498983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:01.498983+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":2953,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:46.753014+0000","last_change":"2024-03-30T15:13:39.069148+0000","last_active":"2024-03-30T15:14:46.753014+0000","last_peered":"2024-03-30T15:14:46.753014+0000","last_clean":"2024-03-30T15:14:46.753014+0000","last_became_active":"2024-03-30T15:13:28.632404+0000","last_became_peered":"2024-03-30T15:13:28.632404+0000","last_unstale":"2024-03-30T15:14:46.753014+0000","last_undegraded":"2024-03-30T15:14:46.753014+0000","last_fullsized":"2024-03-30T15:14:46.753014+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:13:39.069031+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.069031+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.220616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1199,"num_read_kb":1194,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1326,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613950+0000","last_change":"2024-03-30T15:14:09.726829+0000","last_active":"2024-03-30T15:14:49.613950+0000","last_peered":"2024-03-30T15:14:49.613950+0000","last_clean":"2024-03-30T15:14:49.613950+0000","last_became_active":"2024-03-30T15:14:09.726541+0000","last_became_peered":"2024-03-30T15:14:09.726541+0000","last_unstale":"2024-03-30T15:14:49.613950+0000","last_undegraded":"2024-03-30T15:14:49.613950+0000","last_fullsized":"2024-03-30T15:14:49.613950+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:36.922218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:11:36.922218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1758,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596198+0000","last_change":"2024-03-30T15:14:29.204333+0000","last_active":"2024-03-30T15:14:42.596198+0000","last_peered":"2024-03-30T15:14:42.596198+0000","last_clean":"2024-03-30T15:14:42.596198+0000","last_became_active":"2024-03-30T15:13:03.514904+0000","last_became_peered":"2024-03-30T15:13:03.514904+0000","last_unstale":"2024-03-30T15:14:42.596198+0000","last_undegraded":"2024-03-30T15:14:42.596198+0000","last_fullsized":"2024-03-30T15:14:42.596198+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.204254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.204254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.733748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1642,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953081+0000","last_change":"2024-03-30T15:14:24.106210+0000","last_active":"2024-03-30T15:14:43.953081+0000","last_peered":"2024-03-30T15:14:43.953081+0000","last_clean":"2024-03-30T15:14:43.953081+0000","last_became_active":"2024-03-30T15:13:19.276949+0000","last_became_peered":"2024-03-30T15:13:19.276949+0000","last_unstale":"2024-03-30T15:14:43.953081+0000","last_undegraded":"2024-03-30T15:14:43.953081+0000","last_fullsized":"2024-03-30T15:14:43.953081+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:24.106039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:24.106039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.706646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1738,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953042+0000","last_change":"2024-03-30T15:14:12.151419+0000","last_active":"2024-03-30T15:14:43.953042+0000","last_peered":"2024-03-30T15:14:43.953042+0000","last_clean":"2024-03-30T15:14:43.953042+0000","last_became_active":"2024-03-30T15:13:19.271491+0000","last_became_peered":"2024-03-30T15:13:19.271491+0000","last_unstale":"2024-03-30T15:14:43.953042+0000","last_undegraded":"2024-03-30T15:14:43.953042+0000","last_fullsized":"2024-03-30T15:14:43.953042+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:12.151317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:12.151317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.046779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"909'50","reported_seq":2067,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.293457+0000","last_change":"2024-03-30T15:14:36.416421+0000","last_active":"2024-03-30T15:14:45.293457+0000","last_peered":"2024-03-30T15:14:45.293457+0000","last_clean":"2024-03-30T15:14:45.293457+0000","last_became_active":"2024-03-30T15:13:03.543824+0000","last_became_peered":"2024-03-30T15:13:03.543824+0000","last_unstale":"2024-03-30T15:14:45.293457+0000","last_undegraded":"2024-03-30T15:14:45.293457+0000","last_fullsized":"2024-03-30T15:14:45.293457+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'38","last_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_deep_scrub":"905'38","last_deep_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.416317+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:42.691935+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1632,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595705+0000","last_change":"2024-03-30T15:12:20.989844+0000","last_active":"2024-03-30T15:14:42.595705+0000","last_peered":"2024-03-30T15:14:42.595705+0000","last_clean":"2024-03-30T15:14:42.595705+0000","last_became_active":"2024-03-30T14:57:56.271447+0000","last_became_peered":"2024-03-30T14:57:56.271447+0000","last_unstale":"2024-03-30T15:14:42.595705+0000","last_undegraded":"2024-03-30T15:14:42.595705+0000","last_fullsized":"2024-03-30T15:14:42.595705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:20.989740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:12:20.989740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":2008,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952244+0000","last_change":"2024-03-30T15:13:29.044762+0000","last_active":"2024-03-30T15:14:43.952244+0000","last_peered":"2024-03-30T15:14:43.952244+0000","last_clean":"2024-03-30T15:14:43.952244+0000","last_became_active":"2024-03-30T15:13:28.632600+0000","last_became_peered":"2024-03-30T15:13:28.632600+0000","last_unstale":"2024-03-30T15:14:43.952244+0000","last_undegraded":"2024-03-30T15:14:43.952244+0000","last_fullsized":"2024-03-30T15:14:43.952244+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:29.044665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:29.044665+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1726,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596534+0000","last_change":"2024-03-30T15:13:03.523181+0000","last_active":"2024-03-30T15:14:42.596534+0000","last_peered":"2024-03-30T15:14:42.596534+0000","last_clean":"2024-03-30T15:14:42.596534+0000","last_became_active":"2024-03-30T15:13:03.522675+0000","last_became_peered":"2024-03-30T15:13:03.522675+0000","last_unstale":"2024-03-30T15:14:42.596534+0000","last_undegraded":"2024-03-30T15:14:42.596534+0000","last_fullsized":"2024-03-30T15:14:42.596534+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.840893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":2070,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952336+0000","last_change":"2024-03-30T15:14:12.746871+0000","last_active":"2024-03-30T15:14:43.952336+0000","last_peered":"2024-03-30T15:14:43.952336+0000","last_clean":"2024-03-30T15:14:43.952336+0000","last_became_active":"2024-03-30T15:14:12.746444+0000","last_became_peered":"2024-03-30T15:14:12.746444+0000","last_unstale":"2024-03-30T15:14:43.952336+0000","last_undegraded":"2024-03-30T15:14:43.952336+0000","last_fullsized":"2024-03-30T15:14:43.952336+0000","mapping_epoch":883,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:27.092990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:27.092990+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.631034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1664,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969221+0000","last_change":"2024-03-30T15:13:52.590016+0000","last_active":"2024-03-30T15:14:42.969221+0000","last_peered":"2024-03-30T15:14:42.969221+0000","last_clean":"2024-03-30T15:14:42.969221+0000","last_became_active":"2024-03-30T15:13:52.589696+0000","last_became_peered":"2024-03-30T15:13:52.589696+0000","last_unstale":"2024-03-30T15:14:42.969221+0000","last_undegraded":"2024-03-30T15:14:42.969221+0000","last_fullsized":"2024-03-30T15:14:42.969221+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:46.745785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:11:46.745785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.354389+0000","last_change":"2024-03-30T15:14:36.036176+0000","last_active":"2024-03-30T15:14:50.354389+0000","last_peered":"2024-03-30T15:14:50.354389+0000","last_clean":"2024-03-30T15:14:50.354389+0000","last_became_active":"2024-03-30T15:13:52.594698+0000","last_became_peered":"2024-03-30T15:13:52.594698+0000","last_unstale":"2024-03-30T15:14:50.354389+0000","last_undegraded":"2024-03-30T15:14:50.354389+0000","last_fullsized":"2024-03-30T15:14:50.354389+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:36.036047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.036047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:57.199133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"909'26","reported_seq":1796,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.229987+0000","last_change":"2024-03-30T15:13:29.494270+0000","last_active":"2024-03-30T15:14:45.229987+0000","last_peered":"2024-03-30T15:14:45.229987+0000","last_clean":"2024-03-30T15:14:45.229987+0000","last_became_active":"2024-03-30T15:13:28.631530+0000","last_became_peered":"2024-03-30T15:13:28.631530+0000","last_unstale":"2024-03-30T15:14:45.229987+0000","last_undegraded":"2024-03-30T15:14:45.229987+0000","last_fullsized":"2024-03-30T15:14:45.229987+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'22","last_scrub_stamp":"2024-03-30T15:13:29.494150+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:13:29.494150+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1605,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353667+0000","last_change":"2024-03-30T15:14:17.334654+0000","last_active":"2024-03-30T15:14:50.353667+0000","last_peered":"2024-03-30T15:14:50.353667+0000","last_clean":"2024-03-30T15:14:50.353667+0000","last_became_active":"2024-03-30T15:13:28.614801+0000","last_became_peered":"2024-03-30T15:13:28.614801+0000","last_unstale":"2024-03-30T15:14:50.353667+0000","last_undegraded":"2024-03-30T15:14:50.353667+0000","last_fullsized":"2024-03-30T15:14:50.353667+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.334571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:55.918411+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.334571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:37.752428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1765,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594789+0000","last_change":"2024-03-30T15:13:36.535601+0000","last_active":"2024-03-30T15:14:42.594789+0000","last_peered":"2024-03-30T15:14:42.594789+0000","last_clean":"2024-03-30T15:14:42.594789+0000","last_became_active":"2024-03-30T15:13:19.290979+0000","last_became_peered":"2024-03-30T15:13:19.290979+0000","last_unstale":"2024-03-30T15:14:42.594789+0000","last_undegraded":"2024-03-30T15:14:42.594789+0000","last_fullsized":"2024-03-30T15:14:42.594789+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:36.535520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.535520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:45.609458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1741,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.951905+0000","last_change":"2024-03-30T15:14:31.060531+0000","last_active":"2024-03-30T15:14:43.951905+0000","last_peered":"2024-03-30T15:14:43.951905+0000","last_clean":"2024-03-30T15:14:43.951905+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:14:43.951905+0000","last_undegraded":"2024-03-30T15:14:43.951905+0000","last_fullsized":"2024-03-30T15:14:43.951905+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:31.060450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:31.060450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.835750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":1021,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595214+0000","last_change":"2024-03-30T15:13:32.864468+0000","last_active":"2024-03-30T15:14:42.595214+0000","last_peered":"2024-03-30T15:14:42.595214+0000","last_clean":"2024-03-30T15:14:42.595214+0000","last_became_active":"2024-03-30T15:13:28.623160+0000","last_became_peered":"2024-03-30T15:13:28.623160+0000","last_unstale":"2024-03-30T15:14:42.595214+0000","last_undegraded":"2024-03-30T15:14:42.595214+0000","last_fullsized":"2024-03-30T15:14:42.595214+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:13:32.864377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:13:32.864377+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1266,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614825+0000","last_change":"2024-03-30T15:14:13.185434+0000","last_active":"2024-03-30T15:14:49.614825+0000","last_peered":"2024-03-30T15:14:49.614825+0000","last_clean":"2024-03-30T15:14:49.614825+0000","last_became_active":"2024-03-30T15:13:28.639814+0000","last_became_peered":"2024-03-30T15:13:28.639814+0000","last_unstale":"2024-03-30T15:14:49.614825+0000","last_undegraded":"2024-03-30T15:14:49.614825+0000","last_fullsized":"2024-03-30T15:14:49.614825+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:13.185353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:13.185353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.512661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1370,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615031+0000","last_change":"2024-03-30T15:14:09.726789+0000","last_active":"2024-03-30T15:14:49.615031+0000","last_peered":"2024-03-30T15:14:49.615031+0000","last_clean":"2024-03-30T15:14:49.615031+0000","last_became_active":"2024-03-30T15:14:09.726470+0000","last_became_peered":"2024-03-30T15:14:09.726470+0000","last_unstale":"2024-03-30T15:14:49.615031+0000","last_undegraded":"2024-03-30T15:14:49.615031+0000","last_fullsized":"2024-03-30T15:14:49.615031+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:47.455580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.455580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"909'55","reported_seq":2018,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.291109+0000","last_change":"2024-03-30T15:13:42.932913+0000","last_active":"2024-03-30T15:14:45.291109+0000","last_peered":"2024-03-30T15:14:45.291109+0000","last_clean":"2024-03-30T15:14:45.291109+0000","last_became_active":"2024-03-30T15:13:19.293582+0000","last_became_peered":"2024-03-30T15:13:19.293582+0000","last_unstale":"2024-03-30T15:14:45.291109+0000","last_undegraded":"2024-03-30T15:14:45.291109+0000","last_fullsized":"2024-03-30T15:14:45.291109+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'43","last_scrub_stamp":"2024-03-30T15:13:42.932748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.932748+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.764445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1671,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841484+0000","last_change":"2024-03-30T15:13:28.784472+0000","last_active":"2024-03-30T15:14:49.841484+0000","last_peered":"2024-03-30T15:14:49.841484+0000","last_clean":"2024-03-30T15:14:49.841484+0000","last_became_active":"2024-03-30T15:13:28.624999+0000","last_became_peered":"2024-03-30T15:13:28.624999+0000","last_unstale":"2024-03-30T15:14:49.841484+0000","last_undegraded":"2024-03-30T15:14:49.841484+0000","last_fullsized":"2024-03-30T15:14:49.841484+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.784351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.784351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1740,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595840+0000","last_change":"2024-03-30T15:12:16.942168+0000","last_active":"2024-03-30T15:14:42.595840+0000","last_peered":"2024-03-30T15:14:42.595840+0000","last_clean":"2024-03-30T15:14:42.595840+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:14:42.595840+0000","last_undegraded":"2024-03-30T15:14:42.595840+0000","last_fullsized":"2024-03-30T15:14:42.595840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:12:16.942087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:12:16.942087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1601,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841910+0000","last_change":"2024-03-30T15:14:16.682735+0000","last_active":"2024-03-30T15:14:49.841910+0000","last_peered":"2024-03-30T15:14:49.841910+0000","last_clean":"2024-03-30T15:14:49.841910+0000","last_became_active":"2024-03-30T15:13:03.545212+0000","last_became_peered":"2024-03-30T15:13:03.545212+0000","last_unstale":"2024-03-30T15:14:49.841910+0000","last_undegraded":"2024-03-30T15:14:49.841910+0000","last_fullsized":"2024-03-30T15:14:49.841910+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.682638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.682638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:40.967052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1262,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614700+0000","last_change":"2024-03-30T15:14:00.635645+0000","last_active":"2024-03-30T15:14:49.614700+0000","last_peered":"2024-03-30T15:14:49.614700+0000","last_clean":"2024-03-30T15:14:49.614700+0000","last_became_active":"2024-03-30T15:14:00.632424+0000","last_became_peered":"2024-03-30T15:14:00.632424+0000","last_unstale":"2024-03-30T15:14:49.614700+0000","last_undegraded":"2024-03-30T15:14:49.614700+0000","last_fullsized":"2024-03-30T15:14:49.614700+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:38.079399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:11:38.079399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"909'21","reported_seq":1357,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614825+0000","last_change":"2024-03-30T15:13:28.633451+0000","last_active":"2024-03-30T15:14:49.614825+0000","last_peered":"2024-03-30T15:14:49.614825+0000","last_clean":"2024-03-30T15:14:49.614825+0000","last_became_active":"2024-03-30T15:13:28.629693+0000","last_became_peered":"2024-03-30T15:13:28.629693+0000","last_unstale":"2024-03-30T15:14:49.614825+0000","last_undegraded":"2024-03-30T15:14:49.614825+0000","last_fullsized":"2024-03-30T15:14:49.614825+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:08:12.601790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:08:12.601790+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596506+0000","last_change":"2024-03-30T15:12:59.144443+0000","last_active":"2024-03-30T15:14:42.596506+0000","last_peered":"2024-03-30T15:14:42.596506+0000","last_clean":"2024-03-30T15:14:42.596506+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:14:42.596506+0000","last_undegraded":"2024-03-30T15:14:42.596506+0000","last_fullsized":"2024-03-30T15:14:42.596506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_clean_scrub_stamp":"2024-03-30T15:12:59.144310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1613,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353681+0000","last_change":"2024-03-30T15:13:28.205972+0000","last_active":"2024-03-30T15:14:50.353681+0000","last_peered":"2024-03-30T15:14:50.353681+0000","last_clean":"2024-03-30T15:14:50.353681+0000","last_became_active":"2024-03-30T15:13:19.284117+0000","last_became_peered":"2024-03-30T15:13:19.284117+0000","last_unstale":"2024-03-30T15:14:50.353681+0000","last_undegraded":"2024-03-30T15:14:50.353681+0000","last_fullsized":"2024-03-30T15:14:50.353681+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.205876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.205876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:54.117502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1301,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613552+0000","last_change":"2024-03-30T15:14:30.194970+0000","last_active":"2024-03-30T15:14:49.613552+0000","last_peered":"2024-03-30T15:14:49.613552+0000","last_clean":"2024-03-30T15:14:49.613552+0000","last_became_active":"2024-03-30T15:13:52.588594+0000","last_became_peered":"2024-03-30T15:13:52.588594+0000","last_unstale":"2024-03-30T15:14:49.613552+0000","last_undegraded":"2024-03-30T15:14:49.613552+0000","last_fullsized":"2024-03-30T15:14:49.613552+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:30.194832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:30.194832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:32.945100+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1610,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841322+0000","last_change":"2024-03-30T15:14:00.635095+0000","last_active":"2024-03-30T15:14:49.841322+0000","last_peered":"2024-03-30T15:14:49.841322+0000","last_clean":"2024-03-30T15:14:49.841322+0000","last_became_active":"2024-03-30T15:14:00.634711+0000","last_became_peered":"2024-03-30T15:14:00.634711+0000","last_unstale":"2024-03-30T15:14:49.841322+0000","last_undegraded":"2024-03-30T15:14:49.841322+0000","last_fullsized":"2024-03-30T15:14:49.841322+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:37.755201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.755201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.272904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"909'51","reported_seq":1853,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.316545+0000","last_change":"2024-03-30T15:13:37.663625+0000","last_active":"2024-03-30T15:14:45.316545+0000","last_peered":"2024-03-30T15:14:45.316545+0000","last_clean":"2024-03-30T15:14:45.316545+0000","last_became_active":"2024-03-30T15:13:19.285236+0000","last_became_peered":"2024-03-30T15:13:19.285236+0000","last_unstale":"2024-03-30T15:14:45.316545+0000","last_undegraded":"2024-03-30T15:14:45.316545+0000","last_fullsized":"2024-03-30T15:14:45.316545+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'35","last_scrub_stamp":"2024-03-30T15:13:37.663475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.663475+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.956850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953166+0000","last_change":"2024-03-30T15:12:19.812425+0000","last_active":"2024-03-30T15:14:43.953166+0000","last_peered":"2024-03-30T15:14:43.953166+0000","last_clean":"2024-03-30T15:14:43.953166+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:14:43.953166+0000","last_undegraded":"2024-03-30T15:14:43.953166+0000","last_fullsized":"2024-03-30T15:14:43.953166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:19.812343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:58:23.951507+0000","last_clean_scrub_stamp":"2024-03-30T15:12:19.812343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":102,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353649+0000","last_change":"2024-03-30T15:14:14.317102+0000","last_active":"2024-03-30T15:14:50.353649+0000","last_peered":"2024-03-30T15:14:50.353649+0000","last_clean":"2024-03-30T15:14:50.353649+0000","last_became_active":"2024-03-30T15:13:52.587447+0000","last_became_peered":"2024-03-30T15:13:52.587447+0000","last_unstale":"2024-03-30T15:14:50.353649+0000","last_undegraded":"2024-03-30T15:14:50.353649+0000","last_fullsized":"2024-03-30T15:14:50.353649+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:14.317001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:14.317001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.560518+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1734,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952088+0000","last_change":"2024-03-30T15:14:15.099798+0000","last_active":"2024-03-30T15:14:43.952088+0000","last_peered":"2024-03-30T15:14:43.952088+0000","last_clean":"2024-03-30T15:14:43.952088+0000","last_became_active":"2024-03-30T15:13:19.271586+0000","last_became_peered":"2024-03-30T15:13:19.271586+0000","last_unstale":"2024-03-30T15:14:43.952088+0000","last_undegraded":"2024-03-30T15:14:43.952088+0000","last_fullsized":"2024-03-30T15:14:43.952088+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.099699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.099699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:18.098833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1654,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965500+0000","last_change":"2024-03-30T15:12:22.500804+0000","last_active":"2024-03-30T15:14:42.965500+0000","last_peered":"2024-03-30T15:14:42.965500+0000","last_clean":"2024-03-30T15:14:42.965500+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:14:42.965500+0000","last_undegraded":"2024-03-30T15:14:42.965500+0000","last_fullsized":"2024-03-30T15:14:42.965500+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:12:22.500717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:22.500717+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1580,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353387+0000","last_change":"2024-03-30T15:13:36.165499+0000","last_active":"2024-03-30T15:14:50.353387+0000","last_peered":"2024-03-30T15:14:50.353387+0000","last_clean":"2024-03-30T15:14:50.353387+0000","last_became_active":"2024-03-30T15:13:28.646999+0000","last_became_peered":"2024-03-30T15:13:28.646999+0000","last_unstale":"2024-03-30T15:14:50.353387+0000","last_undegraded":"2024-03-30T15:14:50.353387+0000","last_fullsized":"2024-03-30T15:14:50.353387+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:36.165416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.165416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:05.472693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"909'24","reported_seq":1762,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841002+0000","last_change":"2024-03-30T15:14:25.778635+0000","last_active":"2024-03-30T15:14:49.841002+0000","last_peered":"2024-03-30T15:14:49.841002+0000","last_clean":"2024-03-30T15:14:49.841002+0000","last_became_active":"2024-03-30T15:14:00.632108+0000","last_became_peered":"2024-03-30T15:14:00.632108+0000","last_unstale":"2024-03-30T15:14:49.841002+0000","last_undegraded":"2024-03-30T15:14:49.841002+0000","last_fullsized":"2024-03-30T15:14:49.841002+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:14:25.778547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:25.778547+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:32.231962+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1745,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953206+0000","last_change":"2024-03-30T15:14:22.044920+0000","last_active":"2024-03-30T15:14:43.953206+0000","last_peered":"2024-03-30T15:14:43.953206+0000","last_clean":"2024-03-30T15:14:43.953206+0000","last_became_active":"2024-03-30T15:13:03.550868+0000","last_became_peered":"2024-03-30T15:13:03.550868+0000","last_unstale":"2024-03-30T15:14:43.953206+0000","last_undegraded":"2024-03-30T15:14:43.953206+0000","last_fullsized":"2024-03-30T15:14:43.953206+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.044816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.044816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.372065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1629,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952802+0000","last_change":"2024-03-30T15:10:58.066365+0000","last_active":"2024-03-30T15:14:43.952802+0000","last_peered":"2024-03-30T15:14:43.952802+0000","last_clean":"2024-03-30T15:14:43.952802+0000","last_became_active":"2024-03-30T14:57:56.273443+0000","last_became_peered":"2024-03-30T14:57:56.273443+0000","last_unstale":"2024-03-30T15:14:43.952802+0000","last_undegraded":"2024-03-30T15:14:43.952802+0000","last_fullsized":"2024-03-30T15:14:43.952802+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:58.066275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:10:58.066275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":76,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840532+0000","last_change":"2024-03-30T15:14:15.728874+0000","last_active":"2024-03-30T15:14:49.840532+0000","last_peered":"2024-03-30T15:14:49.840532+0000","last_clean":"2024-03-30T15:14:49.840532+0000","last_became_active":"2024-03-30T15:14:09.725078+0000","last_became_peered":"2024-03-30T15:14:09.725078+0000","last_unstale":"2024-03-30T15:14:49.840532+0000","last_undegraded":"2024-03-30T15:14:49.840532+0000","last_fullsized":"2024-03-30T15:14:49.840532+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:15.728771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:15.728771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.523390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"833'10","reported_seq":1983,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952755+0000","last_change":"2024-03-30T15:13:04.947263+0000","last_active":"2024-03-30T15:14:43.952755+0000","last_peered":"2024-03-30T15:14:43.952755+0000","last_clean":"2024-03-30T15:14:43.952755+0000","last_became_active":"2024-03-30T15:13:03.549786+0000","last_became_peered":"2024-03-30T15:13:03.549786+0000","last_unstale":"2024-03-30T15:14:43.952755+0000","last_undegraded":"2024-03-30T15:14:43.952755+0000","last_fullsized":"2024-03-30T15:14:43.952755+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'10","last_scrub_stamp":"2024-03-30T15:13:04.947176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:04.947176+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.127393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"909'39","reported_seq":1794,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.249748+0000","last_change":"2024-03-30T15:14:24.914466+0000","last_active":"2024-03-30T15:14:45.249748+0000","last_peered":"2024-03-30T15:14:45.249748+0000","last_clean":"2024-03-30T15:14:45.249748+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:14:45.249748+0000","last_undegraded":"2024-03-30T15:14:45.249748+0000","last_fullsized":"2024-03-30T15:14:45.249748+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'27","last_scrub_stamp":"2024-03-30T15:14:24.914288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:24.914288+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:43.784084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1754,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.953341+0000","last_change":"2024-03-30T15:13:36.062294+0000","last_active":"2024-03-30T15:14:43.953341+0000","last_peered":"2024-03-30T15:14:43.953341+0000","last_clean":"2024-03-30T15:14:43.953341+0000","last_became_active":"2024-03-30T15:13:28.632685+0000","last_became_peered":"2024-03-30T15:13:28.632685+0000","last_unstale":"2024-03-30T15:14:43.953341+0000","last_undegraded":"2024-03-30T15:14:43.953341+0000","last_fullsized":"2024-03-30T15:14:43.953341+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:13:36.062171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.062171+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.071532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1726,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596332+0000","last_change":"2024-03-30T15:13:44.911542+0000","last_active":"2024-03-30T15:14:42.596332+0000","last_peered":"2024-03-30T15:14:42.596332+0000","last_clean":"2024-03-30T15:14:42.596332+0000","last_became_active":"2024-03-30T15:13:28.625385+0000","last_became_peered":"2024-03-30T15:13:28.625385+0000","last_unstale":"2024-03-30T15:14:42.596332+0000","last_undegraded":"2024-03-30T15:14:42.596332+0000","last_fullsized":"2024-03-30T15:14:42.596332+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.911436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:47.030057+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.911436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:47.418564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1665,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969180+0000","last_change":"2024-03-30T15:14:21.534146+0000","last_active":"2024-03-30T15:14:42.969180+0000","last_peered":"2024-03-30T15:14:42.969180+0000","last_clean":"2024-03-30T15:14:42.969180+0000","last_became_active":"2024-03-30T15:13:52.592711+0000","last_became_peered":"2024-03-30T15:13:52.592711+0000","last_unstale":"2024-03-30T15:14:42.969180+0000","last_undegraded":"2024-03-30T15:14:42.969180+0000","last_fullsized":"2024-03-30T15:14:42.969180+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:21.534007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.534007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.099017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"909'18","reported_seq":102,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.264955+0000","last_change":"2024-03-30T15:14:21.380980+0000","last_active":"2024-03-30T15:14:45.264955+0000","last_peered":"2024-03-30T15:14:45.264955+0000","last_clean":"2024-03-30T15:14:45.264955+0000","last_became_active":"2024-03-30T15:14:00.631854+0000","last_became_peered":"2024-03-30T15:14:00.631854+0000","last_unstale":"2024-03-30T15:14:45.264955+0000","last_undegraded":"2024-03-30T15:14:45.264955+0000","last_fullsized":"2024-03-30T15:14:45.264955+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'12","last_scrub_stamp":"2024-03-30T15:14:21.380891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.380891+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:26.416878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1751,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.965579+0000","last_change":"2024-03-30T15:13:31.683113+0000","last_active":"2024-03-30T15:14:42.965579+0000","last_peered":"2024-03-30T15:14:42.965579+0000","last_clean":"2024-03-30T15:14:42.965579+0000","last_became_active":"2024-03-30T15:13:28.623365+0000","last_became_peered":"2024-03-30T15:13:28.623365+0000","last_unstale":"2024-03-30T15:14:42.965579+0000","last_undegraded":"2024-03-30T15:14:42.965579+0000","last_fullsized":"2024-03-30T15:14:42.965579+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.682971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:45.427234+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.682971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1728,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595988+0000","last_change":"2024-03-30T15:14:27.928955+0000","last_active":"2024-03-30T15:14:42.595988+0000","last_peered":"2024-03-30T15:14:42.595988+0000","last_clean":"2024-03-30T15:14:42.595988+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:14:42.595988+0000","last_undegraded":"2024-03-30T15:14:42.595988+0000","last_fullsized":"2024-03-30T15:14:42.595988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:27.928851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:27.928851+0000","last_clean_scrub_stamp":"2024-03-30T15:14:27.928851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.764039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1725,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952854+0000","last_change":"2024-03-30T15:13:44.201709+0000","last_active":"2024-03-30T15:14:43.952854+0000","last_peered":"2024-03-30T15:14:43.952854+0000","last_clean":"2024-03-30T15:14:43.952854+0000","last_became_active":"2024-03-30T15:13:03.537074+0000","last_became_peered":"2024-03-30T15:13:03.537074+0000","last_unstale":"2024-03-30T15:14:43.952854+0000","last_undegraded":"2024-03-30T15:14:43.952854+0000","last_fullsized":"2024-03-30T15:14:43.952854+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.201602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.201602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.021081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1612,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.842022+0000","last_change":"2024-03-30T15:13:03.530702+0000","last_active":"2024-03-30T15:14:49.842022+0000","last_peered":"2024-03-30T15:14:49.842022+0000","last_clean":"2024-03-30T15:14:49.842022+0000","last_became_active":"2024-03-30T15:13:03.529999+0000","last_became_peered":"2024-03-30T15:13:03.529999+0000","last_unstale":"2024-03-30T15:14:49.842022+0000","last_undegraded":"2024-03-30T15:14:49.842022+0000","last_fullsized":"2024-03-30T15:14:49.842022+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:13.914480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:12:13.914480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"909'21","reported_seq":1356,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615424+0000","last_change":"2024-03-30T15:13:28.646408+0000","last_active":"2024-03-30T15:14:49.615424+0000","last_peered":"2024-03-30T15:14:49.615424+0000","last_clean":"2024-03-30T15:14:49.615424+0000","last_became_active":"2024-03-30T15:13:28.642687+0000","last_became_peered":"2024-03-30T15:13:28.642687+0000","last_unstale":"2024-03-30T15:14:49.615424+0000","last_undegraded":"2024-03-30T15:14:49.615424+0000","last_fullsized":"2024-03-30T15:14:49.615424+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:07:34.553452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:07:34.553452+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1745,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952455+0000","last_change":"2024-03-30T15:13:46.180286+0000","last_active":"2024-03-30T15:14:43.952455+0000","last_peered":"2024-03-30T15:14:43.952455+0000","last_clean":"2024-03-30T15:14:43.952455+0000","last_became_active":"2024-03-30T15:13:03.548886+0000","last_became_peered":"2024-03-30T15:13:03.548886+0000","last_unstale":"2024-03-30T15:14:43.952455+0000","last_undegraded":"2024-03-30T15:14:43.952455+0000","last_fullsized":"2024-03-30T15:14:43.952455+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:46.180140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:46.180140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:00.861407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":822,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969115+0000","last_change":"2024-03-30T15:14:23.521674+0000","last_active":"2024-03-30T15:14:42.969115+0000","last_peered":"2024-03-30T15:14:42.969115+0000","last_clean":"2024-03-30T15:14:42.969115+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:14:42.969115+0000","last_undegraded":"2024-03-30T15:14:42.969115+0000","last_fullsized":"2024-03-30T15:14:42.969115+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.521577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.521577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.497032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1583,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.354045+0000","last_change":"2024-03-30T15:13:27.166879+0000","last_active":"2024-03-30T15:14:50.354045+0000","last_peered":"2024-03-30T15:14:50.354045+0000","last_clean":"2024-03-30T15:14:50.354045+0000","last_became_active":"2024-03-30T15:13:19.279299+0000","last_became_peered":"2024-03-30T15:13:19.279299+0000","last_unstale":"2024-03-30T15:14:50.354045+0000","last_undegraded":"2024-03-30T15:14:50.354045+0000","last_fullsized":"2024-03-30T15:14:50.354045+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:27.166790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:27.166790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:52.798993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1253,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615123+0000","last_change":"2024-03-30T15:13:52.590156+0000","last_active":"2024-03-30T15:14:49.615123+0000","last_peered":"2024-03-30T15:14:49.615123+0000","last_clean":"2024-03-30T15:14:49.615123+0000","last_became_active":"2024-03-30T15:13:52.589809+0000","last_became_peered":"2024-03-30T15:13:52.589809+0000","last_unstale":"2024-03-30T15:14:49.615123+0000","last_undegraded":"2024-03-30T15:14:49.615123+0000","last_fullsized":"2024-03-30T15:14:49.615123+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:25.719723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:25.719723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:54.190231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"909'21","reported_seq":1766,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.285541+0000","last_change":"2024-03-30T15:14:29.924425+0000","last_active":"2024-03-30T15:14:45.285541+0000","last_peered":"2024-03-30T15:14:45.285541+0000","last_clean":"2024-03-30T15:14:45.285541+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:14:45.285541+0000","last_undegraded":"2024-03-30T15:14:45.285541+0000","last_fullsized":"2024-03-30T15:14:45.285541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:14:29.924285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.924285+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:47.854739+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1310,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614446+0000","last_change":"2024-03-30T15:14:00.634621+0000","last_active":"2024-03-30T15:14:49.614446+0000","last_peered":"2024-03-30T15:14:49.614446+0000","last_clean":"2024-03-30T15:14:49.614446+0000","last_became_active":"2024-03-30T15:14:00.634218+0000","last_became_peered":"2024-03-30T15:14:00.634218+0000","last_unstale":"2024-03-30T15:14:49.614446+0000","last_undegraded":"2024-03-30T15:14:49.614446+0000","last_fullsized":"2024-03-30T15:14:49.614446+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:45.094026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:11:45.094026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"878'15","reported_seq":1903,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952959+0000","last_change":"2024-03-30T15:14:00.650824+0000","last_active":"2024-03-30T15:14:43.952959+0000","last_peered":"2024-03-30T15:14:43.952959+0000","last_clean":"2024-03-30T15:14:43.952959+0000","last_became_active":"2024-03-30T15:14:00.638157+0000","last_became_peered":"2024-03-30T15:14:00.638157+0000","last_unstale":"2024-03-30T15:14:43.952959+0000","last_undegraded":"2024-03-30T15:14:43.952959+0000","last_fullsized":"2024-03-30T15:14:43.952959+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'13","last_scrub_stamp":"2024-03-30T15:13:06.936305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.936305+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.127412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1633,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841921+0000","last_change":"2024-03-30T15:13:34.747694+0000","last_active":"2024-03-30T15:14:49.841921+0000","last_peered":"2024-03-30T15:14:49.841921+0000","last_clean":"2024-03-30T15:14:49.841921+0000","last_became_active":"2024-03-30T15:13:19.280923+0000","last_became_peered":"2024-03-30T15:13:19.280923+0000","last_unstale":"2024-03-30T15:14:49.841921+0000","last_undegraded":"2024-03-30T15:14:49.841921+0000","last_fullsized":"2024-03-30T15:14:49.841921+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:34.747586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:34.747586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:56.200164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1262,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614658+0000","last_change":"2024-03-30T15:14:17.194256+0000","last_active":"2024-03-30T15:14:49.614658+0000","last_peered":"2024-03-30T15:14:49.614658+0000","last_clean":"2024-03-30T15:14:49.614658+0000","last_became_active":"2024-03-30T15:13:28.637092+0000","last_became_peered":"2024-03-30T15:13:28.637092+0000","last_unstale":"2024-03-30T15:14:49.614658+0000","last_undegraded":"2024-03-30T15:14:49.614658+0000","last_fullsized":"2024-03-30T15:14:49.614658+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:17.194169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:17.194169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:19.028824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"909'35","reported_seq":1724,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.283147+0000","last_change":"2024-03-30T15:14:32.245733+0000","last_active":"2024-03-30T15:14:45.283147+0000","last_peered":"2024-03-30T15:14:45.283147+0000","last_clean":"2024-03-30T15:14:45.283147+0000","last_became_active":"2024-03-30T14:57:56.275725+0000","last_became_peered":"2024-03-30T14:57:56.275725+0000","last_unstale":"2024-03-30T15:14:45.283147+0000","last_undegraded":"2024-03-30T15:14:45.283147+0000","last_fullsized":"2024-03-30T15:14:45.283147+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"903'25","last_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_deep_scrub":"903'25","last_deep_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_clean_scrub_stamp":"2024-03-30T15:14:32.245633+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:39.205989+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1750,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952492+0000","last_change":"2024-03-30T15:14:29.136087+0000","last_active":"2024-03-30T15:14:43.952492+0000","last_peered":"2024-03-30T15:14:43.952492+0000","last_clean":"2024-03-30T15:14:43.952492+0000","last_became_active":"2024-03-30T15:13:19.277421+0000","last_became_peered":"2024-03-30T15:13:19.277421+0000","last_unstale":"2024-03-30T15:14:43.952492+0000","last_undegraded":"2024-03-30T15:14:43.952492+0000","last_fullsized":"2024-03-30T15:14:43.952492+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.135910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.135910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:49.766429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"852'12","reported_seq":1987,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.726811+0000","last_change":"2024-03-30T15:13:30.869874+0000","last_active":"2024-03-30T15:14:42.726811+0000","last_peered":"2024-03-30T15:14:42.726811+0000","last_clean":"2024-03-30T15:14:42.726811+0000","last_became_active":"2024-03-30T15:13:29.615517+0000","last_became_peered":"2024-03-30T15:13:29.615517+0000","last_unstale":"2024-03-30T15:14:42.726811+0000","last_undegraded":"2024-03-30T15:14:42.726811+0000","last_fullsized":"2024-03-30T15:14:42.726811+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"852'12","last_scrub_stamp":"2024-03-30T15:13:30.869737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:30.869737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.802749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1315,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614723+0000","last_change":"2024-03-30T15:14:00.649300+0000","last_active":"2024-03-30T15:14:49.614723+0000","last_peered":"2024-03-30T15:14:49.614723+0000","last_clean":"2024-03-30T15:14:49.614723+0000","last_became_active":"2024-03-30T15:14:00.638394+0000","last_became_peered":"2024-03-30T15:14:00.638394+0000","last_unstale":"2024-03-30T15:14:49.614723+0000","last_undegraded":"2024-03-30T15:14:49.614723+0000","last_fullsized":"2024-03-30T15:14:49.614723+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:11:40.102459+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:11:40.102459+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1600,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841331+0000","last_change":"2024-03-30T15:13:06.782014+0000","last_active":"2024-03-30T15:14:49.841331+0000","last_peered":"2024-03-30T15:14:49.841331+0000","last_clean":"2024-03-30T15:14:49.841331+0000","last_became_active":"2024-03-30T15:13:03.509862+0000","last_became_peered":"2024-03-30T15:13:03.509862+0000","last_unstale":"2024-03-30T15:14:49.841331+0000","last_undegraded":"2024-03-30T15:14:49.841331+0000","last_fullsized":"2024-03-30T15:14:49.841331+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:06.781928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:06.781928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"909'32","reported_seq":1738,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841200+0000","last_change":"2024-03-30T15:14:36.715235+0000","last_active":"2024-03-30T15:14:49.841200+0000","last_peered":"2024-03-30T15:14:49.841200+0000","last_clean":"2024-03-30T15:14:49.841200+0000","last_became_active":"2024-03-30T15:13:03.536364+0000","last_became_peered":"2024-03-30T15:13:03.536364+0000","last_unstale":"2024-03-30T15:14:49.841200+0000","last_undegraded":"2024-03-30T15:14:49.841200+0000","last_fullsized":"2024-03-30T15:14:49.841200+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:36.715131+0000","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2024-03-30T14:59:20.453594+0000","last_clean_scrub_stamp":"2024-03-30T15:14:36.715131+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:16:01.194138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1680,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841554+0000","last_change":"2024-03-30T15:13:04.847959+0000","last_active":"2024-03-30T15:14:49.841554+0000","last_peered":"2024-03-30T15:14:49.841554+0000","last_clean":"2024-03-30T15:14:49.841554+0000","last_became_active":"2024-03-30T15:13:03.539915+0000","last_became_peered":"2024-03-30T15:13:03.539915+0000","last_unstale":"2024-03-30T15:14:49.841554+0000","last_undegraded":"2024-03-30T15:14:49.841554+0000","last_fullsized":"2024-03-30T15:14:49.841554+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:04.847864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:04.847864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1758,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595017+0000","last_change":"2024-03-30T15:14:00.630183+0000","last_active":"2024-03-30T15:14:42.595017+0000","last_peered":"2024-03-30T15:14:42.595017+0000","last_clean":"2024-03-30T15:14:42.595017+0000","last_became_active":"2024-03-30T15:14:00.629833+0000","last_became_peered":"2024-03-30T15:14:00.629833+0000","last_unstale":"2024-03-30T15:14:42.595017+0000","last_undegraded":"2024-03-30T15:14:42.595017+0000","last_fullsized":"2024-03-30T15:14:42.595017+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:57.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:15.144864+0000","last_clean_scrub_stamp":"2024-03-30T15:12:57.294119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1220,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613437+0000","last_change":"2024-03-30T15:14:00.635564+0000","last_active":"2024-03-30T15:14:49.613437+0000","last_peered":"2024-03-30T15:14:49.613437+0000","last_clean":"2024-03-30T15:14:49.613437+0000","last_became_active":"2024-03-30T15:14:00.635207+0000","last_became_peered":"2024-03-30T15:14:00.635207+0000","last_unstale":"2024-03-30T15:14:49.613437+0000","last_undegraded":"2024-03-30T15:14:49.613437+0000","last_fullsized":"2024-03-30T15:14:49.613437+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:08:11.553667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":76,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840708+0000","last_change":"2024-03-30T15:14:29.787668+0000","last_active":"2024-03-30T15:14:49.840708+0000","last_peered":"2024-03-30T15:14:49.840708+0000","last_clean":"2024-03-30T15:14:49.840708+0000","last_became_active":"2024-03-30T15:14:09.726417+0000","last_became_peered":"2024-03-30T15:14:09.726417+0000","last_unstale":"2024-03-30T15:14:49.840708+0000","last_undegraded":"2024-03-30T15:14:49.840708+0000","last_fullsized":"2024-03-30T15:14:49.840708+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.787527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.787527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:37.823723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"909'21","reported_seq":1785,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.203684+0000","last_change":"2024-03-30T15:13:38.465288+0000","last_active":"2024-03-30T15:14:45.203684+0000","last_peered":"2024-03-30T15:14:45.203684+0000","last_clean":"2024-03-30T15:14:45.203684+0000","last_became_active":"2024-03-30T15:13:19.293272+0000","last_became_peered":"2024-03-30T15:13:19.293272+0000","last_unstale":"2024-03-30T15:14:45.203684+0000","last_undegraded":"2024-03-30T15:14:45.203684+0000","last_fullsized":"2024-03-30T15:14:45.203684+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:38.465198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:38.465198+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1760,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952970+0000","last_change":"2024-03-30T15:14:19.969427+0000","last_active":"2024-03-30T15:14:43.952970+0000","last_peered":"2024-03-30T15:14:43.952970+0000","last_clean":"2024-03-30T15:14:43.952970+0000","last_became_active":"2024-03-30T15:13:03.525235+0000","last_became_peered":"2024-03-30T15:13:03.525235+0000","last_unstale":"2024-03-30T15:14:43.952970+0000","last_undegraded":"2024-03-30T15:14:43.952970+0000","last_fullsized":"2024-03-30T15:14:43.952970+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:14:19.969342+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:14:19.969342+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.130177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":871,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615089+0000","last_change":"2024-03-30T15:13:29.620544+0000","last_active":"2024-03-30T15:14:49.615089+0000","last_peered":"2024-03-30T15:14:49.615089+0000","last_clean":"2024-03-30T15:14:49.615089+0000","last_became_active":"2024-03-30T15:13:29.619329+0000","last_became_peered":"2024-03-30T15:13:29.619329+0000","last_unstale":"2024-03-30T15:14:49.615089+0000","last_undegraded":"2024-03-30T15:14:49.615089+0000","last_fullsized":"2024-03-30T15:14:49.615089+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:46.479258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:46.479258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1334,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613659+0000","last_change":"2024-03-30T15:14:29.215562+0000","last_active":"2024-03-30T15:14:49.613659+0000","last_peered":"2024-03-30T15:14:49.613659+0000","last_clean":"2024-03-30T15:14:49.613659+0000","last_became_active":"2024-03-30T15:13:28.634012+0000","last_became_peered":"2024-03-30T15:13:28.634012+0000","last_unstale":"2024-03-30T15:14:49.613659+0000","last_undegraded":"2024-03-30T15:14:49.613659+0000","last_fullsized":"2024-03-30T15:14:49.613659+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:29.215490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:29.215490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.444348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1224,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613097+0000","last_change":"2024-03-30T15:13:31.137742+0000","last_active":"2024-03-30T15:14:49.613097+0000","last_peered":"2024-03-30T15:14:49.613097+0000","last_clean":"2024-03-30T15:14:49.613097+0000","last_became_active":"2024-03-30T15:13:28.626496+0000","last_became_peered":"2024-03-30T15:13:28.626496+0000","last_unstale":"2024-03-30T15:14:49.613097+0000","last_undegraded":"2024-03-30T15:14:49.613097+0000","last_fullsized":"2024-03-30T15:14:49.613097+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:31.137663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:13:31.137663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.017020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"909'27","reported_seq":1691,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.303432+0000","last_change":"2024-03-30T15:10:55.115248+0000","last_active":"2024-03-30T15:14:45.303432+0000","last_peered":"2024-03-30T15:14:45.303432+0000","last_clean":"2024-03-30T15:14:45.303432+0000","last_became_active":"2024-03-30T14:57:56.273020+0000","last_became_peered":"2024-03-30T14:57:56.273020+0000","last_unstale":"2024-03-30T15:14:45.303432+0000","last_undegraded":"2024-03-30T15:14:45.303432+0000","last_fullsized":"2024-03-30T15:14:45.303432+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'19","last_scrub_stamp":"2024-03-30T15:10:55.115152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:10:55.115152+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.127322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1736,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596062+0000","last_change":"2024-03-30T15:14:20.859711+0000","last_active":"2024-03-30T15:14:42.596062+0000","last_peered":"2024-03-30T15:14:42.596062+0000","last_clean":"2024-03-30T15:14:42.596062+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:14:42.596062+0000","last_undegraded":"2024-03-30T15:14:42.596062+0000","last_fullsized":"2024-03-30T15:14:42.596062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.859619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.859619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:36.845652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1737,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952388+0000","last_change":"2024-03-30T15:14:27.114393+0000","last_active":"2024-03-30T15:14:43.952388+0000","last_peered":"2024-03-30T15:14:43.952388+0000","last_clean":"2024-03-30T15:14:43.952388+0000","last_became_active":"2024-03-30T15:13:03.540174+0000","last_became_peered":"2024-03-30T15:13:03.540174+0000","last_unstale":"2024-03-30T15:14:43.952388+0000","last_undegraded":"2024-03-30T15:14:43.952388+0000","last_fullsized":"2024-03-30T15:14:43.952388+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:27.114273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_clean_scrub_stamp":"2024-03-30T15:14:27.114273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:35.103581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1233,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615320+0000","last_change":"2024-03-30T15:14:20.165761+0000","last_active":"2024-03-30T15:14:49.615320+0000","last_peered":"2024-03-30T15:14:49.615320+0000","last_clean":"2024-03-30T15:14:49.615320+0000","last_became_active":"2024-03-30T15:13:28.624430+0000","last_became_peered":"2024-03-30T15:13:28.624430+0000","last_unstale":"2024-03-30T15:14:49.615320+0000","last_undegraded":"2024-03-30T15:14:49.615320+0000","last_fullsized":"2024-03-30T15:14:49.615320+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.165646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.165646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.792269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1661,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595722+0000","last_change":"2024-03-30T15:14:23.181759+0000","last_active":"2024-03-30T15:14:42.595722+0000","last_peered":"2024-03-30T15:14:42.595722+0000","last_clean":"2024-03-30T15:14:42.595722+0000","last_became_active":"2024-03-30T15:13:03.527402+0000","last_became_peered":"2024-03-30T15:13:03.527402+0000","last_unstale":"2024-03-30T15:14:42.595722+0000","last_undegraded":"2024-03-30T15:14:42.595722+0000","last_fullsized":"2024-03-30T15:14:42.595722+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:23.181637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.181637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:28.900759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"909'12","reported_seq":1348,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.126962+0000","last_change":"2024-03-30T15:14:23.386867+0000","last_active":"2024-03-30T15:14:45.126962+0000","last_peered":"2024-03-30T15:14:45.126962+0000","last_clean":"2024-03-30T15:14:45.126962+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:14:45.126962+0000","last_undegraded":"2024-03-30T15:14:45.126962+0000","last_fullsized":"2024-03-30T15:14:45.126962+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8","last_scrub_stamp":"2024-03-30T15:14:23.386767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:14:23.386767+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:52.358728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1322,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615751+0000","last_change":"2024-03-30T15:13:28.626812+0000","last_active":"2024-03-30T15:14:49.615751+0000","last_peered":"2024-03-30T15:14:49.615751+0000","last_clean":"2024-03-30T15:14:49.615751+0000","last_became_active":"2024-03-30T15:13:28.626493+0000","last_became_peered":"2024-03-30T15:13:28.626493+0000","last_unstale":"2024-03-30T15:14:49.615751+0000","last_undegraded":"2024-03-30T15:14:49.615751+0000","last_fullsized":"2024-03-30T15:14:49.615751+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:10.508629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:08:10.508629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1767,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.968299+0000","last_change":"2024-03-30T15:14:34.385293+0000","last_active":"2024-03-30T15:14:42.968299+0000","last_peered":"2024-03-30T15:14:42.968299+0000","last_clean":"2024-03-30T15:14:42.968299+0000","last_became_active":"2024-03-30T15:14:09.727046+0000","last_became_peered":"2024-03-30T15:14:09.727046+0000","last_unstale":"2024-03-30T15:14:42.968299+0000","last_undegraded":"2024-03-30T15:14:42.968299+0000","last_fullsized":"2024-03-30T15:14:42.968299+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:34.385192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:34.385192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:45.959150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1730,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952810+0000","last_change":"2024-03-30T15:13:28.065073+0000","last_active":"2024-03-30T15:14:43.952810+0000","last_peered":"2024-03-30T15:14:43.952810+0000","last_clean":"2024-03-30T15:14:43.952810+0000","last_became_active":"2024-03-30T15:13:19.277971+0000","last_became_peered":"2024-03-30T15:13:19.277971+0000","last_unstale":"2024-03-30T15:14:43.952810+0000","last_undegraded":"2024-03-30T15:14:43.952810+0000","last_fullsized":"2024-03-30T15:14:43.952810+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:28.064921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:13:28.064921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.324547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1649,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969405+0000","last_change":"2024-03-30T15:14:25.522723+0000","last_active":"2024-03-30T15:14:42.969405+0000","last_peered":"2024-03-30T15:14:42.969405+0000","last_clean":"2024-03-30T15:14:42.969405+0000","last_became_active":"2024-03-30T15:13:19.281726+0000","last_became_peered":"2024-03-30T15:13:19.281726+0000","last_unstale":"2024-03-30T15:14:42.969405+0000","last_undegraded":"2024-03-30T15:14:42.969405+0000","last_fullsized":"2024-03-30T15:14:42.969405+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_clean_scrub_stamp":"2024-03-30T15:14:25.522636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:49.743677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"909'32","reported_seq":1787,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.305509+0000","last_change":"2024-03-30T15:14:33.413179+0000","last_active":"2024-03-30T15:14:45.305509+0000","last_peered":"2024-03-30T15:14:45.305509+0000","last_clean":"2024-03-30T15:14:45.305509+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:14:45.305509+0000","last_undegraded":"2024-03-30T15:14:45.305509+0000","last_fullsized":"2024-03-30T15:14:45.305509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'24","last_scrub_stamp":"2024-03-30T15:14:33.413037+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:14:33.413037+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:44.439782+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1197,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614013+0000","last_change":"2024-03-30T15:13:28.631279+0000","last_active":"2024-03-30T15:14:49.614013+0000","last_peered":"2024-03-30T15:14:49.614013+0000","last_clean":"2024-03-30T15:14:49.614013+0000","last_became_active":"2024-03-30T15:13:28.627893+0000","last_became_peered":"2024-03-30T15:13:28.627893+0000","last_unstale":"2024-03-30T15:14:49.614013+0000","last_undegraded":"2024-03-30T15:14:49.614013+0000","last_fullsized":"2024-03-30T15:14:49.614013+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:08:00.523668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:08:00.523668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1318,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615345+0000","last_change":"2024-03-30T15:14:31.203592+0000","last_active":"2024-03-30T15:14:49.615345+0000","last_peered":"2024-03-30T15:14:49.615345+0000","last_clean":"2024-03-30T15:14:49.615345+0000","last_became_active":"2024-03-30T15:13:52.594341+0000","last_became_peered":"2024-03-30T15:13:52.594341+0000","last_unstale":"2024-03-30T15:14:49.615345+0000","last_undegraded":"2024-03-30T15:14:49.615345+0000","last_fullsized":"2024-03-30T15:14:49.615345+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:31.203492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:14:31.203492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:38.217512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1751,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.597716+0000","last_change":"2024-03-30T15:13:44.520601+0000","last_active":"2024-03-30T15:14:42.597716+0000","last_peered":"2024-03-30T15:14:42.597716+0000","last_clean":"2024-03-30T15:14:42.597716+0000","last_became_active":"2024-03-30T15:13:03.517348+0000","last_became_peered":"2024-03-30T15:13:03.517348+0000","last_unstale":"2024-03-30T15:14:42.597716+0000","last_undegraded":"2024-03-30T15:14:42.597716+0000","last_fullsized":"2024-03-30T15:14:42.597716+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:44.520521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:13:44.520521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:59.701443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1668,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969141+0000","last_change":"2024-03-30T15:14:09.727891+0000","last_active":"2024-03-30T15:14:42.969141+0000","last_peered":"2024-03-30T15:14:42.969141+0000","last_clean":"2024-03-30T15:14:42.969141+0000","last_became_active":"2024-03-30T15:14:09.727015+0000","last_became_peered":"2024-03-30T15:14:09.727015+0000","last_unstale":"2024-03-30T15:14:42.969141+0000","last_undegraded":"2024-03-30T15:14:42.969141+0000","last_fullsized":"2024-03-30T15:14:42.969141+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:45.725848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:11:45.725848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.370971+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"909'35","reported_seq":1371,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614934+0000","last_change":"2024-03-30T15:14:00.656230+0000","last_active":"2024-03-30T15:14:49.614934+0000","last_peered":"2024-03-30T15:14:49.614934+0000","last_clean":"2024-03-30T15:14:49.614934+0000","last_became_active":"2024-03-30T15:14:00.638539+0000","last_became_peered":"2024-03-30T15:14:00.638539+0000","last_unstale":"2024-03-30T15:14:49.614934+0000","last_undegraded":"2024-03-30T15:14:49.614934+0000","last_fullsized":"2024-03-30T15:14:49.614934+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'25","last_scrub_stamp":"2024-03-30T15:11:47.028449+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:11:47.028449+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1776,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595052+0000","last_change":"2024-03-30T15:14:34.266129+0000","last_active":"2024-03-30T15:14:42.595052+0000","last_peered":"2024-03-30T15:14:42.595052+0000","last_clean":"2024-03-30T15:14:42.595052+0000","last_became_active":"2024-03-30T15:14:09.726865+0000","last_became_peered":"2024-03-30T15:14:09.726865+0000","last_unstale":"2024-03-30T15:14:42.595052+0000","last_undegraded":"2024-03-30T15:14:42.595052+0000","last_fullsized":"2024-03-30T15:14:42.595052+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:14:34.266038+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:14:34.266038+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:34.734977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"883'15","reported_seq":2027,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969330+0000","last_change":"2024-03-30T15:14:09.732603+0000","last_active":"2024-03-30T15:14:42.969330+0000","last_peered":"2024-03-30T15:14:42.969330+0000","last_clean":"2024-03-30T15:14:42.969330+0000","last_became_active":"2024-03-30T15:14:09.727307+0000","last_became_peered":"2024-03-30T15:14:09.727307+0000","last_unstale":"2024-03-30T15:14:42.969330+0000","last_undegraded":"2024-03-30T15:14:42.969330+0000","last_fullsized":"2024-03-30T15:14:42.969330+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'8","last_scrub_stamp":"2024-03-30T15:11:02.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:11:02.788609+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.370918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1735,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952906+0000","last_change":"2024-03-30T15:13:41.111641+0000","last_active":"2024-03-30T15:14:43.952906+0000","last_peered":"2024-03-30T15:14:43.952906+0000","last_clean":"2024-03-30T15:14:43.952906+0000","last_became_active":"2024-03-30T15:13:19.271296+0000","last_became_peered":"2024-03-30T15:13:19.271296+0000","last_unstale":"2024-03-30T15:14:43.952906+0000","last_undegraded":"2024-03-30T15:14:43.952906+0000","last_fullsized":"2024-03-30T15:14:43.952906+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:13:41.111529+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T15:13:41.111529+0000","last_clean_scrub_stamp":"2024-03-30T15:13:41.111529+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:58.789083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1663,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969001+0000","last_change":"2024-03-30T15:13:52.588733+0000","last_active":"2024-03-30T15:14:42.969001+0000","last_peered":"2024-03-30T15:14:42.969001+0000","last_clean":"2024-03-30T15:14:42.969001+0000","last_became_active":"2024-03-30T15:13:52.588042+0000","last_became_peered":"2024-03-30T15:13:52.588042+0000","last_unstale":"2024-03-30T15:14:42.969001+0000","last_undegraded":"2024-03-30T15:14:42.969001+0000","last_fullsized":"2024-03-30T15:14:42.969001+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:02.667523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:02.667523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:48.371086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"909'24","reported_seq":1715,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:45.311982+0000","last_change":"2024-03-30T15:13:42.509798+0000","last_active":"2024-03-30T15:14:45.311982+0000","last_peered":"2024-03-30T15:14:45.311982+0000","last_clean":"2024-03-30T15:14:45.311982+0000","last_became_active":"2024-03-30T15:13:03.516406+0000","last_became_peered":"2024-03-30T15:13:03.516406+0000","last_unstale":"2024-03-30T15:14:45.311982+0000","last_undegraded":"2024-03-30T15:14:45.311982+0000","last_fullsized":"2024-03-30T15:14:45.311982+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'18","last_scrub_stamp":"2024-03-30T15:13:42.509704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.509704+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.374132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1645,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841263+0000","last_change":"2024-03-30T15:13:33.729411+0000","last_active":"2024-03-30T15:14:49.841263+0000","last_peered":"2024-03-30T15:14:49.841263+0000","last_clean":"2024-03-30T15:14:49.841263+0000","last_became_active":"2024-03-30T15:13:19.285258+0000","last_became_peered":"2024-03-30T15:13:19.285258+0000","last_unstale":"2024-03-30T15:14:49.841263+0000","last_undegraded":"2024-03-30T15:14:49.841263+0000","last_fullsized":"2024-03-30T15:14:49.841263+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:33.729289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_clean_scrub_stamp":"2024-03-30T15:13:33.729289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.358577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1637,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841393+0000","last_change":"2024-03-30T15:13:26.857132+0000","last_active":"2024-03-30T15:14:49.841393+0000","last_peered":"2024-03-30T15:14:49.841393+0000","last_clean":"2024-03-30T15:14:49.841393+0000","last_became_active":"2024-03-30T15:13:19.291186+0000","last_became_peered":"2024-03-30T15:13:19.291186+0000","last_unstale":"2024-03-30T15:14:49.841393+0000","last_undegraded":"2024-03-30T15:14:49.841393+0000","last_fullsized":"2024-03-30T15:14:49.841393+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:26.857055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:26.857055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1645,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969496+0000","last_change":"2024-03-30T15:12:50.677660+0000","last_active":"2024-03-30T15:14:42.969496+0000","last_peered":"2024-03-30T15:14:42.969496+0000","last_clean":"2024-03-30T15:14:42.969496+0000","last_became_active":"2024-03-30T14:57:56.267168+0000","last_became_peered":"2024-03-30T14:57:56.267168+0000","last_unstale":"2024-03-30T15:14:42.969496+0000","last_undegraded":"2024-03-30T15:14:42.969496+0000","last_fullsized":"2024-03-30T15:14:42.969496+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:50.677579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:12:50.677579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1646,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596766+0000","last_change":"2024-03-30T15:14:00.638500+0000","last_active":"2024-03-30T15:14:42.596766+0000","last_peered":"2024-03-30T15:14:42.596766+0000","last_clean":"2024-03-30T15:14:42.596766+0000","last_became_active":"2024-03-30T15:14:00.638153+0000","last_became_peered":"2024-03-30T15:14:00.638153+0000","last_unstale":"2024-03-30T15:14:42.596766+0000","last_undegraded":"2024-03-30T15:14:42.596766+0000","last_fullsized":"2024-03-30T15:14:42.596766+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:01.111602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:01.111602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"909'46","reported_seq":128,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353879+0000","last_change":"2024-03-30T15:13:52.602453+0000","last_active":"2024-03-30T15:14:50.353879+0000","last_peered":"2024-03-30T15:14:50.353879+0000","last_clean":"2024-03-30T15:14:50.353879+0000","last_became_active":"2024-03-30T15:13:52.587538+0000","last_became_peered":"2024-03-30T15:13:52.587538+0000","last_unstale":"2024-03-30T15:14:50.353879+0000","last_undegraded":"2024-03-30T15:14:50.353879+0000","last_fullsized":"2024-03-30T15:14:50.353879+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:37.497100+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:13:37.497100+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:03.512478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1319,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614596+0000","last_change":"2024-03-30T15:13:43.145133+0000","last_active":"2024-03-30T15:14:49.614596+0000","last_peered":"2024-03-30T15:14:49.614596+0000","last_clean":"2024-03-30T15:14:49.614596+0000","last_became_active":"2024-03-30T15:13:28.643964+0000","last_became_peered":"2024-03-30T15:13:28.643964+0000","last_unstale":"2024-03-30T15:14:49.614596+0000","last_undegraded":"2024-03-30T15:14:49.614596+0000","last_fullsized":"2024-03-30T15:14:49.614596+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:43.145050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:13:43.145050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:55.720473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1774,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.597583+0000","last_change":"2024-03-30T15:14:20.084320+0000","last_active":"2024-03-30T15:14:42.597583+0000","last_peered":"2024-03-30T15:14:42.597583+0000","last_clean":"2024-03-30T15:14:42.597583+0000","last_became_active":"2024-03-30T15:14:09.724748+0000","last_became_peered":"2024-03-30T15:14:09.724748+0000","last_unstale":"2024-03-30T15:14:42.597583+0000","last_undegraded":"2024-03-30T15:14:42.597583+0000","last_fullsized":"2024-03-30T15:14:42.597583+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:20.084242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:14:20.084242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:29.241716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1659,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841772+0000","last_change":"2024-03-30T15:13:03.545413+0000","last_active":"2024-03-30T15:14:49.841772+0000","last_peered":"2024-03-30T15:14:49.841772+0000","last_clean":"2024-03-30T15:14:49.841772+0000","last_became_active":"2024-03-30T15:13:03.536453+0000","last_became_peered":"2024-03-30T15:13:03.536453+0000","last_unstale":"2024-03-30T15:14:49.841772+0000","last_undegraded":"2024-03-30T15:14:49.841772+0000","last_fullsized":"2024-03-30T15:14:49.841772+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:33.910931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:11:33.910931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1653,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595246+0000","last_change":"2024-03-30T15:13:34.474161+0000","last_active":"2024-03-30T15:14:42.595246+0000","last_peered":"2024-03-30T15:14:42.595246+0000","last_clean":"2024-03-30T15:14:42.595246+0000","last_became_active":"2024-03-30T15:13:28.632549+0000","last_became_peered":"2024-03-30T15:13:28.632549+0000","last_unstale":"2024-03-30T15:14:42.595246+0000","last_undegraded":"2024-03-30T15:14:42.595246+0000","last_fullsized":"2024-03-30T15:14:42.595246+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:34.474067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:13:34.474067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:44.521823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"909'9","reported_seq":1715,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841112+0000","last_change":"2024-03-30T15:13:03.510230+0000","last_active":"2024-03-30T15:14:49.841112+0000","last_peered":"2024-03-30T15:14:49.841112+0000","last_clean":"2024-03-30T15:14:49.841112+0000","last_became_active":"2024-03-30T15:13:03.509107+0000","last_became_peered":"2024-03-30T15:13:03.509107+0000","last_unstale":"2024-03-30T15:14:49.841112+0000","last_undegraded":"2024-03-30T15:14:49.841112+0000","last_fullsized":"2024-03-30T15:14:49.841112+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'7","last_scrub_stamp":"2024-03-30T15:11:41.878570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:11:41.878570+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1745,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596340+0000","last_change":"2024-03-30T15:14:21.851335+0000","last_active":"2024-03-30T15:14:42.596340+0000","last_peered":"2024-03-30T15:14:42.596340+0000","last_clean":"2024-03-30T15:14:42.596340+0000","last_became_active":"2024-03-30T15:13:03.518001+0000","last_became_peered":"2024-03-30T15:13:03.518001+0000","last_unstale":"2024-03-30T15:14:42.596340+0000","last_undegraded":"2024-03-30T15:14:42.596340+0000","last_fullsized":"2024-03-30T15:14:42.596340+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:21.851223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:14:21.851223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:47.274601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1729,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.952769+0000","last_change":"2024-03-30T15:13:03.529355+0000","last_active":"2024-03-30T15:14:43.952769+0000","last_peered":"2024-03-30T15:14:43.952769+0000","last_clean":"2024-03-30T15:14:43.952769+0000","last_became_active":"2024-03-30T15:13:03.526657+0000","last_became_peered":"2024-03-30T15:13:03.526657+0000","last_unstale":"2024-03-30T15:14:43.952769+0000","last_undegraded":"2024-03-30T15:14:43.952769+0000","last_fullsized":"2024-03-30T15:14:43.952769+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:10:54.102039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:10:54.102039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1740,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595522+0000","last_change":"2024-03-30T15:14:00.638502+0000","last_active":"2024-03-30T15:14:42.595522+0000","last_peered":"2024-03-30T15:14:42.595522+0000","last_clean":"2024-03-30T15:14:42.595522+0000","last_became_active":"2024-03-30T15:14:00.638063+0000","last_became_peered":"2024-03-30T15:14:00.638063+0000","last_unstale":"2024-03-30T15:14:42.595522+0000","last_undegraded":"2024-03-30T15:14:42.595522+0000","last_fullsized":"2024-03-30T15:14:42.595522+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:12:17.973865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:12:17.973865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1646,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594823+0000","last_change":"2024-03-30T15:14:26.138759+0000","last_active":"2024-03-30T15:14:42.594823+0000","last_peered":"2024-03-30T15:14:42.594823+0000","last_clean":"2024-03-30T15:14:42.594823+0000","last_became_active":"2024-03-30T14:57:56.283023+0000","last_became_peered":"2024-03-30T14:57:56.283023+0000","last_unstale":"2024-03-30T15:14:42.594823+0000","last_undegraded":"2024-03-30T15:14:42.594823+0000","last_fullsized":"2024-03-30T15:14:42.594823+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_clean_scrub_stamp":"2024-03-30T15:14:26.138647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:46.059280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"909'21","reported_seq":87,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840356+0000","last_change":"2024-03-30T15:14:09.736900+0000","last_active":"2024-03-30T15:14:49.840356+0000","last_peered":"2024-03-30T15:14:49.840356+0000","last_clean":"2024-03-30T15:14:49.840356+0000","last_became_active":"2024-03-30T15:14:09.725322+0000","last_became_peered":"2024-03-30T15:14:09.725322+0000","last_unstale":"2024-03-30T15:14:49.840356+0000","last_undegraded":"2024-03-30T15:14:49.840356+0000","last_fullsized":"2024-03-30T15:14:49.840356+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:26.727540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:26.727540+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1737,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.596009+0000","last_change":"2024-03-30T15:13:03.533616+0000","last_active":"2024-03-30T15:14:42.596009+0000","last_peered":"2024-03-30T15:14:42.596009+0000","last_clean":"2024-03-30T15:14:42.596009+0000","last_became_active":"2024-03-30T15:13:03.532804+0000","last_became_peered":"2024-03-30T15:13:03.532804+0000","last_unstale":"2024-03-30T15:14:42.596009+0000","last_undegraded":"2024-03-30T15:14:42.596009+0000","last_fullsized":"2024-03-30T15:14:42.596009+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:42.869533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:03:22.212713+0000","last_clean_scrub_stamp":"2024-03-30T15:11:42.869533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1729,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595506+0000","last_change":"2024-03-30T15:13:39.898458+0000","last_active":"2024-03-30T15:14:42.595506+0000","last_peered":"2024-03-30T15:14:42.595506+0000","last_clean":"2024-03-30T15:14:42.595506+0000","last_became_active":"2024-03-30T15:13:28.624284+0000","last_became_peered":"2024-03-30T15:13:28.624284+0000","last_unstale":"2024-03-30T15:14:42.595506+0000","last_undegraded":"2024-03-30T15:14:42.595506+0000","last_fullsized":"2024-03-30T15:14:42.595506+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_clean_scrub_stamp":"2024-03-30T15:13:39.898372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:02.543464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1310,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.614193+0000","last_change":"2024-03-30T15:13:40.224279+0000","last_active":"2024-03-30T15:14:49.614193+0000","last_peered":"2024-03-30T15:14:49.614193+0000","last_clean":"2024-03-30T15:14:49.614193+0000","last_became_active":"2024-03-30T15:13:28.631113+0000","last_became_peered":"2024-03-30T15:13:28.631113+0000","last_unstale":"2024-03-30T15:14:49.614193+0000","last_undegraded":"2024-03-30T15:14:49.614193+0000","last_fullsized":"2024-03-30T15:14:49.614193+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_clean_scrub_stamp":"2024-03-30T15:13:40.224204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1659,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.595250+0000","last_change":"2024-03-30T15:14:09.727337+0000","last_active":"2024-03-30T15:14:42.595250+0000","last_peered":"2024-03-30T15:14:42.595250+0000","last_clean":"2024-03-30T15:14:42.595250+0000","last_became_active":"2024-03-30T15:14:09.726973+0000","last_became_peered":"2024-03-30T15:14:09.726973+0000","last_unstale":"2024-03-30T15:14:42.595250+0000","last_undegraded":"2024-03-30T15:14:42.595250+0000","last_fullsized":"2024-03-30T15:14:42.595250+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:05.038649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:13:05.038649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.945531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"909'46","reported_seq":1609,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:50.353592+0000","last_change":"2024-03-30T15:13:42.244532+0000","last_active":"2024-03-30T15:14:50.353592+0000","last_peered":"2024-03-30T15:14:50.353592+0000","last_clean":"2024-03-30T15:14:50.353592+0000","last_became_active":"2024-03-30T15:13:28.614224+0000","last_became_peered":"2024-03-30T15:13:28.614224+0000","last_unstale":"2024-03-30T15:14:50.353592+0000","last_undegraded":"2024-03-30T15:14:50.353592+0000","last_fullsized":"2024-03-30T15:14:50.353592+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'34","last_scrub_stamp":"2024-03-30T15:13:42.244406+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:13:42.244406+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:52.631648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1647,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.969010+0000","last_change":"2024-03-30T15:13:35.659341+0000","last_active":"2024-03-30T15:14:42.969010+0000","last_peered":"2024-03-30T15:14:42.969010+0000","last_clean":"2024-03-30T15:14:42.969010+0000","last_became_active":"2024-03-30T15:13:19.281642+0000","last_became_peered":"2024-03-30T15:13:19.281642+0000","last_unstale":"2024-03-30T15:14:42.969010+0000","last_undegraded":"2024-03-30T15:14:42.969010+0000","last_fullsized":"2024-03-30T15:14:42.969010+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:35.659115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:13:35.659115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:47.209280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":68,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840300+0000","last_change":"2024-03-30T15:14:09.725445+0000","last_active":"2024-03-30T15:14:49.840300+0000","last_peered":"2024-03-30T15:14:49.840300+0000","last_clean":"2024-03-30T15:14:49.840300+0000","last_became_active":"2024-03-30T15:14:09.725019+0000","last_became_peered":"2024-03-30T15:14:09.725019+0000","last_unstale":"2024-03-30T15:14:49.840300+0000","last_undegraded":"2024-03-30T15:14:49.840300+0000","last_fullsized":"2024-03-30T15:14:49.840300+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:43.743015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:13:43.743015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.009132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":85,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:43.457901+0000","last_change":"2024-03-30T15:14:22.358228+0000","last_active":"2024-03-30T15:14:43.457901+0000","last_peered":"2024-03-30T15:14:43.457901+0000","last_clean":"2024-03-30T15:14:43.457901+0000","last_became_active":"2024-03-30T15:14:00.631826+0000","last_became_peered":"2024-03-30T15:14:00.631826+0000","last_unstale":"2024-03-30T15:14:43.457901+0000","last_undegraded":"2024-03-30T15:14:43.457901+0000","last_fullsized":"2024-03-30T15:14:43.457901+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:22.358146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:14:22.358146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:48.743258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1260,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.613895+0000","last_change":"2024-03-30T15:14:16.147467+0000","last_active":"2024-03-30T15:14:49.613895+0000","last_peered":"2024-03-30T15:14:49.613895+0000","last_clean":"2024-03-30T15:14:49.613895+0000","last_became_active":"2024-03-30T15:13:28.645477+0000","last_became_peered":"2024-03-30T15:13:28.645477+0000","last_unstale":"2024-03-30T15:14:49.613895+0000","last_undegraded":"2024-03-30T15:14:49.613895+0000","last_fullsized":"2024-03-30T15:14:49.613895+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:14:16.147377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:14:16.147377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:15:16.240357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"909'21","reported_seq":1699,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.840670+0000","last_change":"2024-03-30T15:13:36.760503+0000","last_active":"2024-03-30T15:14:49.840670+0000","last_peered":"2024-03-30T15:14:49.840670+0000","last_clean":"2024-03-30T15:14:49.840670+0000","last_became_active":"2024-03-30T15:13:19.279835+0000","last_became_peered":"2024-03-30T15:13:19.279835+0000","last_unstale":"2024-03-30T15:14:49.840670+0000","last_undegraded":"2024-03-30T15:14:49.840670+0000","last_fullsized":"2024-03-30T15:14:49.840670+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'15","last_scrub_stamp":"2024-03-30T15:13:36.760405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:13:36.760405+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1340,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.615729+0000","last_change":"2024-03-30T15:13:52.592354+0000","last_active":"2024-03-30T15:14:49.615729+0000","last_peered":"2024-03-30T15:14:49.615729+0000","last_clean":"2024-03-30T15:14:49.615729+0000","last_became_active":"2024-03-30T15:13:52.592009+0000","last_became_peered":"2024-03-30T15:13:52.592009+0000","last_unstale":"2024-03-30T15:14:49.615729+0000","last_undegraded":"2024-03-30T15:14:49.615729+0000","last_fullsized":"2024-03-30T15:14:49.615729+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:07:35.570970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:07:35.570970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1738,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-30T15:14:42.594724+0000","last_change":"2024-03-30T15:13:03.420409+0000","last_active":"2024-03-30T15:14:42.594724+0000","last_peered":"2024-03-30T15:14:42.594724+0000","last_clean":"2024-03-30T15:14:42.594724+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:14:42.594724+0000","last_undegraded":"2024-03-30T15:14:42.594724+0000","last_fullsized":"2024-03-30T15:14:42.594724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:13:03.420327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:13:03.420327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:43.193826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1661,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T15:14:49.841118+0000","last_change":"2024-03-30T15:13:03.529442+0000","last_active":"2024-03-30T15:14:49.841118+0000","last_peered":"2024-03-30T15:14:49.841118+0000","last_clean":"2024-03-30T15:14:49.841118+0000","last_became_active":"2024-03-30T15:13:03.529100+0000","last_became_peered":"2024-03-30T15:13:03.529100+0000","last_unstale":"2024-03-30T15:14:49.841118+0000","last_undegraded":"2024-03-30T15:14:49.841118+0000","last_fullsized":"2024-03-30T15:14:49.841118+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:11:35.859491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:11:35.859491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:14:50.820202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1199,"num_read_kb":1194,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1538,"num_read_kb":1295,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":2008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005303,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106960,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93638192,"statfs":{"total":95995035648,"available":95885508608,"internally_reserved":0,"allocated":7938048,"data_stored":4150265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299644,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110768,"kb_used_data":7400,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93634384,"statfs":{"total":95995035648,"available":95881609216,"internally_reserved":0,"allocated":7577600,"data_stored":3820247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575127,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106644,"kb_used_data":7436,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93638508,"statfs":{"total":95995035648,"available":95885832192,"internally_reserved":0,"allocated":7614464,"data_stored":3821449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":850,"seq":3650722201618,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40220,"kb_used_data":7200,"kb_used_omap":17,"kb_used_meta":32750,"kb_avail":93704932,"statfs":{"total":95995035648,"available":95953850368,"internally_reserved":0,"allocated":7372800,"data_stored":3690159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17527,"internal_metadata":33536905},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":840,"seq":3607772528660,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40436,"kb_used_data":7672,"kb_used_omap":46,"kb_used_meta":32465,"kb_avail":93704716,"statfs":{"total":95995035648,"available":95953629184,"internally_reserved":0,"allocated":7856128,"data_stored":4091026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47515,"internal_metadata":33244773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117260,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111396,"kb_used_data":8028,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633756,"statfs":{"total":95995035648,"available":95880966144,"internally_reserved":0,"allocated":8220672,"data_stored":4412792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":832,"seq":3573412790295,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40092,"kb_used_data":7200,"kb_used_omap":37,"kb_used_meta":32602,"kb_avail":93705060,"statfs":{"total":95995035648,"available":95953981440,"internally_reserved":0,"allocated":7372800,"data_stored":3689480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38119,"internal_metadata":33385241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659393,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111844,"kb_used_data":8236,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93633308,"statfs":{"total":95995035648,"available":95880507392,"internally_reserved":0,"allocated":8433664,"data_stored":4548652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:14:54.592 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-30T15:14:54.592 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T15:14:54.593 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-30T15:14:54.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-03-30T15:14:54.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[41614]: osdmap e912: 8 total, 8 up, 8 in 2024-03-30T15:14:54.808 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[41614]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[29876]: osdmap e912: 8 total, 8 up, 8 in 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[29876]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:14:54.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:54 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:14:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:54 smithi134 ceph-mon[33714]: osdmap e912: 8 total, 8 up, 8 in 2024-03-30T15:14:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:54 smithi134 ceph-mon[33714]: pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-03-30T15:14:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:54 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:14:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:54 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.753+0000 7f530fde5700 1 -- 172.21.15.28:0/526805693 >> v1:172.21.15.28:6789/0 conn(0x7f530810d960 legacy=0x7f530810fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.753+0000 7f530fde5700 1 -- 172.21.15.28:0/526805693 shutdown_connections 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.753+0000 7f530fde5700 1 -- 172.21.15.28:0/526805693 >> 172.21.15.28:0/526805693 conn(0x7f53080fc9e0 msgr2=0x7f53080fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.753+0000 7f530fde5700 1 -- 172.21.15.28:0/526805693 shutdown_connections 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.753+0000 7f530fde5700 1 -- 172.21.15.28:0/526805693 wait complete. 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 librados: starting msgr at 2024-03-30T15:14:55.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 librados: starting objecter 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 Processor -- start 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 -- start start 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 librados: setting wanted keys 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 librados: calling monclient init 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53081a93d0 con 0x7f530810d960 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53081aa5f0 con 0x7f5308109dc0 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530fde5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f53081ab830 con 0x7f5308101130 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530db81700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f5308101130 0x7f5308106460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:47286/0 (socket says 172.21.15.28:47286) 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530e382700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f530810d960 0x7f53081a7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37490/0 (socket says 172.21.15.28:37490) 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f530db81700 1 -- 172.21.15.28:0/3993913350 learned_addr learned my addr 172.21.15.28:0/3993913350 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f52faffd700 10 client.?.objecter ms_handle_connect 0x7f5308101130 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f52faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f52faffd700 10 client.?.objecter ms_handle_connect 0x7f530810d960 2024-03-30T15:14:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.754+0000 7f52faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 10 client.?.objecter ms_handle_connect 0x7f5308109dc0 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3802828000 0 0) 0x7f53081ab830 con 0x7f5308101130 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f40031f0 con 0x7f5308101130 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1465114832 0 0) 0x7f53081a93d0 con 0x7f530810d960 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53081ab830 con 0x7f530810d960 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393794944 0 0) 0x7f53081aa5f0 con 0x7f5308109dc0 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53081a93d0 con 0x7f5308109dc0 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 695697961 0 0) 0x7f53081ab830 con 0x7f530810d960 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53081aa5f0 con 0x7f530810d960 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2108777898 0 0) 0x7f52f40031f0 con 0x7f5308101130 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53081ab830 con 0x7f5308101130 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5300004730 con 0x7f530810d960 2024-03-30T15:14:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5304001930 con 0x7f5308101130 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3532877073 0 0) 0x7f53081a93d0 con 0x7f5308109dc0 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52f40031f0 con 0x7f5308109dc0 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f52fc001960 con 0x7f5308109dc0 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1929337773 0 0) 0x7f53081aa5f0 con 0x7f530810d960 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 >> v1:172.21.15.28:6790/0 conn(0x7f5308101130 legacy=0x7f5308106460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 >> v1:172.21.15.134:6789/0 conn(0x7f5308109dc0 legacy=0x7f53081a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53081aca70 con 0x7f530810d960 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53081a95a0 con 0x7f530810d960 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f530fde5700 10 client.26007.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53081a9a60 con 0x7f530810d960 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f530fde5700 1 librados: init done 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.755+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5300004a40 con 0x7f530810d960 2024-03-30T15:14:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.756+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5300004ee0 con 0x7f530810d960 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.756+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec005120 con 0x7f530810d960 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.757+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5300003a90 con 0x7f530810d960 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(912..912 src has 654..912) ==== 7561+0+0 (unknown 1969578904 0 0) 0x7f5300089640 con 0x7f530810d960 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 10 client.26007.objecter ms_dispatch 0x7f5308001550 osd_map(912..912 src has 654..912) 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 3 client.26007.objecter handle_osd_map got epochs [912,912] > 0 2024-03-30T15:14:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 3 client.26007.objecter handle_osd_map decoding full epoch 912 2024-03-30T15:14:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 20 client.26007.objecter dump_active .. 0 homeless 2024-03-30T15:14:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.758+0000 7f52faffd700 10 client.26007.objecter ms_handle_connect 0x7f52f40704f0 2024-03-30T15:14:55.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.762+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5300056af0 con 0x7f530810d960 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:14:55.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:14:55.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:55 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:14:55.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:55.983+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f52ec005410 con 0x7f530810d960 2024-03-30T15:14:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:14:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:14:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T15:14:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T15:14:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:14:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:14:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:14:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:55 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:14:56.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.768+0000 7f52faffd700 1 -- 172.21.15.28:0/3993913350 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v913) ==== 101+0+0 (unknown 3981496957 0 0) 0x7f530005a540 con 0x7f530810d960 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 10 librados: watch_flush enter 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 10 librados: watch_flush exit 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 20 client.26007.objecter shutdown clearing up homeless session... 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 10 client.26007.objecter successfully canceled tick 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 >> v1:172.21.15.28:6800/3011431093 conn(0x7f52f40704f0 legacy=0x7f52f40729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 >> v1:172.21.15.28:6789/0 conn(0x7f530810d960 legacy=0x7f53081a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 shutdown_connections 2024-03-30T15:14:56.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.773+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 >> 172.21.15.28:0/3993913350 conn(0x7f53080fc9e0 msgr2=0x7f53081106b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:14:56.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.774+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 shutdown_connections 2024-03-30T15:14:56.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.775+0000 7f530fde5700 1 -- 172.21.15.28:0/3993913350 wait complete. 2024-03-30T15:14:56.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:14:56.775+0000 7f530fde5700 1 librados: shutdown 2024-03-30T15:14:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-03-30T15:14:56.906 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:14:56.906 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:14:56.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T15:14:56.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T15:14:56.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 1.9 scrub starts 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: 1.9 scrub ok 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[29876]: pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 1.9 scrub starts 2024-03-30T15:14:56.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: 1.9 scrub ok 2024-03-30T15:14:56.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:56 smithi028 ceph-mon[41614]: pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-30T15:14:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T15:14:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T15:14:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:14:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:14:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-30T15:14:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 1.9 scrub starts 2024-03-30T15:14:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: 1.9 scrub ok 2024-03-30T15:14:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:56 smithi134 ceph-mon[33714]: pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 774 B/s rd, 0 op/s 2024-03-30T15:14:58.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T15:14:58.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T15:14:58.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: osdmap e913: 8 total, 8 up, 8 in 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 3.d scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[29876]: 3.d scrub ok 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:14:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:14:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: osdmap e913: 8 total, 8 up, 8 in 2024-03-30T15:14:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 3.d scrub starts 2024-03-30T15:14:58.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:57 smithi028 ceph-mon[41614]: 3.d scrub ok 2024-03-30T15:14:58.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:14:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3993913350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-30T15:14:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: osdmap e913: 8 total, 8 up, 8 in 2024-03-30T15:14:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 3.d scrub starts 2024-03-30T15:14:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:57 smithi134 ceph-mon[33714]: 3.d scrub ok 2024-03-30T15:14:58.185 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:58.458 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:14:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:14:58.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[29876]: pgmap v1420: 174 pgs: 174 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:14:59.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:58 smithi028 ceph-mon[41614]: pgmap v1420: 174 pgs: 174 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:14:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:58 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:14:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:58 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:14:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:58 smithi134 ceph-mon[33714]: pgmap v1420: 174 pgs: 174 active+clean; 582 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:00.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:59 smithi028 ceph-mon[41614]: 12.f deep-scrub starts 2024-03-30T15:15:00.167 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:14:59 smithi028 ceph-mon[41614]: 12.f deep-scrub ok 2024-03-30T15:15:00.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:59 smithi028 ceph-mon[29876]: 12.f deep-scrub starts 2024-03-30T15:15:00.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:14:59 smithi028 ceph-mon[29876]: 12.f deep-scrub ok 2024-03-30T15:15:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:59 smithi134 ceph-mon[33714]: 12.f deep-scrub starts 2024-03-30T15:15:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:14:59 smithi134 ceph-mon[33714]: 12.f deep-scrub ok 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[41614]: 1.6 scrub starts 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[41614]: 1.6 scrub ok 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[41614]: 6.1a deep-scrub starts 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[41614]: 6.1a deep-scrub ok 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[41614]: pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[29876]: 1.6 scrub starts 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[29876]: 1.6 scrub ok 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[29876]: 6.1a deep-scrub starts 2024-03-30T15:15:01.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[29876]: 6.1a deep-scrub ok 2024-03-30T15:15:01.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:00 smithi028 ceph-mon[29876]: pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-mon[33714]: 1.6 scrub starts 2024-03-30T15:15:01.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-mon[33714]: 1.6 scrub ok 2024-03-30T15:15:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-mon[33714]: 6.1a deep-scrub starts 2024-03-30T15:15:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-mon[33714]: 6.1a deep-scrub ok 2024-03-30T15:15:01.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-mon[33714]: pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:01.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 -- 172.21.15.28:0/462999858 >> v1:172.21.15.28:6790/0 conn(0x7fdf44100a10 legacy=0x7fdf44100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 -- 172.21.15.28:0/462999858 shutdown_connections 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 -- 172.21.15.28:0/462999858 >> 172.21.15.28:0/462999858 conn(0x7fdf44076230 msgr2=0x7fdf44076640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 -- 172.21.15.28:0/462999858 shutdown_connections 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 -- 172.21.15.28:0/462999858 wait complete. 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 librados: starting msgr at 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.100+0000 7fdf49646700 1 librados: starting objecter 2024-03-30T15:15:01.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 Processor -- start 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 -- start start 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 librados: setting wanted keys 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 librados: calling monclient init 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf441a70a0 con 0x7fdf4410eb10 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf441a82c0 con 0x7fdf44100a10 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf49646700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdf441a9500 con 0x7fdf4410afa0 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf427fc700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fdf4410afa0 0x7fdf441a1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:55034/0 (socket says 172.21.15.28:55034) 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf427fc700 1 -- 172.21.15.28:0/744559867 learned_addr learned my addr 172.21.15.28:0/744559867 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter ms_handle_connect 0x7fdf4410afa0 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter ms_handle_connect 0x7fdf4410eb10 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter ms_handle_connect 0x7fdf44100a10 2024-03-30T15:15:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582426886 0 0) 0x7fdf441a9500 con 0x7fdf4410afa0 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf2c0031f0 con 0x7fdf4410afa0 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169163419 0 0) 0x7fdf441a70a0 con 0x7fdf4410eb10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf441a9500 con 0x7fdf4410eb10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689071901 0 0) 0x7fdf441a82c0 con 0x7fdf44100a10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.101+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf441a70a0 con 0x7fdf44100a10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2814916954 0 0) 0x7fdf2c0031f0 con 0x7fdf4410afa0 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf441a82c0 con 0x7fdf4410afa0 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf38001890 con 0x7fdf4410afa0 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 660847363 0 0) 0x7fdf441a9500 con 0x7fdf4410eb10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf2c0031f0 con 0x7fdf4410eb10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3406755899 0 0) 0x7fdf441a70a0 con 0x7fdf44100a10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf441a9500 con 0x7fdf44100a10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf3c0017f0 con 0x7fdf4410eb10 2024-03-30T15:15:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf34004880 con 0x7fdf44100a10 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3829722799 0 0) 0x7fdf441a82c0 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 >> v1:172.21.15.134:6789/0 conn(0x7fdf44100a10 legacy=0x7fdf441073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 >> v1:172.21.15.28:6789/0 conn(0x7fdf4410eb10 legacy=0x7fdf441a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf441aa740 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf441a72d0 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf49646700 10 client.59216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf441a7810 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf49646700 1 librados: init done 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fdf38003b20 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.102+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fdf38005c60 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.103+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf30005120 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.103+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fdf3801c4a0 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(913..913 src has 842..913) ==== 7561+0+0 (unknown 2231888928 0 0) 0x7fdf380895c0 con 0x7fdf4410afa0 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 10 client.59216.objecter ms_dispatch 0x7fdf44001550 osd_map(913..913 src has 842..913) 2024-03-30T15:15:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 3 client.59216.objecter handle_osd_map got epochs [913,913] > 0 2024-03-30T15:15:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 3 client.59216.objecter handle_osd_map decoding full epoch 913 2024-03-30T15:15:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 20 client.59216.objecter dump_active .. 0 homeless 2024-03-30T15:15:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.104+0000 7fdf2bfff700 10 client.59216.objecter ms_handle_connect 0x7fdf2c070530 2024-03-30T15:15:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.109+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdf38056a10 con 0x7fdf4410afa0 2024-03-30T15:15:01.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.316+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fdf30005410 con 0x7fdf4410afa0 2024-03-30T15:15:01.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.318+0000 7fdf2bfff700 1 -- 172.21.15.28:0/744559867 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v913) ==== 81+0+47827 (unknown 2386695212 0 1419491598) 0x7fdf3805a460 con 0x7fdf4410afa0 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 913, 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8345fd88-eea4-11ee-b647-cb9ed24678a4", 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-03-30T14:48:49.804304+0000", 2024-03-30T15:15:01.321 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-03-30T15:14:56.764968+0000", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-03-30T15:13:27.584413+0000", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-03-30T14:54:05.032145+0000", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5799936, 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 121, 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-30T15:15:01.322 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 345, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:52:17.629184+0000", 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-03-30T15:15:01.324 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "562", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "559", 2024-03-30T15:15:01.325 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.326 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:15:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.9100000858306885, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.9100000858306885, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.329 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.9100000858306885, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.9100000858306885, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:34.798924+0000", 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.330 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-03-30T15:15:01.332 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 72, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.333 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.334 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-30T15:15:01.335 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:35.563381+0000", 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.336 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.337 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.338 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "70", 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.339 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.340 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.342 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-03-30T15:15:01.343 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:37.775241+0000", 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:15:01.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "72", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.346 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.348 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.349 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:39.594196+0000", 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.350 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.351 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "74", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.354 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.25, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.25, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.25, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.25, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.355 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:54:41.660853+0000", 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.357 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "76", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.358 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.359 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-03-30T15:15:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.361 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 12, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-03-30T14:57:46.063758+0000", 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-03-30T15:15:01.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "117", 2024-03-30T15:15:01.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-03-30T15:15:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-03-30T15:15:01.365 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": {}, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-03-30T15:15:01.366 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.89145469665527344, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.88999998569488525, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "eb350905-2b09-408a-9adc-4f2af2766479", 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.367 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 8, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 668, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 850, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 669, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2927075629 2024-03-30T15:15:01.368 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2927075629 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2927075629 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2927075629 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/2927075629", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/2927075629", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/2927075629", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/2927075629", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76bfea8d-68c7-448e-bb99-d3d003567520", 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.371 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:15:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.374 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1850313547 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1850313547", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1850313547", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1850313547", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1850313547", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ad24917c-f5cb-4ca8-8a52-91d2256ff21c", 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 18, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 779, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 840, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 780, 2024-03-30T15:15:01.376 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2294873818 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-03-30T15:15:01.377 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2294873818 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2294873818 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2294873818 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/2294873818", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/2294873818", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/2294873818", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/2294873818", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "4305caf1-0023-438f-8520-87814130b457", 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.380 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.1316375732421875, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 27, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:15:01.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1506157250 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1506157250", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1506157250", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1506157250", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1506157250", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "8180ec45-5260-4cdc-855c-be5302b1465c", 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.384 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 32, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 815, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 832, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 816, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6800", 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4077237787 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6801", 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4077237787 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6803", 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4077237787 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6802", 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4077237787 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6800/4077237787", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6801/4077237787", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6803/4077237787", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6802/4077237787", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "265992a1-46bc-4621-b9db-2b5ba1ee5e45", 2024-03-30T15:15:01.388 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 41, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6804", 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6805", 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 10 librados: watch_flush enter 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 10 librados: watch_flush exit 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 20 client.59216.objecter shutdown clearing up homeless session... 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 10 client.59216.objecter successfully canceled tick 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 >> v1:172.21.15.28:6800/3011431093 conn(0x7fdf2c070530 legacy=0x7fdf2c0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 >> v1:172.21.15.28:6790/0 conn(0x7fdf4410afa0 legacy=0x7fdf441a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 shutdown_connections 2024-03-30T15:15:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 >> 172.21.15.28:0/744559867 conn(0x7fdf44076230 msgr2=0x7fdf44112200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 shutdown_connections 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 -- 172.21.15.28:0/744559867 wait complete. 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:01.321+0000 7fdf49646700 1 librados: shutdown 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6807", 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6806", 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3668151179 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6804/3668151179", 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6805/3668151179", 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6807/3668151179", 2024-03-30T15:15:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6806/3668151179", 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d", 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.394 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 52, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6808", 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.395 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6809", 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6811", 2024-03-30T15:15:01.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6810", 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2135224194 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6808/2135224194", 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6809/2135224194", 2024-03-30T15:15:01.397 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6811/2135224194", 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6810/2135224194", 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "76f25ab9-af2f-4f45-afe3-13ece781d29c", 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-03-30T15:15:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 61, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 899, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6812", 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.399 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6813", 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6815", 2024-03-30T15:15:01.400 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.134:6814", 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3758190298 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.134:6812/3758190298", 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.134:6813/3758190298", 2024-03-30T15:15:01.401 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.134:6815/3758190298", 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.134:6814/3758190298", 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.402 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:52.194112+0000", 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 668 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:37.435073+0000", 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:41.612581+0000", 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 779 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.404 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:13:44.577235+0000", 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:18.115280+0000", 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 815 2024-03-30T15:15:01.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:53.429473+0000", 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.406 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:09.401904+0000", 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701547738038271, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-03-30T15:14:02.933284+0000", 2024-03-30T15:15:01.407 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.17", 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.4", 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-03-30T15:15:01.408 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: 3 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.a", 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.18", 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.410 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.d", 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 5, 2024-03-30T15:15:01.411 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 4 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2199995094": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/545344739": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/4222366292": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/406580277": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2354623613": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2838206829": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:15:01.412 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1249955842": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:15:01.413 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/3720192247": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.413 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1446120238": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:15:01.413 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/429160846": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:15:01.413 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/580061692": "2024-03-31T14:55:17.563475+0000", 2024-03-30T15:15:01.413 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/997592180": "2024-03-31T14:49:29.238923+0000", 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/2767265670": "2024-03-31T14:49:57.293695+0000", 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1590506004": "2024-03-31T14:49:57.293695+0000" 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-03-30T15:15:01.414 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [ 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 13, 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 5 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 25, 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 33, 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.416 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 69, 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.417 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 104, 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 112, 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 124, 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 1, 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 143, 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 147, 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 155, 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 157, 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:15:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 165, 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 3 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 168, 2024-03-30T15:15:01.422 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 4 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 176, 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 179, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 188, 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 195, 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 210, 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 6 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 214, 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 7, 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 234, 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 8, 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 246, 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 10, 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 250, 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 11, 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 2 2024-03-30T15:15:01.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 261, 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 14, 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 325, 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 2, 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 326, 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 3, 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 329, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 4, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 333, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "snaps": [ 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "begin": 6, 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: "length": 1 2024-03-30T15:15:01.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-03-30T15:15:01.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-30T15:15:01.433 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-30T15:15:01.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 1.7 scrub starts 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 1.7 scrub ok 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/744559867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T15:15:01.963 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 1.7 scrub starts 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 1.7 scrub ok 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:01.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:01 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/744559867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:15:01.991 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:01.991 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T15:15:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T15:15:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 1.7 scrub starts 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 1.7 scrub ok 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:01 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/744559867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-30T15:15:02.729 INFO:tasks.thrashosds.thrasher:j is {'epoch': 913, 'fsid': '8345fd88-eea4-11ee-b647-cb9ed24678a4', 'created': '2024-03-30T14:48:49.804304+0000', 'modified': '2024-03-30T15:14:56.764968+0000', 'last_up_change': '2024-03-30T15:13:27.584413+0000', 'last_in_change': '2024-03-30T14:54:05.032145+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 121, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 345, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T14:52:17.629184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '562', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '559', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9100000858306885, 'score_stable': 2.9100000858306885, 'optimal_score': 1, 'raw_score_acting': 2.9100000858306885, 'raw_score_stable': 2.9100000858306885, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-30T14:54:34.798924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-30T14:54:35.563381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-30T14:54:37.775241+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-30T14:54:39.594196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-30T14:54:41.660853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-03-30T14:57:46.063758+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8914546966552734, 'average_primary_affinity': 0.8899999856948853, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'eb350905-2b09-408a-9adc-4f2af2766479', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 668, 'up_from': 850, 'up_thru': 899, 'down_at': 669, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 2927075629}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 2927075629}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 2927075629}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 2927075629}]}, 'public_addr': '172.21.15.28:6809/2927075629', 'cluster_addr': '172.21.15.28:6810/2927075629', 'heartbeat_back_addr': '172.21.15.28:6812/2927075629', 'heartbeat_front_addr': '172.21.15.28:6811/2927075629', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '76bfea8d-68c7-448e-bb99-d3d003567520', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 899, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1850313547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1850313547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1850313547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1850313547}]}, 'public_addr': '172.21.15.28:6805/1850313547', 'cluster_addr': '172.21.15.28:6806/1850313547', 'heartbeat_back_addr': '172.21.15.28:6808/1850313547', 'heartbeat_front_addr': '172.21.15.28:6807/1850313547', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'ad24917c-f5cb-4ca8-8a52-91d2256ff21c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 779, 'up_from': 840, 'up_thru': 899, 'down_at': 780, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 2294873818}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 2294873818}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 2294873818}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 2294873818}]}, 'public_addr': '172.21.15.28:6801/2294873818', 'cluster_addr': '172.21.15.28:6802/2294873818', 'heartbeat_back_addr': '172.21.15.28:6804/2294873818', 'heartbeat_front_addr': '172.21.15.28:6803/2294873818', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4305caf1-0023-438f-8520-87814130b457', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1316375732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 899, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1506157250}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1506157250}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1506157250}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1506157250}]}, 'public_addr': '172.21.15.28:6813/1506157250', 'cluster_addr': '172.21.15.28:6814/1506157250', 'heartbeat_back_addr': '172.21.15.28:6816/1506157250', 'heartbeat_front_addr': '172.21.15.28:6815/1506157250', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8180ec45-5260-4cdc-855c-be5302b1465c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 32, 'last_clean_end': 815, 'up_from': 832, 'up_thru': 899, 'down_at': 816, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6800', 'nonce': 4077237787}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6801', 'nonce': 4077237787}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 4077237787}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6802', 'nonce': 4077237787}]}, 'public_addr': '172.21.15.134:6800/4077237787', 'cluster_addr': '172.21.15.134:6801/4077237787', 'heartbeat_back_addr': '172.21.15.134:6803/4077237787', 'heartbeat_front_addr': '172.21.15.134:6802/4077237787', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '265992a1-46bc-4621-b9db-2b5ba1ee5e45', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 899, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6804', 'nonce': 3668151179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 3668151179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 3668151179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6806', 'nonce': 3668151179}]}, 'public_addr': '172.21.15.134:6804/3668151179', 'cluster_addr': '172.21.15.134:6805/3668151179', 'heartbeat_back_addr': '172.21.15.134:6807/3668151179', 'heartbeat_front_addr': '172.21.15.134:6806/3668151179', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 899, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6808', 'nonce': 2135224194}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2135224194}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2135224194}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6810', 'nonce': 2135224194}]}, 'public_addr': '172.21.15.134:6808/2135224194', 'cluster_addr': '172.21.15.134:6809/2135224194', 'heartbeat_back_addr': '172.21.15.134:6811/2135224194', 'heartbeat_front_addr': '172.21.15.134:6810/2135224194', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '76f25ab9-af2f-4f45-afe3-13ece781d29c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 899, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6812', 'nonce': 3758190298}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 3758190298}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 3758190298}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.134:6814', 'nonce': 3758190298}]}, 'public_addr': '172.21.15.134:6812/3758190298', 'cluster_addr': '172.21.15.134:6813/3758190298', 'heartbeat_back_addr': '172.21.15.134:6815/3758190298', 'heartbeat_front_addr': '172.21.15.134:6814/3758190298', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:52.194112+0000', 'dead_epoch': 668}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:37.435073+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:41.612581+0000', 'dead_epoch': 779}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:13:44.577235+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:18.115280+0000', 'dead_epoch': 815}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:53.429473+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:09.401904+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-30T15:14:02.933284+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.17', 'osds': [6, 7, 1]}, {'pgid': '6.4', 'osds': [1, 4, 3]}], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.d', 'mappings': [{'from': 2, 'to': 6}, {'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:0/2199995094': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/545344739': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/4222366292': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/406580277': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/2354623613': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/2838206829': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/1249955842': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:6800/3720192247': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:0/1446120238': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/429160846': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:0/580061692': '2024-03-31T14:55:17.563475+0000', '172.21.15.28:6800/997592180': '2024-03-31T14:49:29.238923+0000', '172.21.15.28:6800/2767265670': '2024-03-31T14:49:57.293695+0000', '172.21.15.28:0/1590506004': '2024-03-31T14:49:57.293695+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 13, 'snaps': [{'begin': 2, 'length': 5}]}, {'pool': 25, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 33, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 69, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 104, 'snaps': [{'begin': 7, 'length': 2}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 124, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 147, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 155, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 165, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 168, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 4, 'length': 2}]}, {'pool': 188, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 3, 'length': 2}]}, {'pool': 210, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 214, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 246, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 250, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 261, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 325, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 326, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 329, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 333, 'snaps': [{'begin': 6, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-30T15:15:02.729 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.a 2024-03-30T15:15:02.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 4.a 2024-03-30T15:15:02.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:15:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:15:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[29876]: pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-30T15:15:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:15:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:15:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:02 smithi028 ceph-mon[41614]: pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-30T15:15:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:02 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:15:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:02 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:15:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:02 smithi134 ceph-mon[33714]: pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-03-30T15:15:03.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-30T15:15:03.853 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:04.027 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:04.085 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 1.5 scrub starts 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 1.5 scrub ok 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 5.9 deep-scrub starts 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[41614]: 5.9 deep-scrub ok 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:15:04.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 1.5 scrub starts 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 1.5 scrub ok 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 5.9 deep-scrub starts 2024-03-30T15:15:04.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:03 smithi028 ceph-mon[29876]: 5.9 deep-scrub ok 2024-03-30T15:15:04.149 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:15:04.149 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:15:04.149 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:15:04.150 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:15:04.150 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 1.5 scrub starts 2024-03-30T15:15:04.150 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 1.5 scrub ok 2024-03-30T15:15:04.150 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 5.9 deep-scrub starts 2024-03-30T15:15:04.150 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:03 smithi134 ceph-mon[33714]: 5.9 deep-scrub ok 2024-03-30T15:15:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T15:15:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T15:15:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:15:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:15:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: osdmap e914: 8 total, 8 up, 8 in 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[29876]: pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T15:15:04.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: osdmap e914: 8 total, 8 up, 8 in 2024-03-30T15:15:04.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:04 smithi028 ceph-mon[41614]: pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: osdmap e914: 8 total, 8 up, 8 in 2024-03-30T15:15:05.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:04 smithi134 ceph-mon[33714]: pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- 172.21.15.28:0/2309408155 >> v1:172.21.15.28:6790/0 conn(0x7fb6c410ea80 legacy=0x7fb6c4110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- 172.21.15.28:0/2309408155 shutdown_connections 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- 172.21.15.28:0/2309408155 >> 172.21.15.28:0/2309408155 conn(0x7fb6c40fdb20 msgr2=0x7fb6c40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- 172.21.15.28:0/2309408155 shutdown_connections 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- 172.21.15.28:0/2309408155 wait complete. 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 librados: starting msgr at 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 librados: starting objecter 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 Processor -- start 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 -- start start 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 librados: setting wanted keys 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.581+0000 7fb6cafd4700 1 librados: calling monclient init 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6cafd4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6c419ef80 con 0x7fb6c410aee0 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6cafd4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6c41aeb10 con 0x7fb6c4102250 2024-03-30T15:15:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6cafd4700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb6c41afcf0 con 0x7fb6c410ea80 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c3fff700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb6c410aee0 0x7fb6c419dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50598/0 (socket says 172.21.15.28:50598) 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c3fff700 1 -- 172.21.15.28:0/1657419881 learned_addr learned my addr 172.21.15.28:0/1657419881 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6c410aee0 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6c410ea80 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6c4102250 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237709908 0 0) 0x7fb6c419ef80 con 0x7fb6c410aee0 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6b40031f0 con 0x7fb6c410aee0 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2083889008 0 0) 0x7fb6c41aeb10 con 0x7fb6c4102250 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c419ef80 con 0x7fb6c4102250 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110831517 0 0) 0x7fb6c41afcf0 con 0x7fb6c410ea80 2024-03-30T15:15:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6c41aeb10 con 0x7fb6c410ea80 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431685683 0 0) 0x7fb6b40031f0 con 0x7fb6c410aee0 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6c41afcf0 con 0x7fb6c410aee0 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6b0001890 con 0x7fb6c410aee0 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 876598453 0 0) 0x7fb6c41aeb10 con 0x7fb6c410ea80 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6b40031f0 con 0x7fb6c410ea80 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 952431105 0 0) 0x7fb6c419ef80 con 0x7fb6c4102250 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6c41aeb10 con 0x7fb6c4102250 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6bc004880 con 0x7fb6c410ea80 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6b8001830 con 0x7fb6c4102250 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 855282218 0 0) 0x7fb6c41afcf0 con 0x7fb6c410aee0 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 >> v1:172.21.15.28:6790/0 conn(0x7fb6c410ea80 legacy=0x7fb6c419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.582+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 >> v1:172.21.15.134:6789/0 conn(0x7fb6c4102250 legacy=0x7fb6c4107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c41b0ed0 con 0x7fb6c410aee0 2024-03-30T15:15:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c41aff20 con 0x7fb6c410aee0 2024-03-30T15:15:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fb6b0003b20 con 0x7fb6c410aee0 2024-03-30T15:15:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fb6b0005c60 con 0x7fb6c410aee0 2024-03-30T15:15:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6cafd4700 10 client.26019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6c41b03e0 con 0x7fb6c410aee0 2024-03-30T15:15:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.583+0000 7fb6cafd4700 1 librados: init done 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.584+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c4103730 con 0x7fb6c410aee0 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.584+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fb6b0003680 con 0x7fb6c410aee0 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(914..914 src has 842..914) ==== 7561+0+0 (unknown 2487317655 0 0) 0x7fb6b0089550 con 0x7fb6c410aee0 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 10 client.26019.objecter ms_dispatch 0x7fb6c4001550 osd_map(914..914 src has 842..914) 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 3 client.26019.objecter handle_osd_map got epochs [914,914] > 0 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 3 client.26019.objecter handle_osd_map decoding full epoch 914 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 20 client.26019.objecter dump_active .. 0 homeless 2024-03-30T15:15:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.585+0000 7fb6c1ffb700 10 client.26019.objecter ms_handle_connect 0x7fb6b4070560 2024-03-30T15:15:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.589+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb6b0056a00 con 0x7fb6c410aee0 2024-03-30T15:15:05.783 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:05.783 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:05.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.827+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"} v 0) -- 0x7fb6c4075a70 con 0x7fb6c410aee0 2024-03-30T15:15:05.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.879+0000 7fb6c1ffb700 1 -- 172.21.15.28:0/1657419881 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]=0 clear 4.a pg_upmap_items mapping v915) ==== 116+0+0 (unknown 2612932142 0 0) 0x7fb6b005a450 con 0x7fb6c410aee0 2024-03-30T15:15:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.882+0000 7fb6cafd4700 10 librados: watch_flush enter 2024-03-30T15:15:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.882+0000 7fb6cafd4700 10 librados: watch_flush exit 2024-03-30T15:15:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.882+0000 7fb6cafd4700 20 client.26019.objecter shutdown clearing up homeless session... 2024-03-30T15:15:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.882+0000 7fb6cafd4700 10 client.26019.objecter successfully canceled tick 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 >> v1:172.21.15.28:6800/3011431093 conn(0x7fb6b4070560 legacy=0x7fb6b4072a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 >> v1:172.21.15.28:6789/0 conn(0x7fb6c410aee0 legacy=0x7fb6c419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 shutdown_connections 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 >> 172.21.15.28:0/1657419881 conn(0x7fb6c40fdb20 msgr2=0x7fb6c41116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 shutdown_connections 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.883+0000 7fb6cafd4700 1 -- 172.21.15.28:0/1657419881 wait complete. 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:05.884+0000 7fb6cafd4700 1 librados: shutdown 2024-03-30T15:15:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:clear 4.a pg_upmap_items mapping 2024-03-30T15:15:06.110 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:05 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-03-30T15:15:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:05 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-03-30T15:15:06.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:05 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]: dispatch 2024-03-30T15:15:06.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-03-30T15:15:06.650 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:07.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: osdmap e915: 8 total, 8 up, 8 in 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: 6.e deep-scrub starts 2024-03-30T15:15:07.097 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:06 smithi134 ceph-mon[33714]: 6.e deep-scrub ok 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: osdmap e915: 8 total, 8 up, 8 in 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: 6.e deep-scrub starts 2024-03-30T15:15:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[29876]: 6.e deep-scrub ok 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1657419881' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.a"}]': finished 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: osdmap e915: 8 total, 8 up, 8 in 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: 6.e deep-scrub starts 2024-03-30T15:15:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:06 smithi028 ceph-mon[41614]: 6.e deep-scrub ok 2024-03-30T15:15:07.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:07 smithi134 ceph-mon[33714]: osdmap e916: 8 total, 8 up, 8 in 2024-03-30T15:15:07.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:07 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T15:15:07.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:07 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T15:15:07.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:07 smithi134 ceph-mon[33714]: osdmap e917: 8 total, 8 up, 8 in 2024-03-30T15:15:07.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[41614]: osdmap e916: 8 total, 8 up, 8 in 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[41614]: osdmap e917: 8 total, 8 up, 8 in 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[29876]: osdmap e916: 8 total, 8 up, 8 in 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T15:15:08.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[29876]: osdmap e917: 8 total, 8 up, 8 in 2024-03-30T15:15:08.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:08.248 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:08.249 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:08.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:08.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-03-30T15:15:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:08 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:15:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:08 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:15:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:08 smithi134 ceph-mon[33714]: pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:08 smithi134 ceph-mon[33714]: 5.14 deep-scrub starts 2024-03-30T15:15:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:08 smithi134 ceph-mon[33714]: 5.14 deep-scrub ok 2024-03-30T15:15:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:15:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:15:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[29876]: pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[29876]: 5.14 deep-scrub starts 2024-03-30T15:15:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[29876]: 5.14 deep-scrub ok 2024-03-30T15:15:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:15:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:15:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[41614]: pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[41614]: 5.14 deep-scrub starts 2024-03-30T15:15:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:08 smithi028 ceph-mon[41614]: 5.14 deep-scrub ok 2024-03-30T15:15:10.129 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:09 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:15:10.129 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:09 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:15:10.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:09 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:15:10.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:09 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:15:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:09 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:15:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:09 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:15:10.976 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:10.976 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: 3.2 deep-scrub starts 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: 3.2 deep-scrub ok 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: 4.d deep-scrub starts 2024-03-30T15:15:10.976 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-mon[33714]: 4.d deep-scrub ok 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: 3.2 deep-scrub starts 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: 3.2 deep-scrub ok 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: 4.d deep-scrub starts 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[29876]: 4.d deep-scrub ok 2024-03-30T15:15:11.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: 3.2 deep-scrub starts 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: 3.2 deep-scrub ok 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: 4.d deep-scrub starts 2024-03-30T15:15:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:10 smithi028 ceph-mon[41614]: 4.d deep-scrub ok 2024-03-30T15:15:11.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:11.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-30T15:15:11.657 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-30T15:15:11.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd pool set .mgr pgp_num 11 2024-03-30T15:15:11.883 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:11.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:11 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:15:11.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:11 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:15:11.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:11 smithi134 ceph-mon[33714]: osdmap e918: 8 total, 8 up, 8 in 2024-03-30T15:15:11.905 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:11.916 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:15:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:15:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[41614]: osdmap e918: 8 total, 8 up, 8 in 2024-03-30T15:15:11.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:11.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:15:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:15:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[29876]: osdmap e918: 8 total, 8 up, 8 in 2024-03-30T15:15:11.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:13.017 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:15:13.018 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:15:13.018 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:13.018 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: osdmap e919: 8 total, 8 up, 8 in 2024-03-30T15:15:13.018 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:15:13.018 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:12 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:15:13.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:15:13.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: osdmap e919: 8 total, 8 up, 8 in 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 1 objects/s recovering 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: osdmap e919: 8 total, 8 up, 8 in 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:15:13.214 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:12 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:15:13.557 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:13.558 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 -- 172.21.15.28:0/241549509 >> v1:172.21.15.28:6790/0 conn(0x7f736010aee0 legacy=0x7f736010d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 -- 172.21.15.28:0/241549509 shutdown_connections 2024-03-30T15:15:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 -- 172.21.15.28:0/241549509 >> 172.21.15.28:0/241549509 conn(0x7f73600fdb20 msgr2=0x7f73600fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 -- 172.21.15.28:0/241549509 shutdown_connections 2024-03-30T15:15:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 -- 172.21.15.28:0/241549509 wait complete. 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 librados: starting msgr at 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.810+0000 7f7364d49700 1 librados: starting objecter 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 Processor -- start 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 -- start start 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 librados: setting wanted keys 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 librados: calling monclient init 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73601a7100 con 0x7f7360102250 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73601a8320 con 0x7f736010aee0 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f7364d49700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f73601a9560 con 0x7f736010ea80 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f735e59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7360102250 0x7f73601075a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58802/0 (socket says 172.21.15.28:58802) 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f735e59c700 1 -- 172.21.15.28:0/1661184222 learned_addr learned my addr 172.21.15.28:0/1661184222 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter ms_handle_connect 0x7f7360102250 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter ms_handle_connect 0x7f736010ea80 2024-03-30T15:15:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter ms_handle_connect 0x7f736010aee0 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022133016 0 0) 0x7f73601a7100 con 0x7f7360102250 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f734c003aa0 con 0x7f7360102250 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178103759 0 0) 0x7f73601a9560 con 0x7f736010ea80 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73601a7100 con 0x7f736010ea80 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.811+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911749974 0 0) 0x7f73601a8320 con 0x7f736010aee0 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73601a9560 con 0x7f736010aee0 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3469666676 0 0) 0x7f734c003aa0 con 0x7f7360102250 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73601a8320 con 0x7f7360102250 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7350001930 con 0x7f7360102250 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1658370668 0 0) 0x7f73601a9560 con 0x7f736010aee0 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f734c003aa0 con 0x7f736010aee0 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 680364282 0 0) 0x7f73601a7100 con 0x7f736010ea80 2024-03-30T15:15:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73601a9560 con 0x7f736010ea80 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7348003820 con 0x7f736010aee0 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f73580013f0 con 0x7f736010ea80 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2444816028 0 0) 0x7f73601a8320 con 0x7f7360102250 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 >> v1:172.21.15.28:6790/0 conn(0x7f736010ea80 legacy=0x7f73601a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 >> v1:172.21.15.134:6789/0 conn(0x7f736010aee0 legacy=0x7f73601a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73601aa7a0 con 0x7f7360102250 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73601a9790 con 0x7f7360102250 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f7364d49700 10 client.26025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f73601a9c50 con 0x7f7360102250 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f7364d49700 1 librados: init done 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7350003b40 con 0x7f7360102250 2024-03-30T15:15:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.812+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7350005d10 con 0x7f7360102250 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.813+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7340005120 con 0x7f7360102250 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.814+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f735001c610 con 0x7f7360102250 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(919..919 src has 842..919) ==== 7532+0+0 (unknown 1988632944 0 0) 0x7f7350089380 con 0x7f7360102250 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 10 client.26025.objecter ms_dispatch 0x7f7360001550 osd_map(919..919 src has 842..919) 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 3 client.26025.objecter handle_osd_map got epochs [919,919] > 0 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 3 client.26025.objecter handle_osd_map decoding full epoch 919 2024-03-30T15:15:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 20 client.26025.objecter dump_active .. 0 homeless 2024-03-30T15:15:13.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.815+0000 7f73577fe700 10 client.26025.objecter ms_handle_connect 0x7f734c070490 2024-03-30T15:15:13.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:13.819+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f735005a2a0 con 0x7f7360102250 2024-03-30T15:15:14.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.027+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) -- 0x7f7340005410 con 0x7f7360102250 2024-03-30T15:15:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:13 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T15:15:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:13 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T15:15:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:13 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:15:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:13 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:15:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:13 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:15:14.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-03-30T15:15:14.473 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.910+0000 7f73577fe700 1 -- 172.21.15.28:0/1661184222 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v920) ==== 131+0+0 (unknown 1032777788 0 0) 0x7f735005eb60 con 0x7f7360102250 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.914+0000 7f7364d49700 10 librados: watch_flush enter 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.914+0000 7f7364d49700 10 librados: watch_flush exit 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.914+0000 7f7364d49700 20 client.26025.objecter shutdown clearing up homeless session... 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.914+0000 7f7364d49700 10 client.26025.objecter successfully canceled tick 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.915+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 >> v1:172.21.15.28:6800/3011431093 conn(0x7f734c070490 legacy=0x7f734c072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.915+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 >> v1:172.21.15.28:6789/0 conn(0x7f7360102250 legacy=0x7f73601075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.915+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 shutdown_connections 2024-03-30T15:15:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.915+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 >> 172.21.15.28:0/1661184222 conn(0x7f73600fdb20 msgr2=0x7f7360111bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.916+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 shutdown_connections 2024-03-30T15:15:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.916+0000 7f7364d49700 1 -- 172.21.15.28:0/1661184222 wait complete. 2024-03-30T15:15:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:14.916+0000 7f7364d49700 1 librados: shutdown 2024-03-30T15:15:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 11 2024-03-30T15:15:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:14 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T15:15:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:14 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T15:15:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:14 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T15:15:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:14 smithi134 ceph-mon[33714]: pgmap v1434: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 16 B/s, 1 objects/s recovering 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[41614]: pgmap v1434: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 16 B/s, 1 objects/s recovering 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-30T15:15:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:14 smithi028 ceph-mon[29876]: pgmap v1434: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 16 B/s, 1 objects/s recovering 2024-03-30T15:15:15.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 4.b scrub starts 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 4.b scrub ok 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: osdmap e920: 8 total, 8 up, 8 in 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:15:15.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 4.b scrub starts 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 4.b scrub ok 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: osdmap e920: 8 total, 8 up, 8 in 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:15:15.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:15 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:15:15.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 4.b scrub starts 2024-03-30T15:15:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 4.b scrub ok 2024-03-30T15:15:16.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1661184222' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-30T15:15:16.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: osdmap e920: 8 total, 8 up, 8 in 2024-03-30T15:15:16.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:15:16.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:15:16.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:15:16.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:15 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:15:16.185 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:16.186 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:16.798 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-03-30T15:15:16.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[29876]: 6.11 scrub starts 2024-03-30T15:15:17.036 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:17.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[29876]: 6.11 scrub ok 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[29876]: pgmap v1436: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[41614]: 6.11 scrub starts 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[41614]: 6.11 scrub ok 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[41614]: pgmap v1436: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T15:15:17.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:16 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T15:15:17.158 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:16 smithi134 ceph-mon[33714]: 6.11 scrub starts 2024-03-30T15:15:17.158 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:16 smithi134 ceph-mon[33714]: 6.11 scrub ok 2024-03-30T15:15:17.158 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:16 smithi134 ceph-mon[33714]: pgmap v1436: 174 pgs: 174 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:17.158 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:16 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T15:15:17.158 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:16 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- 172.21.15.28:0/2034354102 >> v1:172.21.15.28:6789/0 conn(0x7f59a4101140 legacy=0x7f59a4101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- 172.21.15.28:0/2034354102 shutdown_connections 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- 172.21.15.28:0/2034354102 >> 172.21.15.28:0/2034354102 conn(0x7f59a40fc9d0 msgr2=0x7f59a40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- 172.21.15.28:0/2034354102 shutdown_connections 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- 172.21.15.28:0/2034354102 wait complete. 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 librados: starting msgr at 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 librados: starting objecter 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 Processor -- start 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 -- start start 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 librados: setting wanted keys 2024-03-30T15:15:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.674+0000 7f59aa8ac700 1 librados: calling monclient init 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59aa8ac700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a41a9420 con 0x7f59a410d970 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59aa8ac700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a41aa600 con 0x7f59a4101140 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59aa8ac700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f59a41ab840 con 0x7f59a4109dd0 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a8e49700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f59a410d970 0x7f59a41a7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58826/0 (socket says 172.21.15.28:58826) 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a37fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f59a4109dd0 0x7f59a41a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:54518/0 (socket says 172.21.15.28:54518) 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a8e49700 1 -- 172.21.15.28:0/1932841787 learned_addr learned my addr 172.21.15.28:0/1932841787 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter ms_handle_connect 0x7f59a410d970 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter ms_handle_connect 0x7f59a4109dd0 2024-03-30T15:15:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter ms_handle_connect 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497506256 0 0) 0x7f59a41a9420 con 0x7f59a410d970 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f598c0031f0 con 0x7f59a410d970 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 740240210 0 0) 0x7f59a41aa600 con 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a41a9420 con 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076565520 0 0) 0x7f59a41ab840 con 0x7f59a4109dd0 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.675+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a41aa600 con 0x7f59a4109dd0 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1547474990 0 0) 0x7f59a41a9420 con 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a41ab840 con 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3617302330 0 0) 0x7f598c0031f0 con 0x7f59a410d970 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a41a9420 con 0x7f59a410d970 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f59940044a0 con 0x7f59a4101140 2024-03-30T15:15:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1348139664 0 0) 0x7f59a41aa600 con 0x7f59a4109dd0 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f598c0031f0 con 0x7f59a4109dd0 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c001db0 con 0x7f59a410d970 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f59980017b0 con 0x7f59a4109dd0 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 640417465 0 0) 0x7f59a41ab840 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6790/0 conn(0x7f59a4109dd0 legacy=0x7f59a41a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6789/0 conn(0x7f59a410d970 legacy=0x7f59a41a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a41aca80 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59a41aa830 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59aa8ac700 10 client.51223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59a41aacf0 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59aa8ac700 1 librados: init done 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f5994001ca0 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.676+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f5994004c10 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.677+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5990005140 con 0x7f59a4101140 2024-03-30T15:15:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f5994001800 con 0x7f59a4101140 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(921..921 src has 842..921) ==== 7532+0+0 (unknown 2037041457 0 0) 0x7f5994087f40 con 0x7f59a4101140 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 10 client.51223.objecter ms_dispatch 0x7f59a4001550 osd_map(921..921 src has 842..921) 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 3 client.51223.objecter handle_osd_map got epochs [921,921] > 0 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 3 client.51223.objecter handle_osd_map decoding full epoch 921 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 20 client.51223.objecter dump_active .. 0 homeless 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1127 (unknown 409899127 0 633015442) 0x7f5994088410 con 0x7f59a4101140 2024-03-30T15:15:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.678+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f598c070550 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5990006b70 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 312356697 0 0) 0x7f5990006b70 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a41ab840 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3367256911 0 0) 0x7f59a41ab840 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5990006b70 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c003ba0 con 0x7f59900029c0 2024-03-30T15:15:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 5983462 0 0) 0x7f5990006b70 con 0x7f59900029c0 2024-03-30T15:15:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.679+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f599000a670 con 0x7f59900029c0 2024-03-30T15:15:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.680+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f599c005b90 con 0x7f59900029c0 2024-03-30T15:15:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.680+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6789/0 conn(0x7f59900029c0 legacy=0x7f5990005450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.680+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6789/0 conn(0x7f59900029c0 legacy=0x7f5990005450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f599000aed0 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832224391 0 0) 0x7f599000aed0 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5990006b70 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1383204422 0 0) 0x7f5990006b70 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f599000aed0 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.687+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c0042e0 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.688+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 733857464 0 0) 0x7f599000aed0 con 0x7f599000aa70 2024-03-30T15:15:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.688+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f599000c130 con 0x7f599000aa70 2024-03-30T15:15:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.689+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.0 v1:172.21.15.28:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f599c004540 con 0x7f599000aa70 2024-03-30T15:15:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.689+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6789/0 conn(0x7f599000aa70 legacy=0x7f5990008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.689+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6789/0 conn(0x7f599000aa70 legacy=0x7f5990008d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.691 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-03-30T15:15:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.689+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5990002330 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998618609 0 0) 0x7f59a41a9420 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f599000aed0 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1584011555 0 0) 0x7f599000aed0 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a41a9420 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c0061a0 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3980822890 0 0) 0x7f59a41a9420 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.690+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f5990012c90 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.691+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 5 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f599c0061c0 con 0x7f5990001f00 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.691+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.134:6789/0 conn(0x7f5990001f00 legacy=0x7f599000b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.691+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.134:6789/0 conn(0x7f5990001f00 legacy=0x7f599000b870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.691+0000 7f59a8e49700 1 -- 172.21.15.28:0/1932841787 reap_dead start 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5990013400 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2866294005 0 0) 0x7f5990013400 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a41a9420 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3273991658 0 0) 0x7f59a41a9420 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5990013400 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c0065f0 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 218768709 0 0) 0x7f5990013400 con 0x7f5990012ff0 2024-03-30T15:15:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.698+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.134:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f59900145e0 con 0x7f5990012ff0 2024-03-30T15:15:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.1 v1:172.21.15.134:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f599c0067e0 con 0x7f5990012ff0 2024-03-30T15:15:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.134:6789/0 conn(0x7f5990012ff0 legacy=0x7f5990011370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.134:6789/0 conn(0x7f5990012ff0 legacy=0x7f5990011370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.701 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-03-30T15:15:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f5990014220 con 0x7f59900135d0 2024-03-30T15:15:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f59900135d0 2024-03-30T15:15:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2844720721 0 0) 0x7f5990014220 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5990013400 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2307736400 0 0) 0x7f5990013400 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5990014220 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.700+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c006e30 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.701+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2486783927 0 0) 0x7f5990014220 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.701+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f599001b140 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.701+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 5 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f599c009670 con 0x7f59900135d0 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.701+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6790/0 conn(0x7f59900135d0 legacy=0x7f59900139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.701+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6790/0 conn(0x7f59900135d0 legacy=0x7f59900139e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.709+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 3) -- 0x7f599001af40 con 0x7f599001b470 2024-03-30T15:15:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.709+0000 7f59a17fa700 10 client.51223.objecter ms_handle_connect 0x7f599001b470 2024-03-30T15:15:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.709+0000 7f59a17fa700 10 client.51223.objecter resend_mon_ops 2024-03-30T15:15:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384286413 0 0) 0x7f599001af40 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5990014220 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 587330112 0 0) 0x7f5990014220 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f599001af40 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f599c0073a0 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3551220537 0 0) 0x7f599001af40 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.710+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 --> v1:172.21.15.28:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f599001c970 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.711+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 <== mon.2 v1:172.21.15.28:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f599c007690 con 0x7f599001b470 2024-03-30T15:15:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.711+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6790/0 conn(0x7f599001b470 legacy=0x7f5990019820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.711+0000 7f59a17fa700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6790/0 conn(0x7f599001b470 legacy=0x7f5990019820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T15:15:17.713 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 10 librados: watch_flush enter 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 10 librados: watch_flush exit 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 20 client.51223.objecter shutdown clearing up homeless session... 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 10 client.51223.objecter successfully canceled tick 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.28:6800/3011431093 conn(0x7f598c070550 legacy=0x7f598c072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 >> v1:172.21.15.134:6789/0 conn(0x7f59a4101140 legacy=0x7f59a40fe850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59a8e49700 1 -- 172.21.15.28:0/1932841787 reap_dead start 2024-03-30T15:15:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 shutdown_connections 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 >> 172.21.15.28:0/1932841787 conn(0x7f59a40fc9d0 msgr2=0x7f59a406a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 shutdown_connections 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 -- 172.21.15.28:0/1932841787 wait complete. 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:17.712+0000 7f59aa8ac700 1 librados: shutdown 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T15:15:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:15:17.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:15:17.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: osdmap e921: 8 total, 8 up, 8 in 2024-03-30T15:15:17.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1932841787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T15:15:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:15:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:15:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:15:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:15:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:15:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:15:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: osdmap e921: 8 total, 8 up, 8 in 2024-03-30T15:15:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:17 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1932841787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T15:15:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:15:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:15:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:15:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: osdmap e921: 8 total, 8 up, 8 in 2024-03-30T15:15:18.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:17 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1932841787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T15:15:18.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 1 1 2024-03-30T15:15:18.494 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:18.808 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:18.808 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 4.3 scrub starts 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 4.3 scrub ok 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:15:19.118 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.14 deep-scrub starts 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.14 deep-scrub ok 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: pgmap v1438: 174 pgs: 174 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.119 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:18 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 4.3 scrub starts 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 4.3 scrub ok 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:15:19.182 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.14 deep-scrub starts 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.14 deep-scrub ok 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: pgmap v1438: 174 pgs: 174 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 4.3 scrub starts 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 4.3 scrub ok 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:15:19.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.14 deep-scrub starts 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.14 deep-scrub ok 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: pgmap v1438: 174 pgs: 174 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:18 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-30T15:15:19.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-30T15:15:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:20.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: 1.1 scrub starts 2024-03-30T15:15:20.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: 1.1 scrub ok 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: osdmap e922: 8 total, 8 up, 8 in 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: 4.18 scrub starts 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[29876]: 4.18 scrub ok 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: 1.1 scrub starts 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: 1.1 scrub ok 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: osdmap e922: 8 total, 8 up, 8 in 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: 4.18 scrub starts 2024-03-30T15:15:20.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:19 smithi028 ceph-mon[41614]: 4.18 scrub ok 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: 1.1 scrub starts 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: 1.1 scrub ok 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: osdmap e922: 8 total, 8 up, 8 in 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: 4.18 scrub starts 2024-03-30T15:15:20.105 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:19 smithi134 ceph-mon[33714]: 4.18 scrub ok 2024-03-30T15:15:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.287+0000 7f93a73dc700 1 -- 172.21.15.28:0/4080362416 >> v1:172.21.15.28:6790/0 conn(0x7f93a010ea60 legacy=0x7f93a0110f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- 172.21.15.28:0/4080362416 shutdown_connections 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- 172.21.15.28:0/4080362416 >> 172.21.15.28:0/4080362416 conn(0x7f93a00fdb00 msgr2=0x7f93a00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- 172.21.15.28:0/4080362416 shutdown_connections 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- 172.21.15.28:0/4080362416 wait complete. 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 librados: starting msgr at 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 librados: starting objecter 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 Processor -- start 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- start start 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 librados: setting wanted keys 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 librados: calling monclient init 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a01a9370 con 0x7f93a0102230 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a01aa590 con 0x7f93a010ea60 2024-03-30T15:15:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.288+0000 7f93a73dc700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a01ab7d0 con 0x7f93a010aec0 2024-03-30T15:15:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a5178700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f93a0102230 0x7f93a01075f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40622/0 (socket says 172.21.15.28:40622) 2024-03-30T15:15:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a5178700 1 -- 172.21.15.28:0/3607996670 learned_addr learned my addr 172.21.15.28:0/3607996670 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter ms_handle_connect 0x7f93a0102230 2024-03-30T15:15:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter ms_handle_connect 0x7f93a010aec0 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter ms_handle_connect 0x7f93a010ea60 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270720383 0 0) 0x7f93a01a9370 con 0x7f93a0102230 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f938c0039c0 con 0x7f93a0102230 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793107736 0 0) 0x7f93a01aa590 con 0x7f93a010ea60 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a01a9370 con 0x7f93a010ea60 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849659088 0 0) 0x7f93a01ab7d0 con 0x7f93a010aec0 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a01aa590 con 0x7f93a010aec0 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3856884942 0 0) 0x7f93a01a9370 con 0x7f93a010ea60 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a01ab7d0 con 0x7f93a010ea60 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1905709809 0 0) 0x7f938c0039c0 con 0x7f93a0102230 2024-03-30T15:15:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a01a9370 con 0x7f93a0102230 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9398001cd0 con 0x7f93a010ea60 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f939c001930 con 0x7f93a0102230 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1982657030 0 0) 0x7f93a01aa590 con 0x7f93a010aec0 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f938c0039c0 con 0x7f93a010aec0 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f93900010b0 con 0x7f93a010aec0 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1614754284 0 0) 0x7f93a01ab7d0 con 0x7f93a010ea60 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 >> v1:172.21.15.28:6790/0 conn(0x7f93a010aec0 legacy=0x7f93a01a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 >> v1:172.21.15.28:6789/0 conn(0x7f93a0102230 legacy=0x7f93a01075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a01aca10 con 0x7f93a010ea60 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93a01ab9a0 con 0x7f93a010ea60 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a73dc700 10 client.51235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f93a01abe60 con 0x7f93a010ea60 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.289+0000 7f93a73dc700 1 librados: init done 2024-03-30T15:15:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.290+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9398002010 con 0x7f93a010ea60 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.290+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9398005040 con 0x7f93a010ea60 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.290+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9384005120 con 0x7f93a010ea60 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.291+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9398003b70 con 0x7f93a010ea60 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.291+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(923..923 src has 842..923) ==== 7561+0+0 (unknown 3801137042 0 0) 0x7f9398088590 con 0x7f93a010ea60 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.291+0000 7f93967fc700 10 client.51235.objecter ms_dispatch 0x7f93a0001550 osd_map(923..923 src has 842..923) 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.291+0000 7f93967fc700 3 client.51235.objecter handle_osd_map got epochs [923,923] > 0 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.291+0000 7f93967fc700 3 client.51235.objecter handle_osd_map decoding full epoch 923 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.292+0000 7f93967fc700 20 client.51235.objecter dump_active .. 0 homeless 2024-03-30T15:15:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.292+0000 7f93967fc700 10 client.51235.objecter ms_handle_connect 0x7f938c0703e0 2024-03-30T15:15:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.296+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f93980559e0 con 0x7f93a010ea60 2024-03-30T15:15:20.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.512+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f9384005410 con 0x7f93a010ea60 2024-03-30T15:15:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.940+0000 7f93967fc700 1 -- 172.21.15.28:0/3607996670 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v924) ==== 113+0+0 (unknown 3686350842 0 0) 0x7f9398059430 con 0x7f93a010ea60 2024-03-30T15:15:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.943+0000 7f93a73dc700 10 librados: watch_flush enter 2024-03-30T15:15:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.943+0000 7f93a73dc700 10 librados: watch_flush exit 2024-03-30T15:15:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.943+0000 7f93a73dc700 20 client.51235.objecter shutdown clearing up homeless session... 2024-03-30T15:15:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.943+0000 7f93a73dc700 10 client.51235.objecter successfully canceled tick 2024-03-30T15:15:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 >> v1:172.21.15.28:6800/3011431093 conn(0x7f938c0703e0 legacy=0x7f938c0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 >> v1:172.21.15.134:6789/0 conn(0x7f93a010ea60 legacy=0x7f93a01a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 shutdown_connections 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 >> 172.21.15.28:0/3607996670 conn(0x7f93a00fdb00 msgr2=0x7f93a0111f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 shutdown_connections 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 -- 172.21.15.28:0/3607996670 wait complete. 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:20.944+0000 7f93a73dc700 1 librados: shutdown 2024-03-30T15:15:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T15:15:20.994 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 6.1f scrub starts 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 6.1f scrub ok 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: osdmap e923: 8 total, 8 up, 8 in 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3607996670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:20.995 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-mon[33714]: from='client.51235 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 6.1f scrub starts 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 6.1f scrub ok 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: osdmap e923: 8 total, 8 up, 8 in 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T15:15:21.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3607996670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[29876]: from='client.51235 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 6.1f scrub starts 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 6.1f scrub ok 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: osdmap e923: 8 total, 8 up, 8 in 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3607996670' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:21.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:20 smithi028 ceph-mon[41614]: from='client.51235 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:21.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:21.312 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:21.313 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:21.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 1 1 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[29876]: from='client.51235 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[29876]: osdmap e924: 8 total, 8 up, 8 in 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[41614]: from='client.51235 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[41614]: osdmap e924: 8 total, 8 up, 8 in 2024-03-30T15:15:21.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:21.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:15:21.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:21 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:15:21.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-03-30T15:15:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:22.219 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:22.341 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:21 smithi134 ceph-mon[33714]: from='client.51235 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:22.341 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:21 smithi134 ceph-mon[33714]: osdmap e924: 8 total, 8 up, 8 in 2024-03-30T15:15:22.341 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:22.341 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:21 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:15:22.341 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:21 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 12.3 scrub starts 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 12.3 scrub ok 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 3.10 scrub starts 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 3.10 scrub ok 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:15:23.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 23 B/s, 0 objects/s recovering 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: osdmap e925: 8 total, 8 up, 8 in 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 12.3 scrub starts 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 12.3 scrub ok 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 3.10 scrub starts 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 3.10 scrub ok 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:15:23.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:15:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 23 B/s, 0 objects/s recovering 2024-03-30T15:15:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:15:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:15:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: osdmap e925: 8 total, 8 up, 8 in 2024-03-30T15:15:23.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:22 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 12.3 scrub starts 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 12.3 scrub ok 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 3.10 scrub starts 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 3.10 scrub ok 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 23 B/s, 0 objects/s recovering 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: osdmap e925: 8 total, 8 up, 8 in 2024-03-30T15:15:23.381 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:22 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- 172.21.15.28:0/885011711 >> v1:172.21.15.28:6790/0 conn(0x7ff6d810eac0 legacy=0x7ff6d8110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- 172.21.15.28:0/885011711 shutdown_connections 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- 172.21.15.28:0/885011711 >> 172.21.15.28:0/885011711 conn(0x7ff6d80fc9d0 msgr2=0x7ff6d80fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- 172.21.15.28:0/885011711 shutdown_connections 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- 172.21.15.28:0/885011711 wait complete. 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 librados: starting msgr at 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 librados: starting objecter 2024-03-30T15:15:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 Processor -- start 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 -- start start 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 librados: setting wanted keys 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.778+0000 7ff6de515700 1 librados: calling monclient init 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6de515700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d810ad70 con 0x7ff6d810afb0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6de515700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a8640 con 0x7ff6d810eac0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6de515700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a9860 con 0x7ff6d81031a0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d7fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff6d81031a0 0x7ff6d8107bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38090/0 (socket says 172.21.15.28:38090) 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d7fff700 1 -- 172.21.15.28:0/3131155984 learned_addr learned my addr 172.21.15.28:0/3131155984 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter ms_handle_connect 0x7ff6d81031a0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter ms_handle_connect 0x7ff6d810afb0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter ms_handle_connect 0x7ff6d810eac0 2024-03-30T15:15:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118837283 0 0) 0x7ff6d810ad70 con 0x7ff6d810afb0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c40031f0 con 0x7ff6d810afb0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237078141 0 0) 0x7ff6d81a9860 con 0x7ff6d81031a0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d810ad70 con 0x7ff6d81031a0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063619909 0 0) 0x7ff6d81a8640 con 0x7ff6d810eac0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d81a9860 con 0x7ff6d810eac0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 862095882 0 0) 0x7ff6d810ad70 con 0x7ff6d81031a0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d81a8640 con 0x7ff6d81031a0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.779+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2229805715 0 0) 0x7ff6c40031f0 con 0x7ff6d810afb0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d810ad70 con 0x7ff6d810afb0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 513945582 0 0) 0x7ff6d81a9860 con 0x7ff6d810eac0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c40031f0 con 0x7ff6d810eac0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6c00018d0 con 0x7ff6d81031a0 2024-03-30T15:15:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6c8001830 con 0x7ff6d810afb0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6d00017b0 con 0x7ff6d810eac0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4286234953 0 0) 0x7ff6d81a8640 con 0x7ff6d81031a0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 >> v1:172.21.15.134:6789/0 conn(0x7ff6d810eac0 legacy=0x7ff6d81a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 >> v1:172.21.15.28:6789/0 conn(0x7ff6d810afb0 legacy=0x7ff6d81082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d81aaac0 con 0x7ff6d81031a0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d81a7690 con 0x7ff6d81031a0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6de515700 10 client.59252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6d81a7b50 con 0x7ff6d81031a0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6de515700 1 librados: init done 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff6c0003b40 con 0x7ff6d81031a0 2024-03-30T15:15:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.780+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff6c0005cb0 con 0x7ff6d81031a0 2024-03-30T15:15:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.781+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b8005120 con 0x7ff6d81031a0 2024-03-30T15:15:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff6c00036a0 con 0x7ff6d81031a0 2024-03-30T15:15:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(925..925 src has 842..925) ==== 7561+0+0 (unknown 3223162620 0 0) 0x7ff6c00893a0 con 0x7ff6d81031a0 2024-03-30T15:15:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 10 client.59252.objecter ms_dispatch 0x7ff6d8001550 osd_map(925..925 src has 842..925) 2024-03-30T15:15:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 3 client.59252.objecter handle_osd_map got epochs [925,925] > 0 2024-03-30T15:15:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 3 client.59252.objecter handle_osd_map decoding full epoch 925 2024-03-30T15:15:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 20 client.59252.objecter dump_active .. 0 homeless 2024-03-30T15:15:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.782+0000 7ff6d57fa700 10 client.59252.objecter ms_handle_connect 0x7ff6c4070470 2024-03-30T15:15:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:23.787+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff6c00567f0 con 0x7ff6d81031a0 2024-03-30T15:15:23.962 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:23.963 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.007+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7ff6b8005c20 con 0x7ff6d81031a0 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:15:24.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:15:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T15:15:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T15:15:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T15:15:24.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:23 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T15:15:24.262 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:23 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T15:15:24.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-03-30T15:15:24.821 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:24.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.958+0000 7ff6d57fa700 1 -- 172.21.15.28:0/3131155984 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v926) ==== 133+0+0 (unknown 930881846 0 0) 0x7ff6c005a240 con 0x7ff6d81031a0 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 10 librados: watch_flush enter 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 10 librados: watch_flush exit 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 20 client.59252.objecter shutdown clearing up homeless session... 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 10 client.59252.objecter successfully canceled tick 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff6c4070470 legacy=0x7ff6c4072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 >> v1:172.21.15.28:6790/0 conn(0x7ff6d81031a0 legacy=0x7ff6d8107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.961+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 shutdown_connections 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.962+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 >> 172.21.15.28:0/3131155984 conn(0x7ff6d80fc9d0 msgr2=0x7ff6d8110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.962+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 shutdown_connections 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.962+0000 7ff6de515700 1 -- 172.21.15.28:0/3131155984 wait complete. 2024-03-30T15:15:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:24.962+0000 7ff6de515700 1 librados: shutdown 2024-03-30T15:15:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T15:15:25.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3131155984' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[29876]: from='client.59252 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[29876]: pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 20 B/s, 0 objects/s recovering 2024-03-30T15:15:25.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3131155984' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[41614]: from='client.59252 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:24 smithi028 ceph-mon[41614]: pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 20 B/s, 0 objects/s recovering 2024-03-30T15:15:25.243 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:24 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3131155984' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.244 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:24 smithi134 ceph-mon[33714]: from='client.59252 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-30T15:15:25.244 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:24 smithi134 ceph-mon[33714]: pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 20 B/s, 0 objects/s recovering 2024-03-30T15:15:25.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 3 1 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: from='client.59252 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: osdmap e926: 8 total, 8 up, 8 in 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:15:25.989 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:25 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:15:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:26.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T15:15:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T15:15:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:15:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:15:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: from='client.59252 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:26.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: osdmap e926: 8 total, 8 up, 8 in 2024-03-30T15:15:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:15:26.102 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:15:26.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T15:15:26.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T15:15:26.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:15:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:15:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: from='client.59252 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-30T15:15:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: osdmap e926: 8 total, 8 up, 8 in 2024-03-30T15:15:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:15:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:25 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:15:26.474 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:26.474 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:27.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T15:15:27.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T15:15:27.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T15:15:27.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T15:15:27.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-30T15:15:27.023 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:15:27.023 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:26 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:15:27.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:15:27.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:15:27.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:26 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:15:27.296 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:27.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.850+0000 7f8b11080700 1 -- 172.21.15.28:0/1497171447 >> v1:172.21.15.28:6790/0 conn(0x7f8b0c10d970 legacy=0x7f8b0c10fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.850+0000 7f8b11080700 1 -- 172.21.15.28:0/1497171447 shutdown_connections 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.850+0000 7f8b11080700 1 -- 172.21.15.28:0/1497171447 >> 172.21.15.28:0/1497171447 conn(0x7f8b0c0fc9f0 msgr2=0x7f8b0c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.850+0000 7f8b11080700 1 -- 172.21.15.28:0/1497171447 shutdown_connections 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.850+0000 7f8b11080700 1 -- 172.21.15.28:0/1497171447 wait complete. 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 librados: starting msgr at 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 librados: starting objecter 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 Processor -- start 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 -- start start 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 librados: setting wanted keys 2024-03-30T15:15:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.851+0000 7f8b11080700 1 librados: calling monclient init 2024-03-30T15:15:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8b11080700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b0c1a93f0 con 0x7f8b0c109dd0 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8b11080700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b0c1aa610 con 0x7f8b0c101140 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8b11080700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8b0c1ab850 con 0x7f8b0c10d970 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8b0b59e700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f8b0c10d970 0x7f8b0c1a7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:38118/0 (socket says 172.21.15.28:38118) 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8b0b59e700 1 -- 172.21.15.28:0/649403050 learned_addr learned my addr 172.21.15.28:0/649403050 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter ms_handle_connect 0x7f8b0c109dd0 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter ms_handle_connect 0x7f8b0c10d970 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter ms_handle_connect 0x7f8b0c101140 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077369318 0 0) 0x7f8b0c1aa610 con 0x7f8b0c101140 2024-03-30T15:15:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8af40039c0 con 0x7f8b0c101140 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487861525 0 0) 0x7f8b0c1a93f0 con 0x7f8b0c109dd0 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c1aa610 con 0x7f8b0c109dd0 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953997233 0 0) 0x7f8b0c1ab850 con 0x7f8b0c10d970 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c1a93f0 con 0x7f8b0c10d970 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3166772653 0 0) 0x7f8af40039c0 con 0x7f8b0c101140 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b0c1ab850 con 0x7f8b0c101140 2024-03-30T15:15:27.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8afc0013d0 con 0x7f8b0c101140 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1541032361 0 0) 0x7f8b0c1aa610 con 0x7f8b0c109dd0 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8af40039c0 con 0x7f8b0c109dd0 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1447916483 0 0) 0x7f8b0c1a93f0 con 0x7f8b0c10d970 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b0c1aa610 con 0x7f8b0c10d970 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b000018b0 con 0x7f8b0c109dd0 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b040013a0 con 0x7f8b0c10d970 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.852+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1170181128 0 0) 0x7f8b0c1ab850 con 0x7f8b0c101140 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 >> v1:172.21.15.28:6790/0 conn(0x7f8b0c10d970 legacy=0x7f8b0c1a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 >> v1:172.21.15.28:6789/0 conn(0x7f8b0c109dd0 legacy=0x7f8b0c1a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c1aca90 con 0x7f8b0c101140 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0c1aba80 con 0x7f8b0c101140 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8b11080700 10 client.51247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b0c1abf40 con 0x7f8b0c101140 2024-03-30T15:15:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8b11080700 1 librados: init done 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8afc001d30 con 0x7f8b0c101140 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.853+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8afc004940 con 0x7f8b0c101140 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.854+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af8005150 con 0x7f8b0c101140 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.854+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8afc01b2b0 con 0x7f8b0c101140 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(927..927 src has 842..927) ==== 7561+0+0 (unknown 3731267110 0 0) 0x7f8afc088120 con 0x7f8b0c101140 2024-03-30T15:15:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 10 client.51247.objecter ms_dispatch 0x7f8b0c001550 osd_map(927..927 src has 842..927) 2024-03-30T15:15:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 3 client.51247.objecter handle_osd_map got epochs [927,927] > 0 2024-03-30T15:15:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 3 client.51247.objecter handle_osd_map decoding full epoch 927 2024-03-30T15:15:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 20 client.51247.objecter dump_active .. 0 homeless 2024-03-30T15:15:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.855+0000 7f8af3fff700 10 client.51247.objecter ms_handle_connect 0x7f8af40703e0 2024-03-30T15:15:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:27.859+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8afc055570 con 0x7f8b0c101140 2024-03-30T15:15:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.079+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f8af8005440 con 0x7f8b0c101140 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 5.d deep-scrub starts 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: 5.d deep-scrub ok 2024-03-30T15:15:28.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[29876]: osdmap e927: 8 total, 8 up, 8 in 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 5.d deep-scrub starts 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: 5.d deep-scrub ok 2024-03-30T15:15:28.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:27 smithi028 ceph-mon[41614]: osdmap e927: 8 total, 8 up, 8 in 2024-03-30T15:15:28.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:15:28.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:15:28.288 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:15:28.289 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:15:28.289 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 5.d deep-scrub starts 2024-03-30T15:15:28.289 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: 5.d deep-scrub ok 2024-03-30T15:15:28.289 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:27 smithi134 ceph-mon[33714]: osdmap e927: 8 total, 8 up, 8 in 2024-03-30T15:15:28.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.969+0000 7f8af3fff700 1 -- 172.21.15.28:0/649403050 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v928) ==== 113+0+0 (unknown 1353179392 0 0) 0x7f8afc058fc0 con 0x7f8b0c101140 2024-03-30T15:15:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.973+0000 7f8b11080700 10 librados: watch_flush enter 2024-03-30T15:15:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.973+0000 7f8b11080700 10 librados: watch_flush exit 2024-03-30T15:15:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.973+0000 7f8b11080700 20 client.51247.objecter shutdown clearing up homeless session... 2024-03-30T15:15:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.973+0000 7f8b11080700 10 client.51247.objecter successfully canceled tick 2024-03-30T15:15:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8af40703e0 legacy=0x7f8af40728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 >> v1:172.21.15.134:6789/0 conn(0x7f8b0c101140 legacy=0x7f8b0c106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 shutdown_connections 2024-03-30T15:15:28.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 >> 172.21.15.28:0/649403050 conn(0x7f8b0c0fc9f0 msgr2=0x7f8b0c0fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 shutdown_connections 2024-03-30T15:15:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 -- 172.21.15.28:0/649403050 wait complete. 2024-03-30T15:15:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:28.974+0000 7f8b11080700 1 librados: shutdown 2024-03-30T15:15:28.977 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T15:15:29.109 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:29.110 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:29.110 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 5.2 scrub starts 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 5.2 scrub ok 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/649403050' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: from='client.51247 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 12.2 deep-scrub starts 2024-03-30T15:15:29.114 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:28 smithi134 ceph-mon[33714]: 12.2 deep-scrub ok 2024-03-30T15:15:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 5.2 scrub starts 2024-03-30T15:15:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 5.2 scrub ok 2024-03-30T15:15:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T15:15:29.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/649403050' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: from='client.51247 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 12.2 deep-scrub starts 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[29876]: 12.2 deep-scrub ok 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 5.2 scrub starts 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 5.2 scrub ok 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T15:15:29.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T15:15:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/649403050' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: from='client.51247 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 12.2 deep-scrub starts 2024-03-30T15:15:29.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:28 smithi028 ceph-mon[41614]: 12.2 deep-scrub ok 2024-03-30T15:15:29.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 3 1 2024-03-30T15:15:29.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-03-30T15:15:29.993 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:30.115 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: 6.a scrub starts 2024-03-30T15:15:30.116 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: 6.a scrub ok 2024-03-30T15:15:30.116 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: from='client.51247 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:30.116 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: osdmap e928: 8 total, 8 up, 8 in 2024-03-30T15:15:30.116 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: 6.1c deep-scrub starts 2024-03-30T15:15:30.116 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:29 smithi134 ceph-mon[33714]: 6.1c deep-scrub ok 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: 6.a scrub starts 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: 6.a scrub ok 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: from='client.51247 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: osdmap e928: 8 total, 8 up, 8 in 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: 6.1c deep-scrub starts 2024-03-30T15:15:30.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[29876]: 6.1c deep-scrub ok 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: 6.a scrub starts 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: 6.a scrub ok 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: from='client.51247 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: osdmap e928: 8 total, 8 up, 8 in 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: 6.1c deep-scrub starts 2024-03-30T15:15:30.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:29 smithi028 ceph-mon[41614]: 6.1c deep-scrub ok 2024-03-30T15:15:31.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:30 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:30 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:15:31.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:30 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:15:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:30 smithi134 ceph-mon[33714]: pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:31.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:30 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[29876]: pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:15:31.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:15:31.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[41614]: pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:15:31.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:30 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:15:31.709 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:31.709 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 -- 172.21.15.28:0/233586377 >> v1:172.21.15.28:6790/0 conn(0x7fa67c1031c0 legacy=0x7fa67c1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 -- 172.21.15.28:0/233586377 shutdown_connections 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 -- 172.21.15.28:0/233586377 >> 172.21.15.28:0/233586377 conn(0x7fa67c0fca30 msgr2=0x7fa67c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 -- 172.21.15.28:0/233586377 shutdown_connections 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 -- 172.21.15.28:0/233586377 wait complete. 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 librados: starting msgr at 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.798+0000 7fa68369a700 1 librados: starting objecter 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 Processor -- start 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 -- start start 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 librados: setting wanted keys 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 librados: calling monclient init 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa67c1a91d0 con 0x7fa67c1031c0 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa67c1aa3f0 con 0x7fa67c10eae0 2024-03-30T15:15:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa68369a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa67c1ab610 con 0x7fa67c10afd0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa681436700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa67c1031c0 0x7fa67c10a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48098/0 (socket says 172.21.15.28:48098) 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa681436700 1 -- 172.21.15.28:0/2404802801 learned_addr learned my addr 172.21.15.28:0/2404802801 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter ms_handle_connect 0x7fa67c1031c0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter ms_handle_connect 0x7fa67c10eae0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852749411 0 0) 0x7fa67c1a91d0 con 0x7fa67c1031c0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa664003aa0 con 0x7fa67c1031c0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter ms_handle_connect 0x7fa67c10afd0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299834839 0 0) 0x7fa67c1aa3f0 con 0x7fa67c10eae0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.799+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c1a91d0 con 0x7fa67c10eae0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158210371 0 0) 0x7fa67c1ab610 con 0x7fa67c10afd0 2024-03-30T15:15:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa67c1aa3f0 con 0x7fa67c10afd0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3563691933 0 0) 0x7fa664003aa0 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa67c1ab610 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa6740045a0 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3942994237 0 0) 0x7fa67c1a91d0 con 0x7fa67c10eae0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa664003aa0 con 0x7fa67c10eae0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387649961 0 0) 0x7fa67c1aa3f0 con 0x7fa67c10afd0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa67c1a91d0 con 0x7fa67c10afd0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa66c0013f0 con 0x7fa67c10eae0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa6700017b0 con 0x7fa67c10afd0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2262502872 0 0) 0x7fa67c1ab610 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 >> v1:172.21.15.28:6790/0 conn(0x7fa67c10afd0 legacy=0x7fa67c1a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 >> v1:172.21.15.134:6789/0 conn(0x7fa67c10eae0 legacy=0x7fa67c1a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c1ac850 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c1aa620 con 0x7fa67c1031c0 2024-03-30T15:15:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa68369a700 10 client.26061.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa67c1aaae0 con 0x7fa67c1031c0 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa68369a700 1 librados: init done 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fa674001b10 con 0x7fa67c1031c0 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.800+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fa674004e00 con 0x7fa67c1031c0 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.801+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa668005120 con 0x7fa67c1031c0 2024-03-30T15:15:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.802+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fa674004de0 con 0x7fa67c1031c0 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.802+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(928..928 src has 842..928) ==== 7561+0+0 (unknown 2613025228 0 0) 0x7fa6740894c0 con 0x7fa67c1031c0 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.802+0000 7fa67a7fc700 10 client.26061.objecter ms_dispatch 0x7fa67c001550 osd_map(928..928 src has 842..928) 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.802+0000 7fa67a7fc700 3 client.26061.objecter handle_osd_map got epochs [928,928] > 0 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.802+0000 7fa67a7fc700 3 client.26061.objecter handle_osd_map decoding full epoch 928 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.803+0000 7fa67a7fc700 20 client.26061.objecter dump_active .. 0 homeless 2024-03-30T15:15:31.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.803+0000 7fa67a7fc700 10 client.26061.objecter ms_handle_connect 0x7fa6640704e0 2024-03-30T15:15:31.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:31.807+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa674056910 con 0x7fa67c1031c0 2024-03-30T15:15:32.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.024+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7fa668005c20 con 0x7fa67c1031c0 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T15:15:32.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:15:32.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:31 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:15:32.283 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:32.283 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T15:15:32.283 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T15:15:32.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T15:15:32.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T15:15:32.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:15:32.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:15:32.284 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:31 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:15:32.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-03-30T15:15:32.547 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T15:15:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.612+0000 7fa67a7fc700 1 -- 172.21.15.28:0/2404802801 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v929) ==== 133+0+0 (unknown 2598980920 0 0) 0x7fa67405a360 con 0x7fa67c1031c0 2024-03-30T15:15:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.616+0000 7fa68369a700 10 librados: watch_flush enter 2024-03-30T15:15:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.617+0000 7fa68369a700 10 librados: watch_flush exit 2024-03-30T15:15:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.617+0000 7fa68369a700 20 client.26061.objecter shutdown clearing up homeless session... 2024-03-30T15:15:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.617+0000 7fa68369a700 10 client.26061.objecter successfully canceled tick 2024-03-30T15:15:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.618+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 >> v1:172.21.15.28:6800/3011431093 conn(0x7fa6640704e0 legacy=0x7fa6640729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.618+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 >> v1:172.21.15.28:6789/0 conn(0x7fa67c1031c0 legacy=0x7fa67c10a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.618+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 shutdown_connections 2024-03-30T15:15:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.618+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 >> 172.21.15.28:0/2404802801 conn(0x7fa67c0fca30 msgr2=0x7fa67c110f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.620+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 shutdown_connections 2024-03-30T15:15:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.621+0000 7fa68369a700 1 -- 172.21.15.28:0/2404802801 wait complete. 2024-03-30T15:15:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:32.621+0000 7fa68369a700 1 librados: shutdown 2024-03-30T15:15:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T15:15:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:15:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:15:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 3.11 scrub starts 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 3.11 scrub ok 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:32 smithi134 ceph-mon[33714]: osdmap e929: 8 total, 8 up, 8 in 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 3.11 scrub starts 2024-03-30T15:15:33.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 3.11 scrub ok 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[29876]: osdmap e929: 8 total, 8 up, 8 in 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 3.11 scrub starts 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 3.11 scrub ok 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:15:33.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:15:33.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2404802801' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-30T15:15:33.225 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:32 smithi028 ceph-mon[41614]: osdmap e929: 8 total, 8 up, 8 in 2024-03-30T15:15:33.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 5 1 2024-03-30T15:15:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:34.170 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:34.171 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: osdmap e930: 8 total, 8 up, 8 in 2024-03-30T15:15:34.773 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:34 smithi134 ceph-mon[33714]: pgmap v1455: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:34.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: osdmap e930: 8 total, 8 up, 8 in 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[41614]: pgmap v1455: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:34.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T15:15:34.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T15:15:34.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:15:34.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:15:34.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: osdmap e930: 8 total, 8 up, 8 in 2024-03-30T15:15:34.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:34 smithi028 ceph-mon[29876]: pgmap v1455: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 37 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- 172.21.15.28:0/2047244126 >> v1:172.21.15.28:6790/0 conn(0x7f859c10aee0 legacy=0x7f859c10d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- 172.21.15.28:0/2047244126 shutdown_connections 2024-03-30T15:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- 172.21.15.28:0/2047244126 >> 172.21.15.28:0/2047244126 conn(0x7f859c0fdb20 msgr2=0x7f859c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- 172.21.15.28:0/2047244126 shutdown_connections 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- 172.21.15.28:0/2047244126 wait complete. 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 librados: starting msgr at 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 librados: starting objecter 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 Processor -- start 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 -- start start 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 librados: setting wanted keys 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.723+0000 7f85a410c700 1 librados: calling monclient init 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f85a410c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f859c107980 con 0x7f859c10ea80 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f85a410c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f859c1a8650 con 0x7f859c102250 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f85a410c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f859c1a9830 con 0x7f859c10aee0 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f85a16a7700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f859c10aee0 0x7f859c104ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:53466/0 (socket says 172.21.15.28:53466) 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f85a16a7700 1 -- 172.21.15.28:0/4089855279 learned_addr learned my addr 172.21.15.28:0/4089855279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter ms_handle_connect 0x7f859c10aee0 2024-03-30T15:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter ms_handle_connect 0x7f859c10ea80 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter ms_handle_connect 0x7f859c102250 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942348560 0 0) 0x7f859c1a9830 con 0x7f859c10aee0 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85880031f0 con 0x7f859c10aee0 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735648996 0 0) 0x7f859c107980 con 0x7f859c10ea80 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f859c1a9830 con 0x7f859c10ea80 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911900672 0 0) 0x7f859c1a8650 con 0x7f859c102250 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f859c107980 con 0x7f859c102250 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3342647205 0 0) 0x7f85880031f0 con 0x7f859c10aee0 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f859c1a8650 con 0x7f859c10aee0 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f858c004610 con 0x7f859c10aee0 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3248197465 0 0) 0x7f859c1a9830 con 0x7f859c10ea80 2024-03-30T15:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85880031f0 con 0x7f859c10ea80 2024-03-30T15:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8594001af0 con 0x7f859c10ea80 2024-03-30T15:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3223543382 0 0) 0x7f859c107980 con 0x7f859c102250 2024-03-30T15:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.724+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f859c1a9830 con 0x7f859c102250 2024-03-30T15:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8598001890 con 0x7f859c102250 2024-03-30T15:15:35.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 1.4 deep-scrub starts 2024-03-30T15:15:35.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 1.4 deep-scrub ok 2024-03-30T15:15:35.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:15:35.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 6.8 scrub starts 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[29876]: 6.8 scrub ok 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 1.4 deep-scrub starts 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 1.4 deep-scrub ok 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 6.8 scrub starts 2024-03-30T15:15:35.731 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:35 smithi028 ceph-mon[41614]: 6.8 scrub ok 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 334502669 0 0) 0x7f859c1a8650 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 >> v1:172.21.15.134:6789/0 conn(0x7f859c102250 legacy=0x7f859c1047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 >> v1:172.21.15.28:6789/0 conn(0x7f859c10ea80 legacy=0x7f859c1a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859c1aaa10 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f859c1a76a0 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f85a410c700 10 client.59270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f859c1a7b60 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f85a410c700 1 librados: init done 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f858c001af0 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.725+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f858c004e80 con 0x7f859c10aee0 2024-03-30T15:15:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.726+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8580005120 con 0x7f859c10aee0 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.726+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f858c0050f0 con 0x7f859c10aee0 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(930..930 src has 842..930) ==== 7561+0+0 (unknown 2244249878 0 0) 0x7f858c089400 con 0x7f859c10aee0 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 10 client.59270.objecter ms_dispatch 0x7f859c001550 osd_map(930..930 src has 842..930) 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 3 client.59270.objecter handle_osd_map got epochs [930,930] > 0 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 3 client.59270.objecter handle_osd_map decoding full epoch 930 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 20 client.59270.objecter dump_active .. 0 homeless 2024-03-30T15:15:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.727+0000 7f8592ffd700 10 client.59270.objecter ms_handle_connect 0x7f8588070430 2024-03-30T15:15:35.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.732+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f858c056990 con 0x7f859c10aee0 2024-03-30T15:15:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:35.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 1.4 deep-scrub starts 2024-03-30T15:15:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 1.4 deep-scrub ok 2024-03-30T15:15:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:15:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:15:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 6.8 scrub starts 2024-03-30T15:15:35.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:35 smithi134 ceph-mon[33714]: 6.8 scrub ok 2024-03-30T15:15:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:35.960+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f8580005410 con 0x7f859c10aee0 2024-03-30T15:15:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.634+0000 7f8592ffd700 1 -- 172.21.15.28:0/4089855279 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v931) ==== 113+0+0 (unknown 1352810672 0 0) 0x7f858c05a3e0 con 0x7f859c10aee0 2024-03-30T15:15:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.638+0000 7f85a410c700 10 librados: watch_flush enter 2024-03-30T15:15:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.638+0000 7f85a410c700 10 librados: watch_flush exit 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.638+0000 7f85a410c700 20 client.59270.objecter shutdown clearing up homeless session... 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.638+0000 7f85a410c700 10 client.59270.objecter successfully canceled tick 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8588070430 legacy=0x7f85880728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 >> v1:172.21.15.28:6790/0 conn(0x7f859c10aee0 legacy=0x7f859c104ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 shutdown_connections 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 >> 172.21.15.28:0/4089855279 conn(0x7f859c0fdb20 msgr2=0x7f859c1116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 shutdown_connections 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 -- 172.21.15.28:0/4089855279 wait complete. 2024-03-30T15:15:36.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:36.639+0000 7f85a410c700 1 librados: shutdown 2024-03-30T15:15:36.643 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T15:15:36.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4089855279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: from='client.59270 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: pgmap v1456: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 30 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:15:36.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:36 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4089855279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: from='client.59270 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: pgmap v1456: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 30 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:15:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4089855279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: from='client.59270 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: pgmap v1456: 174 pgs: 174 active+clean; 315 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 30 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:15:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:36 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:15:37.746 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:37.746 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: 6.1d deep-scrub starts 2024-03-30T15:15:37.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: 6.1d deep-scrub ok 2024-03-30T15:15:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: from='client.59270 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: osdmap e931: 8 total, 8 up, 8 in 2024-03-30T15:15:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: 4.4 scrub starts 2024-03-30T15:15:37.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:37 smithi134 ceph-mon[33714]: 4.4 scrub ok 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: 6.1d deep-scrub starts 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: 6.1d deep-scrub ok 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: from='client.59270 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: osdmap e931: 8 total, 8 up, 8 in 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: 4.4 scrub starts 2024-03-30T15:15:37.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[41614]: 4.4 scrub ok 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: 6.1d deep-scrub starts 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: 6.1d deep-scrub ok 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: from='client.59270 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: osdmap e931: 8 total, 8 up, 8 in 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: 4.4 scrub starts 2024-03-30T15:15:37.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:37 smithi028 ceph-mon[29876]: 4.4 scrub ok 2024-03-30T15:15:38.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 5 1 2024-03-30T15:15:38.622 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:38.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:15:38.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:15:38.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:38.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: pgmap v1458: 174 pgs: 174 active+clean; 315 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 37 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:38.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:15:38.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:38 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:15:38.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:15:38.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:15:38.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:38.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: pgmap v1458: 174 pgs: 174 active+clean; 315 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 37 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: pgmap v1458: 174 pgs: 174 active+clean; 315 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 37 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:15:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:38 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:15:39.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-03-30T15:15:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:39.839 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:15:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:15:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:15:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:15:39.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:15:39.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:39 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:15:39.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:15:39.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:15:39.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:15:39.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:15:39.960 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:15:39.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:39 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:15:40.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 12.18 deep-scrub starts 2024-03-30T15:15:40.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 12.18 deep-scrub ok 2024-03-30T15:15:40.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 12.19 scrub starts 2024-03-30T15:15:40.909 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 12.19 scrub ok 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 3.8 deep-scrub starts 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 3.8 deep-scrub ok 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[41614]: pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 12.18 deep-scrub starts 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 12.18 deep-scrub ok 2024-03-30T15:15:40.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 12.19 scrub starts 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 12.19 scrub ok 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 3.8 deep-scrub starts 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 3.8 deep-scrub ok 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:15:40.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:40 smithi028 ceph-mon[29876]: pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-30T15:15:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 12.18 deep-scrub starts 2024-03-30T15:15:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 12.18 deep-scrub ok 2024-03-30T15:15:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 12.19 scrub starts 2024-03-30T15:15:40.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 12.19 scrub ok 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 3.8 deep-scrub starts 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 3.8 deep-scrub ok 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:15:40.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:40 smithi134 ceph-mon[33714]: pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-30T15:15:41.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:41 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 -- 172.21.15.28:0/3699829251 >> v1:172.21.15.28:6790/0 conn(0x7f9c3c109dd0 legacy=0x7f9c3c10c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 -- 172.21.15.28:0/3699829251 shutdown_connections 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 -- 172.21.15.28:0/3699829251 >> 172.21.15.28:0/3699829251 conn(0x7f9c3c0fc9f0 msgr2=0x7f9c3c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 -- 172.21.15.28:0/3699829251 shutdown_connections 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 -- 172.21.15.28:0/3699829251 wait complete. 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 librados: starting msgr at 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 librados: starting objecter 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.619+0000 7f9c41f55700 1 Processor -- start 2024-03-30T15:15:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 -- start start 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 librados: setting wanted keys 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 librados: calling monclient init 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c3c1a93e0 con 0x7f9c3c10d970 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c3c1aa600 con 0x7f9c3c109dd0 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c41f55700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9c3c1ab840 con 0x7f9c3c101140 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c3b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9c3c101140 0x7f9c3c0fe8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48844/0 (socket says 172.21.15.28:48844) 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c3b7fe700 1 -- 172.21.15.28:0/4154057216 learned_addr learned my addr 172.21.15.28:0/4154057216 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 10 client.?.objecter ms_handle_connect 0x7f9c3c10d970 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 10 client.?.objecter ms_handle_connect 0x7f9c3c101140 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873731638 0 0) 0x7f9c3c1a93e0 con 0x7f9c3c10d970 2024-03-30T15:15:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.620+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c240031f0 con 0x7f9c3c10d970 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515114680 0 0) 0x7f9c3c1ab840 con 0x7f9c3c101140 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c3c1a93e0 con 0x7f9c3c101140 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 10 client.?.objecter ms_handle_connect 0x7f9c3c109dd0 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278302470 0 0) 0x7f9c3c1aa600 con 0x7f9c3c109dd0 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c3c1ab840 con 0x7f9c3c109dd0 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1144630242 0 0) 0x7f9c240031f0 con 0x7f9c3c10d970 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c3c1aa600 con 0x7f9c3c10d970 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9c34001590 con 0x7f9c3c10d970 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4010002286 0 0) 0x7f9c3c1a93e0 con 0x7f9c3c101140 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c240031f0 con 0x7f9c3c101140 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9c2c0018b0 con 0x7f9c3c101140 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1530261465 0 0) 0x7f9c3c1aa600 con 0x7f9c3c10d970 2024-03-30T15:15:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 >> v1:172.21.15.28:6790/0 conn(0x7f9c3c101140 legacy=0x7f9c3c0fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 >> v1:172.21.15.134:6789/0 conn(0x7f9c3c109dd0 legacy=0x7f9c3c1a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c1aca80 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c1aba70 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c41f55700 10 client.26073.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c3c1abf30 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c41f55700 1 librados: init done 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9c34003ea0 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.621+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9c34004d40 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.622+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c102620 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9c34003a00 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(931..931 src has 881..931) ==== 7561+0+0 (unknown 2087961199 0 0) 0x7f9c340886c0 con 0x7f9c3c10d970 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 10 client.26073.objecter ms_dispatch 0x7f9c3c001550 osd_map(931..931 src has 881..931) 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 3 client.26073.objecter handle_osd_map got epochs [931,931] > 0 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 3 client.26073.objecter handle_osd_map decoding full epoch 931 2024-03-30T15:15:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 20 client.26073.objecter dump_active .. 0 homeless 2024-03-30T15:15:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.623+0000 7f9c38ff9700 10 client.26073.objecter ms_handle_connect 0x7f9c24070240 2024-03-30T15:15:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.628+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9c34055b70 con 0x7f9c3c10d970 2024-03-30T15:15:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:41.848+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7f9c3c0fe380 con 0x7f9c3c10d970 2024-03-30T15:15:41.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:41.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:41.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:42.590 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:42.590 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:42.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.665+0000 7f9c38ff9700 1 -- 172.21.15.28:0/4154057216 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v932) ==== 133+0+0 (unknown 3889033295 0 0) 0x7f9c340595c0 con 0x7f9c3c10d970 2024-03-30T15:15:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.669+0000 7f9c41f55700 10 librados: watch_flush enter 2024-03-30T15:15:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.669+0000 7f9c41f55700 10 librados: watch_flush exit 2024-03-30T15:15:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.669+0000 7f9c41f55700 20 client.26073.objecter shutdown clearing up homeless session... 2024-03-30T15:15:42.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.669+0000 7f9c41f55700 10 client.26073.objecter successfully canceled tick 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 12.1d scrub starts 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 12.1d scrub ok 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[29876]: pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:15:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.670+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9c24070240 legacy=0x7f9c24072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.670+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 >> v1:172.21.15.28:6789/0 conn(0x7f9c3c10d970 legacy=0x7f9c3c1a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.671+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 shutdown_connections 2024-03-30T15:15:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.671+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 >> 172.21.15.28:0/4154057216 conn(0x7f9c3c0fc9f0 msgr2=0x7f9c3c110a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.671+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 shutdown_connections 2024-03-30T15:15:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.671+0000 7f9c41f55700 1 -- 172.21.15.28:0/4154057216 wait complete. 2024-03-30T15:15:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:42.671+0000 7f9c41f55700 1 librados: shutdown 2024-03-30T15:15:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T15:15:42.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 12.1d scrub starts 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 12.1d scrub ok 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:42.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:42 smithi134 ceph-mon[33714]: pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:15:42.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:15:42.967 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 12.1d scrub starts 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 12.1d scrub ok 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-30T15:15:42.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:42 smithi028 ceph-mon[41614]: pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-30T15:15:43.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-03-30T15:15:43.873 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 2.1 scrub starts 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 2.1 scrub ok 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: osdmap e932: 8 total, 8 up, 8 in 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:15:43.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 2.1 scrub starts 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 2.1 scrub ok 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: osdmap e932: 8 total, 8 up, 8 in 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:15:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:43 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:15:43.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 6 1 2024-03-30T15:15:43.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 2.1 scrub starts 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 2.1 scrub ok 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/4154057216' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: osdmap e932: 8 total, 8 up, 8 in 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:15:43.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:43 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:15:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:15:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:15:44.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[29876]: pgmap v1462: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:44.723 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:15:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:15:44.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:44 smithi028 ceph-mon[41614]: pgmap v1462: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:44 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:15:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:44 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:15:44.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:44 smithi134 ceph-mon[33714]: pgmap v1462: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:45.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: osdmap e933: 8 total, 8 up, 8 in 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:15:45.860 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: osdmap e933: 8 total, 8 up, 8 in 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:15:45.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:15:45.862 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:45 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:15:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:15:45.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: osdmap e933: 8 total, 8 up, 8 in 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:15:45.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:45 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:15:46.548 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:46.548 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[29876]: 1.a scrub starts 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[29876]: 1.a scrub ok 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[29876]: pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[41614]: 1.a scrub starts 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[41614]: 1.a scrub ok 2024-03-30T15:15:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:46 smithi028 ceph-mon[41614]: pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:46 smithi134 ceph-mon[33714]: 1.a scrub starts 2024-03-30T15:15:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:46 smithi134 ceph-mon[33714]: 1.a scrub ok 2024-03-30T15:15:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:46 smithi134 ceph-mon[33714]: pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-30T15:15:47.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-30T15:15:47.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:47 smithi028 ceph-mon[29876]: 4.e deep-scrub starts 2024-03-30T15:15:47.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:47 smithi028 ceph-mon[29876]: 4.e deep-scrub ok 2024-03-30T15:15:47.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:47 smithi028 ceph-mon[41614]: 4.e deep-scrub starts 2024-03-30T15:15:47.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:47 smithi028 ceph-mon[41614]: 4.e deep-scrub ok 2024-03-30T15:15:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:47 smithi134 ceph-mon[33714]: 4.e deep-scrub starts 2024-03-30T15:15:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:47 smithi134 ceph-mon[33714]: 4.e deep-scrub ok 2024-03-30T15:15:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.330+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/2623554179 >> v1:172.21.15.28:6789/0 conn(0x7f4ccc102230 legacy=0x7f4ccc102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.330+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/2623554179 shutdown_connections 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.330+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/2623554179 >> 172.21.15.28:0/2623554179 conn(0x7f4ccc0fdb00 msgr2=0x7f4ccc0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.330+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/2623554179 shutdown_connections 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.330+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/2623554179 wait complete. 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 librados: starting msgr at 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 librados: starting objecter 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 Processor -- start 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 -- start start 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 librados: setting wanted keys 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 librados: calling monclient init 2024-03-30T15:15:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ccc1a9330 con 0x7f4ccc10ea60 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ccc1aa550 con 0x7f4ccc102230 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd3fa0700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ccc1ab790 con 0x7f4ccc10aec0 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd253d700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ccc10ea60 0x7f4ccc1a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54122/0 (socket says 172.21.15.28:54122) 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4cd253d700 1 -- 172.21.15.28:0/447128073 learned_addr learned my addr 172.21.15.28:0/447128073 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter ms_handle_connect 0x7f4ccc10ea60 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter ms_handle_connect 0x7f4ccc10aec0 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter ms_handle_connect 0x7f4ccc102230 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.331+0000 7f4ccaffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3798893500 0 0) 0x7f4ccc1a9330 con 0x7f4ccc10ea60 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb40031f0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446111054 0 0) 0x7f4ccc1ab790 con 0x7f4ccc10aec0 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ccc1a9330 con 0x7f4ccc10aec0 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078266673 0 0) 0x7f4ccc1aa550 con 0x7f4ccc102230 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ccc1ab790 con 0x7f4ccc102230 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2316475984 0 0) 0x7f4cb40031f0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ccc1aa550 con 0x7f4ccc10ea60 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4cc40018f0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1664163980 0 0) 0x7f4ccc1a9330 con 0x7f4ccc10aec0 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cb40031f0 con 0x7f4ccc10aec0 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2260515456 0 0) 0x7f4ccc1ab790 con 0x7f4ccc102230 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ccc1a9330 con 0x7f4ccc102230 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4cc00019b0 con 0x7f4ccc10aec0 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4cbc0047c0 con 0x7f4ccc102230 2024-03-30T15:15:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 33462811 0 0) 0x7f4ccc1aa550 con 0x7f4ccc10ea60 2024-03-30T15:15:48.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 >> v1:172.21.15.28:6790/0 conn(0x7f4ccc10aec0 legacy=0x7f4ccc1a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:48.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 >> v1:172.21.15.134:6789/0 conn(0x7f4ccc102230 legacy=0x7f4ccc075f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:48.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc1ac9d0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc1ab9c0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4cd3fa0700 10 client.26079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ccc1abe80 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4cd3fa0700 1 librados: init done 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4cc4001cc0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.332+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4cc4005da0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.333+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb8005120 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.334+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f4cc4004160 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(933..933 src has 881..933) ==== 7561+0+0 (unknown 1593766657 0 0) 0x7f4cc4089680 con 0x7f4ccc10ea60 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 10 client.26079.objecter ms_dispatch 0x7f4ccc001550 osd_map(933..933 src has 881..933) 2024-03-30T15:15:48.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 3 client.26079.objecter handle_osd_map got epochs [933,933] > 0 2024-03-30T15:15:48.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 3 client.26079.objecter handle_osd_map decoding full epoch 933 2024-03-30T15:15:48.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 20 client.26079.objecter dump_active .. 0 homeless 2024-03-30T15:15:48.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.335+0000 7f4ccaffd700 10 client.26079.objecter ms_handle_connect 0x7f4cb4070450 2024-03-30T15:15:48.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.339+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4cc40044d0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:48.496 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:48.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.570+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f4cb8005410 con 0x7f4ccc10ea60 2024-03-30T15:15:48.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.683+0000 7f4ccaffd700 1 -- 172.21.15.28:0/447128073 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v934) ==== 113+0+0 (unknown 152314354 0 0) 0x7f4cc4056ad0 con 0x7f4ccc10ea60 2024-03-30T15:15:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.687+0000 7f4cd3fa0700 10 librados: watch_flush enter 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 10 librados: watch_flush exit 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 20 client.26079.objecter shutdown clearing up homeless session... 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 10 client.26079.objecter successfully canceled tick 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 >> v1:172.21.15.28:6800/3011431093 conn(0x7f4cb4070450 legacy=0x7f4cb4072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 >> v1:172.21.15.28:6789/0 conn(0x7f4ccc10ea60 legacy=0x7f4ccc1a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 shutdown_connections 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 >> 172.21.15.28:0/447128073 conn(0x7f4ccc0fdb00 msgr2=0x7f4ccc111b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 shutdown_connections 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.688+0000 7f4cd3fa0700 1 -- 172.21.15.28:0/447128073 wait complete. 2024-03-30T15:15:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:48.689+0000 7f4cd3fa0700 1 librados: shutdown 2024-03-30T15:15:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T15:15:48.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 4.1f scrub starts 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 4.1f scrub ok 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 5.7 scrub starts 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 5.7 scrub ok 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 4.1f scrub starts 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 4.1f scrub ok 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 5.7 scrub starts 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 5.7 scrub ok 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:15:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:48 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 4.1f scrub starts 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 4.1f scrub ok 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 5.7 scrub starts 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 5.7 scrub ok 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:15:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:48 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:49.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 6 1 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 12.b scrub starts 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 12.b scrub ok 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 4.c scrub starts 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 4.c scrub ok 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: osdmap e934: 8 total, 8 up, 8 in 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:15:49.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 4.2 deep-scrub starts 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[29876]: 4.2 deep-scrub ok 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 12.b scrub starts 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 12.b scrub ok 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 4.c scrub starts 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 4.c scrub ok 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: osdmap e934: 8 total, 8 up, 8 in 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 4.2 deep-scrub starts 2024-03-30T15:15:49.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:49 smithi028 ceph-mon[41614]: 4.2 deep-scrub ok 2024-03-30T15:15:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 12.b scrub starts 2024-03-30T15:15:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 12.b scrub ok 2024-03-30T15:15:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 4.c scrub starts 2024-03-30T15:15:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 4.c scrub ok 2024-03-30T15:15:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/447128073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:50.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: osdmap e934: 8 total, 8 up, 8 in 2024-03-30T15:15:50.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:15:50.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:15:50.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 4.2 deep-scrub starts 2024-03-30T15:15:50.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:49 smithi134 ceph-mon[33714]: 4.2 deep-scrub ok 2024-03-30T15:15:50.963 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[41614]: 4.9 scrub starts 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[41614]: 4.9 scrub ok 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[41614]: pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[29876]: 4.9 scrub starts 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[29876]: 4.9 scrub ok 2024-03-30T15:15:50.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:50 smithi028 ceph-mon[29876]: pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:50.965 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:50.965 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:50.973 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:51.010 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:50 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:15:51.010 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:50 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:15:51.010 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:50 smithi134 ceph-mon[33714]: 4.9 scrub starts 2024-03-30T15:15:51.010 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:50 smithi134 ceph-mon[33714]: 4.9 scrub ok 2024-03-30T15:15:51.011 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:50 smithi134 ceph-mon[33714]: pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:51.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:15:51.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[29876]: 12.10 scrub starts 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[29876]: 12.10 scrub ok 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[41614]: 12.10 scrub starts 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[41614]: 12.10 scrub ok 2024-03-30T15:15:51.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:15:51.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:15:51.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-mon[33714]: 12.10 scrub starts 2024-03-30T15:15:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-mon[33714]: 12.10 scrub ok 2024-03-30T15:15:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:15:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:15:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:15:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:52.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[29876]: 4.14 scrub starts 2024-03-30T15:15:52.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[29876]: 4.14 scrub ok 2024-03-30T15:15:52.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[29876]: 12.1f deep-scrub starts 2024-03-30T15:15:52.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[29876]: 12.1f deep-scrub ok 2024-03-30T15:15:52.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[29876]: pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[41614]: 4.14 scrub starts 2024-03-30T15:15:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[41614]: 4.14 scrub ok 2024-03-30T15:15:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[41614]: 12.1f deep-scrub starts 2024-03-30T15:15:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[41614]: 12.1f deep-scrub ok 2024-03-30T15:15:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:52 smithi028 ceph-mon[41614]: pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:52 smithi134 ceph-mon[33714]: 4.14 scrub starts 2024-03-30T15:15:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:52 smithi134 ceph-mon[33714]: 4.14 scrub ok 2024-03-30T15:15:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:52 smithi134 ceph-mon[33714]: 12.1f deep-scrub starts 2024-03-30T15:15:53.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:52 smithi134 ceph-mon[33714]: 12.1f deep-scrub ok 2024-03-30T15:15:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:52 smithi134 ceph-mon[33714]: pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: 12.d deep-scrub starts 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: 12.d deep-scrub ok 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: osdmap e935: 8 total, 8 up, 8 in 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T15:15:53.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: 12.d deep-scrub starts 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: 12.d deep-scrub ok 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: osdmap e935: 8 total, 8 up, 8 in 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T15:15:53.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:53 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T15:15:54.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: 12.d deep-scrub starts 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: 12.d deep-scrub ok 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: osdmap e935: 8 total, 8 up, 8 in 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T15:15:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:53 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- 172.21.15.28:0/332711006 >> v1:172.21.15.28:6789/0 conn(0x7f0b3410eb20 legacy=0x7f0b34110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- 172.21.15.28:0/332711006 shutdown_connections 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- 172.21.15.28:0/332711006 >> 172.21.15.28:0/332711006 conn(0x7f0b340fc9d0 msgr2=0x7f0b340fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- 172.21.15.28:0/332711006 shutdown_connections 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- 172.21.15.28:0/332711006 wait complete. 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 librados: starting msgr at 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 librados: starting objecter 2024-03-30T15:15:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 Processor -- start 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 -- start start 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 librados: setting wanted keys 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.364+0000 7f0b3bd95700 1 librados: calling monclient init 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b3bd95700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b341029d0 con 0x7f0b3410eb20 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b3bd95700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b34102be0 con 0x7f0b3410b010 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b3bd95700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0b341aba60 con 0x7f0b34103200 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b39b31700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f0b34103200 0x7f0b34100e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:59678/0 (socket says 172.21.15.28:59678) 2024-03-30T15:15:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b39b31700 1 -- 172.21.15.28:0/184717391 learned_addr learned my addr 172.21.15.28:0/184717391 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter ms_handle_connect 0x7f0b34103200 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter ms_handle_connect 0x7f0b3410eb20 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter ms_handle_connect 0x7f0b3410b010 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1926262188 0 0) 0x7f0b341aba60 con 0x7f0b34103200 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b200031f0 con 0x7f0b34103200 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147008049 0 0) 0x7f0b341029d0 con 0x7f0b3410eb20 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b341aba60 con 0x7f0b3410eb20 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942818488 0 0) 0x7f0b34102be0 con 0x7f0b3410b010 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b341029d0 con 0x7f0b3410b010 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1420673284 0 0) 0x7f0b200031f0 con 0x7f0b34103200 2024-03-30T15:15:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b34102be0 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0b30001a10 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435980059 0 0) 0x7f0b341029d0 con 0x7f0b3410b010 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b200031f0 con 0x7f0b3410b010 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.365+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1011888385 0 0) 0x7f0b341aba60 con 0x7f0b3410eb20 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b341029d0 con 0x7f0b3410eb20 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0b240017f0 con 0x7f0b3410b010 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0b2c001620 con 0x7f0b3410eb20 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 108522544 0 0) 0x7f0b34102be0 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 >> v1:172.21.15.134:6789/0 conn(0x7f0b3410b010 legacy=0x7f0b34101570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 >> v1:172.21.15.28:6789/0 conn(0x7f0b3410eb20 legacy=0x7f0b341a8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b341acc40 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b341abc30 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b3bd95700 10 client.59288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b341ac0f0 con 0x7f0b34103200 2024-03-30T15:15:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b3bd95700 1 librados: init done 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f0b30003dd0 con 0x7f0b34103200 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.366+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0b30005e20 con 0x7f0b34103200 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.367+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b18005120 con 0x7f0b34103200 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.367+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f0b30003930 con 0x7f0b34103200 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(935..935 src has 881..935) ==== 7561+0+0 (unknown 1082516443 0 0) 0x7f0b30089620 con 0x7f0b34103200 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 10 client.59288.objecter ms_dispatch 0x7f0b34001550 osd_map(935..935 src has 881..935) 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 3 client.59288.objecter handle_osd_map got epochs [935,935] > 0 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 3 client.59288.objecter handle_osd_map decoding full epoch 935 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 20 client.59288.objecter dump_active .. 0 homeless 2024-03-30T15:15:54.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.368+0000 7f0b2affd700 10 client.59288.objecter ms_handle_connect 0x7f0b200704d0 2024-03-30T15:15:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.373+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0b30056ad0 con 0x7f0b34103200 2024-03-30T15:15:54.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.593+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f0b18005c20 con 0x7f0b34103200 2024-03-30T15:15:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.708+0000 7f0b2affd700 1 -- 172.21.15.28:0/184717391 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v936) ==== 133+0+0 (unknown 4145951643 0 0) 0x7f0b3005a520 con 0x7f0b34103200 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 10 librados: watch_flush enter 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 10 librados: watch_flush exit 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 20 client.59288.objecter shutdown clearing up homeless session... 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 10 client.59288.objecter successfully canceled tick 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 >> v1:172.21.15.28:6800/3011431093 conn(0x7f0b200704d0 legacy=0x7f0b20072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 >> v1:172.21.15.28:6790/0 conn(0x7f0b34103200 legacy=0x7f0b34100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 shutdown_connections 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 >> 172.21.15.28:0/184717391 conn(0x7f0b340fc9d0 msgr2=0x7f0b340fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 shutdown_connections 2024-03-30T15:15:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.711+0000 7f0b3bd95700 1 -- 172.21.15.28:0/184717391 wait complete. 2024-03-30T15:15:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:54.712+0000 7f0b3bd95700 1 librados: shutdown 2024-03-30T15:15:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T15:15:54.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: 6.6 scrub starts 2024-03-30T15:15:54.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: 6.6 scrub ok 2024-03-30T15:15:54.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: 12.17 scrub starts 2024-03-30T15:15:54.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: 12.17 scrub ok 2024-03-30T15:15:54.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/184717391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[29876]: from='client.59288 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: 6.6 scrub starts 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: 6.6 scrub ok 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: 12.17 scrub starts 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: 12.17 scrub ok 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/184717391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:54.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:54 smithi028 ceph-mon[41614]: from='client.59288 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: 6.6 scrub starts 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: 6.6 scrub ok 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: 12.17 scrub starts 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: 12.17 scrub ok 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/184717391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:54 smithi134 ceph-mon[33714]: from='client.59288 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-30T15:15:55.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 7 1 2024-03-30T15:15:55.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[41614]: 12.1c scrub starts 2024-03-30T15:15:55.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[41614]: 12.1c scrub ok 2024-03-30T15:15:55.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[41614]: from='client.59288 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:55.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[41614]: osdmap e936: 8 total, 8 up, 8 in 2024-03-30T15:15:55.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[29876]: 12.1c scrub starts 2024-03-30T15:15:55.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[29876]: 12.1c scrub ok 2024-03-30T15:15:55.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[29876]: from='client.59288 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:55.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:55 smithi028 ceph-mon[29876]: osdmap e936: 8 total, 8 up, 8 in 2024-03-30T15:15:56.103 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:55 smithi134 ceph-mon[33714]: 12.1c scrub starts 2024-03-30T15:15:56.103 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:55 smithi134 ceph-mon[33714]: 12.1c scrub ok 2024-03-30T15:15:56.103 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:55 smithi134 ceph-mon[33714]: from='client.59288 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-30T15:15:56.103 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:55 smithi134 ceph-mon[33714]: osdmap e936: 8 total, 8 up, 8 in 2024-03-30T15:15:56.217 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:15:56.217 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:15:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:56.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[41614]: 6.1b scrub starts 2024-03-30T15:15:56.844 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[41614]: 6.1b scrub ok 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[41614]: 5.1b scrub starts 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[41614]: 5.1b scrub ok 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[41614]: pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[29876]: 6.1b scrub starts 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[29876]: 6.1b scrub ok 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[29876]: 5.1b scrub starts 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[29876]: 5.1b scrub ok 2024-03-30T15:15:56.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:56 smithi028 ceph-mon[29876]: pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:56.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-03-30T15:15:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:56 smithi134 ceph-mon[33714]: 6.1b scrub starts 2024-03-30T15:15:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:56 smithi134 ceph-mon[33714]: 6.1b scrub ok 2024-03-30T15:15:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:56 smithi134 ceph-mon[33714]: 5.1b scrub starts 2024-03-30T15:15:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:56 smithi134 ceph-mon[33714]: 5.1b scrub ok 2024-03-30T15:15:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:56 smithi134 ceph-mon[33714]: pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:15:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[29876]: 6.0 scrub starts 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[29876]: 6.0 scrub ok 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[41614]: 6.0 scrub starts 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[41614]: 6.0 scrub ok 2024-03-30T15:15:57.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:15:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:57 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:15:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:57 smithi134 ceph-mon[33714]: 6.0 scrub starts 2024-03-30T15:15:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:57 smithi134 ceph-mon[33714]: 6.0 scrub ok 2024-03-30T15:15:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:57 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:15:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:57 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:15:58.451 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:15:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 3.b scrub starts 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 3.b scrub ok 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 4.1 deep-scrub starts 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 4.1 deep-scrub ok 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 1.8 scrub starts 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: 1.8 scrub ok 2024-03-30T15:15:58.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[41614]: pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 3.b scrub starts 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 3.b scrub ok 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 4.1 deep-scrub starts 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 4.1 deep-scrub ok 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 1.8 scrub starts 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: 1.8 scrub ok 2024-03-30T15:15:58.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:58 smithi028 ceph-mon[29876]: pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 3.b scrub starts 2024-03-30T15:15:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 3.b scrub ok 2024-03-30T15:15:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 4.1 deep-scrub starts 2024-03-30T15:15:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 4.1 deep-scrub ok 2024-03-30T15:15:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 1.8 scrub starts 2024-03-30T15:15:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: 1.8 scrub ok 2024-03-30T15:15:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:58 smithi134 ceph-mon[33714]: pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:15:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.369+0000 7fba8479e700 1 -- 172.21.15.28:0/998116209 >> v1:172.21.15.28:6789/0 conn(0x7fba7c10ea90 legacy=0x7fba7c110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.369+0000 7fba8479e700 1 -- 172.21.15.28:0/998116209 shutdown_connections 2024-03-30T15:15:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.369+0000 7fba8479e700 1 -- 172.21.15.28:0/998116209 >> 172.21.15.28:0/998116209 conn(0x7fba7c0fdb30 msgr2=0x7fba7c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.369+0000 7fba8479e700 1 -- 172.21.15.28:0/998116209 shutdown_connections 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.369+0000 7fba8479e700 1 -- 172.21.15.28:0/998116209 wait complete. 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 librados: starting msgr at 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 librados: starting objecter 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 Processor -- start 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 -- start start 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 librados: setting wanted keys 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 librados: calling monclient init 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba7c1a7170 con 0x7fba7c10aef0 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba7c1a8390 con 0x7fba7c10ea90 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8479e700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fba7c1a95d0 con 0x7fba7c102260 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8253a700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fba7c102260 0x7fba7c107600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52880/0 (socket says 172.21.15.28:52880) 2024-03-30T15:15:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba8253a700 1 -- 172.21.15.28:0/2482272476 learned_addr learned my addr 172.21.15.28:0/2482272476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c10aef0 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c102260 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c10ea90 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913879335 0 0) 0x7fba7c1a7170 con 0x7fba7c10aef0 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba640031f0 con 0x7fba7c10aef0 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716046175 0 0) 0x7fba7c1a95d0 con 0x7fba7c102260 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.370+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba7c1a7170 con 0x7fba7c102260 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591825965 0 0) 0x7fba7c1a8390 con 0x7fba7c10ea90 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba7c1a95d0 con 0x7fba7c10ea90 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1217428971 0 0) 0x7fba640031f0 con 0x7fba7c10aef0 2024-03-30T15:15:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba7c1a8390 con 0x7fba7c10aef0 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fba70001960 con 0x7fba7c10aef0 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 977883671 0 0) 0x7fba7c1a7170 con 0x7fba7c102260 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba640031f0 con 0x7fba7c102260 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fba6c001b70 con 0x7fba7c102260 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4073181262 0 0) 0x7fba7c1a95d0 con 0x7fba7c10ea90 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba7c1a7170 con 0x7fba7c10ea90 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fba740043b0 con 0x7fba7c10ea90 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3116006283 0 0) 0x7fba7c1a8390 con 0x7fba7c10aef0 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 >> v1:172.21.15.28:6790/0 conn(0x7fba7c102260 legacy=0x7fba7c107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 >> v1:172.21.15.134:6789/0 conn(0x7fba7c10ea90 legacy=0x7fba7c1a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7c1aa810 con 0x7fba7c10aef0 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fba7c1a9800 con 0x7fba7c10aef0 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba8479e700 10 client.26091.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:15:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fba7c1a9cc0 con 0x7fba7c10aef0 2024-03-30T15:15:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba8479e700 1 librados: init done 2024-03-30T15:15:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fba70003b20 con 0x7fba7c10aef0 2024-03-30T15:15:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.371+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fba70004a50 con 0x7fba7c10aef0 2024-03-30T15:15:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.372+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba68005120 con 0x7fba7c10aef0 2024-03-30T15:15:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fba70003680 con 0x7fba7c10aef0 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(936..936 src has 881..936) ==== 7561+0+0 (unknown 349772208 0 0) 0x7fba700882d0 con 0x7fba7c10aef0 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 10 client.26091.objecter ms_dispatch 0x7fba7c001550 osd_map(936..936 src has 881..936) 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 3 client.26091.objecter handle_osd_map got epochs [936,936] > 0 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 3 client.26091.objecter handle_osd_map decoding full epoch 936 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 20 client.26091.objecter dump_active .. 0 homeless 2024-03-30T15:15:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.373+0000 7fba7b7fe700 10 client.26091.objecter ms_handle_connect 0x7fba64070480 2024-03-30T15:15:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.378+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fba70055760 con 0x7fba7c10aef0 2024-03-30T15:15:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.600+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fba68005410 con 0x7fba7c10aef0 2024-03-30T15:15:59.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.722+0000 7fba7b7fe700 1 -- 172.21.15.28:0/2482272476 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v937) ==== 113+0+0 (unknown 2130661345 0 0) 0x7fba700591b0 con 0x7fba7c10aef0 2024-03-30T15:15:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.725+0000 7fba8479e700 10 librados: watch_flush enter 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 10 librados: watch_flush exit 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 20 client.26091.objecter shutdown clearing up homeless session... 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 10 client.26091.objecter successfully canceled tick 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 >> v1:172.21.15.28:6800/3011431093 conn(0x7fba64070480 legacy=0x7fba64072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 >> v1:172.21.15.28:6789/0 conn(0x7fba7c10aef0 legacy=0x7fba7c1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 shutdown_connections 2024-03-30T15:15:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.726+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 >> 172.21.15.28:0/2482272476 conn(0x7fba7c0fdb30 msgr2=0x7fba7c111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:15:59.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.727+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 shutdown_connections 2024-03-30T15:15:59.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.727+0000 7fba8479e700 1 -- 172.21.15.28:0/2482272476 wait complete. 2024-03-30T15:15:59.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:15:59.727+0000 7fba8479e700 1 librados: shutdown 2024-03-30T15:15:59.729 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T15:15:59.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 12.e scrub starts 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 12.e scrub ok 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 5.1 scrub starts 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 5.1 scrub ok 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 3.16 scrub starts 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: 3.16 scrub ok 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 12.e scrub starts 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 12.e scrub ok 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 5.1 scrub starts 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 5.1 scrub ok 2024-03-30T15:15:59.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 3.16 scrub starts 2024-03-30T15:15:59.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: 3.16 scrub ok 2024-03-30T15:15:59.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:15:59 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 12.e scrub starts 2024-03-30T15:16:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 12.e scrub ok 2024-03-30T15:16:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 5.1 scrub starts 2024-03-30T15:16:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 5.1 scrub ok 2024-03-30T15:16:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 3.16 scrub starts 2024-03-30T15:16:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: 3.16 scrub ok 2024-03-30T15:16:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:15:59 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:00.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 7 1 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 5.6 scrub starts 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 5.6 scrub ok 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 6.9 scrub starts 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 6.9 scrub ok 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 4.10 scrub starts 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: 4.10 scrub ok 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: osdmap e937: 8 total, 8 up, 8 in 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:00.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 5.6 scrub starts 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 5.6 scrub ok 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 6.9 scrub starts 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 6.9 scrub ok 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 4.10 scrub starts 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: 4.10 scrub ok 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: osdmap e937: 8 total, 8 up, 8 in 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:00.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:00.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:00 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:01.021 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 5.6 scrub starts 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 5.6 scrub ok 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 6.9 scrub starts 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 6.9 scrub ok 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 4.10 scrub starts 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: 4.10 scrub ok 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2482272476' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: osdmap e937: 8 total, 8 up, 8 in 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:01.022 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:00 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:01.144 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-03-30T15:16:01.145 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T15:16:01.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:02.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[41614]: 3.13 scrub starts 2024-03-30T15:16:02.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[41614]: 3.13 scrub ok 2024-03-30T15:16:02.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[41614]: osdmap e938: 8 total, 8 up, 8 in 2024-03-30T15:16:02.013 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:02.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[29876]: 3.13 scrub starts 2024-03-30T15:16:02.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[29876]: 3.13 scrub ok 2024-03-30T15:16:02.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[29876]: osdmap e938: 8 total, 8 up, 8 in 2024-03-30T15:16:02.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:01 smithi134 ceph-mon[33714]: 3.13 scrub starts 2024-03-30T15:16:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:01 smithi134 ceph-mon[33714]: 3.13 scrub ok 2024-03-30T15:16:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:01 smithi134 ceph-mon[33714]: osdmap e938: 8 total, 8 up, 8 in 2024-03-30T15:16:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[41614]: 5.f scrub starts 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[41614]: 5.f scrub ok 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[41614]: 1.3 scrub starts 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[41614]: 1.3 scrub ok 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[41614]: pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[29876]: 5.f scrub starts 2024-03-30T15:16:03.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[29876]: 5.f scrub ok 2024-03-30T15:16:03.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[29876]: 1.3 scrub starts 2024-03-30T15:16:03.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[29876]: 1.3 scrub ok 2024-03-30T15:16:03.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:02 smithi028 ceph-mon[29876]: pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-30T15:16:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:02 smithi134 ceph-mon[33714]: 5.f scrub starts 2024-03-30T15:16:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:02 smithi134 ceph-mon[33714]: 5.f scrub ok 2024-03-30T15:16:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:02 smithi134 ceph-mon[33714]: 1.3 scrub starts 2024-03-30T15:16:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:02 smithi134 ceph-mon[33714]: 1.3 scrub ok 2024-03-30T15:16:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:02 smithi134 ceph-mon[33714]: pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.689+0000 7fd8a2587700 1 -- 172.21.15.28:0/125582950 >> v1:172.21.15.28:6790/0 conn(0x7fd89c10d960 legacy=0x7fd89c10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.689+0000 7fd8a2587700 1 -- 172.21.15.28:0/125582950 shutdown_connections 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.689+0000 7fd8a2587700 1 -- 172.21.15.28:0/125582950 >> 172.21.15.28:0/125582950 conn(0x7fd89c0fc9e0 msgr2=0x7fd89c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.689+0000 7fd8a2587700 1 -- 172.21.15.28:0/125582950 shutdown_connections 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.689+0000 7fd8a2587700 1 -- 172.21.15.28:0/125582950 wait complete. 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 librados: starting msgr at 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 librados: starting objecter 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 Processor -- start 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 -- start start 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 librados: setting wanted keys 2024-03-30T15:16:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 librados: calling monclient init 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c06a990 con 0x7fd89c10d960 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c1a8540 con 0x7fd89c101130 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8a2587700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd89c1a9720 con 0x7fd89c109dc0 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd89b7fe700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fd89c109dc0 0x7fd89c105d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52898/0 (socket says 172.21.15.28:52898) 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd89b7fe700 1 -- 172.21.15.28:0/3875785665 learned_addr learned my addr 172.21.15.28:0/3875785665 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8997fa700 10 client.?.objecter ms_handle_connect 0x7fd89c10d960 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8997fa700 10 client.?.objecter ms_handle_connect 0x7fd89c109dc0 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.690+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504140619 0 0) 0x7fd89c06a990 con 0x7fd89c10d960 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884003aa0 con 0x7fd89c10d960 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 10 client.?.objecter ms_handle_connect 0x7fd89c101130 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281026899 0 0) 0x7fd89c1a9720 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c06a990 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946820962 0 0) 0x7fd89c1a8540 con 0x7fd89c101130 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c1a9720 con 0x7fd89c101130 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 733241330 0 0) 0x7fd89c06a990 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c1a8540 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd890001960 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158332378 0 0) 0x7fd884003aa0 con 0x7fd89c10d960 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd89c06a990 con 0x7fd89c10d960 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd894004980 con 0x7fd89c10d960 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 55257595 0 0) 0x7fd89c1a9720 con 0x7fd89c101130 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd884003aa0 con 0x7fd89c101130 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd88c001830 con 0x7fd89c101130 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3234899747 0 0) 0x7fd89c1a8540 con 0x7fd89c109dc0 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 >> v1:172.21.15.134:6789/0 conn(0x7fd89c101130 legacy=0x7fd89c105670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 >> v1:172.21.15.28:6789/0 conn(0x7fd89c10d960 legacy=0x7fd89c1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c1aa900 con 0x7fd89c109dc0 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c1a98f0 con 0x7fd89c109dc0 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8a2587700 10 client.59300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd89c1a9db0 con 0x7fd89c109dc0 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8a2587700 1 librados: init done 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fd890001c50 con 0x7fd89c109dc0 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.691+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fd890005d10 con 0x7fd89c109dc0 2024-03-30T15:16:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.692+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd888005120 con 0x7fd89c109dc0 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fd890003930 con 0x7fd89c109dc0 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(938..938 src has 881..938) ==== 7561+0+0 (unknown 178391402 0 0) 0x7fd8900893e0 con 0x7fd89c109dc0 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 10 client.59300.objecter ms_dispatch 0x7fd89c001550 osd_map(938..938 src has 881..938) 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 3 client.59300.objecter handle_osd_map got epochs [938,938] > 0 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 3 client.59300.objecter handle_osd_map decoding full epoch 938 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.693+0000 7fd8997fa700 20 client.59300.objecter dump_active .. 0 homeless 2024-03-30T15:16:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.694+0000 7fd8997fa700 10 client.59300.objecter ms_handle_connect 0x7fd884070470 2024-03-30T15:16:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.698+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd890056890 con 0x7fd89c109dc0 2024-03-30T15:16:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.917+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7fd888005c20 con 0x7fd89c109dc0 2024-03-30T15:16:03.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.974+0000 7fd8997fa700 1 -- 172.21.15.28:0/3875785665 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v939) ==== 133+0+0 (unknown 3016719960 0 0) 0x7fd89005a2e0 con 0x7fd89c109dc0 2024-03-30T15:16:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 10 librados: watch_flush enter 2024-03-30T15:16:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 10 librados: watch_flush exit 2024-03-30T15:16:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 20 client.59300.objecter shutdown clearing up homeless session... 2024-03-30T15:16:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 10 client.59300.objecter successfully canceled tick 2024-03-30T15:16:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 >> v1:172.21.15.28:6800/3011431093 conn(0x7fd884070470 legacy=0x7fd884072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 >> v1:172.21.15.28:6790/0 conn(0x7fd89c109dc0 legacy=0x7fd89c105d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 shutdown_connections 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 >> 172.21.15.28:0/3875785665 conn(0x7fd89c0fc9e0 msgr2=0x7fd89c110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 shutdown_connections 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 -- 172.21.15.28:0/3875785665 wait complete. 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:03.976+0000 7fd8a2587700 1 librados: shutdown 2024-03-30T15:16:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T15:16:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[29876]: 5.e scrub starts 2024-03-30T15:16:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[29876]: 5.e scrub ok 2024-03-30T15:16:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3875785665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[29876]: from='client.59300 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[41614]: 5.e scrub starts 2024-03-30T15:16:04.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[41614]: 5.e scrub ok 2024-03-30T15:16:04.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3875785665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:03 smithi028 ceph-mon[41614]: from='client.59300 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:03 smithi134 ceph-mon[33714]: 5.e scrub starts 2024-03-30T15:16:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:03 smithi134 ceph-mon[33714]: 5.e scrub ok 2024-03-30T15:16:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:03 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3875785665' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:03 smithi134 ceph-mon[33714]: from='client.59300 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-30T15:16:04.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 4 1 2024-03-30T15:16:04.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 12.c scrub starts 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 12.c scrub ok 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 4.8 scrub starts 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 4.8 scrub ok 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: from='client.59300 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: osdmap e939: 8 total, 8 up, 8 in 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 4.f scrub starts 2024-03-30T15:16:04.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[29876]: 4.f scrub ok 2024-03-30T15:16:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:05.306 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 12.c scrub starts 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 12.c scrub ok 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 4.8 scrub starts 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 4.8 scrub ok 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: from='client.59300 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: osdmap e939: 8 total, 8 up, 8 in 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 4.f scrub starts 2024-03-30T15:16:05.307 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:04 smithi028 ceph-mon[41614]: 4.f scrub ok 2024-03-30T15:16:05.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 12.c scrub starts 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 12.c scrub ok 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 4.8 scrub starts 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 4.8 scrub ok 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: from='client.59300 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: osdmap e939: 8 total, 8 up, 8 in 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 4.f scrub starts 2024-03-30T15:16:05.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:04 smithi134 ceph-mon[33714]: 4.f scrub ok 2024-03-30T15:16:06.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:05 smithi028 ceph-mon[41614]: 6.c scrub starts 2024-03-30T15:16:06.018 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:05 smithi028 ceph-mon[41614]: 6.c scrub ok 2024-03-30T15:16:06.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:05 smithi028 ceph-mon[29876]: 6.c scrub starts 2024-03-30T15:16:06.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:05 smithi028 ceph-mon[29876]: 6.c scrub ok 2024-03-30T15:16:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:05 smithi134 ceph-mon[33714]: 6.c scrub starts 2024-03-30T15:16:06.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:05 smithi134 ceph-mon[33714]: 6.c scrub ok 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.840+0000 7f1094fcf700 1 -- 172.21.15.28:0/3763277954 >> v1:172.21.15.28:6789/0 conn(0x7f1090102250 legacy=0x7f1090102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.840+0000 7f1094fcf700 1 -- 172.21.15.28:0/3763277954 shutdown_connections 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.840+0000 7f1094fcf700 1 -- 172.21.15.28:0/3763277954 >> 172.21.15.28:0/3763277954 conn(0x7f10900fdb20 msgr2=0x7f10900fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.840+0000 7f1094fcf700 1 -- 172.21.15.28:0/3763277954 shutdown_connections 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.840+0000 7f1094fcf700 1 -- 172.21.15.28:0/3763277954 wait complete. 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 librados: starting msgr at 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 librados: starting objecter 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 Processor -- start 2024-03-30T15:16:06.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 -- start start 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 librados: setting wanted keys 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.841+0000 7f1094fcf700 1 librados: calling monclient init 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f1094fcf700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10901a70d0 con 0x7f1090102250 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f1094fcf700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10901a82f0 con 0x7f109010aee0 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f1094fcf700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f10901a9530 con 0x7f109010ea80 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f108e59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1090102250 0x7f1090107560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44736/0 (socket says 172.21.15.28:44736) 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f108e59c700 1 -- 172.21.15.28:0/1304715724 learned_addr learned my addr 172.21.15.28:0/1304715724 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f108ed9d700 1 --1- 172.21.15.28:0/1304715724 >> v1:172.21.15.28:6790/0 conn(0x7f109010ea80 0x7f10901a5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:52904/0 (socket says 172.21.15.28:52904) 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter ms_handle_connect 0x7f1090102250 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter ms_handle_connect 0x7f109010ea80 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter ms_handle_connect 0x7f109010aee0 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:06.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862900003 0 0) 0x7f10901a9530 con 0x7f109010ea80 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10780031f0 con 0x7f109010ea80 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011371139 0 0) 0x7f10901a70d0 con 0x7f1090102250 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10901a9530 con 0x7f1090102250 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430070350 0 0) 0x7f10901a82f0 con 0x7f109010aee0 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10901a70d0 con 0x7f109010aee0 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 162415352 0 0) 0x7f10780031f0 con 0x7f109010ea80 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10901a82f0 con 0x7f109010ea80 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.842+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1088001bb0 con 0x7f109010ea80 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 279692036 0 0) 0x7f10901a70d0 con 0x7f109010aee0 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10780031f0 con 0x7f109010aee0 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f10840017f0 con 0x7f109010aee0 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4089300077 0 0) 0x7f10901a9530 con 0x7f1090102250 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10901a70d0 con 0x7f1090102250 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f10800045a0 con 0x7f1090102250 2024-03-30T15:16:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1351821923 0 0) 0x7f10901a82f0 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 >> v1:172.21.15.134:6789/0 conn(0x7f109010aee0 legacy=0x7f10901a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 >> v1:172.21.15.28:6789/0 conn(0x7f1090102250 legacy=0x7f1090107560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10901aa770 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10901a9760 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f1094fcf700 10 client.59306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f10901a9c20 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f1094fcf700 1 librados: init done 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f1088003ec0 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.843+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1088005f00 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.844+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1070005120 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.844+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f108801c820 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(939..939 src has 881..939) ==== 7561+0+0 (unknown 94339335 0 0) 0x7f10880896c0 con 0x7f109010ea80 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 10 client.59306.objecter ms_dispatch 0x7f1090001550 osd_map(939..939 src has 881..939) 2024-03-30T15:16:06.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 3 client.59306.objecter handle_osd_map got epochs [939,939] > 0 2024-03-30T15:16:06.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 3 client.59306.objecter handle_osd_map decoding full epoch 939 2024-03-30T15:16:06.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 20 client.59306.objecter dump_active .. 0 homeless 2024-03-30T15:16:06.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.845+0000 7f107f7fe700 10 client.59306.objecter ms_handle_connect 0x7f1078070420 2024-03-30T15:16:06.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:06.849+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1088056b70 con 0x7f109010ea80 2024-03-30T15:16:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.069+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f1070005410 con 0x7f109010ea80 2024-03-30T15:16:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:06 smithi134 ceph-mon[33714]: 5.1e deep-scrub starts 2024-03-30T15:16:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:06 smithi134 ceph-mon[33714]: 5.1e deep-scrub ok 2024-03-30T15:16:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:06 smithi134 ceph-mon[33714]: pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-30T15:16:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:06 smithi134 ceph-mon[33714]: 4.13 deep-scrub starts 2024-03-30T15:16:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:06 smithi134 ceph-mon[33714]: 4.13 deep-scrub ok 2024-03-30T15:16:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[29876]: 5.1e deep-scrub starts 2024-03-30T15:16:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[29876]: 5.1e deep-scrub ok 2024-03-30T15:16:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[29876]: pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-30T15:16:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[29876]: 4.13 deep-scrub starts 2024-03-30T15:16:07.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[29876]: 4.13 deep-scrub ok 2024-03-30T15:16:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[41614]: 5.1e deep-scrub starts 2024-03-30T15:16:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[41614]: 5.1e deep-scrub ok 2024-03-30T15:16:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[41614]: pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-30T15:16:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[41614]: 4.13 deep-scrub starts 2024-03-30T15:16:07.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:06 smithi028 ceph-mon[41614]: 4.13 deep-scrub ok 2024-03-30T15:16:07.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.992+0000 7f107f7fe700 1 -- 172.21.15.28:0/1304715724 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v940) ==== 113+0+0 (unknown 659107491 0 0) 0x7f108805a5c0 con 0x7f109010ea80 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 10 librados: watch_flush enter 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 10 librados: watch_flush exit 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 20 client.59306.objecter shutdown clearing up homeless session... 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 10 client.59306.objecter successfully canceled tick 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 >> v1:172.21.15.28:6800/3011431093 conn(0x7f1078070420 legacy=0x7f10780728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 >> v1:172.21.15.28:6790/0 conn(0x7f109010ea80 legacy=0x7f10901a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.995+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 shutdown_connections 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.996+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 >> 172.21.15.28:0/1304715724 conn(0x7f10900fdb20 msgr2=0x7f1090111f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.996+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 shutdown_connections 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.996+0000 7f1094fcf700 1 -- 172.21.15.28:0/1304715724 wait complete. 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:07.996+0000 7f1094fcf700 1 librados: shutdown 2024-03-30T15:16:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T15:16:08.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1304715724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[29876]: from='client.59306 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[29876]: 6.1e scrub starts 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[29876]: 6.1e scrub ok 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1304715724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[41614]: from='client.59306 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[41614]: 6.1e scrub starts 2024-03-30T15:16:08.244 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[41614]: 6.1e scrub ok 2024-03-30T15:16:08.245 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:07 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:07 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1304715724' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:07 smithi134 ceph-mon[33714]: from='client.59306 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:07 smithi134 ceph-mon[33714]: 6.1e scrub starts 2024-03-30T15:16:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:07 smithi134 ceph-mon[33714]: 6.1e scrub ok 2024-03-30T15:16:08.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:07 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:08.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:08.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 4 1 2024-03-30T15:16:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: 5.3 scrub starts 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: 5.3 scrub ok 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: 4.0 scrub starts 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: 4.0 scrub ok 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: from='client.59306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: osdmap e940: 8 total, 8 up, 8 in 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[29876]: pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:16:09.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: 5.3 scrub starts 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: 5.3 scrub ok 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: 4.0 scrub starts 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: 4.0 scrub ok 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: from='client.59306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: osdmap e940: 8 total, 8 up, 8 in 2024-03-30T15:16:09.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:08 smithi028 ceph-mon[41614]: pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:16:09.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: 5.3 scrub starts 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: 5.3 scrub ok 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: 4.0 scrub starts 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: 4.0 scrub ok 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: from='client.59306 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: osdmap e940: 8 total, 8 up, 8 in 2024-03-30T15:16:09.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:08 smithi134 ceph-mon[33714]: pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-30T15:16:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[41614]: 5.0 scrub starts 2024-03-30T15:16:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[41614]: 5.0 scrub ok 2024-03-30T15:16:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:10.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:10.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[29876]: 5.0 scrub starts 2024-03-30T15:16:10.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[29876]: 5.0 scrub ok 2024-03-30T15:16:10.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:10.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:09 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:09 smithi134 ceph-mon[33714]: 5.0 scrub starts 2024-03-30T15:16:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:09 smithi134 ceph-mon[33714]: 5.0 scrub ok 2024-03-30T15:16:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:09 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:10.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:09 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.845+0000 7f6dcb71a700 1 -- 172.21.15.28:0/2111232600 >> v1:172.21.15.28:6789/0 conn(0x7f6dc4074560 legacy=0x7f6dc4074940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.845+0000 7f6dcb71a700 1 -- 172.21.15.28:0/2111232600 shutdown_connections 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.845+0000 7f6dcb71a700 1 -- 172.21.15.28:0/2111232600 >> 172.21.15.28:0/2111232600 conn(0x7f6dc40fdb20 msgr2=0x7f6dc40fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.845+0000 7f6dcb71a700 1 -- 172.21.15.28:0/2111232600 shutdown_connections 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.845+0000 7f6dcb71a700 1 -- 172.21.15.28:0/2111232600 wait complete. 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 librados: starting msgr at 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 librados: starting objecter 2024-03-30T15:16:10.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 Processor -- start 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 -- start start 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 librados: setting wanted keys 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 librados: calling monclient init 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dc410f780 con 0x7f6dc4075050 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dc41aa850 con 0x7f6dc4110cc0 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dcb71a700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6dc41aba30 con 0x7f6dc4074560 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc94b6700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f6dc4074560 0x7f6dc410c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:46532/0 (socket says 172.21.15.28:46532) 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc94b6700 1 -- 172.21.15.28:0/3570020746 learned_addr learned my addr 172.21.15.28:0/3570020746 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc8cb5700 1 --1- 172.21.15.28:0/3570020746 >> v1:172.21.15.28:6789/0 conn(0x7f6dc4075050 0x7f6dc410cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60794/0 (socket says 172.21.15.28:60794) 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter ms_handle_connect 0x7f6dc4074560 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter ms_handle_connect 0x7f6dc4075050 2024-03-30T15:16:10.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter ms_handle_connect 0x7f6dc4110cc0 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3149523403 0 0) 0x7f6dc41aba30 con 0x7f6dc4074560 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dac0031f0 con 0x7f6dc4074560 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955915383 0 0) 0x7f6dc410f780 con 0x7f6dc4075050 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.846+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dc41aba30 con 0x7f6dc4075050 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3763120338 0 0) 0x7f6dc41aa850 con 0x7f6dc4110cc0 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dc410f780 con 0x7f6dc4110cc0 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4170258215 0 0) 0x7f6dac0031f0 con 0x7f6dc4074560 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dc41aa850 con 0x7f6dc4074560 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6dbc004700 con 0x7f6dc4074560 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 352976387 0 0) 0x7f6dc41aba30 con 0x7f6dc4075050 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6dac0031f0 con 0x7f6dc4075050 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6db8001830 con 0x7f6dc4075050 2024-03-30T15:16:10.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3092553995 0 0) 0x7f6dc41aa850 con 0x7f6dc4074560 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 >> v1:172.21.15.134:6789/0 conn(0x7f6dc4110cc0 legacy=0x7f6dc41a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 >> v1:172.21.15.28:6789/0 conn(0x7f6dc4075050 legacy=0x7f6dc410cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc41acc10 con 0x7f6dc4074560 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc41a98a0 con 0x7f6dc4074560 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dcb71a700 10 client.59312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6dc41a9d60 con 0x7f6dc4074560 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dcb71a700 1 librados: init done 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6dbc001460 con 0x7f6dc4074560 2024-03-30T15:16:10.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.847+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6dbc005200 con 0x7f6dc4074560 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.848+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db0005120 con 0x7f6dc4074560 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6dbc005200 con 0x7f6dc4074560 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(941..941 src has 881..941) ==== 7561+0+0 (unknown 655536233 0 0) 0x7f6dbc089820 con 0x7f6dc4074560 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 10 client.59312.objecter ms_dispatch 0x7f6dc4001550 osd_map(941..941 src has 881..941) 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 3 client.59312.objecter handle_osd_map got epochs [941,941] > 0 2024-03-30T15:16:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 3 client.59312.objecter handle_osd_map decoding full epoch 941 2024-03-30T15:16:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 20 client.59312.objecter dump_active .. 0 homeless 2024-03-30T15:16:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.849+0000 7f6dc27fc700 10 client.59312.objecter ms_handle_connect 0x7f6dac0701b0 2024-03-30T15:16:10.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:10.854+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6dbc001eb0 con 0x7f6dc4074560 2024-03-30T15:16:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:11.072+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7f6db0005c20 con 0x7f6dc4074560 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 6.10 scrub starts 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 6.10 scrub ok 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 3.5 scrub starts 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: 3.5 scrub ok 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: osdmap e941: 8 total, 8 up, 8 in 2024-03-30T15:16:11.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[29876]: pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 6.10 scrub starts 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 6.10 scrub ok 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 3.5 scrub starts 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: 3.5 scrub ok 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: osdmap e941: 8 total, 8 up, 8 in 2024-03-30T15:16:11.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:11 smithi028 ceph-mon[41614]: pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-30T15:16:11.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:11.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 6.10 scrub starts 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 6.10 scrub ok 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 3.5 scrub starts 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: 3.5 scrub ok 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: osdmap e941: 8 total, 8 up, 8 in 2024-03-30T15:16:11.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:11 smithi134 ceph-mon[33714]: pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-03-30T15:16:12.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.012+0000 7f6dc27fc700 1 -- 172.21.15.28:0/3570020746 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v942) ==== 133+0+0 (unknown 2126293021 0 0) 0x7f6dbc056cd0 con 0x7f6dc4074560 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 10 librados: watch_flush enter 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 10 librados: watch_flush exit 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 20 client.59312.objecter shutdown clearing up homeless session... 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 10 client.59312.objecter successfully canceled tick 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6dac0701b0 legacy=0x7f6dac072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 >> v1:172.21.15.28:6790/0 conn(0x7f6dc4074560 legacy=0x7f6dc410c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 shutdown_connections 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 >> 172.21.15.28:0/3570020746 conn(0x7f6dc40fdb20 msgr2=0x7f6dc41103c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.015+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 shutdown_connections 2024-03-30T15:16:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.016+0000 7f6dcb71a700 1 -- 172.21.15.28:0/3570020746 wait complete. 2024-03-30T15:16:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:12.016+0000 7f6dcb71a700 1 librados: shutdown 2024-03-30T15:16:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T15:16:12.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:12.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3570020746' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[41614]: from='client.59312 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:12.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3570020746' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:12 smithi028 ceph-mon[29876]: from='client.59312 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:12 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:12.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:12 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3570020746' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:12 smithi134 ceph-mon[33714]: from='client.59312 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-30T15:16:12.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 2 1 2024-03-30T15:16:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:13.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: 3.1f scrub starts 2024-03-30T15:16:13.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: 3.1f scrub ok 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: from='client.59312 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: osdmap e942: 8 total, 8 up, 8 in 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: 5.12 scrub starts 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[41614]: 5.12 scrub ok 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: 3.1f scrub starts 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: 3.1f scrub ok 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: from='client.59312 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: osdmap e942: 8 total, 8 up, 8 in 2024-03-30T15:16:13.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:13.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: 5.12 scrub starts 2024-03-30T15:16:13.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:13 smithi028 ceph-mon[29876]: 5.12 scrub ok 2024-03-30T15:16:13.430 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=cleanup t=2024-03-30T15:16:13.071823028Z level=info msg="Completed cleanup jobs" duration=3.555545ms 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: 3.1f scrub starts 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: 3.1f scrub ok 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: from='client.59312 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: osdmap e942: 8 total, 8 up, 8 in 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: 5.12 scrub starts 2024-03-30T15:16:13.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:13 smithi134 ceph-mon[33714]: 5.12 scrub ok 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 6.14 scrub starts 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 6.14 scrub ok 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 5.16 deep-scrub starts 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 5.16 deep-scrub ok 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 6.14 scrub starts 2024-03-30T15:16:14.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 6.14 scrub ok 2024-03-30T15:16:14.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 5.16 deep-scrub starts 2024-03-30T15:16:14.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 5.16 deep-scrub ok 2024-03-30T15:16:14.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:16:14.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:14 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:16:14.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 6.14 scrub starts 2024-03-30T15:16:14.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 6.14 scrub ok 2024-03-30T15:16:14.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 5.16 deep-scrub starts 2024-03-30T15:16:14.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 5.16 deep-scrub ok 2024-03-30T15:16:14.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:16:14.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:14 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:16:14.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.869+0000 7efc4bba5700 1 -- 172.21.15.28:0/1277180043 >> v1:172.21.15.28:6789/0 conn(0x7efc44101180 legacy=0x7efc44101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:14.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 -- 172.21.15.28:0/1277180043 shutdown_connections 2024-03-30T15:16:14.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 -- 172.21.15.28:0/1277180043 >> 172.21.15.28:0/1277180043 conn(0x7efc440fca30 msgr2=0x7efc440fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:14.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 -- 172.21.15.28:0/1277180043 shutdown_connections 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 -- 172.21.15.28:0/1277180043 wait complete. 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 librados: starting msgr at 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 librados: starting objecter 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 Processor -- start 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 -- start start 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 librados: setting wanted keys 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.870+0000 7efc4bba5700 1 librados: calling monclient init 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc4bba5700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc441a6030 con 0x7efc44109e10 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc4bba5700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc441a7250 con 0x7efc44101180 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc4bba5700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efc441a8490 con 0x7efc4410d9b0 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc49140700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efc44109e10 0x7efc441a0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60808/0 (socket says 172.21.15.28:60808) 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc49140700 1 -- 172.21.15.28:0/1246448588 learned_addr learned my addr 172.21.15.28:0/1246448588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc4a142700 1 --1- 172.21.15.28:0/1246448588 >> v1:172.21.15.28:6790/0 conn(0x7efc4410d9b0 0x7efc441a46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:46542/0 (socket says 172.21.15.28:46542) 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter ms_handle_connect 0x7efc44109e10 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter ms_handle_connect 0x7efc44101180 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter ms_handle_connect 0x7efc4410d9b0 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4138270189 0 0) 0x7efc441a6030 con 0x7efc44109e10 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc300031f0 con 0x7efc44109e10 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3821724344 0 0) 0x7efc441a7250 con 0x7efc44101180 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc441a6030 con 0x7efc44101180 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741754535 0 0) 0x7efc441a8490 con 0x7efc4410d9b0 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc441a7250 con 0x7efc4410d9b0 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 381792115 0 0) 0x7efc441a6030 con 0x7efc44101180 2024-03-30T15:16:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc441a8490 con 0x7efc44101180 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc40004590 con 0x7efc44101180 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3492497854 0 0) 0x7efc300031f0 con 0x7efc44109e10 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc441a6030 con 0x7efc44109e10 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1791835663 0 0) 0x7efc441a7250 con 0x7efc4410d9b0 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc300031f0 con 0x7efc4410d9b0 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc340018b0 con 0x7efc44109e10 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc3c001490 con 0x7efc4410d9b0 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1642007675 0 0) 0x7efc441a8490 con 0x7efc44101180 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 >> v1:172.21.15.28:6790/0 conn(0x7efc4410d9b0 legacy=0x7efc441a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 >> v1:172.21.15.28:6789/0 conn(0x7efc44109e10 legacy=0x7efc441a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.871+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441a96d0 con 0x7efc44101180 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc441a7480 con 0x7efc44101180 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc4bba5700 10 client.51304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efc441a7940 con 0x7efc44101180 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc4bba5700 1 librados: init done 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7efc40001af0 con 0x7efc44101180 2024-03-30T15:16:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7efc40004d50 con 0x7efc44101180 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.872+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc28005120 con 0x7efc44101180 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.873+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7efc40003ee0 con 0x7efc44101180 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(942..942 src has 881..942) ==== 7561+0+0 (unknown 911485150 0 0) 0x7efc40088310 con 0x7efc44101180 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 10 client.51304.objecter ms_dispatch 0x7efc44001550 osd_map(942..942 src has 881..942) 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 3 client.51304.objecter handle_osd_map got epochs [942,942] > 0 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 3 client.51304.objecter handle_osd_map decoding full epoch 942 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 20 client.51304.objecter dump_active .. 0 homeless 2024-03-30T15:16:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.874+0000 7efc3affd700 10 client.51304.objecter ms_handle_connect 0x7efc300703c0 2024-03-30T15:16:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:14.878+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efc40055760 con 0x7efc44101180 2024-03-30T15:16:15.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:15.096+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7efc28005410 con 0x7efc44101180 2024-03-30T15:16:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:16:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:16:15.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[29876]: pgmap v1487: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-30T15:16:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:16:15.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:16:15.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:15 smithi028 ceph-mon[41614]: pgmap v1487: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-30T15:16:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:15 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:16:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:15 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:16:15.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:15 smithi134 ceph-mon[33714]: pgmap v1487: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-30T15:16:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.025+0000 7efc3affd700 1 -- 172.21.15.28:0/1246448588 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v943) ==== 113+0+0 (unknown 659328787 0 0) 0x7efc400591b0 con 0x7efc44101180 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 10 librados: watch_flush enter 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 10 librados: watch_flush exit 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 20 client.51304.objecter shutdown clearing up homeless session... 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 10 client.51304.objecter successfully canceled tick 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 >> v1:172.21.15.28:6800/3011431093 conn(0x7efc300703c0 legacy=0x7efc30072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 >> v1:172.21.15.134:6789/0 conn(0x7efc44101180 legacy=0x7efc441064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 shutdown_connections 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.029+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 >> 172.21.15.28:0/1246448588 conn(0x7efc440fca30 msgr2=0x7efc44110b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.030+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 shutdown_connections 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.030+0000 7efc4bba5700 1 -- 172.21.15.28:0/1246448588 wait complete. 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:16.030+0000 7efc4bba5700 1 librados: shutdown 2024-03-30T15:16:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T15:16:16.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 1.7 scrub starts 2024-03-30T15:16:16.342 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 1.7 scrub ok 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 3.6 scrub starts 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 3.6 scrub ok 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1246448588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: from='client.51304 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 4.11 scrub starts 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[41614]: 4.11 scrub ok 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 1.7 scrub starts 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 1.7 scrub ok 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 3.6 scrub starts 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 3.6 scrub ok 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1246448588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.343 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: from='client.51304 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 4.11 scrub starts 2024-03-30T15:16:16.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:16 smithi028 ceph-mon[29876]: 4.11 scrub ok 2024-03-30T15:16:16.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 1.7 scrub starts 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 1.7 scrub ok 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 3.6 scrub starts 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 3.6 scrub ok 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1246448588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: from='client.51304 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 4.11 scrub starts 2024-03-30T15:16:16.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:16 smithi134 ceph-mon[33714]: 4.11 scrub ok 2024-03-30T15:16:16.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 2 1 2024-03-30T15:16:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:17.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: 3.1c scrub starts 2024-03-30T15:16:17.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: 3.1c scrub ok 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: from='client.51304 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: osdmap e943: 8 total, 8 up, 8 in 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: pgmap v1489: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: 4.1b scrub starts 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[29876]: 4.1b scrub ok 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: 3.1c scrub starts 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: 3.1c scrub ok 2024-03-30T15:16:17.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: from='client.51304 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:17.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: osdmap e943: 8 total, 8 up, 8 in 2024-03-30T15:16:17.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: pgmap v1489: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-03-30T15:16:17.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: 4.1b scrub starts 2024-03-30T15:16:17.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:17 smithi028 ceph-mon[41614]: 4.1b scrub ok 2024-03-30T15:16:17.430 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: 3.1c scrub starts 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: 3.1c scrub ok 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: from='client.51304 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: osdmap e943: 8 total, 8 up, 8 in 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: pgmap v1489: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 820 B/s rd, 0 op/s 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: 4.1b scrub starts 2024-03-30T15:16:17.431 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:17 smithi134 ceph-mon[33714]: 4.1b scrub ok 2024-03-30T15:16:18.600 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:18.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 6.2 scrub starts 2024-03-30T15:16:18.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 6.2 scrub ok 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 5.1f scrub starts 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 5.1f scrub ok 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 4.16 scrub starts 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: 4.16 scrub ok 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[29876]: pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 6.2 scrub starts 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 6.2 scrub ok 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 5.1f scrub starts 2024-03-30T15:16:18.858 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 5.1f scrub ok 2024-03-30T15:16:18.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 4.16 scrub starts 2024-03-30T15:16:18.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: 4.16 scrub ok 2024-03-30T15:16:18.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:18 smithi028 ceph-mon[41614]: pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.855+0000 7f78f0d64700 1 -- 172.21.15.28:0/1270334976 >> v1:172.21.15.28:6790/0 conn(0x7f78ec10d960 legacy=0x7f78ec10fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.855+0000 7f78f0d64700 1 -- 172.21.15.28:0/1270334976 shutdown_connections 2024-03-30T15:16:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.855+0000 7f78f0d64700 1 -- 172.21.15.28:0/1270334976 >> 172.21.15.28:0/1270334976 conn(0x7f78ec0fc9e0 msgr2=0x7f78ec0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 -- 172.21.15.28:0/1270334976 shutdown_connections 2024-03-30T15:16:18.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 -- 172.21.15.28:0/1270334976 wait complete. 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 librados: starting msgr at 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 librados: starting objecter 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 Processor -- start 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 -- start start 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 librados: setting wanted keys 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.856+0000 7f78f0d64700 1 librados: calling monclient init 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78f0d64700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78ec1a7530 con 0x7f78ec101130 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78f0d64700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78ec1a8710 con 0x7f78ec109dc0 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78f0d64700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78ec1a98f0 con 0x7f78ec10d960 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78ead9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f78ec10d960 0x7f78ec1a5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40834/0 (socket says 172.21.15.28:40834) 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78ead9d700 1 -- 172.21.15.28:0/188416145 learned_addr learned my addr 172.21.15.28:0/188416145 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78ea59c700 1 --1- 172.21.15.28:0/188416145 >> v1:172.21.15.28:6789/0 conn(0x7f78ec101130 0x7f78ec1058d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40706/0 (socket says 172.21.15.28:40706) 2024-03-30T15:16:18.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter ms_handle_connect 0x7f78ec10d960 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter ms_handle_connect 0x7f78ec101130 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter ms_handle_connect 0x7f78ec109dc0 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960015967 0 0) 0x7f78ec1a98f0 con 0x7f78ec10d960 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d40031f0 con 0x7f78ec10d960 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968992459 0 0) 0x7f78ec1a7530 con 0x7f78ec101130 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78ec1a98f0 con 0x7f78ec101130 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035326659 0 0) 0x7f78ec1a8710 con 0x7f78ec109dc0 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78ec1a7530 con 0x7f78ec109dc0 2024-03-30T15:16:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1306509783 0 0) 0x7f78ec1a98f0 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78ec1a8710 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78dc0018b0 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 644471135 0 0) 0x7f78d40031f0 con 0x7f78ec10d960 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78ec1a98f0 con 0x7f78ec10d960 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78e40017b0 con 0x7f78ec10d960 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 728767595 0 0) 0x7f78ec1a7530 con 0x7f78ec109dc0 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78d40031f0 con 0x7f78ec109dc0 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78e0001830 con 0x7f78ec109dc0 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2768656622 0 0) 0x7f78ec1a8710 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 >> v1:172.21.15.28:6790/0 conn(0x7f78ec10d960 legacy=0x7f78ec1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 >> v1:172.21.15.134:6789/0 conn(0x7f78ec109dc0 legacy=0x7f78ec105fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.857+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec1aaad0 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78ec1a7760 con 0x7f78ec101130 2024-03-30T15:16:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78f0d64700 10 client.26121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78ec1a7c20 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78f0d64700 1 librados: init done 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f78dc001c60 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.858+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f78dc005df0 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.859+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78cc005120 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.859+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f78dc003950 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(943..943 src has 881..943) ==== 7561+0+0 (unknown 963223731 0 0) 0x7f78dc089430 con 0x7f78ec101130 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 10 client.26121.objecter ms_dispatch 0x7f78ec001550 osd_map(943..943 src has 881..943) 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 3 client.26121.objecter handle_osd_map got epochs [943,943] > 0 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 3 client.26121.objecter handle_osd_map decoding full epoch 943 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 20 client.26121.objecter dump_active .. 0 homeless 2024-03-30T15:16:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.860+0000 7f78db7fe700 10 client.26121.objecter ms_handle_connect 0x7f78d40703c0 2024-03-30T15:16:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:18.864+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f78dc0568e0 con 0x7f78ec101130 2024-03-30T15:16:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 6.2 scrub starts 2024-03-30T15:16:18.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 6.2 scrub ok 2024-03-30T15:16:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 5.1f scrub starts 2024-03-30T15:16:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 5.1f scrub ok 2024-03-30T15:16:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 4.16 scrub starts 2024-03-30T15:16:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: 4.16 scrub ok 2024-03-30T15:16:18.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:18 smithi134 ceph-mon[33714]: pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.081+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7f78cc005c20 con 0x7f78ec101130 2024-03-30T15:16:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.634+0000 7f78db7fe700 1 -- 172.21.15.28:0/188416145 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v944) ==== 133+0+0 (unknown 1620335912 0 0) 0x7f78dc05a330 con 0x7f78ec101130 2024-03-30T15:16:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.638+0000 7f78f0d64700 10 librados: watch_flush enter 2024-03-30T15:16:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.638+0000 7f78f0d64700 10 librados: watch_flush exit 2024-03-30T15:16:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.638+0000 7f78f0d64700 20 client.26121.objecter shutdown clearing up homeless session... 2024-03-30T15:16:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.638+0000 7f78f0d64700 10 client.26121.objecter successfully canceled tick 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.640+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 >> v1:172.21.15.28:6800/3011431093 conn(0x7f78d40703c0 legacy=0x7f78d4072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.640+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 >> v1:172.21.15.28:6789/0 conn(0x7f78ec101130 legacy=0x7f78ec1058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.641+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 shutdown_connections 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.641+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 >> 172.21.15.28:0/188416145 conn(0x7f78ec0fc9e0 msgr2=0x7f78ec1105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.641+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 shutdown_connections 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.641+0000 7f78f0d64700 1 -- 172.21.15.28:0/188416145 wait complete. 2024-03-30T15:16:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:19.641+0000 7f78f0d64700 1 librados: shutdown 2024-03-30T15:16:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T15:16:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:19 smithi134 ceph-mon[33714]: 4.15 scrub starts 2024-03-30T15:16:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:19 smithi134 ceph-mon[33714]: 4.15 scrub ok 2024-03-30T15:16:19.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:19 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:19.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[29876]: 4.15 scrub starts 2024-03-30T15:16:19.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[29876]: 4.15 scrub ok 2024-03-30T15:16:19.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:19.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[41614]: 4.15 scrub starts 2024-03-30T15:16:19.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[41614]: 4.15 scrub ok 2024-03-30T15:16:19.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:19 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-30T15:16:20.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd reweight 0 1 2024-03-30T15:16:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 3.d deep-scrub starts 2024-03-30T15:16:20.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 3.d deep-scrub ok 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: osdmap e944: 8 total, 8 up, 8 in 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 6.19 scrub starts 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 6.19 scrub ok 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 3.17 scrub starts 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: 3.17 scrub ok 2024-03-30T15:16:20.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:20 smithi134 ceph-mon[33714]: pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 3.d deep-scrub starts 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 3.d deep-scrub ok 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: osdmap e944: 8 total, 8 up, 8 in 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 6.19 scrub starts 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 6.19 scrub ok 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 3.17 scrub starts 2024-03-30T15:16:20.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: 3.17 scrub ok 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[29876]: pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 3.d deep-scrub starts 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 3.d deep-scrub ok 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/188416145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: osdmap e944: 8 total, 8 up, 8 in 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 6.19 scrub starts 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 6.19 scrub ok 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 3.17 scrub starts 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: 3.17 scrub ok 2024-03-30T15:16:20.977 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:20 smithi028 ceph-mon[41614]: pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:21.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[41614]: 1.0 scrub starts 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[41614]: 1.0 scrub ok 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[29876]: 1.0 scrub starts 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[29876]: 1.0 scrub ok 2024-03-30T15:16:21.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:21 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:21 smithi134 ceph-mon[33714]: 1.0 scrub starts 2024-03-30T15:16:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:21 smithi134 ceph-mon[33714]: 1.0 scrub ok 2024-03-30T15:16:21.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:21 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 -- 172.21.15.28:0/3435090205 >> v1:172.21.15.28:6790/0 conn(0x7f126c101130 legacy=0x7f126c101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 -- 172.21.15.28:0/3435090205 shutdown_connections 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 -- 172.21.15.28:0/3435090205 >> 172.21.15.28:0/3435090205 conn(0x7f126c0fc9e0 msgr2=0x7f126c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 -- 172.21.15.28:0/3435090205 shutdown_connections 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 -- 172.21.15.28:0/3435090205 wait complete. 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 librados: starting msgr at 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.394+0000 7f1271018700 1 librados: starting objecter 2024-03-30T15:16:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 Processor -- start 2024-03-30T15:16:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 -- start start 2024-03-30T15:16:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 librados: setting wanted keys 2024-03-30T15:16:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 librados: calling monclient init 2024-03-30T15:16:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f126c1a93e0 con 0x7f126c109dc0 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f126c1aa600 con 0x7f126c10d960 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1271018700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f126c1ab840 con 0x7f126c101130 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f126ad9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f126c101130 0x7f126c1064f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40848/0 (socket says 172.21.15.28:40848) 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f126a59c700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f126c109dc0 0x7f126c1a41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40730/0 (socket says 172.21.15.28:40730) 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f126ad9d700 1 -- 172.21.15.28:0/3879148316 learned_addr learned my addr 172.21.15.28:0/3879148316 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1263fff700 10 client.?.objecter ms_handle_connect 0x7f126c109dc0 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1263fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1263fff700 10 client.?.objecter ms_handle_connect 0x7f126c101130 2024-03-30T15:16:22.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.395+0000 7f1263fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 10 client.?.objecter ms_handle_connect 0x7f126c10d960 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020095556 0 0) 0x7f126c1a93e0 con 0x7f126c109dc0 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1258003aa0 con 0x7f126c109dc0 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2816740150 0 0) 0x7f126c1ab840 con 0x7f126c101130 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f126c1a93e0 con 0x7f126c101130 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24753076 0 0) 0x7f126c1aa600 con 0x7f126c10d960 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f126c1ab840 con 0x7f126c10d960 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3377498901 0 0) 0x7f1258003aa0 con 0x7f126c109dc0 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f126c1aa600 con 0x7f126c109dc0 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f125c0013d0 con 0x7f126c109dc0 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4004914257 0 0) 0x7f126c1a93e0 con 0x7f126c101130 2024-03-30T15:16:22.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1258003aa0 con 0x7f126c101130 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1254004680 con 0x7f126c101130 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4250442660 0 0) 0x7f126c1ab840 con 0x7f126c10d960 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f126c1a93e0 con 0x7f126c10d960 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f1264001af0 con 0x7f126c10d960 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 545070906 0 0) 0x7f126c1aa600 con 0x7f126c109dc0 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 >> v1:172.21.15.28:6790/0 conn(0x7f126c101130 legacy=0x7f126c1064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 >> v1:172.21.15.134:6789/0 conn(0x7f126c10d960 legacy=0x7f126c1a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c1aca80 con 0x7f126c109dc0 2024-03-30T15:16:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f126c1aba70 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1271018700 10 client.26127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f126c1abf30 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1271018700 1 librados: init done 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.396+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f125c001680 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.397+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f125c005bc0 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.397+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f124c005120 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.398+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f125c003930 con 0x7f126c109dc0 2024-03-30T15:16:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(944..944 src has 881..944) ==== 7561+0+0 (unknown 2666388488 0 0) 0x7f125c089220 con 0x7f126c109dc0 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 10 client.26127.objecter ms_dispatch 0x7f126c001550 osd_map(944..944 src has 881..944) 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 3 client.26127.objecter handle_osd_map got epochs [944,944] > 0 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 3 client.26127.objecter handle_osd_map decoding full epoch 944 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 20 client.26127.objecter dump_active .. 0 homeless 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.399+0000 7f1263fff700 10 client.26127.objecter ms_handle_connect 0x7f12580703d0 2024-03-30T15:16:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.403+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f125c0567b0 con 0x7f126c109dc0 2024-03-30T15:16:22.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.620+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f124c005410 con 0x7f126c109dc0 2024-03-30T15:16:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.652+0000 7f1263fff700 1 -- 172.21.15.28:0/3879148316 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v945) ==== 113+0+0 (unknown 3365557045 0 0) 0x7f125c05a200 con 0x7f126c109dc0 2024-03-30T15:16:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 10 librados: watch_flush enter 2024-03-30T15:16:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 10 librados: watch_flush exit 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 20 client.26127.objecter shutdown clearing up homeless session... 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 10 client.26127.objecter successfully canceled tick 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 >> v1:172.21.15.28:6800/3011431093 conn(0x7f12580703d0 legacy=0x7f1258072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.655+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 >> v1:172.21.15.28:6789/0 conn(0x7f126c109dc0 legacy=0x7f126c1a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.656+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 shutdown_connections 2024-03-30T15:16:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.656+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 >> 172.21.15.28:0/3879148316 conn(0x7f126c0fc9e0 msgr2=0x7f126c110af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:22.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.657+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 shutdown_connections 2024-03-30T15:16:22.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.657+0000 7f1271018700 1 -- 172.21.15.28:0/3879148316 wait complete. 2024-03-30T15:16:22.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:22.657+0000 7f1271018700 1 librados: shutdown 2024-03-30T15:16:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T15:16:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:22 smithi134 ceph-mon[33714]: 12.6 scrub starts 2024-03-30T15:16:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:22 smithi134 ceph-mon[33714]: 12.6 scrub ok 2024-03-30T15:16:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:22 smithi134 ceph-mon[33714]: pgmap v1493: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:22.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:22 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[41614]: 12.6 scrub starts 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[41614]: 12.6 scrub ok 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[41614]: pgmap v1493: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[29876]: 12.6 scrub starts 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[29876]: 12.6 scrub ok 2024-03-30T15:16:22.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[29876]: pgmap v1493: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:22.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:22 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:23.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd primary-affinity 0 1 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: 5.9 scrub starts 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: 5.9 scrub ok 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: osdmap e945: 8 total, 8 up, 8 in 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: 6.7 scrub starts 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[29876]: 6.7 scrub ok 2024-03-30T15:16:23.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: 5.9 scrub starts 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: 5.9 scrub ok 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: osdmap e945: 8 total, 8 up, 8 in 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: 6.7 scrub starts 2024-03-30T15:16:23.725 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:23 smithi028 ceph-mon[41614]: 6.7 scrub ok 2024-03-30T15:16:23.743 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: 5.9 scrub starts 2024-03-30T15:16:23.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: 5.9 scrub ok 2024-03-30T15:16:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3879148316' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: osdmap e945: 8 total, 8 up, 8 in 2024-03-30T15:16:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: 6.7 scrub starts 2024-03-30T15:16:23.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:23 smithi134 ceph-mon[33714]: 6.7 scrub ok 2024-03-30T15:16:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:24 smithi134 ceph-mon[33714]: 4.7 scrub starts 2024-03-30T15:16:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:24 smithi134 ceph-mon[33714]: 4.7 scrub ok 2024-03-30T15:16:24.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:24 smithi134 ceph-mon[33714]: pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:24.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[41614]: 4.7 scrub starts 2024-03-30T15:16:24.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[41614]: 4.7 scrub ok 2024-03-30T15:16:24.934 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[41614]: pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:24.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[29876]: 4.7 scrub starts 2024-03-30T15:16:24.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[29876]: 4.7 scrub ok 2024-03-30T15:16:24.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:24 smithi028 ceph-mon[29876]: pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.501+0000 7f074394d700 1 -- 172.21.15.28:0/2675496533 >> v1:172.21.15.28:6790/0 conn(0x7f073c10d9a0 legacy=0x7f073c10fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.501+0000 7f074394d700 1 -- 172.21.15.28:0/2675496533 shutdown_connections 2024-03-30T15:16:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.501+0000 7f074394d700 1 -- 172.21.15.28:0/2675496533 >> 172.21.15.28:0/2675496533 conn(0x7f073c0fca00 msgr2=0x7f073c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.501+0000 7f074394d700 1 -- 172.21.15.28:0/2675496533 shutdown_connections 2024-03-30T15:16:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.501+0000 7f074394d700 1 -- 172.21.15.28:0/2675496533 wait complete. 2024-03-30T15:16:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 librados: starting msgr at 2024-03-30T15:16:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 librados: starting objecter 2024-03-30T15:16:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 Processor -- start 2024-03-30T15:16:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 -- start start 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 librados: setting wanted keys 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 librados: calling monclient init 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f073c106900 con 0x7f073c10d9a0 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f073c1b0e10 con 0x7f073c109e00 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f074394d700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f073c1b1ff0 con 0x7f073c101170 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f07416e9700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f073c101170 0x7f073c104760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:40866/0 (socket says 172.21.15.28:40866) 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f07416e9700 1 -- 172.21.15.28:0/629440621 learned_addr learned my addr 172.21.15.28:0/629440621 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f07327fc700 10 client.?.objecter ms_handle_connect 0x7f073c101170 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.502+0000 7f07327fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 10 client.?.objecter ms_handle_connect 0x7f073c10d9a0 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 10 client.?.objecter ms_handle_connect 0x7f073c109e00 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703098297 0 0) 0x7f073c1b1ff0 con 0x7f073c101170 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07280031f0 con 0x7f073c101170 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335871 0 0) 0x7f073c1b0e10 con 0x7f073c109e00 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f073c1b1ff0 con 0x7f073c109e00 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157894880 0 0) 0x7f073c106900 con 0x7f073c10d9a0 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f073c1b0e10 con 0x7f073c10d9a0 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1765782262 0 0) 0x7f07280031f0 con 0x7f073c101170 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f073c106900 con 0x7f073c101170 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f072c001930 con 0x7f073c101170 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4124471245 0 0) 0x7f073c1b0e10 con 0x7f073c10d9a0 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07280031f0 con 0x7f073c10d9a0 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f0734001b30 con 0x7f073c10d9a0 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 10035063 0 0) 0x7f073c1b1ff0 con 0x7f073c109e00 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f073c1b0e10 con 0x7f073c109e00 2024-03-30T15:16:25.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f07380010b0 con 0x7f073c109e00 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2835993130 0 0) 0x7f073c106900 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 >> v1:172.21.15.134:6789/0 conn(0x7f073c109e00 legacy=0x7f073c104e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 >> v1:172.21.15.28:6789/0 conn(0x7f073c10d9a0 legacy=0x7f073c073cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c1b31d0 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f073c1b1040 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f074394d700 10 client.59336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f073c1b1500 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f074394d700 1 librados: init done 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f072c001d00 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.503+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f072c005d90 con 0x7f073c101170 2024-03-30T15:16:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.504+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0720005120 con 0x7f073c101170 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f072c003950 con 0x7f073c101170 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(945..945 src has 881..945) ==== 7561+0+0 (unknown 2446349413 0 0) 0x7f072c089560 con 0x7f073c101170 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 10 client.59336.objecter ms_dispatch 0x7f073c001550 osd_map(945..945 src has 881..945) 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 3 client.59336.objecter handle_osd_map got epochs [945,945] > 0 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 3 client.59336.objecter handle_osd_map decoding full epoch 945 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 20 client.59336.objecter dump_active .. 0 homeless 2024-03-30T15:16:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.505+0000 7f07327fc700 10 client.59336.objecter ms_handle_connect 0x7f07280704e0 2024-03-30T15:16:25.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.510+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f072c056a10 con 0x7f073c101170 2024-03-30T15:16:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:25.733+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7f0720005c20 con 0x7f073c101170 2024-03-30T15:16:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:25 smithi134 ceph-mon[33714]: 3.3 scrub starts 2024-03-30T15:16:25.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:25 smithi134 ceph-mon[33714]: 3.3 scrub ok 2024-03-30T15:16:25.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:25 smithi028 ceph-mon[29876]: 3.3 scrub starts 2024-03-30T15:16:25.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:25 smithi028 ceph-mon[29876]: 3.3 scrub ok 2024-03-30T15:16:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:25 smithi028 ceph-mon[41614]: 3.3 scrub starts 2024-03-30T15:16:25.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:25 smithi028 ceph-mon[41614]: 3.3 scrub ok 2024-03-30T15:16:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.669+0000 7f07327fc700 1 -- 172.21.15.28:0/629440621 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v946) ==== 133+0+0 (unknown 1063733271 0 0) 0x7f072c05a460 con 0x7f073c101170 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 10 librados: watch_flush enter 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 10 librados: watch_flush exit 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 20 client.59336.objecter shutdown clearing up homeless session... 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 10 client.59336.objecter successfully canceled tick 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 >> v1:172.21.15.28:6800/3011431093 conn(0x7f07280704e0 legacy=0x7f07280729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 >> v1:172.21.15.28:6790/0 conn(0x7f073c101170 legacy=0x7f073c104760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 shutdown_connections 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.672+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 >> 172.21.15.28:0/629440621 conn(0x7f073c0fca00 msgr2=0x7f073c1109d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.673+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 shutdown_connections 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.673+0000 7f074394d700 1 -- 172.21.15.28:0/629440621 wait complete. 2024-03-30T15:16:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:26.673+0000 7f074394d700 1 librados: shutdown 2024-03-30T15:16:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T15:16:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: 3.14 scrub starts 2024-03-30T15:16:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: 3.14 scrub ok 2024-03-30T15:16:26.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/629440621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: from='client.59336 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: 1.6 scrub starts 2024-03-30T15:16:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: 1.6 scrub ok 2024-03-30T15:16:26.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:26 smithi134 ceph-mon[33714]: pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: 3.14 scrub starts 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: 3.14 scrub ok 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/629440621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: from='client.59336 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: 1.6 scrub starts 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: 1.6 scrub ok 2024-03-30T15:16:26.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[29876]: pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: 3.14 scrub starts 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: 3.14 scrub ok 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/629440621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: from='client.59336 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: 1.6 scrub starts 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: 1.6 scrub ok 2024-03-30T15:16:26.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:26 smithi028 ceph-mon[41614]: pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:27.484 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-30T15:16:27.485 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-30T15:16:27.485 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-30T15:16:27.485 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-30T15:16:27.485 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T15:16:27.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 1.9 scrub starts 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 1.9 scrub ok 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 6.b scrub starts 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 6.b scrub ok 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: from='client.59336 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: osdmap e946: 8 total, 8 up, 8 in 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 12.4 scrub starts 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: 12.4 scrub ok 2024-03-30T15:16:27.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[29876]: osdmap e947: 8 total, 8 up, 8 in 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 1.9 scrub starts 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 1.9 scrub ok 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 6.b scrub starts 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 6.b scrub ok 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: from='client.59336 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: osdmap e946: 8 total, 8 up, 8 in 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 12.4 scrub starts 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: 12.4 scrub ok 2024-03-30T15:16:27.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:27 smithi028 ceph-mon[41614]: osdmap e947: 8 total, 8 up, 8 in 2024-03-30T15:16:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 1.9 scrub starts 2024-03-30T15:16:27.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 1.9 scrub ok 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 6.b scrub starts 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 6.b scrub ok 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: from='client.59336 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: osdmap e946: 8 total, 8 up, 8 in 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 12.4 scrub starts 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: 12.4 scrub ok 2024-03-30T15:16:27.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:27 smithi134 ceph-mon[33714]: osdmap e947: 8 total, 8 up, 8 in 2024-03-30T15:16:28.473 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:28.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 6.3 scrub starts 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 6.3 scrub ok 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 12.7 scrub starts 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 12.7 scrub ok 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 12.f scrub starts 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 12.f scrub ok 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 4.b deep-scrub starts 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: 4.b deep-scrub ok 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[29876]: pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:28.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 6.3 scrub starts 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 6.3 scrub ok 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 12.7 scrub starts 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 12.7 scrub ok 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 12.f scrub starts 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 12.f scrub ok 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 4.b deep-scrub starts 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: 4.b deep-scrub ok 2024-03-30T15:16:28.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:28 smithi028 ceph-mon[41614]: pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:28.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 6.3 scrub starts 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 6.3 scrub ok 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 12.7 scrub starts 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 12.7 scrub ok 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 12.f scrub starts 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 12.f scrub ok 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 4.b deep-scrub starts 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: 4.b deep-scrub ok 2024-03-30T15:16:28.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:28 smithi134 ceph-mon[33714]: pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.377+0000 7fcd07824700 1 -- 172.21.15.28:0/682514305 >> v1:172.21.15.28:6789/0 conn(0x7fcd00101140 legacy=0x7fcd00101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.377+0000 7fcd07824700 1 -- 172.21.15.28:0/682514305 shutdown_connections 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.377+0000 7fcd07824700 1 -- 172.21.15.28:0/682514305 >> 172.21.15.28:0/682514305 conn(0x7fcd000fc9f0 msgr2=0x7fcd000fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.377+0000 7fcd07824700 1 -- 172.21.15.28:0/682514305 shutdown_connections 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 -- 172.21.15.28:0/682514305 wait complete. 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 librados: starting msgr at 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 librados: starting objecter 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 Processor -- start 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 -- start start 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 librados: setting wanted keys 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 librados: calling monclient init 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd001a9160 con 0x7fcd00109dd0 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd001aa3a0 con 0x7fcd00101140 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd07824700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcd001ab5e0 con 0x7fcd0010d970 2024-03-30T15:16:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd05dc1700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fcd0010d970 0x7fcd001a7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41468/0 (socket says 172.21.15.28:41468) 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd05dc1700 1 -- 172.21.15.28:0/552663588 learned_addr learned my addr 172.21.15.28:0/552663588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fcd04dbf700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd00109dd0 0x7fcd001a41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33218/0 (socket says 172.21.15.28:33218) 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fccf27fc700 10 client.?.objecter ms_handle_connect 0x7fcd0010d970 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fccf27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fccf27fc700 10 client.?.objecter ms_handle_connect 0x7fcd00109dd0 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.378+0000 7fccf27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 10 client.?.objecter ms_handle_connect 0x7fcd00101140 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711937563 0 0) 0x7fcd001a9160 con 0x7fcd00109dd0 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccec003aa0 con 0x7fcd00109dd0 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065816018 0 0) 0x7fcd001ab5e0 con 0x7fcd0010d970 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd001a9160 con 0x7fcd0010d970 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031293135 0 0) 0x7fcd001aa3a0 con 0x7fcd00101140 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd001ab5e0 con 0x7fcd00101140 2024-03-30T15:16:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3283495658 0 0) 0x7fccec003aa0 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd001aa3a0 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccfc001910 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 791476869 0 0) 0x7fcd001a9160 con 0x7fcd0010d970 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccec003aa0 con 0x7fcd0010d970 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccf80015b0 con 0x7fcd0010d970 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 217458661 0 0) 0x7fcd001ab5e0 con 0x7fcd00101140 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd001a9160 con 0x7fcd00101140 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccf4004470 con 0x7fcd00101140 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4176389911 0 0) 0x7fcd001aa3a0 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 >> v1:172.21.15.28:6790/0 conn(0x7fcd0010d970 legacy=0x7fcd001a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 >> v1:172.21.15.134:6789/0 conn(0x7fcd00101140 legacy=0x7fcd001061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd001ac820 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd001a9390 con 0x7fcd00109dd0 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fcd07824700 10 client.26139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd001a9850 con 0x7fcd00109dd0 2024-03-30T15:16:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fcd07824700 1 librados: init done 2024-03-30T15:16:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fccfc001d00 con 0x7fcd00109dd0 2024-03-30T15:16:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.379+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fccfc005d30 con 0x7fcd00109dd0 2024-03-30T15:16:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.380+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce4005120 con 0x7fcd00109dd0 2024-03-30T15:16:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fccfc003950 con 0x7fcd00109dd0 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(948..948 src has 881..948) ==== 7561+0+0 (unknown 2719743420 0 0) 0x7fccfc0895e0 con 0x7fcd00109dd0 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 10 client.26139.objecter ms_dispatch 0x7fcd00001550 osd_map(948..948 src has 881..948) 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 3 client.26139.objecter handle_osd_map got epochs [948,948] > 0 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 3 client.26139.objecter handle_osd_map decoding full epoch 948 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 20 client.26139.objecter dump_active .. 0 homeless 2024-03-30T15:16:29.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.381+0000 7fccf27fc700 10 client.26139.objecter ms_handle_connect 0x7fccec0704d0 2024-03-30T15:16:29.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.386+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fccfc056a90 con 0x7fcd00109dd0 2024-03-30T15:16:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.589+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcce4005410 con 0x7fcd00109dd0 2024-03-30T15:16:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.591+0000 7fccf27fc700 1 -- 172.21.15.28:0/552663588 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v948) ==== 74+0+24282 (unknown 2129501508 0 2142989725) 0x7fccfc05a4e0 con 0x7fcd00109dd0 2024-03-30T15:16:29.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:16:29.593 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":948,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:48:49.804304+0000","modified":"2024-03-30T15:16:28.667473+0000","last_up_change":"2024-03-30T15:13:27.584413+0000","last_in_change":"2024-03-30T14:54:05.032145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":121,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:52:17.629184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"920","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"559","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T14:54:34.798924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T14:54:35.563381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T14:54:37.775241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T14:54:39.594196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T14:54:41.660853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-30T14:57:46.063758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb350905-2b09-408a-9adc-4f2af2766479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":668,"up_from":850,"up_thru":922,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":2927075629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":2927075629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":2927075629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":2927075629}]},"public_addr":"172.21.15.28:6809/2927075629","cluster_addr":"172.21.15.28:6810/2927075629","heartbeat_back_addr":"172.21.15.28:6812/2927075629","heartbeat_front_addr":"172.21.15.28:6811/2927075629","state":["exists","up"]},{"osd":1,"uuid":"76bfea8d-68c7-448e-bb99-d3d003567520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1850313547}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1850313547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1850313547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1850313547}]},"public_addr":"172.21.15.28:6805/1850313547","cluster_addr":"172.21.15.28:6806/1850313547","heartbeat_back_addr":"172.21.15.28:6808/1850313547","heartbeat_front_addr":"172.21.15.28:6807/1850313547","state":["exists","up"]},{"osd":2,"uuid":"ad24917c-f5cb-4ca8-8a52-91d2256ff21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":779,"up_from":840,"up_thru":899,"down_at":780,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":2294873818}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":2294873818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":2294873818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":2294873818}]},"public_addr":"172.21.15.28:6801/2294873818","cluster_addr":"172.21.15.28:6802/2294873818","heartbeat_back_addr":"172.21.15.28:6804/2294873818","heartbeat_front_addr":"172.21.15.28:6803/2294873818","state":["exists","up"]},{"osd":3,"uuid":"4305caf1-0023-438f-8520-87814130b457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":929,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1506157250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1506157250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1506157250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1506157250}]},"public_addr":"172.21.15.28:6813/1506157250","cluster_addr":"172.21.15.28:6814/1506157250","heartbeat_back_addr":"172.21.15.28:6816/1506157250","heartbeat_front_addr":"172.21.15.28:6815/1506157250","state":["exists","up"]},{"osd":4,"uuid":"8180ec45-5260-4cdc-855c-be5302b1465c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":815,"up_from":832,"up_thru":899,"down_at":816,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6800","nonce":4077237787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6801","nonce":4077237787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6803","nonce":4077237787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6802","nonce":4077237787}]},"public_addr":"172.21.15.134:6800/4077237787","cluster_addr":"172.21.15.134:6801/4077237787","heartbeat_back_addr":"172.21.15.134:6803/4077237787","heartbeat_front_addr":"172.21.15.134:6802/4077237787","state":["exists","up"]},{"osd":5,"uuid":"265992a1-46bc-4621-b9db-2b5ba1ee5e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6804","nonce":3668151179}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6805","nonce":3668151179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6807","nonce":3668151179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6806","nonce":3668151179}]},"public_addr":"172.21.15.134:6804/3668151179","cluster_addr":"172.21.15.134:6805/3668151179","heartbeat_back_addr":"172.21.15.134:6807/3668151179","heartbeat_front_addr":"172.21.15.134:6806/3668151179","state":["exists","up"]},{"osd":6,"uuid":"5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":915,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6808","nonce":2135224194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6809","nonce":2135224194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6811","nonce":2135224194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6810","nonce":2135224194}]},"public_addr":"172.21.15.134:6808/2135224194","cluster_addr":"172.21.15.134:6809/2135224194","heartbeat_back_addr":"172.21.15.134:6811/2135224194","heartbeat_front_addr":"172.21.15.134:6810/2135224194","state":["exists","up"]},{"osd":7,"uuid":"76f25ab9-af2f-4f45-afe3-13ece781d29c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6812","nonce":3758190298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6813","nonce":3758190298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6815","nonce":3758190298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6814","nonce":3758190298}]},"public_addr":"172.21.15.134:6812/3758190298","cluster_addr":"172.21.15.134:6813/3758190298","heartbeat_back_addr":"172.21.15.134:6815/3758190298","heartbeat_front_addr":"172.21.15.134:6814/3758190298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:22.255321+0000","dead_epoch":668},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:00.290556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:52.455743+0000","dead_epoch":779},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:28.306185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:44.167871+0000","dead_epoch":815},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:09.385491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:10.312829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:17.407444+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.17","osds":[6,7,1]},{"pgid":"6.4","osds":[1,4,3]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.d","mappings":[{"from":2,"to":6},{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2199995094":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/545344739":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/4222366292":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/406580277":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/2354623613":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/2838206829":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/1249955842":"2024-03-31T14:49:57.293695+0000","172.21.15.28:6800/3720192247":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/1446120238":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/429160846":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/580061692":"2024-03-31T14:55:17.563475+0000","172.21.15.28:6800/997592180":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/2767265670":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1590506004":"2024-03-31T14:49:57.293695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":2,"length":5}]},{"pool":25,"snaps":[{"begin":2,"length":3}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":69,"snaps":[{"begin":1,"length":1}]},{"pool":104,"snaps":[{"begin":7,"length":2}]},{"pool":112,"snaps":[{"begin":1,"length":1}]},{"pool":124,"snaps":[{"begin":1,"length":2}]},{"pool":143,"snaps":[{"begin":3,"length":1}]},{"pool":147,"snaps":[{"begin":2,"length":1}]},{"pool":155,"snaps":[{"begin":2,"length":2}]},{"pool":157,"snaps":[{"begin":2,"length":3}]},{"pool":165,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":179,"snaps":[{"begin":2,"length":1},{"begin":4,"length":2}]},{"pool":188,"snaps":[{"begin":2,"length":1}]},{"pool":195,"snaps":[{"begin":3,"length":2}]},{"pool":210,"snaps":[{"begin":2,"length":6}]},{"pool":214,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":246,"snaps":[{"begin":10,"length":1}]},{"pool":250,"snaps":[{"begin":11,"length":2}]},{"pool":261,"snaps":[{"begin":14,"length":1}]},{"pool":325,"snaps":[{"begin":2,"length":1}]},{"pool":326,"snaps":[{"begin":3,"length":1}]},{"pool":329,"snaps":[{"begin":4,"length":1}]},{"pool":333,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T15:16:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.593+0000 7fcd07824700 10 librados: watch_flush enter 2024-03-30T15:16:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.593+0000 7fcd07824700 10 librados: watch_flush exit 2024-03-30T15:16:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.593+0000 7fcd07824700 20 client.26139.objecter shutdown clearing up homeless session... 2024-03-30T15:16:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.593+0000 7fcd07824700 10 client.26139.objecter successfully canceled tick 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 >> v1:172.21.15.28:6800/3011431093 conn(0x7fccec0704d0 legacy=0x7fccec072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 >> v1:172.21.15.28:6789/0 conn(0x7fcd00109dd0 legacy=0x7fcd001a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 shutdown_connections 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 >> 172.21.15.28:0/552663588 conn(0x7fcd000fc9f0 msgr2=0x7fcd000ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 shutdown_connections 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 -- 172.21.15.28:0/552663588 wait complete. 2024-03-30T15:16:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:29.594+0000 7fcd07824700 1 librados: shutdown 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[41614]: osdmap e948: 8 total, 8 up, 8 in 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/552663588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:29.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:29.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[29876]: osdmap e948: 8 total, 8 up, 8 in 2024-03-30T15:16:29.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:29 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/552663588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:29 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:29 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:29 smithi134 ceph-mon[33714]: osdmap e948: 8 total, 8 up, 8 in 2024-03-30T15:16:29.930 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:29 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/552663588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:30.343 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T15:16:30.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T15:16:30.595 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:30 smithi134 ceph-mon[33714]: 4.19 scrub starts 2024-03-30T15:16:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:30 smithi134 ceph-mon[33714]: 4.19 scrub ok 2024-03-30T15:16:30.931 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:30 smithi134 ceph-mon[33714]: pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[29876]: 4.19 scrub starts 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[29876]: 4.19 scrub ok 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[29876]: pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[41614]: 4.19 scrub starts 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[41614]: 4.19 scrub ok 2024-03-30T15:16:30.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:30 smithi028 ceph-mon[41614]: pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:31.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:31.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 1.2 scrub starts 2024-03-30T15:16:31.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 1.2 scrub ok 2024-03-30T15:16:31.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 3.18 scrub starts 2024-03-30T15:16:31.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 3.18 scrub ok 2024-03-30T15:16:31.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 6.1a scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 6.1a scrub ok 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 2.2 scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 2.2 scrub ok 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 5.8 scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: 5.8 scrub ok 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 1.2 scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 1.2 scrub ok 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 3.18 scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 3.18 scrub ok 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 6.1a scrub starts 2024-03-30T15:16:31.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 6.1a scrub ok 2024-03-30T15:16:31.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 2.2 scrub starts 2024-03-30T15:16:31.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 2.2 scrub ok 2024-03-30T15:16:31.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 5.8 scrub starts 2024-03-30T15:16:31.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: 5.8 scrub ok 2024-03-30T15:16:31.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:31 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 1.2 scrub starts 2024-03-30T15:16:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 1.2 scrub ok 2024-03-30T15:16:32.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 3.18 scrub starts 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 3.18 scrub ok 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 6.1a scrub starts 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 6.1a scrub ok 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 2.2 scrub starts 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 2.2 scrub ok 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 5.8 scrub starts 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: 5.8 scrub ok 2024-03-30T15:16:32.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:31 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.302+0000 7f4027c53700 1 -- 172.21.15.28:0/1604102609 >> v1:172.21.15.28:6790/0 conn(0x7f4020109dc0 legacy=0x7f402010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.302+0000 7f4027c53700 1 -- 172.21.15.28:0/1604102609 shutdown_connections 2024-03-30T15:16:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.302+0000 7f4027c53700 1 -- 172.21.15.28:0/1604102609 >> 172.21.15.28:0/1604102609 conn(0x7f40200fc9e0 msgr2=0x7f40200fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.302+0000 7f4027c53700 1 -- 172.21.15.28:0/1604102609 shutdown_connections 2024-03-30T15:16:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.302+0000 7f4027c53700 1 -- 172.21.15.28:0/1604102609 wait complete. 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 librados: starting msgr at 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 librados: starting objecter 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 Processor -- start 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 -- start start 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 librados: setting wanted keys 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 librados: calling monclient init 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f402006a900 con 0x7f402010d960 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40201a8540 con 0x7f4020109dc0 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.303+0000 7f4027c53700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f40201a9720 con 0x7f4020101130 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f40259ef700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f4020101130 0x7f4020105650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41480/0 (socket says 172.21.15.28:41480) 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f40259ef700 1 -- 172.21.15.28:0/276733118 learned_addr learned my addr 172.21.15.28:0/276733118 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter ms_handle_connect 0x7f4020101130 2024-03-30T15:16:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter ms_handle_connect 0x7f4020109dc0 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895570382 0 0) 0x7f40201a9720 con 0x7f4020101130 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f400c0035b0 con 0x7f4020101130 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter ms_handle_connect 0x7f402010d960 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098583610 0 0) 0x7f40201a8540 con 0x7f4020109dc0 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40201a9720 con 0x7f4020109dc0 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797590042 0 0) 0x7f402006a900 con 0x7f402010d960 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40201a8540 con 0x7f402010d960 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1109399639 0 0) 0x7f400c0035b0 con 0x7f4020101130 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f402006a900 con 0x7f4020101130 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4010001a10 con 0x7f4020101130 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1018853488 0 0) 0x7f40201a9720 con 0x7f4020109dc0 2024-03-30T15:16:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f400c0035b0 con 0x7f4020109dc0 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3647951106 0 0) 0x7f40201a8540 con 0x7f402010d960 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40201a9720 con 0x7f402010d960 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2313114433 0 0) 0x7f402006a900 con 0x7f4020101130 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.304+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 >> v1:172.21.15.134:6789/0 conn(0x7f4020109dc0 legacy=0x7f4020105d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 >> v1:172.21.15.28:6789/0 conn(0x7f402010d960 legacy=0x7f40201a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40201aa900 con 0x7f4020101130 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40201a8770 con 0x7f4020101130 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4027c53700 10 client.59348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40201a8bd0 con 0x7f4020101130 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4027c53700 1 librados: init done 2024-03-30T15:16:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f4010003dd0 con 0x7f4020101130 2024-03-30T15:16:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.305+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4010005d30 con 0x7f4020101130 2024-03-30T15:16:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.306+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004005120 con 0x7f4020101130 2024-03-30T15:16:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.306+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f4010003930 con 0x7f4020101130 2024-03-30T15:16:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(948..948 src has 881..948) ==== 7561+0+0 (unknown 2719743420 0 0) 0x7f40100895d0 con 0x7f4020101130 2024-03-30T15:16:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 10 client.59348.objecter ms_dispatch 0x7f4020001550 osd_map(948..948 src has 881..948) 2024-03-30T15:16:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 3 client.59348.objecter handle_osd_map got epochs [948,948] > 0 2024-03-30T15:16:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 3 client.59348.objecter handle_osd_map decoding full epoch 948 2024-03-30T15:16:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 20 client.59348.objecter dump_active .. 0 homeless 2024-03-30T15:16:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.307+0000 7f4016ffd700 10 client.59348.objecter ms_handle_connect 0x7f400c070330 2024-03-30T15:16:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.315+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4010056a80 con 0x7f4020101130 2024-03-30T15:16:32.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.522+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4004005410 con 0x7f4020101130 2024-03-30T15:16:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.524+0000 7f4016ffd700 1 -- 172.21.15.28:0/276733118 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v948) ==== 74+0+24282 (unknown 2129501508 0 2142989725) 0x7f401005a4d0 con 0x7f4020101130 2024-03-30T15:16:32.526 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:16:32.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":948,"fsid":"8345fd88-eea4-11ee-b647-cb9ed24678a4","created":"2024-03-30T14:48:49.804304+0000","modified":"2024-03-30T15:16:28.667473+0000","last_up_change":"2024-03-30T15:13:27.584413+0000","last_in_change":"2024-03-30T14:54:05.032145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":121,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T14:52:17.629184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"920","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"559","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-30T14:54:34.798924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":72,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-30T14:54:35.563381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-30T14:54:37.775241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-30T14:54:39.594196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-30T14:54:41.660853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-30T14:57:46.063758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb350905-2b09-408a-9adc-4f2af2766479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":668,"up_from":850,"up_thru":922,"down_at":669,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":2927075629}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":2927075629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":2927075629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":2927075629}]},"public_addr":"172.21.15.28:6809/2927075629","cluster_addr":"172.21.15.28:6810/2927075629","heartbeat_back_addr":"172.21.15.28:6812/2927075629","heartbeat_front_addr":"172.21.15.28:6811/2927075629","state":["exists","up"]},{"osd":1,"uuid":"76bfea8d-68c7-448e-bb99-d3d003567520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1850313547}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1850313547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1850313547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1850313547}]},"public_addr":"172.21.15.28:6805/1850313547","cluster_addr":"172.21.15.28:6806/1850313547","heartbeat_back_addr":"172.21.15.28:6808/1850313547","heartbeat_front_addr":"172.21.15.28:6807/1850313547","state":["exists","up"]},{"osd":2,"uuid":"ad24917c-f5cb-4ca8-8a52-91d2256ff21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":779,"up_from":840,"up_thru":899,"down_at":780,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":2294873818}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":2294873818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":2294873818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":2294873818}]},"public_addr":"172.21.15.28:6801/2294873818","cluster_addr":"172.21.15.28:6802/2294873818","heartbeat_back_addr":"172.21.15.28:6804/2294873818","heartbeat_front_addr":"172.21.15.28:6803/2294873818","state":["exists","up"]},{"osd":3,"uuid":"4305caf1-0023-438f-8520-87814130b457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":929,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1506157250}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1506157250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1506157250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1506157250}]},"public_addr":"172.21.15.28:6813/1506157250","cluster_addr":"172.21.15.28:6814/1506157250","heartbeat_back_addr":"172.21.15.28:6816/1506157250","heartbeat_front_addr":"172.21.15.28:6815/1506157250","state":["exists","up"]},{"osd":4,"uuid":"8180ec45-5260-4cdc-855c-be5302b1465c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":815,"up_from":832,"up_thru":899,"down_at":816,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6800","nonce":4077237787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6801","nonce":4077237787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6803","nonce":4077237787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6802","nonce":4077237787}]},"public_addr":"172.21.15.134:6800/4077237787","cluster_addr":"172.21.15.134:6801/4077237787","heartbeat_back_addr":"172.21.15.134:6803/4077237787","heartbeat_front_addr":"172.21.15.134:6802/4077237787","state":["exists","up"]},{"osd":5,"uuid":"265992a1-46bc-4621-b9db-2b5ba1ee5e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6804","nonce":3668151179}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6805","nonce":3668151179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6807","nonce":3668151179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6806","nonce":3668151179}]},"public_addr":"172.21.15.134:6804/3668151179","cluster_addr":"172.21.15.134:6805/3668151179","heartbeat_back_addr":"172.21.15.134:6807/3668151179","heartbeat_front_addr":"172.21.15.134:6806/3668151179","state":["exists","up"]},{"osd":6,"uuid":"5a6a3949-76b2-47f5-aa49-bde7c0ac2a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":915,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6808","nonce":2135224194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6809","nonce":2135224194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6811","nonce":2135224194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6810","nonce":2135224194}]},"public_addr":"172.21.15.134:6808/2135224194","cluster_addr":"172.21.15.134:6809/2135224194","heartbeat_back_addr":"172.21.15.134:6811/2135224194","heartbeat_front_addr":"172.21.15.134:6810/2135224194","state":["exists","up"]},{"osd":7,"uuid":"76f25ab9-af2f-4f45-afe3-13ece781d29c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":899,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6812","nonce":3758190298}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6813","nonce":3758190298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6815","nonce":3758190298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.134:6814","nonce":3758190298}]},"public_addr":"172.21.15.134:6812/3758190298","cluster_addr":"172.21.15.134:6813/3758190298","heartbeat_back_addr":"172.21.15.134:6815/3758190298","heartbeat_front_addr":"172.21.15.134:6814/3758190298","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:22.255321+0000","dead_epoch":668},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:00.290556+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:52.455743+0000","dead_epoch":779},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:28.306185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:44.167871+0000","dead_epoch":815},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:16:09.385491+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:10.312829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T15:15:17.407444+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.17","osds":[6,7,1]},{"pgid":"6.4","osds":[1,4,3]}],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.d","mappings":[{"from":2,"to":6},{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2199995094":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/545344739":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/4222366292":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/406580277":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/2354623613":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/2838206829":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/1249955842":"2024-03-31T14:49:57.293695+0000","172.21.15.28:6800/3720192247":"2024-03-31T14:55:17.563475+0000","172.21.15.28:0/1446120238":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/429160846":"2024-03-31T14:49:29.238923+0000","172.21.15.28:0/580061692":"2024-03-31T14:55:17.563475+0000","172.21.15.28:6800/997592180":"2024-03-31T14:49:29.238923+0000","172.21.15.28:6800/2767265670":"2024-03-31T14:49:57.293695+0000","172.21.15.28:0/1590506004":"2024-03-31T14:49:57.293695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":13,"snaps":[{"begin":2,"length":5}]},{"pool":25,"snaps":[{"begin":2,"length":3}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":69,"snaps":[{"begin":1,"length":1}]},{"pool":104,"snaps":[{"begin":7,"length":2}]},{"pool":112,"snaps":[{"begin":1,"length":1}]},{"pool":124,"snaps":[{"begin":1,"length":2}]},{"pool":143,"snaps":[{"begin":3,"length":1}]},{"pool":147,"snaps":[{"begin":2,"length":1}]},{"pool":155,"snaps":[{"begin":2,"length":2}]},{"pool":157,"snaps":[{"begin":2,"length":3}]},{"pool":165,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":2}]},{"pool":179,"snaps":[{"begin":2,"length":1},{"begin":4,"length":2}]},{"pool":188,"snaps":[{"begin":2,"length":1}]},{"pool":195,"snaps":[{"begin":3,"length":2}]},{"pool":210,"snaps":[{"begin":2,"length":6}]},{"pool":214,"snaps":[{"begin":7,"length":1}]},{"pool":234,"snaps":[{"begin":8,"length":1}]},{"pool":246,"snaps":[{"begin":10,"length":1}]},{"pool":250,"snaps":[{"begin":11,"length":2}]},{"pool":261,"snaps":[{"begin":14,"length":1}]},{"pool":325,"snaps":[{"begin":2,"length":1}]},{"pool":326,"snaps":[{"begin":3,"length":1}]},{"pool":329,"snaps":[{"begin":4,"length":1}]},{"pool":333,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 10 librados: watch_flush enter 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 10 librados: watch_flush exit 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 20 client.59348.objecter shutdown clearing up homeless session... 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 10 client.59348.objecter successfully canceled tick 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 >> v1:172.21.15.28:6800/3011431093 conn(0x7f400c070330 legacy=0x7f400c0727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.526+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 >> v1:172.21.15.28:6790/0 conn(0x7f4020101130 legacy=0x7f4020105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:32.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.527+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 shutdown_connections 2024-03-30T15:16:32.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.527+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 >> 172.21.15.28:0/276733118 conn(0x7f40200fc9e0 msgr2=0x7f4020110e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:32.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.527+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 shutdown_connections 2024-03-30T15:16:32.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.527+0000 7f4027c53700 1 -- 172.21.15.28:0/276733118 wait complete. 2024-03-30T15:16:32.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:32.527+0000 7f4027c53700 1 librados: shutdown 2024-03-30T15:16:32.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: 6.11 scrub starts 2024-03-30T15:16:32.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: 6.11 scrub ok 2024-03-30T15:16:32.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:16:32.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: 4.6 scrub starts 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: 4.6 scrub ok 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/276733118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[29876]: osdmap e949: 8 total, 8 up, 8 in 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: 6.11 scrub starts 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: 6.11 scrub ok 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: 4.6 scrub starts 2024-03-30T15:16:32.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: 4.6 scrub ok 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/276733118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:32.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:32 smithi028 ceph-mon[41614]: osdmap e949: 8 total, 8 up, 8 in 2024-03-30T15:16:33.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: 6.11 scrub starts 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: 6.11 scrub ok 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: 4.6 scrub starts 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: 4.6 scrub ok 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/276733118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T15:16:33.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:32 smithi134 ceph-mon[33714]: osdmap e949: 8 total, 8 up, 8 in 2024-03-30T15:16:33.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T15:16:33.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T15:16:33.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T15:16:33.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T15:16:33.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T15:16:33.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T15:16:33.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T15:16:33.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T15:16:33.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.806 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.806 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:33.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[41614]: 5.1c scrub starts 2024-03-30T15:16:33.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[41614]: 5.1c scrub ok 2024-03-30T15:16:33.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[41614]: 12.16 scrub starts 2024-03-30T15:16:33.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[41614]: 12.16 scrub ok 2024-03-30T15:16:33.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[29876]: 5.1c scrub starts 2024-03-30T15:16:33.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[29876]: 5.1c scrub ok 2024-03-30T15:16:33.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[29876]: 12.16 scrub starts 2024-03-30T15:16:33.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:33 smithi028 ceph-mon[29876]: 12.16 scrub ok 2024-03-30T15:16:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:33 smithi134 ceph-mon[33714]: 5.1c scrub starts 2024-03-30T15:16:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:33 smithi134 ceph-mon[33714]: 5.1c scrub ok 2024-03-30T15:16:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:33 smithi134 ceph-mon[33714]: 12.16 scrub starts 2024-03-30T15:16:34.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:33 smithi134 ceph-mon[33714]: 12.16 scrub ok 2024-03-30T15:16:34.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[41614]: 5.a scrub starts 2024-03-30T15:16:34.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[41614]: 5.a scrub ok 2024-03-30T15:16:34.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[41614]: 4.1c scrub starts 2024-03-30T15:16:34.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[41614]: 4.1c scrub ok 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[41614]: pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[29876]: 5.a scrub starts 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[29876]: 5.a scrub ok 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[29876]: 4.1c scrub starts 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[29876]: 4.1c scrub ok 2024-03-30T15:16:34.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:34 smithi028 ceph-mon[29876]: pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:34 smithi134 ceph-mon[33714]: 5.a scrub starts 2024-03-30T15:16:35.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:34 smithi134 ceph-mon[33714]: 5.a scrub ok 2024-03-30T15:16:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:34 smithi134 ceph-mon[33714]: 4.1c scrub starts 2024-03-30T15:16:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:34 smithi134 ceph-mon[33714]: 4.1c scrub ok 2024-03-30T15:16:35.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:34 smithi134 ceph-mon[33714]: pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 5.14 scrub starts 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 5.14 scrub ok 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 3.1d scrub starts 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 3.1d scrub ok 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 3.12 scrub starts 2024-03-30T15:16:35.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[41614]: 3.12 scrub ok 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 5.14 scrub starts 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 5.14 scrub ok 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 3.1d scrub starts 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 3.1d scrub ok 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 3.12 scrub starts 2024-03-30T15:16:35.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:35 smithi028 ceph-mon[29876]: 3.12 scrub ok 2024-03-30T15:16:36.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 5.14 scrub starts 2024-03-30T15:16:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 5.14 scrub ok 2024-03-30T15:16:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 3.1d scrub starts 2024-03-30T15:16:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 3.1d scrub ok 2024-03-30T15:16:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 3.12 scrub starts 2024-03-30T15:16:36.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:35 smithi134 ceph-mon[33714]: 3.12 scrub ok 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[29876]: 12.8 scrub starts 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[29876]: 12.8 scrub ok 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[29876]: 12.1 scrub starts 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[29876]: 12.1 scrub ok 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[29876]: pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:36.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[41614]: 12.8 scrub starts 2024-03-30T15:16:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[41614]: 12.8 scrub ok 2024-03-30T15:16:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[41614]: 12.1 scrub starts 2024-03-30T15:16:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[41614]: 12.1 scrub ok 2024-03-30T15:16:36.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:36 smithi028 ceph-mon[41614]: pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:36 smithi134 ceph-mon[33714]: 12.8 scrub starts 2024-03-30T15:16:37.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:36 smithi134 ceph-mon[33714]: 12.8 scrub ok 2024-03-30T15:16:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:36 smithi134 ceph-mon[33714]: 12.1 scrub starts 2024-03-30T15:16:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:36 smithi134 ceph-mon[33714]: 12.1 scrub ok 2024-03-30T15:16:37.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:36 smithi134 ceph-mon[33714]: pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[29876]: 1.5 deep-scrub starts 2024-03-30T15:16:37.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[29876]: 1.5 deep-scrub ok 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[29876]: 3.a scrub starts 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[29876]: 3.a scrub ok 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[29876]: osdmap e950: 8 total, 8 up, 8 in 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[41614]: 1.5 deep-scrub starts 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[41614]: 1.5 deep-scrub ok 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[41614]: 3.a scrub starts 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[41614]: 3.a scrub ok 2024-03-30T15:16:37.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:37 smithi028 ceph-mon[41614]: osdmap e950: 8 total, 8 up, 8 in 2024-03-30T15:16:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:37 smithi134 ceph-mon[33714]: 1.5 deep-scrub starts 2024-03-30T15:16:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:37 smithi134 ceph-mon[33714]: 1.5 deep-scrub ok 2024-03-30T15:16:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:37 smithi134 ceph-mon[33714]: 3.a scrub starts 2024-03-30T15:16:38.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:37 smithi134 ceph-mon[33714]: 3.a scrub ok 2024-03-30T15:16:38.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:37 smithi134 ceph-mon[33714]: osdmap e950: 8 total, 8 up, 8 in 2024-03-30T15:16:38.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:38 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:38.705 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:38 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:38.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:38 smithi028 ceph-mon[29876]: pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:38 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:38.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:38 smithi028 ceph-mon[41614]: pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:38 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:39.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:38 smithi134 ceph-mon[33714]: pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: 6.1f deep-scrub starts 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: 6.1f deep-scrub ok 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: 5.c scrub starts 2024-03-30T15:16:39.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[29876]: 5.c scrub ok 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: 6.1f deep-scrub starts 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: 6.1f deep-scrub ok 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: 5.c scrub starts 2024-03-30T15:16:39.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:39 smithi028 ceph-mon[41614]: 5.c scrub ok 2024-03-30T15:16:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: 6.1f deep-scrub starts 2024-03-30T15:16:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: 6.1f deep-scrub ok 2024-03-30T15:16:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:40.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: 5.c scrub starts 2024-03-30T15:16:40.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:39 smithi134 ceph-mon[33714]: 5.c scrub ok 2024-03-30T15:16:40.808 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[29876]: osdmap e951: 8 total, 8 up, 8 in 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[29876]: 5.5 scrub starts 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[29876]: 5.5 scrub ok 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[29876]: pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[41614]: osdmap e951: 8 total, 8 up, 8 in 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[41614]: 5.5 scrub starts 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[41614]: 5.5 scrub ok 2024-03-30T15:16:40.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:40 smithi028 ceph-mon[41614]: pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:40.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.915+0000 7fbe32c46700 1 -- 172.21.15.28:0/3046464837 >> v1:172.21.15.28:6790/0 conn(0x7fbe2c10d970 legacy=0x7fbe2c10fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.915+0000 7fbe32c46700 1 -- 172.21.15.28:0/3046464837 shutdown_connections 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.915+0000 7fbe32c46700 1 -- 172.21.15.28:0/3046464837 >> 172.21.15.28:0/3046464837 conn(0x7fbe2c0fc9d0 msgr2=0x7fbe2c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.915+0000 7fbe32c46700 1 -- 172.21.15.28:0/3046464837 shutdown_connections 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.915+0000 7fbe32c46700 1 -- 172.21.15.28:0/3046464837 wait complete. 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 librados: starting msgr at 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 librados: starting objecter 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 Processor -- start 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 -- start start 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 librados: setting wanted keys 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 librados: calling monclient init 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbe2c1a9340 con 0x7fbe2c101140 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbe2c1aa580 con 0x7fbe2c109dd0 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe32c46700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbe2c1ab7c0 con 0x7fbe2c10d970 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe311e3700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbe2c10d970 0x7fbe2c1a7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58242/0 (socket says 172.21.15.28:58242) 2024-03-30T15:16:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe311e3700 1 -- 172.21.15.28:0/967244721 learned_addr learned my addr 172.21.15.28:0/967244721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe309e2700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe2c101140 0x7fbe2c1167a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60778/0 (socket says 172.21.15.28:60778) 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter ms_handle_connect 0x7fbe2c10d970 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter ms_handle_connect 0x7fbe2c101140 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter ms_handle_connect 0x7fbe2c109dd0 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089984315 0 0) 0x7fbe2c1a9340 con 0x7fbe2c101140 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe140031f0 con 0x7fbe2c101140 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955399825 0 0) 0x7fbe2c1ab7c0 con 0x7fbe2c10d970 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe2c1a9340 con 0x7fbe2c10d970 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298292946 0 0) 0x7fbe2c1aa580 con 0x7fbe2c109dd0 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.916+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe2c1ab7c0 con 0x7fbe2c109dd0 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1322256304 0 0) 0x7fbe140031f0 con 0x7fbe2c101140 2024-03-30T15:16:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe2c1aa580 con 0x7fbe2c101140 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe1c0018f0 con 0x7fbe2c101140 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 556386717 0 0) 0x7fbe2c1a9340 con 0x7fbe2c10d970 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe140031f0 con 0x7fbe2c10d970 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe240016f0 con 0x7fbe2c10d970 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4017131195 0 0) 0x7fbe2c1ab7c0 con 0x7fbe2c109dd0 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe2c1a9340 con 0x7fbe2c109dd0 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe20001810 con 0x7fbe2c109dd0 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2141166552 0 0) 0x7fbe2c1aa580 con 0x7fbe2c101140 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 >> v1:172.21.15.28:6790/0 conn(0x7fbe2c10d970 legacy=0x7fbe2c1a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 >> v1:172.21.15.134:6789/0 conn(0x7fbe2c109dd0 legacy=0x7fbe2c1a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c1aca00 con 0x7fbe2c101140 2024-03-30T15:16:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c1a9570 con 0x7fbe2c101140 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbe1c001cc0 con 0x7fbe2c101140 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbe1c005d10 con 0x7fbe2c101140 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe32c46700 10 client.26151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbe2c1a9a30 con 0x7fbe2c101140 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.917+0000 7fbe32c46700 1 librados: init done 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.918+0000 7fbe32c46700 10 client.26151.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.918+0000 7fbe32c46700 15 client.26151.objecter _session_command_op_assign -1 1 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.918+0000 7fbe32c46700 20 client.26151.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.918+0000 7fbe32c46700 10 client.26151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.918+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbe2c102620 con 0x7fbe2c101140 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbe1c0040c0 con 0x7fbe2c101140 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(951..951 src has 881..951) ==== 7561+0+0 (unknown 3009242379 0 0) 0x7fbe1c089320 con 0x7fbe2c101140 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 10 client.26151.objecter ms_dispatch 0x7fbe2c001550 osd_map(951..951 src has 881..951) 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 3 client.26151.objecter handle_osd_map got epochs [951,951] > 0 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 3 client.26151.objecter handle_osd_map decoding full epoch 951 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 10 client.26151.objecter checking command 1 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter _get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 15 client.26151.objecter _session_command_op_remove -1 1 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter _get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 15 client.26151.objecter _session_command_op_assign 4 1 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 4 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 10 client.26151.objecter _send_command 1 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.134:6800/4077237787 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe14080760 con 0x7fbe1407ca50 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 20 client.26151.objecter dump_active .. 0 homeless 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 10 client.26151.objecter ms_handle_connect 0x7fbe140703a0 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.919+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=951) ==== 24+0+0 (unknown 775199436 0 0) 0x7fbe1c0897f0 con 0x7fbe2c101140 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.920+0000 7fbe29ffb700 10 client.26151.objecter ms_handle_connect 0x7fbe1407ca50 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.921+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== osd.4 v1:172.21.15.134:6800/4077237787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbe14080760 con 0x7fbe1407ca50 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.921+0000 7fbe29ffb700 10 client.26151.objecter ms_dispatch 0x7fbe2c001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.921+0000 7fbe29ffb700 10 client.26151.objecter _finish_command 1 = system:0 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.921+0000 7fbe29ffb700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 4 2024-03-30T15:16:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.921+0000 7fbe29ffb700 15 client.26151.objecter _session_command_op_remove 4 1 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 10 client.26151.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 15 client.26151.objecter _session_command_op_assign -1 2 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 20 client.26151.objecter _get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 20 client.26151.objecter _get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 15 client.26151.objecter _session_command_op_remove -1 2 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 20 client.26151.objecter get_session s=0x7fbe1407c3a0 osd=4 3 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 15 client.26151.objecter _session_command_op_assign 4 2 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 4 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 10 client.26151.objecter _send_command 2 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 --> v1:172.21.15.134:6800/4077237787 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe2c068b30 con 0x7fbe1407ca50 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe29ffb700 1 -- 172.21.15.28:0/967244721 <== osd.4 v1:172.21.15.134:6800/4077237787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+13 (unknown 0 0 3840509706) 0x7fbe2c068b30 con 0x7fbe1407ca50 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe29ffb700 10 client.26151.objecter ms_dispatch 0x7fbe2c001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe29ffb700 10 client.26151.objecter _finish_command 2 = system:0 2024-03-30T15:16:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe29ffb700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 4 2024-03-30T15:16:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.938+0000 7fbe29ffb700 15 client.26151.objecter _session_command_op_remove 4 2 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 10 librados: watch_flush enter 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 10 librados: watch_flush exit 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 10 client.26151.objecter close_session for osd.4 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 >> v1:172.21.15.134:6800/4077237787 conn(0x7fbe1407ca50 legacy=0x7fbe1407eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 20 client.26151.objecter put_session s=0x7fbe1407c3a0 osd=4 1 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 20 client.26151.objecter shutdown clearing up homeless session... 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 10 client.26151.objecter successfully canceled tick 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbe140703a0 legacy=0x7fbe14072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 >> v1:172.21.15.28:6789/0 conn(0x7fbe2c101140 legacy=0x7fbe2c1167a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe311e3700 1 -- 172.21.15.28:0/967244721 reap_dead start 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 shutdown_connections 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 >> 172.21.15.28:0/967244721 conn(0x7fbe2c0fc9d0 msgr2=0x7fbe2c06ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 shutdown_connections 2024-03-30T15:16:40.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 -- 172.21.15.28:0/967244721 wait complete. 2024-03-30T15:16:40.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:40.939+0000 7fbe32c46700 1 librados: shutdown 2024-03-30T15:16:41.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:40 smithi134 ceph-mon[33714]: osdmap e951: 8 total, 8 up, 8 in 2024-03-30T15:16:41.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:40 smithi134 ceph-mon[33714]: 5.5 scrub starts 2024-03-30T15:16:41.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:40 smithi134 ceph-mon[33714]: 5.5 scrub ok 2024-03-30T15:16:41.061 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:40 smithi134 ceph-mon[33714]: pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:41.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:41.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[29876]: 12.14 scrub starts 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[29876]: 12.14 scrub ok 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[29876]: 4.a scrub starts 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[29876]: 4.a scrub ok 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[41614]: 12.14 scrub starts 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[41614]: 12.14 scrub ok 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[41614]: 4.a scrub starts 2024-03-30T15:16:41.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[41614]: 4.a scrub ok 2024-03-30T15:16:41.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:41 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-mon[33714]: 12.14 scrub starts 2024-03-30T15:16:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-mon[33714]: 12.14 scrub ok 2024-03-30T15:16:42.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-mon[33714]: 4.a scrub starts 2024-03-30T15:16:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-mon[33714]: 4.a scrub ok 2024-03-30T15:16:42.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:41 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[41614]: pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[41614]: 3.0 scrub starts 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[41614]: 3.0 scrub ok 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[41614]: osdmap e952: 8 total, 8 up, 8 in 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[29876]: pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:42.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[29876]: 3.0 scrub starts 2024-03-30T15:16:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[29876]: 3.0 scrub ok 2024-03-30T15:16:42.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:42 smithi028 ceph-mon[29876]: osdmap e952: 8 total, 8 up, 8 in 2024-03-30T15:16:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:42 smithi134 ceph-mon[33714]: pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:42 smithi134 ceph-mon[33714]: 3.0 scrub starts 2024-03-30T15:16:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:42 smithi134 ceph-mon[33714]: 3.0 scrub ok 2024-03-30T15:16:43.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:42 smithi134 ceph-mon[33714]: osdmap e952: 8 total, 8 up, 8 in 2024-03-30T15:16:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:3573412790317 2024-03-30T15:16:43.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T15:16:43.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[29876]: 6.e scrub starts 2024-03-30T15:16:43.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[29876]: 6.e scrub ok 2024-03-30T15:16:43.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[29876]: 12.9 scrub starts 2024-03-30T15:16:43.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[29876]: 12.9 scrub ok 2024-03-30T15:16:43.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[41614]: 6.e scrub starts 2024-03-30T15:16:43.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[41614]: 6.e scrub ok 2024-03-30T15:16:43.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[41614]: 12.9 scrub starts 2024-03-30T15:16:43.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:43 smithi028 ceph-mon[41614]: 12.9 scrub ok 2024-03-30T15:16:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:43 smithi134 ceph-mon[33714]: 6.e scrub starts 2024-03-30T15:16:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:43 smithi134 ceph-mon[33714]: 6.e scrub ok 2024-03-30T15:16:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:43 smithi134 ceph-mon[33714]: 12.9 scrub starts 2024-03-30T15:16:44.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:43 smithi134 ceph-mon[33714]: 12.9 scrub ok 2024-03-30T15:16:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 -- 172.21.15.28:0/717507827 >> v1:172.21.15.28:6790/0 conn(0x7f30e410eab0 legacy=0x7f30e4110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 -- 172.21.15.28:0/717507827 shutdown_connections 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 -- 172.21.15.28:0/717507827 >> 172.21.15.28:0/717507827 conn(0x7f30e40fc9e0 msgr2=0x7f30e40fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 -- 172.21.15.28:0/717507827 shutdown_connections 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 -- 172.21.15.28:0/717507827 wait complete. 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.707+0000 7f30ec110700 1 librados: starting msgr at 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 librados: starting objecter 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 Processor -- start 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 -- start start 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 librados: setting wanted keys 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 librados: calling monclient init 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e40fec80 con 0x7f30e4103190 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e410ad10 con 0x7f30e410eab0 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30ec110700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f30e41a9640 con 0x7f30e410afa0 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e96ab700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f30e410afa0 0x7f30e4108230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58256/0 (socket says 172.21.15.28:58256) 2024-03-30T15:16:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e96ab700 1 -- 172.21.15.28:0/2189931992 learned_addr learned my addr 172.21.15.28:0/2189931992 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter ms_handle_connect 0x7f30e410afa0 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter ms_handle_connect 0x7f30e4103190 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505683019 0 0) 0x7f30e41a9640 con 0x7f30e410afa0 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30cc0031f0 con 0x7f30e410afa0 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922107405 0 0) 0x7f30e40fec80 con 0x7f30e4103190 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e41a9640 con 0x7f30e4103190 2024-03-30T15:16:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter ms_handle_connect 0x7f30e410eab0 2024-03-30T15:16:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085882963 0 0) 0x7f30e410ad10 con 0x7f30e410eab0 2024-03-30T15:16:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e40fec80 con 0x7f30e410eab0 2024-03-30T15:16:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.708+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3610997769 0 0) 0x7f30cc0031f0 con 0x7f30e410afa0 2024-03-30T15:16:44.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e410ad10 con 0x7f30e410afa0 2024-03-30T15:16:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30d8001930 con 0x7f30e410afa0 2024-03-30T15:16:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2592606608 0 0) 0x7f30e41a9640 con 0x7f30e4103190 2024-03-30T15:16:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30cc0031f0 con 0x7f30e4103190 2024-03-30T15:16:44.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30dc001230 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3880254159 0 0) 0x7f30e40fec80 con 0x7f30e410eab0 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e41a9640 con 0x7f30e410eab0 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30d4001b80 con 0x7f30e410eab0 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4111591766 0 0) 0x7f30cc0031f0 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 >> v1:172.21.15.28:6790/0 conn(0x7f30e410afa0 legacy=0x7f30e4108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 >> v1:172.21.15.134:6789/0 conn(0x7f30e410eab0 legacy=0x7f30e41a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e41aa820 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30e41a9870 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30ec110700 10 client.26157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30e41a9d30 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30ec110700 1 librados: init done 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f30dc003b20 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.709+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f30dc0048e0 con 0x7f30e4103190 2024-03-30T15:16:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30ec110700 10 client.26157.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30ec110700 15 client.26157.objecter _session_command_op_assign -1 1 2024-03-30T15:16:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30ec110700 20 client.26157.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30ec110700 10 client.26157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f30d0002920 con 0x7f30e4103190 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.710+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f30dc01b120 con 0x7f30e4103190 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(952..952 src has 881..952) ==== 7561+0+0 (unknown 3875643232 0 0) 0x7f30dc0881f0 con 0x7f30e4103190 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 10 client.26157.objecter ms_dispatch 0x7f30e4001550 osd_map(952..952 src has 881..952) 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 3 client.26157.objecter handle_osd_map got epochs [952,952] > 0 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 3 client.26157.objecter handle_osd_map decoding full epoch 952 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 10 client.26157.objecter checking command 1 2024-03-30T15:16:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter _get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 15 client.26157.objecter _session_command_op_remove -1 1 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter _get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 15 client.26157.objecter _session_command_op_assign 3 1 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 4 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 10 client.26157.objecter _send_command 1 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6813/1506157250 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30d4001400 con 0x7f30cc07cb80 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 20 client.26157.objecter dump_active .. 0 homeless 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=952) ==== 24+0+0 (unknown 1915582963 0 0) 0x7f30dc088660 con 0x7f30e4103190 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 10 client.26157.objecter ms_handle_connect 0x7f30cc070450 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.711+0000 7f30e2ffd700 10 client.26157.objecter ms_handle_connect 0x7f30cc07cb80 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.712+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== osd.3 v1:172.21.15.28:6813/1506157250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f30d4001400 con 0x7f30cc07cb80 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.712+0000 7f30e2ffd700 10 client.26157.objecter ms_dispatch 0x7f30e4001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.712+0000 7f30e2ffd700 10 client.26157.objecter _finish_command 1 = system:0 2024-03-30T15:16:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.712+0000 7f30e2ffd700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 4 2024-03-30T15:16:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.712+0000 7f30e2ffd700 15 client.26157.objecter _session_command_op_remove 3 1 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 10 client.26157.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 15 client.26157.objecter _session_command_op_assign -1 2 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 20 client.26157.objecter _get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 20 client.26157.objecter _get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 15 client.26157.objecter _session_command_op_remove -1 2 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 20 client.26157.objecter get_session s=0x7f30cc07c4d0 osd=3 3 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 15 client.26157.objecter _session_command_op_assign 3 2 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 4 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 10 client.26157.objecter _send_command 2 2024-03-30T15:16:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.732+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 --> v1:172.21.15.28:6813/1506157250 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f30d00011f0 con 0x7f30cc07cb80 2024-03-30T15:16:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30e2ffd700 1 -- 172.21.15.28:0/2189931992 <== osd.3 v1:172.21.15.28:6813/1506157250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 685278730) 0x7f30d00011f0 con 0x7f30cc07cb80 2024-03-30T15:16:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30e2ffd700 10 client.26157.objecter ms_dispatch 0x7f30e4001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30e2ffd700 10 client.26157.objecter _finish_command 2 = system:0 2024-03-30T15:16:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30e2ffd700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 4 2024-03-30T15:16:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30e2ffd700 15 client.26157.objecter _session_command_op_remove 3 2 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 10 librados: watch_flush enter 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 10 librados: watch_flush exit 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 10 client.26157.objecter close_session for osd.3 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 >> v1:172.21.15.28:6813/1506157250 conn(0x7f30cc07cb80 legacy=0x7f30cc07f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 20 client.26157.objecter put_session s=0x7f30cc07c4d0 osd=3 1 2024-03-30T15:16:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 20 client.26157.objecter shutdown clearing up homeless session... 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 10 client.26157.objecter successfully canceled tick 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 >> v1:172.21.15.28:6800/3011431093 conn(0x7f30cc070450 legacy=0x7f30cc072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 >> v1:172.21.15.28:6789/0 conn(0x7f30e4103190 legacy=0x7f30e4107b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ea6ad700 1 -- 172.21.15.28:0/2189931992 reap_dead start 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 shutdown_connections 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 >> 172.21.15.28:0/2189931992 conn(0x7f30e40fc9e0 msgr2=0x7f30e4110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 shutdown_connections 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 -- 172.21.15.28:0/2189931992 wait complete. 2024-03-30T15:16:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.733+0000 7f30ec110700 1 librados: shutdown 2024-03-30T15:16:44.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[41614]: 4.3 scrub starts 2024-03-30T15:16:44.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[41614]: 4.3 scrub ok 2024-03-30T15:16:44.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[41614]: pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:44.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[29876]: 4.3 scrub starts 2024-03-30T15:16:44.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[29876]: 4.3 scrub ok 2024-03-30T15:16:44.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:44 smithi028 ceph-mon[29876]: pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:44 smithi134 ceph-mon[33714]: 4.3 scrub starts 2024-03-30T15:16:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:44 smithi134 ceph-mon[33714]: 4.3 scrub ok 2024-03-30T15:16:45.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:44 smithi134 ceph-mon[33714]: pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 -- 172.21.15.28:0/180394055 >> v1:172.21.15.28:6790/0 conn(0x7f8bb010d100 legacy=0x7f8bb010f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 -- 172.21.15.28:0/180394055 shutdown_connections 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 -- 172.21.15.28:0/180394055 >> 172.21.15.28:0/180394055 conn(0x7f8bb00772c0 msgr2=0x7f8bb00776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 -- 172.21.15.28:0/180394055 shutdown_connections 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 -- 172.21.15.28:0/180394055 wait complete. 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 librados: starting msgr at 2024-03-30T15:16:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.347+0000 7f8bb6b32700 1 librados: starting objecter 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 Processor -- start 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 -- start start 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 librados: setting wanted keys 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 librados: calling monclient init 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bb01a4e90 con 0x7f8bb0105680 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bb01a60b0 con 0x7f8bb0109500 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb6b32700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8bb01a72f0 con 0x7f8bb010d100 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb48ce700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8bb0105680 0x7f8bb01098e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60854/0 (socket says 172.21.15.28:60854) 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb48ce700 1 -- 172.21.15.28:0/3208964039 learned_addr learned my addr 172.21.15.28:0/3208964039 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8bb50cf700 1 --1- 172.21.15.28:0/3208964039 >> v1:172.21.15.28:6790/0 conn(0x7f8bb010d100 0x7f8bb01a3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58294/0 (socket says 172.21.15.28:58294) 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter ms_handle_connect 0x7f8bb0105680 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter ms_handle_connect 0x7f8bb010d100 2024-03-30T15:16:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961921813 0 0) 0x7f8bb01a4e90 con 0x7f8bb0105680 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba0004370 con 0x7f8bb0105680 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter ms_handle_connect 0x7f8bb0109500 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688378140 0 0) 0x7f8bb01a72f0 con 0x7f8bb010d100 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb01a4e90 con 0x7f8bb010d100 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266093874 0 0) 0x7f8bb01a60b0 con 0x7f8bb0109500 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.348+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb01a72f0 con 0x7f8bb0109500 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1277530130 0 0) 0x7f8ba0004370 con 0x7f8bb0105680 2024-03-30T15:16:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bb01a60b0 con 0x7f8bb0105680 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ba4001960 con 0x7f8bb0105680 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4093271326 0 0) 0x7f8bb01a4e90 con 0x7f8bb010d100 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ba0004370 con 0x7f8bb010d100 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ba80047f0 con 0x7f8bb010d100 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3651334683 0 0) 0x7f8bb01a72f0 con 0x7f8bb0109500 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bb01a4e90 con 0x7f8bb0109500 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8b9c001870 con 0x7f8bb0109500 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 405471345 0 0) 0x7f8bb01a60b0 con 0x7f8bb0105680 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 >> v1:172.21.15.28:6790/0 conn(0x7f8bb010d100 legacy=0x7f8bb01a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 >> v1:172.21.15.134:6789/0 conn(0x7f8bb0109500 legacy=0x7f8bb0109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb01a8530 con 0x7f8bb0105680 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb01a7520 con 0x7f8bb0105680 2024-03-30T15:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f8ba4003b20 con 0x7f8bb0105680 2024-03-30T15:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f8ba4005d10 con 0x7f8bb0105680 2024-03-30T15:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8bb6b32700 10 client.26169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.349+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bb01a79e0 con 0x7f8bb0105680 2024-03-30T15:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 1 librados: init done 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 10 client.26169.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 15 client.26169.objecter _session_command_op_assign -1 1 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 20 client.26169.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 10 client.26169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.350+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8bb0101d90 con 0x7f8bb0105680 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f8ba4003680 con 0x7f8bb0105680 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(952..952 src has 881..952) ==== 7561+0+0 (unknown 3875643232 0 0) 0x7f8ba4089390 con 0x7f8bb0105680 2024-03-30T15:16:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 10 client.26169.objecter ms_dispatch 0x7f8bb0001550 osd_map(952..952 src has 881..952) 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 3 client.26169.objecter handle_osd_map got epochs [952,952] > 0 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 3 client.26169.objecter handle_osd_map decoding full epoch 952 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 10 client.26169.objecter checking command 1 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter _get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 15 client.26169.objecter _session_command_op_remove -1 1 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter _get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 15 client.26169.objecter _session_command_op_assign 1 1 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 4 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 10 client.26169.objecter _send_command 1 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6805/1850313547 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ba0082a60 con 0x7f8ba007ed50 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 20 client.26169.objecter dump_active .. 0 homeless 2024-03-30T15:16:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.352+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=952) ==== 24+0+0 (unknown 1915582963 0 0) 0x7f8ba40897d0 con 0x7f8bb0105680 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.353+0000 7f8badffb700 10 client.26169.objecter ms_handle_connect 0x7f8ba00726a0 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.353+0000 7f8badffb700 10 client.26169.objecter ms_handle_connect 0x7f8ba007ed50 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.354+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== osd.1 v1:172.21.15.28:6805/1850313547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ba0082a60 con 0x7f8ba007ed50 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.354+0000 7f8badffb700 10 client.26169.objecter ms_dispatch 0x7f8bb0001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.354+0000 7f8badffb700 10 client.26169.objecter _finish_command 1 = system:0 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.354+0000 7f8badffb700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 4 2024-03-30T15:16:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.354+0000 7f8badffb700 15 client.26169.objecter _session_command_op_remove 1 1 2024-03-30T15:16:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 10 client.26169.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 15 client.26169.objecter _session_command_op_assign -1 2 2024-03-30T15:16:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 20 client.26169.objecter _get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 20 client.26169.objecter _get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 15 client.26169.objecter _session_command_op_remove -1 2 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 20 client.26169.objecter get_session s=0x7f8ba007e6a0 osd=1 3 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 15 client.26169.objecter _session_command_op_assign 1 2 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 4 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 10 client.26169.objecter _send_command 2 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.371+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 --> v1:172.21.15.28:6805/1850313547 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8bb0068b30 con 0x7f8ba007ed50 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8badffb700 1 -- 172.21.15.28:0/3208964039 <== osd.1 v1:172.21.15.28:6805/1850313547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2059778697) 0x7f8bb0068b30 con 0x7f8ba007ed50 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8badffb700 10 client.26169.objecter ms_dispatch 0x7f8bb0001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8badffb700 10 client.26169.objecter _finish_command 2 = system:0 2024-03-30T15:16:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8badffb700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 4 2024-03-30T15:16:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8badffb700 15 client.26169.objecter _session_command_op_remove 1 2 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 10 librados: watch_flush enter 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 10 librados: watch_flush exit 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 10 client.26169.objecter close_session for osd.1 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 >> v1:172.21.15.28:6805/1850313547 conn(0x7f8ba007ed50 legacy=0x7f8ba00811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 20 client.26169.objecter put_session s=0x7f8ba007e6a0 osd=1 1 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 20 client.26169.objecter shutdown clearing up homeless session... 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 10 client.26169.objecter successfully canceled tick 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 >> v1:172.21.15.28:6800/3011431093 conn(0x7f8ba00726a0 legacy=0x7f8ba0074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 >> v1:172.21.15.28:6789/0 conn(0x7f8bb0105680 legacy=0x7f8bb01098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb50cf700 1 -- 172.21.15.28:0/3208964039 reap_dead start 2024-03-30T15:16:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 shutdown_connections 2024-03-30T15:16:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 >> 172.21.15.28:0/3208964039 conn(0x7f8bb00772c0 msgr2=0x7f8bb0110780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 shutdown_connections 2024-03-30T15:16:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.372+0000 7f8bb6b32700 1 -- 172.21.15.28:0/3208964039 wait complete. 2024-03-30T15:16:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:45.373+0000 7f8bb6b32700 1 librados: shutdown 2024-03-30T15:16:45.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 3.f scrub starts 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 3.f scrub ok 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 12.1e scrub starts 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 12.1e scrub ok 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 3.2 deep-scrub starts 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: 3.2 deep-scrub ok 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 3.f scrub starts 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 3.f scrub ok 2024-03-30T15:16:45.765 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 12.1e scrub starts 2024-03-30T15:16:45.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 12.1e scrub ok 2024-03-30T15:16:45.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 3.2 deep-scrub starts 2024-03-30T15:16:45.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: 3.2 deep-scrub ok 2024-03-30T15:16:45.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:45.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:45 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 -- 172.21.15.28:0/56627224 >> v1:172.21.15.28:6789/0 conn(0x7f777410aef0 legacy=0x7f777410d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 -- 172.21.15.28:0/56627224 shutdown_connections 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 -- 172.21.15.28:0/56627224 >> 172.21.15.28:0/56627224 conn(0x7f77740fdb30 msgr2=0x7f77740fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 -- 172.21.15.28:0/56627224 shutdown_connections 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 -- 172.21.15.28:0/56627224 wait complete. 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 librados: starting msgr at 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.010+0000 7f7779d05700 1 librados: starting objecter 2024-03-30T15:16:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 Processor -- start 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 -- start start 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 librados: setting wanted keys 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 librados: calling monclient init 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77741078b0 con 0x7f777410aef0 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7774107ac0 con 0x7f7774102260 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f7779d05700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f77741a9670 con 0x7f777410ea90 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f776affd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f777410aef0 0x7f7774104e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60874/0 (socket says 172.21.15.28:60874) 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f776affd700 1 -- 172.21.15.28:0/3494526305 learned_addr learned my addr 172.21.15.28:0/3494526305 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter ms_handle_connect 0x7f777410aef0 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter ms_handle_connect 0x7f777410ea90 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:46.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter ms_handle_connect 0x7f7774102260 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701733734 0 0) 0x7f77741078b0 con 0x7f777410aef0 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7760003aa0 con 0x7f777410aef0 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465001367 0 0) 0x7f77741a9670 con 0x7f777410ea90 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.011+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77741078b0 con 0x7f777410ea90 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573612235 0 0) 0x7f7774107ac0 con 0x7f7774102260 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77741a9670 con 0x7f7774102260 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 557776540 0 0) 0x7f7760003aa0 con 0x7f777410aef0 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7774107ac0 con 0x7f777410aef0 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f775c004550 con 0x7f777410aef0 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1183985631 0 0) 0x7f77741078b0 con 0x7f777410ea90 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7760003aa0 con 0x7f777410ea90 2024-03-30T15:16:46.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f776c0013f0 con 0x7f777410ea90 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638922977 0 0) 0x7f77741a9670 con 0x7f7774102260 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77741078b0 con 0x7f7774102260 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7764001830 con 0x7f7774102260 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2685433738 0 0) 0x7f7774107ac0 con 0x7f777410aef0 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 >> v1:172.21.15.28:6790/0 conn(0x7f777410ea90 legacy=0x7f77741a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 >> v1:172.21.15.134:6789/0 conn(0x7f7774102260 legacy=0x7f7774104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77741aa850 con 0x7f777410aef0 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77741a98a0 con 0x7f777410aef0 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f7779d05700 10 client.26175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f77741a9d60 con 0x7f777410aef0 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f7779d05700 1 librados: init done 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f775c003b20 con 0x7f777410aef0 2024-03-30T15:16:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.012+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f775c0050d0 con 0x7f777410aef0 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.013+0000 7f7779d05700 10 client.26175.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.013+0000 7f7779d05700 15 client.26175.objecter _session_command_op_assign -1 1 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.013+0000 7f7779d05700 20 client.26175.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.013+0000 7f7779d05700 10 client.26175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.013+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7774103740 con 0x7f777410aef0 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f775c0053c0 con 0x7f777410aef0 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(953..953 src has 881..953) ==== 7561+0+0 (unknown 3896252173 0 0) 0x7f775c0895d0 con 0x7f777410aef0 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 10 client.26175.objecter ms_dispatch 0x7f7774001550 osd_map(953..953 src has 881..953) 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 3 client.26175.objecter handle_osd_map got epochs [953,953] > 0 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 3 client.26175.objecter handle_osd_map decoding full epoch 953 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 10 client.26175.objecter checking command 1 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter _get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 15 client.26175.objecter _session_command_op_remove -1 1 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter _get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 15 client.26175.objecter _session_command_op_assign 2 1 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 4 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 10 client.26175.objecter _send_command 1 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6801/2294873818 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7760080930 con 0x7f776007cc20 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 20 client.26175.objecter dump_active .. 0 homeless 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_get_version_reply(handle=1 version=953) ==== 24+0+0 (unknown 2149676045 0 0) 0x7f775c089a40 con 0x7f777410aef0 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.014+0000 7f77717fa700 10 client.26175.objecter ms_handle_connect 0x7f77600704f0 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 10 client.26175.objecter ms_handle_connect 0x7f776007cc20 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== osd.2 v1:172.21.15.28:6801/2294873818 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7760080930 con 0x7f776007cc20 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 10 client.26175.objecter ms_dispatch 0x7f7774001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 10 client.26175.objecter _finish_command 1 = system:0 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 4 2024-03-30T15:16:46.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.015+0000 7f77717fa700 15 client.26175.objecter _session_command_op_remove 2 1 2024-03-30T15:16:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 client.26175.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 15 client.26175.objecter _session_command_op_assign -1 2 2024-03-30T15:16:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter _get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter _get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 15 client.26175.objecter _session_command_op_remove -1 2 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter get_session s=0x7f776007c570 osd=2 3 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 15 client.26175.objecter _session_command_op_assign 2 2 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 4 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 client.26175.objecter _send_command 2 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 --> v1:172.21.15.28:6801/2294873818 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7774068b30 con 0x7f776007cc20 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f77717fa700 1 -- 172.21.15.28:0/3494526305 <== osd.2 v1:172.21.15.28:6801/2294873818 2 ==== command_reply(tid 2: 0 ) ==== 8+0+13 (unknown 0 0 110240887) 0x7f7774068b30 con 0x7f776007cc20 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f77717fa700 10 client.26175.objecter ms_dispatch 0x7f7774001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f77717fa700 10 client.26175.objecter _finish_command 2 = system:0 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f77717fa700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 4 2024-03-30T15:16:46.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f77717fa700 15 client.26175.objecter _session_command_op_remove 2 2 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 librados: watch_flush enter 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 librados: watch_flush exit 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 client.26175.objecter close_session for osd.2 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 >> v1:172.21.15.28:6801/2294873818 conn(0x7f776007cc20 legacy=0x7f776007f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter put_session s=0x7f776007c570 osd=2 1 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 20 client.26175.objecter shutdown clearing up homeless session... 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 10 client.26175.objecter successfully canceled tick 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 >> v1:172.21.15.28:6800/3011431093 conn(0x7f77600704f0 legacy=0x7f77600729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 >> v1:172.21.15.28:6789/0 conn(0x7f777410aef0 legacy=0x7f7774104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.032+0000 7f7773fff700 1 -- 172.21.15.28:0/3494526305 reap_dead start 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.033+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 shutdown_connections 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.033+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 >> 172.21.15.28:0/3494526305 conn(0x7f77740fdb30 msgr2=0x7f7774111c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.033+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 shutdown_connections 2024-03-30T15:16:46.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.033+0000 7f7779d05700 1 -- 172.21.15.28:0/3494526305 wait complete. 2024-03-30T15:16:46.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:46.033+0000 7f7779d05700 1 librados: shutdown 2024-03-30T15:16:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 3.f scrub starts 2024-03-30T15:16:46.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 3.f scrub ok 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 12.1e scrub starts 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 12.1e scrub ok 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 3.2 deep-scrub starts 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: 3.2 deep-scrub ok 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:46.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:45 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[41614]: 6.1c scrub starts 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[41614]: 6.1c scrub ok 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[41614]: osdmap e953: 8 total, 8 up, 8 in 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[41614]: pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[29876]: 6.1c scrub starts 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[29876]: 6.1c scrub ok 2024-03-30T15:16:46.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[29876]: osdmap e953: 8 total, 8 up, 8 in 2024-03-30T15:16:46.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:46 smithi028 ceph-mon[29876]: pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:16:47.012 INFO:teuthology.orchestra.run.smithi028.stdout:115964117284 2024-03-30T15:16:47.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.948+0000 7f95a3fa6700 1 -- 172.21.15.28:0/3221931769 >> v1:172.21.15.28:6790/0 conn(0x7f959c100720 legacy=0x7f959c100b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.948+0000 7f95a3fa6700 1 -- 172.21.15.28:0/3221931769 shutdown_connections 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.948+0000 7f95a3fa6700 1 -- 172.21.15.28:0/3221931769 >> 172.21.15.28:0/3221931769 conn(0x7f959c077370 msgr2=0x7f959c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.948+0000 7f95a3fa6700 1 -- 172.21.15.28:0/3221931769 shutdown_connections 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.948+0000 7f95a3fa6700 1 -- 172.21.15.28:0/3221931769 wait complete. 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 librados: starting msgr at 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 librados: starting objecter 2024-03-30T15:16:47.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 Processor -- start 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 -- start start 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 librados: setting wanted keys 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 librados: calling monclient init 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f959c1a9340 con 0x7f959c10ea70 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f959c1aa560 con 0x7f959c10af00 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a3fa6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f959c1ab7a0 con 0x7f959c100720 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a1d42700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f959c100720 0x7f959c107540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58282/0 (socket says 172.21.15.28:58282) 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f95a1d42700 1 -- 172.21.15.28:0/298161798 learned_addr learned my addr 172.21.15.28:0/298161798 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter ms_handle_connect 0x7f959c100720 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter ms_handle_connect 0x7f959c10ea70 2024-03-30T15:16:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter ms_handle_connect 0x7f959c10af00 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862514332 0 0) 0x7f959c1ab7a0 con 0x7f959c100720 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95880039c0 con 0x7f959c100720 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106834814 0 0) 0x7f959c1a9340 con 0x7f959c10ea70 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.949+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959c1ab7a0 con 0x7f959c10ea70 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504966392 0 0) 0x7f959c1aa560 con 0x7f959c10af00 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f959c1a9340 con 0x7f959c10af00 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3716132887 0 0) 0x7f95880039c0 con 0x7f959c100720 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f959c1aa560 con 0x7f959c100720 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f95980017b0 con 0x7f959c100720 2024-03-30T15:16:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4074351471 0 0) 0x7f959c1ab7a0 con 0x7f959c10ea70 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95880039c0 con 0x7f959c10ea70 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9594001bf0 con 0x7f959c10ea70 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1945822514 0 0) 0x7f959c1a9340 con 0x7f959c10af00 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f959c1ab7a0 con 0x7f959c10af00 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f958c001a30 con 0x7f959c10af00 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1824273083 0 0) 0x7f959c1aa560 con 0x7f959c100720 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 >> v1:172.21.15.134:6789/0 conn(0x7f959c10af00 legacy=0x7f959c1a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 >> v1:172.21.15.28:6789/0 conn(0x7f959c10ea70 legacy=0x7f959c1a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c1ac9e0 con 0x7f959c100720 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f959c1a9570 con 0x7f959c100720 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f95a3fa6700 10 client.59366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f959c1a9a30 con 0x7f959c100720 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f95a3fa6700 1 librados: init done 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f95980044f0 con 0x7f959c100720 2024-03-30T15:16:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.950+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9598004ee0 con 0x7f959c100720 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.951+0000 7f95a3fa6700 10 client.59366.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.951+0000 7f95a3fa6700 15 client.59366.objecter _session_command_op_assign -1 1 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.951+0000 7f95a3fa6700 20 client.59366.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.951+0000 7f95a3fa6700 10 client.59366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.951+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9580002920 con 0x7f959c100720 2024-03-30T15:16:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f95980039c0 con 0x7f959c100720 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(952..952 src has 881..952) ==== 7561+0+0 (unknown 3875643232 0 0) 0x7f9598089450 con 0x7f959c100720 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 10 client.59366.objecter ms_dispatch 0x7f959c001550 osd_map(952..952 src has 881..952) 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 3 client.59366.objecter handle_osd_map got epochs [952,952] > 0 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 3 client.59366.objecter handle_osd_map decoding full epoch 952 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 10 client.59366.objecter checking command 1 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter _get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 15 client.59366.objecter _session_command_op_remove -1 1 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter _get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 15 client.59366.objecter _session_command_op_assign 7 1 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 4 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 10 client.59366.objecter _send_command 1 2024-03-30T15:16:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.134:6812/3758190298 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95880807b0 con 0x7f958807caa0 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 20 client.59366.objecter dump_active .. 0 homeless 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=952) ==== 24+0+0 (unknown 1915582963 0 0) 0x7f9598089920 con 0x7f959c100720 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.952+0000 7f9592ffd700 10 client.59366.objecter ms_handle_connect 0x7f95880703f0 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.953+0000 7f9592ffd700 10 client.59366.objecter ms_handle_connect 0x7f958807caa0 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.954+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== osd.7 v1:172.21.15.134:6812/3758190298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f95880807b0 con 0x7f958807caa0 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.954+0000 7f9592ffd700 10 client.59366.objecter ms_dispatch 0x7f959c001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.954+0000 7f9592ffd700 10 client.59366.objecter _finish_command 1 = system:0 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.954+0000 7f9592ffd700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 4 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.954+0000 7f9592ffd700 15 client.59366.objecter _session_command_op_remove 7 1 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 10 client.59366.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 15 client.59366.objecter _session_command_op_assign -1 2 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 20 client.59366.objecter _get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 20 client.59366.objecter _get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 15 client.59366.objecter _session_command_op_remove -1 2 2024-03-30T15:16:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 20 client.59366.objecter get_session s=0x7f958807c3f0 osd=7 3 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 15 client.59366.objecter _session_command_op_assign 7 2 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 4 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 10 client.59366.objecter _send_command 2 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.970+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 --> v1:172.21.15.134:6812/3758190298 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f95800011f0 con 0x7f958807caa0 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f9592ffd700 1 -- 172.21.15.28:0/298161798 <== osd.7 v1:172.21.15.134:6812/3758190298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1942476598) 0x7f95800011f0 con 0x7f958807caa0 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f9592ffd700 10 client.59366.objecter ms_dispatch 0x7f959c001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f9592ffd700 10 client.59366.objecter _finish_command 2 = system:0 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f9592ffd700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 4 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f9592ffd700 15 client.59366.objecter _session_command_op_remove 7 2 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 10 librados: watch_flush enter 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 10 librados: watch_flush exit 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 10 client.59366.objecter close_session for osd.7 2024-03-30T15:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 >> v1:172.21.15.134:6812/3758190298 conn(0x7f958807caa0 legacy=0x7f958807ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 20 client.59366.objecter put_session s=0x7f958807c3f0 osd=7 1 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 20 client.59366.objecter shutdown clearing up homeless session... 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 10 client.59366.objecter successfully canceled tick 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 >> v1:172.21.15.28:6800/3011431093 conn(0x7f95880703f0 legacy=0x7f95880728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 >> v1:172.21.15.28:6790/0 conn(0x7f959c100720 legacy=0x7f959c107540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a2543700 1 -- 172.21.15.28:0/298161798 reap_dead start 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 shutdown_connections 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 >> 172.21.15.28:0/298161798 conn(0x7f959c077370 msgr2=0x7f959c110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 shutdown_connections 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 -- 172.21.15.28:0/298161798 wait complete. 2024-03-30T15:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:44.971+0000 7f95a3fa6700 1 librados: shutdown 2024-03-30T15:16:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:46 smithi134 ceph-mon[33714]: 6.1c scrub starts 2024-03-30T15:16:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:46 smithi134 ceph-mon[33714]: 6.1c scrub ok 2024-03-30T15:16:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:46 smithi134 ceph-mon[33714]: osdmap e953: 8 total, 8 up, 8 in 2024-03-30T15:16:47.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:46 smithi134 ceph-mon[33714]: pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:16:47.771 INFO:teuthology.orchestra.run.smithi028.stdout:261993005327 2024-03-30T15:16:47.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T15:16:48.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 4.1a scrub starts 2024-03-30T15:16:48.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 4.1a scrub ok 2024-03-30T15:16:48.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 12.11 scrub starts 2024-03-30T15:16:48.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 12.11 scrub ok 2024-03-30T15:16:48.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 4.1e scrub starts 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: 4.1e scrub ok 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[41614]: osdmap e954: 8 total, 8 up, 8 in 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 4.1a scrub starts 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 4.1a scrub ok 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 12.11 scrub starts 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 12.11 scrub ok 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 4.1e scrub starts 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: 4.1e scrub ok 2024-03-30T15:16:48.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:47 smithi028 ceph-mon[29876]: osdmap e954: 8 total, 8 up, 8 in 2024-03-30T15:16:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 4.1a scrub starts 2024-03-30T15:16:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 4.1a scrub ok 2024-03-30T15:16:48.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 12.11 scrub starts 2024-03-30T15:16:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 12.11 scrub ok 2024-03-30T15:16:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 4.1e scrub starts 2024-03-30T15:16:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: 4.1e scrub ok 2024-03-30T15:16:48.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:47 smithi134 ceph-mon[33714]: osdmap e954: 8 total, 8 up, 8 in 2024-03-30T15:16:48.365 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:48.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.468+0000 7f3419db2700 1 -- 172.21.15.28:0/28840974 >> v1:172.21.15.28:6790/0 conn(0x7f3414103190 legacy=0x7f3414103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 -- 172.21.15.28:0/28840974 shutdown_connections 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 -- 172.21.15.28:0/28840974 >> 172.21.15.28:0/28840974 conn(0x7f34140fc9e0 msgr2=0x7f34140fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 -- 172.21.15.28:0/28840974 shutdown_connections 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 -- 172.21.15.28:0/28840974 wait complete. 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 librados: starting msgr at 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 librados: starting objecter 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 Processor -- start 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 -- start start 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 librados: setting wanted keys 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.469+0000 7f3419db2700 1 librados: calling monclient init 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3419db2700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34141a9370 con 0x7f3414103190 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3419db2700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34141aa590 con 0x7f341410eab0 2024-03-30T15:16:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3419db2700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f34141ab7d0 con 0x7f341410afa0 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f34137fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3414103190 0x7f341410a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60886/0 (socket says 172.21.15.28:60886) 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f34137fe700 1 -- 172.21.15.28:0/1923362291 learned_addr learned my addr 172.21.15.28:0/1923362291 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3412ffd700 1 --1- 172.21.15.28:0/1923362291 >> v1:172.21.15.28:6790/0 conn(0x7f341410afa0 0x7f34141a42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:58308/0 (socket says 172.21.15.28:58308) 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter ms_handle_connect 0x7f341410afa0 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter ms_handle_connect 0x7f3414103190 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter ms_handle_connect 0x7f341410eab0 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193148898 0 0) 0x7f34141ab7d0 con 0x7f341410afa0 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3400003aa0 con 0x7f341410afa0 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009860125 0 0) 0x7f34141a9370 con 0x7f3414103190 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34141ab7d0 con 0x7f3414103190 2024-03-30T15:16:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938218739 0 0) 0x7f34141aa590 con 0x7f341410eab0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34141a9370 con 0x7f341410eab0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1655533995 0 0) 0x7f3400003aa0 con 0x7f341410afa0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34141aa590 con 0x7f341410afa0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f33fc0013d0 con 0x7f341410afa0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3979066034 0 0) 0x7f34141ab7d0 con 0x7f3414103190 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3400003aa0 con 0x7f3414103190 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3404004600 con 0x7f3414103190 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2838687811 0 0) 0x7f34141a9370 con 0x7f341410eab0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34141ab7d0 con 0x7f341410eab0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f340c001af0 con 0x7f341410eab0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2122898926 0 0) 0x7f34141aa590 con 0x7f341410afa0 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 >> v1:172.21.15.134:6789/0 conn(0x7f341410eab0 legacy=0x7f34141a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 >> v1:172.21.15.28:6789/0 conn(0x7f3414103190 legacy=0x7f341410a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34141aca10 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.470+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34141a95a0 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 10 client.59384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f34141a9a60 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 1 librados: init done 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f33fc001680 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f33fc005c60 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 10 client.59384.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 15 client.59384.objecter _session_command_op_assign -1 1 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 20 client.59384.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 10 client.59384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.471+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f33f4002920 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.472+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f33fc003930 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(954..954 src has 881..954) ==== 7561+0+0 (unknown 4185558970 0 0) 0x7f33fc089110 con 0x7f341410afa0 2024-03-30T15:16:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 10 client.59384.objecter ms_dispatch 0x7f3414001550 osd_map(954..954 src has 881..954) 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 3 client.59384.objecter handle_osd_map got epochs [954,954] > 0 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 3 client.59384.objecter handle_osd_map decoding full epoch 954 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 10 client.59384.objecter checking command 1 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter _get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 15 client.59384.objecter _session_command_op_remove -1 1 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter _get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 15 client.59384.objecter _session_command_op_assign 6 1 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 4 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 10 client.59384.objecter _send_command 1 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.134:6808/2135224194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3400080740 con 0x7f340007ca30 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 20 client.59384.objecter dump_active .. 0 homeless 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=954) ==== 24+0+0 (unknown 2480477438 0 0) 0x7f33fc0895b0 con 0x7f341410afa0 2024-03-30T15:16:48.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 10 client.59384.objecter ms_handle_connect 0x7f34000702f0 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.473+0000 7f3410ff9700 10 client.59384.objecter ms_handle_connect 0x7f340007ca30 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.474+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== osd.6 v1:172.21.15.134:6808/2135224194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3400080740 con 0x7f340007ca30 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.474+0000 7f3410ff9700 10 client.59384.objecter ms_dispatch 0x7f3414001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.474+0000 7f3410ff9700 10 client.59384.objecter _finish_command 1 = system:0 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.474+0000 7f3410ff9700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 4 2024-03-30T15:16:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.474+0000 7f3410ff9700 15 client.59384.objecter _session_command_op_remove 6 1 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 client.59384.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 15 client.59384.objecter _session_command_op_assign -1 2 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter _get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter _get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 15 client.59384.objecter _session_command_op_remove -1 2 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter get_session s=0x7f340007c380 osd=6 3 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 15 client.59384.objecter _session_command_op_assign 6 2 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 4 2024-03-30T15:16:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 client.59384.objecter _send_command 2 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 --> v1:172.21.15.134:6808/2135224194 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f33f40011f0 con 0x7f340007ca30 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3410ff9700 1 -- 172.21.15.28:0/1923362291 <== osd.6 v1:172.21.15.134:6808/2135224194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1166788469) 0x7f33f40011f0 con 0x7f340007ca30 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3410ff9700 10 client.59384.objecter ms_dispatch 0x7f3414001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3410ff9700 10 client.59384.objecter _finish_command 2 = system:0 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3410ff9700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 4 2024-03-30T15:16:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3410ff9700 15 client.59384.objecter _session_command_op_remove 6 2 2024-03-30T15:16:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 librados: watch_flush enter 2024-03-30T15:16:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 librados: watch_flush exit 2024-03-30T15:16:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 client.59384.objecter close_session for osd.6 2024-03-30T15:16:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 >> v1:172.21.15.134:6808/2135224194 conn(0x7f340007ca30 legacy=0x7f340007eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter put_session s=0x7f340007c380 osd=6 1 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 20 client.59384.objecter shutdown clearing up homeless session... 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.491+0000 7f3419db2700 10 client.59384.objecter successfully canceled tick 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 >> v1:172.21.15.28:6800/3011431093 conn(0x7f34000702f0 legacy=0x7f34000727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 >> v1:172.21.15.28:6790/0 conn(0x7f341410afa0 legacy=0x7f34141a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3413fff700 1 -- 172.21.15.28:0/1923362291 reap_dead start 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 shutdown_connections 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 >> 172.21.15.28:0/1923362291 conn(0x7f34140fc9e0 msgr2=0x7f34140fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 shutdown_connections 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 -- 172.21.15.28:0/1923362291 wait complete. 2024-03-30T15:16:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:48.492+0000 7f3419db2700 1 librados: shutdown 2024-03-30T15:16:49.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:48 smithi134 ceph-mon[33714]: 3.c scrub starts 2024-03-30T15:16:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:48 smithi134 ceph-mon[33714]: 3.c scrub ok 2024-03-30T15:16:49.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:48 smithi134 ceph-mon[33714]: pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[29876]: 3.c scrub starts 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[29876]: 3.c scrub ok 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[29876]: pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[41614]: 3.c scrub starts 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[41614]: 3.c scrub ok 2024-03-30T15:16:49.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:48 smithi028 ceph-mon[41614]: pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:16:49.528 INFO:teuthology.orchestra.run.smithi028.stdout:223338299668 2024-03-30T15:16:49.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T15:16:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.740+0000 7f93a7854700 1 -- 172.21.15.28:0/455515421 >> v1:172.21.15.28:6790/0 conn(0x7f93a010eaf0 legacy=0x7f93a0110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.740+0000 7f93a7854700 1 -- 172.21.15.28:0/455515421 shutdown_connections 2024-03-30T15:16:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.740+0000 7f93a7854700 1 -- 172.21.15.28:0/455515421 >> 172.21.15.28:0/455515421 conn(0x7f93a00fc9e0 msgr2=0x7f93a00fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.740+0000 7f93a7854700 1 -- 172.21.15.28:0/455515421 shutdown_connections 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.740+0000 7f93a7854700 1 -- 172.21.15.28:0/455515421 wait complete. 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 librados: starting msgr at 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 librados: starting objecter 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 Processor -- start 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 -- start start 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 librados: setting wanted keys 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 librados: calling monclient init 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a010ab70 con 0x7f93a01031d0 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a010ad80 con 0x7f93a010eaf0 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a7854700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f93a01a9640 con 0x7f93a010afe0 2024-03-30T15:16:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a4def700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f93a010afe0 0x7f93a01080d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34414/0 (socket says 172.21.15.28:34414) 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f93a4def700 1 -- 172.21.15.28:0/3589485842 learned_addr learned my addr 172.21.15.28:0/3589485842 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter ms_handle_connect 0x7f93a010afe0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter ms_handle_connect 0x7f93a01031d0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter ms_handle_connect 0x7f93a010eaf0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471805013 0 0) 0x7f93a01a9640 con 0x7f93a010afe0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.741+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93880031f0 con 0x7f93a010afe0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028262614 0 0) 0x7f93a010ab70 con 0x7f93a01031d0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a01a9640 con 0x7f93a01031d0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 229409297 0 0) 0x7f93a010ad80 con 0x7f93a010eaf0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a010ab70 con 0x7f93a010eaf0 2024-03-30T15:16:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1578279375 0 0) 0x7f93880031f0 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a010ad80 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9394001930 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 776139406 0 0) 0x7f93a01a9640 con 0x7f93a01031d0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93880031f0 con 0x7f93a01031d0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f93900014b0 con 0x7f93a01031d0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4008021081 0 0) 0x7f93a010ab70 con 0x7f93a010eaf0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93a01a9640 con 0x7f93a010eaf0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f93980043e0 con 0x7f93a010eaf0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 367373710 0 0) 0x7f93a010ad80 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 >> v1:172.21.15.134:6789/0 conn(0x7f93a010eaf0 legacy=0x7f93a01a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 >> v1:172.21.15.28:6789/0 conn(0x7f93a01031d0 legacy=0x7f93a01079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a01aa860 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93a01a9870 con 0x7f93a010afe0 2024-03-30T15:16:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f93a7854700 10 client.59390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f93a01a9d30 con 0x7f93a010afe0 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f93a7854700 1 librados: init done 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9394001d00 con 0x7f93a010afe0 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.742+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9394005df0 con 0x7f93a010afe0 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.743+0000 7f93a7854700 10 client.59390.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.743+0000 7f93a7854700 15 client.59390.objecter _session_command_op_assign -1 1 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.743+0000 7f93a7854700 20 client.59390.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.743+0000 7f93a7854700 10 client.59390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.743+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f938c002920 con 0x7f93a010afe0 2024-03-30T15:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f939401c630 con 0x7f93a010afe0 2024-03-30T15:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(954..954 src has 881..954) ==== 7561+0+0 (unknown 4185558970 0 0) 0x7f9394089600 con 0x7f93a010afe0 2024-03-30T15:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 10 client.59390.objecter ms_dispatch 0x7f93a0001550 osd_map(954..954 src has 881..954) 2024-03-30T15:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 3 client.59390.objecter handle_osd_map got epochs [954,954] > 0 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 3 client.59390.objecter handle_osd_map decoding full epoch 954 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 10 client.59390.objecter checking command 1 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter _get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 15 client.59390.objecter _session_command_op_remove -1 1 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter _get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 15 client.59390.objecter _session_command_op_assign 0 1 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 4 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 10 client.59390.objecter _send_command 1 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6809/2927075629 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9398000fb0 con 0x7f938807cb50 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 20 client.59390.objecter dump_active .. 0 homeless 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.744+0000 7f939e7fc700 10 client.59390.objecter ms_handle_connect 0x7f9388070410 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.745+0000 7f939e7fc700 10 client.59390.objecter ms_handle_connect 0x7f938807cb50 2024-03-30T15:16:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.746+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== osd.0 v1:172.21.15.28:6809/2927075629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9398000fb0 con 0x7f938807cb50 2024-03-30T15:16:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.746+0000 7f939e7fc700 10 client.59390.objecter ms_dispatch 0x7f93a0001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.746+0000 7f939e7fc700 10 client.59390.objecter _finish_command 1 = system:0 2024-03-30T15:16:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.746+0000 7f939e7fc700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 4 2024-03-30T15:16:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.746+0000 7f939e7fc700 15 client.59390.objecter _session_command_op_remove 0 1 2024-03-30T15:16:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.747+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_get_version_reply(handle=1 version=954) ==== 24+0+0 (unknown 2480477438 0 0) 0x7f93940544f0 con 0x7f93a010afe0 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 10 client.59390.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 15 client.59390.objecter _session_command_op_assign -1 2 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 20 client.59390.objecter _get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 20 client.59390.objecter _get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 15 client.59390.objecter _session_command_op_remove -1 2 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 20 client.59390.objecter get_session s=0x7f938807c4a0 osd=0 3 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 15 client.59390.objecter _session_command_op_assign 0 2 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 4 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 10 client.59390.objecter _send_command 2 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 --> v1:172.21.15.28:6809/2927075629 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f938c0011f0 con 0x7f938807cb50 2024-03-30T15:16:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f939e7fc700 1 -- 172.21.15.28:0/3589485842 <== osd.0 v1:172.21.15.28:6809/2927075629 2 ==== command_reply(tid 2: 0 ) ==== 8+0+13 (unknown 0 0 674847311) 0x7f938c0011f0 con 0x7f938807cb50 2024-03-30T15:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f939e7fc700 10 client.59390.objecter ms_dispatch 0x7f93a0001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f939e7fc700 10 client.59390.objecter _finish_command 2 = system:0 2024-03-30T15:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f939e7fc700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 4 2024-03-30T15:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.763+0000 7f939e7fc700 15 client.59390.objecter _session_command_op_remove 0 2 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 10 librados: watch_flush enter 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 10 librados: watch_flush exit 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 10 client.59390.objecter close_session for osd.0 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 >> v1:172.21.15.28:6809/2927075629 conn(0x7f938807cb50 legacy=0x7f938807eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 20 client.59390.objecter put_session s=0x7f938807c4a0 osd=0 1 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 20 client.59390.objecter shutdown clearing up homeless session... 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 10 client.59390.objecter successfully canceled tick 2024-03-30T15:16:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9388070410 legacy=0x7f93880728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 >> v1:172.21.15.28:6790/0 conn(0x7f93a010afe0 legacy=0x7f93a01080d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a5df1700 1 -- 172.21.15.28:0/3589485842 reap_dead start 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 shutdown_connections 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 >> 172.21.15.28:0/3589485842 conn(0x7f93a00fc9e0 msgr2=0x7f93a0110f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 shutdown_connections 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 -- 172.21.15.28:0/3589485842 wait complete. 2024-03-30T15:16:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:49.764+0000 7f93a7854700 1 librados: shutdown 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[41614]: 5.10 scrub starts 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[41614]: 5.10 scrub ok 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[41614]: 3.10 deep-scrub starts 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[41614]: 3.10 deep-scrub ok 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[29876]: 5.10 scrub starts 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[29876]: 5.10 scrub ok 2024-03-30T15:16:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[29876]: 3.10 deep-scrub starts 2024-03-30T15:16:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:49 smithi028 ceph-mon[29876]: 3.10 deep-scrub ok 2024-03-30T15:16:50.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:49 smithi134 ceph-mon[33714]: 5.10 scrub starts 2024-03-30T15:16:50.228 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:49 smithi134 ceph-mon[33714]: 5.10 scrub ok 2024-03-30T15:16:50.228 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:49 smithi134 ceph-mon[33714]: 3.10 deep-scrub starts 2024-03-30T15:16:50.229 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:49 smithi134 ceph-mon[33714]: 3.10 deep-scrub ok 2024-03-30T15:16:50.610 INFO:teuthology.orchestra.run.smithi028.stdout:55834575151 2024-03-30T15:16:50.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T15:16:50.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[29876]: 3.1e scrub starts 2024-03-30T15:16:50.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[29876]: 3.1e scrub ok 2024-03-30T15:16:50.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[29876]: pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:50.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[41614]: 3.1e scrub starts 2024-03-30T15:16:50.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[41614]: 3.1e scrub ok 2024-03-30T15:16:50.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:50 smithi028 ceph-mon[41614]: pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:51.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:50 smithi134 ceph-mon[33714]: 3.1e scrub starts 2024-03-30T15:16:51.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:50 smithi134 ceph-mon[33714]: 3.1e scrub ok 2024-03-30T15:16:51.069 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:50 smithi134 ceph-mon[33714]: pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.204+0000 7f483c3d4700 1 -- 172.21.15.28:0/3891599927 >> v1:172.21.15.28:6790/0 conn(0x7f483410eb50 legacy=0x7f4834110f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.204+0000 7f483c3d4700 1 -- 172.21.15.28:0/3891599927 shutdown_connections 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.204+0000 7f483c3d4700 1 -- 172.21.15.28:0/3891599927 >> 172.21.15.28:0/3891599927 conn(0x7f48340fc9e0 msgr2=0x7f48340fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.204+0000 7f483c3d4700 1 -- 172.21.15.28:0/3891599927 shutdown_connections 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.204+0000 7f483c3d4700 1 -- 172.21.15.28:0/3891599927 wait complete. 2024-03-30T15:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 librados: starting msgr at 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 librados: starting objecter 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 Processor -- start 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 -- start start 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 librados: setting wanted keys 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 librados: calling monclient init 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f48341a7090 con 0x7f483410b040 2024-03-30T15:16:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f48341a82b0 con 0x7f483410eb50 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483a170700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f4834103230 0x7f483410a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34430/0 (socket says 172.21.15.28:34430) 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483996f700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f483410b040 0x7f48341a1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45412/0 (socket says 172.21.15.28:45412) 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483a170700 1 -- 172.21.15.28:0/3067007849 learned_addr learned my addr 172.21.15.28:0/3067007849 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f48341a94f0 con 0x7f4834103230 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter ms_handle_connect 0x7f483410b040 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter ms_handle_connect 0x7f483410eb50 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter ms_handle_connect 0x7f4834103230 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042086118 0 0) 0x7f48341a7090 con 0x7f483410b040 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.205+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48200031f0 con 0x7f483410b040 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311790434 0 0) 0x7f48341a82b0 con 0x7f483410eb50 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48341a7090 con 0x7f483410eb50 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60183954 0 0) 0x7f48341a94f0 con 0x7f4834103230 2024-03-30T15:16:51.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48341a82b0 con 0x7f4834103230 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3111449244 0 0) 0x7f48341a7090 con 0x7f483410eb50 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48341a94f0 con 0x7f483410eb50 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f482c001bc0 con 0x7f483410eb50 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 299921128 0 0) 0x7f48200031f0 con 0x7f483410b040 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48341a7090 con 0x7f483410b040 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3597144280 0 0) 0x7f48341a82b0 con 0x7f4834103230 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48200031f0 con 0x7f4834103230 2024-03-30T15:16:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f4824001a10 con 0x7f483410b040 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f48300017f0 con 0x7f4834103230 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 995982954 0 0) 0x7f48341a94f0 con 0x7f483410eb50 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 >> v1:172.21.15.28:6790/0 conn(0x7f4834103230 legacy=0x7f483410a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 >> v1:172.21.15.28:6789/0 conn(0x7f483410b040 legacy=0x7f48341a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48341aa730 con 0x7f483410eb50 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48341a84e0 con 0x7f483410eb50 2024-03-30T15:16:51.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f483c3d4700 10 client.51382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48341a89a0 con 0x7f483410eb50 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f483c3d4700 1 librados: init done 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f482c004a20 con 0x7f483410eb50 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.206+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f482c004f00 con 0x7f483410eb50 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.207+0000 7f483c3d4700 10 client.51382.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.207+0000 7f483c3d4700 15 client.51382.objecter _session_command_op_assign -1 1 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.207+0000 7f483c3d4700 20 client.51382.objecter _get_session osd=-1 returning homeless 2024-03-30T15:16:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.207+0000 7f483c3d4700 10 client.51382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:51.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.207+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4818002950 con 0x7f483410eb50 2024-03-30T15:16:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.208+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f482c003b00 con 0x7f483410eb50 2024-03-30T15:16:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(954..954 src has 881..954) ==== 7561+0+0 (unknown 4185558970 0 0) 0x7f482c0884b0 con 0x7f483410eb50 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 10 client.51382.objecter ms_dispatch 0x7f4834001550 osd_map(954..954 src has 881..954) 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 3 client.51382.objecter handle_osd_map got epochs [954,954] > 0 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 3 client.51382.objecter handle_osd_map decoding full epoch 954 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 10 client.51382.objecter checking command 1 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter _get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 15 client.51382.objecter _session_command_op_remove -1 1 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter _get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 15 client.51382.objecter _session_command_op_assign 5 1 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 4 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 10 client.51382.objecter _send_command 1 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6804/3668151179 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4820084d80 con 0x7f4820081070 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 20 client.51382.objecter dump_active .. 0 homeless 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_get_version_reply(handle=1 version=954) ==== 24+0+0 (unknown 2480477438 0 0) 0x7f482c088890 con 0x7f483410eb50 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 10 client.51382.objecter ms_handle_connect 0x7f48200748c0 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.209+0000 7f482b7fe700 10 client.51382.objecter ms_handle_connect 0x7f4820081070 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.210+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== osd.5 v1:172.21.15.134:6804/3668151179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4820084d80 con 0x7f4820081070 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.210+0000 7f482b7fe700 10 client.51382.objecter ms_dispatch 0x7f4834001550 command_reply(tid 1: 0 ) 2024-03-30T15:16:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.210+0000 7f482b7fe700 10 client.51382.objecter _finish_command 1 = system:0 2024-03-30T15:16:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.210+0000 7f482b7fe700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 4 2024-03-30T15:16:51.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.210+0000 7f482b7fe700 15 client.51382.objecter _session_command_op_remove 5 1 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 10 client.51382.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 15 client.51382.objecter _session_command_op_assign -1 2 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 20 client.51382.objecter _get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 20 client.51382.objecter _get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 15 client.51382.objecter _session_command_op_remove -1 2 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 20 client.51382.objecter get_session s=0x7f48200809c0 osd=5 3 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 15 client.51382.objecter _session_command_op_assign 5 2 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 4 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 10 client.51382.objecter _send_command 2 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.230+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 --> v1:172.21.15.134:6804/3668151179 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4818001220 con 0x7f4820081070 2024-03-30T15:16:51.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f482b7fe700 1 -- 172.21.15.28:0/3067007849 <== osd.5 v1:172.21.15.134:6804/3668151179 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 2724008840) 0x7f4818001220 con 0x7f4820081070 2024-03-30T15:16:51.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f482b7fe700 10 client.51382.objecter ms_dispatch 0x7f4834001550 command_reply(tid 2: 0 ) 2024-03-30T15:16:51.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f482b7fe700 10 client.51382.objecter _finish_command 2 = system:0 2024-03-30T15:16:51.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f482b7fe700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 4 2024-03-30T15:16:51.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f482b7fe700 15 client.51382.objecter _session_command_op_remove 5 2 2024-03-30T15:16:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f483c3d4700 10 librados: watch_flush enter 2024-03-30T15:16:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f483c3d4700 10 librados: watch_flush exit 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f483c3d4700 10 client.51382.objecter close_session for osd.5 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.231+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 >> v1:172.21.15.134:6804/3668151179 conn(0x7f4820081070 legacy=0x7f4820083510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 20 client.51382.objecter put_session s=0x7f48200809c0 osd=5 1 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 20 client.51382.objecter shutdown clearing up homeless session... 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 10 client.51382.objecter successfully canceled tick 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 >> v1:172.21.15.28:6800/3011431093 conn(0x7f48200748c0 legacy=0x7f4820076d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 >> v1:172.21.15.134:6789/0 conn(0x7f483410eb50 legacy=0x7f48341a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483a971700 1 -- 172.21.15.28:0/3067007849 reap_dead start 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 shutdown_connections 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 >> 172.21.15.28:0/3067007849 conn(0x7f48340fc9e0 msgr2=0x7f48340fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 shutdown_connections 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 -- 172.21.15.28:0/3067007849 wait complete. 2024-03-30T15:16:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:51.232+0000 7f483c3d4700 1 librados: shutdown 2024-03-30T15:16:51.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:16:51.816 INFO:teuthology.orchestra.run.smithi028.stdout:3607772528683 2024-03-30T15:16:51.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T15:16:51.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 4.18 deep-scrub starts 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 4.18 deep-scrub ok 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 1.4 scrub starts 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 1.4 scrub ok 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 12.3 deep-scrub starts 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: 12.3 deep-scrub ok 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 4.18 deep-scrub starts 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 4.18 deep-scrub ok 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 1.4 scrub starts 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 1.4 scrub ok 2024-03-30T15:16:51.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 12.3 deep-scrub starts 2024-03-30T15:16:51.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: 12.3 deep-scrub ok 2024-03-30T15:16:51.843 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:51 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:52.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 4.18 deep-scrub starts 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 4.18 deep-scrub ok 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 1.4 scrub starts 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 1.4 scrub ok 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 12.3 deep-scrub starts 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: 12.3 deep-scrub ok 2024-03-30T15:16:52.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:51 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:16:52.334 INFO:teuthology.orchestra.run.smithi028.stdout:3650722201642 2024-03-30T15:16:52.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T15:16:52.754 INFO:teuthology.orchestra.run.smithi028.stdout:176093659418 2024-03-30T15:16:52.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T15:16:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:52.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[29876]: 6.15 scrub starts 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[29876]: 6.15 scrub ok 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[29876]: 5.15 scrub starts 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[29876]: 5.15 scrub ok 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[29876]: pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:52.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[41614]: 6.15 scrub starts 2024-03-30T15:16:52.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[41614]: 6.15 scrub ok 2024-03-30T15:16:52.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[41614]: 5.15 scrub starts 2024-03-30T15:16:52.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[41614]: 5.15 scrub ok 2024-03-30T15:16:52.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:52 smithi028 ceph-mon[41614]: pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:52 smithi134 ceph-mon[33714]: 6.15 scrub starts 2024-03-30T15:16:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:52 smithi134 ceph-mon[33714]: 6.15 scrub ok 2024-03-30T15:16:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:52 smithi134 ceph-mon[33714]: 5.15 scrub starts 2024-03-30T15:16:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:52 smithi134 ceph-mon[33714]: 5.15 scrub ok 2024-03-30T15:16:53.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:52 smithi134 ceph-mon[33714]: pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:16:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 5.4 scrub starts 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 5.4 scrub ok 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 6.17 scrub starts 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 6.17 scrub ok 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 12.15 scrub starts 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 12.15 scrub ok 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 6.1 scrub starts 2024-03-30T15:16:54.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:53 smithi134 ceph-mon[33714]: 6.1 scrub ok 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 5.4 scrub starts 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 5.4 scrub ok 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 6.17 scrub starts 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 6.17 scrub ok 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 12.15 scrub starts 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 12.15 scrub ok 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 6.1 scrub starts 2024-03-30T15:16:54.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[29876]: 6.1 scrub ok 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 5.4 scrub starts 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 5.4 scrub ok 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 6.17 scrub starts 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 6.17 scrub ok 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 12.15 scrub starts 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 12.15 scrub ok 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 6.1 scrub starts 2024-03-30T15:16:54.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:53 smithi028 ceph-mon[41614]: 6.1 scrub ok 2024-03-30T15:16:54.348 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:54.349 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:54.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 4.d scrub starts 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 4.d scrub ok 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 4.4 deep-scrub starts 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 4.4 deep-scrub ok 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 5.d scrub starts 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: 5.d scrub ok 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[41614]: pgmap v1519: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-30T15:16:54.912 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 4.d scrub starts 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 4.d scrub ok 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 4.4 deep-scrub starts 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 4.4 deep-scrub ok 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 5.d scrub starts 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: 5.d scrub ok 2024-03-30T15:16:54.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:54 smithi028 ceph-mon[29876]: pgmap v1519: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 4.d scrub starts 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 4.d scrub ok 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 4.4 deep-scrub starts 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 4.4 deep-scrub ok 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 5.d scrub starts 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: 5.d scrub ok 2024-03-30T15:16:55.082 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:54 smithi134 ceph-mon[33714]: pgmap v1519: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-30T15:16:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:16:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:55 smithi134 ceph-mon[33714]: osdmap e955: 8 total, 8 up, 8 in 2024-03-30T15:16:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:55 smithi134 ceph-mon[33714]: 4.1d scrub starts 2024-03-30T15:16:56.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:55 smithi134 ceph-mon[33714]: 4.1d scrub ok 2024-03-30T15:16:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:55 smithi134 ceph-mon[33714]: 1.1 scrub starts 2024-03-30T15:16:56.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:55 smithi134 ceph-mon[33714]: 1.1 scrub ok 2024-03-30T15:16:56.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[29876]: osdmap e955: 8 total, 8 up, 8 in 2024-03-30T15:16:56.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[29876]: 4.1d scrub starts 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[29876]: 4.1d scrub ok 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[29876]: 1.1 scrub starts 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[29876]: 1.1 scrub ok 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[41614]: osdmap e955: 8 total, 8 up, 8 in 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[41614]: 4.1d scrub starts 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[41614]: 4.1d scrub ok 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[41614]: 1.1 scrub starts 2024-03-30T15:16:56.222 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:55 smithi028 ceph-mon[41614]: 1.1 scrub ok 2024-03-30T15:16:57.032 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 3.11 deep-scrub starts 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 3.11 deep-scrub ok 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 6.a deep-scrub starts 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 6.a deep-scrub ok 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 5.1d scrub starts 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: 5.1d scrub ok 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[41614]: pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 3.11 deep-scrub starts 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 3.11 deep-scrub ok 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 6.a deep-scrub starts 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 6.a deep-scrub ok 2024-03-30T15:16:57.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 5.1d scrub starts 2024-03-30T15:16:57.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: 5.1d scrub ok 2024-03-30T15:16:57.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:56 smithi028 ceph-mon[29876]: pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-30T15:16:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 3.11 deep-scrub starts 2024-03-30T15:16:57.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 3.11 deep-scrub ok 2024-03-30T15:16:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 6.a deep-scrub starts 2024-03-30T15:16:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 6.a deep-scrub ok 2024-03-30T15:16:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 5.1d scrub starts 2024-03-30T15:16:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: 5.1d scrub ok 2024-03-30T15:16:57.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:56 smithi134 ceph-mon[33714]: pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.754+0000 7f36011f6700 1 -- 172.21.15.28:0/2434682736 >> v1:172.21.15.28:6790/0 conn(0x7f35fc10aee0 legacy=0x7f35fc10d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.754+0000 7f36011f6700 1 -- 172.21.15.28:0/2434682736 shutdown_connections 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.754+0000 7f36011f6700 1 -- 172.21.15.28:0/2434682736 >> 172.21.15.28:0/2434682736 conn(0x7f35fc0fdb20 msgr2=0x7f35fc0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.754+0000 7f36011f6700 1 -- 172.21.15.28:0/2434682736 shutdown_connections 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.754+0000 7f36011f6700 1 -- 172.21.15.28:0/2434682736 wait complete. 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 librados: starting msgr at 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 librados: starting objecter 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 Processor -- start 2024-03-30T15:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 -- start start 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 librados: setting wanted keys 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 librados: calling monclient init 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.755+0000 7f36011f6700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35fc1a6fe0 con 0x7f35fc10ea80 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35fc1a8200 con 0x7f35fc102250 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f35fc1a9440 con 0x7f35fc10aee0 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35fb59e700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35fc10ea80 0x7f35fc1a56a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45424/0 (socket says 172.21.15.28:45424) 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35fb59e700 1 -- 172.21.15.28:0/3531178428 learned_addr learned my addr 172.21.15.28:0/3531178428 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35fa59c700 1 --1- 172.21.15.28:0/3531178428 >> v1:172.21.15.28:6790/0 conn(0x7f35fc10aee0 0x7f35fc1a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter ms_handle_connect 0x7f35fc10ea80 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter ms_handle_connect 0x7f35fc10aee0 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter ms_handle_connect 0x7f35fc102250 2024-03-30T15:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4237591630 0 0) 0x7f35fc1a6fe0 con 0x7f35fc10ea80 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e4003aa0 con 0x7f35fc10ea80 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904088484 0 0) 0x7f35fc1a9440 con 0x7f35fc10aee0 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35fc1a6fe0 con 0x7f35fc10aee0 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155192936 0 0) 0x7f35fc1a8200 con 0x7f35fc102250 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35fc1a9440 con 0x7f35fc102250 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120114899 0 0) 0x7f35e4003aa0 con 0x7f35fc10ea80 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35fc1a8200 con 0x7f35fc10ea80 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35f4001cc0 con 0x7f35fc10ea80 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1592070720 0 0) 0x7f35fc1a6fe0 con 0x7f35fc10aee0 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e4003aa0 con 0x7f35fc10aee0 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35f0004610 con 0x7f35fc10aee0 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1200943606 0 0) 0x7f35fc1a9440 con 0x7f35fc102250 2024-03-30T15:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35fc1a6fe0 con 0x7f35fc102250 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35ec001a30 con 0x7f35fc102250 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2964012109 0 0) 0x7f35fc1a8200 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 >> v1:172.21.15.28:6790/0 conn(0x7f35fc10aee0 legacy=0x7f35fc1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 >> v1:172.21.15.134:6789/0 conn(0x7f35fc102250 legacy=0x7f35fc107480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc1aa680 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc1a9670 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 10 client.26199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f35fc1a9b30 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.756+0000 7f36011f6700 1 librados: init done 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.757+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f35f4003ec0 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.757+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f35f40060e0 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.757+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e8005120 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.758+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f35f401c780 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.758+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(955..955 src has 904..955) ==== 7561+0+0 (unknown 3485977719 0 0) 0x7f35f4089790 con 0x7f35fc10ea80 2024-03-30T15:16:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.758+0000 7f35e3fff700 10 client.26199.objecter ms_dispatch 0x7f35fc001550 osd_map(955..955 src has 904..955) 2024-03-30T15:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.758+0000 7f35e3fff700 3 client.26199.objecter handle_osd_map got epochs [955,955] > 0 2024-03-30T15:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.758+0000 7f35e3fff700 3 client.26199.objecter handle_osd_map decoding full epoch 955 2024-03-30T15:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.759+0000 7f35e3fff700 20 client.26199.objecter dump_active .. 0 homeless 2024-03-30T15:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.759+0000 7f35e3fff700 10 client.26199.objecter ms_handle_connect 0x7f35e4070520 2024-03-30T15:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.763+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f35f4056c40 con 0x7f35fc10ea80 2024-03-30T15:16:57.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 12.1b scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 12.1b scrub ok 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 6.16 scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 6.16 scrub ok 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 3.7 scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 3.7 scrub ok 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 5.11 scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 5.11 scrub ok 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 1.a scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 1.a scrub ok 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 4.12 scrub starts 2024-03-30T15:16:57.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[29876]: 4.12 scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 12.1b scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 12.1b scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 6.16 scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 6.16 scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 3.7 scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 3.7 scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 5.11 scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 5.11 scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 1.a scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 1.a scrub ok 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 4.12 scrub starts 2024-03-30T15:16:57.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:57 smithi028 ceph-mon[41614]: 4.12 scrub ok 2024-03-30T15:16:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.972+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f35e8005410 con 0x7f35fc10ea80 2024-03-30T15:16:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.972+0000 7f35e3fff700 1 -- 172.21.15.28:0/3531178428 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+14 (unknown 1823589948 0 2426921137) 0x7f35f405a690 con 0x7f35fc10ea80 2024-03-30T15:16:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:3573412790320 2024-03-30T15:16:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.975+0000 7f36011f6700 10 librados: watch_flush enter 2024-03-30T15:16:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 10 librados: watch_flush exit 2024-03-30T15:16:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 20 client.26199.objecter shutdown clearing up homeless session... 2024-03-30T15:16:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 10 client.26199.objecter successfully canceled tick 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 >> v1:172.21.15.28:6800/3011431093 conn(0x7f35e4070520 legacy=0x7f35e40729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 >> v1:172.21.15.28:6789/0 conn(0x7f35fc10ea80 legacy=0x7f35fc1a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 shutdown_connections 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 >> 172.21.15.28:0/3531178428 conn(0x7f35fc0fdb20 msgr2=0x7f35fc10ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 shutdown_connections 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 -- 172.21.15.28:0/3531178428 wait complete. 2024-03-30T15:16:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:16:57.976+0000 7f36011f6700 1 librados: shutdown 2024-03-30T15:16:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 12.1b scrub starts 2024-03-30T15:16:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 12.1b scrub ok 2024-03-30T15:16:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 6.16 scrub starts 2024-03-30T15:16:58.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 6.16 scrub ok 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 3.7 scrub starts 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 3.7 scrub ok 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 5.11 scrub starts 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 5.11 scrub ok 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 1.a scrub starts 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 1.a scrub ok 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 4.12 scrub starts 2024-03-30T15:16:58.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:57 smithi134 ceph-mon[33714]: 4.12 scrub ok 2024-03-30T15:16:58.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:16:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:16:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:58 smithi134 ceph-mon[33714]: 5.2 deep-scrub starts 2024-03-30T15:16:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:58 smithi134 ceph-mon[33714]: 5.2 deep-scrub ok 2024-03-30T15:16:59.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:58 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3531178428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T15:16:59.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:58 smithi134 ceph-mon[33714]: pgmap v1522: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[29876]: 5.2 deep-scrub starts 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[29876]: 5.2 deep-scrub ok 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3531178428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[29876]: pgmap v1522: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[41614]: 5.2 deep-scrub starts 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[41614]: 5.2 deep-scrub ok 2024-03-30T15:16:59.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3531178428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T15:16:59.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:58 smithi028 ceph-mon[41614]: pgmap v1522: 174 pgs: 174 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 6.12 scrub starts 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 6.12 scrub ok 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 3.1 scrub starts 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 3.1 scrub ok 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 6.d scrub starts 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 6.d scrub ok 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 5.b scrub starts 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[29876]: 5.b scrub ok 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 6.12 scrub starts 2024-03-30T15:17:00.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 6.12 scrub ok 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 3.1 scrub starts 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 3.1 scrub ok 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 6.d scrub starts 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 6.d scrub ok 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 5.b scrub starts 2024-03-30T15:17:00.149 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:16:59 smithi028 ceph-mon[41614]: 5.b scrub ok 2024-03-30T15:17:00.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 6.12 scrub starts 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 6.12 scrub ok 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 3.1 scrub starts 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 3.1 scrub ok 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 6.d scrub starts 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 6.d scrub ok 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 5.b scrub starts 2024-03-30T15:17:00.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:16:59 smithi134 ceph-mon[33714]: 5.b scrub ok 2024-03-30T15:17:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[41614]: 2.0 scrub starts 2024-03-30T15:17:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[41614]: 2.0 scrub ok 2024-03-30T15:17:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[41614]: 12.13 scrub starts 2024-03-30T15:17:00.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[41614]: 12.13 scrub ok 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[41614]: pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[29876]: 2.0 scrub starts 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[29876]: 2.0 scrub ok 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[29876]: 12.13 scrub starts 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[29876]: 12.13 scrub ok 2024-03-30T15:17:00.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:00 smithi028 ceph-mon[29876]: pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:01.080 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:00 smithi134 ceph-mon[33714]: 2.0 scrub starts 2024-03-30T15:17:01.080 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:00 smithi134 ceph-mon[33714]: 2.0 scrub ok 2024-03-30T15:17:01.080 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:00 smithi134 ceph-mon[33714]: 12.13 scrub starts 2024-03-30T15:17:01.080 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:00 smithi134 ceph-mon[33714]: 12.13 scrub ok 2024-03-30T15:17:01.080 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:00 smithi134 ceph-mon[33714]: pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:01.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.751+0000 7f03eb017700 1 -- 172.21.15.28:0/3994433847 >> v1:172.21.15.28:6790/0 conn(0x7f03e4109e10 legacy=0x7f03e410c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.751+0000 7f03eb017700 1 -- 172.21.15.28:0/3994433847 shutdown_connections 2024-03-30T15:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.751+0000 7f03eb017700 1 -- 172.21.15.28:0/3994433847 >> 172.21.15.28:0/3994433847 conn(0x7f03e40fca30 msgr2=0x7f03e40fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.751+0000 7f03eb017700 1 -- 172.21.15.28:0/3994433847 shutdown_connections 2024-03-30T15:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.751+0000 7f03eb017700 1 -- 172.21.15.28:0/3994433847 wait complete. 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 librados: starting msgr at 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 librados: starting objecter 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 Processor -- start 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 -- start start 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 librados: setting wanted keys 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 librados: calling monclient init 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03e410fa50 con 0x7f03e4101180 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03e410fc60 con 0x7f03e410d9b0 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03eb017700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03e41ab990 con 0x7f03e4109e10 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e3fff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f03e4109e10 0x7f03e410e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41902/0 (socket says 172.21.15.28:41902) 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e3fff700 1 -- 172.21.15.28:0/359870221 learned_addr learned my addr 172.21.15.28:0/359870221 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter ms_handle_connect 0x7f03e4101180 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter ms_handle_connect 0x7f03e4109e10 2024-03-30T15:17:01.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter ms_handle_connect 0x7f03e410d9b0 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.752+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167626725 0 0) 0x7f03e410fa50 con 0x7f03e4101180 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03d40031f0 con 0x7f03e4101180 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361985262 0 0) 0x7f03e41ab990 con 0x7f03e4109e10 2024-03-30T15:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e410fa50 con 0x7f03e4109e10 2024-03-30T15:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427587845 0 0) 0x7f03e410fc60 con 0x7f03e410d9b0 2024-03-30T15:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e41ab990 con 0x7f03e410d9b0 2024-03-30T15:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1355013894 0 0) 0x7f03d40031f0 con 0x7f03e4101180 2024-03-30T15:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03e410fc60 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03d80013d0 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 68100220 0 0) 0x7f03e410fa50 con 0x7f03e4109e10 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03d40031f0 con 0x7f03e4109e10 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03d0004610 con 0x7f03e4109e10 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 389278249 0 0) 0x7f03e41ab990 con 0x7f03e410d9b0 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03e410fa50 con 0x7f03e410d9b0 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03dc001af0 con 0x7f03e410d9b0 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 927324508 0 0) 0x7f03e410fc60 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 >> v1:172.21.15.28:6790/0 conn(0x7f03e4109e10 legacy=0x7f03e410e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 >> v1:172.21.15.134:6789/0 conn(0x7f03e410d9b0 legacy=0x7f03e41a8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e41acb70 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03e41abb60 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f03d8001680 con 0x7f03e4101180 2024-03-30T15:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f03d8004ac0 con 0x7f03e4101180 2024-03-30T15:17:01.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03eb017700 10 client.26205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:01.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f03e41ac020 con 0x7f03e4101180 2024-03-30T15:17:01.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.753+0000 7f03eb017700 1 librados: init done 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.754+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4102660 con 0x7f03e4101180 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.755+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f03d8003930 con 0x7f03e4101180 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.755+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(955..955 src has 904..955) ==== 7561+0+0 (unknown 3485977719 0 0) 0x7f03d80881c0 con 0x7f03e4101180 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.755+0000 7f03e1ffb700 10 client.26205.objecter ms_dispatch 0x7f03e4001550 osd_map(955..955 src has 904..955) 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.755+0000 7f03e1ffb700 3 client.26205.objecter handle_osd_map got epochs [955,955] > 0 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.755+0000 7f03e1ffb700 3 client.26205.objecter handle_osd_map decoding full epoch 955 2024-03-30T15:17:01.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.756+0000 7f03e1ffb700 20 client.26205.objecter dump_active .. 0 homeless 2024-03-30T15:17:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.756+0000 7f03e1ffb700 10 client.26205.objecter ms_handle_connect 0x7f03d40704b0 2024-03-30T15:17:01.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.760+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f03d8055670 con 0x7f03e4101180 2024-03-30T15:17:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.965+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f03e410f440 con 0x7f03e4101180 2024-03-30T15:17:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.965+0000 7f03e1ffb700 1 -- 172.21.15.28:0/359870221 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+14 (unknown 92182286 0 2476634961) 0x7f03d80590c0 con 0x7f03e4101180 2024-03-30T15:17:01.967 INFO:teuthology.orchestra.run.smithi028.stdout:3607772528686 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 10 librados: watch_flush enter 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 10 librados: watch_flush exit 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 20 client.26205.objecter shutdown clearing up homeless session... 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 10 client.26205.objecter successfully canceled tick 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 >> v1:172.21.15.28:6800/3011431093 conn(0x7f03d40704b0 legacy=0x7f03d4072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.967+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 >> v1:172.21.15.28:6789/0 conn(0x7f03e4101180 legacy=0x7f03e410df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.968+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 shutdown_connections 2024-03-30T15:17:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.968+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 >> 172.21.15.28:0/359870221 conn(0x7f03e40fca30 msgr2=0x7f03e406a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.968+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 shutdown_connections 2024-03-30T15:17:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.968+0000 7f03eb017700 1 -- 172.21.15.28:0/359870221 wait complete. 2024-03-30T15:17:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:01.968+0000 7f03eb017700 1 librados: shutdown 2024-03-30T15:17:02.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 5.18 scrub starts 2024-03-30T15:17:02.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 5.18 scrub ok 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 2.1 deep-scrub starts 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 2.1 deep-scrub ok 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 5.7 deep-scrub starts 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: 5.7 deep-scrub ok 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 5.18 scrub starts 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 5.18 scrub ok 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 2.1 deep-scrub starts 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 2.1 deep-scrub ok 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 5.7 deep-scrub starts 2024-03-30T15:17:02.163 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: 5.7 deep-scrub ok 2024-03-30T15:17:02.164 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:01 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:02.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 5.18 scrub starts 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 5.18 scrub ok 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 2.1 deep-scrub starts 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 2.1 deep-scrub ok 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 5.7 deep-scrub starts 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: 5.7 deep-scrub ok 2024-03-30T15:17:02.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:01 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: 3.15 scrub starts 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: 3.15 scrub ok 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: 3.1a scrub starts 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: 3.1a scrub ok 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/359870221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: pgmap v1524: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:17:02.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: 3.15 scrub starts 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: 3.15 scrub ok 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: 3.1a scrub starts 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: 3.1a scrub ok 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/359870221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: pgmap v1524: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:17:02.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:02 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:17:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: 3.15 scrub starts 2024-03-30T15:17:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: 3.15 scrub ok 2024-03-30T15:17:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: 3.1a scrub starts 2024-03-30T15:17:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: 3.1a scrub ok 2024-03-30T15:17:03.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/359870221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T15:17:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: pgmap v1524: 174 pgs: 174 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:17:03.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:02 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:17:04.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: osdmap e956: 8 total, 8 up, 8 in 2024-03-30T15:17:04.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 12.1a scrub starts 2024-03-30T15:17:04.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 12.1a scrub ok 2024-03-30T15:17:04.014 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 12.e scrub starts 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 12.e scrub ok 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 6.f scrub starts 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[41614]: 6.f scrub ok 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: osdmap e956: 8 total, 8 up, 8 in 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 12.1a scrub starts 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 12.1a scrub ok 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 12.e scrub starts 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 12.e scrub ok 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 6.f scrub starts 2024-03-30T15:17:04.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:03 smithi028 ceph-mon[29876]: 6.f scrub ok 2024-03-30T15:17:04.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: osdmap e956: 8 total, 8 up, 8 in 2024-03-30T15:17:04.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 12.1a scrub starts 2024-03-30T15:17:04.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 12.1a scrub ok 2024-03-30T15:17:04.183 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 12.e scrub starts 2024-03-30T15:17:04.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 12.e scrub ok 2024-03-30T15:17:04.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 6.f scrub starts 2024-03-30T15:17:04.184 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:03 smithi134 ceph-mon[33714]: 6.f scrub ok 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 -- 172.21.15.28:0/3037687547 >> v1:172.21.15.28:6790/0 conn(0x7fec6410ead0 legacy=0x7fec64110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 -- 172.21.15.28:0/3037687547 shutdown_connections 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 -- 172.21.15.28:0/3037687547 >> 172.21.15.28:0/3037687547 conn(0x7fec640fdb50 msgr2=0x7fec640fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 -- 172.21.15.28:0/3037687547 shutdown_connections 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 -- 172.21.15.28:0/3037687547 wait complete. 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 librados: starting msgr at 2024-03-30T15:17:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.299+0000 7fec68d0c700 1 librados: starting objecter 2024-03-30T15:17:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 Processor -- start 2024-03-30T15:17:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 -- start start 2024-03-30T15:17:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 librados: setting wanted keys 2024-03-30T15:17:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 librados: calling monclient init 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec64076450 con 0x7fec6410af30 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec641aaa90 con 0x7fec6410ead0 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec68d0c700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fec641abcb0 con 0x7fec641022a0 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec6259c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fec641022a0 0x7fec64074540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41908/0 (socket says 172.21.15.28:41908) 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec6259c700 1 -- 172.21.15.28:0/788376892 learned_addr learned my addr 172.21.15.28:0/788376892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter ms_handle_connect 0x7fec6410af30 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter ms_handle_connect 0x7fec641022a0 2024-03-30T15:17:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter ms_handle_connect 0x7fec6410ead0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879357406 0 0) 0x7fec64076450 con 0x7fec6410af30 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec4c0035b0 con 0x7fec6410af30 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952258961 0 0) 0x7fec641abcb0 con 0x7fec641022a0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec64076450 con 0x7fec641022a0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.300+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382245289 0 0) 0x7fec641aaa90 con 0x7fec6410ead0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec641abcb0 con 0x7fec6410ead0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1966837144 0 0) 0x7fec4c0035b0 con 0x7fec6410af30 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec641aaa90 con 0x7fec6410af30 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec580018d0 con 0x7fec6410af30 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1484261226 0 0) 0x7fec64076450 con 0x7fec641022a0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec4c0035b0 con 0x7fec641022a0 2024-03-30T15:17:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec54001930 con 0x7fec641022a0 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4050814275 0 0) 0x7fec641abcb0 con 0x7fec6410ead0 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec64076450 con 0x7fec6410ead0 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec5c0017b0 con 0x7fec6410ead0 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3756748907 0 0) 0x7fec641aaa90 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 >> v1:172.21.15.28:6790/0 conn(0x7fec641022a0 legacy=0x7fec64074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 >> v1:172.21.15.134:6789/0 conn(0x7fec6410ead0 legacy=0x7fec641a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec641acef0 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec641aacc0 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec68d0c700 10 client.26211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec641ab180 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec68d0c700 1 librados: init done 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fec58003b40 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.301+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fec58004a30 con 0x7fec6410af30 2024-03-30T15:17:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.302+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec44005120 con 0x7fec6410af30 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fec580036a0 con 0x7fec6410af30 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(956..956 src has 904..956) ==== 7561+0+0 (unknown 3799479668 0 0) 0x7fec58088130 con 0x7fec6410af30 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 10 client.26211.objecter ms_dispatch 0x7fec64001550 osd_map(956..956 src has 904..956) 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 3 client.26211.objecter handle_osd_map got epochs [956,956] > 0 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 3 client.26211.objecter handle_osd_map decoding full epoch 956 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 20 client.26211.objecter dump_active .. 0 homeless 2024-03-30T15:17:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.303+0000 7fec537fe700 10 client.26211.objecter ms_handle_connect 0x7fec4c0704b0 2024-03-30T15:17:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.308+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fec580555c0 con 0x7fec6410af30 2024-03-30T15:17:04.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.513+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fec44005410 con 0x7fec6410af30 2024-03-30T15:17:04.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.513+0000 7fec537fe700 1 -- 172.21.15.28:0/788376892 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3438605469) 0x7fec58059010 con 0x7fec6410af30 2024-03-30T15:17:04.515 INFO:teuthology.orchestra.run.smithi028.stdout:55834575154 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 10 librados: watch_flush enter 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 10 librados: watch_flush exit 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 20 client.26211.objecter shutdown clearing up homeless session... 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 10 client.26211.objecter successfully canceled tick 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 >> v1:172.21.15.28:6800/3011431093 conn(0x7fec4c0704b0 legacy=0x7fec4c072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 >> v1:172.21.15.28:6789/0 conn(0x7fec6410af30 legacy=0x7fec64074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 shutdown_connections 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 >> 172.21.15.28:0/788376892 conn(0x7fec640fdb50 msgr2=0x7fec6410a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 shutdown_connections 2024-03-30T15:17:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 -- 172.21.15.28:0/788376892 wait complete. 2024-03-30T15:17:04.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:04.515+0000 7fec68d0c700 1 librados: shutdown 2024-03-30T15:17:05.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[41614]: 6.8 deep-scrub starts 2024-03-30T15:17:05.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[41614]: 6.8 deep-scrub ok 2024-03-30T15:17:05.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[41614]: pgmap v1526: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:17:05.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/788376892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T15:17:05.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[29876]: 6.8 deep-scrub starts 2024-03-30T15:17:05.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[29876]: 6.8 deep-scrub ok 2024-03-30T15:17:05.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[29876]: pgmap v1526: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:17:05.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:04 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/788376892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T15:17:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:04 smithi134 ceph-mon[33714]: 6.8 deep-scrub starts 2024-03-30T15:17:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:04 smithi134 ceph-mon[33714]: 6.8 deep-scrub ok 2024-03-30T15:17:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:04 smithi134 ceph-mon[33714]: pgmap v1526: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-30T15:17:05.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:04 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/788376892' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T15:17:05.203 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575151 got 55834575154 for osd.1 2024-03-30T15:17:05.203 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:05.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3607772528683 got 3607772528686 for osd.2 2024-03-30T15:17:05.657 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.880+0000 7fbf33aef700 1 -- 172.21.15.28:0/1856272506 >> v1:172.21.15.28:6790/0 conn(0x7fbf2c109dc0 legacy=0x7fbf2c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.880+0000 7fbf33aef700 1 -- 172.21.15.28:0/1856272506 shutdown_connections 2024-03-30T15:17:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.880+0000 7fbf33aef700 1 -- 172.21.15.28:0/1856272506 >> 172.21.15.28:0/1856272506 conn(0x7fbf2c0fc9e0 msgr2=0x7fbf2c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:05.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:05 smithi028 ceph-mon[29876]: 12.2 scrub starts 2024-03-30T15:17:05.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:05 smithi028 ceph-mon[29876]: 12.2 scrub ok 2024-03-30T15:17:05.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:05 smithi028 ceph-mon[41614]: 12.2 scrub starts 2024-03-30T15:17:05.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:05 smithi028 ceph-mon[41614]: 12.2 scrub ok 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.880+0000 7fbf33aef700 1 -- 172.21.15.28:0/1856272506 shutdown_connections 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 -- 172.21.15.28:0/1856272506 wait complete. 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 librados: starting msgr at 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 librados: starting objecter 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 Processor -- start 2024-03-30T15:17:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 -- start start 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 librados: setting wanted keys 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 librados: calling monclient init 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf2c10fc70 con 0x7fbf2c109dc0 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf2c1aa930 con 0x7fbf2c101130 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.881+0000 7fbf33aef700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf2c1abb10 con 0x7fbf2c10d960 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf3208c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fbf2c10d960 0x7fbf2c1a8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41926/0 (socket says 172.21.15.28:41926) 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf3208c700 1 -- 172.21.15.28:0/3053417947 learned_addr learned my addr 172.21.15.28:0/3053417947 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:05.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf3108a700 1 --1- 172.21.15.28:0/3053417947 >> v1:172.21.15.28:6789/0 conn(0x7fbf2c109dc0 0x7fbf2c10e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49098/0 (socket says 172.21.15.28:49098) 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter ms_handle_connect 0x7fbf2c109dc0 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter ms_handle_connect 0x7fbf2c10d960 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter ms_handle_connect 0x7fbf2c101130 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193773819 0 0) 0x7fbf2c1abb10 con 0x7fbf2c10d960 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf180031f0 con 0x7fbf2c10d960 2024-03-30T15:17:05.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347991953 0 0) 0x7fbf2c10fc70 con 0x7fbf2c109dc0 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf2c1abb10 con 0x7fbf2c109dc0 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856748715 0 0) 0x7fbf2c1aa930 con 0x7fbf2c101130 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf2c10fc70 con 0x7fbf2c101130 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 666331028 0 0) 0x7fbf180031f0 con 0x7fbf2c10d960 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf2c1aa930 con 0x7fbf2c10d960 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbf24001bf0 con 0x7fbf2c10d960 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2090150662 0 0) 0x7fbf2c1abb10 con 0x7fbf2c109dc0 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf180031f0 con 0x7fbf2c109dc0 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbf1c001430 con 0x7fbf2c109dc0 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2145334902 0 0) 0x7fbf2c10fc70 con 0x7fbf2c101130 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf2c1abb10 con 0x7fbf2c101130 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbf280018d0 con 0x7fbf2c101130 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 556359935 0 0) 0x7fbf2c1aa930 con 0x7fbf2c10d960 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 >> v1:172.21.15.134:6789/0 conn(0x7fbf2c101130 legacy=0x7fbf2c10de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 >> v1:172.21.15.28:6789/0 conn(0x7fbf2c109dc0 legacy=0x7fbf2c10e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.882+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2c1accf0 con 0x7fbf2c10d960 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2c1abd40 con 0x7fbf2c10d960 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf33aef700 10 client.59420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf2c1ac200 con 0x7fbf2c10d960 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf33aef700 1 librados: init done 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fbf24003ec0 con 0x7fbf2c10d960 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.883+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fbf24005fa0 con 0x7fbf2c10d960 2024-03-30T15:17:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.884+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf10005120 con 0x7fbf2c10d960 2024-03-30T15:17:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.884+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fbf24003a20 con 0x7fbf2c10d960 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(956..956 src has 904..956) ==== 7561+0+0 (unknown 3799479668 0 0) 0x7fbf240896c0 con 0x7fbf2c10d960 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 10 client.59420.objecter ms_dispatch 0x7fbf2c001550 osd_map(956..956 src has 904..956) 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 3 client.59420.objecter handle_osd_map got epochs [956,956] > 0 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 3 client.59420.objecter handle_osd_map decoding full epoch 956 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 20 client.59420.objecter dump_active .. 0 homeless 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.885+0000 7fbf22ffd700 10 client.59420.objecter ms_handle_connect 0x7fbf18070390 2024-03-30T15:17:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:05.889+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbf24056b10 con 0x7fbf2c10d960 2024-03-30T15:17:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.095+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbf10005410 con 0x7fbf2c10d960 2024-03-30T15:17:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.095+0000 7fbf22ffd700 1 -- 172.21.15.28:0/3053417947 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 3551951586) 0x7fbf2405a560 con 0x7fbf2c10d960 2024-03-30T15:17:06.097 INFO:teuthology.orchestra.run.smithi028.stdout:115964117288 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 10 librados: watch_flush enter 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 10 librados: watch_flush exit 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 20 client.59420.objecter shutdown clearing up homeless session... 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 10 client.59420.objecter successfully canceled tick 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 >> v1:172.21.15.28:6800/3011431093 conn(0x7fbf18070390 legacy=0x7fbf18072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 >> v1:172.21.15.28:6790/0 conn(0x7fbf2c10d960 legacy=0x7fbf2c1a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 shutdown_connections 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 >> 172.21.15.28:0/3053417947 conn(0x7fbf2c0fc9e0 msgr2=0x7fbf2c110840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 shutdown_connections 2024-03-30T15:17:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 -- 172.21.15.28:0/3053417947 wait complete. 2024-03-30T15:17:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.097+0000 7fbf33aef700 1 librados: shutdown 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.103+0000 7f9a665c9700 1 -- 172.21.15.28:0/455697572 >> v1:172.21.15.28:6789/0 conn(0x7f9a60109500 legacy=0x7f9a6010b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.103+0000 7f9a665c9700 1 -- 172.21.15.28:0/455697572 shutdown_connections 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.103+0000 7f9a665c9700 1 -- 172.21.15.28:0/455697572 >> 172.21.15.28:0/455697572 conn(0x7f9a60077370 msgr2=0x7f9a600fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.103+0000 7f9a665c9700 1 -- 172.21.15.28:0/455697572 shutdown_connections 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.103+0000 7f9a665c9700 1 -- 172.21.15.28:0/455697572 wait complete. 2024-03-30T15:17:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 librados: starting msgr at 2024-03-30T15:17:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 librados: starting objecter 2024-03-30T15:17:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 Processor -- start 2024-03-30T15:17:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 -- start start 2024-03-30T15:17:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 librados: setting wanted keys 2024-03-30T15:17:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 librados: calling monclient init 2024-03-30T15:17:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a601a9390 con 0x7f9a6010d0d0 2024-03-30T15:17:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a601aa570 con 0x7f9a60109500 2024-03-30T15:17:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a665c9700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9a601ab770 con 0x7f9a60106650 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a5ffff700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f9a60106650 0x7f9a6010b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41942/0 (socket says 172.21.15.28:41942) 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.104+0000 7f9a5ffff700 1 -- 172.21.15.28:0/388527942 learned_addr learned my addr 172.21.15.28:0/388527942 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter ms_handle_connect 0x7f9a6010d0d0 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter ms_handle_connect 0x7f9a60106650 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter ms_handle_connect 0x7f9a60109500 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 617185627 0 0) 0x7f9a601a9390 con 0x7f9a6010d0d0 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a4c004370 con 0x7f9a6010d0d0 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436560347 0 0) 0x7f9a601ab770 con 0x7f9a60106650 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a601a9390 con 0x7f9a60106650 2024-03-30T15:17:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111043578 0 0) 0x7f9a601aa570 con 0x7f9a60109500 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a601ab770 con 0x7f9a60109500 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437445271 0 0) 0x7f9a4c004370 con 0x7f9a6010d0d0 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a601aa570 con 0x7f9a6010d0d0 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1586569412 0 0) 0x7f9a601a9390 con 0x7f9a60106650 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a4c004370 con 0x7f9a60106650 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a58001b70 con 0x7f9a6010d0d0 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a50001890 con 0x7f9a60106650 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4031774766 0 0) 0x7f9a601ab770 con 0x7f9a60109500 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a601a9390 con 0x7f9a60109500 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a480045e0 con 0x7f9a60109500 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 892388302 0 0) 0x7f9a601aa570 con 0x7f9a6010d0d0 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 >> v1:172.21.15.28:6790/0 conn(0x7f9a60106650 legacy=0x7f9a6010b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 >> v1:172.21.15.134:6789/0 conn(0x7f9a60109500 legacy=0x7f9a6010ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a601ac9b0 con 0x7f9a6010d0d0 2024-03-30T15:17:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.105+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a601ab9a0 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.106+0000 7f9a665c9700 10 client.26223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.106+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a601abe60 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.106+0000 7f9a665c9700 1 librados: init done 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.106+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f9a58003ea0 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.106+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f9a58004de0 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.107+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a40005120 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.107+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f9a5801b460 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(956..956 src has 904..956) ==== 7561+0+0 (unknown 3799479668 0 0) 0x7f9a58088460 con 0x7f9a6010d0d0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 10 client.26223.objecter ms_dispatch 0x7f9a60001550 osd_map(956..956 src has 904..956) 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 3 client.26223.objecter handle_osd_map got epochs [956,956] > 0 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 3 client.26223.objecter handle_osd_map decoding full epoch 956 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 20 client.26223.objecter dump_active .. 0 homeless 2024-03-30T15:17:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.108+0000 7f9a5d7fa700 10 client.26223.objecter ms_handle_connect 0x7f9a4c0704e0 2024-03-30T15:17:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.112+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9a580559f0 con 0x7f9a6010d0d0 2024-03-30T15:17:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:05 smithi134 ceph-mon[33714]: 12.2 scrub starts 2024-03-30T15:17:06.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:05 smithi134 ceph-mon[33714]: 12.2 scrub ok 2024-03-30T15:17:06.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.320+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9a40005410 con 0x7f9a6010d0d0 2024-03-30T15:17:06.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.320+0000 7f9a5d7fa700 1 -- 172.21.15.28:0/388527942 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 2975695610) 0x7f9a58059440 con 0x7f9a6010d0d0 2024-03-30T15:17:06.322 INFO:teuthology.orchestra.run.smithi028.stdout:261993005331 2024-03-30T15:17:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 10 librados: watch_flush enter 2024-03-30T15:17:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 10 librados: watch_flush exit 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 20 client.26223.objecter shutdown clearing up homeless session... 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 10 client.26223.objecter successfully canceled tick 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 >> v1:172.21.15.28:6800/3011431093 conn(0x7f9a4c0704e0 legacy=0x7f9a4c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 >> v1:172.21.15.28:6789/0 conn(0x7f9a6010d0d0 legacy=0x7f9a601a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 shutdown_connections 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 >> 172.21.15.28:0/388527942 conn(0x7f9a60077370 msgr2=0x7f9a6010fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 shutdown_connections 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 -- 172.21.15.28:0/388527942 wait complete. 2024-03-30T15:17:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:06.322+0000 7f9a665c9700 1 librados: shutdown 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3053417947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[29876]: 6.1d scrub starts 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[29876]: 6.1d scrub ok 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[29876]: pgmap v1527: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/388527942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T15:17:06.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3053417947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T15:17:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[41614]: 6.1d scrub starts 2024-03-30T15:17:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[41614]: 6.1d scrub ok 2024-03-30T15:17:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[41614]: pgmap v1527: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:06.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:06 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/388527942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T15:17:07.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117284 got 115964117288 for osd.3 2024-03-30T15:17:07.151 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3053417947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T15:17:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:06 smithi134 ceph-mon[33714]: 6.1d scrub starts 2024-03-30T15:17:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:06 smithi134 ceph-mon[33714]: 6.1d scrub ok 2024-03-30T15:17:07.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:06 smithi134 ceph-mon[33714]: pgmap v1527: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:07.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:06 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/388527942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T15:17:07.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3573412790317 got 3573412790320 for osd.4 2024-03-30T15:17:07.603 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[29876]: 6.5 scrub starts 2024-03-30T15:17:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[29876]: 6.5 scrub ok 2024-03-30T15:17:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[29876]: 3.4 scrub starts 2024-03-30T15:17:07.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[29876]: 3.4 scrub ok 2024-03-30T15:17:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[41614]: 6.5 scrub starts 2024-03-30T15:17:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[41614]: 6.5 scrub ok 2024-03-30T15:17:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[41614]: 3.4 scrub starts 2024-03-30T15:17:07.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:07 smithi028 ceph-mon[41614]: 3.4 scrub ok 2024-03-30T15:17:08.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005327 got 261993005331 for osd.7 2024-03-30T15:17:08.046 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:08.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:07 smithi134 ceph-mon[33714]: 6.5 scrub starts 2024-03-30T15:17:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:07 smithi134 ceph-mon[33714]: 6.5 scrub ok 2024-03-30T15:17:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:07 smithi134 ceph-mon[33714]: 3.4 scrub starts 2024-03-30T15:17:08.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:07 smithi134 ceph-mon[33714]: 3.4 scrub ok 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.234+0000 7fe773088700 1 -- 172.21.15.28:0/603770250 >> v1:172.21.15.28:6790/0 conn(0x7fe76c109db0 legacy=0x7fe76c10c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.234+0000 7fe773088700 1 -- 172.21.15.28:0/603770250 shutdown_connections 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.234+0000 7fe773088700 1 -- 172.21.15.28:0/603770250 >> 172.21.15.28:0/603770250 conn(0x7fe76c0fc9f0 msgr2=0x7fe76c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.234+0000 7fe773088700 1 -- 172.21.15.28:0/603770250 shutdown_connections 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.234+0000 7fe773088700 1 -- 172.21.15.28:0/603770250 wait complete. 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 librados: starting msgr at 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 librados: starting objecter 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 Processor -- start 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 -- start start 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 librados: setting wanted keys 2024-03-30T15:17:08.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 librados: calling monclient init 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe76c0fea40 con 0x7fe76c109db0 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe76c0fec50 con 0x7fe76c10d950 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe773088700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe76c1ab930 con 0x7fe76c101120 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe770e24700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7fe76c101120 0x7fe76c0fcea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:41964/0 (socket says 172.21.15.28:41964) 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe770e24700 1 -- 172.21.15.28:0/2134144569 learned_addr learned my addr 172.21.15.28:0/2134144569 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter ms_handle_connect 0x7fe76c109db0 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter ms_handle_connect 0x7fe76c101120 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter ms_handle_connect 0x7fe76c10d950 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853003160 0 0) 0x7fe76c0fea40 con 0x7fe76c109db0 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.235+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe75c0031f0 con 0x7fe76c109db0 2024-03-30T15:17:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341303903 0 0) 0x7fe76c1ab930 con 0x7fe76c101120 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe76c0fea40 con 0x7fe76c101120 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631776016 0 0) 0x7fe76c0fec50 con 0x7fe76c10d950 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe76c1ab930 con 0x7fe76c10d950 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1677501901 0 0) 0x7fe75c0031f0 con 0x7fe76c109db0 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe76c0fec50 con 0x7fe76c109db0 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe758004640 con 0x7fe76c109db0 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1864450079 0 0) 0x7fe76c0fea40 con 0x7fe76c101120 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe75c0031f0 con 0x7fe76c101120 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe760001870 con 0x7fe76c101120 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3842006468 0 0) 0x7fe76c1ab930 con 0x7fe76c10d950 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe76c0fea40 con 0x7fe76c10d950 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe764001db0 con 0x7fe76c10d950 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3540913647 0 0) 0x7fe76c0fec50 con 0x7fe76c109db0 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 >> v1:172.21.15.28:6790/0 conn(0x7fe76c101120 legacy=0x7fe76c0fcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 >> v1:172.21.15.134:6789/0 conn(0x7fe76c10d950 legacy=0x7fe76c1a81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76c1acb10 con 0x7fe76c109db0 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe76c1abb00 con 0x7fe76c109db0 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7fe758003ba0 con 0x7fe76c109db0 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe773088700 10 client.26229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7fe758004ed0 con 0x7fe76c109db0 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe76c1abfc0 con 0x7fe76c109db0 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.236+0000 7fe773088700 1 librados: init done 2024-03-30T15:17:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.237+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76c102600 con 0x7fe76c109db0 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.238+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7fe758003e50 con 0x7fe76c109db0 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.238+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(956..956 src has 904..956) ==== 7561+0+0 (unknown 3799479668 0 0) 0x7fe758088380 con 0x7fe76c109db0 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.238+0000 7fe769ffb700 10 client.26229.objecter ms_dispatch 0x7fe76c001550 osd_map(956..956 src has 904..956) 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.238+0000 7fe769ffb700 3 client.26229.objecter handle_osd_map got epochs [956,956] > 0 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.238+0000 7fe769ffb700 3 client.26229.objecter handle_osd_map decoding full epoch 956 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.239+0000 7fe769ffb700 20 client.26229.objecter dump_active .. 0 homeless 2024-03-30T15:17:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.239+0000 7fe769ffb700 10 client.26229.objecter ms_handle_connect 0x7fe75c070550 2024-03-30T15:17:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.244+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe7580557d0 con 0x7fe76c109db0 2024-03-30T15:17:08.372 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:17:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:17:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.451+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe76c0fe3d0 con 0x7fe76c109db0 2024-03-30T15:17:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.451+0000 7fe769ffb700 1 -- 172.21.15.28:0/2134144569 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+14 (unknown 574334944 0 248510911) 0x7fe758059220 con 0x7fe76c109db0 2024-03-30T15:17:08.453 INFO:teuthology.orchestra.run.smithi028.stdout:3650722201645 2024-03-30T15:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 10 librados: watch_flush enter 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 10 librados: watch_flush exit 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 20 client.26229.objecter shutdown clearing up homeless session... 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 10 client.26229.objecter successfully canceled tick 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 >> v1:172.21.15.28:6800/3011431093 conn(0x7fe75c070550 legacy=0x7fe75c072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 >> v1:172.21.15.28:6789/0 conn(0x7fe76c109db0 legacy=0x7fe76c0fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 shutdown_connections 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 >> 172.21.15.28:0/2134144569 conn(0x7fe76c0fc9f0 msgr2=0x7fe76c110910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 shutdown_connections 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 -- 172.21.15.28:0/2134144569 wait complete. 2024-03-30T15:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:08.454+0000 7fe773088700 1 librados: shutdown 2024-03-30T15:17:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:08 smithi134 ceph-mon[33714]: 3.9 scrub starts 2024-03-30T15:17:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:08 smithi134 ceph-mon[33714]: 3.9 scrub ok 2024-03-30T15:17:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:08 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:17:09.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:08 smithi134 ceph-mon[33714]: pgmap v1528: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:09.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:08 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/2134144569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T15:17:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[29876]: 3.9 scrub starts 2024-03-30T15:17:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[29876]: 3.9 scrub ok 2024-03-30T15:17:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:17:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[29876]: pgmap v1528: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:09.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/2134144569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T15:17:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[41614]: 3.9 scrub starts 2024-03-30T15:17:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[41614]: 3.9 scrub ok 2024-03-30T15:17:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-30T15:17:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[41614]: pgmap v1528: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:09.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:08 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/2134144569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T15:17:09.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3650722201642 got 3650722201645 for osd.0 2024-03-30T15:17:09.261 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1380873378 >> v1:172.21.15.28:6790/0 conn(0x7f6c6810ea80 legacy=0x7f6c68110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1380873378 shutdown_connections 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1380873378 >> 172.21.15.28:0/1380873378 conn(0x7f6c680fdb20 msgr2=0x7f6c680fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1380873378 shutdown_connections 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1380873378 wait complete. 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 librados: starting msgr at 2024-03-30T15:17:09.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 librados: starting objecter 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 Processor -- start 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 -- start start 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 librados: setting wanted keys 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.694+0000 7f6c6ec60700 1 librados: calling monclient init 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c6ec60700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681a9130 con 0x7f6c6810ea80 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c6ec60700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681aa350 con 0x7f6c68102250 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c6ec60700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6c681ab590 con 0x7f6c6810aee0 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c6d1fd700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6c6810ea80 0x7f6c681a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33094/0 (socket says 172.21.15.28:33094) 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c6d1fd700 1 -- 172.21.15.28:0/1323852085 learned_addr learned my addr 172.21.15.28:0/1323852085 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter ms_handle_connect 0x7f6c6810aee0 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter ms_handle_connect 0x7f6c6810ea80 2024-03-30T15:17:09.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter ms_handle_connect 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 394239136 0 0) 0x7f6c681aa350 con 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c58003aa0 con 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904521718 0 0) 0x7f6c681ab590 con 0x7f6c6810aee0 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c681aa350 con 0x7f6c6810aee0 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1629004256 0 0) 0x7f6c681a9130 con 0x7f6c6810ea80 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c681ab590 con 0x7f6c6810ea80 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 463638348 0 0) 0x7f6c58003aa0 con 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c681a9130 con 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c5c001960 con 0x7f6c68102250 2024-03-30T15:17:09.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 170200182 0 0) 0x7f6c681aa350 con 0x7f6c6810aee0 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c58003aa0 con 0x7f6c6810aee0 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.695+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4140564657 0 0) 0x7f6c681ab590 con 0x7f6c6810ea80 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c681aa350 con 0x7f6c6810ea80 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c540017b0 con 0x7f6c6810aee0 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c60004750 con 0x7f6c6810ea80 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3320214191 0 0) 0x7f6c681a9130 con 0x7f6c68102250 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 >> v1:172.21.15.28:6790/0 conn(0x7f6c6810aee0 legacy=0x7f6c681a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 >> v1:172.21.15.28:6789/0 conn(0x7f6c6810ea80 legacy=0x7f6c681a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c681ac7d0 con 0x7f6c68102250 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c681a9360 con 0x7f6c68102250 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6c5c003b20 con 0x7f6c68102250 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6c5c004b00 con 0x7f6c68102250 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c6ec60700 10 client.51421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c681a9820 con 0x7f6c68102250 2024-03-30T15:17:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.696+0000 7f6c6ec60700 1 librados: init done 2024-03-30T15:17:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.697+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c68103730 con 0x7f6c68102250 2024-03-30T15:17:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.697+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6c5c003680 con 0x7f6c68102250 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(956..956 src has 904..956) ==== 7561+0+0 (unknown 3799479668 0 0) 0x7f6c5c088250 con 0x7f6c68102250 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 10 client.51421.objecter ms_dispatch 0x7f6c68001550 osd_map(956..956 src has 904..956) 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 3 client.51421.objecter handle_osd_map got epochs [956,956] > 0 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 3 client.51421.objecter handle_osd_map decoding full epoch 956 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 20 client.51421.objecter dump_active .. 0 homeless 2024-03-30T15:17:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.698+0000 7f6c65ffb700 10 client.51421.objecter ms_handle_connect 0x7f6c58070630 2024-03-30T15:17:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.703+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6c5c0556e0 con 0x7f6c68102250 2024-03-30T15:17:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.919+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6c68068b30 con 0x7f6c68102250 2024-03-30T15:17:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.920+0000 7f6c65ffb700 1 -- 172.21.15.28:0/1323852085 <== mon.1 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 133664032) 0x7f6c5c059130 con 0x7f6c68102250 2024-03-30T15:17:09.921 INFO:teuthology.orchestra.run.smithi028.stdout:176093659421 2024-03-30T15:17:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 10 librados: watch_flush enter 2024-03-30T15:17:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 10 librados: watch_flush exit 2024-03-30T15:17:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 20 client.51421.objecter shutdown clearing up homeless session... 2024-03-30T15:17:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 10 client.51421.objecter successfully canceled tick 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6c58070630 legacy=0x7f6c58072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 >> v1:172.21.15.134:6789/0 conn(0x7f6c68102250 legacy=0x7f6c68107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 shutdown_connections 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 >> 172.21.15.28:0/1323852085 conn(0x7f6c680fdb20 msgr2=0x7f6c680fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.922+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 shutdown_connections 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.923+0000 7f6c6ec60700 1 -- 172.21.15.28:0/1323852085 wait complete. 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.923+0000 7f6c6ec60700 1 librados: shutdown 2024-03-30T15:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 -- 172.21.15.28:0/1981853641 >> v1:172.21.15.28:6790/0 conn(0x7f700810aec0 legacy=0x7f700810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 -- 172.21.15.28:0/1981853641 shutdown_connections 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 -- 172.21.15.28:0/1981853641 >> 172.21.15.28:0/1981853641 conn(0x7f70080fdb00 msgr2=0x7f70080fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 -- 172.21.15.28:0/1981853641 shutdown_connections 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 -- 172.21.15.28:0/1981853641 wait complete. 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 librados: starting msgr at 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.908+0000 7f700d9d7700 1 librados: starting objecter 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 Processor -- start 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 -- start start 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 librados: setting wanted keys 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 librados: calling monclient init 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70081a9310 con 0x7f700810ea60 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70081aa530 con 0x7f700810aec0 2024-03-30T15:17:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700d9d7700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f70081ab770 con 0x7f7008102230 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f70077fe700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f700810ea60 0x7f70081a79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33110/0 (socket says 172.21.15.28:33110) 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f70077fe700 1 -- 172.21.15.28:0/3310235076 learned_addr learned my addr 172.21.15.28:0/3310235076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter ms_handle_connect 0x7f700810ea60 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter ms_handle_connect 0x7f7008102230 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter ms_handle_connect 0x7f700810aec0 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651296414 0 0) 0x7f70081ab770 con 0x7f7008102230 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff0003aa0 con 0x7f7008102230 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614001269 0 0) 0x7f70081a9310 con 0x7f700810ea60 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.909+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70081ab770 con 0x7f700810ea60 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044288302 0 0) 0x7f70081aa530 con 0x7f700810aec0 2024-03-30T15:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70081a9310 con 0x7f700810aec0 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 264219119 0 0) 0x7f6ff0003aa0 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70081aa530 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6ff80013d0 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3614773797 0 0) 0x7f70081ab770 con 0x7f700810ea60 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff0003aa0 con 0x7f700810ea60 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7000001bb0 con 0x7f700810ea60 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4142321943 0 0) 0x7f70081a9310 con 0x7f700810aec0 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70081ab770 con 0x7f700810aec0 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6ffc003820 con 0x7f700810aec0 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 298335700 0 0) 0x7f70081aa530 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 >> v1:172.21.15.134:6789/0 conn(0x7f700810aec0 legacy=0x7f70081a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 >> v1:172.21.15.28:6789/0 conn(0x7f700810ea60 legacy=0x7f70081a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70081ac9b0 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70081ab9a0 con 0x7f7008102230 2024-03-30T15:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700d9d7700 10 client.59444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70081abe60 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700d9d7700 1 librados: init done 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f6ff8001060 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.910+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f6ff8004a00 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.911+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff4005120 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.912+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f6ff8003930 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.912+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(957..957 src has 904..957) ==== 7561+0+0 (unknown 3981114649 0 0) 0x7f6ff8088150 con 0x7f7008102230 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.912+0000 7f700c9d5700 10 client.59444.objecter ms_dispatch 0x7f7008001550 osd_map(957..957 src has 904..957) 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.912+0000 7f700c9d5700 3 client.59444.objecter handle_osd_map got epochs [957,957] > 0 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.912+0000 7f700c9d5700 3 client.59444.objecter handle_osd_map decoding full epoch 957 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.913+0000 7f700c9d5700 20 client.59444.objecter dump_active .. 0 homeless 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.913+0000 7f700c9d5700 10 client.59444.objecter ms_handle_connect 0x7f6ff0070530 2024-03-30T15:17:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:09.917+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6ff80555a0 con 0x7f7008102230 2024-03-30T15:17:10.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.124+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6ff4005410 con 0x7f7008102230 2024-03-30T15:17:10.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[41614]: 4.17 scrub starts 2024-03-30T15:17:10.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[41614]: 4.17 scrub ok 2024-03-30T15:17:10.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[41614]: 12.5 scrub starts 2024-03-30T15:17:10.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[41614]: 12.5 scrub ok 2024-03-30T15:17:10.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[29876]: 4.17 scrub starts 2024-03-30T15:17:10.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[29876]: 4.17 scrub ok 2024-03-30T15:17:10.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[29876]: 12.5 scrub starts 2024-03-30T15:17:10.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:09 smithi028 ceph-mon[29876]: 12.5 scrub ok 2024-03-30T15:17:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.125+0000 7f700c9d5700 1 -- 172.21.15.28:0/3310235076 <== mon.2 v1:172.21.15.28:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 819888838) 0x7f6ff8058ff0 con 0x7f7008102230 2024-03-30T15:17:10.129 INFO:teuthology.orchestra.run.smithi028.stdout:223338299672 2024-03-30T15:17:10.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 10 librados: watch_flush enter 2024-03-30T15:17:10.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 10 librados: watch_flush exit 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 20 client.59444.objecter shutdown clearing up homeless session... 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 10 client.59444.objecter successfully canceled tick 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 >> v1:172.21.15.28:6800/3011431093 conn(0x7f6ff0070530 legacy=0x7f6ff00729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 >> v1:172.21.15.28:6790/0 conn(0x7f7008102230 legacy=0x7f70081074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 shutdown_connections 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 >> 172.21.15.28:0/3310235076 conn(0x7f70080fdb00 msgr2=0x7f7008111b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 shutdown_connections 2024-03-30T15:17:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 -- 172.21.15.28:0/3310235076 wait complete. 2024-03-30T15:17:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:10.127+0000 7f700d9d7700 1 librados: shutdown 2024-03-30T15:17:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:09 smithi134 ceph-mon[33714]: 4.17 scrub starts 2024-03-30T15:17:10.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:09 smithi134 ceph-mon[33714]: 4.17 scrub ok 2024-03-30T15:17:10.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:09 smithi134 ceph-mon[33714]: 12.5 scrub starts 2024-03-30T15:17:10.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:09 smithi134 ceph-mon[33714]: 12.5 scrub ok 2024-03-30T15:17:10.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 6.13 scrub starts 2024-03-30T15:17:10.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 6.13 scrub ok 2024-03-30T15:17:10.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 12.d scrub starts 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 12.d scrub ok 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 12.0 scrub starts 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 12.0 scrub ok 2024-03-30T15:17:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: osdmap e957: 8 total, 8 up, 8 in 2024-03-30T15:17:10.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/1323852085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T15:17:10.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 5.f deep-scrub starts 2024-03-30T15:17:10.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 5.f deep-scrub ok 2024-03-30T15:17:10.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: purged_snaps scrub starts 2024-03-30T15:17:10.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: purged_snaps scrub ok 2024-03-30T15:17:10.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3310235076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T15:17:10.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: pgmap v1530: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:10.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 6.4 scrub starts 2024-03-30T15:17:10.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[29876]: 6.4 scrub ok 2024-03-30T15:17:10.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 6.13 scrub starts 2024-03-30T15:17:10.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 6.13 scrub ok 2024-03-30T15:17:10.956 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 12.d scrub starts 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 12.d scrub ok 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 12.0 scrub starts 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 12.0 scrub ok 2024-03-30T15:17:10.957 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: osdmap e957: 8 total, 8 up, 8 in 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/1323852085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 5.f deep-scrub starts 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 5.f deep-scrub ok 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: purged_snaps scrub starts 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: purged_snaps scrub ok 2024-03-30T15:17:10.958 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3310235076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T15:17:10.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: pgmap v1530: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:10.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 6.4 scrub starts 2024-03-30T15:17:10.959 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:10 smithi028 ceph-mon[41614]: 6.4 scrub ok 2024-03-30T15:17:10.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659418 got 176093659421 for osd.5 2024-03-30T15:17:10.980 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 6.13 scrub starts 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 6.13 scrub ok 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 12.d scrub starts 2024-03-30T15:17:11.084 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 12.d scrub ok 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 12.0 scrub starts 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 12.0 scrub ok 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: osdmap e957: 8 total, 8 up, 8 in 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/1323852085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 5.f deep-scrub starts 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 5.f deep-scrub ok 2024-03-30T15:17:11.085 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: purged_snaps scrub starts 2024-03-30T15:17:11.086 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: purged_snaps scrub ok 2024-03-30T15:17:11.086 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3310235076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T15:17:11.086 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: pgmap v1530: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:11.086 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 6.4 scrub starts 2024-03-30T15:17:11.086 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:10 smithi134 ceph-mon[33714]: 6.4 scrub ok 2024-03-30T15:17:11.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:17:11.560 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299668 got 223338299672 for osd.6 2024-03-30T15:17:11.561 DEBUG:teuthology.parallel:result is None 2024-03-30T15:17:11.561 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T15:17:11.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:17:11.812 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 12.b deep-scrub starts 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 12.b deep-scrub ok 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: osdmap e958: 8 total, 8 up, 8 in 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 5.17 scrub starts 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 5.17 scrub ok 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 12.18 scrub starts 2024-03-30T15:17:12.075 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[41614]: 12.18 scrub ok 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 12.b deep-scrub starts 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 12.b deep-scrub ok 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: osdmap e958: 8 total, 8 up, 8 in 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 5.17 scrub starts 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 5.17 scrub ok 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 12.18 scrub starts 2024-03-30T15:17:12.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:11 smithi028 ceph-mon[29876]: 12.18 scrub ok 2024-03-30T15:17:12.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 12.b deep-scrub starts 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 12.b deep-scrub ok 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: osdmap e958: 8 total, 8 up, 8 in 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 5.17 scrub starts 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 5.17 scrub ok 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: from='client.24463 v1:172.21.15.134:0/1952441076' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 12.18 scrub starts 2024-03-30T15:17:12.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:11 smithi134 ceph-mon[33714]: 12.18 scrub ok 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 5.13 scrub starts 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 5.13 scrub ok 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 12.1d scrub starts 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 12.1d scrub ok 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 4.5 scrub starts 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 4.5 scrub ok 2024-03-30T15:17:12.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: pgmap v1532: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 4.2 scrub starts 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[29876]: 4.2 scrub ok 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 5.13 scrub starts 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 5.13 scrub ok 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 12.1d scrub starts 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 12.1d scrub ok 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 4.5 scrub starts 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 4.5 scrub ok 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: pgmap v1532: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:17:12.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 4.2 scrub starts 2024-03-30T15:17:12.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:12 smithi028 ceph-mon[41614]: 4.2 scrub ok 2024-03-30T15:17:13.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 5.13 scrub starts 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 5.13 scrub ok 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 12.1d scrub starts 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 12.1d scrub ok 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 4.5 scrub starts 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 4.5 scrub ok 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: pgmap v1532: 174 pgs: 174 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 4.2 scrub starts 2024-03-30T15:17:13.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:12 smithi134 ceph-mon[33714]: 4.2 scrub ok 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 -- 172.21.15.28:0/1389288838 >> v1:172.21.15.28:6789/0 conn(0x7f7d90102250 legacy=0x7f7d90102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 -- 172.21.15.28:0/1389288838 shutdown_connections 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 -- 172.21.15.28:0/1389288838 >> 172.21.15.28:0/1389288838 conn(0x7f7d900fdae0 msgr2=0x7f7d900fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 -- 172.21.15.28:0/1389288838 shutdown_connections 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 -- 172.21.15.28:0/1389288838 wait complete. 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 librados: starting msgr at 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.623+0000 7f7d95171700 1 librados: starting objecter 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 Processor -- start 2024-03-30T15:17:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 -- start start 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 librados: setting wanted keys 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 librados: calling monclient init 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d901a93c0 con 0x7f7d9010ea80 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d901aa600 con 0x7f7d90102250 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d95171700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7d901ab840 con 0x7f7d9010aee0 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d8e59c700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f7d9010aee0 0x7f7d901a42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48520/0 (socket says 172.21.15.28:48520) 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d8e59c700 1 -- 172.21.15.28:0/2702619824 learned_addr learned my addr 172.21.15.28:0/2702619824 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter ms_handle_connect 0x7f7d9010ea80 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter ms_handle_connect 0x7f7d9010aee0 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter ms_handle_connect 0x7f7d90102250 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920200159 0 0) 0x7f7d901a93c0 con 0x7f7d9010ea80 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d7c0031f0 con 0x7f7d9010ea80 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453135737 0 0) 0x7f7d901ab840 con 0x7f7d9010aee0 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d901a93c0 con 0x7f7d9010aee0 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502574764 0 0) 0x7f7d901aa600 con 0x7f7d90102250 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.624+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d901ab840 con 0x7f7d90102250 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1540563753 0 0) 0x7f7d901a93c0 con 0x7f7d9010aee0 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d901aa600 con 0x7f7d9010aee0 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7d78001830 con 0x7f7d9010aee0 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 713443362 0 0) 0x7f7d7c0031f0 con 0x7f7d9010ea80 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d901a93c0 con 0x7f7d9010ea80 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1570949140 0 0) 0x7f7d901ab840 con 0x7f7d90102250 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d7c0031f0 con 0x7f7d90102250 2024-03-30T15:17:13.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7d80001590 con 0x7f7d9010ea80 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7d88001840 con 0x7f7d90102250 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4170207041 0 0) 0x7f7d901aa600 con 0x7f7d9010aee0 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 >> v1:172.21.15.134:6789/0 conn(0x7f7d90102250 legacy=0x7f7d901073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 >> v1:172.21.15.28:6789/0 conn(0x7f7d9010ea80 legacy=0x7f7d901a7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d901aca80 con 0x7f7d9010aee0 2024-03-30T15:17:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d901aba70 con 0x7f7d9010aee0 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d95171700 10 client.59450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d901abf30 con 0x7f7d9010aee0 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d95171700 1 librados: init done 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f7d78001c40 con 0x7f7d9010aee0 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.625+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f7d78005d30 con 0x7f7d9010aee0 2024-03-30T15:17:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.626+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d90103730 con 0x7f7d9010aee0 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.627+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f7d780040c0 con 0x7f7d9010aee0 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.627+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 8 ==== osd_map(958..958 src has 904..958) ==== 7561+0+0 (unknown 4228670894 0 0) 0x7f7d78089470 con 0x7f7d9010aee0 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.627+0000 7f7d87fff700 10 client.59450.objecter ms_dispatch 0x7f7d90001550 osd_map(958..958 src has 904..958) 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.627+0000 7f7d87fff700 3 client.59450.objecter handle_osd_map got epochs [958,958] > 0 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.627+0000 7f7d87fff700 3 client.59450.objecter handle_osd_map decoding full epoch 958 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.628+0000 7f7d87fff700 20 client.59450.objecter dump_active .. 0 homeless 2024-03-30T15:17:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.628+0000 7f7d87fff700 10 client.59450.objecter ms_handle_connect 0x7f7d7c0703e0 2024-03-30T15:17:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.632+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mon.2 v1:172.21.15.28:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7d78005d10 con 0x7f7d9010aee0 2024-03-30T15:17:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.836+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d9010e850 con 0x7f7d7c0703e0 2024-03-30T15:17:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.849+0000 7f7d87fff700 1 -- 172.21.15.28:0/2702619824 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453270 (unknown 2965378022 0 3662861614) 0x7f7d9010e850 con 0x7f7d7c0703e0 2024-03-30T15:17:13.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.853+0000 7f7d95171700 10 librados: watch_flush enter 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.853+0000 7f7d95171700 10 librados: watch_flush exit 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.853+0000 7f7d95171700 20 client.59450.objecter shutdown clearing up homeless session... 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.853+0000 7f7d95171700 10 client.59450.objecter successfully canceled tick 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 >> v1:172.21.15.28:6800/3011431093 conn(0x7f7d7c0703e0 legacy=0x7f7d7c0728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 >> v1:172.21.15.28:6790/0 conn(0x7f7d9010aee0 legacy=0x7f7d901a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 shutdown_connections 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 >> 172.21.15.28:0/2702619824 conn(0x7f7d900fdae0 msgr2=0x7f7d90111c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 shutdown_connections 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 -- 172.21.15.28:0/2702619824 wait complete. 2024-03-30T15:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:13.854+0000 7f7d95171700 1 librados: shutdown 2024-03-30T15:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:17:14.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 12.12 scrub starts 2024-03-30T15:17:14.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 12.12 scrub ok 2024-03-30T15:17:14.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 4.1f deep-scrub starts 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 4.1f deep-scrub ok 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 3.19 scrub starts 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[41614]: 3.19 scrub ok 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 12.12 scrub starts 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 12.12 scrub ok 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 4.1f deep-scrub starts 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 4.1f deep-scrub ok 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 3.19 scrub starts 2024-03-30T15:17:14.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:13 smithi028 ceph-mon[29876]: 3.19 scrub ok 2024-03-30T15:17:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 12.12 scrub starts 2024-03-30T15:17:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 12.12 scrub ok 2024-03-30T15:17:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 4.1f deep-scrub starts 2024-03-30T15:17:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 4.1f deep-scrub ok 2024-03-30T15:17:14.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 3.19 scrub starts 2024-03-30T15:17:14.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:13 smithi134 ceph-mon[33714]: 3.19 scrub ok 2024-03-30T15:17:14.647 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-03-30T15:17:12.256581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2902,"num_read_kb":2654,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":2333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2621,"ondisk_log_size":2621,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":633460,"kb_used_data":18296,"kb_used_omap":108,"kb_used_meta":614611,"kb_avail":749327756,"statfs":{"total":767960285184,"available":767311622144,"internally_reserved":0,"allocated":18735104,"data_stored":6274085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111128,"internal_metadata":629362152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001850"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":2148,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472233+0000","last_change":"2024-03-30T15:16:26.417679+0000","last_active":"2024-03-30T15:17:03.472233+0000","last_peered":"2024-03-30T15:17:03.472233+0000","last_clean":"2024-03-30T15:17:03.472233+0000","last_became_active":"2024-03-30T15:13:55.601989+0000","last_became_peered":"2024-03-30T15:13:55.601989+0000","last_unstale":"2024-03-30T15:17:03.472233+0000","last_undegraded":"2024-03-30T15:17:03.472233+0000","last_fullsized":"2024-03-30T15:17:03.472233+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:26.417567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.417567+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:43.746799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1764,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793150+0000","last_change":"2024-03-30T15:16:26.895669+0000","last_active":"2024-03-30T15:17:02.793150+0000","last_peered":"2024-03-30T15:17:02.793150+0000","last_clean":"2024-03-30T15:17:02.793150+0000","last_became_active":"2024-03-30T15:14:00.633295+0000","last_became_peered":"2024-03-30T15:14:00.633295+0000","last_unstale":"2024-03-30T15:17:02.793150+0000","last_undegraded":"2024-03-30T15:17:02.793150+0000","last_fullsized":"2024-03-30T15:17:02.793150+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:26.895521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.895521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.969586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"909'30","reported_seq":1798,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832309+0000","last_change":"2024-03-30T15:15:48.546754+0000","last_active":"2024-03-30T15:17:10.832309+0000","last_peered":"2024-03-30T15:17:10.832309+0000","last_clean":"2024-03-30T15:17:10.832309+0000","last_became_active":"2024-03-30T15:13:03.535865+0000","last_became_peered":"2024-03-30T15:13:03.535865+0000","last_unstale":"2024-03-30T15:17:10.832309+0000","last_undegraded":"2024-03-30T15:17:10.832309+0000","last_fullsized":"2024-03-30T15:17:10.832309+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'30","last_scrub_stamp":"2024-03-30T15:15:48.546626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:48.546626+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:15.199581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1204,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941517+0000","last_change":"2024-03-30T15:15:57.450490+0000","last_active":"2024-03-30T15:17:02.941517+0000","last_peered":"2024-03-30T15:17:02.941517+0000","last_clean":"2024-03-30T15:17:02.941517+0000","last_became_active":"2024-03-30T15:15:33.641307+0000","last_became_peered":"2024-03-30T15:15:33.641307+0000","last_unstale":"2024-03-30T15:17:02.941517+0000","last_undegraded":"2024-03-30T15:17:02.941517+0000","last_fullsized":"2024-03-30T15:17:02.941517+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:57.450409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.450409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.067800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"926'12","reported_seq":1142,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793208+0000","last_change":"2024-03-30T15:16:53.980500+0000","last_active":"2024-03-30T15:17:02.793208+0000","last_peered":"2024-03-30T15:17:02.793208+0000","last_clean":"2024-03-30T15:17:02.793208+0000","last_became_active":"2024-03-30T15:13:03.543934+0000","last_became_peered":"2024-03-30T15:13:03.543934+0000","last_unstale":"2024-03-30T15:17:02.793208+0000","last_undegraded":"2024-03-30T15:17:02.793208+0000","last_fullsized":"2024-03-30T15:17:02.793208+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'12","last_scrub_stamp":"2024-03-30T15:16:53.980402+0000","last_deep_scrub":"926'12","last_deep_scrub_stamp":"2024-03-30T15:15:26.532092+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.980402+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.653007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1803,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833198+0000","last_change":"2024-03-30T15:16:42.781431+0000","last_active":"2024-03-30T15:17:10.833198+0000","last_peered":"2024-03-30T15:17:10.833198+0000","last_clean":"2024-03-30T15:17:10.833198+0000","last_became_active":"2024-03-30T15:13:52.596681+0000","last_became_peered":"2024-03-30T15:13:52.596681+0000","last_unstale":"2024-03-30T15:17:10.833198+0000","last_undegraded":"2024-03-30T15:17:10.833198+0000","last_fullsized":"2024-03-30T15:17:10.833198+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.781280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:06.712626+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.781280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:12.621398+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":1128,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177137+0000","last_change":"2024-03-30T15:15:57.921848+0000","last_active":"2024-03-30T15:17:03.177137+0000","last_peered":"2024-03-30T15:17:03.177137+0000","last_clean":"2024-03-30T15:17:03.177137+0000","last_became_active":"2024-03-30T15:13:19.287238+0000","last_became_peered":"2024-03-30T15:13:19.287238+0000","last_unstale":"2024-03-30T15:17:03.177137+0000","last_undegraded":"2024-03-30T15:17:03.177137+0000","last_fullsized":"2024-03-30T15:17:03.177137+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:15:57.921757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.921757+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:08.958057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1753,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793278+0000","last_change":"2024-03-30T15:15:37.666540+0000","last_active":"2024-03-30T15:17:02.793278+0000","last_peered":"2024-03-30T15:17:02.793278+0000","last_clean":"2024-03-30T15:17:02.793278+0000","last_became_active":"2024-03-30T15:13:19.284720+0000","last_became_peered":"2024-03-30T15:13:19.284720+0000","last_unstale":"2024-03-30T15:17:02.793278+0000","last_undegraded":"2024-03-30T15:17:02.793278+0000","last_fullsized":"2024-03-30T15:17:02.793278+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:37.666436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:37.666436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"917'46","reported_seq":220,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832382+0000","last_change":"2024-03-30T15:16:53.797229+0000","last_active":"2024-03-30T15:17:10.832382+0000","last_peered":"2024-03-30T15:17:10.832382+0000","last_clean":"2024-03-30T15:17:10.832382+0000","last_became_active":"2024-03-30T15:14:09.722049+0000","last_became_peered":"2024-03-30T15:14:09.722049+0000","last_unstale":"2024-03-30T15:17:10.832382+0000","last_undegraded":"2024-03-30T15:17:10.832382+0000","last_fullsized":"2024-03-30T15:17:10.832382+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'46","last_scrub_stamp":"2024-03-30T15:16:53.797109+0000","last_deep_scrub":"917'46","last_deep_scrub_stamp":"2024-03-30T15:15:10.695652+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.797109+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.519746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1863,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792697+0000","last_change":"2024-03-30T15:16:36.918150+0000","last_active":"2024-03-30T15:17:02.792697+0000","last_peered":"2024-03-30T15:17:02.792697+0000","last_clean":"2024-03-30T15:17:02.792697+0000","last_became_active":"2024-03-30T15:13:03.529657+0000","last_became_peered":"2024-03-30T15:13:03.529657+0000","last_unstale":"2024-03-30T15:17:02.792697+0000","last_undegraded":"2024-03-30T15:17:02.792697+0000","last_fullsized":"2024-03-30T15:17:02.792697+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:36.918048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:16:36.918048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.527008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1863,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791469+0000","last_change":"2024-03-30T15:16:39.417277+0000","last_active":"2024-03-30T15:17:02.791469+0000","last_peered":"2024-03-30T15:17:02.791469+0000","last_clean":"2024-03-30T15:17:02.791469+0000","last_became_active":"2024-03-30T15:13:28.634488+0000","last_became_peered":"2024-03-30T15:13:28.634488+0000","last_unstale":"2024-03-30T15:17:02.791469+0000","last_undegraded":"2024-03-30T15:17:02.791469+0000","last_fullsized":"2024-03-30T15:17:02.791469+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:39.417175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:16:39.417175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:48.461959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":211,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.946863+0000","last_change":"2024-03-30T15:17:02.946900+0000","last_active":"2024-03-30T15:17:02.946863+0000","last_peered":"2024-03-30T15:17:02.946863+0000","last_clean":"2024-03-30T15:17:02.946863+0000","last_became_active":"2024-03-30T15:13:52.592738+0000","last_became_peered":"2024-03-30T15:13:52.592738+0000","last_unstale":"2024-03-30T15:17:02.946863+0000","last_undegraded":"2024-03-30T15:17:02.946863+0000","last_fullsized":"2024-03-30T15:17:02.946863+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.946771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.946771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:28.519318+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1764,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791770+0000","last_change":"2024-03-30T15:16:21.513928+0000","last_active":"2024-03-30T15:17:02.791770+0000","last_peered":"2024-03-30T15:17:02.791770+0000","last_clean":"2024-03-30T15:17:02.791770+0000","last_became_active":"2024-03-30T15:13:28.634783+0000","last_became_peered":"2024-03-30T15:13:28.634783+0000","last_unstale":"2024-03-30T15:17:02.791770+0000","last_undegraded":"2024-03-30T15:17:02.791770+0000","last_fullsized":"2024-03-30T15:17:02.791770+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:21.513795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:21.513795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.239547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"909'32","reported_seq":1838,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833586+0000","last_change":"2024-03-30T15:15:46.530178+0000","last_active":"2024-03-30T15:17:10.833586+0000","last_peered":"2024-03-30T15:17:10.833586+0000","last_clean":"2024-03-30T15:17:10.833586+0000","last_became_active":"2024-03-30T15:13:03.539403+0000","last_became_peered":"2024-03-30T15:13:03.539403+0000","last_unstale":"2024-03-30T15:17:10.833586+0000","last_undegraded":"2024-03-30T15:17:10.833586+0000","last_fullsized":"2024-03-30T15:17:10.833586+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:15:46.530019+0000","last_deep_scrub":"909'32","last_deep_scrub_stamp":"2024-03-30T15:15:46.530019+0000","last_clean_scrub_stamp":"2024-03-30T15:15:46.530019+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:15.982894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1766,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833197+0000","last_change":"2024-03-30T15:17:07.815541+0000","last_active":"2024-03-30T15:17:10.833197+0000","last_peered":"2024-03-30T15:17:10.833197+0000","last_clean":"2024-03-30T15:17:10.833197+0000","last_became_active":"2024-03-30T15:13:19.284770+0000","last_became_peered":"2024-03-30T15:13:19.284770+0000","last_unstale":"2024-03-30T15:17:10.833197+0000","last_undegraded":"2024-03-30T15:17:10.833197+0000","last_fullsized":"2024-03-30T15:17:10.833197+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:07.815442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:17:07.815442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:33.218891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1840,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178365+0000","last_change":"2024-03-30T15:16:01.772236+0000","last_active":"2024-03-30T15:17:03.178365+0000","last_peered":"2024-03-30T15:17:03.178365+0000","last_clean":"2024-03-30T15:17:03.178365+0000","last_became_active":"2024-03-30T15:13:03.533222+0000","last_became_peered":"2024-03-30T15:13:03.533222+0000","last_unstale":"2024-03-30T15:17:03.178365+0000","last_undegraded":"2024-03-30T15:17:03.178365+0000","last_fullsized":"2024-03-30T15:17:03.178365+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:01.772100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:50.439008+0000","last_clean_scrub_stamp":"2024-03-30T15:16:01.772100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:08.202094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":167,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941092+0000","last_change":"2024-03-30T15:16:04.449108+0000","last_active":"2024-03-30T15:17:02.941092+0000","last_peered":"2024-03-30T15:17:02.941092+0000","last_clean":"2024-03-30T15:17:02.941092+0000","last_became_active":"2024-03-30T15:15:33.644118+0000","last_became_peered":"2024-03-30T15:15:33.644118+0000","last_unstale":"2024-03-30T15:17:02.941092+0000","last_undegraded":"2024-03-30T15:17:02.941092+0000","last_fullsized":"2024-03-30T15:17:02.941092+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:04.449025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:04.449025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.310647+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":1136,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177348+0000","last_change":"2024-03-30T15:16:56.972146+0000","last_active":"2024-03-30T15:17:03.177348+0000","last_peered":"2024-03-30T15:17:03.177348+0000","last_clean":"2024-03-30T15:17:03.177348+0000","last_became_active":"2024-03-30T15:13:19.288713+0000","last_became_peered":"2024-03-30T15:13:19.288713+0000","last_unstale":"2024-03-30T15:17:03.177348+0000","last_undegraded":"2024-03-30T15:17:03.177348+0000","last_fullsized":"2024-03-30T15:17:03.177348+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:56.972041+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.972041+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.980469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1764,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791236+0000","last_change":"2024-03-30T15:16:27.513604+0000","last_active":"2024-03-30T15:17:02.791236+0000","last_peered":"2024-03-30T15:17:02.791236+0000","last_clean":"2024-03-30T15:17:02.791236+0000","last_became_active":"2024-03-30T15:13:19.288815+0000","last_became_peered":"2024-03-30T15:13:19.288815+0000","last_unstale":"2024-03-30T15:17:02.791236+0000","last_undegraded":"2024-03-30T15:17:02.791236+0000","last_fullsized":"2024-03-30T15:17:02.791236+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.513453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.513453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.246772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"909'46","reported_seq":1930,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791316+0000","last_change":"2024-03-30T15:16:04.273891+0000","last_active":"2024-03-30T15:17:02.791316+0000","last_peered":"2024-03-30T15:17:02.791316+0000","last_clean":"2024-03-30T15:17:02.791316+0000","last_became_active":"2024-03-30T15:13:03.503223+0000","last_became_peered":"2024-03-30T15:13:03.503223+0000","last_unstale":"2024-03-30T15:17:02.791316+0000","last_undegraded":"2024-03-30T15:17:02.791316+0000","last_fullsized":"2024-03-30T15:17:02.791316+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:04.273778+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_clean_scrub_stamp":"2024-03-30T15:16:04.273778+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.182774+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1666,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940479+0000","last_change":"2024-03-30T15:15:39.424741+0000","last_active":"2024-03-30T15:17:02.940479+0000","last_peered":"2024-03-30T15:17:02.940479+0000","last_clean":"2024-03-30T15:17:02.940479+0000","last_became_active":"2024-03-30T15:15:33.636169+0000","last_became_peered":"2024-03-30T15:15:33.636169+0000","last_unstale":"2024-03-30T15:17:02.940479+0000","last_undegraded":"2024-03-30T15:17:02.940479+0000","last_fullsized":"2024-03-30T15:17:02.940479+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.424659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:39.424659+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.424659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:03.048167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"883'17","reported_seq":220,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832444+0000","last_change":"2024-03-30T15:16:03.458534+0000","last_active":"2024-03-30T15:17:10.832444+0000","last_peered":"2024-03-30T15:17:10.832444+0000","last_clean":"2024-03-30T15:17:10.832444+0000","last_became_active":"2024-03-30T15:14:09.730624+0000","last_became_peered":"2024-03-30T15:14:09.730624+0000","last_unstale":"2024-03-30T15:17:10.832444+0000","last_undegraded":"2024-03-30T15:17:10.832444+0000","last_fullsized":"2024-03-30T15:17:10.832444+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"883'17","last_scrub_stamp":"2024-03-30T15:16:03.458418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.458418+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.411861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1828,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177718+0000","last_change":"2024-03-30T15:16:58.981488+0000","last_active":"2024-03-30T15:17:03.177718+0000","last_peered":"2024-03-30T15:17:03.177718+0000","last_clean":"2024-03-30T15:17:03.177718+0000","last_became_active":"2024-03-30T15:13:28.623165+0000","last_became_peered":"2024-03-30T15:13:28.623165+0000","last_unstale":"2024-03-30T15:17:03.177718+0000","last_undegraded":"2024-03-30T15:17:03.177718+0000","last_fullsized":"2024-03-30T15:17:03.177718+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.981346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.981346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:28.406953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1111,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833750+0000","last_change":"2024-03-30T15:16:36.740818+0000","last_active":"2024-03-30T15:17:10.833750+0000","last_peered":"2024-03-30T15:17:10.833750+0000","last_clean":"2024-03-30T15:17:10.833750+0000","last_became_active":"2024-03-30T15:13:55.606453+0000","last_became_peered":"2024-03-30T15:13:55.606453+0000","last_unstale":"2024-03-30T15:17:10.833750+0000","last_undegraded":"2024-03-30T15:17:10.833750+0000","last_fullsized":"2024-03-30T15:17:10.833750+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:36.740711+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:16:36.740711+0000","last_clean_scrub_stamp":"2024-03-30T15:16:36.740711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:53.118332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1419,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940771+0000","last_change":"2024-03-30T15:16:35.623690+0000","last_active":"2024-03-30T15:17:02.940771+0000","last_peered":"2024-03-30T15:17:02.940771+0000","last_clean":"2024-03-30T15:17:02.940771+0000","last_became_active":"2024-03-30T15:15:33.636152+0000","last_became_peered":"2024-03-30T15:15:33.636152+0000","last_unstale":"2024-03-30T15:17:02.940771+0000","last_undegraded":"2024-03-30T15:17:02.940771+0000","last_fullsized":"2024-03-30T15:17:02.940771+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:35.623600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.623600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.474534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1750,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832895+0000","last_change":"2024-03-30T15:15:58.503036+0000","last_active":"2024-03-30T15:17:10.832895+0000","last_peered":"2024-03-30T15:17:10.832895+0000","last_clean":"2024-03-30T15:17:10.832895+0000","last_became_active":"2024-03-30T15:13:03.538990+0000","last_became_peered":"2024-03-30T15:13:03.538990+0000","last_unstale":"2024-03-30T15:17:10.832895+0000","last_undegraded":"2024-03-30T15:17:10.832895+0000","last_fullsized":"2024-03-30T15:17:10.832895+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:58.502893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:15:58.502893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.939796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"909'50","reported_seq":1881,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940688+0000","last_change":"2024-03-30T15:16:07.496122+0000","last_active":"2024-03-30T15:17:02.940688+0000","last_peered":"2024-03-30T15:17:02.940688+0000","last_clean":"2024-03-30T15:17:02.940688+0000","last_became_active":"2024-03-30T15:15:33.637486+0000","last_became_peered":"2024-03-30T15:15:33.637486+0000","last_unstale":"2024-03-30T15:17:02.940688+0000","last_undegraded":"2024-03-30T15:17:02.940688+0000","last_fullsized":"2024-03-30T15:17:02.940688+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'50","last_scrub_stamp":"2024-03-30T15:16:07.496011+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.496011+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.254292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":194,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941686+0000","last_change":"2024-03-30T15:16:56.919811+0000","last_active":"2024-03-30T15:17:02.941686+0000","last_peered":"2024-03-30T15:17:02.941686+0000","last_clean":"2024-03-30T15:17:02.941686+0000","last_became_active":"2024-03-30T15:15:33.638063+0000","last_became_peered":"2024-03-30T15:15:33.638063+0000","last_unstale":"2024-03-30T15:17:02.941686+0000","last_undegraded":"2024-03-30T15:17:02.941686+0000","last_fullsized":"2024-03-30T15:17:02.941686+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:16:56.919733+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.919733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.747537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832032+0000","last_change":"2024-03-30T15:16:17.454301+0000","last_active":"2024-03-30T15:17:10.832032+0000","last_peered":"2024-03-30T15:17:10.832032+0000","last_clean":"2024-03-30T15:17:10.832032+0000","last_became_active":"2024-03-30T15:14:09.722923+0000","last_became_peered":"2024-03-30T15:14:09.722923+0000","last_unstale":"2024-03-30T15:17:10.832032+0000","last_undegraded":"2024-03-30T15:17:10.832032+0000","last_fullsized":"2024-03-30T15:17:10.832032+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:17.454196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_clean_scrub_stamp":"2024-03-30T15:16:17.454196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.956248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"930'129","reported_seq":383,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940907+0000","last_change":"2024-03-30T15:16:50.902234+0000","last_active":"2024-03-30T15:17:02.940907+0000","last_peered":"2024-03-30T15:17:02.940907+0000","last_clean":"2024-03-30T15:17:02.940907+0000","last_became_active":"2024-03-30T15:15:33.643729+0000","last_became_peered":"2024-03-30T15:15:33.643729+0000","last_unstale":"2024-03-30T15:17:02.940907+0000","last_undegraded":"2024-03-30T15:17:02.940907+0000","last_fullsized":"2024-03-30T15:17:02.940907+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":3,"last_scrub":"930'129","last_scrub_stamp":"2024-03-30T15:16:50.902107+0000","last_deep_scrub":"930'129","last_deep_scrub_stamp":"2024-03-30T15:15:34.351447+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.902107+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.267576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1596,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792049+0000","last_change":"2024-03-30T15:16:42.952231+0000","last_active":"2024-03-30T15:17:02.792049+0000","last_peered":"2024-03-30T15:17:02.792049+0000","last_clean":"2024-03-30T15:17:02.792049+0000","last_became_active":"2024-03-30T15:13:19.290216+0000","last_became_peered":"2024-03-30T15:13:19.290216+0000","last_unstale":"2024-03-30T15:17:02.792049+0000","last_undegraded":"2024-03-30T15:17:02.792049+0000","last_fullsized":"2024-03-30T15:17:02.792049+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.694038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"930'13","reported_seq":1950,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940870+0000","last_change":"2024-03-30T15:16:08.536433+0000","last_active":"2024-03-30T15:17:02.940870+0000","last_peered":"2024-03-30T15:17:02.940870+0000","last_clean":"2024-03-30T15:17:02.940870+0000","last_became_active":"2024-03-30T15:15:33.631577+0000","last_became_peered":"2024-03-30T15:15:33.631577+0000","last_unstale":"2024-03-30T15:17:02.940870+0000","last_undegraded":"2024-03-30T15:17:02.940870+0000","last_fullsized":"2024-03-30T15:17:02.940870+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'13","last_scrub_stamp":"2024-03-30T15:16:08.536334+0000","last_deep_scrub":"899'12","last_deep_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_clean_scrub_stamp":"2024-03-30T15:16:08.536334+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.672000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"909'49","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832984+0000","last_change":"2024-03-30T15:15:57.535166+0000","last_active":"2024-03-30T15:17:10.832984+0000","last_peered":"2024-03-30T15:17:10.832984+0000","last_clean":"2024-03-30T15:17:10.832984+0000","last_became_active":"2024-03-30T15:13:03.533274+0000","last_became_peered":"2024-03-30T15:13:03.533274+0000","last_unstale":"2024-03-30T15:17:10.832984+0000","last_undegraded":"2024-03-30T15:17:10.832984+0000","last_fullsized":"2024-03-30T15:17:10.832984+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'49","last_scrub_stamp":"2024-03-30T15:15:57.535043+0000","last_deep_scrub":"909'49","last_deep_scrub_stamp":"2024-03-30T15:15:57.535043+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.535043+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:13.493728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830125+0000","last_change":"2024-03-30T15:16:15.018712+0000","last_active":"2024-03-30T15:17:10.830125+0000","last_peered":"2024-03-30T15:17:10.830125+0000","last_clean":"2024-03-30T15:17:10.830125+0000","last_became_active":"2024-03-30T15:13:28.645096+0000","last_became_peered":"2024-03-30T15:13:28.645096+0000","last_unstale":"2024-03-30T15:17:10.830125+0000","last_undegraded":"2024-03-30T15:17:10.830125+0000","last_fullsized":"2024-03-30T15:17:10.830125+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:16:15.018575+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.018575+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.714826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1845,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.471791+0000","last_change":"2024-03-30T15:16:27.392994+0000","last_active":"2024-03-30T15:17:03.471791+0000","last_peered":"2024-03-30T15:17:03.471791+0000","last_clean":"2024-03-30T15:17:03.471791+0000","last_became_active":"2024-03-30T15:13:19.274607+0000","last_became_peered":"2024-03-30T15:13:19.274607+0000","last_unstale":"2024-03-30T15:17:03.471791+0000","last_undegraded":"2024-03-30T15:17:03.471791+0000","last_fullsized":"2024-03-30T15:17:03.471791+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.392887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.392887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:28.347790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1150,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793322+0000","last_change":"2024-03-30T15:16:14.901218+0000","last_active":"2024-03-30T15:17:02.793322+0000","last_peered":"2024-03-30T15:17:02.793322+0000","last_clean":"2024-03-30T15:17:02.793322+0000","last_became_active":"2024-03-30T15:14:03.598713+0000","last_became_peered":"2024-03-30T15:14:03.598713+0000","last_unstale":"2024-03-30T15:17:02.793322+0000","last_undegraded":"2024-03-30T15:17:02.793322+0000","last_fullsized":"2024-03-30T15:17:02.793322+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:14.901112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:14.901112+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.946869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1597,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790209+0000","last_change":"2024-03-30T15:16:14.032321+0000","last_active":"2024-03-30T15:17:02.790209+0000","last_peered":"2024-03-30T15:17:02.790209+0000","last_clean":"2024-03-30T15:17:02.790209+0000","last_became_active":"2024-03-30T15:13:28.643398+0000","last_became_peered":"2024-03-30T15:13:28.643398+0000","last_unstale":"2024-03-30T15:17:02.790209+0000","last_undegraded":"2024-03-30T15:17:02.790209+0000","last_fullsized":"2024-03-30T15:17:02.790209+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:14.032241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:16:14.032241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.632287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"851'10","reported_seq":1687,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829903+0000","last_change":"2024-03-30T15:16:07.092266+0000","last_active":"2024-03-30T15:17:10.829903+0000","last_peered":"2024-03-30T15:17:10.829903+0000","last_clean":"2024-03-30T15:17:10.829903+0000","last_became_active":"2024-03-30T15:13:28.632781+0000","last_became_peered":"2024-03-30T15:13:28.632781+0000","last_unstale":"2024-03-30T15:17:10.829903+0000","last_undegraded":"2024-03-30T15:17:10.829903+0000","last_fullsized":"2024-03-30T15:17:10.829903+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'10","last_scrub_stamp":"2024-03-30T15:16:07.092180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.092180+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.903881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"909'35","reported_seq":1913,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790797+0000","last_change":"2024-03-30T15:15:49.136333+0000","last_active":"2024-03-30T15:17:02.790797+0000","last_peered":"2024-03-30T15:17:02.790797+0000","last_clean":"2024-03-30T15:17:02.790797+0000","last_became_active":"2024-03-30T15:13:03.519156+0000","last_became_peered":"2024-03-30T15:13:03.519156+0000","last_unstale":"2024-03-30T15:17:02.790797+0000","last_undegraded":"2024-03-30T15:17:02.790797+0000","last_fullsized":"2024-03-30T15:17:02.790797+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:15:49.136240+0000","last_deep_scrub":"909'35","last_deep_scrub_stamp":"2024-03-30T15:15:49.136240+0000","last_clean_scrub_stamp":"2024-03-30T15:15:49.136240+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:05.813463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1849,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177975+0000","last_change":"2024-03-30T15:16:09.715199+0000","last_active":"2024-03-30T15:17:03.177975+0000","last_peered":"2024-03-30T15:17:03.177975+0000","last_clean":"2024-03-30T15:17:03.177975+0000","last_became_active":"2024-03-30T15:13:19.286426+0000","last_became_peered":"2024-03-30T15:13:19.286426+0000","last_unstale":"2024-03-30T15:17:03.177975+0000","last_undegraded":"2024-03-30T15:17:03.177975+0000","last_fullsized":"2024-03-30T15:17:03.177975+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.715116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.715116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.243266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1458,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828751+0000","last_change":"2024-03-30T15:15:56.177795+0000","last_active":"2024-03-30T15:17:10.828751+0000","last_peered":"2024-03-30T15:17:10.828751+0000","last_clean":"2024-03-30T15:17:10.828751+0000","last_became_active":"2024-03-30T15:13:52.589257+0000","last_became_peered":"2024-03-30T15:13:52.589257+0000","last_unstale":"2024-03-30T15:17:10.828751+0000","last_undegraded":"2024-03-30T15:17:10.828751+0000","last_fullsized":"2024-03-30T15:17:10.828751+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:56.177713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:56.177713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:20.151657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1158,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793229+0000","last_change":"2024-03-30T15:16:25.903992+0000","last_active":"2024-03-30T15:17:02.793229+0000","last_peered":"2024-03-30T15:17:02.793229+0000","last_clean":"2024-03-30T15:17:02.793229+0000","last_became_active":"2024-03-30T15:14:03.596165+0000","last_became_peered":"2024-03-30T15:14:03.596165+0000","last_unstale":"2024-03-30T15:17:02.793229+0000","last_undegraded":"2024-03-30T15:17:02.793229+0000","last_fullsized":"2024-03-30T15:17:02.793229+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:25.903862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:25.903862+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:29.470975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1403,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941074+0000","last_change":"2024-03-30T15:15:48.496708+0000","last_active":"2024-03-30T15:17:02.941074+0000","last_peered":"2024-03-30T15:17:02.941074+0000","last_clean":"2024-03-30T15:17:02.941074+0000","last_became_active":"2024-03-30T15:15:33.639666+0000","last_became_peered":"2024-03-30T15:15:33.639666+0000","last_unstale":"2024-03-30T15:17:02.941074+0000","last_undegraded":"2024-03-30T15:17:02.941074+0000","last_fullsized":"2024-03-30T15:17:02.941074+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:48.496626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:52.952177+0000","last_clean_scrub_stamp":"2024-03-30T15:15:48.496626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1847,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792959+0000","last_change":"2024-03-30T15:16:57.942625+0000","last_active":"2024-03-30T15:17:02.792959+0000","last_peered":"2024-03-30T15:17:02.792959+0000","last_clean":"2024-03-30T15:17:02.792959+0000","last_became_active":"2024-03-30T15:13:28.622538+0000","last_became_peered":"2024-03-30T15:13:28.622538+0000","last_unstale":"2024-03-30T15:17:02.792959+0000","last_undegraded":"2024-03-30T15:17:02.792959+0000","last_fullsized":"2024-03-30T15:17:02.792959+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:57.942520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:57.942520+0000","last_clean_scrub_stamp":"2024-03-30T15:16:57.942520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:18.299259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"909'47","reported_seq":1482,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829301+0000","last_change":"2024-03-30T15:16:43.923599+0000","last_active":"2024-03-30T15:17:10.829301+0000","last_peered":"2024-03-30T15:17:10.829301+0000","last_clean":"2024-03-30T15:17:10.829301+0000","last_became_active":"2024-03-30T15:13:28.645038+0000","last_became_peered":"2024-03-30T15:13:28.645038+0000","last_unstale":"2024-03-30T15:17:10.829301+0000","last_undegraded":"2024-03-30T15:17:10.829301+0000","last_fullsized":"2024-03-30T15:17:10.829301+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'47","last_scrub_stamp":"2024-03-30T15:16:43.923503+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:16:43.923503+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.448641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1886,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:07.238793+0000","last_change":"2024-03-30T15:17:07.238815+0000","last_active":"2024-03-30T15:17:07.238793+0000","last_peered":"2024-03-30T15:17:07.238793+0000","last_clean":"2024-03-30T15:17:07.238793+0000","last_became_active":"2024-03-30T15:13:52.591330+0000","last_became_peered":"2024-03-30T15:13:52.591330+0000","last_unstale":"2024-03-30T15:17:07.238793+0000","last_undegraded":"2024-03-30T15:17:07.238793+0000","last_fullsized":"2024-03-30T15:17:07.238793+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:07.238727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:17:07.238727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:23.384284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1877,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790258+0000","last_change":"2024-03-30T15:16:53.258159+0000","last_active":"2024-03-30T15:17:02.790258+0000","last_peered":"2024-03-30T15:17:02.790258+0000","last_clean":"2024-03-30T15:17:02.790258+0000","last_became_active":"2024-03-30T15:13:52.588705+0000","last_became_peered":"2024-03-30T15:13:52.588705+0000","last_unstale":"2024-03-30T15:17:02.790258+0000","last_undegraded":"2024-03-30T15:17:02.790258+0000","last_fullsized":"2024-03-30T15:17:02.790258+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:53.258067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.258067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.904027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":1145,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790516+0000","last_change":"2024-03-30T15:16:02.236881+0000","last_active":"2024-03-30T15:17:02.790516+0000","last_peered":"2024-03-30T15:17:02.790516+0000","last_clean":"2024-03-30T15:17:02.790516+0000","last_became_active":"2024-03-30T15:13:03.514491+0000","last_became_peered":"2024-03-30T15:13:03.514491+0000","last_unstale":"2024-03-30T15:17:02.790516+0000","last_undegraded":"2024-03-30T15:17:02.790516+0000","last_fullsized":"2024-03-30T15:17:02.790516+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:02.236788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:02.236788+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.182637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1372,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829662+0000","last_change":"2024-03-30T15:17:02.945340+0000","last_active":"2024-03-30T15:17:10.829662+0000","last_peered":"2024-03-30T15:17:10.829662+0000","last_clean":"2024-03-30T15:17:10.829662+0000","last_became_active":"2024-03-30T15:13:28.642223+0000","last_became_peered":"2024-03-30T15:13:28.642223+0000","last_unstale":"2024-03-30T15:17:10.829662+0000","last_undegraded":"2024-03-30T15:17:10.829662+0000","last_fullsized":"2024-03-30T15:17:10.829662+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.945257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.945257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.541819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1841,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178417+0000","last_change":"2024-03-30T15:17:00.987807+0000","last_active":"2024-03-30T15:17:03.178417+0000","last_peered":"2024-03-30T15:17:03.178417+0000","last_clean":"2024-03-30T15:17:03.178417+0000","last_became_active":"2024-03-30T15:13:28.622981+0000","last_became_peered":"2024-03-30T15:13:28.622981+0000","last_unstale":"2024-03-30T15:17:03.178417+0000","last_undegraded":"2024-03-30T15:17:03.178417+0000","last_fullsized":"2024-03-30T15:17:03.178417+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.987712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:00.987712+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.987712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.697391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"909'36","reported_seq":1485,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829831+0000","last_change":"2024-03-30T15:16:31.987233+0000","last_active":"2024-03-30T15:17:10.829831+0000","last_peered":"2024-03-30T15:17:10.829831+0000","last_clean":"2024-03-30T15:17:10.829831+0000","last_became_active":"2024-03-30T15:13:52.591080+0000","last_became_peered":"2024-03-30T15:13:52.591080+0000","last_unstale":"2024-03-30T15:17:10.829831+0000","last_undegraded":"2024-03-30T15:17:10.829831+0000","last_fullsized":"2024-03-30T15:17:10.829831+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'36","last_scrub_stamp":"2024-03-30T15:16:31.987144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:31.987144+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.954546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":3204,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:06.976284+0000","last_change":"2024-03-30T15:16:59.628887+0000","last_active":"2024-03-30T15:17:06.976284+0000","last_peered":"2024-03-30T15:17:06.976284+0000","last_clean":"2024-03-30T15:17:06.976284+0000","last_became_active":"2024-03-30T15:13:28.632404+0000","last_became_peered":"2024-03-30T15:13:28.632404+0000","last_unstale":"2024-03-30T15:17:06.976284+0000","last_undegraded":"2024-03-30T15:17:06.976284+0000","last_fullsized":"2024-03-30T15:17:06.976284+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:16:59.628774+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:16:59.628774+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:23.632450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1339,"num_read_kb":1334,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1438,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828969+0000","last_change":"2024-03-30T15:16:58.945156+0000","last_active":"2024-03-30T15:17:10.828969+0000","last_peered":"2024-03-30T15:17:10.828969+0000","last_clean":"2024-03-30T15:17:10.828969+0000","last_became_active":"2024-03-30T15:14:09.726541+0000","last_became_peered":"2024-03-30T15:14:09.726541+0000","last_unstale":"2024-03-30T15:17:10.828969+0000","last_undegraded":"2024-03-30T15:17:10.828969+0000","last_fullsized":"2024-03-30T15:17:10.828969+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.945077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.945077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.682988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1861,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791583+0000","last_change":"2024-03-30T15:15:48.140991+0000","last_active":"2024-03-30T15:17:02.791583+0000","last_peered":"2024-03-30T15:17:02.791583+0000","last_clean":"2024-03-30T15:17:02.791583+0000","last_became_active":"2024-03-30T15:13:03.514904+0000","last_became_peered":"2024-03-30T15:13:03.514904+0000","last_unstale":"2024-03-30T15:17:02.791583+0000","last_undegraded":"2024-03-30T15:17:02.791583+0000","last_fullsized":"2024-03-30T15:17:02.791583+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:48.140911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:15:48.140911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1745,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.471880+0000","last_change":"2024-03-30T15:15:52.413591+0000","last_active":"2024-03-30T15:17:03.471880+0000","last_peered":"2024-03-30T15:17:03.471880+0000","last_clean":"2024-03-30T15:17:03.471880+0000","last_became_active":"2024-03-30T15:13:19.276949+0000","last_became_peered":"2024-03-30T15:13:19.276949+0000","last_unstale":"2024-03-30T15:17:03.471880+0000","last_undegraded":"2024-03-30T15:17:03.471880+0000","last_fullsized":"2024-03-30T15:17:03.471880+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:52.413467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:52.413467+0000","last_clean_scrub_stamp":"2024-03-30T15:15:52.413467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:07.327152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1849,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472021+0000","last_change":"2024-03-30T15:16:52.596504+0000","last_active":"2024-03-30T15:17:03.472021+0000","last_peered":"2024-03-30T15:17:03.472021+0000","last_clean":"2024-03-30T15:17:03.472021+0000","last_became_active":"2024-03-30T15:13:19.271491+0000","last_became_peered":"2024-03-30T15:13:19.271491+0000","last_unstale":"2024-03-30T15:17:03.472021+0000","last_undegraded":"2024-03-30T15:17:03.472021+0000","last_fullsized":"2024-03-30T15:17:03.472021+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.596404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.596404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.480407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"909'50","reported_seq":2198,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793492+0000","last_change":"2024-03-30T15:15:42.753997+0000","last_active":"2024-03-30T15:17:02.793492+0000","last_peered":"2024-03-30T15:17:02.793492+0000","last_clean":"2024-03-30T15:17:02.793492+0000","last_became_active":"2024-03-30T15:13:03.543824+0000","last_became_peered":"2024-03-30T15:13:03.543824+0000","last_unstale":"2024-03-30T15:17:02.793492+0000","last_undegraded":"2024-03-30T15:17:02.793492+0000","last_fullsized":"2024-03-30T15:17:02.793492+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'50","last_scrub_stamp":"2024-03-30T15:15:42.753852+0000","last_deep_scrub":"905'38","last_deep_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_clean_scrub_stamp":"2024-03-30T15:15:42.753852+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:07.797800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":179,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940835+0000","last_change":"2024-03-30T15:16:44.809724+0000","last_active":"2024-03-30T15:17:02.940835+0000","last_peered":"2024-03-30T15:17:02.940835+0000","last_clean":"2024-03-30T15:17:02.940835+0000","last_became_active":"2024-03-30T15:15:33.641927+0000","last_became_peered":"2024-03-30T15:15:33.641927+0000","last_unstale":"2024-03-30T15:17:02.940835+0000","last_undegraded":"2024-03-30T15:17:02.940835+0000","last_fullsized":"2024-03-30T15:17:02.940835+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:44.809644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:44.809644+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.809644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.841106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":2120,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472274+0000","last_change":"2024-03-30T15:16:20.445759+0000","last_active":"2024-03-30T15:17:03.472274+0000","last_peered":"2024-03-30T15:17:03.472274+0000","last_clean":"2024-03-30T15:17:03.472274+0000","last_became_active":"2024-03-30T15:13:28.632600+0000","last_became_peered":"2024-03-30T15:13:28.632600+0000","last_unstale":"2024-03-30T15:17:03.472274+0000","last_undegraded":"2024-03-30T15:17:03.472274+0000","last_fullsized":"2024-03-30T15:17:03.472274+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:20.445559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:20.445559+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.440558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1837,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178118+0000","last_change":"2024-03-30T15:16:22.877115+0000","last_active":"2024-03-30T15:17:03.178118+0000","last_peered":"2024-03-30T15:17:03.178118+0000","last_clean":"2024-03-30T15:17:03.178118+0000","last_became_active":"2024-03-30T15:13:03.522675+0000","last_became_peered":"2024-03-30T15:13:03.522675+0000","last_unstale":"2024-03-30T15:17:03.178118+0000","last_undegraded":"2024-03-30T15:17:03.178118+0000","last_fullsized":"2024-03-30T15:17:03.178118+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:22.877012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_clean_scrub_stamp":"2024-03-30T15:16:22.877012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.056997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":2183,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472438+0000","last_change":"2024-03-30T15:16:30.455124+0000","last_active":"2024-03-30T15:17:03.472438+0000","last_peered":"2024-03-30T15:17:03.472438+0000","last_clean":"2024-03-30T15:17:03.472438+0000","last_became_active":"2024-03-30T15:14:12.746444+0000","last_became_peered":"2024-03-30T15:14:12.746444+0000","last_unstale":"2024-03-30T15:17:03.472438+0000","last_undegraded":"2024-03-30T15:17:03.472438+0000","last_fullsized":"2024-03-30T15:17:03.472438+0000","mapping_epoch":883,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:30.455010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.455010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:46.323598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1775,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793547+0000","last_change":"2024-03-30T15:16:27.849055+0000","last_active":"2024-03-30T15:17:02.793547+0000","last_peered":"2024-03-30T15:17:02.793547+0000","last_clean":"2024-03-30T15:17:02.793547+0000","last_became_active":"2024-03-30T15:13:52.589696+0000","last_became_peered":"2024-03-30T15:13:52.589696+0000","last_unstale":"2024-03-30T15:17:02.793547+0000","last_undegraded":"2024-03-30T15:17:02.793547+0000","last_fullsized":"2024-03-30T15:17:02.793547+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.848932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:58.461031+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.848932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:28.704658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":203,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791433+0000","last_change":"2024-03-30T15:15:59.123222+0000","last_active":"2024-03-30T15:17:02.791433+0000","last_peered":"2024-03-30T15:17:02.791433+0000","last_clean":"2024-03-30T15:17:02.791433+0000","last_became_active":"2024-03-30T15:13:52.594698+0000","last_became_peered":"2024-03-30T15:13:52.594698+0000","last_unstale":"2024-03-30T15:17:02.791433+0000","last_undegraded":"2024-03-30T15:17:02.791433+0000","last_fullsized":"2024-03-30T15:17:02.791433+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:59.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:26.949114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"909'26","reported_seq":1907,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791148+0000","last_change":"2024-03-30T15:16:23.607161+0000","last_active":"2024-03-30T15:17:02.791148+0000","last_peered":"2024-03-30T15:17:02.791148+0000","last_clean":"2024-03-30T15:17:02.791148+0000","last_became_active":"2024-03-30T15:13:28.631530+0000","last_became_peered":"2024-03-30T15:13:28.631530+0000","last_unstale":"2024-03-30T15:17:02.791148+0000","last_undegraded":"2024-03-30T15:17:02.791148+0000","last_fullsized":"2024-03-30T15:17:02.791148+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'26","last_scrub_stamp":"2024-03-30T15:16:23.607038+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:16:23.607038+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.725064+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1714,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791993+0000","last_change":"2024-03-30T15:17:00.896285+0000","last_active":"2024-03-30T15:17:02.791993+0000","last_peered":"2024-03-30T15:17:02.791993+0000","last_clean":"2024-03-30T15:17:02.791993+0000","last_became_active":"2024-03-30T15:13:28.614801+0000","last_became_peered":"2024-03-30T15:13:28.614801+0000","last_unstale":"2024-03-30T15:17:02.791993+0000","last_undegraded":"2024-03-30T15:17:02.791993+0000","last_fullsized":"2024-03-30T15:17:02.791993+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.896184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:00.896184+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.896184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.207822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1876,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.789971+0000","last_change":"2024-03-30T15:16:42.437894+0000","last_active":"2024-03-30T15:17:02.789971+0000","last_peered":"2024-03-30T15:17:02.789971+0000","last_clean":"2024-03-30T15:17:02.789971+0000","last_became_active":"2024-03-30T15:13:19.290979+0000","last_became_peered":"2024-03-30T15:13:19.290979+0000","last_unstale":"2024-03-30T15:17:02.789971+0000","last_undegraded":"2024-03-30T15:17:02.789971+0000","last_fullsized":"2024-03-30T15:17:02.789971+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.437798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.437798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:55.186767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1852,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:06.757257+0000","last_change":"2024-03-30T15:17:06.757297+0000","last_active":"2024-03-30T15:17:06.757257+0000","last_peered":"2024-03-30T15:17:06.757257+0000","last_clean":"2024-03-30T15:17:06.757257+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:17:06.757257+0000","last_undegraded":"2024-03-30T15:17:06.757257+0000","last_fullsized":"2024-03-30T15:17:06.757257+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:06.757173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:17:06.757173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:09.141961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":1133,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177202+0000","last_change":"2024-03-30T15:16:54.928083+0000","last_active":"2024-03-30T15:17:03.177202+0000","last_peered":"2024-03-30T15:17:03.177202+0000","last_clean":"2024-03-30T15:17:03.177202+0000","last_became_active":"2024-03-30T15:13:28.623160+0000","last_became_peered":"2024-03-30T15:13:28.623160+0000","last_unstale":"2024-03-30T15:17:03.177202+0000","last_undegraded":"2024-03-30T15:17:03.177202+0000","last_fullsized":"2024-03-30T15:17:03.177202+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:54.927931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:54.927931+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:11.285212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1370,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829229+0000","last_change":"2024-03-30T15:16:03.117244+0000","last_active":"2024-03-30T15:17:10.829229+0000","last_peered":"2024-03-30T15:17:10.829229+0000","last_clean":"2024-03-30T15:17:10.829229+0000","last_became_active":"2024-03-30T15:13:28.639814+0000","last_became_peered":"2024-03-30T15:13:28.639814+0000","last_unstale":"2024-03-30T15:17:10.829229+0000","last_undegraded":"2024-03-30T15:17:10.829229+0000","last_fullsized":"2024-03-30T15:17:10.829229+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:03.117142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.117142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.279353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1482,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829748+0000","last_change":"2024-03-30T15:16:39.956634+0000","last_active":"2024-03-30T15:17:10.829748+0000","last_peered":"2024-03-30T15:17:10.829748+0000","last_clean":"2024-03-30T15:17:10.829748+0000","last_became_active":"2024-03-30T15:14:09.726470+0000","last_became_peered":"2024-03-30T15:14:09.726470+0000","last_unstale":"2024-03-30T15:17:10.829748+0000","last_undegraded":"2024-03-30T15:17:10.829748+0000","last_fullsized":"2024-03-30T15:17:10.829748+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:39.956558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:39.956558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:41.349718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"954'65","reported_seq":2167,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177726+0000","last_change":"2024-03-30T15:16:53.939980+0000","last_active":"2024-03-30T15:17:03.177726+0000","last_peered":"2024-03-30T15:17:03.177726+0000","last_clean":"2024-03-30T15:17:03.177726+0000","last_became_active":"2024-03-30T15:13:19.293582+0000","last_became_peered":"2024-03-30T15:13:19.293582+0000","last_unstale":"2024-03-30T15:17:03.177726+0000","last_undegraded":"2024-03-30T15:17:03.177726+0000","last_fullsized":"2024-03-30T15:17:03.177726+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'65","last_scrub_stamp":"2024-03-30T15:16:53.939834+0000","last_deep_scrub":"954'65","last_deep_scrub_stamp":"2024-03-30T15:16:53.939834+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.939834+0000","objects_scrubbed":10,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.253773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1784,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833011+0000","last_change":"2024-03-30T15:16:24.540552+0000","last_active":"2024-03-30T15:17:10.833011+0000","last_peered":"2024-03-30T15:17:10.833011+0000","last_clean":"2024-03-30T15:17:10.833011+0000","last_became_active":"2024-03-30T15:13:28.624999+0000","last_became_peered":"2024-03-30T15:13:28.624999+0000","last_unstale":"2024-03-30T15:17:10.833011+0000","last_undegraded":"2024-03-30T15:17:10.833011+0000","last_fullsized":"2024-03-30T15:17:10.833011+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:24.540431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:16:24.540431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.899577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1851,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177548+0000","last_change":"2024-03-30T15:16:30.707679+0000","last_active":"2024-03-30T15:17:03.177548+0000","last_peered":"2024-03-30T15:17:03.177548+0000","last_clean":"2024-03-30T15:17:03.177548+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:17:03.177548+0000","last_undegraded":"2024-03-30T15:17:03.177548+0000","last_fullsized":"2024-03-30T15:17:03.177548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:16:30.707574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.707574+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:35.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1503,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941625+0000","last_change":"2024-03-30T15:15:53.452852+0000","last_active":"2024-03-30T15:17:02.941625+0000","last_peered":"2024-03-30T15:17:02.941625+0000","last_clean":"2024-03-30T15:17:02.941625+0000","last_became_active":"2024-03-30T15:15:33.640955+0000","last_became_peered":"2024-03-30T15:15:33.640955+0000","last_unstale":"2024-03-30T15:17:02.941625+0000","last_undegraded":"2024-03-30T15:17:02.941625+0000","last_fullsized":"2024-03-30T15:17:02.941625+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:53.452677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:53.452677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:05.005507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1374,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829467+0000","last_change":"2024-03-30T15:16:50.961821+0000","last_active":"2024-03-30T15:17:10.829467+0000","last_peered":"2024-03-30T15:17:10.829467+0000","last_clean":"2024-03-30T15:17:10.829467+0000","last_became_active":"2024-03-30T15:14:00.632424+0000","last_became_peered":"2024-03-30T15:14:00.632424+0000","last_unstale":"2024-03-30T15:17:10.829467+0000","last_undegraded":"2024-03-30T15:17:10.829467+0000","last_fullsized":"2024-03-30T15:17:10.829467+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:50.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:50.961744+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.713880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"947'25","reported_seq":1489,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829503+0000","last_change":"2024-03-30T15:16:28.015564+0000","last_active":"2024-03-30T15:17:10.829503+0000","last_peered":"2024-03-30T15:17:10.829503+0000","last_clean":"2024-03-30T15:17:10.829503+0000","last_became_active":"2024-03-30T15:15:20.081455+0000","last_became_peered":"2024-03-30T15:15:20.081455+0000","last_unstale":"2024-03-30T15:17:10.829503+0000","last_undegraded":"2024-03-30T15:17:10.829503+0000","last_fullsized":"2024-03-30T15:17:10.829503+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"947'25","last_scrub_stamp":"2024-03-30T15:16:28.015469+0000","last_deep_scrub":"947'25","last_deep_scrub_stamp":"2024-03-30T15:16:28.015469+0000","last_clean_scrub_stamp":"2024-03-30T15:16:28.015469+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.014319+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1848,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178045+0000","last_change":"2024-03-30T15:16:47.849349+0000","last_active":"2024-03-30T15:17:03.178045+0000","last_peered":"2024-03-30T15:17:03.178045+0000","last_clean":"2024-03-30T15:17:03.178045+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:17:03.178045+0000","last_undegraded":"2024-03-30T15:17:03.178045+0000","last_fullsized":"2024-03-30T15:17:03.178045+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:47.849253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_clean_scrub_stamp":"2024-03-30T15:16:47.849253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.026665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1722,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790113+0000","last_change":"2024-03-30T15:16:33.041540+0000","last_active":"2024-03-30T15:17:02.790113+0000","last_peered":"2024-03-30T15:17:02.790113+0000","last_clean":"2024-03-30T15:17:02.790113+0000","last_became_active":"2024-03-30T15:13:19.284117+0000","last_became_peered":"2024-03-30T15:13:19.284117+0000","last_unstale":"2024-03-30T15:17:02.790113+0000","last_undegraded":"2024-03-30T15:17:02.790113+0000","last_fullsized":"2024-03-30T15:17:02.790113+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:33.041458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:33.041458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.183468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1405,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829081+0000","last_change":"2024-03-30T15:15:59.194418+0000","last_active":"2024-03-30T15:17:10.829081+0000","last_peered":"2024-03-30T15:17:10.829081+0000","last_clean":"2024-03-30T15:17:10.829081+0000","last_became_active":"2024-03-30T15:13:52.588594+0000","last_became_peered":"2024-03-30T15:13:52.588594+0000","last_unstale":"2024-03-30T15:17:10.829081+0000","last_undegraded":"2024-03-30T15:17:10.829081+0000","last_fullsized":"2024-03-30T15:17:10.829081+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:59.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1723,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833930+0000","last_change":"2024-03-30T15:17:04.884143+0000","last_active":"2024-03-30T15:17:10.833930+0000","last_peered":"2024-03-30T15:17:10.833930+0000","last_clean":"2024-03-30T15:17:10.833930+0000","last_became_active":"2024-03-30T15:14:00.634711+0000","last_became_peered":"2024-03-30T15:14:00.634711+0000","last_unstale":"2024-03-30T15:17:10.833930+0000","last_undegraded":"2024-03-30T15:17:10.833930+0000","last_fullsized":"2024-03-30T15:17:10.833930+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:04.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:28.641088+0000","last_clean_scrub_stamp":"2024-03-30T15:17:04.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.956394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"909'51","reported_seq":1990,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792883+0000","last_change":"2024-03-30T15:16:40.821388+0000","last_active":"2024-03-30T15:17:02.792883+0000","last_peered":"2024-03-30T15:17:02.792883+0000","last_clean":"2024-03-30T15:17:02.792883+0000","last_became_active":"2024-03-30T15:15:06.901815+0000","last_became_peered":"2024-03-30T15:15:06.901815+0000","last_unstale":"2024-03-30T15:17:02.792883+0000","last_undegraded":"2024-03-30T15:17:02.792883+0000","last_fullsized":"2024-03-30T15:17:02.792883+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'51","last_scrub_stamp":"2024-03-30T15:16:40.821264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:40.821264+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.625137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1848,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.473308+0000","last_change":"2024-03-30T15:16:19.416619+0000","last_active":"2024-03-30T15:17:03.473308+0000","last_peered":"2024-03-30T15:17:03.473308+0000","last_clean":"2024-03-30T15:17:03.473308+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:17:03.473308+0000","last_undegraded":"2024-03-30T15:17:03.473308+0000","last_fullsized":"2024-03-30T15:17:03.473308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.416483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:19.416483+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.416483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.017287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":211,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790542+0000","last_change":"2024-03-30T15:16:58.985136+0000","last_active":"2024-03-30T15:17:02.790542+0000","last_peered":"2024-03-30T15:17:02.790542+0000","last_clean":"2024-03-30T15:17:02.790542+0000","last_became_active":"2024-03-30T15:13:52.587447+0000","last_became_peered":"2024-03-30T15:13:52.587447+0000","last_unstale":"2024-03-30T15:17:02.790542+0000","last_undegraded":"2024-03-30T15:17:02.790542+0000","last_fullsized":"2024-03-30T15:17:02.790542+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.985041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.985041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:12.649229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1845,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.752925+0000","last_change":"2024-03-30T15:17:03.752962+0000","last_active":"2024-03-30T15:17:03.752925+0000","last_peered":"2024-03-30T15:17:03.752925+0000","last_clean":"2024-03-30T15:17:03.752925+0000","last_became_active":"2024-03-30T15:13:19.271586+0000","last_became_peered":"2024-03-30T15:13:19.271586+0000","last_unstale":"2024-03-30T15:17:03.752925+0000","last_undegraded":"2024-03-30T15:17:03.752925+0000","last_fullsized":"2024-03-30T15:17:03.752925+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:03.752852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:03.752852+0000","last_clean_scrub_stamp":"2024-03-30T15:17:03.752852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:24.902351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1765,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792435+0000","last_change":"2024-03-30T15:16:35.887068+0000","last_active":"2024-03-30T15:17:02.792435+0000","last_peered":"2024-03-30T15:17:02.792435+0000","last_clean":"2024-03-30T15:17:02.792435+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:17:02.792435+0000","last_undegraded":"2024-03-30T15:17:02.792435+0000","last_fullsized":"2024-03-30T15:17:02.792435+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:16:35.886913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.886913+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.524801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1689,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.789715+0000","last_change":"2024-03-30T15:16:30.991737+0000","last_active":"2024-03-30T15:17:02.789715+0000","last_peered":"2024-03-30T15:17:02.789715+0000","last_clean":"2024-03-30T15:17:02.789715+0000","last_became_active":"2024-03-30T15:13:28.646999+0000","last_became_peered":"2024-03-30T15:13:28.646999+0000","last_unstale":"2024-03-30T15:17:02.789715+0000","last_undegraded":"2024-03-30T15:17:02.789715+0000","last_fullsized":"2024-03-30T15:17:02.789715+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.991652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.991652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.637334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"909'24","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832986+0000","last_change":"2024-03-30T15:15:49.544265+0000","last_active":"2024-03-30T15:17:10.832986+0000","last_peered":"2024-03-30T15:17:10.832986+0000","last_clean":"2024-03-30T15:17:10.832986+0000","last_became_active":"2024-03-30T15:14:00.632108+0000","last_became_peered":"2024-03-30T15:14:00.632108+0000","last_unstale":"2024-03-30T15:17:10.832986+0000","last_undegraded":"2024-03-30T15:17:10.832986+0000","last_fullsized":"2024-03-30T15:17:10.832986+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'24","last_scrub_stamp":"2024-03-30T15:15:49.544151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:49.544151+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.475104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1849,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.473083+0000","last_change":"2024-03-30T15:15:45.282142+0000","last_active":"2024-03-30T15:17:03.473083+0000","last_peered":"2024-03-30T15:17:03.473083+0000","last_clean":"2024-03-30T15:17:03.473083+0000","last_became_active":"2024-03-30T15:13:03.550868+0000","last_became_peered":"2024-03-30T15:13:03.550868+0000","last_unstale":"2024-03-30T15:17:03.473083+0000","last_undegraded":"2024-03-30T15:17:03.473083+0000","last_fullsized":"2024-03-30T15:17:03.473083+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:45.281907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:45.281907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.843114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":167,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941331+0000","last_change":"2024-03-30T15:16:26.586028+0000","last_active":"2024-03-30T15:17:02.941331+0000","last_peered":"2024-03-30T15:17:02.941331+0000","last_clean":"2024-03-30T15:17:02.941331+0000","last_became_active":"2024-03-30T15:15:33.637445+0000","last_became_peered":"2024-03-30T15:15:33.637445+0000","last_unstale":"2024-03-30T15:17:02.941331+0000","last_undegraded":"2024-03-30T15:17:02.941331+0000","last_fullsized":"2024-03-30T15:17:02.941331+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:26.585926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.585926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:32.659121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":188,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829718+0000","last_change":"2024-03-30T15:17:09.804465+0000","last_active":"2024-03-30T15:17:10.829718+0000","last_peered":"2024-03-30T15:17:10.829718+0000","last_clean":"2024-03-30T15:17:10.829718+0000","last_became_active":"2024-03-30T15:14:09.725078+0000","last_became_peered":"2024-03-30T15:14:09.725078+0000","last_unstale":"2024-03-30T15:17:10.829718+0000","last_undegraded":"2024-03-30T15:17:10.829718+0000","last_fullsized":"2024-03-30T15:17:10.829718+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:09.804360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:09.804360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:11.997500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"913'11","reported_seq":2123,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.473110+0000","last_change":"2024-03-30T15:16:22.453685+0000","last_active":"2024-03-30T15:17:03.473110+0000","last_peered":"2024-03-30T15:17:03.473110+0000","last_clean":"2024-03-30T15:17:03.473110+0000","last_became_active":"2024-03-30T15:13:03.549786+0000","last_became_peered":"2024-03-30T15:13:03.549786+0000","last_unstale":"2024-03-30T15:17:03.473110+0000","last_undegraded":"2024-03-30T15:17:03.473110+0000","last_fullsized":"2024-03-30T15:17:03.473110+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'11","last_scrub_stamp":"2024-03-30T15:16:22.453583+0000","last_deep_scrub":"913'11","last_deep_scrub_stamp":"2024-03-30T15:15:03.185977+0000","last_clean_scrub_stamp":"2024-03-30T15:16:22.453583+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.111159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"909'39","reported_seq":1897,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177156+0000","last_change":"2024-03-30T15:16:03.751833+0000","last_active":"2024-03-30T15:17:03.177156+0000","last_peered":"2024-03-30T15:17:03.177156+0000","last_clean":"2024-03-30T15:17:03.177156+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:17:03.177156+0000","last_undegraded":"2024-03-30T15:17:03.177156+0000","last_fullsized":"2024-03-30T15:17:03.177156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'39","last_scrub_stamp":"2024-03-30T15:16:03.751714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.751714+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:26.662581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1866,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.473300+0000","last_change":"2024-03-30T15:16:44.554751+0000","last_active":"2024-03-30T15:17:03.473300+0000","last_peered":"2024-03-30T15:17:03.473300+0000","last_clean":"2024-03-30T15:17:03.473300+0000","last_became_active":"2024-03-30T15:13:28.632685+0000","last_became_peered":"2024-03-30T15:13:28.632685+0000","last_unstale":"2024-03-30T15:17:03.473300+0000","last_undegraded":"2024-03-30T15:17:03.473300+0000","last_fullsized":"2024-03-30T15:17:03.473300+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:16:44.554646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.554646+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:10.334583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1837,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177775+0000","last_change":"2024-03-30T15:16:55.962099+0000","last_active":"2024-03-30T15:17:03.177775+0000","last_peered":"2024-03-30T15:17:03.177775+0000","last_clean":"2024-03-30T15:17:03.177775+0000","last_became_active":"2024-03-30T15:13:28.625385+0000","last_became_peered":"2024-03-30T15:13:28.625385+0000","last_unstale":"2024-03-30T15:17:03.177775+0000","last_undegraded":"2024-03-30T15:17:03.177775+0000","last_fullsized":"2024-03-30T15:17:03.177775+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:55.961998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:55.961998+0000","last_clean_scrub_stamp":"2024-03-30T15:16:55.961998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.499264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1768,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793483+0000","last_change":"2024-03-30T15:15:51.797990+0000","last_active":"2024-03-30T15:17:02.793483+0000","last_peered":"2024-03-30T15:17:02.793483+0000","last_clean":"2024-03-30T15:17:02.793483+0000","last_became_active":"2024-03-30T15:13:52.592711+0000","last_became_peered":"2024-03-30T15:13:52.592711+0000","last_unstale":"2024-03-30T15:17:02.793483+0000","last_undegraded":"2024-03-30T15:17:02.793483+0000","last_fullsized":"2024-03-30T15:17:02.793483+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:51.797863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:51.797863+0000","last_clean_scrub_stamp":"2024-03-30T15:15:51.797863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.163129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"909'18","reported_seq":213,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:08.012114+0000","last_change":"2024-03-30T15:17:08.012176+0000","last_active":"2024-03-30T15:17:08.012114+0000","last_peered":"2024-03-30T15:17:08.012114+0000","last_clean":"2024-03-30T15:17:08.012114+0000","last_became_active":"2024-03-30T15:14:00.631854+0000","last_became_peered":"2024-03-30T15:14:00.631854+0000","last_unstale":"2024-03-30T15:17:08.012114+0000","last_undegraded":"2024-03-30T15:17:08.012114+0000","last_fullsized":"2024-03-30T15:17:08.012114+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'18","last_scrub_stamp":"2024-03-30T15:17:08.011989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:17:08.011989+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:14.699233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1862,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792376+0000","last_change":"2024-03-30T15:16:48.918691+0000","last_active":"2024-03-30T15:17:02.792376+0000","last_peered":"2024-03-30T15:17:02.792376+0000","last_clean":"2024-03-30T15:17:02.792376+0000","last_became_active":"2024-03-30T15:13:28.623365+0000","last_became_peered":"2024-03-30T15:13:28.623365+0000","last_unstale":"2024-03-30T15:17:02.792376+0000","last_undegraded":"2024-03-30T15:17:02.792376+0000","last_fullsized":"2024-03-30T15:17:02.792376+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:48.918557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:48.918557+0000","last_clean_scrub_stamp":"2024-03-30T15:16:48.918557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.691429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1831,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177836+0000","last_change":"2024-03-30T15:16:12.764014+0000","last_active":"2024-03-30T15:17:03.177836+0000","last_peered":"2024-03-30T15:17:03.177836+0000","last_clean":"2024-03-30T15:17:03.177836+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:17:03.177836+0000","last_undegraded":"2024-03-30T15:17:03.177836+0000","last_fullsized":"2024-03-30T15:17:03.177836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.763892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:12.763892+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.763892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.161884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1836,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472722+0000","last_change":"2024-03-30T15:16:51.644001+0000","last_active":"2024-03-30T15:17:03.472722+0000","last_peered":"2024-03-30T15:17:03.472722+0000","last_clean":"2024-03-30T15:17:03.472722+0000","last_became_active":"2024-03-30T15:13:03.537074+0000","last_became_peered":"2024-03-30T15:13:03.537074+0000","last_unstale":"2024-03-30T15:17:03.472722+0000","last_undegraded":"2024-03-30T15:17:03.472722+0000","last_fullsized":"2024-03-30T15:17:03.472722+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:51.643878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_clean_scrub_stamp":"2024-03-30T15:16:51.643878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.652355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1725,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833617+0000","last_change":"2024-03-30T15:16:44.766708+0000","last_active":"2024-03-30T15:17:10.833617+0000","last_peered":"2024-03-30T15:17:10.833617+0000","last_clean":"2024-03-30T15:17:10.833617+0000","last_became_active":"2024-03-30T15:13:03.529999+0000","last_became_peered":"2024-03-30T15:13:03.529999+0000","last_unstale":"2024-03-30T15:17:10.833617+0000","last_undegraded":"2024-03-30T15:17:10.833617+0000","last_fullsized":"2024-03-30T15:17:10.833617+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:44.766607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.766607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:49.304648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"909'21","reported_seq":1468,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830278+0000","last_change":"2024-03-30T15:16:18.042445+0000","last_active":"2024-03-30T15:17:10.830278+0000","last_peered":"2024-03-30T15:17:10.830278+0000","last_clean":"2024-03-30T15:17:10.830278+0000","last_became_active":"2024-03-30T15:13:28.642687+0000","last_became_peered":"2024-03-30T15:13:28.642687+0000","last_unstale":"2024-03-30T15:17:10.830278+0000","last_undegraded":"2024-03-30T15:17:10.830278+0000","last_fullsized":"2024-03-30T15:17:10.830278+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:18.042315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:18.042315+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:22.375270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1856,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472676+0000","last_change":"2024-03-30T15:16:55.649016+0000","last_active":"2024-03-30T15:17:03.472676+0000","last_peered":"2024-03-30T15:17:03.472676+0000","last_clean":"2024-03-30T15:17:03.472676+0000","last_became_active":"2024-03-30T15:13:03.548886+0000","last_became_peered":"2024-03-30T15:13:03.548886+0000","last_unstale":"2024-03-30T15:17:03.472676+0000","last_undegraded":"2024-03-30T15:17:03.472676+0000","last_fullsized":"2024-03-30T15:17:03.472676+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:55.648905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:55.648905+0000","last_clean_scrub_stamp":"2024-03-30T15:16:55.648905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.805796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":925,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793399+0000","last_change":"2024-03-30T15:15:39.725739+0000","last_active":"2024-03-30T15:17:02.793399+0000","last_peered":"2024-03-30T15:17:02.793399+0000","last_clean":"2024-03-30T15:17:02.793399+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:17:02.793399+0000","last_undegraded":"2024-03-30T15:17:02.793399+0000","last_fullsized":"2024-03-30T15:17:02.793399+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.725645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.725645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1692,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792239+0000","last_change":"2024-03-30T15:16:12.024253+0000","last_active":"2024-03-30T15:17:02.792239+0000","last_peered":"2024-03-30T15:17:02.792239+0000","last_clean":"2024-03-30T15:17:02.792239+0000","last_became_active":"2024-03-30T15:13:19.279299+0000","last_became_peered":"2024-03-30T15:13:19.279299+0000","last_unstale":"2024-03-30T15:17:02.792239+0000","last_undegraded":"2024-03-30T15:17:02.792239+0000","last_fullsized":"2024-03-30T15:17:02.792239+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.024110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.024110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:13.583496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1357,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829675+0000","last_change":"2024-03-30T15:15:41.246466+0000","last_active":"2024-03-30T15:17:10.829675+0000","last_peered":"2024-03-30T15:17:10.829675+0000","last_clean":"2024-03-30T15:17:10.829675+0000","last_became_active":"2024-03-30T15:13:52.589809+0000","last_became_peered":"2024-03-30T15:13:52.589809+0000","last_unstale":"2024-03-30T15:17:10.829675+0000","last_undegraded":"2024-03-30T15:17:10.829675+0000","last_fullsized":"2024-03-30T15:17:10.829675+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:41.246373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:41.246373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"909'21","reported_seq":1869,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178142+0000","last_change":"2024-03-30T15:16:18.845698+0000","last_active":"2024-03-30T15:17:03.178142+0000","last_peered":"2024-03-30T15:17:03.178142+0000","last_clean":"2024-03-30T15:17:03.178142+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:17:03.178142+0000","last_undegraded":"2024-03-30T15:17:03.178142+0000","last_fullsized":"2024-03-30T15:17:03.178142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:18.845576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:18.845576+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.137639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829102+0000","last_change":"2024-03-30T15:16:35.057933+0000","last_active":"2024-03-30T15:17:10.829102+0000","last_peered":"2024-03-30T15:17:10.829102+0000","last_clean":"2024-03-30T15:17:10.829102+0000","last_became_active":"2024-03-30T15:14:00.634218+0000","last_became_peered":"2024-03-30T15:14:00.634218+0000","last_unstale":"2024-03-30T15:17:10.829102+0000","last_undegraded":"2024-03-30T15:17:10.829102+0000","last_fullsized":"2024-03-30T15:17:10.829102+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:35.057856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.057856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.068511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"930'17","reported_seq":2018,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941023+0000","last_change":"2024-03-30T15:16:34.639572+0000","last_active":"2024-03-30T15:17:02.941023+0000","last_peered":"2024-03-30T15:17:02.941023+0000","last_clean":"2024-03-30T15:17:02.941023+0000","last_became_active":"2024-03-30T15:15:33.635657+0000","last_became_peered":"2024-03-30T15:15:33.635657+0000","last_unstale":"2024-03-30T15:17:02.941023+0000","last_undegraded":"2024-03-30T15:17:02.941023+0000","last_fullsized":"2024-03-30T15:17:02.941023+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'17","last_scrub_stamp":"2024-03-30T15:16:34.639485+0000","last_deep_scrub":"917'16","last_deep_scrub_stamp":"2024-03-30T15:15:08.284872+0000","last_clean_scrub_stamp":"2024-03-30T15:16:34.639485+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.137297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1745,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832571+0000","last_change":"2024-03-30T15:16:52.813306+0000","last_active":"2024-03-30T15:17:10.832571+0000","last_peered":"2024-03-30T15:17:10.832571+0000","last_clean":"2024-03-30T15:17:10.832571+0000","last_became_active":"2024-03-30T15:13:19.280923+0000","last_became_peered":"2024-03-30T15:13:19.280923+0000","last_unstale":"2024-03-30T15:17:10.832571+0000","last_undegraded":"2024-03-30T15:17:10.832571+0000","last_fullsized":"2024-03-30T15:17:10.832571+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.813177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.813177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:53.163111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1366,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829399+0000","last_change":"2024-03-30T15:15:54.236287+0000","last_active":"2024-03-30T15:17:10.829399+0000","last_peered":"2024-03-30T15:17:10.829399+0000","last_clean":"2024-03-30T15:17:10.829399+0000","last_became_active":"2024-03-30T15:13:28.637092+0000","last_became_peered":"2024-03-30T15:13:28.637092+0000","last_unstale":"2024-03-30T15:17:10.829399+0000","last_undegraded":"2024-03-30T15:17:10.829399+0000","last_fullsized":"2024-03-30T15:17:10.829399+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:54.236135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:54.236135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"909'35","reported_seq":1794,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941125+0000","last_change":"2024-03-30T15:15:51.463270+0000","last_active":"2024-03-30T15:17:02.941125+0000","last_peered":"2024-03-30T15:17:02.941125+0000","last_clean":"2024-03-30T15:17:02.941125+0000","last_became_active":"2024-03-30T15:15:33.636512+0000","last_became_peered":"2024-03-30T15:15:33.636512+0000","last_unstale":"2024-03-30T15:17:02.941125+0000","last_undegraded":"2024-03-30T15:17:02.941125+0000","last_fullsized":"2024-03-30T15:17:02.941125+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:15:51.463172+0000","last_deep_scrub":"903'25","last_deep_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_clean_scrub_stamp":"2024-03-30T15:15:51.463172+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:08.443834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1853,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472051+0000","last_change":"2024-03-30T15:16:00.450389+0000","last_active":"2024-03-30T15:17:03.472051+0000","last_peered":"2024-03-30T15:17:03.472051+0000","last_clean":"2024-03-30T15:17:03.472051+0000","last_became_active":"2024-03-30T15:13:19.277421+0000","last_became_peered":"2024-03-30T15:13:19.277421+0000","last_unstale":"2024-03-30T15:17:03.472051+0000","last_undegraded":"2024-03-30T15:17:03.472051+0000","last_fullsized":"2024-03-30T15:17:03.472051+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:00.450281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:00.450281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:29.621262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"852'12","reported_seq":2126,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178048+0000","last_change":"2024-03-30T15:16:51.915466+0000","last_active":"2024-03-30T15:17:03.178048+0000","last_peered":"2024-03-30T15:17:03.178048+0000","last_clean":"2024-03-30T15:17:03.178048+0000","last_became_active":"2024-03-30T15:13:29.615517+0000","last_became_peered":"2024-03-30T15:13:29.615517+0000","last_unstale":"2024-03-30T15:17:03.178048+0000","last_undegraded":"2024-03-30T15:17:03.178048+0000","last_fullsized":"2024-03-30T15:17:03.178048+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"852'12","last_scrub_stamp":"2024-03-30T15:16:51.915353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:51.915353+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.827580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1427,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829629+0000","last_change":"2024-03-30T15:16:56.913496+0000","last_active":"2024-03-30T15:17:10.829629+0000","last_peered":"2024-03-30T15:17:10.829629+0000","last_clean":"2024-03-30T15:17:10.829629+0000","last_became_active":"2024-03-30T15:14:00.638394+0000","last_became_peered":"2024-03-30T15:14:00.638394+0000","last_unstale":"2024-03-30T15:17:10.829629+0000","last_undegraded":"2024-03-30T15:17:10.829629+0000","last_fullsized":"2024-03-30T15:17:10.829629+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:16:56.913412+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.913412+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.718985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1712,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832698+0000","last_change":"2024-03-30T15:16:56.785419+0000","last_active":"2024-03-30T15:17:10.832698+0000","last_peered":"2024-03-30T15:17:10.832698+0000","last_clean":"2024-03-30T15:17:10.832698+0000","last_became_active":"2024-03-30T15:13:03.509862+0000","last_became_peered":"2024-03-30T15:13:03.509862+0000","last_unstale":"2024-03-30T15:17:10.832698+0000","last_undegraded":"2024-03-30T15:17:10.832698+0000","last_fullsized":"2024-03-30T15:17:10.832698+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.785316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.785316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.566594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"909'32","reported_seq":1842,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832050+0000","last_change":"2024-03-30T15:16:06.372815+0000","last_active":"2024-03-30T15:17:10.832050+0000","last_peered":"2024-03-30T15:17:10.832050+0000","last_clean":"2024-03-30T15:17:10.832050+0000","last_became_active":"2024-03-30T15:13:03.536364+0000","last_became_peered":"2024-03-30T15:13:03.536364+0000","last_unstale":"2024-03-30T15:17:10.832050+0000","last_undegraded":"2024-03-30T15:17:10.832050+0000","last_fullsized":"2024-03-30T15:17:10.832050+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:16:06.372644+0000","last_deep_scrub":"909'32","last_deep_scrub_stamp":"2024-03-30T15:16:06.372644+0000","last_clean_scrub_stamp":"2024-03-30T15:16:06.372644+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.579620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1792,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833268+0000","last_change":"2024-03-30T15:16:25.572404+0000","last_active":"2024-03-30T15:17:10.833268+0000","last_peered":"2024-03-30T15:17:10.833268+0000","last_clean":"2024-03-30T15:17:10.833268+0000","last_became_active":"2024-03-30T15:13:03.539915+0000","last_became_peered":"2024-03-30T15:13:03.539915+0000","last_unstale":"2024-03-30T15:17:10.833268+0000","last_undegraded":"2024-03-30T15:17:10.833268+0000","last_fullsized":"2024-03-30T15:17:10.833268+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:25.572269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:25.572269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:41.788269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1869,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790777+0000","last_change":"2024-03-30T15:16:12.440873+0000","last_active":"2024-03-30T15:17:02.790777+0000","last_peered":"2024-03-30T15:17:02.790777+0000","last_clean":"2024-03-30T15:17:02.790777+0000","last_became_active":"2024-03-30T15:14:00.629833+0000","last_became_peered":"2024-03-30T15:14:00.629833+0000","last_unstale":"2024-03-30T15:17:02.790777+0000","last_undegraded":"2024-03-30T15:17:02.790777+0000","last_fullsized":"2024-03-30T15:17:02.790777+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.440755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:53.189233+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.440755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:42.196677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1215,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940559+0000","last_change":"2024-03-30T15:16:31.616407+0000","last_active":"2024-03-30T15:17:02.940559+0000","last_peered":"2024-03-30T15:17:02.940559+0000","last_clean":"2024-03-30T15:17:02.940559+0000","last_became_active":"2024-03-30T15:15:33.639037+0000","last_became_peered":"2024-03-30T15:15:33.639037+0000","last_unstale":"2024-03-30T15:17:02.940559+0000","last_undegraded":"2024-03-30T15:17:02.940559+0000","last_fullsized":"2024-03-30T15:17:02.940559+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:31.616316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:16:31.616316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:59.497808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":188,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831989+0000","last_change":"2024-03-30T15:17:02.856511+0000","last_active":"2024-03-30T15:17:10.831989+0000","last_peered":"2024-03-30T15:17:10.831989+0000","last_clean":"2024-03-30T15:17:10.831989+0000","last_became_active":"2024-03-30T15:14:09.726417+0000","last_became_peered":"2024-03-30T15:14:09.726417+0000","last_unstale":"2024-03-30T15:17:10.831989+0000","last_undegraded":"2024-03-30T15:17:10.831989+0000","last_fullsized":"2024-03-30T15:17:10.831989+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.856410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.856410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.551174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"909'21","reported_seq":1896,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790887+0000","last_change":"2024-03-30T15:16:57.234494+0000","last_active":"2024-03-30T15:17:02.790887+0000","last_peered":"2024-03-30T15:17:02.790887+0000","last_clean":"2024-03-30T15:17:02.790887+0000","last_became_active":"2024-03-30T15:13:19.293272+0000","last_became_peered":"2024-03-30T15:13:19.293272+0000","last_unstale":"2024-03-30T15:17:02.790887+0000","last_undegraded":"2024-03-30T15:17:02.790887+0000","last_fullsized":"2024-03-30T15:17:02.790887+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:57.234407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:57.234407+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.354154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1871,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472791+0000","last_change":"2024-03-30T15:17:01.695994+0000","last_active":"2024-03-30T15:17:03.472791+0000","last_peered":"2024-03-30T15:17:03.472791+0000","last_clean":"2024-03-30T15:17:03.472791+0000","last_became_active":"2024-03-30T15:13:03.525235+0000","last_became_peered":"2024-03-30T15:13:03.525235+0000","last_unstale":"2024-03-30T15:17:03.472791+0000","last_undegraded":"2024-03-30T15:17:03.472791+0000","last_fullsized":"2024-03-30T15:17:03.472791+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:17:01.695877+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:17:01.695877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:07.698187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":818,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940902+0000","last_change":"2024-03-30T15:15:38.381992+0000","last_active":"2024-03-30T15:17:02.940902+0000","last_peered":"2024-03-30T15:17:02.940902+0000","last_clean":"2024-03-30T15:17:02.940902+0000","last_became_active":"2024-03-30T15:15:33.637711+0000","last_became_peered":"2024-03-30T15:15:33.637711+0000","last_unstale":"2024-03-30T15:17:02.940902+0000","last_undegraded":"2024-03-30T15:17:02.940902+0000","last_fullsized":"2024-03-30T15:17:02.940902+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:38.381905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:15:38.381905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1438,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828744+0000","last_change":"2024-03-30T15:16:09.084905+0000","last_active":"2024-03-30T15:17:10.828744+0000","last_peered":"2024-03-30T15:17:10.828744+0000","last_clean":"2024-03-30T15:17:10.828744+0000","last_became_active":"2024-03-30T15:13:28.634012+0000","last_became_peered":"2024-03-30T15:13:28.634012+0000","last_unstale":"2024-03-30T15:17:10.828744+0000","last_undegraded":"2024-03-30T15:17:10.828744+0000","last_fullsized":"2024-03-30T15:17:10.828744+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.084819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.084819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.458424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1328,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828867+0000","last_change":"2024-03-30T15:15:39.275512+0000","last_active":"2024-03-30T15:17:10.828867+0000","last_peered":"2024-03-30T15:17:10.828867+0000","last_clean":"2024-03-30T15:17:10.828867+0000","last_became_active":"2024-03-30T15:13:28.626496+0000","last_became_peered":"2024-03-30T15:13:28.626496+0000","last_unstale":"2024-03-30T15:17:10.828867+0000","last_undegraded":"2024-03-30T15:17:10.828867+0000","last_fullsized":"2024-03-30T15:17:10.828867+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.275423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.275423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"909'27","reported_seq":1761,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940895+0000","last_change":"2024-03-30T15:16:15.509415+0000","last_active":"2024-03-30T15:17:02.940895+0000","last_peered":"2024-03-30T15:17:02.940895+0000","last_clean":"2024-03-30T15:17:02.940895+0000","last_became_active":"2024-03-30T15:15:33.639967+0000","last_became_peered":"2024-03-30T15:15:33.639967+0000","last_unstale":"2024-03-30T15:17:02.940895+0000","last_undegraded":"2024-03-30T15:17:02.940895+0000","last_fullsized":"2024-03-30T15:17:02.940895+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'27","last_scrub_stamp":"2024-03-30T15:16:15.509294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.509294+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:25.707634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1839,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177990+0000","last_change":"2024-03-30T15:15:58.876389+0000","last_active":"2024-03-30T15:17:03.177990+0000","last_peered":"2024-03-30T15:17:03.177990+0000","last_clean":"2024-03-30T15:17:03.177990+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:17:03.177990+0000","last_undegraded":"2024-03-30T15:17:03.177990+0000","last_fullsized":"2024-03-30T15:17:03.177990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:58.876286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:58.876286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.136820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1848,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472546+0000","last_change":"2024-03-30T15:16:48.526416+0000","last_active":"2024-03-30T15:17:03.472546+0000","last_peered":"2024-03-30T15:17:03.472546+0000","last_clean":"2024-03-30T15:17:03.472546+0000","last_became_active":"2024-03-30T15:13:03.540174+0000","last_became_peered":"2024-03-30T15:13:03.540174+0000","last_unstale":"2024-03-30T15:17:03.472546+0000","last_undegraded":"2024-03-30T15:17:03.472546+0000","last_fullsized":"2024-03-30T15:17:03.472546+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:48.526316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_clean_scrub_stamp":"2024-03-30T15:16:48.526316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.444275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1171,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941441+0000","last_change":"2024-03-30T15:15:44.441499+0000","last_active":"2024-03-30T15:17:02.941441+0000","last_peered":"2024-03-30T15:17:02.941441+0000","last_clean":"2024-03-30T15:17:02.941441+0000","last_became_active":"2024-03-30T15:15:33.640822+0000","last_became_peered":"2024-03-30T15:15:33.640822+0000","last_unstale":"2024-03-30T15:17:02.941441+0000","last_undegraded":"2024-03-30T15:17:02.941441+0000","last_fullsized":"2024-03-30T15:17:02.941441+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:44.441419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:44.441419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1763,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790138+0000","last_change":"2024-03-30T15:15:39.161068+0000","last_active":"2024-03-30T15:17:02.790138+0000","last_peered":"2024-03-30T15:17:02.790138+0000","last_clean":"2024-03-30T15:17:02.790138+0000","last_became_active":"2024-03-30T15:13:03.527402+0000","last_became_peered":"2024-03-30T15:13:03.527402+0000","last_unstale":"2024-03-30T15:17:02.790138+0000","last_undegraded":"2024-03-30T15:17:02.790138+0000","last_fullsized":"2024-03-30T15:17:02.790138+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.160952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:39.160952+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.160952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:03.387528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"909'12","reported_seq":1451,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940641+0000","last_change":"2024-03-30T15:15:59.439735+0000","last_active":"2024-03-30T15:17:02.940641+0000","last_peered":"2024-03-30T15:17:02.940641+0000","last_clean":"2024-03-30T15:17:02.940641+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:17:02.940641+0000","last_undegraded":"2024-03-30T15:17:02.940641+0000","last_fullsized":"2024-03-30T15:17:02.940641+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'12","last_scrub_stamp":"2024-03-30T15:15:59.439586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.439586+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.626484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1434,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830041+0000","last_change":"2024-03-30T15:16:19.986348+0000","last_active":"2024-03-30T15:17:10.830041+0000","last_peered":"2024-03-30T15:17:10.830041+0000","last_clean":"2024-03-30T15:17:10.830041+0000","last_became_active":"2024-03-30T15:13:28.626493+0000","last_became_peered":"2024-03-30T15:13:28.626493+0000","last_unstale":"2024-03-30T15:17:10.830041+0000","last_undegraded":"2024-03-30T15:17:10.830041+0000","last_fullsized":"2024-03-30T15:17:10.830041+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.986167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.986167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:42.816121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1878,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792955+0000","last_change":"2024-03-30T15:16:56.944613+0000","last_active":"2024-03-30T15:17:02.792955+0000","last_peered":"2024-03-30T15:17:02.792955+0000","last_clean":"2024-03-30T15:17:02.792955+0000","last_became_active":"2024-03-30T15:14:09.727046+0000","last_became_peered":"2024-03-30T15:14:09.727046+0000","last_unstale":"2024-03-30T15:17:02.792955+0000","last_undegraded":"2024-03-30T15:17:02.792955+0000","last_fullsized":"2024-03-30T15:17:02.792955+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.944523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.944523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.648879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1841,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.471791+0000","last_change":"2024-03-30T15:16:58.649422+0000","last_active":"2024-03-30T15:17:03.471791+0000","last_peered":"2024-03-30T15:17:03.471791+0000","last_clean":"2024-03-30T15:17:03.471791+0000","last_became_active":"2024-03-30T15:13:19.277971+0000","last_became_peered":"2024-03-30T15:13:19.277971+0000","last_unstale":"2024-03-30T15:17:03.471791+0000","last_undegraded":"2024-03-30T15:17:03.471791+0000","last_fullsized":"2024-03-30T15:17:03.471791+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.649324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.649324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.680061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1752,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793696+0000","last_change":"2024-03-30T15:15:53.835754+0000","last_active":"2024-03-30T15:17:02.793696+0000","last_peered":"2024-03-30T15:17:02.793696+0000","last_clean":"2024-03-30T15:17:02.793696+0000","last_became_active":"2024-03-30T15:13:19.281726+0000","last_became_peered":"2024-03-30T15:13:19.281726+0000","last_unstale":"2024-03-30T15:17:02.793696+0000","last_undegraded":"2024-03-30T15:17:02.793696+0000","last_fullsized":"2024-03-30T15:17:02.793696+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:53.835649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_clean_scrub_stamp":"2024-03-30T15:15:53.835649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:22.015341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"909'32","reported_seq":1890,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793654+0000","last_change":"2024-03-30T15:15:47.766531+0000","last_active":"2024-03-30T15:17:02.793654+0000","last_peered":"2024-03-30T15:17:02.793654+0000","last_clean":"2024-03-30T15:17:02.793654+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:17:02.793654+0000","last_undegraded":"2024-03-30T15:17:02.793654+0000","last_fullsized":"2024-03-30T15:17:02.793654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:15:47.766416+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-03-30T14:56:12.032038+0000","last_clean_scrub_stamp":"2024-03-30T15:15:47.766416+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:12.417393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1143,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940600+0000","last_change":"2024-03-30T15:16:30.565832+0000","last_active":"2024-03-30T15:17:02.940600+0000","last_peered":"2024-03-30T15:17:02.940600+0000","last_clean":"2024-03-30T15:17:02.940600+0000","last_became_active":"2024-03-30T15:15:33.636878+0000","last_became_peered":"2024-03-30T15:15:33.636878+0000","last_unstale":"2024-03-30T15:17:02.940600+0000","last_undegraded":"2024-03-30T15:17:02.940600+0000","last_fullsized":"2024-03-30T15:17:02.940600+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.565747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.565747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.777062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830204+0000","last_change":"2024-03-30T15:16:05.120670+0000","last_active":"2024-03-30T15:17:10.830204+0000","last_peered":"2024-03-30T15:17:10.830204+0000","last_clean":"2024-03-30T15:17:10.830204+0000","last_became_active":"2024-03-30T15:13:52.594341+0000","last_became_peered":"2024-03-30T15:13:52.594341+0000","last_unstale":"2024-03-30T15:17:10.830204+0000","last_undegraded":"2024-03-30T15:17:10.830204+0000","last_fullsized":"2024-03-30T15:17:10.830204+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:05.120583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:05.120583+0000","last_clean_scrub_stamp":"2024-03-30T15:16:05.120583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.767478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1861,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:06.195070+0000","last_change":"2024-03-30T15:17:06.195093+0000","last_active":"2024-03-30T15:17:06.195070+0000","last_peered":"2024-03-30T15:17:06.195070+0000","last_clean":"2024-03-30T15:17:06.195070+0000","last_became_active":"2024-03-30T15:13:03.517348+0000","last_became_peered":"2024-03-30T15:13:03.517348+0000","last_unstale":"2024-03-30T15:17:06.195070+0000","last_undegraded":"2024-03-30T15:17:06.195070+0000","last_fullsized":"2024-03-30T15:17:06.195070+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:06.195011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:36.256518+0000","last_clean_scrub_stamp":"2024-03-30T15:17:06.195011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.476090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1779,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793575+0000","last_change":"2024-03-30T15:16:32.845860+0000","last_active":"2024-03-30T15:17:02.793575+0000","last_peered":"2024-03-30T15:17:02.793575+0000","last_clean":"2024-03-30T15:17:02.793575+0000","last_became_active":"2024-03-30T15:14:09.727015+0000","last_became_peered":"2024-03-30T15:14:09.727015+0000","last_unstale":"2024-03-30T15:17:02.793575+0000","last_undegraded":"2024-03-30T15:17:02.793575+0000","last_fullsized":"2024-03-30T15:17:02.793575+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:32.845790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:16:32.845790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.352493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"909'35","reported_seq":1483,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829950+0000","last_change":"2024-03-30T15:16:46.902473+0000","last_active":"2024-03-30T15:17:10.829950+0000","last_peered":"2024-03-30T15:17:10.829950+0000","last_clean":"2024-03-30T15:17:10.829950+0000","last_became_active":"2024-03-30T15:14:00.638539+0000","last_became_peered":"2024-03-30T15:14:00.638539+0000","last_unstale":"2024-03-30T15:17:10.829950+0000","last_undegraded":"2024-03-30T15:17:10.829950+0000","last_fullsized":"2024-03-30T15:17:10.829950+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:16:46.902346+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.902346+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.280878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1879,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790408+0000","last_change":"2024-03-30T15:15:45.124257+0000","last_active":"2024-03-30T15:17:02.790408+0000","last_peered":"2024-03-30T15:17:02.790408+0000","last_clean":"2024-03-30T15:17:02.790408+0000","last_became_active":"2024-03-30T15:14:09.726865+0000","last_became_peered":"2024-03-30T15:14:09.726865+0000","last_unstale":"2024-03-30T15:17:02.790408+0000","last_undegraded":"2024-03-30T15:17:02.790408+0000","last_fullsized":"2024-03-30T15:17:02.790408+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:15:45.124169+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:15:45.124169+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:10.739463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"883'15","reported_seq":2166,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793610+0000","last_change":"2024-03-30T15:16:17.902741+0000","last_active":"2024-03-30T15:17:02.793610+0000","last_peered":"2024-03-30T15:17:02.793610+0000","last_clean":"2024-03-30T15:17:02.793610+0000","last_became_active":"2024-03-30T15:14:09.727307+0000","last_became_peered":"2024-03-30T15:14:09.727307+0000","last_unstale":"2024-03-30T15:17:02.793610+0000","last_undegraded":"2024-03-30T15:17:02.793610+0000","last_fullsized":"2024-03-30T15:17:02.793610+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"883'15","last_scrub_stamp":"2024-03-30T15:16:17.902638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:17.902638+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.729614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1845,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.472190+0000","last_change":"2024-03-30T15:16:45.605024+0000","last_active":"2024-03-30T15:17:03.472190+0000","last_peered":"2024-03-30T15:17:03.472190+0000","last_clean":"2024-03-30T15:17:03.472190+0000","last_became_active":"2024-03-30T15:13:19.271296+0000","last_became_peered":"2024-03-30T15:13:19.271296+0000","last_unstale":"2024-03-30T15:17:03.472190+0000","last_undegraded":"2024-03-30T15:17:03.472190+0000","last_fullsized":"2024-03-30T15:17:03.472190+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:16:45.604906+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T15:15:29.346296+0000","last_clean_scrub_stamp":"2024-03-30T15:16:45.604906+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.947882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1774,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.793237+0000","last_change":"2024-03-30T15:16:52.953093+0000","last_active":"2024-03-30T15:17:02.793237+0000","last_peered":"2024-03-30T15:17:02.793237+0000","last_clean":"2024-03-30T15:17:02.793237+0000","last_became_active":"2024-03-30T15:13:52.588042+0000","last_became_peered":"2024-03-30T15:13:52.588042+0000","last_unstale":"2024-03-30T15:17:02.793237+0000","last_undegraded":"2024-03-30T15:17:02.793237+0000","last_fullsized":"2024-03-30T15:17:02.793237+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.953000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.953000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:18.161461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"909'24","reported_seq":1793,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941410+0000","last_change":"2024-03-30T15:16:54.880822+0000","last_active":"2024-03-30T15:17:02.941410+0000","last_peered":"2024-03-30T15:17:02.941410+0000","last_clean":"2024-03-30T15:17:02.941410+0000","last_became_active":"2024-03-30T15:15:33.635097+0000","last_became_peered":"2024-03-30T15:15:33.635097+0000","last_unstale":"2024-03-30T15:17:02.941410+0000","last_undegraded":"2024-03-30T15:17:02.941410+0000","last_fullsized":"2024-03-30T15:17:02.941410+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'24","last_scrub_stamp":"2024-03-30T15:16:54.880726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:54.880726+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.891309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1757,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833870+0000","last_change":"2024-03-30T15:17:01.847312+0000","last_active":"2024-03-30T15:17:10.833870+0000","last_peered":"2024-03-30T15:17:10.833870+0000","last_clean":"2024-03-30T15:17:10.833870+0000","last_became_active":"2024-03-30T15:13:19.285258+0000","last_became_peered":"2024-03-30T15:13:19.285258+0000","last_unstale":"2024-03-30T15:17:10.833870+0000","last_undegraded":"2024-03-30T15:17:10.833870+0000","last_fullsized":"2024-03-30T15:17:10.833870+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:01.847214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_clean_scrub_stamp":"2024-03-30T15:17:01.847214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:27.898144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1749,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832378+0000","last_change":"2024-03-30T15:16:32.650660+0000","last_active":"2024-03-30T15:17:10.832378+0000","last_peered":"2024-03-30T15:17:10.832378+0000","last_clean":"2024-03-30T15:17:10.832378+0000","last_became_active":"2024-03-30T15:13:19.291186+0000","last_became_peered":"2024-03-30T15:13:19.291186+0000","last_unstale":"2024-03-30T15:17:10.832378+0000","last_undegraded":"2024-03-30T15:17:10.832378+0000","last_fullsized":"2024-03-30T15:17:10.832378+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:32.650568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:32.650568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.328400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":167,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941426+0000","last_change":"2024-03-30T15:16:38.726895+0000","last_active":"2024-03-30T15:17:02.941426+0000","last_peered":"2024-03-30T15:17:02.941426+0000","last_clean":"2024-03-30T15:17:02.941426+0000","last_became_active":"2024-03-30T15:15:33.640737+0000","last_became_peered":"2024-03-30T15:15:33.640737+0000","last_unstale":"2024-03-30T15:17:02.941426+0000","last_undegraded":"2024-03-30T15:17:02.941426+0000","last_fullsized":"2024-03-30T15:17:02.941426+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:38.726705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:38.726705+0000","last_clean_scrub_stamp":"2024-03-30T15:16:38.726705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.512080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1757,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178141+0000","last_change":"2024-03-30T15:16:40.775179+0000","last_active":"2024-03-30T15:17:03.178141+0000","last_peered":"2024-03-30T15:17:03.178141+0000","last_clean":"2024-03-30T15:17:03.178141+0000","last_became_active":"2024-03-30T15:14:00.638153+0000","last_became_peered":"2024-03-30T15:14:00.638153+0000","last_unstale":"2024-03-30T15:17:03.178141+0000","last_undegraded":"2024-03-30T15:17:03.178141+0000","last_fullsized":"2024-03-30T15:17:03.178141+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:40.775084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:17.708825+0000","last_clean_scrub_stamp":"2024-03-30T15:16:40.775084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.308345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"909'46","reported_seq":237,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.792187+0000","last_change":"2024-03-30T15:16:33.995126+0000","last_active":"2024-03-30T15:17:02.792187+0000","last_peered":"2024-03-30T15:17:02.792187+0000","last_clean":"2024-03-30T15:17:02.792187+0000","last_became_active":"2024-03-30T15:13:52.587538+0000","last_became_peered":"2024-03-30T15:13:52.587538+0000","last_unstale":"2024-03-30T15:17:02.792187+0000","last_undegraded":"2024-03-30T15:17:02.792187+0000","last_fullsized":"2024-03-30T15:17:02.792187+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:33.995035+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:16:33.995035+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.472462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1423,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829321+0000","last_change":"2024-03-30T15:15:44.205503+0000","last_active":"2024-03-30T15:17:10.829321+0000","last_peered":"2024-03-30T15:17:10.829321+0000","last_clean":"2024-03-30T15:17:10.829321+0000","last_became_active":"2024-03-30T15:13:28.643964+0000","last_became_peered":"2024-03-30T15:13:28.643964+0000","last_unstale":"2024-03-30T15:17:10.829321+0000","last_undegraded":"2024-03-30T15:17:10.829321+0000","last_fullsized":"2024-03-30T15:17:10.829321+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:44.205405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:44.205405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1884,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.791333+0000","last_change":"2024-03-30T15:16:56.249850+0000","last_active":"2024-03-30T15:17:02.791333+0000","last_peered":"2024-03-30T15:17:02.791333+0000","last_clean":"2024-03-30T15:17:02.791333+0000","last_became_active":"2024-03-30T15:14:09.724748+0000","last_became_peered":"2024-03-30T15:14:09.724748+0000","last_unstale":"2024-03-30T15:17:02.791333+0000","last_undegraded":"2024-03-30T15:17:02.791333+0000","last_fullsized":"2024-03-30T15:17:02.791333+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.651647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1772,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832504+0000","last_change":"2024-03-30T15:16:07.404860+0000","last_active":"2024-03-30T15:17:10.832504+0000","last_peered":"2024-03-30T15:17:10.832504+0000","last_clean":"2024-03-30T15:17:10.832504+0000","last_became_active":"2024-03-30T15:13:03.536453+0000","last_became_peered":"2024-03-30T15:13:03.536453+0000","last_unstale":"2024-03-30T15:17:10.832504+0000","last_undegraded":"2024-03-30T15:17:10.832504+0000","last_fullsized":"2024-03-30T15:17:10.832504+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:07.404772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.404772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:20.453216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1764,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790665+0000","last_change":"2024-03-30T15:17:00.206475+0000","last_active":"2024-03-30T15:17:02.790665+0000","last_peered":"2024-03-30T15:17:02.790665+0000","last_clean":"2024-03-30T15:17:02.790665+0000","last_became_active":"2024-03-30T15:13:28.632549+0000","last_became_peered":"2024-03-30T15:13:28.632549+0000","last_unstale":"2024-03-30T15:17:02.790665+0000","last_undegraded":"2024-03-30T15:17:02.790665+0000","last_fullsized":"2024-03-30T15:17:02.790665+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.206355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.206355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:08.088059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"909'9","reported_seq":1827,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829438+0000","last_change":"2024-03-30T15:16:16.486714+0000","last_active":"2024-03-30T15:17:10.829438+0000","last_peered":"2024-03-30T15:17:10.829438+0000","last_clean":"2024-03-30T15:17:10.829438+0000","last_became_active":"2024-03-30T15:13:03.509107+0000","last_became_peered":"2024-03-30T15:13:03.509107+0000","last_unstale":"2024-03-30T15:17:10.829438+0000","last_undegraded":"2024-03-30T15:17:10.829438+0000","last_fullsized":"2024-03-30T15:17:10.829438+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'9","last_scrub_stamp":"2024-03-30T15:16:16.486610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:16.486610+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.092622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1848,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.178017+0000","last_change":"2024-03-30T15:16:15.770653+0000","last_active":"2024-03-30T15:17:03.178017+0000","last_peered":"2024-03-30T15:17:03.178017+0000","last_clean":"2024-03-30T15:17:03.178017+0000","last_became_active":"2024-03-30T15:13:03.518001+0000","last_became_peered":"2024-03-30T15:13:03.518001+0000","last_unstale":"2024-03-30T15:17:03.178017+0000","last_undegraded":"2024-03-30T15:17:03.178017+0000","last_fullsized":"2024-03-30T15:17:03.178017+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:15.770517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.770517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:30.386306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1840,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.473338+0000","last_change":"2024-03-30T15:16:09.553470+0000","last_active":"2024-03-30T15:17:03.473338+0000","last_peered":"2024-03-30T15:17:03.473338+0000","last_clean":"2024-03-30T15:17:03.473338+0000","last_became_active":"2024-03-30T15:13:03.526657+0000","last_became_peered":"2024-03-30T15:13:03.526657+0000","last_unstale":"2024-03-30T15:17:03.473338+0000","last_undegraded":"2024-03-30T15:17:03.473338+0000","last_fullsized":"2024-03-30T15:17:03.473338+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.553371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:52.210556+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.553371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.555329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1851,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177378+0000","last_change":"2024-03-30T15:16:19.885058+0000","last_active":"2024-03-30T15:17:03.177378+0000","last_peered":"2024-03-30T15:17:03.177378+0000","last_clean":"2024-03-30T15:17:03.177378+0000","last_became_active":"2024-03-30T15:14:00.638063+0000","last_became_peered":"2024-03-30T15:14:00.638063+0000","last_unstale":"2024-03-30T15:17:03.177378+0000","last_undegraded":"2024-03-30T15:17:03.177378+0000","last_fullsized":"2024-03-30T15:17:03.177378+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.884926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.884926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.230878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":75,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940980+0000","last_change":"2024-03-30T15:15:49.474817+0000","last_active":"2024-03-30T15:17:02.940980+0000","last_peered":"2024-03-30T15:17:02.940980+0000","last_clean":"2024-03-30T15:17:02.940980+0000","last_became_active":"2024-03-30T15:15:33.641239+0000","last_became_peered":"2024-03-30T15:15:33.641239+0000","last_unstale":"2024-03-30T15:17:02.940980+0000","last_undegraded":"2024-03-30T15:17:02.940980+0000","last_fullsized":"2024-03-30T15:17:02.940980+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:49.474729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_clean_scrub_stamp":"2024-03-30T15:15:49.474729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"909'21","reported_seq":199,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833295+0000","last_change":"2024-03-30T15:16:46.769066+0000","last_active":"2024-03-30T15:17:10.833295+0000","last_peered":"2024-03-30T15:17:10.833295+0000","last_clean":"2024-03-30T15:17:10.833295+0000","last_became_active":"2024-03-30T15:14:09.725322+0000","last_became_peered":"2024-03-30T15:14:09.725322+0000","last_unstale":"2024-03-30T15:17:10.833295+0000","last_undegraded":"2024-03-30T15:17:10.833295+0000","last_fullsized":"2024-03-30T15:17:10.833295+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:46.768964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.768964+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:10.905169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1847,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177787+0000","last_change":"2024-03-30T15:16:34.754386+0000","last_active":"2024-03-30T15:17:03.177787+0000","last_peered":"2024-03-30T15:17:03.177787+0000","last_clean":"2024-03-30T15:17:03.177787+0000","last_became_active":"2024-03-30T15:13:03.532804+0000","last_became_peered":"2024-03-30T15:13:03.532804+0000","last_unstale":"2024-03-30T15:17:03.177787+0000","last_undegraded":"2024-03-30T15:17:03.177787+0000","last_fullsized":"2024-03-30T15:17:03.177787+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:34.754272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:51.737365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:34.754272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:47.288934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1832,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177527+0000","last_change":"2024-03-30T15:15:55.946031+0000","last_active":"2024-03-30T15:17:03.177527+0000","last_peered":"2024-03-30T15:17:03.177527+0000","last_clean":"2024-03-30T15:17:03.177527+0000","last_became_active":"2024-03-30T15:13:28.624284+0000","last_became_peered":"2024-03-30T15:13:28.624284+0000","last_unstale":"2024-03-30T15:17:03.177527+0000","last_undegraded":"2024-03-30T15:17:03.177527+0000","last_fullsized":"2024-03-30T15:17:03.177527+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:55.945915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_clean_scrub_stamp":"2024-03-30T15:15:55.945915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.940104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1414,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829235+0000","last_change":"2024-03-30T15:15:35.211426+0000","last_active":"2024-03-30T15:17:10.829235+0000","last_peered":"2024-03-30T15:17:10.829235+0000","last_clean":"2024-03-30T15:17:10.829235+0000","last_became_active":"2024-03-30T15:13:28.631113+0000","last_became_peered":"2024-03-30T15:13:28.631113+0000","last_unstale":"2024-03-30T15:17:10.829235+0000","last_undegraded":"2024-03-30T15:17:10.829235+0000","last_fullsized":"2024-03-30T15:17:10.829235+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:35.211330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_clean_scrub_stamp":"2024-03-30T15:15:35.211330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1770,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:03.177220+0000","last_change":"2024-03-30T15:16:46.869868+0000","last_active":"2024-03-30T15:17:03.177220+0000","last_peered":"2024-03-30T15:17:03.177220+0000","last_clean":"2024-03-30T15:17:03.177220+0000","last_became_active":"2024-03-30T15:14:09.726973+0000","last_became_peered":"2024-03-30T15:14:09.726973+0000","last_unstale":"2024-03-30T15:17:03.177220+0000","last_undegraded":"2024-03-30T15:17:03.177220+0000","last_fullsized":"2024-03-30T15:17:03.177220+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:46.869774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.869774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.813815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"909'46","reported_seq":1677,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941171+0000","last_change":"2024-03-30T15:16:29.599273+0000","last_active":"2024-03-30T15:17:02.941171+0000","last_peered":"2024-03-30T15:17:02.941171+0000","last_clean":"2024-03-30T15:17:02.941171+0000","last_became_active":"2024-03-30T15:15:33.636090+0000","last_became_peered":"2024-03-30T15:15:33.636090+0000","last_unstale":"2024-03-30T15:17:02.941171+0000","last_undegraded":"2024-03-30T15:17:02.941171+0000","last_fullsized":"2024-03-30T15:17:02.941171+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:29.599167+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:16:29.599167+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:48.802092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1566,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.941430+0000","last_change":"2024-03-30T15:16:49.907781+0000","last_active":"2024-03-30T15:17:02.941430+0000","last_peered":"2024-03-30T15:17:02.941430+0000","last_clean":"2024-03-30T15:17:02.941430+0000","last_became_active":"2024-03-30T15:15:33.639084+0000","last_became_peered":"2024-03-30T15:15:33.639084+0000","last_unstale":"2024-03-30T15:17:02.941430+0000","last_undegraded":"2024-03-30T15:17:02.941430+0000","last_fullsized":"2024-03-30T15:17:02.941430+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:49.907687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:16:49.907687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.074815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":180,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829818+0000","last_change":"2024-03-30T15:17:00.823479+0000","last_active":"2024-03-30T15:17:10.829818+0000","last_peered":"2024-03-30T15:17:10.829818+0000","last_clean":"2024-03-30T15:17:10.829818+0000","last_became_active":"2024-03-30T15:14:09.725019+0000","last_became_peered":"2024-03-30T15:14:09.725019+0000","last_unstale":"2024-03-30T15:17:10.829818+0000","last_undegraded":"2024-03-30T15:17:10.829818+0000","last_fullsized":"2024-03-30T15:17:10.829818+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.823351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.823351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.759639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":188,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.940893+0000","last_change":"2024-03-30T15:15:55.442821+0000","last_active":"2024-03-30T15:17:02.940893+0000","last_peered":"2024-03-30T15:17:02.940893+0000","last_clean":"2024-03-30T15:17:02.940893+0000","last_became_active":"2024-03-30T15:14:00.631826+0000","last_became_peered":"2024-03-30T15:14:00.631826+0000","last_unstale":"2024-03-30T15:17:02.940893+0000","last_undegraded":"2024-03-30T15:17:02.940893+0000","last_fullsized":"2024-03-30T15:17:02.940893+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:55.442737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:55.442737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.618043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1364,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828872+0000","last_change":"2024-03-30T15:15:50.268046+0000","last_active":"2024-03-30T15:17:10.828872+0000","last_peered":"2024-03-30T15:17:10.828872+0000","last_clean":"2024-03-30T15:17:10.828872+0000","last_became_active":"2024-03-30T15:13:28.645477+0000","last_became_peered":"2024-03-30T15:13:28.645477+0000","last_unstale":"2024-03-30T15:17:10.828872+0000","last_undegraded":"2024-03-30T15:17:10.828872+0000","last_fullsized":"2024-03-30T15:17:10.828872+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:50.267958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:50.267958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.789129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"954'25","reported_seq":1815,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833025+0000","last_change":"2024-03-30T15:16:50.828853+0000","last_active":"2024-03-30T15:17:10.833025+0000","last_peered":"2024-03-30T15:17:10.833025+0000","last_clean":"2024-03-30T15:17:10.833025+0000","last_became_active":"2024-03-30T15:13:19.279835+0000","last_became_peered":"2024-03-30T15:13:19.279835+0000","last_unstale":"2024-03-30T15:17:10.833025+0000","last_undegraded":"2024-03-30T15:17:10.833025+0000","last_fullsized":"2024-03-30T15:17:10.833025+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'25","last_scrub_stamp":"2024-03-30T15:16:50.828730+0000","last_deep_scrub":"954'25","last_deep_scrub_stamp":"2024-03-30T15:16:50.828730+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.828730+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.570947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1452,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829940+0000","last_change":"2024-03-30T15:16:11.161531+0000","last_active":"2024-03-30T15:17:10.829940+0000","last_peered":"2024-03-30T15:17:10.829940+0000","last_clean":"2024-03-30T15:17:10.829940+0000","last_became_active":"2024-03-30T15:13:52.592009+0000","last_became_peered":"2024-03-30T15:13:52.592009+0000","last_unstale":"2024-03-30T15:17:10.829940+0000","last_undegraded":"2024-03-30T15:17:10.829940+0000","last_fullsized":"2024-03-30T15:17:10.829940+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:11.161451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:11.161451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.387338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1849,"reported_epoch":956,"state":"active+clean","last_fresh":"2024-03-30T15:17:02.790162+0000","last_change":"2024-03-30T15:16:13.467502+0000","last_active":"2024-03-30T15:17:02.790162+0000","last_peered":"2024-03-30T15:17:02.790162+0000","last_clean":"2024-03-30T15:17:02.790162+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:17:02.790162+0000","last_undegraded":"2024-03-30T15:17:02.790162+0000","last_fullsized":"2024-03-30T15:17:02.790162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:13.467406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:16:13.467406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.471291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1773,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833725+0000","last_change":"2024-03-30T15:16:30.669711+0000","last_active":"2024-03-30T15:17:10.833725+0000","last_peered":"2024-03-30T15:17:10.833725+0000","last_clean":"2024-03-30T15:17:10.833725+0000","last_became_active":"2024-03-30T15:13:03.529100+0000","last_became_peered":"2024-03-30T15:13:03.529100+0000","last_unstale":"2024-03-30T15:17:10.833725+0000","last_undegraded":"2024-03-30T15:17:10.833725+0000","last_fullsized":"2024-03-30T15:17:10.833725+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.669525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:59.705700+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.669525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.867855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1339,"num_read_kb":1334,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1076,"ondisk_log_size":1076,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005332,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105760,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639392,"statfs":{"total":95995035648,"available":95886737408,"internally_reserved":0,"allocated":2449408,"data_stored":887810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299673,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105532,"kb_used_data":2040,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639620,"statfs":{"total":95995035648,"available":95886970880,"internally_reserved":0,"allocated":2088960,"data_stored":557792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575156,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105444,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639708,"statfs":{"total":95995035648,"available":95887060992,"internally_reserved":0,"allocated":2125824,"data_stored":558994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":850,"seq":3650722201647,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34872,"kb_used_data":2092,"kb_used_omap":17,"kb_used_meta":32750,"kb_avail":93710280,"statfs":{"total":95995035648,"available":95959326720,"internally_reserved":0,"allocated":2142208,"data_stored":575091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17527,"internal_metadata":33536905},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":840,"seq":3607772528688,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34944,"kb_used_data":2300,"kb_used_omap":46,"kb_used_meta":32465,"kb_avail":93710208,"statfs":{"total":95995035648,"available":95959252992,"internally_reserved":0,"allocated":2355200,"data_stored":820810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47515,"internal_metadata":33244773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117289,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106160,"kb_used_data":2668,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93638992,"statfs":{"total":95995035648,"available":95886327808,"internally_reserved":0,"allocated":2732032,"data_stored":1150337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":832,"seq":3573412790324,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34744,"kb_used_data":2092,"kb_used_omap":37,"kb_used_meta":32602,"kb_avail":93710408,"statfs":{"total":95995035648,"available":95959457792,"internally_reserved":0,"allocated":2142208,"data_stored":574412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38119,"internal_metadata":33385241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659422,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106004,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639148,"statfs":{"total":95995035648,"available":95886487552,"internally_reserved":0,"allocated":2699264,"data_stored":1148839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:17:14.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T15:17:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[29876]: from='client.59450 v1:172.21.15.28:0/2702619824' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[29876]: 6.18 scrub starts 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[29876]: 6.18 scrub ok 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[29876]: pgmap v1533: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[41614]: from='client.59450 v1:172.21.15.28:0/2702619824' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[41614]: 6.18 scrub starts 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[41614]: 6.18 scrub ok 2024-03-30T15:17:14.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:14 smithi028 ceph-mon[41614]: pgmap v1533: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:14 smithi134 ceph-mon[33714]: from='client.59450 v1:172.21.15.28:0/2702619824' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:15.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:14 smithi134 ceph-mon[33714]: 6.18 scrub starts 2024-03-30T15:17:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:14 smithi134 ceph-mon[33714]: 6.18 scrub ok 2024-03-30T15:17:15.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:14 smithi134 ceph-mon[33714]: pgmap v1533: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:16.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:15 smithi028 ceph-mon[29876]: 6.14 deep-scrub starts 2024-03-30T15:17:16.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:15 smithi028 ceph-mon[29876]: 6.14 deep-scrub ok 2024-03-30T15:17:16.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:15 smithi028 ceph-mon[41614]: 6.14 deep-scrub starts 2024-03-30T15:17:16.029 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:15 smithi028 ceph-mon[41614]: 6.14 deep-scrub ok 2024-03-30T15:17:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:15 smithi134 ceph-mon[33714]: 6.14 deep-scrub starts 2024-03-30T15:17:16.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:15 smithi134 ceph-mon[33714]: 6.14 deep-scrub ok 2024-03-30T15:17:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.603+0000 7f3188e64700 1 -- 172.21.15.28:0/895891575 >> v1:172.21.15.28:6789/0 conn(0x7f3184109dc0 legacy=0x7f318410c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.603+0000 7f3188e64700 1 -- 172.21.15.28:0/895891575 shutdown_connections 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.603+0000 7f3188e64700 1 -- 172.21.15.28:0/895891575 >> 172.21.15.28:0/895891575 conn(0x7f31840fc9e0 msgr2=0x7f31840fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.603+0000 7f3188e64700 1 -- 172.21.15.28:0/895891575 shutdown_connections 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.603+0000 7f3188e64700 1 -- 172.21.15.28:0/895891575 wait complete. 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 librados: starting msgr at 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 librados: starting objecter 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 Processor -- start 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 -- start start 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 librados: setting wanted keys 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.604+0000 7f3188e64700 1 librados: calling monclient init 2024-03-30T15:17:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f3188e64700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31841a6070 con 0x7f3184109dc0 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f3188e64700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31841a7290 con 0x7f3184101130 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f3188e64700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f31841a84d0 con 0x7f318410d960 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f3182d9d700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7f318410d960 0x7f31841a4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:48538/0 (socket says 172.21.15.28:48538) 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f3182d9d700 1 -- 172.21.15.28:0/1291102110 learned_addr learned my addr 172.21.15.28:0/1291102110 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter ms_handle_connect 0x7f3184101130 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter ms_handle_connect 0x7f3184109dc0 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter ms_handle_connect 0x7f318410d960 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230822667 0 0) 0x7f31841a7290 con 0x7f3184101130 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31700031f0 con 0x7f3184101130 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575069315 0 0) 0x7f31841a84d0 con 0x7f318410d960 2024-03-30T15:17:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31841a7290 con 0x7f318410d960 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654227487 0 0) 0x7f31841a6070 con 0x7f3184109dc0 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31841a84d0 con 0x7f3184109dc0 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 691926401 0 0) 0x7f31700031f0 con 0x7f3184101130 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31841a6070 con 0x7f3184101130 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f316c001a30 con 0x7f3184101130 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4231517764 0 0) 0x7f31841a7290 con 0x7f318410d960 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31700031f0 con 0x7f318410d960 2024-03-30T15:17:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f317c001bb0 con 0x7f318410d960 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.605+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1547395882 0 0) 0x7f31841a84d0 con 0x7f3184109dc0 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31841a7290 con 0x7f3184109dc0 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f3174001880 con 0x7f3184109dc0 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2376025449 0 0) 0x7f31841a6070 con 0x7f3184101130 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 >> v1:172.21.15.28:6790/0 conn(0x7f318410d960 legacy=0x7f31841a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 >> v1:172.21.15.28:6789/0 conn(0x7f3184109dc0 legacy=0x7f31841a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31841a9710 con 0x7f3184101130 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31841a62a0 con 0x7f3184101130 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f3188e64700 10 client.51436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f31841a6760 con 0x7f3184101130 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f3188e64700 1 librados: init done 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7f316c001ce0 con 0x7f3184101130 2024-03-30T15:17:16.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.606+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7f316c004aa0 con 0x7f3184101130 2024-03-30T15:17:16.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.607+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3164005120 con 0x7f3184101130 2024-03-30T15:17:16.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.607+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7f316c003930 con 0x7f3184101130 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 8 ==== osd_map(958..958 src has 904..958) ==== 7561+0+0 (unknown 4228670894 0 0) 0x7f316c088230 con 0x7f3184101130 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 10 client.51436.objecter ms_dispatch 0x7f3184001550 osd_map(958..958 src has 904..958) 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 3 client.51436.objecter handle_osd_map got epochs [958,958] > 0 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 3 client.51436.objecter handle_osd_map decoding full epoch 958 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 20 client.51436.objecter dump_active .. 0 homeless 2024-03-30T15:17:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.608+0000 7f317b7fe700 10 client.51436.objecter ms_handle_connect 0x7f31700704d0 2024-03-30T15:17:16.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.612+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mon.1 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f316c0556c0 con 0x7f3184101130 2024-03-30T15:17:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.812+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 --> v1:172.21.15.28:6800/3011431093 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3164000b90 con 0x7f31700704d0 2024-03-30T15:17:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.824+0000 7f317b7fe700 1 -- 172.21.15.28:0/1291102110 <== mgr.14598 v1:172.21.15.28:6800/3011431093 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453399 (unknown 2965378022 0 1517093323) 0x7f3164000b90 con 0x7f31700704d0 2024-03-30T15:17:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-30T15:17:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 10 librados: watch_flush enter 2024-03-30T15:17:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 10 librados: watch_flush exit 2024-03-30T15:17:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 20 client.51436.objecter shutdown clearing up homeless session... 2024-03-30T15:17:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 10 client.51436.objecter successfully canceled tick 2024-03-30T15:17:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 >> v1:172.21.15.28:6800/3011431093 conn(0x7f31700704d0 legacy=0x7f3170072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 >> v1:172.21.15.134:6789/0 conn(0x7f3184101130 legacy=0x7f3184106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 shutdown_connections 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 >> 172.21.15.28:0/1291102110 conn(0x7f31840fc9e0 msgr2=0x7f31841105b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 shutdown_connections 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.827+0000 7f3188e64700 1 -- 172.21.15.28:0/1291102110 wait complete. 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:16.828+0000 7f3188e64700 1 librados: shutdown 2024-03-30T15:17:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-30T15:17:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[29876]: 12.1f scrub starts 2024-03-30T15:17:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[29876]: 12.1f scrub ok 2024-03-30T15:17:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[29876]: 1.8 scrub starts 2024-03-30T15:17:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[29876]: 1.8 scrub ok 2024-03-30T15:17:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[29876]: pgmap v1534: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[41614]: 12.1f scrub starts 2024-03-30T15:17:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[41614]: 12.1f scrub ok 2024-03-30T15:17:16.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[41614]: 1.8 scrub starts 2024-03-30T15:17:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[41614]: 1.8 scrub ok 2024-03-30T15:17:16.975 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:16 smithi028 ceph-mon[41614]: pgmap v1534: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:17.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:16 smithi134 ceph-mon[33714]: 12.1f scrub starts 2024-03-30T15:17:17.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:16 smithi134 ceph-mon[33714]: 12.1f scrub ok 2024-03-30T15:17:17.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:16 smithi134 ceph-mon[33714]: 1.8 scrub starts 2024-03-30T15:17:17.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:16 smithi134 ceph-mon[33714]: 1.8 scrub ok 2024-03-30T15:17:17.182 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:16 smithi134 ceph-mon[33714]: pgmap v1534: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-30T15:17:17.572 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1534,"stamp":"2024-03-30T15:17:16.257801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596323,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2907,"num_read_kb":2659,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":2333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2621,"ondisk_log_size":2621,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":633256,"kb_used_data":18440,"kb_used_omap":108,"kb_used_meta":614611,"kb_avail":749327960,"statfs":{"total":767960285184,"available":767311831040,"internally_reserved":0,"allocated":18882560,"data_stored":6367817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111128,"internal_metadata":629362152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002105"},"pg_stats":[{"pgid":"1.9","version":"86'128","reported_seq":2152,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829346+0000","last_change":"2024-03-30T15:16:26.417679+0000","last_active":"2024-03-30T15:17:10.829346+0000","last_peered":"2024-03-30T15:17:10.829346+0000","last_clean":"2024-03-30T15:17:10.829346+0000","last_became_active":"2024-03-30T15:13:55.601989+0000","last_became_peered":"2024-03-30T15:13:55.601989+0000","last_unstale":"2024-03-30T15:17:10.829346+0000","last_undegraded":"2024-03-30T15:17:10.829346+0000","last_fullsized":"2024-03-30T15:17:10.829346+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:26.417567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.417567+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:43.746799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1768,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831384+0000","last_change":"2024-03-30T15:16:26.895669+0000","last_active":"2024-03-30T15:17:10.831384+0000","last_peered":"2024-03-30T15:17:10.831384+0000","last_clean":"2024-03-30T15:17:10.831384+0000","last_became_active":"2024-03-30T15:14:00.633295+0000","last_became_peered":"2024-03-30T15:14:00.633295+0000","last_unstale":"2024-03-30T15:17:10.831384+0000","last_undegraded":"2024-03-30T15:17:10.831384+0000","last_fullsized":"2024-03-30T15:17:10.831384+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:26.895521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.895521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.969586+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"909'30","reported_seq":1798,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832309+0000","last_change":"2024-03-30T15:15:48.546754+0000","last_active":"2024-03-30T15:17:10.832309+0000","last_peered":"2024-03-30T15:17:10.832309+0000","last_clean":"2024-03-30T15:17:10.832309+0000","last_became_active":"2024-03-30T15:13:03.535865+0000","last_became_peered":"2024-03-30T15:13:03.535865+0000","last_unstale":"2024-03-30T15:17:10.832309+0000","last_undegraded":"2024-03-30T15:17:10.832309+0000","last_fullsized":"2024-03-30T15:17:10.832309+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'30","last_scrub_stamp":"2024-03-30T15:15:48.546626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:48.546626+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:15.199581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1208,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822895+0000","last_change":"2024-03-30T15:15:57.450490+0000","last_active":"2024-03-30T15:17:10.822895+0000","last_peered":"2024-03-30T15:17:10.822895+0000","last_clean":"2024-03-30T15:17:10.822895+0000","last_became_active":"2024-03-30T15:15:33.641307+0000","last_became_peered":"2024-03-30T15:15:33.641307+0000","last_unstale":"2024-03-30T15:17:10.822895+0000","last_undegraded":"2024-03-30T15:17:10.822895+0000","last_fullsized":"2024-03-30T15:17:10.822895+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:57.450409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:11.536298+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.450409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.067800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"926'12","reported_seq":1147,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831764+0000","last_change":"2024-03-30T15:16:53.980500+0000","last_active":"2024-03-30T15:17:10.831764+0000","last_peered":"2024-03-30T15:17:10.831764+0000","last_clean":"2024-03-30T15:17:10.831764+0000","last_became_active":"2024-03-30T15:13:03.543934+0000","last_became_peered":"2024-03-30T15:13:03.543934+0000","last_unstale":"2024-03-30T15:17:10.831764+0000","last_undegraded":"2024-03-30T15:17:10.831764+0000","last_fullsized":"2024-03-30T15:17:10.831764+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'12","last_scrub_stamp":"2024-03-30T15:16:53.980402+0000","last_deep_scrub":"926'12","last_deep_scrub_stamp":"2024-03-30T15:15:26.532092+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.980402+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.653007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1803,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833198+0000","last_change":"2024-03-30T15:16:42.781431+0000","last_active":"2024-03-30T15:17:10.833198+0000","last_peered":"2024-03-30T15:17:10.833198+0000","last_clean":"2024-03-30T15:17:10.833198+0000","last_became_active":"2024-03-30T15:13:52.596681+0000","last_became_peered":"2024-03-30T15:13:52.596681+0000","last_unstale":"2024-03-30T15:17:10.833198+0000","last_undegraded":"2024-03-30T15:17:10.833198+0000","last_fullsized":"2024-03-30T15:17:10.833198+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.781280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:06.712626+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.781280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:12.621398+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"86'128","reported_seq":1131,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829464+0000","last_change":"2024-03-30T15:15:57.921848+0000","last_active":"2024-03-30T15:17:10.829464+0000","last_peered":"2024-03-30T15:17:10.829464+0000","last_clean":"2024-03-30T15:17:10.829464+0000","last_became_active":"2024-03-30T15:13:19.287238+0000","last_became_peered":"2024-03-30T15:13:19.287238+0000","last_unstale":"2024-03-30T15:17:10.829464+0000","last_undegraded":"2024-03-30T15:17:10.829464+0000","last_fullsized":"2024-03-30T15:17:10.829464+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:15:57.921757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.921757+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1765,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832218+0000","last_change":"2024-03-30T15:17:09.008054+0000","last_active":"2024-03-30T15:17:10.832218+0000","last_peered":"2024-03-30T15:17:10.832218+0000","last_clean":"2024-03-30T15:17:10.832218+0000","last_became_active":"2024-03-30T15:13:19.284720+0000","last_became_peered":"2024-03-30T15:13:19.284720+0000","last_unstale":"2024-03-30T15:17:10.832218+0000","last_undegraded":"2024-03-30T15:17:10.832218+0000","last_fullsized":"2024-03-30T15:17:10.832218+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:09.007956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:09.007956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:17.246911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"917'46","reported_seq":220,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832382+0000","last_change":"2024-03-30T15:16:53.797229+0000","last_active":"2024-03-30T15:17:10.832382+0000","last_peered":"2024-03-30T15:17:10.832382+0000","last_clean":"2024-03-30T15:17:10.832382+0000","last_became_active":"2024-03-30T15:14:09.722049+0000","last_became_peered":"2024-03-30T15:14:09.722049+0000","last_unstale":"2024-03-30T15:17:10.832382+0000","last_undegraded":"2024-03-30T15:17:10.832382+0000","last_fullsized":"2024-03-30T15:17:10.832382+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'46","last_scrub_stamp":"2024-03-30T15:16:53.797109+0000","last_deep_scrub":"917'46","last_deep_scrub_stamp":"2024-03-30T15:15:10.695652+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.797109+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.519746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1867,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830593+0000","last_change":"2024-03-30T15:16:36.918150+0000","last_active":"2024-03-30T15:17:10.830593+0000","last_peered":"2024-03-30T15:17:10.830593+0000","last_clean":"2024-03-30T15:17:10.830593+0000","last_became_active":"2024-03-30T15:13:03.529657+0000","last_became_peered":"2024-03-30T15:13:03.529657+0000","last_unstale":"2024-03-30T15:17:10.830593+0000","last_undegraded":"2024-03-30T15:17:10.830593+0000","last_fullsized":"2024-03-30T15:17:10.830593+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:36.918048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:02:02.104245+0000","last_clean_scrub_stamp":"2024-03-30T15:16:36.918048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.527008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1868,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821732+0000","last_change":"2024-03-30T15:16:39.417277+0000","last_active":"2024-03-30T15:17:10.821732+0000","last_peered":"2024-03-30T15:17:10.821732+0000","last_clean":"2024-03-30T15:17:10.821732+0000","last_became_active":"2024-03-30T15:13:28.634488+0000","last_became_peered":"2024-03-30T15:13:28.634488+0000","last_unstale":"2024-03-30T15:17:10.821732+0000","last_undegraded":"2024-03-30T15:17:10.821732+0000","last_fullsized":"2024-03-30T15:17:10.821732+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:39.417175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:46.232465+0000","last_clean_scrub_stamp":"2024-03-30T15:16:39.417175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:48.461959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820199+0000","last_change":"2024-03-30T15:17:02.946900+0000","last_active":"2024-03-30T15:17:10.820199+0000","last_peered":"2024-03-30T15:17:10.820199+0000","last_clean":"2024-03-30T15:17:10.820199+0000","last_became_active":"2024-03-30T15:13:52.592738+0000","last_became_peered":"2024-03-30T15:13:52.592738+0000","last_unstale":"2024-03-30T15:17:10.820199+0000","last_undegraded":"2024-03-30T15:17:10.820199+0000","last_fullsized":"2024-03-30T15:17:10.820199+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.946771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.946771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:28.519318+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1768,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823215+0000","last_change":"2024-03-30T15:16:21.513928+0000","last_active":"2024-03-30T15:17:10.823215+0000","last_peered":"2024-03-30T15:17:10.823215+0000","last_clean":"2024-03-30T15:17:10.823215+0000","last_became_active":"2024-03-30T15:13:28.634783+0000","last_became_peered":"2024-03-30T15:13:28.634783+0000","last_unstale":"2024-03-30T15:17:10.823215+0000","last_undegraded":"2024-03-30T15:17:10.823215+0000","last_fullsized":"2024-03-30T15:17:10.823215+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:21.513795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:21.513795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.239547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"909'32","reported_seq":1838,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833586+0000","last_change":"2024-03-30T15:15:46.530178+0000","last_active":"2024-03-30T15:17:10.833586+0000","last_peered":"2024-03-30T15:17:10.833586+0000","last_clean":"2024-03-30T15:17:10.833586+0000","last_became_active":"2024-03-30T15:13:03.539403+0000","last_became_peered":"2024-03-30T15:13:03.539403+0000","last_unstale":"2024-03-30T15:17:10.833586+0000","last_undegraded":"2024-03-30T15:17:10.833586+0000","last_fullsized":"2024-03-30T15:17:10.833586+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:15:46.530019+0000","last_deep_scrub":"909'32","last_deep_scrub_stamp":"2024-03-30T15:15:46.530019+0000","last_clean_scrub_stamp":"2024-03-30T15:15:46.530019+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:15.982894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1766,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833197+0000","last_change":"2024-03-30T15:17:07.815541+0000","last_active":"2024-03-30T15:17:10.833197+0000","last_peered":"2024-03-30T15:17:10.833197+0000","last_clean":"2024-03-30T15:17:10.833197+0000","last_became_active":"2024-03-30T15:13:19.284770+0000","last_became_peered":"2024-03-30T15:13:19.284770+0000","last_unstale":"2024-03-30T15:17:10.833197+0000","last_undegraded":"2024-03-30T15:17:10.833197+0000","last_fullsized":"2024-03-30T15:17:10.833197+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:07.815442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:33.014124+0000","last_clean_scrub_stamp":"2024-03-30T15:17:07.815442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:33.218891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1852,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832833+0000","last_change":"2024-03-30T15:17:10.040562+0000","last_active":"2024-03-30T15:17:10.832833+0000","last_peered":"2024-03-30T15:17:10.832833+0000","last_clean":"2024-03-30T15:17:10.832833+0000","last_became_active":"2024-03-30T15:13:03.533222+0000","last_became_peered":"2024-03-30T15:13:03.533222+0000","last_unstale":"2024-03-30T15:17:10.832833+0000","last_undegraded":"2024-03-30T15:17:10.832833+0000","last_fullsized":"2024-03-30T15:17:10.832833+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:10.040462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:10.040462+0000","last_clean_scrub_stamp":"2024-03-30T15:17:10.040462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:32.905796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":171,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822212+0000","last_change":"2024-03-30T15:16:04.449108+0000","last_active":"2024-03-30T15:17:10.822212+0000","last_peered":"2024-03-30T15:17:10.822212+0000","last_clean":"2024-03-30T15:17:10.822212+0000","last_became_active":"2024-03-30T15:15:33.644118+0000","last_became_peered":"2024-03-30T15:15:33.644118+0000","last_unstale":"2024-03-30T15:17:10.822212+0000","last_undegraded":"2024-03-30T15:17:10.822212+0000","last_fullsized":"2024-03-30T15:17:10.822212+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:04.449025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:04.449025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.310647+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"86'128","reported_seq":1139,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830314+0000","last_change":"2024-03-30T15:16:56.972146+0000","last_active":"2024-03-30T15:17:10.830314+0000","last_peered":"2024-03-30T15:17:10.830314+0000","last_clean":"2024-03-30T15:17:10.830314+0000","last_became_active":"2024-03-30T15:13:19.288713+0000","last_became_peered":"2024-03-30T15:13:19.288713+0000","last_unstale":"2024-03-30T15:17:10.830314+0000","last_undegraded":"2024-03-30T15:17:10.830314+0000","last_fullsized":"2024-03-30T15:17:10.830314+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:56.972041+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:09:40.697341+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.972041+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.980469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1768,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822770+0000","last_change":"2024-03-30T15:16:27.513604+0000","last_active":"2024-03-30T15:17:10.822770+0000","last_peered":"2024-03-30T15:17:10.822770+0000","last_clean":"2024-03-30T15:17:10.822770+0000","last_became_active":"2024-03-30T15:13:19.288815+0000","last_became_peered":"2024-03-30T15:13:19.288815+0000","last_unstale":"2024-03-30T15:17:10.822770+0000","last_undegraded":"2024-03-30T15:17:10.822770+0000","last_fullsized":"2024-03-30T15:17:10.822770+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.513453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.176673+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.513453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.246772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"909'46","reported_seq":1935,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822923+0000","last_change":"2024-03-30T15:16:04.273891+0000","last_active":"2024-03-30T15:17:10.822923+0000","last_peered":"2024-03-30T15:17:10.822923+0000","last_clean":"2024-03-30T15:17:10.822923+0000","last_became_active":"2024-03-30T15:13:03.503223+0000","last_became_peered":"2024-03-30T15:13:03.503223+0000","last_unstale":"2024-03-30T15:17:10.822923+0000","last_undegraded":"2024-03-30T15:17:10.822923+0000","last_fullsized":"2024-03-30T15:17:10.822923+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:04.273778+0000","last_deep_scrub":"475'34","last_deep_scrub_stamp":"2024-03-30T15:13:07.392488+0000","last_clean_scrub_stamp":"2024-03-30T15:16:04.273778+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.182774+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1670,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820288+0000","last_change":"2024-03-30T15:15:39.424741+0000","last_active":"2024-03-30T15:17:10.820288+0000","last_peered":"2024-03-30T15:17:10.820288+0000","last_clean":"2024-03-30T15:17:10.820288+0000","last_became_active":"2024-03-30T15:15:33.636169+0000","last_became_peered":"2024-03-30T15:15:33.636169+0000","last_unstale":"2024-03-30T15:17:10.820288+0000","last_undegraded":"2024-03-30T15:17:10.820288+0000","last_fullsized":"2024-03-30T15:17:10.820288+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.424659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:39.424659+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.424659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"883'17","reported_seq":220,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832444+0000","last_change":"2024-03-30T15:16:03.458534+0000","last_active":"2024-03-30T15:17:10.832444+0000","last_peered":"2024-03-30T15:17:10.832444+0000","last_clean":"2024-03-30T15:17:10.832444+0000","last_became_active":"2024-03-30T15:14:09.730624+0000","last_became_peered":"2024-03-30T15:14:09.730624+0000","last_unstale":"2024-03-30T15:17:10.832444+0000","last_undegraded":"2024-03-30T15:17:10.832444+0000","last_fullsized":"2024-03-30T15:17:10.832444+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"883'17","last_scrub_stamp":"2024-03-30T15:16:03.458418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.458418+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.411861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1832,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830425+0000","last_change":"2024-03-30T15:16:58.981488+0000","last_active":"2024-03-30T15:17:10.830425+0000","last_peered":"2024-03-30T15:17:10.830425+0000","last_clean":"2024-03-30T15:17:10.830425+0000","last_became_active":"2024-03-30T15:13:28.623165+0000","last_became_peered":"2024-03-30T15:13:28.623165+0000","last_unstale":"2024-03-30T15:17:10.830425+0000","last_undegraded":"2024-03-30T15:17:10.830425+0000","last_fullsized":"2024-03-30T15:17:10.830425+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.981346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:14.201056+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.981346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:28.406953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"86'128","reported_seq":1111,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833750+0000","last_change":"2024-03-30T15:16:36.740818+0000","last_active":"2024-03-30T15:17:10.833750+0000","last_peered":"2024-03-30T15:17:10.833750+0000","last_clean":"2024-03-30T15:17:10.833750+0000","last_became_active":"2024-03-30T15:13:55.606453+0000","last_became_peered":"2024-03-30T15:13:55.606453+0000","last_unstale":"2024-03-30T15:17:10.833750+0000","last_undegraded":"2024-03-30T15:17:10.833750+0000","last_fullsized":"2024-03-30T15:17:10.833750+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:36.740711+0000","last_deep_scrub":"86'128","last_deep_scrub_stamp":"2024-03-30T15:16:36.740711+0000","last_clean_scrub_stamp":"2024-03-30T15:16:36.740711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:53.118332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1423,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820882+0000","last_change":"2024-03-30T15:16:35.623690+0000","last_active":"2024-03-30T15:17:10.820882+0000","last_peered":"2024-03-30T15:17:10.820882+0000","last_clean":"2024-03-30T15:17:10.820882+0000","last_became_active":"2024-03-30T15:15:33.636152+0000","last_became_peered":"2024-03-30T15:15:33.636152+0000","last_unstale":"2024-03-30T15:17:10.820882+0000","last_undegraded":"2024-03-30T15:17:10.820882+0000","last_fullsized":"2024-03-30T15:17:10.820882+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:35.623600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:36.165947+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.623600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.474534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1750,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832895+0000","last_change":"2024-03-30T15:15:58.503036+0000","last_active":"2024-03-30T15:17:10.832895+0000","last_peered":"2024-03-30T15:17:10.832895+0000","last_clean":"2024-03-30T15:17:10.832895+0000","last_became_active":"2024-03-30T15:13:03.538990+0000","last_became_peered":"2024-03-30T15:13:03.538990+0000","last_unstale":"2024-03-30T15:17:10.832895+0000","last_undegraded":"2024-03-30T15:17:10.832895+0000","last_fullsized":"2024-03-30T15:17:10.832895+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:58.502893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:15:58.502893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.939796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"909'50","reported_seq":1885,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820775+0000","last_change":"2024-03-30T15:16:07.496122+0000","last_active":"2024-03-30T15:17:10.820775+0000","last_peered":"2024-03-30T15:17:10.820775+0000","last_clean":"2024-03-30T15:17:10.820775+0000","last_became_active":"2024-03-30T15:15:33.637486+0000","last_became_peered":"2024-03-30T15:15:33.637486+0000","last_unstale":"2024-03-30T15:17:10.820775+0000","last_undegraded":"2024-03-30T15:17:10.820775+0000","last_fullsized":"2024-03-30T15:17:10.820775+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'50","last_scrub_stamp":"2024-03-30T15:16:07.496011+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-30T14:56:10.715316+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.496011+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.254292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"96'2","reported_seq":198,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823799+0000","last_change":"2024-03-30T15:16:56.919811+0000","last_active":"2024-03-30T15:17:10.823799+0000","last_peered":"2024-03-30T15:17:10.823799+0000","last_clean":"2024-03-30T15:17:10.823799+0000","last_became_active":"2024-03-30T15:15:33.638063+0000","last_became_peered":"2024-03-30T15:15:33.638063+0000","last_unstale":"2024-03-30T15:17:10.823799+0000","last_undegraded":"2024-03-30T15:17:10.823799+0000","last_fullsized":"2024-03-30T15:17:10.823799+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:16:56.919733+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:37.777339+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.919733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.747537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":180,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832032+0000","last_change":"2024-03-30T15:16:17.454301+0000","last_active":"2024-03-30T15:17:10.832032+0000","last_peered":"2024-03-30T15:17:10.832032+0000","last_clean":"2024-03-30T15:17:10.832032+0000","last_became_active":"2024-03-30T15:14:09.722923+0000","last_became_peered":"2024-03-30T15:14:09.722923+0000","last_unstale":"2024-03-30T15:17:10.832032+0000","last_undegraded":"2024-03-30T15:17:10.832032+0000","last_fullsized":"2024-03-30T15:17:10.832032+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:17.454196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:48.594615+0000","last_clean_scrub_stamp":"2024-03-30T15:16:17.454196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.956248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"930'129","reported_seq":387,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822052+0000","last_change":"2024-03-30T15:16:50.902234+0000","last_active":"2024-03-30T15:17:10.822052+0000","last_peered":"2024-03-30T15:17:10.822052+0000","last_clean":"2024-03-30T15:17:10.822052+0000","last_became_active":"2024-03-30T15:15:33.643729+0000","last_became_peered":"2024-03-30T15:15:33.643729+0000","last_unstale":"2024-03-30T15:17:10.822052+0000","last_undegraded":"2024-03-30T15:17:10.822052+0000","last_fullsized":"2024-03-30T15:17:10.822052+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":3,"last_scrub":"930'129","last_scrub_stamp":"2024-03-30T15:16:50.902107+0000","last_deep_scrub":"930'129","last_deep_scrub_stamp":"2024-03-30T15:15:34.351447+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.902107+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.267576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1600,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821007+0000","last_change":"2024-03-30T15:16:42.952231+0000","last_active":"2024-03-30T15:17:10.821007+0000","last_peered":"2024-03-30T15:17:10.821007+0000","last_clean":"2024-03-30T15:17:10.821007+0000","last_became_active":"2024-03-30T15:13:19.290216+0000","last_became_peered":"2024-03-30T15:13:19.290216+0000","last_unstale":"2024-03-30T15:17:10.821007+0000","last_undegraded":"2024-03-30T15:17:10.821007+0000","last_fullsized":"2024-03-30T15:17:10.821007+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.952143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:48.979438+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.952143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.694038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"930'13","reported_seq":1955,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821024+0000","last_change":"2024-03-30T15:16:08.536433+0000","last_active":"2024-03-30T15:17:10.821024+0000","last_peered":"2024-03-30T15:17:10.821024+0000","last_clean":"2024-03-30T15:17:10.821024+0000","last_became_active":"2024-03-30T15:15:33.631577+0000","last_became_peered":"2024-03-30T15:15:33.631577+0000","last_unstale":"2024-03-30T15:17:10.821024+0000","last_undegraded":"2024-03-30T15:17:10.821024+0000","last_fullsized":"2024-03-30T15:17:10.821024+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'13","last_scrub_stamp":"2024-03-30T15:16:08.536334+0000","last_deep_scrub":"899'12","last_deep_scrub_stamp":"2024-03-30T15:14:28.186699+0000","last_clean_scrub_stamp":"2024-03-30T15:16:08.536334+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.672000+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"909'49","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832984+0000","last_change":"2024-03-30T15:15:57.535166+0000","last_active":"2024-03-30T15:17:10.832984+0000","last_peered":"2024-03-30T15:17:10.832984+0000","last_clean":"2024-03-30T15:17:10.832984+0000","last_became_active":"2024-03-30T15:13:03.533274+0000","last_became_peered":"2024-03-30T15:13:03.533274+0000","last_unstale":"2024-03-30T15:17:10.832984+0000","last_undegraded":"2024-03-30T15:17:10.832984+0000","last_fullsized":"2024-03-30T15:17:10.832984+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'49","last_scrub_stamp":"2024-03-30T15:15:57.535043+0000","last_deep_scrub":"909'49","last_deep_scrub_stamp":"2024-03-30T15:15:57.535043+0000","last_clean_scrub_stamp":"2024-03-30T15:15:57.535043+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:13.493728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"141'2","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830125+0000","last_change":"2024-03-30T15:16:15.018712+0000","last_active":"2024-03-30T15:17:10.830125+0000","last_peered":"2024-03-30T15:17:10.830125+0000","last_clean":"2024-03-30T15:17:10.830125+0000","last_became_active":"2024-03-30T15:13:28.645096+0000","last_became_peered":"2024-03-30T15:13:28.645096+0000","last_unstale":"2024-03-30T15:17:10.830125+0000","last_undegraded":"2024-03-30T15:17:10.830125+0000","last_fullsized":"2024-03-30T15:17:10.830125+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-03-30T15:16:15.018575+0000","last_deep_scrub":"141'2","last_deep_scrub_stamp":"2024-03-30T14:58:25.170239+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.018575+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.714826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1849,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830546+0000","last_change":"2024-03-30T15:16:27.392994+0000","last_active":"2024-03-30T15:17:10.830546+0000","last_peered":"2024-03-30T15:17:10.830546+0000","last_clean":"2024-03-30T15:17:10.830546+0000","last_became_active":"2024-03-30T15:13:19.274607+0000","last_became_peered":"2024-03-30T15:13:19.274607+0000","last_unstale":"2024-03-30T15:17:10.830546+0000","last_undegraded":"2024-03-30T15:17:10.830546+0000","last_fullsized":"2024-03-30T15:17:10.830546+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.392887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:42.150834+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.392887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:28.347790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"86'128","reported_seq":1154,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831882+0000","last_change":"2024-03-30T15:16:14.901218+0000","last_active":"2024-03-30T15:17:10.831882+0000","last_peered":"2024-03-30T15:17:10.831882+0000","last_clean":"2024-03-30T15:17:10.831882+0000","last_became_active":"2024-03-30T15:14:03.598713+0000","last_became_peered":"2024-03-30T15:14:03.598713+0000","last_unstale":"2024-03-30T15:17:10.831882+0000","last_undegraded":"2024-03-30T15:17:10.831882+0000","last_fullsized":"2024-03-30T15:17:10.831882+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:14.901112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:14.901112+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.946869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1601,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822104+0000","last_change":"2024-03-30T15:16:14.032321+0000","last_active":"2024-03-30T15:17:10.822104+0000","last_peered":"2024-03-30T15:17:10.822104+0000","last_clean":"2024-03-30T15:17:10.822104+0000","last_became_active":"2024-03-30T15:13:28.643398+0000","last_became_peered":"2024-03-30T15:13:28.643398+0000","last_unstale":"2024-03-30T15:17:10.822104+0000","last_undegraded":"2024-03-30T15:17:10.822104+0000","last_fullsized":"2024-03-30T15:17:10.822104+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:14.032241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:47.018083+0000","last_clean_scrub_stamp":"2024-03-30T15:16:14.032241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.632287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"851'10","reported_seq":1687,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829903+0000","last_change":"2024-03-30T15:16:07.092266+0000","last_active":"2024-03-30T15:17:10.829903+0000","last_peered":"2024-03-30T15:17:10.829903+0000","last_clean":"2024-03-30T15:17:10.829903+0000","last_became_active":"2024-03-30T15:13:28.632781+0000","last_became_peered":"2024-03-30T15:13:28.632781+0000","last_unstale":"2024-03-30T15:17:10.829903+0000","last_undegraded":"2024-03-30T15:17:10.829903+0000","last_fullsized":"2024-03-30T15:17:10.829903+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'10","last_scrub_stamp":"2024-03-30T15:16:07.092180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.092180+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.903881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"909'35","reported_seq":1925,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:12.311313+0000","last_change":"2024-03-30T15:17:12.311374+0000","last_active":"2024-03-30T15:17:12.311313+0000","last_peered":"2024-03-30T15:17:12.311313+0000","last_clean":"2024-03-30T15:17:12.311313+0000","last_became_active":"2024-03-30T15:13:03.519156+0000","last_became_peered":"2024-03-30T15:13:03.519156+0000","last_unstale":"2024-03-30T15:17:12.311313+0000","last_undegraded":"2024-03-30T15:17:12.311313+0000","last_fullsized":"2024-03-30T15:17:12.311313+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:17:12.311225+0000","last_deep_scrub":"909'35","last_deep_scrub_stamp":"2024-03-30T15:15:49.136240+0000","last_clean_scrub_stamp":"2024-03-30T15:17:12.311225+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:27.338458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1852,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830848+0000","last_change":"2024-03-30T15:16:09.715199+0000","last_active":"2024-03-30T15:17:10.830848+0000","last_peered":"2024-03-30T15:17:10.830848+0000","last_clean":"2024-03-30T15:17:10.830848+0000","last_became_active":"2024-03-30T15:13:19.286426+0000","last_became_peered":"2024-03-30T15:13:19.286426+0000","last_unstale":"2024-03-30T15:17:10.830848+0000","last_undegraded":"2024-03-30T15:17:10.830848+0000","last_fullsized":"2024-03-30T15:17:10.830848+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.715116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:55:41.454365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.715116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.243266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1458,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828751+0000","last_change":"2024-03-30T15:15:56.177795+0000","last_active":"2024-03-30T15:17:10.828751+0000","last_peered":"2024-03-30T15:17:10.828751+0000","last_clean":"2024-03-30T15:17:10.828751+0000","last_became_active":"2024-03-30T15:13:52.589257+0000","last_became_peered":"2024-03-30T15:13:52.589257+0000","last_unstale":"2024-03-30T15:17:10.828751+0000","last_undegraded":"2024-03-30T15:17:10.828751+0000","last_fullsized":"2024-03-30T15:17:10.828751+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:56.177713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:56.177713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:20.151657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"86'128","reported_seq":1162,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831486+0000","last_change":"2024-03-30T15:16:25.903992+0000","last_active":"2024-03-30T15:17:10.831486+0000","last_peered":"2024-03-30T15:17:10.831486+0000","last_clean":"2024-03-30T15:17:10.831486+0000","last_became_active":"2024-03-30T15:14:03.596165+0000","last_became_peered":"2024-03-30T15:14:03.596165+0000","last_unstale":"2024-03-30T15:17:10.831486+0000","last_undegraded":"2024-03-30T15:17:10.831486+0000","last_fullsized":"2024-03-30T15:17:10.831486+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:25.903862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:25.903862+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:29.470975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1415,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822713+0000","last_change":"2024-03-30T15:17:10.088244+0000","last_active":"2024-03-30T15:17:10.822713+0000","last_peered":"2024-03-30T15:17:10.822713+0000","last_clean":"2024-03-30T15:17:10.822713+0000","last_became_active":"2024-03-30T15:15:33.639666+0000","last_became_peered":"2024-03-30T15:15:33.639666+0000","last_unstale":"2024-03-30T15:17:10.822713+0000","last_undegraded":"2024-03-30T15:17:10.822713+0000","last_fullsized":"2024-03-30T15:17:10.822713+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:10.088164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:10.088164+0000","last_clean_scrub_stamp":"2024-03-30T15:17:10.088164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:30.501374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1851,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831031+0000","last_change":"2024-03-30T15:16:57.942625+0000","last_active":"2024-03-30T15:17:10.831031+0000","last_peered":"2024-03-30T15:17:10.831031+0000","last_clean":"2024-03-30T15:17:10.831031+0000","last_became_active":"2024-03-30T15:13:28.622538+0000","last_became_peered":"2024-03-30T15:13:28.622538+0000","last_unstale":"2024-03-30T15:17:10.831031+0000","last_undegraded":"2024-03-30T15:17:10.831031+0000","last_fullsized":"2024-03-30T15:17:10.831031+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:57.942520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:57.942520+0000","last_clean_scrub_stamp":"2024-03-30T15:16:57.942520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:18.299259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"909'47","reported_seq":1482,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829301+0000","last_change":"2024-03-30T15:16:43.923599+0000","last_active":"2024-03-30T15:17:10.829301+0000","last_peered":"2024-03-30T15:17:10.829301+0000","last_clean":"2024-03-30T15:17:10.829301+0000","last_became_active":"2024-03-30T15:13:28.645038+0000","last_became_peered":"2024-03-30T15:13:28.645038+0000","last_unstale":"2024-03-30T15:17:10.829301+0000","last_undegraded":"2024-03-30T15:17:10.829301+0000","last_fullsized":"2024-03-30T15:17:10.829301+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'47","last_scrub_stamp":"2024-03-30T15:16:43.923503+0000","last_deep_scrub":"95'27","last_deep_scrub_stamp":"2024-03-30T14:56:24.457559+0000","last_clean_scrub_stamp":"2024-03-30T15:16:43.923503+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.448641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1890,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821533+0000","last_change":"2024-03-30T15:17:07.238815+0000","last_active":"2024-03-30T15:17:10.821533+0000","last_peered":"2024-03-30T15:17:10.821533+0000","last_clean":"2024-03-30T15:17:10.821533+0000","last_became_active":"2024-03-30T15:13:52.591330+0000","last_became_peered":"2024-03-30T15:13:52.591330+0000","last_unstale":"2024-03-30T15:17:10.821533+0000","last_undegraded":"2024-03-30T15:17:10.821533+0000","last_fullsized":"2024-03-30T15:17:10.821533+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:07.238727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:17:07.238727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:23.384284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1881,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.819666+0000","last_change":"2024-03-30T15:16:53.258159+0000","last_active":"2024-03-30T15:17:10.819666+0000","last_peered":"2024-03-30T15:17:10.819666+0000","last_clean":"2024-03-30T15:17:10.819666+0000","last_became_active":"2024-03-30T15:13:52.588705+0000","last_became_peered":"2024-03-30T15:13:52.588705+0000","last_unstale":"2024-03-30T15:17:10.819666+0000","last_undegraded":"2024-03-30T15:17:10.819666+0000","last_fullsized":"2024-03-30T15:17:10.819666+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:53.258067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:46.390100+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.258067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.904027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"86'128","reported_seq":1150,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820004+0000","last_change":"2024-03-30T15:16:02.236881+0000","last_active":"2024-03-30T15:17:10.820004+0000","last_peered":"2024-03-30T15:17:10.820004+0000","last_clean":"2024-03-30T15:17:10.820004+0000","last_became_active":"2024-03-30T15:13:03.514491+0000","last_became_peered":"2024-03-30T15:13:03.514491+0000","last_unstale":"2024-03-30T15:17:10.820004+0000","last_undegraded":"2024-03-30T15:17:10.820004+0000","last_fullsized":"2024-03-30T15:17:10.820004+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":3,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:02.236788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:02.236788+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.182637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1372,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829662+0000","last_change":"2024-03-30T15:17:02.945340+0000","last_active":"2024-03-30T15:17:10.829662+0000","last_peered":"2024-03-30T15:17:10.829662+0000","last_clean":"2024-03-30T15:17:10.829662+0000","last_became_active":"2024-03-30T15:13:28.642223+0000","last_became_peered":"2024-03-30T15:13:28.642223+0000","last_unstale":"2024-03-30T15:17:10.829662+0000","last_undegraded":"2024-03-30T15:17:10.829662+0000","last_fullsized":"2024-03-30T15:17:10.829662+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.945257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.945257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.541819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1844,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832934+0000","last_change":"2024-03-30T15:17:00.987807+0000","last_active":"2024-03-30T15:17:10.832934+0000","last_peered":"2024-03-30T15:17:10.832934+0000","last_clean":"2024-03-30T15:17:10.832934+0000","last_became_active":"2024-03-30T15:13:28.622981+0000","last_became_peered":"2024-03-30T15:13:28.622981+0000","last_unstale":"2024-03-30T15:17:10.832934+0000","last_undegraded":"2024-03-30T15:17:10.832934+0000","last_fullsized":"2024-03-30T15:17:10.832934+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.987712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:00.987712+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.987712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.697391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"909'36","reported_seq":1485,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829831+0000","last_change":"2024-03-30T15:16:31.987233+0000","last_active":"2024-03-30T15:17:10.829831+0000","last_peered":"2024-03-30T15:17:10.829831+0000","last_clean":"2024-03-30T15:17:10.829831+0000","last_became_active":"2024-03-30T15:13:52.591080+0000","last_became_peered":"2024-03-30T15:13:52.591080+0000","last_unstale":"2024-03-30T15:17:10.829831+0000","last_undegraded":"2024-03-30T15:17:10.829831+0000","last_fullsized":"2024-03-30T15:17:10.829831+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'36","last_scrub_stamp":"2024-03-30T15:16:31.987144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:31.987144+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.954546+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"478'6","reported_seq":3213,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:11.984887+0000","last_change":"2024-03-30T15:16:59.628887+0000","last_active":"2024-03-30T15:17:11.984887+0000","last_peered":"2024-03-30T15:17:11.984887+0000","last_clean":"2024-03-30T15:17:11.984887+0000","last_became_active":"2024-03-30T15:13:28.632404+0000","last_became_peered":"2024-03-30T15:13:28.632404+0000","last_unstale":"2024-03-30T15:17:11.984887+0000","last_undegraded":"2024-03-30T15:17:11.984887+0000","last_fullsized":"2024-03-30T15:17:11.984887+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'6","last_scrub_stamp":"2024-03-30T15:16:59.628774+0000","last_deep_scrub":"478'6","last_deep_scrub_stamp":"2024-03-30T15:04:50.115166+0000","last_clean_scrub_stamp":"2024-03-30T15:16:59.628774+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:23.632450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074409499999999996,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1344,"num_read_kb":1339,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1438,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828969+0000","last_change":"2024-03-30T15:16:58.945156+0000","last_active":"2024-03-30T15:17:10.828969+0000","last_peered":"2024-03-30T15:17:10.828969+0000","last_clean":"2024-03-30T15:17:10.828969+0000","last_became_active":"2024-03-30T15:14:09.726541+0000","last_became_peered":"2024-03-30T15:14:09.726541+0000","last_unstale":"2024-03-30T15:17:10.828969+0000","last_undegraded":"2024-03-30T15:17:10.828969+0000","last_fullsized":"2024-03-30T15:17:10.828969+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.945077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.945077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.682988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1873,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.819476+0000","last_change":"2024-03-30T15:17:10.354073+0000","last_active":"2024-03-30T15:17:10.819476+0000","last_peered":"2024-03-30T15:17:10.819476+0000","last_clean":"2024-03-30T15:17:10.819476+0000","last_became_active":"2024-03-30T15:13:03.514904+0000","last_became_peered":"2024-03-30T15:13:03.514904+0000","last_unstale":"2024-03-30T15:17:10.819476+0000","last_undegraded":"2024-03-30T15:17:10.819476+0000","last_fullsized":"2024-03-30T15:17:10.819476+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:10.353989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:35.208776+0000","last_clean_scrub_stamp":"2024-03-30T15:17:10.353989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:40.174945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1757,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831272+0000","last_change":"2024-03-30T15:17:09.791067+0000","last_active":"2024-03-30T15:17:10.831272+0000","last_peered":"2024-03-30T15:17:10.831272+0000","last_clean":"2024-03-30T15:17:10.831272+0000","last_became_active":"2024-03-30T15:13:19.276949+0000","last_became_peered":"2024-03-30T15:13:19.276949+0000","last_unstale":"2024-03-30T15:17:10.831272+0000","last_undegraded":"2024-03-30T15:17:10.831272+0000","last_fullsized":"2024-03-30T15:17:10.831272+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:09.790965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:52.413467+0000","last_clean_scrub_stamp":"2024-03-30T15:17:09.790965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:11.763450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1853,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831176+0000","last_change":"2024-03-30T15:16:52.596504+0000","last_active":"2024-03-30T15:17:10.831176+0000","last_peered":"2024-03-30T15:17:10.831176+0000","last_clean":"2024-03-30T15:17:10.831176+0000","last_became_active":"2024-03-30T15:13:19.271491+0000","last_became_peered":"2024-03-30T15:13:19.271491+0000","last_unstale":"2024-03-30T15:17:10.831176+0000","last_undegraded":"2024-03-30T15:17:10.831176+0000","last_fullsized":"2024-03-30T15:17:10.831176+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.596404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.596404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.480407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"909'50","reported_seq":2211,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:12.023822+0000","last_change":"2024-03-30T15:17:12.023822+0000","last_active":"2024-03-30T15:17:12.023822+0000","last_peered":"2024-03-30T15:17:12.023822+0000","last_clean":"2024-03-30T15:17:12.023822+0000","last_became_active":"2024-03-30T15:13:03.543824+0000","last_became_peered":"2024-03-30T15:13:03.543824+0000","last_unstale":"2024-03-30T15:17:12.023822+0000","last_undegraded":"2024-03-30T15:17:12.023822+0000","last_fullsized":"2024-03-30T15:17:12.023822+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'50","last_scrub_stamp":"2024-03-30T15:17:12.023681+0000","last_deep_scrub":"905'38","last_deep_scrub_stamp":"2024-03-30T15:14:36.416317+0000","last_clean_scrub_stamp":"2024-03-30T15:17:12.023681+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:17.892104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":183,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821284+0000","last_change":"2024-03-30T15:16:44.809724+0000","last_active":"2024-03-30T15:17:10.821284+0000","last_peered":"2024-03-30T15:17:10.821284+0000","last_clean":"2024-03-30T15:17:10.821284+0000","last_became_active":"2024-03-30T15:15:33.641927+0000","last_became_peered":"2024-03-30T15:15:33.641927+0000","last_unstale":"2024-03-30T15:17:10.821284+0000","last_undegraded":"2024-03-30T15:17:10.821284+0000","last_fullsized":"2024-03-30T15:17:10.821284+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:44.809644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:44.809644+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.809644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.841106+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"86'128","reported_seq":2124,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831855+0000","last_change":"2024-03-30T15:16:20.445759+0000","last_active":"2024-03-30T15:17:10.831855+0000","last_peered":"2024-03-30T15:17:10.831855+0000","last_clean":"2024-03-30T15:17:10.831855+0000","last_became_active":"2024-03-30T15:13:28.632600+0000","last_became_peered":"2024-03-30T15:13:28.632600+0000","last_unstale":"2024-03-30T15:17:10.831855+0000","last_undegraded":"2024-03-30T15:17:10.831855+0000","last_fullsized":"2024-03-30T15:17:10.831855+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:20.445559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:20.445559+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.440558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1840,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832212+0000","last_change":"2024-03-30T15:16:22.877115+0000","last_active":"2024-03-30T15:17:10.832212+0000","last_peered":"2024-03-30T15:17:10.832212+0000","last_clean":"2024-03-30T15:17:10.832212+0000","last_became_active":"2024-03-30T15:13:03.522675+0000","last_became_peered":"2024-03-30T15:13:03.522675+0000","last_unstale":"2024-03-30T15:17:10.832212+0000","last_undegraded":"2024-03-30T15:17:10.832212+0000","last_fullsized":"2024-03-30T15:17:10.832212+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:22.877012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:11:47.840893+0000","last_clean_scrub_stamp":"2024-03-30T15:16:22.877012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.056997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"86'128","reported_seq":2187,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829951+0000","last_change":"2024-03-30T15:16:30.455124+0000","last_active":"2024-03-30T15:17:10.829951+0000","last_peered":"2024-03-30T15:17:10.829951+0000","last_clean":"2024-03-30T15:17:10.829951+0000","last_became_active":"2024-03-30T15:14:12.746444+0000","last_became_peered":"2024-03-30T15:14:12.746444+0000","last_unstale":"2024-03-30T15:17:10.829951+0000","last_undegraded":"2024-03-30T15:17:10.829951+0000","last_fullsized":"2024-03-30T15:17:10.829951+0000","mapping_epoch":883,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":884,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:30.455010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.455010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:46.323598+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1779,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832214+0000","last_change":"2024-03-30T15:16:27.849055+0000","last_active":"2024-03-30T15:17:10.832214+0000","last_peered":"2024-03-30T15:17:10.832214+0000","last_clean":"2024-03-30T15:17:10.832214+0000","last_became_active":"2024-03-30T15:13:52.589696+0000","last_became_peered":"2024-03-30T15:13:52.589696+0000","last_unstale":"2024-03-30T15:17:10.832214+0000","last_undegraded":"2024-03-30T15:17:10.832214+0000","last_fullsized":"2024-03-30T15:17:10.832214+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:27.848932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:58.461031+0000","last_clean_scrub_stamp":"2024-03-30T15:16:27.848932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:28.704658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":207,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821365+0000","last_change":"2024-03-30T15:15:59.123222+0000","last_active":"2024-03-30T15:17:10.821365+0000","last_peered":"2024-03-30T15:17:10.821365+0000","last_clean":"2024-03-30T15:17:10.821365+0000","last_became_active":"2024-03-30T15:13:52.594698+0000","last_became_peered":"2024-03-30T15:13:52.594698+0000","last_unstale":"2024-03-30T15:17:10.821365+0000","last_undegraded":"2024-03-30T15:17:10.821365+0000","last_fullsized":"2024-03-30T15:17:10.821365+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:59.123136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:52.804238+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.123136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:26.949114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"909'26","reported_seq":1911,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822658+0000","last_change":"2024-03-30T15:16:23.607161+0000","last_active":"2024-03-30T15:17:10.822658+0000","last_peered":"2024-03-30T15:17:10.822658+0000","last_clean":"2024-03-30T15:17:10.822658+0000","last_became_active":"2024-03-30T15:13:28.631530+0000","last_became_peered":"2024-03-30T15:13:28.631530+0000","last_unstale":"2024-03-30T15:17:10.822658+0000","last_undegraded":"2024-03-30T15:17:10.822658+0000","last_fullsized":"2024-03-30T15:17:10.822658+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'26","last_scrub_stamp":"2024-03-30T15:16:23.607038+0000","last_deep_scrub":"106'18","last_deep_scrub_stamp":"2024-03-30T14:57:26.184813+0000","last_clean_scrub_stamp":"2024-03-30T15:16:23.607038+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:36.725064+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1718,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820817+0000","last_change":"2024-03-30T15:17:00.896285+0000","last_active":"2024-03-30T15:17:10.820817+0000","last_peered":"2024-03-30T15:17:10.820817+0000","last_clean":"2024-03-30T15:17:10.820817+0000","last_became_active":"2024-03-30T15:13:28.614801+0000","last_became_peered":"2024-03-30T15:13:28.614801+0000","last_unstale":"2024-03-30T15:17:10.820817+0000","last_undegraded":"2024-03-30T15:17:10.820817+0000","last_fullsized":"2024-03-30T15:17:10.820817+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.896184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:00.896184+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.896184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.207822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059980899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1880,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820805+0000","last_change":"2024-03-30T15:16:42.437894+0000","last_active":"2024-03-30T15:17:10.820805+0000","last_peered":"2024-03-30T15:17:10.820805+0000","last_clean":"2024-03-30T15:17:10.820805+0000","last_became_active":"2024-03-30T15:13:19.290979+0000","last_became_peered":"2024-03-30T15:13:19.290979+0000","last_unstale":"2024-03-30T15:17:10.820805+0000","last_undegraded":"2024-03-30T15:17:10.820805+0000","last_fullsized":"2024-03-30T15:17:10.820805+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:42.437798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:04:53.983055+0000","last_clean_scrub_stamp":"2024-03-30T15:16:42.437798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:55.186767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1856,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831610+0000","last_change":"2024-03-30T15:17:06.757297+0000","last_active":"2024-03-30T15:17:10.831610+0000","last_peered":"2024-03-30T15:17:10.831610+0000","last_clean":"2024-03-30T15:17:10.831610+0000","last_became_active":"2024-03-30T14:54:43.359726+0000","last_became_peered":"2024-03-30T14:54:43.359726+0000","last_unstale":"2024-03-30T15:17:10.831610+0000","last_undegraded":"2024-03-30T15:17:10.831610+0000","last_fullsized":"2024-03-30T15:17:10.831610+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:06.757173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:17:06.757173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:09.141961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"86'128","reported_seq":1137,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830134+0000","last_change":"2024-03-30T15:16:54.928083+0000","last_active":"2024-03-30T15:17:10.830134+0000","last_peered":"2024-03-30T15:17:10.830134+0000","last_clean":"2024-03-30T15:17:10.830134+0000","last_became_active":"2024-03-30T15:13:28.623160+0000","last_became_peered":"2024-03-30T15:13:28.623160+0000","last_unstale":"2024-03-30T15:17:10.830134+0000","last_undegraded":"2024-03-30T15:17:10.830134+0000","last_fullsized":"2024-03-30T15:17:10.830134+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":559,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":4,"last_scrub":"86'128","last_scrub_stamp":"2024-03-30T15:16:54.927931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:52:18.365875+0000","last_clean_scrub_stamp":"2024-03-30T15:16:54.927931+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:11.285212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1370,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829229+0000","last_change":"2024-03-30T15:16:03.117244+0000","last_active":"2024-03-30T15:17:10.829229+0000","last_peered":"2024-03-30T15:17:10.829229+0000","last_clean":"2024-03-30T15:17:10.829229+0000","last_became_active":"2024-03-30T15:13:28.639814+0000","last_became_peered":"2024-03-30T15:13:28.639814+0000","last_unstale":"2024-03-30T15:17:10.829229+0000","last_undegraded":"2024-03-30T15:17:10.829229+0000","last_fullsized":"2024-03-30T15:17:10.829229+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:03.117142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.117142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.279353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1482,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829748+0000","last_change":"2024-03-30T15:16:39.956634+0000","last_active":"2024-03-30T15:17:10.829748+0000","last_peered":"2024-03-30T15:17:10.829748+0000","last_clean":"2024-03-30T15:17:10.829748+0000","last_became_active":"2024-03-30T15:14:09.726470+0000","last_became_peered":"2024-03-30T15:14:09.726470+0000","last_unstale":"2024-03-30T15:17:10.829748+0000","last_undegraded":"2024-03-30T15:17:10.829748+0000","last_fullsized":"2024-03-30T15:17:10.829748+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:39.956558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:39.956558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:41.349718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"954'65","reported_seq":2171,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829758+0000","last_change":"2024-03-30T15:16:53.939980+0000","last_active":"2024-03-30T15:17:10.829758+0000","last_peered":"2024-03-30T15:17:10.829758+0000","last_clean":"2024-03-30T15:17:10.829758+0000","last_became_active":"2024-03-30T15:13:19.293582+0000","last_became_peered":"2024-03-30T15:13:19.293582+0000","last_unstale":"2024-03-30T15:17:10.829758+0000","last_undegraded":"2024-03-30T15:17:10.829758+0000","last_fullsized":"2024-03-30T15:17:10.829758+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'65","last_scrub_stamp":"2024-03-30T15:16:53.939834+0000","last_deep_scrub":"954'65","last_deep_scrub_stamp":"2024-03-30T15:16:53.939834+0000","last_clean_scrub_stamp":"2024-03-30T15:16:53.939834+0000","objects_scrubbed":10,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:05.253773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1784,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833011+0000","last_change":"2024-03-30T15:16:24.540552+0000","last_active":"2024-03-30T15:17:10.833011+0000","last_peered":"2024-03-30T15:17:10.833011+0000","last_clean":"2024-03-30T15:17:10.833011+0000","last_became_active":"2024-03-30T15:13:28.624999+0000","last_became_peered":"2024-03-30T15:13:28.624999+0000","last_unstale":"2024-03-30T15:17:10.833011+0000","last_undegraded":"2024-03-30T15:17:10.833011+0000","last_fullsized":"2024-03-30T15:17:10.833011+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:24.540431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:07:31.968649+0000","last_clean_scrub_stamp":"2024-03-30T15:16:24.540431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:39.899577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"72'2","reported_seq":1854,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830662+0000","last_change":"2024-03-30T15:16:30.707679+0000","last_active":"2024-03-30T15:17:10.830662+0000","last_peered":"2024-03-30T15:17:10.830662+0000","last_clean":"2024-03-30T15:17:10.830662+0000","last_became_active":"2024-03-30T14:54:36.317957+0000","last_became_peered":"2024-03-30T14:54:36.317957+0000","last_unstale":"2024-03-30T15:17:10.830662+0000","last_undegraded":"2024-03-30T15:17:10.830662+0000","last_fullsized":"2024-03-30T15:17:10.830662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2024-03-30T15:16:30.707574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:35.298668+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.707574+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:35.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001207638,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1507,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823508+0000","last_change":"2024-03-30T15:15:53.452852+0000","last_active":"2024-03-30T15:17:10.823508+0000","last_peered":"2024-03-30T15:17:10.823508+0000","last_clean":"2024-03-30T15:17:10.823508+0000","last_became_active":"2024-03-30T15:15:33.640955+0000","last_became_peered":"2024-03-30T15:15:33.640955+0000","last_unstale":"2024-03-30T15:17:10.823508+0000","last_undegraded":"2024-03-30T15:17:10.823508+0000","last_fullsized":"2024-03-30T15:17:10.823508+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:53.452677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:53.452677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1374,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829467+0000","last_change":"2024-03-30T15:16:50.961821+0000","last_active":"2024-03-30T15:17:10.829467+0000","last_peered":"2024-03-30T15:17:10.829467+0000","last_clean":"2024-03-30T15:17:10.829467+0000","last_became_active":"2024-03-30T15:14:00.632424+0000","last_became_peered":"2024-03-30T15:14:00.632424+0000","last_unstale":"2024-03-30T15:17:10.829467+0000","last_undegraded":"2024-03-30T15:17:10.829467+0000","last_fullsized":"2024-03-30T15:17:10.829467+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:50.961744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:50.961744+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.961744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.713880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"947'25","reported_seq":1489,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829503+0000","last_change":"2024-03-30T15:16:28.015564+0000","last_active":"2024-03-30T15:17:10.829503+0000","last_peered":"2024-03-30T15:17:10.829503+0000","last_clean":"2024-03-30T15:17:10.829503+0000","last_became_active":"2024-03-30T15:15:20.081455+0000","last_became_peered":"2024-03-30T15:15:20.081455+0000","last_unstale":"2024-03-30T15:17:10.829503+0000","last_undegraded":"2024-03-30T15:17:10.829503+0000","last_fullsized":"2024-03-30T15:17:10.829503+0000","mapping_epoch":922,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":923,"parent":"0.0","parent_split_bits":0,"last_scrub":"947'25","last_scrub_stamp":"2024-03-30T15:16:28.015469+0000","last_deep_scrub":"947'25","last_deep_scrub_stamp":"2024-03-30T15:16:28.015469+0000","last_clean_scrub_stamp":"2024-03-30T15:16:28.015469+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.014319+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1851,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833498+0000","last_change":"2024-03-30T15:16:47.849349+0000","last_active":"2024-03-30T15:17:10.833498+0000","last_peered":"2024-03-30T15:17:10.833498+0000","last_clean":"2024-03-30T15:17:10.833498+0000","last_became_active":"2024-03-30T14:54:37.325140+0000","last_became_peered":"2024-03-30T14:54:37.325140+0000","last_unstale":"2024-03-30T15:17:10.833498+0000","last_undegraded":"2024-03-30T15:17:10.833498+0000","last_fullsized":"2024-03-30T15:17:10.833498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:47.849253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:12:59.144310+0000","last_clean_scrub_stamp":"2024-03-30T15:16:47.849253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.026665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1726,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820033+0000","last_change":"2024-03-30T15:16:33.041540+0000","last_active":"2024-03-30T15:17:10.820033+0000","last_peered":"2024-03-30T15:17:10.820033+0000","last_clean":"2024-03-30T15:17:10.820033+0000","last_became_active":"2024-03-30T15:13:19.284117+0000","last_became_peered":"2024-03-30T15:13:19.284117+0000","last_unstale":"2024-03-30T15:17:10.820033+0000","last_undegraded":"2024-03-30T15:17:10.820033+0000","last_fullsized":"2024-03-30T15:17:10.820033+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:33.041458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:33.041458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.183468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1405,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829081+0000","last_change":"2024-03-30T15:15:59.194418+0000","last_active":"2024-03-30T15:17:10.829081+0000","last_peered":"2024-03-30T15:17:10.829081+0000","last_clean":"2024-03-30T15:17:10.829081+0000","last_became_active":"2024-03-30T15:13:52.588594+0000","last_became_peered":"2024-03-30T15:13:52.588594+0000","last_unstale":"2024-03-30T15:17:10.829081+0000","last_undegraded":"2024-03-30T15:17:10.829081+0000","last_fullsized":"2024-03-30T15:17:10.829081+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:59.194339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.194339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1723,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833930+0000","last_change":"2024-03-30T15:17:04.884143+0000","last_active":"2024-03-30T15:17:10.833930+0000","last_peered":"2024-03-30T15:17:10.833930+0000","last_clean":"2024-03-30T15:17:10.833930+0000","last_became_active":"2024-03-30T15:14:00.634711+0000","last_became_peered":"2024-03-30T15:14:00.634711+0000","last_unstale":"2024-03-30T15:17:10.833930+0000","last_undegraded":"2024-03-30T15:17:10.833930+0000","last_fullsized":"2024-03-30T15:17:10.833930+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:04.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:28.641088+0000","last_clean_scrub_stamp":"2024-03-30T15:17:04.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.956394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"909'51","reported_seq":1994,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830900+0000","last_change":"2024-03-30T15:16:40.821388+0000","last_active":"2024-03-30T15:17:10.830900+0000","last_peered":"2024-03-30T15:17:10.830900+0000","last_clean":"2024-03-30T15:17:10.830900+0000","last_became_active":"2024-03-30T15:15:06.901815+0000","last_became_peered":"2024-03-30T15:15:06.901815+0000","last_unstale":"2024-03-30T15:17:10.830900+0000","last_undegraded":"2024-03-30T15:17:10.830900+0000","last_fullsized":"2024-03-30T15:17:10.830900+0000","mapping_epoch":915,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":916,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'51","last_scrub_stamp":"2024-03-30T15:16:40.821264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:40.821264+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.625137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1852,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831440+0000","last_change":"2024-03-30T15:16:19.416619+0000","last_active":"2024-03-30T15:17:10.831440+0000","last_peered":"2024-03-30T15:17:10.831440+0000","last_clean":"2024-03-30T15:17:10.831440+0000","last_became_active":"2024-03-30T14:54:37.328777+0000","last_became_peered":"2024-03-30T14:54:37.328777+0000","last_unstale":"2024-03-30T15:17:10.831440+0000","last_undegraded":"2024-03-30T15:17:10.831440+0000","last_fullsized":"2024-03-30T15:17:10.831440+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.416483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:19.416483+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.416483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.017287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":215,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821503+0000","last_change":"2024-03-30T15:16:58.985136+0000","last_active":"2024-03-30T15:17:10.821503+0000","last_peered":"2024-03-30T15:17:10.821503+0000","last_clean":"2024-03-30T15:17:10.821503+0000","last_became_active":"2024-03-30T15:13:52.587447+0000","last_became_peered":"2024-03-30T15:13:52.587447+0000","last_unstale":"2024-03-30T15:17:10.821503+0000","last_undegraded":"2024-03-30T15:17:10.821503+0000","last_fullsized":"2024-03-30T15:17:10.821503+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.985041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.985041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:12.649229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1849,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830383+0000","last_change":"2024-03-30T15:17:03.752962+0000","last_active":"2024-03-30T15:17:10.830383+0000","last_peered":"2024-03-30T15:17:10.830383+0000","last_clean":"2024-03-30T15:17:10.830383+0000","last_became_active":"2024-03-30T15:13:19.271586+0000","last_became_peered":"2024-03-30T15:13:19.271586+0000","last_unstale":"2024-03-30T15:17:10.830383+0000","last_undegraded":"2024-03-30T15:17:10.830383+0000","last_fullsized":"2024-03-30T15:17:10.830383+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:03.752852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:17:03.752852+0000","last_clean_scrub_stamp":"2024-03-30T15:17:03.752852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:24.902351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"114'7","reported_seq":1769,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829161+0000","last_change":"2024-03-30T15:16:35.887068+0000","last_active":"2024-03-30T15:17:10.829161+0000","last_peered":"2024-03-30T15:17:10.829161+0000","last_clean":"2024-03-30T15:17:10.829161+0000","last_became_active":"2024-03-30T14:57:55.109573+0000","last_became_peered":"2024-03-30T14:57:55.109573+0000","last_unstale":"2024-03-30T15:17:10.829161+0000","last_undegraded":"2024-03-30T15:17:10.829161+0000","last_fullsized":"2024-03-30T15:17:10.829161+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-03-30T15:16:35.886913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.886913+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.524801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1693,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.819362+0000","last_change":"2024-03-30T15:16:30.991737+0000","last_active":"2024-03-30T15:17:10.819362+0000","last_peered":"2024-03-30T15:17:10.819362+0000","last_clean":"2024-03-30T15:17:10.819362+0000","last_became_active":"2024-03-30T15:13:28.646999+0000","last_became_peered":"2024-03-30T15:13:28.646999+0000","last_unstale":"2024-03-30T15:17:10.819362+0000","last_undegraded":"2024-03-30T15:17:10.819362+0000","last_fullsized":"2024-03-30T15:17:10.819362+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.991652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:06.652343+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.991652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.637334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"909'24","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832986+0000","last_change":"2024-03-30T15:15:49.544265+0000","last_active":"2024-03-30T15:17:10.832986+0000","last_peered":"2024-03-30T15:17:10.832986+0000","last_clean":"2024-03-30T15:17:10.832986+0000","last_became_active":"2024-03-30T15:14:00.632108+0000","last_became_peered":"2024-03-30T15:14:00.632108+0000","last_unstale":"2024-03-30T15:17:10.832986+0000","last_undegraded":"2024-03-30T15:17:10.832986+0000","last_fullsized":"2024-03-30T15:17:10.832986+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'24","last_scrub_stamp":"2024-03-30T15:15:49.544151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:49.544151+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.475104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1853,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831083+0000","last_change":"2024-03-30T15:15:45.282142+0000","last_active":"2024-03-30T15:17:10.831083+0000","last_peered":"2024-03-30T15:17:10.831083+0000","last_clean":"2024-03-30T15:17:10.831083+0000","last_became_active":"2024-03-30T15:13:03.550868+0000","last_became_peered":"2024-03-30T15:13:03.550868+0000","last_unstale":"2024-03-30T15:17:10.831083+0000","last_undegraded":"2024-03-30T15:17:10.831083+0000","last_fullsized":"2024-03-30T15:17:10.831083+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:45.281907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:45.281907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.843114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":171,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822995+0000","last_change":"2024-03-30T15:16:26.586028+0000","last_active":"2024-03-30T15:17:10.822995+0000","last_peered":"2024-03-30T15:17:10.822995+0000","last_clean":"2024-03-30T15:17:10.822995+0000","last_became_active":"2024-03-30T15:15:33.637445+0000","last_became_peered":"2024-03-30T15:15:33.637445+0000","last_unstale":"2024-03-30T15:17:10.822995+0000","last_undegraded":"2024-03-30T15:17:10.822995+0000","last_fullsized":"2024-03-30T15:17:10.822995+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:26.585926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:02.270821+0000","last_clean_scrub_stamp":"2024-03-30T15:16:26.585926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:32.659121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":188,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829718+0000","last_change":"2024-03-30T15:17:09.804465+0000","last_active":"2024-03-30T15:17:10.829718+0000","last_peered":"2024-03-30T15:17:10.829718+0000","last_clean":"2024-03-30T15:17:10.829718+0000","last_became_active":"2024-03-30T15:14:09.725078+0000","last_became_peered":"2024-03-30T15:14:09.725078+0000","last_unstale":"2024-03-30T15:17:10.829718+0000","last_undegraded":"2024-03-30T15:17:10.829718+0000","last_fullsized":"2024-03-30T15:17:10.829718+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:09.804360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:09.804360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:11.997500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"913'11","reported_seq":2128,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830068+0000","last_change":"2024-03-30T15:16:22.453685+0000","last_active":"2024-03-30T15:17:10.830068+0000","last_peered":"2024-03-30T15:17:10.830068+0000","last_clean":"2024-03-30T15:17:10.830068+0000","last_became_active":"2024-03-30T15:13:03.549786+0000","last_became_peered":"2024-03-30T15:13:03.549786+0000","last_unstale":"2024-03-30T15:17:10.830068+0000","last_undegraded":"2024-03-30T15:17:10.830068+0000","last_fullsized":"2024-03-30T15:17:10.830068+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'11","last_scrub_stamp":"2024-03-30T15:16:22.453583+0000","last_deep_scrub":"913'11","last_deep_scrub_stamp":"2024-03-30T15:15:03.185977+0000","last_clean_scrub_stamp":"2024-03-30T15:16:22.453583+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.111159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"909'39","reported_seq":1900,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829642+0000","last_change":"2024-03-30T15:16:03.751833+0000","last_active":"2024-03-30T15:17:10.829642+0000","last_peered":"2024-03-30T15:17:10.829642+0000","last_clean":"2024-03-30T15:17:10.829642+0000","last_became_active":"2024-03-30T14:54:39.643304+0000","last_became_peered":"2024-03-30T14:54:39.643304+0000","last_unstale":"2024-03-30T15:17:10.829642+0000","last_undegraded":"2024-03-30T15:17:10.829642+0000","last_fullsized":"2024-03-30T15:17:10.829642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'39","last_scrub_stamp":"2024-03-30T15:16:03.751714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:03.751714+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:26.662581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"69'2","reported_seq":1870,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831898+0000","last_change":"2024-03-30T15:16:44.554751+0000","last_active":"2024-03-30T15:17:10.831898+0000","last_peered":"2024-03-30T15:17:10.831898+0000","last_clean":"2024-03-30T15:17:10.831898+0000","last_became_active":"2024-03-30T15:13:28.632685+0000","last_became_peered":"2024-03-30T15:13:28.632685+0000","last_unstale":"2024-03-30T15:17:10.831898+0000","last_undegraded":"2024-03-30T15:17:10.831898+0000","last_fullsized":"2024-03-30T15:17:10.831898+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-30T15:16:44.554646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.554646+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:10.334583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1840,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831103+0000","last_change":"2024-03-30T15:16:55.962099+0000","last_active":"2024-03-30T15:17:10.831103+0000","last_peered":"2024-03-30T15:17:10.831103+0000","last_clean":"2024-03-30T15:17:10.831103+0000","last_became_active":"2024-03-30T15:13:28.625385+0000","last_became_peered":"2024-03-30T15:13:28.625385+0000","last_unstale":"2024-03-30T15:17:10.831103+0000","last_undegraded":"2024-03-30T15:17:10.831103+0000","last_fullsized":"2024-03-30T15:17:10.831103+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:55.961998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:55.961998+0000","last_clean_scrub_stamp":"2024-03-30T15:16:55.961998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.499264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1772,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832116+0000","last_change":"2024-03-30T15:15:51.797990+0000","last_active":"2024-03-30T15:17:10.832116+0000","last_peered":"2024-03-30T15:17:10.832116+0000","last_clean":"2024-03-30T15:17:10.832116+0000","last_became_active":"2024-03-30T15:13:52.592711+0000","last_became_peered":"2024-03-30T15:13:52.592711+0000","last_unstale":"2024-03-30T15:17:10.832116+0000","last_undegraded":"2024-03-30T15:17:10.832116+0000","last_fullsized":"2024-03-30T15:17:10.832116+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:51.797863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:51.797863+0000","last_clean_scrub_stamp":"2024-03-30T15:15:51.797863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.163129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"909'18","reported_seq":217,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823642+0000","last_change":"2024-03-30T15:17:08.012176+0000","last_active":"2024-03-30T15:17:10.823642+0000","last_peered":"2024-03-30T15:17:10.823642+0000","last_clean":"2024-03-30T15:17:10.823642+0000","last_became_active":"2024-03-30T15:14:00.631854+0000","last_became_peered":"2024-03-30T15:14:00.631854+0000","last_unstale":"2024-03-30T15:17:10.823642+0000","last_undegraded":"2024-03-30T15:17:10.823642+0000","last_fullsized":"2024-03-30T15:17:10.823642+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'18","last_scrub_stamp":"2024-03-30T15:17:08.011989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:17:08.011989+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:14.699233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829894+0000","last_change":"2024-03-30T15:16:48.918691+0000","last_active":"2024-03-30T15:17:10.829894+0000","last_peered":"2024-03-30T15:17:10.829894+0000","last_clean":"2024-03-30T15:17:10.829894+0000","last_became_active":"2024-03-30T15:13:28.623365+0000","last_became_peered":"2024-03-30T15:13:28.623365+0000","last_unstale":"2024-03-30T15:17:10.829894+0000","last_undegraded":"2024-03-30T15:17:10.829894+0000","last_fullsized":"2024-03-30T15:17:10.829894+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:48.918557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:48.918557+0000","last_clean_scrub_stamp":"2024-03-30T15:16:48.918557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:51.691429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1834,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831346+0000","last_change":"2024-03-30T15:16:12.764014+0000","last_active":"2024-03-30T15:17:10.831346+0000","last_peered":"2024-03-30T15:17:10.831346+0000","last_clean":"2024-03-30T15:17:10.831346+0000","last_became_active":"2024-03-30T14:54:41.352449+0000","last_became_peered":"2024-03-30T14:54:41.352449+0000","last_unstale":"2024-03-30T15:17:10.831346+0000","last_undegraded":"2024-03-30T15:17:10.831346+0000","last_fullsized":"2024-03-30T15:17:10.831346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.763892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:12.763892+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.763892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:14.161884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1840,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830965+0000","last_change":"2024-03-30T15:16:51.644001+0000","last_active":"2024-03-30T15:17:10.830965+0000","last_peered":"2024-03-30T15:17:10.830965+0000","last_clean":"2024-03-30T15:17:10.830965+0000","last_became_active":"2024-03-30T15:13:03.537074+0000","last_became_peered":"2024-03-30T15:13:03.537074+0000","last_unstale":"2024-03-30T15:17:10.830965+0000","last_undegraded":"2024-03-30T15:17:10.830965+0000","last_fullsized":"2024-03-30T15:17:10.830965+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:51.643878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:09:42.843021+0000","last_clean_scrub_stamp":"2024-03-30T15:16:51.643878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.652355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1725,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833617+0000","last_change":"2024-03-30T15:16:44.766708+0000","last_active":"2024-03-30T15:17:10.833617+0000","last_peered":"2024-03-30T15:17:10.833617+0000","last_clean":"2024-03-30T15:17:10.833617+0000","last_became_active":"2024-03-30T15:13:03.529999+0000","last_became_peered":"2024-03-30T15:13:03.529999+0000","last_unstale":"2024-03-30T15:17:10.833617+0000","last_undegraded":"2024-03-30T15:17:10.833617+0000","last_fullsized":"2024-03-30T15:17:10.833617+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:44.766607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:44.766607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:49.304648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"909'21","reported_seq":1468,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830278+0000","last_change":"2024-03-30T15:16:18.042445+0000","last_active":"2024-03-30T15:17:10.830278+0000","last_peered":"2024-03-30T15:17:10.830278+0000","last_clean":"2024-03-30T15:17:10.830278+0000","last_became_active":"2024-03-30T15:13:28.642687+0000","last_became_peered":"2024-03-30T15:13:28.642687+0000","last_unstale":"2024-03-30T15:17:10.830278+0000","last_undegraded":"2024-03-30T15:17:10.830278+0000","last_fullsized":"2024-03-30T15:17:10.830278+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:18.042315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:18.042315+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:22.375270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1860,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829783+0000","last_change":"2024-03-30T15:16:55.649016+0000","last_active":"2024-03-30T15:17:10.829783+0000","last_peered":"2024-03-30T15:17:10.829783+0000","last_clean":"2024-03-30T15:17:10.829783+0000","last_became_active":"2024-03-30T15:13:03.548886+0000","last_became_peered":"2024-03-30T15:13:03.548886+0000","last_unstale":"2024-03-30T15:17:10.829783+0000","last_undegraded":"2024-03-30T15:17:10.829783+0000","last_fullsized":"2024-03-30T15:17:10.829783+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:55.648905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:55.648905+0000","last_clean_scrub_stamp":"2024-03-30T15:16:55.648905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.805796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":937,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:11.017291+0000","last_change":"2024-03-30T15:17:11.017317+0000","last_active":"2024-03-30T15:17:11.017291+0000","last_peered":"2024-03-30T15:17:11.017291+0000","last_clean":"2024-03-30T15:17:11.017291+0000","last_became_active":"2024-03-30T15:05:42.322629+0000","last_became_peered":"2024-03-30T15:05:42.322629+0000","last_unstale":"2024-03-30T15:17:11.017291+0000","last_undegraded":"2024-03-30T15:17:11.017291+0000","last_fullsized":"2024-03-30T15:17:11.017291+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:11.017208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:17:11.017208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:28.247742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1696,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821928+0000","last_change":"2024-03-30T15:16:12.024253+0000","last_active":"2024-03-30T15:17:10.821928+0000","last_peered":"2024-03-30T15:17:10.821928+0000","last_clean":"2024-03-30T15:17:10.821928+0000","last_became_active":"2024-03-30T15:13:19.279299+0000","last_became_peered":"2024-03-30T15:13:19.279299+0000","last_unstale":"2024-03-30T15:17:10.821928+0000","last_undegraded":"2024-03-30T15:17:10.821928+0000","last_fullsized":"2024-03-30T15:17:10.821928+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.024110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.024110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:13.583496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1357,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829675+0000","last_change":"2024-03-30T15:15:41.246466+0000","last_active":"2024-03-30T15:17:10.829675+0000","last_peered":"2024-03-30T15:17:10.829675+0000","last_clean":"2024-03-30T15:17:10.829675+0000","last_became_active":"2024-03-30T15:13:52.589809+0000","last_became_peered":"2024-03-30T15:13:52.589809+0000","last_unstale":"2024-03-30T15:17:10.829675+0000","last_undegraded":"2024-03-30T15:17:10.829675+0000","last_fullsized":"2024-03-30T15:17:10.829675+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:41.246373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:41.246373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"909'21","reported_seq":1872,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831096+0000","last_change":"2024-03-30T15:16:18.845698+0000","last_active":"2024-03-30T15:17:10.831096+0000","last_peered":"2024-03-30T15:17:10.831096+0000","last_clean":"2024-03-30T15:17:10.831096+0000","last_became_active":"2024-03-30T14:54:39.642328+0000","last_became_peered":"2024-03-30T14:54:39.642328+0000","last_unstale":"2024-03-30T15:17:10.831096+0000","last_undegraded":"2024-03-30T15:17:10.831096+0000","last_fullsized":"2024-03-30T15:17:10.831096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:18.845576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:18.845576+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:23.137639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829102+0000","last_change":"2024-03-30T15:16:35.057933+0000","last_active":"2024-03-30T15:17:10.829102+0000","last_peered":"2024-03-30T15:17:10.829102+0000","last_clean":"2024-03-30T15:17:10.829102+0000","last_became_active":"2024-03-30T15:14:00.634218+0000","last_became_peered":"2024-03-30T15:14:00.634218+0000","last_unstale":"2024-03-30T15:17:10.829102+0000","last_undegraded":"2024-03-30T15:17:10.829102+0000","last_fullsized":"2024-03-30T15:17:10.829102+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:35.057856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:35.057856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.068511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"930'17","reported_seq":2023,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821992+0000","last_change":"2024-03-30T15:16:34.639572+0000","last_active":"2024-03-30T15:17:10.821992+0000","last_peered":"2024-03-30T15:17:10.821992+0000","last_clean":"2024-03-30T15:17:10.821992+0000","last_became_active":"2024-03-30T15:15:33.635657+0000","last_became_peered":"2024-03-30T15:15:33.635657+0000","last_unstale":"2024-03-30T15:17:10.821992+0000","last_undegraded":"2024-03-30T15:17:10.821992+0000","last_fullsized":"2024-03-30T15:17:10.821992+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'17","last_scrub_stamp":"2024-03-30T15:16:34.639485+0000","last_deep_scrub":"917'16","last_deep_scrub_stamp":"2024-03-30T15:15:08.284872+0000","last_clean_scrub_stamp":"2024-03-30T15:16:34.639485+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.137297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1745,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832571+0000","last_change":"2024-03-30T15:16:52.813306+0000","last_active":"2024-03-30T15:17:10.832571+0000","last_peered":"2024-03-30T15:17:10.832571+0000","last_clean":"2024-03-30T15:17:10.832571+0000","last_became_active":"2024-03-30T15:13:19.280923+0000","last_became_peered":"2024-03-30T15:13:19.280923+0000","last_unstale":"2024-03-30T15:17:10.832571+0000","last_undegraded":"2024-03-30T15:17:10.832571+0000","last_fullsized":"2024-03-30T15:17:10.832571+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.813177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.813177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:53.163111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1366,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829399+0000","last_change":"2024-03-30T15:15:54.236287+0000","last_active":"2024-03-30T15:17:10.829399+0000","last_peered":"2024-03-30T15:17:10.829399+0000","last_clean":"2024-03-30T15:17:10.829399+0000","last_became_active":"2024-03-30T15:13:28.637092+0000","last_became_peered":"2024-03-30T15:13:28.637092+0000","last_unstale":"2024-03-30T15:17:10.829399+0000","last_undegraded":"2024-03-30T15:17:10.829399+0000","last_fullsized":"2024-03-30T15:17:10.829399+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:54.236135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:54.236135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"909'35","reported_seq":1798,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822309+0000","last_change":"2024-03-30T15:15:51.463270+0000","last_active":"2024-03-30T15:17:10.822309+0000","last_peered":"2024-03-30T15:17:10.822309+0000","last_clean":"2024-03-30T15:17:10.822309+0000","last_became_active":"2024-03-30T15:15:33.636512+0000","last_became_peered":"2024-03-30T15:15:33.636512+0000","last_unstale":"2024-03-30T15:17:10.822309+0000","last_undegraded":"2024-03-30T15:17:10.822309+0000","last_fullsized":"2024-03-30T15:17:10.822309+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:15:51.463172+0000","last_deep_scrub":"903'25","last_deep_scrub_stamp":"2024-03-30T15:14:32.245633+0000","last_clean_scrub_stamp":"2024-03-30T15:15:51.463172+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1857,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830320+0000","last_change":"2024-03-30T15:16:00.450389+0000","last_active":"2024-03-30T15:17:10.830320+0000","last_peered":"2024-03-30T15:17:10.830320+0000","last_clean":"2024-03-30T15:17:10.830320+0000","last_became_active":"2024-03-30T15:13:19.277421+0000","last_became_peered":"2024-03-30T15:13:19.277421+0000","last_unstale":"2024-03-30T15:17:10.830320+0000","last_undegraded":"2024-03-30T15:17:10.830320+0000","last_fullsized":"2024-03-30T15:17:10.830320+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:00.450281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:00.450281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:29.621262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"852'12","reported_seq":2130,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830740+0000","last_change":"2024-03-30T15:16:51.915466+0000","last_active":"2024-03-30T15:17:10.830740+0000","last_peered":"2024-03-30T15:17:10.830740+0000","last_clean":"2024-03-30T15:17:10.830740+0000","last_became_active":"2024-03-30T15:13:29.615517+0000","last_became_peered":"2024-03-30T15:13:29.615517+0000","last_unstale":"2024-03-30T15:17:10.830740+0000","last_undegraded":"2024-03-30T15:17:10.830740+0000","last_fullsized":"2024-03-30T15:17:10.830740+0000","mapping_epoch":851,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":852,"parent":"0.0","parent_split_bits":0,"last_scrub":"852'12","last_scrub_stamp":"2024-03-30T15:16:51.915353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:51.915353+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.827580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"96'2","reported_seq":1427,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829629+0000","last_change":"2024-03-30T15:16:56.913496+0000","last_active":"2024-03-30T15:17:10.829629+0000","last_peered":"2024-03-30T15:17:10.829629+0000","last_clean":"2024-03-30T15:17:10.829629+0000","last_became_active":"2024-03-30T15:14:00.638394+0000","last_became_peered":"2024-03-30T15:14:00.638394+0000","last_unstale":"2024-03-30T15:17:10.829629+0000","last_undegraded":"2024-03-30T15:17:10.829629+0000","last_fullsized":"2024-03-30T15:17:10.829629+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-30T15:16:56.913412+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-30T14:56:35.331150+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.913412+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.718985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1712,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832698+0000","last_change":"2024-03-30T15:16:56.785419+0000","last_active":"2024-03-30T15:17:10.832698+0000","last_peered":"2024-03-30T15:17:10.832698+0000","last_clean":"2024-03-30T15:17:10.832698+0000","last_became_active":"2024-03-30T15:13:03.509862+0000","last_became_peered":"2024-03-30T15:13:03.509862+0000","last_unstale":"2024-03-30T15:17:10.832698+0000","last_undegraded":"2024-03-30T15:17:10.832698+0000","last_fullsized":"2024-03-30T15:17:10.832698+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.785316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.785316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:00.566594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"909'32","reported_seq":1842,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832050+0000","last_change":"2024-03-30T15:16:06.372815+0000","last_active":"2024-03-30T15:17:10.832050+0000","last_peered":"2024-03-30T15:17:10.832050+0000","last_clean":"2024-03-30T15:17:10.832050+0000","last_became_active":"2024-03-30T15:13:03.536364+0000","last_became_peered":"2024-03-30T15:13:03.536364+0000","last_unstale":"2024-03-30T15:17:10.832050+0000","last_undegraded":"2024-03-30T15:17:10.832050+0000","last_fullsized":"2024-03-30T15:17:10.832050+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:16:06.372644+0000","last_deep_scrub":"909'32","last_deep_scrub_stamp":"2024-03-30T15:16:06.372644+0000","last_clean_scrub_stamp":"2024-03-30T15:16:06.372644+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:24.579620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1792,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833268+0000","last_change":"2024-03-30T15:16:25.572404+0000","last_active":"2024-03-30T15:17:10.833268+0000","last_peered":"2024-03-30T15:17:10.833268+0000","last_clean":"2024-03-30T15:17:10.833268+0000","last_became_active":"2024-03-30T15:13:03.539915+0000","last_became_peered":"2024-03-30T15:13:03.539915+0000","last_unstale":"2024-03-30T15:17:10.833268+0000","last_undegraded":"2024-03-30T15:17:10.833268+0000","last_fullsized":"2024-03-30T15:17:10.833268+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:25.572269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:25.572269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:41.788269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1873,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820926+0000","last_change":"2024-03-30T15:16:12.440873+0000","last_active":"2024-03-30T15:17:10.820926+0000","last_peered":"2024-03-30T15:17:10.820926+0000","last_clean":"2024-03-30T15:17:10.820926+0000","last_became_active":"2024-03-30T15:14:00.629833+0000","last_became_peered":"2024-03-30T15:14:00.629833+0000","last_unstale":"2024-03-30T15:17:10.820926+0000","last_undegraded":"2024-03-30T15:17:10.820926+0000","last_fullsized":"2024-03-30T15:17:10.820926+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:12.440755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:53.189233+0000","last_clean_scrub_stamp":"2024-03-30T15:16:12.440755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:42.196677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1219,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820239+0000","last_change":"2024-03-30T15:16:31.616407+0000","last_active":"2024-03-30T15:17:10.820239+0000","last_peered":"2024-03-30T15:17:10.820239+0000","last_clean":"2024-03-30T15:17:10.820239+0000","last_became_active":"2024-03-30T15:15:33.639037+0000","last_became_peered":"2024-03-30T15:15:33.639037+0000","last_unstale":"2024-03-30T15:17:10.820239+0000","last_undegraded":"2024-03-30T15:17:10.820239+0000","last_fullsized":"2024-03-30T15:17:10.820239+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:31.616316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:11.553667+0000","last_clean_scrub_stamp":"2024-03-30T15:16:31.616316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:59.497808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":188,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831989+0000","last_change":"2024-03-30T15:17:02.856511+0000","last_active":"2024-03-30T15:17:10.831989+0000","last_peered":"2024-03-30T15:17:10.831989+0000","last_clean":"2024-03-30T15:17:10.831989+0000","last_became_active":"2024-03-30T15:14:09.726417+0000","last_became_peered":"2024-03-30T15:14:09.726417+0000","last_unstale":"2024-03-30T15:17:10.831989+0000","last_undegraded":"2024-03-30T15:17:10.831989+0000","last_fullsized":"2024-03-30T15:17:10.831989+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:02.856410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:17:02.856410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.551174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"909'21","reported_seq":1900,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821125+0000","last_change":"2024-03-30T15:16:57.234494+0000","last_active":"2024-03-30T15:17:10.821125+0000","last_peered":"2024-03-30T15:17:10.821125+0000","last_clean":"2024-03-30T15:17:10.821125+0000","last_became_active":"2024-03-30T15:13:19.293272+0000","last_became_peered":"2024-03-30T15:13:19.293272+0000","last_unstale":"2024-03-30T15:17:10.821125+0000","last_undegraded":"2024-03-30T15:17:10.821125+0000","last_fullsized":"2024-03-30T15:17:10.821125+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:57.234407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:57.234407+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.354154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"93'2","reported_seq":1875,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831079+0000","last_change":"2024-03-30T15:17:01.695994+0000","last_active":"2024-03-30T15:17:10.831079+0000","last_peered":"2024-03-30T15:17:10.831079+0000","last_clean":"2024-03-30T15:17:10.831079+0000","last_became_active":"2024-03-30T15:13:03.525235+0000","last_became_peered":"2024-03-30T15:13:03.525235+0000","last_unstale":"2024-03-30T15:17:10.831079+0000","last_undegraded":"2024-03-30T15:17:10.831079+0000","last_fullsized":"2024-03-30T15:17:10.831079+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'2","last_scrub_stamp":"2024-03-30T15:17:01.695877+0000","last_deep_scrub":"93'2","last_deep_scrub_stamp":"2024-03-30T15:07:33.297820+0000","last_clean_scrub_stamp":"2024-03-30T15:17:01.695877+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:07.698187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":830,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:11.115929+0000","last_change":"2024-03-30T15:17:11.115956+0000","last_active":"2024-03-30T15:17:11.115929+0000","last_peered":"2024-03-30T15:17:11.115929+0000","last_clean":"2024-03-30T15:17:11.115929+0000","last_became_active":"2024-03-30T15:15:33.637711+0000","last_became_peered":"2024-03-30T15:15:33.637711+0000","last_unstale":"2024-03-30T15:17:11.115929+0000","last_undegraded":"2024-03-30T15:17:11.115929+0000","last_fullsized":"2024-03-30T15:17:11.115929+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:11.115855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:17:11.115855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:31.049898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1438,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828744+0000","last_change":"2024-03-30T15:16:09.084905+0000","last_active":"2024-03-30T15:17:10.828744+0000","last_peered":"2024-03-30T15:17:10.828744+0000","last_clean":"2024-03-30T15:17:10.828744+0000","last_became_active":"2024-03-30T15:13:28.634012+0000","last_became_peered":"2024-03-30T15:13:28.634012+0000","last_unstale":"2024-03-30T15:17:10.828744+0000","last_undegraded":"2024-03-30T15:17:10.828744+0000","last_fullsized":"2024-03-30T15:17:10.828744+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.084819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.084819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.458424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1328,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828867+0000","last_change":"2024-03-30T15:15:39.275512+0000","last_active":"2024-03-30T15:17:10.828867+0000","last_peered":"2024-03-30T15:17:10.828867+0000","last_clean":"2024-03-30T15:17:10.828867+0000","last_became_active":"2024-03-30T15:13:28.626496+0000","last_became_peered":"2024-03-30T15:13:28.626496+0000","last_unstale":"2024-03-30T15:17:10.828867+0000","last_undegraded":"2024-03-30T15:17:10.828867+0000","last_fullsized":"2024-03-30T15:17:10.828867+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:39.275423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:46.540109+0000","last_clean_scrub_stamp":"2024-03-30T15:15:39.275423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"909'27","reported_seq":1765,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820698+0000","last_change":"2024-03-30T15:16:15.509415+0000","last_active":"2024-03-30T15:17:10.820698+0000","last_peered":"2024-03-30T15:17:10.820698+0000","last_clean":"2024-03-30T15:17:10.820698+0000","last_became_active":"2024-03-30T15:15:33.639967+0000","last_became_peered":"2024-03-30T15:15:33.639967+0000","last_unstale":"2024-03-30T15:17:10.820698+0000","last_undegraded":"2024-03-30T15:17:10.820698+0000","last_fullsized":"2024-03-30T15:17:10.820698+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'27","last_scrub_stamp":"2024-03-30T15:16:15.509294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.509294+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:25.707634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1842,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831477+0000","last_change":"2024-03-30T15:15:58.876389+0000","last_active":"2024-03-30T15:17:10.831477+0000","last_peered":"2024-03-30T15:17:10.831477+0000","last_clean":"2024-03-30T15:17:10.831477+0000","last_became_active":"2024-03-30T14:54:37.330351+0000","last_became_peered":"2024-03-30T14:54:37.330351+0000","last_unstale":"2024-03-30T15:17:10.831477+0000","last_undegraded":"2024-03-30T15:17:10.831477+0000","last_fullsized":"2024-03-30T15:17:10.831477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:58.876286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:58.876286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.136820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1852,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832111+0000","last_change":"2024-03-30T15:16:48.526416+0000","last_active":"2024-03-30T15:17:10.832111+0000","last_peered":"2024-03-30T15:17:10.832111+0000","last_clean":"2024-03-30T15:17:10.832111+0000","last_became_active":"2024-03-30T15:13:03.540174+0000","last_became_peered":"2024-03-30T15:13:03.540174+0000","last_unstale":"2024-03-30T15:17:10.832111+0000","last_undegraded":"2024-03-30T15:17:10.832111+0000","last_fullsized":"2024-03-30T15:17:10.832111+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:48.526316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:10:19.989982+0000","last_clean_scrub_stamp":"2024-03-30T15:16:48.526316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.444275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1183,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823105+0000","last_change":"2024-03-30T15:17:09.041537+0000","last_active":"2024-03-30T15:17:10.823105+0000","last_peered":"2024-03-30T15:17:10.823105+0000","last_clean":"2024-03-30T15:17:10.823105+0000","last_became_active":"2024-03-30T15:15:33.640822+0000","last_became_peered":"2024-03-30T15:15:33.640822+0000","last_unstale":"2024-03-30T15:17:10.823105+0000","last_undegraded":"2024-03-30T15:17:10.823105+0000","last_fullsized":"2024-03-30T15:17:10.823105+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:09.041423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:17:09.041423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:18.175739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1776,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:11.341446+0000","last_change":"2024-03-30T15:17:11.341446+0000","last_active":"2024-03-30T15:17:11.341446+0000","last_peered":"2024-03-30T15:17:11.341446+0000","last_clean":"2024-03-30T15:17:11.341446+0000","last_became_active":"2024-03-30T15:13:03.527402+0000","last_became_peered":"2024-03-30T15:13:03.527402+0000","last_unstale":"2024-03-30T15:17:11.341446+0000","last_undegraded":"2024-03-30T15:17:11.341446+0000","last_fullsized":"2024-03-30T15:17:11.341446+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:11.341329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:39.160952+0000","last_clean_scrub_stamp":"2024-03-30T15:17:11.341329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:27.493776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"909'12","reported_seq":1455,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820668+0000","last_change":"2024-03-30T15:15:59.439735+0000","last_active":"2024-03-30T15:17:10.820668+0000","last_peered":"2024-03-30T15:17:10.820668+0000","last_clean":"2024-03-30T15:17:10.820668+0000","last_became_active":"2024-03-30T15:01:40.313386+0000","last_became_peered":"2024-03-30T15:01:40.313386+0000","last_unstale":"2024-03-30T15:17:10.820668+0000","last_undegraded":"2024-03-30T15:17:10.820668+0000","last_fullsized":"2024-03-30T15:17:10.820668+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'12","last_scrub_stamp":"2024-03-30T15:15:59.439586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:15:59.439586+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.626484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1434,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830041+0000","last_change":"2024-03-30T15:16:19.986348+0000","last_active":"2024-03-30T15:17:10.830041+0000","last_peered":"2024-03-30T15:17:10.830041+0000","last_clean":"2024-03-30T15:17:10.830041+0000","last_became_active":"2024-03-30T15:13:28.626493+0000","last_became_peered":"2024-03-30T15:13:28.626493+0000","last_unstale":"2024-03-30T15:17:10.830041+0000","last_undegraded":"2024-03-30T15:17:10.830041+0000","last_fullsized":"2024-03-30T15:17:10.830041+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.986167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:00:01.470075+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.986167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:42.816121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1882,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829721+0000","last_change":"2024-03-30T15:16:56.944613+0000","last_active":"2024-03-30T15:17:10.829721+0000","last_peered":"2024-03-30T15:17:10.829721+0000","last_clean":"2024-03-30T15:17:10.829721+0000","last_became_active":"2024-03-30T15:14:09.727046+0000","last_became_peered":"2024-03-30T15:14:09.727046+0000","last_unstale":"2024-03-30T15:17:10.829721+0000","last_undegraded":"2024-03-30T15:17:10.829721+0000","last_fullsized":"2024-03-30T15:17:10.829721+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.944523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.944523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.648879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1845,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830378+0000","last_change":"2024-03-30T15:16:58.649422+0000","last_active":"2024-03-30T15:17:10.830378+0000","last_peered":"2024-03-30T15:17:10.830378+0000","last_clean":"2024-03-30T15:17:10.830378+0000","last_became_active":"2024-03-30T15:13:19.277971+0000","last_became_peered":"2024-03-30T15:13:19.277971+0000","last_unstale":"2024-03-30T15:17:10.830378+0000","last_undegraded":"2024-03-30T15:17:10.830378+0000","last_fullsized":"2024-03-30T15:17:10.830378+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:58.649324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:27.932491+0000","last_clean_scrub_stamp":"2024-03-30T15:16:58.649324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.680061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1756,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832458+0000","last_change":"2024-03-30T15:15:53.835754+0000","last_active":"2024-03-30T15:17:10.832458+0000","last_peered":"2024-03-30T15:17:10.832458+0000","last_clean":"2024-03-30T15:17:10.832458+0000","last_became_active":"2024-03-30T15:13:19.281726+0000","last_became_peered":"2024-03-30T15:13:19.281726+0000","last_unstale":"2024-03-30T15:17:10.832458+0000","last_undegraded":"2024-03-30T15:17:10.832458+0000","last_fullsized":"2024-03-30T15:17:10.832458+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:53.835649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:25.522636+0000","last_clean_scrub_stamp":"2024-03-30T15:15:53.835649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:22.015341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"909'32","reported_seq":1902,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:13.021931+0000","last_change":"2024-03-30T15:17:13.021986+0000","last_active":"2024-03-30T15:17:13.021931+0000","last_peered":"2024-03-30T15:17:13.021931+0000","last_clean":"2024-03-30T15:17:13.021931+0000","last_became_active":"2024-03-30T14:54:39.338516+0000","last_became_peered":"2024-03-30T14:54:39.338516+0000","last_unstale":"2024-03-30T15:17:13.021931+0000","last_undegraded":"2024-03-30T15:17:13.021931+0000","last_fullsized":"2024-03-30T15:17:13.021931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'32","last_scrub_stamp":"2024-03-30T15:17:13.021850+0000","last_deep_scrub":"909'32","last_deep_scrub_stamp":"2024-03-30T15:17:13.021850+0000","last_clean_scrub_stamp":"2024-03-30T15:17:13.021850+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:30.469335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1147,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820573+0000","last_change":"2024-03-30T15:16:30.565832+0000","last_active":"2024-03-30T15:17:10.820573+0000","last_peered":"2024-03-30T15:17:10.820573+0000","last_clean":"2024-03-30T15:17:10.820573+0000","last_became_active":"2024-03-30T15:15:33.636878+0000","last_became_peered":"2024-03-30T15:15:33.636878+0000","last_unstale":"2024-03-30T15:17:10.820573+0000","last_undegraded":"2024-03-30T15:17:10.820573+0000","last_fullsized":"2024-03-30T15:17:10.820573+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.565747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.565747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:33.777062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1422,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830204+0000","last_change":"2024-03-30T15:16:05.120670+0000","last_active":"2024-03-30T15:17:10.830204+0000","last_peered":"2024-03-30T15:17:10.830204+0000","last_clean":"2024-03-30T15:17:10.830204+0000","last_became_active":"2024-03-30T15:13:52.594341+0000","last_became_peered":"2024-03-30T15:13:52.594341+0000","last_unstale":"2024-03-30T15:17:10.830204+0000","last_undegraded":"2024-03-30T15:17:10.830204+0000","last_fullsized":"2024-03-30T15:17:10.830204+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:05.120583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:05.120583+0000","last_clean_scrub_stamp":"2024-03-30T15:16:05.120583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.767478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1866,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822393+0000","last_change":"2024-03-30T15:17:06.195093+0000","last_active":"2024-03-30T15:17:10.822393+0000","last_peered":"2024-03-30T15:17:10.822393+0000","last_clean":"2024-03-30T15:17:10.822393+0000","last_became_active":"2024-03-30T15:13:03.517348+0000","last_became_peered":"2024-03-30T15:13:03.517348+0000","last_unstale":"2024-03-30T15:17:10.822393+0000","last_undegraded":"2024-03-30T15:17:10.822393+0000","last_fullsized":"2024-03-30T15:17:10.822393+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:06.195011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:36.256518+0000","last_clean_scrub_stamp":"2024-03-30T15:17:06.195011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.476090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1783,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831925+0000","last_change":"2024-03-30T15:16:32.845860+0000","last_active":"2024-03-30T15:17:10.831925+0000","last_peered":"2024-03-30T15:17:10.831925+0000","last_clean":"2024-03-30T15:17:10.831925+0000","last_became_active":"2024-03-30T15:14:09.727015+0000","last_became_peered":"2024-03-30T15:14:09.727015+0000","last_unstale":"2024-03-30T15:17:10.831925+0000","last_undegraded":"2024-03-30T15:17:10.831925+0000","last_fullsized":"2024-03-30T15:17:10.831925+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:32.845790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:06:14.085299+0000","last_clean_scrub_stamp":"2024-03-30T15:16:32.845790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:58.352493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"909'35","reported_seq":1483,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829950+0000","last_change":"2024-03-30T15:16:46.902473+0000","last_active":"2024-03-30T15:17:10.829950+0000","last_peered":"2024-03-30T15:17:10.829950+0000","last_clean":"2024-03-30T15:17:10.829950+0000","last_became_active":"2024-03-30T15:14:00.638539+0000","last_became_peered":"2024-03-30T15:14:00.638539+0000","last_unstale":"2024-03-30T15:17:10.829950+0000","last_undegraded":"2024-03-30T15:17:10.829950+0000","last_fullsized":"2024-03-30T15:17:10.829950+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'35","last_scrub_stamp":"2024-03-30T15:16:46.902346+0000","last_deep_scrub":"186'15","last_deep_scrub_stamp":"2024-03-30T14:59:10.626365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.902346+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.280878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"600'2","reported_seq":1891,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:13.286787+0000","last_change":"2024-03-30T15:17:13.286816+0000","last_active":"2024-03-30T15:17:13.286787+0000","last_peered":"2024-03-30T15:17:13.286787+0000","last_clean":"2024-03-30T15:17:13.286787+0000","last_became_active":"2024-03-30T15:14:09.726865+0000","last_became_peered":"2024-03-30T15:14:09.726865+0000","last_unstale":"2024-03-30T15:17:13.286787+0000","last_undegraded":"2024-03-30T15:17:13.286787+0000","last_fullsized":"2024-03-30T15:17:13.286787+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'2","last_scrub_stamp":"2024-03-30T15:17:13.286721+0000","last_deep_scrub":"600'2","last_deep_scrub_stamp":"2024-03-30T15:07:30.031154+0000","last_clean_scrub_stamp":"2024-03-30T15:17:13.286721+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:41.062923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"883'15","reported_seq":2171,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832397+0000","last_change":"2024-03-30T15:16:17.902741+0000","last_active":"2024-03-30T15:17:10.832397+0000","last_peered":"2024-03-30T15:17:10.832397+0000","last_clean":"2024-03-30T15:17:10.832397+0000","last_became_active":"2024-03-30T15:14:09.727307+0000","last_became_peered":"2024-03-30T15:14:09.727307+0000","last_unstale":"2024-03-30T15:17:10.832397+0000","last_undegraded":"2024-03-30T15:17:10.832397+0000","last_fullsized":"2024-03-30T15:17:10.832397+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"883'15","last_scrub_stamp":"2024-03-30T15:16:17.902638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:17.902638+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:27.729614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"107'2","reported_seq":1850,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.831112+0000","last_change":"2024-03-30T15:16:45.605024+0000","last_active":"2024-03-30T15:17:10.831112+0000","last_peered":"2024-03-30T15:17:10.831112+0000","last_clean":"2024-03-30T15:17:10.831112+0000","last_became_active":"2024-03-30T15:13:19.271296+0000","last_became_peered":"2024-03-30T15:13:19.271296+0000","last_unstale":"2024-03-30T15:17:10.831112+0000","last_undegraded":"2024-03-30T15:17:10.831112+0000","last_fullsized":"2024-03-30T15:17:10.831112+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'2","last_scrub_stamp":"2024-03-30T15:16:45.604906+0000","last_deep_scrub":"107'2","last_deep_scrub_stamp":"2024-03-30T15:15:29.346296+0000","last_clean_scrub_stamp":"2024-03-30T15:16:45.604906+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.947882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1778,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832108+0000","last_change":"2024-03-30T15:16:52.953093+0000","last_active":"2024-03-30T15:17:10.832108+0000","last_peered":"2024-03-30T15:17:10.832108+0000","last_clean":"2024-03-30T15:17:10.832108+0000","last_became_active":"2024-03-30T15:13:52.588042+0000","last_became_peered":"2024-03-30T15:13:52.588042+0000","last_unstale":"2024-03-30T15:17:10.832108+0000","last_undegraded":"2024-03-30T15:17:10.832108+0000","last_fullsized":"2024-03-30T15:17:10.832108+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:52.953000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:52.953000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:18.161461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"909'24","reported_seq":1797,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822987+0000","last_change":"2024-03-30T15:16:54.880822+0000","last_active":"2024-03-30T15:17:10.822987+0000","last_peered":"2024-03-30T15:17:10.822987+0000","last_clean":"2024-03-30T15:17:10.822987+0000","last_became_active":"2024-03-30T15:15:33.635097+0000","last_became_peered":"2024-03-30T15:15:33.635097+0000","last_unstale":"2024-03-30T15:17:10.822987+0000","last_undegraded":"2024-03-30T15:17:10.822987+0000","last_fullsized":"2024-03-30T15:17:10.822987+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'24","last_scrub_stamp":"2024-03-30T15:16:54.880726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:54.880726+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:15.891309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1757,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833870+0000","last_change":"2024-03-30T15:17:01.847312+0000","last_active":"2024-03-30T15:17:10.833870+0000","last_peered":"2024-03-30T15:17:10.833870+0000","last_clean":"2024-03-30T15:17:10.833870+0000","last_became_active":"2024-03-30T15:13:19.285258+0000","last_became_peered":"2024-03-30T15:13:19.285258+0000","last_unstale":"2024-03-30T15:17:10.833870+0000","last_undegraded":"2024-03-30T15:17:10.833870+0000","last_fullsized":"2024-03-30T15:17:10.833870+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:01.847214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:52.361880+0000","last_clean_scrub_stamp":"2024-03-30T15:17:01.847214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:27.898144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1749,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832378+0000","last_change":"2024-03-30T15:16:32.650660+0000","last_active":"2024-03-30T15:17:10.832378+0000","last_peered":"2024-03-30T15:17:10.832378+0000","last_clean":"2024-03-30T15:17:10.832378+0000","last_became_active":"2024-03-30T15:13:19.291186+0000","last_became_peered":"2024-03-30T15:13:19.291186+0000","last_unstale":"2024-03-30T15:17:10.832378+0000","last_undegraded":"2024-03-30T15:17:10.832378+0000","last_fullsized":"2024-03-30T15:17:10.832378+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:32.650568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:16:32.650568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.328400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":171,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823724+0000","last_change":"2024-03-30T15:16:38.726895+0000","last_active":"2024-03-30T15:17:10.823724+0000","last_peered":"2024-03-30T15:17:10.823724+0000","last_clean":"2024-03-30T15:17:10.823724+0000","last_became_active":"2024-03-30T15:15:33.640737+0000","last_became_peered":"2024-03-30T15:15:33.640737+0000","last_unstale":"2024-03-30T15:17:10.823724+0000","last_undegraded":"2024-03-30T15:17:10.823724+0000","last_fullsized":"2024-03-30T15:17:10.823724+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:38.726705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:16:38.726705+0000","last_clean_scrub_stamp":"2024-03-30T15:16:38.726705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:54.512080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1760,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833712+0000","last_change":"2024-03-30T15:16:40.775179+0000","last_active":"2024-03-30T15:17:10.833712+0000","last_peered":"2024-03-30T15:17:10.833712+0000","last_clean":"2024-03-30T15:17:10.833712+0000","last_became_active":"2024-03-30T15:14:00.638153+0000","last_became_peered":"2024-03-30T15:14:00.638153+0000","last_unstale":"2024-03-30T15:17:10.833712+0000","last_undegraded":"2024-03-30T15:17:10.833712+0000","last_fullsized":"2024-03-30T15:17:10.833712+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:40.775084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:15:17.708825+0000","last_clean_scrub_stamp":"2024-03-30T15:16:40.775084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:01.308345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"909'46","reported_seq":241,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.821756+0000","last_change":"2024-03-30T15:16:33.995126+0000","last_active":"2024-03-30T15:17:10.821756+0000","last_peered":"2024-03-30T15:17:10.821756+0000","last_clean":"2024-03-30T15:17:10.821756+0000","last_became_active":"2024-03-30T15:13:52.587538+0000","last_became_peered":"2024-03-30T15:13:52.587538+0000","last_unstale":"2024-03-30T15:17:10.821756+0000","last_undegraded":"2024-03-30T15:17:10.821756+0000","last_fullsized":"2024-03-30T15:17:10.821756+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:33.995035+0000","last_deep_scrub":"153'22","last_deep_scrub_stamp":"2024-03-30T14:58:37.879074+0000","last_clean_scrub_stamp":"2024-03-30T15:16:33.995035+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:57.472462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1423,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829321+0000","last_change":"2024-03-30T15:15:44.205503+0000","last_active":"2024-03-30T15:17:10.829321+0000","last_peered":"2024-03-30T15:17:10.829321+0000","last_clean":"2024-03-30T15:17:10.829321+0000","last_became_active":"2024-03-30T15:13:28.643964+0000","last_became_peered":"2024-03-30T15:13:28.643964+0000","last_unstale":"2024-03-30T15:17:10.829321+0000","last_undegraded":"2024-03-30T15:17:10.829321+0000","last_fullsized":"2024-03-30T15:17:10.829321+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:44.205405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:15:44.205405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1889,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.822254+0000","last_change":"2024-03-30T15:16:56.249850+0000","last_active":"2024-03-30T15:17:10.822254+0000","last_peered":"2024-03-30T15:17:10.822254+0000","last_clean":"2024-03-30T15:17:10.822254+0000","last_became_active":"2024-03-30T15:14:09.724748+0000","last_became_peered":"2024-03-30T15:14:09.724748+0000","last_unstale":"2024-03-30T15:17:10.822254+0000","last_undegraded":"2024-03-30T15:17:10.822254+0000","last_fullsized":"2024-03-30T15:17:10.822254+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:56.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:56:33.193180+0000","last_clean_scrub_stamp":"2024-03-30T15:16:56.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:03.651647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1772,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.832504+0000","last_change":"2024-03-30T15:16:07.404860+0000","last_active":"2024-03-30T15:17:10.832504+0000","last_peered":"2024-03-30T15:17:10.832504+0000","last_clean":"2024-03-30T15:17:10.832504+0000","last_became_active":"2024-03-30T15:13:03.536453+0000","last_became_peered":"2024-03-30T15:13:03.536453+0000","last_unstale":"2024-03-30T15:17:10.832504+0000","last_undegraded":"2024-03-30T15:17:10.832504+0000","last_fullsized":"2024-03-30T15:17:10.832504+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:07.404772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:16:07.404772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:20.453216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1768,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820122+0000","last_change":"2024-03-30T15:17:00.206475+0000","last_active":"2024-03-30T15:17:10.820122+0000","last_peered":"2024-03-30T15:17:10.820122+0000","last_clean":"2024-03-30T15:17:10.820122+0000","last_became_active":"2024-03-30T15:13:28.632549+0000","last_became_peered":"2024-03-30T15:13:28.632549+0000","last_unstale":"2024-03-30T15:17:10.820122+0000","last_undegraded":"2024-03-30T15:17:10.820122+0000","last_fullsized":"2024-03-30T15:17:10.820122+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.206355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:08:21.207705+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.206355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:08.088059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"909'9","reported_seq":1827,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829438+0000","last_change":"2024-03-30T15:16:16.486714+0000","last_active":"2024-03-30T15:17:10.829438+0000","last_peered":"2024-03-30T15:17:10.829438+0000","last_clean":"2024-03-30T15:17:10.829438+0000","last_became_active":"2024-03-30T15:13:03.509107+0000","last_became_peered":"2024-03-30T15:13:03.509107+0000","last_unstale":"2024-03-30T15:17:10.829438+0000","last_undegraded":"2024-03-30T15:17:10.829438+0000","last_fullsized":"2024-03-30T15:17:10.829438+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'9","last_scrub_stamp":"2024-03-30T15:16:16.486610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:16.486610+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:45.092622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1851,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833368+0000","last_change":"2024-03-30T15:16:15.770653+0000","last_active":"2024-03-30T15:17:10.833368+0000","last_peered":"2024-03-30T15:17:10.833368+0000","last_clean":"2024-03-30T15:17:10.833368+0000","last_became_active":"2024-03-30T15:13:03.518001+0000","last_became_peered":"2024-03-30T15:13:03.518001+0000","last_unstale":"2024-03-30T15:17:10.833368+0000","last_undegraded":"2024-03-30T15:17:10.833368+0000","last_fullsized":"2024-03-30T15:17:10.833368+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:15.770517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:15.770517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:30.386306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1844,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830267+0000","last_change":"2024-03-30T15:16:09.553470+0000","last_active":"2024-03-30T15:17:10.830267+0000","last_peered":"2024-03-30T15:17:10.830267+0000","last_clean":"2024-03-30T15:17:10.830267+0000","last_became_active":"2024-03-30T15:13:03.526657+0000","last_became_peered":"2024-03-30T15:13:03.526657+0000","last_unstale":"2024-03-30T15:17:10.830267+0000","last_undegraded":"2024-03-30T15:17:10.830267+0000","last_fullsized":"2024-03-30T15:17:10.830267+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:09.553371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:52.210556+0000","last_clean_scrub_stamp":"2024-03-30T15:16:09.553371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:16.555329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1854,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830511+0000","last_change":"2024-03-30T15:16:19.885058+0000","last_active":"2024-03-30T15:17:10.830511+0000","last_peered":"2024-03-30T15:17:10.830511+0000","last_clean":"2024-03-30T15:17:10.830511+0000","last_became_active":"2024-03-30T15:14:00.638063+0000","last_became_peered":"2024-03-30T15:14:00.638063+0000","last_unstale":"2024-03-30T15:17:10.830511+0000","last_undegraded":"2024-03-30T15:17:10.830511+0000","last_fullsized":"2024-03-30T15:17:10.830511+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:19.884926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:59:57.093513+0000","last_clean_scrub_stamp":"2024-03-30T15:16:19.884926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.230878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":87,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:12.092328+0000","last_change":"2024-03-30T15:17:12.092349+0000","last_active":"2024-03-30T15:17:12.092328+0000","last_peered":"2024-03-30T15:17:12.092328+0000","last_clean":"2024-03-30T15:17:12.092328+0000","last_became_active":"2024-03-30T15:15:33.641239+0000","last_became_peered":"2024-03-30T15:15:33.641239+0000","last_unstale":"2024-03-30T15:17:12.092328+0000","last_undegraded":"2024-03-30T15:17:12.092328+0000","last_fullsized":"2024-03-30T15:17:12.092328+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:12.092273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:26.138647+0000","last_clean_scrub_stamp":"2024-03-30T15:17:12.092273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:17.667367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"909'21","reported_seq":199,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833295+0000","last_change":"2024-03-30T15:16:46.769066+0000","last_active":"2024-03-30T15:17:10.833295+0000","last_peered":"2024-03-30T15:17:10.833295+0000","last_clean":"2024-03-30T15:17:10.833295+0000","last_became_active":"2024-03-30T15:14:09.725322+0000","last_became_peered":"2024-03-30T15:14:09.725322+0000","last_unstale":"2024-03-30T15:17:10.833295+0000","last_undegraded":"2024-03-30T15:17:10.833295+0000","last_fullsized":"2024-03-30T15:17:10.833295+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'21","last_scrub_stamp":"2024-03-30T15:16:46.768964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:38.312097+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.768964+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:10.905169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1851,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830581+0000","last_change":"2024-03-30T15:16:34.754386+0000","last_active":"2024-03-30T15:17:10.830581+0000","last_peered":"2024-03-30T15:17:10.830581+0000","last_clean":"2024-03-30T15:17:10.830581+0000","last_became_active":"2024-03-30T15:13:03.532804+0000","last_became_peered":"2024-03-30T15:13:03.532804+0000","last_unstale":"2024-03-30T15:17:10.830581+0000","last_undegraded":"2024-03-30T15:17:10.830581+0000","last_fullsized":"2024-03-30T15:17:10.830581+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:34.754272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:51.737365+0000","last_clean_scrub_stamp":"2024-03-30T15:16:34.754272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:47.288934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1835,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829827+0000","last_change":"2024-03-30T15:15:55.946031+0000","last_active":"2024-03-30T15:17:10.829827+0000","last_peered":"2024-03-30T15:17:10.829827+0000","last_clean":"2024-03-30T15:17:10.829827+0000","last_became_active":"2024-03-30T15:13:28.624284+0000","last_became_peered":"2024-03-30T15:13:28.624284+0000","last_unstale":"2024-03-30T15:17:10.829827+0000","last_undegraded":"2024-03-30T15:17:10.829827+0000","last_fullsized":"2024-03-30T15:17:10.829827+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:55.945915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:39.898372+0000","last_clean_scrub_stamp":"2024-03-30T15:15:55.945915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.940104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1414,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829235+0000","last_change":"2024-03-30T15:15:35.211426+0000","last_active":"2024-03-30T15:17:10.829235+0000","last_peered":"2024-03-30T15:17:10.829235+0000","last_clean":"2024-03-30T15:17:10.829235+0000","last_became_active":"2024-03-30T15:13:28.631113+0000","last_became_peered":"2024-03-30T15:13:28.631113+0000","last_unstale":"2024-03-30T15:17:10.829235+0000","last_undegraded":"2024-03-30T15:17:10.829235+0000","last_fullsized":"2024-03-30T15:17:10.829235+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:35.211330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:13:40.224204+0000","last_clean_scrub_stamp":"2024-03-30T15:15:35.211330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1773,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.830111+0000","last_change":"2024-03-30T15:16:46.869868+0000","last_active":"2024-03-30T15:17:10.830111+0000","last_peered":"2024-03-30T15:17:10.830111+0000","last_clean":"2024-03-30T15:17:10.830111+0000","last_became_active":"2024-03-30T15:14:09.726973+0000","last_became_peered":"2024-03-30T15:14:09.726973+0000","last_unstale":"2024-03-30T15:17:10.830111+0000","last_undegraded":"2024-03-30T15:17:10.830111+0000","last_fullsized":"2024-03-30T15:17:10.830111+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:46.869774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:16:46.869774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:04.813815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"909'46","reported_seq":1681,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820875+0000","last_change":"2024-03-30T15:16:29.599273+0000","last_active":"2024-03-30T15:17:10.820875+0000","last_peered":"2024-03-30T15:17:10.820875+0000","last_clean":"2024-03-30T15:17:10.820875+0000","last_became_active":"2024-03-30T15:15:33.636090+0000","last_became_peered":"2024-03-30T15:15:33.636090+0000","last_unstale":"2024-03-30T15:17:10.820875+0000","last_undegraded":"2024-03-30T15:17:10.820875+0000","last_fullsized":"2024-03-30T15:17:10.820875+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"909'46","last_scrub_stamp":"2024-03-30T15:16:29.599167+0000","last_deep_scrub":"93'22","last_deep_scrub_stamp":"2024-03-30T14:56:05.629449+0000","last_clean_scrub_stamp":"2024-03-30T15:16:29.599167+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:48.802092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1570,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.823369+0000","last_change":"2024-03-30T15:16:49.907781+0000","last_active":"2024-03-30T15:17:10.823369+0000","last_peered":"2024-03-30T15:17:10.823369+0000","last_clean":"2024-03-30T15:17:10.823369+0000","last_became_active":"2024-03-30T15:15:33.639084+0000","last_became_peered":"2024-03-30T15:15:33.639084+0000","last_unstale":"2024-03-30T15:17:10.823369+0000","last_undegraded":"2024-03-30T15:17:10.823369+0000","last_fullsized":"2024-03-30T15:17:10.823369+0000","mapping_epoch":929,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":930,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:49.907687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:27.930731+0000","last_clean_scrub_stamp":"2024-03-30T15:16:49.907687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:02.074815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":180,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829818+0000","last_change":"2024-03-30T15:17:00.823479+0000","last_active":"2024-03-30T15:17:10.829818+0000","last_peered":"2024-03-30T15:17:10.829818+0000","last_clean":"2024-03-30T15:17:10.829818+0000","last_became_active":"2024-03-30T15:14:09.725019+0000","last_became_peered":"2024-03-30T15:14:09.725019+0000","last_unstale":"2024-03-30T15:17:10.829818+0000","last_undegraded":"2024-03-30T15:17:10.829818+0000","last_fullsized":"2024-03-30T15:17:10.829818+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:17:00.823351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:40.325599+0000","last_clean_scrub_stamp":"2024-03-30T15:17:00.823351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:25.759639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":192,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.820239+0000","last_change":"2024-03-30T15:15:55.442821+0000","last_active":"2024-03-30T15:17:10.820239+0000","last_peered":"2024-03-30T15:17:10.820239+0000","last_clean":"2024-03-30T15:17:10.820239+0000","last_became_active":"2024-03-30T15:14:00.631826+0000","last_became_peered":"2024-03-30T15:14:00.631826+0000","last_unstale":"2024-03-30T15:17:10.820239+0000","last_undegraded":"2024-03-30T15:17:10.820239+0000","last_fullsized":"2024-03-30T15:17:10.820239+0000","mapping_epoch":875,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:55.442737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:42.332090+0000","last_clean_scrub_stamp":"2024-03-30T15:15:55.442737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:21.618043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1364,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.828872+0000","last_change":"2024-03-30T15:15:50.268046+0000","last_active":"2024-03-30T15:17:10.828872+0000","last_peered":"2024-03-30T15:17:10.828872+0000","last_clean":"2024-03-30T15:17:10.828872+0000","last_became_active":"2024-03-30T15:13:28.645477+0000","last_became_peered":"2024-03-30T15:13:28.645477+0000","last_unstale":"2024-03-30T15:17:10.828872+0000","last_undegraded":"2024-03-30T15:17:10.828872+0000","last_fullsized":"2024-03-30T15:17:10.828872+0000","mapping_epoch":850,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":851,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:15:50.267958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:57:46.318935+0000","last_clean_scrub_stamp":"2024-03-30T15:15:50.267958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:17.789129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"954'25","reported_seq":1815,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833025+0000","last_change":"2024-03-30T15:16:50.828853+0000","last_active":"2024-03-30T15:17:10.833025+0000","last_peered":"2024-03-30T15:17:10.833025+0000","last_clean":"2024-03-30T15:17:10.833025+0000","last_became_active":"2024-03-30T15:13:19.279835+0000","last_became_peered":"2024-03-30T15:13:19.279835+0000","last_unstale":"2024-03-30T15:17:10.833025+0000","last_undegraded":"2024-03-30T15:17:10.833025+0000","last_fullsized":"2024-03-30T15:17:10.833025+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'25","last_scrub_stamp":"2024-03-30T15:16:50.828730+0000","last_deep_scrub":"954'25","last_deep_scrub_stamp":"2024-03-30T15:16:50.828730+0000","last_clean_scrub_stamp":"2024-03-30T15:16:50.828730+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:18:13.570947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1452,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.829940+0000","last_change":"2024-03-30T15:16:11.161531+0000","last_active":"2024-03-30T15:17:10.829940+0000","last_peered":"2024-03-30T15:17:10.829940+0000","last_clean":"2024-03-30T15:17:10.829940+0000","last_became_active":"2024-03-30T15:13:52.592009+0000","last_became_peered":"2024-03-30T15:13:52.592009+0000","last_unstale":"2024-03-30T15:17:10.829940+0000","last_undegraded":"2024-03-30T15:17:10.829940+0000","last_fullsized":"2024-03-30T15:17:10.829940+0000","mapping_epoch":867,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":868,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:11.161451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T14:54:36.304294+0000","last_clean_scrub_stamp":"2024-03-30T15:16:11.161451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:34.387338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1853,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.819569+0000","last_change":"2024-03-30T15:16:13.467502+0000","last_active":"2024-03-30T15:17:10.819569+0000","last_peered":"2024-03-30T15:17:10.819569+0000","last_clean":"2024-03-30T15:17:10.819569+0000","last_became_active":"2024-03-30T14:54:41.350155+0000","last_became_peered":"2024-03-30T14:54:41.350155+0000","last_unstale":"2024-03-30T15:17:10.819569+0000","last_undegraded":"2024-03-30T15:17:10.819569+0000","last_fullsized":"2024-03-30T15:17:10.819569+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:13.467406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:05:27.054691+0000","last_clean_scrub_stamp":"2024-03-30T15:16:13.467406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:19.471291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1773,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-30T15:17:10.833725+0000","last_change":"2024-03-30T15:16:30.669711+0000","last_active":"2024-03-30T15:17:10.833725+0000","last_peered":"2024-03-30T15:17:10.833725+0000","last_clean":"2024-03-30T15:17:10.833725+0000","last_became_active":"2024-03-30T15:13:03.529100+0000","last_became_peered":"2024-03-30T15:13:03.529100+0000","last_unstale":"2024-03-30T15:17:10.833725+0000","last_undegraded":"2024-03-30T15:17:10.833725+0000","last_fullsized":"2024-03-30T15:17:10.833725+0000","mapping_epoch":832,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":833,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T15:16:30.669525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T15:14:59.705700+0000","last_clean_scrub_stamp":"2024-03-30T15:16:30.669525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T15:17:56.867855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1344,"num_read_kb":1339,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":2228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1076,"ondisk_log_size":1076,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005333,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105792,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639360,"statfs":{"total":95995035648,"available":95886704640,"internally_reserved":0,"allocated":2473984,"data_stored":903432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":52,"seq":223338299674,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105436,"kb_used_data":2064,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639716,"statfs":{"total":95995035648,"available":95887069184,"internally_reserved":0,"allocated":2113536,"data_stored":573414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575157,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105468,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639684,"statfs":{"total":95995035648,"available":95887036416,"internally_reserved":0,"allocated":2150400,"data_stored":574616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":850,"seq":3650722201647,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34872,"kb_used_data":2092,"kb_used_omap":17,"kb_used_meta":32750,"kb_avail":93710280,"statfs":{"total":95995035648,"available":95959326720,"internally_reserved":0,"allocated":2142208,"data_stored":575091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17527,"internal_metadata":33536905},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":840,"seq":3607772528689,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34852,"kb_used_data":2324,"kb_used_omap":46,"kb_used_meta":32465,"kb_avail":93710300,"statfs":{"total":95995035648,"available":95959347200,"internally_reserved":0,"allocated":2379776,"data_stored":836432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47515,"internal_metadata":33244773},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117290,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106060,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639092,"statfs":{"total":95995035648,"available":95886430208,"internally_reserved":0,"allocated":2756608,"data_stored":1165959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":832,"seq":3573412790324,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34744,"kb_used_data":2092,"kb_used_omap":37,"kb_used_meta":32602,"kb_avail":93710408,"statfs":{"total":95995035648,"available":95959457792,"internally_reserved":0,"allocated":2142208,"data_stored":574412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38119,"internal_metadata":33385241},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659423,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106032,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639120,"statfs":{"total":95995035648,"available":95886458880,"internally_reserved":0,"allocated":2723840,"data_stored":1164461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1178,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T15:17:17.579 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-30T15:17:17.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb shell --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-30T15:17:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T15:17:18.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[41614]: from='client.51436 v1:172.21.15.28:0/1291102110' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:18.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[41614]: 3.1b scrub starts 2024-03-30T15:17:18.092 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[41614]: 3.1b scrub ok 2024-03-30T15:17:18.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[29876]: from='client.51436 v1:172.21.15.28:0/1291102110' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:18.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[29876]: 3.1b scrub starts 2024-03-30T15:17:18.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:17 smithi028 ceph-mon[29876]: 3.1b scrub ok 2024-03-30T15:17:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:17 smithi134 ceph-mon[33714]: from='client.51436 v1:172.21.15.28:0/1291102110' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T15:17:18.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:17 smithi134 ceph-mon[33714]: 3.1b scrub starts 2024-03-30T15:17:18.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:17 smithi134 ceph-mon[33714]: 3.1b scrub ok 2024-03-30T15:17:18.356 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: ::ffff:172.21.15.134 - - [30/Mar/2024:15:17:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[41614]: 3.e scrub starts 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[41614]: 3.e scrub ok 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[41614]: 12.a scrub starts 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[41614]: 12.a scrub ok 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[41614]: pgmap v1535: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[29876]: 3.e scrub starts 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[29876]: 3.e scrub ok 2024-03-30T15:17:18.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[29876]: 12.a scrub starts 2024-03-30T15:17:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[29876]: 12.a scrub ok 2024-03-30T15:17:18.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:18 smithi028 ceph-mon[29876]: pgmap v1535: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-30T15:17:19.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:18 smithi134 ceph-mon[33714]: 3.e scrub starts 2024-03-30T15:17:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:18 smithi134 ceph-mon[33714]: 3.e scrub ok 2024-03-30T15:17:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:18 smithi134 ceph-mon[33714]: 12.a scrub starts 2024-03-30T15:17:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:18 smithi134 ceph-mon[33714]: 12.a scrub ok 2024-03-30T15:17:19.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:18 smithi134 ceph-mon[33714]: pgmap v1535: 174 pgs: 174 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-30T15:17:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 -- 172.21.15.28:0/1135611870 >> v1:172.21.15.28:6790/0 conn(0x7ff4ec10af10 legacy=0x7ff4ec10d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 -- 172.21.15.28:0/1135611870 shutdown_connections 2024-03-30T15:17:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 -- 172.21.15.28:0/1135611870 >> 172.21.15.28:0/1135611870 conn(0x7ff4ec0fdb10 msgr2=0x7ff4ec0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:19.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 -- 172.21.15.28:0/1135611870 shutdown_connections 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 -- 172.21.15.28:0/1135611870 wait complete. 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 librados: starting msgr at 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.653+0000 7ff4f38dd700 1 librados: starting objecter 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 Processor -- start 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 -- start start 2024-03-30T15:17:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 librados: setting wanted keys 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 librados: calling monclient init 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4ec1a9380 con 0x7ff4ec102280 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4ec1aa5a0 con 0x7ff4ec10af10 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f38dd700 1 -- --> v1:172.21.15.28:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff4ec1ab7e0 con 0x7ff4ec10eab0 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f1e7a700 1 --1- >> v1:172.21.15.28:6790/0 conn(0x7ff4ec10eab0 0x7ff4ec1a7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6790/0 says I am v1:172.21.15.28:36618/0 (socket says 172.21.15.28:36618) 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f1679700 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4ec102280 0x7ff4ec1075b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33326/0 (socket says 172.21.15.28:33326) 2024-03-30T15:17:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4f1e7a700 1 -- 172.21.15.28:0/3025038977 learned_addr learned my addr 172.21.15.28:0/3025038977 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter ms_handle_connect 0x7ff4ec102280 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter ms_handle_connect 0x7ff4ec10eab0 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter ms_handle_connect 0x7ff4ec10af10 2024-03-30T15:17:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 10 client.?.objecter resend_mon_ops 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904796533 0 0) 0x7ff4ec1a9380 con 0x7ff4ec102280 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d40031f0 con 0x7ff4ec102280 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.2 v1:172.21.15.28:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3277385887 0 0) 0x7ff4ec1ab7e0 con 0x7ff4ec10eab0 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4ec1a9380 con 0x7ff4ec10eab0 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.1 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1811700496 0 0) 0x7ff4ec1aa5a0 con 0x7ff4ec10af10 2024-03-30T15:17:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4ec1ab7e0 con 0x7ff4ec10af10 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 311569158 0 0) 0x7ff4d40031f0 con 0x7ff4ec102280 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4ec1aa5a0 con 0x7ff4ec102280 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.654+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4e4001830 con 0x7ff4ec102280 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.2 v1:172.21.15.28:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1279034257 0 0) 0x7ff4ec1a9380 con 0x7ff4ec10eab0 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4d40031f0 con 0x7ff4ec10eab0 2024-03-30T15:17:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.2 v1:172.21.15.28:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4dc001d90 con 0x7ff4ec10eab0 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2724143147 0 0) 0x7ff4ec1aa5a0 con 0x7ff4ec102280 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 >> v1:172.21.15.28:6790/0 conn(0x7ff4ec10eab0 legacy=0x7ff4ec1a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 >> v1:172.21.15.134:6789/0 conn(0x7ff4ec10af10 legacy=0x7ff4ec1a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec1aca20 con 0x7ff4ec102280 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ec1aba10 con 0x7ff4ec102280 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4f38dd700 10 client.26259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T15:17:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4ec1abed0 con 0x7ff4ec102280 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4f38dd700 1 librados: init done 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 5 ==== config(26 keys) ==== 1081+0+0 (unknown 1218034440 0 0) 0x7ff4e4003b40 con 0x7ff4ec102280 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.655+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3823158759 0 0) 0x7ff4e4005c30 con 0x7ff4ec102280 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.656+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d8005120 con 0x7ff4ec102280 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.656+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 7 ==== mgrmap(e 22) ==== 91796+0+0 (unknown 1307011771 0 0) 0x7ff4e40036a0 con 0x7ff4ec102280 2024-03-30T15:17:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 8 ==== osd_map(958..958 src has 904..958) ==== 7561+0+0 (unknown 4228670894 0 0) 0x7ff4e4089580 con 0x7ff4ec102280 2024-03-30T15:17:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 10 client.26259.objecter ms_dispatch 0x7ff4ec001550 osd_map(958..958 src has 904..958) 2024-03-30T15:17:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 3 client.26259.objecter handle_osd_map got epochs [958,958] > 0 2024-03-30T15:17:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 3 client.26259.objecter handle_osd_map decoding full epoch 958 2024-03-30T15:17:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 20 client.26259.objecter dump_active .. 0 homeless 2024-03-30T15:17:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.657+0000 7ff4ea7fc700 10 client.26259.objecter ms_handle_connect 0x7ff4d4070120 2024-03-30T15:17:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.661+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff4e40569d0 con 0x7ff4ec102280 2024-03-30T15:17:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.867+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 --> v1:172.21.15.28:6789/0 -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7ff4d8005410 con 0x7ff4ec102280 2024-03-30T15:17:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.870+0000 7ff4ea7fc700 1 -- 172.21.15.28:0/3025038977 <== mon.0 v1:172.21.15.28:6789/0 10 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v36) ==== 118+0+0 (unknown 2460078154 0 0) 0x7ff4e405a420 con 0x7ff4ec102280 2024-03-30T15:17:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.872+0000 7ff4f38dd700 10 librados: watch_flush enter 2024-03-30T15:17:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 10 librados: watch_flush exit 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 20 client.26259.objecter shutdown clearing up homeless session... 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 10 client.26259.objecter successfully canceled tick 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 >> v1:172.21.15.28:6800/3011431093 conn(0x7ff4d4070120 legacy=0x7ff4d40725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 >> v1:172.21.15.28:6789/0 conn(0x7ff4ec102280 legacy=0x7ff4ec1075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 shutdown_connections 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 >> 172.21.15.28:0/3025038977 conn(0x7ff4ec0fdb10 msgr2=0x7ff4ec111fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 shutdown_connections 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 -- 172.21.15.28:0/3025038977 wait complete. 2024-03-30T15:17:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-30T15:17:19.873+0000 7ff4f38dd700 1 librados: shutdown 2024-03-30T15:17:20.180 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 3.8 scrub starts 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 3.8 scrub ok 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 4.1 deep-scrub starts 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 4.1 deep-scrub ok 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 5.1a scrub starts 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 5.1a scrub ok 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 12.19 scrub starts 2024-03-30T15:17:20.181 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:19 smithi134 ceph-mon[33714]: 12.19 scrub ok 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 3.8 scrub starts 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 3.8 scrub ok 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 4.1 deep-scrub starts 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 4.1 deep-scrub ok 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 5.1a scrub starts 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 5.1a scrub ok 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 12.19 scrub starts 2024-03-30T15:17:20.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[29876]: 12.19 scrub ok 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 3.8 scrub starts 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 3.8 scrub ok 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 4.1 deep-scrub starts 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 4.1 deep-scrub ok 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 5.1a scrub starts 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 5.1a scrub ok 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 12.19 scrub starts 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-mon[41614]: 12.19 scrub ok 2024-03-30T15:17:20.224 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE Bus STOPPING 2024-03-30T15:17:20.225 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T15:17:20.225 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE Bus STOPPED 2024-03-30T15:17:20.225 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE Bus STARTING 2024-03-30T15:17:20.225 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE Serving on http://:::9283 2024-03-30T15:17:20.225 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:19 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:19] ENGINE Bus STARTED 2024-03-30T15:17:20.690 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T15:17:20.700 INFO:tasks.cephadm:Teardown begin 2024-03-30T15:17:20.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T15:17:20.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T15:17:20.765 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T15:17:20.765 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T15:17:20.786 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T15:17:20.823 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T15:17:20.824 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-30T15:17:20.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: 12.1c deep-scrub starts 2024-03-30T15:17:20.973 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: 12.1c deep-scrub ok 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: pgmap v1536: 174 pgs: 1 active+clean+scrubbing, 173 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: 5.19 scrub starts 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[41614]: 5.19 scrub ok 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: 12.1c deep-scrub starts 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: 12.1c deep-scrub ok 2024-03-30T15:17:20.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:17:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: pgmap v1536: 174 pgs: 1 active+clean+scrubbing, 173 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: 5.19 scrub starts 2024-03-30T15:17:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 ceph-mon[29876]: 5.19 scrub ok 2024-03-30T15:17:20.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:20 smithi028 systemd[1]: Stopping Ceph mon.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='client.? v1:172.21.15.28:0/3025038977' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='mgr.14598 v1:172.21.15.28:0/3536930181' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T15:17:21.095 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: 12.1c deep-scrub starts 2024-03-30T15:17:21.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: 12.1c deep-scrub ok 2024-03-30T15:17:21.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: from='mgr.14598 ' entity='mgr.y' 2024-03-30T15:17:21.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: pgmap v1536: 174 pgs: 1 active+clean+scrubbing, 173 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-30T15:17:21.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: 5.19 scrub starts 2024-03-30T15:17:21.096 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:20 smithi134 ceph-mon[33714]: 5.19 scrub ok 2024-03-30T15:17:21.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:21 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a[29853]: 2024-03-30T15:17:21.131+0000 7fe76c004700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:21.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:21 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a[29853]: 2024-03-30T15:17:21.131+0000 7fe76c004700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T15:17:21.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:21 smithi028 podman[325785]: 2024-03-30 15:17:21.163771366 +0000 UTC m=+0.148283340 container died b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-30T15:17:21.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:17:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug there is no tcmu-runner data available 2024-03-30T15:17:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:21 smithi028 podman[325785]: 2024-03-30 15:17:21.530479773 +0000 UTC m=+0.514991737 container cleanup b79c1a623a0b058a74e6bc14e6d33c203bb91b3c95064c8a4660b1116b63fbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-30T15:17:21.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 30 15:17:21 smithi028 bash[325785]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T15:17:21.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-30T15:17:22.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:22.001 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-30T15:17:22.001 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-30T15:17:22.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.c 2024-03-30T15:17:22.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:22 smithi028 systemd[1]: Stopping Ceph mon.c for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c[41591]: 2024-03-30T15:17:22.352+0000 7fa9e6251700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c[41591]: 2024-03-30T15:17:22.352+0000 7fa9e6251700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:22 smithi028 podman[325915]: 2024-03-30 15:17:22.548481116 +0000 UTC m=+0.325993034 container died 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE Bus STOPPING 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE Bus STOPPED 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE Bus STARTING 2024-03-30T15:17:22.724 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE Serving on http://:::9283 2024-03-30T15:17:22.725 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:22 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: [30/Mar/2024:15:17:22] ENGINE Bus STARTED 2024-03-30T15:17:23.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 podman[325915]: 2024-03-30 15:17:23.069029033 +0000 UTC m=+0.846540993 container cleanup 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:17:23.223 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 bash[325915]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c 2024-03-30T15:17:23.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-30T15:17:23.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 podman[325943]: 2024-03-30 15:17:23.396075206 +0000 UTC m=+0.851949402 container remove 0ffb0d24979a31aa8631928e2fa0a72d30cfd316b58294c6a37c17ca18db6a3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-30T15:17:23.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.c.service: Deactivated successfully. 2024-03-30T15:17:23.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 systemd[1]: Stopped Ceph mon.c for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:17:23.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Mar 30 15:17:23 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.c.service: Consumed 32.208s CPU time. 2024-03-30T15:17:23.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:23.675 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-30T15:17:23.675 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-30T15:17:23.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.b 2024-03-30T15:17:24.007 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:23 smithi134 systemd[1]: Stopping Ceph mon.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:24.306 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-b[33689]: 2024-03-30T15:17:24.005+0000 7fbe5e4f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:24.306 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-b[33689]: 2024-03-30T15:17:24.005+0000 7fbe5e4f6700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T15:17:24.306 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:24 smithi134 podman[171879]: 2024-03-30 15:17:24.078509515 +0000 UTC m=+0.204586814 container died 1e3e5360cc5947739d616968b883eebafaea0c38938d9200319311f68ad87312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-30T15:17:24.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:24 smithi134 podman[171879]: 2024-03-30 15:17:24.495782517 +0000 UTC m=+0.621859852 container cleanup 1e3e5360cc5947739d616968b883eebafaea0c38938d9200319311f68ad87312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-30T15:17:24.682 INFO:journalctl@ceph.mon.b.smithi134.stdout:Mar 30 15:17:24 smithi134 bash[171879]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mon-b 2024-03-30T15:17:24.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-30T15:17:24.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:24.955 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-30T15:17:24.955 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-30T15:17:24.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T15:17:25.269 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 systemd[1]: Stopping Ceph mgr.y for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:25.609 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T15:17:25.267+0000 7f6c4250b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:25.610 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y[30132]: 2024-03-30T15:17:25.267+0000 7f6c4250b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-30T15:17:25.610 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 podman[326057]: 2024-03-30 15:17:25.299737685 +0000 UTC m=+0.150107081 container died e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-30T15:17:25.973 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 podman[326057]: 2024-03-30 15:17:25.766254256 +0000 UTC m=+0.616623645 container cleanup e78af481317da22f3d9704caee944cf762f11c5c34571a29951eca10e9321c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e50bbcdd1630c7760b708f3a5b4d38a121e66cb, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:17:25.973 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Mar 30 15:17:25 smithi028 bash[326057]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-y 2024-03-30T15:17:26.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-30T15:17:26.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:26.272 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-30T15:17:26.272 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-30T15:17:26.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T15:17:26.561 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:26 smithi134 systemd[1]: Stopping Ceph mgr.x for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:26.852 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:26 smithi134 podman[172015]: 2024-03-30 15:17:26.591276713 +0000 UTC m=+0.135156628 container died 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:17:27.144 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:26 smithi134 podman[172015]: 2024-03-30 15:17:26.958505756 +0000 UTC m=+0.502385672 container cleanup 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T15:17:27.144 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:26 smithi134 bash[172015]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x 2024-03-30T15:17:27.394 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:27 smithi134 podman[172033]: 2024-03-30 15:17:27.143149988 +0000 UTC m=+0.547406438 container remove 7e54a4aec29127e9982dd867710a879f08a7bb6bc20befde7304b841d798cf64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-30T15:17:27.395 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Mar 30 15:17:27 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-30T15:17:27.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-30T15:17:27.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:27.439 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-30T15:17:27.439 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T15:17:27.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T15:17:27.723 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:27 smithi028 systemd[1]: Stopping Ceph osd.0 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:28.223 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:17:27.756+0000 7f1a7954f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:28.223 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:17:27.756+0000 7f1a7954f700 -1 osd.0 958 *** Got signal Terminated *** 2024-03-30T15:17:28.223 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:27 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0[302110]: 2024-03-30T15:17:27.756+0000 7f1a7954f700 -1 osd.0 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:17:33.573 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:33 smithi028 podman[326188]: 2024-03-30 15:17:33.283390361 +0000 UTC m=+5.652276773 container died 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-30T15:17:33.973 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:33 smithi028 podman[326188]: 2024-03-30 15:17:33.754333059 +0000 UTC m=+6.123219460 container cleanup 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:17:33.973 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:33 smithi028 bash[326188]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T15:17:33.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:33 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:17:33.974 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:33 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:17:34.379 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326210]: 2024-03-30 15:17:34.022760882 +0000 UTC m=+0.736941223 container remove 5d5041fb233cb5c1f8d0425cd3bc2e41f9886be59be83a656e41abc26aceae7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-30T15:17:34.633 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.281669196 +0000 UTC m=+0.018125732 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:17:34.633 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.390116654 +0000 UTC m=+0.126573178 container create 339ab2370be13f8f64234d0eb45954588324d071e48c1792a8dae8aa29e6e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-30T15:17:34.633 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.626273255 +0000 UTC m=+0.362729777 container init 339ab2370be13f8f64234d0eb45954588324d071e48c1792a8dae8aa29e6e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-30T15:17:34.633 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.632128409 +0000 UTC m=+0.368584938 container start 339ab2370be13f8f64234d0eb45954588324d071e48c1792a8dae8aa29e6e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T15:17:34.927 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.717383083 +0000 UTC m=+0.453839622 container attach 339ab2370be13f8f64234d0eb45954588324d071e48c1792a8dae8aa29e6e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-30T15:17:35.223 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 30 15:17:34 smithi028 podman[326307]: 2024-03-30 15:17:34.926587207 +0000 UTC m=+0.663043735 container died 339ab2370be13f8f64234d0eb45954588324d071e48c1792a8dae8aa29e6e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T15:17:35.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T15:17:35.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:35.675 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T15:17:35.676 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T15:17:35.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T15:17:36.012 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:35 smithi028 systemd[1]: Stopping Ceph osd.1 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:36.473 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:36 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T15:17:36.010+0000 7f46cb8bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:36.473 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:36 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T15:17:36.010+0000 7f46cb8bd700 -1 osd.1 958 *** Got signal Terminated *** 2024-03-30T15:17:36.473 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:36 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1[52376]: 2024-03-30T15:17:36.010+0000 7f46cb8bd700 -1 osd.1 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:17:41.618 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:41 smithi028 podman[326440]: 2024-03-30 15:17:41.329344553 +0000 UTC m=+5.427130433 container died 58a6408f57d552d21327919a8ccf48aa4d2212e086d0f0ed4ac5c33f44000b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-4e50bbc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-30T15:17:41.973 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:41 smithi028 podman[326440]: 2024-03-30 15:17:41.83266385 +0000 UTC m=+5.930449739 container cleanup 58a6408f57d552d21327919a8ccf48aa4d2212e086d0f0ed4ac5c33f44000b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:17:41.973 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:41 smithi028 bash[326440]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T15:17:42.386 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326464]: 2024-03-30 15:17:42.067461364 +0000 UTC m=+0.731993352 container remove 58a6408f57d552d21327919a8ccf48aa4d2212e086d0f0ed4ac5c33f44000b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-03-30T15:17:42.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.288715408 +0000 UTC m=+0.018376381 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:17:42.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.410106971 +0000 UTC m=+0.139767974 container create 024f32ef96ec7f571c60c2f52a9a4f8e0cd9fc4b49fd28ba5e36a7b31127a7b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2) 2024-03-30T15:17:42.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.629042815 +0000 UTC m=+0.358703782 container init 024f32ef96ec7f571c60c2f52a9a4f8e0cd9fc4b49fd28ba5e36a7b31127a7b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-30T15:17:42.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.634752387 +0000 UTC m=+0.364413344 container start 024f32ef96ec7f571c60c2f52a9a4f8e0cd9fc4b49fd28ba5e36a7b31127a7b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True) 2024-03-30T15:17:42.722 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.720587424 +0000 UTC m=+0.450248381 container attach 024f32ef96ec7f571c60c2f52a9a4f8e0cd9fc4b49fd28ba5e36a7b31127a7b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T15:17:42.973 INFO:journalctl@ceph.osd.1.smithi028.stdout:Mar 30 15:17:42 smithi028 podman[326563]: 2024-03-30 15:17:42.936695747 +0000 UTC m=+0.666356703 container died 024f32ef96ec7f571c60c2f52a9a4f8e0cd9fc4b49fd28ba5e36a7b31127a7b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-30T15:17:43.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T15:17:43.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:43.606 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T15:17:43.606 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T15:17:43.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T15:17:43.931 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:43 smithi028 systemd[1]: Stopping Ceph osd.2 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:43.931 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:43.576Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:17:43.931 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:43.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:17:43.931 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:43.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:17:44.223 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:17:43.929+0000 7fa9e8a49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:44.223 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:17:43.929+0000 7fa9e8a49700 -1 osd.2 958 *** Got signal Terminated *** 2024-03-30T15:17:44.223 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2[300243]: 2024-03-30T15:17:43.929+0000 7fa9e8a49700 -1 osd.2 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:17:49.572 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:49 smithi028 podman[326693]: 2024-03-30 15:17:49.245757853 +0000 UTC m=+5.415050870 container died dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:17:49.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:49 smithi028 podman[326693]: 2024-03-30 15:17:49.743721385 +0000 UTC m=+5.913014444 container cleanup dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, ceph=True) 2024-03-30T15:17:49.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:49 smithi028 bash[326693]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T15:17:50.355 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326715]: 2024-03-30 15:17:50.012043304 +0000 UTC m=+0.763943542 container remove dca7860df8718d93afe3fc342b2219272256c13d1d2f704573eea75bab8b98a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS) 2024-03-30T15:17:50.666 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.354507931 +0000 UTC m=+0.107937448 container create 7f2dc44706eb9f529ccb966d5cb8ba30f496ed340fee1c6b878a7f7b1d2d9ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:17:50.666 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.264410442 +0000 UTC m=+0.017839933 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:17:50.666 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.573254748 +0000 UTC m=+0.326684224 container init 7f2dc44706eb9f529ccb966d5cb8ba30f496ed340fee1c6b878a7f7b1d2d9ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-30T15:17:50.666 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.578862915 +0000 UTC m=+0.332292391 container start 7f2dc44706eb9f529ccb966d5cb8ba30f496ed340fee1c6b878a7f7b1d2d9ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-30T15:17:50.666 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.664707393 +0000 UTC m=+0.418136869 container attach 7f2dc44706eb9f529ccb966d5cb8ba30f496ed340fee1c6b878a7f7b1d2d9ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-30T15:17:50.973 INFO:journalctl@ceph.osd.2.smithi028.stdout:Mar 30 15:17:50 smithi028 podman[326811]: 2024-03-30 15:17:50.874419226 +0000 UTC m=+0.627848696 container died 7f2dc44706eb9f529ccb966d5cb8ba30f496ed340fee1c6b878a7f7b1d2d9ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:17:51.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T15:17:51.607 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:51.608 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T15:17:51.608 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T15:17:51.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T15:17:51.933 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:51 smithi028 systemd[1]: Stopping Ceph osd.3 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:52.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3[64832]: 2024-03-30T15:17:51.931+0000 7f765251a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:17:52.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3[64832]: 2024-03-30T15:17:51.931+0000 7f765251a700 -1 osd.3 958 *** Got signal Terminated *** 2024-03-30T15:17:52.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:51 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3[64832]: 2024-03-30T15:17:51.931+0000 7f765251a700 -1 osd.3 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:17:53.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:53 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:53.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:17:53.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:53 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:53.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:17:53.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:17:53 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:17:53.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:17:55.180 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:54 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:54.848+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:17:55.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:54 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:54.939+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:55.491 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:17:55.233Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-30T15:17:55.491 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:17:55.233Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-30T15:17:55.491 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:17:55.234Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-30T15:17:55.491 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:17:55.234Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-03-30T15:17:56.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:55.898+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:56.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:55.805+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:17:56.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:17:55 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:17:55.980+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:17:56.723 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:56 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3[64832]: 2024-03-30T15:17:56.233+0000 7f764ab06700 -1 osd.3 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.682133+0000 front 2024-03-30T15:17:32.682228+0000 (oldest deadline 2024-03-30T15:17:56.182209+0000) 2024-03-30T15:17:57.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:56.929+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:57.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:17:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:17:56.992+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:17:57.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:56 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:56.789+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:17:57.556 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:57 smithi028 podman[326942]: 2024-03-30 15:17:57.26362815 +0000 UTC m=+5.434885930 container died 5950835d28b6d6020ec97aad81d48ca4e401bf5c377dea6e877d241a59d5a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:17:57.973 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:57 smithi028 podman[326942]: 2024-03-30 15:17:57.712500069 +0000 UTC m=+5.883757852 container cleanup 5950835d28b6d6020ec97aad81d48ca4e401bf5c377dea6e877d241a59d5a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:17:57.973 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:57 smithi028 bash[326942]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T15:17:58.003 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:57 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:57.921+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:58.004 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:57 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:57.766+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:17:58.004 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:17:57 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:17:57.679+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:17:58.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:57 smithi028 podman[326964]: 2024-03-30 15:17:57.972161497 +0000 UTC m=+0.701689014 container remove 5950835d28b6d6020ec97aad81d48ca4e401bf5c377dea6e877d241a59d5a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-30T15:17:58.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:17:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:17:58.002+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:17:58.723 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327059]: 2024-03-30 15:17:58.311668546 +0000 UTC m=+0.017924669 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:17:58.723 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327059]: 2024-03-30 15:17:58.414883251 +0000 UTC m=+0.121139395 container create 452612cf7e3720f7ce589f044aa097a3b55297424207cab109b3cbe5ed60aa3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-03-30T15:17:58.723 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327059]: 2024-03-30 15:17:58.633605785 +0000 UTC m=+0.339861884 container init 452612cf7e3720f7ce589f044aa097a3b55297424207cab109b3cbe5ed60aa3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-30T15:17:58.723 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327059]: 2024-03-30 15:17:58.639561915 +0000 UTC m=+0.345818016 container start 452612cf7e3720f7ce589f044aa097a3b55297424207cab109b3cbe5ed60aa3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T15:17:58.968 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:58.903+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:58.969 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:17:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:17:58.699+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:17:58.969 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:58.752+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:17:59.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327059]: 2024-03-30 15:17:58.733674628 +0000 UTC m=+0.439930731 container attach 452612cf7e3720f7ce589f044aa097a3b55297424207cab109b3cbe5ed60aa3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-4e50bbc) 2024-03-30T15:17:59.223 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 30 15:17:58 smithi028 podman[327105]: 2024-03-30 15:17:58.9600113 +0000 UTC m=+0.018306467 container died 452612cf7e3720f7ce589f044aa097a3b55297424207cab109b3cbe5ed60aa3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-30T15:17:59.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:17:58 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:17:58.966+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:17:59.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T15:17:59.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:17:59.643 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T15:17:59.643 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T15:17:59.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T15:17:59.927 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:17:59.863+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:17:59.927 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:17:59.733+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:17:59.927 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:59 smithi134 systemd[1]: Stopping Ceph osd.4 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:17:59.927 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:59.782+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:00.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:17:59.924+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:00.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:59.962+0000 7fa967cd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:18:00.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:59.962+0000 7fa967cd4700 -1 osd.4 958 *** Got signal Terminated *** 2024-03-30T15:18:00.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:17:59 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:17:59.962+0000 7fa967cd4700 -1 osd.4 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:18:01.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:00.902+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:01.182 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:00.918+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:01.182 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:00.759+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:01.182 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:00 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:18:00.741+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:02.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:01.852+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:02.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:01.903+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:02.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:01.903+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:02.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:18:01.727+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:02.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:01 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:01.770+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:03.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:02.899+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:03.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:02.806+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:03.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:18:02.734+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:03.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:02.939+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:03.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:02 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:02.939+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:03.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:03 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:03.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:03.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:03 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:03.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:03.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:03 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:03.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:04.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:03.783+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:04.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:18:03.735+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:04.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:03.863+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:04.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:03.863+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:04.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:03.981+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:04.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:03.981+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:05.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:04.907+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:05.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:04.907+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:05.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:04.770+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:05.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:04.770+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:05.181 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4[159057]: 2024-03-30T15:18:04.760+0000 7fa9642c8700 -1 osd.4 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:30.699461+0000 front 2024-03-30T15:17:30.699427+0000 (oldest deadline 2024-03-30T15:17:54.799413+0000) 2024-03-30T15:18:05.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:04.948+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:05.182 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:04 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:04.948+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:05.546 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:05 smithi134 podman[172174]: 2024-03-30 15:18:05.294714904 +0000 UTC m=+5.455052500 container died 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-30T15:18:05.875 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:05 smithi134 podman[172174]: 2024-03-30 15:18:05.718891825 +0000 UTC m=+5.879229420 container cleanup 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-03-30T15:18:05.875 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:05 smithi134 bash[172174]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T15:18:05.875 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:05.722+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:05.875 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:05.722+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:06.180 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:05 smithi134 podman[172197]: 2024-03-30 15:18:05.929160113 +0000 UTC m=+0.636213548 container remove 7da5e611076b0805ae2d8ceb79b9a4f1c55903707047d95962d8a71617ce0d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-30T15:18:06.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:05.873+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:06.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:05.873+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:06.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:05.937+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:06.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:05 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:05.937+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:06.625 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.235845151 +0000 UTC m=+0.017672082 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:18:06.625 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.363299257 +0000 UTC m=+0.145126170 container create c6216824851179a065d8525fcfe797c048bc2f5a7ef68ff3ee0f76149a477c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:18:06.625 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.541533475 +0000 UTC m=+0.323360400 container init c6216824851179a065d8525fcfe797c048bc2f5a7ef68ff3ee0f76149a477c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-30T15:18:06.625 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.54746903 +0000 UTC m=+0.329295944 container start c6216824851179a065d8525fcfe797c048bc2f5a7ef68ff3ee0f76149a477c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-4e50bbc) 2024-03-30T15:18:06.625 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.624070241 +0000 UTC m=+0.405897159 container attach c6216824851179a065d8525fcfe797c048bc2f5a7ef68ff3ee0f76149a477c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-30T15:18:06.923 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:06.718+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:06.923 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:06.718+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:06.923 INFO:journalctl@ceph.osd.4.smithi134.stdout:Mar 30 15:18:06 smithi134 podman[172296]: 2024-03-30 15:18:06.837259102 +0000 UTC m=+0.619086019 container died c6216824851179a065d8525fcfe797c048bc2f5a7ef68ff3ee0f76149a477c2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-30T15:18:07.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:06.921+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:07.180 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:06.921+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:07.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:06.947+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:07.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:06 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:06.947+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:07.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T15:18:07.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:07.563 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T15:18:07.563 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T15:18:07.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T15:18:07.880 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:07.689+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:07.881 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:07.689+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:07.881 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 systemd[1]: Stopping Ceph osd.5 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:07.881 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:07.878+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:07.881 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:07.879+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:08.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:07.961+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:08.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:07.961+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:08.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:07.891+0000 7f91c3375700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:18:08.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:07.891+0000 7f91c3375700 -1 osd.5 958 *** Got signal Terminated *** 2024-03-30T15:18:08.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:07 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:07.891+0000 7f91c3375700 -1 osd.5 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:18:09.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:08.694+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:09.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:08.694+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:09.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:08.854+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:09.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:08.854+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:09.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:08.927+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:09.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:08 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:08.927+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:09.930 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:09.827+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:09.931 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:09.827+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:09.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:09.653+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:09.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:09.653+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:10.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:09.953+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:10.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:09 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:09.953+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:10.930 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:10.782+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:10.930 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:10.782+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:10.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:10.663+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:10.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:10.663+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:11.433 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:10.984+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:11.433 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:10.984+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:11.434 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:10 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:10.984+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:11.930 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:11.825+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:11.931 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:11.825+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:11.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:11.615+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:11.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:11.615+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:11.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:11 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:11.615+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:12.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.030+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:12.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.030+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:12.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.030+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:12.841 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:12.575+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:12.841 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:12.575+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:12.841 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:12.575+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:13.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:12.840+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:31.080631+0000 front 2024-03-30T15:17:31.080615+0000 (oldest deadline 2024-03-30T15:17:53.980364+0000) 2024-03-30T15:18:13.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:12.840+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:39.881523+0000 front 2024-03-30T15:17:39.881427+0000 (oldest deadline 2024-03-30T15:18:03.381401+0000) 2024-03-30T15:18:13.181 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5[45155]: 2024-03-30T15:18:12.840+0000 7f91bf969700 -1 osd.5 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:48.082631+0000 front 2024-03-30T15:17:48.082648+0000 (oldest deadline 2024-03-30T15:18:12.782487+0000) 2024-03-30T15:18:13.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.982+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:13.182 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.982+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:13.182 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:12 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:12.982+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:13.600 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:13 smithi134 podman[172428]: 2024-03-30 15:18:13.339295636 +0000 UTC m=+5.555284603 container died 4d12eaf5597fd004fc5b796b6ec439ee77a03bc2dd27b5e2a6305eccd73d698d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:18:13.931 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:13 smithi134 podman[172428]: 2024-03-30 15:18:13.772797161 +0000 UTC m=+5.988786171 container cleanup 4d12eaf5597fd004fc5b796b6ec439ee77a03bc2dd27b5e2a6305eccd73d698d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-30T15:18:13.931 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:13 smithi134 bash[172428]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T15:18:13.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:13.599+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:13.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:13.599+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:13.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:13 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:13.599+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:13.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:13.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:13.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:13.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:13.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:13 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:13.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:14.334 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:13 smithi134 podman[172450]: 2024-03-30 15:18:13.949034984 +0000 UTC m=+0.611101444 container remove 4d12eaf5597fd004fc5b796b6ec439ee77a03bc2dd27b5e2a6305eccd73d698d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-30T15:18:14.334 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:14.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:14.334 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:14.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:14.334 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:14.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:14.603 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.333220884 +0000 UTC m=+0.103557925 container create 2204bb8cf32ba31b89c1a27cf108c8285cd102b27f980b54c46e7b45ceec385d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:18:14.603 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.247640995 +0000 UTC m=+0.017978058 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:18:14.603 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.518695999 +0000 UTC m=+0.289033040 container init 2204bb8cf32ba31b89c1a27cf108c8285cd102b27f980b54c46e7b45ceec385d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:18:14.603 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.524056002 +0000 UTC m=+0.294393049 container start 2204bb8cf32ba31b89c1a27cf108c8285cd102b27f980b54c46e7b45ceec385d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-30T15:18:14.603 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.602123856 +0000 UTC m=+0.372460908 container attach 2204bb8cf32ba31b89c1a27cf108c8285cd102b27f980b54c46e7b45ceec385d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-30T15:18:14.931 INFO:journalctl@ceph.osd.5.smithi134.stdout:Mar 30 15:18:14 smithi134 podman[172542]: 2024-03-30 15:18:14.817524221 +0000 UTC m=+0.587861266 container died 2204bb8cf32ba31b89c1a27cf108c8285cd102b27f980b54c46e7b45ceec385d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc) 2024-03-30T15:18:14.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:14.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:14.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:14.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:14.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:14 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:14.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:15.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.015+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:15.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.015+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:15.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.015+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:15.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T15:18:15.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:15.544 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T15:18:15.544 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T15:18:15.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T15:18:15.871 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 systemd[1]: Stopping Ceph osd.6 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:15.872 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:15.581+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:15.872 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:15.581+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:15.872 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:15.581+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:16.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.869+0000 7f92afd7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:18:16.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.869+0000 7f92afd7f700 -1 osd.6 958 *** Got signal Terminated *** 2024-03-30T15:18:16.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:15 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:15.869+0000 7f92afd7f700 -1 osd.6 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:18:16.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:16.028+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:16.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:16.028+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:16.181 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:16.028+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:16.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:16.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:16.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:16.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:16.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:16 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:16.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:17.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:17.016+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:17.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:17.016+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:17.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:17.016+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:17.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:17.664+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:17.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:17.664+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:17.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:17.664+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:17.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:17 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:17.664+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:18.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:18.034+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:18.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:18.034+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:18.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:18.034+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:18.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:18.654+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:18.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:18.654+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:18.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:18.654+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:18.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:18 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:18.654+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:19.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:19.042+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:19.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:19.042+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:19.431 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:19.042+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:19.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:19.666+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:19.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:19.666+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:19.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:19.666+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:19.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:19 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:19.666+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:20.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:20.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.285751+0000 front 2024-03-30T15:17:32.285771+0000 (oldest deadline 2024-03-30T15:17:55.185612+0000) 2024-03-30T15:18:20.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:20.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:36.787371+0000 front 2024-03-30T15:17:36.787276+0000 (oldest deadline 2024-03-30T15:18:01.487237+0000) 2024-03-30T15:18:20.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6[51106]: 2024-03-30T15:18:20.029+0000 7f92a836b700 -1 osd.6 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.888620+0000 front 2024-03-30T15:17:47.888664+0000 (oldest deadline 2024-03-30T15:18:10.188518+0000) 2024-03-30T15:18:20.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:20.650+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:20.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:20.650+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:20.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:20.650+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:20.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:20 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:20.650+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:21.680 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:21 smithi134 podman[172674]: 2024-03-30 15:18:21.417405275 +0000 UTC m=+5.652754080 container died 1d4ae1455474a7bc5e03f091d2fe11af0b441e7ceecab5e1c1b864b1a7100d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-30T15:18:21.681 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:21.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:21.681 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:21.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:21.681 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:21.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:21.681 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:21 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:21.613+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:22.044 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:21 smithi134 podman[172674]: 2024-03-30 15:18:21.850243057 +0000 UTC m=+6.085591849 container cleanup 1d4ae1455474a7bc5e03f091d2fe11af0b441e7ceecab5e1c1b864b1a7100d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-30T15:18:22.044 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:21 smithi134 bash[172674]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T15:18:22.430 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172697]: 2024-03-30 15:18:22.043326697 +0000 UTC m=+0.624883839 container remove 1d4ae1455474a7bc5e03f091d2fe11af0b441e7ceecab5e1c1b864b1a7100d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-03-30T15:18:22.747 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:22.632+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:22.747 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:22.632+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:22.632+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:22 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:22.632+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.469051178 +0000 UTC m=+0.111999693 container create f2c34ff39687c864e12b4f0ff6f90beb04741d40422f150219de4abcfeaa82bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.374949361 +0000 UTC m=+0.017897895 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.680067221 +0000 UTC m=+0.323015739 container init f2c34ff39687c864e12b4f0ff6f90beb04741d40422f150219de4abcfeaa82bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.685298858 +0000 UTC m=+0.328247375 container start f2c34ff39687c864e12b4f0ff6f90beb04741d40422f150219de4abcfeaa82bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-4e50bbc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-30T15:18:22.748 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.746590654 +0000 UTC m=+0.389539174 container attach f2c34ff39687c864e12b4f0ff6f90beb04741d40422f150219de4abcfeaa82bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:18:23.180 INFO:journalctl@ceph.osd.6.smithi134.stdout:Mar 30 15:18:22 smithi134 podman[172793]: 2024-03-30 15:18:22.970390913 +0000 UTC m=+0.613339441 container died f2c34ff39687c864e12b4f0ff6f90beb04741d40422f150219de4abcfeaa82bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-30T15:18:23.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T15:18:23.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:23.724 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T15:18:23.724 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T15:18:23.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T15:18:23.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:23 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:23.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:23.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:23 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:23.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:23.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:23 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:23.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:23.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:23 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:23.628+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:23.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:23 smithi134 systemd[1]: Stopping Ceph osd.7 for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:23.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:23.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:23.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:23.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:23.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:23 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:23.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:24.430 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.055+0000 7fecbaa36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:18:24.430 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.055+0000 7fecbaa36700 -1 osd.7 958 *** Got signal Terminated *** 2024-03-30T15:18:24.430 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.055+0000 7fecbaa36700 -1 osd.7 958 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T15:18:24.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.670+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:24.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.670+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:24.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.670+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:24.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:24 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:24.670+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:25.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:25 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:25.622+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:25.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:25 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:25.622+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:25.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:25 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:25.622+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:25.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:25 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:25.622+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:26.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:26 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:26.646+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:26.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:26 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:26.646+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:26.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:26 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:26.646+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:26.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:26 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:26.646+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:27.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:27 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:27.609+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:27.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:27 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:27.609+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:27.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:27 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:27.609+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:27.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:27 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:27.609+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:28.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:28 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:28.658+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6811 osd.0 since back 2024-03-30T15:17:32.231370+0000 front 2024-03-30T15:17:32.231551+0000 (oldest deadline 2024-03-30T15:17:56.931366+0000) 2024-03-30T15:18:28.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:28 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:28.658+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-03-30T15:17:38.032400+0000 front 2024-03-30T15:17:38.032469+0000 (oldest deadline 2024-03-30T15:18:03.932292+0000) 2024-03-30T15:18:28.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:28 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:28.658+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6803 osd.2 since back 2024-03-30T15:17:47.433412+0000 front 2024-03-30T15:17:47.433361+0000 (oldest deadline 2024-03-30T15:18:11.533308+0000) 2024-03-30T15:18:28.931 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:28 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7[57422]: 2024-03-30T15:18:28.658+0000 7fecb3022700 -1 osd.7 958 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-03-30T15:17:53.834473+0000 front 2024-03-30T15:17:53.834515+0000 (oldest deadline 2024-03-30T15:18:17.334483+0000) 2024-03-30T15:18:29.845 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:29 smithi134 podman[172924]: 2024-03-30 15:18:29.46872856 +0000 UTC m=+5.517160392 container died 511070c21ad1301c7070675e5ebe50964546c31e88b956218c7121d57f70a754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-30T15:18:30.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:29 smithi134 podman[172924]: 2024-03-30 15:18:29.844552658 +0000 UTC m=+5.892984490 container cleanup 511070c21ad1301c7070675e5ebe50964546c31e88b956218c7121d57f70a754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-4e50bbc, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-30T15:18:30.180 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:29 smithi134 bash[172924]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T15:18:30.181 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[172959]: 2024-03-30 15:18:30.012603364 +0000 UTC m=+0.542484984 container remove 511070c21ad1301c7070675e5ebe50964546c31e88b956218c7121d57f70a754 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-4e50bbc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-30T15:18:30.658 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.396826961 +0000 UTC m=+0.101644715 container create a0e85a0b8529a7661854507983c5fb83fec5b477b50482fb3aa6affc6329e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-30T15:18:30.659 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.312823222 +0000 UTC m=+0.017640968 image pull b0c4478f1d83795df4d538bf108aee5fac57194348fb16dcdbb3115f3e2fe95e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d 2024-03-30T15:18:30.659 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.565891367 +0000 UTC m=+0.270709123 container init a0e85a0b8529a7661854507983c5fb83fec5b477b50482fb3aa6affc6329e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-4e50bbc, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-30T15:18:30.659 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.571540617 +0000 UTC m=+0.276358363 container start a0e85a0b8529a7661854507983c5fb83fec5b477b50482fb3aa6affc6329e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5) 2024-03-30T15:18:30.659 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.657211204 +0000 UTC m=+0.362028963 container attach a0e85a0b8529a7661854507983c5fb83fec5b477b50482fb3aa6affc6329e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-30T15:18:30.930 INFO:journalctl@ceph.osd.7.smithi134.stdout:Mar 30 15:18:30 smithi134 podman[173054]: 2024-03-30 15:18:30.860171334 +0000 UTC m=+0.564989080 container died a0e85a0b8529a7661854507983c5fb83fec5b477b50482fb3aa6affc6329e76c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-4e50bbc, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-03-30T15:18:31.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T15:18:31.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:31.575 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T15:18:31.575 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-30T15:18:31.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@rgw.foo.a 2024-03-30T15:18:31.874 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:31 smithi028 systemd[1]: Stopping Ceph rgw.foo.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:32.223 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:31 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a[70129]: 2024-03-30T15:18:31.871+0000 7fce5fef3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T15:18:32.223 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:31 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a[70129]: 2024-03-30T15:18:31.871+0000 7fce6e772ac0 -1 shutting down 2024-03-30T15:18:33.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:33 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:33.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:33.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:33 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:33.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:33.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:33 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:33.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:42.164 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:41 smithi028 bash[327191]: time="2024-03-30T15:18:41Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-30T15:18:42.164 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:41 smithi028 podman[327191]: 2024-03-30 15:18:41.895591749 +0000 UTC m=+10.123818310 container died 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-4e50bbc, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:18:42.473 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:42 smithi028 podman[327191]: 2024-03-30 15:18:42.377914494 +0000 UTC m=+10.606141094 container cleanup 145b86532a3df7e822429d347f6dbec017f247b90477de7b27e54efe24489742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-4e50bbc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-03-30T15:18:42.473 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Mar 30 15:18:42 smithi028 bash[327191]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-rgw-foo-a 2024-03-30T15:18:42.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@rgw.foo.a.service' 2024-03-30T15:18:42.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:42.858 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-30T15:18:42.858 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-30T15:18:42.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@prometheus.a 2024-03-30T15:18:43.167 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:42 smithi134 systemd[1]: Stopping Ceph prometheus.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-30T15:18:43.423 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.165Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.167Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.167Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a[69906]: ts=2024-03-30T15:18:43.167Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-30T15:18:43.424 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 podman[173184]: 2024-03-30 15:18:43.175809778 +0000 UTC m=+0.117129534 container died 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:43.680 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 podman[173184]: 2024-03-30 15:18:43.612671304 +0000 UTC m=+0.553991088 container cleanup 79037295fdfefb6a6f2a28b8ea6cea0851a61482ba1b39a13155faef25f99794 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:43.681 INFO:journalctl@ceph.prometheus.a.smithi134.stdout:Mar 30 15:18:43 smithi134 bash[173184]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-prometheus-a 2024-03-30T15:18:43.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:43.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:43.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:43.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi028.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.28:8443: connect: connection refused" 2024-03-30T15:18:43.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:43 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:43.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi134.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.134:8443: connect: connection refused" 2024-03-30T15:18:44.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@prometheus.a.service' 2024-03-30T15:18:44.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T15:18:44.086 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-30T15:18:44.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T15:18:44.287 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T15:18:45.973 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:45 smithi028 systemd[1]: Stopping Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:46.418 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a[84897]: ts=2024-03-30T15:18:46.127Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-30T15:18:46.418 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 podman[327704]: 2024-03-30 15:18:46.158528998 +0000 UTC m=+0.139716288 container died 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:46.723 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 podman[327704]: 2024-03-30 15:18:46.57509483 +0000 UTC m=+0.556282125 container cleanup 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:46.723 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 bash[327704]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a 2024-03-30T15:18:47.193 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 podman[327721]: 2024-03-30 15:18:46.776450068 +0000 UTC m=+0.613647162 container remove 7af297ac5d64ed4617ecb1022ae0ca2d43dc42b984994e3f044a24d673ad4b01 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:47.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 podman[327721]: 2024-03-30 15:18:46.852195313 +0000 UTC m=+0.689392362 volume remove 2668a34bb0e9cd1d369d70e8bf3b2bb34dd8d10c21a1f8771fa0df26fbc2e609 2024-03-30T15:18:47.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-03-30T15:18:47.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:18:47.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Mar 30 15:18:46 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@alertmanager.a.service: Consumed 1.869s CPU time. 2024-03-30T15:18:47.473 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:47 smithi028 systemd[1]: Stopping Ceph node-exporter.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:47.473 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:47 smithi028 podman[327842]: 2024-03-30 15:18:47.442677918 +0000 UTC m=+0.111003211 container died 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:47.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:47 smithi028 podman[327842]: 2024-03-30 15:18:47.87098628 +0000 UTC m=+0.539311609 container cleanup 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:47.973 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:47 smithi028 bash[327842]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a 2024-03-30T15:18:48.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:48 smithi028 podman[327861]: 2024-03-30 15:18:48.064278855 +0000 UTC m=+0.622697189 container remove 8a056a7641492a707179f881be4e2c8456fd6a5e2efa444dc50f8fe38aa08d9b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-30T15:18:48.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:48 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-30T15:18:48.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:48 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-30T15:18:48.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:48 smithi028 systemd[1]: Stopped Ceph node-exporter.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:18:48.438 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Mar 30 15:18:48 smithi028 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.a.service: Consumed 4.071s CPU time. 2024-03-30T15:18:48.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T15:18:49.093 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T15:18:50.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:18:50 smithi134 systemd[1]: Stopping Ceph iscsi.iscsi.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:18:50.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:18:50 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a[62662]: debug Shutdown received 2024-03-30T15:19:00.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:00 smithi134 bash[173610]: time="2024-03-30T15:19:00Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-30T15:19:00.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:00 smithi134 podman[173610]: 2024-03-30 15:19:00.561955653 +0000 UTC m=+10.115277766 container died 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-4e50bbc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-30T15:19:01.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 podman[173610]: 2024-03-30 15:19:01.023646673 +0000 UTC m=+10.576968791 container cleanup 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-30T15:19:01.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 bash[173610]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a 2024-03-30T15:19:01.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 podman[173636]: 2024-03-30 15:19:01.308675469 +0000 UTC m=+0.744267686 container remove 289875051a2b42dd33ca9a8c5883ee5d5a810fcbb93c0ad990278bb9809960f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8f3809717761e681477c3ef1d8d2c03114aee2c597cd7380b93b8c10f47fcb5d, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-4e50bbc, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-30T15:19:01.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-30T15:19:01.784 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-30T15:19:01.784 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 systemd[1]: Stopped Ceph iscsi.iscsi.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:19:01.784 INFO:journalctl@ceph.iscsi.iscsi.a.smithi134.stdout:Mar 30 15:19:01 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@iscsi.iscsi.a.service: Consumed 3.667s CPU time. 2024-03-30T15:19:03.144 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:02 smithi134 systemd[1]: Stopping Ceph grafana.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:19:03.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=server t=2024-03-30T15:19:03.143286505Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-30T15:19:03.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=ticker t=2024-03-30T15:19:03.143641787Z level=info msg=stopped last_tick=2024-03-30T15:19:00Z 2024-03-30T15:19:03.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a[66535]: logger=tracing t=2024-03-30T15:19:03.143646201Z level=info msg="Closing tracing" 2024-03-30T15:19:03.400 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 podman[173922]: 2024-03-30 15:19:03.153710493 +0000 UTC m=+0.104383526 container died 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T15:19:03.658 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 podman[173922]: 2024-03-30 15:19:03.47336917 +0000 UTC m=+0.424042205 container cleanup 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T15:19:03.659 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 bash[173922]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a 2024-03-30T15:19:03.931 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 podman[173941]: 2024-03-30 15:19:03.658200961 +0000 UTC m=+0.506404689 container remove 4876f7c82d6f816addaa235204152e11dccb0c417bc00a3c0ee7e294f330bceb (image=quay.io/ceph/grafana:9.4.12, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-grafana-a, maintainer=Grafana Labs ) 2024-03-30T15:19:03.931 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-03-30T15:19:03.931 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 systemd[1]: Stopped Ceph grafana.a for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:19:03.931 INFO:journalctl@ceph.grafana.a.smithi134.stdout:Mar 30 15:19:03 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@grafana.a.service: Consumed 14.746s CPU time. 2024-03-30T15:19:04.334 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 systemd[1]: Stopping Ceph node-exporter.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4... 2024-03-30T15:19:04.612 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 podman[174059]: 2024-03-30 15:19:04.333089464 +0000 UTC m=+0.119423034 container died 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T15:19:04.930 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 podman[174059]: 2024-03-30 15:19:04.710994659 +0000 UTC m=+0.497328270 container cleanup 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T15:19:04.930 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 bash[174059]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b 2024-03-30T15:19:05.195 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 podman[174078]: 2024-03-30 15:19:04.936903301 +0000 UTC m=+0.607149524 container remove 0f12b321d03c0c98b75351ab103a076bfc5e475b0155565ef25625ecbcd9b57a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-30T15:19:05.195 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:04 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-30T15:19:05.195 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:05 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-30T15:19:05.195 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:05 smithi134 systemd[1]: Stopped Ceph node-exporter.b for 8345fd88-eea4-11ee-b647-cb9ed24678a4. 2024-03-30T15:19:05.195 INFO:journalctl@ceph.node-exporter.b.smithi134.stdout:Mar 30 15:19:05 smithi134 systemd[1]: ceph-8345fd88-eea4-11ee-b647-cb9ed24678a4@node-exporter.b.service: Consumed 3.808s CPU time. 2024-03-30T15:19:05.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T15:19:05.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T15:19:05.727 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T15:19:05.729 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi028/crash 2024-03-30T15:19:05.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T15:19:05.791 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T15:19:05.791 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-03-30T15:19:05.793 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi134/crash 2024-03-30T15:19:05.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T15:19:05.831 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T15:19:05.832 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-03-30T15:19:05.832 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T15:19:05.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-30T15:19:05.866 INFO:teuthology.orchestra.run.smithi028.stdout:2024-03-30T15:00:00.000177+0000 mon.a (mon.0) 1711 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:19:05.866 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-30T15:19:05.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[SEC\]' /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-30T15:19:05.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]' /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-30T15:19:06.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[WRN\]' /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-03-30T15:19:06.081 INFO:teuthology.orchestra.run.smithi028.stdout:2024-03-30T15:00:00.000177+0000 mon.a (mon.0) 1711 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-30T15:19:06.081 INFO:tasks.cephadm:Compressing logs... 2024-03-30T15:19:06.081 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T15:19:06.124 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-03-30T15:19:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log 2024-03-30T15:19:06.153 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-30T15:19:06.153 INFO:teuthology.orchestra.run.smithi134.stderr:: No such file or directory 2024-03-30T15:19:06.153 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-03-30T15:19:06.153 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-03-30T15:19:06.153 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-03-30T15:19:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-03-30T15:19:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-03-30T15:19:06.154 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-03-30T15:19:06.157 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log 2024-03-30T15:19:06.157 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-03-30T15:19:06.157 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.4.log: /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-03-30T15:19:06.157 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-03-30T15:19:06.157 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/tcmu-runner.log 2024-03-30T15:19:06.159 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: 94.8% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log 91.7% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/tcmu-runner.log: 67.6% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/tcmu-runner.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr: 81.5% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T15:19:06.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-03-30T15:19:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-03-30T15:19:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.0.log: 94.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-30T15:19:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log 2024-03-30T15:19:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-03-30T15:19:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-03-30T15:19:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log: 94.9% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-03-30T15:19:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-03-30T15:19:06.179 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 85.8% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T15:19:06.180 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T15:19:06.188 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-03-30T15:19:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log: 91.8% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-03-30T15:19:06.195 INFO:teuthology.orchestra.run.smithi028.stderr: 94.8% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/objectstore_tool.7.log.gz 2024-03-30T15:19:06.204 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 90.8% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-03-30T15:19:06.414 INFO:teuthology.orchestra.run.smithi028.stderr: 93.5% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-client.rgw.foo.a.log.gz 2024-03-30T15:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-03-30T15:19:07.628 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-03-30T15:19:08.059 INFO:teuthology.orchestra.run.smithi028.stderr: 91.5% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-03-30T15:19:09.526 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-03-30T15:19:13.719 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-03-30T15:19:14.284 INFO:teuthology.orchestra.run.smithi028.stderr: 94.0% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-03-30T15:19:14.628 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-03-30T15:19:14.773 INFO:teuthology.orchestra.run.smithi028.stderr: 94.0% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-03-30T15:19:15.087 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-03-30T15:19:15.128 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-03-30T15:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-03-30T15:19:15.447 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-30T15:19:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m9.311s 2024-03-30T15:19:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m38.310s 2024-03-30T15:19:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m1.354s 2024-03-30T15:19:15.797 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/8345fd88-eea4-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-03-30T15:19:15.799 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-30T15:19:15.799 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m9.661s 2024-03-30T15:19:15.799 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m36.171s 2024-03-30T15:19:15.799 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m1.356s 2024-03-30T15:19:15.799 INFO:tasks.cephadm:Archiving logs... 2024-03-30T15:19:15.800 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi028/log 2024-03-30T15:19:15.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T15:19:21.024 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi134/log 2024-03-30T15:19:21.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T15:19:26.407 INFO:tasks.cephadm:Removing cluster... 2024-03-30T15:19:26.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 --force 2024-03-30T15:19:26.606 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T15:19:26.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8345fd88-eea4-11ee-b647-cb9ed24678a4 --force 2024-03-30T15:19:27.093 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 8345fd88-eea4-11ee-b647-cb9ed24678a4 2024-03-30T15:19:27.398 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T15:19:27.398 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T15:19:27.414 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T15:19:27.429 INFO:tasks.cephadm:Teardown complete 2024-03-30T15:19:27.429 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T15:19:27.438 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T15:19:27.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T15:19:27.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T15:19:27.539 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T15:19:27.548 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T15:19:27.548 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T15:19:27.551 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T15:19:27.563 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-30T15:19:27.567 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T15:19:27.567 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-30T15:19:27.567 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 13 -155us[ -170us] +/- 74ms 2024-03-30T15:19:27.568 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 +127us[ +111us] +/- 72ms 2024-03-30T15:19:27.568 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +41us[ +25us] +/- 64ms 2024-03-30T15:19:27.568 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T15:19:27.568 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 -180us[ -180us] +/- 74ms 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 7 377 77 +159us[ +146us] +/- 71ms 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +40us[ +40us] +/- 64ms 2024-03-30T15:19:27.571 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T15:19:27.572 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T15:19:27.580 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T15:19:27.581 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T15:19:27.609 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T15:19:27.695 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T15:19:27.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T15:19:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:19:27.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-30T15:19:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:19:27.808 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T15:19:27.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T15:19:27.843 INFO:teuthology.task.internal:Duration was 2286.361259 seconds 2024-03-30T15:19:27.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T15:19:27.875 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T15:19:27.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T15:19:27.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T15:19:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T15:19:27.921 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T15:19:28.260 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T15:19:28.260 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-03-30T15:19:28.261 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T15:19:28.320 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-03-30T15:19:28.320 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T15:19:28.340 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T15:19:28.340 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T15:19:28.363 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T15:19:28.409 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T15:19:28.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T15:19:28.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T15:19:31.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T15:19:31.487 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T15:19:31.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T15:19:31.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T15:19:31.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T15:19:31.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T15:19:31.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T15:19:31.581 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-03-30T15:19:31.616 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-03-30T15:19:31.637 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T15:19:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:19:31.651 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T15:19:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T15:19:31.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T15:19:31.702 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T15:19:31.703 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi028 2024-03-30T15:19:31.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T15:19:31.815 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207/remote/smithi134 2024-03-30T15:19:31.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T15:19:31.860 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T15:19:31.860 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T15:19:31.863 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T15:19:31.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T15:19:31.921 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T15:19:31.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T15:19:32.007 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T15:19:32.007 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T15:19:32.010 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T15:19:32.022 INFO:teuthology.orchestra.run.smithi028.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 15:19 /home/ubuntu/cephtest 2024-03-30T15:19:32.025 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 15:19 /home/ubuntu/cephtest 2024-03-30T15:19:32.026 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T15:19:32.044 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T15:19:32.111 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T15:19:32.138 DEBUG:teuthology.nuke:shortname: smithi028 2024-03-30T15:19:32.138 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T15:19:32.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:29:15.400215', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVqhWiaFRuPQmVUMJ9krG0HCU4jtyqrP3sL5xmlpkxyQzDhtnAtOPugxh6qmPPrimk3UnLRB2/rJn6khSFzfgA='} 2024-03-30T15:19:32.164 DEBUG:teuthology.nuke:shortname: smithi134 2024-03-30T15:19:32.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T15:19:32.188 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-29_21:08:02-orch-squid-distro-default-smithi/7630207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 14:29:15.401955', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/zvpFpaQZcES9AAcTpvqAyiJ1CSm//iJurNQLeN7X0z0W/XK7ZGZaUVVslamF7yXjfMiSSjvzuOYsQFs5Bqjw='} 2024-03-30T15:19:32.205 INFO:teuthology.orchestra.console.smithi028:Power off 2024-03-30T15:19:32.205 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T15:19:32.232 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-03-30T15:19:32.232 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:32.247 INFO:teuthology.orchestra.console.smithi134:Power off 2024-03-30T15:19:32.248 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T15:19:32.257 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-30T15:19:32.268 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-03-30T15:19:32.268 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:32.295 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-30T15:19:36.258 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:36.296 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:36.368 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-30T15:19:36.406 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-30T15:19:40.372 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:40.408 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T15:19:40.484 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-03-30T15:19:40.518 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-03-30T15:19:40.586 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-03-30T15:19:40.619 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-03-30T15:19:40.723 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 2286.3612587451935 failure_reason: '"2024-03-30T15:00:00.000177+0000 mon.a (mon.0) 1711 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-30T15:19:40.724 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T15:19:40.817 INFO:teuthology.run:FAIL